Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper showcases an analysis of Hyperparameter Optimization for Automated Reinforcement Learning. Within the analysis of this process, the authors showcased the need for dynamic configurations for hyperparameters in AutoRL. Using IGPR and discretized modality plots showcased the dynamic hyperparameter landscape for DQN and SAC.

overall_review: Strengths:

The authors introduce a new pipeline to showcase hyperparameter landscapes for dynamic hyperparameters with discrete training steps, or more specifically for AutoRL problems. The paper also showcases definite proof for the use of dynamic hyperparameter configurations for AutoRL algorithms. 

Weaknesses:

One weakness is the lack of analysis of the dynamic hyperparameter landscapes. The follow-on shown in the paper for this behavior was that this proves that dynamic hyperparameter landscapes should be prioritized, not how the hyperparameters change or how they can be predicted. This more important aspect to the field here is the characterization of the hyperparameter landscape. Discussing potential ways to analyze these landscapes would be beneficial. Another potential way to provide more information is to run more experiments to show different hyperparameter landscapes. 
One other potential weakness in this paper is the use of Sobol sampling. Grid sampling can be non-optimal, but in a landscape analysis study, grid search can provide a good baseline to study the complexities of the landscape.

review_summary: This paper conducted several experiments into the hyperparameter landscape of AutoRL algorithms in order to prove the need for dynamic HP configurations, and to identify a novel way to track and analyze these HP landscapes. While interesting, this paper lacks some substantial contribution.
actions_required_to_increase_overall_recommendation: My main suggestion is to provide more information on these hyperparameter landscapes within this paper. Specifically by plotting more scenarios with different algorithms. The use of a small proof for the need for dynamic hyperparameter configurations is interesting, but not impactful enough to warrant acceptance.

Page 8, figure 6 description: Disretized -> Discretized
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper performs an empirical investigation of hyperparameter loss landscapes for two RL tasks (Cartpole and the Hopper environment from MuJoCo) using two algorithms (DQN and SAC). The authors focus on the dependence of each method/environment on hyperparameters at different points during training, motivated by prior empirical results showing the dynamically modifying hyperparameters during training can be beneficial compared to using static hyperparameters.

The authors visualized the hyperparameter loss landscapes at iterations 50k, 100k, and 150k for DQN, and at iterations 125k, 250k, 375k, and 500k for SAC. They found that the landscapes vary significantly at these points in training, supporting the notion of using hyperparameter schedules. They also plotted a measure of multimodality of the returns produced by any given hyperparameter configuration, across the hyperparameter space, and found that configurations become more multimodal as training progresses.
overall_review: **Pros**

* The paper studies an interesting problem, trying to understand the impact of hyperparameters over the course of training for RL tasks. Insights into the effects of these hyperparameters may be useful to understand why practical approaches that dynamically update the hyperparameters work well.

* The results showing that the hyperparameter loss landscapes change substantially during training support the idea that hyperparameter schedules are important in RL.


**Cons**

* Although I appreciate that running a sweep over different hyperparameters is not cheap, the empirical evaluation is still fairly limited, only investigating two algorithms (DQN and SAC) on two tasks (Cartpole and Hopper). These tasks are small-scale, so it is not clear whether the hyperparameter loss surfaces for larger-scale tasks would follow a similar pattern (e.g., perhaps the trends would be different, or they could be more or less pronounced).

* The paper does not provide practical takeaways or actionable recommendations about which schedules to use in practical situations.

* Overall, the clarity of the presentation could be improved. Figure 1 is not very well-designed and does not add much to the paper. The reader can only guess that the symbols in the "data collection" box are related to the data collection procedure from Figure 2 after having read several pages further in the paper. Also, Figure 1 seems too generic to be useful, because the steps of 1) collecting data for a performance dataset, 2) using that data to construct hyperparameter landscapes; and 3) analyzing that landscape, are very broad, and not specific to RL; they could just as well be applied to arbitrary ML tasks. From Figure 1, it is not clear what the "search space" on the left-hand side refers to. In the caption of Figure 1, it is not clear what "modeled with landscape functions" mean?

* The organization of the writing is awkward. There is a lot of space devoted to explaining the basics of RL (Section 3.1), that could likely be summarized in one paragraph. At the same time, important details like the sampling procedure for hyperparameters are not expanded on clearly. The paper could use another pass over the writing to reduce wordiness and improve clarity.

* In Section 4, it states that "we imitate an optimal AutoRL optimizer that would always go for the best possible hyperparameter configuration schedule." The way this is described, it seems like a chicken-and-egg problem: how does one know what the best hyperparameter configuration schedule is without first knowing how the hyperparameter loss landscapes look under different conditions?

* Why are the hyperparameter configurations sampled using the complicated scrambled Sobol sampling strategy? As this paper only considers three hyperparameters for each algorithm, it should be possible to run a simple grid search, which would be more interpretable.

* Figure 2 is hard to parse, and not clearly described in the main text. In addition, the caption is completely uninformative. The font size is too small, and the arrows between blocks are not easy to interpret (owing to the lack of expanation in the caption). Why does it look like Step 5 in Figure 2 is determining the best configuration based on $\mathcal{D}\_{ls(1)}$? In the "Training and Evaluation" paragraph, it states that policies are selected based on their final performance, implying that in each stage the selection is based on $\mathcal{D}\_{\text{final}(i)}$, not $\mathcal{D}\_{ls(i)}$.

* It is not made clear what the "middle surfaces" are in Figure 3.

* What do the white triangles denote in Figures 3 and 4? Are these the hyperparameters that were evaluated? They appear to be unevenly distributed in the space, and do not seem representative enough to generate informative contours. Because there are few hyperparameters here, would a simple grid search be applicable? This would yield an even distribution of points across the range that is plotted, and I think may be better suited for such contour plots.

* Why are 3 timesteps used for DQN, while 4 are used for SAC? How were the timesteps chosen?

* I think that the last paragraph of Section 5.2 (starting at line 269) is unnecessary; it only makes vague statements.

* In Section 4.2, modality is not explained clearly enough. Does multimodality here refer to the distribution of performance values obtaiend when using the same hyperparameters with different random seeds? Also, the "folding test of unimodality" should be explained further here, as this is central to the quantitative results.

* The submission did not provide source code.


**Minor**

* L121: "Thus, An" --> "Thus, an"

* Eq. 3 can easily fit on one line.

* In Eq. 3, I don't think that $\mathcal{J}$ or $\Pi$ are defined anywhere.

* The first two sentences in Section 3.3 are redundant. In addition, the rest of Section 3.3 is wordy and seems overly drawn out.

* In Section 4.1, it is not stated what the $ls$ in $t_{ls}$ stands for.

* In Section 3.3, I think $\pi_Z$ needs to be defined (e.g., as the policy that results from applying algorithm $Z$).

* L49: "The goal of AutoRL is to solve to facilitate" --> "The goal of AutoRL is to facilitate"

* L21: "As the research" --> "As research"

* L171: "we consider s set" --> "we consider a set"

* L188: "then initial the configuration" --> should this be "then initialize the configuration"?

* L205: "of the Sobol' sampler" --> "of the Sobol sampler"

* L243: "train them on" --> "train them with"

* L246: Typo, "10000" should be "100,000"

* L287: "further analyses is" --> "further analyses are"

* L303: "factore" --> "factor"

* L305: "being in contrast" --> "in contrast"
review_summary: Overall, this paper presents a fairly straightforward, visualization-centric, empirical investigation into hyperparameter loss landscapes for RL algorithms. The authors do not provide many takeaways from the results, other than that they support the idea of using hyperparameter schedules. There are no takeaways regarding what types of schedules would be useful, and what properties of the tasks would affect the optimal schedules. The experiments are limited, as they focus on two small-scale RL tasks, which may not be representative of larger, real-world problems. I do not think that the contribution of the paper is substantial enough to meet the bar for acceptance.
actions_required_to_increase_overall_recommendation: I think that the writing and results need to be presented more clearly, and the diagrams (e.g., Figures 1 and 2) should be revised to improve clarity. I would encourage the authors to provide more analysis on larger-scale RL tasks and investigate which types of hyperparameter schedules are best for a given problem setting, to provide more practical takeaways.
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless paper with major impact and strong evaluation, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: The paper focuses on the study of dynamic landscapes of hyperparameters during the training of RL algorithms. It focuses on DQN and SAC as representative DeepRL algorithms, and Cartpole and Hopper-v3 as the target environment for the study.  
Paper introduces a pipeline for methodical collection of experiences with the hyperparameters during the "phases" of DeepRL training. The results are then visualized using IGPR and modality plots primarily in the paper and ILM, ICE plots shared in the appendix. 
The paper highlights the change in optimality peaks across the phases and provides practical recommendation for HPO for DeepRL pipelines.  
overall_review: The paper provides useful insights into the landscape for HPO for DeepRL algorithms. Clearly defined experiments and shared results are valuable for the DeepRL community and will provide more concrete steps toward the realization of AutoRL. 
The minor limitation of the study is in terms of the number and characteristics of the environments. It will be interesting to see if the results shared in the paper hold for more complicated and diverse environments (eg. discrete environment - Atari etc).
review_summary: Great paper with fundamental (better understanding of the evolution of HPO landscape during training) and practical (a helpful guide to plan HPO for DeepRL algorithms) impact.
actions_required_to_increase_overall_recommendation: Further diversification of the environments under study.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper provides an empirical analysis of the hyperparameter landscapes for AutoRL. The authors collect a dataset using DQN and SAC on two RL environments that comprises trajectories collected from various hyperparameter and seed combinations, and divide the RL training into phases so as to study the temporal variation of the landscapes. From these empirical results, the authors conclude that the hyperparameter landscapes vary strongly over time, which is consistent with the assumptions that various AutoRL algorithms already depend on, but such phenomenon also questions the applicability of the multi-fidelity approaches in the context of RL. The finding that many hyperparameter landscapes are highly multi-modal also further hints the complexity of the AutoRL problems.
overall_review: ## Strengths
- The paper addresses an important problem in AutoRL, and fills the gap of providing empirical support for an assumption widely used in the literature, and as such it is definitely relevant to the AutoRL/ML audience for this conference.
- The methods presented are technically sound, and the findings are in general well-supported and could be useful for future AutoRL works.

## Weaknesses
- On the flip side of Strengths point 1, the paper largely stops at only confirming the assumptions held by the community but did not go further. As I mentioned in `Potential Impact`, I do expect an analysis paper like this to provide further insights especially on their implications on the existing AutoRL practices and suggestions for future AutoRL directions. I feel this is particularly true given that the analysis techniques used in this paper are not novel in their own right, and thus the resulting analyses and insights should form the bulk of this paper's contributions.
review_summary: My initial recommendation is a borderline reject: while the paper shows promise and is largely technically sound, the suggested revisions may be material, and thus the paper would benefit from another round of reviewing.

I will reconsider the score based on the author feedback and the other reviews.
actions_required_to_increase_overall_recommendation: Please answer my questions and address my comments especially in the `Impacts` section.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The article tries to improve the impact of hyperparameters on Reinforcement Learning (RL) via Automated RL (AutoRL).
The article proposes an approach to build and analyze hyperparameter landscapes at multiple points in time throughout training, showing strong variations over time for different RL algorithms in various environments. This supports the theory that hyperparameters should be dynamically adjusted during training and provides the potential for more insights on AutoRL problems through landscape analyses.
overall_review: In general, I think this is an interesting paper, with clear methods and concrete results.
I hope the authors could address my questions, especially how the RL community would concretely benefit from this landscape analysis.
review_summary: Based on my understanding and questions, I would rate weak acceptance.
actions_required_to_increase_overall_recommendation: I am willing to adjust my score based on the authors' responses if they could properly address my questions above.
review_title: null

Decision: Accept(Main Track)