Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The paper proposed using RL (specifically PPO algorithm) to generate learned learning rate schedules for SGD. It demonstrated faster convergence compared with fixed learning rate scheduler on image classification tasks.  
strengths_and_weaknesses: ## Strength:
1. Interesting application of RL on an important problem of learning rate scheduler. 
2. Demonstrated faster convergence on MNIST and CIFAR-100 datasets. 
## Weaknesses:
1. Baseline of SLS is a bit weak. It would be interesting to see compare it with some other learning rate schedulers. Only the final loss is not intuitive. 
2. The number of epochs is relatively small for all the experiments. 
3. The tasks are not performed on larger datasets. Also, it's unclear how different batch sizes will affect the conclusion/observations.
4. It's unclear whether the problem has an MDP structure since the state and action spaces are not defined. It will be better to demonstrate the parameters and network architectures chosen for PPO as well. 


suggested_changes: ## Suggested changes
1. Make some ablation studies on the reward, and state space formulation. 
2. It is important to show the formulation of MDP on this problem and show how your choices generalize to different tasks and would be reproducible in different settings (e.g. batch sizes). 
3. Clearly state the gain on wall-clock time for training to a certain accuracy (time-to-accuracy), and final test accuracy,  and show more epochs on larger datasets. These metrics will make the contribution much clearer.
4. Generate some other learning rate schedules with different algorithms other than PPO (maybe TRPO or A3C), what kind of results do you see?
5. On the reward formulation, it's not necessary to use two weighting parameters. It would be interesting to show some results by changing these parameters.
review_title: null

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: The paper proposes training a RL agent to decide the learning rate schedule of an optimizer. The agent receives a reward for quickly lowering the validation loss while being stable. They compare the proposed method against prior fixed schedules for learning rates such as one cycle LR.
strengths_and_weaknesses: Strengths
- The problem is of relevance to the community. RL does present a potential way to tune the learning rate of the optimizer for better performance. 
- The paper shows promise as an initial step for better learning rate schedulers since they perform better than OneCycleLR on the MNIST classification task.

Weaknesses
- See suggested changes. 
suggested_changes: ## Major changes
- Table 1 shows that SGD without any learning rate schedule does better than the proposed method and OneCycleLR. So it is difficult to justify using RL for this task. One way you can justify it is if the grid search for LRs required more compute than the RL training. However the authors have not mentioned this. This would strengthen their claims.
- The authors should compare the computational cost of the proposed methods. RL can be sample inefficient, and in this setting one episode is one training run. So the time to train the RL agent can be so large than naive grid search is a much better solution. 
- In a similar vein, the authors have not mentioned much about the training details for the RL agent (for eg, final number of training steps for the results, reward obtained at the end of training). I would like to see these details in the paper, either in the main text or the appendix.
- The authors have said that the proposed method is sensitive to the RL agent hyperparameters, but have not included the final hyperparameters used for training. This can make it difficult to reproduce their results.
- Only one seed/run has been presented in the paper. It would be better to run multiple seeds (for eg. 5 seeds) and present the result with error bars. 
- For the comparison with SRS, it is difficult to discern the difference in final validation loss between the RL agent and SRS from the plot. Presenting it in a table would aid the presentation. 

## Minor Concerns
- The citation style in the text seems to be incorrect. The cited work should appear in parenthesis, for example. (Metz et al 2019), unless it is used as a noun in the sentence.
review_title: null

Meta-review: metareview: The author proposes a RL-based scheduler to determine the optimal learning rate. However, this work is not novel and not solid. Also, it misses theoretical proof. Also, the presentation is poor.
confidence: 5: The area chair is absolutely certain
Decision: Invite to revise