Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents an approach to distill several "teacher" optimizer into a "student" optimizer through learning to optimize (L2O). They compared three different approaches for doing such distillation, proposed to use random and adversarial perturbation to help with stability, and compared with the analytical optimizers that it distills from and the L2O baselines and showed superior performance.

main_review: Strength:

1. The idea of distilling from multiple "teacher" optimizer is novel although distilling from one "teacher" optimizer has been proposed before as imitation learning in L2O. 

2. The three mechanisms proposed to distill from several "teachers" are interesting, especially the optimal choice amalgamation, which could shed light some further improvements for L2O.

3. The use of random and adversarial pertubation on weights to improve robustness is a nice technique to help with the stability that is a big issue in L2O. 

Weakness:

1. The comparison of the amalgamated optimizer with the analytical optimizers are interesting, but the advantage of the analytical optimizers is that they tend to work reasonably well for a wide range of problem or model sizes. Since the amalgamated optimizer is still learned on specific settings, I wonder how far it could generalize and when would such generalization break? For example, if the model size or the training steps is 10X or 100X larger, would the learned optimizer still perform well? These could shed some light on the limitation of current approaches and inspire future works. 

2. The "oracle optimizer" is using the best validation loss among the analytical optimizer's in the pool, but a stronger oracle might be the optimal choice optimizer that uses the trained LSTM to pick which optimizer to use at each step. It would help to add this stronger baseline into the analysis and comparison in the experiments since it can help understand whether the bottleneck is in the optimal choice LSTM or the distillation process. 

3. It might help to add more details about the Optimal Choice Amalgamation to help with reproducibility. For example, "the LSTM takes the outputs of each optimizer in the pool, the layer type, and time step as inputs", I wish there are more descriptions (maybe in the supplementary material) about the details such as how the inputs are encoded and how often are the LSTM updated to help with reproducibility. It would also be very helpful if the code can open sourced. 

4. It would be helpful to add some experiments to justify that it is necessary to distill from multiple optimizers instead of just one since the "small" setting just uses two optimizers anyway and the gain of the proposed approach from baseline might come more from the better stability in training due to perturbation. 

summary_of_the_review: This work presents a novel approach called optimizer amalgamation that distills multiple "teacher optimizer" into a learned "student" optimizer. They compared three different approaches and proposed to use perturbation to improve stability in L2O. The result is promising compared to previous baselines, but it would be helpful if more analyses (see weakness section) can be done to understand the importance and necessity of different components and more details of the proposed approach can be released to help with reproducibility.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses the problem of selecting a neural network optimizer from a pool of possible optimizers. They propose three variants of a meta-algorithm which combines optimizers from the pool, whereby a differentiable meta-loss is defined on the training loss achieved by selection protocol. They show that their algorithm, equipped with weight space training noise leads to better performance on a variety of problems.
main_review: Strengths 
* The problem is well motivated as a solution to choosing from the large pool of possible optimizers. It naturally follows from prior work on knowledge distillation and amalgamation to instead distill from multiple optimizers. 
* Empirically, optimizer amalgamation performs better (and sometimes significantly so) than baselines from learning to optimize. It also performs favorably to the best choice of analytic optimizer.
* Paper is clearly written and easy to follow.

Weaknesses
* It is a bit unclear how well an amalgamated optimizer can generalize to various settings -- do we need to amalgamate for each specific problem?
* I have some concerns with the experiments discussed in the questions below.
* Overall, the experiments are conducted in relatively small settings. It would be useful to know whether an optimizer learned with amalgamation could be used in a large experiments. Such experiments do not need to be extensive, but would improve the significance of the paper.

Questions
1. It would be interesting to explore further the choice of optimizer pool and its effects on performance. This can be done by choosing subsets of the six optimizers you consider. SGD and Adam would be especially interesting given their popularity.
2. In the case of mean amalgamation, do you observe the phenomena of the optimizer sticking to one of the teachers?
3. Can the weight space perturbations be applied to the baselines to improve their stability?

Minor
* reference under knowledge distillation is undefined
summary_of_the_review: I think this paper is marginally above the acceptance threshold. The algorithm proposed is well motivated for a specific task, and the authors conduct experiments with various ablations to their methods. I have some concerns listed in the questions, but overall believe this to be a good first step in tackling the problem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new optimizer amalgamation method to combine a pool of optimizers into one in order to achieve stronger problem-specific performance. Three differentiable amalgamation mechanisms are designed and stabilization methods are explored. The proposed method is empirically shown to be effective when compared to a large number of baselines. 


main_review: Strengths
=

1. The paper is generally well-written and easy to follow. 

2. The proposed optimizer amalgamation method is interesting and effective. 

3. The experiment is comprehensive and supports the main claim of the paper. 



Weakness
=
Selecting an appropriate optimizer for a given problem is not a new research topic, and many papers exist in the areas of algorithm selection, algorithm portfolio, and meta-learning. I think one weakness of the paper is that it lacks the discussion about the related work, and how the paper can be better positioned in the literature. 

summary_of_the_review: Although the research topic is old, the proposed optimizer amalgamation method looks interesting and the experimental results verified the effectiveness of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new problem called Optimizer Amalgamation and made an attempt to obtain a more powerful learned optimizer from several analytical optimizers. More concretely, three amalgamation losses are designed to train the amalgamated optimizer. In the meanwhile, two types of noise, random gaussian perturbation and projected gradient descent are incorporated into the training objective to increase the stability of the optimizer. The evaluation part compared the amalgamated optimizer with its original teachers, i.e., those analytical optimizers, and some L2O baselines on image classification tasks.
main_review: Strengths:
- This paper is mostly clear and organized, except for some points mentioned in Weaknesses below.
- The paper made an interesting attempt to distill the knowledge from analytical optimizers with three amalgamation methods and conducted comprehensive experiments to show the effectiveness of the amalgamated optimizer.


Weaknesses:
- There are some places that the paper did not describe the content clearly, especially for the definition of stability. When to use $Y_{ij}^\text{val}$ and $Y_{ij}^\text{train}$ in Eq. (9) was not discussed and thus it is confusing to me how to derive the meta-stability and evaluation stability.
- In terms of both the validation loss and the validation accuracy, the amalgamated optimizer did not have obvious advantages over the oracle optimizer, as shown in Figure 4 and Figure 9, which makes the method not very practical.
- Although the authors claimed that they tried to distill the knowledge from traditional optimizers, it is also related to imitation learning. However, there was no discussion about the different between knowledge distillation and imitation learning when trying to learn the optimization pattern from analytical optimizers. Some imitation baselines might be incorporated into the meta-loss as well.
- I am wondering whether first initializing the learned optimizer via the proposed amalgamation methods or imitation learning methods and then training it with the meta loss will have better performance. The reason is that randomly initialized optimizer is hard to train at the beginning and might disturb the whole training trajectory.
- It seems that adversarial perturbation performed similarly as random perturbation. Since adversarial perturbation requires more computing resources, then why not just use random perturbation? Are there any other advantages of adversarial perturbation?
- I am also interested in the performance of the choice optimizer C. Maybe the authors can expand on it, such as comparing C with the analytical optimizers it learns from. 
- I don't quite understand the choice of l2-log loss. Is there any related paper using this type of distillation loss?  Maybe the authors can try the gradient matching loss in [1], which is shown to be effective for dataset distillation and works with a similar number of parameters.
- The time and memory cost for training the amalgamated optimizer is not reported, which is important for the practicality of the method. 

[1] Zhao, B., Mopuri, K. R., & Bilen, H. (2020). Dataset condensation with gradient matching. https://openreview.net/forum?id=mSAKhLYLSsl

summary_of_the_review: Generally, I think the paper is marginally below the acceptance threshold. Although optimzer amalgamation is interesting, it seems expensive to train such an optimizer based on my experience and the authors did not report any time/memory complexity. Besides, the performance is not improved obviously compared with the best analytical optimizer, and the paper lacks the detailed discussion about imitation learning.
review_title: null

Meta-review: This paper introduces a meta-learning approach to "amalgamate" optimizers. The reviewers all found the idea interesting and unanimously found it to be acceptable for publication. In particular, I appreciate that the authors expanded their results to include more larger problems. One of the outstanding questions that would be interesting to address in future work is the use of tuned learning rate schedules.
Decision: Accept (Poster)