Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper propose to solve a very important problem, meta-learning hyperparameters in pre-training stage, followed by fine-tuning a target task. Considering that the hyperparameters can be high-dimensional, the authors consider gradient-based hyperparameter optimization (HO) methods such as IFT based methods and unrolled differentiation. The main difficulty comes from the huge computational cost of dealing with long PT and FT trajectory and repeating such meta-optimization steps. Authors thus simply reduce the number of gradient steps used for each PT and FT stage. Yet, the experimental results demonstrate the effectiveness of the proposed methods over some of the simple baselines.
main_review: == Pros ==

1. I think the paper tackles the very important and interesting problem, meta-learning of PT hyperparameters followed by FT. As far as I know, there are few literatures that explicitly tackle this problem. The problem is challenging because both PT and FT involve long optimization trajectory in practice, leading the optimization problem computationally very expensive. Therefore, I would say that the motivation of this paper is very clear and important, in terms of extending the current range of meta-learning and hyperparameter-optimization study.

2. The paper is well written and easy to understand, combining the two most popular techniques for HO (IFT and unrolled diff.). 

3.  The experimental results show that although the proposed method is simple and straightfoward, it could gain some improvements over the baselines they considered.


== Cons ==

1. As far as I understand, the main difficulty should come from the long PT and FT optimization trajectory used in practice (e.g. each at least thousands of SGD steps). However, the authors simply set each of them to very short number of steps, such as $P=10$ and $K=1$ for both of the real-world experiments. It will lead to short horizon bias, with the solution being less appealing. For example, Shin et al. recently proposed a method that can increase the frequency of meta-updates even with long inner-optimization trajectories. It would be better if the authors could think of some ways to deal with the short horizon bias in any way, because exactly that is the main challenge of this problem.

2. The proposed method is fairly straightforward to think of, combining the two existing techniques for HO and unrolled optimization (MAML). I agree that the method is reasonable, but not sure how much it is technically contributing. Also, as mentioned in section 8 Scopes and Limitations, IFT based methods cannot handle hyper-parameters that do not affect the PT optimization landscape. 

3. In the experiments, I think the baselines are too few. Specifically, there are no baselines that can learn the hyperparameter $\phi$. What if we learn $\phi$ only with the PT dataset based on conventional HO framework? We may split the $D_\text{PT}$ into $D_\text{PT}^\text{train}$ and $D_\text{PT}^\text{val}$, and optimize $\phi$ with Neumann IFT method as you did. I expect it will work well because the authors reported good performance on Partial FT-data setting, where we only use a fraction of FT tasks for meta-training and meta-test with the exclusive set of FT tasks no seen during the meta-training. It means that whatever tasks we use for learning $\phi$, the learned $\phi$ will generalize well to unseen tasks. If it works well, then the importance of meta-learning over PT-FT framework will become questionable.

4. (minor comment) There is no qualitative analysis. It would be interesting to visualize the learned $\phi$ and give the readers some intuition how it helped with the performance (or any other visualization).

5. (minor comment) Missing reference for meta-learning of self-supervised learning : Kang et al.

= References = 
- Shin et al., Large-Scale Meta-Learning with Continual Trajectory Shifting, ICML 2021
- Kang et al., Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation, EMNLP 2020


limitations_and_societal_impact: The authors adequately adressed their limitations and social impract.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work seeks to extend gradient-based HPO to the two stage setting of pretraining + finetuning. While the method used isn't original it is used in an original problem set up.
main_review: -----------------------------------PROS
- The PT+FT paradigm considered is original and relevant
- the paper is mostly clear
- Considering Full FT access and Partial FT access is a good thing

-----------------------------------CONS

MAJOR

- You explain that you update psi_FT online, which may remove memory issues with BPTT, but has been shown to lead to biased (greedy) solutions ("Short horizon bias", Wu2018). These don't converge to hypergradients close to the actual (full horizon) hypergradients we care about. This is a major limitation to your model since the gradients that come from the finetuning stage are in my experience almost as good as random noise, and yet this isn't listed in your Limitations section. A toy model where you can use full-horizon BPTT for both PT and FT would allow you to measure how accurate your algorithm is at approximating hypergradients as the number of steps increase in the PT and FT stage.
- While experiments seem sensible, I feel like the experiments I would need to see to make sure your method is competitive with sota methods aren't included. For instance, I would have liked to see comparisons with sota methods in multi-task learning and/or semi-supervised learning and/or domain adaptation for common image datasets like CIFAR-10 where lots of other methods have been applied. This is because the main difficulties of gradient-based HPO (e.g. gradient degradation) come in for a large number of steps in the PT/FT stages. In line 196 you seem to be using a very short horizon of P=10, which is nowhere near the 10^4 gradient steps you'd need for a CIFAR10-like dataset.  
- There is also an issue of novelty, since the combination of implicit differentiation and BPTT is fairly straightforward (just multiplication as per chain rule) and one may argue doesn't really make up a "new" algorithm in itself. 

MINOR

- Your toy experiments in section 4 are both experiments that don't require your PT + FT 2-stage method. Indeed, these have been done where the properties you apply to the FT data are simply applied to the validation data instead. It would be nice to have experiments where ONLY your 2-stage approach is sensible. 
- I found Figure 1 somewhat confusing. For instance I'm not sure what "Finetuning Data" + "Sample" labels achieve, why the "meta parameters" are connected to "Pre-training" on top of the "Model" and "Pre-train Dataset" labels since potentially the "meta parameters" could include "Model" and "Pre-training dataset". I understood your set up from the equations in 2.1, although a bit more details would have saved me time. For example in line 78 you could explicitely state the the final weights of the PT stage are used as the init weights of the FT stage, such that the final weights of the FT stage minimize the FT loss. 
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper describes a gradient-based hyperparameter optimization for differentiable pre-training hyperparameters in a pre-training and fine-tuning setup. The proposed approach approximately differentiates through the two-stage learning process. The authors show the benefit of their approach in two experiments, where they outperform multiple baselines.  
main_review: Originality: In my opinion, this kind of meta-learning on two-stage setups is not new (see ref. [67] and [68]) but the more general approach is novel. It feels a bit like MAML [15] for differentiable hyperparameters (MAML is mentioned in the related work). The difference to related work is described and related work is cited. 

Clarity: I had issues following this paper. While in the abstract and introduction the authors talk about “hyperparameters” they mention only in the limitations section that the hyperparameters have to be differentiable. When I read hyperparameters of neural network training, I think mainly on parameters like the learning rate. The abstract promises an HPO method for hyperparameters in general without mentioning this limitation.
In the second section in the paragraph “Notation”, the authors introduce uppercase theta, uppercase psi, and uppercase phi as “spaces of parameters for models” without any further description on the difference between these “spaces”.  Two pages later in line 123, the authors name phi “encoder parameters” and psi “decoder parameters” but without mentioning that they assume an encoder-decoder setup. 
In the abstract and introduction, the authors write about optimizing neural networks while in the problem setup they describe the more general case of “parametric model”. 
I wasn’t able to follow their algorithm due to the lack of knowledge about theta, psi, and phi. 
In the experiment section, it is unclear where the definition of the loss comes from and how P and K are tuned. 
I would propose to clarify what kind of hyperparameters are addressed by this approach and rework the problem formulation and notation.

Quality: I can’t assess the methods since I didn’t understand the algorithm in detail due to the lack of knowledge about theta, psi, and phi. The method is evaluated in two experiments with good results.
	 
Significance: In my opinion, the experiments are very specialized for a general tone in the abstract and introduction. I couldn't follow the notation easily. Even if I am not a domain expert for gradient-based hyperparameter optimization this shouldn’t be the case in my opinion. In the current state of this paper, I find it hard to make use of the approach. 
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a gradient-based algorithm which improve the first stage of two-stage pre-trained and fine-tune process with meta-learning techniques. The major contribution is proposing to use additional parameters called meta-parameters to augment pre-trained stage and use a novel gradient-based algorithm to learn meta-parameters through the two-stage process. The experimental results demonstrate the significant improvement of the proposed approach over the standard two-stage PT and FT process.
main_review: == Pros ==
- Using meta-parameters to improve pre-trained model has enough novelty, and further research on this open problem is meaningful.
- The authors propose a novel gradient-based algorithm to optimize meta-parameters.
- The proposed approach achieves significant improvements on both tasks.

== Cons ==
- Meta-parameters as neural networks could have much more representative power than weight of task or temporal transformation, I'd like to see more complex scenarios where this approach can be applied.
- Despite the experiment setting in this paper may be new, three still need some other meta-learning techniques as baseline if possible. Because as far as I know, there are many gradient-based meta-learning algorithms that are not used in PT and FT process for now.
- Meta-learning techniques usually requires much more computing resources and time, an analysis on GPU memory and execution speed would be appreciated.
- Some symbols may be confused for me, such as line 4 in Algorithm 1.
limitations_and_societal_impact: No societal impact.
review_title: null

Meta-review: Pretrain-finetuning is an important and popular learning framework for modern deep learning models, and has achieved significant success. How to learn some hyperparameter of the framework is important yet not well explored. This paper proposes a meta learning strategy for this via direct and implicit differentiation to perform gradient descent. The technique is not new but has not been applied to this problem.

All reviewers seem to consider the paper to be in the borderline. They raised a number of questions including the setting of the number of PT/FT steps, if it will cause short horizon bias, and also asked for some additional baselines to demonstrate the effectiveness of the proposed method. Most of the questions, in my opinion, have been well addressed in the rebuttal with further explanation and extra experiments. Thus I would recommend acceptance of the paper. One concern I have is that it seems the scale of the data in the experiments is not large enough (from what I understand in the paper, though the statistics of some data have not been provided). I guess investigating the large-scale setting is important because this is where the PT/FT framework really reveals its power. I would recommend the authors to revise the paper according to the comments from the reviewers, and consider extra large-scale experiments (such as the ImageNet scale data) to make the paper stronger.
Decision: Accept (Poster)