Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a hyperparameter optimization algorithm in meta-learning, where parameters w/o being involved in the inner loop optimization are treated as hyperparameters. The proposed algorithms approximate the second-order hypergradients via knowledge distillation. They further evaluate the effectiveness on tinyImageNet and CIFAR100.

main_review: The proposed method is well-motivated, and the paper is clearly written. My major concerns are about the experiments:

1. In this paper, the authors evaluate the performance on two image classification datasets. It might be more convincing if the authors can evaluate the performance on more kinds of tasks (e.g., regression tasks). Besides, can HyperDistill boost the performance under different meta-learning algorithms? More analysis will make the findings more convincing. 

2. HyperDistill is a general hyperparameter optimization method. And the authors mention that the proposed method is easy to be applied on traditional hyperparameter optimization (HPO) tasks. Could you evaluate the effectiveness of the proposed method on some supervised learning datasets and compare it with HPO algorithms.

3. Minor: are the tinyImagenet and CIFAR100 tasks few-shot learning tasks?

---- After Rebuttal---

I am happy with the authors' response and decide to keep my score.

summary_of_the_review: The motivation of the proposed method is clear, but more comprehensive experiments are needed to verify the effectiveness of Hyperdistill.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel hyperparameter optimization algorithm in meta-learning to overcome previous limitations of only being able to see a longer horizon and not being scalable to high dimensional hyperparameters. In particular, the authors propose to distill the hypergradient second-order term into a one-step Jacobioan-vector product. The authors show that, with their approximated algorithm, it is possible to perform hyperparameter optimization for higher dimensional hyperparameters and longer horizon length even in an online setting. Empirically, the authors show the advantages of the proposed approach in several benchmark datasets.
main_review: Pros: 
* The paper is well-written and easy to follow. The introduction and related work well summarize the current state of the problem and motivate the proposed method.
* The empirical experiments are rigorous and well supports the claims made in the paper. 
 
Concerns:
* One question I had was on the definition of the short-horizon bias. I believe that the short-horizon bias occurs for a particular type of hyperparameter such as learning rate. For many regularization hyperparameters, my understanding was we still don’t know the greedy update performs worse in the long run. I think a clarification of the definition in the main text would be helpful. 
* In Figure 2, what would happen to the convergence if we vary \gamma? As the \gamma gets smaller, does the convergence look more similar to the 1-step or DrMAD? I think this experiment would strengthen the claim that HyperDistill is robust against short-horizon bias.
* What are additional hyperparameters HyperDistill introduce? Does HyperDistill robust to this new set of hyperparameters? Since the theory part of the algorithm mostly relies on several approximations/assumptions, I believe that the ablation study on all datasets would be helpful. 

Minor comments/questions: 
* In the introduction, it says “computing hypergradients before convergence does not guarantee the quality of the hypergradients”. I was wondering if this statement is true. The IFT works the same way even if the training loss is a constant above 0. 
* In the experiment section, under the task distribution paragraph, TinyImageNet is not cited properly.
* In appendix E, it would be helpful to directly cite the original paper.
* In Figure 7, DrMAD stays at 1 constant. Does it mean that it is not learning anything?

summary_of_the_review: Overall, I vote for accepting. I believe that the paper is well-written, well-motivated and the claims made in the paper are well justified empirically. Moreover, the content is relevant to the ICLR community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper develops a practical gradient-based hyperparameter optimization method, HyperDistill, that meets the following criteria
  a) scalability in hyperparameter dimension and memory constraints, 
  b) accuracy (hyper-gradient update terms do not depend on only the last step of gradient updates)
  c) applicability to the online setting.

The main difficulty lies in estimating the gradient of the weights with respect to the hyperparameters. The authors do so by approximating it as a single Jacobian-vector product, using a "distilled" weight and dataset pair.

Experimentally, HyperDistill achieves better validation losses and higher quality true hypergradient estimates than a variety of recent, relevant baselines, and does in an efficient fashion.
main_review: Strengths
-------------
- This paper is very clearly presented; the background is appropriately detailed and provides sufficient information to compare HyperDistill to the relevant literature. The theory underlying the distillation process is well-explained and easy to follow; the experiments are well-described, easy to interpret, and illustrate well the various advantages of HyperDistill as it compares to other modern methods.

- HyperDistill addresses the shortcomings that are pointed out in the other modern gradient-based HO methods (scalability; limited horizon; applicability to the online setting). In practice, across a variety of datasets, HyperDistill consistently achieves lower validation loss and improves test accuracy.

- HyperDistill is in part motivated by having a long-term horizon (in comparison to the one-step lookahead approach). The authors compare explicitly to this baseline, and show that HyperDistill achieves a better reconstruction of the gradient, and recover one-step lookahead as a special case.

- The set of experiments is Experimentally, HyperDistill improves over the many baselines that are considered by the authors are well-designed, consider a variety of relevant baselines, and presents unambiguously the advantages of HyperDistill.

Weaknesses
-----------------
The main weakness that I see in this work lies in the creation of the distilled dataset D (section 4.2). The authors elide over the difficulty of building this dataset, mentioning only in passing that defining a distance over datasets is difficult. This could be done more carefully, starting by defining precisely what space the minimization problem (11) operates over, and then explicitly solving (or bounding the gap of the proposed solution to) the problem.

Questions / comments
------------------------------
- Nit: it might be worth clarifying the dimensionality and spaces that \pi, \w_t, and \D_t lie in e.g., in (4).

- Similarly, I would clarify the norm being used in (4), especially as the choice of norm becomes important for the distilled dataset later on.

- Some typos: "longer horizonS" in Section 1, "identitcal" in Section 2, "completely ignoreS" in Section 5
summary_of_the_review: The paper is clearly written, the theoretical justification for the paper is well-presented and intuitive, and experiments confirm the value of the proposed method.

My only criticism is the comparative lack of care with which the distilled dataset is analyzed; this stands out particularly in contrast to how the  rest of the technical difficulties are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new online hyperparameter optimization algorithm. Applying meta-learning for hyperparameter optimization is reasonable and interesting but suffers from the second-order gradient computation. Implicit Function Theorem and Unrolled differentiation can be used to approximate the meta-gradient but also causes various problems. In this paper, the authors propose an interesting method by approximating the second-order approximation with knowledge distillation. 
main_review: Pos:
1, The topic is interesting and important which can be impactful in machine learning research community.
2, The idea is reasonable and novel. Applying knowledge distillation in second-order gradient computation is novel.

Cons:
1, The written can be more clear by simplifying the notations.
2, The experiments can be extensive by applying the methods in more datasets, for instance, meta-dataset.
summary_of_the_review: I think the topic is novel and impactful and the idea is novel and interesting. Experiments show the effectiveness of the method. Hence, I recommend acceptance.
review_title: null

Meta-review: This paper presents a novel methodology for performing meta learning for gradient-based hyperparameter optimization.  The approach overcomes limitations (scaling, e.g.) of previous methods through distilling the gradients of the hyperparameters.  The paper received 4 reviews, of which all were positive (6, 6, 8, 8).  The reviewers appreciated the technical clarity of the paper and found the proposed approach sensible, novel, technically sophisticated and effective.  The main concerns were regarding the comprehensiveness of the experiments and technical presentation of the dataset distillation.  It seems that the reviewers found the author response (lots of results were added) satisfactory regarding these points.  Thus the recommendation is to accept.
Decision: Accept (Spotlight)