Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper proposes *Bootstrapped Meta-Learning,* a new meta-learning algorithm for hyperparameter optimization. Drawing inspiration from temporal difference learning techniques in reinforcement learning, the meta-learner is asked to predict the result of additional unrolled steps of the optimization process, by minimizing a carefully selected distance to a target generated during training. This allows for longer meta-learning optimization horizons, without the need for differentiation through longer optimization trajectories. The method is tested for hyperparameter optimization for reinforcement learning, including learning the exploration hyperparameter for a behaviour policy, and in multi-task meta-learning.
main_review: **Originality:** The method proposed in the paper is, to the best of my knowledge, novel. The related work section adequately connects the algorithm with existing work in similar directions. 

**Significance:** The empirical results show considerable improvement w.r.t. well-performing baselines; moreover, the general idea behind the method could inspire future research.

**Rigour:** Both the theoretical results and the experimental protocols seem sound and solid to me.

**Strengths**

- The method is based on a conceptually compelling and inspiring idea.
- The empirical results are remarkable. The ablations and additional experiments (also from the Appendix) help in understanding what matters in the practical algorithm, as well as highlighting the important parts of the contribution.
- In particular, being able to meta-learn hyperparameters for a behaviour policy can open up new avenues for exploration in reinforcement learning.
- The theoretical results have a clear scope (although not so large) and provide some easy to understand local improvement guarantees.
- The paper is well-written and generally easy to follow.

**Minor Concerns / Questions**

- I believe the name *matching function* makes the presentation of the method a little bit harder to digest. Since the function is a pseudometric (i.e., the larger it is, the larger the distance from the target), it should really be called with a name that reminds the reader of this nature (e.g., *mismatch function*).
- I enjoyed the theoretical results, but it is a pity that they only deal with targets of specific forms and, especially, with $L=1$ only. Ideally, theoretical result with a dependency on $L$ would shed some light on the benefits and limitations of longer bootstrapping horizons.
- Can the authors elaborate on the connection between the way the bootstrapping target is formed in their method and traditional temporal difference learning? In particular, the grounding role of that subtracted gradient "nudging the trajectory in a descent direction" is the same as the one of the reward in temporal difference learning; but, while the reward is at the beginning of the trajectory, the grounding is here at the end of the optimization subtrajectory. Is there any mathematical connection beyond the general shared motivation?
- As briefly touched upon in some passages of the paper, when the underlying function is highly nonlinear, there is the risk that the bootstrapping mechanism can lead the optimization process in worse areas of the landscape. For instance, if the function in Figure 1 had a bump/plateaux where $\tilde w$ is, the bootstrapping mechanism would cause more troubles than standard meta-gradients. Why is this not happening in practice?

-------
*After rebuttal*: I am happy with the answer the authors provided and the update to the paper, which will help the readers understand the relationship between the proposed method and TD-learning. Overall, the improvements make me believe that this paper should be highlighted at the conference, to give other researchers working in the field the possibility to get inspired by this new idea. I am thus raising my score to 10.
summary_of_the_review: The algorithmic and empirical contributions of the paper, as well as the theoretical grounding, largely justify in my opinion its acceptance at the conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ‌‌The paper presents a new meta-learning algorithm to address two shortcomings of standard meta-optimization algorithms: curvature (the meta-learner's objective is typically constrained to the same type of geometry as the learner), and limited evaluation (the meta-objective is evaluated only with-in a K-step horizon, ignoring future learning dynamics). The proposed algorithm addresses these two issues by minimizing the distance to a bootstrapped target under a chosen metric. Empirically, the new algorithm  achieved a new state-of-the art for model-free agents on the Atari ALE benchmark and yielded gains in multi-task meta-learning. Theoretically, some guarantees on performance improvements are provided.

main_review: **‌Originality**

The paper is original and novel, proposing a new algorithm to overcome two shortcomings of standard meta-optimization algorithms: curvature mismatch and limited evaluation.

**Quality**

The paper is technically sound, and claims are backed by solid experiments in the reinforcement learning and multi-task meta-learning evaluation setting.

**Clarity**

The paper is clear, however, the algorithm description in section 3 is very abstract. I think the paper would benefit from a running example and a dedicated section and pseudo-code describing the algorithm and how it can be instantiated in different experimental settings.

**Significance**

The work is significant and will benefit the reinforcement and meta-learning community by addressing some of the limitation of the current meta-learning algorithms.

**Limitations**

- The theoretical analysis is limited to noiseless 1-step target updates.
- The experimental evaluation in the multi-task meta-learning setting is limited to only compare with MAML on computer vision applications. 

**Questions to Authors**

- Some engineering / handcrafting is still required by the machine learning practitioner to select what "target" the meta-learner is going to optimize, as well as the proper "metric" for the meta-learner to optimize for. Could the authors comment a bit about what heuristics they used when making these decisions? and whether the automation for this process is possible or not? 
- What would it take to extend the analysis beyond 1-step noiseless target updates?
- How does the performance of BMG compare to alternative meta-learning algorithms like R2D2, Meta-OPT-net and prototypical networks? Have the authors experimented with other meta-learning benchmarks beyond image classification?

**Minor Typos**

- Abstract: "show that metric" -> "show that a metric"

summary_of_the_review: ‌‌
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper broadly considers meta-learning, a.k.a. bilevel optimization, across single-task, multi-task, supervised learning, and reinforcement learning settings. The authors aim to resolve two issues with the standard outer-loop gradient-based optimization of the meta-parameters (assuming a differentiable inner-loop): first, since the meta-learning objective is typically computed from learner parameters after applying up to $K$ inner-loop updates, the meta-optimization is myopic in that it does not optimize for further inner-loop improvement after $K$ steps; second, since the functional form of the learner's objective $f$ is used to drive the outer-loop updates, the meta-learning objective inherits the curvature of $f$. The main algorithmic contribution consists of a family of meta-learning objectives called bootstrapped meta-learning, in which meta-parameters are optimized to bring post-inner-loop learner parameters $x^{(K)}$ closer to a bootstrap target (which are also learner parameters) computed from $x^{(K)}$. The authors show that bootstrapped meta-learning generalizes the "direct" (my terminology) gradient-based meta-parameter optimization used in many previous works, recovering it when using specific choices for the bootstrap computation function and learner parameter matching function. With certain strong assumptions, the authors theoretically motivate the use of gradient-based bootstrap target functions for bootstrapped meta-learning in terms of optimization progress. The authors make several experimental contributions: they use bootstrapped meta-learning to achieve state-of-the-art model-free performance on Atari-57, demonstrate the viability of bootstrapped meta-learning in few-shot image classification on miniImageNet, and show that the more flexible, general form of bootstrapped meta-learning can enable meta-learning parameters that do not appear in the computation graph for the task objective, e.g. meta-learning the exploration rate of the behavior policy in $\epsilon$-greedy $Q$-learning.
main_review: Strengths
- The proposed idea is simple.
- The writing is clear.
- To my knowledge, the proposed idea is novel yet concretely linked to many past works by virtue of generalizing them. The authors give the precise form to recover MG from BMG.
- The proposed idea results in strong improvement of the STACX agent in Atari-57, resulting in a state-of-the-art result (caveat: for model-free agents; the gap to model-based agents remains large).
- The authors demonstrate that the proposed idea is suitable in few-shot image classification, a popular application of meta-learning for few-shot learning.
- The authors run informative ablation studies that support the intuition behind the benefit of BMG: resolving curvature and mitigating myopia.

Weaknesses
- Given that you say BMG is compatible with any update function (so long as it is differentiable in the meta-parameters), it would be nice to have some experiments on learned sequence model update rules (e.g. RNN). All current experiments use update rules with a fixed functional form.
- I am not putting much weight on section 4 ("Performance Guarantees") given the gap between its assumptions and results vs. what is actually implemented, and the restriction to local optimization.

Minor comments
- p. 3, target bootstrap paragraph, 3rd line: are we missing a learning rate for the expression of the target?
- p. 5, the actor-critic RL objective and Eq. 4: as-is, we are always minimizing policy entropy; is there a sign error?
summary_of_the_review: I believe that the meta-learning community will find this paper interesting. It provides new insights into formulating meta-learning models and gives examples of such insights being applied to obtain empirical gains. The paper is well-written and features exemplary empirical execution.

POST-REBUTTAL: I have updated my score to 10 and confidence to 5. I think this paper should get an oral, if not best paper. It is the best in my batch and is arguably the best I have read all year. It may be the best paper I have ever peer-reviewed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents Bootstrapped meta-gradients (BMG), an extension of typical meta-gradients (MG) for the task tuning meta-parameters that control the learning process (i.e. update step) of a learner.  In general terms, MG applies a (meta-)parameterised update rule to a learner for K steps, and then backpropagates through these updates to update the meta-parameters in the direction that improves the performance of the adapted learner.  The authors identify two limitations to this approach, and propose BMG to address them. (i) MG is myopic, in the sense that it does not account for future learning dynamics beyond these K steps, therefore BMG proposes to bootstrap a target from the K-step parameters (in practice by continuing to optimise w.r.t. parameterised update rule for L-1 steps, and then taking a final step w.r.t. a fixed objective to ground the signal). (ii) MG updates are necessarily restricted to be within the geometry of the parameterised learning process.  In contrast, BMG introduces a matching function to measure the distance between the learners K-step parameters and bootstrapped target in an arbitrary (and hopefully more-suitable space).

After framing the problem and BMG, the authors provide a discussion of the necessary conditions for BMG to guarantee performance improvements, though ultimately the presented algorithm is justified empirically using experiments on (i) a toy RL problem, (ii) the Atari RL test suite, (iii) multi-task few-shot adaptation on an image recognition task.  In all settings, BMG provides significant improvement over meta-learning baselines — most impressively achieving a new SOTA on Atari.  Moreover, key features of BMG are highlighted, including the ability to extend the meta-learning horizon without increasing the number of updates steps through which we must backpropagate, and that behavioural parameters outside of the update rule (specifically, epsilon in epsilon-greedy exploration) can also be meta-learned.
main_review: **Strengths**

-  The paper is generally clear and well written.  It is content dense in and certain areas would benefit from more detailed discussions.  However, the appendices are very detailed and provide answers to almost all of the questions that arise during first reading.  In my suggestions below I’ve highlighted a few areas I feel this content would be especially beneficial (and one pare where the main text could be stripped-back as there is always a trade-off with fixed page limits).

- The key ideas of BMG, bootstrapping a target to combat myopic MG updates and using matching functions to improve the meta-learning dynamics, are novel and well motivated.  I strongly believe that these ideas will be of interest to the broader meta-learning community and will likely lead to multiple future research directions.

- The empirical results are strong and thorough.  The toy-model grid world in Sec 5.1 effectively highlights key properties of the proposed algorithm — such as the ability of BMG to exploit longer meta-learning horizons — and extensive testing on Atari environments (Sec 5.2) shows a significant improvement of STACX and leaves little doubt of the efficacy of BMG for self-tuning algorithms.   Moreover, the demonstrated performance improvement of BMG over MAML (sec 6) suggests broad possible applications and further endorses the proposed algorithm.

- BMG also has a couple of nice properties besides raw performance: (i) achieving less myopic updates without having to backpropagate through many update steps of the inner-loop parameters and (ii) allowing for the meta-learning of “behavioural parameters” that are not used in the learning rule (exploration epsilon is used as the example).  Whilst (i) is clearly desirable in the pursuit of computational efficiency, (ii) is very intriguing and opens the door to new applications of meta-learning (indeed, whilst it is mentioned in the abstract and summaries, I believe this point is slightly undersold in the main text and that this experiment could be presented in more detail than the single paragraph at the end of Sec 5.1 it is given — however, given the page limit I can understand the authors predicament).

**Weakenesses**

- My primary concern is not with the content of the paper, but that the it can be quite difficult to intuitively link the numerous experiments back to intuition or interpretation of the results.  I believe this was largely because the exact methodologies are difficult to follow and, indeed, I found it was essential to refer to the appendices repeatedly to fully unpack the experiments and appreciate the results.  Whilst I am sympathetic to space constraints, I believe the authors would be well served to provide more detailed descriptions in the main text or, ideally, an algorithm box.

- I find the implementation and implications of the experiment on multi-task few-shot learning (Sec 6) unclear in the following regards:
	- I do not understand the intuition of why a “hot” expert transforms more information than a “cold” expert, and, moreover, why BMG is able to use this to improve performance.  Could the authors clarify these points.
	- I feel this section in particular suffers from a lack of formal introduction of the task and methodology.  For example, the adaptation seems to be defined as a single step, whereas the appendix (D.2) notes that 10 are used during meta-testing.  Concretely, I think a formal description of the training procedure for BMG would be appropriate in the main text.

- I do not find the analysis presented in Sec 4 (“Performance Guarantees”) especially insightful.  My reading is that, whilst the MG update presented in Lemma 1 can guarantee local improvement, practical implementations of BMG do not.  Empirically, however, grounding the bootstrapped target with a single final step on the meta-objective is sufficient for good performance.  This conclusion is evident from the experiments themselves, and so I would have no issue if Sec 4 was in the appendix.

**Errata**

- Sec 3, paragraph starting “To tackle myopia…”: the inline equation for $\tilde{x}$ is missing brackets around the step counters for $x^{(K+L-1)}$.
- Sec 5.2, second last paragraph: typo - “K is more sensitive curvature and the quality of data”.
- Sec 6, sub-section BMG: typo - “raising the temperature in the expect allows”.
- Sec 6, sub-section Setup: typo - “For 50 meta-updates and beyond..”: this should be 50k meta-updates I believe.
summary_of_the_review: Overall I believe this paper proposed an insightful and novel approach to addressing the stated limitations of typical MG approaches.  The authors do a good job of motivating the key innovations proposed and, given the significant research interest in MG’s in the past few years, it is reasonable to assume that the methodologies presented will be of broad interest.  Moreover, the experimental results are impressive and clearly demonstrate the improved performance of BMG and analyse where this comes from.  I do believe that the experimental results would be better served with more detailed descriptions of the problem settings and methodologies, however given the overall level of detail presented in the appendices I do not doubt there validity or significance.  Even so, for the stated reasons of novelty, interest and potential impact, I believe the paper is suitable for acceptance in the current form.
review_title: null

Meta-review: This paper addresses a meta-learning method which involves bilevel optimization. It is claimed that two limitations (myopia of MG and restricted consideration of geometry of search space) that most of existing methods have can be resolved by the MBG with a properly chosen pseudo-metric. The algorithm first bootstraps a target from the meta- learner, then optimizes the meta-learner by minimizing the distance to that target under a chosen pseudo-metric. The authors also establish conditions that guarantee performance improvements and show that metric can be sued to control meta-optimization. All the reviewers agree that the idea is interesting and experiments well support it. Authors did a good job in the rebuttal phase, resolving most of concerns raised by reviewers, leading that two of reviewers raised their score. While the current theoretical results are limited to a simple case where L=1$, the method is attractive for meta-learning community. All reviewers agree to champion this paper. Congratulations on a nice work.
Decision: Accept (Oral)