Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: -Summary-
The paper proposes a method for the sequential meta-learning problem. The author meta learn not only model parameters but also learning rate vectors for parameter blocks. To this end, the meta-learn model finds appropriate model parameters and adaptive learning rate vectors that capture task-general information. Overall experiments are performed on few-shot meta-learning settings with sequential domains (datasets).

-Pros-
- Optimizing fine-grained learnable learning rate vectors for manually grouped parameter blocks is reasonable.
- The performance of the proposed model significantly outperforms baselines which naively combined existing few-shot meta-learning and continual learning approaches.


-Cons-
- The approach is too simple (Adding learnable strength vector weights for gradient update of conv. blocks) and heuristic. And core hyperparameters are manually decided, like # of blocks per Conv. layer and the size of memory.
- Lack of analysis. There is a lack of concrete insight into how does adaptive learning rate weights mitigating forgetting. 
- The method only considers multi-head continual learning problems for task-incremental learning. The majority of recent impressing CL works considers further realistic and applicable to the broader areas, called class-incremental learning problem that task oracle isn't given during training/inference of the model.
- The method is only performed on simple CNN architectures. It needs to be validated on further modern deep neural network architectures. And, while the construction of CNN in this paper, most of the well-known CNN architectures have a different number of filters per layer. In this case, the strategy to split blocks can be important for pursuing a better model. However, there is no discussion/analysis of the problem.
- Meta-learning with bilevel optimization might require an additional computational cost.

-Comments-
- Experiments on domain ordering are interesting. I see that recent CL works consider evaluations on multiple domains(datasets) like "Hard Attention on Task" (HAT) paper. It would be interesting to perform analysis of the task(domain)-order sensitivity like 'order-normalized performance disparity' (OPD) in [1], which can be beneficial for understanding backward-forward transfer during continual learning under the domain shift.
- Citation of the XtarNet is duplicated. Please combine them. 

[1] Yoon, Jaehong, et al. "Scalable and Order-robust Continual Learning with Additive Parameter Decomposition.", ICLR 2020.
review_title: Review of the Paper 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work focuses on sequential adaptation of a model without forgetting. Their goal is to minimize the catastrophic interference of the model while learning a new few-shot task coming from a different domain. To that end, they introduce a problem setup where the model receives a sequence of few-shot tasks from different domains. 

I am not sure I entirely understand that the motivation and the proposed algorithm in this work. Overall the algorithm resembles replay-based continual learning approaches. Also, the algorithm feels similar to Online Meta-Learning. How does adaptation of learning rate address catastrophic interference? Please take a look at my comments below. Maybe, it is helpful to provide a real task for this setup. Having that said, I appreciate the authors' effort put into evaluating multiple baselines.

Comments and questions:
-What is the value of N (i.e how many tasks were shown to the model?)? I am curious about different runs with varied N, for each model including baselines.
-Maybe discuss memory and accuracy trade-off? Some baselines don’t require a replay buffer nor iterative re-training.
-Please provide references on the following statements - Most existing works focus on developing the generalization ability under a single context/domain. Recently, it has been shown that catastrophic forgetting often occurs when transferring a meta-learning model to a new context.
-How does the proposed setup and algorithm compare to those of “Finn, C. Online Meta-Learning”?
-Some missing references: “Schmidhuber, J. A neural network that embeds its own meta-levels.”, “Santoro, A. Meta-Learning with Memory-Augmented Neural Networks”, “Munkhdalai, T. Meta Networks,” etc.
-Please note that miniImageNet subset was introduced in Matching Nets.
-The first sentence in Section 2.1 seems to describe an application of meta-learning to few-shot learning rather than meta-learning itself. There is also a line of works on memory-based meta-learning (Mikulik, V. Meta-trained agents implement Bayes-optimal agents).

review_title: Needs some clarity

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

At the heart of this paper are two separate contributions. The first is a new online meta-learning problem setting where the meta-learner acts on a sequence of few-shot learning *domains*, as opposed to tasks within a single domain. The second is a method for meta-learning with this form of domain shift.

As far as I know, this is the first benchmark that tackles online meta-learning over few-shot domains. This is a much needed push in the direction of obtaining harder continual learning benchmark - I'm quite excited about this line of work and would encourage the authors to continue to pursue this line of work and include longer sequences of domains to truly test the continual capabilities of our current methods. 

The authors also propose a meta-learner for the newly introduced problem setup. This meta-learner makes use of two well-known concepts form the literature, experience replay and learning-rate adaptation. In particular, the model's parameters are adapted to the current task (few-shot) using meta-learned learning rates. These learning rates are meta-learned over all tasks in the memory in a multi-task loss function.

Pros:
- A new benchmark that is considerably more challenging than previous online meta-learning benchmarks.
- Extensive benchmarking of a wide range of baselines from both meta-learning and continual learning.
- Strong performance from the proposed method.

Cons:
- Unclear claims to novelty.
- Confusing writing at critical steps.

Recommendation: reject

Motivation: 
I believe the proposed benchmark would be of service to the community and should eventually be published at a peer-review venue, however the current manuscript contains critical issues that prevent me from recommending acceptance. I'm open to changing my score should the authors address my concerns below. 

Main concerns:

- *Contributions:* The authors list three contributions. Of these, the first relates to both adapting parameters and learning rates and the second relates to using layer-wise meta-learned learning rates. Both are well known approaches that have been explored extensively in the literature. Meta-learning learning rates was proposed in [1] and have since been explored in a variety of contexts. Adapting both layers and learning rates have also been explored in a variety of forms in episodic meta-learning (e.g. [2]). Further, recent meta-learners [e.g. 3, 4] learn model parameters for fast task adaptation and meta-learn optimiser parameters to avoid catastrophic forgetting, precisely what is claim as a contribution here. This is not to say that there are no algorithmic contributions in this paper, but rather that it is impossible to tease out what the authors claim as their own contribution to the field.

- *Technical correctness:* The writing becomes very dense on pages 4 and 5, where the method is introduced, making it hard to understand precisely what the authors propose. In particular, they describe their method as a bi-level optimisation problem, but Eqs. 1-4 do not support this description.  Both losses have the same input arguments (they only differ in the expectation over tasks) and hence represent a multi-task setup. This is not a well defined problem because the learning rates have no effect on the loss unless the gradient update is taken into account. Similarly, the authors use a single \theta to denote model parameters, but in Figure 1 and in the text mention that they treat the lower layers of the CNN differently from the top (task-specific layers), as in [5]. This is not described in the algorithm, where it appears as if all model parameters are being tuned to the current task and all layer-learning rates are being meta-learned on replays of previous tasks.

Minor concerns:

- *Clarity:* the manuscript would benefit from a simpler and clearer presentation of the problem setup. At the end of the day, the paper proposes to meta-learn over a sequence of few-shot learning domains, as opposed to a sequence of tasks from the same domain. This will require a higher degree of generalisation from meta-learners, and should stress our current methods. There is no need to make the motivation more intricate than that.  Similarly for the proposed method, a simpler and more direct presentation would greatly help the reader understand what is being proposed and what is being borrowed from previous works. I'm still unclear as to what 'double adaptation' means. Is it just that both learning rates and model parameters are being tuned?

- *Citations:* while the related work section contains a wealth of citations, the introduction makes sweeping claims (such as "it has been shown that catastrophic forgetting often occurs when transferring a meta-learning model to a new context") that should be substantiated with appropriate citations. It is also a bit unfair to invalidate all previous works as not applicable to this problem setup because of "high variability underlying a large number of dynamically formed few-shot tasks". This is an empirical matter and not a an absolute fact.   

- *Experiments:* generally, the experimental section is sound and features a broad set of baselines and conducts a battery of ablations. While this is great, the one ablation I was hoping for was to see what happens if you change the order of the domains such that miniImagenet and CIFAR are at the end. In general, having one (or both, as in this case) as the first two tasks means the meta-learner can learn good representations for downstream tasks pretty much immediately. This seems to precisely counter-act the goal of introducing a harder benchmark. Finally, I'm a little unclear as to whether catastrophic forgetting is measured 0-shot without re-adapation, or few-shot by allowing adaptation to tasks from past domains given \theta^j_q?

Typos:
- catastrophe forgetting -> catastrophic forgetting
- scarifying ->degrading(?)

Post Rebuttal 

The authors have improved the context of their work and clarified their proposed method. While the technical novelty is somewhat limited, the proposed method does well and the benchmark introduced herein should be of interest to the community. As such I have increased my score.

References
[1] Li et. al. 2017. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning.
[2] Lee et. al. 2018. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace.
[3] Flennerhag et. al. 2019. Meta-Learning with Warped Gradient Descent.
[4] Gupta et. al. 2020. La-MAML: Look-ahead Meta Learning for Continual Learning.
[5] Javed et. al. 2019. Meta-Learning Representations for Continual Learning.
review_title: An interesting contribution that requires some clarifications

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a new instantiation of Continual Few-Shot learning (CFSL) with multiple domains coined sequential domain meta-learning.
They also propose an extension of the meta-SGD [1] algorithm to CFSL where the learning rates are learned on tasks from past domains to alleviate forgetting.
Both the setting and the idea have potential, but the paper still needs some work and considerable weaknesses.
Mainly, the setting needs further motivation and clarifications, and the method needs some justification as well as stronger performance. 

On the setting:
- 1.1 The setting's motivations are self-driving car- and dialogue system-like scenarios. The authors claim that these are aligned with the newly proposed setting. In these examples, however, the hypothetical methods would need to adapt to the changing data distribution in an **online** manner and without an explicit support set. In the new setting, methods are tested at the end of training on all past tasks, i.e. long **after** the methods were in the tested domains and with a different model. When a new setting is proposed, it should be clearly motivated. I suggest the authors either improve the current motivations or adapt their settings to better align with real-life continual-learning scenarios. For the latter, the authors could, e.g., report online cumulative performance on all domains with domain revisits, similarly to [2].

- 1.2 It seems like the methods have access to the task label to properly chose their output head (or domain-dependent parameters)? If this is so, the methods are operating in the task-aware setting, where forgetting can be solved by freezing and growing the network. In that case, the authors should add this as a baseline. If they cannot beat it, they should show on what metrics they can outperform it, e.g. maybe their method is more computationally efficient. Also, task-aware doesn't seem to be aligned with the proposed motivations.

- 1.3 The meta-test protocol (ALgorithm 2) is unclear. Are the fast weights retrained? If so, how? Are the learned learning rates $\lambda_q$ ever used? If not, why are they learned?



On the method:
- 2.1 The authors rush to explain their double adaptation method but their mechanism is not justified anywhere. Specifically, why would you learn the learning rate on past tasks? and why would you backprop through $\theta_q^j$ which is adapted to domain $q$?

- 2.2 The authors should acknowledge that learning the learning rate (or double adaptation) is a well-known trick now. They should explain how they extend this method and change the tone of the text such that their proposed method seems less novel.

- Going back to 1.3, in ALgo 2, are the learned learning rates ever used? Because algorithm 2 only uses $\theta_N^M$ i.e the model adapted to domain $N$. 

On the experiment:

- 3.1 why didn't the authors run a hyper-parameter search?

- 3.2 Table 3 ablates their proposed double adaptation on past domain scheme, which is the novel part of their method. The gains are however relatively small. Furthermore, with the large number of hyper-parameters and without a hyper-parameter search, the gains could be noise introduced by the authors iterating on their method by not on the baselines. I suggest the authors run a hyperparameter sensitivity analysis. 

- 3.3 The authors explain a new memory selection mechanism that is unclear and doesn't work well yet. I encourage the authors to remove that section and Table 6, or significantly improve it. Also, on "We then divide the losses into Q clusters with kmeans. ":  if the losses are scalars, why are you using k-means?



Other concerns:
- figure 2 seems unconstructive. I suggest the authors remove it or move it to the Appendix.

- put lines in the algorithms.

- The introduction is hard to follow. Also, the 4th paragraph should be moved and merged into the Related Work section.

- In the related work,  the authors claim that [2] is part of a group of papers that operates in a single domain. However, this is not true for [2] which operates in multiple domains within one experiment, e.g. the Omniglot / MNIST / Fashion-MNIST experiment. Also, [3] is a Meta-Continual learning method and not an incremental few-shot learning one.

typos: 
- in abstract and elsewhere: "catastrophe" forgetting--> "catastrophic"




_____

**POST REBUTTAL**

The authors have provided some clarifications. I suggest they use them to improve the paper.
I'm increasing my score to a 5, thus still not in favor of acceptance.
_______

[1] Zhenguo Li, Fengwei Zhou, F. Chen, and H. Li.  Meta-sgd:  Learning to learn quickly for few shot learning. ArXiv, abs/1707.09835, 2017.

[2] Massimo Caccia, P. Rodr ́ıguez, O. Ostapenko, Fabrice Normandin, Min Lin, L. Caccia, Issam H.Laradji, I. Rish, Alexandre Lacoste, D. V ́azquez, and Laurent Charlin.  Online fast adaptation and knowledge accumulation: a new approach to continual learning.ArXiv, abs/2003.05856, 2020.

[3] Khurram   Javed   and   Martha   White.Meta-learning   representations   for   continual   learn-ing.InAdvances   in   Neural   Information   Processing   Systems   32,    pp.   1820–1830.CurranAssociates,Inc.,2019.URLhttp://papers.nips.cc/paper/8458-meta-learning-representations-for-continual-learning.pdf.
review_title: potentially interesting idea but needs more work

Meta-review: The paper proposes a sequential meta-learning method over few-shot sequential domains, which meta learns both model parameters and learning rate vectors to capture task-general representations.

Reviewers raised many insightful and constructive comments. The main themes are as follows:
- The problem setting needs further motivation and clarifications, to make it more realistic and applicable.
- The novelty is relatively weak, e.g. the approach is too simple, and learning the learning rate is a common trick.
- The method needs great effort for better presentation and justification. The current presentation simply lists several equations in a dense way without detailed explanation. Some main claims such as mitigating catastrophic forgetting are not elaborated extensively.

AC scanned through the paper and agreed with the reviewers' main points. Authors' rebuttal in general did not address these concerns to the satisfaction. For example, even after revision, the readability of this paper is not good enough. The authors are encouraged to perform a thorough revision.
Decision: Reject