Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method to continually adapt a deep network to smoothly-changing tasks, by meta-learning a regularizer online. This regularizer consists of a L2 loss that binds the online-adapted parameters and (meta-learned) reference parameters together.

strength_and_weaknesses: 
Strengths: The motivation for the method is clear and makes sense, given the problem setting of online (continual) learning. The method seems novel, and has good experimental results in the analyzed settings.

Weaknesses:
- The main weakness is a series of mistakes in the equations, which makes them hard to parse, and the textual description of the method not being very clear.
- The appendix seems to be missing (!) so important implementation details are missing; this harms the reproducibility.
- Some discussion of related work is missing, but this is a smaller issue.
- Comparison with the most related Continual Learning methods would be better, namely those that also train online with a buffer of past samples: Tiny Episodic Memories, and GDumb.

The paper would benefit from discussing the relation between the proposed method, of a fast-learning network trailed by a (meta-learned) slow-learning "anchor" network, and others with a similar set-up. One such example is:
- Zhang et al., "Lookahead Optimizer: k steps forward, 1 step back", NeurIPS 2019
Other works that focus on learning by "self-distillation" (where usually the two models are described as teacher and student) should also be discussed:
- Caron et al., "Emerging Properties in Self-Supervised Vision Transformers", 2021

clarity,_quality,_novelty_and_reproducibility: 
There are several aspects of the paper that are unclear or have mistakes, described below. The method seems novel and the paper would be high quality if it were not for the unclear aspects/mistakes. There are important implementation details omitted, as it seems that the appendix was cut off.

I will now list the unclear parts, mostly equation mistakes:

- In the FTML equation (after eq. 5), phi is used in both argmins. This is nonsensical, because it means that the first argmin does not contain its minimizee variable at all; and if it was meant to be implicitly present via the second line, its reuse in the second line's argmin prevents this. This makes the whole equation impossible to parse.

- In the same equation, the LHS of the second line uses time index t-1, but the RHS does not -- so what changes between iterations? I can guess that Tau might need to be indexed using t, but this is unclear.

- In the first equation of section 5.2 (the authors really ought to number their equations), the first term in brackets does not contain theta; so the gradient of this term w.r.t. theta will be zero (wiping out this term). If there is a dependence, how is it achieved? This needs to be made explicit in the equation.

- Algorithm 1, lines 6-7: Missing j index in D (minor).

- Algorithm 1, lines 9-10: The notation \phi_\theta is used for the first time without explanation. Furthermore, it is used inconsistently -- in line 10, the theta is missing. If this was not used in all the equations so far, what role does it play here? This needs to be harmonized or explained in the text.

- Page 3: In the discussion of Continual Learning (via meta-learning), contrasting "batch-mode meta-learning" with the proposal should be made more clear, since at this point one does not have enough context to understand what is meant by "batch-mode" in this case.

- Page 5: "Note that this is subtly different from FTL in the classic online learning setting, due to the difference in problem formulation." - This is not enough; what is this classic setting and how is this different?

- Page 5: "Although FTML can enable fast adaptation, the model needs to be “reset” at each task, essentially creating a “branch” on each task and maintaining two independent learning processes: an
adaptation process, whose result is discarded completely at the end of the task, and a meta-training process, which does not influence the current task at all, and is only used for forward transfer into future tasks." This whole phrase is very hard to parse. It also does not follow the next conclusion that the method is "not fully online".


summary_of_the_review: 
The paper seems to present a nice method but requires some work to make the explanations more clear and correct notation mistakes. The missing appendix gives the impression that the submission was a bit rushed. It could also be made more complete in terms of related work. As it is, this paper is very borderline, but can be published if the authors can visibly correct all the mentioned mistakes, and improve it generally.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes an approach to perform meta-learning without task boundaries.
strength_and_weaknesses: **Strengths**
- the paper is globally clear and well writen
- the topic is interesting to the ICLR community

**Weaknesses**
- absence of any theoretical justification
clarity,_quality,_novelty_and_reproducibility: Clarity: good
Novelty: minor

summary_of_the_review: I am puzzled with this submission as I cannot find anything to comment on. For instance, the authors introduce the notion of regret (by the way, it seems from it that the goal is to maximize the loss functions ?! See also eq.(1)), but for what reason? It is never used afterwards, neither to state a bound or motivate the heuristics proposed. Clearly, the contribution proposed is not enough to warrant acceptance at ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes an extension of model adaptation via meta learning in an online fashion. This paper particularly targets the problem model adaptation in the environment with changing tasks and input distributions. With the ability for fast adaptation, the proposed method called as Fully-Online Meta-Learning (FOML) adapt the model parameters quickly and continuously for any incoming tasks. The experiments show that FOML can continuously adapt to a new task quickly and beat prior methods on meta-learning on several challenging datasets  e.g., Rainbow-MNIST, CIFAR100, and CELEBA.
strength_and_weaknesses: 
Strengths:
- The problem of learning from constant stream of data with distinct tasks in this work is challenging and relevant to rapid online adaptation of neural networks for many real-world applications e.g., traffic predictions systems.
-  The formulation for online meta-learning is somewhat novel and uses no task boundary knowledge to learn from newly incoming tasks.
- The experiments show the effectiveness and superiority of the proposed method to tackle online meta-learning problems compared to other prior arts e.g., FTML, LwF, iCARL, and MOCA.


Weaknesses:
- The definition and the role of task boundaries are not very clear for fully online meta learning and prior methods. Is this the boundary that has a task id indicating a specific task? What are “well-separated tasks”? How do the tasks for FOML have no boundaries? As understood from the FTML paper, Standard Online Meta Learning also does not need this additional knowledge to run the algorithm and adapt the model.  Does “task boundary” here mean that the evaluation task might come from past tasks i.e., not necessarily from the currently training tasks? Also, if there is no specified boundaries between tasks, can we derive the problem as standard online learning?
- The “gradual changes of tasks” also need to be explained in more detail. If there are no clear boundaries, how are tasks verified changed overtime? In my opinion, the motivation described in introduction does not perfectly align with the setting. It is vague to capture the gradual changes but discarding the concept of continual learning. The strong assumption in meta learning is about the stationarity of the data distribution. The gradual change of its distribution is known as continual learning problems.
- The meta directional update is conceptually similar to the formulation in Elastic Weight Consolidation (EWC) and Reptile. There is no head-to-head discussion with these two methods and direct comparison in terms of formulation. Especially, This work has no empirical comparison with these methods (that can be applied directly to the problem at hand).
- It is not straightforward to undertand the empirical results in Figure 3 with a little description on the experimental setup. The competitive advantage of FOML is to learn without task boundaries, but the problem still has task boundaries and the same partition as in the previous work  by Finn et al. (2019) to compare between online algorithms. In my opinion, even though the task information is not explicitly given for learning purposes but FOML is exposed with task information indirectly due to the setup described in Section 6, especially with the task partitions.
- There is a big question: why does TOE consistently perform poorly on average among all other methods? Why is this method performing really bad given the fact that it can access all past tasks? As mentioned in the paper, this approach trains the model using all tasks in the large buffer. 
- There is minimum information about LwF, iCARL, and MOCA on how they are implemented for the online meta-learning setup. Please elaborate on how to adapt these methods for the online meta-learning setting. The supplementary material also does not contain this information.
- Also, the use of a Siamese Network needs to be explained for reproducibility. Moreover, there are no citations for this type of network as well.
- There is an inconsistency in writing the proposed method. In formulation, there is a number $K$ for regularization but it is omitted in Algorithm 1.
- The paper needs some corrections:
-- Please fix the manuscript with numbered equations. The number stops at 5.
-- Typo and grammatical errors: varaint, a online data stream, evaulate.

clarity,_quality,_novelty_and_reproducibility: There some unclear messages in problem description, experiments, and comparison to SOTA. The novelty also needs to be justified by providing discussion and comparison with EWC and Reptile. There are some missing information needed to replicate the results from other methods and the proposed method.
summary_of_the_review: This paper has some potentials to address online meta-learning problems with no task boundaries. However, there are some unclear messages and problem descriptions in text. Also, there are some inconsistencies between the claims and the experiments to prove them.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method to tackle a new online meta learning setting where the task boundaries are not known (no access or costly to obtain) or do not exist (e.g. gradual change). 

Meta learning consists of the meta-training step and the adaptation step. During the adaptation step, instead of re-setting the task parameter to the meta-parameter at the beginning of the task (e.g. requiring to know the task boundary to know when to reset), the proposed method uses regularisation to pull parameter back gradually and softly (see Fig. 1) thus does not require to know the task boundaries. 

Experiments show that the proposed method can learn new tasks faster than the state-of-the-art online learning methods on Rainbow-MNIST, CIFAR100 and CELEBA datasets.
strength_and_weaknesses: Strength: 
1) it tackles a new problem (or at least in a new setting);
2) the proposed method makes sense;
3) presentation is clear;

Weakness:
1) more ablation study will be helpful particularly because this is a new setting. For example, an ablation study to show when task boundaries are accessible, can the proposed method discover/recover task boundaries well? One way to check is to see if the regularised pulled back task parameter intersects with the meta-parameter at or near the ground truth task boundaries. 
2) a couple of concrete real applications to show task boundaries are not accessible. Note task boundaries are different from distribution shifts, and in most cases the users know the task has changed. I struggle to come up with a real example of task boundaries not accessible in real applications. 
clarity,_quality,_novelty_and_reproducibility: Quality, clarify and originality of the work are excellent. 
summary_of_the_review: I am happy to accept the paper at the moment and open to change when I see other reviews and rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper tackles a novel problem of fully online meta-learning. The main difference of the proposed problem setting from the previous online meta-learning problem is that it does not assume clear boundaries between the tasks and the task distribution can gradually change over time. To tackle this new problem, the authors propose a meta-learning framework similar to Reptile with a data buffer, which regularizes the task-specific parameters to be close to the meta-parameters. The experimental validation of the method on multiple benchmarks verifies the effectiveness of the proposed method.

The paper received split reviews, with three of the reviewers recommending, or leaning toward rejection and one recommending an acceptance. The reviewers considered the tackled problem as interesting as well as challenging and relevant to many real-world problems, the proposed online meta-learning framework with no task boundary as somewhat novel, and the experimental validation as convincing.

However, the reviewers had the following critical concerns
- Although the paper proposes a novel problem setting, its definition is rather unclear - the role of the task boundaries and the gradual change of the tasks, are not well defined with formal descriptions.
- The proposed regularization-based meta-learning framework is similar to Reptile, but there is no discussion or comparison against it. 
- The experimental section does not provide much insight into how the proposed method works, as it is lacking in-depth analysis.
- There is minimal information of how the baselines are modified for the fully online meta-learning setting. 
- Despite the similarity of the proposed problem to task-free continual learning, there is no comparison against state-of-the-art continual learning methods.

The authors’ responses dealt with some of the issues, but did not provide clear answers to some of the questions (missing comparison with Reptile and task-free continual learning methods). Therefore, the reviewers remained unconvinced even after the rebuttal and kept their original ratings on the paper.  

I also find the paper not yet fully ready for publication - 1) it is unclear how the proposed problem differs from task-free continual learning (is it assuming a stationary task distribution as Reviewer a6XH mentioned?), 2) how the proposed method differs from Reptile besides the lack of task boundaries, 3) the paper provides little insights as to how the proposed method works. However the authors did not provide clear answers to these questions. Some of the arguments from the authors are also inaccurate, such as the statement that continual learning mostly focuses on catastrophic forgetting with little emphasis on rapid adaptation, or forward transfer. However, these have been important goals for continual learning since its inception. 
justification_for_why_not_higher_score: - The paper lacks clear descriptions of the problem setting, and how it differs from task-free continual learning. 
- There is little discussion of, and experimental comparison against relevant works on task-free continual learning and Reptile, which has a very similar objective.
Decision: Reject