Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose FOML, a new online learning algorithm based on MAML. It introduces two new features. 1. Instead of resetting the inner loop weights back to the meta-weights after each task, the online weights are always kept and updated. 2. It introduces a regularizer that aims to keep the online weights always close to the meta-weights and the other way around. Rainbow MNIST and online CIFAR100  experiments show that FOML outperforms TFS, TOE, FTL, and FTML. In ablation experiments, the authors show that FOML woks better for higher numbers of inner update iterations as well as the importance of the meta-update.
main_review: # Strengths
* The proposed method is simple and it performs better than FTML and other previous approaches.
* In general, the text is clear and easy to read.
* The authors provide an Algorithm.
* Implementation details are provided in the Appendix.

# Weaknesses
* Since FOML continuously updates the online weights, it is not clear what would happen if an OOD task is encountered. It would be interesting to see some ablation experiment or some discussion about this scenario, as well as possible ways to deal with it.
* Some examples of non-cited works that tackle this problem are [A] or [B], which try to address this problem and also introduce new versions of MAML. In fact, it seems like [A] could be highly related to your approach. In their case, rather than optimizing a regularizer, they optimize the inner loop learning rate, which in the end, has the effect of preventing the online weights from drifting too much when the new encountered task has a high interference. 

[A] Gupta, Gunshi, Karmesh Yadav, and Liam Paull. "La-maml: Look-ahead meta learning for continual learning." arXiv preprint arXiv:2007.13904 (2020).
[B] Caccia, Massimo, et al. "Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning." Advances in Neural Information Processing Systems 33 (2020).


## Typos
* Pag 4. algorithms tries -> algorithms try
* Pag 4. with a pretrained weights -> with pretrained weights
* Pag 5. necessarily results -> necessarily result
* Pag 5. we will B to denote -> we will use B to denote
* Pag 7. changings -> changing
* Pag 8. see Appendix ??
* Pag 9. depends on -> depend on
summary_of_the_review: The authors present a simple approach that is sound, effective in a specific online continual learning scenario, where a replay buffer is kept, and new tasks are not completely unrelated to previous tasks. On the other hand, the authors only provide experiments on MNIST and CIFAR, and do not provide any additional information on how their algorithm would handle more challenging situations. 

Overall, the proposed method is interesting for the research community but the authors should provide more information on how it would behave under more challenging scenarios, discuss its limitations, and include in the text a more complete comparison with other setups that justifies for a non-expert reader why they did not compare with Harrison et al. 2019, He et al. 2019, [A] ...
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends meta-learning algorithm for continual online learning. Different from previous meta-learning based online continual learning, this algorithm removes assumption of knowing task boundaries of data beforehand. The proposed approach extends the previous online MAML (FTML), by removing the discrete 'resets' of task-specific parameters. One key difference besides removing reset is to change the replay buffer (memory) strategy used by meta-updates, where prior approach always sample data of all previous tasks and the proposed approach samples all data points seen so far (so there is no need to know which task we are currently solving).

As a result, this adapted algorithm cope with data of continuous task boundaries. To validate the proposed approach, authors conducted experiments on a synthetic continual image recognition benchmark (of 1,200 tasks) based on CIFAR100, where they outperforms previous continual meta-learning based approach.
main_review: Strength: 
1. Overall speaking, the writing is clear for people to understand the motivation, the background, the proposed method, and experiments. 
2. The algorithmic changes to previous FTML are incremental (to suit the new assumption) but the justification behinds these design choices are reasonable. 
3. Experiments on synthetic datasets validated the design choices to some degree, as the proposed approach achieves better fitting performances.

Weakness/Question to Authors (order does not matter):
- I am less familiar with the literature and would like to understand the evaluation metric on both datasets. How was the error rate / test error being calculated? Are they essentially the regret that represents your fitting performance? Do you also measure backward transfer performances and forward transfer performances? 
- In meta-update, with a large K, you essentially need to store K copies of all network parameters? Then how is the storage cost comparing your method to previous approaches?
- Why don't you also compare to other non-meta-learning based online continual learning? This would help people to understand the status quo of meta-learning based online continual learning. It is okay even though you can not outperform them.
- What would be the optimal fitting performance? Say, for all  the1200 tasks, you first perform offline batch training on all of them sufficiently, and then fine-tune on each of those 1200 tasks? How close would your model's performance to this optimal performance (of the same model achitecture)?
- Did you compare different strategy for the buffer, such as reservoir buffer vs. fIFO buffer vs other buffer?
- Why don't you consider realistic online continual learning datasets, such as [1] and [2]?

  - [1] Drinking from a Firehose: Continual Learning with Web-scale Natural Language. 
  - [2] Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data

Minor Comments: 
1. Section 1 Paragraph 1: "… simultaneously that data for each new task is also used" missing verb after simultanously?
2. Section 1 Paragraph 1: "fall short of the goal of creating an effective and adaptation system" -> "effective adaptation system"?
3. Section 3 Paragraph of "Online Learning": I am not sure why loss function is also a sequence of time step. Then, what would the loss function be if there is an concrete example. Isn't the loss function determined by the data (and label)? Maybe I misunderstand?
4. Equation of FTL, maybe it would be more clear to reader if you rewrite phi as phi_{TOE}^{t-1}.
5. Figure 3. Does not your approach called FOML? The method name in the legend is OML
6. Implementation Details. Broken appendix hyperlink.
summary_of_the_review: I am not an expert in this area so I believe that my opinion should be discounted. Based on my understanding, I think this is a solid and well-written paper, with extensive results that demonstrates the improvement of presented approach to its comparing prior work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an online learning approach that utilizes the meta-learning framework. The proposed method - FOML has two main components: a regularizer that does not allow the updated parameters of the model to deviate too far from the previously learned set and a meta update on the parameters using the online updates, and a validate set drawn from the buffer of examples seen in the past. FOML is evaluated on two datasets and compared against a few other similar approaches and baselines. The results suggest faster adaptation of FOML compared to previous methods.

main_review: Strengths

The approach is quite simple and intuitive. 

The paper is written clearly and is easy to understand.

Concerns

Lack of clarity on the problem statement: The authors cast the problem as meta-learning that must be done online. I am not convinced with this problem statement. As I understand, the goal in meta-learning (particularly the MAML framework adopted by the authors) is to learn a good initialization for quickly adapting to a new task. However, the current problem seems to be more on the lines of online learning and avoiding catastrophic forgetting, as there is no adaptation step after meta-training. The authors explicitly mention that their goal is not to solve catastrophic forgetting. But then, what is the goal? If I am mistaken, and there is an adaptation step(akin to meta-test), it would be interesting to see the number of adaptation steps required by the various models to adapt to the new task. 

Missing literature: Online learning/continual learning is a well-studied research area. There is a large body of literature on regularization-based methods (e.g., Kirkpatrick et al. 2017, Ritter et al., 2018), memory-based methods, online continual learning (Caccia et al. 2020), etc. I find the discussion on the related work to be quite narrow, missing out on approaches trying to achieve similar goals. Furthermore, evaluating these related works over and above the few methods listed in the experiments is also important.

Weak experimental setup: The specially curated datasets (and the tasks) appear quite simple (even the online-CIFAR100 task). Existing literature has used more complex sequences of tasks (from miniimagenet, tieredimagenet). Experiments on these datasets are warranted to conclude FOML’s efficacy.

Finally, the lack of theoretical justification on the effectiveness of the FOML is also a cause of concern.  

Minor comments

Page 5 last paragraph - To this end, we well \script{B} to denote - we use \script{B} to denote.

Implementation Details Section on Page 8 - Appendix information is missing.

summary_of_the_review: Overall, while the paper is well-written, I have doubts about the motivation behind the problem statement. I am concerned about the missing relevant literature and inadequate experiments.

--- Post rebuttal update

I thank the authors for the detailed response to clarify some of my concerns. I also read the other reviews and the response from the authors. The authors have clarified the existence of the problem statement citing prior work. I agree with the existence of the problem statement. However, I am not entirely convinced with the motivation. Experiments on real-world applications requiring online meta-learning would have significantly strengthened the contribution. I acknowledge that the experiments comparing other online-learning algorithms (such as LwF, iCaRL) and algorithms such as MOCA is a step in the right direction, but requires more analysis (using more datasets).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an online meta learning method which can remove the discrete task boundaries. Different from existing online meta-learning method, FOML utilizes online learning to remove the task boundaries. 

The FOML is able to continually update online parameters with new datapoint, and meanwhile perform meta-gradient updates on a set of meta-parameters using a buffer of previous data. The authors compare the model to state-of-the-art like FTML on benchmarks such as Rainbow-MNIST and CIFAR100 datasets. Experimental results show that FOML learns to adapt faster and can achieve lower error rates on classification tasks. However, it lacks sufficient experiments and comparison with baseline methods to support the claim that FOML can learn without the task boundaries.

main_review: Strengths

1. The idea uses online learning to remove the task boundary is interesting
2. The paper is reasonably written with clear background and diagrams for the overall architecture. It’s well written and easy to follow. 
3. The experiments demonstrate that FOML outperformed FTML

Weakness
1. The idea is not new, authors are missing some closely relevant and major literature in online meta-learning [1] and meta-learning without task boundaries.[2]
2. I might have misunderstood the definition of ‘reset’ in the last paragraph of Section 4; as stated FTML needs to be reset at each task and the meta-training process does not influence the current task at all; But in algorithm 1 of the paper FTML, the model uses the result of meta-update to evaluate performance at data points within round t, and meanwhile the result of the adaptation process is recorded. In algorithm 1, why calculate predictions on the train set and validation set? Are they for loss? I do not see the utilization of them. And in lines 9 and 12, what’s the meaning of \L_{task}? Is this loss related to specific tasks?  The FTML uses \phi to meta-update \theta and then uses the \theta to update \phi. The proposed method utilizes the \theta to regularize and update \phi, and then update \theta by \phi. It is not clear to me how the task boundaries are removed. More explanations are expected.  
3. The experiments are weak. For example, there is no performance evaluation of the proposed FOML in Figure 3 which makes your claim unsupportive. I assume the OML represents the FOML? I’m not sure about that.  
4. According to my previous point, important baselines are missing, the work [2] also targets meta-learning without the task boundaries.
5. Is there any mathematical guarantee to show that FOML can learn without task boundaries? If not, more experiments on extra datasets are expected to support this claim.  
6. Some minor presentation issues, e.g., 
- In Implementation Details, there is an error of “Please see Appendix ??”
- For figure 3, the method name should be FOML rather than OML in the diagram.
- In A.2 Implementation Details-Online CIFAR100, all the images should from ‘CIFAR100’ instead of Rainbow-MNIST.
- For the caption of figure 1, online parameters should be \phi and meta-parameters should be \theta.


Reference

[1] Denevi, G., Stamos, D., Ciliberto, C. and Pontil, M., 2019. Online-within-online meta-learning. In ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019) (Vol. 32, pp. 1-11). Neural Information Processing Systems (NeurIPS 2019).
[2] Harrison, J., Sharma, A., Finn, C. and Pavone, M., 2020. Continuous Meta-Learning without Tasks. Advances in Neural Information Processing Systems, 33.

summary_of_the_review: This paper proposes a method named FOML which is a further step of the existing method FTML that tries to remove the task boundary. However, such an idea is not new and there is already a similar work that exists. Experiments are not strong enough to support the claim.
review_title: null

Meta-review: The paper propose a Fully Online Meta-Learning (FOML) method which extend MAML for continual learning in a fully online learning  without requiring the knowledge of the task boundaries. Experiments show that FOML was able to learn new tasks faster than several existing online learning methods on Rainbow-MNIST, and CIFAR100 datasets. 

There are a few major concerns from reviewers. One concern is about the lack of clarity on the problem statement: The authors cast the problem as meta-learning that must be done in a fully online setting, but it requires to store all the training data in a buffer storing all the training data seen so far, which contradicts to the principle of “online learning”. Another major weakness is the poorly written literature survey, which missed to cite a large body of related work in continual learning and online-meta-learning (such as Online Continual Learning, task-free continual learning, continual learning without task boundaries, etc). These should at least be discussed carefully if not fully compared in the empirical studies. Also experiments are quite weak in both settings, datasets and rather out-of-date baselines. Finally, there also lacks of theoretical justification or analysis. 

Therefore, the paper is not recommended for acceptance in its current form. I hope authors found the review comments informative and can improve their paper by addressing these review comments carefully in future submissions.
Decision: Reject