Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper extends the online meta-learning (OML) model with the ER during the meta-training. Also, the paper proposes a replacement buffer policy for samples replacement from the reservoir. Instead of storing the raw samples, since the backbone model is static during the meta-test, it's better to store the feature representations that paper does. 
main_review: 1: Integrating the experience replay (ER) during the meta-training phase is the key contribution. The paper mentioned that using ER during meta-test only but not meta train creates inconsistency. What do you mean by inconsistency? During the meta-test and meta-train, the classes are disjoint; how consistency is defined?

2: The paper extends the OML with ER, which does not seem a significant contribution. The buffer replacement policy is novel, and it may be useful for the other replay based approaches. PSS improves the model performance compared to the Ring, GSS and Bilevel. 

3: In Algorithm-2, meta-training procedures are provided for the ER, while meta-test procedures are unclear. How are meta-tests performed? Would you please provide the Algorithm for the same?

4: The direct comparison with OML is not fair since OML does not leverage on ER. It seems that results are trivial since compared to without ER, replay always improves the performance. Also, since during the meta-test, only classifiers are trained, therefore it will be beneficial to store the feature representation is also trivial.

5: Another recent work [a] also improves the representation over the OML for continual learning; I request the author, please discuss the result with [a].

[a]: knowledge consolidation based class incremental online learning with limited data, IJCAI-21
summary_of_the_review: Paper has some novel part but the main contributions compared to OML is not significant.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper looks at the problem of catastrophic forgetting in continual learning. They propose a method that extends OML to use ER both during training and testing. In the ER, they store the embedding of the samples instead of the samples themselves to avoid the interference caused by the batch nature of ER and the online nature of OML. They use meta-learning to select samples that are useful for avoiding catastrophic forgetting to be in the ER instead of reservoir sampling. They predict the difference in loss in predicting the class caused by removing a sample from replay and use that score to decide what to remove.
main_review: Strengths
1. The paper identifies some issues with one of the current methods for continual learning and propose simple extensions to reduce the effect of those issues
2. The empirical evaluations/visualizations show clear improvement in performance by adding the proposed ideas
3. The paper is written very well. Easy to read and understand.

Questions/Weaknesses
1. Could you elaborate a bit more on the inconsistencies (how exactly it affects learning/performance) of using ER only during testing? Is the inconsistency of concern here the inconsistency in inner loop updates between the meta-training and meta-testing?
2. [Representation replay] 
a. Wouldn’t not using ER for RLN potentially lead to forgetting?
b. Wouldn’t representation replay still have the interference issue for PLN? 
c. Is there a tradeoff that is being made here? If so any thoughts on what exactly it is and how to choose that tradeoff?
3. Isn’t the target for the prediction in PSS always shifting? If thats the case, what is the rationale behind not updating the FC during meta-testing training? Wouldn't the loss for the same sample change over the course of meta-testing training?
4. In ER, Would having multiple, different (batches of data) updates from the R + new sample alleviate the problem of losing the information from the new sample? 
5. Not clear if the contributions of the paper are significant enough for ICLR  or very incremental. Could you clarify the contributions of the papers in more detail please, including the issues identified, the ideas contributed (novel and point out where ideas were borrowed and applied in a different setting) and evaluations etc.
6. Are there methods other than ANML for the setup of interest in the paper following OML which was proposed in 2019? While evaluating the proposed changes with OML and variations of OML is essential, it is also important to compare with other methods that look at the same setting to situate the proposed method in the literature. 

Other Comments
1. [Related works section] Note that apart from regularization and rehearsal based methods, there are also several architecture based (modular networks type and parameter isolation methods) that have been proposed for avoiding catastrophic forgetting in continual learning.

summary_of_the_review: Simple extension to a current method. Unclear about the significance of the contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the meta-continual learning problem. Representations replay instead of samples replay are used to improve the performance of meta-continual learning. And a new strategy called Predictive Sample Selection is used to select samples into the replay buffer. Experiments demonstrate the effectiveness of the proposed methods. 
main_review: Strengths:
1. The idea of using representations instead of samples for buffer is interesting. 
2. The Predictive Sample Selection method is effective.

Weaknesses:
1. Lack of experiments or theoretical analysis to support using representations instead of samples. 
2. Sample selection method lacks concrete analysis.
summary_of_the_review: The topic is intersting. However, the contribution of this paper is minor. It is a combination of several incremental improvements.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to  integrate ER into meta-training of online aware meta-learning (OML) Javed & White (2019). The authors propose to store the samples’ representations, not the samples themselves into a replay buffer.  In addition, the authors propose a sample selection method with a meta-learned Predictive Sample Selection to select the most informative examples. Experimental results on several datasets demonstrate the effectiveness of the proposed method. 
main_review: Strengths:

* The authors proposed improvement for OML by integrating ER into  meta-training. 
* Experiment shows the effectiveness of the proposed method. 



Weakness:


* The technical novelty of the proposed method is limited and is incremental compared to online aware meta-learning (OML). The proposed method is just a simple improvement based on the framework of OML. The difference is that the proposed method  integrates ER  into the meta-training of OML 

* The clarity of presentation could be improved.  For example, the terms meta-training training, meta-training testing, meta-testing training are not comfortable to read. It would be better to use some common simple words that people widely used to improve the clarity of presentations. 


* For the proposed representation replay, there is a lack of concrete analysis why the proposed method of integrating ER into meta-training could better alleviate catastrophic forgetting compared to OML. It is unclear to see the benefit and necessity of using ER during meta-training only from the derived gradients. 

* Sample selection method lacks in-depth analysis about how different task samples are selected. What are the sample selection results for different seeds?  Is the proposed sample selection method sensitive to data streams or task ordering?

* For the experiment, lack of comparisons to some existing methods which consider the memory selection, e.g., [1]


Reference: 

[1] Online Continual Learning with Maximal Interfered Retrieval. Rahaf Aljundi, et al. NeurIPS 2019

summary_of_the_review: The technical contributions of this paper is limited compared to OML. Also, the presentation clarity, detailed analysis of the proposed method, experiments need to be improved.
review_title: null

Meta-review: Addressing the problem of catastrophic forgetting in continual learning, this paper extends OML to use experience replay (ER) during training, instead of the original approach which uses ER during test phase only. The paper proposes a policy for samples replacement from the reservoir. Experiments show the superiority of the approach in three standard benchmarks compared with several baselines. 

Reviewers were unanimously concerned that the technical contribution of the paper is not sufficient. The authors addressed several issues, including experiments to compare with additional baselines, but the technical novelty remains limited for an ICLR publication. 
The paper cannot be accepted at its current form.
Decision: Reject