Improving Meta-Continual Learning Representations with Representation Replay

Abstract: Continual learning often suffers from catastrophic forgetting. Recently, meta-continual learning algorithms use meta-learning to learn how to continually learn. A recent state-of-the-art is online aware meta-learning (OML). This can be further improved by incorporating experience replay (ER) into its meta-testing. However, the use of ER only in meta-testing but not in meta-training suggests that the model may not be optimally meta-trained. In this paper, we remove this inconsistency in the use of ER and improve continual learning representations by integrating ER also into meta-training. We propose to store the samples' representations, instead of the samples themselves, into the replay buffer. This ensures the batch nature of ER  does not conflict with the online-aware nature of OML. Moreover, we introduce a meta-learned sample selection scheme to replace the widely used reservoir sampling to populate the replay buffer. This allows the most significant samples to be stored, rather than relying on randomness. Class-balanced modifiers are further added to the sample selection scheme to ensure each class has sufficient samples stored in the replay buffer. Experimental results on a number of real-world meta-continual learning benchmark data sets demonstrate that the proposed method outperforms the state-of-the-art. Moreover, the learned representations have better clustering structures and are more discriminative.

Reviews and author responses:

The paper extends the online meta-learning (OML) model with the ER during the meta-training. Also, the paper proposes a replacement buffer policy for samples replacement from the reservoir. Instead of storing the raw samples, since the backbone model is static during the meta-test, it's better to store the feature representations that paper does.  1: Integrating the experience replay (ER) during the meta-training phase is the key contribution. The paper mentioned that using ER during meta-test only but not meta train creates inconsistency. What do you mean by inconsistency? During the meta-test and meta-train, the classes are disjoint; how consistency is defined?

2: The paper extends the OML with ER, which does not seem a significant contribution. The buffer replacement policy is novel, and it may be useful for the other replay based approaches. PSS improves the model performance compared to the Ring, GSS and Bilevel. 

3: In Algorithm-2, meta-training procedures are provided for the ER, while meta-test procedures are unclear. How are meta-tests performed? Would you please provide the Algorithm for the same?

4: The direct comparison with OML is not fair since OML does not leverage on ER. It seems that results are trivial since compared to without ER, replay always improves the performance. Also, since during the meta-test, only classifiers are trained, therefore it will be beneficial to store the feature representation is also trivial.

5: Another recent work [a] also improves the representation over the OML for continual learning; I request the author, please discuss the result with [a].

[a]: knowledge consolidation based class incremental online learning with limited data, IJCAI-21 Paper has some novel part but the main contributions compared to OML is not significant.


 Thanks for the response. The author partially addresses my concern, also compares the proposed model with the recent paper. 
Still, the lack of significant contributions is the main concern. I would like to upgrade my score but still, I believe that paper is not up to the publication stage. 

 Thank you very much for your response and answering my question. My concern regarding the significance of the contribution still remains and therefore not able to raise my score beyond the current recommendation. 

This paper proposes to  integrate ER into meta-training of online aware meta-learning (OML) Javed & White (2019). The authors propose to store the samples’ representations, not the samples themselves into a replay buffer.  In addition, the authors propose a sample selection method with a meta-learned Predictive Sample Selection to select the most informative examples. Experimental results on several datasets demonstrate the effectiveness of the proposed method.  Strengths:

* The authors proposed improvement for OML by integrating ER into  meta-training. 
* Experiment shows the effectiveness of the proposed method. 



Weakness:


* The technical novelty of the proposed method is limited and is incremental compared to online aware meta-learning (OML). The proposed method is just a simple improvement based on the framework of OML. The difference is that the proposed method  integrates ER  into the meta-training of OML 

* The clarity of presentation could be improved.  For example, the terms meta-training training, meta-training testing, meta-testing training are not comfortable to read. It would be better to use some common simple words that people widely used to improve the clarity of presentations. 


* For the proposed representation replay, there is a lack of concrete analysis why the proposed method of integrating ER into meta-training could better alleviate catastrophic forgetting compared to OML. It is unclear to see the benefit and necessity of using ER during meta-training only from the derived gradients. 

* Sample selection method lacks in-depth analysis about how different task samples are selected. What are the sample selection results for different seeds?  Is the proposed sample selection method sensitive to data streams or task ordering?

* For the experiment, lack of comparisons to some existing methods which consider the memory selection, e.g., [1]


Reference: 

[1] Online Continual Learning with Maximal Interfered Retrieval. Rahaf Aljundi, et al. NeurIPS 2019
 The technical contributions of this paper is limited compared to OML. Also, the presentation clarity, detailed analysis of the proposed method, experiments need to be improved. 

 Thanks for the authors' feedback, due to the limited nolvety and lack of concrete analysis, I can only increase the score to 5. 

 Thank you very much for your insightful feedback. We would like to address your concerns in the following.

---

**"The terms meta-training training, meta-training testing,meta-testing training are not comfortable to read"**:

These are standard terminologies that have been used in the meta-learning/meta-continual learning papers, such as (Kavumba et al., 2019), (Beaulieu et al., 2020), (Frikha et al., 2020), and (Wang et al., 2021).

---

**"What are the sample selection results for different seeds? Is the proposed sample selection method sensitive to data streams or task ordering?"**:

Note that the sample selection algorithm (Algorithm 3) does not involve any added randomness. We suppose the reviewer is talking about the meta-testing phase, which is repeated 50 times (as mentioned in Section C in the appendix). However, the FC in Algorithm 3, which is trained during meta-training, is fixed during meta-testing, and thus always produces the same output for a given sample. Therefore, as the samples will always have the same scores during meta-testing, the sample selection results for any given batch of samples will also be the same.

---

**"Lack of comparisons to some existing methods which consider the memory selection, e.g., (Aljundi et al., 2019)"**:

There might be some misunderstanding. MIR in (Aljundi et al., 2019) is used to select samples from the replay buffer into the replay batch, but not on selecting samples into the replay buffer (as is done by PSS). Thus, MIR and PSS are orthogonal, and so we did not compare in the experiments. In contrast, as shown in Figure 5, we have compared with the SOTA samples selection methods.

---

(Aljundi et al., 2019): Online Continual Learning with Maximal Interfered Retrieval, NeurIPS-19

(Frikha et al., 2020): ARCADe: A Rapid Continual Anomaly Detector, ICPR-20

(Kavumba et al., 2019): Learning to Learn to be Right for the Right Reasons, NAACL-HLT-21

(Wang et al., 2021): Do What Nature Did To Us: Evolving Plastic Recurrent Neural Networks For Task Generalization

 Thank you very much for your insightful feedback. We would like to address your concerns in the following.

---

**"Sample selection method lacks concrete analysis"**:

Performing a concrete analysis is very difficult for sample selection methods for the replay buffer because it will require the knowledge of exactly how much each sample benefits the model when it is being replayed to the model at each step of the training process, in order to compare and evaluate the sample selection methods. Therefore, other existing sample selection methods, such as (Rebuffi et al., 2017), (Isele and Cosgun, 2018), and (Aljundi et al., 2019), also do not have concrete mathematical analysis.

---

(Aljundi et al., 2019): Task-free continual learning, CVPR-19

(Isele and Cosgun, 2018): Selective experience replay for lifelong learning, AAAI-18

(Rebuffi et al., 2017): iCaRL: Incremental classifier and representation learning, CVPR-17



 We thank the reviewers for their insightful feedback. We would like to address the concerns raised by the reviewers in the following.

---

**"What do you mean by inconsistency? (How exactly it affects learning/performance) of using ER only during testing"**:

Experience replay (ER) helps the model to rehearse on past samples so that catastrophic forgetting may be alleviated. Therefore, when ER is used with OML in the meta-testing training phase, the PLN is allowed to review past tasks to improve the overall performance. Nevertheless, there exists inconsistency for OML, which means that the meta-testing training makes use of ER while the meta-training training does not. In the proposed method, ER is also used in the meta-training training. The meta-updates can then optimize the RLN to learn representations that are suitable for training with ER, leading to further improvements in the performance. 

---

**"Clarify the contributions"**:

In this paper, we identified the following issues:
1. OML does not make use of any replay strategies in the meta-training phase to overcome catastrophic forgetting in continual learning.
2. Reservoir sampling depends on randomness and does not select samples that are good for continual learning
	 to be stored in the replay buffer.

Ideas contributed:
1. Representation replay is applied to meta-continual learning in order to ensure the RLN receives online updates to learn better representations for each sample.
2. Predictive Sample Selection (PSS) is introduced to select the best samples to be stored in the replay buffer.

Evaluations:
1. The proposed OMREP outperforms SOTA meta-continual learning methods such as ANML, OML, and its variants.
2. The proposed PSS outperforms SOTA replay samples selection methods including reservoir sampling (Vitter, 1985), ring buffer (Chaudhry et al., 2021), greedy sample selection (GSS) (Aljundi et al., 2019), and bilevel optimization (Borsos et al., 2020).

---

**"Lack of experiments or theoretical analysis to support using representations instead of samples... Lack of concrete analysis why the proposed method of integrating ER into meta-training could better alleviate catastrophic forgetting compared to OML"**:

On theoretical analysis, Propositions 1 (in Section 3.1) shows that the meta loss’s gradient and Hessian contributions from the new sample $X_j$ may be canceled out by the replay samples in the same batch $B_j$.  Note that the replay batch $B_j$ contains samples from other tasks, thus is possible to contain samples that directly interfere with the learning of the new sample. Therefore, this hinders the meta-gradients to update the RLN to learn the representations for the new sample, in turn losing crucial information on a subset of samples in the meta-training set.

On the experimental results, Sections 4.2 shows that OMREP (which is using representations for replay) outperforms OMER (which is using samples for replay) on all three datasets. As illustrated in Section 4.3, this is due to the RLN being able to learn representations that have a better clustering structure and are more discriminative.

---

**"Are there methods other than ANML... Another recent work (Karim et al., 2021) also improves the representation over the OML for continual learning"**:

As suggested, we added a comparison with KCCIOL (Karim et al., 2021) in the following. Experiments are performed in the same setting as (Karim et al., 2021), and the results for KCCIOL are copied from (Karim et al., 2021). As can be seen, OMREP has much better performance than KCCIOL on both *Split-Omniglot* and *Mini-ImageNet*. This is because Representation Replay allows the RLN to learn better representations that are more suitable for training in conjunction with ER, and PSS further selects the best samples to be
stored in the replay buffer, thus allowing the performance to be improved significantly.

Classification accuracy (mean$\pm$std) averaged across 50 random meta-testing trajectories:

|Classes|ANML|OML|KCCIOL|OMREP|
|:-:|:-:|:-:|:-:|:-:|
|10|89.8 $\pm$ 4.4|93.9 $\pm$ 4.7|95.4 $\pm$ 3.4|**95.6 $\pm$ 2.9**|
|100|81.8 $\pm$ 2.0|82.3 $\pm$ 2.3|81.4 $\pm$ 2.3|**93.1 $\pm$ 1.1**|
|200|76.7 $\pm$ 1.9|75.7 $\pm$ 1.7|72.6 $\pm$ 1.5|**88.9 $\pm$ 1.2**|

Table 1: *Split-Omniglot*

| Classes|ANML|OML|KCCIOL|OMREP|
|:-:|-|-|-|-|
|8|18.5 $\pm$ 1.8|46.7 $\pm$ 3.5|45.3 $\pm$ 4.9|**56.3 $\pm$ 3.3**|
|10|17.0 $\pm$ 1.6|41.3 $\pm$ 3.0|42.7 $\pm$ 4.6|**49.0 $\pm$ 3.5**|
|12|16.2 $\pm$ 1.3|37.7 $\pm$ 2.4|39.4 $\pm$ 3.0|**44.4 $\pm$ 2.3**|
|14|15.4 $\pm$ 1.1|35.3 $\pm$ 1.7|37.0 $\pm$ 3.7|**39.9 $\pm$ 1.6**|
|16|14.2 $\pm$ 0.9|31.9 $\pm$ 1.1|33.4 $\pm$ 2.1|**34.9 $\pm$ 1.6**|
|18|13.6 $\pm$ 0.7|29.6 $\pm$ 0.8|31.9 $\pm$ 2.2|**35.6 $\pm$ 1.1**|
|20|13.1 $\pm$ 0.6|27.6 $\pm$ 0.6|29.7 $\pm$ 1.4|**33.9 $\pm$ 0.5**|

Table 2: *Mini-ImageNet*

---

(Karim et al., 2021): Knowledge Consolidation based Class Incremental Online Learning with Limited Data, IJCAI-21

 Thank you very much for your insightful feedback. We would like to address your concerns in the following.

---

**"Not using ER for RLN potentially lead to forgetting?"**:

Recall that the RLN is used to learn useful features. After these features are learned in meta-training, the RLN is frozen during meta-testing. Since the RLN parameters do not change throughout the meta-testing phase, the RLN does not forget the features learned from the meta-training, and thus it does not suffer from forgetting and does not require ER.

---

**"Representation replay still have the interference issue for PLN?... Is there a tradeoff that is being made here?"**:

When performing ER, training with the replay samples simulates learning with all the previously learned tasks in the PLN. Therefore, the PLN should not only learn with the new sample but also with the replay batch. This mixing of samples from the old and new tasks is intentional in order to overcome catastrophic forgetting when learning new tasks. Hence, there is no tradeoff here.

---

**"Isn’t the target for the prediction in PSS always shifting?"**:

Yes, the target for PSS is always shifting, which is desirable in the continual learning setting as the usefulness of samples varies throughout the training process, so selecting which samples to remain in the replay buffer should be dependent on the model's current state. 

---

**"What is the rationale behind not updating the FC during meta-testing training?"**:

Recall that in the FC update (Section 3.2), one has to evaluate $\ell_R$  (which is the model's loss on class $Y_r$ when trained on the current batch of representations $R$ with $(f_\theta(X_r); Y_r)$). While we have access to all the meta-training samples belonging to class $Y_r$ in the meta-training phase, the meta-testing training samples arrive sequentially and are not stored as in the continual learning setting. Thus, the loss $\ell_R$ on class $Y_r$ cannot be evaluated to update the FC. 

---

**"Wouldn't the loss for the same sample change over the course of meta-testing training?"**:

Yes, for a  given sample, it is possible for its loss to change over the course of meta-testing training, which would mean that the sample is relatively more useful at different parts of the training procedure. This is indeed beneficial as samples are continually arriving and the model is continually updated in the continual learning setting.

---

**"Would having multiple, different (batches of data) updates from the R + new sample alleviate the problem of losing the information from the new sample?"**:

As suggested by the reviewer, we added a variant (denoted "OMER(Multiple)") which uses three separate replay batches along with the new sample at each update in the meta-training training phase. The experiment is performed on the *CIFAR-FS* dataset. Results are shown in the figure in this URL: https://i.ibb.co/ygqqxFD/cifar-multiple.png (which corresponds to Figure 3(c) in the submission). As can be seen, having multiple batches (OMER(Multiple)) leads to little improvement over OMER. Thus, the problem of losing information from the new sample cannot be alleviated with the use of multiple batches.

 Thank you very much for your insightful feedback. We would like to address your concerns in the following.

---

**"Meta-test procedures are unclear"**:

As mentioned in Section 2.3 of the submission: "The *meta-testing training* phase uses the training set in $T$, with the same procedure as in meta-training training (online updating the PLN by each sample in $T$ once). The model is then evaluated in *meta-testing testing* using the test set of $T$."

To improve clarity, the pseudo-code for meta-testing will be added to the appendix in the revised version.

---

**"Direct comparison with OML is not fair since OML does not leverage on ER... Replay always improves the performance"**:

We did have compared with OML(ER), which is OML using ER in meta-testing. However, as shown in Figure 2, OML(ER) does not have a good performance on the more complex datasets (*mini-ImageNet* and *CIFAR-FS*), and so including ER does not always improve performance. Moreover, we also compared with a naive integration of OML with ER during meta-training (as introduced in Section 3.1.1). As shown in Figure 3, the resultant method (OMER) also has a worse performance when compared to the proposed OMREP.

---

**"Since during the meta-test, only classifiers are trained, therefore it will be beneficial to store the feature representation is also trivial"**:

Note that storing the feature representation during meta-testing has been done in OML (and is called OML(ER) in the experiments), and that is **NOT** our contribution. Indeed, as suggested by the reviewer, storing the feature representation is trivial to meta-testing, because it is exactly the same as replaying the inputs as the RLN is frozen (only classifiers are trained). However, it is not trivial to meta-training (which is what we are addressing in this paper) since it enables online training for the RLN, which is the key to learning good representations for the samples in OML.

The paper looks at the problem of catastrophic forgetting in continual learning. They propose a method that extends OML to use ER both during training and testing. In the ER, they store the embedding of the samples instead of the samples themselves to avoid the interference caused by the batch nature of ER and the online nature of OML. They use meta-learning to select samples that are useful for avoiding catastrophic forgetting to be in the ER instead of reservoir sampling. They predict the difference in loss in predicting the class caused by removing a sample from replay and use that score to decide what to remove. Strengths
1. The paper identifies some issues with one of the current methods for continual learning and propose simple extensions to reduce the effect of those issues
2. The empirical evaluations/visualizations show clear improvement in performance by adding the proposed ideas
3. The paper is written very well. Easy to read and understand.

Questions/Weaknesses
1. Could you elaborate a bit more on the inconsistencies (how exactly it affects learning/performance) of using ER only during testing? Is the inconsistency of concern here the inconsistency in inner loop updates between the meta-training and meta-testing?
2. [Representation replay] 
a. Wouldn’t not using ER for RLN potentially lead to forgetting?
b. Wouldn’t representation replay still have the interference issue for PLN? 
c. Is there a tradeoff that is being made here? If so any thoughts on what exactly it is and how to choose that tradeoff?
3. Isn’t the target for the prediction in PSS always shifting? If thats the case, what is the rationale behind not updating the FC during meta-testing training? Wouldn't the loss for the same sample change over the course of meta-testing training?
4. In ER, Would having multiple, different (batches of data) updates from the R + new sample alleviate the problem of losing the information from the new sample? 
5. Not clear if the contributions of the paper are significant enough for ICLR  or very incremental. Could you clarify the contributions of the papers in more detail please, including the issues identified, the ideas contributed (novel and point out where ideas were borrowed and applied in a different setting) and evaluations etc.
6. Are there methods other than ANML for the setup of interest in the paper following OML which was proposed in 2019? While evaluating the proposed changes with OML and variations of OML is essential, it is also important to compare with other methods that look at the same setting to situate the proposed method in the literature. 

Other Comments
1. [Related works section] Note that apart from regularization and rehearsal based methods, there are also several architecture based (modular networks type and parameter isolation methods) that have been proposed for avoiding catastrophic forgetting in continual learning.
 Simple extension to a current method. Unclear about the significance of the contribution. 

The paper studies the meta-continual learning problem. Representations replay instead of samples replay are used to improve the performance of meta-continual learning. And a new strategy called Predictive Sample Selection is used to select samples into the replay buffer. Experiments demonstrate the effectiveness of the proposed methods.  Strengths:
1. The idea of using representations instead of samples for buffer is interesting. 
2. The Predictive Sample Selection method is effective.

Weaknesses:
1. Lack of experiments or theoretical analysis to support using representations instead of samples. 
2. Sample selection method lacks concrete analysis. The topic is intersting. However, the contribution of this paper is minor. It is a combination of several incremental improvements.

Meta-review: Addressing the problem of catastrophic forgetting in continual learning, this paper extends OML to use experience replay (ER) during training, instead of the original approach which uses ER during test phase only. The paper proposes a policy for samples replacement from the reservoir. Experiments show the superiority of the approach in three standard benchmarks compared with several baselines. 

Reviewers were unanimously concerned that the technical contribution of the paper is not sufficient. The authors addressed several issues, including experiments to compare with additional baselines, but the technical novelty remains limited for an ICLR publication. 
The paper cannot be accepted at its current form.

Decision: Reject