Title: Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
Authors: Elahe Arani
Fahad Sarfraz
Bahram Zonooz
Venue: ICLR 2022
Keywords: Continual Learning
Catastrophic Forgetting
Complementary Learning Systems Theory
Experience Replay

Abstract: Humans excel at continually learning from an ever-changing environment whereas it remains a challenge for deep neural networks which exhibit catastrophic forgetting. The complementary learning system (CLS) theory suggests that the interplay between rapid instance-based learning and slow structured learning in the brain is crucial for accumulating and retaining knowledge. Here, we propose CLS-ER, a novel dual memory experience replay (ER) method which maintains short-term and long-term semantic memories that interact with the episodic memory. Our method employs an effective replay mechanism whereby new knowledge is acquired while aligning the decision boundaries with the semantic memories. CLS-ER does not utilize the task boundaries or make any assumption about the distribution of the data which makes it versatile and suited for ``general continual learning''. Our approach achieves state-of-the-art performance on standard benchmarks as well as more realistic general continual learning settings.


Review discussion:
Strengths:
1. The proposed method is well motivated and novel in the context of continual learning. Maintaining multiple models for different purposes is interesting and is shown to be helpful for continual learning.
2. The writing and organization is clear. It is easy to understand the proposed algorithm and the details.
3. Different continual learning settings are considered, the proposed approach shows better results across all the settings. The analysis of different models also make it easy to understand how the method works.
Weaknesses:
1. One of the main drawbacks of the proposed approach is that three models need to be maintained during training which incur a heavy cost.   One question is that if reducing the individual model size while keeping the whole model size the same, what's the performance in this case?
2. The proposed method has a lot of hyperparameters to tune, which may be a problem if the method is applied to a different task.
More questions:
1. Why update the plastic and stable models stochastically?
2. In table 1, why CLS-ER is better than JOINT on S-MNIST? JOINT should be the upper bound performance?
3. The authors make a connection to flat minima, if explicitly enforcing flat minima, would the proposed method can still do better?
NO.
Thank you for your encouraging remarks and valuable suggestions. We are glad you found it interesting and hope to address your queries below.
> **One of the main drawbacks of the proposed approach is that three models need to be maintained during training which incur a heavy cost. One question is that if reducing the individual model size while keeping the whole model size the same, what's the performance in this case?**
We acknowledge that the memory complexity of our method is higher because of the additional cost of maintaining the semantic memories. Thank you for the valuable suggestion, we will explore this idea in addition to more efficient ways of modeling the semantic memories. (Please see the Q1 response to Reviewer MUyX).
> **The proposed method has a lot of hyperparameters to tune, which may be a problem if the method is applied to a different task.**
This is a valid concern and while our method does have a number of hyperparameters, we believe that the complementary nature of the individual components in our method and its ability to attain similar performance for different choices of hyperparameters,  reduces the required fine-tuning effort considerably.
Specifically, Section B.5 shows that it is not highly sensitive to the particular choice of hyperparameters as different settings can attain similar performance.
Also, because of the complementary nature of the components, we can keep the majority of the parameters fixed (e.g. $\lambda$, $\alpha_S$, $\alpha_S$ and $r_S$) and only finetune the remaining parameters (e.g. $r_P$) which facilitates hyperparameter tuning significantly. Table S.4 further shows that for the vast majority of our experiments we use the same parameters for different buffer sizes.
> **Why update the plastic and stable models stochastically?**
Thank you for bringing this point, we will provide detail in the revision. Updating the semantic memories stochastically compared to a deterministic approach (e.g updating the plastic model every 3rd epoch and stable model every 6th iteration) reduces the overlap in the snapshots of the working model and leads to more diversity in semantic memories and hence higher complementary knowledge. Interesting, noise is a salient component of the computational strategy of the brain [1] and permeates every level of the nervous system [2] which makes stochastic updates more biologically plausible. Empirically we also found that stochastic update performs better than fixed periodic update.
> **In table 1, why CLS-ER is better than JOINT on S-MNIST? JOINT should be the upper bound performance?**
This is a good point and it is indeed the case that JOINT acts as the upper bound and our method achieves performance close to JOINT only for the simpler MNIST settings. For S-MNIST, while the average for CLS-ER is only slightly higher, it is within 1 standard deviation of the JOINT performance. We also conjecture that perhaps because of the relative simplicity of the task, with a higher buffer size (i.e. 5120), there might be sufficient samples to learn a good classifier and our method provides better regularization to avoid overfitting and hence leads to slightly higher generalization.
> **The authors make a connection to flat minima, if explicitly enforcing flat minima, would the proposed method can still do better?**
Thank you for the suggestion. We are not aware of any method that explicitly enforces flatter minima besides perhaps higher entropy regularization[3]. We believe that additional regularizations on top of our method can be an interesting avenue for future research.
Please let us know if we failed to address your concerns or if you require further information or explanation.
**References:**
[1] Maass, Wolfgang. "Noise as a resource for computation and learning in networks of spiking neurons." Proceedings of the IEEE 102.5 (2014): 860-880.
[2] Faisal, A. Aldo, Luc PJ Selen, and Daniel M. Wolpert. "Noise in the nervous system." Nature reviews neuroscience 9.4 (2008): 292-303.
[3] Chaudhari, Pratik, et al. "Entropy-sgd: Biasing gradient descent into wide valleys." Journal of Statistical Mechanics: Theory and Experiment 2019.12 (2019): 124018.
The method is very simple to understand and mimicking the CLS theory with the hippocampus and neocortex is interesting.
Compared to existing methods, is it a fair comparison in terms of training cost such as using two models including plastic and stable models and using reply data to train the working model?
I think more analysis about plastic and stable models are needed because these model act like hippocampus and neocortex based on CLS theory. What is optimal semantic? Just select the logit with the highest softmax between the two models?
NO.
Thank you for your feedback. We are glad that you found the idea of incorporating the hippocampus and neocortex into DNNs interesting as it was the main goal of our study to show the potential of mimicking the complementary learning system for enabling efficient continual learning. We hope to address your concerns below.
> **Q1: Compared to existing methods, is it a fair comparison in terms of training cost such as using two models including plastic and stable models, and using reply data to train the working model?**
We acknowledge that the memory complexity of our method is higher because of the additional cost of maintaining the semantic memories. This is, however, still more efficient compared to some knowledge distillation or a mixture of experts approaches that maintain a separate network for each task [1] or dynamic neural networks where the network parameters grow with the tasks. Our method maintains a constant size and can scale to a large number of tasks.
Also, we would like to emphasize that while the training cost is higher, our method does not require any additional computations for inference as only the stable model is used, requiring a single forward pass similar to the other methods. Our goal was to bridge the gap between DNN and the brain which utilizes multiple memory systems and show the potential benefits of incorporating such a dual memory architecture. In future work, we aim to further explore the design space for brain-inspired algorithms for mimicking the hippocampus and neocortex considering both the faithfulness to the brain and the energy efficiency.
> **Q2: I think more analysis about plastic and stable models are needed because these model act like hippocampus and neocortex based on CLS theory. What is optimal semantic? Just select the logit with the highest softmax between the two models?**
We apologize for the confusion and will provide more intuition in the revision. The plastic and stable model aim to mimic the rapid and slow adaptation of information respectively which we aim to achieve through the design of exponential moving averaged models with different update frequencies and window sizes. Section B.2 and Figures S1 and S2 in the supplementary, which show the task-wise performance of each of the models as training progresses, demonstrate empirically that our approach does indeed achieve the intended purpose.
Optimal semantic information refers to the consolidated structural knowledge encoded in the model logits for the efficient replay which account for the consolidation of feature space and adaptation of the decision boundaries of the previous tasks compared to the sub-optimal local logits in DER++. We highlight the benefits of using these consolidated logits instead of the local logits in DER++ in our results section and provide intuition for how they add value in each of the evaluation settings.
We aim to extract the optimal consolidated logits for replay for a memory sample, which by design would come from the stable model for a memory sample from an earlier task and the plastic model for a recent task (as can be seen in the task-wise performance of the semantic memories). In line with our goal to provide a general incremental learning method, we avoid using the task boundaries to guide our selection and instead use the simple approach of utilizing the performance of the semantic memories as the selection criterion which works well in practice as our method is able to achieve the intended distribution of performance with the stable and plastic model.
Further research into the selection criterion is an interesting research avenue for future work.
We hope to have addressed your concerns and would be happy to provide more details.
I enjoyed reading this publication. I'm not very familiar with the latest CL methods but the authors address various related works and evaluate all models thoroughly. I think the setting without task boundaries is harder and more interesting since the real world also doesn't come with clear task boundaries. The paper is following recent work towards robust and extensive evaluation which is more difficult but will help the CL community to make real progress.
The empirical upper bound "JOINT" is very instructive and highlights that there is still room for progress.
That said, the JOIN performance seems low given the excellent performance of other models on MNIST, CIFAR, Imagenet datasets. Is this because the models are very small? Does CLS-ER work also with larger models?
The flat mima discussion is less essential but is probably missing a reference to Flat Minima, Hochreiter et al 1997.
Instead of the interesting section 6 I'd have preferred some ablation experiments (though the component analysis is appreciated). E.g. are two teacher networks necessary? How much worse is the model if one teacher network is used (with a tuned rate)? What if there were three networks (each with their own rate)?
I did not understand how samples are removed from the buffer (the episodic memory). Are samples removed eventually? If yes, how and where is that explained? If not, what is the space complexity of algorithm 1?
NO.
Thank you for your time and valuable feedback. It is quite encouraging that you enjoyed reading our work and found the evaluation convincing. We hope to address your concerns below.
> **the JOIN performance seems low given the excellent performance of other models on MNIST, CIFAR, Imagenet datasets. Is this because the models are very small? Does CLS-ER work also with larger models?**
Thank you for pointing this out, it would be great though if you could clarify a bit on the reference models so that we could provide a more specific response as there are several factors that can affect the model performance, including model size, training scheme, data augmentations, and regularization techniques.
For a fair comparison, we keep uniform training settings so that we can disentangle the effect of the continual learning method from these other factors. We follow the experimental settings (model and training scheme) from [1] and provide the baseline results from their study.
Following them, we use a small fully connected network for MNIST based settings as it has sufficient capacity given the data complexity and a ResNet-18 model for CIFAR and TinyImageNet settings which is a common choice for in literature, as well as for continual learning methods.
We have not tried our method for models larger than ResNet-18, but we see no reason for the method not to scale up to larger models.
In fact, we would argue that larger models would provide a higher capacity for learning a large number of tasks and richer representations in the semantic memories. Please let us know if we should provide more details.
> **The flat minima discussion is less essential but is probably missing a reference to Flat Minima, Hochreiter et al 1997.**
Thank you for sharing the citation. We will add it in the revision.
> **Instead of the interesting section 6 I'd have preferred some ablation experiments (though the component analysis is appreciated). E.g. are two teacher networks necessary? How much worse is the model if one teacher network is used (with a tuned rate)? What if there were three networks (each with their own rate)?**
This is a good suggestion that would provide valuable insights. We opted for two semantic memories specifically as our study aims to mimic the fast and slow learning mechanisms in the hippocampus and neocortex. We had evaluated the method using a single teacher model and found it to perform sub-par with the dual memory counterpart as one model fails to keep the performance on both the recent and earlier tasks together i.e there is an inherent trade-off. Tuning the semantic memory to adapt to the recent changes comes at the cost of performance on earlier tasks and vice versa. We will add these results in the supplementary.
We did not try our method for more than two semantic memories (teachers) as we did not have a good intuition for the role of additional semantic memories.
However, our method is flexible enough to incorporate more than two teachers, and exploring their effect could be interesting.
> **I did not understand how samples are removed from the buffer (the episodic memory). Are samples removed eventually? If yes, how and where is that explained? If not, what is the space complexity of algorithm 1?**
Sorry for the lack of clarity on this, we will add more details of reservoir sampling in the revision as well as add the algorithm in the supplementary. The samples are indeed removed from the buffer. A fixed-size memory buffer is maintained using reservoir sampling which assigns equal probability to each sample in the stream for being represented in the buffer.
Accordingly, each sample in the data stream has an equal probability of being added to the memory buffer by replacing the existing samples in the buffer at random. Reservoir sampling does not assign any priority to the samples being added or replaced from the buffer.
Please let us know if we failed to address any of your concerns and if we can provide any additional information.
Thank you for your response. After reading the other reviews and your responses I still recommend acceptance. For future work, I'd recommend to "upgrade" the model size, training scheme, etc to better performing models.
That said, I see no reason why the contributions of this paper would not apply there.
One thing that bothers me is that your decision to use two models is solely based on CLS theory. I think it is fine to use neuroscience as an inspirational source but these models are far far far removed from simulations of real neurological systems like the hippocampus.
Furthermore, the goal is not to simulate such systems but to perform well on the classification/regression problem.
Thus, _all_ your central design decisions ought to be based on (mathematical) theory or empirical data (either yours or others). In the context of this paper and this conference, "because CLS theory says so" is in my opinion not enough reason to do things a certain way.
That said, your results are convincing and I have otherwise no critique, thus my recommendation to accept.
Strengths:
1, Semantic memory is the first leveraged in incremental learning, which is interesting in my opinion. The contribution of this paper is novel.
2, The codes are released, which guarantees the reproduction of the proposed method.
3, The experiment results prove the effectiveness of the proposed method, which obtains impressive performance.
Weaknesses:
1, Can this method be applied to online continual learning task?
2, The experiments for domain-IL are only evaluated in one dataset. Experiment results on more datasets should be shown in this paper.
3, For Class-IL, some experiments with different base classes should be shown in this paper.
NO.
Thank you for your kind words and constructive feedback. We are glad that you found the method interesting and hope to address your questions below.
> **Can this method be applied to online continual learning tasks?**
This is an interesting question and actually a research direction that we aim to extend our method to in the future. We would also like to emphasize that the experiments on MNIST based settings (S-MNIST, R-MNIST, P-MNIST, and MNIST-360) can be considered under the online continual learning scenario as we only train the network for 1 epoch, and thereby the model only sees the data for each task once as the common single epoch protocol in online continual learning. Our results on these settings demonstrate the potential of CLS-ER as an efficient online continual learning method.
Furthermore, as highlighted by [1] in their supplementary section F.3, we also believe that it is necessary to set the number of epochs per task considering the dataset complexity in order to disentangle the effects of catastrophic forgetting from those of underfitting. We share their suggestion that future CL works should strive for realism by designing experimental settings which are in line with the guidelines of General Continual Learning[2] which is the goal of our study rather than adopting the single-epoch protocol.
As such, we do not see any limitation of our method that would impede its application in more challenging online continual learning settings.  Thank you for bringing this point, we will highlight it more in the revision.
> **The experiments for domain-IL are only evaluated in one dataset. Experiment results on more datasets should be shown in this paper.**
We are sorry that you got this impression, we will make it clearer in the revision. We had also evaluated our method on Permuted-MNIST which together with Rotated MNIST are the two most common benchmark settings for domain-IL. We had included the results in the supplementary section B.1 and our method is effective under this setting as well. We had excluded it from our main paper as, in accordance with [2], we do not consider it as a viable evaluation protocol as it violates the cross-task resemblance desiderata and deviates from the goals of continual learning.
Furthermore, the MNIST-360 setting can, in part, be considered as evaluating the method on domain-IL as it exposes the model to a smooth rotational distribution shift in addition to a sharp distribution shift (details in supplementary section A.3). We believe that the performance of our method on these three settings collectively demonstrate the effectiveness of CLS-ER for the domain-IL setting. Thank you for bringing this to our attention. For completion, we will add the results of Permuted-MNIST in the main table and highlight the domain-IL learning aspect of MNIST-360 more.
> **For Class-IL, some experiments with different base classes should be shown in this paper.**
We are not sure if we understood the comment correctly, please let us know if our response fails to address your concern, we would be happy to provide more details.
For the Class-IL setting, we follow the protocol in literature whereby we start with a randomly initialized model (without any pre-training) and train them on a sequence of tasks with a disjoint set of classes. This is in contrast to the common approach in many online continual learning settings, where the model is initially trained on a large set of base classes.
We hope to have addressed your concerns and would incorporate your suggestions in the revision. Please let us know if we can provide more details or explanations.
**References:**
[1] Buzzega, Pietro, et al. "Dark Experience for General Continual Learning: a Strong, Simple Baseline." 34th Conference on Neural Information Processing Systems (NeurIPS 2020). 2020.
[2] Farquhar, Sebastian, and Yarin Gal. "Towards robust evaluations of continual learning." arXiv preprint arXiv:1805.09733 (2018).
We thank the reviewers for their valuable feedback and suggestions. We have incorporated their suggestions into the revised submission and made the following changes:
- We performed additional ablation experiments to compare the performance of our method with a single semantic memory (Section D).
- Added the Permuted-MNIST results to the main text (Table 1).
- Highlighted the application of our method on online continual learning setting in the main text (and more details in Section A.4).
- Elaborated the details of reservoir sampling in the main text and added the algorithm in the appendix (Section B).
- Explained the reason for stochastic update of semantic memories (Section 3.3, 4th Paragraph).
- Provided clarification on optimal semantic memories (Section 3.3. 2nd Paragraph)
- Added missing citation for Flat Minima.
We hope that the revision and individual responses addressed the concerns of the reviewers. We would be happy to address any further concerns or questions.

Meta-review:
role: PC
data: comment: The manuscript proposes an experience replay method that supports two time scales of memory, as in complementary learning systems from the cognitive sciences literature. The authors demonstrate that their method on a wide range of benchmarks and after the rebuttal demonstrate it on one additional benchmark.
The reviewers raised a number of questions and concerns around additional experiments (benchmarks and ablations), online training, the cost of training, number of hyperparameters, further analysis, clarifications, and citations. The authors address the majority of these concerns during the rebuttal period, and overall the reviewers were in favour of acceptance. Therefore, I recommend acceptance of this work.

Decision: Accept (Poster)