Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work builds upon the notion of a complementary learning system that consists of multiple memories: episodic memory and the semantic memory in this case. Although this notion is utilized in many works, the main contribution of this work is in designing the episodic memory to contain samples from the current batch that are pre-selected based on their distance from the mean of an errors of all samples along the training trajectory. These errors are also used to modulate the learning. The results on Seq-CIFAR-10, Seq-CIFAR-100 and GCIL show good accuracies in low memory buffer scenarios as well as in the presence of label corruptions. 
strength_and_weaknesses: Strengths: 
The idea of modulating the learning based on the error is well motivated and using it to create an episodic memory buffer seems improve the continual learning accuracy in the clean as well as label corruption settings

Weaknesses: 

•	There is a significant overhead introduced by the dual memory mechanism so it will be fair to compare with other single memory approaches by maintaining a fixed overall memory. 

•	In general, the related works need to be more comprehensive and explicit in explaining how the proposed work is different from the literature and compare if needed [6] : The idea of error sensitivity-based modulation seems to be explored before in the local learning-based approaches [1,2,3]. Why only compare with replay-based approached when comparing the performance. Other non-replay-based approaches have shown superior performance and have the advantage of not requiring the memory buffer [2,4].

•	Not clear if the improved results hold when a larger memory buffer (eg., 5k) is used.

•	Only a single metric (accuracy) is used for comparison. Other metrics such as average forgetting [5] shall be used to

•	Effect of task sequence not considered! do the results hold when the task sequence is changed?

•	Why does ESMER have lower accuracy (on test data) for the task it is trained on? As seen in figure 3 diagonal elements.

•	The ablation does not show the effect of just keeping the error sensitivity modulation without the semantic memory or reservoir sampling.

•	Do the results on label corruption (fig2) hold for Cifar-100 data as well?

•	Does this approach work in the online continual learning setting?

[1] Dellaferrera, G., & Kreiman, G. (2022). Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass. arXiv preprint arXiv:2201.11665.

[2] Madireddy, S., Yanguas-Gil, A., & Balaprakash, P. (2020). Neuromodulated neural architectures with local error signals for memory-constrained online continual learning. arXiv preprint arXiv:2007.08159.

[3] Kudithipudi, Dhireesha, et al. "Biological underpinnings for lifelong learning machines." Nature Machine Intelligence 4.3 (2022): 196-210.

[4] Li, S., Du, Y., van de Ven, G. M., & Mordatch, I. (2020). Energy-based models for continual learning. arXiv preprint arXiv:2011.12216.

[5] Mai, Zheda, et al. "Online continual learning in image classification: An empirical survey." Neurocomputing 469 (2022): 28-51.

[6] Pham, Q., Liu, C., & Hoi, S. (2021). Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34, 16131-16144.
clarity,_quality,_novelty_and_reproducibility: Clarity needs to be improved. For example, the Fig 1 have notations that are not introduced before the figure so it’s hard to parse the information from it.  It is not clear how the hyperparameters in this approach can be optimized for the continual learning setting when considering different datasets.
summary_of_the_review: •	The related works and consequently the approaches compared needs to be updated.

•	The effect of increase in the memory for dual memory approaches need to be considered when comparing with other approaches.

•	Alternative continual learning metrics and task sequence need to be considered.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new experience replay method using error sensitivity as a modulation method in continual learning. The proposed method keeps track of the classification loss during continual learning and when the newly-received data incurs a high classification loss, it gets downweighed to reduce its effect. In this way, the learned representations are more stable and suffer less from abrupt drift during task transition. The paper also adopts a semantic memory (a momentum-updated model) to further stabilize the continual learning process, following a prior work, CLS-ER. Experiments show that the proposed method achieves better performance in both standard continual learning benchmarks and settings with label noise. 
strength_and_weaknesses: Strength:
1. The proposed method considers the large errors incurred by data from new tasks as a potential reason for abrupt presentation drift in continual learning. Based on this insight, the paper proposes to downweigh the data points with large errors to reduce the drift and the experiments show that it is very effective. 
2. The experiments show that the performance is strong, surpassing CLS-ER even with one fewer model to keep track of. (Only the stable model instead of both plastic and stable models in CLS-ER)
3. The experiments on continual learning with noisy labels are interesting. 
4. The paper is well-written and easy to follow. 

Weakness:
1. The proposed method is only tested on small and relatively simple datasets like CIFAR10 and CIFAR100. I think the results are more complete and convincing if results on tiny/Mini/Full ImageNet are also reported and compared. 
2. The proposed method has high complexity and many hyperparameters. 

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and it’s very clear. 

Quality: The paper is of high quality with a well-motivated method and good support by experiments.  

Novelty: The method is novel. I believe the concept of error sensitivity will be inspiring to the continual learning community. 

Reproducibility: The authors promise to release the code and most details are well documented in the paper and the appendix. 

summary_of_the_review: This is a great paper presenting a novel method and strong results. It would be better if more experiments can be performed on some more complex datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: = Update after author response =

I thank the authors for their clarifications. I maintain  my recommendation for acceptance.

= Original review = 

The paper proposes a new approach to replay-based continual learning. The purpose of this approach is to prevent catastrophic representation drift caused by the large errors that occur when new tasks are introduced.

The approach maintains a slow-changing version  of the main learning network, clips sample losses if they are too far above a running average of previous losses (as estimated by the slow-changing network), penalizes disagreement with the slow-changing version for replay loss, and actually uses the slow-changing network at inference. 

The method is compared to different variants of the so-called ER method  (another replay-based method) and found to improve performance.
strength_and_weaknesses: - Strengths:

The method seems novel and interesting, and the results are promising.

- Weaknesses:

One possible concern is that all baselines used for comparison are variants of the ER method. Since this is a bit beyond my  field I can't really assess whether this is an adequate representation of existing replay-based methods. If this is acceptable to other reviewers, then I can't see anything much wrong with the substance of paper.

As a minor point, the presentation is at times a bit confusing and could be improved, see below.
clarity,_quality,_novelty_and_reproducibility: The paper is reasonably understandable, but a few points should be clarified:

- Generally: It took me a while to understand that the paper uses the word "memory" for what seems to be networks, or learners. This is confusing because this term is usually associated with buffers of items (samples or  returns). 

In fact, the paper itself actually seems to use "semantic memory" in both senses !  E.g. in the line just after Eq. 3, where it seems to apply to a buffer of l_s values.

I would strongly recommend using the term "network" or "learner" to refer to the actual networks theta_s and theta_w,  and reserve the word "memory" for buffers of samples or losses.


- p. 5: "a task warm-up period during which the running estimate is not updated"

Does this imply that even though the method doesn't require access to task ID, it does require access to task boundaries / notifications of task changes? If so, this should be explicitly mentioned in the introduction. 


- Greek 'alpha' and lowercase Latin 'a' in eq. 6 are very similar and a bit confusing! Maybe some different letters could be used?

- Please do explain acronyms! For example, in Class-IL / Task-IL - what is "IL" ? (I know it's "incremental learning", but it should really be explicitly stated in the paper, unless I missed it). Also, in the Related Work, what do ER and DER stand for ? 

- The paper explains the various kinds of Class-IL, but not Task-IL, which is used in Table 1 (unless I missed it?) 

- Figure 4a: Can you confirm whether the red curve shows "accuracy" with regard to the wrong/noised label, and therefore lower red curves are better? If soe this should be explicitly stated somewhere.

summary_of_the_review: This new method for improving stability of continual learning seems interesting and promising. If more knowledgeable reviewers are satisfied that the set of baseline methods used for performance comparison is representative, I have no objection to acceptance (pending clarifications as requested above).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new replay-based method for continual learning. The proposed ESMER is a dual memory rehearsal-based system which has a working model, a stable model, and a buffer which incorporates error-sensitivity-based reservoir sampling. The proposed algorithm works better than the existing replay-based methods for CL.

Reviewers asked several questions in their reviews and the authors provided convincing answers to all of them. The authors also added more experimental results that sound convincing. I recommend an acceptance for this paper.

However, I strongly encourage the authors to add imagenet experiments to the final version of the paper since cifar-10 and cifar-100 are totally not sufficient for current standards in continual learning. 
justification_for_why_not_higher_score: This is a good paper. I am recommending a poster since the results are only based on cifar-10 and cifar-100. It is a promising approach that needs more work to become impactful.
justification_for_why_not_lower_score: There is no reason to reject this paper. It is a solid paper that is worth publishing.
Decision: Accept: poster