Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an online Experience replay (ER) based continual learning technique to mitigate forgetting of the past data. The paper considers a repeated rehearsal setting that allows training for multiple steps on each batch of data (previously proposed by Aljuni et al. 2019). In this repeated rehearsal setting, the paper studies the source of replay-memory overfitting. 

The paper proposes a formulation of ER loss function that captures not only the dependence on the number of samples from the current and past tasks (shown in previous work), but also accounts for the ratio of incoming data size and the replay-memory size (novel contribution claim). When the relative size of replay-memory is smaller as compared to the size of the incoming task, memory overfitting can occur. As the number of previously-seen tasks increases, the memory overfitting worsens. 

Further, the paper analyzes the training process for any given task and observes that repeated rehearsal leads to a quick drop in the training loss of the incoming data. Therefore, during the later iterations of training, the memory loss dominates the incoming data loss, and reduces the regularization effect of incoming data. The authors describe this phenomenon as an underfitting-overfitting dilemma – where repeated rehearsal can reduce underfitting of incoming data but at the same time can cause overfitting of the replay-memory data. 

In order to solve the over-fitting problem, the authors propose data-augmentation of both the incoming data and the replay-data during mini-batch training. A RL-based HPO scheme is employed during training to tune the two key hyper-parameters – number of repeated rehearsals and strength of augmentation. 

The proposed method is evaluated on a suite of computer-vision tasks and is shown to outperform ER baselines. 

strengths_and_weaknesses: Strengths: 
1. The paper analyzes the replay-memory overfitting problem in detail. 
2. The paper combines existing techniques of repeated rehearsal with data-augmentation to alleviate overfitting on replay-memory data.
3. The paper performs ablation studies to understand the effect of repeated rehearsal and augmentation. 
4. Use of RL-based HPO during online learning is an interesting direction.  
5. The paper is mostly well-written. 

Weaknesses:
1. The paper does not cite and compare against Dark ER [Buzzega et al. 2020]. This prior work also employs data-augmentation and is also applicable to the online learning setting.  
2. Previous ER formulations often use an additional hyper-parameter that balances the loss between incoming data and the new data. Not sure why this hyper-parameter is missing in the problem formulation. 
3. I could not find any example of the type of augmentations used in the main paper. 
4. Writing improvements:
   (a) It is difficult to follow Figure 1 due to smaller font and lack of enough details in the figure caption.
   (b) Typo in Line 213. 

questions: 1. The training loss for the incoming batch in Figure 2 (a) goes down very quickly. Why do you not observe overfitting on the incoming training batch during repeated rehearsal (say k=10)?
2. It is not clear to me what is the reward function used for RL-based HPO. 
3. What is the compute overhead of RL-based HPO?

limitations: The authors mention some limitations of RL-based HPO but do not specify the the compute overhead.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the effect of data augmentation in online continual learning. It proposes *repeated augmented rehearsal*, a simple method that adds data augmentation when using multiple iterations on the same batch in online continual learning.
strengths_and_weaknesses: ### Strengths

1. I consider the simplicity of the proposed approach a major strength. Continual learning research suffers from a flood of overly complex methods without proper ablation studies. Revisiting the seemingly simple baselines, with proper ablation studies, is an important contribution!

2. The empirical evaluation is of high quality, includes most relevant baselines and competing methods, and provides a number of ablation studies. (A slight caveat is that the same architecture, ResNet18, is used on all datasets.)

3. The results of the proposed method in the online continual learning setting are very convincing. RAR provides significant performance boosts across the board and can be combined with any rehearsal-based CL method.



### Weaknesses

1. I think this paper has one major flaw. It assumes the ER formulation in Eq. (2), which weights loss contributions from memory and incoming batch equally. One can easily upweight or downweight the memory by reweighting the first gradient term in Eq. (2), which controls the memory overfitting issue and (if done properly) can also remove the undesirable dependence on the task size. Therefore, in some sense the paper highlights a problem of vanilla ER, but neglects the most straight-forward (partial) remedy. Since the goal of the paper is to revisit a simple baseline, this constitutes a major shortcoming in my opinion.

2. I find it somewhat regrettable that the authors chose to exclusively focus on the *online* continual learning setting, which seems of limited practical relevance to me. (One can always buffer data from a stream into mega-batches and run an "offline" continual learning algorithm on the resulting "pseudo-tasks".) The key questions the paper studies, the effect of data augmentation in rehearsal-based continual learning, is just as relevant in the "offline" CL setting.

3. Finally, I want to register that the resulting method is of limited novelty, simply combining repeated ER with data augmentation. (Personally, I don't think that is a big caveat, but should be mentioned.) I would also add that augmentation can only ever be a partial remedy. It can ameliorate the memory overfitting issue and is certainly useful in practice. But, in some sense, it will never tackle the fundamental problem.

4. Minor Comments

    - The notation in Eq. (2) is sloppy, since the examples are indexed with $i$, but this is not the summation index of the sum. Why not just $\sum_{x, y \in \mathcal{B}_t}$? This would also avoid the currently overloaded notation for data points from memory and incoming batch.
    - Calling Theorem 1 a Theorem is a bit much in my opinion. I would go with a more modest "Proposition", but I guess that is a question of personal taste.

### Rating

I explicitly commend the effort to revisit simple, principled baselines. The paper is well-written and the empirical evaluation is sound. Unfortunately, I have to recommend rejection for this paper. I think omitting the option to reweight the memory loss (Weakness 1) is a fundamental shortcoming when trying to study the issue of memory overfitting in continual learning.

### Update after rebuttal

The authors' rebuttal has resolved my main concern regarding the reweighted ER baseline. I have increased my score accordingly.
questions: 1. Did you consider the (partial) remedy of reweighting the memory loss term in Eq. (2)? If yes, why did you not include this option in your method? If no, do you see any reasons not to use this simple (partial) remedy in conjunction with data augmentation?

2. Why did you choose to focus exclusively on the online CL setting? Do you think it is of great practical relevance? Would you agree that some of the main questions the paper is raising are also relevant in the offline CL setting?
limitations: I think the limitations of the paper could be discussed more honestly. Some of the consideration around the "regularization effect" of incoming data are mostly anecdotal (e.g., Figures 1 and 2) but that is not clearly stated. Also, data augmentation seems to be limited to image data, which could have been mentioned in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors first point out an underfitting vs overfitting dilemma in online continual learning (OCL) with experience replay (ER) on the data distribution and on the samples in the replay buffer, respectively.

They show that the amount of iteration on an incoming batch in Repeated ER helps with the underfitting of the data at the expense of more overfitting on the buffer. This is both proved theoretically and empirically.

The authors propose to combine data augmentation with repeated ER, coined repeated augmented rehearsal (RAR). The motivation for the data augmentation is that it should reduce the buffer's overfitting.

The authors also point out the open problem of Hyperparameter Tuning in OCL. Accordingly, they propose an RL-based hyperparameter tuning algorithm that adapts the hyperparameter in an online manner.

The method is then empirically validated against relevant ER baselines on relevant benchmarks. The paper includes useful ablations and analyses of the results.


strengths_and_weaknesses: Theorem 1 is helpful in showcasing how different OCL problem characteristics will impact the underfitting/overfitting dilemma, specifically the data to buffer size ratio.

The repeated rehearsal analysis is useful for the field. 
The loss landscape analysis helps the reader to grasp the increasing overfitting problem as the number of iterations (k) increases. 
The regularization effect of the incoming data is also something useful, and contrary to the aforementioned observation, I have never given this one previous thoughts.

The method (RAR) is simple and effective. 

I also appreciate that the others addressed the open problem of hyperparameter-tuning in OCL with an adaptive RL approach. Its adaptive feature could be particularly useful in OCL with very long task sequences.

The plasticity/stability was particularly interesting.

questions: None for now
limitations: The main limitation I see is that the method requires prior knowledge of data augmentation techniques, which makes it less general than other OCL methods.
Could someone w/o prior knowledge in e.g. audio, RL, use the same RandAugmentatiomn technique? If not, a general protocol to tackle different modalities would be useful.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper addresses a rehearsal-based approach for online continual learning. It gives theoretical insights on overfitting and unfitting problem in rehearsal-based approaches. Inspired by the investigation, the paper proposed a simple, novel method called repeated augmented rehearsal. 
strengths_and_weaknesses: Strengths:

- strong theoretical analysis of rehearsal-based methods from the perspective of empirical risk minimization. 
- the experimental results support the effectiveness of the suggested method. 

Weaknesses: 
- No experiments on large-scale datasets such as ImageNet-1K. 
questions: When the RL agent is removed, what is recommended to keep a good performance?  
limitations: No experiments on large-scale datasets such as ImageNet-1K.
review_title: null

Meta-review: metareview: This work considers rehearsal-based methods in continual learning and revisits revisits the rehearsal dynamics. Most reviewers praised the analysis of overfitting/underfitting in replay-based methods and the presentation. The proposed repeated rehearsal with data-augmentation was shown to be effective in empirical evaluations and the ablation studies. Finally, the authors addressed on of the main concerns raised by the reviewers, namely the relation to the reweighted ER baseline, during the rebuttal.



confidence: Certain
Decision: Accept