Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors proposed a new benchmark protocol (i-Blurry) for continual learning. In this benchmark protocol, the class distribution is class incremental and has blurry task boundaries, and the training is online. They also propose a new method, CILB. This method contains three important components: “sample importance memory”, “memory-only training”, and “adaptive LR scheduling.” Extensive experimental results are provided to show the effectiveness of the proposed method. 
main_review: ### Strengths 

- The proposed benchmark protocol, “i-Blurry”, is reasonable and interesting. It is important to analyse the results when some of the classes are not disjoint in continual learning tasks. 

- The authors provided extensive experimental results.

- The proposed method is technically sound and proven to be effective by the empirical results.

- The paper is well-organized and easy to follow.

&nbsp;

### Weaknesses

- ***The technical contributions of this paper are somewhat weak.*** In this paper, the authors mainly propose three components in the method section: “sample importance memory”, “memory-only training”, and “adaptive LR scheduling”. For the first component, the main idea is to find the optimal exemplars that can minimise the total loss so that it could be regarded as a simplified version of “mnemonics” (Liu et al., 2020). For the second component, it can be summarised as “sample importance memory”+GDumb (Prabhu et al. 2020). For the third component, adaptive LR can benefit all classification models. The authors don’t explain why it is important in continual learning tasks. 

-  ***This paper is not self-contained.*** Alg.3 and Section A.3 are not some additional information and results. I cannot understand the proposed method without reading Alg.3 and Section A.3. Thus the authors should definitely include these parts in the main paper.

- ***The authors only provide experiment results on small-scale datasets.*** Most continual learning papers, such as iCaRL (Rebuffi et al., 2017) and BiC (Wu et al., 2019), provide the results on large-scale datasets (e.g., ImageNet-1k). As the authors trying to establish a new benchmark protocol, it is important to provide the results on large-scale datasets.

- ***An ablation study on the number of tasks should be provided.*** In many continual learning papers (iCaRL (Rebuffi et al., 2017), BiC (Wu et al., 2019), etc.), the number of tasks will significantly influence the continual learning performance. However, I don’t find an ablation study on the number of tasks in this paper. I don’t even see an explanation about how the authors choose that hyperparameter. 
summary_of_the_review: Overall, I think this is an interesting paper. The authors design a new benchmark for continual learning and propose a simple yet effective method. My primary concern is that “how likely is it for following researchers to refer to this benchmark?” As there are already many different benchmark protocols in continual learning, I think the following researchers will tend to choose the benchmark protocol already used in many popular papers. Nevertheless, I still think the paper might be useful for the continual learning community. 

My rating is “borderline accept”, and I will consider upgrading my rating if the authors successfully address my questions.

&nbsp;

### Post-rebuttal update
The authors addressed most of my concerns in the rebuttal. They also provided the results I asked for in the revision (e.g., the results on ImageNet-1k). I think I tend to accept this submission. So I upgrade my rating to eight.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new problem setup in continual learning. As the title suggests, the paper focuses on online, task-free, class incremental, task blurry learning with any-time inference. The authors also came up with new baselines and importance-based memory management. They empirically tested their methods in the proposed problem setup.
main_review: The paper has excellent plots for the new problem setup. However, without further clarification of the following concepts/comments, the significance of the new setup could be weak.
* **The paper claimed the new setup is both “task-free” and “class-incremental.”** In my opinion, the two setups are not compatible with each other. Specifically, they are different at the output layer in supervised learning. 
* **“Any-time inference” may require more rigorous justifications; so as to the new metric.** Only one baseline demonstrated the incapability of making any-time inference in Table 1. The significance of any-time inference needs more evidence. 
* **Thm. 1 may suffer from catastrophic forgetting.** Thm. 1 can be seen as a performance-driven memory management strategy. However, it has nothing to do with catastrophic forgetting. In Fig. 4, the proposed method has a projection of performance drop in the later phase. In the long run, it is hard to tell whether the proposed method will outperform the baselines. This performance drop could be attributed to the catastrophic forgetting of Thm. 1.
* **The paper doesn’t discuss the algorithm complexity**, which could be O(M) in general, where M is the memory size.

summary_of_the_review: There are unsolved concerns about the concepts/comments in the current version. Without addressing them, the proposed problem setup and method could not be that significant for continual learning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a more realistic setting (called *i-Blurry*) for continual learning (CL) that generalizes the *blurry* and *disjoint*  settings proposed in prior work. The disjoint setting assumes that there is no class that appears in multiple tasks and the blurry setting assumes that no new classes are seen after the first task. In the *i-Blurry* setting, one can have overlapping classes across tasks as well as new classes appearing in each task. This setting is also **online** and thus the paper is interested in continuous model evaluation i.e., any-time inference too. It proposes a metric which calculates the area under the accuracy curve during training for the same. 

For the new configuration, the paper proposes a strong baseline and a new algorithm called CLIB. CLIB is a memory-based CL method that refines its memory by throwing out *least important* samples and updating it with more important ones. The sample importance is calculated as the expected decrease in training loss when the sample is used for training. CLIB is also equipped with a data-driven adaptive learning rate scheduling scheme which provides some additional performance benefits. 

The results showcase that CLIB is able to outperform other online CL methods and the baseline by large margins for various instantiations of their i-Blurry setting, *including the disjoint and Blurry setting*, on the CIFAR-10, CIFAR-100 and TinyImagenet datasets. An ablation study shows the main benefits of the proposed method come from the sample importance based memory scheme. 
main_review: ### Strengths:

1. The proposed method works well on a various instantiations of the i-Blurry-N-M setup. The proposed algorithms is better in the blurry (N=0) setting and performs comparably well in disjoint setting (N=100) as well. 

2. The ablation study highlights the main component in the proposed approach, the memory management scheme that relies on sample importance, that results in the highest benefits. 

3. Lots of supporting experiments are provided in the appendix that further corroborate most claims. 

4. Good organization of related work and great figures.  

### Weaknesses:

1. I am unsure if the proposed approach of obtaining sample importance is entirely novel. A discussion regarding this is necessary as it is likely that similar ideas present (beyond CL for instance) in literature. 

2. A natural language description of Theorem 1 is missing. 

3. I felt that more insights into the calculation of importance are required in the main text. A note about the efficiency/runtime needs to be specified. 
 
**Personally**, I think dedicating a portion of the main-text (Section 4) in discussing these would probably be more beneficial than describing memory only training and adaptive lr, as they seemingly play only a minor role (as indicated by the ablations).

4. No information as to why TinyImagenet is challenging for all methods is provided. There is also no justification provided for choosing TinyImagenet over Imagenet (RM, BiC and GDumb all show their results on Imagenet). 

5. Since the i-Blurry setting is an **online** one, the choice of hyperparameters are likely to affect the results greatly. As such hyperparameter choices (updates per sample, memory size and batchsize) need to justified. Specifically, were the hyperparameters chosen consistent with previous work? 

6. For a fixed $M$, is it reasonable to expect that performance of all algorithms would increase (in general) with increasing N? This can be used to justify choosing $N$ to be from $0, 50, 100$ (would also be helpful reference for future works). 

**Suggestion** (not a weakness): The paper can also report the performance on some (soft) upper bounds. For instance, what if the task was not restricted to be online. What is the performance when the entire training data is available at once (non CL scenario)?


##### There are some typos:

* Section 4.2 (page 5): Arguing that considering ... lead to training efficacy **decrease**.

* Section 4.1 (page 5): For the LR scheduling, other **CL methods use either (1) exponential decay** or (2) constant LR ...

* Last line on page 4 is missing a period (full-stop)
    
* Also, Table 3 $A_{avg}$ on CIFAR-100 has the same numbers for the last two rows, which might be a typo.
summary_of_the_review: ### Basis for the scores

1. The experiments justify most of the information present in the paper. Some additional information regarding the sample importance based memory management scheme is required. Also, some decisions regarding hyperparameters need to justified, given the *i-Blurry* setting is online and the main motivation of the paper is to propose and solve a more practical CL setting. 

2. I'm not sure about the novelty of the sample-importance based memory management scheme and a discussion regarding prior work related to sample-importance is required. The proposed setting on *i-Blurry* is marginally novel as it is a generalization of the disjoint and blurry configurations. Any-time inference and $A_{AUC}$ are  likely novel.

3. Empirical results shown cannot be easily (and extensively) compared to some prior work as there isn't a strict adherence to prior dataset and hyperparameter choices. Some justification for the decisions is required. 

4. Insights into the proposed CLIB method along with a nuanced discussion regarding the differences between the algorithms is lacking. There is no description regarding efficiency and scalability.

## Update
The authors clarify almost all of my questions pertaining to the paper. The paper now contains experiments on Imagenet-1k with additional insights into the algorithm and its efficiency. As such, I've improved my scores.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an experience replay method for continual learning (CL) whose main innovation is a memory-management algorithm that updates the memory when new samples arrive in a way that preserves the most "useful" samples (wrt to their effect on the loss) while promoting class balance. The method is evaluated on the trickiest CL setting - class incremental, online, and task-free - where it outperforms baselines on CIFAR-10 and -100 datasets.
main_review: The main contribution of the paper, I believe, is the memory update method, which (as stated above) updates the memory when new samples arrive in a way that preserves the most "useful" samples (wrt to their effect on the loss) while promoting class balance. The authors also train exclusively on samples drawn from the memory, rather than on batches split between memory and newly arrived samples, which seems to be advantageous. And the authors use a learning-rate scheduling method that also gives some advantages. The memory management method is novel, as far as I'm aware, and the overall method performs very well. Evaluation is carried out in the class incremental, online, task-free setting, which is a good decision in my opinion. So many other methods fall short under these conditions.  So I think the paper deserves to be accepted.

The authors make a big deal out of their "blurry" sampling method for evaluation. I think it's a good approach, but isn't really a big contribution in its own right.

Presumably, following memory update, the batch that is trained on is sampled uniformly from the memory. Is that right?

Space permitting, it would be nice to see a more detailed comparison with related experience replay methods. In particular, Aljundi et al's MIR method also uses a kind of importance weighting. It would be good to see the differences spelled out, and to see some sort of explanation for why the present method performs better.
summary_of_the_review: A nice paper describing an effective and somewhat novel replay-based CL method, evauated in a demanding setting.
review_title: null

Meta-review: The authors propose a new continual-learning setting with a few distinguishing features: 1) the task boundaries are blurry (in other words, past task samples can reappear); 2) training is online; and 3) evaluation using online accuracy (instead of average accuracy). The authors also propose a useful method for this scenario and benchmark it using four different datasets.

The first round of review pointed to two main limitations of the manuscript. 
+ The authors only provided small-scale experiments. The reviewers argued that for the setup and method to have an impact having good results using larger-scale data would go a long way. 
+ Whether “task-free” and “class-incremental” were compatible. 

For the former, the authors were very reactive and provided results using a standard "ImageNet for CL" dataset. 

For the latter, I must thank the authors and also the reviewers for discussing this thoroughly. In the end, my understanding is that there was a reconciliation that both were in fact compatible, but the reviewer suggested that this be discussed very clearly by the authors. I second this suggestion. The CL field given its many slightly different settings might be partly to blame here (reviewer Vfw2 made a similar comment, and I also thank them for playing a role in resolving the issue).


A few additional thoughts: 
+ I believe that more general setups in CL are worthwhile even in the absence of any immediate applications. This is especially true since some of the standard CL assumptions do not seem to be well motivated. However, I find that claiming that something is more realistic requires grounding (e.g. a set of examples from the "real world" or a specific domain/setting). I know the authors backed some of their claims with references, but different real-world problems will come with different limitations and I would be hesitant to use phrases such as "most real-world" settings without thorough justification.
+ While different from the core of your work, I believe the framework proposed in this other recent paper has similar goals (although the setup allows pre-training and is not online). Might be worth knowing about it in case you do not: 
Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning, NeurIPS 2020
https://papers.nips.cc/paper/2020/file/c0a271bc0ecb776a094786474322cb82-Paper.pdf

All in all, this is a good contribution that proposes an interesting and rich setting along with a good baseline method for it. I strongly encourage the authors to follow through on their promise to provide the community with code, dataset splits, Kaggle leaderboard, etc., as a way to maximize the impact of their work.
Decision: Accept (Poster)