Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a study on a different flavor of continual learning scenarios than previously considered in the literature. Precisely, one in which data separation is not strict among concepts, and later tasks can revisit one of the previously introduced classes. The authors name this "class incremental with repetitions" (CIL).

The authors show that in these scenarios storing and replaying data is less important (which is not surprising), unless it corrects for imbalanced exposures to the different classes. Therefore the authors propose a sampling method which balances the class representation during training.

Experiments on the standard computer vision tasks show comparisons between domain incremental, class incremental and class incremental with repetitions.
strength_and_weaknesses: Strengths:
 - novelty of the analysis
 - relevant CL algorithms analysed under the different scenarios
 - well written paper

Weaknesses:
 - the results are not necessarily surprising; e.g. it is known that balancing classes in training batches benefits performance / optimization
clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear about the details, therefore I consider it's easily reproducible.
summary_of_the_review: The CL community would benefit from an analysis of a more natural stream of tasks than the extreme cases considered so far. Due to the clear exposure, interesting analysis and experiments I suggest the paper should be accepted.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on class-incremental with repetition (CIR), which ranges from class- and domain-incremental learning. For building streams and managing repetition over time, the authors propose two CIR generators (i.e., slot-based and sampling-based), which are from two aspects to generate data streams. Moreover, the authors propose a frequency-aware (FA) storage policy tailored to CIR scenarios. Further, the authors perform the first comprehensive evaluation on CIFAR 100 and TinyImageNet to compare other replay approaches in the CIR scenario, and then show the effectiveness of FA.

strength_and_weaknesses: Strengths
1. The idea of CIR is absorbing and reasonable. Because for realistic data streams, it is natural to occurrences previously seen classes in new coming samples. Further, they designed a paradigm to build a stream with repetition via the proposed generators. 
Weaknesses

1. The novelty of this paper is insufficient. In my view, this paper conducted a sampling procedure via two generators. As for frequency-aware replay, its improvement is dealing with unbalanced data. What is neglected in this paper, however, is that there have been numerous works on adaptative store policy in continual learning.

2. The writing of this paper is not clear enough to follow. Firstly, the explanation of the discrepancy between CIR and traditional incremental learning (i.e., class- and domain-incremental) is not clear. Figure 1 replaced with actual objects may give a more intuitive understanding of the authors' motivation. Moreover, it is necessary to use a coherent algorithm to introduce the overall procedure, however, absent in the main text. Finally, the conclusion part is tedious and should be refined.

3. The lack of formulas in this paper makes comprehending have more barriers. For example, the generators build streams part lacks the complementary mathematical explanation. 

4. The authors should provide comprehensive theoretical analyses of their method, in my view, which is necessary for solid work.

clarity,_quality,_novelty_and_reproducibility: Overall, this paper is not high-quality work.
Clarity: This paper is not easy to follow. The figures in this paper don't help others understand the authors' motivation. For example, Figure 1 does not make sense in intuitively showing the difference between CIR and CI (or DI). Besides, the authors don't provide a visual explanation of the slot-based generator. The tedious conclusion part makes it difficult to grip the main contributions of this paper.

Novelty: This paper is not rich in novelty, and lacks relevant theoretical analyses. The main contribution of this paper, in my view, is to build a CIR data stream. Recently, analogous works with frequency-aware replay are not rare.

Reproducibility: The authors promise to provide open-source code and the algorithm.

summary_of_the_review: This is not a high-quality paper without a clear expression and lack of solid theoretical analyses, which could not stimulate other research works. So, I recommend rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: For the possible repetition in Continual Learning, the paper makes contributes in the following three areas: (1) The author proposes two stochastic scenario generators that produce a wide range of CIR scenarios starting from a single dataset and a few control parameters. (2) The author conducts a comprehensive evaluation of repetition in CL by studying the behavior of existing CL strategies under different CIR scenarios. (3) The author presents a novel replay strategy that exploits repetition and counteracts the natural imbalance present in the stream.
strength_and_weaknesses: Strengths: The paper proposes two practical CIR generators and performs a comprehensive series of comparative experiments of existing CL methods, which provides the CL area with a referenceable basis for further research. 

Weaknesses: The analysis of the FA storage policy is not sufficient. In the paper, the FA method is only proposed based on ER. Can the FA method be applied to other CL methods? How does the ER-FA method perform when the dataset is not unbalanced? Does FA cause a decrease in class accuracy for frequent classes? The author may want to respond to the weakness the reviewer raised.

clarity,_quality,_novelty_and_reproducibility: In terms of algorithms, the paper describes the steps of the algorithms in detail so that the algorithms can be clearly understood.

In terms of experimental results, the paper provides multiple experiments in various settings and uses high-quality graphs to present the experimental results.

In terms of novelty, the paper proposes two novel CLR generators and an improved FA method on ER method.

In terms of reproducibility, the author promises to provide an open-source implementation of our generators and algorithms with the scripts needed to reproduce the results reported in the paper. 

In summary, the paper is well above average in terms of clarity, quality, novelty, and reproducibility. 

summary_of_the_review: Class-Incremental with Repetition (CIR) scenarios are important in realistic scenarios but have not been sufficiently studied so far. This paper proposes two practical CIR generators and performs a comprehensive series of comparative experiments of existing CL methods, which provides the CL area with a referenceable basis for further research. Therefore, it is recommended to accept this paper if the questions about the weaknesses can be answered.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary of Contribution

This paper describes an scenario for Continual Learning with Repetition (CIR), a scenario in which task distributions are not disjoint (which distinguishes it from the class-incremental scenario). Class-incremental and domain-incremental learning can be thought of as extreme cases of CIR. The authors propose two stream generators: The *slot* generator which does not repeat examples, and the *sampling* generator which instead allows repetition of samples. The authors additionally propose an approach to exemplar replay (Frequency-Away Replay) to address the CIL task. Experimental results are given on multiple benchmarks using the two proposed generators.

# Strengths

The main strength of the paper is its proposal of new scenarios for class-incremental learning which can be seen as a relaxation of the (somewhat rigid) protocols used in the majority of works on incremental learning. The experimental evaluation, and comparison with non-balancing baselines confirm the importance of taking repetition into consideration.

# Weaknesses

+ **Marginal Contribution**: The main contribution of the work is the definition of the two stream generators that can be used to generate novel scenarios for class-incremental learning with repetition. The proposed approach to CIL (Frequency-Aware Replay) is a very straightforward approach, and similar rebalancing approaches have been applied to class-incremental learning with long-tailed class distributions (and learning for class-imbalanced problems in general) and thus is only marginally novel.

+ **Clarity**: There are some details of the proposed scenarios and frequency-aware replay that are not precisely described in mathematical detail. Some aspects regarding clarity were addressed during the discussion phase.

+ **Lacking Theoretical Foundations and Insights**: The paper is basically an illustration of how to model incremental learning with non-disjoint tasks. As such, it does not provide any particularly insightful theoretical analysis of the *phenomenon* of incremental learning with repetition and how it must be addressed (beyond frequency-based sampling of exemplars).

# Summary

The significant contribution of this work lies in the definition of new class-incremental learning scenarios with repetition. Although appreciable from the perspective of continual learning in general, the contributions to the theory and practice of incremental learning (to CIR and CIL) are very small and not particularly novel. Rebalancing based on class frequencies is a very well-known approach, and this paper does not provide any new insights into the problem in the continual learning scenario.

justification_for_why_not_higher_score: The technical and theoretical contributions of the work are very marginal. The only significant contribution of the paper are the new IL scenarios with repetition, which is not significant enough to meet the bar for acceptance at ICLR.
Decision: Reject