Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The setting of this paper is to use wild data (free data) as an additional resource to do class-incremental learning. This paper proposes an online placebo (free images) selection policy to select good images from the free data stream. The selected images are used for knowledge distillation. Thus the proposed method can be combined with existing CIL methods to improve their performances. To train the evaluation policy, this paper introduces an online reinforcement learning algorithm, which reduces training computation costs compared with offline reinforcement learning. Experiments show that the proposed method can enhance the performances of many existing methods.
strength_and_weaknesses: Strengths:
(1) This paper proposes an evaluation function to select good images from the free data stream. The proposed methods can provide a plug-in component for existing CIL methods.
(2) The experiments validate the effectiveness of the proposed framework.
(3) The paper is well organized, clearly motivated, and contributes novelty.

Weaknesses:
(1) The results are not so significant. There are some existing works using wild/free images as additional data for incremental learning. Compared with previous works, this work is a universal plug-in component to enhance many existing CIL works. However, this paper uses the methods without additional data as baselines, which obviously have improvements. In fact, one intuitive way of using free images is to find a confidence score to select good images or generate pseudo labels, which should be the true baseline. From Table.3 and A1, we can find that "Higher confidence" seems a baseline that does not use the proposed method but also can select good images for knowledge distillation. The gaps between the proposed method and "Higher confidence" is even close to random errors for LUCIR-AANet and iCaRL. From Table.3, using the proposed methods gets only 0.58% improvement in average accuracy for iCaRL, and the results for LUCIR-AANet is only 0.37%, which means "Higher confidence" is the basic baseline that should be compared with. This indicates that the overall good performance is largely due to the additional data, even a simple confident selection metric can provide relatively good improvements. 
(2) It would be helpful to report the mean and standard deviation to take out any concerns related to randomness.
(3) How to use ImageNet-1K as additional dataset for modified ResNet-32? Because the image size of CIFAR-100 is 32x32. 

clarity,_quality,_novelty_and_reproducibility: Generally speaking, this paper is clear and easy to follow.
summary_of_the_review: There are concerns about the results. The baseline methods compared are without additional data, which is somehow unfair. Moreover, the proposed offline RL method takes extra costs but only brings marginal improvement.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method to leverage free unlabeled image as "placebos" to calculate knowledge distillation loss for class incremental learning. A corresponding online learning algorithm and a memory reusing strategy are proposed to avoid extra memory usage. Comprehensive experiments on benchmark datasets demonstrate the effectiveness of the method.
strength_and_weaknesses: The main strength of the paper is the idea of levering free/public available data to help class incremental learning.

However, there are multiple weaknesses that influence the overall quality of the paper:
- I actually have a fundamental question of the problem setting. Since the idea is to use the placebos of old classes, is it possible to totally get rid of old classes, i.e., rehearsal-free? 
- In (4), the old class exemplars still play an important role in both the rehearsal loss (first term) and the distillation loss (second term). What if the exemplars are totally removed from (4)?
- I doubt that the results in Table 1 and 2 are strictly fair. Since saving the free data batch requires additional memory, shouldn't the competing methods get the same amount of memory to save buffered examples? For example, if $|\mathcal{U}|  + |\mathcal{P}| = 1200$, the buffer size for other methods should be $1200$ + "buffer size used in the results". 
- The ablation study in Table 3 is very unclear.
- The layout of Figure 1 needs improvement, especially (b).
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the paper is good.
The novelty and originality of the work is good, but the empirical evaluation is questionable.
The code is available so it should be reproducible.
summary_of_the_review: The paper presents an interesting idea of levering free/public available data to help class incremental learning. However, the empirical evaluation of the method itself needs careful check and explanation. Therefore, I recommend weak reject initially.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using placebos (chosen from a free image stream such as Google Images) in class incremental learning. It formulates the policy training process as an online Markov Decision Process, and achieve improvement over prior works.
strength_and_weaknesses: Strength:
1. The paper is written well.
2. Clear improvement is achieved over baseline.

Weakness:
1. The assumption of accessing free image stream (like Google Images) is NOT carefully validated. If the free image stream includes images of old classes or highly correlated to old classes, this essentially increases the size of memory which is not fair.  A clear distinction in terms of statistics between memory and placebo should be provided. (Please do NOT just say they are from different data sources.)

2. Another CL setup is missing: the 0th phrase does not have half of classes, but just the number of incremental class (e.g. 5 or 10 classes).
clarity,_quality,_novelty_and_reproducibility: The major concern is if accessing free image stream (like Google Images) is valid, which significantly affect the quality of this paper. However this is not provided.
summary_of_the_review: Reject is rated, given the crucial assumption of accessing free image stream is not carefully validated. Another concern is missing an important setup (see weakness).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to address the catastrophic forgetting problem in class-incremental learning (CIL). As most of the old samples are not accessible in CIL, the authors propose to use unlabeled external data (i.e., placebos) to compute the knowledge distillation loss to consolidate the old knowledge. To achieve online placebo selections, the authors propose to formulate the CIL task as an online MDP and design a novel online learning algorithm for learning an online policy to control the placebo selection. Further, they propose a mini-batch-based memory-reusing strategy to avoid extra memory usage. The authors conduct extensive experiments on three CIL datasets, CIFAR-100, ImageNet-100, and ImageNet-1k, and show their method achieves state-of-the-art performance. 
strength_and_weaknesses: Strengths

1. The proposed online MDP formulation for CIL is novel and interesting. I think it makes sense to formulate the CIL as an online MDP and learn a policy to control the learning process, e.g., placebo selection. I think this formulation can also be used to learn policies to adjust other components in CIL. 

2. The idea of using extra unlabeled data is interesting and reasonable. Because unlabeled data can be obtained easily from the Internet without much effort. I think this is a promising direction to improve performance when training with imbalanced data, e.g., CIL. 

3. This paper is well-written and easy to follow. The authors clearly explain their motivation in the introduction. Besides, the authors also provide extensive figures and visualizations to make their explanations clearer. For example, Figure 1(b) clearly visualizes the problem when using the new data to compute the KD loss. Figure 1(c) demonstrates the effectiveness of the proposed placebo selection method. 

4. Extensive experiment results are provided to show the effectiveness of the proposed method. For example, in Table 1, the authors combine their method with five different CIL baselines and achieve consistent improvements. It is impressive to see that their method can work with many different methods and KD losses. In Table 2, the authors show their method achieves state-the-art performance on three datasets. Furthermore, the authors also provide detailed ablation results and visualizations. In summary, I think the experiments are very impressive. 

5. The authors provide open-source code and detailed configures. These materials are helpful for the following researchers. 

Weaknesses

1. It would be better to move more training details (e.g., Appendix E training configures) from the appendices to the main papers. Because I think these details are very important for reproducing the results. 

2. The authors should also compare some famous CIL methods (e.g., LUCIR (Hou et al., 2019) and iCaRL (Rebuffi et al., 2017)) in Table 2 because they are very important baselines for class-incremental learning. 

3. The authors can provide more visualization results for different classes (like Figure 3(a)) in the appendix. These visualization results can help the following researchers to understand how the placebo images work in the class-incremental learning models. 

4. Some recent related papers published in CVPR 2022 should also be discussed and compared. For example, [1] also uses the same benchmark protocol and should be compared in Table 2. 

[1] Joseph, K. J., et al. "Energy-based Latent Aligner for Incremental Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

clarity,_quality,_novelty_and_reproducibility: Overall, this paper is a high-quality work. 

Clarity: This paper is well-written and easy to follow. The authors use a lot of figures and visualizations to help the audience to understand the motivation and the proposed method.

Novelty: This work is novel as it proposes an online MDP formulation for CIL, a novel online learning algorithm to train a policy for the online MDP, and a mini-batch-based memory reusing strategy to avoid extra memory usage. 

Reproducibility: The authors provide open-source code in the supplementary and training details in the appendix. 

summary_of_the_review: Overall, I think this is a high-quality paper with a novel method and extensive experiment results. The proposed direction, selecting extra data to help to improve CIL, is promising in my view. Further, this paper achieves state-of-the-art performance and provides open-source code. So, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is proposing to utilize unlabelled data in class-incremental learning setting. The proposed method partially follows online learning with partial information over MDP formalism. Although the authors discuss this setting in detail, resulting application of EXP3 is still somewhat unnatural. Four experts reviewed the paper independently and 3 out of 4 thinks the paper has major issues which needs to be fixed before publication:
- Its empirical design should be improved significantly. Currently, the baselines are rather simplistic and not utilizing  unlabelled data. Although they are not applied to class incremental learning setting, many methods can utilize unlabelled data like contrastive approaches. Evaluation such simple baselines is necessary to justify the complexity of the proposed method.
- The setting needs to be justified. First of all, the unlabelled data stream and its relationship to the labelled stream should be clarified further. What are the assumptions needed?, are they realistic? is there any real application which would benefit from this method?
- Although the setting is significantly different than classical class-incremental learning, there is a very minor performance improvement. This is extremely surprising as using significant amount of additional information does not simply improve results. This should be explained and studied.
justification_for_why_not_higher_score: The paper is simply not ready to be published. There are major issues which need to be fixed before publication: 1) justification of the proposed new problem, 2) better empirical study with closer baselines and fair comparisons as well as results from simple ML methods, 3) explanation for lack of performance.
Decision: Reject