Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to apply distillation loss on unlabeled data to tackle the catastrophic forgetting problem for class incremental learning. In each training iteration, this work first retrieves a candidate pool of unlabeled data from a free image stream and then adopts a learned RL strategy to select high-quality unlabeled data to augment the sampled mini-batch. It conducts extensive experiments and achieves consistent improvements on CIFAR-100, ImageNet-100, and ImageNet-1k datasets.
main_review: Pros
- The idea of introducing unlabeled placebo data for alleviating catastrophic forgetting seems interesting and novel for class incremental learning. Such extra training data does not require additional data memory.  
- This paper is well-written and easy to follow.
- The proposed method achieves strong performances across multiple benchmarks, in particular for high-resolution image classification tasks, with higher memory efficiency for old data.

Concerns
- The related work is a bit lacking and many recent related works are missing:

  [A] DER: Dynamically Expandable Representation for Class Incremental Learning (SOTA Method, better than PODNet+AANet)

  [B] On Learning the Geodesic Path for Incremental Learning (better than or comparable to PODNet+AANet on CIFAR100)

- Some of the previous work has explored other types of unlabeled data in the class incremental learning, which should be discussed:

  [C] Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild (ICCV2019)

  [D] More Classifiers, Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning (ECCV2020)

- The effect of RL seems limited. In Table 3, for the method LUCIR+AANets, the performance of RL is only marginally better than w/o RL (<0.5%).  It is unclear if the RL strategy is necessary. 

- Some of the experiment details are unclear/missing: 
1) The experimental setting of Figure 1 seems unclear: Does the experiment of 'KD on new' keep old data? What are the values of temperature and distillation coefficients and how does the method determine them?
2) The values of some hyperparameters are missing:  What is the batch size for training? How many samples are taken from the free image stream (|\mathcal{U}|) as candidates? In each iteration, how many unlabeled data samples (K) are selected from \mathcal{U} for training?
3) RL comparison: In Table.3, What sampling policy is used in No.8 w/o RL? Is it random sampling? 
summary_of_the_review: The proposed CIL strategy, which exploits distillation on unlabeled placebo data, seems novel and effective. However, several SOTA methods are missing in the related work, and the experimental settings and the efficacy of RL need to be clarified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to sample placebo data from a free image stream and use them for knowledge distillation during class-incremental learning. For effective sampling, an RL algorithm is also proposed. Experimental results show that the proposed method improves state-of-the-art methods.
main_review: - The RL-based sampling algorithm looks interesting. The paper is well-written, but some parts could be improved (see suggestions below).

- The first contribution, leveraging "unlabeled placebo data from a free image stream to improve the KD effect in CIL models" has already been studied in a prior work, [Lee et al.]. The second contribution, the RL-based sampling algorithm is not in this prior work, so maybe the authors can take [Lee et al.] as another baseline, though it seems the main performance gain is from the usage of the free image stream rather than the sampling algorithm (by looking at the ablation study in Table 3 & performance gain shown in [Lee et al.]).

- Knowledge distillation can be done in many different forms, so the formulation should be clarified. I assume that the KL divergence loss is used, as 1) they referred to a series of works using it like LwF and iCaRL, and 2) Fig 1 matches the input of the CE loss and KD loss.

- Fig 1(b) is technically incorrect. The KL divergence loss is defined only when two inputs are probability vectors. You should normalize the inputs. If the loss is really computed in this way, the method is technically incorrect.

- Fig 2 (and even the paragraph referencing Fig 2) is not self-explainable, e.g., a description about the way to construct placebos is necessary.

- Could you elaborate more on models used in No. 8 and 9 in Table 3?

[Lee et al.] Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild. In ICCV, 2019.

Just to note, my initial recommendation is under an assumption that the authors address all my concerns well.

## Post rebuttal

As other reviewers agreed, this paper missed some important prior works like [Lee et al.] and [Zhang et al.] at the time of submission. The first contribution "to compute the KD loss using placebo data chosen from a free image stream" is duplicated with them, so the abstract and contribution paragraph in the intro had to be significantly rewritten but they didn't.

However, apart from it, additional comparisons made during the rebuttal period is impressive. Also, I think the second contribution with an RL algorithm is interesting, and the ablation study on it (row 8 and 9 in Table 3) justify its effectiveness properly.

summary_of_the_review: The idea of sampling data from a free image stream for class-incremental learning is not new, but the RL-based sampling algorithm looks new. The paper is well-written, but some parts could be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to use a separate available data stream (called placebos) to distill information during the continual learning. They show that this is almost as efficient as using ‘old data’, and much more efficient than using ‘new data’ (the normal distillation setting). To balance the different losses they propose a reinforcement learning algorithm that judges the quality of placebos.  They obtain good results on several standard datasets. 
main_review: Positive points:
a) the paper confirms earlier observations that external data can significantly improve results on a wide range of KD-based CL methods.

b) paper is well written, and motivation (fig 1 is nice)

c) ablation shows that RL improves results (though as said above I would have liked more insights and stronger baselines).

The paper has the following weaknesses:

a) Related work: the usage of external data has been studied before in continual learning. I am aware of two relevant works:

[1] Zhang, J., Zhang, J., Ghosh, S., Li, D., Tasci, S., Heck, L., Zhang, H. and Kuo, C.C.J., 2020. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1131-1140).
[2] Lee, K., Lee, K., Shin, J. and Lee, H., 2019. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 312-321).

These two methods (and especially [2]) already show the main contribution of the proposed paper, namely that external data can help significantly improve CL. [2] also provides a method to sample from the stream (called 'Confidence calibration for sampling.'). Comparison with these methods is of great importance to evaluate the proposed method. Also, the novelty claim should be adjusted according to these works. 

b) I think the RL algorithm was interesting. However, I would have liked to see a more in-depth analysis of its results, comparison also to simpler baselines (choosing only samples with higher confidence for example). 

c) It would be nice to also see results for the equal-size split on CIFAR 100 (i.e. 10 tasks of 10 classes). I do not see why the proposed method would perform worse in this setting (because of the RL training ?). Anyways even if underperforming for this setting, it would be insightful to include it. 
summary_of_the_review: Regretfully the authors missed very important related work already showing the potential of external data. This drastically reduces the novelty of the paper. This work also proposes a method to select data from the data stream which could be compared to the proposed RL algorithm. Given that comparison with the most relevant related work is missing, I recommend rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper uses data chosen from a free image stream (without labels) to supply knowledge (by the "placebo" samples) for class incremental learning. The empirical results shows that the proposed method exhibits good accuracy in large and high resolution image benchmark datasets such as ImageNet-1k with small memory budget. The paper also presents comprehensive ablation study for each proposed component.
main_review: **Strengths**
- S1: Novel idea of using unlabeled data for knowledge distillation for class incremental learning (CIL).
- S2: Compelling empirical performance with large datasets.
- S3: Clear presentations (i.e., figures, texts and equations)

**Weaknesses**
- W1: Out of protocol of CIL -- using large unlabeled data requires large supplementary data source and require additional computational cost, which may limits the applicability of the CIL method. The rationale behind this claim is that the CIL is for the realistic recognition scenario that the data is given in a stream and the model update should be on the fly, adaptively to the new data. If the computation of incremental update step is large, the benefit of the CIL could disappear.
- W2: Large unlabeled source may or may not contain sufficient information (i.e., even the future information) depending on the choice of source of unlabeled data -- Google searched image could be the superset of the ImageNet-1k as ImageNet dataset is collected from Google search. Thus, the choice of unlabeled data might be the key to success of the proposed method, not the design of the method. Lack of study about the source is a weak point of significance of the proposed method
- W3: The evaluation protocol of using small memory budget is largely beneficial to the proposed method. As the proposed method is using large unlabeled data as an extra source of information, the small memory budget is easily compensated by the proposed method while other methods inevitably performs poorly due to lack of information source.
- W4: Similar to W3, the combined size of episodic memory (i.e., containing samples from the tasks) and the placebo buffer P (i.e., containing samples from the free image stream) should be matched to the episodic memory size in other comparable methods. The memory budget of the compared methods is not clearly stated. In addition, the "budget" and "non-budget" in the ablation study are also not clearly described (what do you mean by "keep the strict memory budget in each phase"? 
- W5: The proposed method seems promoting learning new knowledge. As there is a measure to evaluate the ability to learn new knowledge (e.g., intransigence (Chaudhry et al., 2018)), it is suggested to empirically validate the benefit of the proposed approach by the measure.
- W6: As the method employs reinforcement learning (RL), the computational cost would be considerable. But there is no study about the computational complexity.
summary_of_the_review: Given the strength and the weaknesses, the paper has decent novelty of using extra free data stream for improving continual learning methods but may be less fairly evaluated (i.e., the evaluation protocol is designed favorably towards the proposed method) with other compared methods.
review_title: null

Meta-review: This paper presents a novel method for class-incremental learning (CIL) with the help of placebo data chosen from a free image stream. Such placebo data are unlabeled and easy to obtain in practice. To adaptively generate phase-specific functions as the accurate estimation of placebos' quality for KD, this paper applies reinforcement learning based on the constraints of the CIL. The effectiveness of the method has been verified on multiple datasets, including ImageNet-1k and ImageNet-Subset with both lower memory and higher accuracy than baselines. The major concern is about the novelty that the unlabeled auxiliary data is not quite new for CIL despite the minor difference in settings and methods. Moreover, the improvements over the baselines are not significant enough, which is a minor concern.
Decision: Reject