Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates a semi-supervised continual learning (SSCL) setting and proposes a new method called DistillMatch for this setting. The major contributions are: (1) The authors carefully design a realistic SSCL setting where object-object correlations between labeled and unlabeled sets are maintained through a label super-class structure. And then, they develop the DistillMatch method combining knowledge distillation, pseudo-labels, out of distribution detection, and consistency regularization. (2) They show that  DistillMatch outperforms other existing methods on CIFAR-100 dataset, and ablation study results are shown also. 

However, there are some downsides that should be considered before its publication. (1) In abstract the authors claim that they can significantly reduce the memory budget (of labeled training data) by leveraging unlabeled data (perhaps with large volume). This motivation seems to be contradictive. (2) From a methodological viewpoint, the proposed DistillMatch method is just a combination of existing methods (listed as in above). So where is the novelty of this "new" method? (3) In experiments, the chosen baseline algorithm is very weak. There are some strong baseline methods such as GEM, A-GEM, and ER. So I wonder to know the real improvements over state-of-the-art methods for continual learning. (4) The label super-class structure existed in CIFAR-100 has been used in their experiments. But this is not very common for other more realistic datasets such as miniImageNet. If there is no super-class structure, we don't know how to apply the proposed DistillMatch method. 

In summary, I think this semi-supervised continual learning setting is interesting, but the proposed DistillMatch method can not persuade me that this method is a novel significant contribution to this problem. So at present time I believe there is much room for the authors to improve their method before publication. 
review_title: The proposed DistillMatch method appears to be a combination of knowledge distillation, out of distribution detection, consistency regularization and several other small tricks.

final_score: rating: 6: Marginally above acceptance threshold
review_content: - Summary:
This paper proposes class-incremental learning with unlabeled data correlated to labeled data, and a method to tackle it. The task can be considered as a variant of [Lee et al.], which has no assumption on the unlabeled dataset, while this paper assumes the correlation between labeled and unlabeled dataset explicitly. The proposed method is inspired by state-of-the-art class-incremental learning, semi-supervised learning, and out-of-distribution (OoD) detection methods: local distillation [Li and Hoiem], OoD detection [Hsu et al.], consistency regularization and pseudo labeling (or hard distillation) [Sohn et al.], and loss balancing based on class statistics [Lee et al.]. Experimental results support that the proposed method outperforms prior works in the proposed task.

- Reasons for score:
1. Extending continual learning to the semi-supervised setting is natural, given that the extension to self-taught learning has already been considered in [Lee et al.]. However, I cannot agree that semi-supervised learning is more realistic than self-taught learning, which is emphasized throughout the paper 18 times. In an early work of [Raina et al.], self-taught learning is proposed to make the scenario of learning with unlabeled data "widely applicable to many practical learning problems." [Oliver et al.] also argued that "(unlabeled data from out-of-distribution) violates the strict definition of semi-supervised learning, but it nevertheless represents a common use-case for semi-supervised learning (for example, augmenting a face recognition dataset with unlabeled images of people not in the labeled set)." I am not saying that semi-supervised learning is unrealistic, but the argument in this paper sounds overclaimed. I believe both semi-supervised and self-taught learning are realistic in some cases. I also recommend to provide real world scenarios that the proposed task (correlation between labeled and unlabeled data exists and no memory for coreset is available) is useful in practice.
2. The proposed method is not novel, which is essentially the combination of state-of-the-art methods in relevant tasks. But I do not discount this much, because this work would be valuable as the proposed task is interesting but not investigated before. However, the name of task might need to be changed, because a similar name, "semi-supervised incremental learning" is already taken by a kind of semi-supervised learning, which incrementally incorporates unlabeled data to training.
3. Though the improvement over prior class-incremental learning methods is impressive, the overall performance is still too low. In fact, the scale of the experimental setting is too small, so I doubt it is scalable. All experiments are bounded on CIFAR-100, and even only 20% of training data are used as labeled one. Frankly, in this small-scale setting (in both number of data and image resolution), keeping all data is just fine, as the coreset size is negligible compared to the model size. I recommend to experiment in large-scale settings, e.g., on ImageNet. Also, I recommend to compare the oracle setting as well, which keeps all previous training data.
4. In addition to small-scale experimental setting, the architecture is larger than the prior work [Lee et al.]: WRN-28-2 vs. WRN-16-2. In the worst case scenario, it is possible that the best performance of the proposed method is simply from the complexity of their learning objective, i.e., all methods overfit to training data, but the proposed method did not have enough updates to overfit to them.
5. In Figure 3, why do GD and DM not have a coreset? I think there is no reason to give an unfair constraint to them. I recommend to draw curves with respect to increasing number of coreset for those methods as well.
6. Could you provide results on the self-taught learning setting like [Lee et al.]? It would also be interesting to see the performance of the proposed method in the setting.
7. Hyperparameter sweep results provided in Table 4 are either minimum or maximum of the range, so you could improve the performance by enlarging the range.

- Minor Comments:
8. Subscripts of theta often are dropped. Is theta equal to $\theta_{n,1:n}$?
9. "the parameters of no more than three models" -> I believe it is four, because you need to temporarily store gradients during training.
10. $\hat{q}$ is not a probability vector, which makes eq. (2) mathematically do not make sense.
11. Citation format issue: you can use \citet for noun and \citep for adverb.
12. typo on page 5: statoe -> state
13. Table 4: what is TPR here? threshold for consistency regularization?

[Raina et al.] Self-taught Learning: Transfer Learning from Unlabeled Data. In ICML, 2007.

[Li and Hoiem] Learning without forgetting. In TPAMI, 2017.

[Oliver et al.] Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. In NeurIPS, 2018.

[Lee et al.] Overcoming catastrophic forgetting with unlabeled data in the wild. In ICCV, 2019.

[Hsu et al.] Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In CVPR, 2020.

[Sohn et al.] Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.

**After rebuttal**

I'd like to thank authors for their efforts to address my concerns. They have addressed most of them, so I increased my score from 5 to 6.

However, there are two concerns that couldn't be resolved during the rebuttal period:

(1) I am still not sure if the proposed task is practical. At glance it looks realistic, but I couldn't find a detailed scenario that can only be solved by the proposed task. Any real world scenario I can think of is closer to [Lee et al.], which is a prior work of this paper. Authors provided an exploring robot example in the thread of responses, but I think [Lee et al.] fits better for the provided one. I recommend authors to find a concrete use-case in real-world applications, which can only be solved by the proposed setting (or at least [Lee et al.] is not applicable; in the revised intro, you may emphasize that there are some real-world problems that [Lee et al.] is not applicable but yours is). R1 and R4 seem to have a similar concern.

(2) the scale of experiment is too small. As CIFAR-10/100 have a limited number of data for your purpose,  you can borrow some data from tinyimages (FYI, CIFAR-10/100 are a subset of 80M tinyimages) or focus on ImageNet.

I am okay with the lack of novelty on the proposed method. For a newly proposed task, I think proposing a simple and effective baseline is good enough. However, because of the two concerns above, I cannot strongly agree with its acceptance.


review_title: The proposed task is interesting, but more experiments are required

final_score: rating: 7: Good paper, accept
review_content: The paper presents a novel semi-supervised continual learning (SSCL) setting, where labeled data is scarce and unlabeled data is plentiful. The proposed framework is built on pseudo-labeling, consistency regularization, Out-of-Distribution (OoD) detection,
and knowledge distillation in order to reduce the catastrophic forgetting in the proposed setting.

The paper is in general clear and well-written. The contributions are clearly highlighted and the proposed approach is conveniently compared with other state of the art methods, demonstrating its superiority.

Positive aspects: 
-  the definition of a realistic, semi-supervised setting for continual learning
- a novel approach for continual learning in order to cope with 'catastrophic forgetting'
- the proposed approach is memory efficient, since it does not need exemplars to replay past tasks

Negative aspects:
- the OoD implemented in this paper rejects the unknown samples. In other words, all unknown samples are considered a single class. It would have been a plus to distinguish between several unknown classes and somehow introduce them in the framework
- the lack of recabilibration step after a number of tasks (in the case of pseudo-labeled samples), could lead to an undesired error propagation which is not quantified in the paper 

However, I have some questions:
1. What is the relationship between the 'fi' and 'theta' models (section 4)? Are they completely separate or there is a relationship between them?
For instance, when 'theta' is extended with a new task, is 'fi' extended accordingly? Or is 'fi' trained off-line from the beginning (with all tasks)?
2. There are some different source of errors: distilation, pseudo-labels... Do you perform any kind of system re-calibration? After how many tasks? I mean, do you make a study of error propagation of pseudo-labeled data? Or at some point do you have a human-in-the-loop to correct mis-classification? What is the mis-classification error of pseudo-labeled samples?
3. Do you assume that labeled and unlabeled data come from different distributions or you have a single distribution which is 
divided in labeled and unlabeled data at the beginning of the process?
4. Does your scenario foresee that when learning a new task T, all the previous tasks are represented (1..T-1) in the unlabebeld data or only a subpart? (i.e. kind of selective replay)
5. When the number of tasks increases, the number of unlabeled data per task remains constant or is scaled accordingly (i.e. reduced) ?
6. Would be interesting to test your approach in a real-world scenario, i.e. robot navigation.
review_title: Recommendation to Accept

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper comes up with a novel scenario where the unlabled data are available as well as labeled data in the continual learning scenario.

### Overall
- Based on my understanding, the major contribution is the proposal of a task scenario, aka, experimental setting. The novelty of DistillMatch is an incremental modification of previous work. 
- The task setting sidesteps the learning with non-stationarity problem than solving it.
- Further, this setting potentially makes the task easier for the proposed method. To verify whether this is true, more information are needed.
- The presentation of the paper needs polishing, I listed a few points below.

### Pros
- The novel scenario of semisupervised continual learning is proposed. The argument is that in several realistic scenarios old data are often re-observed without label (The funiture labeling example). Therefore instead of storing a coreset, one may make use of the unlabeled data for pseudo-rehearsal/distillation. It is reasonable to make use of it when this assumption is true.
- With the setting the author proposed, the DistillMatch method is able to perform better than previous methods. 

### Cons
1. The novelty mostly comes from the task scenario, the DistillMatch method is incremental.
2. Although SSCL is a new scenario, and the author argues it is more realistic. IMO taking this assumption sidesteps the problem of continual learning rather than solving it. The central problem of continual learning IMO is to learn under non-stationary distribution, the assumption made in this submission makes the distribution more stationary.
3. It is true that this assumption should be utilized when available. However, the only dataset used is manually constructed from CIFAR100, contradicting the initial motivation to move towards a more realistic scenario.
4. There's a lack of information on how the compared methods are adapted to the new scenario. 
I searched the supplementary but failed to find a detailed documentation. With the given information, it is hard to tell whether the comparison is fair. My concerns are following,
**increasing from 3 -> 4 as this point is resolved in the rebuttal**
  - In the RandomClasses setting, it is stated that no coreset is used, if the compared methods depends on coreset to replay, it would be unfair. If that's the case, the only conclusion we can draw is that replay is better than no replay, which seems trivial to me.
  - GD depends on internet crawled data, is it replaced with the unlabeled data since it is available in the experiment setting? If not, then I think it is just the setting that favors DistillMatch.
  - With the above said, I suggest the author to list clearly the objectives, replay buffer sizes or even pseudo code for each of the compared method and their own method in a table, which will help the reader identify what major component in the proposed method is making the contribution.

Regarding the quality and clarity,
I found myself confused and making guesses sometimes while reading it.

To list a few:
- introduction paragraph 2, ... to determine which unlabeled data is relevant to the incremental task ..., I guess the incremental task means learning the newly observed data, but then for rehearsal we'll pick the unlabeled data which is from the distribution of past tasks.
- section 1, ... save up to 0.23 stored images per processed image over naive rehearsal (compared to Lee) ..., here seems Lee et al is the naive rehearsal. But then "which only saved 0.08" confuses me, seems to be saying Lee saves 0.08 compared to naive rehearsal.
- section 3, ... where data distributions reflect object class correlations between, and among, the labeled and unlabeled data distributions ... not enough information to infer what "reflect" and "object class correlation" means here.
- section 4, ... Let S_{n-1} denote the score of our OoD detector for valid classes of our pseudo-label model ... what is the "valid classes" needs to be clarified. As I understand it, S_{n-1} measures how likely the unlabeled data is in the distribution of past tasks.
- Super class / Parent class are not defined clear enough.
review_title: Proposes a semisupervised task scenario for continual learning

Meta-review: This paper proposes a semi-supervised setting to reduce memory budget in replay-based continual learning.
It uses unlabeled data in the environment for replaying which requires no storage, and generates pseudo-labels where unlabeled data is connected to labeled one.
The method was validated on the proposed tasks.

Pros:
- The semi-supervised continual learning setting is novel and interesting.
- The proposed approach is memory efficient, since it does not need exemplars to replay past tasks.

Cons:
- The scale of experiment is small. It lacks evaluation in real world environment.
- The novelty is limited, because it is a combination of existing technologies: pseudo-labeling, consistency regularization, Out-of-Distribution (OoD) detection, and knowledge distillation.
- The comparison might not be fair due to different settings.

The authors addressed the fairness and scalability with additional experiments
and leave some suggestions of reviewers for future work.
R3 had a concern on the error propagation of pseudo-labels which I also share. The authors agreed that this is a challenge for all CL methods.

In summary, the reviews are mixed. All reviewers agree that the semi-supervised continual learning setting is novel and interesting, and some have concerns on scalability and novelty of the method which I also share. So at present time I believe there is much room for the authors to improve their method and experiments before publication.

Decision: Reject