Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer
Authors: James Smith
Jonathan C Balloch
Yen-Chang Hsu
Zsolt Kira
Venue:  (2021)
Keywords: continual learning
semi-supervised learning

Abstract: Rehearsal is a critical component for class-incremental continual learning, yet it requires a substantial memory budget. Our work investigates whether we can significantly reduce this memory budget by leveraging unlabeled data from an agent's environment in a realistic and challenging continual learning paradigm. Specifically, we explore and formalize a novel semi-supervised continual learning (SSCL) setting, where labeled data is scarce yet non-i.i.d. unlabeled data from the agent's environment is plentiful. Importantly, data distributions in the SSCL setting are realistic and therefore reflect object class correlations between, and among, the labeled and unlabeled data distributions. We show that a strategy built on pseudo-labeling, consistency regularization, Out-of-Distribution (OoD) detection, and knowledge distillation reduces forgetting in this setting. Our approach, DistillMatch, increases performance over the state-of-the-art by no less than 8.7% average task accuracy and up to a 54.5% increase in average task accuracy in SSCL CIFAR-100 experiments. Moreover, we demonstrate that DistillMatch can save up to 0.23 stored images per processed unlabeled image compared to the next best method which only saves 0.08. Our results suggest that focusing on realistic correlated distributions is a significantly new perspective, which accentuates the importance of leveraging the world's structure as a continual learning strategy.

Under review as a conference paper at ICLR 2021
MEMORY-EFFICIENT SEMI-SUPERVISED CONTINUAL LEARNING: THE WORLD IS ITS OWN REPLAY BUFFER
Anonymous authors Paper under double-blind review
ABSTRACT
Rehearsal is a critical component for blass-incrementee continual learning, yet it requires a substantial memory budget. Our work investigates whether we can significantly reduce this memory budget by leveraging unlabeled data from an agent's environment in a realistic and challenging continual learning paradigm. Specifically, we explore and formalize novel semi-supervised continual learning (SSCL) setting, where labeled data scarce yet non-i.i.d. unlabeled data from the agent's environment is plentiful. Importantly, data distributions in the SSCL setting are realistic and therefore reflect object class correlations between, and among, the labeled and unlabeled data distributions. We show that strategy built on pseudo-labeling, consistency regularization, Dut-of-Distribution (OoD) detec- tion, and knowledge distillation reduces forgetting in this setting. Our approach, DistilIMatch, increases performance over the state-of-the-ee by no less than 8.7% average task accuracy and up 54.5% increase in average task accuracy in SSCL CIFAR-100 experiments. Moreover, we demonstrate that DistillMatch can save up to 0.23 stored images per processed unlabeled image compared to the next best method which only saves 0.08. Our results suggest that focusing on realistic correlated distributions is a significantly new perspective, which accentuates the importance of leveraging the world's structure continual learning strategy.
INTRODUCTION
Computer vision models in the real-world are often frozen and not updated after deployment, yet they may encounter novel data in the environment. Unlike the typical supervised learning setting, class-incremental continual learning challenges the learner to incorporate new information as it sequentially encounters new object classes without forgetting previously-acqurre knowledge (catastrophic forgetting). Research has shown that rehearsal of prior classes is critical component for dlass-incrementa continual learning (Hsu et al.) 2018 van de Ven & Tolias 2019). Unfortunately. rehearsal requires substantial memory budget, either in the form of coreset of stored experiences or a separate learned model to generate samples from past experiences. This is not acceptable for nemoory-oostrainee applications which cannot afford to increase the size of their memory they encounter new classes.
Instead, consider a novel real-world setting where an incremental learner's labeled task data is a product of its environment and the learner encounters a vast stream of unlabeled data in addition the labeled task data. In such setting (visualized in Figure D. the unlabeled datastream is intrinsically correlated to each learning tasks due to the underlying structure of the environment. We explore many ways in which this correlation may exist. For example, when an incremental learner is tasked to learn samples of the previously-unseen class at time i in the real world, examples of e may be encountered in the environment (in unlabeled form) during some future task. In such setting, incremental learner could use the unlabeled data in its environment as source of memory-free rehearsal, though it would need a method to determine which unlabeled data relevant to the incremental task (i.e. detecting in-distribution data).
We formalize this realistic paradigm in the semi-supervised continual learning (SSCL) setting, wherein unlabeled and labeled data are not i.i.d. as they correlated through the underlying structure of the environment. We propose and conduct experiments over a realistic setting in which this correlation may exist, in the form of label super-class structure (e.g. unlabeled examples

---

Under review as a conference paper at ICLR 2021
Dar
Convitt
Daso Storaga
Apunti
Continual Learning Aigert
Candraal
Schome A: Internal Replay
Scheme B: External
Schama Reallsttle Raplay
Replay from Web
Environment
Figure I: Unlike standard replay (scheme A) which requires a substantial memory budget, explore the potential of an unlabeled datastream serve as replay and significantly reduce the required memory budget. Unlike previous work which requires access to an external datastream uncorrelated to the environment (scheme B), we consider the datastream to be a product of the continual learning agent's 's environment (scheme C).
of household furniture such chairs, couches, and tables will appear while learning the labeled examples of household electrical devices such as lamp, keyboard, and television (Krizhevsky et al. 2009 Zhu & Bain 2017)) We measure the final-task accuracy A, the accuracy over all tasks 12, and the coreset memory required to attain a specific level of Q accuracy over several realistic SSCL settings. Our experiments demonstrate that state-of-the-ar continual learning methods (Lee et al. 2019) perform inconsistently in the novel SSCL paradigm with no prior method performing "best" across all settings. This leads us to ask "How can an approach to catastrophic forgetting be robust to several realistic, memory-constrained continual learning scenarios?"
To answer the above question, we propose a novel learning approach that works well in both the simple (i.e., no correlations) and realistic SSCL settings: Mathh. We leverage unlabeled data not only for knowledge distillation (in which the distilling model is fixed), but also for a supervepe loss (in which the supervisory signal can adapt during training the new task). Key to our approach is that we address the distribution mismatch between the labeled and unlabeled data (Oliver et al. 2018) with setting). Compared to nearest prior state-of-the-an (all methods from (Lee et al. 2019)) configured work well as possible in the novel SSCL setting, our experiment scenarios by as much as a 54.5 % increase in Ω and no less than an 8.7% increase. Furthermore, we find that our method can save up to 0.23 stored images per processed unlabeled image over naive rehearsal (compared to (Lee et al. 2019) which only saved 0.08 stored images per processed unlabeled image). summary, we make the following contributions:
(OoD) detection (and are the first to do so in the continual learning
outperform the state-of-the-art in all of
1. We propose the realistic semi-supervee -uperviue continual learning (SSCL) setting, where object-object correlations between labeled and unlabeled sets are maintained through label suber--uppppeee super-class structure. We show that state-of-the the art continual learning methods perform inconsistently in the SSCL setting (i.e. no baseline method is "best" across all settings).
We propose a novel continual learning method DistillMatch for the SSCL setting leveraging pseudo-labeling, strong data augmentations, and baselines, DistillMatch achieves superior performance on a majority of metrics for 8/8 experiments and results in substantial memory budget savings.
detection. Compared to the
2 BACKGROUND AND RELATED WORK
Several related methods leverage distillation losses on past tasks to mitigate catastrophic forgetting using soft labels from a frozen copy the previous
Knowledge Distillation in Continual Learning:
task's model (Castro et al., 2018 Hou et al. 2018 Li & Hoiem 2017 Rebuffi et al.] 2017). For example, learning using two teachers, with one teacher distilling knowledge from previous tasks and another distilling knowledge from the current task, has been found to increase adaptability to task while preserving knowledge on the previous tasks (Hou et al. 2018 Lee et al. 2019). Class- balancing and fine-tuning have been used encourage the model's 's final predicted class distribution
2

---

Under review as a conference paper at ICLR 2021
to be balanced across all tasks (Castro et al. 2018 Lee et al. 2019). These methods are related in that they rely on distillation losses to mitigate catastrophic forgetting, but the losses are designed to distill knowledge about specific local tasks and cannot discriminate between classes from different tasks (crucial for class incremental learning). More context on where our work fits into the greater body of continual learning research is provided in Appendix H.
Global distillation (GD) introduces a global teacher which provides a knowledge ensemble from both the past tasks and current task (Lee et al. 2019). This addresses a crucial shorteoming of common knowledge distillation methods which do not reconcile information from the local tasks (i.e. the groups of object classes presented sequentially to the learner) with the global task (i.e. all object classes seen at any time). GD leverages a large stream of uncorrelated unlabeled data from sources such as data mining social media or web data (Lee et al. 2019 to boost its distillation performance. Similar to GD, we leverage an unlabeled datastream to mitigate forgetting, but we take the perspective that this datastream is from the agent's environment and reflects object-object correlation structures imposed by the world (i.e. correlations between the task data and the unlabeled data).
Out of Distribution Detection: Leveraging unlabeled data for rehearsal is key to our work, but can contain mix of classes not in the distribution of the data seen by the learner thus far. Therefore, we include out-of-Distribution (OoD) Detection (Hsu et al. 2020 Lee et al. 2018 Liang et al. 2017) to select unlabeled data corresponding to the classes our learner has seen so far with high confidence. Semantic Ool detection is a difficult challenge (Hsu et al. 2020p and we do not have access to any known OoD data to calibrate our confidence. We therefore build Confidence (DeConf) (Hsu et al. 2020), which can be calibrated using only in-distribution training data. The method consists of decomposed confidence scoring with a learned temperature scaling in addition to input pre-processing. For further details, the reader is referred to (Hsu et al. (020).
recent method, Decomposed
Semi-Supervised Learning: In semi-supervised learning (which motivates the SSCL setting), mod- els are given a (typically small) amount of labeled data and leverage unlabeled data to boost per- formance. This is an active area of research given that large, labeled datasets are expensive, but most applications have access to plentiful, cheap unlabeled data. There are several approaches to
eeiissuerviised learning (Berthelot et al. 2019 Lee 2013) Kingma et al. 2014 Kuo et al., 2019) Miyato et al. 2018 Oliver al.] 2018 Sohn et al.) 2020 SSpringebberg] 2015 Tarvainen & Valpola 2017) which involve balancing supervised loss E, applied to the labeled data with an unsupervised
loss Eut applied to unlabeled data. Additional details these methods are provided in Appendix H
3 SSCL SETTING
In caassinncrementl continual learning, a model is gradually introduced to labeled data corresponding to M semantic object classes Cl2 c..... CM over series of N tasks, where tasks are non-overlapping subsets of classes. We use the notation T, to denote the set of classes introduced in task n, with [T,] denoting the number of object classes in task n. Each class appears in only single task, and the
goal is to incrementally learn to classify new object classes as they are introduced while retaining performance on previously learned classes. The plas-inncrementl learning setting (Hsu al. (2088) is a challenging continual learning settings because no task indexes are provided to the learner during inference and the learner must support classification across all classes seen up to task n.
We extend the class-incremental continual learning setting in the realistic semi-supervised continual learning (SSCL) setting, where data distributions reflect existing object class correlations between, and among, the labeled and unlabeled data distributions. The amount of labeled data in this setting is drastically reduced as is common in supervieee learning. For example, our experiments reduce the number of labeled examples per class by 80% compared to a prior setting (Lee et al.) 2019). At task n, we denote batches of labeled data as X₁, {(T,, yb): batches of unlabeled data as Ud {uu: b E (1, hyperparameter describing the relative size of Xr, to Un The goal in task n is to learn a model On which predicts object class labels for any query input over all classes seen in the current and previous tasks (Ti UT2 U... U T.)) The index n on 6, indicates that our model is updated each task; i.e. f refers to model from the previous task and f., refers to the model from the current task.
(1,.... B) y6 E E7m) and ,BB). Here, B refers to batch-size and μ is
To simulate an environment where unlabeled and labeled data are naturally correlated, we leverage well-defined relationships between objects derived from super-class structure (i.e. various animals
3

---

Under review as a conference paper at ICLR 2021
Uniabeledd Data Supar Chios
MistilMatch Modal, Tosk
[Seettor)
Oonaitiancy ReguArtzadaon Equater H
Strvnn Aspronsiiion
Coneleet
Cless Retaread dR Lase Squilber (J, db
Figure 2: Diagram of DistillMatchh, our proposed algorithm for semi superviie continual learning. We use unlabeled data from the agent's environment to ground the current model in the past task while simultancously learning new knowledge. This done with combination of local distillation, consistency regularization, and pscudo-labeling unlabeled data with confident predictions from an Mii-o--Distribution Detector.
within one super-class). We use the CIFAR-100 dataset (Krizhevsky et al. 2009) because object correlations among classes and parent classes, crucial to our experiments, are well defined and explored Zhu & Bain 2017). This dataset contains eight unbalanced super-classes, which we use to simulate realistic data environments. Each super-class contains number of parent classes (e.g. one super-class contains the parent classes flowers, fruits/vegetables, and trees). There are 20 parent classes total which form the 20 continual learning tasks, with each parent class consisting of five object classes (e.g. flowers parent class consists of orchids, poppies, roses, sunflowers, tulips). For a single task, when learner is being shown labeled training data from one of the parent classes (e.g. flowers, fruit/vegetables, or trees), the unlabeled data for this task will contain examples from the entire super-class (e.g. flowers, fruits/vegetalles, and trees).
SSCL with this realistic super-class "environment" structure is our main setting, but we also explore several other correlation combinations, including the simple SSCL setting without any super-class structure. We use the following terminology to describe the correlations of the tasks (i.c. labeled data): RandomClass Tasks, where no correlations exist in task classes, and ParentClass Tasks, where tasks are introduced by CIFAR-106 parent classes (i.e. each task is to learn the five classes of single CIFAR-100 parent class). For the unlabeled data distribution we have: Uniform Unlabeled, where all classes are uniformly dsitributed in unlabeled data for all tasks, PositiveSuperclas Unlabeled, where the unlabeled data of each tasks consists of the parent classes in the same super-class as the current task, NegativeSSuprrclas Unlabeled, where the unlabeled data of each tasks consists of parent classes from different super-class as the current task, and data of each task consists of 20 randomly sampled classes (roughly equal to the average class size in a super-class). Further discussion and details, including figures depicting example streams for each task sequence, are provided in Appendix
where the unlabeled
4 APPROACH
Local Distillation Preserve Past Tasks 1... Using Unlabeled Data: Our approach, sum- marized in Figure 2 is distillation-baser approach and therefore uses a standard local distillation loss (as in prior methods). The intuition of knowledge distillation is that the current model should make similar predictions to previous models over the set of classes associated with the previous tasks. Refining prior notation, this loss depends on 0i,:: the model of 0 at time i that has been trained up to, and including, data with the classes from task n. For example, On, refers to the model trained during task and its logits associated with all tasks up to and including class n. Let denote po(y x) as the predicted class distribution produced by model 9 for input æ. Using this notation, last during the training of task is given as:
1
last - Σ
B Σ
Last Last (po,, (y ((ra)), Po.,.(0 (y (xx))))
where Last is distance loss such KL divergence, and denotes weak data augmentations such as random horizontal flips and crops. This loss acts as a regularization penalty to encourage the current model make similar predictions the previous model for all tasks 1 1.
4

---

Under review as a conference paper at ICLR 2021
OoD Detector Training and Calibration: We train and calibrate OoD detector in the SSCL setting in order to identify unlabeled data previously seen by the agent. OoD detection calculates scalar score S(us) for unlabeled input Ub, and rejects up where TOoD is a calibrated threshold. OoD scores in our SSCL setting include the time index nt S,, (un), and are generated from separate Ool model on using the method in (Hsu al.. 2020). We use unlabeled data considered as in-distribution for a hard distillation loss, and we use all unlabeled data for soft knowledge distillation and consistency losses. We use separate model because calibrating the decision threshold TOaD requires labeled hold-out data, which we cannot afford to sacrifice in our main classification model given we are already working in a limited labeled data regime. We hold-out 50% of the labeled training data (across all tasks) when training on for this calibration decision. At the end of each task n, we calibrate TOoD to operate at a d% true-positive ratio (TPR) using the hold-out labeled data, where å is scalar hyperparameter. For computational efficiency, we exclude unlabeled losses when training Φn. We note that our implementation uses the same number of models (and therefore parameters) as used in the prior kate-of-the-an method
-utoof-distribution if ((,)) < TOoD.
GD Lee et al. 2019).
Confidence- Based Hard Distillation Preserve Past Tasks 1...n 1 Using Unlabeled Data:
In our experiments, we found that eq. ① works well at preserving performance on local tasks, but does not work well on distilling knowledge from the global task (i.e. object classes from tasks 1 nn). We demonstrate hard distillation (cross entropy loss with a one-hot vector label) preferable over soft distillation because hard distillation distills knowledge across all classes (i.c. the global task). Specifically, we distill global knowledge from task n - 1 into task n by pseudo-label unlabeled data by generating predicted class distributions for unlabeled images as: qb.n-1 Po.-1.1:n-11 (y/ | a (()))), and one-hot pseudo-labed as -b.n-1 = arg max(9t.n-)) We identify highly confident data on task
using frozen copy of 1n11,1:n-1
1 using the DeConf OoD detection described in Section 2 Let S,- (us) denote the score of our OoD detector for past task classes of our pseudo-label model, 0,-1,1sm-1- Then, our confidence-basee hard distillation loss for
the pseudo-labeled data Ept becomes:
BB Σ Bpt 6m
Ept
1(Sn-1(u)) TooD) ( a(u)))
where (p q) is the cross-entropy between probability distributions p and q and Bpt is the number of pseudo-labeled examples in the given batch identified with OoD detection. We normalize the batch by Bp and not μB so that Ept can be class balanced alongside the labeled training data (described in the next subsection). Note that we only need to retain -1-1m at task n..
Semi-Supervised Class Balancing Balance Past Tasks 1... n 1 with New Task n Using Labeled and Unlabeled Data: Our method is designed to work even in the absence of a coreset (stored images) when unlabeled data from past tasks available. This is due the design choice of including hard pseule-labeled data from a frozen model copy. A problem arises from this approach in sensitivity to class imbalance, however. Specifically, the distribution of classes for a given batch becomes imbalanced when considering both labels y and pseudo-labels fh.n-1. This is because the number of examples per class in the labeled training data is unlikely to equal the number of examples per class in the pseudo-labeled training data, given these come from two different distributions. To address this, we weight loss components proportionally to the distribution of both labeled training data and hard pseudo-labeles unlabeled data. Specifically, scale the gradient computed from data with label or pseudo-label l E .1,,..,,A, ..) by:
w(k) K (HHO. ((I,8) E X + Ku E UnJ X 1y k}] Hu Un -n-1 k}
Consistency Regularization - Learn New Task n. Using Unlabeled Data: We examine the effects of a consistency loss introduced in FixMatch (Sohn et al. 2020) in the SSCL setting to leverage unlabeled data for learning the current task (rather than only preserving knowledge on the past tasks). This loss enforces consistency between weakly and strongly augmented versions of unlabeled data which increases robust decisions on highly confident unlabeled examples. Strong data augmentations, denoted as A, include RandAAggennt (Cubuk et al., 2019). The model generates a predicted class distribution from weakly-augmented version of the unlabeled image: qb po...... (y (u.)).
5

---

Under review as a conference paper at ICLR 2021
Using generated one-hot pseudo-label 9b arg max(q)). the unlabeled loss tui is calculated as:
1 J1B 1 (mmx(()) 2 TFM))CE((0 po.. (y A((u))))
fut
(4)
where TEM is confidence threshold (scalar hyperparameter) above which pseudo-label is retained. Final Loss Our final, balanced loss looall given as:
Lcotal
(pd) (Co + Ept) + dudlau + Aastldst
(5)
B+ Bpd
8, = Σ
w(yb) Lcw(.b. Po.. a(x))))
(6)
b=1
BBB
pxt w( 1( 2 (c (y a(w)))
Σ
where bu is taken from eq. 4 l, is the supervised task loss, last is from eq. Q. and Xmot, Adse are
hyperparameters which weight of unlabeled consistency loss and local distillation, respectively.
EXPERIMENTS
We evaluate DistillMatch and state-of-the-arr baselines under several realistic SSLC scenarios with the CIFAR-100 dataset using 100 labeled examples per class. We choose recent distillation methods which
can leverage the unlabeled data with distillation losses: Distillation and Retrospection (DR) (Hou
al.
2018). End-to-Ene incremental learning (E2E) CCstro et al., 2018), and Global Distillation
(GD)
Lee et al. 2019). Similar to GD (Lee et al. 2019 we do not compare to model-regularization
methods (i.e., methods which penalize changes to model parameters, discussed Appendix
because distillation methods have been found to perform better in the inssemnncreettall learning setting (Lee et al. 2019p. Besides, these methods are orthogonal to our contribution and could be combined with our approach (and the competing approaches) for better performance. We use implementations DR and E2E from (Lee et al., 2019). which are adapted from incremental task learning to incremental class learning. These implementations of DR and E2E use the unlabeled data for their respective knowledge distillation loss(es), as shown Appendix !| We also compare to neural network trained only with cross entropy loss labeled data (Base).
5.1 METRICS
We evaluate methods using: (I) final performance, or the performance with respect to all past classes after having seen all N tasks (referred to as A,,i:);; and (II) Ω or the average (over all tasks) normalized task accuracy with respect As before, we use index i to index tasks through time and index to index tasks with respect to test/validation data (for example, Ai, describes the accuracy of our model after task on task n data). Specifically:
offline oracle method (Hayes & Kanan 2019).
A
(0(xx,0..) = T.))
Dreat
(a,y)
N N
Tm
A
Ω
(9)
Tha
Aoffline
==
designed to evaluate the global task and is therefore computed with respect to all previous classes. For the final task accuracy in results, will denote ANTIN simply AN.

---

Under review as a conference paper at ICLR 2021
Table 1: Results (%) CIFAR- 100 with 20% Labeled Data for (a) RandomClass Tasks for various task sizes (mo Coreset, Uniform Unlabeled Data Distribution) and (b) ParentClass Task various coreset sizes (20 Tasks, Uniform Unlabeled Data Distribution). Results are reported as an average of 3 runs, with std provided in the supplementary material. UB refers to the upper bound, given by the offline oracle.
(b)
Tasks
5
20
Coreset
400
Metric
Metric
Av
Q
Ax
Ω
Ax
Ω
Ax
An
(1)
(1)
UB
56.7
100
56.7
100
56.7
100
UB
56.7
100
56.7
100
Base
15.6
52.5
8.2
34.7
4.3
22.0
Basc
3.5
18.5
14.6
53.4
E2E
12.5
46.1
7.5
32.3
4.0
21.1
E2E
3.2
18.1
19.5
59.3
DR
16.0
53.7
8.3
36.4
4.3
22.4
DR
3.7
19.4
20.1
57.8
GD
32.1
69.9
21.4
60.0
13.4
42.7
GD
10.5
37.4
21.4
57.7
DM
44.8
84.4
37.5
76.9
21.0
60.8
DM
20.8
57.8
24.4
67.5
5.2
OTHER DETAILS
We do not tune hyperparameters on the full task set because tuning hyperparameters with hold out data from all tasks may violate the principal of continual learning that states each task in visited only once (van de Ven & Tolias 2019). Following (Lee et al. 2019,, include coreset for many experiments (although we show our method does not need a coreset when certain object correlations are present between unlabeled and labeled data) which used for both the labeled cross entropy loss and the distillation loss. Given that our SSL tasks use 20% of the labeled data from (Lee et al. (019)) we also reduce the coreset 20% of their coreset size (from 2000 to 400). Notice that no methods have a memory budget for unlabeled data (i.e. the unlabeled data is from stream and discarded after use rather than stored). We include supplementary details and metrics in our Appendix: additional ablations A additional experiment details (B), hyperparameter selection ( full results including standard deviation and plots (D. OoD performance and super- super-parent class associations
5.3 RESULTS
We do not need to store coreset data in an environment with uniform unlabeled data. We first evaluate our methods in a scenario where unlabeled data is drawn from a uniform distribution over all classes (Table [a]. This represents the situation where labeled task data is presented sequentially by a teacher but unlabeled data from all tasks is freely available. We evaluate these experiments with no coreset for replay given that unlabeled examples for all classes are present each task. Across these experiments, we see that our proposed method (DM) establishes strong performance for SSCL. With respect to final classification accuracy AN and average normalized task accuracy Q, we see that DM has considerably higher performance. The local distillation baselines E2E and DR perform no better than Base, which makes sense because local distillation does not distill the global task. The results suggest that when the unlabeled data distribution contains all past task classes, methods which distill across the global task (GD and DM) achieve high performance despite having no coreset.
Class correlations in the labeled task data negatively affect performance of all distillation meth- ods. We explore a scenario where object correlations exist in the class distribution of the labeled data but not the unlabeled data (Table lb). This extends the previous scenario by considering learning tasks of similar object types. We evaluate these experiments with and without a coreset to make direct comparisons with both the previous experiment and the next experiment. Compared to the previous experiments, we see that the introduction of super-classes entails more difficult problem, as every method decreases performance; yet DM largely outperforms all other methods. For example, in Table [bb (no coreset), GD results in 37.4% Ω and DM results in .87 Ω indicating 54.5% increase over state of the art. Overall, these results pose an interesting question of why parent class tasks would negatively affect performance; plan to more fully explore this in future works.
The DistillMatch approach robust to several realistic unlabeled data correlations; the other methods are not. We evaluate our methods with several unlabeled data distributions (beyond uniform distribution) with the above ParentClass Tasks (Table 2a The unlabeled data correlations represent the realistic SSCL scenario, vastly different perspective compared prior experimental methodologies. We evaluate these experiments with a coreset of 400 images for replay and show that our method is state of art in each realistic scenario. Surprisingly, we found that the global
7

---

Under review as a conference paper at ICLR 2021
Table 2: Results (%) CIFAR- 100 with 20% Labeled Data for (a) ParentClass Task CIFAR-100 with 20% Labeled Data for various unlabeled data distributions (20 Tasks, with 400 image Coreset) and (b) Selected Ablation Studies (RandomClass Tasks with Uniform Unlabeled Data Distribution, 10 Tasks, no Coreset). Results are reported as an average of runs, with std provided in the supplementary material. For (b), Each row represents part of method which is removed as part of the study. UB refers to the upper bound, given by the offiin oracle.
(a)
UL Data Corr.
Random Sample
Positive
Negative
(b)
Ablation
An
Metric (1)
Av
Ω
Ax
Ω
Ax
Ω
Ept eq. 7
7.7
32.0
UB
56.7
100
56.7
100
56.7
100
mr(k) eq. (
30.2
69.6
Base
14.6
53.4
14.6
53.4
14.6
53.4
Ext eq. (4)
33.3
71.2
E2F
18.9
59.4
19.9
60.1
19.8
60.0
East - eq. T
35.2
74.1
DR
18.8
62.8
20.1
62.1
19.9
61.8
Full Method
37.5
76.9
GD
17.9
50.2
18.1
50.5
21.3
59.9
DM
19.7
63.3
20.7
64.8
22.4
65.1
Figure 52 (9) Coreset Size for Base, with GD and DM (no corset) plotted horizontally. We show base requires roughly 935 and 338 stored images to match the performance of DM and GD, respectfully. This is equivalent to 0.23 and 0.08 stored images per unlabeled image (calculated using the number of unlabeled images per task this scenario, which is 4,000).
(ven)
Base
200
Conrent Sizw (log araled)
distillation method actually performs worse than Base. This can be explained in that the global distillation loss expects the unlabeled data to contain examples which are either the distribution of the past tasks or out of distribution for the global task (i.e. it assumes the unlabeled data does not contain examples the current task), and actually hurts performance when this assumption not held. Our method, on the other hand, performs the same or better compared to the other methods (DR, E2E) which have no such an assumption. These results suggest that our reminding based solution is the only evaluated method which performs well in memory constrained setting regardless of the correlations between unlabeled and labeled data.
The DistillMatch approach saves memory. Here, perform study to quantify the memory budget savings from our method (Figure BD. We see that DM performs considerably higher than the leading baseline, GD, in the RandomClass Tasks with Uniform Unlabeled Data Distribution scenario (10 task). All other methods require a large coreset to match DM. We specifically find the necessary coresct size for Base required match our performance. We show base requires roughly 935 and 338 stored images to match the performance of DM and GD, respectfully. This is equivalent to 0.23 and 0.08 stored images per unlabeled image (calculated using the number of unlabeled images per task in this scenario, which is 4000).
Each component of DistillMatch contributes its performance gains. We perform an extensive ablation study of our method in Table 2h for the scenario with coreset (further ablations are found in Appendix-P AA. We show that while semi-supervised consistency loss 4 class balaneing B) and soft distillation loss (T) are all important to our method, the most important contribution is the hard distillation loss 77. In summary, we find that our design performs well across a range of conditions, namely with different amounts of coresets (including none at all), well as under different unlabeled distributions. This is a key aspect of our experimental setting, which better reflects the flexibility that is necessary from a continual learning algorithm.
CONCLUSIONS
We formalize the SSCL setting, which mirrors the underlying structure in the real world where a continual agent's learning task is product of its environment, and determine its effect on the continual learning problem. To address the unique challenges of the SSCL setting, we propose
8

---

Under review as a conference paper at ICLR 2021
novel learning approach that works within these constraints, DistillMatch, notably outperforming closest prior art. Our approach consists of pseudo-labeling and consistency regularization, distillation (OoD) detection for confidence selection. Our analysis
for continual learning, and approach performs well in memory constrained setting regardless
shows that our reminding-band of the correlations between unlabeled and labeled data, unlike existing approaches. challenge for future work is increasing the effectiveness of semantic OoD detection, and exploring better techniques for calibrating OoD detectors in continual (online) manner. We acknowledge other concurrent works push realistic continual learning settings as well (Mundt et al. 2020 contributions are orthogonal to our setting.
Ren et al. 2020). but the

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 11849-11866 1860, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems, pp. 11816- 11825, 2019b.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.09571, 2020.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 5050-5060, 2019.
Francisco M Castro, Manuel Marin-Jiménes Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 233-248, 2018
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint 19Xiv:1902.100
2019b.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. arXiv preprint /909 137/9, 2019.
Sayna Ebrahimi, Mohamed Elhoseiny. Trevor Darrell, and Marcus Rohrbach. Uncertainty- continual learning with bayesian neural networks. arXiv preprint 1906. 2019.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning, arXiv preprint 2rXiv:2003.09553, 2020.
Alexander Gepperth and Cem Karaoguz. Incremental learning with self-organizing maps. 2017 12th International Workshop on Self-Organizing Maps and Learning Vector Quantization, Clustering and Data Visualization (WSOM), pp. 1-8,22 2017.
Tyler Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. arXiv preprint arXiv: 10XX::/900.01520. 2019.
Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9769-9776. IEEE, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv: 05201, 2019.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Lifelong learning via
progressive distillation and retrospection. In Proceedings of the European Conference Computer Vision (ECCV), pp. 437-452, 2018.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv: /810. 12488, 201 88
10

---

Under review as a conference paper at ICLR 2021
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. arXiv preprint arXiv:2002.11297, 2020.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, pp. 1818 1818-1828, 2019.
Longlong Jing and Yingli Tian. supe-supe visual feature learning with deep neural networks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv: 10368, 2017.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. International Conference on Learning Representations (ICLR), 2018.
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting neural networks. AAAI Conference on Artificial Intelligence, 2018.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pp. 3581 13589, Cambridge, MA, USA, 2014. MIT Press.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwingg et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Tech Report, 2009.
Chia-Wer Kuo, Chih-Yao Ma, Jia-Bin Huang, and Zsolt Kira. Manifold graph with learned prototypes for semi-supervised image classification. arXiv preprint
2019.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Mong-Hyun Lee. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 312-321, 2019.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributiin samples and adversarial attacks. In Advances in Neural Information Processing Systems, pp. 7 7167-7177, 2018.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. arXiv preprint arXiv:2001.00639 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947) 2017.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv: 1706::/776.0260000 2017.
Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. Generative feature replay for class-incremental learning. arXiv preprint 091999000..01199 2020.
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. arXiv preprint arXiv: (70.000000033500 2017.

---

Under review as a conference paper at ICLR 2021
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning.
In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17. pp. 6479, USA, 2017. Curran Associates Inc. ISBN 978- 17181-1108-6096-4
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single-ineremmentll-assse scenarios. Neural Networks, 1 116:56-73, 2019.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Martin Mundt, Yong Won Hong. Iuliaa Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge active and open world learning. arXiv preprint arXiv:2009.01797, 2020.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Advances in Neural Information Processing Systems, pp. 1235-32446 2018.
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54 71, 2019. ISSN 0893-6080. doi: https://doi.org/ www.1//...emmee2201....o.
Sylvestre-Alvise Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR' 17, pp. 5533-5542, 2017.
Mengye Ren, Michael L luzzolino, Michael C Mozer, and Richard S Zemel. Wandering within world: Online contextualized few-shot learning, arXiv preprint arXiv:2007.04546, 2020.
Anthony Robins. Catastrophic forgetting, rehearsal and pscudorchearsal. Connection Science, 7(2): 146. 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 348-358, 2019.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.04671, 2016.
Gobinda Saha, Isha Garg, Aayush Ankit, and Kaushik Roy. Structured compression and sharing of representational space for continual learning. arXiv preprint arXiv:2001.08650, 2020.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In I. Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2990-2999. Curran Associates, Inc., 2017.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying sun-viuper learning with consistency and confidence. arXiv preprint arXiv:200J.07683 2020.
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv: 1.06390, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: targets improve semi-supervised deep learning results. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett (cds.), Advances in Neural Information Processing Systems 30, pp. 1195-1204. Curran Associates, Inc., 2017.
consistency
Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In International Conference Learning Representations, 2019.
12

---

Under review as a conference paper at ICLR 2021
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734. 2019.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv: 00095, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
Xinqi Zhu and Michael Bain. B-cnn: branch convolutional neural network for hierarchical classifica- tion, arXiv preprint 1709v 09890, 2017.
13


Reviews:
[{"review_id": "z-PeszUupN7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates a semi-supervised continual learning (SSCL) setting and proposes a new method called DistillMatch for this setting. The major contributions are: (1) The authors carefully design a realistic SSCL setting where object-object correlations between labeled and unlabeled sets are maintained through a label super-class structure. And then, they develop the DistillMatch method combining knowledge distillation, pseudo-labels, out of distribution detection, and consistency regularization. (2) They show that  DistillMatch outperforms other existing methods on CIFAR-100 dataset, and ablation study results are shown also. \n\nHowever, there are some downsides that should be considered before its publication. (1) In abstract the authors claim that they can significantly reduce the memory budget (of labeled training data) by leveraging unlabeled data (perhaps with large volume). This motivation seems to be contradictive. (2) From a methodological viewpoint, the proposed DistillMatch method is just a combination of existing methods (listed as in above). So where is the novelty of this \"new\" method? (3) In experiments, the chosen baseline algorithm is very weak. There are some strong baseline methods such as GEM, A-GEM, and ER. So I wonder to know the real improvements over state-of-the-art methods for continual learning. (4) The label super-class structure existed in CIFAR-100 has been used in their experiments. But this is not very common for other more realistic datasets such as miniImageNet. If there is no super-class structure, we don't know how to apply the proposed DistillMatch method. \n\nIn summary, I think this semi-supervised continual learning setting is interesting, but the proposed DistillMatch method can not persuade me that this method is a novel significant contribution to this problem. So at present time I believe there is much room for the authors to improve their method before publication. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "L1bB59KXOOn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary:\nThis paper proposes class-incremental learning with unlabeled data correlated to labeled data, and a method to tackle it. The task can be considered as a variant of [Lee et al.], which has no assumption on the unlabeled dataset, while this paper assumes the correlation between labeled and unlabeled dataset explicitly. The proposed method is inspired by state-of-the-art class-incremental learning, semi-supervised learning, and out-of-distribution (OoD) detection methods: local distillation [Li and Hoiem], OoD detection [Hsu et al.], consistency regularization and pseudo labeling (or hard distillation) [Sohn et al.], and loss balancing based on class statistics [Lee et al.]. Experimental results support that the proposed method outperforms prior works in the proposed task.\n\n- Reasons for score:\n1. Extending continual learning to the semi-supervised setting is natural, given that the extension to self-taught learning has already been considered in [Lee et al.]. However, I cannot agree that semi-supervised learning is more realistic than self-taught learning, which is emphasized throughout the paper 18 times. In an early work of [Raina et al.], self-taught learning is proposed to make the scenario of learning with unlabeled data \"widely applicable to many practical learning problems.\" [Oliver et al.] also argued that \"(unlabeled data from out-of-distribution) violates the strict definition of semi-supervised learning, but it nevertheless represents a common use-case for semi-supervised learning (for example, augmenting a face recognition dataset with unlabeled images of people not in the labeled set).\" I am not saying that semi-supervised learning is unrealistic, but the argument in this paper sounds overclaimed. I believe both semi-supervised and self-taught learning are realistic in some cases. I also recommend to provide real world scenarios that the proposed task (correlation between labeled and unlabeled data exists and no memory for coreset is available) is useful in practice.\n2. The proposed method is not novel, which is essentially the combination of state-of-the-art methods in relevant tasks. But I do not discount this much, because this work would be valuable as the proposed task is interesting but not investigated before. However, the name of task might need to be changed, because a similar name, \"semi-supervised incremental learning\" is already taken by a kind of semi-supervised learning, which incrementally incorporates unlabeled data to training.\n3. Though the improvement over prior class-incremental learning methods is impressive, the overall performance is still too low. In fact, the scale of the experimental setting is too small, so I doubt it is scalable. All experiments are bounded on CIFAR-100, and even only 20% of training data are used as labeled one. Frankly, in this small-scale setting (in both number of data and image resolution), keeping all data is just fine, as the coreset size is negligible compared to the model size. I recommend to experiment in large-scale settings, e.g., on ImageNet. Also, I recommend to compare the oracle setting as well, which keeps all previous training data.\n4. In addition to small-scale experimental setting, the architecture is larger than the prior work [Lee et al.]: WRN-28-2 vs. WRN-16-2. In the worst case scenario, it is possible that the best performance of the proposed method is simply from the complexity of their learning objective, i.e., all methods overfit to training data, but the proposed method did not have enough updates to overfit to them.\n5. In Figure 3, why do GD and DM not have a coreset? I think there is no reason to give an unfair constraint to them. I recommend to draw curves with respect to increasing number of coreset for those methods as well.\n6. Could you provide results on the self-taught learning setting like [Lee et al.]? It would also be interesting to see the performance of the proposed method in the setting.\n7. Hyperparameter sweep results provided in Table 4 are either minimum or maximum of the range, so you could improve the performance by enlarging the range.\n\n- Minor Comments:\n8. Subscripts of theta often are dropped. Is theta equal to $\\theta_{n,1:n}$?\n9. \"the parameters of no more than three models\" -> I believe it is four, because you need to temporarily store gradients during training.\n10. $\\hat{q}$ is not a probability vector, which makes eq. (2) mathematically do not make sense.\n11. Citation format issue: you can use \\citet for noun and \\citep for adverb.\n12. typo on page 5: statoe -> state\n13. Table 4: what is TPR here? threshold for consistency regularization?\n\n[Raina et al.] Self-taught Learning: Transfer Learning from Unlabeled Data. In ICML, 2007.\n\n[Li and Hoiem] Learning without forgetting. In TPAMI, 2017.\n\n[Oliver et al.] Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. In NeurIPS, 2018.\n\n[Lee et al.] Overcoming catastrophic forgetting with unlabeled data in the wild. In ICCV, 2019.\n\n[Hsu et al.] Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In CVPR, 2020.\n\n[Sohn et al.] Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.\n\n**After rebuttal**\n\nI'd like to thank authors for their efforts to address my concerns. They have addressed most of them, so I increased my score from 5 to 6.\n\nHowever, there are two concerns that couldn't be resolved during the rebuttal period:\n\n(1) I am still not sure if the proposed task is practical. At glance it looks realistic, but I couldn't find a detailed scenario that can only be solved by the proposed task. Any real world scenario I can think of is closer to [Lee et al.], which is a prior work of this paper. Authors provided an exploring robot example in the thread of responses, but I think [Lee et al.] fits better for the provided one. I recommend authors to find a concrete use-case in real-world applications, which can only be solved by the proposed setting (or at least [Lee et al.] is not applicable; in the revised intro, you may emphasize that there are some real-world problems that [Lee et al.] is not applicable but yours is). R1 and R4 seem to have a similar concern.\n\n(2) the scale of experiment is too small. As CIFAR-10/100 have a limited number of data for your purpose,  you can borrow some data from tinyimages (FYI, CIFAR-10/100 are a subset of 80M tinyimages) or focus on ImageNet.\n\nI am okay with the lack of novelty on the proposed method. For a newly proposed task, I think proposing a simple and effective baseline is good enough. However, because of the two concerns above, I cannot strongly agree with its acceptance.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HEP6e7aiTZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper comes up with a novel scenario where the unlabled data are available as well as labeled data in the continual learning scenario.\n\n### Overall\n- Based on my understanding, the major contribution is the proposal of a task scenario, aka, experimental setting. The novelty of DistillMatch is an incremental modification of previous work. \n- The task setting sidesteps the learning with non-stationarity problem than solving it.\n- Further, this setting potentially makes the task easier for the proposed method. To verify whether this is true, more information are needed.\n- The presentation of the paper needs polishing, I listed a few points below.\n\n### Pros\n- The novel scenario of semisupervised continual learning is proposed. The argument is that in several realistic scenarios old data are often re-observed without label (The funiture labeling example). Therefore instead of storing a coreset, one may make use of the unlabeled data for pseudo-rehearsal/distillation. It is reasonable to make use of it when this assumption is true.\n- With the setting the author proposed, the DistillMatch method is able to perform better than previous methods. \n\n### Cons\n1. The novelty mostly comes from the task scenario, the DistillMatch method is incremental.\n2. Although SSCL is a new scenario, and the author argues it is more realistic. IMO taking this assumption sidesteps the problem of continual learning rather than solving it. The central problem of continual learning IMO is to learn under non-stationary distribution, the assumption made in this submission makes the distribution more stationary.\n3. It is true that this assumption should be utilized when available. However, the only dataset used is manually constructed from CIFAR100, contradicting the initial motivation to move towards a more realistic scenario.\n4. There's a lack of information on how the compared methods are adapted to the new scenario. \nI searched the supplementary but failed to find a detailed documentation. With the given information, it is hard to tell whether the comparison is fair. My concerns are following,\n**increasing from 3 -> 4 as this point is resolved in the rebuttal**\n  - In the RandomClasses setting, it is stated that no coreset is used, if the compared methods depends on coreset to replay, it would be unfair. If that's the case, the only conclusion we can draw is that replay is better than no replay, which seems trivial to me.\n  - GD depends on internet crawled data, is it replaced with the unlabeled data since it is available in the experiment setting? If not, then I think it is just the setting that favors DistillMatch.\n  - With the above said, I suggest the author to list clearly the objectives, replay buffer sizes or even pseudo code for each of the compared method and their own method in a table, which will help the reader identify what major component in the proposed method is making the contribution.\n\nRegarding the quality and clarity,\nI found myself confused and making guesses sometimes while reading it.\n\nTo list a few:\n- introduction paragraph 2, ... to determine which unlabeled data is relevant to the incremental task ..., I guess the incremental task means learning the newly observed data, but then for rehearsal we'll pick the unlabeled data which is from the distribution of past tasks.\n- section 1, ... save up to 0.23 stored images per processed image over naive rehearsal (compared to Lee) ..., here seems Lee et al is the naive rehearsal. But then \"which only saved 0.08\" confuses me, seems to be saying Lee saves 0.08 compared to naive rehearsal.\n- section 3, ... where data distributions reflect object class correlations between, and among, the labeled and unlabeled data distributions ... not enough information to infer what \"reflect\" and \"object class correlation\" means here.\n- section 4, ... Let S_{n-1} denote the score of our OoD detector for valid classes of our pseudo-label model ... what is the \"valid classes\" needs to be clarified. As I understand it, S_{n-1} measures how likely the unlabeled data is in the distribution of past tasks.\n- Super class / Parent class are not defined clear enough.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3zdGcKtO1QT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel semi-supervised continual learning (SSCL) setting, where labeled data is scarce and unlabeled data is plentiful. The proposed framework is built on pseudo-labeling, consistency regularization, Out-of-Distribution (OoD) detection,\nand knowledge distillation in order to reduce the catastrophic forgetting in the proposed setting.\n\nThe paper is in general clear and well-written. The contributions are clearly highlighted and the proposed approach is conveniently compared with other state of the art methods, demonstrating its superiority.\n\nPositive aspects: \n-  the definition of a realistic, semi-supervised setting for continual learning\n- a novel approach for continual learning in order to cope with 'catastrophic forgetting'\n- the proposed approach is memory efficient, since it does not need exemplars to replay past tasks\n\nNegative aspects:\n- the OoD implemented in this paper rejects the unknown samples. In other words, all unknown samples are considered a single class. It would have been a plus to distinguish between several unknown classes and somehow introduce them in the framework\n- the lack of recabilibration step after a number of tasks (in the case of pseudo-labeled samples), could lead to an undesired error propagation which is not quantified in the paper \n\nHowever, I have some questions:\n1. What is the relationship between the 'fi' and 'theta' models (section 4)? Are they completely separate or there is a relationship between them?\nFor instance, when 'theta' is extended with a new task, is 'fi' extended accordingly? Or is 'fi' trained off-line from the beginning (with all tasks)?\n2. There are some different source of errors: distilation, pseudo-labels... Do you perform any kind of system re-calibration? After how many tasks? I mean, do you make a study of error propagation of pseudo-labeled data? Or at some point do you have a human-in-the-loop to correct mis-classification? What is the mis-classification error of pseudo-labeled samples?\n3. Do you assume that labeled and unlabeled data come from different distributions or you have a single distribution which is \ndivided in labeled and unlabeled data at the beginning of the process?\n4. Does your scenario foresee that when learning a new task T, all the previous tasks are represented (1..T-1) in the unlabebeld data or only a subpart? (i.e. kind of selective replay)\n5. When the number of tasks increases, the number of unlabeled data per task remains constant or is scaled accordingly (i.e. reduced) ?\n6. Would be interesting to test your approach in a real-world scenario, i.e. robot navigation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a semi-supervised setting to reduce memory budget in replay-based continual learning.
It uses unlabeled data in the environment for replaying which requires no storage, and generates pseudo-labels where unlabeled data is connected to labeled one.
The method was validated on the proposed tasks.

Pros:
- The semi-supervised continual learning setting is novel and interesting.
- The proposed approach is memory efficient, since it does not need exemplars to replay past tasks.

Cons:
- The scale of experiment is small. It lacks evaluation in real world environment.
- The novelty is limited, because it is a combination of existing technologies: pseudo-labeling, consistency regularization, Out-of-Distribution (OoD) detection, and knowledge distillation.
- The comparison might not be fair due to different settings.

The authors addressed the fairness and scalability with additional experiments
and leave some suggestions of reviewers for future work.
R3 had a concern on the error propagation of pseudo-labels which I also share. The authors agreed that this is a challenge for all CL methods.

In summary, the reviews are mixed. All reviewers agree that the semi-supervised continual learning setting is novel and interesting, and some have concerns on scalability and novelty of the method which I also share. So at present time I believe there is much room for the authors to improve their method and experiments before publication.

Decision: Reject