A distinct unsupervised reference model from the environment helps continual learning
Authors: Seyyed AmirHossein Ameli Kalkhoran
Mohammadamin Banayeeanzade
Mahdi Samiei
Mahdieh Soleymani Baghshah
Venue: Submitted to ICLR 2023 (2023)
Keywords: continual Learning
open-set semi-supervised continual learning
knowledge distillation

Abstract: The existing continual learning methods are mainly focused on fully-supervised scenarios and are still not able to take advantage of unlabeled data available in the environment. Some recent works tried to investigate semi-supervised continual learning (SSCL) settings in which the unlabeled data are available, but it is only from the same distribution as the labeled data. This assumption is still not general enough for real-world applications and restricts the utilization of unsupervised data. In this work, we introduce Open-Set Semi-Supervised Continual Learning (OSSCL), a more realistic semi-supervised continual learning setting in which out-of-distribution (OoD) unlabeled samples in the environment are assumed to coexist with the in-distribution ones. Under this configuration, we present a model with two distinct parts: (i) the reference network captures general-purpose and task-agnostic knowledge in the environment by using a broad spectrum of unlabeled samples, (ii) the learner network is designed to learn task-specific representations by exploiting supervised samples. The reference model both provides a pivotal representation space and also segregates unlabeled data to exploit them more efficiently. By performing a diverse range of experiments, we show the superior performance of our model compared with other competitors and prove the effectiveness of each component of the proposed model.

Under review as a conference paper at ICLR 2023
A DISTINCT UNSUPERVISED REFERENCE MODEL FROM THE ENVIRONMENT HELPS CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The existing continual learning methods are mainly focused on fully-supervised scenarios and are still not able to take advantage of unlabeled data available in the environment. Some recent works tried to investigate semi-supervised continual learning (SSCL) settings in which the unlabeled data are available, but it is only from the same distribution as the labeled data. This assumption is still not general enough for real-world applications and restricts the utilization of unsupervised data. In this work, we introduce Open-Set Semi-Supervised Continual Learning (OSSCL), more realistic semi-supervised continual learning setting in which out- of-distribution (OoD) unlabeled samples in the environment are assumed to cocxist with the in-distribution ones. Under this configuration, we present model with distinct parts: (i) the reference network captures general-purposs and task-agnostic knowledge in the environment by using broad spectrum of unlabeled samples, (ii) the learner network is designed to learn task-specific representations by exploiting supervised samples. The reference model both provides a pivotal representation space and also segregates unlabeled data to exploit them more efficiently. By performing a diverse range of experiments, we show the superior performance of our model compared with other competitors and prove the effectiveness of each component of the proposed model.
INTRODUCTION
In a real-world continual learning (CL) problem, the agent has to learn from a non-i.idd stream of samples with serious restrictions on storing data. In this case, the agent must be prone to catastrophic
forgetting during training (French 1999.. The existing CL methods are mainly focused on supervised
scenarios and can be categorized into three main approaches (Parisi et al., 2019): (i) Replay-base based methods reuse samples from previous tasks either by keeping raw samples in limited memory
buffer (Rebuffi et al. 2017 Lopez-Paz & Ranzato 2017 Aljundi et al., 2019) or by generating pseudo-samples from previous classes (Shin et al.) [0]]]0011 Wu et al. 2018)) van de Ven et al. 2020). (ii)
Regularization-basse methods aim to maintain the stability of the network across tasks by penalizing
deviation from the previously learned representations or parameters (Nguyen et al., 2018) Cha et al.
2021 Rebuffi et al. 2017: Li & Hoiem 2016). (iii) Methods based parameter isolation dedicate
distinct parameters to each task by introducing new task-specific weights or masks (Rusu et al. 2016
Yoon et
2018 Wortsman et al. 2020).
Humans, as intelligent agents, are constantly in contact with tons of unsupervised data being endlessly streamed in the environment that can be used facilitate concept learning in the brain (Zhuang et al. 202T, Bi & Poo 1998, Hinton & Sejnowskil 1999). With this in mind, an important but less explored issue in many practical CL applications how to effectively utilize a vast stream of unlabeled data along with limited labeled samples. Recently, efforts have been made in this direction leading to the investigation of three different configurations; WWang et al. (002)) introduced a very restricted scenario for semi-supervised continual learning in which the unsupervised data are only from the classes which are being learned at the current time step. On the other hand, Lee et al. (2019) introduced configuration that is "more similar to self-taught learning rather than semi-supervised learning". In fact, they introduced a setting in which the model is exposed to plenty of labeled samples which is necessary assumption for their model to achieve good performance; in addition, their model has access to a large corpse of unsupervised data in an environment that typically does not include samples related to the current CL problem. By adopting this idea, Smith et al. (202T) proposed

---

Under review as a conference paper at ICLR 2023
more realistic setting by assuming limitation the number of supervised samples available for the training. In addition to that, they assumed the existence of a shared hidden hierarchy between the supervised and unsupervised samples, which is not necessarily true for practical applications.
In this work, we will first propose a general scenario to unity the mentioned configurations into more realistic setting called Open-Set Bemi-Supervised Continual Learning (OSSCL). In this scenario, the agent can observe unsupervised data from two sources: (i) Related unsupervised data, which are sampled from the same distribution as the supervised dataset, and (ii) Unrelated unsupervised data which have a different distribution from the classes of the current CL problem. The in-distribution unsupervised samples can be from the classes that are being solved, have been solved at previous time steps, or are going to be solved in the future.
Previous CL works in which unlabeled data was available alongside labeled data, mainly utilized unlabeled data by creating pseudo-labels for them using a model which is trained by labeled samples (Lee et al. 2019] Smith et al. [02T] WWan et al. 202T). Those unlabeled data with their pseudo-labed were used directly in the training procedure. However, due to the fact that labeled data are scarce in realistic scenarios, the pseudo-labeling process will be inaccurate and creates highly noisy labels. Therefore, we present a novel method to learn in the OSSCL setting which alleviates the mentioned problem and utilizes unlabeled data effectively.
Our proposed model, which is consisted of an Unsupervised Reference network and Supervised Learner network (URSL), can effectively absorb information by leveraging contrastive learning techniques combined with knowledge distillation methods in the representation space. While the reference network is mainly responsible for learning general knowledge from unlabeled data, the learner network is expected capture task-specific information from few supervised samples using a contrastive loss function. In addition, the learner retains a close connection to the reference network to utilize the essential related information provided by unsupervised samples. At the same time, the representation space learned in the reference network can be utilized to provide an -uitt--diitriuutionn detector that segregates unlabeled data to employ the filtered ones more properly in the training procedure of the learner model. In short, our main contributions are as follows:
We propose OSSCL as realistic semi-supervised continual learning scenario that an intelligent agent encounters in practical applications (Section 2
We propose novel stal-structured model that is suitable for learning in the mentioned scenario and can effectively exploit unlabeled samples (Section 3).
We show the superiority of our method in several benchmarks and different combinations unlabeled samples. our model achieves state-of-the-ar accuracy with a notable gap compared to the baselines and previous methods (Section
2 PRELIMINARIES
In this work, we consider the training dataset to consist of two parts; the supervised dataset D Prup is a sequence of T tasks (Ti, T2.. Tr). At time step i, the model only has access to T, {(xx, y: No where 24 i.ddd P(X|ps) denotes training sample and y; represents its corresponding label. We consider K separate classes at each task and follow the common class-incremental setting as is
shown to be the most challenging scenario for evaluation. Given training loss l and the network parameters 0, the training objective at time step t is defined as 0" arg ming N. MiN,, ((x;, Vi, O).
On the other hand, the unsupervised dataset D Dunsup is a sequence of T sets (U,, L4,, contain- ing only unlabeled data points. We assume that 24 represents the unsupervised data available in the environment at time step f, which is accessible by the model along with T,. Based on the OSSCL setting which is general framework, we assume that the unsupervised dataset is composed of two parts: (i) The related part, also called the in-distribution set, is consisted of unsupervised samples generated from the same distribution as D.up. In order to maintain generality, we assume that this set consists not only of unsupervised samples related to the current supervised task but also of the other tasks of the CL problem that have either been observed in previous time steps will be observed in the future. (ii) The unrelated data points, also called the oft-of--istributte samples, are a set of unsupervised data sampled from the distribution Q, which is not necessarily the distribution from
2

---

Under review as a conference paper at ICLR 2023
Ursupeevieed famplles Unrdined
Den Detection
Dut of distibbttion samples
dist lbus on samples (LA)
fscudal samples
Envienment
Lruneup
Protonypss
Ul
UR,
UR2
Exp
u..7
SL
S12
She
Supervicor
Coup
Th Superteed Teak
A schematic of the method and configuration. The unsupervised reference (UR)). supervised learner (ST,), labeled data (Ti), and related and unrelated unlabeled data (4,)) at time step t are shown on the left while the OoD segregation module is shown on the right of the figure.
Figure 1:
which the supervised samples have been generated. the next section, we will propose novel method to perform in this configuration, and in Section 4 a variety of experiments are provided to show the effectiveness of our model.
METHOD
Learning continually from Dsup has been widely explored by the community. Meanwhile, unlike deep models, humans are less hungry for supervised data. Although they observe large volume of data during their lifetime, only small and insignificant portion of this data is labeled. It is believed that the considerable human ability to learn with few instances is due to the rich representations learned from the large volumes of unsupervised observations (Zhuang et al. 2021 Bi & Poo 1998, Hinton & Sejnowski 1999). Here, we aim to explore the benefits of using D Punsup and its impact on empowering the continual Tearner. Specifically, we will show how Pansup will promote representation learning in addition to providing positive forward/backward transfer in the continual learning process.
We propose our URSL model, which is consisted two parts: 1) The general task-agnostic reference network, which is responsible for absorbing information from unsupervised data in the environment, and 2) the learner network, which is designed to capture knowledge from a few supervised samples while it is also guided by the reference network. The notation UR, and $ SL used to respectively demonstrate the reference and learner network instances at time step / (refer to I Algorithm [ for overview).
We employ a contrastive representation learning approach for training both the reference and the learner networks. This approach has been proven be a proper solution for supervised CL problems. Indeed, some previous works in CL claimed that classifier heads placed on top of the representation network are the serious sources of catastrophic forgetting (Ramasesh et al., 202T Banayeeanzade
et al. 2021 Cha et al. 2021) ), therefore, Co (Cha et al. 2021) presented a supervised contrastive loss avoid this problem. We utilize contrastive representation learning as unified approach for training both the reference and the learner networks, which allows information to flow between these networks easily. Combined with knowledge distillation techniques applied in the representation space, this approach provides a convenient tool to exploit the most out of unsupervised samples.
Our model is also equipped with an exemplar memory M to randomly store a portion of supervised samples from previous tasks (Lopez-Paz & Ranzato 2017 Rebuffi et al. 2017). The stored samples will contribute to the training the learner network. After the final time step, these samples are also used to train classifier head top of the representation space of the learner network. It is noteworthy that our model does not store unlabeled data in its own memory since this data is always found in abundance in the environment, and this makes model needless of large memory.
3

---

Under review as a conference paper at ICLR 2023
3.1 REFERENCE NETWORK
The unsupervised reference network UR, X' - IR is general-purpose feature extractor responsi- ble for encoding all kinds of unsupervised information available the environment. The network is composed of an encoder f. and projector g. responsible for embedding input x in the representation on the unit d-dimensional Euclidean sphere and 0, represents the model parameters at time step f. Considering a batch B C Us with size N, the SimCLR (Chen ctt al. 2020a) loss function used for training the network can be written as:
space by= (ʃ o g)o, o, (x) where
big = log exp($x.8//T)
Lunsup (0:;T) = 2N Σ 1,28 + 122
1 ke @xp(E,&+/r)
where z2 and 2 are the representations of two different augmentations of the same image E B and the temperature hyperparameter.
3.2 SEGREGATING UNSUPERVISED SAMPLES
In this section, we show how to segregate unlabeled samples by employing the reference network and supervised samples. Although unsupervised samples can play an important role in both learning the representation space and controlling changes in this space through time, naive approaches to incorporating these samples into the training of the learner network can lead to inferior performance due to the existence of unrelated samples among unlabeled ones. Therefore, we will first explain the OoD detection method, which is designed to segregate unlabeled data and incorporate them more properly in the continual learning process the learner network. To efficiently segregate unsupervised data, we employ prototypical-base OoD detection method (Park et al. (02T) in the representation space of the reference network using samples in Te U U.M. It is noteworthy that the representation space of the reference network chosen for OoD detection since it provides better sample discrimination than any other representation space obtained by training over small number of labeled samples. Additionally, this approach eliminates the need to train another network specialized in OoD detection in contrast to the previous works (Chen et al. 2020b Huang et al. 2021 Saito et al. 2021).
At time step t, our OoD method creates P = (Pi. P......... P Pk } set of K x t prototypes representing the centroids of observed classes so far, which extracted using the labeled data available in Te U UM:
Ptc = v0
Σ (BT.))
En
(E::
where A is set of augmentations meant form different views of a real image, and s is the operator that projects vectors into the unit d-dimensional sphere. We also define the score operator S (pt 2) max c c(P),2) 2) where denotes the cosine similarity measure. This operator takes prototypes in addition to sample in the representation space and calculates the score of its most probable assignment. With this mind, we consider 5 as the scores of the labeled data obtained by passing TUM through the (Pt,)) operator, i.e. S (S(P), (fog)e, (x))]= E T.MM.. By considering as hyperparameter, we define a threshold Tid mean (5)) + Taavar (S)) on the scores of unlabeled data to specify in-distribution samples Uf {æ|x U4, S(Pt,()))))
Tia). Furthermore, we assign pseudo-labels to the unsupervised samples on which we have superior confidence by defining higher threshold Tpl mean (8)) ) Vpp/ar (S)). with the hyperparameter 7pl. and prepare pseudo-labeled samples as T, {<x,, 0) E S(Pt > Tpl: 9 = arg max; x)) other words, an unsupervised sample with similarity value higher than Tpl class prototype is pseudo- labeled to that class. However, to reduce pscudo-labeling noise, we do not utilize pseudo-labels directly during the training procedure. Those pscudo-labels are used to identify whether this unlabeled data is from past classes not. Samples of T mainly used compensate for the small number of supervised samples in the memory, as further explained in the next section. We provide a detailed investigation of the performance of the Ool) module in Appendix C
4

---

Under review as a conference paper at ICLR 2023
3.3 LEARNER NETWORK
Similar to the reference network, the learner network SL, : X' + Rd is a feature extractor with the form (ʃ p. (x) where şir denotes the model parameters time step The training of the learner network is done using three mechanisms:
Supervised Training: Following Co²L (Cha et al. 2021). we will use an asymmetric supervised version of the contrastive loss function to train the learner network. By considering a supervised batch B {z24, wk) ) which is sampled from Tr U M U Fr and applying augmentation policy to form two different views of real samples, we can write the supervised contrastive loss as follow:
Cxup(Prit) Σ
[ne Kil
Σ log jEC:
exp(
1 Eeg exp(&..8e/T)
where Or is the new classes the current time step t, and S are the other samples of the current batch with the same label y/-
The existence of Ti is crucial for learning proper representation since only a small amount of labeled data available during continual learning. In fact, Co intends to prevent overfitting to the small number of past task samples stored in the memory by proposing the asymmetric supervised contrastive loss that utilizes samples from the memory only as negative samples (Cha et al. (02T). However, when the labeled data are limited, even employing the past samples in M, as negative samples, still may cause overfitting. Therefore, enrich M by T. diversify the samples from previous classes.
Knowledge Transfer Through Time: The loss function in Eq. allows the model discriminate between new and previous classes. However, it is not sufficient to maintain the discrimination power of the learner network among previous tasks. Therefore to avoid catastrophic forgetting, at each time step f, we use an instance-wise relation distillation (IRD) loss to transfer knowledge from the previous time step to the current model (Cha et al., 2021). This self-distillation technique, which is also compatible with the contrastive representation learning approach, retains the old knowledge by maintaining the samples' similarity in the representation space of learner network. To this end,
first, we sample a batch B from T U J T, augment each sample x; twice to create - T25. and then calculate the instance- wise similarity vector as:
p (ā: r) = [P ...0...... Pi.i Pi Pi2N]] where Pij pest (xp(E
exp(a,
By computing probabilities for both SL, and SL,-1, we can write time distillation loss as:
22 p(ā log log 2.ottaserrrrrrr.pptt........
CTD(pU
(5)
where 7' and 7" represent the distillation-specifie temperatures for the previous model and the current
model, respectively.
The reference network encounters numerous unsupervised samples throughout its training and is expected to learn rich representation space using the objective introduced in Section 3.1 This representation used as guidance for the learner network, and the knowledge can be transferred to the learerr network using an IRD loss similar to the Eq. 5
Knowledge Transfer from Reference:
2N Σ - O', log (e, PP,, T").
(6)
This distillation is applied to the learner network based on the samples in To U M U It is noteworthy that this distillation, rather than using all of the unsupervised samples in U4 only uses the unsupervised samples U4 which seems to be related the training of the learner network.
5

---

Under review as a conference paper at ICLR 2023
3.4 THE URSL ALGORITHM
In summary, the model receives two sets of samples at each time step: T. and U4 The reference network is trained Uz using the self-supervised loss function introduced in Eq. T Then, an OoD detection and pseudo-labeling technique introduced in Section 3.2 are used to segregate unsupervised samples in U4. Finally, the learner network trained based on the weighted aggregation of three loss functions introduced in Section 3.3 by defining 7 and À hyperparameters:
L.(())) = (1P63)
+ (,,
Algorithm URSL:
Unsupervised Reference and Supervised Learner
Require: A supervised dataset Ssup {Ta)== and an unsupervised dataset D Punsup {1 1: initialize U Ro and SLo respectively with random parameters Og and
2: for f. =. do
3: Initialize 02 t- - and (t 41-1
4:
Update 0. based on U, minimize Lumsup(O T) (Eq.
5:
Extract Pt using T, UM (Eq.
6:
Compute S from Tr U.M (Section. 3.2
7:
Compute Tia mean (Sf) Taavar (Si] and Tpl mean (S) + Var (S))
8: Prepare F and Lhe based on Tid. Tpls and scores of Uy (Section 3.2) while not done do
9:
Sample a batch from T, TUMUU,
11:
Compute L. < Lamp based on B (Eq. 3
i then
13:
based on B (Eq. 5 Update Lg t L. XCxD(pps pc, T',"") based on a batch from Te U MuM (Eq.
Update L, L. wwpppClliliiinino
15:
Update sP: Pt - VV
Update M such that the number of samples for each class is the same.
17:
Train the classifier head using Tr UM
4 EXPERIMENTS
Benchmark Scenario: To demonstrate the effectiveness of our method, we have performed several experiments in this section. We use two datasets for each experiment: the main and the peripheral. A small portion of the main dataset, which is determined by P. is selected as supervised data, the rest considered as related unsupervised data, and all samples of the peripheral dataset are considered (probably) unrelated unlabeled data. At each time step, 9000 examples from each unsupervised dataset are randomly sampled, shuffled together, and fed into the model as unsupervised data. In Appendix we provide the results of experiments in which the number of datasets inside 24, is greater than two datasets, and the environment is even more realistic.
The hyperparameters of our model are not dependent the experiment configuration, and general and consistent solution for all conditions is provided. We conducted wide range of experiments to demonstrate the model's robustness in various scenarios. In our experiments.
used the CIFARI 10, CIFAR 10 (Krizhevsky et al. 2009). and Tiny-ImageNet (Le & Yang 2013) datasets as the main or peripheral datasets, which are commonly used datasets in the open semi-supervised learning literature (Chen et al. 2020b Huang et al., 2021 Yu et al., 2020); moreover, the settings of our experiments are known as the "cross dataset" setting in the open-set superviue literature (Chen) al. 2020b). We have utilized ResNet- 18 architecture as the backbone of both networks with two-layer MLP on its head as the projector. The input images for the model are 32 x 32 pixels in the notation _M] to show the size of the supervised memory introduced in Section 3 Further experimental setups and details are provided in Appendix
size. Additionally,
Baselines: L (Cha al.) 2021) be seen as simplified version of URSL in which there is reference network and no means for using unsupervised samples. Therefore, we propose modified version of Co Lo in which the model is trained jointly by employing both supervised and

---

Under review as a conference paper at ICLR 2023
Table 1: Accuracy of different models on the CIFARIO dataset.
Unsupervised
Method
Setting
Dataset
Co²L
Co
Co
GD
DM
URSL
P 0.01
CIFARI00
26.5
36.02a.2
46.62o.3
24.0
30.1 10.7
58.2
I 50
Imay-ema 26.5+1.6
33.0+1.s
42.7
24.3+1.4
29.7+a.1
51.0
P 0.1
CIFARI00
58.3
52.7+1.0
62.4
48.1+1.a
59.6
72.8
MM| 200
Imay-lmagenet 58.3
42.2.2.1
61.3
47.3+14
42.1
72.8
CoPL.
GEM
iCaRI.
P =
None
69.5
29.2
49.9
IM| 200
Table 2: Accuracy of different models on the CIFAR100 dataset.
Unsupervised
Method
Setting
Dataset
Co'L
Co²L.j
Co²L.p
GD
DM
URSL
P = 0.05
CIFARIO
15.9
20.4
21.0
11.8
24.210
30.4
[M| = 500
Tiny-lmagenet
15.9
16.9
21.5+0.2
11.5+a.
28. 28.1+1.2
30.5
P = 0.1
CIFAR10
25.1
26.910.1
28.3
16.7
33.611.0
37.5
M 1000
Tiny-Imagenet
25.1+0.1
28.4+1.4
28.9
15.9
38.5
37.2
Co²L
GEM
iCaRL
P
None
35.1
22.4
34.4 4
M 1000
an unsupervised contrastive loss on the supervised and unsupervised data, respectively. In another baseline, Co*L-p, we only pre-train the model with unsupervised data available in the first time step and ignore the unsupervised data in the subsequent steps avoid possible conflict with the supervised loss during continual learning. There are also two other baselines in the prior works that seem consistent with the OSSCL setting due the presence of OoD detection module. GD (Lee et al. 2019) trained an Ool) module to recognize unlabeled data from previous classes among the entire unlabeled dataset. This in-distribution data was only used to combat catastrophic forgetting. DM (Smith et al. 2021) mainly changed GD setting through defining some policies over unlabeled data by using superclasses of the CIFAR 100 and using the FixMatch method (Sohn al. 2020). On the other side, we also report results of fully supervised continual learning for two popular continual learning models, GEM and iCaRL, and also the sfate-of-the L. These methods have access to all samples of the related dataset labeled ones during continual learning but cannot use unlabeled samples from any source.
4.1 RESULTS
Tables and 3 show the classification accuracy at the final time step when the main datasets are selected as CIFAR10. C1FAR100, and Imagee,, respectively. almost all the experiments, URSL outperforms all other baselines. There are two reasons for the superiority of URSL over GD and DM: (i) Unlike GD and DM, which train OoD detection with a small number of labeled samples, OoD detection of URSL is based the representation of the reference network, which is trained with a large amount of unlabeled data and has high discrimination power. (ii) GD only uses these unlabeled data to solve the forgetting, while URSL uses those to transfer rich representation from the reference network to the learner network. Although CopL-p and Co2L-j improved Co²L, URSL outperformed them in all scenarios, showing the effectiveness of the proposed ideas compared with
7

---

Under review as a conference paper at ICLR 2023
Table Accuracy of different models the Imane-e dataset.
Unsupervised
Method
Setting
Dataset
Co²L
Co²L-j
Co²L.p
GD
DM
URSL
P 0.05
CIFARIO
8.4
11.020.
12.8
4.54
4.8go.
17.2
|M| 1000
CIFARI00
8.4
10.8+0.s
12.9+a.
5.7
4.4+0.s
17.5
P 0.1
CIFARIO
10.l
17.6
18.4
7.5
5.6
21.9
M 2000
CIFARI00
15.0
18.4+0.7
18.6+a.
7.9
5.4
20.8+1.0
Co²1.
GEM
iCaRI.
P =
None
22.540.5
17.4.0.a
18.2
M 2000
Table Ablation of Eq. 7 on CIFARI00 classification with CIFARIO dataset as peripheral.
URSL w/o
URSL w/o
URSL w/o
Version
Only Loup
Only LKD
URSL
Coup
LTD
LKD
Acc.(%)
25.7+0.2
28.4
28.9
19.1
28.2
30.4
the naive approaches for incorporating unlabeled data. Furthermore, URSL achieved comparable even better results than state- tf-t--a-tt full-supervised CL methods. This phenomenon suggests that URSL benefit from unsupervised samples to mitigate the forgetting of previous classes or to learn a general representation that is proper for learning the classes that will be observed in the future.
We provide multiple benchmarks in Appendix D to show the robustness and power of method. For instance, After and Before scenarios. in which the unlabeled related samples are respectively restricted to the future and past classes of the main dataset, prove that our method has positive forward and backward transfer. In a Non-I.I.D. scenario, we examined our method in an environment in which only a fraction of the classes of the main dataset are present in Ue at each time step. Additionally, Appendix [ indicates that our method can achieve remarkable performance even in situations in which the ratio of the number of related unsupervised samples to number of unrelated unsupervised samples is very low.
4.2 ABLATION STUDIES
In this section, we conducted experiments to demonstrate the contribution of different components of the model the final performance. To that end, have selected CIFARI00 as the main dataset,
CIFARIO as the peripheral dataset, P = 0.05. and [ = 500. Table g indicates the model's performance in the experiments created by ablations over the losses of the model presented in Eq.
Effect of Cuup: Coup induces the representation of the learner network to discriminate between classes directly by using supervised contrastive loss and labels; therefore, As the results suggest, this loss is important and contributes to the performance of the model. Adding Loup to the URSL w/o Coup version is increased the performance by 4.7%. Moreover, although LKD provides great discrimination for the learner network and achieves 28. 28.2% accuracy, adding Loup to this version still enhances the performance.
Effect of CTD: The role of LTD is to transfer previously Icarned knowledge and reduce forgetting. The performance the model is increased from 19.1% to 28.9% only by adding LTD to Only Caup version. Furthermore, Although LKD reduces forgetting in another way, adding LTD to URSL without LTD p increased performance from 28.4% to 30.4%. All mentioned comparisons indicate that this loss can effectively help the model avoid forgetting.
Effect of D LKD is a new way to utilize unlabeled samples of the environment. This loss intends transfer the reference network's 's rich knowledge about the environment to the learner network. As
8

---

Under review as a conference paper at ICLR 2023
Table 5: Ablation of OoD CIFARI00 classification with CIFAR10 dataset as peripheral.
Experiment
Variant 1
Variant 2
Variant 3
Variant 4
Eqs.
and
TUM
TUM
TUMUTE TUMUT
Data
Eq. 6
TUMU
TUMU
TUMUU, TUMUU
Acc.(%)
20.7 24.5.0.4
29.2
30.4
it is shown, using this loss alone train the leamnr network achieves great performance. In addition, adding LKD to Only Loup increases performance from 19, 1.1% to 28.4%. Although LKD and LTD both reduce forgetting in different ways and have overlap in their function, adding LKD URSL without LKD still boosts the performance.
is worth mentioning that performance reduction from Only LKD version to URSL without Cuup version is because of equality of LKD and LTD coefficients, X, and y, respectively. The high ratio of prevents the model from learning new tasks and reduces the plasticity of the model.
The next study, provided in Table indicates the importance of the segregation module in providing Ti and Us to the model. In the below paragraphs, the results are discussed:
Effect of 7;: Due to the fact that there exists a variety of images that differed from the current classes among unlabeled data, we are not able to use unlabeled data naively in Eqs. and Therefore, In experiments, we investigate the effect adding Te to the data of Eqs. 3 and 5 The results indicate the importance of adding Tr in preventing the overfitting caused by the limited number of data from the past classes. Remarkable boost in the performance of Variant and Variant can be seen by
comparing them with Varian/ 2, and Variant 4, respectively.
Effect of L4, Eq. 6 is designed to transfer the rich representation of the reference network to the learner network. The results show that adding Ue this loss naively and without segregation leads to the transfer of irrelevant knowledge to the learner network. That unrelated transferred knowledge to tasks prevents the model from learning discriminative representation for the target tasks. Segregation of le boosts the performance of Variant 1, and Variant 3 by 3.8% and 1.2 1.2%, respectively.
CONCLUSION
In this paper, we present OSSCL, a novel setting for continual learning which is more realistic than previously studied settings. The setting assumes that the agent has access large number of unsupervised data in the environment, some of which are relevant to tasks due to the similarity between surroundings and tasks. As a possible solution for this setting, we presented novel model, consisting of supervised learner and an unsupervised reference network, to effectively utilize both supervised and unsupervised samples. The learner network benefits from three loss functions; the supervised loss function, which is formed based on limited supervised samples and segregated unsupervised samples, knowledge distillation through time, and representational guidance from the reference network. URSL has outperformed other state-of-the-art continual learning models with considerable margin. The experiments and ablation studies demonstrate the superiority of the model and the effectiveness of each of its components.
REFERENCES
Dosovitskiy Alexey, Philipp Fischer, Jost Tobias, Martin Riedmiller Springenberg, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Trans. Pattern Analysis and Machine Intelligence, 99, 2015.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.

---

Under review as a conference paper at ICLR 2023
Yogesh Balaji, Mehrdad Farajtabar, Dong Yin, Alex Mott, and Ang Li. The effectiveness of memory replay in large scale continual learning. arXiv preprint arXiv:2010.02418, 2020.
Mohammadamin Banayecanzade, Rasoul Mirzaiezadeh, Hosein Hasani, and Mahdieh Soleymani. Generative vs. discriminative: Rethinking the meta-continual learning. In Advances in Neural Information Processing Systems, volume 34, pp. 21 21592-21600 604. Curran Associates, Inc., 2021.
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218-888 02-8222722 2021.
Guo-qiang Bi and Mu-ming Poo. Synaptic modifications in cultured hippocampal neurons: depen- dence spike timing, synaptic strength, and postsynaptic cell type. Journal of neuroscience, 18 (24): 144:4-004411472 1998.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912- 9924, 2020.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference Compuler Vision (ICCV), pp. 9516-9525, October 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint trXiv:2002.05709 2020a.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3569-3576, 2020b.
Francesco De Comité, François Denis, Rémi Gilleron, and Fabien Letouzey. Positive and unlabeled examples help learning. International conference on algorithmic learning theory, pp. 219-230. Springer, 1999.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv: 1rX.0:1606.00704, 2016.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 213-220, 2008
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-133) 1999.
Saurabh Garg, Yifan Wu, Alexander J Smola, Sivaraman Balakrishnan, and Zachary Lipton. Mixture proportion estimation and pu learning: A modern approach. Advances in Neural Information Processing Systems, 34:8532-8544, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1rXiv:1803.07726 2018.
Gregory Griffin, Alex Holub, and Pictro Perona. Caltech-256 object category dataset. 2007.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory- based lifelong learning. Advances Neural Information Processing Systems. 33:1023-1035. 2020.
Geoffrey Hinton and Terrence Sejnowski. Unsupervised learning: foundations of neural computa- fion. MIT press, 1999.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint S.t3:05503.023//1 2(7), 2015.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 83 8310-8319, 2021.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning, Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02222. 2016.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks, In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In ICCV, 2019.
Sang-Wor Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Wor Woo Ha, and Byoung-Tai -aa Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Chongxuan Li, Taufik Xu, Jun Zhu, and Bo Zhang, Triple generative adversarial nets. Advances in neural information processing systems, 30, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, pp. 614-629. Springer, 2016.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, pp. 6467-6476. Curran Associates, Inc., 2017.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. Europeun conference on computer vision, pp. 69-84. Springer, 2016.
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. Neural Networks, 113:54-71, 2019. ISSN 0893-6080.
Jongjin Park, Sukmin Yun. Jongheon Jeong, and Jinwoo Shin. Opencos: Contrastive semi-supervised learning for handling open-set unlabeled data. CoRR, 0bb/22107.08943 2021.
Hieu Pham. Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 11557-1 11567-11568, 2021.

---

Under review as a conference paper at ICLR 2023
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pp. 524-540. Springer, 2020.
Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference Learning Representa- fions, 2021.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. Advances in neural information processing systems, 28, 2015.
S. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5543, 2017.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks, 2016.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with open-set consistency regularization. Advances in Neural Information Processing Systems, 34: 25956-25967, 2021.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE international conference on computer vision, pp. 3400-3400 2017.
James Smith, Jonathan Balloch, Yen-Chang Hsu, and Zsolt Kira. Memory-efficien cemi-supervised continual learning: The world is its own replay buffer. arXiv preprint Accepted for publication IJCNN 2021.
2021.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying comi-supereised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv: 066390, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 99-108, 2022.
Gido M. van de Ven, Hava T. Siegelmann, and Andreas S. Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11(1), August 2020.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103. 113, 2008
Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, and Jun Zhu. Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning. 2021.
12

---

Under review as a conference paper at ICLR 2023
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, volume 33. pp. 15173-15184, Curran Associates, Inc., 2020.
Chenshen Wu, Luis Herranz, Xialei Liu, yaxing wang, Joost van de Weijer, and Bogdan Raducanu. Memory replay gans: Learning to generate new categories without forgetting. In Advances in Neural Information Processing Systems, volume 31, pp. 5962-5972. Curran Associates, Inc., 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33:6256-6266 6268, 2020a.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10687-10698, 2020b.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In European Conference Computer Vision, pp. 138-454. Springer, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelli- gence. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987-3995, PMLR, 2017.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael Frank, James DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), 2021.
13

---

Under review as a conference paper at ICLR 2023
Table 6: The Running time of a single task for the URSL and baselines for CIFAR100 classification with CIFARIO as peripheral dataset
Co²L.p
URSL
Method
Lo
GD
DM
Pretrain (only once)
learning task
Reference
Learner
Training (iin(mi)ttes)
12
63
6.5
8.5
14.5
15.75
12
A RELATED WORKS
Continual Learning Three families of approaches have been proposed to address the issue of forgetting in continual learning. Replay-based methods reuse samples from previous tasks either by keeping raw samples
limited memory buffer (Rebuffi et al., 2017: Lopez-Paz & Ranzato Aljundi et al., 2019) or by synthesizing pseudo-samples from past classes (Shin al. [017]] Wu et al. 2018 van de Ven et al. 2020). Regularization-ben based methods aim to maintain the network's parameters stable across tasks by penalizing deviation from the important parameters for
the previously learned tasks (Nguyen et af., (2088) Lee et al. 2017] Zenke et al., 2017 Cha et al.]
2021 Rebuffi al. 2017). Methods based on parameter isolation dedicate different parameters to each task by introducing new task-specific weights or masks (Rusu et al. 2016 Yoon et al. 2018
Wortsman al. 2020).
Methods based on parameter isolation suffer either from extensive resource usage or capacity shortage when the number of tasks is large. Regularization-base methods are promising when the number of tasks is small; however, as the number of tasks increases, they become more prone to catastrophic forgetting and failure. However, replay-based methods have shown promising results in general continual learning settings. This work can be categorized as a replay-based method.
Self-supervised Learning Self-supervised learning methods are being explored to learn a represen- tation using unlabeled data such that the learned representation will be able convey meaningful semantic or structural information. Based on this, various ideas, such as distortion (Alexey et al. 2015 Gidaris al. 2018), jigsaw puzzles (Noroozi & Favaro 2016), colorization (Zhang al. 2016), and generative modeling (Vincent et al. 2008), have been investigated. Meanwhile, con- trastive learning has played significant role recent developments of self-supervised representation learning. Contrastive learning involves learning embedding space in which samples (e.g., crops) from the same instance (e.g., an image) are pulled together, and samples from different instances are pushed apart. Early work in this field incorporated some form of instance-level classification with contrastive learning and was successful in some cases. The results of recent methods such as SimCLR (Chen et al. 2020a), SwAV (Caron al. 2020), and BYOL (Grill et al. 2020) are comparable to those produced by the state-of-the-art supervised methods.
Knowledge Distilation Knowledge distillation aims to transfer knowledge from teacher model to student model without losing too much generalization power (Hinton et al. 2015). The idea was adapted in continual learning tasks to alleviate catastrophic forgetting by keeping the network's responses to the samples from the old tasks unchanged while updating it with new training samples (Shmelkov et al. 2017: Rebulfi et al. 2017
Hoiem 2016). iCaRL (Rebuifi et al. 2017) applics distillation loss maintain the probability vector of the last model outputs in learning new tasks, while UCIR (Hou et al. 2019) maximizes the cosine similarity between the embedded features the last model and the current model. L (Cha al. 2021) proposed novel instance-wise relation distillation loss for continual learning that maintain features' relation between batch samples in the representation space.
Semi-supervised learning In practical scenarios, the number of labeled data is limited; therefore, training the models using such limited labeled data leads to low performance. Due to this fact, semi-supervised learning methods try utilize the unlabeled data among the labeled data to achieve better performance. There are three main categories of semi-supervised training methods: generative, consistency regularization, and pseudo-labeling methods. A generative method can learn implicit and
14

---

Under review as a conference paper at ICLR 2023
Table 7: hyperparameters search space
Parameters
Values
Es
{100, 200
Er EIy1 T
{200, 300, 400}
{100, 200
{0.1, 0.5)
X
(0.05, 0.2, 0.4)
(Mka- Tpp)
[(-4, -2), (-2, 0), (-2, 2)) (SGD momentum, Adam) {0.1, 0.013
Optimizer
Initial Learning rate
transferable features of data in order to model data distributions more accurately in supervised tasks
(Springenberg) | Dumoulin al.) 2016 [i et al.) 2017). Consistency regularization describes a category of methods in which the model's prediction should not change significantly if realistic perturbation is applied to the unlabeled data samples (Rasmus et al. 2015 Laine Aila [016]]
Tarvainen & Valpola 2017). By pseudo-labeling, trained model on the labeled set is utilized to provide pseudo-labels for a portion of unlabeled data in order to produce additional training examples that can be used as labeled samples in the training data set (Lee et al. 2013 Xie et al. 2020b Pham jet al. 2021). UDA (Xie et al. 2020a) and Fix.attc (Sohn et al. 2020) are two examples of recent brilliant works in semi-supervised learning. UDA (Xie et al., (200aa) employs data augmentation methods as perturbations for consistency training and encourages the consistency between predictions on the original and augmented unsupervised samples. In the FixMatch method (Sohn al. 2020), consistency regularization and pscudo-labeling are combined, and cross-entropy loss is used to calculate both supervised and unsupervised losses.
Open-set Semi-supervised Learning Most semi-supervised learning methods assume that labeled and unlabeled data share the same label space. Nevertheless, in the Open-set Semi-supervised Learning setting, unlabeled data can contain categories that aren't present in the labeled data, i.c.. outliers, which can adversely affect the performance of SSL algorithms. In UASD (Chen et al.) 2020b), soft targets produced by averaging predictions from some temporally ensembled networks, and distii--dittibuttinn samples are detected using a simple threshold applied to the largest prediction score. Using cross-modal matching strategy, Huang et al. (202I) trained a network to predict whether a data sample matches a one-hot class label or not. By using this module, they filter out samples that have low matching scores with all possible class labels. In Saito et al. (202T), inlier confidence scores were calculated using one-vs-all (OVA) classifiers. Furthermore, coft-consistency regularization loss also applied to enhance the OVA-classifier's smoothness. thereby improving outlier detection.
Detection Previous works semi-supervised settings utilized an out-of- distribution detector in order to filter relevant unlabeled data. Some methods train K-way classifier, assign pseudo-labels to unlabeled data and incorporate them in the training procedure. Due to the neural networks' overconfidence over even noisy data (Hsu al. 2020), these methods use specific techniques to alleviate this phenomenon. Lee et al. 2019) trained a classifier using the confidence calibration technique in order lower confidence over unseen data. In this work, they sampled a bunch of random data from a massive dataset like ImageNet and applied loss to reduce the model confidence on them. [mmith al. (202T) used another technique called DeConf, by which they data. calibrated probabilities only using in-distribution data and without needing out-of-distribun In another type of method called " "Learning from Positive and Unlabeled Data" (Comité et al.) 1999) Elkan & Noto 2008 Garg et al. 2021). authors train a binary classifier that demonstrates whether each input is in in-distribution or not. Garg et al. (2021) proposed an iterative two-stage method in which first they estimate that determines the mixture proportion of positive data among unlabeled data. Then, they train a classifier using estimated a. They iterated these two stages until a convergence criterion satisfied.
15

---

Under review as a conference paper at ICLR 2023
Table 8: Chosen hyperparameters for URSL.
Parameters
Values
Es
200
Er,
400
Ery,
100
Batch size
512
0.1
T'
0.01
7"
0.2
0.2
?
0.2
Maa. 7p))
(24
Optimizer
Adam
Initial Learning rate
0.01
Minimum learning rate
le-4
B DETAILS OF EXPERIMENTAL SETUPS
B.1 DATASET DETAILS
The CIFARIO dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. If this dataset is used the main continual dataset, we randomly split it into 5 tasks with 2 classes per task. For this dataset, we used the ratios of 0.01 and P 0.1 for the supervised samples, respectively equal to and 500 samples per class.
The CIFAR100 dataset contains 100 classes with 500 training and 100 test samples for each class. Fach supervised task includes the training samples of 10 classes if this dataset is used as the main continual dataset. In Table 2 of the main paper, we used P = 0.05 and = 0.1 configurations for this dataset, corresponding to 25 and 50 training samples per class.
Imay-em is a subset the Imagenet dataset which contains 200 classes, 100000 training samples, and 10000 test samples. Before using the dataset, we downsize the input images from 64x64 to 32x32 in order to make all image sizes equal. We split the dataset into 10 equally sized supervised tasks. Similar to CIFAR100, this dataset is used with P 0.05 and P 0.1 ratios which equivalent to 25 and 50 training samples per class.
Caltech256 is object recognition dataset that contains 30607 real-world images from 257 categories. Images sizes are different from each other, and the minimum number of images per category is 80 images. We only use this dataset in Appendix(F]] Fo increase the number of datasets in U4, in order to diversify the objects in unlabeled samples and provide a more realistic environment.
B.2 TRAINING DETAILS
As explained in the main paper, we used two datasets for each experiment: (i) the main dataset to construct the supervised and related unsupervised samples and (ii) the peripheral dataset to provide unrelated unsupervised samples, At each time step, 9000 unlabeled samples are provided from the main dataset and 9000 unlabeled samples from the peripheral dataset.
In our experiments, the ResNet- 18 architecture is used as the encoder for our method, as well as all other baselines. In our method, starting from random initialization, the reference network is trained for Ër, = 400 epochs at time step t = 1 to converge to a good representation. However, for subsequent time steps, it would only be trained for Ers. 100 epochs. On the other hand, the learner network is trained for Es = 200 epochs in all time steps like all the baseline methods whose main number of epochs is 200. The mean and standard deviation of results are obtained over 3 runs.
In Table the required running times to train a single epoch of different models are reported. These results recorded on GeForce RTX 3080 Ti GPU.
16

---

Under review as a conference paper at ICLR 2023
Table 9: After and Before Benchmarks of CIFAR100 classification with the CIFAR10 dataset as peripheral
Only Supervised
Setting
After
Before
Acc.(%)
15.9
19.1
20.0+o.1
Table 10: Only Related and Only Unrelated Benchmarks of CIFARIO0 classification with CIFAR10 dataset as peripheral
Only Supervised
Only Unrelated
Only Related
Setting
OSSCL
Acc.(%)
15.9
19.6
29.4
30.4
B.3 TUNING THE HYPERPARAMETERS
We created a validation set for all three main datasets by selecting 10% of the training samples at random and then performed hyperparameter search according to Table7] Tabld8 shows the chosen hyperparameters obtained either by considering the validation results or by adapting from the Co²L paper. The strength of our proposed method that the selected hyperparameters different scenarios, and we used single configuration for all experiments. For Co²L, DM, and GD, we used the optimal hyperparameters Co and Co used a similar set of hyperparameters as URSL except for the new hyperparameter introduced in Co where the unsupervised loss coefficient was set to 1.
invariant across
the authors reported it in the original papers. In addition.
B.4 AUGMENTATIONS
To increase the diversity training samples, following previous works (Cha et al. 2021 Chen et al.) 2020a), used the following augmentation techniques for all data:
RandomResizedCrop The image randomly cropped with the seale [0.2, 1] and then the cropped image will be resized to 32 32.
2. Randonnorizzotalliin Each image is flipped horizontally with probability p 0.5, independently from other samples.
3. Color,Jitte:: The brightness, contrast, saturation, and hue of each image are changed with probability p 0.8, with maximum strength [0.4, 0.4, 0.4, 0.1], respectively.
4. RandomGrayscale: Images are converted to grayscale with probability = 0.2.
B.5 TRAINING CLASSIFIER
At the end of the training, we trained a linear classifier on the learner network's encoder head for 100 epochs using all memory data and the last time step labeled data Ty. M. We used Weighted Random Sampler to draw mini-batches due to class imbalance in labeled data.
C THE PERFORMANCE OF THE OoD DETECTION
In this section, we evaluate the performance first one, the number of related and unrelated data is 9,000 each, and for the other one, this number is 4,500. Precision and AUROC metric diagrams for the OoD detection module have been shown in those two settings (during the time steps) in Figures and 3 It can be that the performance of the OoD detection module is improved over time due to the fact that sees more classes and can detect class boundaries more precisely.
OoD detection module two scenarios. For
17

---

Under review as a conference paper at ICLR 2023
0.8
0.7
0.6
AMRCC
0.5
0.4
0.2
0.1
0.0
30
60
90
Number of Accumulsted Clesses
Task Number
Figure 2: (left) AUROC of OoD detection based the number of the main dataset seen classes for CIFAR 100 classification with CIFARIO dataset as peripheral when the number of related and unrelated data are 9000 (right) The precision of OoD detection at each task of CIFAR 100 classification with the CIFAR 10 dataset as peripheral when the number of related and unrelated data is 9000.
0.8
0.7
0.76
0.72
0.5
0.4
0.68
0.2
0.64
0.1
60
90
10
Number Accumulsted Clesses
Number
Figure 3: (left) AUROC of OoD detection based on the number of the main dataset seen classes for CIFAR 100 classification with CIFARTO dataset as peripheral when the number of related and unrelated data are 4500 (right) The precision of OoD detection at each task of CIFAR 100 classification with the CIFARIO dataset as peripheral when the number related and unrelated data is 4500
D OTHER BENCHMARKS
In the "Other Benchmarks" section of the paper, we investigated different configurations to demon- strate the effectiveness and robustness of the URSL model in dealing with various conditions. All benchmarks are clarified, and the results are analyzed further below.
After and Before In these two benchmarks, assume that there unlabeled data. The difference between these two settings is the presence of classes from the main dataset in the unlabeled data any time step. More specifically, in the After scenario, unrelated samples are only from classes of the current time step and future classes from subsequent time steps provided. However, in the Before scenario, the unlabeled data from only previous classes presented. This experiment was designed to show that the URSL model can benefit from a positive forward/backward knowledge transfer from the unsupervised samples the supervised tasks. As shown in Table9] in the Before scenario, it seems that visiting unlabeled data of previous classes helps to mitigate catastrophic forgetting (positive backward transfer). In contrast, in the After scenario, the model learns decent representation space which is beneficial for learning the new coming classes (positive forward transfer).
unrelated samples among
Only Related and Only Unrelated To investigate the effect each type of unlabeled data on the model's functionality, we defined Only Related and Only Unrelated settings. As their names suggest, in the former, all unlabeled data at each task only contains main dataset samples. In contrast, in the latter, all unlabeled data only from the peripheral dataset. In Table [0]] comparing Only Unrelated with Only Supervised shows that even unrelated samples improve performance by enriching the representation of the reference network and providing pivot model that prevents the
18

---

Under review as a conference paper at ICLR 2023
Table 11: Non-L.LD. Benchmarks of CIFARI00 classification with CIFAR10 dataset peripheral
Only
Non-L.DD. (25 %)
Non-L.L..
Setting
Supervised
(50 %)
OSSCL
Acc.(%)
15.9
24.9+
27.4
30.4-o.2
Table 12: The effect of the number of related and unrelated samples on CIFARI00 classification with CIFARIO dataset as the peripheral dataset.
Related-Unrelated (samples)
1000-9000
4500-4500
4500-9000
9000-4500
9000-9000
Accuracy(%)
21.0a..1
26.8
27.0
30.0
30.4
learner network from high accumulative changes during the continual learning process. Also, the Only Related scenario demonstrates the effectiveness of existing related samples among unlabeled data in performance, and when compared with the OSSCL setting, the effectiveness of unrelated samples can be understood. It is worth mentioning that although most of the improvement due to incorporating related samples, accessing pure related unlabeled data is not usually a realistic assumption. Instead, they are among huge stream of unlabeled data containing unrelated samples. Therefore, we considered both related and unrelated datasets as unlabeled samples (in the OSSCL setting) and showed that properly employing these datasets (in URSL) further improves the results compared with the OnlyRelated case.
Non-L.L.. The OSSCL scenario considers I.I.D. assumption on the related unsupervised samples available in the environment. To challenge this assumption, we introduced a new benchmark in which the related data is generated only from portion of the supervised classes each time step. For example. in the Non-I.I.D. (50 %) experiment, the related unsupervised dataset, only includes the samples from half of the supervised classes, which are randomly selected at each time step. As shown in Table [1] the URSL model still demonstrates good performance even with this limited access to the related unlabeled samples.
E NUMBER OF RELATED AND UNRELATED SAMPLES
In this section, we investigated the effect of the number and ratio the related and unrelated samples among unlabeled data. In contrast other baselines, URSL is able utilize unrelated unlabeled samples to boost final performance even with an imbalanced number of related and unrelated unlabeled sets. Table 12 shows that increasing unrelated samples improves results slightly while increasing related samples provides the model with more in-distribution samples to improve its performance and combat catastrophic forgetting.
F MORE COMPLICATED ENVIRONMENTS
In this section, we examine the performance of our model in even more realistic environments by conducting more experiments in scenarios in which the unlabeled data is comprised of multiple datasets. Table 13 shows the performance experiments. Besides the datasets we used in the main experiments, we also used Caltech256 (Griffin et al.) 2007 in our experiments. At each experiment, we add 9000 samples from each dataset sampled randomly to the T.. The results suggest that our model is robust to variety of unlabeled data and performs well in more realistic scenarios in which the model is exposed to plenty of unlabeled samples that most which are not related to its target tasks.
19

---

Under review as a conference paper at ICLR 2023
Table 13: The results of using multiple datasets in U stimulate realistic environment.
Dataset1
Dataset2
Dataset3
Dataset4
Acc.
CIFARIO
CIFAR 100
30.4
CI
CIFAR
Tiny-Iman Imagenet
30. +0.4
CIFARIO
CIFAR 100
Caltech256
31.4 40.5
CI
CIFAR 100
Tiny-lmagenet
Caltech256
31.620.7
Table 14: effect different architectures for the reference and the learner network on CIFARI00 classification with CIFARIO dataset as the peripheral dataset.
Reference (#parameters)
Learner (#parameters)
Accuracy(%)
ResNet- 18 (11.1M)
ResNet-18 (11.IM)
31 31.2
ResNet-34 (21.2M)
ResNet-18 (11.1M)
31.4 31.42a.
ResNet-18 (II.IM)
WideResNet-40-22 (2.24M)
30.7 20.0
ResNet-50 (23.5M)
WideResNet-28-10 (36.48M)
31.1
G THE REFERENCE AND THE LEARNER ARCHITECTURES
The authors of Co (Cha et al. 2021) used ResNet-18 18 as the feature extractor architecture of their model. Following this design choice, we used the same architecture for both the learner and reference networks as well as all other models and baselines in all experiments to ensure a fair comparison. In this section, investigate the effect of changing the architecture for the learner and the reference networks as reported in Table 14 As expected, the model's performance slightly increases as the number of model parameters grows. Moreover, deep ResNet architectures compared with wide ResNet architectures achieved better performance.
It is noteworthy that although used batch size of 512 in all of our experiments in other sections, the experiments in this section are performed with a batch size of 128 to meet the memory limit requirement, in addition providing fair comparison.
H MEMORY BUFFER SELECTION ALGORITHM
Selecting the suitable samples to be stored in the memory is active area of research in continual learning (Bang et al. 2021 Tiwari et al. 2022 Isele & Cosgun 2018). However, the purpose of our research was focus on memory selection policies. Therefore, have used random policy as
it is widely adopted in many CL works (Prabhu et al. 2020 Guo et al. 2020 Balaji et al. 2020
It is noteworthy that the segregation of unlabeled data provides more diverse data than what exists in the memory buffer from past classes. Nevertheless, because the stored samples in the memory buffer play an important role segregating the unsupervised samples we conducted experiments using different selection algorithms for memory buffer samples. In addition to the "random" selection method, we defined three other selection strategies:
Low-confidence: select the data which the model has low confidence
High-confidence: select the data on which the model has high confidence
Rainbow (Bang et al. 2021): select from all the ranges of confidence. This algorithm calculates a confidence score for each sample and sorts all scores; then, selects some data by considering the presence of samples from all ranges of model confidence.
Table 15 shows the performance of all algorithms:
As be seen, the "Random" selection algorithm outperforms both the "High-confidence" and cow-confidence" selection strategies by good margin. Moreover, the "Rainbow" achieves similar results as the "Random" strategy.
20

---

Under review as a conference paper at ICLR 2023
Table 15: effect of different algorithms for data selection for memory buffer CIFAR100 classifica- tion with CIFARIO dataset as the peripheral dataset.
Algorithm
Low-confidence
high-confidence
Rainbow
Random
Accuracy(%)
67.9+a.9
69.7+1.%%
72.5+a.c % 72.8
Table 16: Comparison of URSL with URSL with full-pretraining
URSL with Pretrain
Method
URSL
Acc.(%)
30.4
31.3
FURTHER EXPERIMENTS
1.1 PRETRAINING THE REFERENCE NETWORK
The reference network expected gradually absorb unsupervised knowledge from the environment. In this section, we designed an experiment to show the success of the reference network in continually learning the unsupervised samples. In this experiment, the reference network is first pre-trained with all unsupervised samples before starting the learning of the first supervised task. Next, the reference network is frozen, and its parameters are maintained throughout the entire learning procedure. Other training details and learning mechanisms of the learner network are the same as the original URSL model.
Table 16 demonstrates that pretraining only slightly improves the URSL results. This suggests that the unsupervised samples available in the environment are sufficient for the reference network to learn proper representation even if the data is observed continually.
1.2 SELF-SUPERVISED METHOD
In whole experiments, used NT-Xent loss (Sohn 2016), a popular and straightforward contrastive loss, which is widely used in self-supervised learning literature (Chen et al.) 2020a) and achieved remarkable performances for training the reference network. However, our model and algorithm perform well regardless of the self-supervised loss used to train the reference network. To demonstrate it, we compared the results of our model with experiments in which the reference network trained by BYOL (Grill et al., 2020). different self-supervise algorithm from NT-Xent. Tables 17 and 18 report the results of runs for two different scenarios.
The advantage of BYOL over SimCLR is that it does not need negative data in training. Indeed, in our experiments, datasets have less diverse samples than huge datasets such as ImageNet; therefore, we expect B BOOL to perform better than SimCLR. The empirical results are also confirmation of this point. However, BYOL needs more time to obtain comparable results.
J LIMITATIONS
In our method, there exist several limitations. First, we have to keep a minimum number of main dataset samples from each class in the memory buffer in order to create more precise prototypes; Second, due to the non-parallelism of the training phase of the teacher network and the student network, the time complexity our method is higher than other methods and baselines. Furthermore, model performs worse if the number of samples for each class becomes rigorously imbalanced.
There exist other limitations related to the Open-Set Semi- Sumi-iupere Continual Learning scenario. Although this configuration seems more realistic than the previous works in literature, there may still exist situations in which the assumptions of OSSCL do not hold. For example, an agent may have limited access to both related and unrelated unlabeled samples in the environment. This will
21

---

Under review as a conference paper at ICLR 2023
Table 17: Comparison between NT-Xeent and BYOL performance CIFAR10 classification with uny-Imagenet dataset as peripheral
Main dataset
Peripheral dataset
SSL method
Accuracy (%)
Time cost (mins)
CIFARIO CIFARIO
Tiny- -ny-emtee Tiny-Imagenet
SimCLR BYOL
72.8+a.0 %
136 mins
73.1+1.0%8 %
258 mins
Table 18: Comparison between NT-Xent and BYOL performance CIFARI00 classification with CIFARIO dataset as peripheral
Main dataset
Peripheral dataset
SSL method
Accuracy (%)
Time cost (mins)
CIFAR100
CIFARIO CIFAR10
SimCLR BYOL
30.4 %
219 mins
CIFARIO0
32.3 %
445 mins
lead to poor performance of model since it is designed perform in situation where plenty of unsupervised data exists.
K CODE AND DATA AVAILABILITY
The source code to reproduce the results of this paper is attached to this document. In this repository, there exists README file containing instructions and configuration details. Moreover, the licenses of the freely available datasets and used source codes are also available in the README file.
22


Reviews:
[{"review_id": "wrRRawA33k", "reviewer": "Reviewer_GwGC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Open-Set Semi-Supervised Continual Learning, where the experimental setting is essentially a class-incremental learning scenario with unlabeled data from both in- and out-of-distribution. The proposed method consists of two components: the unsupervised reference model learns with the SimCLR loss and in charge of training representation and thresholding unlabeled data, and the supervised model learns with the supervised contrastive loss, previous model, and the unsupervised reference model. Experimental results show the effectiveness of the proposed method in small image benchmarks.", "summary_of_the_review": "I appreciate thorough experimental results, but the proposed scenario and the experimental setting do not seem really match, and ablation study on the hyperparameter choice is not enough. I think this is a good paper if they limit their scenario to low-shot learning. Please answer my concerns above.\n\n**post-rebuttal**\n\nAfter reading the rebuttal, I decided not to change my rating, as I cannot agree with the validity of the proposed scenario.\n\nThe authors claimed that \"The difference between OSSCL and the setting of Lee et al. [1] is that their setting is fully supervised, which makes the setting unrealistic.\"\n\nI don't specifically agree with this argument. Once you incorporate unlabeled dataset, there is no significant difference between fully-supervised and semi-supervised, as we have both labeled and unlabeled data in both cases.\nThe proposed setting assumes that a portion of unlabeled dataset is guaranteed to share the same data distribution with the labeled dataset, which is more unrealistic to me. Rather, exposing a large unlabeled dataset to each time step of continual learning is more realistic. [1] used 80M Tiny Images as an unlabeled dataset, which is the source of CIFAR datasets. This means that unlabeled data in their setting includes data sharing a similar inductive bias to CIFAR in a natural way.\n\nThe scenario of [2] makes sense in some cases, as they limited the setting of each time step to the conventional semi-supervised learning; as authors said in the rebuttal, \"unlabeled data at all time steps are drawn from the same dataset as the labeled data had been drawn from\" in [2]'s setting.\n\nHowever, this paper distinguishes main-labeled as labeled dataset and a union of main-unlabeled and peripheral (OOD) datasets, but at last, we end up with having a pair of labeled and unlabeled dataset, which is essentially not different from [1].\nBut the size of both labeled and unlabeled dataset is significantly smaller than [1], so I think the experimental setting in this paper is somewhat toy-ish and seems not scalable.\n\nAgain, I could not find the reason why we need to use \"only 10% of CIFAR samples\" as a labeled dataset, to make a portion of unlabeled data to share the same data distribution with the labeled dataset.\nThis not only makes the setting unrealistic, but also toy-ish, as we have a very small number of 32x32 training data per time step.\n\nAlso, after reviewing the paper again, an additional comment on Table 3 is that, the overall performance is too low, so I do not think the comparison is so meaningful.", "strength_and_weaknesses": "Strengths\n\n+ The proposed scenario is interesting.\n\n+ The proposed method outperforms other methods in most cases.\n\nWeaknesses\n\n- It is hard to say the authors introduce a new scenario on continual learning. To me, the authors are just re-branding the setting proposed in prior works. Please make a more thorough comparison with prior works, [Lee et al.] and [Smith et al.] on the scenario. The authors have not explicitly claimed novelty on the scenario, but writing in abstract and intro is somewhat misleading, if they did not mean to claim the novelty.\n\n- In fact, I could not see much difference on the setting from the prior work from [Lee et al.]. Unless specifically focusing on few- or low-shot setting, limiting the number of training data per class would not be a concern in the real-world setting. Rather, the scalability of the method would be more interesting and important. Based on the experimental result on the first few tables, the scalability with respect to the dataset size is partially confirmed, but the largest setting is still too small; only 10% of CIFAR. However, based on the experimental result on Table 14, the proposed method is not scalable with respect to the model size.\n\n- Similar to above, I could not find the reason why the number of unlabeled data should be limited to only 9k per \"time step\" (this is confusing, \"time step\" sounds like one iteration or even smaller unit. I personally think \"stage\" would be a better term). There are plenty of unlabeled data in the open world, and there is no reason to randomly sample only 9k of them. Indeed, based on Table 13, the proposed model seems not scalable much with respect to the number of unlabeled data.\n\n- Ablation studies on the hyperparameters are required. For example, the choice of $\\eta_{id}$ and $\\eta_{pl}$ seems critical to the performance of the proposed method.\n\n- Training time is different over methods. Are authors sure if all methods are converged or early-stopped to show their best performance?\n\n- $L_{KD}$ takes $\\theta_t$ as a fixed set of parameters but missed.\n\n- FixMatch and other few papers appear multiple times in the reference section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hqSgFv4cdg", "reviewer": "Reviewer_HjYX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on a version of a continual learning problem where for each continual task a learner is given a set of labeled instance, and a set of unlabeled instances in which some are relevant to the task and some are not.  The authors make the claim that this version of the continual learning task is most realistic in practice and is not explicitly studied in prior work. The authors propose a method called URSL that 1) Trains a \"reference\" network using a contrastive objective on unsupervised data 2) Uses the reference network to select task relevant instance from the unsupervised data 3) Trains a separate supervised learning model using labeled instances, instances from a memory buffer, and instances determined as task-relevant (but used as negative samples).  The authors perform a series of experiments where the supervised labels are from one data set and the unsupervised data is from an entirely different data set.  URSL performs slightly better than most baselines when it comes to accuracy of the model after the final task is learned, but in some cases performs worse.\n", "summary_of_the_review": "While the authors propose to focus on perhaps the most realistic version of the continual learning problem, I feel that the proposed method is not strong in technical novelty and doesn’t introduce significant methodological or theoretic contributions, which makes empirical performance a focus.  Unfortunately, I also did not find the empirical results compelling by not performing consistently better than baselines where no unsupervised data is used.  For these reasons, I argue for rejection.", "strength_and_weaknesses": "Strengths\n1. I feel the framing of the continual learning problem where many unsupervised instances are provided for each task is a practically useful framing of continual learning.\n2. The ablation study is a necessary part of an evaluation for URSL as it is complex and includes a number of additional terms in the supervised learning objective.\n\nWeaknesses:\n1. Overall, the empirical results are not very compelling.  The proposed method is rather complex but simpler methods that use no unsupervised data are fairly competitive to the point where one method that uses no unsupervised data actually outperforms URSL.  It is hard to argue for the practical utility of URSL given the relatively low or nonexistent performance increase over some of the baselines.\n2. I find the overall experimental set up is somewhat limited.  First, accuracy results are reported only for the final task.  In continual learning the goal is to learn as tasks are presented to the learner.  Accuracy of intermediate tasks is a relevant and important result to report in the main body of the paper.  Second, the continual learning task presented has many variants based on how much supervised/unsupervised data is given and what data specifically is given as unsupervised.  The authors explore only a limited set of these, so it is hard to really understand the breadth of applicability of URSL.  Even simplifying the experiments by synthetically including known task-relevant data at some percentage to the unsupervised data to confirm the selection criteria is finding that data to learn from would be enlightening.\n3. URSL borrows considerably from prior work.  As far as I can tell the basic procedures used to train each model are taken from prior work (namely SimCLR and Co^2L).  The main contributions is the method for sampling task-relevant features for Co^2L for the supervised model and two additional objective terms for knowledge transfer.  Without much principled justification (though some intuitive justification is provided), I do not find this work particularly novel at a technical level.\n\nA few additional comments:\n1. This problem seems highly related to PU learning (Garg et al; NeurIPS 2021 for example).  It would be good to see it discussed even briefly.\n2. I don’t understand the intuition behind how thresholds are set for selecting unsupervised instances.  The thresholds are set relative to a mean similarity.  This seems to result in cases where an entire batch could be dissimilar and still many instances can be selected.  Said another way, if one batch has a low mean similarity (many instances are dissimilar to the prototype) and another batch has high mean similarity (many instances are similar to the prototype), the same number of instances can be selected for both if their \\eta parameters and variances are the same.  This seems counter-intuitive.\n3. What does the following statement mean: “It is noteworthy that the representation space of the reference network is chosen for OoD detection since it provides better sample discrimination than any other representation space obtained by training over a small number of labeled samples.”? How can such a claim be justified?\n4. Why two thresholds for selection?  Intuitively if an instance is deemed task-relevant, it should have a task-specific pseudo label, yes?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JQo6OQSaeY", "reviewer": "Reviewer_4qUo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the URSL model “Unsupervised Reference network and a Supervised Learner network“, which consists of two parts: 1) The general task-agnostic reference network, which is responsible for absorbing information from unsupervised data in the environment, and 2) the learner network, which is designed to capture knowledge from a few supervised samples while it is also guided by the reference network. They utilize contrastive representation learning as a unified approach for training both the reference and the learner networks, which allows information to flow between these networks. They also combine with knowledge distillation techniques applied in the representation space to take advantage of unsupervised samples.", "summary_of_the_review": "This is an ok paper that I feel is not strong enough. As I mentioned, my concern is related to the novelty. I like that the authors provide extensive experiments with different settings. Some discussion could be explained in more detail such as: if the model overcomes catastrophic forgetting by nature “the way it is constructed”, and the transferability and scalability of the model to a large number of tasks.", "strength_and_weaknesses": "Strength:  \n- The paper is well organized and well written.\n- The proposed model performs well compared with STOA. The experiments are fair and well executed. \n\nWeaknesses:\n- The novelty of the proposed approach is limited, most of the contributions are based on previous work such as: the SimCLR (Chenet al., 2020a) loss function, and an instance-wise relation distillation (IRD) loss to transfer knowledge from the previous time step to the current model (Cha et al., 2021).\n- In (Further Experiments section 1.1) the authors said “In this experiment, the reference network is first pre-trained with all unsupervised samples before starting the learning of the first supervised task. Next, the reference network is frozen, and its parameters are maintained throughout the entire learning procedure.” Does this apply to avoid catastrophic forgetting?\n- How does increasing the number of stored samples in the memory affect the performance? \n- It would be nice if the authors can discuss the transfer and ability to scale to streams with a hundred tasks for the proposed model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "467bs6LWI0l", "reviewer": "Reviewer_aMML", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a novel setting for continual learning called OSSCL, which assumes that the agent has access to a large number of unsupervised data in the environment with some of which are relevant to tasks. To solve the proposed task, the author introduced a dual network, consisting of a supervised learner and an unsupercised reference network. The dual model can effectively utilize both supervised and unsupervised samples in the data. Different elements in the model are evaluated by ablation studies. The proposed model also outperform some other SOTA continual learning models.", "summary_of_the_review": "The model itself is based on many previous studies but the performance seems good, compared to previous models run by the authors themselves. ", "strength_and_weaknesses": "Strength:\n\nThe paper is clearly written and easy to read. The experiment part is solid and the idea of using a reference network from the environment is interesting. Each part of the losses the author introduced in their model is discussed by ablation studies. \n\nWeaknesses:\n\n1, Many of the components in the network training, such as the designing of loss functions, are borrowed from previous studies, which makes the work incremental. The authors should highlight the novelty they made compared to previous studies, instead of creating a problem (designing a paradigm, OSSCL) which has not be investigate previously, but only studied by the authors themselve. \n\n2, In the experiment, the main and the perioheral dataset do share some underlying common distribution, and hence the out-of-distribution samples are not truly out-of-distribution samples. The authors should justify this point. \n\n3, Citations need to be checked. Some of the citations are duplicate. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0w2bEMz7Kxj", "reviewer": "Reviewer_G7Pp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new continual learning algorithm that exploits unsupervised data in the environment. The model consists of two main parts: (i) reference network that learns knowledge from unlabeled data, and (ii) learner network for solving specific task of interest by utilizing labeled data. The reference network is trained using self-supervised loss function and then out-of-distribution detection technique is applied to segregate unlabeled data. The learner network is trained using three loss functions that transfer knowledge through time, reference and utilize supervised samples. The method is tested on CIFAR-10, CIFAR-100 and Tiny-ImageNet datasets. It is shown to outperform existing methods. \n", "summary_of_the_review": "Overall, this a good paper and presents an interesting and effective method for continual learning by exploiting unlabeled data in a realistic setting in which unlabeled data can contain out-of-distribution samples. The method outperforms baselines by a large margin.\n", "strength_and_weaknesses": "Strengths:\n- The paper is well organized and well motivated.\n- The ability of continual learning methods to utilize unlabeled data is important and the proposed method solves the problem in an effective way. The proposed approach based on reference and learner networks is interesting.\n- The experiments are well designed and the method achieves large improvements over existing methods. Moreover, ablation study shows that each of the components of the objective function is essential.\n\nWeaknesses/questions:\n- The method has many hyperparameters and systematic analysis of the robustness to their selection is missing. It is not clear how to select these parameters.\n- Some parts of the work need to be better explained. All parameters defined in the equations need to be defined which is not currently the case. The intuition behind each part of the objective function should be presented.\n- Are there any assumptions on the proportion of the in-distribution compared to out-of-distribution unlabeled samples? Can the performance be degraded if unlabeled data is very different from the data that the model was trained on?\n- Is the model applicable to large-scale datasets such as ImageNet? How does it compare to baselines in terms of scalability?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject