Representational Continuity for Unsupervised Continual Learning
Authors: Divyam Madaan
Jaehong Yoon
Yuanchun Li
Yunxin Liu
Sung Ju Hwang
Venue: ICLR 2022 Oral (2022)
Keywords: Continual Learning
Representational Learning
Deep Learning

Abstract: Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (LUMP), a simple yet effective technique that interpolates between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations. 

Published as a conference paper at ICLR 2022
REPRESENTATIONAL CONTINUITY FOR UNSUPERVISED CONTINUAL LEARNING
Divyam Madaan * Jaehong Y.o
2.3 f Yuanchun Lis.6 Yunxin Liu
Sung Ju Hwang?+
New York University KAIST² Microsoft Research AITRICS
Institute for Al Industry Research (AIR) Tsinghua University
di divyam. madaan@nyu edu, (jaehong. yoon, s jhwang82 @kaiist ac..a.c.... kr
Liyuanchun@air.c tsinghua. edu. cn, liuyunxin@air tsinghua, edu. on
ABSTRACT
Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where learn the feature representations show that reliance on annotated data is not necessary for continual learning. We conduct systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better out-of- distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (LUMP), a simple yet effective technique that interpolates between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations. We release our code online.
unlabelled sequence of tasks and
INTRODUCTION
Recently continual learning (Thrun, 1995) has gained lot of attention in the deep learning community due to its ability to continually learn on a sequence of non-stationary tasks (Kumar & Daume III, 2012; Li & Hoiem, 2016) and close proximity to the human learning process (Flesch et al., 2018). 8). However, the inability of the learner to prevent forgetting of the knowledge learnt from the previous tasks has been long-standing problem (McCloskey & Cohen, 1989; Goodfellow et al., 2013). To address this problem, a large body of methods (Rusu et al., 2016; Zenke et al., 2017; Yoon et al., 2018; Li al., 2019; Aljundi et al., 2019; Buzzega al., 2020) have been proposed; however, all these methods focus on the supervised learning paradigm, but obtaining high-quality labels is expensive and often impractical in real-world scenarios. In contrast, CL for unsupervised representation learning has received limited attention in the community. Although Rao et al. (2019) instantiated a continual unsupervised representation learning framework (CURL), it not scalable for nighrresolution tasks, as it is composed of MLP encoders/decoderrs and simple MoG generative replay. This is evident in their limited empirical evaluation using digit-based gray-scale datasets.
Meanwhile, a set of directions have shown huge potential to tackle the representation learning problem without labels (He et al., 2020; Chen al., 2020a; Grill al., 2020; Chen et al., 2020b; Chen & He, 2021; Zbontar et al., 2021) by aligning contrastive pairs of training instances or maximizing the similarity between two augmented views of each image. However, a common assumption for existing methods is the availability of large amount of unbiased and unlabelled datasets to learn the feature representations. We argue that this assumption is not realistic for most of the real-time applications, including self-driving cars (Bojarski et al., 2016), medical applications (Kelly et al., 2019) and conversational agents (Li et al., 2020). The collected datasets are often limited in size during the initial training phase (Finn et al., 2017), and datasets/tasks change continuously with time.
"Corresoondigg author. The work was done while the author was an intern at Microsoft Research.

---

Published as a conference paper at ICLR 2022
Viewi Bed
Diu Learne
Ma
M/px
Self Supervised
Dal
Classitter Buy.
loss
1/kz
View2
grants
SUPERVISED CONTINUAL LEARNING (SCL)
UNSUPERVISED CONTINUAL LEARNING (UCL)
Figure 1: Illustration of supervised and unsupervised continual learning. The objective of SCL is to leam the ability to classify labeled images in current task while preserving the past tasks' knowledge, where the tasks are non-iid to each other. On the other hand, UCI. aims to learn the representation of images without the presence of labels and the model learns general-purposs representations during sequential training.
To accommodate such continuous shifts in data distributions, representation learning models need to increment the knowledge without losing the representations learned in the past. With this motivation, we attempt to bridge the gap between unsupervised representation learning and continual learning to address the challenge of learning the representations on a sequence of tasks. Specifically, we focus on unsupervised continual learning (UCL), where the goal of the continual learner is to learn the representations from stream of unlabelled data instances without forgetting (see Figure 1). To this end, we extend various existing SCL strategies to the unsupervised continual learning framework and analyze the performance of current state-of-the-art representation learning techniques: SimSiam (Chen & He, 2021) and BarlowTwins (Zbontar et al., 2021) for UCL. Surprisingly, we observe that the unsupervised representations are comparatively more robust to catastrophic forgetting across all datasets and simply finetuning on the sequence of tasks can outperform various state-of saate-of-the-are ff-the-art continual learning alternatives. Furthermore, we show that UCL generalize better to various out of distribution tasks and outperforms SCL. for few-shot training scenarios (Section 5.2).
We demystify the robustness of unsupervised representations by investigating the feature similarity, measured by centered kernel alignment (CKA) (Kornblith et al., 2019) between two independent UCL and SCL models and between UCL and SCL models. We notice that two unsupervised model representations have a relatively high feature similarity compared two supervised representations. Furthermore, in all cases, two models have high similarity in lower layers indicating that they learn similar low-level features. Further, we measure the & distance between model parameters (Neyshabur et al., 2020) and visually compare the feature representations learned by different SCL. and UCL. strategies. We observe that UCL obtains human perceptual feature patterns for previous tasks, demonstrating their effectiveness to alleviate catastrophic forgetting (Section 5.3). We conjecture that this is due to their characteristic ability to learn general-purpor features (Doersch et al., 2020), which makes them transfer better and comparatively more robust to catastrophic forgetting.
To gain further insights, we visualize the loss landscape (Li et al., 2018) of the UCL and SCL models and observe that UCL obtains flatter and smoother loss landscape than SCL. Additionally, propose a simple yet effective technique coined Lifelong Unsupervised Mixup (LUMP), which utilizes mixup (Zhang et al., 2018) 8) for unlabelled training instances. In particular, LUMP interpolates between the current task examples and examples from previous instances to minimize catastrophic forgetting. We emphasize that LUMP is easy to implement, does not require additional hyperparameters, and simply trains on the interpolated instances. To this end, LUMP requires little, or no modification to methods effectively minimizes catastrophic forgetting even with uniformly selecting the examples from replay buffer. We show that LUMP with UCL outperforms the otate-of- the-art supervised continual learning methods across multiple experimental settings with significantly lower catastrophic forgetting. In summary, our contributions are as follows:
existing rehearsal-
We attempt to bridge the gap between continual learning and representation learning and tackle the two crucial problems of continual learning with unlabelled data and representation learning on sequence of tasks.
Systematic quantitative analysis shows that UCL achieves better performance over SCL with significantly lower catastrophic forgetting on Sequential CIFAR-10, CIFAR-100, and Tiny- ImageNet. Additionally, we evaluate distribbittt tasks and few- training demonstrating the expressive power of unsupervised representations.
We provide visualization of the representations and loss landscapes, which show that UCL learns discriminative, human perceptual patterns and achieves flatter and smoother loss landscape. Furthermore, we propose Lifelong Unsupervised Mixup (LUMP) for UCL, which effectively alleviates catastrophic forgetting and provides better qualitative interpretations.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Continual learning. We can partition the existing continual learning methods into three categories. The regularization approaches (Li Hoiem, 2016; Zenke et al., 2017: Schwarz al., 2018; Ahn et al., 2019) impose regularization constraint to the objective that mitigates catastrophic forgetting. The architectural approaches (Rusu et al., 2016; Yoon et al., 2018; Li et al., 2019) avoid this problem by including task-specific parameters and allowing the expansion of the network during continual learning. The rehearsal approaches (Rebuffi et al., 2017: Rolnick et al., 2019; Aljundi et al., 2019) allocate small memory buffer to store and replay the examples from the previous task. However, all these methods are confined to supervised learning, which limits their application in real-life problems. Rao al. (2019); Smith et al. (2021) tackled the problem of continual unsupervised representation learning; however, their methods are restricted to simple low-resolution tasks and not scalable to large-scale continual learning datasets.
Representational learning. A large number of works have addressed the unsupervised learning prob- lem in the standard machine learning framework. Specifically, contrastive learning frameworks (He et al., 2020; Chen et al., 2020a; Grill et al., 2020; Chen et al., 2020b;c) that learn the representations by measuring the similarities of positive and negative pairs have gained lot of attention the com- munity. However, all these methods require large batches and negative sample pairs, which restrict the scalability of these networks. Chen & He (2021) tackled these limitations and proposed SimSiam, that use standard Siamese networks (Bromley et al., 1994) with the stop-gradient operation to prevent the collapsing of Siamese networks to a constant. Recently, Zbontar et al. (2021) formulated an objective that pushes the cross- corssscorrlltoonn matrix between the embeddings of distorted versions of a sample closer to the identity matrix. However, all these methods assume the presence of large datasets for pre-training, which is impractical in real-world applications. In contrast, we tackle the problem of incremental representational learning and learn the representations sequentially while maximizing task adaptation and minimizing catastrophic forgetting.
PRELIMINARIES
3.1 PROBLEM SETUP
We consider the continual learning setting, where we learn on continuum of data consisting of T tasks T1.T = (Ti.. Tr). In supervised continual learning, each task consists task descriptor T E (1... T and its corresponding dataset D, = ( (=1)) } with n,, examples. Each input- pair (æ:,r, )i.r) E X, x Vr, where (Xr V-) is an unkown data distribution. Let us consider network fe: X, - D parametrized by θ = {wi (wr)/// where R.' and L denote D-dimensional embedding space and number of layers respectively. The classifier is denoted by has R² + Vr. The network error using cross entropy loss (CE) for SCL with finetuning can be formally defined as:
FINETUUE CE (hv (Je ( ,Bi,))
(1)
In this work, we assume the absence of label supervision during training and focus on unsupervised continual learning. In particular, each task consists of U, { Ti,r E X, with n, examples. Our aim is to learn the representations fe : Xx + R on a sequence of tasks while preserving the knowledge of the previous tasks. We introduce the representation learning framework and propose LUMP Section 4 to learn unsupervised representations while effectively mitigating catastrophic forgetting.
3.2 LEARNING PROTOCOL AND EVALUATION METRICS
Currently, the traditional continual learning strategies follow the standard training protocol, where we learn the network representations fe X, + Vi, on a sequence of tasks. In contrast, our goal is to learn the feature representations fe Xr + R,P, so we follow a two-step learning protocol to obtain the model predictions. First, we pre-train the representations on a sequence of tasks Tir (T... Tr) to obtain the representations. Next, evaluate the quality of our pre-trained representations using a K-nearest neighbor (KNN) classifier (Wu et al., 2018) following the setup in Chen al. (2020a); Chen & He (2021); Zbontar et al. (2021).
3

---

Published as a conference paper at ICLR 2022
To validate knowledge transfer of the learned representations, adopt the metries from the SCL literature (Chaudhry et al., 2019b; Mirzadeh et al., 2020). Let ar, denote the test accuracy of task after learning task T, using a KNN on frozen pre-trained representations task T,. More formally, we can define the metrics to evaluate the continually learned representations as follow:
Average accuracy is the average test accuracy of all the tasks completed until the continual learning of task A, = Σi= Or,
2. Average Forgetting is the average performance decrease of each task between its maximum accu-
racy and accuracy the completion of training: F T T-1 max (Ar,i ar,a)
UNSUPERVISED CONTINUAL LEARNING
4.1 CONTINUOUS REPRESENTATION LEARNING WITH SEQUENTIAL TASKS
To learn feature representations, contrastive learning (Chen et al., 2020a;b; He et al., 2020) maximizes the similarity of representations between the images of the same views (positive pairs) and minimizes the similarity between images of different views (negative pairs). However, these methods require large batches, negative sample pairs (Chen et al., 2020a;b), or architectural modifications (He et al., 2020; Chen et al., 2020c), or non-differentiable operators (Caron et al., 2020), which makes their application difficult for continual learning scenarios. In this work, we focus on SimSiam (Chen & He, 2021) and BarlowTwins (Zbontar et al., 2021), which tackle these limitations and achieve state-of-the-art performance on standard representation learning benchmarks.
SimSiam (Chen & He, 2021)
variant of Siamese networks (Bromley et al., 1994) for learning input data representations. It consists of an encoder network fe. which is composed of backbone network, and shared across a projection MLP and prediction MLP head h(-). Specifically, SimSiam minimizes the cosine-similarii between the output vectors of the projector and the predictor MLP across two different augmentations for an image. Initially, we consider FINETUNE, which a a naive CL baseline that minimizes the cosine-similarity between the projector output (2)_r fe (z! )) and the predictor output (pī, h(fe(z? on sequence of tasks follows:
CINETUNE CCL
2 D(p stopgradppaaa.... + 2 D(v stopgrad(zz)))
1
2
pl.r
z
where D(P 21,1
llp
ll
diy, and -²,- are two randomly augmented views of an input example 2i,r E Tr and ||-1|2 denotes the f2-norm. Note that, the stopgrad is crucial component in SimSiam to prevent the trivial solutions obtained by Siamese networks. Due to its simplicity and effectiveness, we chose Simsiam to analyze the performance of unsupervised representations for continual learning.
BarlowTwing (Zbontar et al., 2021) minimizes the redundancy between the embedding vector components of the distorted versions of an instance while conserving the maximum information inspired from Barlow (1961). In particular, the objective function eliminates the SimSiam s stopgrad component and instead makes the cross-correlation matrix computed between the outputs of two identical networks closer to the identity matrix. Let C be the cross-correlation matrix between the outputs of two Siamese branches along the batch dimension and Z₁ and Ze denote the batch embeddings of the distorted views for all images of batch from the current task (z, U. The objective function for UCL with finetuning and Barlow Twins can then be defined as:
ΣB 2ts + /E (Eb,
FINETUNE [ C1:) X ". CoJ², where Cij it
(3)
UCL
is a positive constant trading off the importance of the invariance and redundancy reduction terms of the loss, and j denote the network's 's output vector dimensions. Similar to SimSiam, Barlow Twins is simple, easy to implement, and can be applied to existing continual learning strategies with little or modification.
4

---

Published as a conference paper at ICLR 2022
4.2 PRESERVING REPRESENTATIONAL CONTINUITY: A VIEW OF EXISTING SCL METHODS
Learning feature representations from labelled instances on a sequence of tasks has been long studied in continual learning. However, the majority of these learning strategies are not directly applicable to UCL. To compare with the regularivation-based strategies, we extend Synaptic Intelli- gence (SI) (Zenke et al., 2017) UCL and consider the online per-synapse consolidation during the entire training trajectory of the unsupervised representations. For architectural- based strategies, we in- vestigate Progressive Neural Networks (PNN) (Rusu et al., 2016) and learn the feature representations progressively using the representations learning frameworks proposed in Section 4.1.
We also formulate Dark Experience Replay (DER) (Buzzega et al., 2020) for UCL. DER for SCL alleviates catastrophic forgetting by matching the network logits across sequence of tasks during the optimization trajectory. Notably, the loss for SCL-DER can be defined as follow:
DER SCL
SCLL
FINETUNE + a (x.))) [|softmax(p) - softmax(hy
(4)
where p = (x)
CIINETUNE denotes the cross-entropy loss on the current task (see Equation (1)) SCI.
and random examples are selected using reservoir sampling from the replay-buffer buffer M. Since, we do not have access to the labels for UCL, we cannot minimize the aforementioned objective.
Instead, we utilize the output of the projected output by the backbone network to preserve the knowledge of the past tasks over the entire training trajectory. In particular, DER for UCL consists of combination of two terms. The first term learns the representations using SimSiam from Equation (2) or Barlow Twins from Equation (3) and the second term minimizes the Euclidean distance between the projected outputs minimize catastrophic forgetting. More formally, UCL-DER minimizes the following loss:
UCL
PINETUNE + a- LG-EG)-M - UCL
(5)
However, the performance of the rehearsal-based methods is sensitive the choice of a and often requires supervised training setup, task identities, and boundaries. To tackle this issue, we propose Lifelong Unsupervised Mixup in the subsequent subsection, which interpolates between the current and past task instances to mitigate catastrophic forgetting effectively.
4.3 LIFELONG UNSUPERVISED MIXUP
The standard Mixup (Zhang et al., 2018) training constructs virtual training examples based on the principle of Vicinal Risk Minimization. In particular, let (ara, ys) and (xz, U2) denote two random feature-target pairs sampled from the training data distribution and let (ā, D) denote the interpolated feature-target pair in the vicinity of these examples; mixup then minimizes the following objective:
MIXUP (ž, 1) CE (hy (Je (T)). .7)
where ã = X - I, + (1 A)-A) Iz and 5 = A y. + (1 Vj-
~ Beta(a, a), for a E (0, In this work, we focus on lifelong self-supervised learning and propose Lifelong Unsupervised Mixup (LUMP) that utilizes mixup for UCL by incorporating the instances stored in the replay-buffer from the previous tasks into the vicinal distribution. In particular, LUMP interpolates between the examples of the current task (Ixrr) E U, and random examples selected using uniform sampling from the replay buffer, which encourages the model to behave linearly across a sequence of tasks. More formally, LUMP minimizes the objective in Equation (2) and Equation (3) on the following interpolated instances ãi,r for the current task T:
ft, A Ii.r + (1 -1-33 X) IJ.M,
(7)
where Zj.M M denotes the example selected using uniform sampling from replay buffer M. The interpolated examples not only augments the past tasks' instances in the replay buffer but also approximates a regularized loss minimization (Zhang et al., 2021). During UCL, LUMP enhances the robustness of learned representation by revisiting the attributes of the past task that are similar to the current task. Recently, Kim et al. (2020); Lee et al. (2021); Verma et al. (2021); Shen et al. (2022) also employed mixup for contrastive learning. Our work is different from these existing works in that our objective is different, and we focus unsupervised continual learning. To this end, LUMP successively mitigates catastrophic forgetting and learns discriminative & human-perceptuull features over the current state-of-the-art SCL strategies (see Table 1 and Figure 4).
5

---

Published as a conference paper at ICLR 2022
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Baselines. We compare with multiple supervised and unsupervised continual learning baselines across different categories of continual learning methods.
Supervised continual learning. FINETUNE is a vanilla supervised learning method trained on a sequence of tasks without regularization or episodic memory and MULTITASK optimizes the model on complete data. For regularization-basse CL methods, we compare against SI (Zenke et al., 2017) and AGEM (Chaudhry et al., 2019a). We include PNN (Rusu et al., 2016) for architecture-basee methods. Lastly, we consider GSS (Aljundi et al., 2019) that populates the replay-buffer using solid-angle minimization and DER (Buzzega et al., 2020) matches the network logits sampled through the optimization trajectory for rehearsal during continual learning.
2. Unsupervised continual learning. We consider the unsupervised variants of various SCL base- lines to show the utility of the unsupervised renprisseeepppeeeeeeeeeee representations for sequential learning. Specifically, we use SIMSIAM (Chen & He, 2021) and BARLOW (Zbontar al., 2021), which are the ofatc-of-the-arr representational learning techniques for learning the unsupervised continual representations. We compare with FINETUNE and MULTITASK following the supervised learning baselines, and SI (Zenke et al., 2017), PNN (Rusu et al., 2016) for unsupervised regularization and architecture CL methods respectively. For rehearsal-baser method, we compare with the UCL variant of DER (Buzzega et al., 2020) described in Section 4.2
Datasets. We compare the performance of SCL and UCL on various continual learning benchmarks using single-head ResNet-18 (He et al., 2016) architecture. Split CIFAR-10 (Krizhevsky, 2012) consists of two random classes out of the ten classes for each task. Split CIFAR-100 (Krizhevsky, 2012) consists of five random classes out of the 100 classes for each task. Split Tmay-ImageNeet is a variant of the ImageNet dataset (Deng et al., 2009) containing five random classes out of the 100 classes for each task with the images sized 64 x 64 pixels.
Training and evaluation setup. We follow the hyperparameter setup of Buzzega et al. (2020) for all the SCL strategies and tune them for the UCL representation learning strategies. All the learned representations are evaluated with KNN classifier (Wu et al., 2018) across three independent runs. Further, we use the papeepparaeeters obtained by SimSiam for training UCL strategies with Barlow Twins to analyze the sensitivity of UCL to Syper-parameters and for fair comparison between different methods. We train all the UCL methods for 200 epochs and evaluate with the KNN classifier (Wu et al., 2018). We provide the hyper- syper-parametter in detail in Table A.5.
5.2 QUANTITATIVE RESULTS
Evaluation on SimSiam. Table shows the evaluation results for supervised and unsupervised rep- resentations learnt by SimSiam (Chen & He, 2021) across various continual learning strategies. In all cases, continual learning with unsupervised representations achieves significantly better performance than supervised representations with substantially lower forgetting. For instance, SI with UCL obtains better performance and 68%, 54%, and 44% lower forgetting relative to the best-perfont pest-performig SCL strategy on Split CIFAR-10. Split CIFAR-100, and Split IingeImm,,Net, respectively. Surprisingly, FINETUNE with UCL achieves higher performance and significantly lower forgetting in comparison all SCL strategies except DER. Furthermore, LUMP improves upon the UCL strategies: 2.8% and 5.9% relative increase in accuracy and 15% and 57.1% relative decrease in forgetting on Split CIFAR- 1-0 and Split ImageNe,,,,, respectively.
Evaluation on BarlowT SwilowTTinn. To verify that unsupervised representations are indeed more robust to catastrophic forgetting, we train BarlowTwins (Zbontar et al., 2021) on a sequence of tasks. We notice that the representations learned with Barlow Twins substantially improve the accuracy and forgetting over SCL: 71.4%, 39.7% and 73.2% decrease in forgetting with FINETUNE on Split CIFAR- 10, Split CIFAR- 100 and Split Timy-Ima respectively. Similarly, we observe that SI, and DER are more robust to catastrophic forgetting; however, PNN underperforms on complicated tasks since feature accumulation using adaptor modules is insufficient to construct useful representations for current task adaptation. Interestingly, representations learnt with Barlow Twins achieve lower forgetting for FINETUNE, DER and LUMP than SimSiam with comparable accuracy across all the datasets.

---

Published as a conference paper at ICLR 2022
Table Accuracy and forgetting the learnt representations on Split CIFAR-IO, Split CIFAR- 100 and Split Tiny- ImageNet on Resnet-18 18 architecture with KNN classifier (Wu et al., 2018). All the values are measured by computing mean and standard deviation across three trials. The best and second-best results are highlighted in bold and underline respectively.
METHOD
SPLIT CIFAR-I0
SPLIT CIFAR-100
SPLIT TINY- MON-IMAGENET
ACCURACY FORGETTING ACCURACY FORGETTING ACCURACY FORGETTING
SUPERVISED CONTINUAL LEARNING
FINETUNE
82.87 0.47)
14.26 0.5))
61.08 0:44)
31.23
(= 0.41)
53.10 (= 1.57)
33.15 (# 1.22)
PNN (Rusu et al. 2016)
82.74 (11)
66.05 (= 0.86)
64.38 (.9)
Si (Zenke et al,, 2017)
85.18 (66)
11.39 (77)
63,58 0.30 27.98 0.34
44,96 (44
26.29 1.40)
A-GEM (Chandlry et al., 2019a)
82.41 1.24)
13.82 1.27)
59.81 1.77)
30.08
60.45 0224 24.94 1.24)
GSS (Aljundi et al., 2019)
89.49 t= 1.75)
7.50 1.52)
70.78
(= 1.67)
21.28
(= 1.53)
70.96 0.73)
14.76 (# 1.22)
DER (Buzzega et al., 2020)
91.35 (= 0.46)
5.65 (= 0.35)
79.52 (= 1.88)
12.80 (= 1.47)
68.03 (# 0.85)
17.74 (# 0.65)
MULTITASK
97.77 (:15)
93.89 (= 0.78)
91.79 (#: 0.46)
UNSUPERVISED CONTINUAL LEARNING
FINETUNE
90.11 (.12)
5.42 0.08)
75.42 (1 0.78)
10.19 (1 0.37)
71.07 0.200
9.48 0.56)
SIn JAMM
PNN (Rusu et al, 2016)
90.93 0.20)
66.58
1.00)
62.15 (# 1.35)
Si (Zenke et al, 2017)
92.75 (2: 0.06)
1.81 (= 0.11
80.08
1.30)
5,54 (= 1.30)
72.34
(2:0.4)
8.26 (1: 0.4)
DER (Buzzega et al., 2020)
91.22 L1 0.30)
4.63 (1L 0.26)
77.27
(1 0.30)
9.31
(L 0.09)
71.90
(1 1.44)
8,36 (1. 2.06)
LUMP
91.00 (= 0.40)
2.92 (= 0.53)
82.30 (= L.35)
4.71 (= 1.52)
76.66
(:: 2.39)
3.54 (t: 1.04)
MULTITASK
95.76 (12 0.08)
86.31 (22 0.38)
82.89 0.99)
BARLOW NOWTWIN TWNNS
FINETUNE
87.72 (=: 0.82)
4.08 (= 0.56
71.97
(= 0.54)
9.45 (= 1.01)
66.28 (#: 1,21)
8.89 (+: 0.66)
PNN (Rusu et al., 2016)
87.52 [s 0.33)
57.93
(m: 2.95)
48.70 (=: 2.59)
S1 (Zenke et al., 2017)
[m 0.06) 88.67 L1 0.24)
90.21
2.03
(= 0.22)
75.04
(= 0.63)
7.43 (= 0.67)
56.96 (#: 1.48)
17.04 (=) 0.89)
DER (Buzzega et al., 2020)
2.41
(1 0.26)
73.48
(1) 0.53)
7.98 (1) 0.28)
68.56
(1 1.47)
7.87
L1 0.54)
LUMP
90.31 t 0.30)
1.13 0.18)
80.24 1.04)
3.53 0.83)
72.17
0.89)
2.43 1.00)
MULTITASK
95.48 0.14)
87.16 (L) 0,52)
82.42 (1.0
averuge accuracy (%6)
100
accuracy over data size
forgetting over data size HQ-FT
CKA simiterity
1.0
CKA aimilarity
1.0
80
averagg for werrggetingg
40
CKA Sallaaan
LUMe
055
0.5
60
40
0.0
0 3
6.0
0.0
0
Layers
Layers
Layers
2910
200
5.00
500
FINITUNE
(1)
dataset per task
dataset
Figure 3: CKA Feature similarity between two in- dependent UCL models (red), two independent SCL models (blue), and UCL and SCL model (green) for different strategies on Split CIFAR-100 test distribution.
Figure 2: Evaluation on Few-shot training for Split CIFAR- 100 across different number of training instances per task. The results are measured across three independent trials.
Evaluation on Few-shot training. Figure 2 compares the effect of few-shot training on UCL and SCL, where each task has limited number of training instances. Specifically, we conduct the experimental evaluation using 100, 200, 500, and 2500 training instances for each task in split CIFAR- 100 dataset. Surprisingly, we observe that the gap in average accuracy between SCL and UCL methods widens with decrease in the number of training instances. Note that UCL decreases the accuracy by 15. 78% on average with lower forgetting when the number of training instances decreases from 2500 to 100; whereas, SCL. obtains a severe 32.21 %pp deterioration in accuracy. We conjecture that this is an outcome of the discriminative feature embeddings learned by UCL, which discriminates all the images in the dataset and captures more than class-specific information as also observed in Doersch et al. (2020). Furthermore, LUMP improves the performance over all the baselines with significant margin across all few-shot experiments.
Evaluation on OOD datasets. We evaluate the learnt representations on various -istrriiitt (OOD) datasets in Table 2 to measure their generalization to unseen data distributions. In particular, we conduct the OOD evaluation on MNIST (LeCun, 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017). SVHN (Netzer et al., 201 1), CIFAR- 10 and CIFAR- ( (Krizhevsky, 2012) using KNN classifier (Wu et al., 2018). We observe that unsupervised representations outperform the supervised representations in all cases across all the datasets. In particular, the UCL representations learned with Simsiam, and SI on Split- Split-CIFAR- 10 improves the absolute performance over the best pest-performing SCL strategy by 4.58%. 6.09%, 15.26%, and 17.07% on MNIST, FMNIST, SVHN and CIFAR-100 respectively. Further, LUMP trained on Split- Split-CIFAR-LOO 100 outperforms SI across all datasets and obtains comparable performance with Split CIFAR-10 dataset.
7

---

Published as a conference paper at ICLR 2022
Table 2: Comparison accurrcy on of distribution datasets using a KNN classifier (Wu et al., 2018) on pre- trained SCL and UCL representations. We consider MNIST (LeCun, 1998), Fashion MNNIST (FMNIST) (Xiao et al., 2017). SVHN (Netzer et al., 2011) as out of distribution for Split CIFAR- 100 and Split CIFAR-IO. All the values are measured by computing mean and standard deviation across three trials. The hest and second-best results highlighted in bold and underline respectively.
IN-CLASS
SPLIT CIFAR-10
SPLIT CIFAR-100
OUT-OF-CLASS
MNIST
FMNIST
SVHN
CIFAR-100
MNIST
FMNIST
SVHN
CIFAR-10
SUPERVISED CONTINUAL LEARNING
FINETUNE
86.42 1 111
74.47 (054
41,00
17,42 de
75.02
(2
38.05 (2:073)
39. 18 688
St (Zonke 2017)
82.08
42.62
19.14
63.71
40.92
40.41
A-GEM (Chaudhry al., 2019g)
86.07
t= 194
37.77
1:0
16.11
039
320
64.16
(=22)
37.48 L:3)
37.91 1359
GsS (Aljundi et al, 2019)
70.36
c2 354
69.20 2513
33.11
(20
18.21 cm:
76.54
(2040
65.31 (z 1m3
ct
49.41 ct 180
DER (Buzzegs al. 2020)
80:32
70.49
41.48
17.72(1
87.21
(1739)
75,97
59.07
MULTITASK
88.79 tz L.h 79.50 (z03))
41.26 ct 108) 27.68 (t 069 92.29 (#:30))
86.12 180
54.94 ce cm))
54.04 ct (9:3080
UNSUPPEVVSSED CONTINUAL LEARNING
FINETONE
[. 0:9
(_. (31
49.66 580
(1 61hh 0.00
c_.90 am)
50.09 (1 14th
57.15 DM6
SIMSIAM
St (Zenke et all..221
93.72 (::030
82,50 (8))81
$7.88 (=
36.21
(= sol
91.50 10 c 120
80.57 om)
54.07 (#:27))
60.55 (a a50
DER (Buzzega al., 2020)
88:35
79.33
48.83
(355)
30.68
87.96
76.21
47.70
56,26
80.78
150
31.17
183
um
(+:14)
50.13 (+:37))
63.00
0.31
MULTITASK
90.69 o.))
(0.65
47.67 (1 2.1))
39.55
c1 (1:))
90.35 (1:30)
81.11 c_. 1.0
52.20 (1
70.19 (1.
BoRLOW WINS
FINETUNE
86.86 142) 78.37
44.64
130
28.03
(1 053)
76.08
76,82 (2.22cmm
42.95
0M
53.12 (1.19
St (Zenke et al, 2017)
90:31 12 06:
80.58
664)
49.18
(2:00
85.24
78.82
067)
45.18
53.99
DER (Buzzega et al., 2020)
85 2:5)
77.96
(2 0.53
45.68
et 5.9
27.83
(t 0:99
78.08
et 1955
76.67
0.8
44.58
L00)
53.24
ct 6222
LUMP
88.73
81.69
51.53
31,53
90.22
12:0
81,28
(201))
50.24
(2:09)
60.76
MULTITASK
c: 1:31
79.49
c1 3.49
36.33
co 0.29
79.40 1.00)
c1:09)
49.50 c2 0300
5.3 QUALITATIVE ANALYSIS
Similarity in feature and parameter space. We analyze the similarity between the representations learnt between (i) Two independent UCL models, (ii) Two independent SCL models (iii) SCL and UCL models using centered kernel alignment (CKA) (Kornblith al., 2019) in Figure 3, which provides a score between O and I measuring the similarity between a pair of hidden representations. For two
representations 0 X'' - TRd and O₂ + Rd, CKA(O), 22) = TCor( (T)ll [[aw[ where covariances are with respect to the test distribution. Additionally, we measure the & dis- tance (Neyshabur al., 2020) between the parameters of two independent UCL models (see Table 3) and two independent SCL models (see Table 4). First, we observe that the representations learned by two independent UCL methods have high feature similarity and lower &g distance compared to the two independent SCL. methods, demonstrating UCL. representations' robustness. Second, we note that the representations between any two independent models are highly similar in the lower layers indicating that they learn similar high-level features, including edges and shapes; however, the features are dissimilar for the higher modules. Lastly, we see that the representations between a UCL and SCL model are similar in the lower layers but diverge in the higher layers across all CL strategies.
[|Cov( (x)) 1
Visualization of feature space. Next, we visualize the learned features to dissect further the repre- sentations learned by UCL and SCL strategies. Figure 4 shows the visualization of the latent feature maps for tasks To and Ta3 after the completion of continual learning. For To. we observe that the SCL methods are prone catastrophic forgetting, the features appear noisy and do not have coherent patterns. In contrast, the features learned by UCL strategies are perceptually relevant and robust to catastrophic forgetting, with LUMP learning the most distinctive features. Similar to To. we observe that the UCL features are more relevant and distinguishable than SCL for Tia. Note that we randomly selected the examples and feature maps for all visualizations.
Loss landscape visualization. To gain further insights, we visualize the loss landscape of task To after the completion of training on task To and Tı9 for various UCL and SCL strategies in Figure 5. We measure the cross-entropy loss for all methods with randomly initialized linear classifier for a fair evaluation of two different directions. We use the visualization tool from Li et al. (2018) that searches the task loss surface by repeatedly adding random perturbations to model weights. We observe that the loss landscape after To looks quite similar across all the strategies since the forgetting does not exist yet. However, after training Tis. there is a clear difference with the UCL strategies obtaining a flatter and smoother loss landscape because UCL methods are more stable and robust to the forgetting, which hurts the loss landscapes of past tasks for SCL. It is important to observe that LUMP obtains a smoother landscape than other UCL strategies, demonstrating its effectiveness. We defer further analyses for feature and loss landscape visualization to Appendix A.2.
8

---

Published as a conference paper at ICLR 2022
Table 3: & distance between UCL parameters after Table 4: l distance between SCL paraneters after completion training.
completion of training.
MODEL
FINETUNE
S1
DER
MULTITASK
MulirTask
PINETUNE
MAD 75.51 iz won
DER
c5
48.76
(2.9%
219.00 17-
MULTITASK
02
79.95 tua
$7,90 c2 61.22 ce-0
MULTITASK
c2
271.30 wel
271.88 BAN 314 (2-0
SCL-FINETUNE
SCL-St Aoc: 58.9 9 0.2
SCL-GSS Aec: 78-4 1.8
SCL-DER Aa:: 73.1 0.4
SCL-FINETUNE
UCL-SI Ari 76.4 1.8
LUMP (OURS)
APPLE (76)
Aac: 50.7 dt 0.2
Aor: 70.8 at 0.4
Acc: 76.0 d: 2.7
SCI--FINETUNE
SCI-51
SCI-GSS Acc: 59.9 + 2.2
SCI-DER
UCI-FINETUNE
UcL-SS
LUMP (OURS) Acc: 80.8 +
RACCOON (Tis)
Aa:: 50.6 = 1.4
Acc: 48.4 1.0
Au: 76.4 = 2.2
Ac:: 74.6 0.5
As: 78.0 = 1.6
Figure 4: Visualization of feature maps for the second block representations leamt by SCL and UCL strategies (with Simsiam) for ResNet-18 architecture after the completion of CL. for Split CIFAR-100 dataset (n 20).
T19
SCL-FINETUNE
SCL-St
SCL-DER
OCL-FINETUNS
UCL-St
LUMP (OURS)
Acc:54.73 0.25
Acc: 58.59 + 0.20 Ace: 73.13
0.38
Ace: 70.80 0.40
Acc: 76.39 + 1.56
Acc:76.60 2.70
Figure 5: Loss landscape visualization of To after the completion of training task To (top) and Tis (bottom) for Split CIFAR-100 dataset on ResNet-18 architecture. We Simsiam UCL methods.
6 DISCUSSION AND CONCLUSION
This work attempts to bridge the gap between unsupervised representation learning and continual learning. In particular, we establish the following findings for unsupervised continual learning.
Surpassing supervised continual learning. Our empirical evaluation across various CL strategies and datasets shows that UCL representations representations. Furthermore, we notice that UCL generalizes better to OOD tasks and achieves stronger performance on few-shot learning tasks. We propose Lifelong unsupervised mixup (LUMP), which interpolates the unsupervised instances between the current task and past task and obtains higher performance with lower catastrophic forgetting across wide range of tasks.
robust catastrophic forgetting than SCL
Dissecting the learned representations. We conduct a systematic analysis to understand the differ- ences between the representations learned by UCL and SCL strategies. By investigating the similarity between the representations. we observe that UCL and SCL. strategies have high similarities in the lower layers but are dissimilar the higher layers. We also show that UCL representations learn coherent and discriminative patterns and smoother loss landscape than SCL.
Limitations and future work. this work, we do not consider the high-resolution tasks for CL. We intend to evaluate the forgetting of the learnt representations on ImageNet (Deng et al., 2009) in future work, since UCL shows lower catastrophic forgetting and representation learning has made significant progress ImageNet over the past years. In follow-up work, intend to conduct further analysis to understand the behavior of UCL and develop sophisticated methods continually learn unsupervised representations under various setups, such class- classiincrementll task-agnostic CI..

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by Microsoft Research Asia, the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF- 2018RI ARRAASAIOO 1059921), Institute of Information & communications Technology Planning & Evalua- tion (ITTP) grant funded by the Korea government (MSIT) NN..2019-0-000 Graduate School Program (KAIST) and 2021 1-0-01696).. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
Artificial Intelligence
AUTHOR CONTRIBUTIONS
Divyam Madaan conceived of the presented idea, developed the experimental framework, carried out OOD evaluation, CKA visualization and took the lead in writing the manuscript. Jaehong Yoon performed the hyperparameter search, carried out the visualization of loss landscape and feature maps and performed the few-shot training analysis. Yuanchun Li, Yunxin Liu, and Sung Ju Hwang supervised the project.
REFERENCES
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-ban continual learning with adaptive regularization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Horace Barlow. Possible principles underlying the transformations of sensory messages. 1961.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner. Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller. Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.00733 2016.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. 1994.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2019a.
Arslan Chaudhry, Marcus Rohrbach. Mohamed Elhoseiny. Thalaiyasingam Ajanthan, Punect K. Dokania, Philip HS Torr, and Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv: 10486, 2019b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML), 2020a.
Ting Chen, Simon Korbblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
10

---

Published as a conference paper at ICLR 2022
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 042977, 2020c.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE International Conference on Computer Vision and Putlern Recognition (CVPR), 2009.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Timo Flesch, Jan Balaguer, Ronald Dekker, Hamed Nili, and Christopher Summerfield. Comparing continual task learning in minds and machines. Proceedings of the National Academy of Sciences, 2018.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation catastrophic forgetting gradient-baset neural networks. arXiv preprint arXiv: 1312.6211, 2013.
Cean-Basticn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach suler-spe learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE International Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Christopher J Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. Key challenges for delivering clinical impact with artificial intelligence. BMC medicine, 2019.
Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint urXiv:2010.0630 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the International Conference Machine Learning (ICML). PMLR, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multi-task learning, In Proceedings of the International Conference on Machine Learning (ICML), 2012.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. con/exdb/mmist// 1998.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A dogain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), 2018.

---

Published as a conference paper at ICLR 2022
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. Proceedings of the International Conference Machine Learning (ICML), 2019.
Yuanpeng Li, Liang Zhao, Kenneth Church, and Mohamed Elhoseiny. Compositional language continual learning. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. 1989.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, 2011.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is heing transferred in transfer learning? In Advances Neural Information Processing Systems (NeurIPS), 2020.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell, Continual unsupervised representation learning. In Advances in Neural Information Processing Systems, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 04671, 2016.
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki, Agnieszka Strbska-Barwinskee Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning. In Proceedings of the International Conference Machine Learning (ICML), 2018
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing, Un-mix: Rethinking image mixtures for unsupervised visual representation learning. Proceedings of the AAAI National Conference Artificial Intelligence (AAAI), 2022.
James Smith, Cameron Taylor, Seth Baer, and Constantine Dovrolis. Unsupervised progressive learning and the stam architecture. In Proceedings of the International Joint Conference Artificial Intelligence (ICCAI), 2021.
Sebastian Thrun A Lifelong Learning Perspective for Mobile Robot Control. Elsevier, 1995.
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domain- domain-agnostic contrastive learning. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Zhirong Wu. Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non parammemtttic instance discrimination. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2018
12

---

Published as a conference paper at ICLR 2022
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07744 2017.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In Proceedings of the International Conference Learning Representations (ICLR), 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: learning via redundancy reduction. Proceedings of the International Conference on Machine Learning (ICML), 2021.
Self-supervised
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Pas Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
13

---

Published as a conference paper at ICLR 2022
A SUPPLEMENTARY MATERIAL
Organization. In the supplementary material, we provide the implementation details followed by the hyper-parameeer configurations in Appendix A.I. Further, we show the other experiments we conducted and additional visualizations and results in Appendix A.2.
A.1 EXPERIMENTAL DETAILS
Implementations. We use the DER (Buzzega et al., 2020) open-source codebase' for all the experiments. In particular, reproduce all their experimental results for supervised continual learning and use various models with their set of original representations for SimSiam' and Barlow for unsupervised continual learning. We verify our implementation by reproducing the reported results on CIFAR- 10 in the original paper, where train the representations the complete CIFAR-10 dataset and evaluate the test-set using KNN classifier (Wu et al., 2018). In particular, (Wu et al., 2018) stores the features for each instance in the task-level training set discrete memory bank. The optimal feature-Ievel embeddings are then learned by instance-level discrimination, which maximally scatters the features of the training samples. Following prior works in representation learning, we use the task-level training set without any augmentation in the inceemnnceettall setup for the supervised and unsupervised KNN evaluation.
as our baselines. We follow the
Hyperparameter configurations. We use the tuned pappee-assmeters reported by Buzzega al. (2020) for all the SCL experiments. On the other hand, we tune the hyper-parameters for continual learning strategies for UCL. We provide the hyper-parameters setup for UCL. for different datasets in Table A.5. We train all the UCL. methods with a batch size of 256 for 200 epochs, while training the SCL methods with batch size 32 for 50 epochs following Buzzega et al. (2020). We observed that training the SCL methods further lead to degredation performance for all the methods. We use the same set augmentations for both SCL and UCL except that we use RandomRes i zedCrop op with scale in (0.2, 1.0] for UCL (Wu et al., 2018; Chen & He, 2021) and RandomCrop for SCL. For rehearsal- methods, we use the buffer size 200 for Split CIFAR-1O, 10, Split CIFAR-100 and 256 for Split Niny-ImageNet dataset. We a learning rate of 0.03 for SGD optimizer with weight decay 5e-4 and momentum 0.9.
Table A.5: Hyperparameter configurations for all the datasets on ResNet-18 architecture.
METHOD SPLIT CIFAR-10
SPLIT CIFAR-100
SEQ. TINY-IMAGENE
SI
c: 100 ::
0:0.1 E:
0.01 g:1
PNN
wd : 64
wd: 12
wd:8 8
DER
a 0.1
a : 0.1
a : 0.01
LUMP
0.1
A: 0.1
À 0.4
A.2 ADDITIONAL EXPERIMENTS
We provide additional loss landscape on Split CIFAR-100 in Figure A.6 and Figure A.7, Figure A.8 show the second and third block feature visualizations on Split CIFAR- respectively. Figure A.9 shows the feature visualizations for Split Ting-emaeeNet on ResNet- 18 architecture
14

---

Published as a conference paper at ICLR 2022
To
T17
Tis
Ti
SCI-FINETUNE Acc: 51..3 + 0.25
Acc: 58.59 1: 0.20
SCL-DER Acc: 73.13 +: 0.38
13
UCL-FINCTUNE Acc: 70.80 0.40
M0
UCL-SI Acc: 76.39 1: 1.59
LUMP (OLRS) Acc: 76.60 1: 2.70
Figure A.6: Loss landscape visualization of To after the completion of training on task To. Ti7, Tis. and Tig for Split CIFAR- 100 dataset on ResNet-1e 18 architecture. We use Simsiam for UCL methods.
15

---

Published as a conference paper at ICLR 2022
RAW IMAGES To, Tr, Ta, Tia
SCI-FINETUNE Acc:61.08 + 0.04
SCL-Si
Acc:63.58 0.37
SCL-GSS Acc:70.78 ::67 1.67
SCL-DER Acc:79.52 = 1.88
UCL-FINETUNE
Acc:75.42 0.78
UCL-S1 Acc:80.08 1.30
LUMP (OURS) Acc:82.30 + 1.35
Figure A.7: Visualization of feature maps for the second block representations leart by SCL and UCL strategies (with Simsiam) for Resnet-18 18 architecture after the completion of continual learning for Split CIFAR-100 dataset (n 20). The accuracy is the mean across three runs for the corresponding task.
16

---

Published as a conference paper at ICLR 2022
RAW IMAGES To: Ti. T2. Tia
SCL-FINETUNE Acc:61.08 .08 + 0.04
SCL--1 Acc:63.58 + 0.37
SCL-Gss Acc:70.78 + 1.67
SCL-DER Acc:79.52 + 1.88
CII--FINTUUNE Acc:75.42 = 0.78
UCL-S1 Acc:80.08 1.30
LUMP (OURS) Acc:82.30 + 1.35
Figure A.8: Visualization of feature maps the third block representations learnt by SCL and UCL strategies (with Simsiam) for Resnet- 18 architecture after the completion of continual learning for Split CIFAR- dataset (71 20). The accuracy is the mean across three runs for the corresponding task.
17

---

Published as a conference paper at ICLR 2022
Raw IMAGES To, Tr, Ta, Tio
SCI-FINETUNE Acc:53.10 + 1.37
SCL-Si
Acc:44.96 2.41
SCI--GSS Acc:70.96
SUL-DER Acc:68.03 = 0.85
UCL-FINETUNE Acc:71.07 0.20
UCL-SI Acc:72.31 0.12
LUMP (OURS) Acc:76.66 + 2.39
Figure A.9: Visualization of feature maps for the second block representations leart by SCL and UCL strategies (with Simsiam) for Resnet-18 18 architecture after the completion of continual learning for Split Tiny-ImageNet dataset (n 20). The accuracy is the mean across three runs for the corresponding task.
18


Reviews:
[{"review_id": "v2b-HX38Jk", "reviewer": "Reviewer_u1jR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n* Rethinking continual learning with unsupervised representation learning is interesting, and empirical results indicate that most supervised continual learning methods can be improved by the proposed approach.\n\n* A bunch of experiments have been conducted to demonstrate the effectiveness of the proposed approach in various settings. And several visualizations have also been included for a better understanding of the learned features.\n\n### Weaknesses\n\n* **[Missed Comparison with CURL]** \\\nThough the authors criticised that *Continual Unsupervised Representation Learning framework (CURL)* to be limited by digit-based gray-scale datasets, no direct comparison with CURL is done by following their evaluation protocol. Adding this result could better reveal the difference between the proposed method and CURL in terms of effectiveness. \\\nBeside, the evaluation of cluster quality used in CURL seems to be an important evaluation metric in unsupervised continual learning, which has not been used in the paper.\n\n* **[Degraded Performance of DER and MULTITASK]** \\ \nIn the Table 1, we see that the proposed unsupervised continual learning can improve all baseline methods except DER and MULTITASK. A clear explanation about this performance drop should be added.\n\n* **[Qualitative Analysis]** \\ \nWhy the visualization of feature maps stops at $\\mathcal{T_{13}}$, while the loss landscape visualization continues to $\\mathcal{T_{19}}$ ? And in Figure 5, the difference between SCL-DER and LUMP is hard to interpret.\n\n* **[Why Not Directly Use Unsupervised Learned Presentations]** \\\nAs an important purpose of the unsupervised representation learning is to learn a powerful embedding space that can be quickly fine-tuned for latter down-stream tasks. Why don't we consider a baseline where the feature backbone is initialized with SimSiam or Barlow Twins, and directly fine-tune them on a sequence of tasks. This is probably not considered in the standard continual learning, but the results of this baseline could be informative to the community of both domains.", "summary_of_the_paper": "This paper attempts to bridge the gap between **unsupervised representation learning** and **continual learning** by extending various supervised continual learning methods to the unsupervised learning framework.\nIt builds upon two recent unsupervised feature learning techniques (*SimSiam* and *BarlowTwins*) and a powerful data augmentation technique *MixUp*.\nImproved performances have been demonstrated in various experimental settings, together with comprehensive feature visualizations.", "summary_of_the_review": "This paper attempts to rethink the standard continual learning in a new point of view by considering unsupervised representation learning methods. This purpose is interesting, but the major difference between the SCL (supervised continual learning) and UCL (unsupervised continual learning) seems to be just adding a unsupervised representation loss at the backbone and freezing it in the second stage of predicting head finetuning.\nMoreover, some parts of the empirical results are not clearly presented or explained, an improved version of experimental results could be helpful for better validating the contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j9EewvLS7cE", "reviewer": "Reviewer_tbme", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points\n\n* The paper takes one of the most import issues in CL: learning robust representation in unsupervised setting. For me, the problem itself is real and practical.\n\n* The paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of UCL and the proposed  LUMP algorithm over SCL methods.\n\n* Overall, the paper is well written. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty, the method has been well motivated by pointing out the limitations in SOTA methods.\n\n* The authors provide code for reproducing the results in the paper.\n\n\nWeak Points\n\n* The proposed LUMP algorithm is adopted from supervised mixup technique(Zhang et al, 2018). So the novelty is limited.\n\n* The authors conducted extensive experiments in task-incremental setting. It would be interesting to see how UCL and the proposed method perform in class-incremental and task-agnostic CL settings.\n\n* Fig 3: In general, higher layers have lower feature similarity than lower layers, and similarity between UCL models are higher than that of SCLs. However, there is an exception in Layer 4 of DER method -- the similarity of SCL is higher than that of UCL. It is worth some discussion on this exception.\n\n", "summary_of_the_paper": "This paper studies the problem of representation learning in an unsupervised continual learning(UCL) setting. It shows that the representation learned with UCL is more general than the one learned with supervised CL (SCL), and investigates why UCL is more robust to catastrophic forgetting than SCL by analyzing the similarity of learned features and visualizing loss landscape. The authors also propose to apply mixup technique to UCL setting and present a LUMP algorithm to further improve the performance of CL.This paper studies the problem of representation learning in an unsupervised continual learning(UCL) setting. It shows that the representation learned with UCL is more general than the one learned with supervised CL (SCL), and investigates why UCL is more robust to catastrophic forgetting than SCL by analyzing the similarity of learned features and visualizing loss landscape. The authors also propose to apply mixup technique to UCL setting and present a LUMP algorithm to further improve the performance of CL.", "summary_of_the_review": "Overall, I vote for marginally accepting. I like the idea of unsupervised continual learning and handling it by the proposed LUMP method. My major concern is about the limited novelty of the proposed method -- adopted from mixup in supervised learning, and some additional experiments on class-incremental or task-agnostic settings(see weakness above). Hopefully the authors can address my concern in the rebuttal period.\n\n[After rebuttal]\nThe authors addressed most of my concerns, so I would like to raise my score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "h4hc_3ByBV1", "reviewer": "Reviewer_QCJy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- First the paper is well written and easy to read. \n- The experiments are rich and well-chosen to better understand the superiority of unsupervised representations in the context of CL. I especially appreciated the experiments in Fig 2 that investigate the impact of the size of the training dataset.\n- The conclusions are enlightening and will be very helpful to design new supervised or unsupervised CL methods.\n- The code is publicly available and looks clean and  easy to use.\n\nWeaknesses:\n- Some details are unclear: \n   - Sec 5.1: Knn classifier: which set is used for NN? the replay buffer? validation set? \n   - The Knn is used both for supervised and unsupervised experiments, right?\n   - Sec 5.3: more explanations about CKA are required for the reader that is not familiar with this measure\n- I recommend changing the title. \"Rethinking the Representational Continuity\" is much too strong. The conclusions of the paper are great but it does not provoke a real rethinking of the problem.\n- I am not really convinced but the visualization in Fig4. It seems that LUMP has sparser activations. The shapes of the objects are more clearly visible in its feature map. Does it simply mean that it learns lower-level features (similar to edge detector)? Maybe a TSNE visualization would help to see how the features of old tasks are affected when learning a new task.\n- In Fig.5, we can notice that the range of value gets smaller in T19 (from [4.6,5.6] for T0 to [4.4,4.6]). Any idea why?\n\n\n", "summary_of_the_paper": "The paper proposes to tackle the continual learning problem in an unsupervised setting. It shows that recent self-supervised learning methods are efficient tools to learn image representation with lower catastrophic learning problems. Two recent self-supervised methods are evaluated: SimSiam and BarlowTwins. In both cases, the superiority of unsupervised features is demonstrated. \n\nThe widely used mixup method is also adapted to the UCL problem. Straightforwardly, current images are mixed with images of the past tasks sampled from the replay buffer.", "summary_of_the_review": "The paper shows interesting results that confirm the potential of self-supervised learning methods in the context of continual learning. The technical novelty may look incremental but the experimental conclusions are very interesting for the community. The paper is clear and well written.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I8Se-LsJYbC", "reviewer": "Reviewer_synL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* The submission is well written and easy to follow. The proposed concept is well motivated with various quantitative and qualitative justifications. \n* While many unsupervised/self-supervised training approaches require pre-training on massive unlabeled data, the proposed method here works well with the help from Mixup and does not require additional pre-training set, largely making it more applicable to real-world use cases. \n* I especially enjoy reading Sec 5.3 regarding the analyses on feature similarity between different learning approaches, visualization of feature space, and loss landscape visualization. This section provides additional justification besides the absolute accuracy improvement over the supervised continual learning counterparts.\n\nWeaknesses\n* I think the main limitation of using the proposed pipeline in practice is runtime and memory constraints. To run K-NN in a lifelong learning setting is challenging due to ever-growing storage requirements for storing samples coming from different stages. During inference, the algorithm also needs to compute distances between the query and many stored data points. Can the authors shed some light about the runtime and memory comparison between the proposed method and supervised counterparts?\n* It would be great to also compare with other recent supervised continual learning approaches as well such as [A1, A2]\n\n[A1] Zhao et al. Maintaining discrimination and fairness in class incremental learning. CVPR 2020. \\\n[A2] Liu et al. Mnemonics training: Multi-class incremental learning without forgetting. CVPR 2020.\n", "summary_of_the_paper": "Unlike most of the continual learning approaches in the literature that perform supervised training at each learning stage, the authors propose to perform unsupervised representation learning on the sequence of incoming data and then classify the samples at each stage using K-NN. Experiments on standard CIFAR10/100 and Tiny-ImageNet shows that the proposed method alleviates catastrophic forgetting and generalizes better in different scenarios.", "summary_of_the_review": "Overall a good quality submission with novel and interesting ideas. It would be great for the authors to address the weaknesses mentioned above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Exciting work at the intersection of continual learning and representation learning. The reviewers have all commented that the proposed work addresses a number of issues related to catastrophic forgetting, which is very encouraging. The work also shows that the representation learning with the proposed method is more general than the one learned with supervised CL. The reviewers have praised the work as being well-written and with thorough experiments. There was a robust back and forth between the reviewers and the authors during the rebuttal period, in which the authors appear to have addressed most of the concerns. Given the insights, results and potential impact of this work, I think this work definitely should be published at ICLR.
Decision: Accept (Oral)