Year: 2022

Abstract:
Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-ofdistribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (LUMP), a simple yet effective technique that interpolates between the current task and previous tasks’ instances to alleviate catastrophic forgetting for unsupervised representations. We release our code online.

heading: 1 INTRODUCTION
text: Recently continual learning (Thrun, 1995) has gained a lot of attention in the deep learning community due to its ability to continually learn on a sequence of non-stationary tasks (Kumar & Daume III, 2012; Li & Hoiem, 2016) and close proximity to the human learning process (Flesch et al., 2018). However, the inability of the learner to prevent forgetting of the knowledge learnt from the previous tasks has been a long-standing problem (McCloskey & Cohen, 1989; Goodfellow et al., 2013). To address this problem, a large body of methods (Rusu et al., 2016; Zenke et al., 2017; Yoon et al., 2018; Li et al., 2019; Aljundi et al., 2019; Buzzega et al., 2020) have been proposed; however, all these methods focus on the supervised learning paradigm, but obtaining high-quality labels is expensive and often impractical in real-world scenarios. In contrast, CL for unsupervised representation learning has received limited attention in the community. Although Rao et al. (2019) instantiated a continual unsupervised representation learning framework (CURL), it is not scalable for high-resolution tasks, as it is composed of MLP encoders/decoders and a simple MoG generative replay. This is evident in their limited empirical evaluation using digit-based gray-scale datasets.
Meanwhile, a set of directions have shown huge potential to tackle the representation learning problem without labels (He et al., 2020; Chen et al., 2020a; Grill et al., 2020; Chen et al., 2020b; Chen & He, 2021; Zbontar et al., 2021) by aligning contrastive pairs of training instances or maximizing the similarity between two augmented views of each image. However, a common assumption for existing methods is the availability of a large amount of unbiased and unlabelled datasets to learn the feature representations. We argue that this assumption is not realistic for most of the real-time applications, including self-driving cars (Bojarski et al., 2016), medical applications (Kelly et al., 2019) and conversational agents (Li et al., 2020). The collected datasets are often limited in size during the initial training phase (Finn et al., 2017), and datasets/tasks change continuously with time.
∗Corresponding author. The majority of the work was done while the author was a student at KAIST.
To accommodate such continuous shifts in data distributions, representation learning models need to increment the knowledge without losing the representations learned in the past. With this motivation, we attempt to bridge the gap between unsupervised representation learning and continual learning to address the challenge of learning the representations on a sequence of tasks. Specifically, we focus on unsupervised continual learning (UCL), where the goal of the continual learner is to learn the representations from a stream of unlabelled data instances without forgetting (see Figure 1). To this end, we extend various existing SCL strategies to the unsupervised continual learning framework and analyze the performance of current state-of-the-art representation learning techniques: SimSiam (Chen & He, 2021) and BarlowTwins (Zbontar et al., 2021) for UCL. Surprisingly, we observe that the unsupervised representations are comparatively more robust to catastrophic forgetting across all datasets and simply finetuning on the sequence of tasks can outperform various state-of-the-art continual learning alternatives. Furthermore, we show that UCL generalize better to various out of distribution tasks and outperforms SCL for few-shot training scenarios (Section 5.2).
We demystify the robustness of unsupervised representations by investigating the feature similarity, measured by centered kernel alignment (CKA) (Kornblith et al., 2019) between two independent UCL and SCL models and between UCL and SCL models. We notice that two unsupervised model representations have a relatively high feature similarity compared to two supervised representations. Furthermore, in all cases, two models have high similarity in lower layers indicating that they learn similar low-level features. Further, we measure the `2 distance between model parameters (Neyshabur et al., 2020) and visually compare the feature representations learned by different SCL and UCL strategies. We observe that UCL obtains human perceptual feature patterns for previous tasks, demonstrating their effectiveness to alleviate catastrophic forgetting (Section 5.3). We conjecture that this is due to their characteristic ability to learn general-purpose features (Doersch et al., 2020), which makes them transfer better and comparatively more robust to catastrophic forgetting.
To gain further insights, we visualize the loss landscape (Li et al., 2018) of the UCL and SCL models and observe that UCL obtains a flatter and smoother loss landscape than SCL. Additionally, we propose a simple yet effective technique coined Lifelong Unsupervised Mixup (LUMP), which utilizes mixup (Zhang et al., 2018) for unlabelled training instances. In particular, LUMP interpolates between the current task examples and examples from previous instances to minimize catastrophic forgetting. We emphasize that LUMP is easy to implement, does not require additional hyperparameters, and simply trains on the interpolated instances. To this end, LUMP requires little, or no modification to existing rehearsal-based methods effectively minimizes catastrophic forgetting even with uniformly selecting the examples from replay buffer. We show that LUMP with UCL outperforms the state-ofthe-art supervised continual learning methods across multiple experimental settings with significantly lower catastrophic forgetting. In summary, our contributions are as follows:
• We attempt to bridge the gap between continual learning and representation learning and tackle the two crucial problems of continual learning with unlabelled data and representation learning on a sequence of tasks.
• Systematic quantitative analysis shows that UCL achieves better performance over SCL with significantly lower catastrophic forgetting on Sequential CIFAR-10, CIFAR-100, and Tiny-ImageNet. Additionally, we evaluate out-of-distribution tasks and few-shot training demonstrating the expressive power of unsupervised representations.
• We provide visualization of the representations and loss landscapes, which show that UCL learns discriminative, human perceptual patterns and achieves a flatter and smoother loss landscape. Furthermore, we propose Lifelong Unsupervised Mixup (LUMP) for UCL, which effectively alleviates catastrophic forgetting and provides better qualitative interpretations.

heading: 2 RELATED WORK
text: Continual learning. We can partition the existing continual learning methods into three categories. The regularization approaches (Li & Hoiem, 2016; Zenke et al., 2017; Schwarz et al., 2018; Ahn et al., 2019) impose a regularization constraint to the objective that mitigates catastrophic forgetting. The architectural approaches (Rusu et al., 2016; Li et al., 2019; Yoon et al., 2020) avoid this problem by including task-specific parameters and allowing the expansion of the network during continual learning. The rehearsal approaches (Rebuffi et al., 2017; Rolnick et al., 2019; Aljundi et al., 2019) allocate a small memory buffer to store and replay the examples from the previous task. However, all these methods are confined to supervised learning, which limits their application in real-life problems. Rao et al. (2019) tackled the problem of continual unsupervised representation learning and learned task-specific representations on shared parameters; however, their method is restricted to simple low-resolution tasks and not scalable to standard CL benchmark datasets.
Representational learning. A large number of works have addressed the unsupervised learning problem in the standard machine learning framework. Specifically, contrastive learning frameworks (He et al., 2020; Chen et al., 2020a; Grill et al., 2020; Chen et al., 2020b;c) that learn the representations by measuring the similarities of positive and negative pairs have gained a lot of attention in the community. However, all these methods require large batches and negative sample pairs, which restrict the scalability of these networks. Chen & He (2021) tackled these limitations and proposed SimSiam, that use standard Siamese networks (Bromley et al., 1994) with the stop-gradient operation to prevent the collapsing of Siamese networks to a constant. Recently, Zbontar et al. (2021) formulated an objective that pushes the cross-correlation matrix between the embeddings of distorted versions of a sample closer to the identity matrix. However, all these methods assume the presence of large datasets for pre-training, which is impractical in real-world applications. In contrast, we tackle the problem of incremental representational learning and learn the representations sequentially while maximizing task adaptation and minimizing catastrophic forgetting.

heading: 3 PRELIMINARIES

heading: 3.1 PROBLEM SETUP
text: We consider the continual learning setting, where we learn on a continuum of data consisting of T tasks T1:T = (T1 . . . TT ). In supervised continual learning, each task consists a task descriptor τ ∈ {1 . . . T} and its corresponding dataset Dτ = {(xi,τ , yi,τ )nτi=1} with nτ examples. Each inputpair (xi,τ , yi,τ ) ∈ Xτ × Yτ , where (Xτ ,Yτ ) is an unkown data distribution. Let us consider a network fΘ : Xτ → RD parametrized by Θ = {wl}l=Ll=1 , where RD and L denote D-dimensional embedding space and number of layers respectively. The classifier is denoted by hψ : RD → Yτ . The network error using cross entropy loss (CE) for SCL with finetuning can be formally defined as:
LFINETUNESCL = CE (hψ (fΘ (xi,τ ) , τ) , yi,τ ) . (1)
In this work, we assume the absence of label supervision during training and focus on unsupervised continual learning. In particular, each task consists of Uτ = {(xi,τ )nτi=1}, xi,τ ∈ Xτ with nτ examples. Our aim is to learn the representations fΘ : Xτ → RD on a sequence of tasks while preserving the knowledge of the previous tasks. We introduce the representation learning framework and propose LUMP in Section 4 to learn unsupervised representations while effectively mitigating catastrophic forgetting.

heading: 3.2 LEARNING PROTOCOL AND EVALUATION METRICS
text: Currently, the traditional continual learning strategies follow the standard training protocol, where we learn the network representations fΘ : Xτ → Yτ on a sequence of tasks. In contrast, our goal is to learn the feature representations fΘ : Xτ → RD, so we follow a two-step learning protocol to obtain the model predictions. First, we pre-train the representations on a sequence of tasks T1:T = (T . . . TT ) to obtain the representations. Next, we evaluate the quality of our pre-trained representations using a K-nearest neighbor (KNN) classifier (Wu et al., 2018) following the setup in Chen et al. (2020a); Chen & He (2021); Zbontar et al. (2021).
To validate knowledge transfer of the learned representations, we adopt the metrics from the SCL literature (Chaudhry et al., 2019b; Mirzadeh et al., 2020). Let aτ,i denote the test accuracy of task i after learning task Tτ using a KNN on frozen pre-trained representations on task Tτ . More formally, we can define the metrics to evaluate the continually learned representations as follow:
1. Average accuracy is the average test accuracy of all the tasks completed until the continual learning of task τ : Aτ = 1τ ∑τ i=1 aτ,i 2. Average Forgetting is the average performance decrease of each task between its maximum accuracy and accuracy at the completion of training: F = 1T−1 ∑T−1 i=1 maxτ∈{1,...,T} (aτ,i − aT,i)

heading: 4 UNSUPERVISED CONTINUAL LEARNING

heading: 4.1 CONTINUOUS REPRESENTATION LEARNING WITH SEQUENTIAL TASKS
text: To learn feature representations, contrastive learning (Chen et al., 2020a;b; He et al., 2020) maximizes the similarity of representations between the images of the same views (positive pairs) and minimizes the similarity between images of different views (negative pairs). However, these methods require large batches, negative sample pairs (Chen et al., 2020a;b), or architectural modifications (He et al., 2020; Chen et al., 2020c), or non-differentiable operators (Caron et al., 2020), which makes their application difficult for continual learning scenarios. In this work, we focus on SimSiam (Chen & He, 2021) and BarlowTwins (Zbontar et al., 2021), which tackle these limitations and achieve state-of-the-art performance on standard representation learning benchmarks.
SimSiam (Chen & He, 2021) uses a variant of Siamese networks (Bromley et al., 1994) for learning input data representations. It consists of an encoder network fΘ, which is composed of a backbone network, and is shared across a projection MLP and prediction MLP head h(·). Specifically, SimSiam minimizes the cosine-similarity between the output vectors of the projector and the predictor MLP across two different augmentations for an image. Initially, we consider FINETUNE, which is a a naive CL baseline that minimizes the cosine-similarity between the projector output (z1i,τ = fΘ(x 1 i,τ )) and the predictor output (p2i,τ = h(fΘ(x 2 i,τ )) on a sequence of tasks as follows:
LFINETUNEUCL = 1
2 D(p1i,τ ,stopgrad(z 2 i,τ )) +
1 2 D(p2i,τ ,stopgrad(z 1 i,τ )), (2)
where D(p1i,τ , z 2 i,τ ) = − p1i,τ ‖p2i,τ‖2 · z2i,τ ‖z2i,τ‖2 ,
x1i,τ and x 2 i,τ are two randomly augmented views of an input example xi,τ ∈ Tτ and ‖·‖2 denotes the `2-norm. Note that, the stopgrad is a crucial component in SimSiam to prevent the trivial solutions obtained by Siamese networks. Due to its simplicity and effectiveness, we chose Simsiam to analyze the performance of unsupervised representations for continual learning.
BarlowTwins (Zbontar et al., 2021) minimizes the redundancy between the embedding vector components of the distorted versions of an instance while conserving the maximum information inspired from Barlow (1961). In particular, the objective function eliminates the SimSiam stopgrad component and instead makes the cross-correlation matrix computed between the outputs of two identical networks closer to the identity matrix. Let C be the cross-correlation matrix between the outputs of two Siamese branches along the batch dimension and Z1 and Z2 denote the batch embeddings of the distorted views for all images of a batch from the current task (xτ ∈ Uτ ). The objective function for UCL with finetuning and BarlowTwins can then be defined as:
LFINETUNEUCL = ∑ i (1− Cii)2 + λ · ∑ i ∑ j 6=i Cij2, where Cij = ∑ B z 1 B,iz 2 B,j√∑
B (z 1 B,i)
2 √∑
B (z 2 B,j)
2 . (3)
λ is a positive constant trading off the importance of the invariance and redundancy reduction terms of the loss, i and j denote the network’s output vector dimensions. Similar to SimSiam, BarlowTwins is simple, easy to implement, and can be applied to existing continual learning strategies with little or no modification.

heading: 4.2 PRESERVING REPRESENTATIONAL CONTINUITY: A VIEW OF EXISTING SCL METHODS
text: Learning feature representations from labelled instances on a sequence of tasks has been long studied in continual learning. However, the majority of these learning strategies are not directly applicable to UCL. To compare with the regularization-based strategies, we extend Synaptic Intelligence (SI) (Zenke et al., 2017) to UCL and consider the online per-synapse consolidation during the entire training trajectory of the unsupervised representations. For architectural-based strategies, we investigate Progressive Neural Networks (PNN) (Rusu et al., 2016) and learn the feature representations progressively using the representations learning frameworks proposed in Section 4.1.
We also formulate Dark Experience Replay (DER) (Buzzega et al., 2020) for UCL. DER for SCL alleviates catastrophic forgetting by matching the network logits across a sequence of tasks during the optimization trajectory. Notably, the loss for SCL-DER can be defined as follow:
LDERSCL = LFINETUNESCL + α · E(x,p)∼M [ ‖softmax(p)− softmax(hψ(xi,τ ))‖22 ] , (4)
where p = hφτ (x), LFINETUNESCL denotes the cross-entropy loss on the current task (see Equation (1)) and random examples are selected using reservoir sampling from the replay-bufferM. Since, we do not have access to the labels for UCL, we cannot minimize the aforementioned objective.
Instead, we utilize the output of the projected output by the backbone network to preserve the knowledge of the past tasks over the entire training trajectory. In particular, DER for UCL consists of a combination of two terms. The first term learns the representations using SimSiam from Equation (2) or BarlowTwins from Equation (3) and the second term minimizes the Euclidean distance between the projected outputs to minimize catastrophic forgetting. More formally, UCL-DER minimizes the following loss:
LDERUCL = LFINETUNEUCL + α · E(x)∼M [ ‖fΘτ (x)− fΘ(xi,τ )‖ 2 2 ] (5)
However, the performance of the rehearsal-based methods is sensitive to the choice of α and often requires supervised training setup, task identities, and boundaries. To tackle this issue, we propose Lifelong Unsupervised Mixup in the subsequent subsection, which interpolates between the current and past task instances to mitigate catastrophic forgetting effectively.

heading: 4.3 LIFELONG UNSUPERVISED MIXUP
text: The standard Mixup (Zhang et al., 2018) training constructs virtual training examples based on the principle of Vicinal Risk Minimization . In particular, let (xi, yi) and (xj , yj) denote two random feature-target pairs sampled from the training data distribution and let (x̃, ỹ) denote the interpolated feature-target pair in the vicinity of these examples; mixup minimizes the following objective on the interpolated samples:
LMIXUP(x̃, ỹ) = CE (hψ (fΘ (x̃) , τ) , ỹ) , (6) where x̃ = λ · xi + (1− λ) · xj and ỹ = λ · yi + (1− λ) · yj .
λ ∼ Beta(α, α), for α ∈ (0,∞). In this work, we propose Lifelong Unsupervised Mixup (LUMP) that utilizes mixup for unsupervised continual learning by incorporating the instances stored in the replay-buffer from the previous tasks into the vicinal distribution. In particular, LUMP interpolates between the examples of the current task (xi,τ ) ∈ Uτ and random examples selected using uniform sampling from the replay buffer, which encourages the model to behave linearly across a sequence of tasks. More formally, LUMP minimizes the objective in Equation (2) and Equation (3) on the following interpolated instances x̃i,τ for the current task τ :
x̃i,τ = λ · xi,τ + (1− λ) · xj,M, (7) where xj,M ∼ M denotes the example selected using uniform sampling from replay buffer M. The interpolated examples not only augments the past tasks’ instances in the replay buffer but also approximates a regularized loss minimization (Zhang et al., 2021). During UCL, LUMP enhances the robustness of learned representation by revisiting the attributes of the past task that are similar to the current task. To this end, LUMP successively mitigates catastrophic forgetting and learns discriminative & human-perceptual features with significant improvement in performance over the current state-of-the-art SCL strategies (see Table 1 and Figure 4). Furthermore, unlike the existing rehearsal-based methods, it does not require the tuning of additional hyper-parameters.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETUP
text: Baselines. We compare with multiple supervised and unsupervised continual learning baselines across different categories of continual learning methods.
1. Supervised continual learning. FINETUNE is a vanilla supervised learning method trained on a sequence of tasks without regularization or episodic memory and MULTITASK optimizes the model on complete data. For regularization-based CL methods, we compare against SI (Zenke et al., 2017) and AGEM (Chaudhry et al., 2019a). We include PNN (Rusu et al., 2016) for architecture-based methods. Lastly, we consider GSS (Aljundi et al., 2019) that populates the replay-buffer using solid-angle minimization and DER (Buzzega et al., 2020) matches the network logits sampled through the optimization trajectory for rehearsal during continual learning.
2. Unsupervised continual learning. We consider the unsupervised variants of various SCL baselines to show the utility of the unsupervised representations for sequential learning. Specifically, we use SIMSIAM (Chen & He, 2021) and BARLOWTWINS (Zbontar et al., 2021), which are the state-of-the-art representational learning techniques for learning the unsupervised continual representations. We compare with FINETUNE and MULTITASK following the supervised learning baselines, and SI (Zenke et al., 2017), PNN (Rusu et al., 2016) for unsupervised regularization and architecture CL methods respectively. For rehearsal-based method, we compare with the UCL variant of DER (Buzzega et al., 2020) described in Section 4.2
Datasets. We compare the performance of SCL and UCL on various continual learning benchmarks using single-head ResNet-18 (He et al., 2016) architecture. Split CIFAR-10 (Krizhevsky, 2012) consists of two random classes out of the ten classes for each task. Split CIFAR-100 (Krizhevsky, 2012) consists of five random classes out of the 100 classes for each task. Split Tiny-ImageNet is a variant of the ImageNet dataset (Deng et al., 2009) containing five random classes out of the 100 classes for each task with the images sized 64 × 64 pixels. Training and evaluation setup. We follow the hyperparameter setup of Buzzega et al. (2020) for all the SCL strategies and tune them for the UCL representation learning strategies. All the learned representations are evaluated with KNN classifier (Wu et al., 2018) across three independent runs. Further, we use the hyper-parameters obtained by SimSiam for training UCL strategies with BarlowTwins to analyze the sensitivity of UCL to hyper-parameters and for a fair comparison between different methods. We train all the UCL methods for 200 epochs and evaluate with the KNN classifier (Wu et al., 2018). We provide the hyper-parameters in detail in Table A.5.

heading: 5.2 QUANTITATIVE RESULTS
text: Evaluation on SimSiam. Table 1 shows the evaluation results for supervised and unsupervised representations learnt by SimSiam (Chen & He, 2021) across various continual learning strategies. In all cases, continual learning with unsupervised representations achieves significantly better performance than supervised representations with substantially lower forgetting. For instance, SI with UCL obtains better performance and 68%, 54%, and 44% lower forgetting relative to the best-performing SCL strategy on Split CIFAR-10, Split CIFAR-100, and Split Tiny-ImageNet, respectively. Surprisingly, FINETUNE with UCL achieves higher performance and significantly lower forgetting in comparison to all SCL strategies except DER. Furthermore, LUMP improves upon the UCL strategies: 2.8% and 5.9% relative increase in accuracy and 15% and 57.1% relative decrease in forgetting on Split CIFAR-100 and Split Tiny-ImageNet, respectively.
Evaluation on BarlowTwins. To verify that unsupervised representations are indeed more robust to catastrophic forgetting, we train BarlowTwins (Zbontar et al., 2021) on a sequence of tasks. We notice that the representations learned with BarlowTwins substantially improve the accuracy and forgetting over SCL: 71.4%, 69.7% and 73.2% decrease in forgetting with FINETUNE on Split CIFAR-10, Split CIFAR-100 and Split Tiny-ImageNet respectively. Similarly, we observe that SI, and DER are more robust to catastrophic forgetting; however, PNN underperforms on complicated tasks since feature accumulation using adaptor modules is insufficient to construct useful representations for current task adaptation. Interestingly, representations learnt with BarlowTwins achieve lower forgetting for FINETUNE, DER and LUMP than SimSiam with comparable accuracy across all the datasets.

heading: S FINETUNE 87.72 (± 0.32) 4.08 (± 0.56) 71.97 (± 0.54) 9.45 (± 1.01) 66.28 (± 1.23) 8.89 (± 0.66)
text: Figure 2: Evaluation on Few-shot training for Split CIFAR-100 across different number of training instances per task. The results are measured across three independent trials.
Figure 3: CKA Feature similarity between two independent UCL models (red), two independent SCL models (blue), and UCL and SCL model (green) for different strategies on Split CIFAR-100 test distribution.
Evaluation on Few-shot training. Figure 2 compares the effect of few-shot training on UCL and SCL, where each task has a limited number of training instances. Specifically, we conduct the experimental evaluation using 100, 200, 500, and 2500 training instances for each task in split CIFAR-100 dataset. Surprisingly, we observe that the gap in average accuracy between SCL and UCL methods widens with a decrease in the number of training instances. Note that UCL decreases the accuracy by 15.78%p on average with lower forgetting when the number of training instances decreases from 2500 to 100; whereas, SCL obtains a severe 32.21%p deterioration in accuracy. We conjecture that this is an outcome of the discriminative feature embeddings learned by UCL, which discriminates all the images in the dataset and captures more than class-specific information as also observed in Doersch et al. (2020). Furthermore, LUMP improves the performance over all the baselines with a significant margin across all few-shot experiments.
Evaluation on OOD datasets. We evaluate the learnt representations on various out-of-distribution (OOD) datasets in Table 2 to measure their generalization to unseen data distributions. In particular, we conduct the OOD evaluation on MNIST (LeCun, 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), SVHN (Netzer et al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky, 2012) using a KNN classifier (Wu et al., 2018). We observe that unsupervised representations outperform the supervised representations in all cases across all the datasets. In particular, the UCL representations learned with Simsiam, and SI on Split-CIFAR-10 improves the absolute performance over the best-performing SCL strategy by 4.58%, 6.09%, 15.26%, and 17.07% on MNIST, FMNIST, SVHN, and CIFAR-100 respectively. Further, LUMP trained on Split-CIFAR-100 outperforms SI across all datasets and obtains comparable performance with Split CIFAR-10 dataset.

heading: 5.3 QUALITATIVE ANALYSIS
text: Similarity in feature and parameter space. We analyze the similarity between the representations learnt between (i) Two independent UCL models, (ii) Two independent SCL models (iii) SCL and UCL models using centered kernel alignment (CKA) (Kornblith et al., 2019) in Figure 3, which provides a score between 0 and 1 measuring the similarity between a pair of hidden representations. For two representations Θ1 : X → Rd1 and Θ2 : X → Rd1 , CKA(Θ1,Θ2) = ||Cov(Θ1(x),Θ2(x))|| 2 F
||Cov(Θ1(x))||F ·||Cov(Θ2(x))||F , where covariances are with respect to the test distribution. Additionally, we measure the `2 distance (Neyshabur et al., 2020) between the parameters of two independent UCL models (see Table 3) and two independent SCL models (see Table 4). First, we observe that the representations learned by two independent UCL methods have a high feature similarity and lower `2 distance compared to the two independent SCL methods, demonstrating UCL representations’ robustness. Second, we note that the representations between any two independent models are highly similar in the lower layers indicating that they learn similar high-level features, including edges and shapes; however, the features are dissimilar for the higher modules. Lastly, we see that the representations between a UCL and SCL model are similar in the lower layers but diverge in the higher layers across all CL strategies.
Visualization of feature space. Next, we visualize the learned features to dissect further the representations learned by UCL and SCL strategies. Figure 4 shows the visualization of the latent feature maps for tasks T0 and T13 after the completion of continual learning. For T0, we observe that the SCL methods are prone to catastrophic forgetting, as the features appear noisy and do not have coherent patterns. In contrast, the features learned by UCL strategies are perceptually relevant and robust to catastrophic forgetting, with LUMP learning the most distinctive features. Similar to T0, we observe that the UCL features are more relevant and distinguishable than SCL for T13. Note that we randomly selected the examples and feature maps for all visualizations.
Loss landscape visualization. To gain further insights, we visualize the loss landscape of task T0 after the completion of training on task T0 and T19 for various UCL and SCL strategies in Figure 5. We measure the cross-entropy loss for all methods with a randomly initialized linear classifier for a fair evaluation of two different directions. We use the visualization tool from Li et al. (2018) that searches the task loss surface by repeatedly adding random perturbations to model weights. We observe that the loss landscape after T0 looks quite similar across all the strategies since the forgetting does not exist yet. However, after training T19, there is a clear difference with the UCL strategies obtaining a flatter and smoother loss landscape because UCL methods are more stable and robust to the forgetting, which hurts the loss landscapes of past tasks for SCL. It is important to observe that LUMP obtains a smoother landscape than other UCL strategies, demonstrating its effectiveness. We defer further analyses for feature and loss landscape visualization to Appendix A.2.
Table 3: `2 distance between UCL parameters after completion of training.
Table 4: `2 distance between SCL paraneters after completion of training.

heading: 6 DISCUSSION AND CONCLUSION
text: This work attempts to bridge the gap between unsupervised representation learning and continual learning. In particular, we establish the following findings for unsupervised continual learning.
Surpassing supervised continual learning. Our empirical evaluation across various CL strategies and datasets shows that UCL representations are more robust to catastrophic forgetting than SCL representations. Furthermore, we notice that UCL generalizes better to OOD tasks and achieves stronger performance on few-shot learning tasks. We propose Lifelong unsupervised mixup (LUMP), which interpolates the unsupervised instances between the current task and past task and obtains higher performance with lower catastrophic forgetting across a wide range of tasks.
Dissecting the learned representations. We conduct a systematic analysis to understand the differences between the representations learned by UCL and SCL strategies. By investigating the similarity between the representations, we observe that UCL and SCL strategies have high similarities in the lower layers but are dissimilar in the higher layers. We also show that UCL representations learn coherent and discriminative patterns and smoother loss landscape than SCL.
Limitations and future work. In this work, we do not consider the high-resolution tasks for CL. We intend to evaluate the forgetting of the learnt representations on ImageNet (Deng et al., 2009) in future work, since UCL shows lower catastrophic forgetting and representation learning has made significant progress on ImageNet over the past years. In follow-up work, we intend to conduct further analysis to understand the behavior of UCL and develop sophisticated methods to continually learn unsupervised representations under various setups, such as class-incremental or task-agnostic CL.

heading: 7 AUTHOR CONTRIBUTIONS
text: Divyam Madaan conceived of the presented idea, developed the experimental framework, planned the experiments, carried out OOD evaluation, CKA visualization and took the lead in writing the manuscript. Jaehong Yoon performed the hyperparameter search, carried out the visualization of loss landscape and feature maps and performed the few-shot training analysis. Yuanchun Li, Yunxin Liu, and Sung Ju Hwang supervised the project.

heading: A SUPPLEMENTARY MATERIAL
text: Organization. In the supplementary material, we provide the implementation details followed by the hyper-parameter configurations in Appendix A.1. Further, we show the other experiments we conducted and additional visualizations and results in Appendix A.2.
A.1 EXPERIMENTAL DETAILS
Implementations. We use the DER (Buzzega et al., 2020) open-source codebase1 for all the experiments. In particular, we reproduce all their experimental results for supervised continual learning and use various models with their set of hyper-parameters as our baselines. We follow the original representations for SimSiam2 and BarlowTwins3 for unsupervised continual learning. We verify our implementation by reproducing the reported results on CIFAR-10 in the original paper, where we train the representations on the complete CIFAR-10 dataset and evaluate on the test-set using KNN classifier (Wu et al., 2018). In particular, (Wu et al., 2018) stores the features for each instance in the task-level training set in a discrete memory bank. The optimal feature-level embeddings are then learned by instance-level discrimination, which maximally scatters the features of the training samples. Following prior works in representation learning, we use the task-level training set without any augmentation in the task-incremental setup for the supervised and unsupervised KNN evaluation.
Hyperparameter configurations. We use the tuned hyper-parameters reported by Buzzega et al. (2020) for all the SCL experiments. On the other hand, we tune the hyper-parameters for continual learning strategies for UCL. We provide the hyper-parameters setup for UCL for different datasets in Table A.5. We train all the UCL methods with a batch size of 256 for 200 epochs. For rehearsalbased methods, we use the buffer size 200 for Split CIFAR-10, Split CIFAR-100 and 256 for Split Tiny-ImageNet dataset. We use a learning rate of 0.03 for SGD optimizer with weight decay 5e-4 and momentum 0.9.
Table A.5: Hyperparameter configurations for all the datasets on ResNet-18 architecture.
METHOD SPLIT CIFAR-10 SPLIT CIFAR-100 SEQ. TINY-IMAGENET
SI c : 100 ξ : 1 c : 0.1 ξ : 1 c : 0.01 ξ : 1 PNN wd : 64 wd : 12 wd : 8 DER α : 0.1 α : 0.1 α : 0.01 LUMP λ : 0.1 λ : 0.1 λ : 0.4
A.2 ADDITIONAL EXPERIMENTS
We provide additional loss landscape and feature visualizations on Split CIFAR-100 in Figure A.6 and Figure A.7 respectively. Figure A.9 shows the feature visualizations for Split Tiny-ImageNet on ResNet-18 architecture.
1https://github.com/aimagelab/mammoth 2https://github.com/facebookresearch/simsiam 3https://github.com/facebookresearch/barlowtwins

Review summaries:
This paper attempts to bridge the gap between **unsupervised representation learning** and **continual learning** by extending various supervised continual learning methods to the unsupervised learning framework. It builds upon two recent unsupervised feature learning techniques (*SimSiam* and *BarlowTwins*) and a powerful data augmentation technique *MixUp*. Improved performances have been demonstrated in various experimental settings, together with comprehensive feature visualizations.

This paper studies the problem of representation learning in an unsupervised continual learning(UCL) setting. It shows that the representation learned with UCL is more general than the one learned with supervised CL (SCL), and investigates why UCL is more robust to catastrophic forgetting than SCL by analyzing the similarity of learned features and visualizing loss landscape. The authors also propose to apply mixup technique to UCL setting and present a LUMP algorithm to further improve the performance of CL.This paper studies the problem of representation learning in an unsupervised continual learning(UCL) setting. It shows that the representation learned with UCL is more general than the one learned with supervised CL (SCL), and investigates why UCL is more robust to catastrophic forgetting than SCL by analyzing the similarity of learned features and visualizing loss landscape. The authors also propose to apply mixup technique to UCL setting and present a LUMP algorithm to further improve the performance of CL.

The paper proposes to tackle the continual learning problem in an unsupervised setting. It shows that recent self-supervised learning methods are efficient tools to learn image representation with lower catastrophic learning problems. Two recent self-supervised methods are evaluated: SimSiam and BarlowTwins. In both cases, the superiority of unsupervised features is demonstrated.   The widely used mixup method is also adapted to the UCL problem. Straightforwardly, current images are mixed with images of the past tasks sampled from the replay buffer.