Rethinking the Representational Continuity: Towards Unsupervised Continual Learning

1 INTRODUCTION . Recently continual learning ( Thrun , 1995 ) has gained a lot of attention in the deep learning community due to its ability to continually learn on a sequence of non-stationary tasks ( Kumar & Daume III , 2012 ; Li & Hoiem , 2016 ) and close proximity to the human learning process ( Flesch et al. , 2018 ) . However , the inability of the learner to prevent forgetting of the knowledge learnt from the previous tasks has been a long-standing problem ( McCloskey & Cohen , 1989 ; Goodfellow et al. , 2013 ) . To address this problem , a large body of methods ( Rusu et al. , 2016 ; Zenke et al. , 2017 ; Yoon et al. , 2018 ; Li et al. , 2019 ; Aljundi et al. , 2019 ; Buzzega et al. , 2020 ) have been proposed ; however , all these methods focus on the supervised learning paradigm , but obtaining high-quality labels is expensive and often impractical in real-world scenarios . In contrast , CL for unsupervised representation learning has received limited attention in the community . Although Rao et al . ( 2019 ) instantiated a continual unsupervised representation learning framework ( CURL ) , it is not scalable for high-resolution tasks , as it is composed of MLP encoders/decoders and a simple MoG generative replay . This is evident in their limited empirical evaluation using digit-based gray-scale datasets . Meanwhile , a set of directions have shown huge potential to tackle the representation learning problem without labels ( He et al. , 2020 ; Chen et al. , 2020a ; Grill et al. , 2020 ; Chen et al. , 2020b ; Chen & He , 2021 ; Zbontar et al. , 2021 ) by aligning contrastive pairs of training instances or maximizing the similarity between two augmented views of each image . However , a common assumption for existing methods is the availability of a large amount of unbiased and unlabelled datasets to learn the feature representations . We argue that this assumption is not realistic for most of the real-time applications , including self-driving cars ( Bojarski et al. , 2016 ) , medical applications ( Kelly et al. , 2019 ) and conversational agents ( Li et al. , 2020 ) . The collected datasets are often limited in size during the initial training phase ( Finn et al. , 2017 ) , and datasets/tasks change continuously with time . ∗Corresponding author . The majority of the work was done while the author was a student at KAIST . To accommodate such continuous shifts in data distributions , representation learning models need to increment the knowledge without losing the representations learned in the past . With this motivation , we attempt to bridge the gap between unsupervised representation learning and continual learning to address the challenge of learning the representations on a sequence of tasks . Specifically , we focus on unsupervised continual learning ( UCL ) , where the goal of the continual learner is to learn the representations from a stream of unlabelled data instances without forgetting ( see Figure 1 ) . To this end , we extend various existing SCL strategies to the unsupervised continual learning framework and analyze the performance of current state-of-the-art representation learning techniques : SimSiam ( Chen & He , 2021 ) and BarlowTwins ( Zbontar et al. , 2021 ) for UCL . Surprisingly , we observe that the unsupervised representations are comparatively more robust to catastrophic forgetting across all datasets and simply finetuning on the sequence of tasks can outperform various state-of-the-art continual learning alternatives . Furthermore , we show that UCL generalize better to various out of distribution tasks and outperforms SCL for few-shot training scenarios ( Section 5.2 ) . We demystify the robustness of unsupervised representations by investigating the feature similarity , measured by centered kernel alignment ( CKA ) ( Kornblith et al. , 2019 ) between two independent UCL and SCL models and between UCL and SCL models . We notice that two unsupervised model representations have a relatively high feature similarity compared to two supervised representations . Furthermore , in all cases , two models have high similarity in lower layers indicating that they learn similar low-level features . Further , we measure the ` 2 distance between model parameters ( Neyshabur et al. , 2020 ) and visually compare the feature representations learned by different SCL and UCL strategies . We observe that UCL obtains human perceptual feature patterns for previous tasks , demonstrating their effectiveness to alleviate catastrophic forgetting ( Section 5.3 ) . We conjecture that this is due to their characteristic ability to learn general-purpose features ( Doersch et al. , 2020 ) , which makes them transfer better and comparatively more robust to catastrophic forgetting . To gain further insights , we visualize the loss landscape ( Li et al. , 2018 ) of the UCL and SCL models and observe that UCL obtains a flatter and smoother loss landscape than SCL . Additionally , we propose a simple yet effective technique coined Lifelong Unsupervised Mixup ( LUMP ) , which utilizes mixup ( Zhang et al. , 2018 ) for unlabelled training instances . In particular , LUMP interpolates between the current task examples and examples from previous instances to minimize catastrophic forgetting . We emphasize that LUMP is easy to implement , does not require additional hyperparameters , and simply trains on the interpolated instances . To this end , LUMP requires little , or no modification to existing rehearsal-based methods effectively minimizes catastrophic forgetting even with uniformly selecting the examples from replay buffer . We show that LUMP with UCL outperforms the state-ofthe-art supervised continual learning methods across multiple experimental settings with significantly lower catastrophic forgetting . In summary , our contributions are as follows : • We attempt to bridge the gap between continual learning and representation learning and tackle the two crucial problems of continual learning with unlabelled data and representation learning on a sequence of tasks . • Systematic quantitative analysis shows that UCL achieves better performance over SCL with significantly lower catastrophic forgetting on Sequential CIFAR-10 , CIFAR-100 , and Tiny-ImageNet . Additionally , we evaluate out-of-distribution tasks and few-shot training demonstrating the expressive power of unsupervised representations . • We provide visualization of the representations and loss landscapes , which show that UCL learns discriminative , human perceptual patterns and achieves a flatter and smoother loss landscape . Furthermore , we propose Lifelong Unsupervised Mixup ( LUMP ) for UCL , which effectively alleviates catastrophic forgetting and provides better qualitative interpretations . 2 RELATED WORK . Continual learning . We can partition the existing continual learning methods into three categories . The regularization approaches ( Li & Hoiem , 2016 ; Zenke et al. , 2017 ; Schwarz et al. , 2018 ; Ahn et al. , 2019 ) impose a regularization constraint to the objective that mitigates catastrophic forgetting . The architectural approaches ( Rusu et al. , 2016 ; Li et al. , 2019 ; Yoon et al. , 2020 ) avoid this problem by including task-specific parameters and allowing the expansion of the network during continual learning . The rehearsal approaches ( Rebuffi et al. , 2017 ; Rolnick et al. , 2019 ; Aljundi et al. , 2019 ) allocate a small memory buffer to store and replay the examples from the previous task . However , all these methods are confined to supervised learning , which limits their application in real-life problems . Rao et al . ( 2019 ) tackled the problem of continual unsupervised representation learning and learned task-specific representations on shared parameters ; however , their method is restricted to simple low-resolution tasks and not scalable to standard CL benchmark datasets . Representational learning . A large number of works have addressed the unsupervised learning problem in the standard machine learning framework . Specifically , contrastive learning frameworks ( He et al. , 2020 ; Chen et al. , 2020a ; Grill et al. , 2020 ; Chen et al. , 2020b ; c ) that learn the representations by measuring the similarities of positive and negative pairs have gained a lot of attention in the community . However , all these methods require large batches and negative sample pairs , which restrict the scalability of these networks . Chen & He ( 2021 ) tackled these limitations and proposed SimSiam , that use standard Siamese networks ( Bromley et al. , 1994 ) with the stop-gradient operation to prevent the collapsing of Siamese networks to a constant . Recently , Zbontar et al . ( 2021 ) formulated an objective that pushes the cross-correlation matrix between the embeddings of distorted versions of a sample closer to the identity matrix . However , all these methods assume the presence of large datasets for pre-training , which is impractical in real-world applications . In contrast , we tackle the problem of incremental representational learning and learn the representations sequentially while maximizing task adaptation and minimizing catastrophic forgetting . 3 PRELIMINARIES . 3.1 PROBLEM SETUP . We consider the continual learning setting , where we learn on a continuum of data consisting of T tasks T1 : T = ( T1 . . . TT ) . In supervised continual learning , each task consists a task descriptor τ ∈ { 1 . . . T } and its corresponding dataset Dτ = { ( xi , τ , yi , τ ) nτi=1 } with nτ examples . Each inputpair ( xi , τ , yi , τ ) ∈ Xτ × Yτ , where ( Xτ , Yτ ) is an unkown data distribution . Let us consider a network fΘ : Xτ → RD parametrized by Θ = { wl } l=Ll=1 , where RD and L denote D-dimensional embedding space and number of layers respectively . The classifier is denoted by hψ : RD → Yτ . The network error using cross entropy loss ( CE ) for SCL with finetuning can be formally defined as : LFINETUNESCL = CE ( hψ ( fΘ ( xi , τ ) , τ ) , yi , τ ) . ( 1 ) In this work , we assume the absence of label supervision during training and focus on unsupervised continual learning . In particular , each task consists of Uτ = { ( xi , τ ) nτi=1 } , xi , τ ∈ Xτ with nτ examples . Our aim is to learn the representations fΘ : Xτ → RD on a sequence of tasks while preserving the knowledge of the previous tasks . We introduce the representation learning framework and propose LUMP in Section 4 to learn unsupervised representations while effectively mitigating catastrophic forgetting . 3.2 LEARNING PROTOCOL AND EVALUATION METRICS . Currently , the traditional continual learning strategies follow the standard training protocol , where we learn the network representations fΘ : Xτ → Yτ on a sequence of tasks . In contrast , our goal is to learn the feature representations fΘ : Xτ → RD , so we follow a two-step learning protocol to obtain the model predictions . First , we pre-train the representations on a sequence of tasks T1 : T = ( T . . . TT ) to obtain the representations . Next , we evaluate the quality of our pre-trained representations using a K-nearest neighbor ( KNN ) classifier ( Wu et al. , 2018 ) following the setup in Chen et al . ( 2020a ) ; Chen & He ( 2021 ) ; Zbontar et al . ( 2021 ) . To validate knowledge transfer of the learned representations , we adopt the metrics from the SCL literature ( Chaudhry et al. , 2019b ; Mirzadeh et al. , 2020 ) . Let aτ , i denote the test accuracy of task i after learning task Tτ using a KNN on frozen pre-trained representations on task Tτ . More formally , we can define the metrics to evaluate the continually learned representations as follow : 1 . Average accuracy is the average test accuracy of all the tasks completed until the continual learning of task τ : Aτ = 1τ ∑τ i=1 aτ , i 2 . Average Forgetting is the average performance decrease of each task between its maximum accuracy and accuracy at the completion of training : F = 1T−1 ∑T−1 i=1 maxτ∈ { 1 , ... , T } ( aτ , i − aT , i )

Summary: This paper attempts to bridge the gap between **unsupervised representation learning** and **continual learning** by extending various supervised continual learning methods to the unsupervised learning framework. It builds upon two recent unsupervised feature learning techniques (*SimSiam* and *BarlowTwins*) and a powerful data augmentation technique *MixUp*. Improved performances have been demonstrated in various experimental settings, together with comprehensive feature visualizations.