Rethinking the Representational Continuity: Towards Unsupervised Continual Learning

1 INTRODUCTION . Recently continual learning ( Thrun , 1995 ) has gained a lot of attention in the deep learning community due to its ability to continually learn on a sequence of non-stationary tasks ( Kumar & Daume III , 2012 ; Li & Hoiem , 2016 ) and close proximity to the human learning process ( Flesch et al. , 2018 ) . However , the inability of the learner to prevent forgetting of the knowledge learnt from the previous tasks has been a long-standing problem ( McCloskey & Cohen , 1989 ; Goodfellow et al. , 2013 ) . To address this problem , a large body of methods ( Rusu et al. , 2016 ; Zenke et al. , 2017 ; Yoon et al. , 2018 ; Li et al. , 2019 ; Aljundi et al. , 2019 ; Buzzega et al. , 2020 ) have been proposed ; however , all these methods focus on the supervised learning paradigm , but obtaining high-quality labels is expensive and often impractical in real-world scenarios . In contrast , CL for unsupervised representation learning has received limited attention in the community . Although Rao et al . ( 2019 ) instantiated a continual unsupervised representation learning framework ( CURL ) , it is not scalable for high-resolution tasks , as it is composed of MLP encoders/decoders and a simple MoG generative replay . This is evident in their limited empirical evaluation using digit-based gray-scale datasets . Meanwhile , a set of directions have shown huge potential to tackle the representation learning problem without labels ( He et al. , 2020 ; Chen et al. , 2020a ; Grill et al. , 2020 ; Chen et al. , 2020b ; Chen & He , 2021 ; Zbontar et al. , 2021 ) by aligning contrastive pairs of training instances or maximizing the similarity between two augmented views of each image . However , a common assumption for existing methods is the availability of a large amount of unbiased and unlabelled datasets to learn the feature representations . We argue that this assumption is not realistic for most of the real-time applications , including self-driving cars ( Bojarski et al. , 2016 ) , medical applications ( Kelly et al. , 2019 ) and conversational agents ( Li et al. , 2020 ) . The collected datasets are often limited in size during the initial training phase ( Finn et al. , 2017 ) , and datasets/tasks change continuously with time . ∗Corresponding author . The majority of the work was done while the author was a student at KAIST . To accommodate such continuous shifts in data distributions , representation learning models need to increment the knowledge without losing the representations learned in the past . With this motivation , we attempt to bridge the gap between unsupervised representation learning and continual learning to address the challenge of learning the representations on a sequence of tasks . Specifically , we focus on unsupervised continual learning ( UCL ) , where the goal of the continual learner is to learn the representations from a stream of unlabelled data instances without forgetting ( see Figure 1 ) . To this end , we extend various existing SCL strategies to the unsupervised continual learning framework and analyze the performance of current state-of-the-art representation learning techniques : SimSiam ( Chen & He , 2021 ) and BarlowTwins ( Zbontar et al. , 2021 ) for UCL . Surprisingly , we observe that the unsupervised representations are comparatively more robust to catastrophic forgetting across all datasets and simply finetuning on the sequence of tasks can outperform various state-of-the-art continual learning alternatives . Furthermore , we show that UCL generalize better to various out of distribution tasks and outperforms SCL for few-shot training scenarios ( Section 5.2 ) . We demystify the robustness of unsupervised representations by investigating the feature similarity , measured by centered kernel alignment ( CKA ) ( Kornblith et al. , 2019 ) between two independent UCL and SCL models and between UCL and SCL models . We notice that two unsupervised model representations have a relatively high feature similarity compared to two supervised representations . Furthermore , in all cases , two models have high similarity in lower layers indicating that they learn similar low-level features . Further , we measure the ` 2 distance between model parameters ( Neyshabur et al. , 2020 ) and visually compare the feature representations learned by different SCL and UCL strategies . We observe that UCL obtains human perceptual feature patterns for previous tasks , demonstrating their effectiveness to alleviate catastrophic forgetting ( Section 5.3 ) . We conjecture that this is due to their characteristic ability to learn general-purpose features ( Doersch et al. , 2020 ) , which makes them transfer better and comparatively more robust to catastrophic forgetting . To gain further insights , we visualize the loss landscape ( Li et al. , 2018 ) of the UCL and SCL models and observe that UCL obtains a flatter and smoother loss landscape than SCL . Additionally , we propose a simple yet effective technique coined Lifelong Unsupervised Mixup ( LUMP ) , which utilizes mixup ( Zhang et al. , 2018 ) for unlabelled training instances . In particular , LUMP interpolates between the current task examples and examples from previous instances to minimize catastrophic forgetting . We emphasize that LUMP is easy to implement , does not require additional hyperparameters , and simply trains on the interpolated instances . To this end , LUMP requires little , or no modification to existing rehearsal-based methods effectively minimizes catastrophic forgetting even with uniformly selecting the examples from replay buffer . We show that LUMP with UCL outperforms the state-ofthe-art supervised continual learning methods across multiple experimental settings with significantly lower catastrophic forgetting . In summary , our contributions are as follows : • We attempt to bridge the gap between continual learning and representation learning and tackle the two crucial problems of continual learning with unlabelled data and representation learning on a sequence of tasks . • Systematic quantitative analysis shows that UCL achieves better performance over SCL with significantly lower catastrophic forgetting on Sequential CIFAR-10 , CIFAR-100 , and Tiny-ImageNet . Additionally , we evaluate out-of-distribution tasks and few-shot training demonstrating the expressive power of unsupervised representations . • We provide visualization of the representations and loss landscapes , which show that UCL learns discriminative , human perceptual patterns and achieves a flatter and smoother loss landscape . Furthermore , we propose Lifelong Unsupervised Mixup ( LUMP ) for UCL , which effectively alleviates catastrophic forgetting and provides better qualitative interpretations . 2 RELATED WORK . Continual learning . We can partition the existing continual learning methods into three categories . The regularization approaches ( Li & Hoiem , 2016 ; Zenke et al. , 2017 ; Schwarz et al. , 2018 ; Ahn et al. , 2019 ) impose a regularization constraint to the objective that mitigates catastrophic forgetting . The architectural approaches ( Rusu et al. , 2016 ; Li et al. , 2019 ; Yoon et al. , 2020 ) avoid this problem by including task-specific parameters and allowing the expansion of the network during continual learning . The rehearsal approaches ( Rebuffi et al. , 2017 ; Rolnick et al. , 2019 ; Aljundi et al. , 2019 ) allocate a small memory buffer to store and replay the examples from the previous task . However , all these methods are confined to supervised learning , which limits their application in real-life problems . Rao et al . ( 2019 ) tackled the problem of continual unsupervised representation learning and learned task-specific representations on shared parameters ; however , their method is restricted to simple low-resolution tasks and not scalable to standard CL benchmark datasets . Representational learning . A large number of works have addressed the unsupervised learning problem in the standard machine learning framework . Specifically , contrastive learning frameworks ( He et al. , 2020 ; Chen et al. , 2020a ; Grill et al. , 2020 ; Chen et al. , 2020b ; c ) that learn the representations by measuring the similarities of positive and negative pairs have gained a lot of attention in the community . However , all these methods require large batches and negative sample pairs , which restrict the scalability of these networks . Chen & He ( 2021 ) tackled these limitations and proposed SimSiam , that use standard Siamese networks ( Bromley et al. , 1994 ) with the stop-gradient operation to prevent the collapsing of Siamese networks to a constant . Recently , Zbontar et al . ( 2021 ) formulated an objective that pushes the cross-correlation matrix between the embeddings of distorted versions of a sample closer to the identity matrix . However , all these methods assume the presence of large datasets for pre-training , which is impractical in real-world applications . In contrast , we tackle the problem of incremental representational learning and learn the representations sequentially while maximizing task adaptation and minimizing catastrophic forgetting . 3 PRELIMINARIES . 3.1 PROBLEM SETUP . We consider the continual learning setting , where we learn on a continuum of data consisting of T tasks T1 : T = ( T1 . . . TT ) . In supervised continual learning , each task consists a task descriptor τ ∈ { 1 . . . T } and its corresponding dataset Dτ = { ( xi , τ , yi , τ ) nτi=1 } with nτ examples . Each inputpair ( xi , τ , yi , τ ) ∈ Xτ × Yτ , where ( Xτ , Yτ ) is an unkown data distribution . Let us consider a network fΘ : Xτ → RD parametrized by Θ = { wl } l=Ll=1 , where RD and L denote D-dimensional embedding space and number of layers respectively . The classifier is denoted by hψ : RD → Yτ . The network error using cross entropy loss ( CE ) for SCL with finetuning can be formally defined as : LFINETUNESCL = CE ( hψ ( fΘ ( xi , τ ) , τ ) , yi , τ ) . ( 1 ) In this work , we assume the absence of label supervision during training and focus on unsupervised continual learning . In particular , each task consists of Uτ = { ( xi , τ ) nτi=1 } , xi , τ ∈ Xτ with nτ examples . Our aim is to learn the representations fΘ : Xτ → RD on a sequence of tasks while preserving the knowledge of the previous tasks . We introduce the representation learning framework and propose LUMP in Section 4 to learn unsupervised representations while effectively mitigating catastrophic forgetting . 3.2 LEARNING PROTOCOL AND EVALUATION METRICS . Currently , the traditional continual learning strategies follow the standard training protocol , where we learn the network representations fΘ : Xτ → Yτ on a sequence of tasks . In contrast , our goal is to learn the feature representations fΘ : Xτ → RD , so we follow a two-step learning protocol to obtain the model predictions . First , we pre-train the representations on a sequence of tasks T1 : T = ( T . . . TT ) to obtain the representations . Next , we evaluate the quality of our pre-trained representations using a K-nearest neighbor ( KNN ) classifier ( Wu et al. , 2018 ) following the setup in Chen et al . ( 2020a ) ; Chen & He ( 2021 ) ; Zbontar et al . ( 2021 ) . To validate knowledge transfer of the learned representations , we adopt the metrics from the SCL literature ( Chaudhry et al. , 2019b ; Mirzadeh et al. , 2020 ) . Let aτ , i denote the test accuracy of task i after learning task Tτ using a KNN on frozen pre-trained representations on task Tτ . More formally , we can define the metrics to evaluate the continually learned representations as follow : 1 . Average accuracy is the average test accuracy of all the tasks completed until the continual learning of task τ : Aτ = 1τ ∑τ i=1 aτ , i 2 . Average Forgetting is the average performance decrease of each task between its maximum accuracy and accuracy at the completion of training : F = 1T−1 ∑T−1 i=1 maxτ∈ { 1 , ... , T } ( aτ , i − aT , i )

Summary: This paper studies the problem of representation learning in an unsupervised continual learning(UCL) setting. It shows that the representation learned with UCL is more general than the one learned with supervised CL (SCL), and investigates why UCL is more robust to catastrophic forgetting than SCL by analyzing the similarity of learned features and visualizing loss landscape. The authors also propose to apply mixup technique to UCL setting and present a LUMP algorithm to further improve the performance of CL.This paper studies the problem of representation learning in an unsupervised continual learning(UCL) setting. It shows that the representation learned with UCL is more general than the one learned with supervised CL (SCL), and investigates why UCL is more robust to catastrophic forgetting than SCL by analyzing the similarity of learned features and visualizing loss landscape. The authors also propose to apply mixup technique to UCL setting and present a LUMP algorithm to further improve the performance of CL.