How Well Does Self-Supervised Pre-Training Perform with Streaming Data?

1 INTRODUCTION . Recent advances in self-supervised learning ( SSL ) ( He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Jure et al. , 2021 ) demonstrate competitive or even better transfer learning performance on various downstream tasks , compared with supervised pre-training . Although waiving the cost of human labeling , SSL usually requires massive unlabeled data to learn a powerful representation model and benefits from significantly large-scale pre-training data , e.g. , He et al . ( 2020 ) adopted billion-scale data to pre-train better SSL models . The common pre-training practice follows the joint training ( JT ) setup , where massive unlabeled data are collected together before model training . In reality , however , it is usually difficult to access a large amount of collective unlabeled data at once . Instead , real-world data are usually accessed in a streaming fashion , e.g. , data are generated and collected sequentially chunk by chunk ( Delange et al. , 2021 ) , or even decentralized and stored in different servers ( Lange et al. , 2020 ) ; such a learning setup is known as sequential training ( ST ) . Despite much research effort and promising results achieved by JT , it inevitably suffers from heavy data storage , prolonged training time , and finds itself incompetent when training data volume expands over time . For ST , on the other hand , a learner can be sequentially trained with disjoint data chunks , making it much more efficient than JT . How to effectively and efficiently pre-train a representation model under the ST setup has been an open problem . Despite the high efficiency , some continual learning research works ( Goodfellow et al. , 2013 ; Kirkpatrick et al. , 2017 ; Rebuffi et al. , 2017 ) have shown that ST with supervised models tends to suffer from catastrophic forgetting ( McCloskey & Cohen , 1989 ) , having a significant performance degradation on the historical data chunks . Unlike the case of continual learning tasks , in pre-training tasks , one expects the model to well generalize to downstream tasks rather than focusing only on the seen tasks ( Chen et al. , 2020a ) . Nevertheless , how well sequential self-supervised models perform on downstream tasks remains unclear . To fill the research gap , we provide a thorough empirical study on the transfer learning behavior of SSL models pre-trained with streaming data . In the pre-training stage , to mimic real-world data collection scenarios and for the ease of dissection of sequential SSL , we consider streaming data with different degrees of distribution shift . As shown in Figure 1 , we obtain four types of streaming data , including ( 1 ) the instance incremental sequence with negligible data distribution shift , by randomly splitting ImageNet-1K ( Russakovsky et al. , 2015 ) into four identically distributed ( IID ) data chunks , ( 2 ) the random class incremental sequence with moderate data distribution shift , by randomly splitting 1K classes of images into four independent chunks each with 250 classes , ( 3 ) the distant class incremental sequence with severe data distribution shift , by splitting 1K classes of data into four chunks while maximizing the semantical dissimilarity among chunks , and ( 4 ) the domain incremental sequence with severe domain distribution shift , by taking each domain of data in DomainNet ( Peng et al. , 2019 ) as a data chunk . As for the evaluation , we consider three downstream tasks following Ericsson et al . ( 2021 ) , including few-shot evaluation and linear evaluation ( also named many-shot classification ) on 12 image classification datasets ( Kornblith et al. , 2019b ) , and the Pascal VOC ( Everingham et al. , 2010 ) detection task . Through extensive experiments with more than 500 pre-trained models , we thoroughly investigate key roles in sequential SSL , including streaming data , downstream tasks and datasets , continual learning methods , SSL methods , and the method efficiency in terms of time and storage . We also thoroughly investigate the knowledge forgetting behavior of sequential SSL and SL models and provide a comprehensive empirical analysis of the underlying reason . To the best of our knowledge , we are among the first to explore the sequential self-supervised pretraining setting and the first to provide a thorough empirical study on self-supervised pre-training with streaming data . We summarize the takeaways as well as our contributions as : ( 1 ) . Sequential SSL models exhibit the on-par transfer learning performance as joint SSL models on streaming data with negligible or mild distribution shift . As for streaming data with severe distribution shifts or longer sequences , i.e. , the distant class incremental sequence , evident performance gaps exist between sequential SSL and joint SSL models . Such performance gaps , however , can be mitigated effectively and efficiently with unsupervised parameter regularization ( Aljundi et al. , 2018 ) and simple data replay . ( 2 ) . Based on the above finding , we conclude that the standard joint training paradigm may be unnecessary for SSL pre-training . Instead , sequential SSL is performance-competitive but more time-efficient and storage-saving and is well worth considering as the practical practice for self-supervised pre-training with streaming data . ( 3 ) . Compared with supervised learning ( SL ) models , SSL models consistently show smaller performance gaps between ST and JT . Our comprehensive investigation of learned representations demonstrates that sequential SSL models are less prone to catastrophic forgetting than SL models . ( 4 ) . Through the empirical analysis on the sharpness of minima in the loss landscape , we find that SSL models have wider minima than SL models , which we hypothesize is the reason for less forgetting of SSL models . 2 RELATED WORK . Self-supervised learning ( SSL ) . SSL learns useful features by solving various pretext tasks using supervisions generated from unlabeled training data , e.g. , predicting rotations ( Gidaris et al. , 2018 ) , predicting cluster assignments ( Caron et al. , 2018 ) , and solving instance discrimination ( Wu et al. , 2018 ; Chen et al. , 2020a ; He et al. , 2020 ; Grill et al. , 2020 ) . To achieve better performance in the downstream task , recent studies of SSL have made efforts in either upstream pre-training or downstream transfer . Previous works ( Caron et al. , 2019 ; He et al. , 2020 ) have leveraged especially large datasets for pre-training , such as YFCC 100M ( Thomee et al. , 2016 ) and Instagram 1B ( Mahajan et al. , 2018 ) . Some recent works ( Gururangan et al. , 2020 ; Reed et al. , 2021 ) propose to pre-train with the downstream dataset for a better transfer . Our work still focuses on the downstream-agnostic model pre-training . However , in realistic scenarios , access to massive data is often streaming , and how to perform SSL with streaming data has not been studied before , motivating our work . Continual learning . Existing studies of continual learning ( CL ) ( Delange et al. , 2021 ) mainly focus on supervised tasks and can be summarized into three categories , including regularization , replay , and parameter-isolation . In regularization-based CL , knowledge preserving is achieved by regularizing the parameter posterior of the new task not to deviate drastically from the prior ( Aljundi et al. , 2018 ; Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) . Replay-based CL methods overcome forgetting by saving samples of previous tasks in a replay buffer ( Rebuffi et al. , 2017 ; Rolnick et al. , 2019 ; Lopez-Paz & Ranzato , 2017 ; Wang et al. , 2021 ) and using them to regularize the learning of new tasks . Last , isolation-based CL methods leverage different parameters for learning each task to preserve the learned knowledge ( Serra et al. , 2018 ; Mallya & Lazebnik , 2018 ) . Although works ( Rao et al. , 2019 ; Aljundi et al. , 2019 ) explore continual learning for some specific unsupervised tasks , few have studied the transfer learning performance of sequential self-supervised models . 3 PROBLEM SETTING . In pre-training tasks , we train representation models on large-scale datasets , such as ImageNet ( Russakovsky et al. , 2015 ) , and evaluate the transferability of learned representations on various downstream tasks ( Chen et al. , 2020a ) . In our empirical study , we adopt the prevailing MoCo-v2 ( Chen et al. , 2020c ) method to pre-train SSL models with diverse streaming data . Types of streaming data . In pre-training , we consider streaming data with various distribution shifts to mimic practical data collection scenarios . As shown in Figure 1 , each type of streaming data consists of sequential and disjoint data chunks , while collective data cover all available data . ( 1 ) In the instance incremental sequence , streaming data chunks are almost IID , which simulates the scenario where data are continually collected under the same condition . In this case , there is negligible distribution shift among sequential data chunks . ( 2 ) In the random class incremental sequence , data in disjoint chunks belong to different classes , which mimics the scenario where data are collected by random keyword search on the Internet ( Parisi et al. , 2019 ) . Here the distribution shift is moderate . ( 3 ) The distant class incremental sequence is similar to the random class incremental sequence except that the semantic gaps between sequential data chunks in the distant sequence are larger , i.e. , images from different data chunks are semantically dissimilar . This data sequence has severe distribution shift between chunks . It mimics the scenario where data are crawled from websites with different subjects . ( 4 ) In the domain incremental sequence , data chunks are collected from different domains with severe domain distribution shift . A typical example is that large-scale autonomous driving data in Han et al . ( 2021 ) are collected in different domains , such as different weather conditions and cities , but share similar classes . The first three types of streaming data are designed with ImageNet ( Russakovsky et al. , 2015 ) , while the domain incremental sequence consists of five domains in DomainNet ( Peng et al. , 2019 ) . See Appendix A.1 for a detailed description . Model pre-training . With these streaming data , we study both sequential training ( ST ) and joint training ( JT ) for model pre-training . As illustrated in Figure 1 , in sequential training , a model is sequentially trained with streaming data chunks , while in joint training , a model is repeatedly retrained with collective data , i.e. , all seen data chunks . Moreover , we compare SSL with supervised learning ( SL ) and mainly study the following pre-trained models : sequentially trained SSL models ( SSL-ST ) , jointly trained SSL models ( SSL-JT ) , sequentially trained SL models ( SL-ST ) , and jointly trained SL models ( SL-JT ) . See Appendix A.2 for details of the pre-training stage . Transfer to downstream tasks . We evaluate the transfer learning performance of pre-trained models using three typical downstream tasks : many-shot classification , few-shot classification , and object detection . Following Chen et al . ( 2020a ) , we consider 12 classification datasets for downstream evaluation . Specifically , we conduct many-shot classification on all the above 12 datasets but conduct few-shot classification on 11 datasets except VOC2007 , following Ericsson et al . ( 2021 ) . In both types of classification tasks , representations are fixed for evaluation . In addition , we evaluate pre-trained models on the PASCAL VOC detection dataset following He et al . ( 2020 ) . See Appendix A.4 for more details of the downstream evaluation .

Summary: The authors of this submission worked on an interesting and quite practical task, i.e., to study the self-supervised behavior under the continual learning setup. To this end, the authors studied four types of incremental-learning settings, i.e., Instance incremental sequence, Random class incremental sequence, Distant class incremental sequence, Domain incremental sequence. The four types are designed to reflect different incremental-learning scenarios with different semantics.The authors have also conducted extensive empirical validations, including pre-training >500 models on four categories of pre-training streaming data from ImageNet and DomainNet, and evaluating them on three types of downstream tasks and 12 different downstream datasets.