How Well Does Self-Supervised Pre-Training Perform with Streaming Data?

1 INTRODUCTION . Recent advances in self-supervised learning ( SSL ) ( He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Jure et al. , 2021 ) demonstrate competitive or even better transfer learning performance on various downstream tasks , compared with supervised pre-training . Although waiving the cost of human labeling , SSL usually requires massive unlabeled data to learn a powerful representation model and benefits from significantly large-scale pre-training data , e.g. , He et al . ( 2020 ) adopted billion-scale data to pre-train better SSL models . The common pre-training practice follows the joint training ( JT ) setup , where massive unlabeled data are collected together before model training . In reality , however , it is usually difficult to access a large amount of collective unlabeled data at once . Instead , real-world data are usually accessed in a streaming fashion , e.g. , data are generated and collected sequentially chunk by chunk ( Delange et al. , 2021 ) , or even decentralized and stored in different servers ( Lange et al. , 2020 ) ; such a learning setup is known as sequential training ( ST ) . Despite much research effort and promising results achieved by JT , it inevitably suffers from heavy data storage , prolonged training time , and finds itself incompetent when training data volume expands over time . For ST , on the other hand , a learner can be sequentially trained with disjoint data chunks , making it much more efficient than JT . How to effectively and efficiently pre-train a representation model under the ST setup has been an open problem . Despite the high efficiency , some continual learning research works ( Goodfellow et al. , 2013 ; Kirkpatrick et al. , 2017 ; Rebuffi et al. , 2017 ) have shown that ST with supervised models tends to suffer from catastrophic forgetting ( McCloskey & Cohen , 1989 ) , having a significant performance degradation on the historical data chunks . Unlike the case of continual learning tasks , in pre-training tasks , one expects the model to well generalize to downstream tasks rather than focusing only on the seen tasks ( Chen et al. , 2020a ) . Nevertheless , how well sequential self-supervised models perform on downstream tasks remains unclear . To fill the research gap , we provide a thorough empirical study on the transfer learning behavior of SSL models pre-trained with streaming data . In the pre-training stage , to mimic real-world data collection scenarios and for the ease of dissection of sequential SSL , we consider streaming data with different degrees of distribution shift . As shown in Figure 1 , we obtain four types of streaming data , including ( 1 ) the instance incremental sequence with negligible data distribution shift , by randomly splitting ImageNet-1K ( Russakovsky et al. , 2015 ) into four identically distributed ( IID ) data chunks , ( 2 ) the random class incremental sequence with moderate data distribution shift , by randomly splitting 1K classes of images into four independent chunks each with 250 classes , ( 3 ) the distant class incremental sequence with severe data distribution shift , by splitting 1K classes of data into four chunks while maximizing the semantical dissimilarity among chunks , and ( 4 ) the domain incremental sequence with severe domain distribution shift , by taking each domain of data in DomainNet ( Peng et al. , 2019 ) as a data chunk . As for the evaluation , we consider three downstream tasks following Ericsson et al . ( 2021 ) , including few-shot evaluation and linear evaluation ( also named many-shot classification ) on 12 image classification datasets ( Kornblith et al. , 2019b ) , and the Pascal VOC ( Everingham et al. , 2010 ) detection task . Through extensive experiments with more than 500 pre-trained models , we thoroughly investigate key roles in sequential SSL , including streaming data , downstream tasks and datasets , continual learning methods , SSL methods , and the method efficiency in terms of time and storage . We also thoroughly investigate the knowledge forgetting behavior of sequential SSL and SL models and provide a comprehensive empirical analysis of the underlying reason . To the best of our knowledge , we are among the first to explore the sequential self-supervised pretraining setting and the first to provide a thorough empirical study on self-supervised pre-training with streaming data . We summarize the takeaways as well as our contributions as : ( 1 ) . Sequential SSL models exhibit the on-par transfer learning performance as joint SSL models on streaming data with negligible or mild distribution shift . As for streaming data with severe distribution shifts or longer sequences , i.e. , the distant class incremental sequence , evident performance gaps exist between sequential SSL and joint SSL models . Such performance gaps , however , can be mitigated effectively and efficiently with unsupervised parameter regularization ( Aljundi et al. , 2018 ) and simple data replay . ( 2 ) . Based on the above finding , we conclude that the standard joint training paradigm may be unnecessary for SSL pre-training . Instead , sequential SSL is performance-competitive but more time-efficient and storage-saving and is well worth considering as the practical practice for self-supervised pre-training with streaming data . ( 3 ) . Compared with supervised learning ( SL ) models , SSL models consistently show smaller performance gaps between ST and JT . Our comprehensive investigation of learned representations demonstrates that sequential SSL models are less prone to catastrophic forgetting than SL models . ( 4 ) . Through the empirical analysis on the sharpness of minima in the loss landscape , we find that SSL models have wider minima than SL models , which we hypothesize is the reason for less forgetting of SSL models . 2 RELATED WORK . Self-supervised learning ( SSL ) . SSL learns useful features by solving various pretext tasks using supervisions generated from unlabeled training data , e.g. , predicting rotations ( Gidaris et al. , 2018 ) , predicting cluster assignments ( Caron et al. , 2018 ) , and solving instance discrimination ( Wu et al. , 2018 ; Chen et al. , 2020a ; He et al. , 2020 ; Grill et al. , 2020 ) . To achieve better performance in the downstream task , recent studies of SSL have made efforts in either upstream pre-training or downstream transfer . Previous works ( Caron et al. , 2019 ; He et al. , 2020 ) have leveraged especially large datasets for pre-training , such as YFCC 100M ( Thomee et al. , 2016 ) and Instagram 1B ( Mahajan et al. , 2018 ) . Some recent works ( Gururangan et al. , 2020 ; Reed et al. , 2021 ) propose to pre-train with the downstream dataset for a better transfer . Our work still focuses on the downstream-agnostic model pre-training . However , in realistic scenarios , access to massive data is often streaming , and how to perform SSL with streaming data has not been studied before , motivating our work . Continual learning . Existing studies of continual learning ( CL ) ( Delange et al. , 2021 ) mainly focus on supervised tasks and can be summarized into three categories , including regularization , replay , and parameter-isolation . In regularization-based CL , knowledge preserving is achieved by regularizing the parameter posterior of the new task not to deviate drastically from the prior ( Aljundi et al. , 2018 ; Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) . Replay-based CL methods overcome forgetting by saving samples of previous tasks in a replay buffer ( Rebuffi et al. , 2017 ; Rolnick et al. , 2019 ; Lopez-Paz & Ranzato , 2017 ; Wang et al. , 2021 ) and using them to regularize the learning of new tasks . Last , isolation-based CL methods leverage different parameters for learning each task to preserve the learned knowledge ( Serra et al. , 2018 ; Mallya & Lazebnik , 2018 ) . Although works ( Rao et al. , 2019 ; Aljundi et al. , 2019 ) explore continual learning for some specific unsupervised tasks , few have studied the transfer learning performance of sequential self-supervised models . 3 PROBLEM SETTING . In pre-training tasks , we train representation models on large-scale datasets , such as ImageNet ( Russakovsky et al. , 2015 ) , and evaluate the transferability of learned representations on various downstream tasks ( Chen et al. , 2020a ) . In our empirical study , we adopt the prevailing MoCo-v2 ( Chen et al. , 2020c ) method to pre-train SSL models with diverse streaming data . Types of streaming data . In pre-training , we consider streaming data with various distribution shifts to mimic practical data collection scenarios . As shown in Figure 1 , each type of streaming data consists of sequential and disjoint data chunks , while collective data cover all available data . ( 1 ) In the instance incremental sequence , streaming data chunks are almost IID , which simulates the scenario where data are continually collected under the same condition . In this case , there is negligible distribution shift among sequential data chunks . ( 2 ) In the random class incremental sequence , data in disjoint chunks belong to different classes , which mimics the scenario where data are collected by random keyword search on the Internet ( Parisi et al. , 2019 ) . Here the distribution shift is moderate . ( 3 ) The distant class incremental sequence is similar to the random class incremental sequence except that the semantic gaps between sequential data chunks in the distant sequence are larger , i.e. , images from different data chunks are semantically dissimilar . This data sequence has severe distribution shift between chunks . It mimics the scenario where data are crawled from websites with different subjects . ( 4 ) In the domain incremental sequence , data chunks are collected from different domains with severe domain distribution shift . A typical example is that large-scale autonomous driving data in Han et al . ( 2021 ) are collected in different domains , such as different weather conditions and cities , but share similar classes . The first three types of streaming data are designed with ImageNet ( Russakovsky et al. , 2015 ) , while the domain incremental sequence consists of five domains in DomainNet ( Peng et al. , 2019 ) . See Appendix A.1 for a detailed description . Model pre-training . With these streaming data , we study both sequential training ( ST ) and joint training ( JT ) for model pre-training . As illustrated in Figure 1 , in sequential training , a model is sequentially trained with streaming data chunks , while in joint training , a model is repeatedly retrained with collective data , i.e. , all seen data chunks . Moreover , we compare SSL with supervised learning ( SL ) and mainly study the following pre-trained models : sequentially trained SSL models ( SSL-ST ) , jointly trained SSL models ( SSL-JT ) , sequentially trained SL models ( SL-ST ) , and jointly trained SL models ( SL-JT ) . See Appendix A.2 for details of the pre-training stage . Transfer to downstream tasks . We evaluate the transfer learning performance of pre-trained models using three typical downstream tasks : many-shot classification , few-shot classification , and object detection . Following Chen et al . ( 2020a ) , we consider 12 classification datasets for downstream evaluation . Specifically , we conduct many-shot classification on all the above 12 datasets but conduct few-shot classification on 11 datasets except VOC2007 , following Ericsson et al . ( 2021 ) . In both types of classification tasks , representations are fixed for evaluation . In addition , we evaluate pre-trained models on the PASCAL VOC detection dataset following He et al . ( 2020 ) . See Appendix A.4 for more details of the downstream evaluation .

Summary: This paper studies how models perform on downstream image classification tasks when they are trained via self-supervision using streaming data. Streaming data refers to a sequential training setting  (ST) where the entire training data is not available for training at the same time but can be only used in disjoint chunks. When training on one chunk of data, the previous chunks are unavailable for training during this sequential training. When the entire training data is available for training at the same time, it is called joint training (JT). The key insight of the work is that in the presence of little to moderate drift in the data distribution across chunks, the downstream task performance of self-supervised training in sequential setting (SSL-ST) is comparable to that of self-supervised training in joint setting (SSL-JT). While in the presence of severe drift in data distribution across chunks, SSL-ST performs worse than SSL-JT but through the application of continual learning approaches like data replay and MAS (Aljundi et al, 2018), this performance gap can be reduced to a large extent. Moreover, when training is done in a supervised manner (SL), sequential setting (SL-ST) performs much worse than joint setting (SL-JT)  The work experiments on four kinds of sequential settings with varying degrees of data distribution drift across consecutive chunks - (1) instance incremental sequence, where each chunk of unlabeled data is derived from the same set of classes having i.i.d. samples in each chunk, (2) random class incremental sequence, where data in each chunk belongs to a different set of randomly chosen classes, (3) distant class incremental sequence, where each chunk belongs to a different set of classes such that there is a larger semantic gap between the set of classes and (4) domain incremental sequence, where the chunks belong to different domains thereby exhibiting severe distribution shift. For the first three settings, Imagenet-1k is used which is split into 4 chunks as per the setting. For the fourth setting, DomainNet is used. MoCo-v2 is primarily used in the empirical analysis along with some experiments on BYOL are also conducted to show that the trends are observed in other SSL approaches as well.  The work also conducts an analysis on the amount of forgetting using the backward and forward transfer analysis which shows that compared to supervised setting where there is significant amount of catastrophic forgetting, self-supervised sequential setting has much lower forgetting. Further analysis via Centered Kernel Alignment (CKA) suggests that compared to supervised setting, SSL-ST has a higher feature similarity between two sequential models. Even the feature similarity of SSL-ST with SSL-JT is higher than the corresponding similarity of SL-ST with SL-JT.