How Well Does Self-Supervised Pre-Training Perform with Streaming Data?
Authors: Dapeng Hu
Shipeng Yan
Qizhengqiu Lu
Lanqing HONG
Hailin Hu
Yifan Zhang
Zhenguo Li
Xinchao Wang
Jiashi Feng
Venue: ICLR 2022 Poster (2022)
Keywords: Pre-Training
Representation Learning
Continual Learning
Self-Supervised Learning

Abstract: Prior works on self-supervised pre-training focus on the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained. Unfortunately, such a problem setting is often impractical if not infeasible since many real-world tasks rely on sequential learning, e.g., data are decentralized or collected in a streaming fashion. In this paper, we conduct the first thorough and dedicated investigation on self-supervised pre-training with streaming data, aiming to shed light on the model behavior under this overlooked setup. Specifically, we pre-train over 500 models on four categories of pre-training streaming data from ImageNet and DomainNet and evaluate them on three types of downstream tasks and 12 different downstream datasets. Our studies show that, somehow beyond our expectation, with simple data replay or parameter regularization, sequential self-supervised pre-training turns out to be an efficient alternative for joint pre-training, as the performances of the former are mostly on par with those of the latter. Moreover, catastrophic forgetting, a common issue in sequential supervised learning, is much alleviated in sequential self-supervised learning (SSL), which is well justified through our comprehensive empirical analysis on representations and the sharpness of minima in the loss landscape. Our findings, therefore, suggest that, in practice, for SSL, the cumbersome joint training can be replaced mainly by sequential learning, which in turn enables a much broader spectrum of potential application scenarios. 

Published as a conference paper at ICLR 2022
How WELL DOES SELF-SUPERVISED PRE-TRAINING PERFORM WITH STREAMING DATA?
Dapeng Hu*! Shipeng Yan
Qizhengqiu Lu Lanqing Hong* Hailin Hu
Yifan Zhang'
Zhenguo Li
Xinchao Wang' Jiashi Feng'
National University of Singapore
ShanghaaTTec University Huuaei Noah's 's Ark Lab
3 AARC, Huawei Technologies
ABSTRACT
Prior works on self-supervised pre-training focus the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained. Unfortunately, such a problem setting is often im- practical if not infeasible since many real-world tasks rely on sequential learning, e.g., data decentralized we conduct the first thorough and dedicated investigation on self-supervised pre- training with streaming data, aiming shed light on the model behavior under this overlooked setup. Specifically, we pre-train over 500 models on four cate- gories of pre-training streaming data from ImageNet and DomainNet and evaluate them on three types of downstream tasks and 12 different downstream datasets. Our studies show that, somehow beyond our expectation, with simple data replay or parameter regularization, sequential self-supervised pre-training turns out to be an efficient alternative for joint pre- pre-training training, as the performances the former are mostly par with those of the latter. Moreover, catastrophic forgetting, a common issue in sequential supervised learning. is much alleviated sequential self- self-supervised learning (SSL), which is well justified through comprehensive empirical analysis on representations and the sharpness of minima in the loss land- scape. Our findings, therefore, suggest that, in practice, for SSL, the cumbersome joint training can be replaced mainly by sequential learning, which in turn enables a much broader spectrum of potential application scenarios.
collected in streaming fashion. In this paper,
INTRODUCTION
Recent advances in self-supervised learning (SSL) (He et al., 2020; Grill et al., 2020; Caron et al., 2020; Jure et al., 2021) demonstrate competitive or even better transfer learning performance on various downstream tasks, compared with supervised pre-training. Although waiving the cost of human labeling, SSL usually requires massive unlabeled data to learn a powerful representation model and benefits from significantly large-scale pre-train pre-training data, e.g., He et al. (2020) adopted billion-scale data to pre-train better SSL models. The common pre-training practice follows the joint training (JT) setup, where massive unlabeled data are collected together before model training- In reality, however, it is usually difficult to access large amount of collective unlabeled data at once. Instead, real-world data are usually accessed in a streaming fashion, e.g., data are generated and collected sequentially chunk by chunk (Delange et al., 2021), or even decentralized and stored in different servers (Lange et al., 2020); such a learning setup is known as sequential training (ST). Despite much research effort and promising results achieved by JT, it inevitably suffers from heavy data storage, prolonged training time, and finds itself incompetent when training data volume expands over time. For ST, the other hand, learner be sequentially trained with disjoint data chunks, making it much more efficient than JTT.
How to effectively and efficiently pre-train representation model under the ST setup has been an open problem. Despite the high efficiency, some continual learning research works (Goodfellow al., 2013; Kirkpatrick et al., 2017; Rebuffi al., 2017) have shown that ST with supervised mod- els tends to suffer from catastrophic forgetting (McCloskey & Cohen, 1989), having a significant
"Contributed equally: dapeng. hu@u. nus. edu and yanshpEshangha tech.edu.cm

---

Published as a conference paper at ICLR 2022
performance degradation the historical data chunks. Unlike the case of continual learning tasks, in pre-training tasks, one expects the model to well generalize to downstream tasks rather than fo- cusing only the seen tasks (Chen et al., 2020a). Nevertheless, how well sequential selff-uupervied models perform on downstream tasks remains unclear.
Parent
Child
Class
Class
Class 7
Class
Domains
Chunk 1
Chunk 2
Chunk
Chunk 4
Instance
incremental sequence
Random class
incremental sequence
Distant class
Incremental sequence
Domain
incremental sequence
Collective data
Streaming
Figure I: Illustration of streaming data and the corresponding collective data. Different colors denote different classes, and border types distinguish different domains. We use the WordNet Tree (Miller, 1998) measure the semantic similarity of classes. Classes having the same parent or ancestor in WordNet, marked with similar colors, share similar semantics in the class semantic tree.
To fill the research gap, provide thorough empirical study self-supervised pre-training with streaming data. In the pre-training stage, to mimic wollw data collection scenarios and for the better dissection of sequential SSL, consider streaming data with different degrees of distribution shifts. As shown Figure 1, we obtain four types streaming data, including the instance incre- mental sequence with negligible data distribution shifts, by randomly splitting ImageNet-IK 1K (Rus- sakovsky et al., 2015) into four independent and identically distributed (IID) data chunks, the random class incremental sequence with moderate data distribution shifts, by randomly splitting IK classes of images into four disjoint chunks each with 250 classes, the distant class incremental sequence with severe data distribution shifts, by splitting 1K classes of data into four chunks while maximiz- ing the semantical dissimilarity among chunks, and the domain incremental sequence with severe domain distribution shifts, by taking each domain in DomainNet (Peng et al., 2019) as a data chunk.
As for the evaluation, we consider three downstream tasks following Ericsson et al. (2021), includ- ing few-shot evaluation and linear evaluation (also named many-shot classification) on 12 image classification datasets (Kornblith et al., 2019b), and the Pascal VOC (Everingham et al., 2010) de- tection task. Through extensive experiments with more than 500 pre-trained models, we thoroughly investigate key roles in sequential SSL, including streaming data, downstream tasks and datasets, continual learning methods, SSL methods, and the method efficiency in terms of time and storage. We also thoroughly investigate the knowledge forgetting behavior of sequential SSL and supervised learning (SL. models and provide a comprehensive empirical analysis the underlying reason.
To the best of our knowledge, training setting and the first to provide a thorough empirical study on self-supervised pre-training i). Sequential SSL
among the first explore the sequential self-supervised pre-
with streaming data. We summarize the takeaways well as our contributions models exhibit the on-par transfer learning performance as joint SSL models on streaming data with negligible mild distribution shifts. As for streaming data with severe distribution shifts or longer sequences. i.c., the distant class incremental sequence, evident performance gaps exist between sequential SSL and joint SSL models. Such performance gaps, however, can be mitigated effectively and efficiently with unsupervised parameter regularization (Aljundi et al., 2018) and simple data replay. ii) Based on the above finding, we conclude that the standard joint training paradigm may be unnecessary for SSL pre-training. Instead, sequential SSL is performance-competic time-efficient and gorage-ssvingg and well worth considering as the practical practice for self- supervised pre-training with streaming data. fii). Compared with supervised learning (SL) models, SSL models consistently show smaller performance gaps between ST and JT. Our comprehensive investigation of learned representations demonstrates that sequential SSL models are less prone
but more
2

---

Published as a conference paper at ICLR 2022
catastrophic forgetting than SL models. iv). Through the empirical analysis on the sharpness of minima in the loss landscape, we find that SSL models have wider minima than SL models, which we hypothesize is the reason for less forgetting of SSL models.
RELATED WORK
Self-supervieed learning (SSL). SSL learns useful features by solving various pretext tasks using supervisions generated from unlabeled training data, e.g., predicting rotations (Gidaris et al., 2018), predicting cluster assignments (Caron al., 2018), and solving instance discrimination (Wu et al., 2018; Chen et al., 2020a; He et al., 2020; Grill et al., 2020). To achieve better performance in the downstream task, recent studies of SSL have made efforts in either upstream pre-training or downstream transfer (Zhang et al., 2021). Previous works (Caron et al., 2019; He et al., 2020) have leveraged especially large datasets for pre-training, such as YFCC 100M (Thomee al., 2016) and Instagram 1B (Mahajan et al., 2018). Some recent works (Gururangan et al., 2020; Reed et al., 2021) propose to pre-train with the downstream dataset for a better transfer. Our work still focuses on the downstream-agnnsste model pre-training. However, in realistic scenarios, access to massive data is often streaming, and how to perform SSL with streaming data has not been studied before, motivating our work.
Continual learning, Existing studies of continual learning (CL) (Delange et al., 2021) mainly focus on supervised tasks and can be summarized into three categories, including regularization, replay, and parameter regularizing the parameter posterior of the new task not to deviate drastically from the prior (Aljundi et al., 2018; Kirkpatrick et al., 2017; Zenke et al., 2017). Replay-based CL methods overcome forgetting by saving samples of previous tasks in replay buffer (Rebuffi et al., 2017; Rolnick al., 2019; Wang et al., 2021; Yan et al., 2021b) and using them to regularize the learning of new tasks. Last, isolation-based CL methods leverage different parameters for learning each task to preserve the learned knowledge (Serra et al., 2018; Mallya & Lazebnik, 2018). Although works (Rao et al., 2019: Aljundi et al., 2019) explore continual learning for some specific unsupervised tasks, few have studied the transfer learning performance of sequential self-supervise models.
In regularization based CL, knowledge preserving is achieved by
3 PROBLEM SETTING
In pre-training tasks, we train representation models on large-scale datasets, such as ImageNet (Rus- sakovsky et al., 2015), and evaluate the transferability of learned representations on various down- stream tasks (Chen et al., 2020a). In our empirical study, we adopt the prevailing MoCo-v2 (Chen et al., 2020c) method to pre-train SSL models with diverse streaming data.
Types of streaming data. In pre-training, we consider streaming data with various distribution shifts to mimie practical data collection scenarios. As shown in Figure I, each type of streaming data consists of sequential and disjoint data chunks, while collective data cover all available data. In the instance incremental sequence, streaming data chunks are almost IID, which simulates the sce- nario where data continually collected under the same condition. In this case, there is negligible distribution shift among sequential data chunks. In the random class incremental sequence, data in disjoint chunks belong to different classes, which mimics the scenario where data are collected by random keyword search on the Internet (Parisi et al., 2019). Here the distribution shift is moderate. The distant class incremental sequence is similar to the random class incremental sequence except that the semantic gaps between sequential data chunks in the distant sequence are larger, i.e., images from different data chunks are semantically dissimilar. This data sequence has severe distribution shifts between chunks. It mimics the scenario where data are crawled from websites with different subjects. In the domain incremental sequence, data chunks are collected from different domains with severe domain distribution shifts. A typical example is that large-scale autonomous driving data in Han et al. (2021) are collected in different domains, such as different weather conditions and cities, but share similar classes. The first three types of streaming data are designed with Ima- geNet (Russakovsky et al., 2015), while the domain incremental sequence consists of five domains in DomainNet (Peng et al., 2019). See Appendix A.1 for detailed description.
Model pre-training, With these streaming data, we study both sequential training (ST) and joint training (JT) for model pre-training. As illustrated in Figure 1, in sequential training, model is
3

---

Published as a conference paper at ICLR 2022
sequentially trained with streaming data chunks, while in joint training. model is repeatedly trained with collective data, i.., all seen data chunks. Moreover, we compare SSL with supervised learning (SL) and mainly study the following pre- pre-trained models: sequentially trained SSL models (SSL-ST), jointly trained SSL models (SSI~J'), sequentially trained SL. models (SL-ST), and jointly trained SL models (SL-JTT). See Appendix A.2 for details of the pre-training stage.
Maryyars
Faee
5030
Figure 2: Linear and few-shot evaluation results of random class incremental sequence. On the left are the results of each dataset. On the right are averaged results across all left datasets.
Transfer to downstream tasks. We evaluate the transfer learning performance of pre-trained mod- els using three typical downstream tasks: many-shot classification, few-shot classification, and ob- ject detection. Following Chen et al. (2020a), consider 12 classification datasets for downstream evaluation. Specifically, we conduct many-shot classification on all the above 12 datasets but con- 11 datasets except VOC2007, following Eriesson et al. (2021). In
duct few-shot classification both types of classification tasks, representations are fixed for evaluation. In addition, we eval- uate pre-trained models the PASCAL VOC detection dataset following He al. (2020). See Appendix A.4 for more details of the downstream evaluation.
4 DISSECTION OF SEQUENTIAL SELF-SUPERVISED PRE-TRAINING
We pre-train representation models on four types of streaming data and evaluate pre-trained models on downstream datasets with three downstream evaluation tasks. Note that models pre-trained with imageNet-based based streaming data evaluated on all three downstream tasks. Models trained with the domain incremental sequence are only evaluated with few-shot classification, considering the size of DomainNet is only 1/5 ImageNet. We report downstream evaluation results of the ran- dom class incremental sequence, the distant class incremental sequence, and the domain incremen- tal sequence in Figure Figure and Figure 4, respectively. Results of the instance incremental sequence are illustrated in Appendix B. since the performance of SSL models on the instance in- cremental sequence similar to that on the random class incremental sequence without considering the different cases for SI. models. We also evaluate three types of ImageNet-based streaming data on object detection and illustrate results Figure 10 in Appendix.
4.1 How DOES TRANSFER LEARNING PERFORMANCE VARY WITH STREAMING DATA?
We first consider streaming data with various degrees of distribution shifts, i.c., streaming data with negligible distribution shifts such as the instance incremental sequence, streaming data with mod- erate distribution shifts such as the random class incremental sequence, and streaming data with severe distribution shifts such the distant class incremental sequence and the domain incremental sequence. As shown in Figures 2-4, on all types of streaming data, the performance of sequential SSL models generally increases with more streaming chunks, while sequential SL. models do not
4

---

Published as a conference paper at ICLR 2022
Arceta
Catec
Teves
Maryarot
Faee
09938
10077
Arcata
Poocshor
Fiee
GHA
09918
88
Figure 3: Linear and few-shot evaluation results of distant class incremental sequence. On the left are the results of each dataset. On the right are averaged results across all left datasets.
benefit from increasing data chunks. As for the performance each type of streaming data, quential SSL models surprisingly perform comparably to joint SSL models on streaming data with negligible and moderate distribution shifts. On streaming data with severe distribution shifts, the performance of sequential SSL models is evidently inferior to that of joint SSL models. The de- tection evaluation results Figure 10 in Appendix further support this observation. addition, we provide results of streaming data with longer chunks (8) and random distribution shifts in Ap- pendix B.2. Similarly, we find the long sequence leads to visible but not significant gaps between ST and JT models. In contrast, on all types of streaming data, sequential SL models perform espe- cially worse than joint SL models. The above observations denote that, unlike traditional continual learning tasks (Delange al., 2021), although faced with possible visible performance gaps between ST models and JT models, sequential SSL is still performance prommiiing in pre- pre-training tasks with streaming data.
4.2 ARE RESULTS CONSISTENT ACROSS DOWNSTREAM TASKS OR DATASETS?
In pre-training tasks, we pay attention to the generalization of the learned representations to new data or tasks rather than the performance on the training dataset. Taking a closer look the re- sults in Figures 2-4, we observe that, although joint SSL models achieve comparable performance to joint SL models linear evaluation, joint SL models significantly outperform joint SSL mod- els in few-shot evaluation. This observation is also demonstrated in (Tian et al., 2020; Ericsson et al., 2021). The main difference between the two evaluation protocols that linear evaluation involves more fine-tuning than few-shot evaluation, introduced in Appendix A.4. Therefore, the underlying reason for the observation is that supervised features are correlated with labels and more discriminative, thus easy to directly transfer downstream datasets similar to upstream pre-training data (DomainNet or ImageNet). For example, SL models dominate most few-shot object or scene classification tasks but fail on DTD (Cimpoi al., 2014). a texture classification dataset sharing no common classes with ImageNet or DomainNer. In contrast, self-supervised features are more gen- eralized and comprehensive, thus requiring more fine-tuning for desirable downstream transfer. In addition, on some downstream datasets, we have seemingly abnormal observations that ST models may outperform JT models and the model performance may drop with the increase of chunk num- ber. These phenomena are due to the so-called "negative transfer" (Wang et al., 2019), which is also discussed in other model pre-training studies (Newell & Deng, 2020; Gururangan et al., 2020). That is, pre-training with more data chunks does not necessarily benefit a specific downstream dataset if the added training data are irrelevant to the downstream dataset. See Appendix B.3 for a concrete example of "negative transfer" on Oxford-IIIV Pets (Parkhi et al., 2012) in pre- pre-training with stream- ing data. It is observed that sequential SSL models suffer less "negative transfer" than SL models and continual learning methods largely prevent "negative transfer".
5

---

Published as a conference paper at ICLR 2022
Airrotts
Callect
Flumes
Car5
DTD
SRc-OT whe
Food
CHAR20
Sin
Sund37
21,27
3L5T
Figure 4: Few-shot evaluation results of domain incremental sequence. On the left are the results of each dataset. On the right are averaged results across all left datasets.
4.3 Do CONTINUAL LEARNING METHODS HELP SEQUENTIAL SSL?
As observed in Section 4. there still exist obvious performance gaps between sequential SSL mod- els and joint SSL models, on streaming data with severe distribution shifts such as the distant class incremental sequence and the domain incremental sequence. To this end. we study whether contin- ual learning methods can help mitigate such gaps. Specifically, we investigate two classic methods in continual learning. i.c., data replay and MAS (Aljundi et al., 2018), which are effective defy knowledge forgetting in supervised classification tasks (Delange et al., 2021). When using data re- play, we randomly reserve 10% data from each seen data chunk and add them to the current data chunk for model pre-training. We also consider the combination of MAS and data replay, which is referred to as MAS+ in the experiments. We denote SSL models trained with data replay as SSL-ST w/Replay, SSL models trained with MAS SSL-ST w/MAS, and SSL models trained with both methods as SSL-ST w/MAS+. We report downstream evaluation results of the distant class incre- mental sequence in Figure 3 and results of the domain incremental sequence in Figure 4. As shown in Figure 4, data replay can totally eliminate the performance gaps between sequential SSL models and joint SSL models effectiveness of both continual learning methods in improving the transfer learning performance of sequential SSL models trained with streaming data with severe distribution shifts. In short, we find methods devised for supervised continual tasks are especially promising to make sequential SSL models perform comparably to joint SSL models on challenging streaming data. See Appendix A.3 for implementations of MAS and data replay in sequential SSL.
the domain incremental sequence. Results in Figure 3 also validate the
4.4 HOW ABOUT SSL. METHODS OTHER THAN MoCo?
For simplicity, we choose MoCo-v2 (Chen et al., 2020c) in experiments and demonstrate that To verify whether it also holds for other SSL methods,
quential SSL is performance-promising we train BYOL (Grill et al., 2020) models on the challenging distant class incremental sequence, both sequentially and jointly. Results of BYOL are shown in Figure 13 in Appendix B.4. Similar to MoCo-v2, there still exist visible performance gaps between sequential SSL models and corre- sponding joint SSL models. In contrast, SSL models exhibit much smaller performance gaps than SL models, which further validates the potential of sequential SSL in pre-training tasks.
4.5 ANALYSIS OF METHOO EFFICIENCY
We then discuss the time and memory consumption of different training methods of SSL, including sequential training (SSL-ST), ST with data replay (SSL-ST w/Replay), ST with MAS (SSL-ST w/MAS), ST with MAS and data replay (SSL-ST w/MAS+), and joint training (SSL-JT), As shown in Table 1, JT is very time-consuming especially when the data amount is large, while ST is able to save a large amount of time under sequential training scenarios. To be specifie, ST is about 2x faster than JT when there are 2 chunks of data, and is about 4x faster when the number of chunks 4. Moreover, when we use MAS and data replay to improve the performance of ST, the time consumption of SSL increases little but is still significantly faster than JT. As for storage consumption, we can observe a similar phenomenon as shown in Table 1. summary, sequential SSL is much more time-efficient and sarage-saving than JT, especially when the data amount is large or grows quickly. Such a result indicates that sequential SSL is a more favorable choice for real-world pre-training applications, where data come in sequentially and grow daily.

---

Published as a conference paper at ICLR 2022
Table 1: Resource efficiency of considered SSL pre-training methods. We take the distant class incremental sequence as an example and report the training time (h) and required storage (GB) of the model pre-trained with each data chunk. Note that all the following statistics are recorded under the same hardware environment. The lower value means better efficiency.
Table 2:
The comparison of pre-training meth- ods in terms of the transfer performance gap be- tween ST and JT models. We report the aver- aged accuracy gaps of linear evaluation across 12 downstream datasets. The lower, the better.
Accuracy gap Chunk
3
S1.ST (Instanee)
2.26
3.27
4.83
SSL-ST (Insiamce)
0.41
1.02
1.04
Time (Storage)/ / Chunk
2
SL-ST (Random)
5.63
8.73
10.68
SSL-ST (Random)
0.42
0.94
1.13
SSL-SST
16.5 (35)
16.5 (35)
16.6
SSL-ST W/Replay
17,0
18.5 (42)
(060
SL-ST (Distant)
7.77
12.50
15.75
ST WMAS
(35)
18.1 (35)
18.1 (35)
SSL-ST (Distant)
2.34
3.81
4.62
SSL-ST w/MAS+
(3.4(39)
24.4 (42)
26.4 (46)
SSL-ST w/MAS (Distant)
1.82
2.73
3.17
SSL-JI
(701
46.5 (105)
66.6
SSL-ST w/MAS+ (Distant)
1.47
2.01
2.10
Summary. We show the averaged accuracy gaps between ST models and the corresponding JT models under linear evaluation in Table 2, for both SSL and supervised learning (SL). On streaming data with negligible distribution shifts, SI. exhibits evident accuracy gaps while SSL has negligible gaps. On streaming data with moderate distribution shifts, SL exhibits larger accuracy gaps while still keeps the negligible gaps. On streaming data with severe distribution shifts, shows much larger accuracy gaps, while SSL shows mild accuracy gaps. But such accuracy gaps of SSL can be effectively mitigated with simple continual learning methods. To sum up, SSL exhibits significantly smaller performance gaps between ST models and JT models than SL. The above difference between SL and SSL models motivates us to further investigate the forgetting property in Section 5.
SELF-SUPERVISED MODELS FORGET LESS THAN SUPERVISED MODELS
In this section we first analyze the knowledge forgetting of previous tasks from two perspectives. In Section 5.1, 1 we evaluate the transfer ability of both SL and SSL representations via the standard backward and forward transfer analysis in continual learning (Lopez-Paz & Ranzato, 2017). In Section 5.2, we adopt the CKA similarity (Kornblith et al., 2019a) and image reconstruction from features (Zhao et al., 2020) to directly analyze the representation. Last but not least, in Section 5.3, we provide our empirically justified hypothesis for why SSL models forget less than SL models.
5.1 BACKWARD AND FORWARD TRANSFER ANALYSIS OF SEQUENTIAL LEARNING
Following (Lopez-Paz & Ranzato, 2017), we adopt the backward and forward transfer to assess the knowledge transfer in sequential learning. Backward transfer refers to the improvement of performance on previously learned chunks when learning new chunks, where large negative trans- fer is also known as catastrophic forgetting. For- ward transfer measures the improvement in per- formance the novel chunk with the accumu- lation of knowledge from previous chunks. For supervised continual learning, the performance is defined as the accuracy on the associated test set, which is meaningful due to the consis- tency of the training and test sets. Similarly, we conduct the whole analysis based on the per- formance on the pre-training data chunks, instead of performing the evaluation on downstream datasets. Specifically, for the ease of comparison between SL and SSL, we measure the perfor- mance by KNN classification accuracy on the representations of pre- traiirainig chunks, where the labels are provided just for evaluation, similar to Wu et al. (2018). Concretely, the backward transfer BWT = - Er E A3, - -A3, and forward transfer FWT -ー Σ A5,. Ay, metrics used in Yan al. (2021a) where T is the sequence length, Abe refers to the accuracy on the chunk using model learned step where the label space includes all observed classes up to chunk j. and Åy, means the accuracy with the model learned from scratch. The results are shown in Table 3, and
Table 3: Backward and forward transfer analy- sis of sequential learning.
BWT(%)
FWT(%)
Data
Method
Top-1
Top-5
Top-1
Top-5
SL
-9.45
-5.46
8.64
2.81
Instance
SSL
3.61
3.60
7.55
8.63
Random
SL
-20.63
-7.03
-0.34
0.01
SSL
-5.17
-1.36
11.05
4.52
SL.
-40.43
-28.66
4.90
0.47
Distant
SSL
-13.24
-11.06
11.01
3.66
7

---

Published as a conference paper at ICLR 2022
the implementation details are included in the Appendix C.2. We can obtain the following observa- tions about forgetting: i). Learning method: SSL itself is less prone to catastrophic forgetting than SL, especially that SSL achieves positive backward transfer on the instance incremental sequence. illustrates that SSL. is more suitable for streaming data. ii). Types of streaming data: The model suffers progressively severe forgetting when the distribution shift increases for both SSL and SL. cases. iii). Example forgetting; It is observed that forgetting is less severe in top-S classification than top-1 classification, which indicates that the knowledge is not fully forgotten.
5.2 REPRESENTATION MEMORIZATION ANALYSIS SEQUENTIAL. LEARNING
How do features forget in sequential training? We study how learned features forget in sequential training via Centered Kernel Alignment (CKA) (Kornblith et al.,
Chunk
0.05
0:75
-8
Chunk
0.78 0.77
0.86
-66
2019a). CKA is used to measure the similarity between two representations of the same given samples. See Ap- pendix C.3 for details of the CKA similarity. Specifi- cally, we randomly sample 50,000 images from the first data chunk on each type of streaming data. We use these samples and sequentially trained models for CKA sim- ilarity analysis. We report the CKA similarity values on three types of ImageNet-basedr streaming data in Figure 5. Each value is obtained by first extracting features of sam- ples with two different models and then computing the CKA feature similarity value between the two features. On all streaming data, we have three consistent observa- tions about the CKA similarity between sequential mod- els: i). SSL. models all exhibit higher features similarity to the initial model, compared with SI. models. ii). In general, SSL models show higher features similarity be- tween two sequential models in sequential training, com- pared with SL models. iii. Features similarity between two sequential models decrease on streaming data with more severe distribution shifts, for both SSL and SSL. These observations suggest that features of SSL models forget less and evolve more slowly than those of SL models in sequential training.
Chunk
0.70
-4
Churk
SL Random Class
SSL Random Class
Churk:t
Chuek2
-08
Chunk
-04
Churk44
SL Distant Class
SSL Distant Class
-88
-04
-22
Figure 5: CKA scores between sequen- tially trained models.
Image reconstruction by feature inversion for sequential models. Similar to Zhao et al. (2020), in Figure 6, visualize images reconstructed from both SL-ST and SSL-ST features using deep image prior (DIP) (Ulyanov et al., 2018). To be specific, we choose four images in the first data chunk of the challenging distant class incremental sequence and visualize features of four sequentially learned models for both SSL and SL, respectively. As shown in Figure 6, in sequential training, features of SSL models can always perfectly reconstruct the main information original images. In contrast, features of SL models lose more detailed information with more sequential data chunks, which indicates SSL is much better at countering the knowledge forgetting sequential training. Recalling the evolving CKA similarity shown in Figure 5, the perfect reconstruction results of sequential SSL. models do not mean SSL models stop learning in sequential training. Instead, it indicates that SSL does well in learning new knowledge while keeping previous knowledge.
Input Image
Chunk 1
Chunk
Chunk
Chunk 4
Chunk
Chunk 2
Chunk 3
Chunk 4
Raw image
Images reconstructed from SSL models
Images reconstructed from SL models
Figure 6: Images reconstruction by inversing features from both SSL and SL models in sequential pre-training.
8

---

Published as a conference paper at ICLR 2022
Table 4: Comparisons of the sharpness of minima between SI. and SSL models. Lower is better.
Instance
Random Class
Distant Class
== 0.1
6 = 0.3
6 6=
== 0.3
6= 0.1
= 0.3
SL 0.47
0.94
0.21
0.94
0.19
0,94
SSL 0.14
0.68
0.08
0.66
0.12
0.71
5.3 HYPOTHESIS FOR DIFFERENT FORGETTING BEHAVIORS BETWEEN SL AND SSL
In this subsection, we dig into the different forgetting behavior between SL and SSL by analyzing the sharpness of the minima in the loss landscape. Flat minima in the loss landscape are the minima in which the change of losses slow in its neighborhood. Note that the models having flat min- ima in the loss landscape tend to exhibit impressive generalization ability (Keskar et al., 2016). When starting with flat minima, we expect that learning new chunks will have a minor effect on the performance of existing chunks, as escaping the wide basin is difficult. Therefore, hypothe- size that SSL encourages the model to seek out flatter minima, which increases SSL's resistance to catastrophic forgetting. To verify this hypothesis, we conduct experiments compare the sharpness of minima between SL and SSL models, where we apply a widely-used sharpness metric (Keskar al., 2016; Wen al., 2018). Concretely, we first define the neighborhood C, of mimina as:
f= e -c|1012 ||=||2 S e1
where n denotes the number of parameters and refers to the model parameter after training. Be- cause SSL and SL models are trained with different loss objectives, such as cross-entropy loss and the contrastive loss, we cannot directly analyze the sharpness with either objective. Considering that we aim for a representation model, we propose to directly adopt the KNN classifier to evaluate representations of both SL and SSL models. The KNN classification loss can be a discrete proxy of loss functions. Then, the sharpness of loss minima Lo.5 is defined as follows:
((0) (('))((')
$ max g(0', 0) max O'EC.
E'EC,
f(0)
where g0''.)) means the relative loss change from minima to the parameter 0', and the loss function f(0) is the negative KNN classification accuracy with model parameter 0.
As shown in Table 4, SSL indeed discovers flatter minima compared to SL, which verifies our hypothesis and provides explanation for why SSL suffers less forgetting than SL. More imple- mentation details are Appendix C.4. Moreover, we also conduct the visualization of relative loss change g a linear path like (Mirzadeh et al., 2020) in Appendix C.4. The loss change of SSL is slower than that of SL along the linear interpolation path, demonstrating the flatter minima of SSL.
DISCUSSIONS
This paper has conducted the first thorough empirical evaluation to investigate how well self- supervised learning (SSL) performs with various streaming data types and diverse downstream tasks. Our experimental results and the empirical analysis conclude the three main findings: i). Joint training is unnecessary for SSL with streaming data. Instead, sequential training with suitable continual learning strategies is cerffoomnnce-compeeiii as good alternative. ii). Sequential self-supervised pre-training shows better capability of over- coming catastrophic forgetting than sequential supervised pre-training. ūi). We hypothesize that SSL models have flatter minima than SL models in the loss landscape, which seems reasonable for the different forgetting behaviors between SL and SSL models. Morcover, We demonstrate this hypothesis by thorough empirical analysis of the sharpness of minima.
yet more efficient, well worth considering
As for future directions, we first call for more attention to sequential self-supervised learning for understanding its underlying theories of knowledge forgetting and devising better approaches. Also, we recommend considering sequential self-supervised training as a more efficient representation learning paradigm for real-world applications.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work supported by NUS ARTIC Project (Project Reference: ECT-RP2) and NRF Centre for Advanced Robotics Technology Innovation (CARTIN), Dr. Lanqing Hong and Prof. Xinchao Wang are corresponding authors. We thank Mr. Jiawei Du for his help with the sharpness metric and Mr. Yujun Shi for his discussion on the feature uniformity. We also thank the anonymous reviewers for their valuable comments.
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In European Conference on Computer Vision, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning, In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine- graegga visual categorization of birds. In IEEE Confer- ence on Computer Vision and Pattern Recognition, 2014.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative com- ponents with random forests. In European Conference Computer Vision, 2014.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference on Computer Vision, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In IEEE International Conference on Computer Vision, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi- cum-suuperisee learners. In Advances in Neural Information Processing Systems, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint MxXi::2003.04297, 2020c.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 2010.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference Computer Vision and Pattern Recognition Workshop. 2004.
10

---

Published as a conference paper at ICLR 2022
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 1r21v:13/2.6211, 2013.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervise learning. In Advances in Neural Information Processing Systems, 2020.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: adapt language models to domains and tasks. In Anmual Meeting of the Association for Computational Linguistics, 2020.
Jianhua Han, Xiwen Liang, Hang Xu, Kai Chen, Lanqing Hong, Chaoqiang Ye, Wei Zhang, Zhen- guo Li, Chunjing Xu, and Xiaodan Liang. SODA 10M: Towards large-scale object detection benchmark for autonomous driving. arXiv preprint arXiv:2108, 12178, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Zbontar Jure, Jing Li, Misra Ishan, LeCun Yann, and Deny Stephane. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 2021.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A et al. Overcom-
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting neural networks. Proceedings of the National Academy of Sciences, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, 2019a.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In IEEE Conference on Computer Vision and Pattern Recognition, 2019b.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine- grained cars. In Workshop on Fine-Grained Visual Cutegorization, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, University of Tront, 2009.
Matthias De Lange, Xu Jia, Sarah Parisot, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Unsupervised model personalization while preserving privacy and scalability: An open problem. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision, 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- line-grained visual classification of aircraft. arXiv preprint arXiv: 130.55130 2013.

---

Published as a conference paper at ICLR 2022
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In JEEE Conference on Computer Vision and Pattern Recognition, 2018.
Michael McCloskey and Neal J Cohen, Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Molivation. Elsevier, 1989.
George A Miller. WordNet: An Electronic Lexical Database. MIT press, 1998.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations, 2020.
Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? In JEEE Conference Computer Vision and Pattern Recognition, 2020.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1 2018.
German Parisi, Ronald Kemker, Jose I. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. IEEE International Conference on Computer Vision, 2019.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Colorado J Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, et al. Self-supervised pretraining im- proves self-supervised pretraining. In IEEE International Conference on Computer Vision, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 2015.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik Imagenet-21k pretraining for the masses. In International Conference on Learning Representations, 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision. 2015.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference Machine Learning, 2018.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communica- fions of the ACM, 2016.
12

---

Published as a conference paper at ICLR 2022
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? European Conference on Computer Vision, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, and Jun Zhu. Ordisco: Effec- tive and efficient usage of incremental unlabeled data for semi-supervised continual learning. In IEEE Conference Computer Vision and Pattern Recognition, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, 2020.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding neg- ative transfer. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chunpeng Wu, Yiran Chen, and Hai Li. Smoothout: Smoothing out sharp minima to improve generalization in deep learning. arXiv preprint arXiv:1805.07888, 2018
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. IEEE Conference on Computer Vision and Puttern Recog- nition, 2018
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In JEEE Conference on Computer Vision and Puttern Recognition, 2010.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In IEEE Conference Computer Vision and Pattern Recognition, 2021a.
Shipeng Yan, Jiale Zhou, Jiangwei Xie, Songyang Zhang, and Xuming He. An framework for online incremental learning of semantic segmentation. In Proceedings of the 29th ACM Interna- tional Conference on Multimedia, 2021b.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, 2014.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of Machine Learning Research, 2017.
Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, and Jiashi Feng. Unleashing the power of con- trastive supf-su visual models via contrast-regularized fine-tuning. In Advances Neural Information Processing Systems, 2021.
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrim- ination good for transfer learning? International Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A EXPERIMENTAL SETUPS
A.1 TYPES OF STREAMING DATA
We consider four kinds of streaming data for the study, i.e., the instance incremental sequence, the random class incremental sequence, the distant class incremental sequence. and the domain incremental sequence. To exclude the effect of the number of images, we make sure that data chunks in the same sequence have almost the same data amount. Here we provide more details about these data sequences.
Instance incremental sequence. For the instance incremental sequence, we split the Ima- geNet (Russakovsky et al., 2015) training data that consists of 1.28 million images with 1,000 classes into four even chunks. We ensure that each data chunk includes the same 1,000 classes with the same number of images for each class, which means these data chunks are independent and identically distributed (IID).
Random class incremental sequence. For the random class incremental sequence, we randomly split the 1,000 classes ImageNet into four parts where each part has 250 classes, Since each class of ImageNet has around 1,000 images, can directly obtain four data chunks with almost the same amount of images.
Distant class incremental sequence. To explore the data sequence with severe distribution shifts among data chunks, consider the distant class incremental sequence. Follow- ing (Yosinski et al., 2014), rather than randomly splitting the 1,000 classes, we leverage the WordNet Tree (Miller, 1998) to obtain four even data chunks sharing the minimal seman- tic overlapping. We first build 1000 1000 adjacent matrix among the 1,000 classes by setting the value of similar classes as and the value of dissimilar classes as O. To be specific, we
225
200
175
150
classss
125-
100
take classes sharing the common parent node beneath the ninth
50-
depth in the WordNet Tree as similar classes and vice versa. Using the semantic similarity described in the adjacent ma- trix, we then split the 1,000 classes into independent connected components shown in Figure 7. Finally, imbalanced components into four almost even data chunks. Concretely, the first chunk 'A' includes 250 classes of 318,459 images. The second data chunk 'B' includes 250 classes of 321,488 images. The third data chunk 'C' includes 251 classes of 321,533 images. The fourth data chunk 'D' includes 249 classes of 319,687 images.
25
0
400 Connected component
200
merge these
Figure 7: The number of classes for each connected component from the adjacent matrix of 1, 1,000 classes in ImageNet.
Domain incremental sequence. As for the domain incremental sequence, we consider multi- domain dataset called DomainNet (Peng et al., 2019). In our work, we adopt a domain incremental data sequence made of four distant domains including 'sketch', 'real', 'painting' 'quickdraw' and "clipart'. There exist severe domain distribution shifts among data these five domains. Specifi- cally, data in the domain 'quickdraw' mostly contain only lines without visual textures. As a result, images from "quickdraw' are less informative and more visually distinct, compared with images from those four domains, shown in Figure 8. For each domain, randomly select 48,129 im- ages as a data chunk, except for "quickdraw' where we select 47,687 images.
A.2 DETAILS OF PRE-TRAINING
MoCo-v2. For the illustration purpose, adopt prevailing self-supervised learning (SSL) method, MoCo-v2 (Chen et al., 2020c), to investigate the performance of SSL with streaming data. MoCo-v2 uses a Siamese network consisting of two encoders. These two encoders query images and key images, respectively, and share the same architecture where an MLP projec- tion head f. is on top of a backbone network fo. Only the query encoder is updated by the gradients
designed for
14

---

Published as a conference paper at ICLR 2022
Clipart
Painting
Quickdraw
Sketch
Alarm cloc
Airpiane
Figure 8: Example images in the five domains of DomainNet.
backpropagation while the key encoder is updated by the moving average with momentum. MoCo- v2 maintains an additional dictionary as a queue of features for contrastive learning. Specifically, features in the dictionary are progressively updated. The current mini-chunk features from the key encoder are enqueued and the same number of oldest features are dequeued. MoCo-v2 uses In- foNCE (Oord et al., 2018), variant of contrastive loss (CL), to maximize the similarity of features from positive pairs and minimize the similarity of features from negative pairs. The contrastive loss formalized below.
N Σ log
( (/ /T)
La
(3)
( + E,zez e(0T)))1)'
where N the number of samples, zạ is the L2-normalized projected feature from the query encoder, zf is the L2-normalized projected feature of the same input image from the key encoder, Z are the negative history features stored in the dictionary and is the temperature.
Pre-training. For supervipe pre-training, we follow the protocol of MoCo-v2 (Chen et al., 2020c), ie.. using the standard ResNet50 backbone (He et al., 2016). The implementation is based on OpenSeliSup'. For both joint training and sequential training. the number of training epochs 200 for each model training, where the convergence of loss is observed, as shown in Figure 14. As for the queue in MoCo-v2 training, we adopt the same as the original MoCo-v2, i.e., the queue size is 65,536. For data chunks fewer than 65,536 like DomainNet chunks, we store all samples of the current chunk for contrastive learning. In sequential training, the queue is refreshed before training on a new chunk. We never preserve previous data in the queue, to ensure the model training is sequentially conducted special replay buffer is used to preserve a small fraction of data from each previous chunk, like 10% in our experiments. For the instance incremental sequence, consider one random sequence the data are randomly divided. While for the random class incremental sequence, distant class incre- mental sequence, and domain incremental sequence, we experiment with different sequences of data chunks. In particular, considered sequences are obtained through right circular shift operations. For example, if the data sequence length is 4, after splitting all the data into four chunks A, B, C. and D, four sequences, namely A-B-C-D, B-C-D-A, C-D-A-B, and D-A-B-C are used for the sequential pre-training a representation model. The results from different sequences are averaged to obtain the final performance. For comparison, supervised pre-training is also implemented using OpenSelf- Sup following the recommended training protocol of ImageNet. For supervised pre-training the classifier layer is reset at new data chunk.
disjoint chunks. For the possibly used data replay strategy, another
A.3 DETAILS OF CONTINUAL LEARNING METHODS
Continual learning is assumed suffer from catastrophic forgetting of previously learned knowledge in supervised learning (Goodfellow et al., 2013; Kirkpatrick et al., 2017; McCloskey & Cohen, 1989), leading to significant performance degradation of previous tasks. Here we introduce the continual learning techniques we adopt, including data replay (Rolnick et al., 2019; Rebuffi et al., 2017) and regularization- based method, e.g., Memory Aware Synapses (MAS) (Aljundi et al., 2018).
Data replay. Data relay is a simple yet effective method for alleviating the catastrophic forgetting problem during the continual learning process. Specifically, we need to maintain a replay buffer
15

---

Published as a conference paper at ICLR 2022
and store selected subset of samples from each learned task in the buffer. Then just retrain on samples in the replay buffer to revisit old tasks while training the model for a new task.
To perform data replay in the sequential training process, maintain a replay buffer consisting of images sampled from previous data chunks. After finishing the sequential training with each data chunk, we randomly select 10% data of this chunk and store sampled data in the replay buffer. For the sequential training with the current data chunk, we directly mix current data with data in the replay buffer for self-supervised pre-training e.g. MoCo-v2 training using Eq. (3).
MAS. Regularization-baeee methods aim to mitigate the catastrophic forgetting by consolidating previous knowledge with added regularization term in the loss function. One typical unsupervised regularization-batt based method is MAS (Aljundi et al., 2018). Specifically, MAS proposes to compute gradients the squared L2-norm of the encoder output fo as the importance weights of parameters.
(4)
08,
With the parameter regularization term added, the resulting loss function with the coefficient À is shown as below.
[()) La)
Σ Es - 072)²
In MoCo-v2, the query encoder is considered as the representation network, we thus only impose the MAS regularization term on parameters of the query encoder. Specifically, the regularization coefficient À is fixed to be 100. Following the prevailing use of MAS regularization (Aljundi et al., 2018: 2019), we update the MAS importance weights S2 to cover information of each data chunk in the sequential training process. To be specific, after finishing the sequential training with each data chunk, we leverage both the current data chunk and data in the replay buffer to estimate importance weights for the trained model using Eq. (4). Then we update the stored sequential importance weights by a cumulative moving average of current and previous estimated importance weights, following (Aljundi et al., 2019). As for the model training the current data chunk, we apply the parameter regularization using previous importance weights and optimize the model using Eq, (5).
To sum up, besides the sequentially trained model, both above methods require extra storage for sequential sule-super pre-training. For the 10% data replay method, we need to only keep 10% data of each previous data chunk for sequential training. For the MAS regularization method, we only require to save a set of importance weights for the model and then update the importance weights sequentially.
A.4 DETAILS OF DOWNSTREAM TASKS
We evaluate the transfer performance of the pre-trained models using three different downstream tasks. Following (Chen et al., 2020a), we consider 12 diverse image classification datasets including Food-101 (Bossard et al., 2014), CIFAR10 (Krizhevsky et al., 2009), CIFARIO0 (Krizhevsky al., 2009), Birdsnap (Berg et al., 2014), SUN397 (Xiao et al., 2010), Standard Cars (Krause et al., 2013), FGVC Aircraft (Maji et al., 2013), VOC2007 (Everingham et al., 2010). DTD (Cimpoi al., 2014), OXford-III Pets (Parkhi et al., 2012), Caltech- 101 (Fei-Fei et al., 2004) and Oxford 102 Flowers (Nilsback & Zisserman, 2008). On these datasets, we evaluate the pre-trained models via the many-shot classification and the few-shot classification (except VOC2007). Both classification protocols are the same (Ericsson et al., 2021). In addition, we evaluate the pre-trained models on the PASCAL VOC detection task, following the same transfer protocol of MoCo (He et al., 2020). The training data of detection come from VOC2007 and VOC2012, and the test data come from VOC2007.
Many-shot classification. Many-shot classification is a widely used evaluation protocol (Chen et al., 2020b; He et al., 2020). To evaluate the pre-trained representations, a linear classifier is directly added to the pre-trained feature encoder. During the downstream task evaluation, only the added linear classifier is fine-tune tuned using substantial amount of downstream labeled data while the feature encoder is frozen. In this way, the downstream transfer performance can directly reflect the generalization ability of the pre-trained representation models.
16

---

Published as a conference paper at ICLR 2022
Few-shot classification. Few-shot classification reflects how well the pre-trained models perform on downstream tasks in the few-shot learning regime. Specifically, we consider 5-way 5-shot few- shot tasks on 11 downstream classification datasets, following the few-shot setting in (Ericsson et al., 2021). ) Concretely, the pre-trained model fixed for extracting representations. In contrast to many- shot evaluation, in few-shot evaluation, only few downstream labeled data are provided to obtain prototypes for different categories and then the classification is based on the nearest prototype.
Detection. To further evaluate the transferability of the pre-trained models on more downstream scenarios, we consider object detection as a downstream task, where the fine-grained spatial location information is more important, compared with classification tasks. To be specific, we follow the settings in (He et al., 2020), i.c., adopting the Faster-RCNN (Ren et al., 2015) with a backbone of R50-dilated-CC and fine-tuning all layers including the pre-trained representation network.
We perform no peyper-paaameee tuning for few-shot evaluation and detection evaluation. As for the linear evaluation protocol, we adopt the logistic regression and only tune the weight decay value. The inversed weight decay values for all downstream classification datasets are given in Table 5.
Table 5: The inverse regularization strength (weight decay value) used in many-shot logistic re- gression evaluation 12 different downstream classification datasets. SSL models: self-supervised models. SL models: supervised models.
Dataset
SSL Models
SL Models
Aircraft
5623.4133771333877
9,99999985098899
Caltech-101
316227.771256567
13162277621811911
Flowers
31622.7753066721
999 999.999952502551
Pets
999.999952552251
562.341
Cars
5623.4132771333687
17.782794106882072
DTD
1778.2794843157746
0.01778879462551197
Food
977827.9443315772
0.05623412982476388
CIFAR10
316227.771256566
0.05623412982476388
CIFARI00
(00.000002233114424
0.0562341298247638
Birdsnap
1778.77948431372
SUN397
100.00000233517424
0.0177827946252197
VOC2007
9,99999985098839
0.00562341 32237399
B MORE EXPERIMENTAL. RESULTS
B.1 RESULTS OF INSTANCE INCREMENTAL SEQUENCE
Transfer learning results of self-supervised pre-training with the instance incremental sequence are evaluated on all three downstream tasks. For results of both many-shot classification and few-shot classification in Figure 9, we find sequential SSL performs comparably with joint SSL on all down- stream datasets, with the average performance gap between sequential training and joint training less than 1 %,, while there exists evident gaps, more than 4%, between sequential supervised learning and joint supervised learning.
B.2 RESULTS OF LONGER SEQUENCES
We also conduct experiments on more realistic data sequence with longer chunks and random distribution shifts. Concretely, except the 1000 classes in ILSVRC 2012, we also randomly sample 1000 classes from the remaining classes in ImageNet-21K (Ridnik et al., 2021) to build ImageNet- 2K dataset with 2.29M images. Concretely, we randomly split all samples in the ImageNet-2K dataset into 8 chunks with 0.57M images per chunk. Figure 11 summarizes the results on 8-chunk sequence. Besides the similar observations in Section 4.2, we have the following observations for the longer sequence: i). With more sequential chunks, the performance gaps between ST and JT models become larger for both SL and SSL. ii). The performance gaps of SSL are visible but not significant, compared with those of SL. On average, at all steps in sequential training, the gaps of SSL are much smaller than those of SL. iii). The simple data replay with 10% previous data is effective in mitigating the performance gaps of SSL for the longer sequence. To make the simple data
17

---

Published as a conference paper at ICLR 2022
FeveS
CNT
01D
Manyy
Face
OPAREE
89
VO027
Catec
POWES
P60
D10
Figure 9: Linear and few-shot evaluation results of instance incremental sequence. on the left are the results of each dataset. On the right are averaged results across left datasets.
67
1015
63
SSL-ST SSL-JT
78-
SSL-JT
SSL-ST
59
STL-ST
SSL.JT
1
2
3
3
chunks
chunks
(a) Instance incremental sequence
67
AP75
63
SSL-ST SSL-JT
59
SSL-JT
SSL-JT
1
2
3
2
of chunks
# chunks
of chunks
(b) Random class incremental sequence
67:
84
AP75
63
SSL-ST SSL-JT
SSL-ST
SSL-ST
SSL
STL
2
3
of chunks
# chunks
# of chunks
(c) Distant class incremental sequence
Figure 10: Object detection evaluation results of three types of ImageNet-based streaming data.
18

---

Published as a conference paper at ICLR 2022
Niny-shi
edom
Figure 11:
Linear and few-shot evaluation results of 8-chunk ImageNet-2K sequence. on the left are the results of each dataset. On the right are averaged results across all left datasets.
replay sustainable for real streaming data with much longer chunks, we think setting an upper bound on the size of the replay buffer, such as the number of images, is feasible for realistic scenarios. To be specific, we can determine the upper bound of the replay buffer according to the physical storage limitation. Before the replay buffer approaches the upper bound, the tradeoff only between the accuracy and training time efficiency, i.e., storing more data (larger ratio) in the replay buffer results in better accuracy but longer training time. When the number of data in the replay buffer exceeds the upper bound, we can discard some old samples to save space for data sampled from new chunks (Rebuffi et al., 2017). Generally, we believe sequential SSL with longer sequence promising and will benefit from continual learning methods (Delange et al., 2021).
B.3 RESULTS OF NEGATIVE TRANSFER
Pets
Pets
100
100
90
90
Accuracy (96
60
60
Accuracy
50
50
SSL-ST
SSL-ST
SSL-IT
SSL-JT
20
SL-ST
20
SL-JT
SL.JT
10
10.
SSL-ST
SSL-5 W
SSL-ST w/MAS+
Sequential chunks
Sequential chunks
(a) Many-shot
(b) Few-shot
Figure 12: The averaged evaluation results on Pets of models pre-trained with the distant class incremental sequence D-A-B-C.
We observe severe "negative transfer" when pre-training models with distant class incremental quence and evaluating them on Oxford-IIIT Pets (Parkhi et al., 2012). Specifically, as shown in Figure 12, SSL-ST slightly outperforms SSL-JT the second chunk, i.e., chunk A. After chunk A,
19

---

Published as a conference paper at ICLR 2022
Nimpahot
caub
Figure 13:
Linear and few-shot evaluation results of distant incremental sequence for BYOL. on the left are the results of each dataset. On the right are averaged results across all left datasets.
increasing more chunks for SSL-JT models, i.c., chunk B and chunk C, instead leads to significant decrease performance. We make careful comparison between images from chunk and images in Pets. We find that Oxford-III" Pets includes 37 categories of pets like dogs and cats. As men- tioned in Appendix A.1, Chunk includes many relevant classes like 'Maltese dog'. 'Old English sheepdog'. 'Shetland sheepdog', 'Greater Swiss Mountain dog'. 'Bernese mountain dog'. 'French bulldog', 'Eskimo dog'. 'African hunting dog', 'tabby'. 'tiger cat', 'Persian cat', 'Siamese cat', 'Egyptian cat', and 'Madagascar cat'. The other three data chunks, i..., chunk D. B, and C, do not have these classes. Such observations are consistent with the widely accepted belief that "transfer- ring knowledge from the dissimilar source have negative impact the target learner" (Wang et al., 2019). Note that SL. models suffer severer 'negative transfer' than SSL models and continual learning methods can help SSL models significantly reduce the negative transfer, achieving on-par performance to SSL-JT models.
B.4 RESULTS OF BYOL
To evaluate whether sequential training performs well for other SSL methods, we conduct the chal- lenging distant class incremental sequence experiments with BYOL (Grill et al., 2020). The results of BYOL are shown Figure 13. Similar to the observations with MoCo-v2 Section 4.1, se- quential SSL is visibly inferior to joint SSL on streaming data with severe distribution shifts, but sequential SL performs obviously worse than joint SL. In addition, compared with SL models, SSL models show significantly smaller performance gaps between sequential training and joint training.
B.5 SELF-SUPERVISED TRAINING LOSS OVER TIME
Figure 14 shows the self-supervised training loss at each step in sequential training for various types of streaming data. The loss curves illustrate the average MoCo-v2 training loss for each data chunk, i.e. the contrastive loss. We have two observations. First, given type of streaming data like the instance incremental sequence, in the sequential training process, the training loss would spike when moving to new disjoint data chunk. Second, the loss spike value becomes higher with increasing distribution shifts, e.g.. training loss increases from instance incremental learning (7.1), random class incremental learning (7.2), to distant class incremental learning (7.4) at the beginning of step 2. The queue size is 65,536 for MoCo-v2, and the training batch size is 256. We record the training loss value averaged across one training batch and plot the loss data point every 250 training iterations, which means that after the first point in the loss curve, we have a filled queue for contrastive learning. Therefore, the loss spike is not because of the training mechanism, such the queue's refreshing in early training iterations. Instead, it is reasonable to ascribe the loss spike to the distribution shift change of training data, i.e., the new chunk. As for the first chunk, the initial average contrastive
20

---

Published as a conference paper at ICLR 2022
incremental sequence
Chunk 1
Chunk 2
Chunk
Chunk 4
10
7.0
Training Loss
Training Loss
Training Loss
6.9
6.9
6.9
Training
6.8
6.8
6.6
200000
200000
200000
Iteration
Iteration
Random incremental sequence
Chunk 1
Chunk 2
Chunk
Chunk 4
10
7.2
7.1
7.1
7.0
Training Loss
7.0
Training
Training
6.9
6.9
6.8
6.8
6.7-
6.6
6.6
6.6
200000
200000
200000
200000
Iteration
Iteration
Iteration
Iteration
[c] Distant class incremental
Chunk
Chunk
Chunic
Chunkc
7.6
7.75
Training Loss
7.2
7.4
7.50
7.2
Training
7.0
Training
Training
7.0
6.8
7.00
6.75
200000
200000
200000
200000
Iteration
Iteration
Iteration
Iteration
Figure 14: Self-supervised training loss each step of various types of streaming data.
loss is significant (around 10) because we randomly initialize the representation model. As for subsequent chunks, we inherit the representation model from the previous step. Therefore the initial average contrastive loss is not significant, i.e., around 7.2. On each chunk, we train the model for 200 epochs and finally observe the convergence of contrastive loss. We find that the average contrastive loss would generally converge to the value of around 6.6 for all chunks. We adopt the same training mechanism for each chunk, including the difficulty of the instance discrimination task and hyperparameter settings. We perform contrastive learning until convergence on each chunk. As a result, the final loss value is similar for different chunks across different settings.
C MORE EMPIRICAL ANALYSIS
C.I UNIFORMITY ANALYSIS OF REPRESENTATIONS
Uniformity is an important property for good representations (Wang & Isola, 2020). We then com- pare the uniformity of representations between sequential SL and SSL models. Specifically, sample images from chunk a and obtain representations M E gaxd where n denotes the number of samples (50,000) and d denotes the dimension of features (2,048). We first use singular values decomposition (SVD) to extract the 2,048 singular values of representations as below:
M UUVV EE R
21

---

Published as conference paper at ICLR 2022
Cumuattie percontagas
Cumulattin
Curdatter
ST
SSL-ST
0.2
ST
Uniform
Uniform
Uniform
0
500
1000
1500
2000
0.0
500
1000
1500
2000
500
1000
1500
2000
of dimension
of dimension
dimension
(a) Instance
(b) Random Class
(c) Distant Class
Figure 15: Uniformity of representations. SSL learns more uniform representations than SL.
We first rank the singular values in a descending order and then normalize all singular values to have the sum of 1. We show the cumulative percentages of singular values with increasing dimensions in Figure 15. Our assumption is that representations with more even singular values are more uniform. The corner case is that all singular values are the same, then representations distributed equally on all dimensions. From results on three types of streaming data, we find that SSL models have more uniform representations than SL models.
C.2 DETAILS OF BACKWARD TRANSFER AND FORWARD TRANSFER
For accuracy A on chunk j. we first extract the features with the model trained chunk i for all the examples in the chunk j, and then perform KNN classification in the feature space the chunk j. Specifically, we set the number of nearest neighbor k=200 for the KNN classification.
C.3 CKA SIMILARITY ANALYSIS BETWEEN ST MODELS AND JT MODELS
CKA similarity. To further understand the sequential self-supervised pre-training, then take closer look at the learned feature representations during the sequential training process. We leverage the linear centered kernel alignment (CKA) (Kornblith al., 2019a) to measure the similarity of output features between two different representation networks given the same data set as input. If we consider the size of the data as and the feature dimension for two networks dj and d2, respectively. We use the selected data set to extract features X E Rnxd from one representation E R from another representation network.
network and features is 50,000 and both di and d2 are 2,048. We first preprocess the two representation matrices by centering the columns. Then the linear CKA similarity between two representations X and Y can be computed as below:
our experiments, n
IX CKA(X,Y) IXTXII
How are ST models similar to JT models? We evaluate CKA similarity. for each data chunk, between features from the sequentially trained model and features from corresponding jointly trained model. The same 50,000 samples used for CKA features similarity analysis. For exam- plc, as shown Figure 16, at the step of the second data chunk, we compute the CKA similarity value between features of the model jointly trained with the first data chunks and features from the model sequentially trained after the second data chunk. The corresponding CKA similarity value 0.4, which indicates for SL, the difference between the ST model and the JT model is very large. In contrast, SSL has a higher similarity of 0.7 between the ST model and the JT model. Particu- larly, with MAS and data replay, the CKA similarity increases about 0.9, which means the model trained by sequential SSL extracts nearly the same features as the jointly trained model does. Since the analysis
the streaming data with severe distribution shifts, this further reinforces our hy- pothesis that, with the help of suitable continual learning methods, sequential SSL pre-training is promising replace joint training on streaming data with various distribution shifts.
C.4 SHARPNESS ANALYSIS
We first provide more details of the sharpness metric in Eq. (2). We compute the sharpness the respective chunk A for different types of streaming data in ImageNet. Considering that the function not differentiable, sample 0' from C, and 50 times to take the minimal accuracy. For
22

---

Published as a conference paper at ICLR 2022
1.0
CKA Similarity
0.8
0.6
0.4
SL-ST
SSL-ST w/MAS
0.2
SSL-ST
SSL-ST w/MAS+
0.0
1
2
3
4
# of chunks
Figure 16: CKA similarity scores between the sequentially trained models and the corresponding jointly trained models at the step of each data chunk on the distant class incremental sequence. Given images in the first data chunk, this figure shows the similarity of features between ST models and the corresponding JT models w.r.t. each data chunk on the distant class incremental sequence. The higher CKA similarity value, the more similar.
computational efficiency, randomly sample M. data points to perform kNN classification with k=200. We need to clarify that we cannot make comparisons of sharpness values across different sequential settings. Because in different settings, the loss values are calculated on different data samples and the kNN classification tasks are different such as the number of classes to be classified. Therefore, we can only compare the sharpness value between SL and SSL models trained and tested on the same streaming data. Concretely, for the instance incremental sequence, chunk A contains IID samples from ImageNet-1 K, classified into 1000 classes. For the random class incremental sequence, chunk A contains 250 random classes from ImageNet- IK classified into 250 classes. For the distant class incremental sequence, chunk A contains 250 semantically similar classes from ImageNet-IK, classified into 250 classes. We note that in Table 4, the instance incremental sequence has the worst sharpness metric value compared to the other two types of streaming data. This observation is easy to understand. The classification task with 1,000 classes instance incremental data is more challenging than both classification tasks with only 250 classes on the other two types of streaming data. Therefore, the sharpness value of instance incremental data is the highest. Different from the sharpness experiments, transfer learning experiments evaluate the performance of models on downstream tasks. Therefore, there no contradiction between the observations from both kinds of experiments.
For flatness visualization, show the normalized loss along the specified path by performing lin- early interpolation between the model after chunk 1 and the model after chunk 2 for different splits, as shown in Figure 17. We can see that compared SL, loss change slower for SSL along the linear path, which reflects SSL's superiority in terms of flatness. Specifically, the negative value in the SSL curve means that the performance SSL continuously improves along the path for instance incremental learning.
1.0
1.0
Relative change
0.8
SSL-ST
SUSTT
0.8
SSL-ST
SL-ST
Relaaive loss change
0.8
SSL-ST
SL-ST
0.6
0.4
0.4
0.4
0.0
0.0
0.0
-02
0.3.4.3
0.3.00555 0.6
0.3.0.3..4.
Interpolation
Interpolation
Interpolaion
Instance
(b) Random Class
(c) Distant Class
Figure 17: Relation loss change for different interpolations of parameters
23


Reviews:
[{"review_id": "wmeWtyPjurb", "reviewer": "Reviewer_RJbi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n    1. The submission is well grounded. The task studied in the manuscript is highly practical. the motivation of conducting such an analysis is strong, since the incremental learning setup is indeed practical in many real-world scenarios.\n    2. The amount of experiments is enormous and sufficient to draw the conclusion. \n    3. The experimental designs, especially the four categories of incremental learning setups, are interesting and to the point.\n    4. The manuscript is well organized and well written, especially the abstract and introduction that gives a good tutorial.\n\nI have some concerns and comments as follows:\n    1. Please elaborate on the rationale behind the design of the four scenarios, as early as in the introduction. In other words, why these four categories but nothing in-between or beyond?\n    2. It would be good if the authors can provide more insights and potential applications apart from revealing the experimental observations. In other words, what message of the experimental results convey? \n    3. Intuitively, why do self-supervised models forget less than supervised models?\n    4. Given the performances of the downstream tasks, any insights one can have? Like the affinities across tasks?\n", "summary_of_the_paper": "The authors of this submission worked on an interesting and quite practical task, i.e., to study the self-supervised behavior under the continual learning setup. To this end, the authors studied four types of incremental-learning settings, i.e., Instance incremental sequence, Random class incremental sequence, Distant class incremental sequence, Domain incremental sequence. The four types are designed to reflect different incremental-learning scenarios with different semantics.The authors have also conducted extensive empirical validations, including pre-training >500 models on four categories of pre-training streaming data from ImageNet and DomainNet, and evaluating them on three types of downstream tasks and 12 different downstream datasets. ", "summary_of_the_review": "Th authors focus on an interesting and practical task. Prior works have largely overlooked this task and the proposed work is the first to explore along this line. The motivation of conducting such an analysis is strong, since the incremental learning setup is indeed practical in many real-world scenarios. Extensive experiments are also conducted to support the proposed method, which are very solid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "t_h6rrQStw", "reviewer": "Reviewer_QZfY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1.\tThis paper firstly investigates the sequential training situation for self-supervised learning, which is an important problem for the widespread use of self-supervised learning in real world application. They also propose some scenarios for training model with streaming data.\n2.\tThe experimental results show that the problem of catastrophic forgetting due to training with streaming data is less problematic for self-supervised pretraining compared to the supervised pretraining, for all scenarios. \n3.\tThey show that some existing continual learning techniques are effective for self-supervised sequential pretraining. \n\nWeakness\n1.\tThe novelty and technical contribution are limited. Although this paper proposes some scenarios for sequential training with streaming data and shows that the self-supervised learning methods work better than supervised learning baselines under these scenarios, it is hard to find further contribution except this empirical discovery\n2.\tIt seems that the analysis results in Section 5 are not very helpful for understanding. For example, in subsection 5.1, there is no explanation what BWT and FWT mean and how this is related to the catastrophic forgetting issue. Also, it seems that the result of BWT, FWT, CKA and feature reconstructions are the natural result of less catastrophic forgetting, and provide meaningful insights to make a further improvement on sequential self-supervised pretraining. \n", "summary_of_the_paper": "This paper explores a more realistic situation for self-supervised representation learning of training with streaming data. In this paper, four different situations for training with streaming data are investigated. The experimental results show that the self-supervised pretraining less suffers from catastrophic forgetting compared to the supervised pretraining, when the model is trained sequentially using streaming data. The experimental results also show that existing continual learning techniques are also effective for self-supervised learning with streaming data.", "summary_of_the_review": "This paper empirically studied on sequential self-supervised learning problem, which is never done before, and show that self-supervised learning is more robust to catastrophic forgetting problem compared to supervised learning. Although exploring the problem of sequential self-supervised learning is meaningful, it is hard to find insights beyond empirical results to make improvement on the top of the discovery of this work in this paper. As a result, I prone to reject this paper. However, I am open to any objection and further discussion, if the authors make their point and insight of this paper clearer.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kfnOzKhKz5q", "reviewer": "Reviewer_4UNG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. The paper provides an extensive framework for assessing the downstream task performance of sequential self-supervised learning through the four different settings of increasing severity of data distribution drift. I believe this framework unifies the different settings considered by previous works in this domain and provides a comprehensive benchmark for the community to explore the setting of sequential self-supervised learning in further detail.\n\n2. The paper considers downstream performance on several standard benchmark datasets that contributes to the generalization of the insights and analysis to other vision datasets. There is also analysis on both many-shot and few-shot classification tasks which is essential to cover the diverse range of downstream tasks found in practice.\n\n3. Understanding the performance gap between SSL-ST and SSL-JT at different degree of distribution drift allows to identify what level of distribution drift hurts the performance. I find this analysis particularly insightful. Moreover, the paper also experiments with some continual learning approaches to try mitigating this performance reduction. This information can be useful to understand in what type of sequential self-supervised training scenario, there may be a need to experiment with continual learning approaches.\n\n4. The analysis of feature similarity helps to understand that in self-supervised learning, sequential setting behaves quite similar to joint setting in terms of learned feature representations.\n\n5. The additional analysis on space-time efficiency and backward/forward transfer analysis of performing sequential self-supervised learning is also quite extensive and useful to understand the benefits of the approach.\n\nWeakness\n1. It seems that the paper overall has little to no insight of its own to offer on why the sequential self-supervised learning is much closer in performance to joint self-supervised learning. In some places (Sec 4.2 and Sec 6), the paper does provide some explanation to this behavior but that is (1) conjectured in that there are no experiments conducted or theoretically justified if that is indeed the rationale and (2) cited from other papers. The paper does provide empirical evidence on how well SSL-ST is doing compared to SSL-JT but that seems more of an outcome of how the downstream tasks are performing in the two settings and not exactly shedding light on what is making SSL-ST do so well compared to SSL-JT while SL-ST is much worse than SL-JT. I would suggest the authors to add more empirical evidence about why SSL-ST is not so much worse than SSL-JT across the different sequential settings considered. Further, there can be more empirical evidence added to the claims that there is ‘negative transfer’ happening (Sec 4.2) or the features learned by uniformly distributed over feature space (Sec 7) and this is indeed a potential reason for the trends observed.\n\n2. Continual learning settings are very sensitive to the empirical setup. Changes such as the number of classes per chunk and the size of the replay buffer can make a drastic change in the performance. I am not very clear why the sequential setting has just 4 chunks. I would suggest the authors to experiment on much longer sequences such as 10 or 20 chunks. Understanding the impact of having longer sequences is crucial to assess the full extent of forgetting and negative impact of distribution drift. Moreover, tasks like self-supervised learning are only practical when leveraging extremely large amounts of data as shown with Instagram 1B (Mahajan et al, 2018). Such works are already instance incremental and at large scales, having more than 4 chunks will be highly likely.\n\n3. The experiment details lack some important details to validate the soundness of the setup in which experiments were performed. For eg, when using MoCo-v2, what is the size of the queue used for training? Is the queue refreshed before training on a new chunk? Or is it simply preserved and enqueued/dequeued as usual? In the case of the latter, my concern will be that the setup is not really working on disjoint chunks and the observations may not be valid. Further, if possible, I would request the authors if they could provide the training loss curves for the different settings across the chunks. I would expect the training loss to spike whenever the training proceeds to a new disjoint chunk and the spikes should be higher for higher severity of data distribution drift. If this is not the case, there is a likelihood of a leakage between chunks which affects the sanity of the sequential setup.\n\n4. I am curious why there is a drop in SSL-ST wrt SSL-JT in figure 11 for instance incremental sequence setting. In Appendix B (Pretraining), it says that ‘we consider one random sequence as the data are randomly divided’. Does this mean that the way SSL-ST is implemented in an instance incremental setting, it is identical to SSL-JT? Please correct me if I am wrong. But if this is correct, shouldn’t the two curves have no performance difference?", "summary_of_the_paper": "This paper studies how models perform on downstream image classification tasks when they are trained via self-supervision using streaming data. Streaming data refers to a sequential training setting  (ST) where the entire training data is not available for training at the same time but can be only used in disjoint chunks. When training on one chunk of data, the previous chunks are unavailable for training during this sequential training. When the entire training data is available for training at the same time, it is called joint training (JT). The key insight of the work is that in the presence of little to moderate drift in the data distribution across chunks, the downstream task performance of self-supervised training in sequential setting (SSL-ST) is comparable to that of self-supervised training in joint setting (SSL-JT). While in the presence of severe drift in data distribution across chunks, SSL-ST performs worse than SSL-JT but through the application of continual learning approaches like data replay and MAS (Aljundi et al, 2018), this performance gap can be reduced to a large extent. Moreover, when training is done in a supervised manner (SL), sequential setting (SL-ST) performs much worse than joint setting (SL-JT)\n\nThe work experiments on four kinds of sequential settings with varying degrees of data distribution drift across consecutive chunks - (1) instance incremental sequence, where each chunk of unlabeled data is derived from the same set of classes having i.i.d. samples in each chunk, (2) random class incremental sequence, where data in each chunk belongs to a different set of randomly chosen classes, (3) distant class incremental sequence, where each chunk belongs to a different set of classes such that there is a larger semantic gap between the set of classes and (4) domain incremental sequence, where the chunks belong to different domains thereby exhibiting severe distribution shift. For the first three settings, Imagenet-1k is used which is split into 4 chunks as per the setting. For the fourth setting, DomainNet is used. MoCo-v2 is primarily used in the empirical analysis along with some experiments on BYOL are also conducted to show that the trends are observed in other SSL approaches as well.\n\nThe work also conducts an analysis on the amount of forgetting using the backward and forward transfer analysis which shows that compared to supervised setting where there is significant amount of catastrophic forgetting, self-supervised sequential setting has much lower forgetting. Further analysis via Centered Kernel Alignment (CKA) suggests that compared to supervised setting, SSL-ST has a higher feature similarity between two sequential models. Even the feature similarity of SSL-ST with SSL-JT is higher than the corresponding similarity of SL-ST with SL-JT. ", "summary_of_the_review": "I would like to recommend the paper for ‘marginally above the acceptance threshold’ rating. The authors have addressed some of my concerns. Some of my concerns still exist such as how concretely the new analysis using sharpness of loss minima justifies SSL-ST to be compared to SSL-JT. I also have some concerns on the soundness of the loss curve and whether the rise in the loss is due to distribution drift or just recalibration of the SSL queue. Although there is certainly areas to improve, the paper does provide an interesting perspective to compare SSL and SL and also support it with different kinds of analysis that led me to revise the rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ckuka761LnW", "reviewer": "Reviewer_PFFi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper does a thorough empirical study on different types of streaming data and across a large number of datasets.\n- The streaming data setting is important to consider as we train larger pre-trained models and need to update them over time. The new data may or may not have distribution shifts, which this paper shows has an effect on downstream performance vs. re-training the model.\n- I am unaware of previous thorough empirical studies on the effectiveness of self-supervised pre-training on streaming data. \n- The paper gives interesting observations comparing self-supervised continual learning to supervised continual learning, showing differences under mild distribution shift.\n- The results on each individual dataset are plotted clearly. However, the plots do not have error bars.\n- The paper resolves some of the issues they see with continual learning using data replay. However, the replay method used seems to make the data size keep increasing - this makes the comparison a bit unfair since the data replay method will get to see more and more data (and the benefit of continual learning diminishes over time). There should be a tradeoff with seeing new and old data.\n- The paper also tests another self-supervised method, BYOL.\n- The post-hoc analysis of the sequentially trained models shows quite clearly that self-supervised learning suffers less from forgetting for certain types of streaming data (Fig 7).\n\nWeaknesses / Neutral\n- It's unclear what changes from a practical point of view here, since the suggestion is to use simple continual learning methods which are known. The main contributions are the observations across different types of streaming data (which we may not have control over?), but these are somewhat alleviated by employing the continual learning methods regardless of the type of streaming data.\n- There are other papers out there that seem to consider the streaming / continuing pre-training setting for self-supervised learning (https://openaccess.thecvf.com/content_CVPR_2019/papers/Aljundi_Task-Free_Continual_Learning_CVPR_2019_paper.pdf, https://aclanthology.org/2020.acl-main.740.pdf, https://arxiv.org/abs/2103.12718). Often the story here is that doing more pre-training on a more specialized domain can improve the accuracy on the specialized domain. This is a different point to this paper, which is not looking at the accuracy on specific domains over time (which could also be interesting to keep track of).\n- I'm left unsure why there is such a difference between supervised continual learning and self-supervised continual learning. They are definitely measuring different things - with SSL, there is an additional transfer step. Perhaps the finetuning/transfer step takes care of some of the small distribution shift? Some evidence for this: there is a larger gap between streaming and non-streaming when the downstream evaluation is few-shot learning (less finetuning) in Figure 3. Another source of difference could be the loss function - in SSL you need to model all the features whereas in supervised learning, you can focus on features correlated with the label.\n- On the setting, there is possible distribution shift across chunks in the streaming data, but it seems that if taken as a whole, the overall distribution is assumed to be stationary in some sense. Perhaps for future work, it may make sense to consider gradual distribution shift over time (of the overall distribution, which changes the relationship between pretraining and downstream). \n- In Fig 1, the class semantic trees are not really fully explained. For example, what are the parent nodes? In general the Fig 1 caption could be expanded to more fully specify what the reader should get from the figure. The different types of streaming data are also not explained before they are mentioned in the intro.\n- The \"instance incremental\" setting with IID datapoints, when considered in a huge data setting, is identical to the regular pre-training setup  in some cases - for example, in GPT-3, they didn't even finish one epoch of the data. Thus not having a huge gap isn't too surprising particularly in the large data setting where we make very few passes over the dataset. This paper studies this kind of setting and says that in supervised learning, the gap isn't big (https://arxiv.org/abs/2010.08127). It's interesting that in Figure 11 of your paper, there is a gap between supervised pre-training in a streaming or non-streaming fashion for downstream task performance. This could be an interesting distinction / comparison to make.\n- Is there any way to test the claim of \"negative transfer\" in 4.2? Often, when streaming suffers this \"negative transfer\", JT does not seem to, even with the same chunks.\n- At the bottom of page 3, re-training a model from scratch at every time point is referred to as joint training, but perhaps a more common terminology from online learning is \"follow the leader\".\n- In Figure 7 we only have to show the lower triangular part of the matrix", "summary_of_the_paper": "The paper considers the setting where data for self supervised pre-training comes in a streaming fashion, where models are incrementally trained on new data. They test 4 different types of streaming data which have different distribution shift properties, on 12 classification datasets and an object detection dataset, using MoCov2 as the self-supervised algorithm. The paper finds that for certain types of streaming data, the downstream task performance of streaming SSL versus \"joint\" SSL (which is the standard pre-training scheme where all data is available at once) are somewhat similar, while there is a large gap in performance for streaming data types with a large amount of distribution shift. They find that this behavior is unlike the gap between streaming and non-streaming supervised learning, where the gaps are larger in all types of streaming data. Finally, they reduce some of the gaps with data replay and regularization.", "summary_of_the_review": "The paper provides some interesting observations through pretty thorough experiments on an interesting/important setting for large pre-trained models. They show that continual learning methods can improve the issues they found. There isn't much in the way of explanations of the observations, and the methods aren't novel, but the observations are valuable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper performs a comprehensive investigation on self-supervised pre-training with streaming data. Reviewers agreed that the task studied in this paper is highly practical and important, and the analysis is insightful. Meanwhile, reviewers raised some concerns such as empirical setups and insights. In the revised paper, the authors provided more justifications and added more analysis such as few-shot evaluation and uniformity analysis. After the discussion period, most reviewers are positive about this paper.

Overall, I recommend to accept this paper. I encourage the authors to take the review feedback into account in the final version.
Decision: Accept (Poster)