Question: Please elaborate on the rationale behind the design of the four scenarios, as early as in the introduction. In other words, why these four categories but nothing in-between or beyond?
Evidence: Recent advances in self-supervised learning (SSL) demonstrate competitive or even better transfer learning performance on various downstream tasks, compared with supervised pre-training. Although waiving the cost of human labeling, SSL usually requires massive unlabeled data to learn a powerful representation model and benefits from significantly large-scale pre-training data, e.g., adopted billion-scale data to pre-train better SSL models. The common pre-training practice follows the joint training (JT) setup, where massive unlabeled data are collected together before model training. In reality, however, it is usually difficult to access a large amount of collective unlabeled data at once. Instead, real-world data are usually accessed in a streaming fashion, e.g., data are generated and collected sequentially chunk by chunk, or even decentralized and stored in different servers; such a learning setup is known as sequential training (ST). Despite much research effort and promising results achieved by JT, it inevitably suffers from heavy data storage, prolonged training time, and finds itself incompetent when training data volume expands over time. For ST, on the other hand, a learner can be sequentially trained with disjoint data chunks, making it much more efficient than JT.
How to effectively and efficiently pre-train a representation model under the ST setup has been an open problem. Despite the high efficiency, some continual learning research works have shown that ST with supervised models tends to suffer from catastrophic forgetting, having a significant performance degradation on the historical data chunks. Unlike the case of continual learning tasks, in pre-training tasks, one expects the model to well generalize to downstream tasks rather than focusing only on the seen tasks. Nevertheless, how well sequential self-supervised models perform on downstream tasks remains unclear. to measure the semantic similarity of classes. Classes having the same parent or ancestor in WordNet, marked with similar colors, share similar semantics in the class semantic tree.
To fill the research gap, we provide a thorough empirical study on self-supervised pre-training with streaming data. In the pre-training stage, to mimic real-world data collection scenarios and for the better dissection of sequential SSL, we consider streaming data with different degrees of distribution shifts. As shown in Figure, we obtain four types of streaming data, including the instance incremental sequence with negligible data distribution shifts, by randomly splitting ImageNet-1K into four independent and identically distributed (IID) data chunks, the random class incremental sequence with moderate data distribution shifts, by randomly splitting 1K classes of images into four disjoint chunks each with 250 classes, the distant class incremental sequence with severe data distribution shifts, by splitting 1K classes of data into four chunks while maximizing the semantical dissimilarity among chunks, and the domain incremental sequence with severe domain distribution shifts, by taking each domain in DomainNet as a data chunk.
As for the evaluation, we consider three downstream tasks following, including few-shot evaluation and linear evaluation (also named many-shot classification) on 12 image classification datasets, and the Pascal VOC detection task. Through extensive experiments with more than 500 pre-trained models, we thoroughly investigate key roles in sequential SSL, including streaming data, downstream tasks and datasets, continual learning methods, SSL methods, and the method efficiency in terms of time and storage. We also thoroughly investigate the knowledge forgetting behavior of sequential SSL and supervised learning (SL) models and provide a comprehensive empirical analysis of the underlying reason.
To the best of our knowledge, we are among the first to explore the sequential self-supervised pretraining setting and the first to provide a thorough empirical study on self-supervised pre-training with streaming data. We summarize the takeaways as well as our contributions as: i). Sequential SSL models exhibit the on-par transfer learning performance as joint SSL models on streaming data with negligible or mild distribution shifts. As for streaming data with severe distribution shifts or longer sequences, i.e., the distant class incremental sequence, evident performance gaps exist between sequential SSL and joint SSL models. Such performance gaps, however, can be mitigated effectively and efficiently with unsupervised parameter regularization and simple data replay. ii). Based on the above finding, we conclude that the standard joint training paradigm may be unnecessary for SSL pre-training. Instead, sequential SSL is performance-competitive but more time-efficient and storage-saving and is well worth considering as the practical practice for selfsupervised pre-training with streaming data. iii). Compared with supervised learning (SL) models, SSL models consistently show smaller performance gaps between ST and JT. Our comprehensive investigation of learned representations demonstrates that sequential SSL models are less prone to catastrophic forgetting than SL models. iv). Through the empirical analysis on the sharpness of minima in the loss landscape, we find that SSL models have wider minima than SL models, which we hypothesize is the reason for less forgetting of SSL models.
In the pre-training stage, to mimic real-world data collection scenarios and for the better dissection of sequential SSL, we consider streaming data with different degrees of distribution shifts. As shown in Figure 1, we obtain four types of streaming data, including the instance incremental sequence with negligible data distribution shifts, by randomly splitting ImageNet-1K (Russakovsky et al., 2015) into four independent and identically distributed (IID) data chunks, the random class incremental sequence with moderate data distribution shifts, by randomly splitting 1K classes of images into four disjoint chunks each with 250 classes, the distant class incremental sequence with severe data distribution shifts, by splitting 1K classes of data into four chunks while maximizing the semantical dissimilarity among chunks, and the domain incremental sequence with severe domain distribution shifts, by taking each domain in DomainNet (Peng et al., 2019) as a data chunk.
Answer: We have added more introduction to the rationale behind the proposed streaming data in the introduction. We would like to clarify that we have two main considerations behind the proposed four types of streaming data.
The main consideration is to mimic common practical data collection scenarios where streaming data may include different levels of data distribution shift. To this end, we proposed four types of streaming data targeting for four degrees of commonly-seen distribution shift in realistic scenarios.
The second consideration is for the ease of a comprehensive empirical study on pre-training with streaming data. For the first time to provide such a systematic study, we believe using streaming data with various degrees of distribution shifts is beneficial for the community to have a comprehensive understanding of the sequential pre-training problem. Streaming data with other complex distribution shifts may be feasible, e.g., mixing different distribution shifts within one streaming data sequence may be more realistic. We call for more attention on pre-training with streaming data and further extending the types of considered data sequence wherein. But for our work, a study on streaming data containing mixed distribution shifts is not easy to probe into the properties of the studied problem. Therefore, we adopt the design of the four scenarios in our paper.