Question: Given the performances of the downstream tasks, any insights one can have? Like the affinities across tasks?
Evidence: In pre-training tasks, we pay attention to the generalization of the learned representations to new data or tasks rather than the performance on the training dataset. Taking a closer look at the results in Figures, we observe that, although joint SSL models achieve comparable performance to joint SL models in linear evaluation, joint SL models significantly outperform joint SSL models in few-shot evaluation. This observation is also demonstrated in. The main difference between the two evaluation protocols is that linear evaluation involves more fine-tuning than few-shot evaluation, as introduced in Appendix A.4. Therefore, the underlying reason for the observation is that supervised features are correlated with labels and more discriminative, thus easy to directly transfer to downstream datasets similar to upstream pre-training data (DomainNet or ImageNet). For example, SL models dominate most few-shot object or scene classification tasks but fail on DTD, a texture classification dataset sharing no common classes with ImageNet or DomainNet. In contrast, self-supervised features are more generalized and comprehensive, thus requiring more fine-tuning for desirable downstream transfer. In addition, on some downstream datasets, we have seemingly abnormal observations that ST models may outperform JT models and the model performance may drop with the increase of chunk number. These phenomena are due to the so-called "negative transfer", which is also discussed in other model pre-training studies. That is, pre-training with more data chunks does not necessarily benefit a specific downstream dataset if the added training data are irrelevant to the downstream dataset. See Appendix B.3 for a concrete example of "negative transfer" on Oxford-IIIT Pets in pre-training with streaming data. It is observed that sequential SSL models suffer less "negative transfer" than SL models and continual learning methods largely prevent "negative transfer".
In pre-training tasks, we pay attention to the generalization of the learned representations to new data or tasks rather than the performance on the training dataset. Taking a closer look at the results in Figures 2-4, we observe that, although joint SSL models achieve comparable performance to joint SL models in linear evaluation, joint SL models significantly outperform joint SSL models in few-shot evaluation. This observation is also demonstrated in (Tian et al., 2020; Ericsson et al., 2021). The main difference between the two evaluation protocols is that linear evaluation involves more fine-tuning than few-shot evaluation, as introduced in Appendix A.4. Therefore, the underlying reason for the observation is that supervised features are correlated with labels and more discriminative, thus easy to directly transfer to downstream datasets similar to upstream pre-training data (DomainNet or ImageNet). For example, SL models dominate most few-shot object or scene classification tasks but fail on DTD (Cimpoi et al., 2014), a texture classification dataset sharing no common classes with ImageNet or DomainNet. In contrast, self-supervised features are more generalized and comprehensive, thus requiring more fine-tuning for desirable downstream transfer. In addition, on some downstream datasets, we have seemingly abnormal observations that ST models may outperform JT models and the model performance may drop with the increase of chunk number. These phenomena are due to the so-called “negative transfer” (Wang et al., 2019), which is also discussed in other model pre-training studies (Newell & Deng, 2020; Gururangan et al., 2020). That is, pre-training with more data chunks does not necessarily benefit a specific downstream dataset if the added training data are irrelevant to the downstream dataset. See Appendix B.3 for a concrete example of “negative transfer” on Oxford-IIIT Pets (Parkhi et al., 2012) in pre-training with streaming data. It is observed that sequential SSL models suffer less “negative transfer” than SL models and continual learning methods largely prevent “negative transfer”.
Answer: As for insights from the performance of different downstream tasks. We mainly have two interesting observations and provide the corresponding explanations to them in Section 4.2.
As for different downstream tasks, we observe SSL models perform competitively to SL-JT models under many-shot evaluation while SSL models significantly underform SL-JT models under few-shot evaluation. The underlying insight is that supervised features are correlated with labels and more discriminative, thus easy to directly transfer to downstream datasets similar to upstream pre-training data (DomainNet or ImageNet). For example, SL models dominate most few-shot object or scene classification tasks but fail on DTD dataset, a texture classification dataset which shares no common classes with ImageNet or DomainNet. In contrast, self-supervised features are more generalized and comprehensive, thus requiring more fine-tuning for desirable downstream transfer.
As for evaluation with different downstream datasets, we find the sequentially trained model may suffer “negative transfer” on some specific downstream datasets. A concrete example is given in Appendix B.3. The insight behind pre-training with more data chunks does not necessarily benefit a specific downstream dataset if the added training data are irrelevant to the downstream dataset.
It is a great suggestion on the analysis of the affinities across tasks to further understand the downstream tasks. While in this work, we aim to provide a comprehensive study on pre-training with upstream data. Therefore our explorations or ablations mainly focus on model pre-training. As for the downstream tasks, we only use them to evaluate the pre-trained models, following popular pre-training works [3, 4]. We thus leave the study of downstream tasks for future work.