Question: Is there any way to test the claim of "negative transfer" in 4.2? Often, when streaming suffers this "negative transfer", JT does not seem to, even with the same chunks.
Evidence: In pre-training tasks, we pay attention to the generalization of the learned representations to new data or tasks rather than the performance on the training dataset. Taking a closer look at the results in Figures, we observe that, although joint SSL models achieve comparable performance to joint SL models in linear evaluation, joint SL models significantly outperform joint SSL models in few-shot evaluation. This observation is also demonstrated in. The main difference between the two evaluation protocols is that linear evaluation involves more fine-tuning than few-shot evaluation, as introduced in Appendix A.4. Therefore, the underlying reason for the observation is that supervised features are correlated with labels and more discriminative, thus easy to directly transfer to downstream datasets similar to upstream pre-training data (DomainNet or ImageNet). For example, SL models dominate most few-shot object or scene classification tasks but fail on DTD, a texture classification dataset sharing no common classes with ImageNet or DomainNet. In contrast, self-supervised features are more generalized and comprehensive, thus requiring more fine-tuning for desirable downstream transfer. In addition, on some downstream datasets, we have seemingly abnormal observations that ST models may outperform JT models and the model performance may drop with the increase of chunk number. These phenomena are due to the so-called "negative transfer", which is also discussed in other model pre-training studies. That is, pre-training with more data chunks does not necessarily benefit a specific downstream dataset if the added training data are irrelevant to the downstream dataset. See Appendix B.3 for a concrete example of "negative transfer" on Oxford-IIIT Pets in pre-training with streaming data. It is observed that sequential SSL models suffer less "negative transfer" than SL models and continual learning methods largely prevent "negative transfer".
These phenomena are due to the so-called “negative transfer” (Wang et al., 2019), which is also discussed in other model pre-training studies (Newell & Deng, 2020; Gururangan et al., 2020). That is, pre-training with more data chunks does not necessarily benefit a specific downstream dataset if the added training data are irrelevant to the downstream dataset. See Appendix B.3 for a concrete example of “negative transfer” on Oxford-IIIT Pets (Parkhi et al., 2012) in pre-training with streaming data.****We observe severe “negative transfer” when pre-training models with distant class incremental sequence and evaluating them on Oxford-IIIT Pets (Parkhi et al., 2012). Specifically, as shown in Figure 12, SSL-ST slightly outperforms SSL-JT at the second chunk, i.e., chunk A. After chunk A,increasing more chunks for SSL-JT models, i.e., chunk B and chunk C, instead leads to significant decrease in performance. We make careful comparison between images from chunk A and images in Pets. We find that Oxford-IIIT Pets includes 37 categories of pets like dogs and cats. As mentioned in Appendix A.1, Chunk A includes many relevant classes like ‘Maltese dog’, ‘Old English sheepdog’, ‘Shetland sheepdog’, ‘Greater Swiss Mountain dog’, ‘Bernese mountain dog’, ‘French bulldog’, ‘Eskimo dog’, ‘African hunting dog’, ‘tabby’, ‘tiger cat’, ‘Persian cat’, ‘Siamese cat’, ‘Egyptian cat’, and ‘Madagascar cat’. The other three data chunks, i.e., chunk D, B, and C, do not have these classes. Such observations are consistent with the widely accepted belief that “transferring knowledge from the dissimilar source can have a negative impact on the target learner” (Wang et al., 2019). Note that SL models suffer severer ‘negative transfer’ than SSL models and continual learning methods can help SSL models significantly reduce the negative transfer, achieving on-par performance to SSL-JT models.
Answer: Based on your suggestions, we provide more empirical evidence to describe and explain the existence of “negative transfer”. We observe severe “negative transfer'' when pre-training models with distant class incremental sequence and evaluating them on Oxford-IIIT Pets. The “negative transfer” exists in two forms (1) ST models may outperform JT models. (2) The model performance may drop with the increase of chunk number for both ST and JT models. To be specific, on the 4-chunk streaming data, at step 2, we find models sequentially trained by the second data chunk show good performance on the testing data of Pets, for both many-shot and few-shot evaluation. In contrast, when the model is sequentially trained by other data chunks, i.e. step 3 and step4, more data for training instead leads to significant performance degradation for sequential training, and visible performance reduction for joint training. We find the reason is that only the second data chunk has lots of similar classes to the Pets dataset.
The reason that JT models suffer less “negative transfer” is that JT models can access previous data chunks, except for the current irrelevant chunks. For example, in the sequential learning of D-A-B-C, when training with the B chunk, JT models can have D, A, and B chunks for training. The relevant data in chunk A can significantly mitigate the “negative transfer” brought by data in chunk D and B. However, with only data in chunk B, ST models cannot avoid the severe “negative transfer”. See Section 4.2 and Appendix B.3 for more details of “negative transfer”.