Question: Intuitively, why do self-supervised models forget less than supervised models?
Evidence: We first provide more details of the sharpness metric in Eq. (2). We compute the sharpness on the respective chunk A for different types of streaming data in ImageNet. Considering that the function f is not differentiable, we sample θ 0 from C and run 50 times to take the minimal accuracy. For computational efficiency, we randomly sample 0.1M data points to perform kNN classification with k=200. We need to clarify that we cannot make comparisons of sharpness values across different sequential settings. Because in different settings, the loss values are calculated on different data samples and the kNN classification tasks are different such as the number of classes to be classified. Therefore, we can only compare the sharpness value between SL and SSL models trained and tested on the same streaming data. Concretely, for the instance incremental sequence, chunk A contains IID samples from ImageNet-1K, classified into 1000 classes. For the random class incremental sequence, chunk A contains 250 random classes from ImageNet-1K, classified into 250 classes. For the distant class incremental sequence, chunk A contains 250 semantically similar classes from ImageNet-1K, classified into 250 classes. We note that in Table 4, the instance incremental sequence has the worst sharpness metric value compared to the other two types of streaming data. This observation is easy to understand. The classification task with 1,000 classes on instance incremental data is more challenging than both classification tasks with only 250 classes on the other two types of streaming data. Therefore, the sharpness value of instance incremental data is the highest. Different from the sharpness experiments, transfer learning experiments evaluate the performance of models on downstream tasks. Therefore, there is no contradiction between the observations from both kinds of experiments. For flatness visualization, we show the normalized loss along the specified path by performing linearly interpolation between the model after chunk 1 and the model after chunk 2 for different splits, as shown in Figure 17. We can see that compared to SL, loss change is slower for SSL along the linear path, which reflects SSL’s superiority in terms of flatness. Specifically, the negative value in the SSL curve means that the performance of SSL continuously improves along the path for instance incremental learning.
Answer: We provide more insights and empirical evidence on why SSLmodels forget less than SL models, from the perspective of the sharpness of minima in the loss landscape.
Flat minima in the loss landscape are the minima in which the change of losses is slow in its neighborhood. Note that the models having flat minima in the loss landscape tend to exhibit an impressive generalization ability [1]. When starting with flat minima, we expect that learning new chunks will have a minor effect on the performance of existing chunks, as escaping the wide basin is difficult. Therefore, we hypothesize that SSL encourages the model to seek out more flat minima, which increases SSL's resistance to catastrophic forgetting of previous data chunks, further leading to smaller performance gaps between SSL-ST and SSL-JT on downstream tasks.
To verify this hypothesis, we conduct experiments to compare the sharpness of minima between SL and SSL models. Specifically, we measure the sharpness via a standard sharpness metric in [2]. We also conduct model interpolation between sequential models to visualize the linear path in the loss landscape for both SL and SSL models. The experimental results in Section 5.3 demonstrate that SSL indeed discovers the more flat minima compared to SL, which verifies our hypothesis and provides an explanation for why SSL suffers less forgetting than SL. More details about sharpness analysis can be found in Sec 5.3 and Appendix C.4