SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FOOD-101. Similar to CIFAR-10, we experiment on the FOOD101 dataset, consisting of 75,750 train samples and 25,250 test samples across 101 food classes (e.g. 'pizza', 'misso soup', 'waffles'), using VICReg and the RES-5-250 architecture. In Figure 13 (left), we plot the NCC test accuracy for both the sample level, and the semantic classes level, and in Figure 13 (right) we plot the NCC test accuracy of the intermediate layers with respect to the sample level labels. The results are similar to the ones shown in Section 4. Additionally, we present the sample level clustering in the intermediate layers. Here, we also see an improvement throughout the layers and the epochs.
Aircrafts. We applied the SWaV and Barlow Twins algorithms with the RES-5-250 architecture to the Aircrafts dataset, which consists of 10,000 images from 100 aircraft types. In Figure 9 we show the NCC train accuracy at the various intermediate layers.
Tiny Imagenet. We also experiment on more varied datasets, like the Tiny ImageNet dataset, consisting of 100,000 train samples and 10,000 test samples across 200 classes (e.g. 'nematode', 'harmonica', 'syringe'), using VICReg and the RES-10-250 architecture. In Figure 12 we plot the NCC test accuracy for both the sample level, and the semantic classes level, both unnormalized and normalized resp. In Figure 9 we show the NCC train accuracy using the SWav SSL algorithm. The results are similar to the ones shown in the clustering section.
Imagenet. We extend our experimentation to the more complex ImageNet [19] dataset. In Figure 11, we show the linear test accuracy and NCC train accuracy of both SwAV and SimCLR on the ImageNet dataset, along training epochs, with respect to the semantic labels on the final layer of a ResNet-50. Similarly to the supervised setting, as shown in section Section 4, both the linear accuracy and the NCC accuracy dramatically improves during training.
In Figure 14, we present the training losses and linear accuracies for three different hyperparameter selections trained using VICReg and a RES-5-250 network. Specifically, we vary the  $\mu$  (variance regularization) parameter and display the losses and linear accuracy for  $\mu = 5, 25, 100$ , from left to
Figure 11: SwAV and SimCLR cluster ImageNet data with respect to semantic targets. The last layer Linear Test accuracy and NCC train accuracy of an SSL-trained ResNet-50 on Imagenet, on the 1000 original classes.
Figure 12: VICReg clusters the data with respect to semantic targets. The NCC train accuracy of an SSL-trained RES-10-250 on Tiny Imagenet, measured at the sample level and original classes (both un-normalized and normalized).
right, respectively. Both loss terms are normalized by  $\mu$ . In all cases, we used  $\nu = 1$  and  $\lambda = 25$  by default.
Interestingly, the configuration with  $\mu = 5$  exhibits significantly lower performance compared to the default setting ( $\mu = 25$ ). However, the configuration with  $\mu = 100$  achieves performance comparable to the default, despite the network having a considerably higher invariance loss term. This observation further supports our claim that the regularization term plays a crucial role in learning semantic features.
B.4 Experiments with Different SSL Algorithms
In our main text, we primarily focus on the VICReg SSL algorithm. However, to ensure the robustness of our findings across different algorithms, we also trained a network using the SimCLR algorithm [14] with our default RES-5-250 architecture. In Figure 15, we present several experiments that compare the network trained with VICReg to the one trained with SimCLR.
Firstly, Figure 15 (top) illustrates the linear test accuracies of intermediate layers for both algorithms throughout the training epochs with respect to the original classes (left) and the superclasses (right). The darkness of the lines indicates the progression of epochs, with lighter shades representing later epochs. Despite the significant differences between the algorithms, particularly in SimCLR's contrastive nature, we observe similar clustering behavior across different layers and epochs with VICReg consistently achieving better performance. This finding demonstrates that various SSL algorithms yield comparable clustering characteristics.
Figure 13: SSL algorithms cluster the data with respect to semantic targets and invariance to augmentations at intermediate layers. (left) The normalized NCC test accuracy of a VICReg-trained network on FOOD101 with respect to the sample level labels and the original class labels. (right) The NCC test accuracy of the model with respect to the sample level labels of intermediate layers.
Figure 14: The role of the regularization term in SSL training. Each plot depicts the regularization and invariance losses, along with the linear test accuracy, throughout the training process of VICReg with  $\mu = 5, 25, 100$  respectively.
Furthermore, in addition to clustering, we aim to assess the similarity of representations between the two algorithms. In Figure 15 (bottom left), we present the correspondence of the representations with random targets, similar to the approach used in Figure 4 (left), across different training epochs. The darkness of the lines represents the progression of epochs, with lighter shades indicating later epochs. Additionally, in Figure 15 (bottom right), we depict the same correspondence for different intermediate layers, similar to Figure 4 (middle), at the end of the SSL train. Across various layers and epochs, we observe similar behavior between the algorithms. However, it is worth noting that VICReg demonstrates a slightly higher performance in extracting the random targets.
B.5 Random Target Function Architectures
Neural network architectures possess inherent biases that aid in modeling complex functions. Convolutional networks, for example, implicitly encode biases through their structure, which includes locality and translation invariance. As a result, these networks tend to prioritize local patterns over global features. On the other hand, the ViT architecture takes a different approach by treating images as sequences of patches and utilizing self-attention mechanisms. This design introduces a bias that enables long-range dependencies and global context.
An interesting question arises regarding how the choice of the SSL backbone architecture influences the learned representations. To explore this, we examine the alignment of representations with different types of random targets. Specifically, we conduct experiments similar to those illustrated in Figure 4 (left and middle) but employ different random targets based on the ViT architecture [52].
In Figure 16, we monitor the linear test accuracy of VICReg-trained RES-5-250 in recovering both the ResNet-18 targets and the ViT targets. Figure 16 (left) presents the linear test accuracy at different training epochs (20, 40, 100, 400, 1000) from dark to light resp. and Figure 16 (right) shows the linear test accuracy at different intermediate layers (1-5) from dark to light resp. at the end of SSL
Figure 15: SimCLR and VICReg have similar performance. (top) Linear test accuracy in different training epochs, as a function of the intermediate layer, for original classes and superclasses, from left to right resp. (bottom) (left) Linear test accuracy in different training epochs (from dark to light) with respect to different randomness levels. (right) Linear test accuracy in different intermediate layers, at the end of training with respect to different randomness levels.
Figure 16: The implicit bias of the backbone architecture on the learned representations. (left) Linear test accuracy of an SSL-trained RES-5-250 network for extracting ResNet-18 and ViT random target functions with varying degrees of randomness (x-axis) at different epochs (color-coded from dark to bright). (right) Linear test accuracy of an SSL-trained RES-5-250 network for extracting ResNet-18 and ViT random target functions with varying degrees of randomness (x-axis) at different intermediate layers (color-coded from dark to bright).
training. As can be seen, the results with ViT targets are consistent with the claims in Section 5. In other words, the SSL representations exhibit improved alignment during training epochs and across intermediate layers.
However, it is evident that the linear accuracy with respect to the ResNet-18 targets consistently outperforms the accuracy with respect to ViT targets at all training stages, except for the end of the supervised training phase ("0" randomness). This indicates that during the training process, the SSL-trained model acquires representations that are more aligned with those achieved by training convolutional network architectures. This behavior can be attributed to the implicit biases introduced by the backbone, which share similarities with the ones present in the ResNet-18 architecture.
While our study has yielded significant findings, it is not without its limitations. Firstly, our experiments were conducted on select datasets, which inherently possess their unique characteristics and biases. As such, applying our analysis to different datasets may potentially give different results. This reflects the inherent variability and diversity of real-world data and the possible influence of dataset-specific factors on our findings. Secondly, our analysis primarily focuses on the vision domain. While we believe that our findings have substantial implications in this area, the generalizability of our findings to other domains, such as natural language processing or audio processing, remains unverified.
In this paper, our primary objective is to characterize the different properties and aspects of SSL, with the aim of deepening our understanding of the learned representations. SSL has demonstrated its versatility in a wide range of practical downstream applications, including image classification, image segmentation, and various language tasks. While our main focus lies in unraveling the underlying mechanisms of SSL, the insights gained from this research hold the potential to enhance SSL algorithms. Consequently, these improved algorithms could have a significant impact on a diverse set of applications. However, it is crucial to acknowledge the potential risks associated with the misuse of these technologies.