SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Comparison in transfer learning for object detection and semantic segmentation.
Algorithm	VOC detection	COCO detection	Semantic segmentation
AP50	AP	AP75	AP50	AP	AP75	Mean IoU (%)	Pixel acc. (%)
SimSiam	77.0	48.8	52.2	50.7	31.2	32.8	0.2626	65.63
SimSiam w/ GSG	79.8	51.2	55.1	53.2	33.4	35.6	0.3345	76.57
BYOL	79.2	50.3	54.5	52.0	32.5	34.5	0.2615	62.79
BYOL w/ GSG	80.5	52.0	56.4	53.7	33.8	36.1	0.2938	74.78
Table 4: Comparison between explicit and implicit contrastive learning algorithms.
Methodology	Algorithm	b = 1024	b = 512	b = 256
Explicit contrastive	End-to-end [He et al., 2020]	57.3	56.3	54.9
InstDisc [Wu et al., 2018]	54.1	52.0	50.0
MoCo [He et al., 2020]	57.5	56.4	54.7
SimCLR [Chen et al., 2020a]	62.8	60.7	57.5
Implicit contrastive	SimSiam w/ GSG	70.1	69.4	69.9
BYOL w/ GSG	71.9	71.0	71.6
5.2 Transfer learning
An essential goal of representation learning is to obtain a pre-trained backbone that can be transferred to various downstream tasks. To evaluate whether our pre-trained backbones are transferable, we consider image recognition, object detection, and semantic segmentation tasks. We use the ResNet-50 backbones pre-trained on ImageNet. We follow the experimental setup in [Ericsson et al., 2021]. More implementation details can be found in Appendix D.
Image recognition We carry out image recognition tasks on different datasets. For datasets, we adopt widely used benchmark datasets in transfer learning such as CIFAR-10, Aircraft [Maji et al., 2013], Caltech [Fei-Fei et al., 2004], Cars [Krause et al., 2013], DTD [Cimpoi et al., 2014], Flowers [Nilsback and Zisserman, 2008], Food [Bossard et al., 2014], Pets [Parkhi et al., 2012], SUN397 [Xiao et al., 2010], and VOC2007 [Everingham et al., 2010]. These datasets vary in terms of the amount of data or the number of classes. We report the average precision AP at 11 recall levels  $\{0, 0.1, \dots, 1\}$  on VOC2007, mean per-class accuracy on Aircraft, Pets, Caltech, and Flowers, and top-1 accuracy on the rest of the datasets. For the evaluation protocol, we perform the linear evaluation.
Object detection We perform object detection tasks on Pascal-VOC [Everingham et al., 2010] and MS-COCO [Lin et al., 2014]. We use VOC2007 trainval as the training set and VOC2007 test as the test set. We report  $\mathrm{AP}_{50}$ , AP, and  $\mathrm{AP}_{75}$ .  $\mathrm{AP}_{50}$  and  $\mathrm{AP}_{75}$  are average precision with intersection over union (IoU) threshold 0.5 and 0.75, respectively. We freeze the pre-trained backbone except for the last residual block. We use a Feature Pyramid Network [Lin et al., 2017] to extract representations, and a Faster R-CNN [Ren et al., 2015] to predict. We do the experiments on the Detectron2 platform [Wu et al., 2019].
Semantic segmentation We conduct semantic segmentation tasks on MIT ADE20K [Zhou et al., 2019]. We use ResNet-50 as the encoder and use UPerNet [Xiao et al., 2018] (the implementation in the CSAIL semantic segmentation framework [Zhou et al., 2018, 2017]) as the decoder. It is based on Feature Pyramid Network and Pyramid Pooling Module [Zhao et al., 2017]. We train for 30 epochs and test on the validation set. We report mean IoU and pixel accuracy. For the predicted results of each algorithm, refer to Appendix F.
Results Table 2 and 3 show that applying GSG increases the performance consistently. We can conclude that the pre-trained backbones are transferable to different tasks, and our method helps to get better quality representations.
(a) ImageNet, SimSiam
(b) ImageNet, BYOL
(c) CIFAR-10, SimSiam
(d) CIFAR-10, BYOL
Figure 4: Importance of guiding. Depending on how stop-gradient is used, performance is significantly different. It shows the best performance when used along with GSG.
5.3 The number of negative samples
It is known that the performance of contrastive learning algorithms is vulnerable to reducing the number of negative samples [Tian et al., 2020, Wu et al., 2018, He et al., 2020, Chen et al., 2020a]. In this respect, we compare our algorithms to benchmark contrastive learning algorithms (all with the ResNet-50 backbone). Table 4 reports linear evaluation accuracy on ImageNet (the performance of the benchmark algorithms is from [He et al., 2020] and [Chen et al., 2020a]). End-to-end, SimCLR, and our algorithms use samples in the batch as negative samples, so for these algorithms,  $b$  in the table denotes the batch size. InstDisc and MoCo maintain a separate memory bank or dictionary, so for these algorithms,  $b$  in the table denotes the number of negative samples from the memory bank or dictionary. The table shows that our algorithms work well with small batch sizes.
6 Empirical study
In this section, we broaden our understanding of stop-gradient, predictor, and momentum encoder, which are components of SimSiam and BYOL by comparing variants of the algorithms.
6.1 Importance of guiding
We demonstrate that it is crucial to apply stop-gradient in a guided way by changing how stop-gradient is used. We compare our GSG with what we name random stop-gradient and reverse stop-gradient. In random stop-gradient, stop-gradient is randomly applied to one of the two encoders, and a predictor is applied to the other. That is, we randomly select one of the four equations in Equation (6). On the other hand, in reverse stop-gradient, stop-gradient is applied opposite to the intuition in GSG. In other words, we select the remaining loss terms other than the selected loss terms when GSG is used. In Equation (2), two loss terms will be selected to form the loss as follows:
$$
\mathcal {L} = \frac {1}{2} \mathcal {D} \left(p _ {1 2}, \operatorname {s g} \left(z _ {1 1}\right)\right) + \frac {1}{2} \mathcal {D} \left(p _ {2 2}, \operatorname {s g} \left(z _ {2 1}\right)\right). \tag {7}
$$
Therefore, the random stop-gradient is a baseline where stop-gradient is naively applied, and the reverse stop-gradient is the worst-case scenario according to our intuition.
(b) ImageNet, BYOL
Figure 5: Preventing collapse. Unlike existing algorithms, algorithms to which GSG is applied do not collapse even when the predictor is removed.
(d) CIFAR-10, BYOL
Figure 4 shows the results of applying GSG, random stop-gradient, and reverse stop-gradient along with the existing algorithm for SimSiam and BYOL. We observe the  $k$ -NN accuracy at each epoch while training on ImageNet and CIFAR-10. First of all, in all cases, it can be seen that the algorithm applying our GSG outperforms algorithms using other methods. In addition, it can be seen that the performance of the existing algorithm and the algorithm to which random stop-gradient is applied are similar. When applying random stop-gradient, the number of loss terms is doubled, and half of them are randomly selected, so there is expected to be no significant difference from the existing algorithm. In the case of reverse stop-gradient, the performance drops significantly. This highlights the importance of using stop-gradient in a guided manner.
If we look at the case of CIFAR-10 (Figure 4c and Figure 4d), which has relatively much smaller data and is more challenging to train stably, we can obtain some more interesting results. First, when reverse stop-gradient is applied to SimSiam, it collapses, and the accuracy converges to  $10\%$ , which is the chance-level accuracy of CIFAR-10. However, this was not the case for BYOL. This implies that the momentum encoder can help prevent collapse. Note that SimSiam and BYOL are identical in our experimental setup, except that BYOL has a momentum encoder. In addition, in the case of the existing SimSiam and SimSiam with random stop-gradient, the fluctuation of the accuracy at the beginning of training is severe. However, it is relatively less in the case of SimSiam with GSG. This suggests that GSG can help the stability of training.
6.2 Preventing collapse
One of the expected effects of GSG is to prevent collapse by implicitly repelling each other away from negative pairs. It is known that SimSiam and BYOL collapse without a predictor [Chen and He, 2021, Grill et al., 2020]. Then, a natural question is whether SimSiam and BYOL will not collapse even if the predictor is removed when GSG is applied. Figure 5 reports the performance when the predictor is removed. In the case of CIFAR-10, we run up to 400 epochs to confirm complete collapse.
First, the existing algorithms collapse as expected, and the accuracy converges to the chance-level accuracy (0.1% for ImageNet and 10% for CIFAR-10). Interestingly, however, our algorithms do not collapse. This shows that GSG contributes to training stability, which is our method's intrinsic advantage. Nevertheless, the final accuracy is higher when there is a predictor. This indicates that the predictor in SimSiam and BYOL contributes to performance improvement.
We have proposed implicit contrastive representation learning for visual SSL. In this methodology, while representations of positive pairs attract each other, representations of negative pairs are promoted to repel each other. It exploits the asymmetry of network architectures with source and target encoders without contrastive loss. We have instantiated the methodology and presented our guided stop-gradient method, which can be applied to existing SSL algorithms such as SimSiam and BYOL. We have shown that our algorithms consistently perform better than the benchmark asymmetric learning algorithms for various tasks and datasets. We have also shown that our algorithms are more robust to reducing the number of negative samples than the benchmark contrastive learning algorithms. Our empirical study has shown that our method contributes to training stability. We hope our work leads the community to better leverage the asymmetry between source and target encoders and enjoy the advantages of both contrastive learning and asymmetric learning.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16570-16579, 2022.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detector2. https://github.com/facebookresearch/detectron2, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via nonparametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6210-6219, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310-12320. PMLR, 2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal on Computer Vision, 2018.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.