Bag of Image Patch Embedding Behind the Success of Self-Supervised Learning
Authors: Yubei Chen
Adrien Bardes
ZENGYI LI
Yann LeCun
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning (SSL) has recently achieved tremendous empirical advancements in learning image representation. However, our understanding of the principle behind learning such a representation is still limited. This work shows that joint-embedding SSL approaches learn a representation of image patches, which reflects their co-occurrence. Such a connection to co-occurrence modeling can be established formally, and it supplements the prevailing invariance perspective. We empirically show that learning a representation for fixed-scale patches and aggregating local patch representations as the image representation achieves similar or even better results than the baseline methods. We denote this process as {\it BagSSL}. Even with $32\times 32$ patch representation, BagSSL achieves $62\%$ top-1 linear probing accuracy on ImageNet. On the other hand, with a multi-scale pretrained model, we show that the whole image embedding is approximately the average of local patch embeddings. While the SSL representation is relatively invariant at the global scale, we show that locality is preserved when we zoom into local patch-level representation. Further, we show that patch representation aggregation can improve various SOTA baseline methods by a large margin. The patch representation is considerably easier to understand, and this work makes a step to demystify self-supervised representation learning.

Published in Transactions on Machine Learning Research (11/2023)
Bag of Image Patch Embedding Behind the Success of Self- Learning
Yubei Chen 5,+
ybchen@ucdavis edu
Adrien Bardes 1,2,
Zengyi Li 3
Yann LeCun 1,4
Mett FAIR 2 Inria, École normale supérieure, CNRS, PSL Research University
Aiizi, Inc. (Work performed while Redwood Center, UC Berkeley)
CCenter for Data Science, New York University
'ECE Dept., UC Davis (Work performed while Meta AI FAIR and CDS, NYU)
"Equal contribution
Reviewed on OpenReview: http::///ppparrrrre...ec////r/rt
Abstract
Self-supervised learning (SSL) has recently achieved tremendous empirical advancements in learning image representation. However, our understanding of the principle behind learning such representation is still limited. This work shows that joint-embedding SSL approaches learn representation of image patches, which reflects their co-occurrecee Such connection to co-occurrence modeling can be established formally, and it supplements the prevailing invariance perspective. We empirically show that learning representation for fixed scale patches and aggregating local patch representations as the image representation achieves similar or even better results than the baseline methods. We denote this process as BagSSL. Even with 32 x patch representation, BagSSL achieves 62% top-1 lincar probing accuracy on ImageNet. On the other hand, with multi-scale pretrained model, we show that the whole image embedding is approximately the average of local patch embeddings. While the SSL representation is relatively invariant at the global scale, we show that locality is preserved when we zoom into local patch-level representation. Further, we show that patch representation aggregation can improve various SOTA baseline methods by a large margin. The patch representation is considerably easier to understand, and this work makes a step to demystify self-supervised representation learning.
Introduction
In many application domains, self-supervised representation learning experienced tremendous advancements in the past few years. In terms of the quality of learned features, unsupervised learning has caught up with supervised learning or even surpassed the latter in many cases. This trend promiscs unparalleled scalability for driven machine learning in the future. One of the most successful self supervised learning paradigms is voint-embeddie SSL (Wu et al., 2018; Chen et al., 2020a; Grill et al., 2020; Chen et al., 2020b; Bardes et al., 2021; Zbontar et al., 2021; Yeh et al., 2022; Li et al., 2022), which uses Siamese network architecture (Bromley et al., 1993). These methods follow a general goal: to transform different views (augmentation) of the same instance (image) closer in the representation space, and meanwhile, the new space is not collapsed, or in other words, important geometric and stochastic structures are preserved.
While we celebrate the empirical success of SSL, our understanding of this learning process still needs be improved. This work shows that joint-embedding SSL approaches are learning representation

---

Published in Transactions on Machine Learning Research (11/2023)
of image patches, which reflects their co-occurrence. To demonstrate this, we first establish formal connection between joint-embedding SSL and co-oceurrence modeling. Then, we show that learning a representation for fixed scale patches and linearly aggregating patch representations (bag of local features) as the image representation achieves similar or even better results than learning the representation with multi-scale crops. Empirical results are shown with several SSL methods(Chen et al., 2020a; Bardes et al., 2021; Li et al., 2022; Grill et al., 2020) on CTFAR1O, CIFAR100, ImagoNetI00,, and ImageNet-1K datasets. Even with 32 x 32 patch representation, we can achieve 62% top linear probing accuracy on ImageNet 1K. And KNN classifier also works surprisingly well with the aggregated patch feature. These findings resonate with recent works in supervised learning based on local features (Brendel & Bethge, 2018; Dosovitskiy et al., 2021; Trockman & Kolter, 2023). We further show that for baseline SSL methods pretrained with multi- scale crops, the whole-image representation is approximately a linear aggregation of local patch embeddings. The local patch representation is considerably easier to understand, and we provide visualization to show that the representation space preserves locality when we zoom into the local patch representation. These discoveries supplement the prevailing invariance perspective, provide a useful understanding of the success of joint-embedding SSL, and make step to demystify self-supervised representation learning.
2 Related Works
Joint-embedding SSL: Invariance without Collapse. Joint-embeddingg SSL (or instance-based SSL) (Wu et al., 2018) views each of the images as a different class and uses data augmentation (Dosovitskiy et al., 2016) to generate different views from the same image. As the number of classes equals the number of images, it is formulated as a massive classification problem, which may require a huge buffer or memory bank. Later, SimCLR (Chen et al., 2020a) simplifies the technique significantly and uses an InfoNCE- based formulation to restrict the classification within an individual batch. At the same time, it's widely perceived that contrastive learning needs the "bag of tricks," " e.g., large batches, hyperparameter tuning, momentum encoding, memory queues, etc. Later works (Chen & He, 2021; Yeh et al., 2022; HaoChen et al., 2021) show that many of these issues can be casily fixed. Recently, several even simpler non-contrastive learning methods Bardes et al., 2021; Zbontar et al., 2021; Li et al., 2022) are proposed, where one directly pushes the representation of different views from the same instance closer while maintaining a non-collapsing representation space. Joint- embedding SSL methods mostly differ in their means of achieving a non-collapsing solution. This include classification versus negative Comples(Chen et al., 2020a), Siamese networks (He et al., 2020; Grill et al., 2020) and more recently, covariance regularization (Ermolov et al., 2021; Zbontar et al., 2021; Bardes et al., 2021; HaoChen et al., 2021; Li et al., 2022; Bardes et al., 2022). The covariance regularization has also long been used in many classical unsupervised learning methods (Roweis & Saul, 2000; Tenenbaum et al., 2000; Wiskott & Sejnowski, 2002; Chen et al., 2018), also to enforce a non-collapsing solution. There is a duality between the spectral contrastive loss (HaoChen et al., 2021) and the non-contrastive loss, which we briefly discuss the intuition in Appendix B.
All previously mentioned joint-embedding SSL methods pull together representations of different views of the same instance. Intuitively, the representation would eventually be invariant to the transformation that generates those views. We would like to provide further insight into this learning process: The learning objective can be understood as using the inner product to capture the co-occurrence statistics of local image patches. We also provide visualization to study whether the learned representation truly has this invariance property.
Patch-Based Representation. Many works have explored the effectiveness of path-based image features. In the supervised setting, Bagnet(Brendel & Bethge, 2018) and Thiry et al. (2021) showed that aggregation of patch- based features could achieve most of the performance of supervised learning on image datasets. In the unsupervised setting, Gidaris et al. (2020) performs SSL by requiring bag-of-patches representation to be invariant between different views. Due to architectural constraints, vision transformer- methods also naturally use a patch-based representation (He et al., 2022; Bao et al.).
Learning Representation by Modeling the Co-Occurrence Statistics. The use of word vector representation has a long history in NLP, which dates back to the 80s (Rumelhart et al., 1986; Dumais, 2004). Perhaps one of the most famous word embedding results, the word vector arithmetic operation,
2

---

Published in Transactions on Machine Learning Research (11/2023)
was introduced in Mikolov et al. (2013a). Particularly, to learn this embedding, task called "skip-gram" used, where one uses the latent embedding of word to predict the latent embedding of the word vectors in a context. A refinement was proposed in Mikolov et al. (2013b), where a simplified variant of Noise Contrastive Estimation (NCE) was introduced for training the "Skip-gram" model. The task and loss are deeply connected to the SimCLR and its InfoNCE loss. Later, a matrix factorization formulation was proposed in Pennington et al. (2014), which uses a carcfully reprocessed concurrence matrix compared to latent semantic analysis. While the task in Word2Vec and SimCLR is apparently similar, the underlying interpretations are quite different. In joint-embedding SSL methods, one pervasive perception is that the encoding network is trying to build invariance, i.e., different views of the same instance shall be mapped to the same latent embedding. This work supplements this classical opinion and shows that similar to Werd2Vec, joint-enbeddding SSL methods can be understood as building a distributed representation of image patches by modeling the co occurrence statistics.
3 Joint-Embedding SSL Models Image Patch Co- Co-Occurrence
As we discussed earlier, recent works show that there exists duality Garrido et al. (2023) between contrastive SSL and non-contrastive SSL methods, and these methods are essentially solving the same problem. So we only need to establish the connection between co-occurrence statistics modcling and one joint-embedding SSL method. In the following, we show that the spectral contrastive learning HaoChen et al. (2021) loss function models the co-occurrence statistics. We also replace multi-scalc crop augmentation with fixed-scale image patches during SSL training. Thus the learning process is precisely modeling the patch-level co-occurrence statistics. After training, SSL has learned a representation for fixed-scale image patches. The whole-image representation is a linear aggregation of local patch representations.
Joint-embeddingg SSL and Co-Occurrence. Let's assume T1 and 72 are two color-augmented patches sampled from the dataset. We denote their marginal distribution by p(z1) and p(z2), which includes variation due to sampling different images, spatial locations within an image, and random color augmentation. We denote their joint distribution by (( which is the probability that they co-occur within the same image. Σ and 2ω are corresponding embedding vectors of Z and Z2 in the representation space. The representation transform is parameterized by neural network. The next proposition shows that the spectral contrastive loss function is equivalent to co-occurrence statistics modeling. Proposition 3.1. The spectral contrastive loss function Ls:
Ls Bp(t,, .2) ( + (Ep(s))((E)) ( 2))
(1)
is equivalent to the co-occurrence statistics modeling loss function Le:
Lc to= f
WH
p(ā (2) p(Z))((TZ)
dī dī2
(2)
where x = 2,2 and is constant weighting factor.
The proof is rather straightforward and is presented in Appendix A. The first term in Lg ensures that co occurring image patches are embedded closer in the embedding space, and the second term in Lş uniformly pushes the patch embeddings away from each other so that the representation space does not collapse.
Evaluation. After have learned a representation of fix-scale image patches, we can embed all of the image patches "Fi.,..,,Tww within an image into the embedding space. Then, we can obtain the representation Rimg for the whole image by linearly aggregating (averaging) all patch embeddings. The pipeline is shown in Figure 1. E = v) and i = f (); We call h the embedding and I the projection of an image patch, z. {क'} are fixed-scale. The function f(; 0):;8) is deep neural network with paramcters 0, and is typically much simpler neural network with only one or a few fully connected layers and parameters v. Following a convention in joint embedding SSL, we use h as patch representation hather than á' for slightly better accuracy, i.e., Ramy = mean (hi1," ... ..w)). During SSL training, given an image patch E,, the objective tries to pull its projection E, closer to the projections of other co-occurrngg image patches and to push away non-co-occurring image patches' projections. For easier reference, we denote this process as BagSSL
Bag-of-LLcal-Feattee
3

---

Published in Transactions on Machine Learning Research (11/2023)
2
811
81
9
Embedding
h;
Ring
Fiver
Image
Augmentation
Embecdding-roojsction
Augpool
Figure 1: BagSSL pipeline. From each image, fixed-size image patches extracted, color-augmented, and encoded to embedding and projection space. During training, patch projections of co-occurring patches (from the same image) are pulled together while an anti-collapse regularization is applied to push non-co- occurring patch projections away. After training, patch embeddings {h} from the same image averaged to form the image representation Ramg.
4 Empirical Results
In this section, present series of empirical results and test BagSSL with several representative joint- embedding SSL methods (SimCLR(Chen et al., 2020a), VICReg(Bardes et al,, 2021), TCR(Li et al., 2022), and BYrllGrill et al., 2020)) on four standard benchmarks (CIFAR10, CIFAR100, ImageNet-100, and ImageNet-1KK. Through experiments, we have the following observations:
BagSSL achieves similar or even better results than baseline methods. Even with 32 32 patch representation, BagSSL achieves 62% top-1 lincar probing accuracy on ImageNet.
The aggregated patch embedding approximately converges to the whole-image representation in baseline SSL methods with respect to the number of patches aggregated.
Through visualization, we show that the smaller scale patch representation preserves locality better.
We can leverage BagSSL to further improve the representation quality of the baseline methods.
The first two observations show from two different angles that a patch-based SSL representation can po- tentially explains the performance of the baseline methods. The third observation shows how patch repre- sentation supplements the prevailing invariance perspective. And the fourth observation provides practical benefits in addition to understanding.
4.1 BagSSL Versus Baseline Methods
We first test BagSSL with four baseline methods. The only difference between BagSSL and the baselines during training is that we replace multi-scale cropping augmentation with fixed-scale image patches. Simi- larly, two image patches are randomly selected from each image in the batch. While we mainly present linear probing accuracy, Figure shows that k-NN evaluation consistent with linear probing evaluation. During the evaluation, we show standard central crop evaluation results for both BagSSL and baselines. Further, show the multi-patch aggregated evaluation and multi-crop aggregated evaluation. In the tables, "patch" mcans that fixcd-scale patch embeddings are aggregated, and "crop" mcans that multi-scale crop embeddings are aggregated. All implementation details can be found in Appendix C. The main results are highlighted in Table 1, Table 2, Table 3, where BagSSL matches the baselines or surpass them on CIFAR10, CIFAR100, and ImageNet-100 datasets. Figure 3, we show the results on ImageNet- IK. Interestingly, with 32 x 32 patch representation, BagSSL achieves 62% top-1 linear probing accuracy. There is a 5.3% performance gap between Bag and VICReg ImageNet-IK, possibly due to the engineering issue.
4

---

Published in Transactions on Machine Learning Research (11/2023)
62
30
12:12
Ratch Stn. Bigio
Mull Scale Augrinnttiinn
Patch Soa Rxg651
Netth Scale Aug momtartionn
Probing
[bJ
Figure/ 2: Linear probing and kNN evaluation on CIFAR-10 are consistent. We evaluate the perfor- mance of linear classifier (a) and a k-NN classifier (b) for pretraining with various patch sizes and various evaluation setups. During pretraining, BagSSL uses fixed-scale patches, and bascline methods use multi-scale augmentation RandomResizedCrog (min scale, max max_scaa max_scale). For example, RandomResizedGi
(0. (8.08, 1. 1.0) corresponds to randomly and uniformly select crops ranging from 9 x to 32 x 32. The patch size and augmentation crop size range are marked in the figure. The "Central" evaluation is the standard evaluation protocol where the classifier trained and evaluated on single fixed central patches of the image, which is the entire image for CIFAR-10. For the n patch evaluation, the classifier trained and evaluated on the linearly aggregated embedding of patches, sampled with the same scale factor during pretraining. Please note that the "central" evaluation is expected to perform poorly on fix-scale pretraining the model has never seen the entire image during pretraining.
(min_scale,
Table 1: Performance on CIFAR-10 with patch-based and standard multi-Scale SSL pre- training. We evaluate the performance of a linear classifier with BagSSL and baseline SSL methods. BagSSL uses patch-based training, where 14 x 14 image patches are sampled during pretraining. Baseline mothods use multi-scale training, where the patch scale is uniformly sampled between scale 0.08 and 1.0, which means that multi scale crops are sampled uniformly from 9 x 32 patches.
Method
Patch Patch-based troining evaluation
Multi scale training evaluation
Central crop
patch
16 patches
256 patches
Central crop
crop
16 crops
256 crops
SimCLR
46.2
82.1
90.5
90.8
90.2
86.4
91.6
91.8
TCR
46.0
82.2
90.4
90.8
90.1
86.5
91.5
91.8
VICReg
47.1
83.1
90.9
91.2
90.7
87.3
91.9
92.0
BYOL
47.3
83.6
91.3
91.5
90.9
87.8
92.3
92.4
CIFAR. We first provide experimental results the standard CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009) using ResNet-34. During training, fixed-scale image patches used for BagSSL, and baseline methods RandommResizedCCopp(0 to sample from x to 32 x 32 patches as the crops. Sampled patches and crops are upsampled to a uniform 32 x2 resolution before embedding by ResNet-34. The standard evaluation method generates the embedding using the entire image, both during training of the linear classifier and at final evaluation, and is denoted as (Central). An alternative evaluation that an image embedding generated by inputting a certain number of patches (same scale training time and upsampled) into the neural network and aggregating the patch embeddings by averaging. The number of aggregated patches is marked in the Figures and Tables. In Figure 2, we show that lincar probing and k-NN evaluations are consistent. In the following, we mainly present linear probing evaluation. The results on CIFAR-10 and CIFAR-100 are shown in Tables 1 and 2 respectively. The main observation is that pretraining on small patches and evaluating with the averaged embedding performs on par or better than the baseline methods, as highlighted. The central crop evaluation performs significantly worse for patch-based based training expected since the network has never seen a whole image during training.
1.0) augmentation, which means that we randomly choose

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: Performance on CIEAR-100 with patch-based and standard multi-scale SSL pre- training. We evaluate the performance of linear classifier with BagSSL and baseline SSL methods. BagSSL uses patch-based training, where 14 x 14 image patches sampled during pretraining. Baseline methods use mulli-scale training, where the patch scale is uniformly sampled between scale 0.08 and 1.0, which means that multi-scale crops are sampled uniformly from x 9 to 32 x 32 patches.
Method
Patch-based training evaluation
Multi training evaluation
Central crop
1 patch
16 patches
256 patches
Central crop
crop
16 crops
256 crops
SimCLR
34.7
59.4
67.2
67.4
66.8
60.5
68.2
68.3
TCR
34.6
59.2
67.1
67.3
66.8
60.5
68.1
68.3
VICReg
35.5
60.1
68.0
68.3
61.4
69.0
69.3
BYOL
37.4
60.9
68.9
69.2
68.8
62.3
69.7
69.9
Table 3: Performance ImageNeet-100 with patch-based and standard multi-scale SSL pre- training. We evaluate the performance of a linear classifier with BagSSL and baseline SSL methods. BagSSL uses patch-based training, where 100 100 patches are sampled during pretraining. Baseline methods use multi-scale training, where the patch scale uniformly sampled between scale 0.08 and 1.0, which means that multi scale crops are sampled uniformly from 64 x to 224 x 224 patches.
Method
Patch- training evaluation
Multi-scale training evaluation
Central crop
1 patch
16 patches
48 patches
Central crop
1 crop
16 crops
48 crops
SimCLR
41.4
45.7
76.1
76.2
77.5
70.3
78.6
79.0
TCR
41.3
45.6
76.1
76.3
77.3
70.1
78.5
78.8
VICReg
42.1
16.1
76.8
76.9
77.8
70.7
79.1
79.4
BYOL
42.9
47.3
77.9
77.7
78.0
71.1
79.4
80.1
ImageNNet. We also provide experimental results on ImageNet- and ImageNet datasets Deng et al. (2009) with ResNet- 50 and linear probing evaluation protocol. ImageNet 100 and ImageNet results are shown Table and Figure 3(b) respectively. The behavior observed on CIFAR-10 gencralizes to ImageNet-1100 Averaging embeddings of small patches produced by the patch-based pretrained models perform comparably to standard "central" evaluation of the embedding produced by the baseline models, highlighted in Table 3. In Figure 3(b), we show the results ImageNet. The patch-based pretrained model achieves 67. 67.9% top-1 accuracy with 16 patch embedding averaged, whereas the baseline method VICReg achieves 73 2% top-1 accuracy. This 5.3% performance gap might be due to sub-optimal hyperparameters as we did not optimize the hyperparameters for patch-based training pretraining. Interestingly, with 32 x 32 patch representation, BagSSL achieves 62% top-1 linear probing accuracy.
4.2 Patch Embedding Aggregation Approximately Converges to the Whole-Image Embedding
In this experiment, we show that for multi-scale pretrained baseline SSL model, VICReg, linearly aggre- gating the patch embedding approximately converges to the whole-image embedding. This baseline VICReg network checkpoint has been trained on image crops ranging from 64 x 64 to 224 x 224. We randomly select 512 images from the ImageNet dataset. For each image, we first get the 224 x 224 center crop embedding. We also average embeddings similarity between the aagch-agggegated and center-crop embedding. Figure 3(a) shows that the aggregated embedding approximately converges the whole-image embedding as N increases from 16 all the image patches?.
random 100 x 100 patches for each image. Then we calculate the cosine

---

Published in Transactions on Machine Learning Research (11/2023)
(b)
0.255
pran
0.909
0.75
0.850
0.750
0.25
humber aggregated
Pasch
Figure 3: (a) Patch embedding approximate convergence to the instance embedding. For a baseline multi-scalc pretrained VICReg modcl, we show that the patch embedding aggregation approximately converges to the whole-image embedding as the number of aggregated patches increases. We evaluate the cosine similarity between the aggregation of N patch embeddings and the whole whole-image embedding. N is selected from 1, 16 and all possible patches in the image. (b) Linear evaluation ImageNet for various BandonResizedCrop scales. We show the performance of linear classifier for various pretraining and evaluation settings. BagSSL. uses fixed-scale patches, and the baseline method, VICReg, uses multi-scale augmentation RandomResizedCrrop (0. 08, 1. 1.0), which corresponds to randomly and uniformly select crop scale ranging from 64 64 to 224 x 224. For each pretraining setting, we provide three different evaluations: 1) "Central": the standard evaluation, which takes a 224 x 224 crop; 2) "1 patch": this evaluation takes patch or crop following the corresponding pretraining setting; 3) "16 patches": this evaluation takes
randomly selected patches or crops following the corresponding pretraining setting.
Projection
Embedding
Projection
Embedding
(a) 71x71 Patch
(b) 100x100 Patch
Figure 4: Visualization of cosine similarity in the projection space and the embedding space. Query patch indicated by red dash. Projection and Embedding cosine-similarity heatmaps use the same color scaling. The projection vectors are significantly more invariant compared the embedding ones, and the embedding space contains localized information that is shared among similar patches, when the size of the patches is small enough. We can that the embedding space tends to preserve more locality compared to the projection space.
4.3 Patch Representation Preserves Locality
Joint-embedding SSL methods are primarily motivated from invariance perspective. While this perspec- tive is relatively accurate globally, show that locality is preserved when into local patch level representation. In this section, provide nearest neighbor visualization on CIFAR-10 and cosine-sinilarity
"AAl:: extracting overlapped patches with stride and totally aggregate about 1000 patches' embeddings per image.

---

Published in Transactions on Machine Learning Research (11/2023)
Projection KNN
Embedding KNN
Projection KNN
Embedding KNN
Figure 5: Visualization of kNN in the projection space and the embedding space for CIFAR10. Distance is calculated by cosine similarity. The query patch is in the top left corner encircled by red-dash box, the green box indicates patches from other images of the same class. Patches without surrounding box is from the same image as the query. While the nearest neighbors are both from same-category instances, we can see that the embedding space tends preserve the local part information, whereas the projection space may collapse different parts of the same category.
heatmap visualization on ImageNet to show this point and provide deeper understanding of the learned representation.
First, we take baseline VICReg model pretrained on ImageNet, for a given image patch (e.g., circled by red dash boxes in Figure 4), we visualize the cosine-similarity between embedding from this patch and that from the other same-size patches from the same image. The heatmap visualization is normalized to the same scale. While the cosine-similarrit heatmnp for large (100 x 100) patches more invariant, as shown Figure 4(b), the heatmap for smaller (71 71) patches preserves the locality relatively well demonstrated in Figure 4(a).
Next, pre-train model with only 14 14 image patches CIFAR-10 and calculate the projection and embedding vectors of all different image patches from the training set. Then for given 14 x 14 image patch (e.g,, the ones circled by red dash boxes Fig 5), we visualize its k nearest neighbors in terms of cosine- similarity in both the projection and the embedding space. Figure shows the results for two image patches. The patches circled by green boxes are image patches from another image of the same category, whereas the uncircled patches are from the same image.
Overall, observe that the projection vectors significantly more invariant than the embedding vectors. This is apparent from Figure 4 and Figure 5. For the CIFAR kNN patches, NNs in the embedding space are visually much more similar than NNs in the projection space. In the embedding space, the nearest NNs are mostly locally shifted patches of similar "part" information. For projection space, however, many NNs are patches of different "part" information from the same class, E.g., we can see in Figure 5 that an NN of "whecl" in the projection space might be a "door" or a "window". However, the NNs the embedding space all contain "wheel" information. In the second example, the NNs of "horse legs" patch may have different "horse" body parts, whereas the NNs in the embedding space all "horse leg".
The heatmap visualization on ImageNet also illustrates the same phenomenon. The projection vector from patch is highly similar to that from the query patch whenever the patch has enough information to infer the class of the image. For embedding vectors, the similarity area much more localized to the query patch or other patches with similar features (the other leg the dog Figure 4). This general observation is consistent with the results of the visualizations in Bordes al. (2022). A more thorough visualization provided in Appendix E.
8

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4: Linear evaluation with aggregated embedding on ImageNet with models trained with state-of-the-art SSL methods. Using aggregated embedding outperforms embedding from the center crop. Central: Embedding from the center cropped image is used in training and testing using the standard linear evaluation protocol. 1, 16, and 48 crops: The linear classifier is trained and evaluated on the aggregated embedding of 1, 16, and 48 crops, respectively, sampled with the same scale factor range as during pretraining (0.08, 1.0).
Method
Central crop
1 crop
crops
48 crops
SimCLR
69.3
54.3
71.0
71.3
TCR
69.0
54.1
70.7
71.1
VICReg
73.2
57.6
74.2
74.4
BYOL
74.3
59.3
75.4
75.6
Table 5: Evaluation of SOTA SSL models and these models with Iinearl-aggreggated patches embedding enhancement. All the baseline SSL model uses ResNet-18 as the backbone. We apply spatial average pooling on the last layer output of ResNet-18 and treat it as feature. We evaluate the performance of these checkpoints with both a linear classifier and a K-nearest-neighhor (KNN) classifier. For the "Enhancement" evaluation, the KNN classifier is evaluated on the Amearly-aggregate embedding of patches with size 16 16. These patches are sampled using a sliding window with stride 4.
Method
Baseline (KNN) Baseline (Linear) Local Aggregation (KNN) Local Aggregation (Linear)
SimCLR
90.2
90.7
93.1
92.7
VICReg
90.8
91.2
93.1
92.2
BYOL
91.5
92.6
93.5
94.0
4.4 Patch-Based Evaluation Enhances the SSL Baselines
Global Aggregation. The results in Section 4.1 show that the best performance is obtained when the pretraining step uses multi-scalc crops and the evaluation step uses the aggregated patch emboddings. Here, we evaluate the aggreeaggee representation with four baseline methods pretrained on ImageNet to further confirm this performance boost. All models are downloaded from their original repository. Table 4 shows the linear evaluation performance on the validation set of ImageNet using the whole-image embedding and patch- aggrrgate embedding. For all these models, aggregated embedding outperforms whole-image evaluation, often by more than 1%. Also, increasing the number of patches averaged in the aggregation process improves the performance. We do not go beyond 48 patches due to engineering limits on memory and runtime. Still, we hypothesize that a further increase in the number of patches will improve the performance, as demonstrated on CIFAR-10, where 256 patches significantly outperform 16 patches.
Local Aggregation. In this work, the embedding aggregation is global or uses bag-of-local features. The drawback of doing is that spatial information be lost. An alternative way to aggregate patch embeddings is local averaging and concatenating locally averaged embeddings into single long vector. Empirically, such method tends to outperform the global average significantly. To show the results, we use the checkpoints of the SOTA SSL model pretrained on the CIFAR10 dataset from SoloLearn (da Costa et al., 2022) and tested linear and kNN accuracy with local concatenation aggregation. As shown in Table 5, concatenation aggregation further improve the performance of these SOTA SSL model. Even with only 25 patches, the k-NN accuracy of the aggregated embedding outperforms the baseline linear evaluation accuracy by larger margin compared to global aggregation results shown in Table 1.

---

Published in Transactions on Machine Learning Research (11/2023)
5 Discussion
In this paper, we seek to understand the success of joint-embedding SSL methods. We establish a formal connection between joint-embedding SSL and the co-oceurrence of image patches. We demonstrate learning embedding for fixed-size image patches and linear aggregating the local patch embeddings can achieve similar even better performance than the baseline models pretrained with multi-scale crops. On the other hand, with a multi-scale pretrained model, we show that the whole image embedding is approximately the average of local patch embeddings. Civen this, invariance is expected at the global scale. Through visualization, we show that the locality is preserved when we zoom into local patch level representation. These findings supplement the prevailing invariance perspective and show that there is a distributed representation of local image patches behind the success of joint-embedding SSL. The insights from this angle also help us enhance the represcntation quality from the bascline methods.
During training, the computational cost for BagSSL and the corresponding baseline is almost exactly the same. During inference, the computational cost for BagSSL is N time larger than the baseline (central crop eval), where N the number of aggregated patches. Although the eval computation is much higher, this disadvantage is somewhat tangential to the goal of this work, which is to provide an understanding of self- supervised learning. One might question whether BagSSI, increases the representation capacity by having much computation during evaluation. Though BagSSL has much more computation during evaluation, it does not necessarily have much more capacity since linearly aggregate the patch representation, which is essentially bag-of-local-fetuur approach. For example, Table 1 shows that for multi-scale training evaluation, linear aggregating of the crop representations outperforms the default evaluation for the baseline methods. However, the capacity improvement is limited.
There are few limitations to this work. First, we still use the commonly used projector head, whose role is yct to be fully understood. Sccond, we adopted the convention of two-crop training for BagSSL. If joint- embedding SSL aims model the co-occurrence of patches, two local patches would be relatively inefficient and create disadvantage during training. Third, we haven't optimized the hyperparameters thoroughly for BagSSL on ImageNet-1K due to enginecring resource limitations. Fourth, the results from this work only provides a potential explanation to the SSL baselines, and it does not decisively show that the SSL methods is only learning a patch-based representation. Given the second and third limitations, BagSSL results may further improved, given better engineering. We leave these to our future work.
Acknowledgments
Wc thank Zeyu Yun for helping us with the local aggregation evaluation and providing feodback during the preparation of this paper. We thank many colleagues from CCN/CCM@Flatiro Institute for the valuable comments.
References
Hangbo Bao, Li Dong, Songhao Piao, and Furn Wei. Beit: BERT pre-training of image transformers. In The International Conference on Learning Representations, ICLR 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance seriance-invariancce
cotprrriiaanccoovvaancaee covariance regularization for self-supervised learning. In International Conference on Learning Representations, ICLR, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. In Advances in Neural Information Processing Systems, NewrIPS, 2022.
Florian Bordes, Randall Balestriero, and Pascal Vincent. 1lgh fidelity visualization of what your self- supervised representation knows about. Transactions on Machine Learning Research, TMLR, 2022.
Wieland Brendel and Matthias Bethge. Approximating cmns with bag-of-local-featuree models works sur- prisingly well imagenet. In International Conference on Learning Representations, ICLR, 2018.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification us- ing a' siamese' time delay neural network. Advances in Neural Information Processing Systems, NewrIPS, 1993.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, ICML, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning. arviv preprint arxiv:2003 arXiv:2003.42297, 2020b.
Yubei Chen, Dylan Paiton, and Bruno Olshausen The sparse manifold transform. Advances in Neural Information Processing Systems, NeurIPS, 2018.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self supervised methods for visual representation learning. Journal of Machine Learning Research, JMLR, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the Conference Computer Vision and Pattern Recognition, CVPR, 2009.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Trans- actions on Pallern Analysis and Machine Intelligence, PAMI, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In The International Conference on Learning Representations, ICLR, 2021.
Susan T Dumais. Latent semantic analysis. Annual Review of Information Science and Technology, ARIST, 2004.
Aleksandr Ermolov,, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-suervised representation learning. In International Conference on Machine Learning, ICML, 2021.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. The International Conference on Learning Rep- resentations, ICLR, 2023.
Spyros Gidaris, Andrci Bursuc, Nikos Komodakis, Patrick Pérez, and Matthicu Cord. Learning represen- tations by predicting bags of visual words. In Proceedings of the Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
Jean-Bastinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self- supervised learning. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self seff-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, NeurIPS, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation learning. In Proceedings of the Conference on Computer Vision and Pottern Recognition, CVPR, 2020.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In The Conference on Computer Vision and Pattern Recogration, CVPR, 2022.
Alex Krizhevsky, Geoffrey Hinton, and et al. Learning multiple layers of features from tiny images. 2009.
Quoe Le, Alexandre Karpenko, Jiquan Ngiam, and Andrew Ng. Ica with reconstruction cost for efficient overcomplete feature learning- Advances in Neural Information Processing Systems, NeurIPS, 2011.
Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich T Sommer. Neural manifold clustering and embedding. arXiv preprint ar Xr1::2201.10000, 2022.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In The International Conference on Learning Representations, ICLR Workshop Track Proceedings, 2013a.
Tomas Mikolov, Iya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in Newrul Informalion Processing Systems, NewrIPS, 2013b.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word represen- tation. In Proceedings of the Conference on Empérical Methods in Natural Language Processing, EMNLP. 2014.
Sam T Roweis and Lawrence K Saul. Nonlincar dimensionality reduction by locally lincar embedding. Science, 2000.
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back- propagating errors. Nature, 1986.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 2000.
Louis Thiry, Michael Arbel, Eugene Belilovsky, and Edouard Oyallon. The unreasonable effectiveness of patches in deep convolutional kernels methods. In The International Conference on Learning Representa- tions, ICLR, 2021.
Asher Trockman and J. Zico Kolter. Patches are all you need? Transactions on Machine Learning Research, TMLR, 2023.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Newral Computation, 2002.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via -on-paraettric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR, 2018
Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Lyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In The European Conference on Computer Vision, ECCV, 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, ICML, 2021.
12


Reviews:
[{"review_id": "sbPKnPwz4j", "reviewer": "Reviewer_tLRJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The patch-based SSL idea is interesting and it is competitive compared to the standard SSL approach which uses multi-scale crop augmentation.\n2. The patch aggregation method improves even the baseline SSL methods which were trained with multi-scale augmentations.\n3. Overall the writing is clear. \n\n## Weaknesses\n\n1. There are unsubstantiated claims.\na. SSL learns a representation of patches (abstract and introduction) is unsubstantiated as the paper doesn't show it without a doubt in my opinion. The BagSSL method performing similarly or patch aggregation improving SSL methods does not justify this claim.\nb. Proposition 3.1 only shows that patch-based SSL loss is similar to co-occurrence loss and it doesn't show this for a general SSL which uses full image embeddings.\nc. Sec 4.1 claims the performance difference between BagSSL and VICReg is due to engineering issues or suboptimal hyperparameters which is unsubstantiated.\n\n2. BagSSL learns embeddings of patches and therefore, an image embedding is obtained by patch aggregation (ie, an image corresponds to a set of patch embeddings). One could think about this as BagSSL having more capacity to embed an image. Nevertheless, BagSSL is inferior to multi-scale crop augmentation with baseline SSL. Can you comment on this?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a patch-based SSL method named BagSSL. The idea is to learn embeddings of image patches with SSL where patches from the same image are considered positive. They also provide a patch aggregation method (mean of patch embeddings) to obtain image embedding. Results are provided on cifar-10/100, and imagenet-100/1k with SimCLR, TCR, VICReg and BYOL methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clear justification for the main claims would strengthen the paper and position it correctly in the literature.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2aHcteC03J", "reviewer": "Reviewer_5Jcd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n----\n\n1. The proposed method is simple and effective. It improves the performance of various self-supervised learning baseline algorithms. \n2. The paper discusses the relationship between spectral contrastive loss and co-occurrence statistics loss. \n3. The analysis of aggregated patch embedding shows that it converges to the image-level representation of baseline algorithms. \n4. The effectiveness of the proposed method was demonstrated in both single-scale and multi-scale augmentation settings.\n\nWeaknesses\n----\n\n1. The presentation of this paper, especially the writing, needs to be improved. It does not read well.\n2. Table 1,2,3.  do not directly compare with baseline algorithms. For instance, Table 1 shows CIFAR-100 performance. However, the performance is significantly lower than the baselines. SimCLR achieved about 80.2 in the original paper. But Table 2 the proposed method with SimCLR achieved 67.4 in Patch-based training evaluation and 68.3 in multi-scale training evaluation. The authors need to provide experimental results that are compatible with the original self-supervised papers. \n3. Considering the simplicity of the proposed method, the method should provide strong empirical/practical gain. But as discussed above, the proposed method shows poor performance.\n4. Compared to Multi-scale training, the gain from patch-based training is relatively small.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a simple technique that learns image patch embeddings and aggregates local patch embeddings as image-level representations. The authors named the proposed method as BadSSL. Empirically the paper shows that the proposed method improves baseline self-supervised learning methods. Also, the authors prove that the spectral contrastive loss function is equivalent to co-occurence statistics modeling loss function.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Table 5. Does Local Aggregation improve linear as well? The performance of baselines is reported using both KNN and linear. But the performance of the proposed method/Local aggregation with linear was not reported. If the results are included, it will be more comprehensive and fairer.\n* Computational costs. Since the proposed method uses many patches, it might require much larger computing resources. But. the computational costs and efficiency are not explicitly compared/discussed.\n* Does the proposed method use the same data augmentation? It can be viewed as a type of data augmentation.", "broader_impact_concerns": "This work has no broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sN9w3KNcfQ", "reviewer": "Reviewer_QgnZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n\n- **Intuitive approach.** The introduced approach in defining SSL through the lens of co-occurrence of image patches provides a fresh and insightful perspective, potentially reshaping the understanding and application of SSL in image representation.\n\n**Weaknesses.**\n\n- **Computational cost.** While the authors have focused on the performance improvement from patch aggregation, they do not report the increase of computational cost due to the multiple inferences of each patch. To fairly argue the benefit of patch aggregation compared to Central crop (Tables 1 to 3), it should be discussed.\n- **Lack of important baselines.** In Section 4.4, the authors further demonstrate the usefulness of patch embeddings by showing that their aggregation can boost the performance of baseline SSL. However, the similar performance boosting with aggregation can be considered without patch. As the authors already have shown in Tables 1 to 3, crop-based aggregation might be able to outperform patch-based aggregation. But, such results are not provided in Tables 4 and 5.\n- **Different messages between Section 4.1 and 4.2.** The authors have denoted that “averaged embedding performs on par or better than baselines (Central)” in Section 4.1. Also, the improvement is enlarged as the number of patches is increased. At the same time, in Section 4.2, they have also denoted that aggregated embedding converges to the whole-image embedding, which is obtained with center crop. Then, does it mean that the performance of patch aggregation is increased until the specific number of patches, then it start to decreased and converge to performance of central crop?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper challenges the traditional understanding of Self-Supervised Learning (SSL) for image representation, proposing that joint-embedding SSL primarily learns through the co-occurrence of image patches. Introducing the concept of Bag of Self-Supervised Learning (BagSSL), the authors demonstrate improved or comparable results to baseline methods by aggregating local patch representations. They further show that, in a multi-scale pre-trained model, the whole image embedding can be approximated as the average of these local patch embeddings. Evidence is provided that this approach enhances various state-of-the-art baseline methods. The paper argues that patch representation simplifies the understanding of SSL and helps demystify self-supervised representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requests**\n\n1. Discussion about computational cost \n2. Comparison with crop aggregation in Tables 4 and 5", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows that self-supervised learning (SSL) approaches based on Siamese network architecture could be interpreted as learning a representation of image patches, which reflects their co-occurrence. The reviewers originally had a number of concerns due to unsubstantiated claims and insufficient experiments. Through the discussion phasse, these concerns were mostly resolved; consequently, two Reviewers are leaning to accept while the one Reviewer (5Jcd) keeps the original score leaning to the reject. The idea of patch-level contrastive learning is not a new idea as pointed by Reviewer 5Jcd. However, adding a new perspective to analyze SSL methods based on patch-level co-occurrence is interesting. Overall, this is an interesting and useful paper; therefore, I recommend its acceptance. As in the response to author’s reply for Reviewer tlRJ, deeper analysis or rewording the claims should be included in the final draft.
Decision: Accept (poster)