Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding
Authors: Tianyang Hu
Zhili LIU
Fengwei Zhou
Wenjia Wang
Weiran Huang
Venue: ICLR 2023 poster (2023)
Keywords: theoretical understanding
contrastive learning
stochastic neighbor embedding

Abstract: Contrastive learning, especially self-supervised contrastive learning (SSCL), has achieved great success in extracting powerful features from unlabeled data. In this work, we contribute to the theoretical understanding of SSCL and uncover its connection to the classic data visualization method, stochastic neighbor embedding (SNE), whose goal is to preserve pairwise distances. From the perspective of preserving neighboring information, SSCL can be viewed as a special case of SNE with the input space pairwise similarities specified by data augmentation. The established correspondence facilitates deeper theoretical understanding of learned features of SSCL, as well as methodological guidelines for practical improvement. Specifically, through the lens of SNE, we provide novel analysis on domain-agnostic augmentations, implicit bias and robustness of learned features. To illustrate the practical advantage, we demonstrate that the modifications from SNE to $t$-SNE can also be adopted in the SSCL setting, achieving significant improvement in both in-distribution and out-of-distribution generalization.

Published as a conference paper at ICLR 2023
YOUR CONTRASTIVE LEARNING Is SECRETLY DOING STOCHASTIC NEIGHBOR EMBEDDING
Tianyang Hu', Zhili Liu Fengwei Zhou', Wenjia Wang Weiran Huang
Huawei Noah's Ark Lab," Hong Kong University of Science and Technology
3 Hong Kong University of Science and Technology (Guangzhou)
Qing Yuan Research Institute, Shanghai Jiao Tong University
ABSTRACT
Contrastive learning, especially self-supervised contrastive learning (SSCL), has achieved great success in extracting powerful features from unlabeled data. In this work, we contribute to the theoretical understanding of SSCL and uncover its connec- tion to the classic data visualization method, stochastic neighbor embedding (SNE) (Hinton & Roweis 2002), whose goal is to preserve pairwise distances. From the per- spective of preserving neighboringinformatio. information, SSCL can be viewed as a special case of SNE with the input space pairwise similarities specified by data augmentation. The established correspondence facilitates deeper theoretical understanding of learned features of SSCL, as well methodological guidelines for practical improvement. Specifically, through the lens of SNE, we provide novel analysis on -agosic augmentations, implicit bias and robustness of learned features. To illustrate the prac- tical advantage, we demonstrate that the modifications from SNE to I-SNE (Van der Maaten & Hinton 2008) can also be adopted in the SSCL setting, achieving signif- icant improvement in both in-distribution and out-of-distribution generalization.
INTRODUCTION
Recently, contrastive learning, especially self-supervised contrastive learning (SSCL) has drawn massive attention, with many state-of-the-art models following this paradigm in both computer vision
(He et al. 2020a Chen et al. 2020a b Grill et al. 2020 Chen & He 2021| T, Zbontar et al.) 2021)
and natural language processing (Fang et al. 2020 Wu ct al. 2020 Giorgi et al. 2020 Gao et al. 202T Yan et al.) 202TT. In contrast to supervised learning, SSCL learns the representation through
large number of unlabeled data and artificially defined self-supervision signals, i.e., regarding the augmented views of data sample as positive pairs and randomly sampled data as negative pairs. By enforcing the features of positive pairs to align and those of negative pairs to be distant, SSCL produces discriminative features with state-of-the -frt-arft performance for various downstream tasks.
Despite the empirical success, the theoretical understanding is under-explored as to how the learned features depend on the data and augmentation, how different components in SSCL work and what are the implicit biases when there exist multiple empirical loss minimizers. For instance, SSCL methods are widely adopted for pretraining, whose feature mappings are to be utilized for various downstream tasks which are usually out-of-distribution (OOD). The distribution shift poses great challenges for the feature learning process with extra requirement for robustness and OOD generalization (Arjovsky)
al. 2019 Krueger al. 20221 Bai et al. 2021 | He al. 20200b Zhao al. 2023 Dong al. 2022 which demands deeper understanding of the SSCL methods.
The goal of SSCL. is learn the feature representations from data. For this problem, one classic method is SNE (Hinton et al., 2006) and its various extensions. Specially, t-SNE (Van der Maaten Hinton 2008 has become the go-to choice for low-dimensional data visualization. Comparing to SSCL, SNE is far better explored in terms of theoretical understanding (Arora et al. 2018 Linderman & Steinerberger 2019 Cai & Ma 202T). However, its empirical performance is not satisfactory, especially in modern era where data are overly complicated. Both trying to learn feature representations, are there any deep connections between SSCL and SNE? Can SSCL take the advantage of the theoretical soundness of SNE? Can SNE be revived in the modern era by incorporating SSCL?
"Correspondence Weiran Huang (weiran.huang sjtu.edu.cn).

---

Published as a conference paper at ICLR 2023
In this work, we give affirmative answers to the above questions and demonstrate how the connections to SNE can benefit the theoretical understandings of SSCL, as well as provide methodological
guidelines for practical improvement. The main contributions summarized below.
We propose a novel perspective that interprets SSCL methods as a type of SNE methods with the aim of preserving pairwise similarities specified by the data augmentation.
The discovered connection enables deeper understanding of SSCL methods. We provide novel theoretical insights for Nomain-agnossie data augmentation. implicit bias and OOD generalization. Specifically, we show isotropic random noise augmentation induces 12 similarity while mixup noise can potentially adapt to low-dimensional structures of data; we investigate the implicit bias from the angle of order preserving and identified the connection between minimizing the expected Lipschitz constant of the SSCL feature map and SNE with uniformity constraint; we identify that the popular cosine similarity can be harmful for OOD generalization.
Motivated by the SNE perspective, we propose several modifications to existing SSCL methods and demonstrate practical improvements, Besides a ar--weiegee weighting scheme, we advocate to lose the spherical constraint for improved OOD performance and a I-SNE style matching for improved separation. Through comprehensive numerical experiments, we show that the modified L-SimCLR outperforms the baseline with 90% less feature dimensions on CIFAR-IN 10 and (-MoCo-v2 pretrained on ImageNet significantly outperforms in various domain transfer and OOD tasks.
2 PRELIMINARY AND RELATED WORK
For vector æ, ||ا||p denotes its p-norm, for 1 <ps cc. P(A) is the probability of event A. For random vari- able 2, we use P2 and P_ to denote its probability distribution and density respectively. Denote Gaussian distribution by NJ,,E and let L be the d x d identity matrix. the dataset be D, = ,..,"",, CRd where each æ; independently follows distribution Pz. The goal of unsupervised representation learning is to find informative low-dimensional features ",, e <R
Notations. For function f. let || = supaen (z)) and (f)
of Dn, where d, is usually much smaller than d. We use fx) to as the default notation for the feature mapping from IR + IRd, i.c., 2; = fxx)).
Stochastic neighbor embedding. SNE (Hinton & Roweis 2002) is powerful representation learning framework designed for visualizing high-dimensional data in low dimensions by preserving neighboring information. The training process can be conceptually decomposed into the following two steps: (1) calculate the pairwise similarity matrix P PEIR "x for D,; (2) optimize features =1 ,,,,,, such that their pairwise similarity matrix Q RR= matches P. Under the general guidelines lie plentiful details. In|Hinton& & Roweis (2002), the pairwise similarity is modeled as conditional probabilities of my being the neighbor of æ;, which is specified by Gaussian distribution centered at æt, i.c., when i i
exp( x x 1l
Pole Σ A exp( - x2 24
(2.1)
where or is the variance of the Gaussian centered at o,, Similar conditional probabilities Qiıe 's can be defined on the feature space. When matching Q to P. the measurement chosen is the KL-divergence between two conditional probabilities. The overall training objective for SNE is
PaK Qzi
inf ΣΣ
P
(2.2)
i=1j=1
Significant improvements have been made to the classic SNE. Im al. (2018) generalized the KL--divergence to f-divergence and found that different divergences favors different types of structure. Lu et al. (2019) proposed to make P doubly stochastic so that features are less crowded. Most notably, t-SNE Van der Maaten & Hinton| 2008) modified the pairwise similarity by considering joint distribution rather than conditional, and utilizes t-distribution instead of Gaussian in the feature space modeling. is worth noting that SNE belongs a large class of methods called manifold learning (Li et all In this work, we specifically consider SNE. If no confusion arises, we use SNE to denote the specific work offHinton & Roweis (2002) and this type of methods in general interchangeably.
Self-supervised contrastive learning. The key part of SSCL is the construction of positive pairs, or usually referred to different views of the same sample. For each æ; in the training data, denote
2

---

Published as a conference paper at ICLR 2023
its two augmented views to be x and x. Let Dy, = ,-),,..,, D (a" and define
cxp(sim(f((2),(('"")))
1(x1,,2="') = log
DED RED, exp(siml f(
where sim(21; 21, 22) 21 =a |a
denotes the cosine similarity and f temperature pa- rameter. The training objective of the popular SimCLR (Chen et al. 2020a can be written as LInIONCE: = 2n >x I ar()) +((r),)))))))
Recently, various algorithms are proposed to improve the above contrastive learning. To address the need for the large batch size, MoCo (He et al. [2020a Chen al. 2020b) utilizes mnviing-averaged encoder and a dynamic memory bank to store negative representations, making it more device-friendle Grill et al. (2020); Chen & He (202 [Zbontar et al. (2021 [ Chen et (2021) radically discard negative samples in SSCL but still achieve satisfactory transter performance. Another line of works (Caron et al.
2020 2021 Liu et 2022) mines the hierarchy information in data derive more semantically
compact representations, Radford et al. (021]; Yao et al. (2021) even extend the contrastive methods to the multi- data structure to achieve impressive zero-shot classification results.
Theoretical understanding of SSCL. In contrast of the empirical success, theoretical understanding of SSCL is still limited. While most of theoretical works (Arora et al. 2019 Tosh Tosh
al. 2020 Hao HaoChen
2021 202T]
et
2002 Wang et al. 2022 Wen & Li] [222 Wei et al. 2020 Huang et al. 2022 Ji et al. [202T]
Ma 2023) focus on generalization ability on downstream tasks, there are some works studying
specifically the InfONCE loss. One line of works (Oord et al. 2018 Bachman el al. 2019 Hjelm et al.
2018 Tian al. 2019 2020) understand the InfoNCE loss from mutual information perspective, show- ing that the negative InfoNCE is a lower bound of mutual information between positive samples. Other works (Wang & Isola 2020;
al. [02T] [ing et al. (202T) are from the perspective of geometry of embedding space, showing that InfoNCE can be divided into two parts: one controls alignment and the other prevents representation collapse. In this paper, we study SSCL. from the SNE perspective, which, to the best of the authors' knowledge, has no discussion in existing literature. The closest work ours is Balestriero & LeCun (2022), which proposed a unifying framework under the helm of spectral manifold learning. In comparison, work focus specifically on the connection between SSCL and SNE.
SNE PERSPECTIVE OF SSCL
closer look at the training objectives of SNE and SimCLR reveals great resemblance - SimCLR can be seen as a special SNE model. To see this, denote D2n,- = D" U'DY, as the augmented dataset with index 32 = x. and Ø2. x/. If we change the distance the negative cosine similarity and let of T.. Admitting similar conditional probability formulation as in (2. yields that for i j
exp(sim( ((a (a.). 2 Ex ( ((:) (xxx) ( )/T)
(3.1)
By taking
1, if and ã, are positive pairs Pjjs 0, otherwise,
(3.2)
the SNE objective (2.2) can be written as
2n
PJ - (11o
P
log
Rm
which reduces to the SimCLR objective LIMGNCE. up to a constant scaling term only depending on n.
Now that have established the correspondence between SNE and SimCLR, it's clear that the feature learning process of SSCL also follows the two steps of SNE.
(SI) The positive pair construction specifics the similarity matrix P.
(S2) The training process then matches Q to P by minimizing some divergence between the two specified by the training objective, e.g., KI. divergence in SimCLR.
3

---

Published as a conference paper at ICLR 2023
(a) Data Distribution
(b) 2- 2-sphere
(c) Id Fudlrdean
-10
(d) 2-sphere, OOD
(e) 1d Id-Euclidean, OOD
OOD mean shift
-22
Figure 1: Gaussian mixture setting with 5 components. (a) illustration of data with 250 samples. (b) learned features by standard SimCLR with normalization (cosine similarity) to 1-sphere. (c) learned features by modified SimCLR without normalization (/2 similarity). (d, e) feature mapping of the two methods in case of OOD mean shift. The linear classification accuracy is 48.4% in (d) and 100% in (e).
The main difference between SNE and SSCL is the first part, where the P in SNE is usually densely filled by I distance, ignoring the semantic information within rich data like images and texts. In contrast, SSCL omits all traditional distances in IR and only specifies semantic similarity through data augmentations, and the resulting P is sparsely filled only by positive pairs as in (3.2). For structurally rich data such as image or text, the semantic information invariant to wide range transformations. Human's 's prior knowledge of such invariance guides the construction of positive pairs in SSCI., which then learned by the feature mapping.
Remark 3.1 (SNE vs SSCL). We would like to clarify on the main difference between SNE and SSCL that we focus in this work. Although standard SNE (Hinton et al. (006) is parrametrii without explicit feature maps, and is optimized for the whole dataset, these are not the defining properties of SNE. SNE can also utilize explicit feature maps and mini-batch training (Van Der Maaten (009). On the other hand, SSCL can also benefit from larger/full batches (Chen et al. al.] 2020a) and can also be modified to directly optimize the features z, s.. In this work, we omit these subtleties] and focus on the (S1) perspective, which we view as the most significant difference between SNE and SSCL.
3.1 ANALYSIS
In this section, showcase the utility of the SNE perspective, we demonstrate how the feature learning process of SSCL methods, e.g., SimCLR, can become more intuitive and transparent. Specifically, re-derive the alignment and uniformity principle (Wang & Isola 2020) well provide novel analysis on domain- aid the illustration, we device toy examples with simulated Gaussian mixture data.
augmentations, the implicit bias and robustness of learned features. To
Gaussian mixture setting. Let data follow d-dimensional Gaussian mixture distribution with m components where Pa 1 , N o Ia). The special case with d = 2, m = 5, a 0.1 is illustrated in Figure aa) with 250 independent samples. To apply contrastive methods. consider constructing positive pairs by direct sampling, i.e., if æ is from the first component, then we sample another x' N (11, Ia) independently as alternative view for contrast. The negative samples are the same as in standard SimCLR training.
3.1.1 DOMAIN-AGNOSTIC DATA AUGMENTATION
Now that we have established in (S1) that the input space pairwise distance is specified by the data augmentation, a natural question to ask what are the corresponding induced distances. In this section, we investigate this problem for domain-agnose
data augmentations.
The quality of data augmentation has great impact on the performance of SSCL methods, which reflects people's 's prior knowledge on the data. However, when facing new data without any domain knowledge,
All the contrastive losses are written in full hatches for simplicity in this work as we focus on analyzing the optimal solutions of SSCL. methods rather than the optimization process.
4

---

Published as a conference paper at ICLR 2023
we have to rely on domain-agnoste data augmentations, e.g., adding random noises (Verma et al. 2021), for contrast. We first consider using general random noise augmentation. i... for any x E R", let x = +6 where ö follows some distribution with density ((x). Then, for any 2;, the probability density of having & ER as its augmented point can be characterized as Ptlar = (a; and a: form distance. a positive pair|æ:;) = x)). We have the following proposition on Gaussian-
Proposition 3.2 (Gaussian noise injection). If the noise distribution is isotropic Gaussian with mean zero, the induced distance is equivalent to the la distance in R up a monotone transformation.
Another popular noise injection method is the mixup (Zhang et al. 2017), where the augmented data are comprised of convex combinations of the training data. For each T;; a positive pair can be constructed from another Φ; such that x = I; + Maz - x;) and A (0,1) is the hyperparameter usually modeled with Beta distribution. For independent I1 22 Pz. denote the convoluted density of X(x1 222 as PA (a:), which is symmetric around O. Then, if employing mixup for positive pairs
in SSCL, the induced distance can be written as Pa: . Pa ... px -Z2).
Gaussian vs. mixup. Verma et al. (202T) proposed use mixup when domain- sppciiiii information is unattainable and provided supportive analysis on its advantage over isotropic Gaussian noise from the classification generalization error point of view. Through (S1) perspective, we can intuitively explain why data-dependent mixup noises can be potentially better from the perspective of the "curse of dimensionality". Consider the d-dimensional Gaussian mixture setting with m < d separated components. Notice that 11 Mm can take up at most (m - 1) -ddimnnsioa linear sub-space of Rd. Denoted the space spanned by f4; 's as Sn For the light-tailed Gaussian distribution, and the majority of samples will be close S Hence, majority of the convoluted density px px(x:) will also be supported on Sy., so does the corresponding P Thus, the induced distance from mixup will omit irrelevant variations in the complement of S₁, and focus on the oindimensional sub-space Sy, where J4 's actually differ. This effectively reduces the dimension dependence from d to m - 1. In comparison, isotropic Gaussian noise induces la distance for positive pairs with support of Rd which will be much more inefficient, especially when m < d. Since it is well-known that the performance of regression or classification models is strongly influenced by the intrinsic dimension of the input space (Hamm& & Steinwart 2021), keeping the data in a low-dimensional space is preferable.
3.1.2 ALIGNMENT AND UNIFORMITY
Characterizing the learned features of SSCL is of critical importance. Wang & Isola (2020) proposed alignment and uniformity principles for SimCLR type contrastive learning methods. Such results can be intuitively understood through the perspective of (S1) and (S2). Consider the common case where the feature space is (dg s)-sphere. First, (3.2) indicates that only similarities (distances) between positive pairs are non-zero (finite) and all other pairwise similarities (distances) are zero (infinity). Preserving (3.2) requires the features of positive pairs align (cosine similarity tends to 1) and those of negative pairs to be as distant as possible. If in the extreme case where positive pairs match exactly, i.e., f(z,) = fzx for any == 1.,..,m we call it perfect alignment.
If perfect alignment is achieved and the features are constrained on the unit sphere, matching implies pushing n points on the feature space as distant as possible. Maximally separated n points on a d-sphere has been studied in geometry, known as the Tammes problem (Tammes) 1930 Erber & Hockney| 1991 Melisseny| 1998). We say perfect uniformity is achieved all the pairs are maximally separated the sphere. There are some simple cases of the Tammes problem. If d = 2, perfect uniformity can be achieved if the mapped points form a regular polygon. d 2 n - 1, the solution can be given by the vertices of an (n 1) -issimelex, inscribed in an (n 1)-sphere embedded in Rd.. The cosine similarity between any two vertices is -1/(n- 1) and in this case, LIafANCE can attain its lower b As ns->-o oo. the point distribution converges weakly to uniform distribution. As can be seen in Figure Ta. b). perfect alignment and perfect uniformity are almost achieved by standard SimCLR in the Gaussian mixture setting.
3.2
As we will demonstrate in Section 3.1.4 that the spherical feature space can be bad for OOD generalization, adopting of the Euclidean space will change the statement of the uniformity property and can also be analyzed from the SNE perspective. Details can be found in Appendix A.5
Notice that in this case, the optimal feature mapping will contain little information of the data, mapping anchor samples interchangeahle points with identical pairwise distances
5

---

Published as a conference paper at ICLR 2023
3.1.3 IMPLICIT BIAS
Existing theoretical results on SSCL provide justification of its empirical success in classification. However, there is more to it than just separating different classes and many phenomena are left unexplained. Take the popular SimCLR (Chen et al. (020a) on CIFAR-10 as an example, we can consistently observe that the feature similarities within animals (bird, cat, deer, dog, frog. horse) and within objects (airplane, automobile, ship, truck), are significantly higher than those between animals This can be viewed as an implicit bias towards preserving semantic information, which
and objects] 31 might be surprising as we have no supervision on the label information during the training process. However, existing literature on implicit bias is scarce. As advocated in Saunshi et al. (2022), ignoring inductive biases cannot adequately explain the success of contrastive learning. In this section, we provide simple explanation from the perspective of SNE.
For a more concrete illustration, consider training SimCLR in the Gaussian mixture setting with d= 1, d, 2, m= (4, =i, and a = 0.1. Denote the 4 components in ascending order by A,B,C,D. Perfect alignment and uniformity imply that their feature maps (a, b, c. d) on the unit-circle should be vertices of an inscribed square. What left unsaid is their relative order. Clockwise regardless of the initialization, we can observe SimCLR to consistently produce the order a b c-> d.
Sounter-Clockwiie. from a,
Remark 3.3 (Relative ordering and with d = is mainly for illustration, as in one-dimension, the neighboring info is simplified as the order, which much easier to understand. The results remain the same in high dimensions as long as the clusters are well separated with obvious order of clusters. For instance, some relative orders in Figure b) are also stable, e.g., the neighbor of blue will consistently be purple and yellow.
The prder-preserving property showcased
With great resemblance SNE, SSCL methods also exhibit neighbor- prreeerrggg property and identify it as an implicit bias. Such implicit bias can be universal in SSCL and the phenomenon in Figure i.3 is also a manifestation. In deep learning, the implicit bias is usually characterized by either closeness to the initialization (Moroshko et al. 2020 Azulay et al. (02T). or minimizing certain complexity (Razin Cohen [220]]Zhang et al., 2021). In the case SimCLR, we hypothesize the implicit bias as the expected Lipschitz constant, which has deep connections to SNE with uniformity constraint. For feature map f onto the unit-sphere, define
[l f(x')
C()) Ex
(3.3)
x
where the æ2, are independent samples from the data distribution.
Definition 3.4 (SNE with uniformity constraint). Assume data ",,,,,,,,, ElRd the corresponding SNE features ø1, 2n dd are constrained to be the maximally separated points on the (dshhee,,,,,,, we call this problem SNE with uniformity constraint.
The key of SNE matching the pairwise similarity matrices Q to P. When solving SNE with uniformity constraint, the only thing to be optimized is the pairwise correspondence, or ordering of the mapping. We have the following theorem that links the beighbor-preserving property to (f)).
Theorem 3.5. Let "1, In Rd such that ||x; - æ,llı 0 for any i.j and let 21,".,,,, ,2,, R be maximally separated n points on the (d= - 11)---eee Denote P= (Pij)nx and Q (gi))xxe as the corresponding pairwise similarity matrices of x 's and 2, 's respectively. Let denote permutation on [I,...n} and denote all such permutations as T.. Let Q" as the n-permuted matrix Q and define
Ci
Σ
(x(1)=())
and π arrginn (P,, TET
Pij
Then, n² also minimizes |1-QQ where I| is the Frobenius norm and P = (Per Inx is a (monotonically) transformed similarity matrix with p --/ppj-
Theorem 3.5 showcases the relationship between minimizing C(J) and the structure preserving property by considering a special SNE problem, where the pairwise similarity is not modeled by Gaussian standard. Although 4; = =x)) f(z;) )] is unorthodox, it is reasonable since the larger the distance, the smaller the similarity. We have the following corollary to explain the neighbor- preserving property of SSCL and the implicit bias associated with minimizing the complexity C(f).
Figure A.3 illustrates the phenomenon. Details can be found in Appendix A.1

---

Published as a conference paper at ICLR 2023
Corollary 3.6 (Implicit bias of SSCL). When SSCL model achieves perfect alignment and perfect uniformity. if the complexity C(f) is minimized, the resulting feature map preserves pairwise distance in the input space, resembling SNE with uniformity constraint.
Corollary 3.6 links the implicit bias of SSCL to the SNE optimization with uniformity constraint, In the case of perfect alignment and perfect uniformity, SSCL, can be seen as a special SNE problem where the feature "n, must be maximally separated on the unit-sphere. Recall the I-dimension Gaussian case. There are in total 3! = 6 different orderings for the 4 cluster means, among which, a - b- d will give the lowest SNE loss. As can be seen in Figure A.4 both C(f) and the SNE loss are monotonically decreasing during training for the Gaussian mixture setting.
When the alignment uniformity is not perfect, the resulting feature mapping can still be characterized via SNE, with the uniformity constraint relaxed as a form of regularization. In our numerical experiments on the CIFAR-10 10 data, we observe C(J)) to be monotonically decreasing during the training process, supporting our hypothesis. More details can be found in Appendix(A.3 Corollary 3.6 sheds light on the implicit semantic information preserving phenomenon shown in Figure A.3 as in the input space, images of dogs should be closer to images of cats, than airplanes.
3.1.4 TARGETING OOD: EUCLIDEAN VS SPHERICAT.
Almost all SSCL methods require normalization to the unit-sphere and the similarity on the feature space is often the cosine similarity. In comparison, standard SNE methods operate freely on the Euclidean space. In this section, we show that the normalization can hinder the suunctur-prreerving and there is fundamental trade off between in-distribution and oft-of-domain generalization.
[a). Notice that as long as the mixing components are well separated, the learned feature mapping on the sphere will always be the pentagon shape, regardless of the relative locations of the clusters. This is a result of the uniformity property derived under spherical constraint. Distant clusters the input space will be pulled closer while close clusters will be pushed to be more distant, which results in the trade off between in-distribution and out-of-domain generalization. On one hand, close clusters being more separated in the feature space is potentially beneficial for in-distribution classification. On the other hand, the spherical constraint adds to the complexity of the feature mapping. potentially hurting robustness.
Consider the 2-dimensional Gaussian mixture setting as illustrated in Figure D
In the Euclidean space, pushing away negative samples (as distant as possible) will be much easier, since Fl and potentially preserve more structural information. To verify intuition, relax the spherical constraint in the Gaussian mixture setting and change the cosine similarity in SimCLR to the negative la distance in IR. The learned features are shown in Figure Πc). Comparing to F b). we can get the extra information that the purple cluster is far away the others. If we introduce a small mean shift to the data, moving the distribution along each dimension by 1, the resulting feature maps differ significantly in robustness. As illustrated in Figure vs. (c). the stan- dard SimCLR are much less robust to OOD shifts and the resulting classification accuracy degrades to only 48.4%, while that for the modified SimCLR remains 100% The same OOD advantage can also be verified in the CIFAR- 10 to CIFAR- 100 OOD generalization case (details in Appendix C.3 Figure C.8 and large-scale real-world scenarios with MoCo (Chen et al. 2020b) as baseline (details in Section
the feature vectors could diverge towards infinity
IMPROVING SSCL BY SNE
The proposed SNE perspective (S1, can inspire various modifications to existing SSCL methods. In this section, we choose SimCLR For empirical evaluation, we report the test classification accuracy of nearest neighbor classifiers on both simulated data and real datasets. Experiment details can be found in Appendix C
baseline and investigate three straightforward modifications.
4.1 WEIGHTED POSITIVE PAIRS
In practice, positive pairs are constructed from anchors (training data), by i.i.d. data augmentations, random resized crop, random horizontal flip, color jitter, etc. Take random crop as an example, pair 1 and 2 may be from 30%, 80% random crops, respectively. Their similarities should not be treated
'In practice, various regularization, c.g, weight decay, are employed and the resulting features will be bounded.
7

---

Published as a conference paper at ICLR 2023
0.9
82.1
81.9
Accuracy
80.7
80.8
80.9
80.6
Accuracy
0.7
0.5
0.3
SimCLR
71,1
SimCLL
0.1
10
0.1
16
32
Temperature
Feature Dimension
(a) Weighted SimCLR.
(b) SimCLR vs. f-SiMCLR.
Figure 2: Nearest neighbor classification test accuracy on CIFAR-10 with ResNet- 8 after 200 epochs pre-training. (a) "N/A" stands for the baseline SimCLR. The r-axis the temperature for IoU weighting scheme. (b) Comparison between SimCLR and t-SimCLR with different feature dimensions.
as equal, as in typical SSCL methods. Incorporating the disparity in the data augmentation process straightforward in the perspective of SNE, where the InfoNCE loss can be naturally modified as
1
Pa (t),(((,,)))))
2n
The weight pa in P can be specified manually to reflect human's 's prior knowledge. To test out the effect of such modification, we conduct numerical experiments on CIFAR-10 using the standard SimCLR. The weighting scheme is based on the Intersection over Union (IoU) of random resized crops. For each positive pair, let Par x (xpp(oo(z;;,a))) controlling the strength of the weighting scheme, i.e., the bigger the T', the closer to the unweighted state. The CIFAR- 10 test performance be significantly improved to 82. % if choosing ='=1. 11
where r' >>> 0 is a hyperparameter (temperature)
7' is shown in Figure a)). The baseline is 80.7% and can
4.2 T-SIMCLR: L-SNE STYLE MATCHING
Most SSCL algorithms differ mainly in (S2), i.e., defining Q and matching it P. where fruitful results in SNE literature can be mirrored and applied. Now that we have identified the advantage of modeling features in Euclidean spaces in Section 3.1.4] the most promising modification that follows is to introduce I-SNE to SimCLR. Since we are learning oow-dimensional features from high-dimensional data, preserving all pairwise similarities is impossible and the features tend to collapse. This is referred to as the "crowding problem" " in Van der Maaten & Hinton (2008) (see Section 3.2 therein). t-SNE utilizes the heavy-tail t-distribution instead of the light-tail Gaussian, to model Q and encourage separation in feature space. Correspondingly, the training objective LIMfONCE be modified
(1+ ff (ar))
(Lay +1)/2
log Eisjyason (1+1) 1// tar)) (tar +1//2
(4.1)
1=1
where tyy is the degree of freedom for the t-distribution. Besides substituting the cosine similarity to the l2 distance, the key modification is the modeling of feature space similarity Q, from Gaussian to I-distribution suggested Van der Maaten & Hinton (2008) avoid the crowding problem and accommodate the dimension-deffcience in the feature space. We call the modified method (-SimLLR and we expect it to work better, especially when the feature dimension is low, or in the OOD case.
Figure bbb) shows the comparison between SimCLR and t-SimCLR on CIFAR-10 10 with different feature dimensions, where (-SimCLR has significant advantages in all cases and the smaller the de. the larger the gap. Without decreasing the standard d. = 128, t-SimCLR improves the baseline from 80.8% 83.9% and even beats using only d_ with accuracy 81.7%.
Remark 4.1 (Degree of freedom). Standard t-SNE utilizes t-distribution with bu 1, to better accommodate the extreme d, = case. In practice, ty can vary and as d, increases, larger ty might be preferred. We recommend using Lag 5 as the default choice. The performance of Lar vs d, can be found Appendix C
as well as discussion on the fundamental difference between Lyf and r.
Remark 4.2 (Training epochs). For the CIFAR-10 experiments, we reported the results of ResNet-18 after 200 training epochs, similar to the setting of Yeh et al. (202T). We also conducted 1000-epoch experiments and found that our modifications provide consistent improvements throughout the training process, not in terms of speeding up the convergence, but converging to better solutions. Details can be found in Appendix C.1 and Figure C.6
8

---

Published as a conference paper at ICLR 2023
Table 1: Domain transfer results of vanilla MoCo-v2 and t-MoCo-v2.
Method
Aircraft
Birdsnap
Caltech101
Cars
CIFARIO
CIFARI00
DTD
Pets
SUN397
Avg.
MoCo-v
82.75
44.53
83.31
85.24
95.81
72.75
71.22
86.70
56.05
75.37
(-MoCo-v2
82.78
53.46
86.81
86.17
96.04
78.32
69.20
87.95
59.30
77.78
Table 2: OOD accuracies of vanilla MoCo-v2 and I-MoCo-v2 on domain generalization benchmarks.
Method
PACS
VLCS
Office-Home
Avg.
MoCo-v2
58.5
70.4
36.6
55.2
1-MoCo-12
61.3
75.1
42.1
59.5
5 LARGE SCALE EXPERIMENTS
In this section, we apply the same modifications proposed in Section 4.2 to MoCo-v2 (Chen et al. 2020b). as is more device-friendi, to conduct large scale experiments. We name our model Co Both models are pre-trained for 200 epochs on ImageNet following the setting of Chen ct al. (2020b). The linear probing accuracy of t-MoCo-v2 on ImageNet is 67.0%, which is comparable to the MoCo result 67.5%. With the same level of in-distribution classification accuracy, extensive experiments to compare their OOD performance. The results in Table [ and 2 suggest that our modification significantly improves the domain transfer and the OOD generalization ability without sacrificing in mn-distribution accuracy.
conduct
Domain Transfer. We first conduct experiments on the traditional suf-vsupevi domain transfer benchmark. We compare MoCo-v2 and f-MoCo-v2 on Aircraft, Birdsnap. Caltech 101, Cars, CIFAR CIFARTO0, DTD, Pets, and SUN397. We follow transfer settings in Ericcsson et al. (202T) to finetune the pre-trained models. The results are reported in Table Our model L-MoCo-v2 surpasses MoCo-v2 in out 9 datasets, showing a significantly stronger transfer ability. Notice that our model is pre-tr pre-trained with 200 epochs, surprisingly, compared with the original MoCo-v2 model pre-trained with 800 epochs, the fine-tuning results of 1-MoCo-v2 are still better on Birdsnap, Caltech101, CIFAR 100, and SUN397.
-i--o---ommin generalization. As illustrated in Section 3.1.4 standard SSCL methods, e.g., SimCLR, MoCo, etc., could suffer from OOD shift. To demonstrate the advantage of our modification, we investigate the effectiveness our method on OOD generalization benchmarks: PACS Li et al. (2017). VL.CS [ang et al. (2013), Office-Homa Venkateswara et al. (2077). We follow the standard way to conduct the experiment, i.e., choosing one domain as the test domain and using the remaining domains training domains, which named the
protocol. As can be seen in Table our (-MoCo-v2 indicates significant improvement over MoCo-v2. Both experiments indicate our modification exhibits substantial enhancement for domain transfer and OOD generalization ability. Similar to domain transfer scenario, compared with the original MoCo-v2 model pre- trained with 800 epochs, t-MoCo-v2 better on all of the three datasets. More experiment details, including detailed comparisons, are in Appendix
6 DISCUSSION
This work proposes novel perspective that interprets SSCL methods type of SNE methods, which facilitates both deeper theoretical understandings and methodological guidelines for practical improvement. More interpretations of SSCL from preserving the distance between distributions can be found in Appendix B Our analysis has limitations and the insights from SNE are not universally applicable for all SSCL methods, e.g.. Zbontar et al. 2021): Yang al. (2021) don't fit in our framework. However, this work is an interesting addition to existing theoretical works of SSCL and more investigations can be made along this path. While there are various extensions of the classic SNE, in this work, as proof of concept, we mainly showcased practical improvements from t-SNE. We expect more modifications can be developed by borrowing advances the SNE literature, e.g., changing to f f-divergences (Im et al. 2018) or consider optimal transport Bunne et al. (2019) Salmona et al. (202T); Mialon al. (020), On the other hand, standard SNE methods can also borrow existing techniques in SSCL to improve their performance more complicated data, e.g., incorporating data augmentations instead of or on top of pre-defined distances. In this sense, by choosing feature dimension to be 2, various
SSCL methods can also be used data visualization tools (Böhm al.) 2022 Damrich et al. (202)). Specifically on CIFAR- 10, standard t-EE can barely reveal any clusters while our f-Sim-LR with 2 produces much separation among different labels. More details be found in Appendix C.7

---

Published as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 1907, 02890, 2019.
Sanjeev Arora, Wei Hu, and Pravesh K Kothari. An analysis the t-sne algorithm for data visualization. In Conference On Learning Theory, pp. 1455-1462. PMLR, 2018.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
2019
Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning, pp. 468-477. PMLR, 2021.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15535-15545, 2019.
Haoyue Bai, Rui Sun, Langing Hong, Fengwei Zhou, Nanyang Ye, Han-Jia Ye, S-H Gary Chan, and Zhenguo Li. Decaug: Out-of-distribution generalization via decomposed feature representation and semantic augmentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 000666066667 6713, 2021.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive sell-uuuused learning recover global and local spectral embedding methods. arXiv preprint crXi1::2000: 2022.
Jan Niklas Böhm, Philipp Berens, and Dmitry Kobak. Unsupervised visualization of image datasets using contrastive learning. arXiv preprint arXiv:2210.09879 2022.
Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie legelka. Learning generative models across incomparable spaces. In International conference on machine learning, pp. 851-861. PMLR, 2019.
Toony Cai and Rong Ma. Theoretical foundations of t-sne for visualizing high-dimensional clustered data. arXiv preprint (tx.iv:200500555:0 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, 2020.
Kai Chen, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit- Yan Yeung. Multisiam: Self-supervised multi-instance siamese representation learning for autonomous driving. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 7546-7554, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Sebastian Damrich, Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak. From f-sne to umap with contrastive learning. In International Conference Learning Representations, 2022.
Qishi Dong, Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung-Ho Ho Bae, and Zhenguo Li. Zood: Exploiting model zoo for out-of-distributii generalization. arXiv preprint 2rxiv:22/000092:.0 2022.
Erber and GM Hockney. Equilibrium configurations equal charges on a sphere. Journal of Physics A: Mathematical and General, 24(23):L1369, 1991.
10

---

Published as a conference paper at ICLR 2023
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 59.544445423, 2021.
Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. 2013 IEEE International Conference on Computer Vision, pp. 1657-1664, 2013.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cent: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005. 12766, 2020.
Tiayyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv: arXiv:2104.0888 104.0882), 2021.
John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint wrXiv:2006.03654 2020.
cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena
Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Thomas Hamm and Ingo Steinwart. Adaptive learning rates for support vector machines working on data with low intrinsic dimension. The Annals of Statistics, 49(6):313--3180, 2021.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- seff-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34, 2021.
Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations. arXiv preprint arXiv:2204. 02683, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020a.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.0633 2021.
Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, pp. 107383, 2020b.
Geoffrey Hinton and Sam T Roweis. Stochastic neighbor embedding. In NIPS, volume 15, pp. 833-840. Citeseer, 2002.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18: 18:1551155 2006.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchile Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1808 06670, 2018.
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:21 1.00743, 2021.
Daniel Jiwoong Im, Nakul Verma, and Kristin Branson. Stochastic neighbor embedding under f-divergences. arXiv preprint 18I 2018

---

Published as a conference paper at ICLR 2023
Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. arXiv preprint arXiv:2110.02473, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09344) 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University Toronto, 2009.
David Krueger, Ethan Caballero, Joern-Henris Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815-5826 PMLR, 2021.
Di Yongxin Yang, Yi-Zhe Song. and Timothy M. Hospedales. Deeper, broader and artier domain gen- eralization. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5543-5551, 2017.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng. Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In 2021 AAAI Conference on Artificial Intelligence (AAAI), 2021.
Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich T. Sommer. Neural manifold clustering and embedding. ArXiv, abs/2201.10000, 2022..
George C Linderman and Stefan Steinerberger. Clustering with t-sne, provably. SIAM Journal on Mathematics of Data Science, (2):313-332,
Zhili Liu, Jianhua Han, Kai Chen, Lanqing Hong, Hang Xu, Chunjing Xu, and Zhenguo Li. Task-customized self- self-supe pre-training with scalable dynamic routing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1854-1862, 2022.
Yao Lu, Jukka Corander, and Zhirong Yang. Doubly stochastic neighbor embedding on spheres.
Pattern Recognition Letters, 128: 100-1060010000000000 2019.
Jajjun Ma, Tianyang Hu, and Wenjia Wang. Deciphering the projection head: Representation
evaluation self-supervised learning. arXiv preprint arXiv:2301.12189 2023.
JBM Melisseny. How different can colours be? maximum separation of points on spherical octant. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 454(1973):1499-1555 1998.
Facundo Mémoli. Gromov-wasserster distances and the metric approach to object matching. Foundations of computational mathematicss 11(4):417-477, 2011.
Grégoire Mialon, Dexiong Chen, Alexandre d' Aspremont, and Julien Mairal. A trainable optimal trans- port embedding for feature aggregation. In International Conference on Learning Representations (ICLR), 2020.
Edward Moroshko, Blake E Woodworth, Suriya Gunasekar, Jason D Lee, Nati Srebro, and Daniel Soudry. Implicit bias in deep linear classification: Initialization scale vs training accuracy. Advances in neural information processing systems, 33:22182-2221 12-122291,12222221 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.0/807.03748, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. PMLR, 2021.
Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems, 33:21174--21188,,2222 2020.
Antoine Salmona, Julie Delon, and Agnès Desolneux. Gromov-wassersteinn distances between gaussian distributions. arXiv preprint 10x.9:200 2021.
12

---

Published as a conference paper at ICLR 2023
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. arXiv preprint arXiv:2202. 14037, 2022.
Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places exit on the surface of pollen-grains. Recueil des travaux botaniques néerlandais, 27(1): 1930.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint drXiv::906.05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005. 10243, 2020.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. arXiv preprint arXiv:2008. 10150, 2020.
Laurens Van Der Maaten. Learning parametric embedding by preserving local structure. In Artificial intelligence and statistics, pp. 384-391, PMLR, 2009.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (111), 2008
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3385-5394,2011
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards -onaanndoiommnn-nnnittiiii trastive learning. In International Conference on Machine Learning, pp. 10530-10541. PMLR, 2021.
Haonan Wang. Jieyu Zhang, Qi Zhu, and Wei Huang. Augmentation graph contrastive learning. arXiv preprint 220.04440444474
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pp. 9929-9939. PMLR, 2020.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622, 2020.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. arXiv preprint arXiv:2105.155
2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non con-parmettric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-37337 2018.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXhv:2012.15466 2020.
Yuanmeng Yan, Rumei Li, Sirui Wang. Fuzheng Zhang, Wei Wu, and Weiran Xu. Consert: A contrastive framework for self-supervised sentence representation transfer. arXiv preprint arXiv:2105. 11741, 2021.
Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3987-3996, 2021.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li. Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre- pre-training. arXiv 2021.
preprint arXiv:21
Chun- Slun-Hsiao Yeh, Cheng Yao Hong. Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. arXiv preprint arXiv:21 10.06848, 2021.
13

---

Published as a conference paper at ICLR 2023
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032:0,, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv//110094112. 2017.
Xuyang Zhao, Tianqi Du, Yisen Wang, Jun Yao, and Weiran Huang. Arel: Enhancing contrastive
learning with augmentation-rohut representations. In International Conference Learning Representations (ICLR), 2023.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. arXiv preprint arXiv:2102. 088008800.0..00. 2021.
14

---

Published as a conference paper at ICLR 2023
D: aarplanc
2: hird
4. dect
6r fing
7: herse
slip
9: tnick
Figure A.3:
Cosine similarity heat map of learned features from SimCLR on CIFAR-10 dataset. The darker the color, the larger the similarity.
A TECHNICAL DETAILS
A.1 IMPLICIT BIAS OF SIMCLR CIFAR-10.
Figure A.3 plots the cosine similarity heat map of learned features from SimCLR CIFAR-10 dataset. To calculate the similarity
class A (figures denoted by a;) to class (figures denoted by by). we first calculate the mean of b; as b. Then, sum up simm((; and plot with colors. Hence, the similarity matrix shown in Figure A.3 is not symmetric.
A.2 PROOF OF PROPOSITION 3.2
Recall the domain aggnsic data augmentation process. For any æ,, the probability density of having Ee Rdd its augmented point can be characterized as
jac = (a; and ur! = form a positive pair (x;) = p <t)=
For isotropic Gaussian densities with mean and covariance matrix I²I, ø(t x,) o exp(- tapp-1l-a,111 (0,2211//202) which is monotonic with the la distance between and æ,,
A.3 INVESTIGATIONS ON C())
Figures A.4 A.5
A.5 illustrate the evolution of different complexity measurements during the training process under the Gaussian mixture setting and the CIFAR- respectively.
In the Gaussian mixture setting, the feature extractor is a fully connected ReLU network. Besides C(J). also evaluate the popular sum of squared weights. The observations on SimCLR are listed below:
The expected Lipschitz constant C(J) is small in initialization. It first increases (till around 100 iterations) and then consistently decreases. This empirically supports the implicit bias towards minimizing C(f).
C(J) and the sum of squared weights share very similar patterns.
The SNE loss is non-increasing, if we are doing stochastic neighbor embedding using l-disstncc..
In the CIFAR-10 case, the feature extractor is ResNet- 18 plus fully-connected projection layer. The output from ResNet-18 is usually called representation (512 dimensional) and is utilized for downstream tasks while the projection (128 dimension) is used for training. Such representation- projection set up is common in SSCL., Ma et al. (2023) aimed to decipher the projection head and uniformly distributed while the representation
revealed that the projection feature tends to be feature exhibits stronger alignment. Besides C(f). ). we also evaluate the l nonor the representation. The observations for SimCLP R and (-SimCLR CIFAR-10 10 are summarized as below:

---

Published as a conference paper at ICLR 2023
Lossft0
wrights
SNE Loss
Iterations
Figure A.4: Empirical evaluation on the complexity of the learned feature mapping during training under the Gaussian mixture setting. Two complexity measurements are considered, i.e., C(J) as in (3.3) and the SNE loss as (2.2). The SNE loss here only serves as in indicator for how well the pairwise distances preserved. The training objective is the standard InfoNCE loss. The SNE loss decreases quickly until the first 100 iterations and then stays flat.
C(f) for the projection layer shares similar patterns as in the Gaussian mixture case, first increase and then decreases. However, C(f) for the representation layer monotonically decreases.
C(ff) for the projection layer and the 12-norm in the representation layer share almost identical patterns.
Comparing SimCLR, both the the calculated C()) and 12-nomm are much smaller for t-SimCLR.
In conclusion, on one hand, our empirical results demonstrate that the complexity of the feature extractor C(J) does decrease during training and seem be implicitly minimized. On the other hand, its trend is shared with other more popularly used complexity measurements.
A.4 PROOF OF COROLLARY 3.6
this section, we illustrate with rigor how the hypothesized implicit bias can give rise to structure- preserving property of SSCL. Corollary stttsatess that minimizing the (Lipschitz) complexity of the feature mapping will also result in the best match between P and Q (under permutation). To provide more theoretical insight, we present the following lemma in the simpler vector-matching case.
states
Lemma A.1. Let < < Em and () < f11 <... < Yo be two real-valued sequences, normalized such that E xa = Σ - v 1. Consider permutation of {1,-.,,, .m. and denote all such
permutations as T. Then
argmin FET
Va(() 2y.
argmin RET
[r Um(i)) ) ==
where x =1)) all i= 11....
Proof. By the rearrangement inequality, we have
(a((
F2
Similarly,
Mn())
22-2
1=1
1=1
1
1=1
2

---

Published as a conference paper at ICLR 2023
Projection O1f!
7000
Regresentation
Reprosentation [[n]
5000
4000
2000
1000
160
(a) SimCLR on CIFAR-10.
0.35
Representation o(f)
3.30
Representation [Pl
2.
1200
1000
800
600
0.90
0.00
75
120
(b) f-SimCLR on CIFAR-10.
Figure A.5: Empirical evaluation on the complexity of the learned feature mapping during training on CIFAR-10. Two complexity measurements are considered, i.e., C(f) as (3.3) and 1o-morm. Specifically, we calculate the expected Lipschitz constant on both the representation layer (512- dimensional) and the projection layer (128 dimensional). Figure (a) and (b) show the trends (along the 200 training epochs) for SimCLR and (-SimCLP .R respectively.
Lemma A.I gives a vector-version illustration of our Corollary 3.6 stating that minimizing the expected derivative (to zero) of the mapping function f, i.e., Ex((x)) leads to preserving the norm difference the input vector and output vector.
Next, we provide the proof of Theorem 3.5
Proof of Theorem 3.5
Straightforwardly, we can write
2
|P-Q"1|===
Σ
1
Pij
$51
(m( 2
2+2)
lx(()())
Pij
Pij
it
20 (P.QT)
Σ
Ub 2
Pij
i$1
871
Thus, minimizing Ci (P.QQ)) also minimizes
P-
3

---

Published as a conference paper at ICLR 2023
Theorem 3.5 is rennriiiton of Lemmal A.1 Next, provide proof for Corollary 3.6 restated below.
Proof of Corollary 3.6 Recall the SimCLR loss LInfONCE (( where
exp(sim (x:s)mm((xz))
( -llog E LEDD,, (m;) exp(sim (m:)) (m)))((m))
Without loss of generality, let T 1. Notice that [(x; æx) is monotonically decreasing as (a:)) increases, due to the monotonicity function _
I+c with respect to æ: > for any
<> 0. Hence, in order for LInfONCE to be minimized, perfect alignment is required, i.., f(æ,) [ f(x)) for any i I......
With perfect alignment achieved, LInONCE only concerns the pairwise similarity between negative samples fx;;)'s "s, which can be simplified as LinfONCE 2 Lumifurm where
Luniffrrm
log
c+ E expss
t=1
2 log
1+ exp(sim(f(()),.(x;.)))
i=l
1
2 log
1+
Σ
exp(sim(((x:),f(x))))
71-6
Igigj
Luuiform be minimized by mapping x; 's distant as possible, hence the connection to Tammas problem and the uniformity principle.
With sufficient capacity of the feature mapping J, the SimCLR loss can be minimized to its (empirical) global minima. However, such f is not unique since LInONNE is invariant to permutations of mapping relationships from æ; f(x;)) If f% further minimizes C(f) on the sample level, i.e.,
[f((x;) f(x;)]l
f = argmin -
10:-20
ISiSSS
Then, fü also solves type SNE problem with uniformity constraint 3.4) as stated in Theorem 3.5 To see this, if we define 9; - II (x:)) - (xx; ll2 and P.j - æ æz 2, which is reasonable since the larger the distance, the smaller the similarity, we can directly apply the results in Theorem 3.5
we showcase the relationship between minimizing C(f) and structure preserving property by considering special SNE problem, where the pairwise similarity is not modeled by Gaussian as standard, hence the word "resembling" in Corollary 3.6) Although qı = (xx,) (adt;)-((aaa f(xx) is unorthodox, it is reasonable since the larger the distance, the smaller the similarity. If we consider the SNE method as in Hinton et al. (2006), proof does go through directly and demands more complicated analysis. However, our results are still valid in connecting the complexity of the feature map to the pairwise similarity matching.
Remark A.2. As can be seen from Theorem 3.5 and the proof of Corollary 3.6
Our statement in Corollary requires perfect alignment or perfect uniformity. When the assumptions are not perfectly met, we can still obtain insights for the resulting feature mapping. Alignment and uniformity (Wang & Isola| 2020) is not the whole story of contrastive learning, and our identified structure- structure-preservinee preserving property implicitly induced by complexity minimization provides an other angle of the learning process. From this perspective, contrastive learning be thought of as combination of alignment and SNE with uniformity constraint. In Figure A.3] while obtaining approximate alignment and uniformity, the feature mapping also preserves the relative relationships of the clusters (labels).
4

---

Published as a conference paper at ICLR 2023
A.5 ALIGNMENT AND UNIFORMITY OF T- S-MCLR
Due to the change of training objective, we may want to reevaluate the properties of the learned feature from (-SimCLR. We will show that alignment still hold while uniformity changed (to infinity).
Let us consider compact region Ω C Rd and æ; E Q. Let / be the transformation such that the augmented data point æ = (xz,) still in 22. Wang & Isola (2020) showed that the contrastive loss can be decomposed into the alignment loss and the uniformity loss. Zimmermann et al. (2021) further showed that the contrastive loss converges the cross-entropy between latent distributions, where the underlying latent space is assumed to be uniform, and the positive pairs are specified to be an exponential distribution. In this section, we show parallel result, which states that in the population level, the 1-SNE loss is the cross-entropy between two distributions of generating positive pairs.
Theorem A.3. Let (---)) be the cross entropy between distributions. Let p(æ) be the density of æ. p(-|x: be the conditional density of generating a positive pair, and define
gf (=11 (+]f(x)-f(arz) with Cy(x)) /
p(x')
p(x')
--) f(x') dz'.
Then, have
Bm-p(=)
(A.1)
which corresponds to the anpuatioon-levelISSiii.L. loss where
La - 1e ((2))))
L log (1 (1)
Proof. Note that
(p (+1x))
p(z')
p(ee|x))oog (2)]og
dæ' +logCy(x)
1+ [+)]
p(x')
pla' (1]l)g((+1[(()=
p(ar ((1o\(z))log(((a')) dda'' +lo
Jn 1 (1]l((x)- -(x) da'
p(agl (o))lo((++1////// (()))) ) p(al|x)))og(((() po
Jn
Taking expectation with respect to x leads to
Expp(()) H (pppp(((.gg((aaala)
Exep(( Ex log(
Bea -()
[ p(z)p((' (z)) p L() Op:
where
Cp = I (a))))laa( / p(a,,a )og(p(a))) da
does not depend on f.
E Bxppp(a) H (p(-1x)).)((
p(x')
p(x)
p p log
dæ' dx
p(x)
1 1+1lf(x) - f
12
p( (p()pp(a))
JL 1+ (x)) - f(x') dæda
p(x')
p(2,2' llg
dx dx
1+1lf(x) [ f(x')
p(z)p(xz)
99 1+ f() ff='((((11 dæda'.
5

---

Published as a conference paper at ICLR 2023
In Theorem A.3 La is the alignment loss and L, is the uniformity loss. The decomposition is much more natural for t-SimCLR as opposed to that in LInfONCE, mainly due to the change from conditional to joint distribution when modeling the pairwise similarity. Furthermore, if the t-SimCLR loss is minimized, we must have p(-1ær) (0-l))=0-) =qs( (rr((laa (-1=), provided f has sufficient capacity. Note that if ( (a:) = -g (12:), then Pzj. and Q3je are perfectly matched, which indicates that we obtain perfect neighbor embedding.
Theorem A.3 implies that the optimal feature mapping f" satisfies
p(-1æ)= -5- (-12)),
which further implies that for any E Q
p(x') 1+ 1+ x (x') xC(x) 11:00ppppppp "p(a' (()=
Cx-
(+ C(x)) p(x,x')
Cg)
(A.2)
where = f lx)da' Unlike the usual normalized SimCLR, I-SNE does not assume any special structure on f (e.g., |1/l/2 1), thus can go to infinity. Comparing to the finite sample t-SimCLR loss, the population version trickier to analyze. This is because for a given point æ' can be an augmented sample of some x (with probability p(æ'|=)), negative sample of (when we treat x' as another sample point). This reflects the essential difficulty between population and finite samples in contrastive learning, not only for t-SimCLR.
For clustered data, A.2 provides two important messages, provided that the augmentation is not too extreme and the augmented sample æ'' stays in the same cluster the original xc. On one hand, when 21 and 22 belongs to different clusters, the joint density p(æ: = 1,, = =22) will be very small, close zero, which indicates that (x1) for æ and 22 belonging to the same cluster, p(æ 21, = x22 will be relatively large. Hence, the will observe similar clustered structure in the feature space. This is confirmed in the Gaussian mixture setting in Figure T[c), in which case, the problem can oversimplified as mapping 5 points in 2 the unit-circle.
f - (222)l2 is very large, tending infinity. On the other hand,
features of the same cluster will stay close. Overall,
B CONNECTION TO DISTANCE BETWEEN DISTRIBUTIONS
Through the lens of stochastic neighbor embedding, the feature learning process of SSCL methods can be seen as minimizing certain "distances" between distributions in different dimensions. Idcally, the feature should preserve the distributional information about the data. Since the data and the feature do not lie in the same metric space, quantitatively measuring their distributional distance is difficult. Fortunately, there are existing tools 2011 Salmona et al. 2021).
utilize, specifically, i distance (Mémoli
Let X, Z be two Polish spaces, each endowed respectively with probability measures Ps and Pa- Given two measurable cost functions C. X' x X' - R, Ca x Z + R, and D R + I, the
distance be defined
1/p
inf GW (PP.PPPalcc, (TETIPP.PP))
D(c (x,x ),,c, (2,(22)))) P (d,))))
where (Pz.P=) denotes all the joint distributions in x such that the marginals are Pa and p.- Typically, D2c,,cz) is chosen be |c, ccl and ea (x,x") usually chosen be ||x- The key idea of the Cromov-Wassensee comparing marginal distribution to pairwise distributions, which is very similar to the SNE objective. Consider Monge's 's formulation of the optimal transportation problem and let == fz). By choosing cz(21,2,) = log( 2,1+) with Q specified in 33.1. Cu (2,,2)) P3i with P specified in 3.2) and letting D(ca, Ca. -log(c)))lo((e)))) we have
distance to circumvent the dimension mismatch is to change from
GW (PW E.. (D(c.(z,z').., (f (z)).
where the right hand side recovers the expected InfoNCE loss. Hence, the SNE perspective can also be viewed minimizing the Gromov-Wasse Wasserstein distance between p. and pz-
is worth noting that such an interpretation only relates to contrastive learning, not including generative- based self- superviedd learning methods such Masked AutoEncoder (MAE) (He 2021).

---

Published as a conference paper at ICLR 2023
0.9
0.6
0.5
0.4
=-MMLL
200
400
600
800
2000
Figure C.6: Nearest neighbor test accuracy vs. training epochs. SimCLR and (-SimCLR share similar trends and convergence speed.
202
100
75
0.0
0.2
0.1
0.6
0.8
1.0
Figure C.7: The histogram of IoUs for 1000 constructed positive pairs in CIFAR-10. The empirical distribution is almost symmetric around 0.5.
C EXPERIMENT DETAILS
C.I CIFAR-10 SETTINGS
CIFAR-10 10 (Krizhevsky 2009) a colorful image dataset with 50000 training samples and 10000 test samples from 10 categories. We use ResNet-18 18 (He et al. 2016) as the feature extractor, and other settings such as projection head all follow the original settings of SimCLR (Chen et al. 2020a To evaluate the quality of the features, we follow the KNN evaluation protocol (Wu et al. 2018). which computes the cosine similarities in the embedding space between the test image and its nearest neighbors, and make the prediction via weighted voting. We train each model with batch size 256 and 200 epochs for quicker evaluation. For SimCLR, without specifying otherwise, we grid search the tay and with range (1, 2, 10) and 2, 10) respectively.
Ablation of training epochs We also run the SimCLR and (-SimCLR experiments in the more standard 1000 epochs setting. For SimCLR, we use batch size of 512, learning rate 0.3, temperature of 0.7, and weight dacay of 0.0001. For t-SimCLR, we use batch size of 512, learning rate of 0,8, temperature of 10, weight dacay of 0. 0.0002, and ba 5. 5. The nearest neighbor accuracy for SimCLR is 87.2% vs. that for t-SimCLR is 88.8%.
C.2 IMAGE AUGMENTATION
When processing images, several popular augmentations are usually adopted (following the setting in SimCLR Chen et al. (2020a) e.g., random resized crop (crops random portion of image and resize it
7

---

Published as a conference paper at ICLR 2023
0.8
LSimCLR
SimC1R for CIFAR11
0.7
L-SimCLR for CIFAR-100
for CIFAR-100
CIFAR 10
0.20
100
12.8
Feature Dimension
Figure C.8: Extension on Figure bb). Nearest neighbor classification accuracy for SimCLR vs. I-SimCLR on both CIFAR-10 (in-distribution) and CIFAR-100 (out-of-distribution using different feature dimensions.
the original size). horizontal flip, color jitter (randomly change the brightness, contrast, saturation and hue of an image). To illustrate the natural weighting scheme in Section 4.1 we considered random re- sized crop and specifies the weights by the IoU (intersection over union) the positive pair. In particular, two augmented images are created from an anchor image. Each augmentation crops rectangular region of the image, denoted by "1,22 respectively, and their loU is defined by the area of intersection rı (12 divided by the area of the union r1 Ur2. The IoU is always between 0 and 1. In our experiment, we chose the default settings and Figure C.7 illustrates the IoU histogram of 1000 constructed positive pairs.
C.3 DEGREE OF FREEDOM IN t-SIMLLR
Feature dimension efficiency in OOD case. To further investigate the generalization ability of SSCL methods, we devise challenging setting where the model trained on CIFAR-10 and tested on CIFAR- 100 classification. In this case, we evaluate the effect of increasing feature dimensions in the projection layer, as an extension on the CIFAR 10 in-distribution case. The results are shown in Figure C.8 where there are two things to note:
The gain of extra dimensions in the OOD case does vanish later than that in the in-distribution case.
The advantage of SimCLR vs. t-SimCLR very significant with around 10% improvement when d = 128 using nearest neighbor classification, indicating that t-SimCLR produces better separated clusters.
Relationship between tar and dg. The larger the degree of freedom ta.. the less heavy-tail the t-distribution, As de decreases, the crowding problem becomes more severe and as recommended by (Van der Maaten & Hinton, 2008). smaller faf tends work better. We evaluate the sensitivity of tar (1, 5, 10) under different choices of d, (1, 2, 4, 8, 16, 32, 64, 128) in CIFAR- 10 and the results are reported in Figure C.9 As can be seen, when d, is small (,2), ba = outperforms. Comparing ta =5 and Laf 10, the two perform similarly when dg is large (16,32,66 128) but the smaller bar =5 yields better accuracy when d: = 1,2,4.
Tuning temperature vs. tuning ty. As illustrated in Section 4.2 is low, the heavy-tailed t-distribution is a better choice than Gaussian to alleviate the crowding problem.
when the feature space dimension
'hee evaluating by training linear classifiers for 100 epochs, the accuracy for SimCLR is 46.4% and that for t-SimCLR is 48.1 44% (averaged over replications).
8

---

Published as a conference paper at ICLR 2023
0.9
C.7
Accuracy
0.5
df=1
df=5
0.3
df=10
0.1
16
32
64
128
Feature Dimension
Figure C.9: Nearest neighbor classification accuracy CIFAR-10 for (-SimCLR using different feature dimensions and different degrees of freedom (L_df).
Even though tuning the temperature of LInfNNCE, i.., making T larger. can also have the effect of making the distribution less concentrated (T can be and tuning faf are fundamentally different. The former is controlling how fast does the similarity Qi decays as the distance between 2; and zj increases, while the latter serves as a scaling factor, offering constant level modification of the scheme. In our experiments with SimCLR vs t-SimCLR on CIFAR-I0, temperature is tuned as hyperparameter. The difference in
the standard deviation), tuning temperature
never make up the difference between the baseline SimCLR and t-SimCLR. We found r = 0.5 to work better for the base SimCLR while larger works better with our t-SimCLR, We recommend
as the default choice.
C.4 IMAGENET PRE- TRAINING
To show the ability for large scale domain transfer and OOD generalization, we conduct experiments on ImageNet pre- pre-training based on MoCo-v2 with its official implementationf" We follow most of their settings, e.g, data augmentation, 200 epochs pre- pre-training, and optimization strategy, etc. The loss modified according to Section (4.2 and batch normalization is applied along every dimension. We grid search the lyf and with range 5, 10, 15) and (0.2, 2, 10) respectively. Finally we choose tay = 10 and 7 7= to be the optimal hyperparameters. We use this pre-train model as initialization for domain transfer and OOD experiments.
C.5 DOMAIN TRANSFER
We compare MoCo-v2 pre-trained with 800 / 200 epochs and I-MoCo-v2 on Aircraft, Birdsnap, Caltech101, Cars, CIFARIO. 10, CIFARI00, DTD, Pets, and SUN397 in Table We follow the transfer settings in Ericsson al. 2021 finetune the pre-trained models. For datasets Birdsnap, Cars, CIFARIO, CIFARIO0, DTD, and SUN397, we report the top-1 accuracy metric, while for Aircraft, 111, and Pets, report the mean per-class accuracy metric. We also follow (ricsson et al. (2021) to split each dataset into training, validation, and test sets. On each dataset, we perform hyperparameter search follows. (1) We choose the initial learning
grid of 4 log- arithmically spaced values between x 10-4 and x 10-2: (2) We choose the weight decay parameter according to grid of 4 logarithmically spaced values between 1 10-9 and 1 10-3 plus no weight decay; (3) The weight decay values are divided by the learning rate; (4) For each pair of learning rate and weight decay,
according
finetune the pre- pre-trained model for 5000 steps by SGD with Nesterov momentum 0.9, batch size of 64, and cosine annealing learning rate schedule without restarts. As can be seen in Table C.3
our t-MoCo-v2 v2 with 200 epochs even outperform the baseline with 800 epochs on average.
C.6 OOD GENERALIZATION
To demonstrate the advantage of our modification, we also compare MoCo-v2 pre-trained with 800 200 epochs and f-MoCo-v2 on OOD generalization benchmarks: PACS al. (2017). VLCSS et al. (2013), Office-Home Venkateswara et al. (007). ). We follow the standard way to conduct the

---

Published as a conference paper at ICLR 2023
Table C.3: Domain transfer results of vanilla MoCo-v2 and I-MoCo-v2.
Method
Aircraft
Birdsnap
Caltech
Cars
CIFARIO
CIFAR100
SUN397
Avg.
McCo-v2 (800 epochs)
83.80
45.51
83.01
$6,18
96.42
71.69
71.70
89.11
55.61
75.89
McCo-v2 (200 opochs)
82.75
44.53
83.31
85.24
95.81
72.75
71.22
86.70
56.05
75.37
(-MoCo-v2 (200
53.46
86.81
96.04
78.32
59.30
77.78
Table C.4: OOD accuracies vanilla MoCo-v2 and f-MoCo-v2 on domain generalization benchmarks.
Method
PACS
VLCS
Office-Home
Avg.
MoCo-v2 (800 epochs)
58.9
69.8
41.6
56.8
MoCo-v2 (200 epochs)
58.5
70.4
36.6
55.2
t-MoCo-v2 (200 epochs)
61.3
75.1
42.1
59.5
experiments, i.e., choosing one domain the test domain and using the remaining domains training protocol. The top linear classifier is trained on the
domains, which is named the training domains and tested on the test domain. Each domain rotates the test domain and the average accuracy is reported for each dataset in Table C.4 On each dataset, we perform hyperparameter search following BomainBed Gulrajani & Lopez- (202T). We adopt the save-one-domain- cross-validation setup in Bed with 10 experiments for hyperparameter selection and run trials. As can be seen in TableC..4 our (-MoCo-v2 with 200 epochs even significantly outperform the baseline with 800 epochs for all of the three datasets.
C.7 SSCL INSPIRED DATA VISUALIZATION
t-SNE (Van der Maaten & Hinton 2008 and its variants are designed for data visualization. However, for more complicated data, such as colored images, the results are not satisfactory. Using standard I-SNE, the 2D visualization of the 50K training images of CIFAR-10 (labels denoted as 0, I.....9 can be seen in Figure C.10, where different labels are hardly separated. The poor performance of t-SNE on CIFAR-10 10 can be traced back the poor distance choice on images, i.e., fo-norm. Inspired by the success of SSCL for natural images, t-SNE can potentially be improved by incorporating data augmentations.
In light of our perspective (S1), t-SNE can take advantage of the distance specified with 3 and the resulting model essentially our t-SimCLR with feature dimension 2. The visualization from t-SimCLR is shown in Figure C.Il which is much more separated (the nearest neighbor classification feature dimension to 2, various SSCL methods can also be made into data visualizing tools. In Figure C.12 we visualize the outcome from SimCLR (the nearest neighbor classification accuracy on CIFAR test data is 24.8%).
accuracy on CIFAR-10 test data is 56.6 56.6%). By choosing
Similar investigations have been carried in Böhm et al. 2022) Damrich et al. (2022) where they focused specifically on data visualization and stochastic nei ghor embedding.
10

---

Published as a conference paper at ICLR 2023
20
-20
-40
-20
20
Figure C. 10: 50K CIFAR- 10 training images visualization in 2D with t-SNE.
0.2
0.0
-0.4
-0.6
0.0
Figure C.11: 50K CIFAR-10 training images visualization in 2D with the default t-SimCLR.
600
400
3
200
400
-600
-600
-400
200
Figure C212: 50K CIFAR-10 training images visualization in 2D with the SimCRR.
11


Reviews:
[{"review_id": "iDa55W_uFW", "reviewer": "Reviewer_emn1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies contrastive self-supervised learning from the view of SNE. The authors show that InfoNCE loss can be seen as a special case of SNE, by setting the data similarity matrix P according to the data augmentation and embedding matrix Q as the softmax of pairwise similarity. Under the SNE framework, new insights for domain-agnostic data augmentation, implicit bias and OOD generalization are presented. Moreover, from SNE perspective, this paper proposes to improve SimCLR by 1) introducing weighted matrix P to better utilize the augmentation detail and 2) t-SNE style InfoNCE loss, i.e, l2 instead of cosine to improve OOD generalization and exponential to polynomial to improve feature dimension efficiency. Various experiments show that improvement can be achieved in both in-distribution and out-of-distribution generalization.\n", "summary_of_the_review": "This paper is novel and original. The contribution is solid. The SNE perspective is helpful for both theoretical understanding of SSCL as well as guidance for practical improvement. I believe this work is helpful for both the SNE and SSCL communities and may facilitate further exchange between the two.", "strength_and_weaknesses": "Strength: \nThe novelty is strong. The connection between SimCLR and SNE is rarely discussed and the SNE perspective of SSCL is novel to me. The connection is presented in an inspiring way where the authors provided several new insights and demonstrated several SNE inspired modifications to SimCLR. While mainstream self-supervised learning focuses on methodology and benchmark performance, it’s nice to see that this work started by investigating the Gaussian mixture setting. Though oversimplified, the low-dimensional simulated setting does provide plenty of insights and can be explicitly analyzed. The acquired insights are further validated by the following real-data experiments.  \n\nThe implicit bias towards SNE with uniformity constraint (sec 3.1.3) is most interesting to me, which points to a very important, but perhaps overlooked question in understanding contrastive learning. Contrastive learning works extremely well in practice, which is surprising since the learning objective is so simple and has (infinitely many) trivial solutions. I agree with the authors that the discriminative features are mainly learned implicitly. The order preserving phenomenon does shed light on understanding the CL for real data. \nFollowing the authors’ argument, the role of positive pair alignment is merely to simplify the input space, by merging equivalent classes. Then the feature learning process is essentially SNE with uniformity constraint.  This is a very interesting point to me and may call for more investigations. \n\nThe dimensional efficiency (sec 4.2) and OOD generalization are also interesting and are very thought-provoking. In Figure2(b), it’s surprising to see that the dimensionality gains for CIFAR-10 classification are almost non-existent after only 16. It is interesting to see how the dimension affects the performance in the OOD setting. \n\nWeaknesses: \nThe title may have been overclaimed. The paper only concerns the InfoNCE loss (SimCLR, MoCo) and there are various other self-supervised learning methods, e.g., SimSiam, BYOL, etc. How does the connection to SNE carry over to them? Can we observe the same phenomenon? \n\nSome questions/comments:\n- I am curious to see how does feature dimensions affect the performance in the OOD setting. My guess is that the performance gain due to higher dimensions should be much more significant than IID case, e.g., Figure 2(b). \n\n- The order/neighbor preserving phenomenon is not very easy to see in the 2-d Gaussian case. I would suggest the authors make the clusters more extreme so that which should be closer to which is easier to see. \n\n- Typo: The first sentence of sec 3.1.3, results on SSCL **provide**\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FGwSiujzc2k", "reviewer": "Reviewer_RZus", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A novel perspective of self-supervised contrastive learning (SSCL) is investigated. To be specific, this paper interprets SSCL as stochastic neighbor embedding (SNE), and studies implicit bias and out-of-distribution generalization under this framework. The case study about OOD is interesting to me, and also the extension of SSCL based on SNE is impressive. ", "summary_of_the_review": "Overall, this paper is interesting and impressive. If the weaknesses paints can be well addressed, it would be better. ", "strength_and_weaknesses": "Strength:\n\n(1) A novel perspective to interpret SSCL as SNE.\n\n(2) The case study based on the Gaussian mixture model for implicit bias and OOD is impressive.\n\n(3) The two variants of SSCL based on SNE are reasonable, which is interesting for exploring OOD generation of SSCL.\n\nWeaknesses:\n\n(1) It is claimed that \"Can SNE be revived in the modern era by incorporating SSCL?\". It is not discussed in the main manuscripts. \n\n(2) Eq.3.1 is not a good choice. Actually, the conditional probability P_{j|i} is not normalized. It would be a better choice to add the coefficient term 1/2n  as the normalization term for summarization over j.    \n\n(3) How are the negative samples constructed in the case study in Sect.3, such as Gaussian mixture setting? \n\n(4) This whole paper claimed to connect SSCL to SNE, but conducted experiments on MoCo-v2, which is not typically an SSCL method as it does not require negative samples. The connection between SNE and MoCo should be discussed as well. \n\nSome minor typos:\n\n(1) \"two random sampled data as negative pairs\", there are more than two negative pairs. \n\n(2) \"is the is a temperature parameter.\"\n\n(3) In Eq.3.1, it should be $\\tilde{x}_i$ and $\\tilde{x}_j$\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8v-7uw1R9n", "reviewer": "Reviewer_qnUF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. This paper interpreted self-supervised contrastive learning (SSCL) as a type of stochastic neighbor embedding (SNE) methods that preserve the pairwise similarities specified by the data augmentations. \n2. Based on the connection with SNE, this paper provided theoretical insights for domain-agnostic data augmentation, implicit bias, and OOD generalization of SSCL. In particular, for the implicit bias of SSCL, this paper proved the equivalence between minimizing the expected Lipschitz constant of the SSCL feature map and SNE with uniformity constraint. \n3. Motivated by the SNE perspective, this paper proposed modifications to the SSCL methods, including re-weighting positive pairs and t-SNE style matching, and achieved improvements in experiments. ", "summary_of_the_review": "I think building the connection between SSCL and SNE is very important and many techniques in SNE can be applied to SSCL to improve its performance (as already demonstrated in this paper). \n\nMy major concern with this paper is the theoretical part. I found the theoretical analysis in this paper restricted to exceedingly simple toy examples and some arguments are handwavy and lack depth. The connection between the implicit bias of SSCL (minimizing $C(f)$) and SNE with uniformity constraint is pretty interesting. Unfortunately, this paper fails to either prove or empirically demonstrate that SimCLR is minimizing $C(f)$. ", "strength_and_weaknesses": "Strengths:\nI think viewing SSCL as a type of SNE is an exciting and potentially fruitful perspective. Many theoretical analyses and practical techniques can be potentially applied to SSCL to improve its performance. As already demonstrated in this paper, the re-weighting of positive pairs and t-SNE style matching inspired by SNE literature can improve the performance of SSCL.\n\nWeaknesses:\nThe equivalence between SSCL and SNE is not hard to see after a re-writing of the objective function; this connection is interesting but not very surprising. The bulk of this paper focuses on the theoretical analysis of SSCL from the SNE perspective. However, all the theoretical analysis is restricted to toy examples and some arguments are handwavy and lack depth.\n1. The analysis is restricted to a toy example: the gaussian mixture model. The positive samples are generated by sampling from the same component. This is impossible to conduct in SSCL since we cannot access the label information in SSCL and therefore cannot decide which component a sample belongs to. \n2. In the analysis domain-agnostic data augmentation, why is $P(x_1 \\text{ and } x_2 \\text{ form a positive pair })$ equal to the density $\\phi(x_1-x_2)$? For example, suppose the original dataset only contains one sample, all the augmented data are positive pairs, in which case $P(x_1 \\text{ and } x_2 \\text{ form a positive pair })$ always equals $1$ but the density may not be $1$. \n3. Regarding the implicit bias, this paper hypothesized that SSCL minimizes the expected Lipschitz constant $C(f)$. Then it proves that minimizing $C(f)$ in SSCL is equivalent to solving an SNE problem with uniformity constraint. But the SNE problem is not standard ($P$ needs to be transformed in a strange way and $Q$ is also unorthodox). This paper did not prove or give sufficient empirical evidence to show SimCLR is minimizing $C(f)$.\n4. The arguments for the effects of normalization on OOD generalization are very handwavy and the authors did not provide any rigorous theorem for this part. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4aBNpeQj_Mm", "reviewer": "Reviewer_DMAs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper shows a connection between self-supervised contrastive learning (SSCL) and stochastic neighborhood embedding (SNE), a data visualization method based on preserving distances. More formally, the authors show that SSCL is a form of SNE with pairwise distance/similarity defined by the data augmentation. Leveraging this connection and working with a simple mixture of gaussians setting, the authors provide theoretical insights into\n- performance of domain-agnostic augmentations such as mixup\n- uniformity and alignment of the learned features\n- *expected Lipschitz constant* as an implicit bias of SSCL\n- out of distribution robustness with normalized versus unnormalized features\n\nThese theoretical results are accompanied with new practical ideas that give significant improvements over standard SSCL. These ideas are:\n- using a weighted loss with the weighting dependent on the augmentation pair\n- $t$-SSCL in the spirit of $t$-SNE by replacing the gaussian distribution to a heavy tail $t$-distribution\n\nLastly, the paper show the benefits of these modifications in large scale experiments in terms of domain transfer and o.o.d. generalization.", "summary_of_the_review": "Despite the issues with the technical content in the paper, the practical advantages of the proposed viewpoints are pretty strong. Therefore, I lean towards acceptance. I would encourage the authors to clean up the theoretical sections to improve the paper.", "strength_and_weaknesses": "**Strengths**:\n- The connection to SNE, although intuitive, is new and provides useful insights, which the paper explores through different angles.\n- Performance gains from the suggested algorithmic improvements are pretty impressive!\n\n**Weaknesses**:\n- Theoretical results in the paper are not clearly presented, and seem more like intuitions than formal statements. For example, in section 3.1.1, the distance proposition is limited to only the simplistic gaussian noise injection setup, and the difference with mixup is not formalized. Same holds for section 3.1.2. Section 3.1.3 claims that “SSCL exhibit neighbor-preserving property and we identify it as an implicit bias”, but the corollary assumes that if $C(f)$ is minimized then show distances are preserved. It is not clear to me why $C(f)$ would be minimized by SSCL.\n- Comparison to prior work on relating SSCL to spectral clustering and inductive bias of function classes is not discussed (see HaoChen et al. 2021, HaoChen et al. 2022, Saunshi et al. 2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster