Understanding Deep Contrastive Learning via Coordinate-wise Optimization
Authors: Yuandong Tian
Venue: NeurIPS 2022 Accept (2022)
Keywords: contrastive learning
self-supervised learning
representation learning
principal component analysis
loss design
landscape analysis
deep linear network

Abstract: We show that Contrastive Learning (CL) under a broad family of loss functions (including InfoNCE) has a unified formulation of coordinate-wise optimization on the network parameter $\vtheta$ and pairwise importance $\alpha$, where the \emph{max player} $\vtheta$ learns representation for contrastiveness, and the \emph{min player} $\alpha$ puts more weights on pairs of distinct samples that share similar representations. The resulting formulation, called \boldmethod{}, unifies not only various existing contrastive losses, which differ by how sample-pair importance $\alpha$ is constructed, but also is able to extrapolate to give novel contrastive losses beyond popular ones, opening a new avenue of contrastive loss design. These novel losses yield comparable (or better) performance on CIFAR10, STL-10 and CIFAR-100 than classic InfoNCE. Furthermore, we also analyze the max player in detail: we prove that with fixed $\alpha$, max player is equivalent to Principal Component Analysis (PCA) for deep linear network, and almost all local minima are global and rank-1, recovering optimal PCA solutions. Finally, we extend our analysis on max player to 2-layer ReLU networks, showing that its fixed points can have higher ranks. Codes are available in https://github.com/facebookresearch/luckmatters/tree/main/ssl/real-dataset.

Understanding Deep Contrastive Learning via Coordinate-wise Optimization
Yuandong Tian Meta AI (FAIR) yuandong@meta.cm com
Abstract
We show that Contrastive Learning (CL) under broad family of loss functions (including InfoNCE) has unified formulation of coordinate-wise optimization on the network parameter 0 and pairwise importance where the max player 0 learns representation for contrastiveness, and the min player a puts more weights on pairs of distinct samples that share similar representations. The resulting formulation, called a-CL., unifies not only various existing contrastive losses, which differ by how sample-pair importance is constructed, but also is able to extrapolate give novel contrastive losses beyond popular ones, opening a new avenue of contrastive loss design. These novel losses yield comparable (or better) performance on CIFARIO IO, STL- and CIFAR-100 than classic InfoNCE. Furthermore, we also analyze the max player in detail: we prove that with fixed a, max player is equivalent to Principal Component Analysis (PCA) for deep linear network, and almost all local minima are global and rank- recovering optimal PCA solutions. Finally, we extend our analysis on max player to 2-layer ReLU networks, showing that its fixed points can have higher ranks. Codes are available !
Introduction
While contrastive self-supervised learning has been shown to learn good features (Chen et al. 2020 He et al.] 2020 Oord et al.) 2018) and in many cases, comparable with features learned from supervised learning, it remains open problem what features it learns, in particular when deep nonlinear networks are used. Theory on this is quite sparse, mostly focusing on loss function (Arora al. 2019) and treating the networks black-box function approximator.
In this paper, we present novel perspective of contrastive learning (CL) for broad family of contrastive loss functions L(0): minimizing L(O) corresponds to coordinute-wise optimization procedure on an objective & (0) (a) with respect to network parameter and pairwise importance on batch samples, where & (0) is an energy function and R(aa) is a regularizer, both associated with the original contrastive loss C. In this view, the max player 0 learns a representation to maximize the contrastiveness of different samples and keep different augmentation view of the same sample similar, while the min player a puts more weights on pairs of different samples that appear similar in the representation space, subject regularization. Empirically, this formulation, named Pair- weighed Contrastive Learning (a-CL), when coupled with various regularization terms, yields novel contrastive losses that show comparable (or better) performance in CIFAR10 (Krizhevsky et al. 2009)
and 10 (Coates et al. 2011).
We then focus on the behavior of the max player who does representation learning via maximizing the energy function & (0). When the underlying network is deep linear, we show that max E, (0) is the loss function (under re- pe-aaamtterrzation) of Principal Component Analysis (PCA) (Wold et al.) 1987), a century-old unsupervised dimension reduction method. To further show they are equivalent,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

x[[]
x[(]
x[i]
Contrastive Loss
((x))
(x))
Multi-layer network
InfoNCE (Oedd
2018
7 logg(
2/T
MINE Beighazi et al
2018
log(a)
c²
Triplct Schrolt ef
2015
x
+++
z[(]
z[e]
z
Soft Triplet Tian al
2020
7 7log(1
+ x) x)
e"/r=e
N+11 Tuplet Sohn 2010
log
e²
d
Lifted Structured Soup et
log(2)1
++c
Modified Triplct Eqn. 10 Caria et
sigmoid (cz)
Triplct Contrastive Tun. Fe 2021
linear
|inear
Figure I: Problem Setting. Left: Data points (i-th sample a=[i] and its augmented version .fi') sample =[)]) are sent to networks with weights 0, yield outputs =[i], z[1'] and =[z]. From the outputs
compute pairwise squared distance d' between =[i] and =[j] and intra-class squared distance de between x[1] and = =|{"] for contrastive learning with a general family of contrastive loss Ca.s (Eqn. Right: Different existing loss functions corresponds to different monotonous functions á and gb. Here [x]+ max(x,)).
we prove that the nonlinear training dynamics of CL with linear multi-layer feedforward network (MLP) enjoys nice properties: with proper weight normalization, almost all its local optima are global, achieving optimal PCA objective, and are rank- 1. The only difference here that the data augmentation provides negative digen-directions to avoid.
Furthermore, extend analysis to 2-layer ReLU network, explore the difference between the rank-1 PCA solution and the solution learned by a nonlinear network. Assuming the data follow orthogonal mixture model, the 2-layer ReLUU networks enjoy similar dynamics the linear one, except for a special sticky weight rule that keeps the low-layer weights to be non-negative and stays zero when touching zero. In the case of one hidden node, we prove that the solution in ReLU always picks single mode from the mixtures. In the case of multiple hidden nodes, the resulting solution is necessarily ra
Related Work
Contrastive learning. While many contrastive learning techniques (e.g., SimCLR (Chen al. 2020), MoCo (He et al. 2020), PIRL (Misra & Maaten 2020), SwAV (Caron et al. 2020), DeepClus- ter (Caron al. 2018). Barlow Twins (Zbontar et al.) 2021). InstDis (Wu et al., 2018). etc) have been proposed empirically and able to learn good representations for downstream tasks, theoretical study relatively sparse, mostly focusing on loss function itself (Tian et al. 2020bb] HaooCen et al. [021]] Arora et al. 2019), e.g., the relationship of loss functions with mutual information (MI). We are not aware of prior works on joint analysis that combines neural network and loss functions in CL.
Theoretical analysis of deep networks. Many works focus on analysis of deep linear networks in
supervised setting, where label given. (Baldi & Hornik 1989 Zhou & Liang 2018
Kawaguchi
2016) analyze the critical points of linear networks. (Saxe et al. [ Arora et al. 2018)
also analyze
the training dynamics. On the other hand, analyzing nonlinear networks has been a difficult task.
Existing works mostly lie in supervised learning, e.g., teacher-studen setting (Tian) 2020 Allen
al. 2018), landscape (Safran & Shamir 2018). For contrastive learning, recent work (Wen & analyzes the dynamics of 1-layer ReL.U networks with specific weight structure, and (Jing
2021
al.
2022) analyzes the collapsing behaviors 2-layer linear network. To our best knowledge, we
unaware of such analysis networks with layers (linear nonlinear) in the context of CL.
Connection between Principal Component Analysis (PCA) and Self-supervised Learning. (Lee T establishes the statistical connection between non-linear Canonical Component Analysis (CCA) and SimSiam (Chen He, 2020 for any zero-mean encoder, without considering the aspect of training dynamics. In contrast, we reformulate contrastive learning as coordinate- wise optimization procedure with min/max players, in which the max player is reparameterization of PCA optimized with gradient descent, and analyze its training dynamics in the presence of specific neural architectures.
et al.
3 Contrastive Learning as Coordinate-wise Optimization
Notation. Suppose we have N pairs of samples [a+ and ) Both (1 and a=(") are augmented samples from sample i and represents the input batch. These samples sent to

---

neural networks and z[i] and x|²"] are their outputs. The goal of contrastive learning (CL) is to
find the representation to maximize the squared distance di := || = 2 between distinct
samples and j. and minimize the squared distance df == ||]]|] =[r'113/2 / between different data augmentations z[i] and x=("] of the same sample i.
3.1 A general family of contrastive loss
We consider minimizing general family of loss functions Lo.v, where Φ and th are monotonously increasing and differentiable scalar functions (define & = Eÿzi (d dlt)) for notation brevity);
N
min L.. 0 = ( = Σ Φ i=1
Σ Jo
(dF
d.g))
(1)
0
Both and] j run from N. With different & and 15, Eqn. 1 covers many loss functions (Tbl. In particular, setting (xx) log(( x) and ((x) exp(x/T) (T) gives a generalized version of InfoNCE loss (Oord al. 2018):
exp(-df1) log exp( +)) exp(
Cyce
log
h-p(-d6//7 d2//T)
where 6 is some constant related =[1] and z 2|1']. e has been used many works (He et al. 2020; Tian et al. 2020a). Setting yields SimCLR setting (Chen et al. 2020) where the denominator doesn't contains exp(-df /T). This is also used in (Yeh et al. 2021).
3.2 The other side of gradient descent of contrastive loss
To minimize Le. gradient descent follows its negative gradient direction. As first discovery of this work, it turns out that the gradient descent of the loss function L is the gradient ascent direction of another energy function a.:
Theorem 1. For any differential mapping- z = 2(x; 0), gradient descent of Ca.v. is equivalent to gradient ascent of the objective (0) := [=((C](((0))) (0))):
QL..v
DEa
(3)
00
30 a=a(())
Here the pairwise importance a = a(0) = (ar;( is a function of imput batch æ. defined as:
0:g (0) ('.)) (
(4)
where ø', v are derivatives of o, tb. The contrastive covariance C. 1, is defined as:
Cala, bj = Σ Σ an a -b11) s#L
Σ
Σ jzi
(di)-a(""((b1))bb))) (5)
azj
That is, minimizing the loss function (0) can be regarded maximizing the energy function Ea=ag(a(9)) (0) with respect O. Here sg(.) means stop-gradient, i.e., the gradient of 0 is not backpropagated into (0).
Please check Supplementary Materials (SM) for all proofs. From the definition of energy E, (0), is clear that 0i determines the importance of each sample pair æ|i] and æ[j]. For (i.j)-pair that "deserves attention"., aij is large so that it plays large role in the contrastive covariance term. In particular, for InfoNCE loss with the pairwise importance takes the following form:
exp( d2y/T)
0i ΣSA exp(-dl,/T) >
(6)
which means that InfoNCE focuses on (jpiir with small squared distance dy. both & and v linear, then ai const and L is simple subtraction of positive/negative squared distances.

---

From Thm. 1 an important observation is that when propagating gradient w.r.t. θ using the objective (0) during the backward pass, the gradient does not propagate into ( even if (0) is function of 0 in the forward pass. In fact, in Sec. 6 we show that propagating gradient through a(O) yields worse empirical performance. This suggests that a should be treated as an independent variable when optimizing O. It turns out that if (a:) is an exponential function (as in most cases of Tbl. D. this is indeed true and a can be determined by a separate optimization procedure:
Theorem 2. If (x)) ealT, then the corresponding pairwise importance (Eqn. 4 is the solution to the minimization problem:
Σ Oij = 1E aij 2 o 171
ax (0) = arg min (0) - -R(a); A:= a: Vi,
20 (
aEA
Here the regularization R(a) Rn( = T En H(a)) -T -i Ejjt Oig loj- aij-
For InfoNCE, the feasible set A becomes fa : a 0, E, اززز Oij = E/((E + +c) This means that if i- th sample is already well-separated (small iutra-augmentatin
distance d, and large aummenttggttttionnn distance dıj), then & is small, the summation of weights Σjœ; aij associated with sample i is also small and such sample is overall discounted. Setting e 0 reduces sample-agnostic constraint (i.e., ΣØz; Okj = 1).
Thm. leads to a novel perspective of coordinate-wise optimization for Contrastive Learning (CL);
Corollary (Contrastive Learning as Coordinate- Optimization). If (b(x) e//T, minimizing Ls is equivalent to the following iterative procedure: (Min-player o:)
Og = arg min & (0:) - -R(a) aEA
(8a)
(Max-player 0)
0.+1 0: + V VE (0)
(8h)
Intuitively, the max player O (Eqn. 8b) performs one-step gradient ascent for the objective (0)
R(a), learns a representation to maximize the distance of different samples and minimize the distance of the same sample with different augmentations (as suggested by Ca(z, zl). On the other hand, the "min player" a (Eqn. (a)) finds optimal a analytically, assigning high weights on confusing pairs for "max player" to solve.
Relation to max-min formulation. While Corollary I looks very similar to max-min formulation, important differences exist. Different from traditional max-min formulation, in Corollary [ there is asymmetry between O and a. First, O only follows one step update along gradient ascent direction of max0 &. (0), while is solved analytically. Second, due to the stop-gradient operator, the gradient of 0 contains no knowledge on how 0 changes a. This prevents 0 from adapting to a' response on changing 0. Both give advantages to min-player a to find the confusing sample pairs more effectively.
Relation to hard-negative samples. While many previous works (Kalantidis et al. 2020: Robinson ct al. 2021) focus on secking and putting more weights on hard samples, Corollary shows that contrastive losses already have such mechanism at the batch level, focusing on -hard-negative pairs" beyond hard-negative samples.
From this formulation, different pairwise importance a corresponds to different loss functions within the loss family specified by Eqn. [ and choosing among this family (i.e., different á and 1 can be regarded as choosing different a when optimizing the same objective E., (0). Based on this observation, we now propose the following training framework called a-CL:
Definition 1 (Piih-weighed Contrastive Learning (a-CL)). Optimize 0 by gradient ascent: 0z+1 O, + V ( (0), with the energy E.(O), defined in Thm. 7 and pairwise importance a, = (0,).
In a- CL, choosing can be achieved by either implicitly specifying regularizer R(aa) and solve Eqn. Sal or by a direct mapping a = a()) without any optimization. This opens a novel revenue for CL loss design. Initial experiments (Sec. 6) show that a-CL gives comparable (or even better) downstream performance in CIFARIO and 100 compared to vanilla InfoNCE loss.
4 Representation Learning in Deep Linear CL is PCA
In Corollary D. optimizing over a is well-understood, since &, (0) is linear w.r.t. a and R(ax) in general is a (strong) concave function. As a result, a has unique optimal. On the other hand,

---

Contrastive Covariance
Eigeropectrum
Objective time
Singular dynamics
Singular vector alighment
100
1.0
Objedive Fundtion
Snouue Veluee
6-
04-
8-
0.0
2.5
7.5
2.5 5.0 7.5
1000
2000
3000
1000
2000
3000
0.2
1000
2000
3000
Eijen Index
Rerttin
Iteradion
Iteration
Figure 2: Dynamics of CL. with multilayer (T. 5) linear network (DeepLin) with fixed a. Running the training dynamics Saammaaauuiiiiii leads convergence towards the maximal eigenvalue of X For dynamies singular value Wi, the largest singular values (solid lines) converges to while the second largest singular values (dashed lines) decay to 0.
understanding the max player maxe & (0) is important since it performs representation learning in CL. is also a hard problem because non-convex optimization.
We start with specific case when z is deep linear network, i.e., ( where W is the equivalent linear mapping for the deep linear network, and the parameters to optimized. Note that this covers many different kinds of deep linear networks, including VGG-like (Saxe et al. 2014), ResNet-like (Hardt Ma 2017) and DenseNet-like (Huang et al. 2017). For notation brevity, we
define C., [ar] = C. T
Corollary 2 (Representation learning in Deep Linear CL reparameterizes Principal Component Analysis (PCA)). When z (ff)ar with constraint WWT =1, &a is the objective of Principal Component Analysis (PCA) with reparameterization W ((0):
max & (0) 2 (r(W(0)X.WT(OO (0)) S.1. WWT =1
here Xa = Co [2]] the contrastive covariance of input x.
As comparison, in traditional Principal Component Analysis, the objective is (Kokiopoulou et al.) 2011): maxw tr(W' Vaamasamplle [x) ) subject to the constraint WW L, where V Vsample [c] is the empirical covariance of the dataset (here it is one batch). Therefore, X., can be regarded as generalized covariance matrix, possibly containing negative eigenvalues. In the case of supervised CL (i.e,. pairs from the same/different labels treated positiveenegative (Khosla al. 2020 then it is connected with Fisher's ' Linear Discriminant Analysis (Fisher 1936).
Here we show mathematically rigorous connection between CL. and dimensional reduction, as suggested intuitively in (Hadsell et al., 2006). Unlike traditional PCA, due to the presence of data augmentation, while symmetric, the contrastive covariance X, is not necessarily PSD) matrix. Nevertheless, the intuition is the same: to find the direction that corresponds to maximal variation of the data.
While it is interesting discover that CL with deep linear network is essentially reparameterization of PCA, it remains elusive that such reparameterization leads to the same solution of PCA, in particular when the network is deep (and may contain local optima). Also, PCA has an overall end-to-end constraint WWT = I, while in network training, we instead use normalization layers and it unclear whether they are equivalent or not.
In this section, we show for a specific deep linear model, almost all its local maxima of Eqn. are global and it indeed solves PCA.
4.1 A concrete deep linear model
We study a concrete deep linear network with parameters/weights = (Wi)
W,W,11 (i
(10)
Here Wi E R 1, ny is the number nodes at layer 1, =[i] output of æ+i] and similarly z|i"] for (i'). We use 0 to represent the collection of weights at all layers. For convenience, we
define the 1-th layer activation [(] = Wifi-18 With this notation fo[i] = a=ii] is the input and
=|i] W W.f
5

---

We call this setting DeepLin. The Jacobian matrix W == WLWL-1 W41 and W = Wso
Wr. W I.......Wl Lemma 1. The training dynamics in DeepLin is W, =
X Similar to supervised learning (Arora al. 2018 2018b), nearby layers are also balanced: d ae (WiW," WW4.1/W. = O.
Note that Ca[fo] Ca [>]
Du al.
4.2 Normalization Constraints
Note that if we just run the training dynamics (Lemma I without any constraints, ||Wile will go to infinity. Fortunately, empirical works already suggest various ways of normalization to stabilize the network training.
One popular technique CL lg normalization. and before the loss function L (Chen et al.
often put right after the output of the network
2020: Grill et al. 2020
He et al. 2020), i.e., 2[1] mcan (f (f
=| Besides, LayerNorm (Ba et al. 2016) (i.e., f( [f
is
extensively used in Transformer based models s (Xiong et al. 2020). Here we show that for gradient
flow dynamics of MLP models, such normalization layers conserve ||Wil| for any below it, regardless of loss function.
Lemma 2. For MLP, the weight W, is below ly-norm or LayerNorm layer, then dt WWll = O.
Note that Lemma 2 also holds for nonlinear MLP with reversible activations, which includes ReLU (see SM). Therefore. without loss of generality, we consider the following complete objective for max player with DeepLin (here the constraint set of the weights due to normalization):
max & (0) == tr tr(WXxWT)) eEB
+ = (0:
[ 1, SIST)
(11)
2
4.3 Representation Learning with DeepLin is PCA
As of our main contributions, the following theorem asserts that almost all local optimal so- lutions of Eqn. [T are global, and the optimal objective corresponds to the PCA objective. Note that (Kawaguchi) 2016 Laurent & Brecht, 2018) proves bad local optima for deep linear network in supervised learning, while here we give similar results for CL, and additionally we also give the (simple) rank-1 structure of all local optima.
Theorem (Representation Learning with DeepLin is PCA). If Xmxx (Xa) then for any local maximum f E θ of Eqn. // whose W W 1 has distinct maximal eigenvalue:
there exists set of unit vectors {wi so that Wi =V==, U,w,_, Do is the unit eigenvector corresponding to Amas (Xa),
S L, in particular,
0 is global optimal with objective 29 Amax >max((aa) (X.).
Corollary 3. If we additionally use per-filter normalization (i.e., ||w 2 = 1/VTT), then Thm. holds and ii more constrained: [DI) + / 1 T.
Remark. Here we prove that given fixed a. maximizing E, E,(0) gives rank-1 solutions for deep linear network. This conclusion is extension of (Jing et al. 2022), which shows weight collapsing happens 0 is 2-layer linear network and a is fixed. If the pairwise importance a is adversarial, then it may not lead rank-1 solution. In fact, a can magnify minimal eigen-directions and change the eigenstructure of X. continuously, We leave it for future work.
Note that the condition that "W3,,,, has distinct maximal eigenvalue" is important. Otherwise there are counterexamples. For example, consider I-layer linear network z =W and X₂ has duplicated maximal eigenvalues (with u and U2 being corresponding orthogonal eigenvectors), then W.,, W = (i.c., it has degenerated eigenvalues), and for any local maximal W1. its row vector can be arbitrary linear combinations of u, and U2 and thus W, is not rank-1.
Compared to recent works (Ji et al. 2021) that also relates CL with PCA in linear representation setting using constant a, our Theorem 3 has no statistical assumptions on data distribution and augmentation, and operates on vanilla InfoNCE loss and deep architectures.

---

5 How Representation Learning Differs in Two-layer ReLU Network
So far we have shown that the max player maxe (O) (zr(Ca[=(0)) is essentially PCA objective when the input-output mapping = = W W()) is linear. A natural question arises. What is the benefit of CL if its representation learning component has such simple nature? Why can it learn a good representation in practice beyond PCA?
For this, nonlinearity is the key but understanding its role is highly nontrivial. For example, when the neural network model is nonlinear, Thm. T and Corollary [ holds but not Corollary Therefore, there is not even a well-defined Xa due to the fact that multiple hidden nodes can be switched on/off given different data input. Previous works (Safran & Shamir 2018) Du et al. 2018a) also show that with nonlinearity, in supervised learning spurious local optima exist.
Here take a first step analyze nonlinear cases. We study 2-1ayer models with ReLU activation h(x) max 0). We show that with a proper data assumption, the 2-layer model shares a modified version of dynamics with its linear version, and the contrastive covariance term X, (and its eigenstructure) remains well-defined and useful in nonlinear case.
5.1 The 2-layer ReLU network and data model
We consider the bottom-layer weight W1 = [w01,, W12.-... Wik] with wik being the k-th filter. For brevity, let K ny be the number of hidden nodes. We still consider solution in the constraint set e (Eqn. I1. since Lemma still holds for ReI.U networks. This model is named ReLU2Layer.
In addition, we assume the following data model:
Assumption 1 (Orthogonal mixture model within receptive field Rg). There exists a set of orthonor- mal bases {T.m m==1 so that any input data æri]] Em am [i]am satisfies the property that O. [i] is
M
Nonnegative: [ 2 One-hot: for any k; am > 0 for most one mn and Augmentation only
scales E& by (sample-dependent) factor, i.e., x(i'] >[ with 7/1] > 0.
Since all appears in the inner-product with the weight vectors WIk, with rotation of coordination, we can just set 2.m. = e., where en is the one-hot vector with m-th component being 1. In this case, 2 0 is always one-hot vector with only at most only one positive entry.
Intuitively, the model is motivated by sparsity: in each instantiation of æ, there are very small number of activated modes and their linear combination becomes the input signal æ. As we shall see, even with this simple model, the dynamics of ReLU network behaves very differently from the linear case.
With this assumption, we only need to consider nonnegative low-layer weights and Xa is still valid quantity for ReLU2Layer:
Lemma 3 (Evaluation of ReLU2Layer), If Assumption 7 holds, setting w'k = max(w 0) won't 't change the output of ReLU2Layen. Furthermore, if W 2 0, then the formula for linear network W. still works for ReLU2Layer.
Ea = (1r(WW,1,,,
On the other hand, sharing the energy function Ea does not mean ReLU2Layer is completely identical to its linear version. In fact, the dynamics follows its linear counterparts, but with important modifications:
Theorem (Dynamics of ReLU2Layer). If Assumption Tholds, then the dynamics of ReLU2Layer with Wik Z 0 is equivalent to linear dynamics with the Sticky Weight rule: any component that reaches 0 stays 0.
As we will see, this modification leads to very different dynamics and local optima in ReLU2Layer from linear cases, even when there is only one ReLU node.
5.2 Dynamics in One ReLU node
Now consider the dynamics of the simplest case: ReLU2Layer with only I hidden node. In this case, WT,, W>1 is a scalar and thus W,, W₂ tr(W,T W2) = 1. We only need to consider wi E R which is the only weight vector in the lower layer, under the constraint ||alle |/ile || 1 (Eqn. IT]. We denote this setting as ReLU2Layer1Hid.
The dynamics now becomes very different from linear setting. Under linear network, according to Theorem wi converges to the largest eigenvector of X,, = C. [21). For ReLU2Layer situation differs drastically:
7

---

Contrastive covariance X
cover iterations
Wi W3
05
0.
500
2000
Theorem shows that training ReLU2Layer could lead to more diverse hidden weight patterns beyond rank-1 solution obtained in the linear case (shown in right two figures: converged Wi and W2 W2).
Figure 3:
Theorem 5. If Assumption Toolds then in ReLUBLayerIHid w1 Cm for certain
Intuitively, this theorem is achieved by closely tracing the dynamics. When the number of positive entries of wi is more than the linear dynamics always hits the boundary of the polytope wi 2 making one of its entry be zero, and stick to zero due to sticky weight rule. This procedure repeats until there only one survival positive entry in wi
Overall, this simple case already shows that nonlinear landscape can lead to many local optima: for any m, wi Em is one local optimal. Which one the training falls into depends on weight initialization, and critically affects the properties of per-trained models.
5.3 Multiple hidden nodes
For complicated situations like multiple hidden units, completely characterizing the training dynamics becomes hard (if not impossible). Instead, we focus on fixed point analysis.
like Theorem 5
For deep linear model, using multiple hidden units does not lead to any better solutions. According to Thm. 3 at local optimal, W1 V2 Vn This means that the weights Wik, which are row vectors of Wi, just scaled version of the maximal eigenvector th of X.. Moreover, this is independent of the eigenstructure of Xa as long as Amax(Xa) > O.
In ReLU2Layer, the situation is a bit different. Thm. shows that these hidden nodes are (slightly) more diverse. Fig. shows one such example. The intuition here is that in nonlinear case, rank-1 structure of the critical points may be replaced with low-rank structures.
Theorem (ReLU2Layer encourages diversity). If Assumption 7 holds, then for any local optimal (W2, Wi) E θ of ReLU2Layer with either Wi ve,, for some m and 0, or rank(Wi) > 1.
Experiments
We evaluate a-CL framework (Def. 1 in CIFARIO (Krizhevsky et al. 2009) and STL-10 (Coates cet al. 201T) with ResNet18 (He et al. 2016), and compare the downstream performance multiple losses, with regularizers taking the form of R(o) Σ, Σ jot r(Osg) with constraint Σj Oij 1.
Here can be different concave functions:
(a-CL-rH) Entropy regularizer r=((;)) = -Tais log Oiji
(a-CL-n,,) Inverse regularizers (a,())) - (r > 1).
(a-CL-r,)) Square regularizer r, (ar)) = =
Besides, we also compare with the following:
Minimizing InfoNCE or quadratic loss: ming L(0) for L {Lnces Lquadratie).
Setting a as InfoNCE (Eqn. and backpropagates through a(O) with respect O. (a-CL-direct) Directly setting (here p 1):
exp( /T/
O.j Σ, expl-ppppp ((p,/T))
(12)

---

CIFAR-10
STL.-10
100 epochs
300
500 epochs
100 epochs
300 epochs
500 epochs
Equadratic
63.59 = 2.53
73.02 = 0.80
73,58 0.82
55.59 4.00
64.97 1.45
67.28
Lnce
84.06 = 0.30
87.63 0.13
$7.86
78.46 - 0.24
82.49
83.70
backprop a())
83.42 0.25
37.18
0.24
77.88 - 0.17
81.86
83.19 0.16
a-CI-rH
84.27 = 0.24
8.25
0.24
78.53 + 0.35
82.62
83.74
a-Cl-ry
83.72 - 0.19
87.51
87.69
78.22 0.28
82.19
83.47 +
a-CL-r,
84.72 = 0.10
36.62
86.74
76.95 + 1.06
80.64
0.77
81.65
0.59
a-CL-direct
85.11 1 0.19 87.93 0.16/8809 1 0.13
79.32 0.36
0.17
84.05
0.20
Table 1: Comparison over multiple loss formulations (ResNet18 hackhone, batchsize 128).
Top-1 accuracy
with linear evaluation protocol. Temperature 0.5 and learning rate is 0.01. Bold highest performance and blue is second highest. Each setting repeated times with different random seeds.
ResNet18 Backbone
ResNet50 Backbone
CIFAR-100 100
100 epochs
300 epochs
500 epochs
100 epochs
300 epochs
500 epochs
Lnce
55.70 0.37
59.71 0.36
59.89
0.34
60.16 0.48
65.40 0.31
65.53 0.30
a-CL-direct
57.63
0.07
02..0....2.... 0.26
60.27 + 0 29
62.93 + 0.28
65.84 0.
65.87
0.21
100
Lnce
84.06 0.30
87.6 0.13
87.86
0.12
86.39 0.16
89.97 0.14
90.19
a-Cl.-direct
85.11
0.19
0.19
87.93
0.16
88.09
0.13
87.79
+ 0.25
90.41
0.18 0.21
STL-10 10
Ince
78.46 +.25
82.49
83.70 0.12
81.64 0.24
86.57 0.17
87.90
a-CL-direct 79.32 0.36 0.178..55: 7.:8110.6666..::1176666611111111 0.20
83.20
0.25 87.17 0.14 87.85 0.21
Table 2: More experiments with 8/ResNe00 backbone on CIFAR-10, STL-10 and CIFAR-100.
Batchsize is 128. For ResNet18, 8, learning rate is 0.01; for ResNet50, learning rate is 0.001.
For inverse regularizer ry,, we pick y = 2 and r 0.5; for direct-set a, we pickp= and 0.5; for square regularizer, we use J = 5. All training is performed with Adam (Kingma & Ba (014)) optimizer. Code is written in PyTorch and a single modern GPU suffices for the experiments.
The results are shown in Tbl. We can see that (1) backpropagating through ex(O) is worse, justifying our perspective of coordinate optimization, (2) our proposed a-CL. works for different regularizers, (3) using different regularizer leads to comparable or better performance than original InfoNCE Lnce. (4) the pairwise importance does not even need to come from minimization process. Instead, we can directly set a based on pairwise squared distances di and d² For a-CL- direct, the performance is slightly worse if we do not normalize axj (i.e., aij- := exp( exp(-dy/r)) seems that for strong performance, da.y satisfy this condition (e-g., squared regularizer rss) may not work as well.
dr should go to too when 0uj + 0. Regularizers that do not
Tbl. shoows more experiments with different backbones (e.g., ResNet50) and more complicated datasets (e.g., CIFAR-100), Overall, we see consistent gains of a-CL over InfoNCE in early stages of the training (e.g., 1-2 point of absolute percentage gain) and comparable performance at 500 epoch. More ablations on batchsizes and exponent p in Eqn. 12 are provided in Appendix ??
7 Conclusion and Future Work
We provide novel perspective of contrastive learning (CL) via the lens of coordinate-wise opti- mization and propose a unified framework called a-CL that not only covers a broad family of loss functions including InfoNCE, but also allows a direct set of importance of sample pairs. Preliminary experiments on CIFARRIOSTTL-OCCIAARR
show comparable/better performance with the new loss than InfoNCE. Furthermore, we prove that with deep linear networks, the representation learning part is equivalent to Principal Component Analysis (PCA). J. In addition, we also extend our analysis to representation learning in 2-layer ReLU network, shedding light on the important difference in representation learning for linear/nonlinear cases.
Future work. Our framework a-CL turns various loss functions into a unified framework with different choices of pairwise importance a and how to find good choices remains open. Also, we mainly focus on representation learning with fixed pairwise importance However, in the actual training, a and 0 change concurrently. Understanding their interactions is an important next step. Finally, removing Assumption D in ReLU analysis is also open problem to be addressed later.

---

References
Allen-Zhu, Z., Li, Y., and Liang. Y. Learning and generalization in overparameterized neural networks, going beyond two layers. arXiv preprint urXiv:1811.04900 2018.
Arora, S., Cohen, N., and Hazan, E. On the optimization of deep networks: Implicit acceleration by overparameteriratton. In Interaational Conference on Machine Learning, pp. 244-253. PMLR, 2.
Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint (xXiv:1902009222 2019.
Ba. J. L., Kiros, R., and Hinton, G. E. Layer normalization. arXiv preprint 2016.
Baldi, P. and Horiik, K. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58. 1989.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In International Conference on Machine Learning, pp. 531-540. PMLR, 2018.
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. ECCV, 2018.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Chen, X. and He, K. Exploring simple siamese representation learning. In CVPR, 2020.
Coates, A., Ng. A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In International conference on artificial intelligence and statistics, 2011.
Coria, J. M., Bredin, H., Ghannay, S., and Rosset, S. comparison of metric learning loss functions for end-to-end speaker verification. In International Conference on Statistical Language and Speech Processing, pp. 137-148. Springer, 2020.
Du. S., Lee, J., Tian, Y., Singh, A., and Poczos, B. Gradient descent learns one-hidden-layer cmn: Don't t be afraid of spurious local minima. In International Conference on Machine Learning, pp. 1339-1348. PMLR, 2018a.
Du, S. S.. Hu, W., and Lee, D. Algorithmic regularization learning deep homogeneous models: Layers are automatically balanced. arXiv preprint 1806. 1000000000000000,, 2018b.
Fisher, R. A. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2): 179-188, 1936.
Grill, J..B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. Bootstrap your own latent: A new approach to self-supervised learning. NeurIPS, 2020.
Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
HaoChen, J. Z., Wei, C., Gaidon, A., and Ma. T. Provable guarantees for self-supervised deep learning with spectral contrastive loss. NeurIPS, 2021.
Hardt, M. and Ma, Identity matters in deep learning. ICLR, 2017.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
10

---

He, K., Fan, H., Wu, Y. Xie, S.. and Girshick, R. B. Momentum contrast for unsupervised visual representation learning. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-97355 2020.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
4700-4708, 2017.
Ji, W., Deng, Z., Nakada, R., Zou, J., and Zhang, L. The power of contrast for feature learning: A theoretical analysis. arXiv preprint arXiv:2110.02473 2021.
Jing, L., Vincent, P., LeCun, Y.. and Tian, Y. Understanding dimensional collapse in contrastive self-supervised learning. ICLR, 2022.
Kalantidis, Y., Sariyildiz, M. B., Pion, N., Weinzaepfel, P., and Larlus, D. Hard negative mixing for contrastive learning. NeurIPS, 2020.
Kawaguchi, K. Deep learning without poor local minima. NeurIPS, 2016.
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. NeurlPS, 2020.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69883 2014.
Kokiopoulou, E., Chen, J., and Saad, Y. Trace optimization and eigenproblems in dimension reduction methods. Numerical Linear Algebra with Applications, 18(3):555-622, 2011.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Laurent, T. and Brecht, J. Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pp. 2902-2907, PMLR, 2018.
Lee, J. D., Lei, Q. Saunshi, N., and Zhuo, J. Predicting what you already know helps: Provable self-super learning. Advances in Neural Information Processing Systems, 34, 2021.
Misra, I. and Maaten, L. v. d. Self-supervised learning of pretext-invariant representations. In CVPR. 2020.
Oh Song. H., Xiang, Y., Jegelka, S., and Savarese. S. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004-4012 2016.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint htl.:::8070374444 2018
Robinson, J., Chuang, C.-Y., Sra, S., and Jegelka, S. Contrastive learning with hard negative samples. ICLR, 2021.
Safran, I. and Shamir, O. Spurious local minima are common in two-layer relu neural networks. In International Conference Machine Learning, pp. 4433 4441. PMLR, 2018.
Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR, 2014.
Schroff, F., Kalenichenko, D., and Philbin, Facenet: unified embedding for face recognition and clustering. In CVPR, 2015.
Sohn, K. Improved deep metric learning with multi-class n- n-pair loss objective. In Advances in neural information processing systems, pp. 1857-1865, 2016.
Tian, Y. Student specialization in deep relu networks with finite width and input dimension. ICMI., 2020.

---

Tian, Y., Krishnan, D., and Isola. P. Contrastive multiview coding. In Computer Vision-ECCC ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020a.
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. What makes for good views for contrastive learning? NeurIPS, 2020b.
Tian, Y., Yu, L.,, Chen, X., and Ganguli, S. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578, 2020c.
Wen, Z. and Li, Y. Toward understanding the feature learning process of supervise contrastive learning. arXiv preprint arXiv:2105.15134 2021.
Wold, S.. Esbensen, K., and Geladi, P. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1 2(1-3):37-52, 1987.
Wu, Z., Xiong. Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-37422 2018.
Xiong, R., Yang. Y., He, D., Zheng, K., Zheng. S., Xing, C., Zhang, H.,, Lan, Y., Wang, L., and Liu, On layer normalization in the transformer architecture. In International Conference Machine Learning, pp. 10524-10533. PMLR, 2020.
Yeh, C.-H., Hong, C.-Y., Hsu, Y.-.., Liu, T.-L., Chen, Y., and LeCun, Y. Decoupled contrastive learning. arXiv preprint 1..iv::1100064444 2021.
Zhontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint 2rxiv:2/03.032:.) 2021.
Zhou, Y. and Liang, Y. Critical points of linear neural networks: Analytical forms and landscape properties. In International Conference on Learning Representations, 2018.
12


Reviews:
[{"review_id": "if2cOGJsuuc", "reviewer": "Reviewer_pNXS", "summary": "In this paper, the authors provide a different perspective of InfoNCE with a theoretical analysis. A broad family of loss functions could be represented by a coordinate-wise optimization on the network parameter and pairwise importance. The authors connect the relationships between PCA and InfoNCE. Based on this point, the authors demonstrate optimizing the network parameter equals recovering the optimal PCA solution.", "questions": "N/A", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper, although very theoretical, is still very easy to follow. I really enjoyed reading this paper. \n2. The novelty is undoubted. Although the relationship between contrastive learning and PCA is not the first time to propose. This paper gives a very interesting point about the min player and the max player. It also addresses the how the temperature parameter influences the hard samples mining.\n3. This paper gives a much formal reformulation to demonstrate the max player is equivalent to Principal Component Analysis (PCA) for deep linear network, and almost all local minima are global and rank-1, recovering optimal PCA solutions. \n4. The two-layer MLP based analysis provides a good reason for the dimensional collapse in InfoNCE and other methods. \n\nWeakness:\nThe relationship between contrastive learning and PCA is not the first time to propose. In contrastive laplacian eigenmaps (Neurips2021), the random regular graph is used. It is a Laplacian form of PCA. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fgKKSYrRBpk", "reviewer": "Reviewer_2mJ8", "summary": "In this work, authors show that contrastive learning (CL) is actually under a broad family of loss functions, and propose a unified formulation called \\alphaCL. This formulation unifies not only various existing contrastive losses, but also is able to extrapolate to give novel contrastive losses beyond popular existing ones, providing a guide for designing new CL loss.", "questions": "Question:\n\n1.\tI see authors analyze the representation of CL based on two-layer ReLU network in section 5. We know that Relu is local linear function so that it provides some advantages for analysis. How about other functions such as leaky relu?\n\n2.\tIn the experiments, authors propose different formulations for the selection of r. Could you tell readers how you design them, or which should they focus on if they want to design new loss. Such discussions may help readers to design their own losses based on their problems with your theorys. \n", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1.\tAlthough lots of CL loss has been proposed in the past few years motivated by different points, there is no work studying the interactions among them. This work is the first one, to my best knowledge, to reveal the connections or relations among these existing popular CL losses, with efficient proofs and clear descriptions.\n\n2.\tUp to now, the research of CL still remains an open problem what features does CL learn. Especially when we use black-box deep neural networks, it is difficult for us to understand the learned features. Based on the analysis of proposed general CL loss in this work, authors presents some special case to help up understand the representation learning by CL.\n\nWeakness:\n\n1.\tCurrent analysis and discussion are still based on some constrained assumptions, such as assumption 1 in the paper. And the understanding for the learned features of CL is still for some easy special case. \n\n2.\tThe experimental results are very easy but with lots of theoretical analysis in the main paper. For example, the quantitative results are obtained only on CIFAR-10 and STL-10, which is a very small dataset and easy task. Besides, authors just select ResNet-18 as the backbone. Readers may ask how about applying the new formulation in the big dataset such as Imagenet? How about the performance if we select different backbones? How about applying the new proposed formulations for other CL tasks such as graph learning? Moreover, it is better for authors to provide some visualized results to help readers understand their theories.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VUS-0A1EEPT", "reviewer": "Reviewer_n1zS", "summary": "This paper study a general class of loss function for contrastive learning that contains existing losses as special cases, which is formulated as a coordinate-wise optimization. The authors further proved analyses on the max player on deep linear networks and 2-NN.", "questions": "Please see the \"Strengths and Weaknesses\" section.", "limitations": "As discussed in the \"Strengths and Weaknesses\" section, the core analyses on 2-NN relies on a strong Assumption 1, which needs remedy.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ The problem itself is relevant and important for many practitioners in the community.\n+ The methodology is novel and general; the perspective of coordinate-wise optimization is interesting and the analyses are sound.\n+ The paper is well-written and easy to follow.\n\nCons:\n- The experiments is a bit weak: CIFAR-10 and STL-10 are relatively simple datasets. Can the authors also consider more involved datasets such as CIFAR-100?\n- Figure 2 does not have confidence bands (although I noted Table 1 does report standard deviations).\n- Assumption 1 seems a bit restrictive. It might be not satisfied by real-world (even simple ones), which makes Theorem 5 and 6 less interesting.\n\nSummary: although the theoretical analyses on the 2-NN case relies on a strong assumption, I think the problem formulation and the perspective of coordinate optimization are useful to the community. I recommend the authors (1) add experiments on larger real datasets, and add confidence bands in the figures; (2) empirically verify whether Assumption 1 holds; or construct synthetic datasets based on Assumption 1 and baselining it over other methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept