Zero-Mean Regularized Spectral Contrastive Learning: Implicitly Mitigating Wrong Connections in Positive-Pair Graphs
Authors: Xiong Zhou
Xianming Liu
Feilong Zhang
Gang Wu
Deming Zhai
Junjun Jiang
Xiangyang Ji
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive Learning
Unsupervised Domain Adaptation
Learning with Noisy Labels
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a theoretically sound and effective approach for mitigating wrong connections in spectral contrastive learning with positive-pair graphs .

Abstract: Contrastive learning has emerged as a popular paradigm of self-supervised learning that learns representations by encouraging representations of positive pairs to be similar while representations of negative pairs to be far apart. The spectral contrastive loss, in synergy with the notion of positive-pair graphs, offers valuable theoretical insights into the empirical successes of contrastive learning. In this paper, we propose incorporating an additive factor into the term of spectral contrastive loss involving negative pairs. This simple modification can be equivalently viewed as introducing a regularization term that enforces the mean of representations to be zero, which thus is referred to as *zero-mean regularization*. It intuitively relaxes the orthogonality of representations between negative pairs and implicitly alleviates the adverse effect of wrong connections in the positive-pair graph, leading to better performance and robustness. To clarify this, we thoroughly investigate the role of zero-mean regularized spectral contrastive loss in both unsupervised and supervised scenarios with respect to theoretical analysis and quantitative evaluation. These results highlight the potential of zero-mean regularized spectral contrastive learning to be a promising approach in various tasks.

Published as a conference paper at ICLR 2024
ZERO-MEAN REGULARIZED SPECTRAL CONTRASTIVE LEARNING: IMPLICITLY MITIGATING WRONG CON- NECTIONS IN POSITIVE-PAIR GRAPHS
Xiong Zhou, Xianming Liu; Feilong Zhang, Gang Wu, Deming Zhai Junjun Jiang
Faculty of Computing, Harbin Institute of Technology {cs2x, csxm, fizhang, gwu, zhaideming, jiang jun jun) @hit edu. on
Xiangyang Ji
Department of Automation, Tsinghua University xyji@tsinghue enw.ghhuueed.cccc edu.on
ABSTRACT
Contrastive learning has emerged as a popular paradigm of self-supervised learning that learns representations by encouraging representations of positive pairs to be similar while representations of negative pairs to be far apart. The spectral con- trastive loss, in synergy with the notion of positive-pan pair graphs, offers valuable theoretical insights into the empirical successes of contrastive learning. In this pa- per, we propose incorporating an additive factor into the term of spectral contrastive loss involving negative pairs. This simple modification as introducing a regularization term that enforces the mean of representations to be zero, which thus referred to zero-mean regularization. It intuitively relaxes the orthogonality of representations between negative pairs and implicitly alleviates the adverse effect of wrong connections in the positive-pair graph, leading better performance and robustness. To clarify this, we thoroughly investigate the role of zero-mean regularized spectral contrastive loss both unsupervised and supervised scenarios with respect to theoretical analysis and quantitative evaluation. These results highlight the potential of zero-mean regularized spectral contrastive learning to be promising approach in various tasks.
be equívalently viewed
INTRODUCTION
Contrastive learning has emerged one of the most prominent self-supervised learning paradigms, which offers promising representations that can be adapted to diverse downstream tasks [5, 6, 7, 8, 19, 23, 30, 55, 56]. Contrastive losses serve the training objectives for contrastive learning, encouraging the learning of representations invariant to data augmentations by maximizing the similarity between features from different distortions of the same images. In addition unsupervised training, contrastive losses are extended to the fully-supervised setting as an alternative to the classical cross-entropy loss to effectively leverage label information [26]. This extension enables contrastive learning to achieve state-of-the performance on various supervised learning tasks.
Researchers have attempted offer theoretical understanding of the empirical successes of contrastive learning [1, 46, 50, 51, 53]. Some works provide mathematical analysis, which, however, are under an impractical assumption that two views are somewhat independent conditioned on the label [29, 43, 52]. Instead, spectral contrastive loss (SpeCL) presents solid theoretical foundations without requiring conditional independence but on a more realistic data property that there is continuity of the population data within the same class [19, 20]. The core concept in SpeCI. is the positive-pal -pir graph on data, where nodes are augmented samples and the edge between two nodes is weighted as the probability of encountering them positive pair. By applying spectral decomposition the adjacency matrix defined on the population augmentation graph, SpeCL builds the relationship between the eigenvector matrix and the learned representations. SpeCL naturally exhibits sample-
"Correspondence Xianming Liu <esxm @ hit.cdu.cn>

---

Published as a conference paper at ICLR 2024
contrastive property, while the covariance regularization term within the equivalent form [20] can be seen as contrastive between the dimensions of the representations, and thus coincides with the cimeensionn-contrastivve property. Therefore, SpeCL serves as bridge between sample-contrastive and dimension-contrastive properties, offering pathway to unify them. Moreover, SpeCI. resembles state-of-the-art contrastive losses, such Barlow Twins loss [57] and VICReg loss [1, 2].
Although SpeCL has appealing theoretical advantages, there are still two limitations carefully consid- ered in this paper. Firstly, SpeCL intuitively requires the orthogonality of representations between negative pairs. However, relaxing the orthogonality could potentially enhance the discriminativeness of representations. Secondly, the pairwise similarities among the representations learned by SpeCL are determined by the connections within the positive-pair graph. Unfortunately, the presence of false positive pairs, which may be due to noisy views in self-supervised learning [10] or noisy labels in supervised learning [63], can lead to wrong connections (i.e., unfavorable edge weights). Identifying and correcting these wrong connections is scarcely possible since the underlying structure of the graph is unknown. To overcome these limitations, we propose to incorporate an additional factor T into the term of SpeCL involving negative pairs, which is simple yet effective to relax the orthogonality and coincides with the motivation of margin-based losses [12. 33, 34, 54, 61]. By algebraic manipulation, it can be found that our scheme is equivalent to regularizing the mean of representations to be zero, thus referred to zero-mean regularization. Furthermore, of r explicitly performs a uniform reduction in all positive-pair weights, which modifies pairwise similarities between positive pairs and implicitly mitigates the adverse effects of wrong connections.
would establish that the introduction
To demonstrate the effectiveness of zero-mean regularization in mitigating adverse effects of wrong connections, we provide theoretical investigations in both unsupervised and supervised scenarios: unsupervised domain adaptation (UDA) and supervised learning with noisy labels. For contrastive pretraining based UDA, wrong connections occurs when positive pairs are built from different domains and classes, we prove that the incorporation of T tightens the downstream error on target domains in the form of multiplying factor (1 - r)2. For supervised learning with noisy labels, false positive pairs originate from the misguidance of label noise. We first establish the supervised version of SpeCL and show that the global minimizer exhibits a geometric structure similar to the recently discovered Neural Collapse [39], and we prove that zero-mean regularization can mitigate label noise by implicitly reducing mislabeled weights in the noise transition matrix.
The main contributions of our paper can be summarized as follows:
We propose to incorporate an additional factor into spectral contrastive loss involving negative pairs, which is equivalent to the zero-mean regularization term. We also show that zero-mean regularization implicitly performs a uniform weight reduction in positive pairs.
We investigate the role of zero-mean regularized spectral contrastive loss on spectral con- trastive pretraining- based unsupervised domain adaptation and offer theoretical proof that gero-mean regularization can tighten the error bound by multiplying factor less than one.
We establish the supervised version of spectral contrastive loss and derive the closed-fone optimal representations, which resembles the Neural Collapse phenomenon and suggests using class-mean features classifier. We further prove that zero-mean regularization can mitigate label noise by implicitly reducing mislabeled weights in the noise transition matrix.
PRELIMINARY
We consider datasets containing samples drawn from the input space X' with the distribution Pdata that can be partitioned into r different classes with the label function y(-) : X - [r]. The representation
fz) E d is extracted by mapping : X' - Rd that is usually characterized by a number of parameterized compositions, where d is the representation dimensionality.
Spectral Contrastive Loss.
The spectral contrastive loss is formulated as [19]:
L(F) =-2 (((,,tt)~pppp [f(=)T f(x }] Eie )r-paat [((=)
E E [11s(x)= - f(x f(x+)\11 M1
Ex [f())(()T]] 11/ +const,
(2.1)
Roll)
R((()
2

---

Published as a conference paper at ICLR 2024
0.15 0.02
0.15
0.1625
Owl
Owl
1
0.06 0.02
0.7 0.02
0.045
0.773
Dog
Dog
0.06
0.04
0.02
0.06
0.02
0.05
0.025
D.OS
0.06
no45
0.45
Pig
Pis
0.06
0.02
0.06
0.02
0.045
015
0.02
0.1625
0.1625
Tiger
Rabbit
Tigar
Rabbit
(8)
(b)
(c)
(d)
Figure 1: Illustration of the uniform reduction of positive-pair graphs implied in self-supervised learning (a-b) and learning with noisy labels (c-d). (a) Edge weights denote connectivity (probability positive pair). The red dashed edges means adverse connections since
of sampling the endpoints the similarity of two very different samples will hinder the discriminativeness of representations. When these weights in the positive-pair graph are subtracted with 0.02 and then normalized, resulting in (b) The red dashed edges are alleviated as the weights decrease, while the consistency enhanced self-loop weights increases. (c) In the presence of label noise, positive-paii weights for supervised spectral contrastive learning can be viewed as the label noise transition matrix. Here, edges weights denote the probabilities of annotating the sample every labels. Orange arrows indicate the adverse connections which introduce label noise. By subtracting these weights by 0.02 and normalizing them, obtain (d) The orange connections are mitigated as their weights reduced.
where (Ir, x+) Ppos is a pair of augmentations of the same data, x=) is pair of independently random augmented data, and I the identity matrix. The second equality, as identified in [20], reveals that the spectral contrastive loss has two terms: (i) (J),). namely the invariance term, measures the & distance between positive pairs, which can also be expressed as the Dirichlet energy of representations on the positive-pair graph [1]: (ii) R2 (f), restricts the representation covariance towards the identity matrix, similar to Barlow Twins loss [57] and VICReg loss [1, 2]. Spectral contrastive loss naturally falls under the category of a sample-contrastive method, which also exhibits the diinensionn-cntrasttiv characteristic due to the existence of the (1( term.
Limitations. Although spectral contrastive loss achieves promising theoretical and quantitative results, there still some limitations needed be carefully considered: (i) SpeCI. intuitively requires the orthogonality of representation between negative pairs, while the InfoNCE loss [38] used in SimCLR [6] and MoCo [23] aims to make them opposite. Relaxing the orthogonality constraint would bring SpeCL closer to these prior works and may also enhance the discriminativenes. of representations, particularly for supervised contrastive learning Section 3.3. (ii) The pairwise similarities among the representations acquired through SpeCL are dictated by the relationships within the positive-pair graph. However, rectifying incorrect connections proves to a formidable task, given that the inherent structure of the graph remains elusive. SpeCI. does not incorporate any mechanism to mitigate the detrimental impact of connections stemming from erroneous positive pairs, which frequently arise in various tasks due to the inclusion of noisy views [10], noisy labels [63], and other contributing factors, as illustrated in Figure 1.
3 ZERO-MEAN REGULARIZES SPECTRAL CONTRASTIVE LEARNING
In this section, we introduce in detail main contribution zero- mean regularization for SpeCL. We explain its motivation and how it can be applied in SpeCL for alleviating detrimental connections in positive-pair graph on unsupervised domain adaptation and learning with noisy labels. We also provide more clarification in Appendix A.
3.1 ZERO-MEAN REGULARIZATION
We introduce the zero-mean regularization on SpeCI. [19], which is simply incorporating an additional factor the negative part of the original SpeCL. The population objective is formulated as:
L(iTT) = -2 E. (E Prppa [s(x) [(x))(xx)))) + E( ")xpa))<pan [((()] f(x (2=7))+)))))))) (1))))77}))) +
(3.1)
3

---

Published as a conference paper at ICLR 2024
It can be found that, compared with Equation 2. the only modification is the introduction of the additive factor T. To more clearly show the role of r, we further derive its equivalent form:
[(f:T) = Buz [I]((z) - f(x f(x+))1
EE (()))(()]]]1)) +
[ [f())]l + tonst,
Ro(
R((
R2(f)
(3.2)
where R2( regularizes the mean of representations be zero and thus is referred as zero-mean regularization, and 7 2 0 is actually the trade-off parameter that controls the regularization strength.
Remark. Zero-mean regularization can effectively alleviate the limitations of SpeCL: (i) The introduction of r is initially motivated by the observation that SpeCI. requires orthogonality of repre- sentations between negative pairs, which, in the supervised scenario (as described in Section 3.3), is roughly requiring the orthogonality between different classes. However, prior work has demonstrated that the representations learned by cross entropy and mean-squared error during the terminal phase of training usually exhibit the Neural Collapse phenomenon [17, 36, 39, 61, 65], which shows that the representations of different classes are maximally distant and form simplex Equiangular Tight Frame (ETF)'. Intuitively, the introduction of T can encourage the representations of negative pairs to have larger angles and thus makes them more discriminative. (ii) Spectral contrastive learning is equivalent to spectral decomposition on positive-pan pair graph. The introduction of T implicitly reduces all these positive-putt weights uniformly in Equation 3.3, which does not change the mag- nitude relationship between positive-pair weights but alters the ratio between them, as illustrated in Figure I. This explicitly modifies pairwise similarities between representations, which further indicates the benefits in decoupling the class and domain information for contrastive pretraining based unsupervised domain adaptation (cf. Section 3.2) and mitigating label noise by reducing mislabeled weights in the noise transition matrix (ef. Section 3.3.2).
In the following, we introduce in detail zero-mean regularization works in both unsupervised and supervised versions of SpeCI.. We carefully discuss in theory how is helpful in downstream unsupervised domain adaptation and supervised classification with noisy labels.
3.2 UNSUPERVISED DOMAIN ADAPTATION
In the UDA setting, we have access to labeled data from a source domain and unlabeled data from target domains, and the goal is to achieve good performance on target domains [4. 41, 42]. We consider a multi-way classification problem, where is the number of classes, p the number of domains, and n is the number of examples in each class of each domain. The total number of data is N rpn. For data x E X, we use d₂ E [p) and yo [r] to denote domain and class, respectively.
Contrastive Pre-training based UDA. While conventional UDA methods typically leverage the intuition of learning domain-invarian representations [14, 45, 49, 58), recent research by Shen et al. [44] has demonstrated that model first pre-trained on both the source and target domains with SpeCL and then fine-tuned using labeled source data, can yield comparable or superior results to strong UDA methods. Furthermore, the authors have theoretically proven that spectral contrastive pre-training enables the learning of representations that vary substantially across domains, while still generalizing the target domain by disentangling domain and class information, instead of merely satisfying domain invariance. Following the setup of [44], we also consider a stochastic block model (24) for the positive-pair graph, where the probability of existence of an edge between and x' is: (1)
if dz = d and yı = Yr', (2) a if dg of de' and y, = Ye' (3) 3 if de = de and ys * ys', (4) if X be the adjacency matrix of the graph. Let d = p + r 1 be the feature dimension 2 and f X + Rd be the representation model.
de * de and ye # yu'. Let A E R- NNN
An ETF is a collection of vectors V = [1i, U2,... Uy] E dx having equal lengths and equal, maximally separated pair-wise angles. In the setting of < d 1, we have V = C(I --111) [39, 48, 62).
'The reason we assume d = p-r------ is to facilitate a better derivation and arrive at more relevant conclusions.
As depicted in the proof of Theorem 4.1, there are exactly p + r- nonzero eigenvalues Á. By assuming 1, we can easily write the d-rank approximation of Á according to spectral decomposition. This
d = p also reveals that the effective dimension of the feature space is p + r 1. In the scenario where d > p + r 1. we can directly set the values of excess dimensions to zero without compromising the validity our conclusion. However, we acknowledge that this assumption may appear restrictive. In cases where d < p + r - 1, the corresponding derivation becomes intractable, making it challenging to obtain the exact form of Ad-
4

---

Published as a conference paper at ICLR 2024
P
P
P
P P-t
Good Case: a, B >
Bud Case: y
B
(d)
The illustration primarily originates from [44]. (a) Illustrative example in binary classifi- cation with two domains, where each class-domain pair is a node in the positive-pair graph. Edge weights denote the probability of sampling the endpoints as a positive pair. p denotes the probability of sampling a pair of the same domain and class. (b) The edge weights are subtracted by a constant T 7, which does not change the magnitude relationship of the edge weights but simply changes the ratio between them. (c) When a and 3 are greater than y, the features are oriented so that source- trained linear classifier generalizes to the target domain. The class and domain information are disentangled along the vertical and horizontal axes, respectively. (d) When < 7, the target features are flipped and the source-trained classifier can not generalize. As can be seen, the weights of pairs belonging to the same domain and class is relatively increased, while the weights of different domains or classes are relatively diminished, i.e., max max > max(a.B] when p > max (a, 3}. which benefits in terms of decoupling the class and domain information. Furthermore, the con- nections of pairs from different domains and different classes in the graph are also weakened, iee., min min(a.8) (0.9) which helps to improve discriminativeness across domains and classes.
Figure 2:
p-T
The Uniform Weight Reduction in Positive-Pair Graphs. Let EE| denote the total number of edges, according to the definition of SpeCL in Equation 3.1, we can rewrite the loss function as
2
[(x) T f(x')
+ Σ N2
(f(x) [(x') +T)²
E
(3.3)
J/ ( FF) 7|
2
N [
(A-
(A-
TIE 11
+
const,
where F E xN is the matrix whose the z-th row contains f(x)T. We note that the introduction of 7 reduces the positive-pair weights uniformly, i.e., all the entries in A are subtracted by r| As illustrated in Figure 2, subtracting a suitable positive constant from the edge weights preserves their magnitude relationship while altering the ratio between them. Specifically, the weights of pairs belonging to the same domain and class are relatively increased, whereas the weights of pairs from different domains or classes are relatively decreased. This adjustment explicitly modifies pairwise similarities between representations, leading to further benefits in terms of decoupling the class and domain information. Additionally, the connections of pairs from different domains and different classes in the graph are weakened, which enhances discriminativeness across domains and classes. We further provide theoretical analysis that can tighten the downstream error on the target domains.
N2
In UDA, we care about the error on target domains. Let S = {ar E X' dz 1} and T = {(r X d * 1} be the source and target domains, respectively. Given the labeled source domain data, we leam the linear classifier with a pre-trained representation model f:
i = arg min BERdX
Σ E(IIT (xx 11 m/bb))?))
(3.4)
ZES
where T. cy. -. R" is the mean-zero one-hot embedding of the label, n) 0 is the regularization strength. For data E T, we use pred(x) arg max, (TT f(x)) as the predictor, where f achieves the global minimum of Equation 3.3. Let pred E Nxr be the matrix with 8T f(x on its 2-th row and pred, be the matrix by restricting pred to the target domain.
Compared to Shen et al. [44], we provide a more general form of the error bound when training SpeCI. with mern-mean regularization across multiple domains as follows:
5

---

Published as a conference paper at ICLR 2024
Theorem 3.1. Let and E (0. be arbitrary constants. In the above stochastic block model, assume p p> max(a, 3). 7 min(a, }. and r ()) ())))\\((0). Then, there exists
T [1-e, 1}, such that for any Q min min(a-7.B-7))
and regularization strength vj (0. (a-1)e
2rp
with high probability S, we have
(T)
pred, - pred 0o
poly(r,
(3.5)
(P(AlT)) )a+1(T))
where pred, E[pred7] is the expectation of the prediction matrix when achieving the minimum
of the loss in Equation 3.3 with T E [0, 1), (T), and (T+ are the 1-st, d-th and d - 1-th
eigenvalues of A A- N I respectively.
+|
Furthermore, the target error can be bounded by
( - (241(
(AI(T))²
(pre(p)e # yn) SO (P((T))))
poly(r:p),
(3.6)
where poly(r,p) denotes a polynomial function of and p.
In the following, we show that the introduction of T through zero-mean regularization can tighten the error bound:
Proposition 3.2. the setting of Theorem 3.1, if (0)--((2(0) w x1(
3:
then, with probability at least 1 55, we have
Px~T (pred ys) 50 (1 (1-7)²(2
17((aa(0) - (0))2 n poly((rp))
(3.7)
where (0) = np + n(r 1)3 + nlp 1)a + nip 1)(r - 1)y and Ta(0) - (0+1 = min/r(3- 7). }
As can be seen, the eтror decreases when the number of samples n increases, and it also controlled by the difference between the d-th and d + 1-th eigenvalues. This difference, in turn, depends on the gap between the serros-clas/aaerss-ddmaaa connectivities B, and the across-both connectivities y. Furthermore, Proposition 3.2 provides additional insight that the incorporation of tighten the error bound in the form of multiplying factor (1 - =) This theoretical foundation underpins the proposed zero-mean regularization in the UDA task.
3.3 SUPERVISED CLASSIFICATION WITH NOISY LABELS
The primary objective this subsection is to establish the supervised version of SpeCL with zero- mean regularization. We subsequently derive the closed-form optimal representations by minimizing the supervised SpeCL, which leads to a result similar to the neural collapse solution. Furthermore, we demonstrate that zero-mean regularization can mitigate label noise by implicitly reducing mislabeled weights in the noise transition matrix.
3.3.1 SUPERVISED SPECTRAL CONTRASTIVE LOsS
Similar to the supervised contrastive loss [26], we can define the supervised version of SpeCL with the zero-mean regularization for the labeled dataset (X, Y) as
(sup([; = pos (ʃ) + (nie(fiT))
2
1 13
C (f)
ama(())))
f(x))] and Alzs,e))
rn
(3.8)
Cneg
ΣΣΣΣ amalee)) ~((s,))))
[f(x) f(x f(x))
n
i= c=
where Ti, denotes the i-th example the class c, n is the number of examples each class, and A(x: denotes the augmentations of example

---

Published as a conference paper at ICLR 2024
Remark. Note that we retain the condition A within the negative part Lneg, which is raised for two main reasons. Firstly, this condition aligns with the self-supervised form of spectral contrastive loss in Eq. 7, which facilitates the derivation of the positive and negative parts into spectral decomposition as evidenced in the proof of Theorem 3.3; Secondly, the inclusion of the constraint c # l in Lncy would result in that the supervised loss goes infinity as the similarity f(x) f(x)) in Cpos has the potential to become arbitrarily large. By incorporating positive pairs in Lney" can effectively address this issue. This mechanism is analogous to the presence of numerator in the denominator in CE and InfoNCE, serving to ensure training stability.
The supervised version - (f=P((;T) differs from the self-supervised version L(;;T) ) in that pos (J) uses samples in the same class as positive pairs, resulting in class-dependent connections in the positive pair graph for supervised spectral contrastive learning.
SpeCT Theorem 3.3. The global minimum of the supervised SpeCL (f:T((;:) in Equation 3.8 is uniquely
obtained at Vi E [ma], Vc E [r], Va A(z f(x) = ha, where H [ha..... hT] is the minimizer of [ - H I|
A more specific conclusion of Theorem 3.3 is that we have H H r1--- when S d 1, and when r > d 1, H H Pa(r1 -117) where Pa(X) denotes the best d-rank approximation of X. These results provide insights into the structure of the learned representations and highlight the impact of zero-mean regularization on the geometric characteristies of the learned representations. One straightforward observation is that, as T increases, the angle between representations of different classes (i.e, Zhe, hy) = arccos for o j) increases, thereby leading to better discrininativeness When T = and S d + 1, we have ft ft = (r - 1)(1 - =11)) that forms the neural collapse solution proposed in [39].
Remark. Resembling the neural collapse phenomenon [39], the optimal minimizer of supervised SpeCL is characterized by two manifestation for last-layer features when r S d and 7 [0. 1]: (i) Within-class variability collapse. The within-class variation of features becomes negligible as they collapse to their respective means. Specifically, for all classes c and samples x ~ A((i,,)) the feature becomes a constant value fa:) he. leading reduction in with-class variability; (ii) Convergence to an equiangular frame. The vectors the class means converge to having equal length, forming equal-sized angles between any pair. This is represented by the equation H 11 r1 -11 T These two manifestation indicates that the zero-regularized SpeCL exhibits a phenomenon similar to neural collapse even though it exclusively employs features without the linear classifier to calculate loss. Regarding the downstream linear classification, this suggests that we may easily utilize class-meat features as the classifier. Compared Table Table 12, class-mean features as classifier achieve comparable results with the trained classifier on CIFAR-10 and SVHN.
3.3.2 THE IMPLICATION OF MITIGATING LABEL NOISE
As mentioned in Section 3.3, the introduction of r can encourage the discriminativeness of different classes. In the following, we will prove that zero-mean regularization can mitigate label noise. Consider the label noise transition matrix W = (wij)rxr: where wij denotes the probability of flipping the true class j into class Then, for each sample Ti.p. in the true class Y/ E [r], the probability of its label being de E [r] is Wy.d.. To analyze the supervised SpeCL in the presence of label noise, we let C (f:T) denote the expected loss on the label noise model, i.e., Laup (fi =
Cy (f:T, (T,,,(X (X, Y)). where Ý is the flipped labels of Y. We can derive the results similar to Theorem 3.3 as follows:
Theorem 3.4. For the noise transition matrix W defined above, the global minimum of goup (fiT) uniquely obtained Vi E [n), Vc [r], Vx ~ A(Ii,)), f(x) ho where H [h1,... hr] the minimizer of [[( -11 H H p
'We have -2s f(x) f(xx) + + [f(x] f(x*) +(=))++)²= [f(x] f(x (s + const.
'Our analysis covers all choices of representation dimension and the numher of classes, unlike previous work [17, 36, 50] that only considers cases where the feature dimension is larger than the number of classes.
7

---

Published as a conference paper at ICLR 2024
55%
15%
15%
15%
10%
10%
10%
10%
15%
55%
15%
15%
10%
10%
10%
575
45%
5%
5%
8.3%
15%
15%
55%
15%
10%
10%
10%
10%
5%
45%
8.3%
15%
15%
15%
55%
10%
10%
10%
10%
5%
(a) Noise Transition Matrix
(b) Uniform Reduction zl
(c) Reducod Matrix
(d) Re-normalized Matrix
Figure 3: Illustration of zero-mean regularization for mitigating label noise. (a) The noise transition
matrix that is clean-label dominant [63] with the noise rate 45%. (b) Zero-mean regularization causes all elements of the transition matrix to be subtracted by constant 7 = 10%. (c) The mislabeled (off-diagonal) weights reduced be 5% and the clean (on-diagonal) weights reduced to 45%. (d) The re-normalized matrix shows that the overall noise rate is reduced to 25%, suggesting that zero-mean regularization a simple yet effective way to mitigate label noise.
Table I: 1 liner probing (%) and Top-1 validation accuracy (%) of self-supervised learning and supervised learning with the spectral contrastive loss in Equation I. respectively. All results reported by "mean + std" are ran 3 trials. The results of the spectral contrastive loss with zero mean regularization 7 > 0 with better performance than the original spectral contrastive loss r = 0 are highlighted. The best results underlined.
Self-Supervised Learning
Supervised Learning CIFAR- 100
Method
CIFAR-10
CIFAR- -00
SVHN
CIFAR-10
SVHN
SpeCI.
86.22 + 0.12
52.66 + 0.16
89.77 + 0.19
93.47 + 0,46
65.59 + 0.75
96.05 + 0.16
0.1
86.83 = 0.12
53.97 + 0.19
90.26 + 0.07
94.21 + 0.13
68.15 + 0.19
96.22 + 0.05
0.2
86.84 = 0.10
55.27 1 0.02
90.78 1 0.07
94.52 1 0.05
68.04 0.18
96.28 0.05
T =
87.15 0.12
56.37
0.27
91.10
0.13
94.36 1 0.12
69.06
0.34
96.22 0.09
T == 1.0
87.09 1 0.16
56.63 1 0.10
91.24 1 0.09
94.38 1 0.06
69.59 1 0.11
96.25 1 0.10
This theorem reveals that the immediate effect of r is to implicitly reduce the mislabeled weights (wij. i + j)) in the noise transition matrix, which in turn mitigates label noise. For a better understanding. we provide a more direct view for symmetric label noise.
Proposition 3.5. Considering the symmetric label noise [15] in which Wee = 1 (r 1)w for ccc, c', Wee = for # d' and w < 2 is also the global minimizer of L (f: 1T
2211f IfT 2 rw, let f = arg miny Loup (fiT), then vi-nw
Proposition 3.5 indicates that, when 7 2 rw, using S (f:T) training on symmetric noisy labels is equivalent to using Caup reduction 15%, the re-normalized matrix will be an identify matrix.
f: 1-rw training on clean labels. As illustrated in Figure 3, if the uniform
EXPERIMENTS
In this section, we provide extensive experiments on the tasks of contrastive learning, supervised classification, unsupervised domain adaptation, and learning with noisy labels to verify the effec- tiveness of zero-mean regularization on several benchmark datasets with ResNets [21, 22]. More experimental results and details can be found in the Appendix B.
Results Self-Supervised Learning and Supervised Learning. Table reported the linear probing accuracy of self-supervised learned models and the Top-1 accuracy of supervised learned models with spectral contrastive loss in Equation 3.1 and supervised spectral contrastive loss in Equation 3.8. The results show that zero-mo zero-mean regularization can learn more discriminative representations.
Results of Unsupervised Domain Adaptation. We evaluate zero- zero-mean regularization across four digits datasets with different domains: SVHN (S) [37]. MNIST (M) [28]. USPS (U) [25], and MNIST-M (M-M) [14]. We first pre-train models with the modified spectral contrastive loss in Equation 3.1, and then fine-tune a linear classifier composited the models on the source domain.
8

---

Published as a conference paper at ICLR 2024
Table 2: Top-1 accuracy (%) of unsupervised domain adaptation four digit datasets. The results of the spectral contrastive loss with zero mean regularization T 0 with better performance than the original spectral contrastive loss
are highlighted. The best results are underlined.
S-
S-UU
S >M-M
M U
M
M-M
M-M S
Average
Vanilla SpeCI.
88.10
80.37
75.24
94.47
51.19
37.32
71.12
0.1
88.16
82.76
76.10
95.71
72.10
44.78
76.60
0.2
86.44
82.96
76.10
94.87
63.63
37.45
73.58
0.5
88.44
83.16
77.34
90.43
64.68
48.03
75.35
1.0
88.73
83.81
79,81
96.16
59.48
38.73
74.45
Table 3: Top-1 test accuracy (mean 1stdd %) of supervised spectral contrastive loss with zero-mean regularization in Equation 3.8 on benchmark datasets with symmetric label noise. The results of the spectral contrastive loss with zero mean regularization T > 0 with better performance than the original spectral contrastive loss are highlighted. The best results are underlined.
Symmetric Noise Rate
Datasets
Methods
0.2
0.4
0.6
0.8
CE Focal
82.88 = 0.04
68.99 + 0.15
52.27 + 0.20
51.62 + 0.08
60.91 0.07
46.73 0.13
29.47 0.04
13.54 0.03
GCE
91.35 + 0.04
89.14 + 0.01
80.50 + 0.07
52.25 0.03
Vanilla SpeCL
90.80 = 0.01
88.29 + 0.13
85.87 = 0.07
75.51 0.27
T == 0.1
91.18 - 0.07
88.99 + 0.07
86.66 + 0.04
80.29 + 0.39
CIFAR-I0
0.2
91.57 = 0.03
89.13 = 0.12
86.81 = 0.03
80.06 0.39
T == 0.5
91.31 + 0.06
89.49 + 0.02
86.59 + 0.09
78.51 + 0.25
1.0
91.42 + 0.07
0.05 89.26
86.98 0.07
76.63 0.30
CE
60.94 + 0.06
46.07 0.01
30.33 0.02
14.41 0.06
Focal
60.91 + 0.07
46.73 + 0.13
29.47 + 0.04
13.54 + 0.03
GCE
68.91 0.03
64.87 0.07
56.04 0.04
9.13 = 0.00
Vanilla SpeCL
67.87 + 0.05
62.97 + 0.11
57.51 0.02
51.10 + 0.08
0.1
68.63 = 0.09
64.46 = 0.04
59.41 = 0.06
52.32 - 0.09
CIFAR-100
T = 0.2
69.60 = 0.07
65.04 0.07
59.46 - 0.04
53.90 + 0.06
0.5
69.57 + 0.06
65.05 + 0.06
59.17 + 0.09
52.80 0.08
r = 1.0
69.50 + 0.15
65.37 + 0.07
59.41 + 0.01
52.60 + 0.07
Table 2 demonstrates that zero-mean regularization > can help decoupling class information of representations across domains. Experiments on DomainNet [41] are provided in Appendix B.4.
Results Learning with Noisy Labels. To further verify that zero-mean regularization can mitigate label noise. We conduct experiments in the scenario of symmetric label noise, considering baselines CE, Focal loss [32], and GCE [59]. As shown Table 3, r >> performs better than SpeCL in all cases, especially for high noise. This also show that contrastive learning can mitigate label noise.
CONCLUSION
This paper introduces zero-mean regularization to enhance spectral contrastive loss, which equiva- lently incorporates an additive factor into the loss component involving negative pairs and thus makes representations of negative pairs opposite. Provable accuracy guarantees are achieved under linear probe evaluation for contrastive learning with restricted model classes. Additionally, zero-mean regularization reduces positive-pair weights uniformly during contrastive pretraining for unsuper- vised domain adaptation, tightening downstream error on target domains. The supervised version of spectral contrastive loss reveals structure resembling the neural collapse phenomenon, with larger regularization strength indicating improved discriminativeness. Moreover, zero-mean regularization mitigates label noise by implicitly reducing mislabeled weights in the noise transition matrix. These findings highlight the superiority of gero-mean roeeannegggrrng regularization enhancing contrastive learning and its potential applications in various domains.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGE
This work was supported in part by National Natural Science Foundation of China under Grants 92270116, 62071155 and 623B2031, and in part by the Fundamental Research Funds for the Central Universities (Grant NO.HI...CEE::2234433 and HIT.DZZUJ.2220755).
REFERENCES
[1] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In Advances in Neural Information Processing Systems, 2022.
[2] Adrien Bardes, Jean Ponce, and Yann Lecun. Vicreg: Variance-imvarince-covvraannn regularization for self-supervised learning. International Conference on Learning Representations, 2022.
[3] Mikhail Belkin and Partha Niyogi. Laplacian cigenmaps and spectral techniques for embedding and clustering. In Neural Information Processing Systems, 2001.
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains, Machine learning, 79:151-175, 2010.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[6] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Xinlci Chen and Kaiming Hc. Exploring simple siamese representation learning. In Proceedings of the JEEE/CVF Conference Computer Vision and Puttern Recognition, pages 15750-15758, 2021.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020.
[9] Xinlei Chen, Saining Xie, and Kaiming He. empirical study training self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[10] Ching- Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vincet, Neel Joshi, Antonio Torralba. Stefanie Jegelka, and Yale Song. Robust contrastive learning against noisy views. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
June 2022.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. 10111///RR 200.2.0009004
[12] Jiankang Deng, Jia Guo, Niannan Xuc, and Stefanos Zafciriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4690-4699, 2019.
[13] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1 (3):211-218, 1936.
[14] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):096-20330, 2016.
[15] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. Proceedings the AAAI conference artificial intelligence, 31(1), 2017.
[16] can-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learnin learning. Advances neural information processing systems, 33:21271-21284, 2020.
[17] XY Han, Vardan Papyan, and David I. Donoho. Neural collapse under mse loss: Proximity to and dynamics the central path. In International Conference Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2024
[18] Jeft Z HaoChen and Tengyu Ma. A theoretical study of inductive biases in contrastive leamnin. In The Eleventh International Conference on Learning Representations, 2023.
[19] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34: 5000-5011, 2021.
[20] Jeff Z HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transferability of contrastive representations related subpopulations. arXiv preprint arXiv:2204.00 204.683.. 2022.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[22] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 630-645. Spriiner,, 2016.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9729-9738, 38, 2020.
[24] Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social networks, 5(2): 109-137, 1983.
[25] Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions pattern analysis and machine intelligence, 16(5):550- 554, 1994.
[26] Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot. Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 13:1866111 1661- 2020.
[27] Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.
[28] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86( 86(1 1):22)8))22244 1998.
[29] Jason D Lee, Qi Loi, Nikunj Saunshi, and JIACHENG ZHUO Predicting what you already know helps: Provable self supervised leamning In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Newral Information Processing Systems, volume 34, pages Associates, Inc., 2021. URL htwps: /ttp:p/e/////oooeocicceeeeg neurips...........ee... cceppaper_fileeeee ftt.ffieeepppprr//////// .ddf.
Curran
2021/file/
[30] Jason Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable seff-supervised learning. Advances in Neural Information Processing Systems, 34:309-323, 2021.
[31] Jing Lei and Alessandro Rinaldo. Consistency of spectral clustering in stochastic block models. The Annals of Statisties, pages 215 203,2377200 2015.
[32] Tsung Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pages 9980-2988. 2017.
[33] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang. Mingsheng Long. and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 455- Springer, 2020.
[34] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 212-220. 2017.
[35] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL https: http::://////peviieeeeee/ forumm/formmmm iid=
Skq3 9Scxx.
[36] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20 2022.
11

---

Published as a conference paper at ICLR 2024
[37] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. nips workshop on deep learning unsupervised feature learning, 2011.
[38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807:03744 2018.
[39] Vardan Papyan, XY Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117(40))24652-2466.3 2020.
[40] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Pruceedings of the IEEE conference on computer vision and pattern recognition, pages 1944 1952,,199,200 2017.
[41] Xingchao Peng, Qiaxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 1406-1415, 2019.
[42] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint 0rXiv:2008.08833 2020.
[43] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. theoretical analysis of contrastive unsupervised representation learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors. Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 5628 5637. PMLR, 09 15 Jun 2019. URL hts:s::tt.... : / http://oooceengn.gseee www.nnssmmrr.... mom..ppeee press /977 / saunshil 9ttmml.
[44] Kendrick Shen, Robbic M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Perey Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference on Machine Learning, pages 19878. PMLR, 2022.
[45] Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach unsupervised domain adaptation. In Proc. 6th International Conference on Learning Representations, 2018
[46] Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What do we maximize in self-supervised learning? First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022.
[47] Gilbert W Stewart. On the perturbation of pscudo-inverses, projections and lincar least squares problems. SIAM review, 19(4):634-66 662, 1977.
[48] Thomas Strohmer and Robert W Heath Jr. Grassmannian frames with applications to coding and commu- nication. Applied and computational harmonic analysis, 14(3):257-275, 2003.
[49] Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv: 1909. 1825, 2019.
[50] Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via unified gradient framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14431-14440, 2022.
[51] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding superviee learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278 PMLR, 2021.
[52] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. Journal of Machine Learning Research, 11-3,, 2021. URL. http: fijlr..rr hpappss//222 2210020100000000 008 9. httml.
[53] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 1e:11611116466 2021.
[54] Hao Wang. Yitong Wang. Zheng Zhou, Xing Ji. Dihong Gong. Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265-5274 2018
[55] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere, In International Conference on Machine Learning, pages 9929-9933 PMI.R, 2020.
12

---

Published as a conference paper at ICLR 2024
[56] Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 27, 2022, Proceedings, Part XXVI, pages 668 684. Springer, 2022.
[57] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning, pages 12310-12320. PMLR, 2021.
[58] Jingyi Zhang. Jiaxing Huang. Zichen Tian, and Shijian Lu. Spectral unsupervised domain adaptation for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9829-9840, 2022.
[59] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[60] Xiong Zhou, Xianming Liu, Junjun Jiang. Xin Gao, and Xiangyang Ji. Asymmetrie loss functions for lcarning with noisy labels. In International conference machine learning, pages 12846-12856. PMLR, 2021.
[61] Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, Xin Gao, and Xiangyang Ji. Learning towards the largest margins. International Conference on Learning Representations, 2022.
[62] Xiong Zhou, Xianming Liu, Hanzhang Wang, Deming Zhai, Jiangjunjun, and Xiangyang Ji. On the dynamics under the unhinged loss and beyond. Journal of Machine Learning Research, 24(376):1-62
2023. URL http: // jmlr www.oo//.aa...o.. http:/ppprss////eeeeeee /24 42244/23222 2 0771 11.mmll.
[63] Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, and Xiangyang Ji. Asymmetric loss functions for noise-tolerant learning: Theory and applications. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[64] Hao Zhu, Ke Sun, and Peter Koniusz. Contrastive laplacian eigenmaps. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 5682-5695. Curran Associates, Inc., 2021. URL https::/proocediing.. newriss.e......... cappccppppprffllssspppppppe 24120013500000011187222110000002
[65] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 11129820-298835 2021.
13

---

Published as a conference paper at ICLR 2024
Appendix for "Zero-Mean Regularized Sepctral Contrastive Learning"
A MORE CLARIFICATION
A.1 "WEIGHT REDUCTION IN POSITIVE PAIRS" BEYOND ZERO-MEAN REGULARIZATTON
The main idea of zero-mean regularization is to implicitly reduce the weights of wrong connections in the positive-pair graph, as stated throughout the paper. This idea fully depicted in the unsupervised domain adaptation (see Eq. 3.3) and learning with noisy labels (see Theorem 3.4) in Section 3.3.2. In this paper, we focus on the spectral contrastive loss, since it is theoretically sound and easier to analyze. The final manifestation is accordingly expressed in the concise term of the loss-specific zero-mean regularization.
We further attempt apply the core idea of "weight reduction in positive pairs" to the InfoNCE loss that is widely used many popular contrastive learning schemes, such as SimCLR [6]. MoCo [23], and CPC [38]. Specifically, modify the InfoNCE loss by uniformly reducing the importance of positive pair as follows:
(1 T)
log
(1 r)etim((,2t) + Ei=1 essm(ax27)
cstm(orat)-r
log esim(z,xt) + Ei= e eaim(ar27))
(A.1)
log
geim(zzxx)
Σ e )+r)
where sim(21, denotes the similarity between Di and Ø2, -'= 1-T > 0 and r E [0, 1). As can seen, the derived form that adds margin term similar to margin-based losses, particu- larly the negative-maun margin softmax loss [33]. From this point of view, weight reduction in positive pairs coincides with the motivation of margin-basee losses [12, 33, 34, 54, 61] that enlarges the discriminativeness with intuitive decision boundaries.
To empirically validate the efficacy of the modified term InfoNCE loss in Equation A.1, we conduct experiments self- self-supervised learning and supervised learning in Table 4. In these experiments, we utilize CIFAR-10/-10 and SVHN datasets as benchmarks. We believe that these additional experiments only provide further evidence of the versatility embedded within the concept of "weight reduction in wrong positive pairs" but also highlight its potential to enhance the performance of contrastive learning algorithms on real-world datasets.
Table 4: Top-1 liner probing (%) and Top-1 validation accuracy (%) of self-supervised learning and supervised learning with the modified InfoNCE in Equation A.1, respectively. The results of the InfoNCE loss with the margin > 0 obtaining better performance than the original spectral contrastive loss r = 0 are highlighted. The best results are underlined.
Self-Supervised Learning
Supervised Learning
Method
CIFAR-10 CIFAR-100
SVHN
CIFAR-10
CIFAR-100
SVHN
InfoNCE
86.56
55.51
90.77
94.18
71.43
96.11
0.05
86.64
55.68
91.05
94.04
71.80
96.24
r 0.1
86.97
56.07
90.93
94.27
71.56
96.17
0.2
86.65
55.55
90.77
94.19
72.07
96.20
r 0.5
87.32
56.01
90.95
94.17
72.44
96.18
0.9
86.48
55.88
90.95
94.43
72.29
96.16
A.2 DOWNSTREAM ERROR BOUND FOR CONTRASTIVE LEARNING
There are several works focusing on the SpeCL [18, 19. 20]. which provide provable accuracy guarantees under linear probe evaluation in context of an implicit m-way partition of X, Lee, they
14

---

Published as a conference paper at ICLR 2024
disjoint non-empty sub-graphs of X' such that X' = (vx X, let idg be the index such that I E Sia,). Specifically, HaoChen et al. [19] proved the efficacy of SpeCL under the assumption that the representation dimension d exceeds the number of sub sub-graphs Ma [18] extended these results by studying representations learned within a constrained model class F while imposing the condition m > as follows:
Subsequently, HaoChen and
In this work, prove downstream error bound when
Theorem A.1. Let F be class of functions of X - Rd and let f arg min L(fiT) be the minimizer of the SpeCL. Assume that:
1. (e-separability). The probability of a positive pair belonging to two different sets is less than 6, that is, P (id + idgt) + 1
2. (Alignment) The downstream label y(z) is a constant on each S; for i [m]]
3. (F-implementable inner-cluster connection larger than 8). For any f E F and any linear head w E Rdd let function g(z) www f(x). ==ff(x) For any E [m], we have Qs (g) =
(((())))(=*))²
(a.at)-pph (((=))=((')))) 6;
4. (Implementability). There exists a function f f F such that f(x) = vid, for all x Pdata- where {U1, 22...., ...., Um is a set of different vectors that achieve the global minimum of E P.U.U III 27. "==1Ppu and p: Pizmpane (x E S,).
Let Puin [m] Przmpaann (xr E S.) and A = maxi,j ||e u2l12 For m > d, if cA < 1, then there exists linear head W E Rddd which achieves the following downstream error
[[w](s) - reccol1 -
eA(1 VeA)
Exmpaate
(A.2)
28(p
Remark. In Theorem 1. we assume that the model class F is implementable and capable of expressing some vectors V = [Vl...., Um] that achieve the global minimum of || >it Peuce || 27 E peu; 2 (see Lemma A.3 in the appendix for the specific form of V). In [18]. these vectors are concretely specified as one-hot vectors, which actually corresponds to the case of f 0. Otherwise, our error bound under linear probing is characterized Exwppac.. TT((x) +
arg min || f(x) - ve 1²)) and can be further bounded by the error Ex-pptca [[] f(x) [z== ey Moreover, the right-hand
hand side in Equation A.2 can be further tightened once we choose vectors that minimize A = y ||u;- 11. Therefore, our main result in Theorem A. provides a general
bound for spectral contrastive learning when > which extends the prior works and completes the picture of spectral contrastive learning.
Importantly, Theorem A. also reveals a trade-off between inter-partition separability and intra- partition compactness. Intuitively, good inter-partition separability involves maximizing the distance between any two distinct partitions, i.e., maxy miniz; ||u: "y while the notion of intra-partition compactness anticipates the reduction in À maxi. ||0 vy contributing to more stringent bound on the right-hand side of Eq. 3.3. The intricate interdependence between maxy min;yj ||u; vz and miny max4y1 [|e; -u,--1l1 v, elucidates the nuanced nature the trade-off in the understanding of self- supervupeee learning.
A.3 COMPARISON TO CONTRASTIVE LAPLACIAN EIGENMAPS
While there are similaritics between Zero-ment regularized spectral loss (Lero-SpeCL) in Eq. A.2 and contrastive Laplacian eigenmaps (COLES) [64], there are distinct differences that crucial to highlight:
Different Negative Components. The contrastive objective (to be maximized) can generally formulated as the objective J(f) = B(e,xt)<pp,. (xf(), f((=)) n JoPaste (x) f() f)) While the positive components of Proo-SpeLL& and COLES are the same, their negative components differ. The negative component of COLES is f(z))) = f(()) f(x ), while Zero-SpeCL is f(x = -(f()) f(x))
15

---

Published as a conference paper at ICLR 2024
Different Reasons for Covariance Term. Both Zero-SpeCl and COLES introduce the temn [[E=)(=)f(==== - -. The motivation of COLES is to softly satisfy the constraint FTF== that removes an arbitrary scaling factor in the embeddings [3]. In addition, also helps avoid collapsed solutions, since without the constraint / F I, the derived graph Dirichlet energy -T((F AWF) will be minimized when all representations collapse to constant vector. In contrast, the covariance term R1(f) [=.,(())((= I in
Zero-SpeCL appears as a part of an equivalent form in Eq. 3.2.
Different Overall Objective. COLES represents a constrained graph Dirichlet energy mini- mization problem (contrastive Laplacian cigenmaps), i.c., minpTF=1 -Tr(F" AWF). On the other hand, Zero-SpeCI. depiets the low-rank matrix approximation problem ming A - F Fl? (as shown in Eq. 3.3).
A.4 THE ROLE OF
In this paper, we propose setting E [] to balance the regularization strength of zero-mean regularization. Although in Section 3, we do not explicitly constrain 1, theoretical and empirical evidence supports the necessity of such constraint:
In Theorem 3.1, we assume that T < (n(0)- X=(0) < 1 to ensure that the introduction of (1
7 will not alter the value of the d-th eigenvalue in Eq. (C.23). In Proposition 3.2, assume that (0) (2(0) (0)0))x,(0) < 1 to guarantee that the first eigenvalue of A1(0)
À' is (T (1 - (0) (as can be seen, T S essential to avoid the presence of negative eigenvalues of A') Eq. (C.32), thus facilitating a more intuitive comparison A=(0)=Aa(0) (:(0) 1], it requires a more nuanced comparison of the order of these eigenvalues, predictably resulting in an obviously different bound than at 5 0.
with the bound at = 0. While it is possible to draw conclusions within the range of
In Theorem 3.3, we prove that the global minimum of the supervised spectral contrastive loss - HT H When r S d 1, (r - --11-117
is achieved at H = arg min [ H H 1 has minimum of zero only for E [0, 1], corresponding 1 f rl - -11T, where the angle between representations of different classes is arccos
_I_ > 90% However, --y
for r 1, there exists no É that satisfies A T á = rI - T11 as it would lead to paradoxical situation where < 1 A H1 = 1 (rI -11 72
0.
We conducted experiments for r > 1 as presented in Tables 9 and 11 of the Appendix. As observed, consistently better results are obtained with т E [o, 1]. while excessively large values hinder overall performance. Thus, suggest limiting 7 to the range of [0, to balance the regularization and discriminative capabilities.
B MORE EXPERIMENTAL DETAILS AND RESULTS
In this section, present the training details and experimental results several tasks, including self-supervised contrastive learning, supervised contrastive learning, unsupervised domain adaptation, and learning with noisy labels.
B. DATASETS
Small Datasets. In our experiments, we consider several commonly-used datasets, namely CIFAR- 10/-100 [27], SVHN (S) [37], MNIST (M) [28], USPS (U) [25], and MNIST-M (M-M) [14]. For the tasks of self-supervised contrastive learning and supervised classification, we utilize CIF-R-100-100 and SVHN datasets. These datasets are also employed for the evaluation of learning with noisy labels. To investigate unsupervised domain adaptation, consider four domains of digit datasets: SVHN, MNIST. USPS, and MNIST-M. For small datasets, each image is resized to a standardized size of 32 32 with 3 color channels to ensure uniformity in input sizes.
16

---

Published as a conference paper at ICLR 2024
Table 5: Top-1 linear probing of self-supervised learning with the spectral contrastive loss in Equa- tion B.1 under various values of and т. Notably, the results obtained with the spectral contrastive and outperforming the baseline (the original 0) are highlighted. The best results are underlined for clarity.
loss incorporating zero mean regularization
spectral contrastive loss
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
CIFAR-100
SVHN
Method
μ4 1.0
μ 5.0
Vanilla SpeCL
82.54
40,45
84.80
86.17
51.75
88.79
0.1
82.97
42.51
86.35
86.80
52.87
89.36
0.2
83.11
42.98
85.65
86.78
53.13
90.19
0.5
83.22
43.36
86.23
87.23
53.89
91.38
7 1.0
83.78
44.21
86.84
87.46
54.98
91.58
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
CIFAR-100
SVHN
Method
10.0
20.0
Vanilla SpeCL
86.25
52.44
89.58
85.64
51.64
88.82
0.1
86.77
53.98
90.36
85.33
54.26
91.03
0.2
86.93
55.25
90.80
85.62
55.23
91.43
0.5
87.05
56.61
91.10
86.47
56.17
92.01
87.24
56.77
91.12
85.87
55.80
92.13
ImageNet. We also include subset of the ImageNet dataset called ImageNet-1000 100. ImageNet ILSVRC2012 [11] consists of approximately 1.2 million training images and 50,000 validation images, divided into 1,000 classes. However, due to the limited computing resources and time, focus specifically on the first 100 classes of ImageNet, forming the ImageNet- 100 dataset. Fach image in this subset is resized to dimensions of 64 x 64 with 3 color channels.
DomainNet. For unsupervised domain adaptation, we employ the DomainNet dataset [41]. Do- mainNet a large-scale dataset designed specifically for unsupervised domain adaptation tasks. contains approximately 600,000 images distributed among 345 classes across six domains. These six domains included in DomainNet are clipart, painting, real, sketch, infograph, and quickdraw. In our experiments, focus four domains from the cleaned version of DomainNet, including clipart, painting, real, and sketch. Each image in these domains resized to dimensions of 64 x 64 with color channels ensure consistency input sizes.
B.2 TRAINING DETAILS.
Data Augmentation. We adopt the same image augmentation settings as used in the Sim- Siam method [7]. These augmentation techniques include RandomResizedCrop. ColorJitter, Ran- domGrayscale, GaussianBlur, and RandomHorizonallliiy
Network Architectures. For smaller datasets such as CIFAR- 10/-100 and digit datasets (SVHN, MNIST, USPS, MNIST-M), employ the PreAct-ResNet-I1 architecture [22] the backbone network. The projection layers consist a 2-layer MLP with a hidden dimension of 2048 and an output dimension of 1024. For larger-scale datasets such ImageNet and DomainNet, we utilize the ResNet-50 architecture [21] as the backbone network. The projection layers in this case also consist of 2-layer MIP,, with both the hidden dimension and the output dimension set 4096.
Optimization Details. In this paper, we train these self-supervised (or supervised) models for total of 200 epochs. The training process utilizes the SGD optimizer with a momentum of 0.9, learning rate of 0, and weight decay of 5e-4. To dynamically adjust the learning rate throughout the 200 epochs, we use the cosine decay learning rate schedule [35]. For datasets with smaller sizes, set the batch size to 512 I GPU, while for ImageNet- 100 100 and DomainNet, the batch size set to 1536 on 8 GPUs. All experiments were implemented using Py Torch and executed on NVIDIA GTX 2080Ti GPUs.
17

---

Published as a conference paper at ICLR 2024
Table 6: Top-1 validation accuracy of supervised learning with the spectral contrastive loss in
Equation 3.8 under various values of μ and т. Notably, the results obtained with the spectral contrastive loss incorporating zero mean regularization T > 0 and outperforming the baseline (the original spectral contrastive loss
0) are highlighted. The best results are underlined for clarity.
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
CIFAR-100
SVHN
Method
μ4 1.0
μ 5.0
Vanilla SpeCL
87.01
33.18
90.04
93.52
56.13
95.91
0.1
87.69
35.22
95.89
94.28
57.90
95,93
0.2
88.72
39.82
96.04
94.31
57.73
96.06
0.5
89.45
42.38
96.11
94.50
59.26
96.09
7 1.0
89.74
41.69
95.97
94.33
59.56
96.07
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
R-FAR-100
SVHN
Method
10.0
20.0
Vanilla SpeCL
94.05
65.68
95.89
94.14
70.70
96.12
т 0.1
94.39
68.11
96.21
94.28
70.50
96.16
0.2
94.59
68.30
96.23
94.30
71.03
96.25
0.5
94.47
68.74
96.14
94.64
71.04
96.22
1.0
94.43
69.74
96.31
94.06
71.94
95.92
Implementation of Spectral Contrastive Loss. The pseudocode for the spectral contrastive loss, as well as its supervised version in PyTorch-style. is provided Table 7. It is worth noting that, similar to other methods such as SimSiam [7], BYOL [16]. SimCLR [6]. and others, we perform fz-normalization on the final output to compute the loss. Additionally, followed HaoChen al. [19], we introduce an extra lyper-parameter μa (the default value of is 10) to balance the positive and negative components of the loss.
B.3 EXPERIMENTS CONTRASTIVE LEARNING AND SUPERVISED LEARNING
The empirical objective of spectral contrastive loss with zero-mean regularization used in this paper formulated as
a
Lasl (J;T)
E A(z((z)) atmA((;)
[f(x) (
a>a(()) [(([(=) f(x 7) ~A(s,)
(B.1)
where r; denotes the i-th example in the input space, and A(x) denotes the augmentations of example F.
The supervised version of spectral contrastive loss with the zero-mean regularization for the labeled dataset (X, Y)
[sup([:r] = Loup() (f) +
poo (J)
a>a x A(2)())
[(x))]()+)]] and
r
(B.2)
Eneg(f:7)
E anAlaa))
f(x)))
n2
A(agaa)
where Ti,e denotes the i-th example in the class n is the number of examples in each class, and A(xr) denotes the augmentations of example a..
Experimental Details. We conduct experiments self-supervised and supervised spectral con- trastive learning using ReeActt- 18 models pre-trained on CIFAR-10, 10, CIFAR-100, SVHN, and subset of ImageNet comprising the first 100 classes. The training process involves training the networks for 200 epochs. To evaluate the performance of these models in linear classification, employ independent linear classifier on the fixed representations obtained during contrastive
18

---

Published as a conference paper at ICLR 2024
Table 7: PyTorch-like pseudocode of spectral contrastive loss and its supervised version.
Spectral Contractive Loss
class Scceraaloonrasiiiioos (nn. (un.Medlle)>
def iiini _nni__ttse_____ (self, mu=1.0, tau=0.01, 0) su inii_iiii in self.tar - tav self.mu
def (srward zi, z2, labels-None):
If labels is not None, the loss becomes supervised, assert zl. shashape == z2.shape and len shape) ==2 device zevicve
zh.shape [0]
2 = torch. cat [21, 22), dim=0)
F.normalize(z, dim-l)
logits morch.matmul (z, Z.T)
mask-out negative pairs
if labels is None:
mask torch.eye (N, dtype-torch. float32, device-devi repeat (2, 2)
else:
mask = torch.eg (labels, labels.T) .float () .repeat (2, 2)
lcss lcss_partl partl -2 (logits mask) sum mask. sum ()
lcss_part2 self.m (llgits self.tau) mean ()
return loss_part1 loss_part2
Table 8: Top-1 accuracy under linear probe evaluation (%) on maageNet-100. The results of the spectral contrastive loss with zero mean regularization 7 >> 0 with better performance than baseline (the original spectral contrastive loss т 0) are highlighted. The best results are underlined.
Self-Supervised Learning
Method
ImageNet-100
DomainNet
Vanilla SpeCL
40.82 69.32
34.266 / 57.794
= 0.1
40.50 70.38
35.912 59.314
0.2
41.86 71.58
37.037 60.632
== 0.5
42.90 / 72.48
37.844/ 62.039
1.0
38.26 68.10
35.657 60.202
pre-training. This classifier is trained using labeled data. The reported results correspond to the top-1 accuracy achieved by these models.
Experimental Results. Table 5 and Table 6 present the Top-1 accuracy results for self-supervised learned models and supervised learned models, respectively. These results clearly demonstrates that zero-mean regularization consistently outperforms the Baseline (models without zero-mean regularization) across most scenarios. Notably, the best performance is consistently achieved when utilizing zero-mean regularization in all cases. Similar conclusions are also reflected in Table 8 which offers the results of self supervised learning on ImageNet and DomainNet. Moreover, Table 9 reports the results obtained using fixed value of μ 10 and different values of which shows that excessively large values of 7 can hinder the improvement achieved by zero-mean regularization. Therefore, suggest limiting the range of [0. 1].
19

---

Published as a conference paper at ICLR 2024
Table 9: Top-I test accuracy (mean +tt., %) of spectral contrastive learning with fixed value of 10 and various values of r on benchmark datasets with symmetric label noise. Notably, the results obtained with the spectral contrastive loss incorporating zero mean regularization 7 > 0 and
outperforming the baseline (the original spectral contrastive loss 7 = 0) are highlighted. The best results are underlined for clarity.
Self-Supervised Learning
Supervised Learning
Method
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
CIFAR-100
SVHN
Vanilla SpeCL
86.25
52.44
89.58
94.05
65.68
95.89
0.1
86.77
53.98
90.36
94.39
68.11
96.21
0.2
86.93
55.25
90.80
94.59
68.30
96.23
87.05
56.61
91.10
94.47
68.74
96.14
1.0
87.24
56.77
91.12
94.43
69.74
96.31
T 5.0
85,80
54.82
91.04
94.11
68.99
96.07
10.0
82.44
50.02
90.25
93.17
66.21
95.82
T 20.0
73.81
35.96
86.75
89.18
51.98
95.67
B.4 EXPERIMENTS UNSUPERVISED DOMAIN ADAPTATION
We conduct empirical validation of unsupervised domain adaptation using spectral contrastive pretraining on four digit datasets: SVHN (S) [37], MNIST (M) [28]. USPS (U) [25], and MNIST-M (M-M) [14]. To perform the pretraining, we utilize the spectral contrastive loss with zero-mean regularization on the PreAct-ResNetl architecture. The models are pretrained over all the domains for 200 epochs. Subsequently,
train these pretrained models using a linear classifier on the source domains for 10 epochs. The learning rate is set to 0.005, and the weight decay set to 1e-5. The results of these experiments reported in Table 2.
For the experiments on the DomainNet dataset, we follow a similar approach. We pretrain the four domains of DomainNet for 200 epochs. Then, train linear
ResNet-50 architecture classifier on the source domains for 100 epochs with base size 2048. The learning rate is set to 0.1, and the weight decay to le-5. The results of these experiments are provided Table 10.
Table 10: Top-1 accuracy under linear probe evaluation (%) on all individual domain pairs of DomainNet. The results of the spectral contrastive loss with zero mean regularization >>>>>>>> 0 with better performance than baseline (the original spectral contrastive loss 0) are highlighted. The best results are underlined. ERM denotes empirical risk minimization over source domains.
Source
Clipart
Painting
Real
Sketch
Avg
Target
Painting
Real
Sketch Clipart Real Sketch Clipart Painting Sketch Clipart Painting Real
ERM
5.75
13.56
11.45
13.35
17.33
10.95
27.14
17.74
14.61
18.58
6.96
11.03
14.04
Baseline
14.22
21.55
15.97
14.85
25.62
13.92
21.46
22.74
15.03
20.58
14.81
19.15
18.33
t 0.1
14.79
22.69
16.44
15.72
27.05
14.20
22.23
23.04
15.22
21.30
16.26
21.03
19.16
7 = 0.2
14.57
23.10
16.72
15.90
27.69
14.22
22.43
23.41
15.29
21.45
16.62
21.29
19.39
0.5
14.16
22.37
16.49
15.32
26.99
13.44
22.67
22.92
14.72
20.34
15.84
20.62
18.82
7 1.0
11.33
19.20
13.82
12.54
23.21
11.33
19.87
19.71
12.12
16.88
13.30
16.51
15.81
Experimental Results. As depicted in Table 10, the results demonstrate that unsupervised domain adaptation (UDA) utilizing spectral contrastive pretraining yields superior performance compared to the standard empirical risk minimization (ERM) approach. This indicates that spectral contrastive pretraining effectively decouples class information across different domains, leading to improved adaptation. Furthermore, the results indicate that utilizing zero-mean regularization with different values of (0.1. 0.2, and 0.5) outperforms the baselines. This suggests that zero-man cero-mean regularization enables the learning of more discriminative representations, resulting in enhanced performance the unsupervised domain adaptation task.
20

---

Published as a conference paper at ICLR 2024
Table 11: Top-1 validation accuracy (mean +td, %) of supervised spectral contrastive loss with zero-mean regularization in Equation 3.8 on benchmark datasets with symmetric label noise. The results of the spectral contrastive loss with zero mean regularization T 0 with better performance than the original spectral contrastive loss 0 are highlighted. The best results are underlined.
Symmetric Noise Rate
Datasets
Methods
0.2
0.4
0.6
0.8
CE Focal
82.88 + 0.04
68.99 + 0.15
52.27 + 0.20
51.62 + 0.08
60.91 = 0.07
46.73 + 0.13
29.47 = 0.04
13.54 + 0.03
GCE
91.35 0.04
89.14 0.01
80.50 0.07
52.25 0.03
Vanilla SpeCL
90.80 = 0.01
88.29 = 0.13
85.87 = 0.07
75.51 + 0.27
0.1
91.18 = 0.07
88.99 = 0.07
86.66 = 0.04
80.29 + 0.39
CIFAR-10
T == 0.2
91.57 = 0.03
89.13 0.12
86.81 = 0.03
80.06 + 0.39
0.5
91.31 + 0.06
89.49 0.02
86.59 + 0.09
78.51 + 0.25
T -= 1.0
91.42 + 0.07
89.26 + 0.05
86.98 + 0.07
76.63 + 0.30
5.0
90.26 0.02
87.23 0.10
84.36 0.02
76.32 0.07
T = 10.0
84.51 + 0.05
80.52 + 0.13
77.43 + 0.20
68.99 + 0.55
CE
60.94 = 0.06
46.07 = 0.01
30.33 - 0.02
14.41 + 0.06
Focal
60.91 0.07
46.73 0.13
29.47 0.04
13.54 + 0.03
GCE
68.91 + 0.03
64.87 + 0.07
56.04 + 0.04
9.13 + 0.00
Vanilla SpeCL
67.87 + 0.05
62.97 + 0.11
57.51 0.02
51.10 E 0.08
T == 0.1
68.63 = 0.09
64.46 = 0.04
59.41 - 0.06
52.32 = 0.09
CIFAR-100
0.2
69.60 = 0.07
65.04 0.07
59.46 = 0.04
53.90 = 0.06
T T= 0.5
69.57 + 0.06
65.05 + 0.06
59.17 + 0.09
52.80 + 0.08
1.0
69.50 + 0.15
65.37 0.07
59.41 0.01
52.60 0.07
T == 5.0
66.56 = 0.03
60.78 - 0.07
53.96 - 0.09
47.28 + 0.06
T 10.0
56.01 0.08
48.72 0.11
43.96 0.09
38.09 0.12
B.5 EXPERIMENTS ON LEARNING WITH NOISY LABELS
For the experiments on learning with noisy labels, we propose to employ a mixed spectral contrastive loss mitigate label noise, ie., using L + 0.54 The self-supervised part only aligns positive pairs (two random augmentations from the same sample), while the supervised part utilizes positive pairs according noisy labels.
Noise generation. The noisy labels are generated following standard approaches in works [40, 60]. For symmetric label noise, we corrupt the training labels by flipping labels in each class randomly to incorrect labels to incorrect labels in other classes with probability 7 {0.2, 0.4, 0.6, 0.83.
Experimental Results. Table presents the Top-1 validation accuracy in percentage (mean of the last five epochs) for various levels of symmetric label noise. As can be seen, the presence of T S consistently improves upon the baseline (7 0) across all scenarios.
B.6 EXPERIMENTS ON "CLASS-MEAN FEATURES CLASSIFIER"
To demonstrate the feasibility of employing class-mean features as the classifier following pre-training the representation model f with regulaegglr spectral contrastive losses the self-supervised or supervised scenarios, we derive the class means (he directly from the average features of each class in training set. We subsequently evaluate its performance using the class prediction rule arg max. h f(x) for each test sample z. As observed in Table 12, method, termed as "class mean Features as classifiers", achieves comparable performance to the conventional linear probing method which trains a linear classifier CE over tens of epochs.
21

---

Published as a conference paper at ICLR 2024
Table 12: Top-1 liner probing (%) and Top-1 validation accuracy (%) of class-mean classifier on self-supervised learning and supervised learning with zero-regularized spectral contrastive loss, respectively. The results with obtaining better performance than the original spectral contrastive loss 7 0 are highlighted. The best results are underlined.
Self-Supervised Learning
Supervised Learning
Method
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
CIFAR-100
SVHN
SpeCL
84.45
46.78
88.43
92.71
57.01
96.00
т 0.1
85.17
48.14
89.99
94.07
63.55
96.26
0.2
84.90
49.54
89.81
94.38
62.23
96.30
0.5
85.20
49.71
89.67
94.31
63.71
96.21
r 1.0
85.06
49.62
89.98
94.23
63.81
96.09
C PROOF OF THEOREMS, LEMMAS AND PROPOSITIONS
C.I PROOF OF THEOREM A.1
Theorem A.1 Let F be a class of functions of X -+ Rdd and let arg minner L(f;T) be the minimizer of the spectral contrastive loss. Assume that:
1. separabbbl.. The probability of a positive pair belonging to two different sets less than G, that is, (r JrPpas (id2 # idg.) I
2. (Alignment) The downstream label y(z) is a constant on each for i [m]]:
3. (F-immlementable inner-cluster connection larger than 5). For any f E F and any linear
head w E Rd,, let function g(z) = w f(x). For any E [m]) we have Qs (g) =
(aptt)ppoo 8, [((()))
(1(())-((''))) 6;
Caca a BacK
4. (Implementabiliry). There exists a function f f F such that f(x) Vid, for all x Paata where {U1, 222 ..... Um is set of different vectors that achieve the global minimum of E P.u; 112, and p. Przapana (x E S.).
E P,U.U
I 27
Let Pmii min Przmpaano (xr S,) and A max,, ||0 - 1 For m d, if eD <, 1, then there exists linear head W E Rddd which achieves the following downstream error
[|ws(0) - 11c -
eA(1+ VeA)
Ex-pasta
(C.1)
20(pmin
Proof. Let f be the function f()x) = vid. According to Lemma C.2, m > d, the minimum of I| Di Aaais Il 27- 11 pad; is O which is obtained when >m p. and E P. 0, so we know that R2 (f* R2( 0. For the invariance term with respect f*,
we have
Ro(s") Ey [11*())) (x S max ||e u3l12
Pr (x.at) (s,at)~Ppor
(id + id S cA. (C.2)
Define matrix M Expeiaa [f(z) since f = arg sagminee sEF L(f:T) the minimizer of contrastive loss within the functional class, we have Ro(f) Ri(f) 2r Ro(f"), and further obtain
|M Ili R1( < Ro(f") 5 < < 1. Thus, M is a full rank matrix, and we can define function f(x) = M 1/2 and have that
(C.3)
Banpan..l((a)fflle [f(x)f(=) Exrpurra [M 1/2 f() fx)) M -- =I
(C.4)
Let Q Ex-ppan [f(x)f(()) (f) and T, (x) = f(x)) - Qf (x). Since R1 (f()) 0. we know that
(T= Ege-pann [f)) QT (f)) 0. (C.5)
22

---

Published as a conference paper at ICLR 2024
Using the first and third assumptions in Theorem 3.1, we have
[1) TT ||(
E (r,art ymppps
(p: c) B (2 )M [1T5((t) xx((x))12
iE mm]
26.
(ps - c) IBr,arppuaat,
[1TT((T) (2()))
iE/m]
(C.6)
=6- 6. Σ
1
Pr Eg Ee
mptaoaa [|T,(()) - xx(22)
Di
=E/m]
20.
Σ Pi
[1 =)(x')1
Piniu
iE(m]
(1
=26-
Ex Exrpaaaa (x))1
Pmiin
On the other hand, we have
E (n PoPpos
[1((()) -f(xt)||
< E3 co Jmppos
(C.7)
M Ilspec B( [l f(xx))
<( + veA) cA.
Combining Equation C.6 and C.7 we have
eA(1 + VEA
Ex-ppat
(1)))()))1
(C.8)
28 (pimin - c)
By Lemma C.I, we know that there exists a matrix U E Rdxd such that
A11 + VeAyunnn
Earopsaaaa (UM 1(2))
(C.9)
20 (Prin c)
Thus, we define matrix W UM-1/2 then we have
[loas - W f(x)1
eA + VeA Pmin
E Egeeeaca
(C.10)
26(pmin E)
which finishes the proof when d < m.
Lemma C.1. Suppose X Hd and g X - IRd are two functions defined on X such that (f
(9(=)))))((=))(())) =I
(C.11)
Define the projection of f onto g's orthogonal subspace
nt (x) f(z) -a (f(2))(('))
- lg(())
(C.12)
Then, there exist matrix U E Rdxd such that
E Ep [1]g(=) U ff()) En (x)
(C.13)
Proof. Please see the proof in HaoChen and Ma [18, Lemma B.1].
Lemma C.2. For any given weights P1. P1..... P.. > satisfying Σm Ps = 1 and t E [0, 1), the global minimum of Em praco - Lall + 2r En p.vsll 11 (where d is the feature dimensionality) is
(a) O when d, which is obtained if and only V V(V; 1m) [T::
23

---

Published as a conference paper at ICLR 2024
(b) d m + 2T 7² when I d, which is obtained if and only if VVV V =p1 -11mmm 1
where V [Ui,..., Um] and P = diag (P1, (21,P2.... ..... Pm.)
Proof. Obviously, we have 1 17 + 2T D p.ovllé 2 O. where the equality holds if and only if Σ> Ae la and T - pr O. However, impossible to achieve this condition since there are fewer unknowns than equations. Thus, we will analyze the global minimizer while considering the relation between m and d.
d, is almost
If m > d, there exist different vectors that satisfy Ei=IPeeoy Piate la and Ei= P.d = 0, that is, VPV =1m== = [Ta 0).
m $ d. we have
Σ
T
-
La
+ 2T
Σ Piu;
sE/m]
sE/ml
Lt )
Tr
Σ Aiai Le
+ 2 Σ Aid
/E/ml
1E/m]
/E/m²
= Tr
Σ
Σ
P:PjUs0 150 2 Σ
Pruse +Ia
+ 2r
Σ
PePp oj
(E/m] [m]
iE/ml
ml JE[m]
Σ
Σ
(u, (0,TT 0y) Σ
P.u, 04 + d 2T
Σ
Σ
pupjyj by
(C.14)
Emm] jE(m]
rE(m]
(E(m) JE(m]
Σ
Σ
(up 21 - -- -2
Σ
asd Us
sE/m] jE[m]
{E[m]
=d + Σ
(up +T} -72 + Σ
(11::11 + T)² - Σ p:llus112
it
EE {E|m]
{E[m]
=d 72
(op 1/ +))
(p:llu:11 112 + Tp: 1) 2 2r
iドj
kE(m)
=d - m + 2T 72 +
Pips (u, By + + + (p.llua113 + TP: 1)²
mm/m]
Thus, the global minimum of E P.u.U - - / + 2r Σ p.u is d 27 72, which is obtained at
1 T v= --, Vi [m] and i F pillus + Tp: 1 Vi E [m]]
(C.15)
that is, VV = p-1 -l 1 and further
Σ [e(m) Pracee Tall: d - m. m 72, and E (e(m) Prez
=1 2
(C.16)
and this equation requires that T < inf.ERm ET 2T1m IT
2 TPs =1
C.2 PROOF OF THEOREM 3.1 AND PROPOSITION 3.2
Theorem 3.1 Let Ç > 0 and (0, 1/) be arbitrary constants. In the above stochastic block model, assume p max(a, 3), 7 minfa, 3} and -x(0))x(0)) Then, there exists & E e; 1}. (a->)))
A1
such that for any Ω 8, and regularization strength 17 E (0, probability - n 55, we have
with a high 2rp
(7
predy - predd/ 50 ((a(T))) - )et1(7))
poly(r,pp)
(C.17)
24

---

Published as a conference paper at ICLR 2024
where pred, E[pred7] is the expectation of the prediction matrix when achieving the minimun of the loss in Eq. 7 with
[0, 1), (r). >a(T) and )d+1(T) are the 1-st, d-th and d 1-th eigenvalues
of A A A
11 = N
respectively.
Furthermore, the target enorr can be bounded by
( (A1(T))2
P (p-e(died(x)) # yy) So
(A((aa()) - Ady (T))2 poly(r,p),
(C.18)
where poly(r,7) p) denotes a polynomial function of and p.
Proof. Let Ad E R N be the rank-d approximation of the adjacency matrix A, which contains the top-d components of A's SVD decomposition. We use Ad.(T,s) to denote the matrix by restricting Ag the rows corresponding to the source and the columns corresponding to the target. We use As,(S,S) to denote the matrix by restricting Aa to the rows and columns corresponding to the source.
Let Y E RNx be the label matrix where the r-th row it contains the target of the label Uz. Let Ys RlSixr and T7 E |Rl be the matrices by restricting Y to the source and target domains, respectively.
We can rewrite the spectral contrastive loss as
Azr'
1 [(=))((+)+))
[(x) [xx) ((')
Σ
No [(x)
E
22
(C.19)
VN .F) 7
(A-
T|El
111) (v P)
(
+ const.
E
Let A' A
r|
11
À
A= E[A],
1 ĀlN be the expectation of number of edges in the A- - 11 .nd and )x(r) be the i-th eigenvalue of Á'. By Lemma C.3,
E
N2
graph A, À' = -[A]]
E
we have that the prediction on the target domain is
E N2
1) Ys.
prody AAA,T..)) (Aa(s.s)
n 1--151-11
(C.20)
We define the ideal prediction as
n.. 11-181-1 1-1--11 1) 1
predy = N'A(T,S) Ala(s.s) +
E N2
Ys-
(C.21)
We will bound the difference between predy and predy. : For every class [r], define the following error vector
E oe A Ala.T.ss) (ALLSs.s) + N n- 1-1 Yg. AAC(TIs) (Aliss.s)
E N2 17 181 1) yg.
where Yg is the e-th column of Ys-
Using the perturbation bound for the pseudoinverse matrix [47], have
||b^||| < <114' - A'a,T,ssll ( (Abiss.s)
E 17 1S| (51-11 1) ' |YS/|
+
N2
(Aiisses)
E
17-18--1 181-1) - (ALis.s) + NZ 1S) 1) :)'{ ||g|| -IXII
E
+ || ||
+ Nº
n7 |S|
n22
1 V5
n2
2
||A'all
[|YS|| 1|44 - A'al
E |S]
17 E
[S]
(C.22)
25

---

Published as a conference paper at ICLR 2024
where the first inequality is induced by the triangle inequality?, the second inequality in Eq. C.22 is based the facts that |A,, |I || ||A,,CT,s) ST)l I ||As Agll ((AA),s.s)
r)tl
Na |||||| and the upper bound for the difference between two pseudoinverse matrix
N
[B - A S 1+y5
max [|
[BBI) B - | [47, Theorem 3.3].
2
By Lemma C.6, there exists constant C' C()) such that || - -'lllll s CVN with probability at least 1 - N 2C for any > ((1/p)) From now on, we assume this high probability event happens.
Follow some notations in the proof of Lemma C.4, we have Aa(F) = min( )4} and (4+1(T) = Ag. which is based on the facts that the eigenvalue corresponding to the eigenvector I, & I, 8 I of A' satisfies
np + n(r 1)8 + n(p 1)a + n(p 1)(-- + = (1 - ()) Xa(O), (C.23)
when we have (0) np n(r 1)3 + n(p 1)ax + n(p 1)(r 1)7, |g) ANANN
(B ?)? + (ar - 7)p=>n? (p ( + = (Â1 and T < (1
(0)-AA(())
Let A A La(r)-Sa-n(e
min(r(8 - 7), - - If our choice of N further satisfies N 2 2rmC we have ||.'' - -''ll s (Aa(()) - -)4+1(T)), so from Lemma C.5, we have
114' - SO ( (r(
(T) (d(T) (d(()) - )a-1(7) 11A'- - A'll so {d(7)
(
.VN) (C.24)
- (441(7)
By Hoeffding's inequality, with probability at least - 22 we assume | - 18 S N. From now on, we assume this pogb-brobability event happens. The total failure probability so far is
N-2 + 222 N- By the definition of graph, we have | 2 pN² rp
If our choice of N further
satisfies N 2re we have [B] >
eN²
thus
p
2rp
E 1
ON
(C.25)
N
2rp²"
Substituting Equation C.24 and Equation
C.25 into Equation C.22 gives:
(2(((TT) ( (T
|| so
(4+I(F))VN poly(r,p) 1V51-
(C.26)
Summing over all classes e and noticing that || S VN leads to
X1(7)
N 2
50 (T((T(T) )(+1(7))) ppoly(:p))
(C.27)
Moreover, let g =
N2 17
[S] and
{ =
N²
[
(S), we have
d
p
18 El
E EE|
S poly(r,p).
(C.28)
A + p
At
A
By Lemma C.4, we have
A'a(T.s))
(Aa(s.s)
+
E
17-181-11
Ys -'a,
F. (la(s.s) + N2
|8| 181-11
Ys
N2
Xy.
24
||Y||le S
P
18-11-11YTT1e
1
(oly(r,p).
dt + P
At + p
d
VN
(C.29)
'hhat is, ||AB AAB-CD||- = |AB
AD AD CPII < ||] B DII 1|4 C|IIPI
26

---

Published as a conference paper at ICLR 2024
Combining Equation C.27 and Equation C.29, we have
(T) predy predd 5o (P(al(t) - )at2(7))
ppol((,
(C.30)
Notice that the æ-th row of predy is 24 d,tp< Vv.- Since X n(p 3 + (p 1)a (p 1)7)
>
znp(( 7), and &
E 1 |S| S mpnp, we have b1
2 1 where the second
N
X1+
inequality follows the assumption on 7)
Since prod, incorrect on the x-th row only if its difference from the æ-th row of prod, has larger norm than 9?(1 - -),, we know the final total enror on the target domains is bounded by (x))))) of (Aa(())) (Aa)))))) poly(r,p).
Collecting all the requirements of N, this bound holds so long as N 2 S (
(minto-r.8 (miit
rp
which is equivalent to n 2 Ω min(o- 1 as N = rpn.
(1)0))(2(0) 4mlbppaatt6000)0800000- A,(0)
Proposition 3.2 In the setting of Theorem 3.1, ifT S
2:
then, with probability at least 1 - n 5, we have
F PEwT(pred(a)) & vo) 5o ( (1 (1-
-poly(r.)),
(C.31)
17((aa(0) - (0)1
where (0) up + n(r 1)3 + nlp 1)a + n(p 1)(r 1)7 and da(0) )a+1(0)
n min[r(8- - p( 7))
Proof. According the proof of Theorem 3.1, we know that when r < (0)-))3a(0) A1(0)
the first eigenvalue
of A' will be
(T) np + n(r 1)3 n(p 1)a + n(p 1)(r 1)7 2 = (0)) (C.32)
and {a( (0) n(p - 3- a a +))++++++)))+))++ + min(p(a- ()) r(3 r(3, (1(3-7)) and d (T) d (0) mpp - 3 - a + 7). Substituting these results into Theorem 4.1 finishes the proof.
Lemma C.3. For f that achieves the minimum of spectral contrastive loss -2 And ((x') + En, Nor (f(x) f(x'))2 ((x)) (where C >>> 0 is any constant), let
b arg min Exes (16T f(z) 12 + m/lbil?)) be the linear head learned on the source
domain S. Let pred E Nxr be the matrix with bT f(x) as its x-th row and pred- be the matrix by restricting pred to the target domain T, then we have
predy = AA(T.S) (AL(S.s) + N2
C
1--11-111 (S|
111-11) Ys.
(C.33)
where (-)t is the Moore- Penrose inverse. [S] is the number of data in the source domain. and Aa the rank-d approximation of A.
Proof. We can rewrite the spectral contrastive loss as
2 An.' [(x) + Σ S(6))
2
N2
2,30!
(C.34)
N
(1
T ( F
A
FF)
+ const,
C
VNN"
VN
27

---

Published as a conference paper at ICLR 2024
where F R is the matrix which the z-th row contains f(x) T. According the Eckart- Young- Mirsky theorem, the minimizer of the above loss function is F _N. Sa where Sa RRx is matrix such that A = $ Sa-
Let Sa.s E Sd be the matrix obtained by restricting Sa to the rows corresponding to the source data, and Sa,7 be the matrix obtained by restricting S₂ the rows corresponding to the target data. The head learned on the source domain can be expressed as
b arg min Σ (111 f(x) bERRX ZES
U= n mb(1)²
(C.35)
VC
C N2 n. 17-181111 Ys.
Sj.
N
Therefore, the prediction on the target domain prody is
predy FTb - SaTSAs (SusShs + C N2 17. 181-1) Ys
(C.36)
Ad,CT.s) (Adis.s) N2 181-1)1) Ys.
This finishes the proof.
Lemma CA. Let Ā = [[A] -111, where the adjacency matrix A is drawn from the stochastic
block model in Section 4.1. Then, for any > 0. if > max(a, 3}, min(a, B} > ry and T < por po+r3+ (pr rYy, we have
pr
AA (Auis.s) + ET) Ys
x Yr,
(C.37)
x + pe
where d = prr-p-p and X - np - n3 + n(p - 1)a - nnlp 1)7 Furthermore, if we use d to denote the i-th largest eigenvalue of A, we have da deFι n min(r(B - 7),,p(a - 7) and X S np rpp
Proof. By the definition of the stochastic block model, every entry Ags' is in the set of {p T, a T - 7 T). depending on whether x and x' belong to the same domain/class. We can index every node (dz; Var idz), where id, [m] is the index of x within domain d, and class ye- For any integer 2 1, we use 1; to denote the i-dimensional all-one vectors, and 1; = 1 be its normalized unit vector. We use Si-1 denote the i-dimensional unit-norm sphere.
It can be verified that À can be decomposed into the sum of several matrix Kronecker products:
A= -) (1,1,) (1,11))
+ ((,))) (1,11) - +0-BB-at))-h00000
(C.38)
+ (1p1)) 8 (1.))
r (1,1,) (1,1)) (1,14)
As a result, Á has the following four sets of eigenvectors with non-zero eigenvalues:
1 8 I, I,,. The corresponding eigenvalue is Xo A np+ n(r 10:0--117 1)5 +n(p 1)a + n(p 1)(r 1)7 - nprr.
u @ I, s T.., where u E -P and u T 11 = 0. The corresponding eigenvalue is Ao A n(r 1)8 - n(r 1)y.
np
I, 8 & 1,, where v E gr-1 and v T 1, = The corresponding eigenvalue is X A
np n3 nlp 1)x= n(p 1)7-
28

---

Published as a conference paper at ICLR 2024
2 u & In. where E gh-1, S- . _u',, 1p O, and u v' 0. The corresponding
eigenvalue Aş _ np n² na + ny.
Since p > max(3,a), min(a, 3} > y and T < patro+(pr-ppr)) we know that
min Ao: A4, > X1:
(C.39)
and all these cigenvalues are positive. When d = - 1, As will contain exactly the first three sets eigenvectors, since they correspond to the top-d eigenvalues. This suggests that we can write Ad follows
Åα Ao ITT II ITT, + + (1p II,IIT,,,,,, II, + & -L,,-I,,,))8 ITI,)
Restricting to the source domain, we have
Anssss) Ao + 1)
I,T, & III d (I, -I,-IIII,))I,,,
(C.40)
p
By the definition of iseudoinverse, we have
+ ((isss)+E1)) (Aot(p- lide + +5) -1 - III, @ IIT, + ( +5) (l, -1.-I,I,) (I,-I,T,
P
Notice that Ys satisfies (I,I, & InI, Y 0 and ((I, - I,I, 2 In In = Ys. we have
\ ( +5) Ys.
-1
(Aa(s,s) + +D)lYs
(C.41)
We can also write (x,) in the form of Kronecker products as follows:
Ad(x.s) = An p 1, I,T @ T.T,, (o & @ I.I,
1 2 (I, (1, (I,- I.
Again, using the fact that IT,T, o II = and ((I, I,I,, 2 I.T,, Ys, we have
A..(x,s) (Aa(s.s) + +51) Ys =
X pp 1 8 Ys.
(C.42)
Finally, noticing that 1 8 Y finishes the proof.
Lemma C.5. Let Ad and Aa be the rank-d approximations of A and A, respectively, Let X be the i-th largest eigenvalue of Å. || II be the operator norm of matrix ty-norm of a vector. Then when ||A All Xa dat1. we have
AA Àall S (1+
2||A Áll + 2| )
1|A ĀlI-
(C.43)
(Aa X4+1) - ||A Al
Proof. Please see the proof in Shen et al. [44, Lemma 3].
Lemma C.6 (Theorem 5.2 in [31]). Let A be the adjacency matrix of a random graph on nodes in which edges occur independently. Let [] À be the expectation adjacency matrix and assume
that N maxi, Aij 2 log N. Then, for any > 0, there exists a constant = C(E) such that
LA - P S CVN
(C.44)
with probability at least 1 25
29

---

Published as a conference paper at ICLR 2024
C.3 PROOF OF THEOREM 3.3
Theorem 3.3 The global minimum of the supervised spectral contrastive loss Loup (J;T) in
Equation 3.8 is uniquely obtained at f(x) = he. for any i, E [r] and A(ra,)) where A = [h2..... h... is the minimizer of [[ (((l--Tie - - HH' F. More specifically, ifr S d 1, H satisfies A rI --r--11 ; >> d have i H Pa(rI- [[11] where Pa(X) denotes the best d-rank upproximation of X.
Proof. Let hi,e Ea-A(e If he Eim1 hi and H [h1, he,,....i.. hy].
For the positive part pas+ we have
2
2000
(((x)) (Ro-mAlesa))(aa))
r
f=1
2
A (or And [f(())
Σ E - A(ss,t) A(5,))() lf
r
(C.45)
2
Σ i=1
hic
Σ hjo
r
c=1
hall
For the negative part Lneg. have
(f(x) f(x)) + +)]
Lueg
r2n
2
E Al))))
f(x)){(x²)))))))))))))))))))))))
+ +
(C.46)
n2
k=
i=1 j=1
x A(2g,t)
(he h4 + 7)²
where the equality holds if and only if f(x) ) is a constant for any i.j [r], x ~ A(xi,)) and x= A(x,,.)
Combining Equation C.45 and Equation C.46, we have
( 2 1 +
hh hy + +)²
22
c=1
c=
k=
(C.47)
[[rl -11 H H + const
According to Eckart- Young- theorem [13]. the global minimizer it of [[rI - -11T)- HT I| the best d-rank approximation of -
More specifically, if < d + 1, H satisfies A A r1 - - if d 1, we have AT Pa(rI - TI where (x( denotes the best d-rank approximation of X.
C.4 PROOF OF THEOREM 3.4
Theorem 3.4 For the noise transition matrix W defined above, the global minimum of Loop (fiT) uniquely obtained at Vi E [n], Vc [r], Vx ~ A(Ii,)), f(x) = ha where it [h,,,.., hp] is the
minimizer of [[r T11 H H F
30

---

Published as a conference paper at ICLR 2024
Proof. According the definition of the label noise model, we know that (f:T)
C pos (fiT) + L noise,nee (fiT)
(C.48)
where
posse. (f:T)= =- en2
E (ond(t
Σ
Wee E A(as (f
c=1
2
he
Σ Weeh''
2 r
Σ Wee h1 Ne
r
(C.49)
and
Csup
noise,neg (fiT)
Wke a(x(ee) (AA(()())
[f(x) f(x)) +
22
[[(() 1(x ) +))² ((x)))(+++++))
Wke R anA((nn)) a(a((e))e))
(C.50)
((x))
E A(ac))
Alzgen))
(he ho + T))
c=1c=1
where the second equality is based on the fact that Ex=1 Wke' 1. We further have
1
Coup (f:T) ,2 [1r - -111) H H const.
(C.51)
This finishes the proof.
C.5 PROOF OF PROPOSITION 3.5
Proposition 3.5 Considering the symmetric label noise [15] in which Weet = 11- -1-(r-1)wfoo for = d, Wee' for + d, and < J. 1fT 2 rw, let f = arg miny Laop (J;T), then VI-TW f is also
the global minimizer of suu (f: f=
Proof. The symmetric label noise model means that W (1 --r)-ll--- + wll according to Theorem 3.4, know that Vi [n], Ve [r]. Vx Alxrc), f(x) fig where H [h1..., hrp is the minimizer of
[Ir =11 H1 11 _
[| -w]ll- (r rw)l1 H H1|?
(C.52)
= (1 rw) (rI (rl- - (-/ーー H) (--- H)
thus, -I-r fz) is also the minimizer of LeuP( T-re) in accordance with Theorem 3.3.
31


Reviews:
[{"review_id": "M39EwyeOLb", "reviewer": "Reviewer_Ef5A", "summary": "In this point, the authors try to extend spectral contrastive learning with negative pairs. By adding a zero-mean regularization, the loss function relaxes the orthogonality of representations between negative pairs and implicitly alleviates the adverse effect of wrong connections in the positive-pair graph, leading to better performance and robustness. Beyond that, this paper gives a solid theoretical analysis for the regularization.", "questions": "I would like to hear from authors about the relationship between this one and contrastive laplacian eigenmaps. I think the authors give a totally different theoretical analysis based on a very similar thing.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. This paper is very easy to follow. The motivation is very clear, and the methodology is elegant.\n\n2. The perspective of UDA is novel. It provides a different way to observe contrastive learning.\n\n3. The perspective of supervised classification with a noise label is also helpful for contrastive learning.", "weaknesses": "1. The novelty is an issue. According to the Equ 3.2, it is the same as contrastive laplacian eigenmaps (NeurIPS 2021). In contrastive laplacian eigenmaps, they have the same three terms. The main point is the fully connected adjacent matrix (the all-one matrix).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6wDUTP8QfB", "reviewer": "Reviewer_9yLv", "summary": "The authors provide a new regularizer for (spectral) contrastive\nlearning which supposedly improves the quality of the representation.\nThe authors provide experiments to show the quantitative improvements\ndue to their method.", "questions": "Could the approach be extended to other CL techniques?  Do you expect\nit to improve the results similarly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The regularization is well-motivated and intuitively explained.  The\nexperiments seem convincing and suggest that the improvement is\nconsisent across datasets.\n\nThe appendix is extensive.", "weaknesses": "The results on the CIFAR datasets are not state-of-the-art for\ncontrastive learning.  This begs the question whether the regularizer\ncould also be applied to other CL techniques such as SimCLR or others.\nHopefully the regularization would also help in that case.\n\nTable 1 & 2 only report single numbers.  It would be more convincing\nif there was a mean +- std reported or some other statisic computed\nover multiple runs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UXvhbFb8TG", "reviewer": "Reviewer_41KY", "summary": "By incorporating an additive factor into the SpeCL term that involves negative pairs, this paper enforces the mean of representations to be zero. The experimental results and related theoretical analysis suggest that introducing $\\tau$ can improve performance.", "questions": "1. On page 4, there is a missing space in \"thatSpeCL\".\n\n2. It is suggested to include more benchmark methods in the experiments to avoid categorizing it as an enhanced/improved version of SpeCL method, and instead position it as a robust SSL method.\n\n3. The factor $τ$ effectively relaxes the orthogonal constraint on negative pairs. It would be interesting to explore the integration of other methods, such as Barlow Twin, in different ways.\n\n4. In the experiments, it is recommended to analyze the results achieved with different values of $\\tau$ due to its previous detailed analysis. This analysis can provide insights into what can be inferred from the results obtained with different $\\tau$ values.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This paper addresses the issue of negative pairs in contrastive learning, which is a hot topic in the research community. The authors aim to provide a solution to this problem that arises in related CL works.\n\n2.The authors of this paper present a theoretical foundation to support the proposed zero-mean regularization in both the unsupervised domain adaptation (UDA) task and supervised classification with noisy labels.\n\n3.The proposed method's robustness is improved by the impressive results achieved on the supervised classification task, particularly when dealing with symmetric label noise.", "weaknesses": "1. This work appears to be relatively incremental based on Ref. [18] and [19]. Basically, what caught my interest is the representation of spectral embeddings using affinities and features; however, it was previously proposed in [18].\n\n2. Although the authors describe the proposed additive factor as simple yet effective, this work can still be considered progressive research. However, when compared to [18] and [19], it is unclear whether significant progress has been made for this conference.\n\n3. The effectiveness of the proposed method is demonstrated through experiments that involve fewer state-of-the-art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)