Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism
Authors: Zhijian Zhuo
Yifei Wang
Jinwen Ma
Yisen Wang
Venue: ICLR 2023 poster (2023)

Abstract: Recently, a variety of methods under the name of non-contrastive learning (like BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric architectural designs, aligning positive pairs alone is sufficient to attain good performance in self-supervised visual learning. Despite some understandings of some specific modules (like the predictor in BYOL), there is yet no unified theoretical understanding of how these seemingly different asymmetric designs can all avoid feature collapse, particularly considering  methods that also work without the predictor (like DINO). In this work, we propose a unified theoretical understanding for existing variants of non-contrastive learning. Our theory named Rank Differential Mechanism (RDM) shows that all these asymmetric designs create a consistent rank difference in their dual-branch output features. This rank difference will provably lead to an improvement of effective dimensionality and alleviate either complete or dimensional feature collapse. Different from previous theories, our RDM theory is applicable to different asymmetric designs (with and without the predictor), and thus can serve as a unified understanding of existing non-contrastive learning methods. Besides, our RDM theory also provides practical guidelines for designing many new non-contrastive variants. We show that these variants indeed achieve comparable performance to existing methods on benchmark datasets, and some of them even outperform the baselines. Our code is available at \url{https://github.com/PKU-ML/Rank-Differential-Mechanism}.

Published as a conference paper at ICLR 2023
TOWARDS A UNIFIED THEORETICAL UNDERSTAND- ING OF NON-CONTRASTIVE LEARNING VIA RANK DIFFERENTIAL MECHANISM
Zhijian Zhuo Yifei Wang Jinwen Ma Yisen Wang
School of Mathematical Sciences, Peking University
National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University Institute for Artificial Intelligence, Peking University
ABSTRACT
Recently, a variety of methods under the name of non-contrastive learning (like BYOL, SimSiam, SwAV, DINO) show that when equipped with some asym- metric architectural designs, aligning positive pairs alone is sufficient to attain good performance in self-supervised visual learning. Despite some understand- ings of some specific modules (like the predictor in BYOL), there is yet no unified theoretical understanding of how these seemingly different asymmetric designs can all avoid feature collapse, particularly considering methods that also work without the predictor (like DINO). In this work, we propose a unified theoreti- cal understanding for existing variants of non-contrastive learning. Our theory named Rank Differential Mechanism (RDM) shows that all these asymmetric designs create consistent rank difference in their dual-branch output features. This rank difference will provably lead to an improvement of effective dimen- sionality and alleviate either complete or dimensional feature collapse. Different from previous theories, RDM theory is applicable to different asymmetric designs (with and without the predictor), and thus can serve as a unified under- standing of existing non-contrastive learning methods. Besides, our RDM the- ory also provides practical guidelines for designing many new non-contrastive variants. We show that these variants indeed achieve comparable performance to existing methods benchmark datasets, and some of them even outperform the baselines. Our code available at https: /tap:://githi /ttptgggttbb/t///g www.coo//c//ccc/ ML/ Maa---DDffereential-Mcchhhh
INTRODUCTION
Self-supervised learning of visual representations has undergone rapid progress in recent years, par- ticularly due to the rise of contrastive learning (CL) (Oord et al. 2018 Wang et al. 2021). Canonical contrastive learning methods like SimCLR (Chen et al.) 2020) and MoCo (He et al. 2020) utilize both positive samples (for feature alignment) and negative samples (for feature uniformity). Sur- prisingly, researchers notice that CL can also work well by only aligning positive samples, which is referred to as non-contrastive learning. Without the help of negative samples, various techniques are proposed to prevent feature collapse, for example, stop- gradgertintt, momentum encoder, predictor (BYOL (Grill er al.) 2020). SimSiam (Chen & He 202T)), Sinkhorn iterations (SwAV (Caron et al.) 2020), feature centering and sharpening (DINO CCarnn al. 2)2 These above designs all cre- ate a certain of asymmetry between the online branch (with gradient) and the target branch (without gradient) (Wang et al. 2022a).. Empirically, these tricks can successfully alleviate feature collapse and obtain comparable this progress, it is still not clear why these different heuristics can reach the same goal.
even superior performance than canonical contrastive learning. Despite
Some existing works proposed understand some specific non-contrastive techniques, mostly focusing on the predictor head proposed by BYOL (Grill et al. al.] 20220). From an empirical
& He (202T) think that the predictor helps approximate the expectation over augmentations, and Zhang et al. (2022a) take center-residual decomposition of representations for analyzing the col- lapse. From a theoretical perspective, Tian et al. (202I) analyze the dynamics of predictor weights
"Equal Contribution.
"Corrsponning Author: Yisen Wang (yisen, wang@pku.edu.ccm

---

Published as a conference paper at ICLR 2023
BrOL
Sand
SeuWi
DINO
210
375
26
323
10-
26-
O1:
O9
0
Coech
Cooch
Coock
Zoech
Figure 1: The effective rank of the normalized outputs of the online and target branches for four different non-contrastive methods (BYOL, SimSiam, SwAV, and DINO) on CIFAR-IO.
under simple linear networks, and Wen & Li 2022) obtain optimization guarantees for two-layer nonlinear networks. These theoretical discussions often need strong assumptions on the data distri- bution (e.g, standard normal (Tian et al. 2021 and augmentations (e.g. random masking (Wen &
Li 2022 ). Besides, their analyses are often problem-specifie which is hardly extendable other non non-contrastive variants without a predictor, e.g, DINO. Therefore, a natural question is raised here:
Are there any basic principles behind these seemingly different techniques?
In this paper, we make the first attempt in this direction by discovering a common mechanism behind these non-contrastive variants. To get glimpse of it, in Figure
measure the effective rank (Roy & Vetterli, (007) of four different non-contrastive methods (BYOL, SimSiam, SwAV, and DINO). We find the following phenomenons: 1) among different methods, the target branch (orange line) has consistently higher rank than the online branch (blue line); 2) after the initial warmup stage, the rank of the online branch (blue line) consistently improves along the training process. Inspired by this observation, we propose new theoretical understanding of non-contrastive methods, dubbed Rank Differential Mechanism (RDM), where we show that these different techniques essentially behave as a low-pass spectral filter, which is guaranteed to induce the rank difference above and avoid feature collapse along the training. We summarize the contribution of this work as follows:
Asymmetry matters for feature diversity. In contrast to common beliefs, we show that even symmetric architecture can provably alleviate complete feature collapse. However, it still suffers from low feature diversity, collapsing to a very low dimensional subspace. indicates the key role of asymmetry is to avoid dimensional feature collapse.
Asymmetry induces low-pass filters that provably avoid dimensional collapse. Based on theoretical and empirical evidence on real-world data, we point out the common un- derlying mechanism of asymmetric designs in BYOL, SimSiam, SwAV, DINO is that they behave as low-pass online-branch filters, or equivalently, high-pass target-branch filters. We further show that the asymmetry-ine -nnuucd low-pass filter can provably yield the rank collapse (Figure I and prevent feature collapse along the training process.
Principled designs of asymmetry. Following the principle of RDM, we design series of non-contrastive variants empirically verify the effectiveness of our theory. For the online encoder, we show that different variants of low-pass filters can also attain fairly good performance. We also design new kind of targel predictors with high-pass filters. Experiments show that SimSiam with our target predictors can outperform DirectPred (Tan et al. 2021) and achieve comparable or even superior performance to the original SimSiam.
2 RELATED WORK
Non-contrastive Learning. Among existing methods, BYOL GGrill et al. 2020) is the first to show we can alleviate the feature collapse of aligning positive samples along with an online predictor and a momentum encoder. Later, SimSiam (Chen & He (02T) further simplifies this requirement and shows that only the online predictor is enough. As for another thread, SwAV (Caron et al.) 2020) ap- plies Sinkhorn- KonkhopnnKKKoppppp iterations (Cuturi 2013) the target output from an optimal transport view. DINO (Caron et al. (021) further simplities this approach by simply combining feature centering and feature sharpening. Remarkably, all these methods adopt online-target dual- branch architec- ture and gradients from the target branch are detached. Our theory provides a unified understanding of these designs and reveals their common underlying mechanisms. Additional comparison with related work is included in Appendix F
2

---

Published as a conference paper at ICLR 2023
1,0
un ol aigeggrreee
SymSind
of agpcroo/oaa
60-
Samtiam
Samm
0.5-
Sinsiam SymS mélmm
-20
Varilla Stamene
-10
Vari Samees
0.4
Varilis Sammos
0.2
20
20
0
100
200 330
400
0.0
200
400
1500
1930
2000
-25
500
105a
Higenvaluss
(a) Accuracy
(b) Variance
(c) Eigenvalues
(d) Two branches
Figure 2: Comparison of the asymmetric SimSiam with symmetric baselines, SymSimSiam
(ours) and Vamilla Siamese on CIFAR-10, where SymSimSiam adopts Laym (fb) (Eq. 2 while
Vanilla Siamese adopts (So) (a) Linear probing test accuracy. (b) Feature variance of nor-
malized outputs. [c]] Eigenvalues of the correlation matrix of the normalized outputs. (d) Eigen-
values of the correlation matrix of the normalized outputs from two branches of SymSimSiam.
Dimensional Collapse of Self-supervised Representations. Prior to ours, several works also ex- plore the dimensional collapse issue in contrastive learning. (Ermolov et al. (202T), Hua et al. (2021), Weng et al. (2022) and Zhang ct al. (2022b) propose whiteming techniques to alleviate dimensional collapse, similar spintt to Barlow Twins (Zbontar et al. 2021) with feature decorrelation regular- ization. Jing et al. 2022) point out the dimensional co collapse of contrastive learning without using the projector, and propose DirectCLR
direct replacement. Instead, our work mainly focuses on understanding the role of asymmetric designs on overcoming dimensional collapse.
Theoretical Analysis on Contrastive Learning. Saunshi et al. (2019) first establish downstream guarantees for contrastive learning, which are later gradually refined (Nozawa & Sato 2021| Ash cct al. 2022 [ao et al. 2022). (ooh et al. (2021) Lee et al. (202I) also propose similar guarantees on downstream tasks. However, these methods mostly rely on the conditional independence assump- tion that is far from practice. Recently, HHoChen al. (202T) and Wang et al. (2022b) propose an augmentation graph perspective with more practical assumptions, and contribute the generalization ability to the existence of augmentation overlap (which also exists for non-contrastive learning). Wen & Li (2021) analyze the feature dynamics of contrastive learning with shadow ReLU networks.
ASYMMETRY IS THE KEY TO ALLEVIATE DIMENSIONAL COLLAPSE
Prior works tend to believe that asymmetric designs are necessary for avoiding complete feature col- lapse (Zhang et al. 2022a), while we show that a fully symmetric architecture. dubbed SymSimSiam (Symmetric Simple Siamese network), can also avoid complete collapse. Specifically, we simply align the positive pair (z. x)) with symmetric alignment loss,
Caym Ez,, fo(x) fa(x)), f(('t)
where we apply feature centering the output of an encoder f. i.e., fo(-) μ. The feature average 4 fo()) can be computed via mini-batch estimate or exponential moving average as in Batch Normalization (Ioffe & Szegedy (2015) (2015) (see Algorithm[I] []. Theorem states that SymSim- Siam can avoid complete collapse by simultaneously maximizing feature variance Var(fo(z))
Theorem 1. When fo( is lz-normalized, the SymSimSiam objective is equivalent to
Csym = Caym (fe) Var(fe(x)) + +1= -Ex,+ fo fo( - E. (ff(z))))) - 21l2 + (2)
Empirically shown in Figures 2(a] [(b] compared to vanilla Siamese network, SymSimSiam
indeed alleviates complete collapse, i.e., the feature variance is maximized along training and good linear probing accuracy achieved. However, the accuracy of SymSimSiam is also obviously lower than the asymmetric SimSiam (Figure 2(a)). This indicates that symmetric design could alleviate complete collapse, but it may be not enough to prevent dimensional feature collapse. Intuitively, features uniformly distributed on a great circle of a unit sphere have maximal variance while being dimensionally collapsed. As further shown in Figure 2(c) SymSimSian indeed suffers from more severe dimensional collapse compared SimSiam. With few effective dimensions, the encoder network has limited ability to encode rich semantics (HaoChen et al., 202T).
The above SymSimSiam experiments show that with a symmetric architecture, we can easily pre- vent complete collapse while hardly improving the effective feature dimensionality for overcoming
3

---

Published as a conference paper at ICLR 2023
DYCL
Sand
3MD
1.6
0.5
0114-10
8--
014
206
06
Doson
Epoce
Erich
Zoech
Figure 3: Eigenspace alignment of the online and target outputs along the training. Given u, as an eigenvector of Cz, if it is also an eigenvector of Cy. then Cp X,u; is in the same direction as Uy Thus, we measure the alignment of u, by computing the cosine similarity between u, and C U4 and take the average over each ti as the overall eigenspace alignment (details in Appendix B.3
dimensional collapse. Instead, we also notice that SimSiam with asymmetric designs alleviate dimensional collapse and achieve better performance. reflects the fact that the asymmetry in ex- isting non con-contrastive methods is the key to alleviating dimensional collapse, which leads us to the main focus of our paper, iee., demystifying asymmetric designs.
4 THE RANK DIFFERENCE MECHANISM OF ASYMMETRIC DESIGNS
In Figure I we have observed a common mechanism behind the non-contrastive learning methods: these asymmetric designs create positive rank differences between the target and online outputs consistently throughout training. Here, we provide a formal analysis of this phenomenon from both theoretical and empirical sides and show how it helps alleviate dimensional feature collapse.
Problem Setup. Given a set of natural training data X { I Rd), we can draw pair of positive samples (z, .2 from independent random augmentations of natural example E with distribution A(-(T). Their joint distribution satisfies P(x, xx) = P(x+,a0 , = ((r((()))((a) [z). Without loss of generality, following HaoChen et al. (202T). and denote the collection of online and target outputs as p, 2
consider a finite sample space | (can be exponentially large)
Baxk whose æ-th
is P2; Zz, respectively. We consider general alignment loss
(pp) E l(p,, sg(2++))
(3)
to cover different variants of con-contrastive methods. The online and target outputs Pz: 2n are either fz-normalized (BYOL and SimSiam) or softmax-normalized (SwAV and DINO). For the loss function BYOL and SimSiam adopt the mean square error (MSE) loss, while SwAV and DINO adopt the cross entropy (CE) loss. sg(.) denotes stopping the gradients from the operand. whose spectral
For an encoder f. we define its feature correlation matrix C = E (x) decomposition is C VAVT, where V contains unit eigenvectors in columns, and A is the diagonal matrix with descending eigenvalues λ 2 ... 2..2.. X 2 O.
Measure of Dimensional Collapse. well-known measure of the effective feature dimensionality the effective rank (erank) of the feature correlation matrix
(=f Roy & Vetterli
2007]. Specifically, erank(C) = exp( (q)), where ! = (qi,...,
2 DiA. A are the normalized eigenvalues as probability distribution, and H(q) = - E. di log(4) is its Shannon entropy. Compared to the canonical rank, the effective rank is real-valued and invariant to feature scaling. more uniform distribution of eigenvalues has larger effective rank, and vice versa. Thus, the effective rank of C is a proper metric to measure the degree of dimensional feature collapse.
the signal processing literature, spectral filtering process G of a signal f is to apply a scalar function (i.e., spectral filter) g R + IR element-wisely on its eigenvalues in its spectral domain, i.e., u = Gf(x) = Vg(A)V' f(x), where G = Vg(A)VT T is also known as spectral convolution operator. The filtered signal admits C. c== on the property of g. a filter can be categorized as low-pass, high-pass, band-pass or band-stop. Gen- erally speaking, a low-pass filter will amplify large eigenvalues and diminish smaller ones (e.g., monotonically increasing function), and a high-pass filter does the opposite. Many canonical algo- rithms can be seen as special cases of spectral filtering, e.g., PCA-based image denoising is low-pass filtering.
Spectral Filters.
= Eg up Vg(A)AVT Depending
4

---

Published as a conference paper at ICLR 2023
4.1 ASYMMETRIC DESIGNS BEHAVE AS SPECTRAL FILTERS
First of all, we notice that regardless of the existence of asymmetry, the alignment of two-branch outputs in non-contrastive learning will enforce the two-branch output features of the positive pairs to be close each other. Therefore, from spectral viewpoint, a natural hypothesis is that the online and target features will be aligned into the same eigenspace, and only differ slightly in the eigenvalues. We describe this hypothesis formally below.
Definition (Eigenspace alignment). For two matrices A and B, they have aligned eigenspace if
V,, s.t. A= VAvV B = VA,V",
(4)
where V is an orthogonal matrix of eigenvectors and Nay Ab consist non-increasing eigenvalues. Hypothesis 1. During training, non-contrastive learning aligns the eigenspace of three correlation matrices of output features: the online correlation Cp E.Papy the target correlation C₂ E and the feature correlation of positive samples C+ = 2,
Next, we validate this hypothesis from both theoretical and empirical aspects. To begin with, we consider a simplified setting adopted in prior work (Tian et al.) (02T) for the case of analysis: 1) data isotropy, where the natural data distribution p(z) has zero mean and identity covariance, and the augmentation A(x [z) has mean ã and covariance o² 2) linear encoder 2n f(z) Wyz, Ws e Raxk; 3) linear online predictor pa- W W E R*xk, Under this setting, the following lemma shows that for an arbitrary encoder f. the eigenspace of three correlation matrices indeed align well:
Lemma 1.
With the assumptions above as in Tian et al. (2027). when the predictor W minimizes the alignment loss (Eq. 3). we have
= 3V, s.t. VA VA.V CC+=
(5)
where V is an orthogonal matrix and Ap, Az, A4 are diagonal matrices consisting of descending eigenvalues X² AT
.....,, respectively.
Next, we provide an empirical examination of Hypothesis on on real-world data. From Figures 3 can see that there is a consistently high degree of eigenspace alignment between Cp and Cg among all non-contrastive methods | In Appendix E. we further verify the alignment w.r.t C+. Therefore, these methods indeed attain fairly high degree of eigenspace alignment.
A Spectral Filter View. As result of eigenspace alignment, the alignment loss essentially works on mitigating the difference in eigenvalues. Therefore, we can take a spectral perspective on non- contrastive methods, where the asymmetric designs are equivalent to spectral filtering process applied to the target output Zz, or target spectral filtering process applied to the target output pa- As the two cases are equivalent, we mainly take the online filter as an example the discussion below.
Lemma 2. Denote an online filter function g: X2 X² XB that satisfies X² V 1..i=1....4 We can apply a spectral filtering on 2g with g. and get p.= Wz W V VV.) Then, we have Cp C5 E.. In other words, Pa, Pa have the same feature correlation.
This spectral filter view reveals the key difference between symmetric and asymmetric designs in non-contrastive learning. In the symmetric case, the two branches yield almost equal eigenvalues (Figure 2(d). Thus, the alignment loss will quickly diminish and the representations collapse dimen- sionally (Figure Instead, in asymmetric designs, asymmetric components create a difference in eigenvalues such that the target output generally has higher rank than the online output (Figure 4(a)). Therefore, the aligmment loss will not easily diminish (not necessarily decrease; see Figure 9). Instead, the alignment improves feature diversity in an implicit way, as we will show later.
4.2
LOW-PASS PROPERTY OF ASYMMETRY-INDUUCEE SPECTRAL FILTERS
The discussion above reveals that specific asymmetric design behaves as spectral filter when applied to non-contrastive learning. To gain some insights for a unified understanding, we further investigate whether there is a common pattern behind the filters of different asymmetric designs.
To achieve this, we calculate and plot the corresponding online filter g(>=) non-contrastive method. From Figure 4(b) find that the spectral filters indeed look very similar,
/XTAx; of each
As DINO adopts dynamic feature sharpening coefficients, there is larger change of its eigenspace align- compared to others. Nevertheless, the alignment degree is always above 0.7, which is relatively high.
5

---

Published as a conference paper at ICLR 2023
DYCL
SmSiiam
SwdW
DWO
Onire
Oetra
-25
Lag operisue
-100
-22
-30
-35
Operinlun
Eperaaus 7000
(a) Eigenvalues of the feature correction matrices of the online and target outputs.
DVOL
Sm53amm
SwiW
DIVO
57
62-
a2e alz
a1a
0.2
0.50
2.35
colo
cdis
odas
oubes
oakea
01:
5.00 dez dosodks odos eulon
(b) The corresponding online spectral filter function ( VXTAA.
Figure 4: Eigenvalues and spectral filters of each method on CIFAR-I0: top eigenvalues (whose sum is larger than 99.99% of the total sum) are shown, 128 for BYOL and 512 for the other.
Center Snersentii 301
606
626
(a) BYOL
(b) SimSiam
(c) SWAV
(d) DINO
Figure 5: The role of asymmetric designs.
a & (
spectral filters of the ideal optimal predictors,
g" (Aş) = />; (Eq. 6p. Both filters calculated from BYOL and SimSiam are almost monotoni-
cally decreasing. (c) The target filters using different LinkhornKnopp (SK) iterations in SwAV. (d)
The target filters using centering and/or sharpening with different target temperatures (!) in DINO
(the online temperature is set to 0.1).
particularly in the sense that all filter functions are roughly monotonically increasing w.r.f A= This kind of filter is usually called a low-pass filter because it (relatively) enlarges low-frequency com- ponents (large eigenvalues) and shrinks nigh-frequency components (small eigenvalues). Based on this empirical finding, we propose the following hypothesis on the low-pass nature of asymmetry.
Hypothesis 2. Asymmetric modules in non-contrastive learning behave as low-pass online filters. Formally, the corresponding spectral filter g(X²) = /X,/X² is monotonically increusing
We note that could easily apply a high-pass filter to the online output deliberately (which, as we have observed, will likely fail). Therefore, our hypothesis above only applies to asymmetric designs that work well in practice. In the discussion below, further provide theoretical and empirical investigations of why existing asymmetric modules have a low-pass filtering effect.
not suggesting any asymmetric design behaves as low-pass filters, someone
Case I. Online Predictor. One popular kind of non-contrastive method, including BYOL and Sim- Siam, utilizes a learnable online predictor ge R I for architectural asymmetry. One would wonder why such a learnable predictor will behave low-pass filter (Lemma 2). Here, we provide some theoretical insights in the following theorem.
"qquivalently, when viewing spectral filters from the target branch, the asymmetric modules behave as a high-pass target filter because the corresponding filter function h(XP) = /x//X is monotonically decreasing.

---

Published as a conference paper at ICLR 2023
Theorem 2. Under Hypothesis
assume the invertibility of Cz. the optimal predictor is given by where wi = S2 = X² /2 E [0. 11, E [M]. W = =
(6)
Therefore, the spectral property of the learnable filter is determined by the filter function X /AT-
Theorem shows that the predictor essentially learns to predict the correlation C+ between positive samples from one augmented view Cz, i.e, eliminating the augmentation noise and predicting the common features. The following lemma further reveals that 1) the correlation between positive samples is equivalent to the correlation of their underlying natural data E, and 2) the difference between C4 and C2 is equal to the conditional covariance induced by data augmentation A(x[z).
Lemma 3. The following equalities hold:
1. C+ = CC 2 where 2n = 1a
2. C, = C+ where Vajz Egg (Zz 22) (Zo -2)² is the conditional covariance.
In practice, data augmentations mainly cause sigh-freuency noises in the feature space, therefore, the denoising predictor will behave as a low-pass filter. Indeed, Figures [[a) and 5(b) emppricaly show that the filter derived from our theory, AT /X,, aligns well with the actual learned predictor in Figure 4(b) as a low-pass filter.
Case II. Target Transformation. Another kind of con-contrastive method is to apply hand-crafted (instead of learned) transformations in the target branch, such as the Sinkhom-Knopp (SK) iteration in SwAV (Caron et al. 2020) and centering whwreenninaaaagggggg...aa operators in DINO (Caron et al.) 2021 In this case, we further study how these transformations behave as high-pass filters applied to the target output (see footnote of Hypothesis 2) Since it is generally hard to analyze these spatial transformations in the spectral domain, here we empirically study the role of each transformation on the resulting filter. Figure 5(c) shows that SK iterations indeed act like low-pass filters, and one iteration is enough. This explains why few SK iterations already work well in SWAV. As for DINO, we notice that centering operation alone is not enough produce a high-pass filter, which agrees with empirical results in DINO. Meanwhile, we notice that in order to obtain a high-pass filter (monotonically decreasing), it is necessary for DINO to apply temperature smaller than the online branch (< 0.1), which is exactly the feature sharpening technique adopted in DINO (Caron et al.) 2021). These facts show our theory aligns well with empirical results in non-contrastive learning.
4.3 ASYMMETRY- INDUCED LOW-PASS FILTERS SAVE NON CON-CONTRASTIVE LEARNING
In the discussion above, observed common pattern in existing asymmetric designs: their cor- responding spectral filters are all low-pass, Here, we further show that this property so essential that it can provably save con-contraetive learning from the risk of feature collapse by producing the rank difference between outputs and alleviating dimensional collapse during training.
First, let us take a look at its effect on the effective rank of two-branch output features. In the theorem below, we show that low-pass online filters are guaranteed to produce consistently higher effective rank of target features than that of online features, as shown in Figure
Theorem 3. If g(()) = /x/x; is monotonically increasing, or equivalently, M(2)) VXUX is monotonically decreusing, we have erank(C,) S erank(C.). Further, as long g(X²) or h(A?) non-constant, the inequality holds strictly, (C,) < erank(C.).
Meanwhile, we also observe that for each output, its own effective rank is also successfully elevated along this process. This is not coincidence. Below, we theoretically show how the rank difference alleviates dimensional collapse. As the training dynamics of deep neural networks is generally hard to analyze, prior works (Tian et al.) 2021 Wen & Li] (021]) mainly deal with linear or shadow networks with strong assumptions
data distribution, which could be far from practice. As over- parameterized deep neural networks are very expressive, we instead adopt the unconstrained feature setting Miixon et al.) 2022, HaoChen et al., 2021) and consider gradient descent directly in the feature space R' Taking the MSE loss E(u, v) = 2 -l - rll2 as an example, the following theorem shows that the rank difference indeed helps improve the effective rank of online output p.
Theorem 4. Under Hypothesis | when we apply an online spectral filter Pa Wzz zz (Lemma 2
gradient descent with step size < a < gives the following update at the t-th step,
X
X r+++1
(1 (1-a} a)² - <221( + 20 a)h(A?,)) X7,
i- 11=1... 1..,,,,
(7)
7

---

Published as a conference paper at ICLR 2023
Table 1: Linear probing accuracy (%) of SimSiam with different online predictors (including learn- able nonlinear and linear predictors of SimSiam) on CIFAR-I0, CIFAR-100 and ImageNet-100.
CIFAR-10
CIFAR-100
100
IN-100
(400ep)
Predictor
100ep
200ep
400ep
100ep
200ep
400ep
acc
acc@
SimSiam (nonlinear)
74.00
84.39
89.41
46.12
55.51
63.39
77.96
94.26
SimSiam (linear)
72.42
83.67
88.9
42.98
55.69
62.02
77.18
94.02
g(
73.91
82.74
86.51
46.73
54.45
59.59
75.06
93.28
g( log(o)
74.80
80.72
87.52
39.82
48.59
58.08
77.22
94.34
(( = log( +
75.70
80.47
86.76
40.53
47.68
61.15
77.42
93.90
g( = log(1 + o²)
75.51
82.28
87.51
39.19
48.97
59.06
77.22
94.00
Table Linear probing accuracy (%) of SimSiam with different target predictors (including learn- able nonlinear and linear predictors) CIFAR-10, CIFAR-100 and ImageNet-100
CIFAR-10
CIFAR-100
IN-100
(400ep)
Predictor
100ep
200ep
400ep
100ep
200ep
400ep
acc
acc @5
SimSiam (nonlinear)
74.00
84.39
89.41
46.12
55.51
63.39
77.96
94.26
SimSiam (linear)
72.42
83.67
88.90
42.98
55.69
62.02
77.18
94.02
h(o) = o -0.3
77.33
85.57
88.74
48.07
58.07
62.83
77.68
94.16
h(a)
78.17
85.52
88.43
50.61
58.68
62.79
76.74
94.10
h(a) o 0.7
77.98
85.79
88.45
51.04
58.37
62.69
77.28
94.94
h(o) = =o"
77.33
86.44
88.50
51.05
58.87
61.92
77.00
93.32
where h(()= 1/g(X) is high-pass filter because g(A) is low-pass filter (Hypothesis) 2p. Then, the update XP / will nicely correspond to a high-pass filter under either of the two conditions:
1. the learned encoder nearly optimal, i.e., W W in Theorem
2. Xt, 1 which naturally holds with good positive alignment, i.e., 2x Zxt.
Then, according Theorem 3 we have crank( Cp(t+)) crank(C,r.). In other words, the effec- tive rank of online output will keep improving after gradient descent.
Intuitively, the improvement of effective rank is natural consequence of the rank difference. As the online output has a lower effective rank than the target output, optimizing its alignment loss w.r.t the gradient-detacher target sg( =x)) will enforce the online output to improve its effective rank in order match the target output 2g. In this way, the rank difference becomes ladder (cre- ated by asymmetric designs) for non-contrastive methods to gradually improve its effective feature dimensionality and get rid of dimensional feature collapse eventually.
Note on Stop Gradient. Our analysis above also reveals the importance of the stop gradient oper- ation. In particular, when gradients from the target branch are not detached, in order to match the rank of two outputs, minimizing the alignment loss can also be fulfilled by simply pulling down the rank of the target output. this case, the feature rank will never be improved without stop gradient.
PRINCIPLED ASYMMETRIC DESIGNS BASED ON RDM
The discussions above show that the rank differential mechanism provides unified theoretical un- derstanding of different non-contrastive methods. Besides, it also provides a general recipe for designing new non-contrastive variants. As Theorem points out, the key requirement is that the asymmetry can produce a low-pass filter on the online branch, or equivalently, high-pass filter on the target branch. Below, we propose some new variants directly following this principle.
5.1 VARIANTS OF ONLINE LOW-PASS FILTERS
Despite the learnable predictor (Section 4.2, can also directly design online predictors with fixed low-pass filters. For numerical stability, we adopt the singular value decomposition (SVD) of the output compute the eigenvalues and the eigenspace, egg., 2 UΣ with singular values of's. ss As X₂ (o 2 filter that is monotone in is also monotone in a, and vice versa. Specifically, for
8

---

Published as a conference paper at ICLR 2023
Table 3: Linear probing accuracy (%) of SimSiam with different predictors on ImageNet-1K 1k.
SimSiam
Online Low-pase Low-passssa Filter (ours)
Target High-pass, passs Filter (ours)
(Learnable Online Filter)
acc @1 log(1 [ + oa)
g (a)
( = o -0.3
acc @1
acc@5
acc @5
acc@1
acc@5 5
67.97
88.17
64.67
86.13
67.73
88.05
Orline Filtes CIAB10
Targt hars on CPAR-30
Onlns flles C/A33302
Target fars CFAn-200
Sidaam
5mdak
Sudam
Sindunn
slowigh slowgrm planiger sioingg
plamm gett
-20
303
LOC6
22:00
2008
500
1006
3300
1008
12.0
2000
530
3300
15.0
3000
Opetzaaes
OgenvoLes POBC
Opervilis Poce
Figure 6: Eigenvalues of the correlation matrix of the normalized outputs for different variants of filters on CIFAR-10 and CIFAR-100.
an online encoder f + R* we assign W V9(E*)VT, where g(E)) -g(o;) is low-pass
filter that is monotonically increasing w.r.t. a. We note that DirectPred proposed by Tian et (202T) is a special case with (o( = a. Additionally, we consider three variants: 1) g (a) = log( 2)
g2 (A)) = log(a + 1);
1): 3) ga (o) = log(o² + 1). These three new variants are low-pass filters because
they are monotonically increasing with σ 2 0.
We evaluate their performance on four datasets, i.e. CIFAR-IO, CIFAR- (Krizhevsky et al. 2009). ImageNet-100 100 and ImageNet- (Deng et al., 2009). We use ResNet- 18 (He et al. 2016) as the backbone encoder for CIFAR-IO, CIFAR-IOX 100 and ImageNet-100 and adopt ResNet-50 for Imagenet-1k kk following standard practice. And the projector is a three-layer MLP in which BN (Toffe) & Szegedy 2015)) is applied to all layers. We adopt the linear probing task for evaluating the learned representations. More details are included in Appendix B.2 Jssows that our designs of online predictors with different low-pass filters work well in practice, and achieve comparable performance to SimSiam with learnable predictors. In particular, it also significantly outperforms SymSimSiam (Figure 2(a)). showing that rank differences indeed help alleviate dimensional collapse (Figure 6).
5.2 VARIANTS OF TARGET HIGH-PASS FILTERS
In turn, we also consider applying a high-pass filter on the target branch using a target predictor. Compared to the online predictors above, target predictors have additional advantages: with stop gradient, we do not require backpropagation through SVD, which could reduce time overhead (Table 5). Specifically, we consider polynomial high-pass target filters i(a)) = a" with different -1 S p < 0, which are all monotonically decreasing (sce Algorithm 2)
We further evaluate the target filters following the same protocols above. As shown in Table 2 our high-pass target filters can often outperform SimSiam by a large margin with relatively short training time. Especially, on CIFAR-10, the filter h(o) a 00.5 improves the baseline by 5.75% with 100 epochs and the filter h(a) o improves the baseline by 2.77% with 200 epochs. Notably, on 100, our methods outperform the baseline by large margin (8.07%, 3.18,, and 0.81% 1% improvements with 100, 200, and 400 epochs, respectively). Additional results based on the YOL framework can be found in Appendix C
CONCLUSION
In this paper, we presented a unified thcoretical understanding of non-contrastive learning via the rank differential hypothesis. In particular, we showed that existing non-contrastive learning all pro- duce consistent rank difference between the online and the target outputs. Digging deeper into this phenomenon, we theoretically proved that low-pass online filters can yield such a rank differ- ence and improve the effective feature dimensionality along the training. Meanwhile, we provided theoretical and empirical insights on how existing asymmetric designs produce low-pass filters. At last, following the principle of our theory, we designed a series of new online and target filters, and showed that they achieve comparable even superior to existing asymmetric designs.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Jinwen Ma is supported by the National Key Research and Development Program of China under grant 201 8AAA000200 Yisen Wang is partially supported by the National Natural Science Foun- dation of China (62006153), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), and Huawei Technologies Inc.
REFERENCES
Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. In AISTATS, 2022.
Han Bao, Yoshihiro Nagano, and Kento Nozawa. On the surrogate gap between contrastive and supervised losses. In ICML, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurlPS, 2013.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 13 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In ICML, 2021.
Grill, Florian Strub, Florent Altché, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, C. Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self-supervised learning. In NeurIPS, 2020.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurlPS, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In ICCV, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. In NeurlPS, 2021.
10

---

Published as a conference paper at ICLR 2023
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):1-13, 2022.
Kento Nozawa and Issei Sato. Understanding negative samples in instance discriminative self- supervised representation learning. In NeurIPS, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0374 2018.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In EUSIPCO, 2007.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICMI., 2021.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In COLT, 2021.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In CVPR, 2022a.
Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and
Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurlPS, 2021.
Yifei Wang, Qi Zhang, Yisen Wang. Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. ICLR, 2022b.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In ICML,, 2021.
Zixin Wen and Yuanzhi Li. The mechanism of prediction head in non-contrastive self-supervised learning, arXiv preprint arXiv:2205.06222 2022.
Xi Weng, Lei Huang, Lei Zhao, Rao Muhammad Anwer, Salman Khan, and Fahad Shahbaz Khan. An investigation into whitening loss for self-supervised learning. arXiv preprint 22.00:2210.03586, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for ImageNet training. arXiv preprint arXiv: 708803888, 2017.
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X Pham, Chang D Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self- supervised contrastive learning. In ICLR, 2022a.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2022b.
11

---

Published as a conference paper at ICLR 2023
A OMITTED PROOFS
In this section, we present proofs for all lemmas and theorems in the main paper.
A. PROOF OF THEOREM 1
Proof. Since the output of fo is no-nommalieed and - -Ra+s(( ,s, we have Laym (0)
EE f((x)) -E - (f((=)((=)))))))))))(()) (fa (1) fo fo(x + 1|011
and (xr(f(x)) = [x]1(())1 - ||ul = |u1|s Thus, we conclude that faym (0)
-E2,, fo(x) fo(xt) Var(f(x)) +1.
L
A.2 PROOF OF LEMMA T
Proof. Let 2x W and Bot Wyxt the loss functions is
, W) = R2, 1
[ sg((=+)))) )1(
(8)
2
Be,x 2
(Waz +g((z+)) 1)² (Wzz. nα +8b(Zz+))
(9)
-g (3) W We sg((z+) ( W2 - 2n W sg(2x+) sg(zu+)) sg(2z+))
(10)
Ex (Tr(z) W W=z) 2T (sg(zxt) W2z) Tr (sg(ze+) sg((z+))) ))]
(11)
2
2 Ex,ut [Tr (W W=nz)) - -T (W on sg(zzt) + Tr (g((=))))((Zz)))
(12)
2 Tr (W WE.2+2]) - -2 Tr (WE, EWE,,,+222 222 + Tr
E E 1xxt2++2 2n+ =[]]]
(13)
Notice that
C. = B = 2x =I- 1
(14)
(15)
WyFEgBelRTT T WJ
(16)
-WBBx (ix T + +221) W7
(17)
=1+
(18)
and
C Fy, 2
(19)
EgEg, =
(20)
Eg (E 2 (Erja w
(21)
-WTE WT
(22)
W.W,.
(23)
Hence,
L(W,, W) [1+ Tr (WT - 2
(WW,))))
(24)
Taking partial derivative with respect to W, get
ac(W,, W) (1 + o ww,, -WW
(25)
oW
With additional weight decay, we have
aL(W, W)
= WW1+ - -W, + +nW,
(26)
dw
12

---

Published as a conference paper at ICLR 2023
where 7 0 is the coefficient of weight decay. Suppose the spectral decomposition W is W, VAVT, where V an orthogonal matrix and A diagonal matrix consisting of decending eigenvalues "1,-.. dg. Therefore.
C, (1 W,, V(1 +DV =VAVV,
(27)
C+ By, 22 W = VAVT.
(28)
Let @C(W) 0, get W W (11+ W + nl) VA V where Au diag ( - 0²) + n). >c(((1+ n)). It follows that
Cp E.PaP W' _ W
Cr,, =V VAPVT.
(29)
Hence, Cp. Cz2 C. share the same eigenspace.
A.3 PROOF OF LEMMA 2
Proof. By the definition of pz W 2z. know that
Cp Cp=EappP E. W 2 W W, (Enre22)) W W
(30)
Since Wg Vg(A.)VV C. VA,T Cp -VAVV and g(1)) /18/2 we have
C - -=
= Cp.
(31)
A.4 PROOF OF LEMMA 3
Proof. For the first equality, we have
C+ Eg 2x=
EgE.a (1 2x Eg (Ezj22x) (Enj 2x )
Eg2221 C.
(32)
For the first part,
E + Vaja = 2 + Ex
Eujo (2z - 23) (Zz 2z)"|
(33)
Ea T + Be
En E0 (Bar)= 2 za (Ex)aza) T + 22
(34)
2
Eg 2 T
(35)
(36)
Cz.
(37)
A.5 PROOF OF THBOREM 2
Proof. Similar to the derivation in the proof of Lemma I
we have
C(W) E W {1(
(38)
Tr (W [
2T (WE, [WBe,+*a2)) + +r (Btt
(39)
2
Notice that C, 2T
E 2 1 2T1 and
B 2 = E, 22
(40)
= Ex (x) (Bue Zzz
(41)
= Ez T
(42)
Cz.
(43)
13

---

Published as a conference paper at ICLR 2023
Hence,
((W)= - 1
2TH Tr (WTWW,,))) - (WC) Tr (C.)).
(44)
Taking partial derivative with respect to W, we get
OL(W)
WC: C. C.
(45)
aw
Let 8C(W) BW
O, we have W = -C,1 Since C, C have aligned eigenspace V, we have
VÄV + Vaf= = VA
(46)
V2 = V - -A,-A)
(47)
which implies that the corresponding eigenvalues satisfy & = X2 b2 2 where X E, denote the i-th eigenvalues of C, Vaja- respectively. And OC2 = VVAT (VA.VT)~ VVA- VΩVT, where is a diagonal matrix and 2g = A E [0, 1] 1.2....,k k.
A.6 PROOF OF THEOREM 3
Before the proof of Theorem first introduce the following useful lemma.
Lemma 4. Assume that Σ q 1 and q1 2 92 z 2 9k 0. then for any 1 Si < j
S
and any A E (0,pj), it holds that
H(1 42 .2......k)) > (lq 2i + +...... 9j - A...., 96).
(48)
Proof. We first note that
... > H(g........ 9i + A, ..... - A A.....
(49)
q: loo(())))) 93 log(4) (4) > -(: A) log((: + A) (0) A) log( A).
(50)
Define f(x) (q: + 2) log(g: + +z) (g; x) log(g; x) and its first order derivative satisfies
f(xx) = log(g. + x)) + log(4, - x) < 0, Vo E (0, 9).
(51)
Hence, Equation 50 holds. This completes the proof of the lemma.
Proof of Theorem
k, then Ek= of Σi=1 df 1, oi 2 q 2 2 oe and of 2 q 2 ... 2 q.. Without loss of generality,
3 Let of
x//(2 xF) and of x//((t XP), where i = 1,2, ..,...
i-1
we assume that oh qh > O. Because g(^;) = / is monotonically increasing and A 2 \j.
for any 1 i i < j S k, we have
L X
X
X
M/C Xi of X//CE/1 XF) 9
(52)
X/
X3
If g(f)) is constant, it follows that
L o
visisssk.
(53)
0
9j
Combining with Σ oi Dh of 1. we get of = - -1,,,..,,. Hence,
H H(g o..... pi) = H H(i (.....,
(54)
which implies that erank(C,)) = erank (C.).
If g(F)) is non-constant, then g(Af) > 9 (A))). And it follows that
y
qi
(55)
o
14

---

Published as a conference paper at ICLR 2023
Arming with Equations 522 and (55
we get
00
L
(56)
9
T
1=1
t=1
1=1
1=1
which indicates oh < g Similarly, we have of > qì- Hence, m max 2 q. - 1,2.
k}
exists and m < L. Using Equation 52 we have
c 2 ifisism, c <
(57)
if in+ k.
Directly applying Ex= q. = Σx= Kf = 1 gives that
Σ mm+1
-
(58)
20
According to Lemma and Equation (58), we transport the redundancy from right to left, the entropy of the distribution will decrease. The transportation process is described as following:
Step Let <- 1 and] j k.
Step If A min { . 93 og) > then of -g + + and qz + 93 A.
Step If q. = q + --i 1. Else j + j 1.
Step 4 if 2 j we finish this process. Else, we return step 2.
After most l: - 1 loops, .E........ gf) becomes (gf, 2 ....... Equation 58 correctness of the transportation process. According to Lemma 4 we have
ensures the
H H
> H(p &
(59)
which implies that er,) < erank(C.).
A.7 PROOF OF THBBRRE 4
Proof. Since ps (t+1) = Ps - ax(ps - (1) = (1 a)pe (2) + taz and Pa = Wzz.
(0)
c((11) Eypp! (p!(ttD))
(60)
Ex,,rt (1- tae (1- + az2
(61)
= - -I,(1-a) aa) (1)
(150)
+ =
(1()
(2ot)
+ + a - a)
p(0)) (0)
+
(0)
(p())
(62)
= (1 - Expp!
(1)
(p!0)
+a²
(1)
(2x))
+ aa a) WE WE (219) + E Ey (t)
(2())
(63)
(1 a) 20(()) +z a(1 a) (WC+ CT
(64)
It follows that
VA = - (1-a) V + +VVVALVV + 2a(1 - VAAA
(65)
As result, we have
A (++1) = (1 - -)) + + 20(1 A
(66)
i.e.,
X No1 (1 (1-a)3 22)2 a N² 20(1 -.
(67)
15

---

Published as a conference paper at ICLR 2023
Dividing both sides of Equation (67 by No,,, we have
P 11+1
Nor
1
= (1 - a)² + a
+2o=11 -a)
(68)
Aite
a1 a 2a(1 - a))os
(69)
A where he = 1/2%,
X,,
If W = W* cff(cc (c! then (A,,) =
Mi. X,,
Ma-- N.e = 1 a² (2,)
.
Hence,
is monotonically decreasing and non-constant.
If XA, = then N (L-a)² ( - + 2a(1 (),) is monotonically X
decreasing and non-constant (0 <a<
According to Theorem
we have crank (C,n))()))))))))))))))))))))))) > erank(C,..))
B EXPERIMENTAL DETAILS
In this section, we provide the details and hyperparameters for SymSimSiam and variants of spectral filters.
B.I DATASETS
We evaluate the performance of our methods on four benchmark datasets: CIFAR-10. 10, CIFAR-100 KKrihevskyy et al. 2009), ImageNet-1000 and ImageNet-I1 Ik (Deng et al. 2009). CIFAR-10 and CIFAR-T01) small-scale datasets, composed of 32 x 32 images with 10 and T00 classes, respec- tively. ImageNet- 100 is a subset of ImageNet- Ik containing 100 classes.
B.2 IMPLEMENTATION DETAILS
Unless specified otherwise, we follow the default settings solo-learn (da Costa et al. 2022) on CIFAR-10, 10, CIFAR- 100 and ImageNet-1066 100. For ImageNet, our implementation follows the official code of SimSiam (Chen & He 2021), and with a learnable linear predictor is adopted as our baseline. With the original projector, SimSiam with a learnable linear predictor could not work, so we delete the last BN in the projector in this case. And we also list the results of SimSiam with the default predictor (we refer as a learnable nonlinear predictor).
the same settings. For fair comparison, SimSiam
Data augmentations.
The augmentation pipeline RandomResizzddCrp with scale in (0.2, 1.0), RandomHHoizoontlliii with probability 0.5, ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4), hue (0.1)) with probability 0.8 and RandomGray with probability 0.2. For ImageNet-100 100 and ImageNet- 1k. Gaussian blurring (Chen al. 2020)) with an applying probability 0.5 is also used.
Optimization. SGD is used the optimizer with momentum 0.9 and weight decay 1.0 10-5 (1.0 x 10-4 for ImageNet- 1k). The learning rate adopts the linear scaling rule (IrxBatchSizee256) with a base learning rate of 0.5 (0.05 for ImageNet- 1k). After 10 epochs of warmup training, we use the cosine learning rate decay (Loshchilov & Hutter 2017). We use a batch size 256 CIFAR-10 and CIFAR-100; 128 on ImageNet- and 512 on ImageNet-Ik.
Linear evaluation. For the linear evaluation, we evaluate the pre-trained backbone network by training a linear classifier on the frozen representation. For CIFAR-10, CIFAR-100 and ImageNet- 100, the linear classifier is trained using SGD optimizer with momentum = 0.9, batch size = 256 and initial learning rate = 30.0 for 100 epochs. The learning rate is divided by 10 at epochs 60 and 80. For ImageNet-Ik lk, following the official code of SimSiam (Chen & He| (20T), we train the linear classifier for 90 epochs with LARS optimizer (You et al., 2017) with momentum = 0.9, batch size 4096, weight decay O. initial learning rate 0.1 and cosine decay of the learning rate.
16

---

Published as a conference paper at ICLR 2023
B.3 EXPERIMENTAL SETTING FOR FIGURES
For the implementations of B YOO, SimSiam, SwAV and DINO on CIFAR-10 10 and CIFAR- 100, we use codes and all default settings in the open-source library solo-learn (da Costa et al. 2022 For SWAV, we do not store features from the previous batches augment batch features (queue size 0) for the consistency of training loss. We adopt ResNet-18 as the backbone encoder. The dimensions of the outputs of BYOL, SimSiam, SwAV and DINO 256, 2048, 3000 and 4092, respectively.
In Figure 3, we use the alignment metric to measure the eigenspace alignment of the online and target outputs, i.c., Cp and C.. The intuition is that, given u; as an eigenvector of Cs, if it is also an eigenvector of Cp. then Cpu; X/u; is in the same direction as u,. Thus, the higher the cosine similarity between U4 and Cpus, the more aligned between the eigenspace of Cp and C.. Formally, we define the alignment between the eigenspace of Cp and C2 as
u Lull2
Cpus
Alignment C=)
(70)
ICpuill2*
where u; is eigenvector corresponding to the i-th largest eigenvalue A of C.. 1,2..... And m is set to 512 for SimSiam, SwAV and DINO and 128 for BYOL. The sum of the first m eigenvalues is greater than 99.99% of the sum of all eigenvalues. Therefore, we can think that the space span by the first eigenvectors is good approximation to the original eigenspace. And in Figures 4 g the first 512 point pairs are displayed for SimSiam, SWAV and DINO (128 for BYOL).
B.4 PSEUDOCODE
Here, we provide the pseudocode for SymSimSiam (Algorithm T and variants of spectral filters (Algorithm{).
Algorithm SymSimSiam: Pseudocode in a Py Torch-likee like style.
# f: backbone projection mlp
c: center (1-by-k)
m: momentum 0.9
for x in loader:
load a minibatch x with samples
xl, , x2 aug (x) (x)
# two random augmentations
zl. z2 fxxl), f(x2)
# projections, n-by-k
zl = normalize (zl, dim=l)
12 n2-normalize # 12-normalize
z2 = normalize(2) dim=l)
update opdate_center(( (cat (zl 22))
z1, z2 zl-c, z2-c
loss = (2l1 ) sum ( (dim=] 1). 1)). mean ()
loss backward update (f. params)
# SGD update
@ torchh. no-grad (
def update center
= (1-m) z. mean (dim=0)
C ADDITIONAL RESULTS BASED ON BYOL FRAMEWORK
In the main paper, we conduct experiments based on the SimSiam framework. Here, we also gather some results variants target high-pass filters based on the B YOL framework. We adopt ResNet- 18 as the backbone encoder and use the default setting in da Costa et al. (2022).
Table shows that our target high-pass filters can often outperform BYOL with a large margin at earlier epochs. In particular, on CIFAR-10, 10, the target filter h(a) o- 0.3 improves BYOL with learnable linear predictor by 4.92%, 2.68% and 1.00% with 100, 200 and 400 epochs, respectively.
17

---

Published as a conference paper at ICLR 2023
Algorithm 2 Variants of spectral filters: Pseudocode in a PyTorch-like style.
f: backbone projection mlp
# g: spectral filter location: the location of spectral filter (online target)
for x in loader:
load a minibatch x with samples
xl, x2 axg(x), (x)
two random augmentations
zl. z2 f(xl), f(x2)
projections, n-by-k
loss L(z1, z2, location 1/2 L(22 localoon )/2
loss backward () update (f. params)
SGD update
def L(p, 2, location):
loss function online predictor
if location = " "online": with torch gr- ():
u. s, vh (p)
w = @ diag()(s)) @ vh
p = p w.w.ddetchh
else:
target predictor
with torch no go.grad(()
u. 8, vh = sz)
z ==u@ (iag(gl(s))
@ vh # gl()) (t)
detach ()
stop gradient
p = normalize (p, dim=l)
12 normalize
z = normalize (z, (z, dim=1)
# -ormalile
return -(pzz) sum (dim=1). mean ()
Table 4: Linear probing accuracy (%) of BYOL with different target predictors on CIFAR-10/100.
CIFAR10
CIFAR100
Predictor
100ep
200ep
400ep
100ep
200ep
400ep
BYOL (nonlinear predictor)
82.14
87.89
91.24
53.92
60.75
67.41
BYOL (linear predictor)
80.81
86.58
89.73
49.65
57.53
60.66
A(o)) -0.3
85.73
89.26
90.73
57.56
61.89
65.88
h(a 0.5
85,46
89.24
90.30
58.12
62.51
65.16
h(or O 0.7
85.08
88.21
90.01
58.20
62.58
65.72
M(a) = -
84.64
88.29
89.96
58.40
62.06
65.04
On CI0,R-100, the filter (((a) = a is better 8.75% than the baseline with 100 epochs, the filter
h(o) = o improves the baseline by 5.05% with 200 epochs and h(()) = a -0.3 improves the baseline by 5.22% with 400 epochs.
D COST COMPARISON
We compare the training speeds and GPU memory usages of different methods (SimSiam, the online filter g(or log(1 1 + a) and the target filter h(o) a -0.3, ) CIFAR-10. We perform our experiments on single RTX 2080ti GPU.
shown Table compared to the original SimSiam, the GPU memory usage of our methods only increase a little (26 MiB). SimSiam and the target filter h(or a -03 have the same training time for 20 epochs.
18

---

Published as a conference paper at ICLR 2023
Table 5: Training time and memory comparison of different methods CIFAR-10.
Method
Total time for 20 epochs
GPU memory
SimSiam
527
3043 MiB
g(o) log 1 + 0) h(o) 0.3
532
3069 MiB
527
3069 MiB
Table 6: Eigenspace alignment between C₂ and C+.
Method
BYOL
SimSiam
SwAV
DINO
CIFAR-10
0.9897 0.9892
0.9979 0.9972
0,9965 0.9956
0.9734 0.9621
CIFAR-100
E ADDITIONAL VISUALIZATION
E.1 ADDITIONAL EMPIRICAL EVIDENCE FOR EIGENSPACE ALIGNMENT
In the main paper, we have shown that Cp and C, share the same eigenspace. Here, we add empirical evidence for eigenspace alignment between C₂ and C+ on CIFAR-10 10 and CIFAR-100. As shown in Table we can see that all methods have very high alignment between the eigenspace of C2 and C...
E.2 RESULTS ON CIFAR-100
We conduct some experiments CIFAR-100. Figure demonstrates that the target branch always has higher effective rank than the online branch and the rank of the online branch continues to increase after the warmup state in all methods.
In Figure 8(a) we compare the eigenvalues computed from branch outputs. There clear difference in eigenvalues, and the eigenvalue distribution of the target branch is more biased towards larger values. Figure 8(b) shows the spectral filter g(A v Xe /A, where X", X are cigenvalues of online and target correlation matrices Cp. Cz, respectively. The spectral filters of all methods are nearly monotonically increasing w.r.t. X².
E.3 TRAINING DYNAMIC
In Figure 9 we compare the training loss between SimSiam and SymSimSiam on CIFAR-10 and CIFAR-101 We can see that the loss of SimSiam is always larger than SymSimsian and does not consistently decrease, Instead, the alignment loss of SymSimSiam continues to decrease.
306
300
28-
100
18-
Sirge
Sirgt
230
230
Cpach
Epach
Figure 7: The effective rank of the normalized outputs of the online and target branches along the training dynamics on CIFAR-100.
F COMPARISON TO RELATED WORK
Comparison with Tian al. (2021). Although we assume eigenspace alignment as Tian et al. we take very different techniques and arrive at different conclusions, as highlighted below:
Difference Perspectives, Techniques, and Conclusions. As for the goal, Tian al. only consider how predictor helps avoid full collapse. Instead, we first point out that avoiding full collapse is not the key role of asymmetric designs (also achievable with symmetric designs). Thus, focus on the more general dimensional collapse issue and analyze this
19

---

Published as a conference paper at ICLR 2023
DYCL
SmSiiam
SwdW
DWO
Onira
Oetra
Lag opernuu
-2
-22
12-
-25
100
Operinion
Egeraauss 700
(a) Eigenvalues of the feature correction matrices of the online and target outputs.
DYOL
5m33emm
Swil
10-
18-
05-
033
47
0.2
320
3.2
5700 199h
9909
9203
0.991
2701 1317 n951
new. Fice 9803
(b) The corresponding online spectral filter function g(A)) Vx//A.
Figure 8: Rank difference experiments and spectral filters on CIFAR-100.
-0.3
SimSiam SymSimSiam
-.3
SimSiam
SymSimSiam
0.4
-0.4
Training loss
0.5
Training loss
-0.5
-0.6
-0.6
-0.
0.8
-0.8
-0.9
-0.9
100
300
400
200
300
Epoch
Epoch
(a) CIFAR-10
(b) CIFAR-100
Figure 9: Training loss of SimSiam and SymSimSiam on CIFAR- 10 and CIFAR-100.
quantitatively through the change of the effective rank along training. As for the techniques, Tian et al. mainly analyze the linear learning dynamics under strong architectural and data assumptions, while ours focus on the common spectral filtering property that also holds for nonlinear modules and general data distributions. As for the conclusion, we formally show that asymmetric designs will improve effective dimensionality, while Tian et al. only discuss how it avoids full collapse (which is extreme case of dimensional collapse, and a non-full-cole coll-co encoder may still suffer from dimensional collapse).
A unified framework for various asymmetric designs. Tian et al. 's analysis only focuses the predictor in BYOL and SimSiam, and they cannot explain why SWAV and DINO also work without predictors. Our RDM applies all existing asymmetric designs through the unified spectral filter perspective.
General principles for predictor design. Tian et al. propose DirectPred, which is only specific filter. Instead, we point out the core underlying principle, that as long as the online filter is a low-pass filter, it could theoretically avoid feature collapse. We also empirically verify this point by showing that various online low-pass filters can avoid feature collapse.
20

---

Published as a conference paper at ICLR 2023
A More Effective Asymmetric Design through Target Predictor. Based on our RDM theory, we also propose new kind of asymmetric design non-contrastive learning: ap- plying a predictor to the target branch. We show that target predictors achieve better results than online predictors while being more computationally efficient.
Therefore, our analysis improves Tian al. (2021) many aspects and apply to a wider context. And we achieve this with new perspectives and techniques that are quite distinctive from Tian et al. 2021
21


Reviews:
[{"review_id": "YdmciVs9KF", "reviewer": "Reviewer_rcoc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how representation collapse is avoided in non-contrastive methods that employ certain “asymmetrical tricks” rather than using negative samples, despite the existence of trivial solutions.\nIt proposes an explanation through an idea of “rank differential” created by asymmetric designs, i.e. the effective rank of representations from the target branch is consistently higher than that of the online branch, as training progresses.\nThis is empirically verified for methods like BYOL, SimSiam, SwAV, DINO.\nThe paper hypothesizes that this rank differential leads to a continuous increase in the effective rank of representation and thus prevents dimensional collapse; a combination of theory and experiments are used to justify this.\nThe paper further attempts to explain the existence of such a rank differential through a “low-pass filtering” effect of asymmetric designs on the representation spectrum.\nMotivated by these ideas, new non-contrastive variants (that directly set the prediction head) are proposed that achieve comparable performance to existing methods on CIFAR and ImageNet benchmarks.", "summary_of_the_review": "Overall I believe that the paper contributes many interesting insights about non-contrastive methods through the lens of rank differences, and also suggest ways to empirically leverage these findings. However currently I am not convinced about the technical correctness and conclusions about one of the main results (Theorem 4) that aims to prove why rank differential can help avoid dimension collapse, which is one of the main contributions of the paper. Keeping this and some other issues in mind, I am inclined to assign a score of weak reject (keeping in mind all the other contributions).", "strength_and_weaknesses": "**Strengths**\n\nThe paper makes many interesting observations about non-contrastive learning, a promising alternative to contrastive learning that is more computationally efficient and scalable.\n- Symmetric architectures, with feature centering, are shown to avoid complete collapse but not dimension collapse\n- The existence of rank differential in many non-contrastive methods is a new and interesting finding in my opinion\n- The overall story, with eigenspace alignment -> rank difference -> rank improvement seems plausible given the results of the paper. Many claims are supported either by some experiments or some theory, albeit with many assumptions at times to make analysis more convenient.\n  - Eigenspace alignment is empirically verified\n  - Rank difference is empirically verified and justified from eigenspace alignment by further assuming a “low-pass filter” assumption (that is empirically verified)\n  - Finally Theorem 4 proves rank improvement as a consequence of the above two (potentially significant issues with this in weaknesses).\n- The rank differential and low/high pass filtering view inspire new non-contrastive variants that directly set the prediction head rather than training them (similar to DirectPred). These methods perform comparably to previous methods, sometimes outperforming them with fewer epochs.\n\nOverall many interesting insights are provided which can help understand non-contrastive methods better.\n\n\n**Weaknesses**\n\n- My main concern is with Theorem 4 that tries to explain why eigenspace alignment + rank differential can lead to increase in effective rank. In particular, I'm not sure that the result is correct, due to a potential bug in Eq. 22 in the Appendix. While it is assumed that the (unnormalized) covariances $\\Gamma_{t}$ and $\\Gamma_{z}$ of $p$ and $z$ respectively are aligned, that does not imply that $\\mathbb{E} [p^{\\top} z] = \\left(\\Gamma_{p} \\Gamma_{z}\\right)^{1/2}$, as assumed in Eq. 22. (Simple counter example is when $p$ and $z$ are independent, and zero-mean Gaussians say). For the result to hold, you might something stronger that the cross-covariance of $p$ and $z$ are also aligned, which is a much stronger assumption and will change the expression for Eq 4. and Theorem 4 in general.\n\n- Most of the theoretical results are not end-to-end (which is fine) and make many assumptions as the paper goes along, which is sometimes hard to follow. For e.g. why does the assumption in Theorem 5 that $z$ and $\\bar{z}$ have aligned eigenspaces makes sense?\n\nOther comments/questions:\n\n- While Theorem 4 is meant to show that the effective rank keeps increasing, what stops the rank to asymptotically stagnate at a small value and lead to dimension collapse? Furthermore, is there any intuition for why rank differential should always remain high as training progresses (besides empirical evidence)?\n\n- The use of the terms “low/high pass filters” can be confusing at times. If these terms are directly inspired from signal processing (or another field), then the connection should be made clearer. The notation for $g$ and $h$ are also ambiguous, since they are just functions of a single $\\lambda^{z}\\_{i}$ but their definitions also involve $\\lambda^{p}\\_{i}$.\n\n- Figure 4: typo “online branch is more biased towards larger values”. For SimSiam, plot (a) suggests that the ratio of $\\lambda^{p}\\_{i} / \\lambda^{z}\\_{i}$ is not monotonic, but figure (b) doesn’t agree since the ratio seems to be 1 at the ends. Do the spectral filter function plots not include all indices $i$? Is this also true for other plots like Figures 5 and 6?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UHkj_5uubqz", "reviewer": "Reviewer_xJ6R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors provide a new perspective for non-contrastive learning methods. A rank difference mechanism (RDM) theory is proposed to explain how existing non-constrastive learning methods alleviate dimensional feature collapse. More importantly,  RDM theory also provides practical guidelines for designing many new non-contrastive variants. In this paper, authors demonstrate the low-pass filters and high-pass filters for online and target branches respectively. In the experiments, the theory is consistent with the practical settings.", "summary_of_the_review": "Based on the comments above, I am inclined to accept this paper. However, I will also take full account of other reviewers' comments.", "strength_and_weaknesses": "Strength\n1. This paper proposes a novel perspective for non-contrastive learning methods. This perspective can effectively explain the four main approaches in non-contrastive learning.\n2. Some of the more interesting variants were obtained under the new view. These methods are promising and are able to inspire the community.\n3. The experiments are convincing in my opinion.\n\nWeakness:\n1. Although the authors call the method rank difference, and define an effective rank difference to estimate how uniform the eigenvalues are. It is the eigenvalues difference rather than the rank difference. I don't think that's a suitable name.\n2. The eigenspace alignment assumption is a little bit strong, and all conclusions and methods are based on that. This paper is, therefore, more like an extension of ICML2021(Understanding self-supervised learning dynamics without contrastive pairs).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9hB4LSbZrR4", "reviewer": "Reviewer_hvBQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes RDM theory, which is applicable to different asymmetric designs (with and without the predictor), and can serve as a unified understanding of existing non-contrastive learning methods. Besides, the RDM theory also provides practical guidelines for designing many new non-contrastive variants. RDM achieves comparable performance to existing methods on benchmark datasets, and some of them even outperform the baselines.", "summary_of_the_review": "The paper has some merits while I currently still have some concerns and questions as above stated. I would like to see the authors' response.\n\n---post rebuttal:\n\nThe authors have made good job in clarifying their novelty not only to my original concerns but also the other reviewers'. I appreciate the novelty of this work and increase my score from 5 to 6. Yet the performance advantage of this method shall be more carefully and comprehesively discussed. BYOL actually used a non-linear projector while the authors did not show their non-linear version's performance.", "strength_and_weaknesses": "Pros: \n1.\tThe paper analyzes how symmetric and asymmetric frameworks alleviate different collapses (dimensional and complete).\n2.\tRDM designs a new asymmetric design that also work well in practice.\n\nCons:\n1.\tLimited novelty: SymSimSiam is quite similar to the combination of DINO and SimSiam. It seems to me like just borrowing the centering and sharpening techniques of DINO to $l_2$ space (see Algorithm 1).\n2.\tAlignment in eigenspace is similar to whitening operation and the authors may miss some related works [1, 2]. Author may compare RDM with these methods on methodology.\n3.\tThe limited performance on CIFAR and ImageNet. Authors claim the proposed method can get comparable results on these benchmarks. However, I don’t see any result that can lead to this conclusion (For example, with 100 epochs pretraining, RDM gets 85.7% and 58.4% accuracies on CIFAR10 and CIFAR100 datasets). Maybe the authors can comment on this.\n\n[1] Weng X, Huang L, Zhao L, et al. An Investigation into Whitening Loss for Self-supervised Learning[J]. arXiv preprint arXiv:2210.03586, 2022.\n[2] Zhang S, Zhu F, Yan J, et al. Zero-CL: Instance and Feature decorrelation for negative-free symmetric contrastive learning[C]//International Conference on Learning Representations. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster