Modulate Your Spectrum in Self-Supervised Learning
Authors: Xi Weng
Yunhao Ni
Tengwei Song
Jie Luo
Rao Muhammad Anwer
Salman Khan
Fahad Khan
Lei Huang
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learning
whitening
dimensional collapse
spectral transformation
iterative normalization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: The proposed INTL is well motivated, theoretically demonstrated, and empirically validated in avoiding dimensional collapse, and is a promising SSL method in practice.

Abstract: Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL.

Published as a conference paper at ICLR 2024
MODULATE YOUR SPECTRUM IN SELF-SUPERVISED LEARNING
Xi Weng"), Yunhao Ni*!, Tengwei Song', Jie Luo', Rao Muhammad Anwer?, Salman Khan', Fahad Shahbaz Khan Lei Huang X 1 SKLCCSE, Institute of Artificial Intelligence. Beihang University, Beijing, China Mohamed bin Zayed University of Artificial Intelligence, UAE
ABSTRACT
Whitening loss offers a theoretical guarantee against feature collapse in self- supervised learning (SSL) with joint embedding architectures. Typically, it in- volves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally. we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward -qual-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code available https://github..com wttp:///wwwoowww.i/ii///////////
INTRODUCTION
Self-supervised learning (SSL) via joint embedding architectures to learn visual representations has made significant progress over the last several years (Bachman et al., 2019; He et al. 2020 Chen
cet al. [2000a] Chen & He| |02 Bardes et al./ 2022 Oquab et al. 2023 almost outperforming their supervised counterpart on many downstream tasks (Liu et al. 202 Jaiswal et al., 2020 Ranasinghe] iet al. (022). This paradigm addresses to train a dual pair of networks to produce similar embeddings for different views of the same image (Chen & He, 2021). One main challenge with the joint embedding architectures is how to prevent a collapse of the representation, in which the two branches ignore the inputs and produce identical and constant outputs (Chen & He (021). A variety of methods have been proposed successfully avoid collapse, including contrastive learning methods (Wu et al.) He et al. 2020 Saunshi et al. 2019) that attract different views from the same image (positive while pull apart different images (negative pairs), and non-contrastive methods (Grill et al.
2018
pairs
2020
Chen He 202T) that directly match the positive targets without introducing negative pairs.
The collapse problem is further generalized into dimensional collapse (Hua et al,, 2021 Jing et al..
2022) (or informational collapse (Bardes al. 2022)), where the embedding vectors only span divensiinenii subspace and would be highly correlated. In this case, the covariance matrix of embedding has certain zero eigenvalues, which degenerates the representation in SSL. To prevent dimensional collapse, theoretically motivated paradigm, called whitening loss, is proposed by minimizing the distance between embeddings of positive pairs under the condition that embeddings from different views are whitened (Ermolov et al. 2021 Hua et al. (021). One typical implementation of whitening loss is hard whitening (Ermolov 20.] Weng al. 2022) that designs whitening transformation over mini-batch data and imposes the loss on the whitened output (Ermolov et al. 2021 Huua et al.) 2021 Weng et al. (022). We note that the whitening transformation is a function over embedding during forward pass, and modulates the spectrum of embedding implicitly during backward pass when minimizing the objective. This raises questions whether there exist other functions over embedding can avoid collapse? If yes, how the function affects the spectrum of embedding?
This paper proposes spectral transformation (ST), a framework to modulate the spectrum of embed- ding in joint embedding architecture. ST maps the spectrum of embedding to a desired distribution during forward pass, and modulates the spectrum of embedding by implicit gradient update during backward pass (Figure
"qqual contribution
2 correspooniigg author ).n.eccci.ea....cc.c.......

---

Published as a conference paper at ICLR 2024
This framework provides a way to seek for functions beyond whitening transformation that
X: mini-batch
cmboedinn
transformed output
avoid dimensional collapse. We show that whitening transformation is special instance of ST using a power function by definition, and there exist other power functions that can avoid dimensional collapse by our empirical investi- gation (see Section 3.2 for details). We demon- strate that IterNorm (Huang approximating whitening method by using New- ton's 's iterations (Bini et al. 2005 Ye et al., 2020),
Z
F₂()
T-T
Fi !
Z
X3
joint anchisecture
al., 22199) an
Figure 1: The framework using spectral transfor- mation (ST) modulate the spectrum of embed- ding in joint embedding architecture for SSL.
also an instance of ST, and show that IterNorm
with different iteration number corresponds to different ST (see Section 3.2.2 for details). We further theoretically characterize how the spectrum evolves the increasing of iteration number of IterNNom.
We empirically observe that IterNorm suffers from severe dimensional collapse and mostly fails to train the model in SSL unexpectedly, unlike its benefits in approximating whitening for supervised learning (Huang et al., 2019). We thus propose IterNorm with trace loss (INTL), simple solution to address the failure of IterNorm, by adding an extra penalty on the transformed output. Moreover, we theoretically demonstrate that INTL can avoid dimensional collapse, and reveal its mechanism in modulating the spectrum of embedding to be -qual-eigenvalues. We conduct comprehensive experiments and show that INTL is a promising SSL method in practice. Our main contributions are summarized as follows:
We propose spectral transformation. a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show there cxist other functions that can avoid dimensional collapse by empirical observation and intuitive explanation.
We propose new instance of ST, called IterNorm with trace loss (INTL). We theoretically prove that INTL can avoid collapse and modulate the spectrum of embedding towards an equal-eigenvalue distribution during the course of optimization.
INTL's experimental performance on standard benchmarks showcases its high promise as a practical SSL method, consistently achieving or surpassing state-of-the-art methods, even when utilizing a relatively small batch size.
RELATED WORK
Our work is related to the SSL methods that address the feature collapse problem when using joint embedding architectures.
Contrastive learning prevents collapse by attracting positive samples closer, and spreading negative samples apart (Wu et al. 2018: Ye et al. (019). In these methods, negative samples play an important role and need to be well designed (Oord et al. a0.8, Bachman et al. 2019 Henafi 2020 MoCos (He et al. 2020 Chen et al. 2020b) build a memory bank with a momentum encoder to provide consistent negative samples, while SimCLR (Chen et al., 2020a) addresses that more negative samples in batch with strong data augmentations perform better. Our proposed INTL can avoid collapse and work well without negative samples. Additionally, recent work (Zhang et al. 2023) explores the use of data augmentation for contrastive learning through spectrum analysis enhance performance, while our paper focuses on developing a novel non-contrastive method to prevent collapse under standard augmentation.
Non-contrastive learning can be categorized into two groups: asymmetric methods and whitening loss. Asymmetric methods employ asymmetric network architectures to prevent feature collapse without the need for explicit negative pairs (Caron et al. 2018 2020 Li et al., 2021 Grill et al.) 2020; Chen & He 2021). For instance, BYOL (Grill et al. 2020) enhances network stability by appending a predictor after the online network and introducing momentum into the target network. SimSiam (Chen & He, 2021) extends BYOL and emphasizes the importance of stop-gradient to prevent solutions. Other advancements in this realm include cluster assignment prediction using the Sinkhorn-Knon algorithm (Caron et al.) 2020) and the development of asymmetric pipelines with self- self-distillation losses for Vision Transformers (Caron et al., 2021). However, it remains unclear how these asymmetric networks effectively prevent collapse without the inclusion of negative pairs. This has sparked debates surrounding topics such as batch normalization (BN))eetterman & Albrecht 2020 Tian al. 2020bb Richemond et al. 2020) and gooegraadint(Chen & He| [021]/Zaag et al. 2022a). Despite prelimmnryy efforts to analyze training dynamics (Tian et al. (021) and establish
Chen

---

Published as a conference paper at ICLR 2024
: the exact mechanisms behind these methods remain an ongoing area of research. In our work, we address the more intricate challenge of dimensional collapse and theoretically demonstrate that our INTI. method effectively prevents this issue, offering valuable insights into mitigating feature collapse in various scenarios.
connections between non-contrastive and contrastive methods (Tao et al. 2022 Garrido et al. 2023
Whitening loss is theoretically motivated paradigm to prevent dimensional collapse (Ermolov et al. 202I). One typical implementation of whitening loss is hard whitening that designs whitening transformation over mini-batch data and imposes the loss on the whitened output. The designed whitening transformation includes batch whitening W-MSE (Ermolov al. 202T and Shuffled- DBN (Hua et al. (02I), channel whitening in CW-RGP (Weng et al. 2022), and the combination of both in Zero-CI. (Zhnng et al., 2022b). Our proposed ST generalizes whitening transformation and provides frame to modulate the spectrum of embedding. Our INTL can improve these work in training stability and performance, by replacing whitening transformation with IterNorm (Huang et al.) 2019) and imposing an additional trace loss on the transformed output. Furthermore, we theoretically show that our proposed INTL modulates the spectrum of embedding to be equal-eigenvalues.
Another way to implement whitening loss is soft whitening that imposes a whitening penalty as regularization on the embedding, including Barlow Twins (Zbontar et al. 202T). VICReg (Bardes) al., 2022) and CCA-SSG (Zhang et al. 202T). Different from these works, our proposed INTL imposes the trace loss on the approximated whitened output, providing equal-cigenvaluee modulation on the embedding.
There are also theoretical works analyzing how dimensional collapse occurs (Hua et al. 202 Jing et al. 2022) and how it can be avoided by using whitening loss (Hua et al. [021] Weng et al. 2022). The recent works (He & Ozay] 2022 Ghosh al., 2022) further discuss how characterize the magnitude of dimensional collapse, and connect the spectrum of a representation to a power law. They show the coefficient of the power law is strong indicator for the effects of the representation. Different from these works, our theoretical analysis presents a new thought in demonstrating how to avoid dimensional collapse, which provides theoretical basis for our proposed INTL.
SPECTRAL TRANSFORMATION BEYOND WHITENING
3.1 PRELIMINARY AND NOTATION
Joint embedding architectures. Let x denote the input sampled uniformly from a set of images DD and T denote the set of data transformations available for augmentation. We consider a pair of neural networks Fe and For, parameterized by 0 and 0' respectively. They take input two randomly augmented views, x(1 Ti (x) and x(2) Tx(x), where Tı,2 E T; and they output the embedding z(1) Fe (x) and (2) F² (x)2)), The networks are trained with an objective function that minimizes the distance between embeddings obtained from different views of the same image:
L(x, 0) = Ex TiaNT E (FF((T(x)), For (7x(x))).
where ((:--) is loss function. The mean square error (MSE) of nornmaalized vectors as ((2()) 2))
(1)
x(2)
1/ is usually used as the loss function CCen & He 2021). This loss
is also equivalent to the negative cosine similarity, up scale of 1/2 and an optimization irrelevant
constant (Chen & He| (021). This architecture is also called Siamese Network (Chen & He, 202I).
Fg = Fg.. Another variant distinguishes the networks into target network F% and online network Fa. and updates the weight 6' of target network through exponential moving average (EMA) (Chen et al. 2020b Grill et al., 2020) over of online network.
Feature collapse. While minimizing Eqn. trivial solution known as complete collapse could occur such that (x( = c, Vx D. Moreover, weaker collapse condition called dimensional collapse can be easily arrived, for which the projected features collapse into a fow-dimensional manifold. To express dimensional collapse more mathematically. we refer to dimensional collapse as the phenomenon that one or certain eigenvalues of the covariance matrix of feature vectors degenerate to O. Therefore, we can determine the occurrence of dimensional collapse by observing the spectrum of the covariance matrix.
Whitening loss. To address the collapse problem, whitening loss (Ermolov et al. 2021 is pro- posed to minimize Eqn. [ under the condition that embeddings from different views are whitened. Whitening loss provides theoretical guarantee in avoiding (dimensional) collapse, since the embed- ding is whitened with all axes decorrelated (Ermolov et al. 2021 Hua et al. (021). Ermolov et
3

---

Published as a conference paper at ICLR 2024
al. (Ermolov et al. 2021) ) propose to whiten the mini-batch embedding Z E Rdxm using batch whitening (BW) (Huang et al., 2018 Siarohin et al., 2019) and impose the loss on the whitened output Z E Rdxm given the mini-batch inputs X with size of m, as follows:
min (); Ex~D, TT 12 - 1
with 2 -
= Σー z E (1,2),
where Σ
m ZZ" is the covariance matrix of embedding Σ 1 is called the whitening matrix, and calculated either by Cholesky decomposition in (Ermolov et al. (02T) or by eigen-decomposition
in (Hua et al. (021). E.g., zero-phase component analysis (ZCA) whitening (Huang et al. 2018) calculates Σ 2 UA = UT, where A diag(A1...... Aa) and U [u,,.., U.d are the eigenvalues and associated eigenvectors of Σ, i.e. UAU = Σ. One intriguing result shown in (Weng et al. 2022) is that hard whitening can avoid collapse by only constraining the embedding Z to be full-rank, but not whitened.
We note that the whitening transformation is a function over embedding Z during forward pass, and modulates the spectrum of embedding Z implicitly during backward pass when minimizing MSE loss imposed on the whitened output. This raises a question of whether there are other functions over embedding Z that can avoid collapse? If yes, how the function affects the spectrum of embedding Z?
3.2 SPECTRAL TRANSFORMATION
In this section, we extend the whitening transformation to spectral transformation (ST), more general view to characterize the modulation on the spectrum of embedding, and empirically investigate the interaction between the spectrum of the covariance matrix of Z and collapse of the SSL model.
Definition 1. (Spectral Transformation) Given any unary function g(-) in the definition domain X(Z) = {>1, b2,,A,,. .e......uu >a)- Drawing an analogy with whitening, g(-) on the covariance matrix E of embedding Z is defined as g(E) Ug(A)UT, where g(A) diig(g(X())))- We denote the transformation matrix of ST as ØT = g(Z), so that the output of ST is calculated by Z DSTZ=
Ug(A)U"Z Z and the covariance matrix of Z is Ei = -2 = UAg (A)U?.
Based on Definition 1 ST is an abstract framework until g(-) is determined, and its essence is mapping the spectrum A(Z) A(2 (A1g²(An) (X2), >ag² When applied in the context of self-supervised learning, the loss function for ST remains the same as Equ 2 with the only difference being that Z is determined by 9()). Meanwhile, the optimization direction for the embedding spectrum can also be determined when employing gradient- based methods. That is, what spectrum of embedding will be modulated to be during the course of training.
Can we unveil the potential of ST? Our ST framework exhibits uncertainty and diversity, allowing g(-) adopt the guise of any single-variable function within the defined domain, including power functions, exponential functions, iterative functions, and more. Whitening, on the other hand, is a
special and successful instance within ST, where g(-) takes the form of a power function g(A) = 1 This naturally prompts two questions: 1. Could there be other functions, akin to whitening, capable of preventing collapse within the ST framework? If yes, how the function works and affects the spectrum of embedding Z?
3.2.1 SPECTRAL TRANSFORMATION USING POWER FUNCTIONS
With these questions in mind, we embark on a deeper exploration of the mechanics extending beyond whitening, considering more comprehensive transformation g\) >x- E (-o0, +o0) for ST. Based on Definition. this comprehensive power transformation is mapping the spectrum X(Z) to
x(2 (X1² 1-22 2p A22
2p
Empirical observation. Initially, we conduct experiments on 2D dataset, varying the parameter p. and visualize the outputs of the toy models as depicted in Figure a). Our observations indicate that the toy model tends to perform well in avoiding collapse when p falls within the neighborhood of 0.5, specifically in the range of 0.45 to 0.55. However, as p gradually deviates from 0.5, collapse becomes more pronounced. Subsequently, we extend our experiments to real-world datasets to validate these findings. The results presented in Figure (bb) align with the previously observed phenomena. When p is set to either 0.45 0.55, the model maintains high evaluation performance, similar to that
The embedding is usually centralized by performing Z = Z(1 - -1 17) for whitening, and we assume is centralized in this paper for simplifying discussion.
4

---

Published as a conference paper at ICLR 2024
Conditin indiaatio
Accunay
p-04s
90 CJ
94
0-6
055
10
P0G C3
94 045
830
935
(a)
(b)
(c)
Figure 2: Investigate ST using power functions. We choose several from 0 to 1.5. We show (a) the visualization of the toy model output; (b) top-1 and 5-nearest neighbors (5-nn) accuracy on CIFAR-10: (c) condition indicator of embedding Z and transformed output Z on CIFAR-10. We use the inverse of the condition number (IoC) in logarithmic scale with base 10 ( UgloC Ige 2 as the condition indicator. The results on CIFAR-10 are obtained through training with ResNet-18 for 200 epochs and averaged over five runs, with standard deviation shown as error bars. We show the details of experimental setup in Appendix D Similar phenomena can be observed when using other datasets (e.g., ImageNet) and other networks (e.g., ResNet-50).
Ig
of whitening (p- 0.5). This discovery suggests that within the ST framework, there exist other functions capable of successfully preventing collapse, which answers the first question.
For the second question, illustrated in Figure ()), it becomes evident that when p lies in the vicinity of 0.5, the embedding showcases more well-conditioned spectrum, characterized by a smaller condition number (larger IoC). However, when p significantly deviates from 0.5, the spectrum of embedding loses its well-conditioned attributes, closely aligning with the occurrence of embedding collapse. This statement asserts that if g()) is effective in preventing collapse within the ST framework, it will result in the modulation of the embedding spectrum towards a well-conditioned state.
Intuitive explanation. We note that (Weng et al.
2022) implied that whitening loss in Eqn 2 can
be decomposed into two asymmetric losses C 1 )eel? where (()) refers to the whitening matrix of Z, st. represents the stop gradient operation, and Z denotes the whitening output. Each asymmetric loss can be viewed as an online network to match whitened target Z. As more generalized form of whitening, our ST can also extend this decomposition to the loss function. As depicted in Figure ((... when p falls within the range of 0.45 to 0.55, Z exhibits a well-conditioned spectrum, with each eigenvalue approaching 1. In such instances, Z serves as an ideal target for Z to match, enabling the embedding Z to learn favorable spectrum to prevent collapse. Conversely, when p deviates significantly from 0.5, the spectrum of the transformed output loses its well-conditioned characteristics, with Z becoming a detrimental target, ultimately leading to the collapse of the embedding.
(( ()z() )ell + || /
3.2.2 IMPLICIT SPECTRAL TRANSFORMATION USING NEWTON'S 'S ITERATION
However, utilizing the power function g() = X (where is approximately 0.5) within our ST framework is not without its drawbacks. One issue is the potential for numerical instability when computing eigenvalues À and eigenvectors U via eigen-decomposition, particularly when the covariance matrix is illccondt (Paszke et al. 2019). We provide comprehensive experiments and analysis in Appendix DD3 to validate the presence of this problem in SSL.
Naturally, if we could implement a spectral transformation that can modulate the spectrum without the need for explicit calculation of X or U, this issue could be mitigated. In fact, we take note of an approximate whitening method called iterative normalization (IterNorm) (Huang al., 2019). which uses Newton's 's iteration to address the numerical challenges associated with batch whitening in supervised learning. Specifically, given the centered embedding Z. the iteration count T. and the trace-normalized covariance matrix ΣN E/tr(E), IterNorm performs Newton's 's iteration as follows.
Po = P = 2 - - P k = =,,,....,
The whitening matrix Σ- approximated by का = P PT/vtr( (t)) and we have the whitened output = Z. When T - to. Φ7 + Σ and the covariance matrix of 2 will be an identity matrix. Here, we theoretically show that IterNorm is also an instance of spectral transformation as follows. Theorem 1. Define one-variable iterative function fr( satisfying
fx+1(x)= = =f foff 3 0: fo(x) =
5

---

Published as a conference paper at ICLR 2024
The mapping function of IterNorm is g(A) = JT( ( / [r)). IterNorm implicitly maps VA, E X(Z) to X4 = (r(E) de fr (r)E)))
Without calculating À or U,
The proof is provided Appendix B.1 For simplicity, we define the T-whitening function of IterNorm (x) = x xfr" (x), which obtains the spectrum of transformed output. Based on the fact that the covariance matrix of transformed output will be identity when ''' of IterNorm increases to infinity (Bini et al. 2005), thus have
2Y > O, lim hr tr(E)) T-000
=1.
Different iteration numbers T' of IterNorm imply different T-whitening functions hr((). It is interest- ing analyze the characteristics of hr(().
Proposition 1. Given x E (0, 1). v7 E N we have lr((z) E (0, 1) and (1) > 0.
The proof is shown in Appendix A.1 Proposition states hr is monotone increasing function A (r(z)
for E (0, 1) and its range is also in (0, 1). Since
E (0, 1), AY, 0, we have
VT E N, > N3 > - X, Xy > 0.
(5)
Formula indicates that IterNorm maps all non-zero eigenvalues to (0, 1) and preserves monotonicity. Proposition 2. Given x E (0, 1), vT E N, we have h)+1(2) > (x))..
The proof is shown in Appendix A.2
Proposition 2 indicates that IterNorm gradually stretches the eigenvalues towards one as the iteration number T increases. This property of IterNorm theoretically shows that the spectrum of Z will have better condition we use a larger iteration number of IterNorm.
In summary, our analyses theoretically show that IterNorm gradually stretches the eigenvalues towards one as the iteration number T increases, and the smaller the eigenvalue is, the larger required to approach one.
ITERATIVE NORMALIZATION WITH TRACE Loss
expected that IterNorm, as a kind of spectral transformation, can avoid collapse and obtain good performance in SSL, due to its benefits in approximating whitening for supervised learning (Huang al. 2019). However, we empirically observe that IterNorm suffers severe dimensional collapse and mostly fails to train the model in SSL (we postpone the details in Section 4.2)). Based on the analyses in Section 3.2 and 3.2.2] we propose simple solution by adding extra penalty named trace loss on the transformed output Z by IterNorm to ensure well-conditioned spectrum. It is clear that the sum of eigenvalues of Vig is less than or equal to d, we thus propose a trace loss that encourages the trace of Eg to its maximum d, when S m. particular, we design a new method called IterNorm with trace loss (INTL) for optimizing the SSL model as?
d INTL(Z) Σ (1 -E
min AEB
(6)
j=1
where Z F-)) and Ż Itte Eqn. can be viewed as an optimization problem over to encourage the trace of Z to be d.
4.1 THEORETICAL ANALYSIS
In this section, we theoretically prove that INTL can avoid collapse, and INTL modulates the spectrum of embedding towards an equal-eigenvalue distribution during the course of optimization.
d
Note that Σg can be expressed using the T-whitening function hr(()) as Eg = Σ (Z.
where x, = (x/tr(E) 2 0 and Ii 1. When the range of Fo(-) is wide enough, the optimization
"hhe complete loss function of INTL LINTI. LMSE + B Ltrases where the coefficient 9 fixed across all datasets and architectures, and its determination is elaborated in Algorithm of INTL' of Appendix To simplify the discussion, we omit the LMSE term here, without compromising the validity.

---

Published as a conference paper at ICLR 2024
222
16stagnaa
Top poatco
Hightsake tnks
Epecss
211
1:22
1-3
(a)
(b)
(c)
Figure 3: Investigate the effectiveness of IterNorm with and without trace loss. We train the models on CIFAR-10 10 with ResNet-18 18 for 100 epochs. We apply IterNorm with various iteration numbers T. and show the results with (solid lines) and without (dashed lines) trace loss respectively. (a) The spectrum of the embedding Z: (b) The spectrum of the transformed output Z: (c) The top-1 accuracy. (d) indicates that IterNorm (without trace loss) suffers from numeric divergence when using large iteration number, e.g. 7 = 9. It is noteworthy that when T > 11, the loss values are all NAN, making the model unable be trained. Similar phenomena can be observed when using other datasets (e.g. ImageNet) and other networks (e.g., ResNet-50).
problem over (Eqn. 6)) can be transformed the following optimization problem over x (Eqn. without changing the optimal value (please see Appendix
2
for the details of derivation):
min
INTL(x) Σ
Σ 1 (x:)),,
d
s.t.
Σ
T2 2 0, d,
where újs is the j-th elements of vector u; In this formulation, we can prove that our proposed INTL can theoretically avoid collapse, as long as the iteration number T IterNorm larger than zero
Theorem 2. Let x [0, 114, YT E N4. INTL(x) shown in Eqn. is strictly convex function. x" = 12, 112T is the unique minimum point well the optimal solution INTI(x).
The proof is provided in Appendix B.2 Based on Theorem 2 INTL modulates the spectrum of embedding be -qual-eigesvllues during the backward pass, which provides theoretical guarantee to avoid dimensional collapse.
Connection to hard whitening. Hard whitening methods, like W-MSE (Ermolov et al. 2021 and shuffle-DBN (Hua et al. 2021). design whitening transformation over each view and minimize the distances between the whitened outputs from different views. This mechanism modulates the covariance matrix of embedding to be full-rank (Weng al., (022). Our INTL designs an approximated whitening transformation using IterNorm and imposes an additional trace loss penalty on the (approximately) whitened output, which modulates the covariance matrix of embedding having equal eigenvalues.
Connection to soft whitening. Soft whitening methods, like Barlow-Twins (Zbontar et al. 2021 and VICReg (Bardes et al., (022) directly impose whitening penalty as regularization on the embedding. This modulates the covariance matrix of the embedding to be identity (with fixed scalar Y,, e.g., <I). Our INTL imposes the penalty on the transformed output, but can be viewed as implicitly modulating the covariance matrix of the embedding to be identity with free scalar (i.e., having equal eigenvalues).
Intuitively, INTL modulates the spectrum of embedding to be equal-eigenvalues during the backward pass, which is stronger constraint than hard whitening (the full-rank modulation). but weaker constraint than soft whitening (the whitening modulation). This preliminary but new comparison provides a new way to understand the whitening loss in SSL.
4.2 EMPIRICAL ANALYSIS
In this section, we empirically show that IterNorm-only and trace-loss-only fail to avoid collapse, but IterNorm with trace loss can well avoid collapse.
IterNorm fails avoid collapse. In theory. IterNorm can map all non-zero eigenvalues to approach one, with a large enough T. In practice, it usually uses a fixed T, and it is very likely to encounter small eigenvalues during training. In this case, IterNorm cannot ensure the transformed output has a well-conditioned spectrum (Figure 3()), which potentially results dimensional collapse. One may use large T', however, IterNorm will encounter numeric divergence upon further increasing the iteration number T, even though it has converged. E.g., IterNorm suffers from numeric divergence in Figure d) when using = 9, since the maximum cigenvalue of whitened output is around
7

---

Published as a conference paper at ICLR 2024
Table 1: Classification top-1 accuracy of a linear classifier and a 5-nearest neighbors classifier for different loss functions and datasets. The table is mostly inherited from solo-learn (da Costa
al. 2022). All methods are based on ResNet- 18 with two augmented views generated from per
sample and are trained for 1000-epoch on CIFAR- 10/100 with a batch size of 256 and 400-epoch on
ImageNet-100 100 with a batch size of 128.
CIFAR-IÓ
CITAR- 100
ImageNet-Iun
Method
top-1
5-nn
top-1
5-nn
top-1
5-nn
SimCLR (Chen et al.
202(aa)
90.74
85.13
65.78
53.19
77.64
65.78
MoCo V2 (Chen a
22220)
92.94
88.95
69.89
58.09
79.28
70.46
BYOL (Grlll al. 2020)
92.58
87.40
70.46
56.46
80.32
68.94
SwAV CCaron
2020)
89.17
84.18
64.88
53.32
74.28
63.84
SimSiam (Chen He
1022
90.51
86.82
66.04
55.79
78.72
67.92
W-MSE (Emolov
2002
88.67
84.95
61.33
49.65
69.06
58.44
Shuffled-DBi (Hua
2021
91.17
88.95
66.81
$7.27
75.27
67.21
DINO (Caron ei al. 2021
89.52
86.13
66.76
56.24
74.92
64.30
Barlow Twins (Zborila el al. 2021
92.10
88,09
70.90
59.40
80.16
72.14
VICReg (Bardes ct al.
2022
92.07
87.38
68.54
56.32
79.40
71.94
Zero- CL (Zhang ct al. 2022b
90.81
87.51
70.33
59.21
79.26
71.18
CW-RGP (Weng टt al.
2022
92.03
89.67
67.78
58.24
76.96
68.46
INTL (ours)
92.60
90.03
70.88
61.90
81.68
73,46
Table 2: Comparisons on ImageNet linear classification with various training epochs. All methods based ResNet-50 backbone with two augmented views generated from per sample. EMA represents Exponential Moving Average. Given that one the objectives SSL methods to achieve high performance with small batch sizes, it's worth noting that our INTL performs effectively when trained with small batch sizes, such as 256 and 512.
Method
Batch size
EMA
100 eps
200 eps
400 eps
800 eps
SimCLR
4096
No
66.5
68.3
69.8
70.4
4096
No
66.5
69.1
70.7
71.8
SWAV
512
No
65.8
67.9
SimSiam
256
No
68.1
70.0
70.8
71.3
W-MSE
512
No
65.1
66.4
Shuffled-DBN
512
No
65.2
Barlow Twins
2048
No
67.7
72.5
73.2
Zero-CL
1024
No
68.9
72.6
CW-RGP
512
No
67.1
69.6
INTL (ours)
512
69,5
73.1
MoCo v2
256
67.4
69.9
71.0
72.2
BYOL
4096
Yes
66.5
70.6
73.2
74.3
INTL (ours)
256
Yes
69.2
71.5
73.7
74.3
107, significantly large than 1 (we attribute to the numeric divergence, since this result goes against Proposition and 2 and we further validate it by monitoring the transformed output). noteworthy that when T II, the loss values are all NAN, making the model unable to be trained. These problems make IterNorm difficult to avoid dimensional collapse in practice.
The Synergy between IterNorm and trace loss. IterNorm in combination with trace loss demon- strates significant differences compared to IterNorm-only. Our experimental results, as shown in Figure (aa), empirically confirm that INTL effectively prevents dimensional collapse, aligning with the findings of Theorem 2 INTL encourages the uniformity of eigenvalues within the covariance matrix of the embedding Z. resulting in well-conditioned spectra for the transformed output (Fig- ure (bb) and impressive evaluation performance (Figure (c)) even when the iteration count T is as low as 1. To further evaluate the performance of trace-loss-only. we conducte experiments under the same setup. Without IterNorm, prace-loss-only achieves a top-1 accuracy of only 16. 16.15%, indicating significant collapse. Therefore, the efficacy of INTL, well the attainment of optimal solution characterized by equal eigenvalues, is result of the synergy between IterNorm and trace loss.
5 EXPERIMENTS ON STANDARD SSL BENCHMARK
In this section, we conduct experiments on standard SSL benchmarks to validate the effectiveness of proposed INTL. We first evaluate the performance of INTL for classification CIFAR- 10/100 (Krizhevsky) 2009), ImageNet- 100 (Tian et al., 2020a). and ImageNet (Deng al.] 2009). Then we evaluate the effectiveness in transfer learning, for pre-trained model using INTL. We provide the full PyTorc--style algorithm in Appendix well details of implementation and computational overhead in Appendix

---

Published as a conference paper at ICLR 2024
Table 3: Transfer Learning. All competitive unsupervised methods are based on 200-epoch pre- training on ImageNet (IN). The table are mostly inherited from (Chen & He 202TD. Our INTI. is performed with 3 random seeds, with mean and standard deviation reporied.
COCO detection
COCO instance seg.
Method
AP5e
AP
AP
APs0
AP
AP76
Scratch
44.0
26.4
27.8
46.9
29.3
30.8
Supervised
58.2
38.2
41.2
54.7
33.3
35.2
SimCLR
57.7
37.9
40.9
54.6
33.3
35.3
MoCo v2
58.8
39.2
42.5
55.5
34.3
36.6
BYOL
$7.8
37.9
40.9
54.3
33.2
35.0
SWAV
57.6
37.6
40.3
54.2
33.1
35.1
SimSiam
57.5
37.9
40.9
54.2
33.2
35.2
W-MSE (repro.)
60.1
39.2
42.8
56.8
34.8
36,7
Barlow Twins
59.0
39.2
42.5
56.0
34.3
36.5
INTL (ours)
60.9
40.7+0.09
43.7
40.17
57.3+0.08
35.4
37.6
5.1 EVALUATION FOR CLASSIFICATION
Evaluation on small and medium size datasets. We initially train and perform linear evaluation of INTL using ResNet-18 18 as the backbone on CIFAR-10/100 (Krizhevsky 2009) and ImageNet- 100 (Tian et al. 2020a). We strictly adhere to the experimental settings outlined in solo-learn (da Costa et al.
2022) for these datasets. As depicted in Table I INTL achieves remarkable results, with top-1 accuracy of 92.60% on CIFAR-10, 70.88% on CIFAR-IOO, and 81.68% on ImageNet-100. methods reproduced by solo-learn, Furthermore, when employing a 5-nearest neighbors classifier, INTL outperforms other baselines by a significant margin, underscoring its capacity to learn superior representations.
These results are on par with or even surpass the state-of-the
further assess the versatility of INTL, we train it using a ResNet-50 backbone and evaluate its performance using the standard linear evaluation protocol on ImageNet. The results, presented in Table demonstrate the effectiveness of INTL, achieving top-1 accuracy of 69.5%, 71.1%, 72.4%, and 73.1% after pre- training for 100, 200, 400, and 800 epochs, respectively. We observe that our INTL performs even better when utilized in conjunction with the Exponential Moving Average (EMA) technique, employed in YOL and MoCo. This combination yielded top-1 accuracy of 74.3% after 800 epochs of training.
Evaluation on ImageNet.
5.2 TRANSFER TO DOWNSTREAM TASKS
We examine the representation quality by transferring our pre-trained model to other tasks, including COCO (Lin et al. 2014) object detection and instance segmentation. We use the baseline of the detection codebase from MoCo He et al.) 2020) for INTL. The results of baselines shown in Table 3
are mostly inherited from CCen Hell 2021). I). We observe that INTL performs much better than other state-of-the-an approaches on COCO object detection and instance segmentation, which shows the great potential of INTL in transferring downstream tasks.
5.3 ABLATION STUDY
We conducte comprehensive set of ablation experiments to assess the robustness and versatility of our INTL in Appendix(F) These experiments cover various aspects, including batch sizes, embedding dimensions, the use of multi-crop augmentation, semi-supervised training, the choice of Vision Transformer (ViT) backbones and adding trace loss to other methods. Through these experiments, we gain valuable insights into how INTL performs under different conditions and configurations, shedding light on its adaptability and effectiveness in diverse scenarios. The results collectively reinforce the notion that INTL is a robust and flexible self-supervised learning method capable of delivering strong performance across wide range of settings and data representations. Notably, our INTL achieved a remarkable top-1 accuracy of 76.6% on ImageNet linear evaluation with ResNet-50 when employing multi-crop augmentation, surpassing even the common supervised baseline of 76.5%.
CONCLUSION
In this paper, we proposed spectral transformation (ST) framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. Our proposed IterNorm with trace loss (INTL) is well-motivated, theoretically demonstrated, and em- pirically validated in avoiding dimension collapse. Comprehensive experiments have shown the merits of INTL for achieving modulates the spectrum of embedding be equal-eigenvalues during the backward pass, which is a stronger constraint than hard whitening (the full-rank modulation), but a weaker constraint than soft whitening (the whitening modulation). This preliminary but new results provides potential way understand and compare SSL methods.
oftt--of-thcart performance for SSL in practice. We showed that INTL

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was partially supported by the National Science and Technology Major Project under Grant 20222D011633 16310, National Natural Science Foundation of China (Grant No. 62106012), the Fundamental Research Funds for the Central Universities.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility and comprehensiveness of our paper, have included an appendix comprising six main sections. These sections serve various purposes:
Appendix
A contains detailed proofs for the propositions presented in our work.
Appendix provides in-depth proofs for the theorems introduced our research.
Appendix C offers comprehensive view of the INTL algorithm, including detailed formulas and Py Torch-style code for implementation.
Appendix D elaborates on the settings used in our analytical experiments, with reference to Figure and Figure
Appendix furnishes insights into the implementation details and computational intricacies experiments conducted on standard SSL benchmarks, as discussed in Section
Finally, Appendix encompasses comprehensive set of ablation experiments, assessing the robustness and versatility our INTL method across various scenarios.
REFERENCES
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Veriaane-ivarrancceovvvaaanee oeg.://eccoaaaaaana...eeararaaaa for self-supervised learning. In ICLR, 2022.
Dario A. Bini, Nicholas J. Higham, and Beatrice Meini. Algorithms for the matrix pth root. Numerical Algorithms, 2005.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018
Mathilde Caron, Ishan Mísra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint arXiv:2003.nm 04297, 2020b.
ImageNet contributors. Imagenet terms of access. 2020. URL https: net org download
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):166, 2022. URL BRLbatp::// jmrr../m.... org/ wttpm///wwrpppppp///////// httpp//apppp/33332
1155 html
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical
Image Database. In CVPR, 2009.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In ICML, 2021.
10

---

Published as a conference paper at ICLR 2024
Abe Fetterman and Josh Albrecht. Understanding self-supervised and contrastive learning with "bootstrap your own latent" (byol). Technical Report, 2020.
Inc. Flickr. Flickr terms and conditions of use. 2020. URL http::/awwwb.teeeeeeeo techfak
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023.
Arna Ghosh, Arnab Kumar Mondal, Kumar Krishna Agrawal, and Blake A. Richards. Investigating power laws in deep representation learning. arXiv preprint arXIv:2202.05808, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach self-supervised learning. In NeuralPS, 2020.
Bobby He and Mete Ozay. Exploring the gap between collapsed and whitened features in self- supervised learning. In ICML, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In ICCV, 2021.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In CVPR, 2018.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardization towards efficient whitening. In CVPR, 2019.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-super learning. arXiv preprint arXiv:2011.0036 2020.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. ICLR, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR,
abs//4122 10ts/1412.690, 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollar, and Larry Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip Yu. Graph self-supervised learning: A survey. arXiv e-prints, pp. arXiv-2103, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1.XXiv1807.037444 2018.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 230.::2304.01933 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, gegh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2024
Kanehana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Michael Ryoo. Self-supervised video transformer. In CVPR, 2022.
Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv preprint arXiv:2010.10244 2020.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar, A theoretical analysis of contrastive unsupervised representation learning. In ICMI., 2019.
Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening and coloring batch transform for gans. In ICLR, 2019.
Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via A unified gradient framework. In CVPR, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In European conference on computer vision, 2020a.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self self-supervised learning with dual deep networks. CoRR, abs/2010.00578, 2020b.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. JCML, 2021.
Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. IEEE transactions on pattern analysis and machine intelligence, 45(5):5549-5555 2022.
Xi Weng. Lei Huang, Lei Zhao, Rao Muhammad Anwer, Salman Khan, and Fahad Khan. An investigation into whitening loss for self-supervised learning. In NeurlPS, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Chengxi Ye, Matthew Evanusa, Hua He, Anton Mitrokhin, Tom Goldstein, James A. Yorke, Cornelia Fermuller, and Yiannis Aloimonos. Network deconvolution. In ICLR, 2020.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann Lecun, and Stephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICMI., 2021.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. In ICLR, 2022a.
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S. Yu. From canonical correlation analysis to self-supervised graph neural networks. NeurlPS, 2021.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-CL: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2022b.
Yifei Zhang, Hao Zhu, Zixing Song. Piotr Koniusz, and Irwin King. Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1289 11999-1225 11297, 2023.
12

---

Published as a conference paper at ICLR 2024
A PROOFS OF PROPOSITION
A.1 PROOF OF PROPOSITION.
Proposition 1. Given æ E (0, 1). VT E N we have hr(a) (0, 1) and 11z((z) >
Proof. We know the iterative function fx) satisfies
3 fx+1(x)= 2 fx(2)) rff (z),
(8)
We define (x)x)) (x). When 1, it is easy verify VT N. hr = fr(1) = 1. We first prove Jr(1) (x) > and (x) 0 by mathematical induction.
(1) When 7" = 0, we have fo(x) = 1 > 0, and ho(a) = x, (x) = 1 > 0.
(2) Assuming it holds when T k, we have f1((x) 0 and (x) 0. Based on (1(x))
fa (x) fx (z)), we have:
fx)=) +x
Since hx(1) = (x)x > 0 and ha (x) is continuous, we have Vor E (0, 1), hy (x) < 1. We thus obtain:
fx-(()) [/((0)33 xB(x
fx(()) hx(2))
0
(10)
Furthermore, (x) fx+1 + 2x h++ (x)) where
fx+1(1) + 2xfl-1(x))
2 (1(())) + 22ff((x)
3 32
3
2
3
2 (A(00) + +2r((z))
- 2z)
2 = + +201((0)
11 hx (x))[((((()) + 2xf1(x)]
So we have 1 (x) = (z) h=(x)][fa(x)) 2x)(( > 0. Combining the result in Eqn. TO we thus have holds when T = k +
As result, have VT E N, fx( 0 and h4 (z) 0, when (0,
Since hr and (x) is continuous, we have lT(x)) < 1. Besides, we have hr(x) If (x) > 0, then (x) E (0, 1).
A.2 PROOF OF PROPOSITION.2
Proposition Given x E (0, 1). VT E N, we have hT41(()) (x()))
Proof. According to proof of Proposition.1. we have that when (0, and VT E Jr(x) > and hr(x) = (x) 10.
13

---

Published as a conference paper at ICLR 2024
Therefore, we have 1 lT(x)) = [T+1(1) > JT(Ø). obvious that
3
1
fx+1(x) fu(x)
2 f((())
2 xz)(2) fa(x)
1
fa(z)) 2 2z1((1)
2
fx(x)1 xx))
2 (z) hx(2))
> 0
So given (0, VT E N, we have (2)) > hr(x))
B PROOFS OF THEOREM
B.I PROOP OF THEOREM 1.
Theorem 1. Define one-variable iterative function fr()), satisfying
- f (x), 2 0: fo(x) =
The mapping function of IterNorm is
g(A) = (T((T(())) 1/ /tr(E),
so that AY, X(Z), IterNorm maps it to d, (r())) fr (
Proof. Given Σ UAUT, A diag(A ( Aa), U [u _u..... ua]. Following the calculation
steps of IterNorm, we have
A u.u
EN = E/tr(2)
=
(11)
tr tr(2)
Define
1
A
4/ Σ
fT
(12)
/tr(E)
tr Σ
d
Based on 4 =
Σ g(A)uu,; we can prove 1/- = Dr, we will have
s(A)
Vir(z)) (r
Define P = / tr (r then we have 4/ 4T mathematical induction.
P' Pr. We can prove P/ = Pr by
(1) When = 0,
fol (r
1, P' P =
(2) When 2 1, assume that P/--1 = P1 thus
Pr PT-1
p3 1 En
2
p'
7-11
(P/-1)' IN
(P'
14

---

Published as a conference paper at ICLR 2024
According to the definition of P'r-
P Pr-1- Σ fr-1 tr(2))
y T
Because Vi, u, N,.uu. uu; and Vi + j, " Uj O.
EN = (T 1
d
3
Σ
fr-1 1
A tr(E)
)uuu;
T
[
Σ
tr(x)) inu
X
tr
i=1
X
b nuu!?" ? tr(z)
1-11
tr(E)
Therefore, we have
3
1
Pr
PT--
T-1
2
(P/r-11) 33اN
(P
d
d
3
A
d
!T-1
|u:11;
3
nuu!"" T
2
tr
2 t=1
tr())
tr(_)
t=1
d
Σ
3
X
3
A かr(2))
j u.u. T
JT
tr
(r
2
2
tr(ンī)
:=1
Note
A
3
A
1
A
ST(
fT
(r())
2
(r().)
Ir
Σ.)
2
(r(z))
tr(E)
So that
X tr(E))
Pr
JT(
|1 =P/r
We obtain that
d Σ
fr( di
Tr D
)uu T UU vtr(2)
fr tr(2)
U
(r())
tr()))
Thus, the mapping function of IterNorm is g()) = fr((n()) //t/(rr(........)/)).r()) The whitened output is
Ż 27 U Vtr(s)
U Zo. The covariance matrix of Z is ST( Tr(2)
Eg 22T = (r(
U
A
)un,," T
(r())
tr
tr(E)
1=1
So that AX2 A()), IterNorm maps it to 2, Spectral Transformation.
(x( fI (1>)t)))
which is special instance of
B.2 PROOF OF THEOREM 2.
Theorem 2. Let E [0, 1j. VT E N+. INT shown in Eqn. 7 is strictly convex function.
12. x"=
... 2 is the unique minimum point as well as the optimal solution to INTL(x).
Proof. The INTL can be viewed as the following optimization problem:
min #20
INTL(Z) = Σ (1 -1- (E
(13)
j=1
15

---

Published as a conference paper at ICLR 2024
where Z = (-) and Ż (orrNorm(Z)) Eqn. can be viewed as a optimization problem over to encourage the trace of 2 to be d.
Let (#1,-.. .za) = pZZ), where 24 A1/(t(()) as defined in the submitted paper. Z E R (P()) will be surjective from Tedx Dx = fx 10, 11
Id 1]. When the range of (.(.) is wide enough, for example, Fo(-) is surjective from 0 @ to Z IR Rd Here
view Fo as function over 8, since the input is given and fixed. Then ((F(())) surjective from & E θ to x E Dx, meaning that if we find the optimal solution x*, we are able to get the corresponding
8" + subject x" ( (-)). On the contrary, for any 0 E O can get (FFF(-)) E Dx-
Therefore, the optimization expression for minimizing INTL can be written as follows which have the same range and optimal value as Eqn. 6
d Σ E11
d 10i--hr(an (x.))
2
min
INTL(x)-
1=1
(PINtL)
d
(14)
s.t.
Σ Ez==1
24 2 2,0,1 d
We denote the Lagrange function of PINTL is that
d
d
(x; (NTL(X) + Σ az (=((-x))+ +1
Σx;-1 -1
B.2.1 CONVEXITY AND CONCAVITY OF (r((x)
Before calculating extreme points of PINTZ. we first consider the convexity and concavity of hz(()) which is critical to proof.
When T we have ho(x)) x. so (x) 0.
(I) When T 1, we have hi h1(x) =si -f)((x) f/2- f.2 t.". so (1(x))
2) O.
(2) Assume that when T k. (x) 0 holds. We can easily get following propositions by derivation:
(z) 3 1 -(())) 3 (1)())) -
(15)
3
(x) 2 f((()) (f((())) 2 (
(16)
M ( = (f + 2x + (2))))))))) 20
(17)
For convenience in our calculation, let f f(
(x),,, and h hx(x) xa².
split Eqn. 17 into three parts and take Eqn. 15 and
16
into calculation:
3
3
4f++1(2)(z = 4( ah)
a
bh)
2
2
a3 h) (3b
a 3bh)
2x 2x(5641(00) 2x h - "
bh)
2 2
2
= (3b - d² 3bh)²
3 1
3
2xfx+1 (x)))
21 a ah)
I [2e( h) -aab 3xabb?
= an - h) [3e(1 - h) -6a - 6xub²
16

---

Published as a conference paper at ICLR 2024
Considering to construct the form of (x) = 2(2ab + vae + xb²), we first calculate that
4fx+1 (x) (x) (x) + 2r fr (x)
2 (3 66bb 2a² 6abh 3mae(1 - h1) - Gahh 6xb22]]
2 (3 h)[32ac(11 1t) + 6ab(1 h) 3xbb2 h)
3xb² (1 h) 2a 6abh 6xb²h]
(3 h)(1 (x) 2 (3 h))bbb h + 3xb2 + 2a + 6abh)
1
Then we calculate the left part
2x[5k+2(<)]] x²3b- 33hhh
(9zb2 + ra + hrrb2 - - 18rb² + 6xa bh)
(9xb² + att + 9xb2n² 6abh 18xb² + 6abh?)
For convenience, let
S = - (3 - ho)-)) 3xb2 + 2a + 6abh)
1 (3xb2 3mbb2 2ath + 6abh 9xb2h 9xb6² -a* 18abh)
Then we have
2x (x))² S
(3a h + 12xb2 - 24abh - 24xt"h + 12abli² -6a
(h 2) + 4abh + 4xb²h)
3 (h 2) Ara bb + 4x a22
3
(1 2) + 2rab)²
2
14 (x) 21(A(x))]
2
+ (1)(())) (1)((x)) Based on Lemma.1, we know (x) (0, 1). so h (z) O. Therefore, when x (0, 1). then VT N+.
Here we obtain (x) = 1/3 - hx(2)][1 - (x
loT (x:) = x.f7"(x) is strictly concave function that satisfies (x) < and (g()) = 0.
B.2.2
OPTIMAL SOLUTION FOR THE LAGRANGE FUNCTION
Based on Section B.2.1 when Ø E (0, 1), then VT E N4, hr(x)) = fx is strictly concave function that satishes (x) < 0. So hr(x)) a strictly convex function.
Σ [1 (x;))), first. Denote that the Hessen Matrix of d
We discuss 0,(21,' .. ",,. gj gy(21, ... .a) about x
(z1)
(xra)
and the Hessen Matrix of 0f(21 ,xx) about x
V2(23)- V(2g, Vg,) 2g, v2 g3 + ((V
17

---

Published as a conference paper at ICLR 2024
We denote that all eigenvalues of (Vg,)( (Vgj)= are (Vg),,, All eigenvalues are non-negtive, denoting that (Vg,))(Vg)) semi-pos semi-positive.
Now we denote that the Hessen Matrix of INTL(() is
VIINTL(X)
(V((
1=1
= 2 Σ
(Vg)) (Vg,) + Σ 99 V2g
j=1
where
Σ (
2
93 V 9j 2
h4 (xa)g)
d
We denote that (x,) < <0,g; > 0, and uys are not all zeros for a certain i (since Σ 4/: = 1).
(x:)g; > 0 and 2 Σ 9j Vzg d Σ (x)))
Therefore,
must be positive matrix.
j=1
Since Σ (Vg)) ) is semi-positive, then we can denote that V2INTL is positive.
Therefore, [NTL(x) is strictly convex about on 1)
And for (NT is continuous, the minimum point on 1], is the same as that on (0, 1)d.
While the constraints of (PINTL) form convex set, (PINTL) must be convex programming, which means that the KKT point of (PINTI.) is its unique extreme point, and the global minimum point in the same time.
We denote that the KKT conditions of (PINTL) is that
aL = 0,
i = 1, d
dz.
0nl (a(-;;)
0,
i = 1, d
a2 0,
= 1, d
Fj 1=
We can identify one of the solutions to the KKT conditions is that
x = [1 1
a 0
2h (2) Tee ( -1]
18

---

Published as a conference paper at ICLR 2024
easy to identify the last three equations in KKT conditions. As for the first equation, for all I = 1, 1 d. we have
aL
(zx)
(2:4
1]u3.
a4 + #
dxt
i=
1=1
d
22//
1N 7
u3 +
j=1
d
d
= 21/ 7
1
>
7) - 11
1
Σ
u +
j=1
iiml
= 21/ 1
1
7) - -1]uãe +
21/ M
1
2)
2) -1]
Σ
uitt
2
+ p
1 224 d
1
1]
+
Therefore, x" x-= [2..... 1 is the optimal solution to (PINTI.). INTI. promotes the equality of all eigenvalues in the optimization process, which provides a theoretical guarantee to avoid dimensional collapse.
C ALGORITHM OF INTL
The description of our paper is based on batch whitening (BW) (Ermolov et al. 2021 Hua et al. 202T), and it extend similarly for channel whitening (CW) (Weng et al. 2022). where the covariance matrix of Z is calculated as Σ d ZZ" Z. We implement INTL. based on CW, considering CW more effective when the batch size m relatively small.
Given the centralized embedding of two positive pairs (()), (()) Rd and {1,2). use
IterNorm to obtain the approximately whitened output 2 = [2] 2m (w) The loss functions used in our method are
1
2
(1)
2
(2)
LMSE
(18)
171
1
112
1
(2)
1
2 Σ
T
Lirace
(1
2i
(v)
2
(19)
v=1 1=1
LINTL = LMSE + B- Ltrace
(20)
where CMSE indicates MSE of L2 normalizze vectors which minimizes the distance between (1)
and (2)) Here we simplify the expression of Ltrace in Eqn. 6 because off-diagonal elements of >i does not need to be calculated. 3 is the trade-off between LMSE and INTL.
In our experiments, that obtains good performance has only relevant to the batch size. So we fix the iteration number T' to 4 and empirically regress / with various batch sizes and obtain that 3 = 0.01 (logzbs - 3) where bs means the batch size and bs 8. We keep the iteration number T of IterNorm and the coefficient 3 fixed in this form (i.c.. 3 is determined given the batch size) across all the datasets and architectures, so our INTL can be directly applied to other datasets and models without tuning the coefficient.
observe that when the iteration number T of IterNorm is fixed, the coefficient
For clarity, we also describe the algorithm of INTL in Py Torch-sttyle pseudocode, shown in Figure aa).
19

---

Published as a conference paper at ICLR 2024
# F: backbone + projection bs: batch size
aug: random augmentation
for loader: load minibatch with samples
21, 22 f(aug(x)), f(aug(x)) # embedding # transformed output
21 i__hat, 22 22_hat IterNorm(z1), IterNorm(z2)
# trade_off between MSE and trace Loss
trade_off (log2(bs) 3) 0.81
mse norm_nse(z1_hatt z2_hat) MSE
trace _oase_lssss_ TL(z1_hatt TL(22_haat # trace Loss
loss
trade_off trace_loss
return loss
def IterNora(x, iters=4): Iterative Normalization
M, D - x.size() x:
x x.mean(dim=1) reshape(M,
sigma (x x) / (D 1) covariance matrix trace sigma.diagonal
sigma_nors sigma trace normalize sigma
eye(M) identity matrix: m for in ritge(;ters):
(3 matrix_ petrix_power(P, @ sigma_norm) norm)
return P trace.sqrt()
def TL(x):
D = x.size()
d torch. pow sum
(D
t1 d.add_ (-addddd)______ (2) return tl
def norm_mse(xe) x1):
x² normalize (x0) L2-normalize
normalize(x L2-normalize
return 2 (x0 ())suu(ddim-1))mmen
Figure 4: Algorithm of INTL, PyTorch-style Pseudocode.
D ANALYTICAL EXPERIMENTS
D. EXPERIMENTS ON SYNTHETIC DATASET
In section 3.2 of the submitted paper, we conduct experiments on the 2D dataset and report the results on with varying p. Here, we provide the details of the experimental setup, and further show the results of IterNorm (Huang et al. 2019) for SSI. in this 2D dataset.
D.1.1 DETAILS EXPERIMENTAL SETUPS
We synthesize a two-dimensional dataset with isotropic Gaussian blobs containing 512 sample points shown in Figure 5 a). We construct toy Siamese network (a simple three-layer neural network. including three fully connected (FC) layers, with BN and ReLU appended to the first two) as
encoder for this dataset. The dimensions of the network are (2 16) (16 - 16) (16 - 2) that each bracket represents the input and output dimensions of each FC layer respectively. We then MSE as the loss function and do not normalize the features before calculating the loss function.
We train the model by randomly shuffling the data into mini-batches, and set the batch size to 32. We the stochastic gradient descent (SGD) algorithm with learning rate of 0.1. In terms of the data transformation, we only apply Gaussian noise as data augmentation and generate views from each
20

---

Published as a conference paper at ICLR 2024
2D dataset distribution
Initial network output
0.0
-05
-1.
-2.0
-6
-25
-3.
-0.5
(a)
(b)
Figure 5: Visualization our synthetic 2D dataset. We show (a) the distribution of our 2D dataset; (b) the initial output of the toy Siamese network.
sample point in mini-batches. We visualize the output of the initialized network without training in Figure b)). All runs are performed under the same random seed.
Lg eigenvalue
Lg of eigenvalue
-
T-9
Epochs
Epochs
(a)
(b)
Figure 6: Investigate the spectrum of transformed output 2 (solid lines) and the corresponding embedding Z (dashed lines) using IterNorm for SSL with different iteration numbers T. We show the evolution of eigenvalues during training on the toy 2D dataset (Note that there are only two eigenvalues and we ignore the larger one because always remains a high value during training). In particular, (a) shows the results with well-conditioned initial spectrum while (b) with ill- conditioned one.
D.1.2 RESULTS OF ITERNORM FOR SSL
To figure out the failure of IterNorm (Huang al., (2199) for SSL, further conduct experiments to investigate the spectrum of the whitened output using IterNorm on this synthetic 2D dataset for intuitive analyses. The output dimension of the toy model is 2, so there are only two eigenvalues of the covariance matrix of the output. We then track alterations of the two eigenvalues during training. IterNorm can obtain an idealized whitened output with small iteration number (e.g., as recommend in (Huang et al. 2019)) and avoid collapse, if the embedding Z has well-conditioned spectrum 5) (Figure 6(a)). However, the embedding has ill-conditioned spectrum as shown in Figure 6 b), IterNorm fails to pull the small eigenvalue to approach which results in dimensional collapse.
D.2 EXPERIMENTS ON CIFAR-10
In section and 4 of the submitted paper, we conduct several experiments on CIFAR-10 to illustrate our analysis. We provide a brief description of the setup in the caption of Figure and 2 of the submitted paper. Here, describe the details of these experiments. All experiments based on the following training settings, unless otherwise stated in the figures of the submitted paper.
uniformly
A is small, Note X is the maximum 'A well-conditioned spectrum means that the condition number c
X
eigenvalue and da is the minimum one.
21

---

Published as a conference paper at ICLR 2024
-25
-5.0
Lf
125
-7.5
255
Lg Lgofe
1024
-10.00
2048
-10
-12.5
-12
15.0
1000
3000
4000
5000
6000
1000
2000
3000
4000
5000
6000
Iteration
Treration
(a)
(b)
Figure 7: Investigate numerical instability of spectral transformation using power functions for SSL. The numbers in the legend represent embedding dimensions and the batch size fixed to 512. (a) trains models on ImageNet with ResNet-50; (b) trains models on CIFAR-10 with ResNet-18; The models are trained for 6000 iterations, and we track the inverse of condition number (c A in
logarithmic scale with base 10 to judge whether the the covariance matrix is ill ill-conditioned. The models that were interrupted before the end of the training indicate training crash caused by numerical instability.
Training Settings. We use the ResNet- 18 as the encoder (the dimension of encoding is 512.), a two layer MLP with ReLU and BN appended as the projector (the dimension of the hidden layer and embedding are 1024 and 1 128 respectively). The model is trained CIFAR-I 10 with batch size of 256, using Adam optimizer (Kingma & Ba 2014) with a learning rate of 3 x 10 3 and learning rate warm-up for the first 500 iterations and a 0.2 learning rate drop at the last 50 and 25 epochs. The weight decay is set as 10-6 All transformations are performed with positives extracted per image with standard data argumentation (see Section E.3 for details). We use the same evaluation protocol as in W-MSE (Ermolov et al. 2021).
Method Settings. We use MSE of -ooonmmal vectors to be the loss function in all experiments.
Specifically, in Figure 3 of the paper for the experiments of training the models with INTL, we simply set the trade-off parameter between MSE and INTL follows: 3 = 0.05 for T 5, = 0.5 for T and 3 = 5 for T without fine-tuning. The details of INTL algorithm please refer to Section
D.3 NUMERICAL INSTABILITY OF SPECTRAL TRANSFORMATION USING POWER FUNCTIONS
One issue with employing the spectral transformation (A)= - (where p is approximately 0.5) the risk of numerical instability during the calculation of eigenvalues À and eigenvectors U via digen-decomposition. This instability can arise when dealing with an ill-conditioned covariance matrix, as noted in (Paszke et al., 2019). In this study, we empirically validate the existence of this phenomenon in the context of self-supervised pre-training. It's important to mention that we primarily focus on the special case of 0.5, referred as hard whitening, similar phenomena are observed when p is set near 0.5.
To assess the generality of this phenomenon, we conduct experiments on both ImageNet with ResNet- 50 and CIFAR-10 with ResNet-18. We maintain fixed batch size of 512 and manipulate the shape of the covariance matrix by adjusting the embedding dimension d (where the covariance matrix has shape of d x d). The models undergo 6000 iterations, and we monitor the inverse of the condition number (c-1 Ad to ascertain the ill-conditioned nature of the covariance matrix. The experimental
Xi
results, depicted in Figure 7. lead to the following key observations:
(a) Training crashes when the embedding dimension exceeds the batch size (e.g., d = 1024 or 2048). In such cases, the covariance matrix becomes thcoretically singular, and computing the inverse of the eigenvalues introduces numerical errors. However, in practice, the minimum eigenvalue of the covariance matrix is likely very small non-zero value due precision rounding the use of small constant. Consequently, the covariance matrix may already be ill ill-conditioned from the start of training. Both Figure 7 a) and (b) illustrate that when d = 1024 or 2048, the inverse of the condition number is approximately 10-12 10-10, indicating severe ill-conditioning from the beginning, resulting in rapid training breakdown.
22

---

Published as a conference paper at ICLR 2024
Table 4: Parameters used for image augmentations on ImageNet and ImageNet-100.
Parameter
71
crop size
224 X 221
224
221
maximum scale crops
1.0
1.0
minimum scale of crops
0.08
0.08
brightness
0.4
0.4
contrast
0.4
0.4
saturation
0.2
0.2
huc color jitter prob horizontal flip prob
0.1
0.1
0.8
0.8
0.5
0.5
gaussian prob solarization prob
1.0
0.1
0.0
0.2
(b) Training is prone crashing when the embedding dimension equals the batch size (d 512). In such cases, it's challenging to definitively establish whether the covariance matrix singular. However, observations from Figure 7 suggest that the covariance matrix tends towards ill- conditioning when d = 512. The inverse of the condition number progressively decreases during training, eventually leading to training instability.
(c) There is possibilityy of training instability when the embedding dimension is less than the batch size. In these situations, we initially observe that the covariance matrix remains well-conditioned. However, this favorable condition is not consistently maintained throughout training. We notice that well-conditioning suddenly breaks after few iterations, leading to model collapse for d = 64 or d 128. Interestingly, training does not crash when 256. This phenomenon was briefly discussed in (Ermolov et al. 2021), suggesting that stability can be improved by setting m = 2d.
We confirm the presence of numerical instability when employing hard whitening (Ermolov et al.) 202I). as indicated by the above analysis. While one can mitigate this instability empirically by setting mn = 2d, our experiments reveal that training crashes due to numerical instability can still occur at various points during training. In extensive experimentation (with 10 random seeds and longer training iterations), we observed instances of numerical issues-appres approximately 3-4 times-occurring at different stages, including early, mid, or even towards the end of training. Even though it is possible resume training using saved checkpoints in the event of crash, this significantly limits the practical applicability of long-term pre-training.
E DETAILS EXPERIMENTS ON STANDARD SSL BENCHMARK
In this section, we provide the details of implementation and training protocol for the experiments on large-scale ImageNet (Deng et al. 2009). medium-scale ImageNet- 100 (Tian et al. 2020a) and small- scale CIFAR-10/100 (Krizhevsky) (209) classification well transter learning COCO (Lin al. 2014) object detection and instance segmentation. We also provide computational overhead of INTL pre-training on ImageNet.
DATASETS
CIFAR-10 and CIFAR-100 (Krizhevsky 2009), small-scale datasets composed of 32 32
images with 10 and 100 classes, respectively.
ImageNet-100 100 (Tian et al. 2020a). random 100-class subset of ImageNet (Deng et al. 2009).
ImageNet (Deng et al. and 50K test images, spanning over 1000 classes.
2009). the well-known largescale dataset with about 1.3M training images
COC02017 (Lin et al. 2014), large-scale object detection, segmentation, and captioning dataset with 330K images containing 1.5 million object instances.
E.2 EXPERIMENT ON IMAGENET
In section 5.1 of the paper, compare INTL to the state-of-the-art SSL methods large-scale ImageNet classification. Here, we describe the training details these experiments.
23

---

Published as a conference paper at ICLR 2024
Table 5: Parameters used for multi-crop of INTL on ImageNet.
Parameter
Ti
72
73
T4
crop size
224 x 224
224
192 X 192
160
x 128
96 96
maximum scale crops
1.0
0.857
0.714
0.571
minimum scale of crops
0.2
0.2
0.171
0.143
0.114
0.086
brightness
0.4
0.4
0.4
0.4
0.4
0.4
contrast
0.4
0.4
0.4
0.4
0.4
0.4
saturation
0.2
0.2
0.2
0.2
0.2
0.2
huc color jitter prob horizontal flip prob
0.1
0.1
0.1
0.1
0.1
0.1
0.8
0.8
0.8
0.8
0.8
0,8
0.5
0.5
0.5
0.5
0.5
0.5
gaussian prob
0.5
0.5
0.5
0.5
0.5
0.5
solarization prob
0.1
0.1
0.1
0.1
0.1
0.1
Table 6: Parameters used for INTL pre-training on ImageNet- Ne0.
Parameter
Value
max epoch
400
backhone
ResNet-18
projection layers
projection hidden dimension
4096
projection output dimension
4096
optimizer
SGD
SGD momentum
0.9
lcarning rate learning rate warm-up learning rate schedule
0.5
epochs cosine decay
weight decay batch size
2.5e-5
128
Backbone and Projection. We use the ResNet-50 (He et al. 2016) the backbone and the output dimension is 2048. We use 3-layers MLP as the projection: two hidden layers with BN and ReLU applied to it and a linear layer as output. We set dimensions of the hidden layer and embedding to 8192 as our initial experiments followed the settings of VICReg and Barlow Twins, both of which use dimension of 8192 for the projection. Compared to a projection dimension of 2048, using projection dimension of 8 8192 can bring about a 14% improvement in top-1 accuracy for INTL. Therefore, followed this setting in subsequent experiments ImageNet. We report that using projection dimension 8192 requires approximately 18% additional GPU memory and 2% time per epoch compared to using the one of 2048.
Image Transformation Details. In image transformation, We the same augmentation parame- ters as BYL (Grill et al. 2020). Each input image is transformed twice to produce the two distorted views. The image augmentation pipeline consists of the following transformations: random cropping, resizing 224 224, horizontal flipping, color jittering, converting grayscale, Gaussian blurring, and solarization. The details of parameters are shown in Table 4
Optimizer and Learning Rate Schedule. We apply the SGD optimizer, using a learning rate of base-Ir x BatchSize / 256 and cosine decay schedule. The base-Ir for 100-epoch pre-training is 0.5, for 200(400)-epoe) is 0.4 and for 800-epoch is 0.3. The weight decay 10 and the SGD momentum is 0.9. In addition, we use learning rate warm-up for the first epochs of the optimizer.
Evaluation Protocol. For linear classification, we train the linear classifier for 100 epochs with SGD optimizer (using a learning rate of base-Ir x BatchSize 256 with a base-Ir of 0.2) and using MultiStepLR scheduler with y = 0.1 dropping at the last 40 and 20 epochs. Note that when combining INTL with multi-crop in the ablation experiments, the base-Ir is set 0.4. The batch size and weight decay for both are 256 and respectively.
Exponential Moving Average. In the main text, we observe that our INTL can performs even better when utilized in conjunction with the Exponential Moving Average (EMA) technique. We set the base coefficient for momentum updating to 0.996 for all-epoch training. The momentum coefficient follows a cosine increasing schedule with final value 1.0 as BYOL (Grill et al. 2020).
24

---

Published as a conference paper at ICLR 2024
Table 7: Parameters used for INTL pre-training on CIFAR-10/100.
Parameter
Value
epoch
1000
backbone
ResNet-18
projection layers
3
projection hidden dimension
2048
projection output dimension
2048
optimizer
SGD
SGD momentum
0.9
learning rate Iearning rate warm-up learning rate schedule
0.3
epochs
cosine decay le-4
weight decay
batch size
256
E.3 EXPERIMENTS FOR SMALL AND MEDIUM SIZE DATASETS
In section 5.1 of the paper, we provide the classification results of INTL pre-training on small and medium size datasets such as CIFAR-10, CIFAR-106 and ImageNet- 100. Here, We describe the details of implementation and training protocol for the experiments these datasets as follows. For fairness, most of and so on are consistent with solo-learn (da Costa et al. 2022).
we used such as batch size, projection settings, data augmentation
Experimental setup on ImageNet-100. Details of implementation and training protocol for INTL pre-training on ImageNet- 100 are shown in Table 6 The image transformation and evaluation protocol are the same as ones on ImageNet.
Experimental setup CIFAR-10/100. Then Details of implementation and training protocol for INTL pre-training on CIFAR-10/101 are shown in Table The details of image transformation are shown in T For evaluation, we use the same setup of protocol as in W-MSE (Ermolov et al. 202T): training the linear classifier for 500 epochs using the Adam optimizer and the labeled training set of each specific dataset, without data augmentation; the learning rate is exponentially decayed from 10-2 to 10-6 and the weight decay is 5 x 10-5.
In addition, we also evaluate the accuracy of k-nearest neighbors classifier (k-NN, k= 5) these experiments. For other methods, we evaluate the models provided by (da Costa et al. 2022) to obtain k-NN accuracy which does not require additional parameters and training.
E.4 EXPERIMENTS FOR TRANSFER LEARNING
In this part, we describe the training details of experiments for transfer learning. Our implementation based on the released codebase of MoCo (He et al. 2020 for transfer learning to object detection and instance segmentation tasks. We use the default hyper-paramett parameter configurations from the training scripts provided by the codebase for INTL, using our 200-epoch and 800-epoch pre-trained model on ImageNet.
For the experiments of COCO detection and COCO instance segmentation, use Mask R-CNN (Ix schedule) fine-tuned in COCO 2017 train, evaluated COCO 2017 val. The Mask R- R-CNN model is with the C4-backbone. Our INTL is performed with 3 random seeds, with mean and standard deviation reported.
E.5 COMPUTATIONAL OVERHEAD
In Table 9] report compute and GPU memory requirements based on our implementation for different settings on ImageNet with ResNet-50. The batch size is 256, and we train each model with 2 A A100-PCIE-40GB GPUs, using mixed precision and py-torch optimized version of synchronized batch-normalization layers.
wtwp:y/siicceeooieeeeeeeeeeeeeee under the CC-BY-NC 4.0 license.
25

---

Published as a conference paper at ICLR 2024
Table Parameters used for image augmentations on CIFAR-10/100.
Parameter
71
72
crop size
32 32
32 32
maximum scale of crops
minimum scale of crops
0.08
0.08
brightness
0.4
0.4
contrast
0.4
0.4
saturation
0.2
0,2
huc color jitter prob horizontal flip prob gaussian prob
0.1
0.1
0.8
0.8
0.5
0.5
solarization prob
0.0
0.2
Table 9: Computational cost. We report time and GPU memory requirements of our implementation for INTL trained per epoch on ImageNet with ResNet-50.
Method
EMA
Multi-Crop
time /1 epoch
peak memory / GPU
No
No
29min11
16.0
Yes
No
24min46
11.8 G
INTL
No
Yes
57min33
25.9 G
Yes
Yes
50min52
21.2 G
F ABLATION STUDY
In this section, we conduct a comprehensive set of ablation experiments to assess the robustness and versatility of our INTL. These experiments cover various aspects, including batch sizes, embedding dimensions, the use multi-crop augmentation, semi-supervised training, the choice Vision Transformer (ViT) backbones and adding trace loss to other methods.
Batch size. Most SSL methods, including certain whitening- based methods, are known to sensitive to batch sizes, e.g. SimCLR (Chen et al. 2020a).
Table 10: Effect of batch sizes for INTL. We train 100 epoch ImageNet and provide the Top-1 accuracy using linear evaluation. The embedding dimension is fixed to 8192.
SwAV (Caron et al.) (2020) and W-MSE Ermolov
et al. 2021) all require large batch size (e.g. 4096)
to work well. We then test the robustness of INTL
Bs
32
64
128
256
512
1024
batch sizes. We train INTI. on ImageNet for 100
aCc.
64.2
66.4
68.7
69.5
69.7
epochs with various batch sizes ranging from 32 to 1024. As shown in Table. [0] even if the batch size is low 32 64, INTL still maintains good performance. At the same time, when the batch size increases, the accuracy INTL also improved. These results indicate that INTL has good robustness to batch sizes and can adapt to various scenarios that constrain the training batch size.
Embedding dimension. Embedding dimension, the output dimension of the projection, is also key element for most self-supervised learning methods, which may have a significant impact on training results. As illustrated in (Zbontar et al. (021). Bar- low Twins very sensitive to embedding dimen- sion and it requires large dimension (e.g. 8192 or 16384) work well. We also test the robustness of INTL to embedding dimensions. Following the setup of (Chen et al.) (220aa and (Zbontar et al.) 2021). we train INTL on ImageNet for 300 epochs with the dimension ranging from 64 to 16384. As shown in Figure. 8 even when the embedding di- mension low as 64 or 128. INTL still achieves good results. These results show that INTL also has strong robustness to embedding dimensions.
72.5
70.0
65.0
62.5
INTL (ours)
Barlow Twins
SimCLR
64
128
255
512
1024
2048
4096
8192
16584
Embedding dimension
Figure 8: Ablation experiments for varying em- bedding dimensions. The batch size is fixed to 256.
Multi-Crop In the main text experiments, employ the standard augmentation, which generates two augmented views for each sample. It's worth noting that multi-crop strategics, such as the
26

---

Published as conference paper at ICLR 2024
Table 1I: Ablation experiments on ImageNet linear classification with EMA and multi-crop. All are
based on ResNet-50 backbone.
Method
Bs
EMA
Multi-Crop
100 eps
200 eps
400 eps
800 eps
SWAV
4096
No
No
66.5
69.1
70.7
71.8
INTL (ours)
512
No
No
69.5
71.1
72.4
73.1
SWAV
4096
No
Yes
72.1
73.9
74.6
75.3
SWAV
256
No
Yes
72.7
74.3
INTL (ours)
256
No
Yes
72.4
74.3
74.9
CLSA
256
Yes
No
69.4
72.2
INTL (ours)
256
Yes
No
69.2
71.5
73.7
74.3
DINO
4080
Yes
Yes
75.3
CLSA
256
Yes
Yes
73.3
76.2
INTL (ours)
256
Yes
Yes
73.5
75.2
76.1
76.6
Table 12: Semi-supervised classification on top of the fine-tuned representations from 1 1% and 10% of ImageNet samples.
Epoch
Bs
Semi-supervised
Method
Top-1
Top-5
1%
10%
1%
10%
Supervised
120
256
25.4
56.4
48.4
80.4
SimCLR
800
4096
48.3
63.6
75.5
87,8
BYOL
1000
4096
53.2
68.8
78.4
89.0
SwAV
800
4096
53,9
70.2
78.5
89.9
Barlow Twins
1000
2048
55.0
69.7
79.2
89.3
VICReg
1000
2048
54,8
69.5
79.4
89.5
INTL (ours)
800
512
55.0
69.4
80.8
89.8
used by SwAV (Caron et al. 2020), are widely recognized for enhancing the performance of SSL methods. For instance, SwAV achieves a remarkable Top-1 accuracy of 75.3% with multi-crop. Therefore, we also conduct experiments with INTL using multi-crop We apply an efficient multi-crop approach that generates views for each image, with sizes of 2 224 + 192 + 160 + 128 96, which is similar to the approach used by CLSA (Wang Qil 2022). ). (Detailed parameter settings are provided in Table 5). The results are shown in Table 11. When INTL is paired with multi-crop top-1 accuracy. For instance, after
augmentation, it consistently achieve notable improvements 800 epochs of pre-training, INTL attains an impressive top-1 accuracy of 76.6%, even surpassing the common supervised baseline of 76.5%. The incorporation of multi-crop augmentation enhances the performance of INTL, making it promising method for self-supervised representation learning across a range of experimental setups.
Semi-supervised training. For semi-supervised classification, we fine-tune our pre-trained INTL backbone and train the linear classifier on ImageNet for 20 epochs. We employ subsets of size 1% and 10%, following the same split as SimCLR. The optimization is performed using the SGD optimizer with a base learning rate of 0.006 for the backbone and 0.2 for the classifier, along with cosine decay schedule. The semi- cani-suprrvised results on the ImageNet validation dataset are presented in Table 12 demonstrating that INTL performs well in semi-supervised training scenarios.
Vison Transformer backbones. We conduct additional experiments using vision transformer (ViT) backbones for INTL. For comparison, we reproduce five other method under the same settings. The results are shown in Figure illustrating that INTL maintains strong performance when ViTs are used as backbones. This suggests that INTL exhibits robust generalization capabilities across different network architectures.
Barlow Twins/VICReg with Trace loss. We conducte experiments on CIFAR- 10/100 and ImageNet-100 to assess the impact of adding trace loss to Barlow Twins and VICReg, follow- ing the experimental setup outlined in Table 2 of our paper. We trained the models on 1IFAR-10/100 for 200 epochs and on ImageNet-100 for 100 epochs. The coefficient of trace loss empirically suitable value for both methods. The results are presented in the Table [3] We observed that adding trace loss to Barlow Twins had a minor positive effect on performance, while introducing it to VICReg significantly reduced performance, particularly ImageNet- 100. We hypothesize that this discrepancy may arise from the influence of trace loss on the regularization strength these
to 0.01, an
27

---

Published as a conference paper at ICLR 2024
Accunacy
Accunacy
Vidg
Results CIFAR-I0
(b) Results CIFAR-100
(c) Results ImageNet-100
Figure 9: Abalation experiments using vision transformer (ViT) backbones. We train INTL as well as other methods (including SimCLR, W-MSE, DINO, Barlow Twins, and VICReg) for comparison when using ViTs as backbones. Our training setup involved ViT-tiny for 200 epochs on CIFAR-10/1N and ViT-small for 100 epochs ImageNet- -00. 100. The settings were kept consistent with DINO, with the exception of the embedding dimension for W-MSE, which was set 64, while other methods used 2048. We evaluated their classification performance using both linear classifier and 5-nearest neighbors classifier. The results for CIFAR-10, CIFAR-100, and ImageNet-100 presented in panels (a), (b), and (c) respectively.
Table 13: Evaluate the performance of adding trace loss BeeilooTwins//ICIRC
CIFAR-10
CIFAR-100
Image 100
Method
top-1
5-nn
top-1
5-mn
top-1
5-nn
Barlow Twins
80.43
76.68
51.60
42.71
58.34
50.21
Barlow Twins trace loss
80.45
76.32
51.66
43.94
59.78
50.45
VICReg
83.14
79.62
55.96
46.71
66.01
57.76
VICReg + trace loss
81.67
78.74
54.75
46.24
63.54
55.18
methods. It can either disrupt the existing balance, leading to reduced performance, or achieve more favorable balance, resulting in improved performance.
G LICENSES OF DATASETS
ImageNet (Deng et al. 2009) is subject to the ImageNet terms of access: (contributors) 2020)
COCO (Lin et al. 2014). The annotations are under the Creative Commons Attribution 4.0 License. The images are subject to the Flickr terms of use (Flickr 2020).
28


Reviews:
[{"review_id": "Zj6T8KNTVp", "reviewer": "Reviewer_YNa9", "summary": "This paper presents a new approach, Spectral Transformation (ST), for self-supervised learning, and proposes a new training algorithm named IterNorm with trace loss (INTL). The basic idea of the paper is to balance the spectrum of the covariance matrices for the learned features which is often ill-posed. Theoretical and empirical results are provided as well for demonstrating the performance.", "questions": "see my comment", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear writing with many experimental results.", "weaknesses": "To me, I do not see any obvious weakness of the proposed approach. Motivated by the whitening, the paper presents a nice and logical development of the approach. However, I do not see a very strong point neither that can make this paper stand out compared with the literature. I suggested the authors to further emphasize the key contributions: What really makes your approach better than others such as MoCo and SimCLR? How about computational speed (this seems not to be discussed in both paper and appendix)? \n\nI’d like to increase my rate if the authors can convince me at this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82PviRYSda", "reviewer": "Reviewer_AFis", "summary": "This paper proposed a new self-supervised learning framework called spectral transformation (ST) to modulate the spectrum of embedding to avoid dimensional collapse. To be specific, they introduced a novel ST instance named IterNorm with trace loss (INTL). Theoretically, this paper proved that INTL can modulate the spectrum of embeddings toward equal eigenvalues and prevent dimensional collapse. Empirically, the authors showed that INTL can obtain state-of-the-art performance for SSL on real-world datasets.", "questions": "see my comments above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The dimensional collapse is an important problem in contrastive learning and the analysis in this paper is insightful.\n2. The theoretical analysis and empirical results cooperate well. The improvements on real-world datasets are significant, especially in transfer learning tasks.", "weaknesses": "1. As analyzed in this paper, both whitening methods and INTL are instances of spectral transformations. However, it seems that INTL outperforms whitening in every task. So what are the disadvantages of whitening methods? It would be better to provide more theoretical and empirical comparisons between them.\n2. The motivation behind the trace loss is a little confusing. Is it possible to provide a more detailed discussion?\n3. It seems that INTL shows superior performance in 5-nn accuracy than linear probing accuracy. Are there any intuitive explanations for that?\n4. There are some typos. For example, in p.4, ‘Eqn. 13 can be viewed as an optimization problem over …’ should be replaced with ‘Eqn.6 …’.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GDvLvjwTtC", "reviewer": "Reviewer_UjJ7", "summary": "- The authors propose a framework referred to as Spectral Transformation (ST) to modulate the spectrum of embedding and to seek functions beyond whitening that can avoid dimensional collapse.\n- Additionally, The authors introduce a novel ST instance named IterNorm with trace loss (INTL) to prevent collapse and modulate the spectrum of embedding toward equal eigenvalues during optimization.\n- The extensive experiments on ImageNet classification and COCO object detection demonstrate the effectiveness of INTL in learning superior representations.", "questions": "- Could the authors compare the INTL method with the baseline [1] if possible?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- (+) The authors show the novel points of the proposed methods, INTL, while comparing them with previous methods such as hard and soft whitening.\n- (+) The authors show the empirical observations of IterNorm, which map all non-zero eigenvalues to approach one, with large enough iterations (T).", "weaknesses": "- (-) The authors seem to have a missing baseline [1] in SSL. The baseline looks similar to INTL from the viewpoint of spectral adjusting.\n    - [1] Exploring the Gap between Collapsed & Whitened Features in SSL-ICML2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)