Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization
Authors: Junlin He
Jinxiao Du
Wei Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Self-supervised learning;  Orthogonality Regularization; Dimensional collapse
Primary area: machine_vision
TL;DR: Orthogonality Regularization is the first time to be used in Self-supervised learning(SSL), which  prevents dimension collapse of weight matrices, hidden features and representations and thus improves performance of SSL.

Abstract: Self-supervised learning (SSL) has rapidly advanced in recent years, approaching the performance of its supervised counterparts through the extraction of representations from unlabeled data. However, dimensional collapse, where a few large eigenvalues dominate the eigenspace, poses a significant obstacle for SSL. When dimensional collapse occurs on features (e.g. hidden features and representations), it prevents features from representing the full information of the data; when dimensional collapse occurs on weight matrices, their filters are self-related and redundant, limiting their expressive power.
Existing studies have predominantly concentrated on the dimensional collapse of representations, neglecting whether this can sufficiently prevent the dimensional collapse of the weight matrices and hidden features. 
To this end, we first time propose a mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matrices, thus safeguarding against the dimensional collapse of weight matrices, hidden features, and representations. Our empirical investigations demonstrate that OR significantly enhances the performance of SSL methods across diverse benchmarks, yielding consistent gains with both CNNs and Transformer-based architectures.

Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization
Junlin He
Jinxiao Du
The Hong Kong Polytechnic University Hong Kong SAR, China junlinspeed he@connect poltoonnectpoly..hit polyu.hk
The Hong Kong Polytechnic University Hong Kong SAR, China jinxiao dww.onneet.t..eoooomcc pocict..oolu.hht
Wei Ma"
The Hong Kong Polytechnic University Hong Kong SAR, China
wei W. www.mmyl.ll..cmmmmmmmmmmmmmm edu. .k
Abstract
Self-supervised learning (SSL) has rapidly advanced in recent years, approaching the performance of its supervised counterparts through the extraction of represen- tations from unlabeled data. However, dimensional collapse, where a few large eigenvalues dominate the eigenspace, poses a significant obstacle for SSL. When dimensional collapse occurs on features (e.g. hidden features and representations), it prevents features from representing the full information of the data; when di- mensional collapse occurs on weight matrices, their filters are self-related and redundant, limiting their expressive power. Existing studies have predominantly concentrated on the dimensional collapse of representations, neglecting whether this can sufficiently prevent the dimensional collapse of the weight matrices and hidden features. To this end, we first time propose a mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matri- ces, thus safeguarding against the dimensional collapse of weight matrices, hidden features, and representations. Our empirical investigations demonstrate that OR significantly enhances the performance of SSL methods across diverse benchmarks, yielding consistent gains with both CNNs and Transformerrbased architectures. Our code will be released at https///ittbb.commmmmmaaannnOOnn S
Introduction
Self-supervised learning (SSL) has established itself indispensable paradigm in machine learning, motivated by the expensive costs of human annotation and the abundant quantities of unlabeled data. SSL endeavors produce meaningful representations without the guidance of labels. Recent developments have witnessed joint-embedding SSL methods achieving, or even exceeding
the supervised counterparts (Misra & Maaten 2020 [ardes al. 2022 Caron al. 2020 Chen
Fan, Girshick & Hc 2020 Chen, Kornblith, Norouzi & Hinton 2020 CCen & He 2021 Dwibedi
al. HaoChen
al. 202 2| et al. 2020 He & Ozay 2022 Ting al. 2021 Zhou. Xiong
& Hoil 2020 Jing et al. 2020 Balestriero et al. 2023 Grill et al. 2020 Zbontar et al.
al. 2021). The efficacy these methods hinges on two pivotal principles: 1) the ability learn augmentation-inaalee
representations, and 2) the prevention of complete collapse, where all inputs are encoded to a constant vector.
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Efforts to forestall complete collapse have been diverse, including contrastive methods with both positive and negative pairs (He al. 2020 Chen, Korblith, Norouzi & Hinton 2020 Chen al.
2021 and non-contrastive methods utilizing techniques such as self-distillation (Caron al. 2021
Grill et al. 2020, Chen He 2021), clustering (Caron et al. 2018 2020 Pang et al. 2022) and
feature whitening (Bardes et al. 2022 Zbontar et 2021 Weng et al. 2022 (023). Notwithstanding
these methods are prone to dimensional collapse, phenomenon where a few large eigenvalues
dominate the eigenspace. Dimensional collapse can occur both features (e.g. hidden features and representations) and weight matrices.
Encoder
Augmented Input
Block1
Block2
Representations
Xaug1
W1
Hidden
W
Haug1
Representations
Weight matrices
Existing methods
Orthogonality Regularization
augg1
W
W
Hidden features
Hidden features
Hidden features
Hidden festures
Uncontrolled
Guaranteed
Weight matrices
by theory
Representations
W1
W2
Z Zaug1
Figure 1: Illustration of dimensional collapse in SSL. We augmented input Xang1 as example: we assume that the encoder contains two basic blocks, each containing linear operation (e.g.. linear layer or convolutional layer) and an activation function. Dimensional collapse can occur in weight matrices (W1, W2), hidden features, and the finally obtained representations. Existing methods directly on representations and expect to affect hidden features and weight matrices indirectly, which has guarantee in theory; our method directly constrains weight matrices and indirectly influences hidden features and representations, which can be guaranteed by theoretical analysis.
To prevent dimensional collapse representations, depicted in Figure existing methods include modifying representations in downstream tasks (He & Ozay 2022), whitening representations directly (i.e. removing the projector) (Jing et al. (02T), incorporating regularizers on representations during pretraining (Huang et al. 2024 Hua et al. (021). However, whether they sufficiently prevent the dimensional collapse of weight matrices and hidden features remains unknown (i.c., no theoretical guarantee) (Pasand al. 2024). In Appendix A.I we further demonstrate that whitening representa- tions directly to eliminate the dimensional collapse of representations cannot adequately remove the dimensional collapse of weight matrices.
To address these challenges, we first time propose mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretrain- ing. natural that OR prevents the dimensional collapse of weight matrices as ensures weight matrices orthogonality, keeps the correlation between its filters as low as possible, and lets each filter have a norm of For features (e.g. hidden features and representations), orthogonal weight matrices promote uniform eigenvalue distributions and thus prevent the domination of eigenspaces by

---

limited number of large eigenvalues, as theoretically substantiated by Huang et al. 20188) Yoshida & Miyato (2017), Rodriguez et al. (2016).
In our study, we introduce and assess the effects of two leading orthogonality regularizers, Soft Orthogonality (SO) and Spectral Restricted Isometry Property Regularization (SRIP), on methods. We examine their integration with 13 modern SSL methods from Solo-learn and LightSSL. spanning both contrastive and non-contrastive methods (Chen, Fan, Girshick & He 2020 Chen et al. 2021 Grill
ct al. 2020 Caron et al. 2021 Dwibedi et al. 2021). Our findings indicate a consistent enhancement in linear probe accuracy on CIFAR-IO I00 using both CNNs and Transformer- based architectures and OR exhibits a good scaling law at the model scale. Furthermore, when applied to BYOL trained on IMAGENET-IK Ik, OR significantly improves the downstream performance on both classification and object detection tasks, suggesting its applicability to large-scale SSL settings. Remarkably, OR achieves these enhancements without necessitating modifications to existing SSL architectures or hyperparameters.
In summary, we present three major contributions:
We systematically study the phenomenon of dimensional collapse in SSL, including how feature whitening and network depth affect the dimensional collapse of weight matrices and hidden features.
We first time introduce orthogonal regularization (OR) as solution to prevent the di- mensional collapse of weight matrices, hidden features, and representations during SSL pretraining.
Our extensive experimental analysis demonstrates OR's substantial role in enhancing the performance of state-of-the-art joint-embedding SSL methods with a wide spectrum of backbones.
2 Related Work
2.1 Sell-Supervised Learning
Self-supervised Learning (SSL) aims to learn meaningful representations from unlabeled data. Existing SSL methods can be broadly classified into two categories: generative and joint embedding methods. This paper concentrates on joint-embedding methods, which learn representations by aligning the embeddings of different augmented views of the same instance. Joint-embedding methods further subdivide into contrastive and non-contrastive methods. Contrastive methods, such as those proposed by He et al. (2020). Chen, Kornblith, Norouzi & Hinton (2020), Chen et al. (2021), treat each sample as distinct class and leverage the InfoNCE loss (Oord et al. 2018) to bring representations of positive pairs closer together while distancing those of negative pairs in the feature space. These methods generally require substantial number of negative samples for effective learning. In contrast, non-contrastive methods eschew the use of negative samples. They instead employ various techniques such as deff-distillatio (Caron et al, 2021] Grill et al. (020)) CCen He 2021), clustering (Caron et al. 2018 2020 Pang al. 2022) and feature whitening (Bardes et al. 2022 Zbontar et al. 2021 Weng et al. 2022 2023). Our empirical findings indicate that incorporating
OR enhances the performance of both contrastive and non-contrastive SSL methods. The exploration of its effects on generative methods remains for future work.
2.2 Dimensional Collapse in SSL
Dimensional collapse plagues both generative and joint embedding SSL methods (Zhang al. 2022, Jing et al. 2021 Zhang et al. 2021 Tian et al. 2021). To prevent the dimensional collapse of representations, existing work has typically focused on imposing constraints on the covariance matrix of the representations, including modifying representations in downstream tasks (He & Ozay 2022). removing the projector (Jing al. 2021), incorporating regularizers on representations during pretraining (Huang et al. | Hua et al. 2021). However, these strategies face challenges such as performance degradation upon removing the projector, not addressing collapse during pre-training, and failing to prevent dimensional collapse in hidden features and weight matrices within the encoder (referred to Appendix A.T) This motivates us to regularize the weight matrices of the DNNs directly in SSL.
3

---

2.3 Orthogonality Regularization
Orthonomaality regularization, which is applicd in linear transformations, can improve the generaliza- tion and training stability of DNNs (Xie et al. 2017 Huang et al. 2018 Saxe et al. 2013). OR has demonstrated its effects tasks including superviseed/semi-sueervee
image classification, image retrieval, unsupervised inpainting, image generation, and adversarial training (Bansal et al. 2018 Balestriero et al. 2018 [Balestriero & Baraniuk 2020 Xie al. [017] HHuan et al. 2018). Efforts to utilize orthogonality in network training have included penalizing the deviation of the gram matrix of each weight matrice from the identity matrix (Xie et al. 2017 Bansal et al. 2018
Balestriero al.
2018 Kim & Yun 2022) and employing orthogonal initialization (Xie et al. 2017 Saxe et al. 2013). For more stringent norm preservation, some studies transform the convolutional layer into a doubly block-Toeplitz (DBT) matrix and enforce orthogonality (Qi et al. 2020 Wang et al. 2020)
In this work, first time investigate the efficacy of two orthogonality regularizers, Soft Orthogonality (SO) and Spectral Restricted Isometry Property (SRIP) in SSL (Bansal et al,2208)). These regularizers aim to minimize the distance between the gram matrix of each weight matrix and the identity matrix-measure measured Frobenius and spectral norms, respectively.
Preliminaries
3.1 Settings of Self-supervised Learning
In this section, we present the general settings for joint-embedding SSL methods. We consider large unlabelled dataset X E IR xD comprising N samples each of dimensionality D. The objective of SSL methods is construct an effective encoder f that transforms raw data into meaningful representations Z = f(X). where Z E N and M denotes the representation dimensionality. The learning process of SSL methods visually represented in Figure 2 where data augmentations transform X into two augmented views Xmugl Xuuga E PDN A typical joint- loint-embedding SSL architecture encompasses an encoder f and a projector p. These components yield encoder features 7uug1
and Zuuga f(Xamz)), as well as projection features Haug p( Zzugi Jand Hugz = Zuuzz) During training. the parameters of and p are optimized via backpropagation to minimize the discrepancy between Haugi and Hugg2 To prevent the encoder f from producing constant feature vector, contrastive methods utilize negative samples, and non-contrastive methods employ strategies such as the self-distillation technique.
The efficacy of the encoder f is usually assessed by the performance of the C-class classification as downstream tasks. Specifically, given labeled dataset containing samples X, E Sx and their corresponding labels Y, E SxC, where S the sample number. Then, linear layer parameterized by W. E CCM is appended on top of the learned representations Z, f(X.), and thus the classification task can be fulfilled by minimizing the cross-entropy between sof tmax(g( Zz.) and Y,. There are two strategies for the fine-tuning: 1) non-linear fine-tuning, which trains both and f in the downstream tasks, and 2) linear evaluation, which freezes f and only trains (referred to as the linear probe).
Augmented
Projector
Inputs
Encoder
Representations
Projector
features
Xaug1
P
Hawg
Inputs
X
Augimentation
Objective
Xmugz
Z.ugga
p
Hang?
Augmented
Encoder
Representations
Projector
Projector
Inputs
teatures
Figure Illustration of joint-embedding SSL methods. This is a general structure. Differrnt augmented inputs be passed either by shared weight Encoder and Projector or by independent Encoder and Projector, depending on different SSL methods.

---

3.2 Orthogonality Regularizers
We introduce two orthogonality regularizers: Soft Orthogonality (SO) and Spectral Restricted Isome- try Property Regularization (SRIP), which are seamlessly integrable with linear and convolutional layers.
Consider a weight matrix W& E Reupptx in a linear layer, where inpul and oulput denote the number of input and output features, respectively. In line with Bansal et al. (2018). Xie al. (2017). Huang et al. (2018), we reshape the convolutional filter to two-dimensional weight matrix W Pmppptx ourput while we still use the same notation W for consistency. To be specific, imput S H x Cnn and output = Cout, with Cin and Cout being the number of input and output channels, and S and H representing the width and height of the filter, respectively.
The SO regularizer encourages the weight matrix W to approximate orthogonality by minimizing the distance between its Gram matrix and the identity matrix. This quantified by the Frobenius norm as follows:
|| - -11
if input > output,
SO(W)= =
|| - -1p
otherwise,
where 7 is the identity matrix of appropriate size.
The SRIP regularizer employs the spectral norm to measure the deviation from orthogonality, which is defined as:
[oWTW-- - I), input > output,
SRIP(W)
(2)
OWWT - I), otherwise.
where (-) denotes the spectral norm operator. Due the high computational cost posed by the spectral norm, the power iteration method (Yoshida & Miyato 2017 Bansal et al. 2018) with two iterations used for the estimation. The process for estimating ( W
is:
||0||
u = (WTW D)v, e (WTW- - Tyu, oW -
(3)
||u||2
where E Rispat is a vector initialized randomly from normal distribution.
4 Incorporating OR into SSL
This section details the integration of OR with SSL methods. To be specific, we employ OR across the encoder, targeting both convolutional and linear layers during pretraining. We represent the SSL method's 's loss function LossssL. Our overall optimization objective is the minimization of the combined loss equation:
Loss LosssSL + 7 LossoR,
where LOSSOR is defined as Σwer SO(W) or E SRIP depending on the selected orthogonality regularizers. The term y serves as hyperparameter that balances the SSL objective and OR loss. Notably, we only perform OR on the weight matrices located within the linear and convolutional layers of the encoder f. OR provides a versatile regularization strategy for the encoder f. facilitating its application across various SSL methods without necessitating modifications to the network designs or existing training protocols.
Analysis of Dimensional Collapse and the Effects of OR in SSL
In this section, we show that dimensional collapse happens not only to the representations (i.e. output of the encoder), but also to weight matrices and hidden features of the encoder. We also compare the feature whitening technique used by one previous method, VICREG (Bardes al. (2022), with OR. Migrating to BYOL, we find that the feature whitening technique only solves the dimensional collapse at the feature level, but instead accelerates the collapse of the weight matrices, and even leads to lower performance of the downstream tasks as shown in Table [ In contrast, OR can
5

---

eliminate the dimensional collapse of weight matrices and thus the dimensional collapse of hidden features and representations.
In Appendix [A.1] we further reveal that the original VICREG has a dimensional collapse problem with its weight matrices, which could not be solved by removing its projector. Adding OR to VICREG eliminates this problem and boosts the performance.
Table I: Comparison of the feature whitening technique from VICREG and SO on CIFAR-10.
Methods
BOYL
BYOL with SO BYOL with feature whitening
Top-1
92.92 99.83
93.04
92.66
Top-5
99.84
99.79
To study the eigenspace of matrix, we utilize the normalized eigenvalues defined in He & Ozay 2022):
Definition (Normalized eigenvalues) Given a specific matrix T E RRDD where N is the sam- ple number and D is the feature dimension, we first obtain its covariance matrix ET E TR DxD Then we perform an eigenvalue decomposition on ΣT to obtain its eigenvalues {Xī D
{AT.... .A........ ....."""-- ..!".... .XT} descending order. And we obtain the normalized eigenvalues by di- viding all eigenvalues by the max eigenvalue XT, denoted as {XT /XT ..................... XT / ....... .X//A?)
1i=1 To simplify the denotation, we reuse { to denote normalized eigenvalues of
These normalized eigenvalues are less than or equal to 1, where a larger value in one dimension indicates more information contained, and vice versa. We argue that if normalized eigenvalues drop very quickly, this means that only a few dimensions in the eigenspace contain meaningful information and also means that dimensional collapse has occurred.
We train three BYO (Grill et al. 2020) models, without OR, with the feature whitening technique (e.g. Variance and Covariance regularization) from VICREG and with OR, respectively. We choose randomly initialized ResNet18 as the backbone (i.c. encoder) and train the three models on CIFAR-10 for 1.000 epochs, following the same recipe of Da Costa al, (022). Importantly, SO is selected as the orthogonality regularize, and y is set to le 6. For the feature whitening technique, we impose the Variance and Covariance regularization from VICREG on the output of the predictor in BOL as two additional loss terms, the former to ensure the informativeness of individual dimensions and the latter to reduce the correlation between dimensions. Following the solo-learn settings, we set the two loss term hyperparameters Tvic and Toic 0.004, and then tune the Vvic from le 3 to le 5.
After training, we calculate the normalized eigenvalues of both weight matrices and features (e.g. input features, hidden features, representations). ResNet18 contains four basic blocks, each containing four convolutional layers, and we visualize the normalized eigenvalues of the last convolutional layer in each block. Hidden features are the outputs of four basic blocks in ResNet18. We use the first batch in the test set as input features with batchsize 4, 4,096 and feature dimension 3072 332*32*3). Results are analyzed in the following sections.
5.1 Dimensional Collapse of Weight Matrices
We first examine the weight matrices of the encoder. Similar to 5.1 all the weight matrices are viewed two-dimensional matrices, and on top of them, we calculate their normalized eigenvalues. For specific weight matrix W E inpput output in a neural network layer, we denote {*," her normalized eigenvalues of this layer.
output the
As shown in Figure 3 X and Y axis is the index and i-th values of {AW putput respectively. is clear that with OR, the eigenvalues of the convolutional layers of different depths decay more slowly, which means that their filters are less redundant and more diverse. In particular, we note that the deepest convolutional layer (i.e. layer4_512) has a much faster decay rate of the eigenvalues compared to the other convolutional layers the absence OR. Notably, the feature whitening technique does not alleviate this phenomenon. OR could significantly improve this. OR requires the weight matrix to be as orthogonal as possible, which means that the diagonal elements of its covariance matrix are as identical possible, and the off-diagonal elements will be close to 0. Then

---

ctt/10, 2012
represertation ratton, 512
Normalised eigenvaltue
Iddden Netts ay 72064
Iddder, feort) (uerr)
6.1
0.1
Normaliees
0.01
0.11
12pr2_128
0.001
100
512
0.001
10
300
1000
73984
Eigenvalue rank
genvvllue rank
(a) Eigenvalues weights (without OR)
(b) Eigenvalues features (without OR)
Normalised eigenvalue
Nidden (eatalayyrr) 73064
Nidden, (eors] 34562
6.1
0.1
Iedden feuta (ayer) 20736
Nidtie, fedtee_ Byeroo 12000
0.01
1yyer, 544
122e2_112
Iyerl
1210
0.001
0.001
10
100
73984
Eigenvatue rank
gggnvvllue rank
(c) Eigenvalues weights (with feature whitening)
(d) Eigenvalues of features (with feature whitening)
1ra2p_3012
eigenveiue
Normalised eigenvellue
hiddlnn feetst
0.1
Iidden, featal (edoo/lyyrrr,227775
hiddee, (wwrs( 178m 2-
leper..5
1epe13 233
byen 512
2-15
0.001
0.001
Eigenvelue rank
Egenvvaluee
(e) Eigenvalues of weights (with OR)
(i) Eigenvalues of features (with OR)
Figure 3: Eigenspectra of both weights and features within the encoder (ResNet18). The features collected on the first batch of the test set (batchsize 4,096). We pretrain BYOL without OR, with feature whitening from VICREG, and with OR on CIFAR-10. The x-axis and y-axis are both log-scaled. The solid line represents that all eigenvalues are positive, the dashed line represents the existence of eigenvalues that are non- mossiiiieee, and the number of eigenvalues is represented behind the underline.
the eigenvalue decomposition of such covariance matrix will have elements in {A," W potpput that are all close to (verified Appendix A.2).
5.2 Dimensional Collapse of Features
As the weight matrices become orthogonal, the distribution of their outputs stabilizes, thereby preventing the dimensional collapse of hidden features and representations. This property has been demonstrated in vector form by Huang et al.) (018)) and is now presented in matrix form:
Proposition 1 For specific weight matrix W e Rinput and X R xinput, comprising N samples each of dimensionality input. We denote and Š as the sample means of X and S, re- spectively. Let XW, where WTW T. The covariance mairix of X is Σx (X (X-X)"(XXXX N-1
(1) If X and Σx I²1, then
and Es = o² (2) If input output, we have
aL ||S]] | 12. (3) Given the back- pack-propagated gradient 75 have aL 0s
||
0L
1|
7

---

The first point of Proposition] T illuustaaess that in each layer of DNNs, the orthogonal weight matrix preserves the normalization and de-correlation of the output S, assuming the input is whitened. This reveals that as the network gets deeper, the hidden features of each layer and the final representations do not tend to collapse. Moreover, orthogonal filters maintain the norm of both the output and the back-propagated gradient information in DNNs, as demonstrated by the second and third points of Proposition [
To verify that the dimensional collapse of features can be eliminated by OR, we visualize the normalized eigenvalues of features (input features, hidden features, and representations) as shown in Figure 3 Without the OR constraint, features located in deeper layers (i.e. representations) will have the fastest eigenvalues decay rate, and the distributions of eigenvalues of hidden features vary considerably at different depths. After adding the OR, it can be seen that the decay rates of hidden features at layers 3 and 4 are almost the same, while the eigenvalues of representations decay much more slowly. We also visualize the representations in Appendix [A.3] which also verifies that the dimensional collapse of the representations is mitigated. Interestingly, the effect of OR on the feature level is similar to that of the feature whitening technique, however, the latter is unable to eliminate the dimensional collapse in the weight matrices.
Numerical Experiments
We study the effects of OR on SSL methods through extensive experiments. we first demonstrate that OR improves the classification accuracy on CIFAR-10, CIFAR-100 100, and IMAGENET 100, and the improvement is consistent across different backbones and SSL methods. On the large-scale dataset IMAGENET-IK (Deng et al. 2009). OR boosts the classification accuracy on both in-distribution and out-distribution datasets (i.e. transfer learning datasets), demonstrating consistent improvement. Moreover, OR also enhances the performance in downstream tasks(e.g. object detection).
Baseline methods and datasets. We evaluated the effect of adding OR to 13 modern SSL methods, including 6 methods implemented by solo-Icarn (MOCOv2plus, MOCOv3, DINO, NNB YOL, B YOL, VICREG) (Chen He 202T, Chen et al. 2021 Grill et al. 2020 Dwibedi et al. 2021 Caron et al. >021) and 10 methods implemented by LightlySSL. (Barlow Toolloowwwing Twins, BOL,, DCL. DCLW, DINO, Moco, NNCLR, SimCLR, SimSiam, SwaV) (Zbontar et al. 200TT, Yeh et al. 2022 Caron et al. 2020 Chen & He/ 2021). We pretrain SSL methods on CIFAR-I0, CIFAR- 100, IMAGENET-ION and IMAGENET-IK and evaluate transfer learning scenarios on datasets including CIFAR-100.
CIFA-10 (Krizhevsky et al. 2009), Food-10 (Bossard et al. 2014), Flowers-102 (Xia et al. 2017),
DTD (Sharan et al. 2014), GTSRB (Haloi 2015). We evaluate the objection detection task on PASCAL VOC2007 and VOC2012 (Everingham et al. 2010). Detailed descriptions of datasets and baseline SSL methods are shown in Appendix|A.A. A.4 and A.5] respectively.
Training and evaluation settings. For each SSL method, we use the original settings in solo- learn (Da Costa et al. 2022) and LightlySSL. These settings include the network structure, loss function, training policy (training epochs, optimizers, and learning rate schedulers) and data augmen- tation policy. The splits the training and test set follow torchvision Marcel & Rodriguez (2010). For all the classification tasks, we report the linear probe or KNN accuracy; for the objection detection task, we perform non-linear fine-tuning. Details of training, parameter tuning, and evaluation are presented in Appendix A.6 It is worth noting that the Solo-learn and LightlySSL setups are not the same as the official implementation of the SSL methods, e.g., there is no use of multi-crop augmentation in DINO, and there is no exceptionally long training epoch. We leave experiments on migrating OR to the official implementation for future work.
Recipe of adding OR. For OR, y of SRIP is tuned from (1e 3, le 4. - 5} and of SO is tuned from {le - 5, le 6, le 7) on validation set. When you want to add OR to your SSL pre-training, you simply pass the encoder into the loss function, and then you just need to set of the OR according to the backbone and regularizer you use as shown in T Appendix A.6
6.1 OR is Suitable for Different Backbones and SSL Methods
After pretraining on CIFRA-100, for each SSL method, we report the corresponding classification accuracy as shown in Table Both two orthogonality regularizers consistently improve the linear classification accuracy. Note that OR boosts the performance of both constrastive (MoCov2plus,
8

---

Mocov3) and non-contrastive methods(BYOL. NNBYOL. DINO) as they are all susceptible to dimensional collapse (Zhang et al. 2022 Dng et al. 2021 Zhang et al. 2021 Tian et al. 202T). Non-contrastive methods gain more improvements in contrast to contrastive methods. When we use ResNet18, 8, MOCOv3 improves 3% on Top-1 I accuracy while DINO and NNBYOL improve 6% and 5%, respectively. OR can also boost the performance of the Sota method like BYOL by 1%. When we scale to ResNet 50 and WideResnet28w2, OR consistently boosts their performance. Moreover, the additional time overhead of adding OR to SSL is low compared to the original training time (referred to A.7
Table 2: Classification aceuracy CIFAR-100 (CNN backbones). SSL methods (in Solo- trained with or without OR on CIFAR-100. The best results are in bold, the second best in italics.
Methods
MoCOv2plus
MOCO+3
DINO
NNBYOL
BYOL
Encoder
Regularizer
Top-1
1op-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-3
Top-1
Top-3
69.51
91.32
67.13
89.83
59.13
86.41
68.87
90.98
71.15
92.17
ResNet18
SO
69.72
91.56
68.63
90.85
61.41
87.94
71.40
92.12
72.15
92.48
SRIP
69.67
91.92
69.37
90.88
62.75
88.37
71.97
92.80
71.52
92.49
73.70
93.16
66.87
89.47
52.67
80,68
72.31
92.94
74.20
93.44
ResNet50
SO
73.40
93.45
69.22
90.76
57.30
84.10
72.87
92.77
74.57
93,83
SRIP
73.32
93.20
70.30
90.97
59.93
86,20
72.97
92.91
74.39
93.61
61.80
87.69
57.73
84.82
53,42
82.62
64,00
89.49
60.87
86.96
WideResnet28w2
SO
62.09
87.87
58.86
85.69
56.07
84.87
64.50
89.22
60.96
87.34
SRIP
61.37
87.94
58.79
85.17
53.93
83.27
64.23
89.00
61.10
87.31
In addition to CNN backbones, OR is also able to improve SSL performance on Transformer-based backbones (e.g., VIT) (Han et al. 2022 Dosovitskiy et al. 2020 Zhou et 2021]. We pretrain DINO on CIFAR-100 with different depths of VITs. As shown in Table 3 with the increasing depth of the VIT, the original DINO performance is increasing, and OR is able to increase their performance even further, which exhibits good scaling law. Interestingly. under the Transformer-bate OR able improve performance more (up 12%) compared CNN backbones. This is consistent with some existing studies (RoyChowdhury et al. (017) that linear layers are more likely to have redundant filters than convolutional layers DNNs, L..., more prone to dimensional collapse.
architecture,
Table 3: Classification accuracy 100 (VITs). DINO (in Solo-learn is trained with or without OR on CIFAR-100.
Encoder
VIT-tiny
VIT-small
VIT-base
Regularizer
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
54.16
83.37
62.02
87.80
64.12
88.83
SO
60.84
87.65
64.95
89.47
66.91
90.42
To evaluate OR on more SSL methods, under the LightSSI. framework, we test SO on CIFAR- 100 IMAGENET-100, and IMAGENET-1K. OR still consistently improves the performance of various SSL methods as shown in Table 4
Table 4: Performance of SSL methods on LightlySSL. ResNet18 is used on CIFRA-10, CIFAR- 100 and IMAGENET-100, and ResNet50 is employed on IMAGENET-IK.
Methods
CIFAR-I0 (Epoch 200)
CIFAR- 10 (Epoch 400)
IMAGENET-IOX
IMAGENET-IK
original
with SO
original
with SO
original
with SO
original
with SO
Barlow Twins
83,58
84.78
85.91
86.25
56.6
57.0
BYOL
86.94
87.01
89.64
90.02
51.7
52.1
DCL.
84.10
85.99
$6.27
DCLW
82,42
82.73
85.25
85.71
DINO
59.77
60.40
Moco
85.19
85.32
NNCLR
82.31
82.34
SimCLR
84.55
84.84
SimSiam
79.27
84.31
SwaV
83.10
83.67
6.2 OR Works on Large-scale Dataset
We demonstrate the effects of OR on the large-scale dataset IMAGENET-IK. Specifically, we pre-train three BYOL models- BYOL without OR, BYOL with SO, and BYOL with SRIP on IMAGENET-IK

---

(with ResNet50), For each testing classification dataset, we report the accuracy as shown in Table and 6
Table 5: Classification and objection detection performance. BYOL is trained with or without OR on IMAGENET-IK (ResNer50 with batchsize 128, Epoch 100). The best results are in bold, the second best in italics.
Dataset
IMAGENET-IK
Dataset
VOC 2007+2012
Regularizer
Top-1
Top-5
Pretraining methods
AP
AP30)
AP15
65.81
87.06
Scratch
33.8
60.8
33.1
67.8d
BYOL without
44.74
76.04
SRIP
67.91
88,20
BYOL with SO
$3.81
81.46
59.43
Table 6: Classification accuracy on transfer learning datasets.
Dataset
FoodT01
Hlowes102
DTD
GTSRB
CIFAR-10
CITAR- T00
Regularizer
Top-1
Top-5
Top-1
Top-3
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
59.838
83.945
73.817
91.039
68.777
91.596
63.539
92.035
79.7
99.12
52.09
81.89
SO
63.624
86,444
78.956
93.397
70.851
92.819
68.725
93.99/
83,58
99.41
57.57
85,38
SRIP
63.628
86.420
79.33
94.243
70.426
92.234
69.256
94.347
84.26
99.39
57.84
85.87
We can observe that OR not only improves the accuracy on IMAGENET-IK but also on all the transfer learning datasets. OR improves TOP-1 accuracy by 3% in IMAGENET-Ik and by 3% to 9% in each transfer learning datasets. The transfer learning task evaluates the generality of the encoder as it has to encode samples from various out-of-distribution domains with categories that may not have seen during pretraining. OR also significantly improves SSL's performance in the objection detection task by 20% on AP. The above results are close Chen et al. (2021) Da Costa et al. (2022). Weng et al. (2023) where also training only 100 epochs.
Table 7: Classification accuracy on IMAGENET-IK Ik (pretrained with different epochs and batchsizes).
Dataset
IMAGENET-IK Ik
Pretraining settings
Epoch 100 batchsize 128
Epoch 200 batchsize 256
Regularizer
Top-1
Top-5
Top-1
Top-5
65.81
87.06
69.76
89.10 10
SO
67.84
88.18
70.16
89.47
Considering that training epoch and batchsize during pretraining significantly impact the performance Chen et al.] (202I). Huang et al. (2024). Lavoie et al.] (2022), we further increase the training epoch(200) and batchsize(256) and scale up the learning rate accordingly. As shown in Table 7 OR consistently improves the performance.
7
Conclusions
The existing studies focus on the dimensional collapse of representations and overlook whether weight matrices and hidden features also undergo dimensional collapse. We first time propose mitigation approach to employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matrices, thus safeguarding against the dimensional collapse of weights, hidden features, and representations. Our empirical investigations demonstrate that OR significantly enhances SSL method performance across diverse benchmarks, yielding consistent gains with both CNNs and Transformer-basee archi- tectures as the backbones. Importantly, the time complexity and required efforts on fine-tuning are low and the performance improvement is significant, enabling it to become useful plug-in in in various SSL methods.
In terms of future research, we wish to examine the effect of OR on other pre-training foundation models, such as vision generative SSL models such as MAE (He (022). auto- regression models like GPTs and LLaMAs (Radford et al. 2018 2019 Brown et al. 2020 Touvron et al. 2023), and Contrastive Language-Iman
Pre-training models (Radford al. 2021 et al. 2022). believe OR is a pluggable and useful module to boost the performance of vision and language foundation models. In fact, this paper the first to test the effectiveness of OR in Transformer-basee architecture and is reasonable to believe that it will perform well in these domains.
10

---

Acknowledgments
The work described in this paper was supported by the National Natural Science Foundation of China (No. 52102385), grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU/25209221 and PolyU/15206322), and grants from the Otto Poon Charitable Foundation Smart Cities Research Institute (SCRI) at the Hong Kong Polytechnic University (Project No. PO043552). The contents of this article reflect the views of the authors, who are responsible for the facts and accuracy of the information presented herein.
References
Balestriero, R. & Baraniuk, R. G. (2020), 'Mad max: Affine spline insights into deep learning', Proceedings of the IEEE 109(5), 704-727.
Balestriero, R., Ibrahim, M., Sobal, V. Morcos, A., Shekhar, s., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y. et al. (2023), 'A cookbook of self-supervised learning', arXiv preprint arXiv:2304. 12210.
Balestriero, R. et al. (2018), A spline theory of deep learning, in 'International Conference on Machine Learning', PMLR, pp. 374-383.
Bansal, N., Chen, X. & Wang, Z. (2018), 'Can gain more from orthogonality regularizations in training deep networks?", Advances in Newral Information Processing Systems 31.
Bardes, A., Ponce, J. & LeCun, Y. (2022), Variaaceiinaarincc-ccaaaaini regularization for self-
supervised learning', /CLR, Vicreg 1, 2.
Bossard, L., Guillaumin, M. & Van Gool, (2014), Food-101 pood-101-mining discriminative components with random forests, in "Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part VI 13', Springer, pp. 446-461.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020). 'Language models are few-shot learners', Advances in neural information processing systems 33, 1877-1901.
Caron, M., Bojanowski, P., Joulin, A. Douze, M. (2018), Deep clustering for unsupervised learning of visual features, in 'Proceedings of the European conference computer vision (ECCV)', pp. 139.149.
Caron, M., Misra, I,, Mairal, J., Goyal, P., Bojanowski, P. & Joulin, A. (2020), 'Unsupervised learning of visual features by contrasting cluster assignments', Advances neural information processing systems 33, 99 9912-9924.
Caron, M.. Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P. & Joulin, A. (2021), Emerging properties self-supervised vision transformers, in 'Proceedings of the IEEE/CVF international conference on computer vision', pp. 9650-9660.
Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. (2020), A simple framework for contrastive learning of visual representations, in 'International conference on machine learning'. PMLR, pp. 1597-1607.
Chen, X., Fan, H., Girshick, R. & He, K. (2020), 'Improved baselines with momentum contrastive learning'. arXiv preprint arXhv:2003.04297.
Chen, X. & He, K. (2021), Exploring simple siamese representation learning, in PProceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 15750-15758.
Chen, X., Xic, & He, K. (2021), An empirical study of training self-supervised vision transformers. in 'Proceedings of the IEEE/CVF international conference on computer vision', pp. 8640-96449
Da Costa, V. G. T., Fini, E., Nabi, M., Sebe, N. & Ricci, E. (2022), 'solo-learn: A library of self- supervised methods for visual representation learning'. Journal of Machine Learning Research 23(56), 1-6.

---

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K. & Fei-Fei, L. (2009), Imagenet: A large-scun scale hierarchical image database, in '2009 IEEE conference on computer vision and pattern recognition', leee, pp. 248-255.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. et al. (2020), 'An image is worth 16x16 words: Transformers for image recognition scale', arXiv preprint arXiv:2010.11929
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, & Zisserman, A. (2021), With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, in Proceedings of the IEEE/CVF International Conference on Computer Vision', pp. 9588-9597
Everingham, M., Van Gool, L., Williams, C. K., Winn, & Zisserman, A. (2010). 'The pascal visual object classes (voc) challenge', International journal of computer vision 88, 303-338.
Girshick, R., Donahue, J., Darrell, T. & Malik, (2014), Rich feature hierarchies for accurate object detection and semantic segmentation, in 'Proceedings of the IEEE conference on computer vision and pattern recognition' pp. 580-587.
Grill, J...., Strub, F., Altché, F., Tallec, C.,, Richemond, P., Buchatskaya, E.,, Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M. et al. (2020), 'Bootstrap your own latent-a new approach to self-supervised learning', Advances in neural information processing systems 33,
Haloi, M. (2015), 'Traffic sign classification using deep inception based convolutional networks', arXiv preprint arXiv:1511.02992.
Han, K., Wang. Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y. et al. (2022), 'A survey on vision transformer', IEEE transactions on pattern analysis and machine intelligence 45(1), 87-110.
HaoChen, J. Z., Wei, C., Gaidon, A. & Ma, T. (2021), 'Provable guarantees for self-supervised deep learning with spectral contrastive loss', Advances in Neural Information Processing Systems 34, 5000-5011.
He, B. & Ozay, M. (2022), Exploring the gap between collapsed & whitened features in self- supervised learning, in 'International Conference on Machine Learning,, PMLR, pp. 8613-8634.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P. & Girshick, R. (2022), Masked autoencoders are scalable vision learners, 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 16000-16009,
He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. (2020), Momentum contrast for unsupervised visual representation learning, in 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 9729-97335
Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y. & Zhao, H. (2021), On feature decorrelation in self-supervised learning, in 'Procedding of the IEEE/CVF International Conference on Computer Vision pp. 5598-9608.
Huang, H., Campello, R. J., Erfani, S. M., Ma, X., Houle, M. E. & Bailey, J. (2024), 'Ldreg: Local dimensionality regularized self- learning', arXiv preprint arXiv:2401. 10474.
Huang, L., Liu, X., Lang, B., Yu, A., Wang, Y. & Li, B. (2018), Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, in 'Proceedings of the AAAI Conference on Artificial Intelligence', Vol. 32.
Jing, L.., Vincent, P., LeCun, Y. & Tian, Y. (2021), 'Understanding dimensional collapse in contrastive self-supervised learning', arXiv preprint 10009:21
Jing, L., Zbontar, J. et al. (2020), 'Implicit rank-minimizing autoencoder', Advances in Neural Information Processing Systems 33, 14736-14746.
Kim, & Yun, S.-Y. (2022), 'Revisiting orthogonality regularization: a study for convolutional
neural networks in image classification', IEEE Access 10, 69741-69749.
12

---

Krizhevsky, A., Hinton, G. et al. (2009), 'Learning multiple layers of features from tiny images'.
Lavoie, S., Tsirigotis, C., Schwarzer, M.. Vani, A., Noukhovitch, M., Kawaguchi, K. & Courville, A. (2022), 'Simplicial embeddings in self-supervised learning and downstream classification arXiv preprint arXiv:2204.00616.
J. Li, D., Xiong, C. & Hoi, S. (2022), Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 'International conference on machine learning'. PMLR, pp. 12888-12900
Li, J., Zhou, P., Xiong, C. & Hoi, S. C. (2020), Prototypical contrastive learning of unsupervised representations" arXiv preprint arXiv:2005.0499 04966
Li, X., Chen, S. & Yang. J. (2020), Understanding the disharmony between weight normalization family and weight decay, in PProceddings of the AAAI Conference on Artificial Intelligence', Vol. 34, pp. 4722-
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, & Zitnick, (2014), Microsoft coco: Common objects in context, in 'Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13', Springer, pp. 740-755.
Marcel, S. & Rodriguez, Y. (2010), Torchvision the machine-vision package of torch, in 'Proceedings of the 18th ACM international conference on Multimedia', pp. 1485-1488.
McInnes, I.., Healy, J. & Melville, J. (2018), 'Umap: Uniform manifold approximation and projection for dimension reduction', arXiv preprint 1802.03420
Misra, I. & Maaten, L. d. (2020), Self-supervised learning of peeext-invariint representations, in 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 6707- 6717.
Oord, A. v. d., Li, Y. & Vinyals, O. (2018), 'Representation learning with contrastive predictive coding'. arXiv preprint arXiv: 1807.0378.
Pang, B., Zhang, Y., Li, Y., Cai, J. & Lu, C. (2022), Unsupervised visual representation learning by synchronous momentum grouping, in 'European Conference on Computer Vision', Springer, pp. 265-282.
Pasand, A. S.S Moravej, R., Biparva, M. & Ghodsi, A. (2024), 'Werank: Towards rank degradation pre- vention for self-supervised learning using weight regularization', arXiv preprint arXiv:2402.09586
Qi, H., You, C., Wang, X., Ma, Y. & Malik, (2020), Deep isometric learning for visual recognition, in 'International conference on machine learning', PMLR, pp. 7824-7835.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, al. (2021), 'Clip: Learning transferable visual models from natural language supervision', arXiv preprint arXiv:2103.000
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. et al. (2018), 'Improving language under- standing by generative pre-training'.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, et al. (2019), 'Language models are unsupervised multitask learners", OpenAl blog 188), 9.
Rodríguez, P., Gonzalez, J., Cucurull, G., Gonfaus, J. M. & Roca, X. (2016), 'Regularizing cnns with locally constrained decorrelations' arXiv preprint arXiv:1611.01966
RoyChowdhury, A., Sharma, P., Learned-Miller, E. Roy, A. (2017), Reducing duplicate filters in deep neural networks, in 'NIPS workshop deep learning: Bridging theory and practice', Vol. 1, p. 6.
Saxe, A. M., McClelland, L., & Ganguli, S. (2013), 'Exact solutions to the nonlinear dynamics of learning in deep linear neural networks", arXiv preprint arXiv:1312.6122
13

---

Sharan, L., Rosenholtz, R. & Adelson, E. H. (2014), 'Accuracy and speed of material categorization in real-world images', Journal of vision 14(9). 12-12.
Tian, Y., Chen, X. & Ganguli, S. (2021), Understanding self-supervised learning dynamics without contrastive pairs, in 'International Conference on Machine Learning', PMLR, pp. 10268-10278.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, et al. (2023), 'Llama: Open and efficient foundation language models', arXiv preprint ar arXhv:2302.33771
Wang. J., Chen, Y., Chakraborty, R. & Yu, X. (2020), Orthogonal convolutional neural networks, in PProceddnngs of the IEEE/CVF conference on computer vision and pattern recognition', pp. 11505- 11515.
Weng, X., Huang. I.., Zhao, I.,, Anwer, R., Khan, H. & Shahbaz Khan, F. (2022), 'An investigation into whitening loss for self-supervised learning', Advances in Neural Information Processing Systems 35,
Weng, X., Ni, Y., Song, T., Luo. J., Anwer, R. M., Khan, S., Khan, F. S. & Huang, L. (2023), 'Modulate your spectrum in self sulf-supe learning', arXiv preprint arXiv:2305.16789
Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y. Girshick, R. (2019), 'Detectron2', https://github comn/faaebboorreerccceeeeeeeee
Xia, X., Xu, C. & Nan, B. (2017), Inception-v3 for flower classification, in '2017 2nd international conference on image, vision and computing (ICIVC)', IEEE, pp. 783-787.
Xie, D., Xiong, J. & Pu, (2017), All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation, in 'Proceedings the IEEE Conference on Computer Vision and Pattern Recognition', pp. 6176- 6185.
Ych, C.-H., Hong, C.-Y,, Hsu, Y.-C., Liu, T.-L., Chen, Y. & LeCun, Y. (2022). Decoupled contrastive learning, in 'European conference on computer vision', Springer, pp. 668-684.
Yoshida, Y. & Miyato, T. (2017), 'Spectral norm regularization for improving the generalizability of deep learning' arXiv preprint arXiv: /705. 10941
Zbontar, J., Jing, L., Misra, I., LeCun, Y. Deny, S. (2021), Barlow twins: Self-supervised learning via redundancy reduction, in "International conference on machine learning', PMLR, pp. 12310- 12320.
Zhang, Q. Wang, Y. Wang, Y. (2022), 'How mask matters: Towards theoretical understandings of
masked autoencoders' Advances in Neural Information Processing Systems 35, 27127-271399
Zhang, S., Zhu, F., Yan, J., Zhao, R. & Yang, X. (2021), Zero-cl: Instance and feature decorrela- tion for negative-free symmetric contrastive learning, in 'International Conference on Learning Representations'.
Zhou, D.,. Kang. B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q. & Feng, (2021), 'Deepvit: Towards deeper vision transformer', arXiv preprint a7Xiv:2103.11886.
14

---

A Appendix supplemental material
A.1 Effects of Representation Whitening on the Encoder
In this section, we explore the effect of whitening representations on hidden features and weight matrices the encoder. To be specific, similar the settings of we train three VICREG Bardes et al. (2022) models: original VICREG, VICREG without projector (Li, Chen & Yang 2020). and VICREG with OR.
Original VICREG adds two regularization terms (variance and covariance regularization) to whiten the projector features. We use Xaug1 example to introduce them.
Variance regularization. The variance regularization term ensures that each dimension of the learned representation Z maintains non-trivial variance. This is critical to prevent the collapse of dimensions, where a model might ignore certain informative variations in the data. Mathematically, the variance regularization can be expressed follows:
Lyar
1 D
Σ d=1
m - S(za,
(5)
where D is the dimensionality of Zaug1 f(Xaugi), 2a represents the d-th dimension of Zaugl S(2a, Var(=z)) + is the regularized standard deviation of zu across different samples, and a threshold parameter that dictates the minimum desired standard deviation for each dimension.
Covariance regularization. The covariance regularization term is designed to decorrelate the different dimensions of Zaug1. By minimizing the off-diagonal elements of the covariance matrix of Zang1 this term helps ensure that different dimensions capture distinct aspects of the data, thereby preventing redundancy in the representation. The covariance regularization is defined as:
Lov
Σ (Cov(2i,,)))
(6)
where Cov(z;, (zi, 2j) denotes the covariance between the i-th and j-th dimensions of Zangt This term effectively encourages the representation to have orthogonal dimensions, which is beneficial for learning independent features.
As for the VICREG without projector, we discard the projector and apply the SSL objective directly to the representations, which ensures that the representations are whitened (no dimensional collapse in representations), i.e., minimize the correlation among dimensions and make each dimension rich in information. For OR, choose SO the regularizer and set y as le 6. We then experimentally observe that guaranteeing that dimensional collapses do not occur in representations or projector features (i.e., VICREG without projector and projector features) does not guarantee that dimensional collapses do not occur in weight matrices in the encoder. Moreover, discarding the projector even damages the performance of the original VICREG, while OR still boosts the performance as shown in Table g
Table 8: Performance comparison of different VICREG configurations.
Models
VICREG without Projector
VICREG
VICREG with SO
Top-1 Top-10
88.64 64
91.64
92.35 99.79
99.57
99.73
As shown in Figure 4 the weight matrices of the original VICREG suffer from dimensional collapse and whitening representations directly (i.e., getting rid of the projector) even makes the eigenvalue decay faster for the weight matrices. OR can alleviate this phenomenon.
A.2 Visualization of Weight Matrices
In this section, layer4 of ResNet18 pretrained with BYOL CIFAR-10 is visualized. To be specific. we calculate the correlation coefficient matrix of the weight matrix and then plot the HeatMap of the correlation coefficient matrix and the results of Spectral Biclustering. As shown in Figure 5
15

---

Normalised eigervellue
6.1
0.1
Normalies
(1a18 dfrro_inags 2072
102 Cottaataaiiti..3 2112
0.01
0.01
3:dden Pratolayera
12pr2_128
Jestelaapr11__0001
frstolayerli
0.001
100
512
0.001
10
300
1000
73984
Eigenvalue rank
genvvllue rank
(a) Eigenvalues weights VICREG)
Eigenvalues of features(VICREG)
Normalised eigenvalue
Nidden (watalayerr) 73064
Iidden, (wers 34562
6.1
0.1
Iidden festa ay 2013 20136
Nidtie, festte Byeroo 12000
0.01
1ayer, .54
122er_112
1212
0.001
0.001
10
100
Eigenvatue rank
Eigenvalue rank
(c) Eigenvalues of VeighI without projector) (d) Eigenvalues of features without projector)
1rapp_3012
eigenvelue
Normalised eigenvvllue
Iiddinn feets[
0.1
Iidden, featal 2aoolllyyrrl2277732
hidden, (wwns( 1786 1-
0.01
leper.
1225
1epe13 233
byen 512
2-15
0.001
0.031
Eigenvelue rank
Eggenvellue rank
(e) Eigenvalues of Weights with SO)
(1) Eigenvalues of features(VICREE with SO)
Figure 4: Eigenspectra of both weights and features within the encoder (ResNet18). The features are collected the first batch of the test (batchsize 4096). We pretrain original VICREG, VICREG without projecto, and VICREG with OR on CIFAR-10 10. The x-axis and y-axis are both log-scaled. The solid line represents that all eigenvalues are positive, the dashed line represents the existence of eigenvalues that are non-positive, and the number of eigenvalues is represented behind the underline.
HeatMap can intuitively indicate that the correlation of non-diagonal elements is constrained to by OR. The Biclustering results show an obvious blocky structure, which means that there is clustering between filters, and the weight matrix is low-rank and redundant.
A.3 Visualization of Representations
We used BYOL (ResNet18) for pretraining on CIFAR-10 10. After pretraining, we perform dimension reduction and visualization of learned representations using UMAP (McInnes et al. 2018). As shown in Figure in the absence of OR, there is a tendency for the cluster centers of each category to move closer together and more outliers appear. This is due the fact that in the absence of OR, BYOL produces representations dominated by some extremely large eigenvalues (i.e. dimensional collapse), which is consistent with results in Section 5.2
16

---

Correlation Matrix Heatmap
Correlation Matrik Heatmao
100
100
300
400
400
500
500
100
200
300
400
500
100
200
300
400
500
(a) HcatMap (without OR)
(b) HeatMap (with OR)
Biclustered Matrix
Biclustered
0
100
200
300
400
500
100
200
300
400
500
100
100
200
200
300
300
400
500
500
Biclustering (without OR)
(d) Biclustering (with OR)
Figure 5: HeatMap is the visualization of the absolute value of the correlation coefficients among filters of the weight matrix (layer4). Biclustering is the visualization of the results of spectral biclustering. can be seen that OR significantly reduces the correlation and removes the clustering patterns among filters from the heatmap and biclustering, respectively.
17

---

(a) BYOL without OR
(b) BYOL with OR (SO)
Figure 6: Visualization of Representations
18

---

A.4 Datasets
We utilized several datasets for pretraining and evaluating SSL methods. Below we provide a detailed description of these datasets;
IMAGENET-IK (Dnn et al. 2009): A large dataset contains 1,281,167 training images, 50,000 validation images, and 100,000 test images, which spans 1000 object classes,
IMAGENET-100 (Deng al. 2009): subdataset of IMAGENET-IK, KK containing 100 classes with 1000 training data and 300 test data per class.
CIFAR-10 (Krizhevsky et al. 2009): Comprising 60,000 images in 10 classes, with each class containing 6,000 images. The split includes 50,000 training images and 10,000 test images.
CIFAR-100 (Krizhevsky et al. 2009): This dataset consists of 60,000 images divided into 100 classes, with 600 images per class. The dataset is split into 50,000 training images and 10,000 test images.
Food-101 (Bossard al. 2014): This dataset includes 101,000 images of food dishes categorized into 101 classes, with each class having approximately 1,000 images.
Flowers-102 (Xia et al. 2017): Contains 8,189 images of flowers from 102 different categories. Each class consists between 40 and 258 images.
DTD (Sharan al. 2014): The Describable Textures Dataset (DTD) includes 5,640 images
categorized into 47 different texture categories.
GTSRB (Haloi 2015): The German Traffic Sign Recognition Benchmark (GTSRB) dataset consists of over 50,000 images of traffic signs across 43 categories.
PASCAL VOC2007 and VOC2012 (Lin et al. 2014): Used for evaluating objection tasks, this dataset includes complex everyday scenes with annotated objects in their natural context. The objection detection task contains 20 categories. We use the VOC2007 and VOC2012 train-val (16551 images) as the training set and then report the performance on the VOC2007 test set (4952 images).
Each dataset was carefully curated to support the training and validation of our models, ensuring a comprehensive evaluation across various image classification and segmentation tasks.
A.5 Joint-emb SSL methods
Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations without the need for labeled data. This appendix provides concise overview of several SSL methods used in this paper.
MOCOv2, introduced by Chen He (02I)) on top of MOCO's momentum encoder and the use of the dynamic dictionary with queue to store negative samples (He et al. 2020), adds the MLP projection head and more data augmentation. Compared to MOCOv2, MOCOv2plus uses a symmetric similarity loss.
MoCov3 (Chen et al. 2021) makes some improvements on the basis of v1/2, firstly, because the batchsize is large enough when training V3, the memory queue removed, and the negative samples are sampled directly from the batch. Secondly, symmetric contrastive loss is used, and finally, an extra prediction head added the original encoder, which is two-layer fully connected layer.
Bootstrap Your Own Latent (BYOL), proposed by Grill et al. (2020), introduces novel approach SSL that does not rely on negative pairs. Instead, BYOL employs dual-network architecture where the encoder learns to predict the representations of the momentum encoder. Through series of updates (i.e. EMA), where the momentum encoder gradually assimilates the encoder's ' weights, B BYOL effectively learns robust representations. The success of BYO depends not only on the EMA, but also on its additional projector and the BN in the projector to avoid a complete collapse of the encoder. This method challenges the conventional wisdom that contrastive learning requires negative pairs, opening new avenues for SSL research.
19

---

Expanding the ideas of BYOL, NNBYOL (Dwibedi et al. (021) introduces the concept using nearest neighbors to augment the learning process. By leveraging the similarities between different instances in the dataset, NNBYOL. aims to refine the quality of the learned representations further. This approach underscores the potential of incorporating instance- level information into the SSL framework, enhancing the discriminability and robustness of the resulting models.
DINO (Caron al. 2021) uses a self-distilling architecture. The outputs of the teacher networks (i.e. the momentum encoder) are subjected to centering operation by averaging over batch, and each network outputs a K-dimensional feature that is normalized using Softmax. The similarity between the student model (i.e. the encoder) and the teacher model is then computed using cross-entropy loss the objective function. stop-gr stop-gradient operator is used on the teacher model to block the propagation of the gradient, and only the gradient is passed the student model make it update parameters. The teacher model updated using the weights of the student model (i.e. EMA).
Barlow Twins computes the correlation matrix between the embeddings of two different views of the same sample and avoids collapse by making it close possible the unit matrix. This approach makes the embeddings between the two views of the sample as similar as possible while minimizing the redundancy between vector components.
VICREG avoids the complete collapse problem with variance and covariance regularization.
DCL and DCLW removes the NPC effect of infoNCE loss by getting rid of the positive term from the denominator and thus significantly improves the learning efficiency
SimSiam achieves very strong baseline without using large batchsize, negative samples, momentum encoder using only the stop-gradie gradient operation.
SwaV trained by predicting the clustering assignment of another view and also introduces multi-crop, which increases the number of views by reducing the image size without increasing the extra memory and computational requirements.
SimCLR establishes simple and effective architecture for contrastive learning by increasing the batchsize, augmenting the data and adding a nonlinear projector after the representation.
A.6 Hyper-parameters of Pretraining and Evaluation
For each SSL method, use the original settings of Solo-learn and LightlySSL (Da Costa et al. 20022). These settings include the network structure, loss function, training policy, and data augmentation policy. Considering that we use numerous SSL methods and that our setup is exactly the same as them, please go to their official implementation.
For OR, the appropriate regularization term y generally depends only on the backbone used by SSL and the orthogonality regularizer (SRIP or SO) chosen. As shown in Table when you want add OR to your SSL pre-training, you simply pass the encoder into the loss function, and then you just need set 7 of the OR according to the backbone and regularizer you use.
Table 9; The recipe of adding OR
Encoder
Regularizer
Regularization term
SO
lc-6
ResNet18
SRIP
le-3
SO
Ic-6
ResNet50
SRIP
le-3
SO
le-6,le-7
WideResner28w2
SRIP
le-4,le-5
VIT-tiny
SO
le-5
VIT-small
SO
le-5
VIT-Base
SO
Ic-6
For the classification tasks, due to computational constraints, we do not perform non-linear fine-tuning
classification tasks. Instead, we perform linear probe or KNN to evaluate the quality obtained
representations as typically done in the literature (Huang et al. 2024 Li. Chen & Yang 2020
Lavoie
20

---

et al. 2022). To be specific, for each SSL method and dataset, after pretraining, We train linear classifer on top frozen representations of the training set. Then we report the Top-1 and Top-5 linear classification accuracy on the test set. When training the linear classifier, we use 100 epochs, weight decay 0.0005, learning rate 0.1 (we divide the learning rate by factor of 10 on Epoch 60 and 100), batchsize 256, and SGD with Nesterov momentum as optimizer (In IMAGENET-1K 1k, we use batchsize 128 and learning rate 0.2).
For the object detection task, we perform nonlinear fine-tuning on ResNer50 in RCNN-C4 Girshick al. 2014) with batchsize 9 and base learning rate 0.01. We use the detectron2 (Wu et 2019). following the MOCO-v1 (He et al. 2020) official implementation exactly.
A.7 Time Cost of OR
Implementing OR requires computing the OR loss in the backbone each gradient update, count the time overhead required by the different backbones to compute OR at one time, and we have averaged over 10 times shown in Table A.7 In the pre-training phase, the time overhead of OR is only related to the backbone and the steps that need to be updated, IMAGENET-IK (100 epochs, batchsize 128) has total of 62599 steps, and CIFAR- 100 (1000 epochs. batchsize 256) has total of 194999 steps. As you can see, compared to the original pre-training overhead of dozens and hundreds of hours, the additional time added by OR very small, steadily improving SSL's performance, Notably, if we use larger batchsize such as 4096, our time overhead will be reduced by 64 on IMAGENET-1K and 16 on CIFAR-100.
Table 10: Time cost of OR
Encoder
ResNe118
ResNet50
WiddRes
VIT-tiny
VIT-small
VIT-base
Overhead of SO
0.016s
0.022s
0.008s
0.019s
0.024x
0.045s
Single step
Overhead of SRIP
0.012s
0.030s
0.012s
0.029s
0.034s
0.054s
Orig)
12h 26m
18h 10m
36m
11m
13h 18m
1d 8h
CIFAR-100
Overhcad of SO
0.87h
1.19h
0.43h
1,03h
1.30h
2,4hh
Overhead of SRIP
0.64h
1.62h
0.65h
1.57h
1.84h
2.92h
Original(BYOL)
3d 17h
IMAGENET-IK
Overhcad of SO
0.38h
Overhcad SRIP
0.52h
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we explained that dimensional collapse would exist weight matrices and features, which would damage the performance of SSL. We used OR to eliminate the dimensional collapse in SSL. In addition, consistent improvement results were achieved under each benchmark.In In the introduction, list our 3 contributions in detail.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our approach requires the introduction of additional computational overhead, which discussed in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
22

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We just expressed the existing theory in the vector form to the matrice form.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer:[Yes
Justification: All of our experimental codes including how divide the generated data, training and testing are all the same with open-sourced Solo-learn. And we' ve also attached them in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We include instructions in the appendix. including how to download and divide the data set, how to execute training and test scripts, and how to visualize the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppp.cc/pubbbcguuuceeee Codeeubmii ssi criionPPoiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have explained data division and hyperparameters in detail in both Ap- pendix and code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We only conducted multiple averaging operations in the statistical OR time overhead, and the experiments in the main paper did not carry out multiple averaging operations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We compared the time of OR in Appendix, and explained that our experiments were all completed on 3090 GPUs.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripss http:///wwrr...cpppplll/l//etcaa itw.eddeeeescnniieieeienneeee
Answer: [Yes]
Justification: The data sets are all public data sets, and we strictly follow existing studies, which will not have bad impact on the society.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our goal is to improve the performance the vision foundation model, which will have positive impact various downstream tasks.
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our approach addresses the potential dimensional collapse SSL without raising their risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We referenced the framework we used and the data set we used, In the code, we listed the URL of each data and the code environment we needed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "wulKkThwg5", "reviewer": "Reviewer_KCGP", "summary": "Dimensional collapse, characterized by the dominance of a few large eigenvalues, is a phenomenon in self-supervised learning (SSL) that can lead to redundant features and weight matrices. To address this, the authors propose orthogonal regularization (OR) during pretraining to promote orthogonality in weight matrices, rather than using common approaches that regularize features via the objective (e.g., Barlow Twins, VicReg). The authors conduct experiments with five SSL methods from the SoloLearn framework to demonstrate the benefit of OR.", "questions": "Memory Overhead of OR: How much memory overhead does orthogonal regularization (OR) introduce? \n\nExplanation for Lower Gamma in Larger Models: The paper uses a lower gamma for larger models, but why? One might think that larger models would require more regularization. \n\nResults for Barlow Twins and VicReg: Why were results for Barlow Twins or VicReg not reported? Is the OR approach redundant when combined with these methods, which also aim to achieve orthogonality?\n\nBalanced Eigenvalues and Performance Discrepancy: The eigenvalues of VicReg without a predictor (Appendix Fig 4) appear more balanced than those of BYOL OR (Fig 3), yet VicReg's performance is far worse.", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and provides a good introduction to the topic of representation collapse. The motivation for the approach is clearly stated, and the experimental setting is well documented. With small architectures like ResNet18, the effect of orthogonal regularization (OR) appears impressive at first glance. Although OR introduces some computational overhead, it is relatively small and may diminish further with larger batch sizes.", "weaknesses": "The soundness of some experiments seems doubtful. Results for WideResNet28x2 are consistently below those for ResNet18. Additionally, the results using Dino are generally poor, leading to all ViT experiments being below any ResNet baseline.\n\nThe performance of BYOL with OR, when trained for only 100 epochs with a batch size of 64, is even below the reported accuracy in SoloLearn. Considering the optimal training duration for BYOL is approximately 1000 epochs with a batch size of 4096, this comparison appears to be in a severely undertrained regime. Slightly longer training, 200 epochs with a batch size of 256, shows that the benefit of the OR approach diminishes rapidly.\n\nMinor issue: Line 44 states, \"We evaluate the effect of adding OR in seven modern SSL methods during pretraining,\" but results are shown for only five methods.\n\nAdditionally, since SSL approaches like Barlow Twins and VicReg aim to achieve orthogonality as well, there should be an experiment evaluating the effect of OR on these methods.\n\nUsing one learning rate (LR) for all methods in linear probing might not be optimal. It is cheap to attach multiple linear probing heads and find the most suitable LR for each method to ensure fair comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n> The soundness of some experiments\n\nThanks for pointing out this. Solo-learn only provides the official config file (yaml) for resnet18 on cifar100, on which we replaced the backbone (VITs, ResNet50,WideResnet) and added the OR. To ensure a fair comparison, we did not modify any other hyperparameters. Replacing a backbone may require tuning these parameters to achieve the best results. For now, we just want to show that OR can be plug-and-play under all kinds of backbone.\nMore importantly, we believe our OR method could boost the performance under different hyperparameter settings. \nPlease check the performance table in the general response.\n\n>Additionally, the results using Dino\n\nIn fact, VIT-small **outperforms** all the ResNet backbone results in DINO under the cifar100 experiment, please refer to Tables 1 and 2 in the paper.  Results of DINO in the ResNet series are poor, and we believe this is due to the parameters in the config files of its repository. For example, in DINO, its default config uses the lars optimizer, warmup\\_teacher\\_temperature\\_epochs: 0, warmup\\_start\\_lr: 3e-05. However, the optimizer used for checkpoint results is SGD, lars=Ture, warmup\\_teacher\\_temperature\\_epochs: 50, warmup\\_start\\_lr: 3e-03. These differences in hyperparameters cause our results to be different from their reported results. Our baseline and OR methods run entirely under their officially provided hyperparameter config.  To ensure fair comparisons, we did not modify the parameters in the original configs.  \n\n>The performance of BYOL with OR\n\nWe checked the parameters in its checkpoint and BYOL trained on ImageNet with a batch size of 128 instead of 64 in its yaml config file. We think this is the reason for the different results.\nOur current results on ImageNet1k show that SO can perform better with small batchsize and small epochs. Although the benefit of OR on ImageNet1k is smaller after increasing the epoch on ImageNet1k, it still improves transfer learning substantially.\n\nAs for the results for larger epochs and larger batch sizes, the experiments of DINO and DINO (SO) on ImageNet with larger batchsize (2048) and epoch (300) are provided.In the follow-up work, we will conduct more experiments to fully test the effect of OR. \n**Please check the performance table in the general response.**\n\n>Minor issue:\n\nSorry for our mistake. The SSL methods we have now include **6** methods implemented by solo-learn (MOCOv2plus, MOCOv3, DINO, NNBYOL, BYOL, VICREG) and **10** methods implemented by the LIGHTLY framework (BarlowTwins, BYOL, DCL, DCLW, DINO,  Moco, NNCLR, SimCLR, SimSiam, SwaV). Removing duplicates, there are in total **13** different SSL methods. \n\n>Additionally, since SSL \n\nYour suggestions are very constructive. We tried to add OR to VICREG and Barlow Twins (resnet18, cifar10). **Both gained a boost from OR**, suggesting that simply requiring the output of the projector to be whitened is not enough to avoid weight matrix collapse. We additionally visualized the weights of the former, please see the pdf material of the general response.\n\n>Using one learning rate (LR) \n\nThanks for the suggestion, which is very reasonable. The representations produced by different methods may require different learning rates for the downstream tasks. We are only using the solo-learn settings for now. Considering the time constraints during the rebuttal period, we will try to adjust the learning rate later to get the best results.   \nFor fair comparisons, we added the results of 10 SSL methods using KNN classification under the LIGHTLY framework to visualize the quality of the representations better. \n\n\n>Memory Overhead of OR: How much memory overhead does orthogonal regularization (OR) introduce?\n\nFor a specific backbone, OR traverses the parameters (linear and convolutional layers) in it.  \nLet's assume that the largest W is a square matrix with shape n\\*n, then the memory overhead needed is to compute W.T @ W which is also an n\\*n matrix, and to introduce an identical matrix I, which is also an n\\*n matrix, and storing the distance between the two requires an n\\*n matrix. So in total, the memory overhead of the forward propagation is   \n4 \\*n\\*n\\*(Floating Point Precision)/4.\nUsing BarlowTwins in LIGHTLY with resnet18 as an example, the maximum memory usage without OR is 4.87GB, with SO it is 4.89GB. Therefore, the memory overhead of OR is quite low.\n\n>Explanation for Lower Gamma \n\nYou are asking a very good question to promote understanding of OR loss.   \nBecause our OR is a penalized **summation of all parameters**, the larger the number of backbone parameters, the OR loss will increase rapidly, in order to be balanced with the SSL loss, we have only appropriately scaled down the Gmma by a multiplicity of 0.1.   We checked the SO loss of the randomly initialized ResNet18 and ResNet50 backbones (gamma= 1), which is 5873.4062 for the former and 217609.7031 for the latter, and the increase in the number of model parameters brings about a nearly **40-fold** increase in OR loss.\n\n>Balanced Eigenvalues and Performance Discrepancy\n\nYour notice is well worth further discussion. The whiteness (balanced eigenvalues) of representations and their performance do not grow in tandem, see Table 4 in \\[1\\].  \nWe visualized the original VICREG, and we can see that the original VICREG is much better than the version with the projector removed, but the whiteness degree of its representations is not as high as the one with the projector removed. Please see the pdf material of the general response.   \n\n\\[1\\] He, Bobby, and Mete Ozay. \"Exploring the gap between collapsed & whitened features in self-supervised learning.\" *International Conference on Machine Learning*. PMLR, 2022\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBxioQgXv1", "reviewer": "Reviewer_gNSa", "summary": "This paper aims to improve self-supervised learning (SSL) by introducing orthogonality regularization (OR) during pretraining to prevent dimensional collapse. OR improses orthogonality constraint on weight matrices, not only preventing dimensional collapse of representation but also regularizing the weights from collapsing. The experimental results show that the benefit of the proposed method is consistent and widely applicable across datasets and architecture choices.", "questions": "* In practice, a practitioner has to choose between SO and SRIP. What would be the authors' recommendation? \n* Do you expect that applying dropout on the encoder's hidden neurons would result in a similar effect, as dropout is also meant to prevent feature co-adaptation? It is quite likely that the existing encoders have already employed dropout, so I wonder if dropout is insufficient to prevent the collapse.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* This paper effectively communicates both the problem it addresses and the solution it presents. The motivation and solutions are compelling and presented in an easily understandable manner.\n* The benefit of the proposed method appears to be consistent and significant.", "weaknesses": "* More discussion on the difference between the proposed orthogonal regularization methods would have been helpful. On the surface, the two methods are almost equally good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n\n>More discussion on the difference between the proposed orthogonal regularization methods would have been helpful. On the surface, the two methods are almost equally good.\n\nThanks for the advice, it helps a lot for those who intend to use OR\\!  \nYes, both are almost the same in terms of effectiveness. SRIP is more time-consuming and takes up more memory than SO.  \nSo we chose SO for very time-consuming experiments (ImageNet and ImageNet100). Additionally, SRIP has theoretically **weaker (softer)** regularization constraints than SO because it uses the spectral norm instead of the L2 norm.\n\nBased on our experience with large-scale datasets and OR's results for the feature whitening method VICREG, we recommend SO. We add 10 SSL methods in the LIGHTLY framework, all of which use SO, and all of which obtain improved results. Please check the performance table in the general response.\n\n>Do you expect that applying dropout on the encoder's hidden neurons would result in a similar effect, as dropout is also meant to prevent feature co-adaptation? It is quite likely that the existing encoders have already employed dropout, so I wonder if dropout is insufficient to prevent the collapse.\n\nThanks for raising this question. Indeed, dropout is also able to prevent feature co-adaptation. In SSL, there is no dropout in ResNet, but dropout is enabled in VITs. Our OR approach significantly improves the DINO with VITs (Table 2 in the paper), suggesting that single dropout cannot adequately prevent dimensional collapse at the three levels of the weight matrix, intermediate features, and representations.  \nWe can visualize Dropout's impact on all three in future work. Exploring the role of Dropout and OR in SSL is very interesting for further research.\n\nIn fact, because of the widespread use of Dropout in the Transformer architecture, there has been some exploration of the impact of dropout on the features of SSL using the Transformer architecture. Dropout encourages the use of global features rather than local features in the Transformer architectures \\[1\\].\n\n\\[1\\] Luo, Jian, et al. \"Dropout regularization for self-supervised learning of transformer encoder speech representation.\" *arXiv preprint arXiv:2107.04227* (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m2Hwx4BiDW", "reviewer": "Reviewer_sksr", "summary": "This work proposes a method to reduce the dimensional collapse issue in self-supervised pretrained networks. Dimensional collapse occurs when a few large eigenvalues dominate the eigenspace. While previous research focused solely on the output representations of the pretrained networks, this work shows that dimensional collapse also occurs on the intermediate representations and weight matrices.\n\nTo address this issue, the authors use orthogonal regularization (OR) objectives for the weight matrices in linear layers (dense or convolutional) within the network. The OR objective promotes orthogonality in the weight matrices by minimizing the distance between the Gram matrix of each weight matrix and the identity matrix. They demonstrate that using the OR regularization objective leads to a smoother decay of eigenvalues in the (intermediate) representations and weight matrices of a self-supervised pretrained ResNet18.\n\nThe authors evaluate the OR regularization objective by pretraining various backbones (RN18, RN50, WideResNet28w2, and ViT-tiny/small/base) with multiple joint-embedding SSL methods (MoCov2, MoCov3, DINO, BYOL, NNBYOL) on CIFAR-100. They then train linear classification models on the frozen features, observing improved accuracy in nearly all cases with OR. Finally, they pretrain a RN50 backbone with BYOL on ImageNet-1k and evaluate it through linear probing on ImageNet and fine-tuning on Pascal object detection, showing improved results with OR.", "questions": "I would like the authors to address my concerns (W1), (W2), (W3) and (W4), as described in the weaknesses section. Some more specific questions:\n- About W1: can the authors provide a comparison of vanilla BYOL, BYOL with OR, and BYOL with whitening in terms of their effects on the eigenvalues of intermediate representations and weight matrices.\n- About W3: What is the reason for the differences in the results reported in this work and in in solo-learn (https://github.com/vturrisi/solo-learn)?\n- About W4: Can the authors demonstrate that OR improves even well-tuned baselines like BYOL with 1000 epochs (74.3 in BYOL paper) or MoCo-v2 with 800 epochs (71.1% in https://arxiv.org/pdf/2003.04297), or MoCo-v3 (74.6 for RN50 see: https://github.com/facebookresearch/moco-v3). Any of them would be good.", "limitations": "Yes, they are discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The used method of reducing dimensional collapse of SSL methods is simple and can be easily integrated on various SSL methods and backbones.\n- The strongest aspect of this work is that it shows that OR consistently improves various joint-embedding SSL methods and backbones. However, the improvements (in linear accuracy) are small in the case of well-tuned baselines.", "weaknesses": "- **(W1)** The introduction's claimed contribution, \"We **systematically study** the phenomenon of dimensional collapse in SSL, including how whitening representations and network depth affect the dimensional collapse of weight matrices and hidden features,\" seems to me to be an exaggeration. In the main paper, there is only one experiment pretraining ResNet18 with BYOL w/ and w/o OR on CIFAR-10, and another experiment in the supplementary material using VICReg, which are not directly comparable. Given the emphasis on this contribution, a more appropriate approach would be to include an analysis of whitening's impact on dimensional collapse in the main paper (e.g., adding another column with whitening-based results in Figure 3). This analysis should be conducted under the same conditions as the experiments in Figure 3, for example, using BYOL with a projector and whitening. This setup would enable a straightforward comparison of vanilla BYOL, BYOL with OR, and BYOL with whitening in terms of their effects on the eigenvalues of intermediate representations and weight matrices.\n- **(W2)**: **Limited technical contribution**: as discussed in the related work sections of this work, orthogonality regularization of the weight matrices within neural network has been extensively studied before (related work subsection 2.3). In particular, the OR objectives studied here are directly taken from Bansal et al. 2018, and straight-forwardly applied to the examined self-supervised approaches, without any modification at all. So, the main novelty of this work is to apply in a plug-and-play way the OR regularization objectives of Bansal et al. 2018 on self-supervised methods.\n- **(W3)** About the CIFAR100 results: in the results provided in solo-learn (https://github.com/vturrisi/solo-learn), which the authors also use for their experiments, some baselines (i.e., SSL methods that do not use OR) have higher results than what is reported in this work and often higher even to the results reported in this work with the OR regularization objective.  In particular, this is the case for MoCov2+ (69.89% in solo-learn vs 69.51% here), MoCov3 (68.83% in solo-learn vs 67.13% here), DINO (66.76% in solo-learn vs 59.13% in here). For BYOL however, is the other way around (70.46% in solo-learn vs 71.15% in here). What is the reason for the differences in the results?\n- **(W4)** The improvement that OR offers over a well-tuned BYOL baseline on ImageNet (200 pretraining epochs and 256 batch-size) is marginal: from 69.76 to 70.16. It seems that it offers significant performance improvements in mostly un-tuned baselines, like the results when pretraining BYOL for 100 and for 64 batch-size. In reality, even 200 epochs are not adequate for pretraining BYOL that, its performance reaches 74.3 for 1000 epochs (see BYOL paper). So, it is unclear whether OR could offer any improvement for such long pretraining settings that are typically used in SSL. \n- **(W5)** Not my primary concern, but although OR is shown to provide a small improvement on the linear accuracy of some SSL methods (e.g., going from 69.76 to 70.16 for BYOL pretrained with 200 epochs on ImageNet-1k), no evidence is shown that OR actually avoids dimension collapse in settings where otherwise (i.e., without OR) the SSL method will completely collapse and fail to learn any meaningful representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n\n>(W1)\n\nThank you for your very constructive suggestions\\! Following your suggestions, we added more experiments of BYOL with vic loss (feature whitening their predictor's output). Simply adding feature whitening to the original SSL method will even degrade its performance, in contrast, our plug-and-play OR method can improve the performance.\nWe also visualize weights and features of BYOL w/ vic, VICREG, and VICREG w/ OR (please see the pdf material), which are not able to avoid dimensional collapse of their weight matrices by feature whitening alone.\nPlease check the performance table and see the pdf material of the general response. \n\n>(W2)\n\nThanks for the comment, in which we respectfully disagree. Although OR has been explored under supervised and semi-supervised learning, this paper is the first time to explore its effect on dimensional collapse in SSL and attempt to turn it into a plug-and-play method for enhancing SSL. It is also the first time that OR has been applied to a transformer architecture.   \nWe only employed the widely used form of OR, but we wanted to share it with the community given its plug-and-play and interpretability in SSL methods.\n\nTo further highlight our contribution in a more systematic study of dimensional collapse in SSL, we have added relevant experiments and visualizations based on your suggestions:  \nBYOL, BYOL with OR, and BYOL with whitening in terms of their effects on the eigenvalues of intermediate representations and weight matrices.\n\nThe contribution of this paper is rephrased and summarized as follows: From our observations, although existing methods such as VICREG deal with the collapse of features via whitening features, they can not prevent the dimensional collapse of the weight matrices. We are not only the first work to systematically study dimensional collapse at all levels in SSL but also provide new solutions to address the dimensional collapse by constraining the weight matrix.\n\nOR is just one of the methods we use to constrain the weight matrix in SSL, as it is intuitive to avoid the collapse of the weight matrix and has deterministic effects on the features.  \nWe only employed the widely used form of OR, but we wanted to share it with the community given its plug-and-play and interpretability in SSL methods.\n\n>(W3)\n\nThanks for pointing this out. We followed the latest Solo-learn config file exactly and ran the results directly. We carefully compared the config in its GitHub repository with the config provided by checkpoints and found that there are differences in its hyperparameters, which is most likely the reason why our reproduced results are not the same as its report results. \n\nFor example, in DINO, its default config uses the lars optimizer, warmup\\_teacher\\_temperature\\_epochs: 0, warmup\\_start\\_lr: 3e-05. However, the optimizer used for reporting results is SGD, lars=Ture, warmup\\_teacher\\_temperature\\_epochs: 50, warmup\\_start\\_lr: 3e-03.  These differences in hyperparameters result in our results being different from the results of their checkpoint reporting.\n\nOur baseline and OR methods run entirely under their officially provided hyperparameter configs. **It is difficult to make direct use of the parameters in its checkpoints. solo-learn has been updated and its hyperparameter form has been changed.**\n\nFortunately, our OR methods are plug-and-play enough to compare the impact of SO with 10 SSL methods in another open-source SSL framework Lightly, which includes BarlowTwins , BYOL ,DCL, DCLW , DINO, Moco, NNCLR, SimCLR, SimSiam, SwaV. \nIn our environment, its original baseline results are higher, and OR can further enhance these SSL methods.  \nPlease check the performance table and see the pdf material of the general response. \n\n>(W4) \n\nThank you for your suggestions\\! Our current results on Imagenet show that SO is able to improve more with small batchsize and small epochs.  Although the benefit of OR on ImageNet1k is smaller after increasing the epoch on ImageNet1k, it still improves transfer learning substantially.\nAs for the results for larger epochs and larger batchsize, due to our limited time and computational resources,  the results of DINO and DINO (SO) on ImageNet with larger batchsize (2048) and epoch (300) are provided. In the follow-up work, we will definitely make up more epoch experiments to fully test the effect of OR.  \nIn addition, we added the results of various SSL methods on LIGHTLY framework cifar10, resnet18, 200 epochs, and 400 epochs, the OR can get the same improvement.   \nAlso, we report  BarlowTwins and BYOL results under Imagenet100 (ResNet 18, epoch 200, batch size 256).  \nPlease check the performance table in the general response.\n\n>(W5)\n\nBy visualizing weight matrices and features, we can see that OR avoids the dimensional collapse of weights as well as features (Figure 3 in the paper), and it even likewise improves on feature whitening methods such as VICREG and BarlowTwins.  Please check the performance table in the general response.\n\nWithout OR, the current SSL methods will not completely collapse as they introduced techniques such as using negative samples, self-distillation, clustering, and feature whitening. In this paper, we visualize the decaying eigenspace of weight matrices and features to observe the dimensional collapse, and we demonstrate the OR can effectively mitigate, if not prevent, the dimensional collapse.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CA6R2Oys6T", "reviewer": "Reviewer_2nFS", "summary": "In order to combat the problem of dimensionality collapse in self-supervised learning, the authors of this work introduce orthogonal regularization on encoders during pretraining. Taking inspiration from previous work on supervised orthogonality regularization, the authors study the impact of OR on eigenvalues of weights when considering self-supervised learning and find that OR learned weights/features contain more meaningful features and avoid dimensional collapse better. The recommended result seems to be effective for both contrastive and non-contrastive SSL methods.", "questions": "1) For section 3.2 under what case would we have input > output ? \n2) Could the authors provide some explanation for what entails whitening of the features ?", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall I think the idea is simple and the paper is well written. Particularly the observations regarding the eigenvalues of SSL features and weights and the impact of OR are informative. In addition, I believe the OR improvements on transfer learning and downstream tasks are considerable and could become a staple part of SSL.", "weaknesses": "1) I would like to point out the improvements of OR on SSL methods is not as major on the simple datasets such as Cifar100 set according to Table 1. However, the changes in larger datasets with BYOL are more noticeable but I would like to ask why results for other methods were not included for larger datasets. \n2)  Particularly with regards to Fig 3, to me it doesn’t look as if the feature eigenvalue changes much with the OR except for the representations. Perhaps a combination of other OR methods could help ? Or adding orthogonality regularization with the features. \n3) With regards to classification accuracy on Table 5, higher batch size and longer training somewhat diminish the impact of OR improvement. Perhaps this result could be used to encourage OR when training with smaller batches or less epochs is required ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n\n>I would like to point out the improvements of OR on SSL methods is not as major on the simple datasets such as Cifar100 set according to Table 1\\. However, the changes in larger datasets with BYOL are more noticeable but I would like to ask why results for other methods were not included for larger datasets.\n\nThis is because training on large-scale data takes too much time (3 days for 100 epochs), and then BYOL presents the most stable and best performance on cifar100, so we only tested BYOL for now.  \nDue to our limited time and computational resources, the results of DINO and DINO (SO) on ImageNet with larger batchsize (2048) and epoch (300) are provided.  \nAlso, we report  BarlowTwins and BYOL results under Imagenet100 (ResNet 18, epoch 200, batch size 256).  \nPlease check the performance table in the general response.\n\n>Particularly with regards to Fig 3, to me it doesn’t look as if the feature eigenvalue changes much with the OR except for the representations. Perhaps a combination of other OR methods could help ? Or adding orthogonality regularization with the features.  \n\nInterestingly, the effects of OR on intermediate features do not appear to be as large as that of representations.  \nWe visualize intermediate features of VICREG (based on feature whitening), and VICREG\\_OR. Although OR can improve the performance of VICREG, we can find that the change in their intermediate features and even the representation is not obvious, but the weight matrix has a significant change.   \nPerhaps we need to train a deeper network to observe the effect of OR on the deeper features, we leave this part for future work.  \nPlease check the performance table and see the pdf material of the general response. \n\n>With regards to classification accuracy on Table 5, higher batch size and longer training somewhat diminish the impact of OR improvement. Perhaps this result could be used to encourage OR when training with smaller batches or less epochs is required ?  \n\nThank you for your suggestions\\! Our current results on Imagenet show that SO is able to improve more with small batchsize and small epochs. \nTo test the effects of OR with large batchsize and long epochs, the results of DINO and DINO (SO) on ImageNet with larger batchsize (2048) and epoch (300) are provided.  \nIn the follow-up work, we will definitely make up more epoch experiments to fully test the effect of OR.  \nIn addition, we added the results of various SSL methods on LIGHTLY framework cifar10, (resnet18), 200 epochs, and 400 epochs, the OR can get the same improvement.   \nPlease check the performance table in the general response.\n\n>For section 3.2 under what case would we have input \\> output ?\n\nFor convolutional layers, input \\= convolution kernel area \\* input feature dimension, output \\= output feature dimension, e.g. for the first convolution in ResNet18, input \\= 7\\*7\\*3, output \\= 64, and input\\>output.  \nFor linear layers, input is greater than output as long as the input feature dimension is greater than the output feature dimension.\n\n>Could the authors provide some explanation for what entails whitening of the features?  \n\nOR causes features to whiten because the orthogonalized weight matrices, which make the feature spectra more consistent across depth layers of features, prevent deeper features (e.g., representations) from collapsing and showing a whitened nature \\[1,2\\].  \nWe can observe that the distribution of features in each layer is much closer after adding the OR constraints (Table 3 (d) brown and yellow lines in the paper). \n\n\\[1\\] Huang, Lei, et al. \"Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 32\\. No. 1\\. 2018\\.  \n\\[2\\] Wang, Jiayun, et al. \"Orthogonal convolutional neural networks.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2020\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ADkoSRaat2", "reviewer": "Reviewer_Yvo9", "summary": "Driven by the dimensional collapse phenomenon observed in self-supervised learning, this research adopts a methodology designed to prevent the reduction in dimensionality of weight matrices. The empirical findings indicate a reduction in dimensional collapse in certain instances.", "questions": "1. What are the advantages of regularizing weights over regularization of representation?\n\n2. What is the performance of SRIP in Table 2 and 5?", "limitations": "Please refer to the weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The proposed method seamlessly integrates with existing SSL methods.", "weaknesses": "The proposed method lacks novelty, essentially applying the loss formulation from Barlow Twins to the weight matrix.\n\nMoreover, the experimental results are insufficient to substantiate the efficacy of the method due to the following reasons:\n\n1. Many of the SSL methodologies utilized to assess the OR are no longer considered state-of-the-art. Therefore, it is recommended to evaluate the OR using state-of-the-arts methods.\n\n2. Table 1 reveals that the effectiveness of SO or SRIP appears to exhibit considerable variability, rendering the proposed methods unstable.\n\n3. Tables 3 and 5 exclusively demonstrate the evaluation of OR for the BYOL method, omitting the efficacy assessments of other SSL methods. Additionally, in Table 4, the datasets are characterized as small-scale, which contradicts the claims made by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback. In addition to the general response, we address your itemized concerns here.\n\n>The proposed method lacks novelty, essentially applying the loss formulation from Barlow Twins to the weight matrix.\n\nMethods like Barlow Twins try to avoid dimensional collapse of intermediate features as well as weights by whitening the projector output.  \nOur OR, on the other hand, acts directly on the weight matrix to indirectly avoid the dimensional collapse of features.  \nThe implementation and motivation of these two methods are different, and no one has previously explored the dimensional collapse of the weight matrix in SSL, nor has anyone attempted to constrain the weight matrix in SSL to obtain an improvement in the results (except for weight decay).\n\nIn a broader sense, OR is just one of the methods we use to constrain the weight matrix in SSL, as it is intuitive to avoid the collapse of the weight matrix and has deterministic effects on the features. We only employed the widely used form of OR, but we wanted to share it with the community given its plug-and-play and interpretability in SSL methods.\n\nWe complement the experiments based on feature whitening methods such as Barlow Twins and VICRGE, OR can further improve their performance. We also visualize and analyze BYOL w/ vic, VICREG, and VICREG w/ OR, which are not able to avoid dimensional collapse of their weight matrices by feature whitening alone. Simply adding feature whitening to the original SSL method will even degrade its performance, unlike our plug-and-play OR.   \nPlease check the performance table and see the pdf material of the general response. \n\n>Many of the SSL methodologies utilized to assess the OR are no longer considered state-of-the-art. Therefore, it is recommended to evaluate the OR using state-of-the-arts methods.\n\nThank you for your advice. The SSL methods we have now include **6** methods implemented by solo-learn (MOCOv2plus, MOCOv3, DINO, NNBYOL, BYOL, VICREG) and **10** methods implemented by the LIGHTLY framework (BarlowTwins, BYOL, DCL, DCLW, DINO,  Moco, NNCLR, SimCLR, SimSiam, SwaV). Removing duplicates, there are in total **13** different SSL methods.  \nPlease check the performance table in the general response.\n\nWe are currently experimenting as many SSL methods as possible with that have been built as benchmarks for fair comparison (LIGHTLY, solo-learn). If there are  SOTA SSL methods that we still need to include, please let us know and we will try to add OR to it.\n\n>Table 1 reveals that the effectiveness of SO or SRIP appears to exhibit considerable variability, rendering the proposed methods unstable.  \n\n\nOur ORs, both SO and SRIP, work on all kinds of SSLs and all kinds of backbones. Only when MOCOv2 plus replaces the backbone with ResNet50, OR makes the Top-1 go down, but its Top-5 all goes up.   \nPlease note that we did not adjust any hyperparameters when changing the backbone or adding OR.\n\n>Tables 3 and 5 exclusively demonstrate the evaluation of OR for the BYOL method, omitting the efficacy assessments of other SSL methods. \n\nThis is because training on large-scale data takes too much time (3 days for 100 epochs), and then BYOL presents the most stable and best performance on cifar100, so we only tested BYOL for now.  \nDue to our limited time and computational resources, the results of DINO and DINO (SO) on ImageNet with larger batchsize (2048) and epoch (300) are provided.  \nAlso, we report  BarlowTwins and BYOL results under Imagenet100 (ResNet 18, epoch 200, batch size 256).  \nPlease check the performance table in the general response.  \n\t\t\n\n>Additionally, in Table 4, the datasets are characterized as small-scale, which  contradicts the claims made by the authors.\n\nTable 4 refers to the results of SSL models pre-trained on large-scale datasets to perform linear probing on small datasets, which is consistent with numerous SSL studies. We will add more descriptions to the table later to avoid ambiguity.  \n\t\n\n\n>What are the advantages of regularizing weights over regularization of representation?\n\nWe supplement the experiments based on feature whitening methods such as Barlow Twins and VICRGE, **OR can further improve their performance**. We also visualize and analyze BYOL w/ vic, VICREG, and VICREG w/ OR, which **are not able to avoid dimensional collapse of their weight matrices by feature whitening alone**. Simply adding feature whitening to the original SSL method will **even degrade** its performance, unlike our plug-and-play OR.   \nPlease check the performance table and see the pdf material of the general response. \n\n>What is the performance of SRIP in Table 2 and 5?  \n\nThe VITs experiment in Table 2 and the BYOLl 200 epoch experiment in Table 5 were too time-consuming, so we only tested the effect of SO. The results of SO are very similar to those of SRIP, with shorter time and less memory usage. In general, we advocate the use of SO for large models. However, we will still add the results of SRIP with the same settings in the later version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their questions and constructive feedback.   Here, we respond to the five core issues of common interest: 

**Solo-learning baseline settings**
The baseline results of our report are the results of the newest official config file, which are different from the checkpoint results.  **This is because the configs in the checkpoint itself are different from the newest official config file**. For example, in DINO, its default config uses the lars optimizer, warmup\_teacher\_temperature\_epochs: 0, warmup\_start\_lr: 3e-05. However, the optimizer used for checkpoint results is SGD, lars=Ture, warmup\_teacher\_temperature\_epochs: 50, warmup\_start\_lr: 3e-03.   

**More SSL methods under different hyperparameter settings**  
The SSL methods we have now include **6** methods implemented by solo-learn (MOCOv2plus, MOCOv3, DINO, NNBYOL, BYOL, VICREG) and **10** methods implemented by the LIGHTLY framework (BarlowTwins, BYOL, DCL, DCLW, DINO,  Moco, NNCLR, SimCLR, SimSiam, SwaV). Removing duplicates, there are in total **13** different SSL methods.
Under the LIGHTLY framework, we tested the effect of SO under CIFAR10, ImageNet100, and ImageNet1K. 

|  KNN Top1 (CIFAR10) | Batch 512 Epoch 200 |  |  | Batch 512 Epoch 400 |  |
| :---- | :---- | :---- | :---- | :---- | :---- |
|  |  Report |  Reproduced |  w/ SO |  Reproduced |  w/ SO |
| Barlow Twins | 81.9 | 83.58 | **84.78** | 85.91 | **86.25** |
| BYOL | 86.8 | 86.94 | **87.01** | 89.64 | **90.02** |
| DCL | 84.0 | 83.38 | **84.10** | 85.95 | **86.27** |
| DCLW | 82.4 | 82.42 | **82.73** | 85.25 | **85.71** |
| DINO | 81.3 | 81.87 | **81.95** | \\ | \\ |
| Moco | 84.7 | 85.19 | **85.32** | \\ | \\ |
| NNCLR | 81.5 | 82.31 | **82.34** | \\ | \\ |
| SimCLR | 84.8 | 84.55 | **84.84** | \\ | \\ |
| SimSiam | 76.4 | 79.27 | **84.31** | \\ | \\ |
| SwaV | 84.2 | 83.10 | **83.67** | \\ | \\ |

we can see that even under different implementations with different hyperparameter settings (in our environment, our reproduced results are even higher), OR can still have a stable improvement in SSL performance.

**Relationships with feature whitening**  
Feature whitening methods like Barlow Twins and VICREG aim to avoid dimensional collapse of intermediate features as well as weights by whitening the output of the projector.  
Our OR, on the other hand, acts directly on the weight matrix to indirectly avoid the dimensional collapse of features.

Thanks for the suggestions of Reviewer KCGP, sksr, Yvo9. We supplement the experiments based on feature whitening methods such as Barlow Twins (LIGHTLY) and VICREG (Solo-learn), **OR can further improve their performance**. We also visualize weights and features of  BYOL w/ vic, VICREG, and VICREG w/ OR (**please see the pdf material**), which are not able to avoid dimensional collapse of their weight matrices by feature whitening alone.   
Simply adding feature whitening to the original SSL method will even degrade its performance, in contrast, our plug-and-play OR method can improve the performance. 
| vicreg w/o Projector | vicreg | vicreg \+ SO (1e-6) | vicreg \+ SRIP (1e-3) | BOYL | BYOL+ SO (1e-6) | BYOL+ vic (1e-3) | BYOL+ vic (1e-4) | BYOL+ vic (1e-5) |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| 88.64 | 91.64 | **92.35** | 92.22 | 92.92 | **93.04** | 92.35 | 92.38 | 92.66 |
| 99.57 | 99.73 | **99.79** | 99.67 | 99.83 | **99.84** | 99.85 | 99.82 | 99.79 |

**Contributions**  
From our observations, although existing methods such as VICREG deal with the collapse of features via whitening features, they can not prevent the dimensional collapse of the weight matrices. We are not only the first work to systematically study dimensional collapse at all levels in SSL but also provide new solutions to address the dimensional collapse by constraining the weight matrix. 

In a broader sense, OR is just one of the methods we use to constrain the weight matrix in SSL, as it is intuitive to avoid the collapse of the weight matrix and has deterministic effects on the features. We only employed the widely used form of OR, but we wanted to share it with the community given its plug-and-play and interpretability in SSL methods.

**Larger datasets with larger settings**  
We understand that examining the effect of OR under larger batchsize, more epochs, especially the large-scale dataset ImageNet is necessary.   
Due to our limited time and computational resources,  We have to choose to complete the following experiments.
1. ImageNet100, Barrow Twins and BYOL in the LIGHTLY framework (ResNet18 epoch 200 batchsize 256\)

|  | Barrow Twins w/SO | Barrow Twins  (Reproduced) | Barrow Twins   | BYOL w/SO | BYOL  (Reproduced)  | BYOL   |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| KNN Top1 | **57.0** | 56.6 | 46.5 | **52.1** | 51.7 | 43.9 |

2. ImageNet1k, DINO in the LIGHTLY framework (ResNet50 epoch 300 batchsize 2048\)

|  | DINO w/SO | DINO (Reproduced) |
| :---- | :---- | :---- |
| KNN Top1 | **60.40** | 59.77 |
| KNN Top5 | **85.80** | 85.38 |
| Linear Top1 | **72.18** | 71.88 |
| Linear Top5 | **90.47** | 90.2 |

3. ImageNet1k, BYOL in the Solo-learn framework:  Transfer Learning

|  | ImageNet Top1 | Flowers102 Top1  |
| :---- | :---- | :---- |
| Batchsize 64  Epoch 100 | 65.81 | 73.817 |
| w/ SO | **67.84** | **78.956** |
| Btachsize 256 Epoch 200 | 69.76 | 76.452 |
| w/ SO | **70.16**  | **80.371** |

Although the benefit of OR on ImageNet1k is smaller after increasing the epoch on ImageNet1k, it still improves transfer learning substantially.

Overall, we would like to thank the reviewers once again for your detailed and effective suggestions. Improving and supplementing the experiments based on your suggestions will undoubtedly make our article more convincing.

Meta-review: **Summary of the paper:**
The paper proposes weight regularization in order to avoid feature dimensionality collapse in self-supervised learning. It reguralizes the weights by adding a loss function, of two possible types, that encourage orthogonality of the weight matrix as well as a uniform spectral domain. It shows on several architectures, a few SSL algorithms, and a couple of datasets, that the addition of their loss often improves the downstream performance.

**Summary of the reviews:**
The reviewers found the paper well-written, and the proposed method simple but well-motivated. They mostly found the results convincing in that the proposed loss function usually shows benefits over its absence. 

On the other hand, the reviewers had doubts about the soundness of the experimental setup when in comes to the paper’s results on the established methods and architectures. They also pointed a lack of comparison with other methods to avoid collapse such as Barlow Twins and VIC regularization. Two reviewers found the technical contribution to be low. 

**Summary of the rebuttal and discussions:**
The authors point out the subpar results is due to the plugging of architectures into their used codebase without hyperparameter optimization or, in the case of DINO, due to the lack of the essential multicrop augmentation. They further offered new experiments by adding their loss function on top of VICreg and BT. 

**Consolidation report:**

During the discussion the reviewers were mostly (but not fully) convinced regarding the occasional subpar results of the baselines, especially given that the authors promised to improve and reflect on these discrepancies in the final version. As a result they all moved their ratings towards acceptance. 

Dimensionality collapse is an important issue with SSL which is an important field, this can compensate for the lack of general technical contribution pointed out by reviewers

**Recommendation:**
All in all, the AC agrees with the unanimous view of the reviewers and recommends acceptance.

In the final version, it is important to,
- as promised, improve and reflect upon the reported results as was partially done during the rebuttal.
- cite the following [a] paper that is published at NeurIPS 2023 workshops and is directly related to the proposed method.

[a] “WERank: Rank Degradation Prevention for Self-Supervised Learning via Weight Regularization” NeurIPS 2023 workshop on Self-Supervised Learning: Theory and Practice
Decision: Accept (poster)