Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization

Authors: junlin  he
jinxiao  du
wei  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:25:03

Abstract: Self-supervised learning (SSL) has rapidly advanced in recent years, approaching the performance of its supervised counterparts through the extraction of representations from unlabeled data. However, dimensional collapse, where a few large eigenvalues dominate the eigenspace, poses a significant obstacle for SSL. When dimensional collapse occurs on features (e.g. hidden features and representations), it prevents features from representing the full information of the data; when dimensional collapse occurs on weight matrices, their filters are self-related and redundant, limiting their expressive power.
Existing studies have predominantly concentrated on the dimensional collapse of representations, neglecting whether this can sufficiently prevent the dimensional collapse of the weight matrices and hidden features. 
To this end, we first time propose a mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matrices, thus safeguarding against the dimensional collapse of weight matrices, hidden features, and representations. Our empirical investigations demonstrate that OR significantly enhances the performance of SSL methods across diverse benchmarks, yielding consistent gains with both CNNs and Transformer-based architectures.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) has established itself as an indispensable paradigm in machine learning, motivated by the expensive costs of human annotation and the abundant quantities of unlabeled data. SSL endeavors to produce meaningful representations without the guidance of labels. Recent developments have witnessed joint-embedding SSL methods achieving, or even exceeding the supervised counterparts (Misra & Maaten 2020 , Bardes et al. 2022 , Caron et al. 2020 , Chen, Fan, Girshick & He 2020 , Chen, Kornblith, Norouzi & Hinton 2020 , Chen & He 2021 , Dwibedi et al. 2021 , HaoChen et al. 2021 , He et al. 2020 , He & Ozay 2022 , Jing et al. 2021 , Li, Zhou, Xiong & Hoi 2020 , Jing et al. 2020 , Balestriero et al. 2023 , Grill et al. 2020 , Zbontar et al. 2021 , Chen et al. 2021) . The efficacy of these methods hinges on two pivotal principles: 1) the ability to learn augmentation-invariant representations, and 2) the prevention of complete collapse, where all inputs are encoded to a constant vector.Efforts to forestall complete collapse have been diverse, including contrastive methods with both positive and negative pairs (He et al. 2020 , Chen, Kornblith, Norouzi & Hinton 2020 , Chen et al. 2021 ) and non-contrastive methods utilizing techniques such as self-distillation (Caron et al. 2021 , Grill et al. 2020 , Chen & He 2021) , clustering (Caron et al. 2018 , 2020 , Pang et al. 2022 ) and feature whitening (Bardes et al. 2022 , Zbontar et al. 2021 , Weng et al. 2022 , 2023) . Notwithstanding, these methods are prone to dimensional collapse, a phenomenon where a few large eigenvalues dominate the eigenspace. Dimensional collapse can occur on both features (e.g. hidden features and representations) and weight matrices.Figure 1 : Illustration of dimensional collapse in SSL. We use one augmented input X aug1 as an example: we assume that the encoder contains two basic blocks, each containing a linear operation (e.g., a linear layer or convolutional layer) and an activation function. Dimensional collapse can occur in weight matrices (W 1 , W 2 ), hidden features, and the finally obtained representations. Existing methods act directly on representations and expect to affect hidden features and weight matrices indirectly, which has no guarantee in theory; our method directly constrains weight matrices and indirectly influences hidden features and representations, which can be guaranteed by theoretical analysis.To prevent dimensional collapse of representations, as depicted in Figure 1 , existing methods include modifying representations in downstream tasks (He & Ozay 2022) , whitening representations directly (i.e. removing the projector) (Jing et al. 2021) , incorporating regularizers on representations during pretraining (Huang et al. 2024 , Hua et al. 2021) . However, whether they sufficiently prevent the dimensional collapse of weight matrices and hidden features remains unknown (i.e., no theoretical guarantee) (Pasand et al. 2024) . In Appendix A.1, we further demonstrate that whitening representations directly to eliminate the dimensional collapse of representations cannot adequately remove the dimensional collapse of weight matrices.To address these challenges, we first time propose a mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. It is natural that OR prevents the dimensional collapse of weight matrices as it ensures weight matrices orthogonality, keeps the correlation between its filters as low as possible, and lets each filter have a norm of 1. For features (e.g. hidden features and representations), orthogonal weight matrices can promote uniform eigenvalue distributions and thus prevent the domination of eigenspaces by a limited number of large eigenvalues, as theoretically substantiated by Huang et al. (2018) , Yoshida & Miyato (2017) , Rodríguez et al. (2016) .In our study, we introduce and assess the effects of two leading orthogonality regularizers, Soft Orthogonality (SO) and Spectral Restricted Isometry Property Regularization (SRIP), on SSL methods. We examine their integration with 13 modern SSL methods from Solo-learn and LightSSL, spanning both contrastive and non-contrastive methods (Chen, Fan, Girshick & He 2020 , Chen et al. 2021 , Grill et al. 2020 , Caron et al. 2021 , Dwibedi et al. 2021) . Our findings indicate a consistent enhancement in linear probe accuracy on CIFAR-100 using both CNNs and Transformer-based architectures and OR exhibits a good scaling law at the model scale. Furthermore, when applied to BYOL trained on IMAGENET-1k, OR significantly improves the downstream performance on both classification and object detection tasks, suggesting its applicability to large-scale SSL settings. Remarkably, OR achieves these enhancements without necessitating modifications to existing SSL architectures or hyperparameters.In summary, we present three major contributions:• We systematically study the phenomenon of dimensional collapse in SSL, including how feature whitening and network depth affect the dimensional collapse of weight matrices and hidden features.• We first time introduce orthogonal regularization (OR) as a solution to prevent the dimensional collapse of weight matrices, hidden features, and representations during SSL pretraining.• Our extensive experimental analysis demonstrates OR's substantial role in enhancing the performance of state-of-the-art joint-embedding SSL methods with a wide spectrum of backbones.2 Related Work 2.1 Self-Supervised LearningSelf-supervised Learning (SSL) aims to learn meaningful representations from unlabeled data.Existing SSL methods can be broadly classified into two categories: generative and joint embedding methods. This paper concentrates on joint-embedding methods, which learn representations by aligning the embeddings of different augmented views of the same instance. Joint-embedding methods further subdivide into contrastive and non-contrastive methods. Contrastive methods, such as those proposed by He et al. (2020) , Chen, Kornblith, Norouzi & Hinton (2020) , Chen et al. (2021) , treat each sample as a distinct class and leverage the InfoNCE loss (Oord et al. 2018) to bring representations of positive pairs closer together while distancing those of negative pairs in the feature space. These methods generally require a substantial number of negative samples for effective learning. In contrast, non-contrastive methods eschew the use of negative samples. They instead employ various techniques such as self-distillation (Caron et al. 2021 , Grill et al. 2020 , Chen & He 2021) , clustering (Caron et al. 2018 , 2020 , Pang et al. 2022 ) and feature whitening (Bardes et al. 2022 , Zbontar et al. 2021 , Weng et al. 2022 , 2023) . Our empirical findings indicate that incorporating OR enhances the performance of both contrastive and non-contrastive SSL methods. The exploration of its effects on generative methods remains for future work. 
dimensional collapse in ssl: 
 Dimensional collapse plagues both generative and joint embedding SSL methods (Zhang et al. 2022 , Jing et al. 2021 , Zhang et al. 2021 , Tian et al. 2021) . To prevent the dimensional collapse of representations, existing work has typically focused on imposing constraints on the covariance matrix of the representations, including modifying representations in downstream tasks (He & Ozay 2022) , removing the projector (Jing et al. 2021) , incorporating regularizers on representations during pretraining (Huang et al. 2024 , Hua et al. 2021 ). However, these strategies face challenges such as performance degradation upon removing the projector, not addressing collapse during pre-training, and failing to prevent dimensional collapse in hidden features and weight matrices within the encoder (referred to Appendix A.1). This motivates us to regularize the weight matrices of the DNNs directly in SSL.Orthonormality regularization, which is applied in linear transformations, can improve the generalization and training stability of DNNs (Xie et al. 2017 , Huang et al. 2018 , Saxe et al. 2013) . OR has demonstrated its effects on tasks including supervised/semi-supervised image classification, image retrieval, unsupervised inpainting, image generation, and adversarial training (Bansal et al. 2018 , Balestriero et al. 2018 , Balestriero & Baraniuk 2020 , Xie et al. 2017 , Huang et al. 2018) . Efforts to utilize orthogonality in network training have included penalizing the deviation of the gram matrix of each weight matrice from the identity matrix (Xie et al. 2017 , Bansal et al. 2018 , Balestriero et al. 2018 , Kim & Yun 2022) and employing orthogonal initialization (Xie et al. 2017 , Saxe et al. 2013) .For more stringent norm preservation, some studies transform the convolutional layer into a doubly block-Toeplitz (DBT) matrix and enforce orthogonality (Qi et al. 2020 , Wang et al. 2020) .In this work, we first time investigate the efficacy of two orthogonality regularizers, Soft Orthogonality (SO) and Spectral Restricted Isometry Property (SRIP) in SSL (Bansal et al. 2018 ). These regularizers aim to minimize the distance between the gram matrix of each weight matrix and the identity matrix-measured in Frobenius and spectral norms, respectively.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Our approach requires the introduction of additional computational overhead, which is discussed in Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Guidelines: 
settings of self-supervised learning: 
 In this section, we present the general settings for joint-embedding SSL methods. We consider a large unlabelled dataset X ∈ R N ×D , comprising N samples each of dimensionality D. The objective of SSL methods is to construct an effective encoder f that transforms raw data into meaningful representations Z = f (X), where Z ∈ R N ×M and M denotes the representation dimensionality.The learning process of SSL methods is visually represented in Figure 2 , where data augmentations transform X into two augmented views X aug1 , X aug2 ∈ R D×N . A typical joint-embedding SSL architecture encompasses an encoder f and a projector p. These components yield encoder features Z aug1 = f (X aug1 ) and Z aug2 = f (X aug2 ), as well as projection features H aug1 = p(Z aug1 )and H aug2 = p(Z aug2 ). During training, the parameters of f and p are optimized via backpropagation to minimize the discrepancy between H aug1 and H aug2 . To prevent the encoder f from producing a constant feature vector, contrastive methods utilize negative samples, and non-contrastive methods employ strategies such as the self-distillation technique.The efficacy of the encoder f is usually assessed by the performance of the C-class classification as downstream tasks. Specifically, given a labeled dataset containing samples X s ∈ R S×D and their corresponding labels Y s ∈ R S×C , where S is the sample number. Then, a linear layer g parameterized by W c ∈ R C×M is appended on top of the learned representations Z s = f (X s ), and thus the classification task can be fulfilled by minimizing the cross-entropy between softmax(g(Z s )) and Y s . There are two strategies for the fine-tuning: 1) non-linear fine-tuning, which trains both g and f in the downstream tasks, and 2) linear evaluation, which freezes f and only trains g (referred to as the linear probe).Figure 2 : Illustration of joint-embedding SSL methods. This is a general structure. Different augmented inputs can be passed either by shared weight Encoder and Projector or by independent Encoder and Projector, depending on different SSL methods.We introduce two orthogonality regularizers: Soft Orthogonality (SO) and Spectral Restricted Isometry Property Regularization (SRIP), which are seamlessly integrable with linear and convolutional layers.Consider a weight matrix W ∈ R input×output in a linear layer, where input and output denote the number of input and output features, respectively. In line with Bansal et al. (2018) , Xie et al. (2017) , Huang et al. (2018) , we reshape the convolutional filter to a two-dimensional weight matrix W ∈ R input×output , while we still use the same notation W for consistency. To be specific, input = S × H × C in and output = C out , with C in and C out being the number of input and output channels, and S and H representing the width and height of the filter, respectively.The SO regularizer encourages the weight matrix W to approximate orthogonality by minimizing the distance between its Gram matrix and the identity matrix. This is quantified by the Frobenius norm as follows:EQUATIONwhere I is the identity matrix of appropriate size.The SRIP regularizer employs the spectral norm to measure the deviation from orthogonality, which is defined as:EQUATIONwhere σ(•) denotes the spectral norm operator. Due to the high computational cost posed by the spectral norm, the power iteration method (Yoshida & Miyato 2017 , Bansal et al. 2018 ) with two iterations is used for the estimation. The process for estimating σ(W T W -I) is:EQUATIONwhere v ∈ R input is a vector initialized randomly from a normal distribution.• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
incorporating or into ssl: 
 This section details the integration of OR with SSL methods. To be specific, we employ OR across the encoder, targeting both convolutional and linear layers during pretraining. We represent the SSL method's loss function as Loss SSL . Our overall optimization objective is the minimization of the combined loss equation:EQUATIONwhere Loss OR is defined as W ∈f SO(W ) or W ∈f SRIP (W ), depending on the selected orthogonality regularizers. The term γ serves as a hyperparameter that balances the SSL objective and OR loss. Notably, we only perform OR on the weight matrices located within the linear and convolutional layers of the encoder f . OR provides a versatile regularization strategy for the encoder f , facilitating its application across various SSL methods without necessitating modifications to the network designs or existing training protocols. 
analysis of dimensional collapse and the effects of or in ssl: 
 In this section, we show that dimensional collapse happens not only to the representations (i.e. output of the encoder), but also to weight matrices and hidden features of the encoder. We also compare the feature whitening technique used by one previous method, VICREG (Bardes et al. 2022) , with OR. Migrating to BYOL, we find that the feature whitening technique only solves the dimensional collapse at the feature level, but instead accelerates the collapse of the weight matrices, and it even leads to lower performance of the downstream tasks as shown in Table 1 . In contrast, OR can eliminate the dimensional collapse of weight matrices and thus the dimensional collapse of hidden features and representations.In Appendix A.1, we further reveal that the original VICREG has a dimensional collapse problem with its weight matrices, which could not be solved by removing its projector. Adding OR to VICREG eliminates this problem and boosts the performance. To study the eigenspace of a matrix, we utilize the normalized eigenvalues defined in He & Ozay (2022) :Definition 1 (Normalized eigenvalues) Given a specific matrix T ∈ R N ×D , where N is the sample number and D is the feature dimension, we first obtain its covariance matrix Σ T ∈ R D×D . Then we perform an eigenvalue decomposition on Σ T to obtain its eigenvalues{λ T i } D i=1 = {λ T 1 , • • • , λ T i , • • • , λ T D } in descending order.And we obtain the normalized eigenvalues by dividing all eigenvalues by the max eigenvalue λ T 1 , denoted as{λ T 1 /λ T 1 , • • • , λ T i /λ T 1 , • • • , λ T D /λ T 1 }.To simplify the denotation, we reuse {λ T i } D i=1 to denote normalized eigenvalues of T .These normalized eigenvalues are less than or equal to 1, where a larger value in one dimension indicates more information contained, and vice versa. We argue that if normalized eigenvalues drop very quickly, this means that only a few dimensions in the eigenspace contain meaningful information and also means that dimensional collapse has occurred.We train three BYOL (Grill et al. 2020 ) models, without OR, with the feature whitening technique (e.g. Variance and Covariance regularization) from VICREG and with OR, respectively. We choose randomly initialized ResNet18 as the backbone (i.e. encoder) and train the three models on CIFAR-10 for 1,000 epochs, following the same recipe of Da Costa et al. (2022) . Importantly, SO is selected as the orthogonality regularize, and γ is set to 1e -6. For the feature whitening technique, we impose the Variance and Covariance regularization from VICREG on the output of the predictor in BYOL as two additional loss terms, the former to ensure the informativeness of individual dimensions and the latter to reduce the correlation between dimensions. Following the solo-learn settings, we set the two loss term hyperparameters to γ vic and γ vic * 0.004, and then tune the γ vic from 1e -3 to 1e -5.After training, we calculate the normalized eigenvalues of both weight matrices and features (e.g. input features, hidden features, representations). ResNet18 contains four basic blocks, each containing four convolutional layers, and we visualize the normalized eigenvalues of the last convolutional layer in each block. Hidden features are the outputs of four basic blocks in ResNet18. We use the first batch in the test set as input features with batchsize 4,096 and feature dimension 3072 (32*32*3).Results are analyzed in the following sections.We first examine the weight matrices of the encoder. Similar to 5.1, all the weight matrices are viewed as two-dimensional matrices, and on top of them, we can calculate their normalized eigenvalues. For a specific weight matrix W ∈ R input×output in a neural network layer, we denote {λ W i } output i=i as the normalized eigenvalues of this layer.As shown in Figure 3 , X and Y axis is the i index and i-th values of {λ W i } output i=i , respectively. It is clear that with OR, the eigenvalues of the convolutional layers of different depths decay more slowly, which means that their filters are less redundant and more diverse. In particular, we note that the deepest convolutional layer (i.e. layer4_512) has a much faster decay rate of the eigenvalues compared to the other convolutional layers in the absence of OR. Notably, the feature whitening technique does not alleviate this phenomenon. OR could significantly improve this. OR requires the weight matrix to be as orthogonal as possible, which means that the diagonal elements of its covariance matrix are as identical as possible, and the off-diagonal elements will be close to 0. Then the eigenvalue decomposition of such a covariance matrix will have elements in {λ W i } output i=i that are all close to 1 (verified in Appendix A.2).As the weight matrices become orthogonal, the distribution of their outputs stabilizes, thereby preventing the dimensional collapse of hidden features and representations. This property has been demonstrated in vector form by Huang et al. (2018) and is now presented in matrix form:Proposition 1 For a specific weight matrix W ∈ R input×output and X ∈ R N ×input , comprising N samples each of dimensionality input. We denote X and S as the sample means of X and S, respectively. Let S = XW , whereW T W = I. The covariance matrix of X is Σ X = (X-X) T •(X-X) N -1 . (1) If X = 0 and Σ X = σ 2 I, then S = 0 and Σ S = σ 2 I. (2) If input = output, we have ∥S∥ 2 F = ∥X∥ 2 F . (3) Given the back-propagated gradient ∂L ∂S , we have ∂L ∂S 2 F = ∂L ∂X 2 F .The first point of Proposition 1 illustrates that in each layer of DNNs, the orthogonal weight matrix preserves the normalization and de-correlation of the output S, assuming the input is whitened. This reveals that as the network gets deeper, the hidden features of each layer and the final representations do not tend to collapse. Moreover, orthogonal filters maintain the norm of both the output and the back-propagated gradient information in DNNs, as demonstrated by the second and third points of Proposition 1.To verify that the dimensional collapse of features can be eliminated by OR, we visualize the normalized eigenvalues of features (input features, hidden features, and representations) as shown in Figure 3 . Without the OR constraint, features located in deeper layers (i.e. representations) will have the fastest eigenvalues decay rate, and the distributions of eigenvalues of hidden features vary considerably at different depths. After adding the OR, it can be seen that the decay rates of hidden features at layers 3 and 4 are almost the same, while the eigenvalues of representations decay much more slowly. We also visualize the representations in Appendix A.3, which also verifies that the dimensional collapse of the representations is mitigated. Interestingly, the effect of OR on the feature level is similar to that of the feature whitening technique, however, the latter is unable to eliminate the dimensional collapse in the weight matrices. 
numerical experiments: 
 We study the effects of OR on SSL methods through extensive experiments. we first demonstrate that OR improves the classification accuracy on CIFAR-10, CIFAR-100, and IMAGENET100, and the improvement is consistent across different backbones and SSL methods. On the large-scale dataset IMAGENET-1k (Deng et al. 2009) , OR boosts the classification accuracy on both in-distribution and out-distribution datasets (i.e. transfer learning datasets), demonstrating consistent improvement. Moreover, OR also enhances the performance in downstream tasks(e.g. object detection).Baseline methods and datasets. We evaluated the effect of adding OR to 13 modern SSL methods, including 6 methods implemented by solo-learn (MOCOv2plus In addition to CNN backbones, OR is also able to improve SSL performance on Transformer-based backbones (e.g., VIT) (Han et al. 2022 , Dosovitskiy et al. 2020 , Zhou et al. 2021 ). We pretrain DINO on CIFAR-100 with different depths of VITs. As shown in Table 3 , with the increasing depth of the VIT, the original DINO performance is increasing, and OR is able to increase their performance even further, which exhibits a good scaling law. Interestingly, under the Transformer-based architecture, OR is able to improve performance more (up to 12%) compared to CNN backbones. This is consistent with some existing studies (RoyChowdhury et al. 2017) that linear layers are more likely to have redundant filters than convolutional layers in DNNs, i.e., more prone to dimensional collapse. We demonstrate the effects of OR on the large-scale dataset IMAGENET-1k. Specifically, we pre-train three BYOL models-BYOL without OR, BYOL with SO, and BYOL with SRIP on IMAGENET-1k (with ResNet50). For each testing classification dataset, we report the accuracy as shown in Table 5 and 6.  
conclusions: 
 The existing studies focus on the dimensional collapse of representations and overlook whether weight matrices and hidden features also undergo dimensional collapse. We first time propose a mitigation approach to employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matrices, thus safeguarding against the dimensional collapse of weights, hidden features, and representations.Our empirical investigations demonstrate that OR significantly enhances SSL method performance across diverse benchmarks, yielding consistent gains with both CNNs and Transformer-based architectures as the backbones. Importantly, the time complexity and required efforts on fine-tuning are low and the performance improvement is significant, enabling it to become a useful plug-in in various SSL methods.In terms of future research, we wish to examine the effect of OR on other pre-training foundation models, such as vision generative SSL models such as MAE (He et al. 2022) , auto-regression models like GPTs and LLaMAs (Radford et al. 2018 , 2019 , Brown et al. 2020 , Touvron et al. 2023) , and Contrastive Language-Image Pre-training models (Radford et al. 2021 , Li et al. 2022) . We believe OR is a pluggable and useful module to boost the performance of vision and language foundation models. In fact, this paper is the first to test the effectiveness of OR in a Transformer-based architecture and it is reasonable to believe that it will perform well in these domains.A Appendix / supplemental material A.1 Effects of Representation Whitening on the EncoderIn this section, we explore the effect of whitening representations on hidden features and weight matrices in the encoder. To be specific, similar to the settings of 5, we train three VICREG Bardes et al. (2022) models: original VICREG, VICREG without projector (Li, Chen & Yang 2020), and VICREG with OR.Original VICREG adds two regularization terms (variance and covariance regularization) to whiten the projector features. We use X aug1 as an example to introduce them.Variance regularization. The variance regularization term ensures that each dimension of the learned representation Z maintains a non-trivial variance. This is critical to prevent the collapse of dimensions, where a model might ignore certain informative variations in the data. Mathematically, the variance regularization can be expressed as follows:EQUATION)where D is the dimensionality of Z aug1 = f (X aug1 ), z d represents the d-th dimension of Z aug1 , S(z d , ϵ) = Var(z d )) + ϵ is the regularized standard deviation of z d across different samples, and γ is a threshold parameter that dictates the minimum desired standard deviation for each dimension.Covariance regularization. The covariance regularization term is designed to decorrelate the different dimensions of Z aug1 . By minimizing the off-diagonal elements of the covariance matrix of Z aug1 , this term helps ensure that different dimensions capture distinct aspects of the data, thereby preventing redundancy in the representation. The covariance regularization is defined as:EQUATIONwhere Cov(z i , z j ) denotes the covariance between the i-th and j-th dimensions of Z aug1 . This term effectively encourages the representation to have orthogonal dimensions, which is beneficial for learning independent features.As for the VICREG without projector, we discard the projector and apply the SSL objective directly to the representations, which ensures that the representations are whitened (no dimensional collapse in representations), i.e., minimize the correlation among dimensions and make each dimension rich in information. For OR, we choose SO as the regularizer and set γ as 1e -6. We then experimentally observe that guaranteeing that dimensional collapses do not occur in representations or projector features (i.e., VICREG without projector and projector features) does not guarantee that dimensional collapses do not occur in weight matrices in the encoder. Moreover, discarding the projector even damages the performance of the original VICREG, while OR still boosts the performance as shown in Table 8 . As shown in Figure 4 , the weight matrices of the original VICREG suffer from dimensional collapse and whitening representations directly (i.e., getting rid of the projector) even makes the eigenvalue decay faster for the weight matrices. OR can alleviate this phenomenon. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The data sets we use are all public data sets, and we strictly follow existing studies, which will not have a bad impact on the society.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our goal is to improve the performance of the vision foundation model, which will have a positive impact on various downstream tasks.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our approach addresses the potential dimensional collapse in SSL without raising their risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We referenced the framework we used and the data set we used. In the code, we listed the URL of each data set and the code environment we needed.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not introduce new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, layer4 of ResNet18 pretrained with BYOL on CIFAR-10 is visualized. To be specific, we calculate the correlation coefficient matrix of the weight matrix and then plot the HeatMap of the correlation coefficient matrix and the results of Spectral Biclustering. As shown in Figure 5 , HeatMap can intuitively indicate that the correlation of non-diagonal elements is constrained to 0 by OR. The Biclustering results show an obvious blocky structure, which means that there is clustering between filters, and the weight matrix is low-rank and redundant.We used BYOL (ResNet18) for pretraining on CIFAR-10. After pretraining, we perform dimension reduction and visualization of learned representations using UMAP (McInnes et al. 2018) . As shown in Figure 6 , in the absence of OR, there is a tendency for the cluster centers of each category to move closer together and more outliers appear. This is due to the fact that in the absence of OR, BYOL produces representations dominated by some extremely large eigenvalues (i.e. dimensional collapse), which is consistent with results in Section 5.2. We utilized several datasets for pretraining and evaluating SSL methods. Below we provide a detailed description of these datasets:• IMAGENET-1k (Deng et al. 2009 • GTSRB (Haloi 2015) : The German Traffic Sign Recognition Benchmark (GTSRB) dataset consists of over 50,000 images of traffic signs across 43 categories.• PASCAL VOC2007 and VOC2012 (Lin et al. 2014 ): Used for evaluating objection tasks, this dataset includes complex everyday scenes with annotated objects in their natural context. The objection detection task contains 20 categories. We use the VOC2007 and VOC2012 train-val (16551 images) as the training set and then report the performance on the VOC2007 test set (4952 images).Each dataset was carefully curated to support the training and validation of our models, ensuring a comprehensive evaluation across various image classification and segmentation tasks.Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations without the need for labeled data. This appendix provides a concise overview of several SSL methods used in this paper.• MOCOv2, introduced by Chen & He (2021), on top of MOCO's momentum encoder and the use of the dynamic dictionary with a queue to store negative samples (He et al. 2020) , adds the MLP projection head and more data augmentation. Compared to MOCOv2, MOCOv2plus uses a symmetric similarity loss.• MoCov3 (Chen et al. 2021 ) makes some improvements on the basis of v1/2, firstly, because the batchsize is large enough when training V3, the memory queue is removed, and the negative samples are sampled directly from the batch. Secondly, symmetric contrastive loss is used, and finally, an extra prediction head is added to the original encoder, which is a two-layer fully connected layer.• Bootstrap Your Own Latent (BYOL), proposed by Grill et al. (2020) , introduces a novel approach to SSL that does not rely on negative pairs. Instead, BYOL employs a dual-network architecture where the encoder learns to predict the representations of the momentum encoder. Through a series of updates (i.e. EMA), where the momentum encoder gradually assimilates the encoder's weights, BYOL effectively learns robust representations. The success of BYOL depends not only on the EMA, but also on its additional projector and the BN in the projector to avoid a complete collapse of the encoder. This method challenges the conventional wisdom that contrastive learning requires negative pairs, opening new avenues for SSL research.• Expanding on the ideas of BYOL, NNBYOL (Dwibedi et al. 2021 ) introduces the concept of using nearest neighbors to augment the learning process. By leveraging the similarities between different instances in the dataset, NNBYOL aims to refine the quality of the learned representations further. This approach underscores the potential of incorporating instancelevel information into the SSL framework, enhancing the discriminability and robustness of the resulting models. • DINO (Caron et al. 2021 ) uses a self-distilling architecture. The outputs of the teacher networks (i.e. the momentum encoder) are subjected to a centering operation by averaging over a batch, and each network outputs a K-dimensional feature that is normalized using Softmax. The similarity between the student model (i.e. the encoder) and the teacher model is then computed using cross-entropy loss as the objective function. A stop-gradient operator is used on the teacher model to block the propagation of the gradient, and only the gradient is passed to the student model to make it update its parameters. The teacher model is updated using the weights of the student model (i.e. EMA). • Barlow Twins computes the correlation matrix between the embeddings of two different views of the same sample and avoids collapse by making it as close as possible to the unit matrix. This approach makes the embeddings between the two views of the sample as similar as possible while minimizing the redundancy between vector components. • VICREG avoids the complete collapse problem with variance and covariance regularization.• DCL and DCLW removes the NPC effect of infoNCE loss by getting rid of the positive term from the denominator and thus significantly improves the learning efficiency • SimSiam achieves a very strong baseline without using large batchsize, negative samples, or momentum encoder using only the stop-gradient operation. • SwaV is trained by predicting the clustering assignment of another view and also introduces multi-crop, which increases the number of views by reducing the image size without increasing the extra memory and computational requirements. • SimCLR establishes a simple and effective architecture for contrastive learning by increasing the batchsize, augmenting the data and adding a nonlinear projector after the representation.For each SSL method, we use the original settings of Solo-learn and LightlySSL (Da Costa et al. 2022 ). These settings include the network structure, loss function, training policy, and data augmentation policy. Considering that we use numerous SSL methods and that our setup is exactly the same as them, please go to their official implementation.For OR, the appropriate regularization term γ generally depends only on the backbone used by SSL and the orthogonality regularizer (SRIP or SO) chosen. As shown in Table 9 , when you want to add OR to your SSL pre-training, you simply pass the encoder into the loss function, and then you just need to set γ of the OR according to the backbone and regularizer you use. For the classification tasks, due to computational constraints, we do not perform non-linear fine-tuning in classification tasks. Instead, we perform a linear probe or KNN to evaluate the quality of obtained representations as typically done in the literature (Huang et al. 2024 , Li, Chen & Yang 2020 , Lavoie et al. 2022) . To be specific, for each SSL method and dataset, after pretraining, We train a linear classifier on top of frozen representations of the training set. Then we report the Top-1 and Top-5 linear classification accuracy on the test set. When training the linear classifier, we use 100 epochs, weight decay to 0.0005, learning rate 0.1 (we divide the learning rate by a factor of 10 on Epoch 60 and 100), batchsize 256, and SGD with Nesterov momentum as optimizer (In IMAGENET-1k, we use batchsize 128 and learning rate 0.2).For the object detection task, we perform nonlinear fine-tuning on ResNet50 in RCNN-C4 (Girshick et al. 2014) with batchsize 9 and base learning rate 0.01. We use the detectron2 (Wu et al. 2019) , following the MOCO-v1 (He et al. 2020 ) official implementation exactly.Implementing OR requires computing the OR loss in the backbone at each gradient update, we count the time required by the different backbones to compute OR at one time, and we have averaged over 10 times as shown in Table A .7. In the pre-training phase, the time overhead of OR is only related to the backbone and the steps that need to be updated, IMAGENET-1k (100 epochs, batchsize 128) has a total of 62599 steps, and CIFAR-100 (1000 epochs, batchsize 256) has a total of 194999 steps. As you can see, compared to the original pre-training overhead of dozens and hundreds of hours, the additional time added by OR is very small, steadily improving SSL's performance. Notably, if we use a larger batchsize such as 4096, our time overhead will be reduced by 64 on IMAGENET-1k and 16 on CIFAR-100. Justification: In the abstract, we explained that dimensional collapse would exist in weight matrices and features, which would damage the performance of SSL. We used OR to eliminate the dimensional collapse in SSL. In addition, consistent improvement results were achieved under each benchmark.In the introduction, we list our 3 contributions in detail.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: **Summary of the paper:**
The paper proposes weight regularization in order to avoid feature dimensionality collapse in self-supervised learning. It reguralizes the weights by adding a loss function, of two possible types, that encourage orthogonality of the weight matrix as well as a uniform spectral domain. It shows on several architectures, a few SSL algorithms, and a couple of datasets, that the addition of their loss often improves the downstream performance.

**Summary of the reviews:**
The reviewers found the paper well-written, and the proposed method simple but well-motivated. They mostly found the results convincing in that the proposed loss function usually shows benefits over its absence. 

On the other hand, the reviewers had doubts about the soundness of the experimental setup when in comes to the paper’s results on the established methods and architectures. They also pointed a lack of comparison with other methods to avoid collapse such as Barlow Twins and VIC regularization. Two reviewers found the technical contribution to be low. 

**Summary of the rebuttal and discussions:**
The authors point out the subpar results is due to the plugging of architectures into their used codebase without hyperparameter optimization or, in the case of DINO, due to the lack of the essential multicrop augmentation. They further offered new experiments by adding their loss function on top of VICreg and BT. 

**Consolidation report:**

During the discussion the reviewers were mostly (but not fully) convinced regarding the occasional subpar results of the baselines, especially given that the authors promised to improve and reflect on these discrepancies in the final version. As a result they all moved their ratings towards acceptance. 

Dimensionality collapse is an important issue with SSL which is an important field, this can compensate for the lack of general technical contribution pointed out by reviewers

**Recommendation:**
All in all, the AC agrees with the unanimous view of the reviewers and recommends acceptance.

In the final version, it is important to,
- as promised, improve and reflect upon the reported results as was partially done during the rebuttal.
- cite the following [a] paper that is published at NeurIPS 2023 workshops and is directly related to the proposed method.

[a] “WERank: Rank Degradation Prevention for Self-Supervised Learning via Weight Regularization” NeurIPS 2023 workshop on Self-Supervised Learning: Theory and Practice

Reviews: review: paper_summary: paper_summary: Dimensional collapse, characterized by the dominance of a few large eigenvalues, is a phenomenon in self-supervised learning (SSL) that can lead to redundant features and weight matrices. To address this, the authors propose orthogonal regularization (OR) during pretraining to promote orthogonality in weight matrices, rather than using common approaches that regularize features via the objective (e.g., Barlow Twins, VicReg). The authors conduct experiments with five SSL methods from the SoloLearn framework to demonstrate the benefit of OR.
strength_weakness: strength_weakness: The soundness of some experiments seems doubtful. Results for WideResNet28x2 are consistently below those for ResNet18. Additionally, the results using Dino are generally poor, leading to all ViT experiments being below any ResNet baseline.

The performance of BYOL with OR, when trained for only 100 epochs with a batch size of 64, is even below the reported accuracy in SoloLearn. Considering the optimal training duration for BYOL is approximately 1000 epochs with a batch size of 4096, this comparison appears to be in a severely undertrained regime. Slightly longer training, 200 epochs with a batch size of 256, shows that the benefit of the OR approach diminishes rapidly.

Minor issue: Line 44 states, "We evaluate the effect of adding OR in seven modern SSL methods during pretraining," but results are shown for only five methods.

Additionally, since SSL approaches like Barlow Twins and VicReg aim to achieve orthogonality as well, there should be an experiment evaluating the effect of OR on these methods.

Using one learning rate (LR) for all methods in linear probing might not be optimal. It is cheap to attach multiple linear probing heads and find the most suitable LR for each method to ensure fair comparisons.
questions: questions: Memory Overhead of OR: How much memory overhead does orthogonal regularization (OR) introduce? 

Explanation for Lower Gamma in Larger Models: The paper uses a lower gamma for larger models, but why? One might think that larger models would require more regularization. 

Results for Barlow Twins and VicReg: Why were results for Barlow Twins or VicReg not reported? Is the OR approach redundant when combined with these methods, which also aim to achieve orthogonality?

Balanced Eigenvalues and Performance Discrepancy: The eigenvalues of VicReg without a predictor (Appendix Fig 4) appear more balanced than those of BYOL OR (Fig 3), yet VicReg's performance is far worse.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper aims to improve self-supervised learning (SSL) by introducing orthogonality regularization (OR) during pretraining to prevent dimensional collapse. OR improses orthogonality constraint on weight matrices, not only preventing dimensional collapse of representation but also regularizing the weights from collapsing. The experimental results show that the benefit of the proposed method is consistent and widely applicable across datasets and architecture choices.
strength_weakness: strength_weakness: * More discussion on the difference between the proposed orthogonal regularization methods would have been helpful. On the surface, the two methods are almost equally good.
questions: questions: * In practice, a practitioner has to choose between SO and SRIP. What would be the authors' recommendation? 
* Do you expect that applying dropout on the encoder's hidden neurons would result in a similar effect, as dropout is also meant to prevent feature co-adaptation? It is quite likely that the existing encoders have already employed dropout, so I wonder if dropout is insufficient to prevent the collapse.
limitations: limitations: NA
paper_summary: paper_summary: This work proposes a method to reduce the dimensional collapse issue in self-supervised pretrained networks. Dimensional collapse occurs when a few large eigenvalues dominate the eigenspace. While previous research focused solely on the output representations of the pretrained networks, this work shows that dimensional collapse also occurs on the intermediate representations and weight matrices.

To address this issue, the authors use orthogonal regularization (OR) objectives for the weight matrices in linear layers (dense or convolutional) within the network. The OR objective promotes orthogonality in the weight matrices by minimizing the distance between the Gram matrix of each weight matrix and the identity matrix. They demonstrate that using the OR regularization objective leads to a smoother decay of eigenvalues in the (intermediate) representations and weight matrices of a self-supervised pretrained ResNet18.

The authors evaluate the OR regularization objective by pretraining various backbones (RN18, RN50, WideResNet28w2, and ViT-tiny/small/base) with multiple joint-embedding SSL methods (MoCov2, MoCov3, DINO, BYOL, NNBYOL) on CIFAR-100. They then train linear classification models on the frozen features, observing improved accuracy in nearly all cases with OR. Finally, they pretrain a RN50 backbone with BYOL on ImageNet-1k and evaluate it through linear probing on ImageNet and fine-tuning on Pascal object detection, showing improved results with OR.
strength_weakness: strength_weakness: - **(W1)** The introduction's claimed contribution, "We **systematically study** the phenomenon of dimensional collapse in SSL, including how whitening representations and network depth affect the dimensional collapse of weight matrices and hidden features," seems to me to be an exaggeration. In the main paper, there is only one experiment pretraining ResNet18 with BYOL w/ and w/o OR on CIFAR-10, and another experiment in the supplementary material using VICReg, which are not directly comparable. Given the emphasis on this contribution, a more appropriate approach would be to include an analysis of whitening's impact on dimensional collapse in the main paper (e.g., adding another column with whitening-based results in Figure 3). This analysis should be conducted under the same conditions as the experiments in Figure 3, for example, using BYOL with a projector and whitening. This setup would enable a straightforward comparison of vanilla BYOL, BYOL with OR, and BYOL with whitening in terms of their effects on the eigenvalues of intermediate representations and weight matrices.
- **(W2)**: **Limited technical contribution**: as discussed in the related work sections of this work, orthogonality regularization of the weight matrices within neural network has been extensively studied before (related work subsection 2.3). In particular, the OR objectives studied here are directly taken from Bansal et al. 2018, and straight-forwardly applied to the examined self-supervised approaches, without any modification at all. So, the main novelty of this work is to apply in a plug-and-play way the OR regularization objectives of Bansal et al. 2018 on self-supervised methods.
- **(W3)** About the CIFAR100 results: in the results provided in solo-learn (https://github.com/vturrisi/solo-learn), which the authors also use for their experiments, some baselines (i.e., SSL methods that do not use OR) have higher results than what is reported in this work and often higher even to the results reported in this work with the OR regularization objective.  In particular, this is the case for MoCov2+ (69.89% in solo-learn vs 69.51% here), MoCov3 (68.83% in solo-learn vs 67.13% here), DINO (66.76% in solo-learn vs 59.13% in here). For BYOL however, is the other way around (70.46% in solo-learn vs 71.15% in here). What is the reason for the differences in the results?
- **(W4)** The improvement that OR offers over a well-tuned BYOL baseline on ImageNet (200 pretraining epochs and 256 batch-size) is marginal: from 69.76 to 70.16. It seems that it offers significant performance improvements in mostly un-tuned baselines, like the results when pretraining BYOL for 100 and for 64 batch-size. In reality, even 200 epochs are not adequate for pretraining BYOL that, its performance reaches 74.3 for 1000 epochs (see BYOL paper). So, it is unclear whether OR could offer any improvement for such long pretraining settings that are typically used in SSL. 
- **(W5)** Not my primary concern, but although OR is shown to provide a small improvement on the linear accuracy of some SSL methods (e.g., going from 69.76 to 70.16 for BYOL pretrained with 200 epochs on ImageNet-1k), no evidence is shown that OR actually avoids dimension collapse in settings where otherwise (i.e., without OR) the SSL method will completely collapse and fail to learn any meaningful representation.
questions: questions: I would like the authors to address my concerns (W1), (W2), (W3) and (W4), as described in the weaknesses section. Some more specific questions:
- About W1: can the authors provide a comparison of vanilla BYOL, BYOL with OR, and BYOL with whitening in terms of their effects on the eigenvalues of intermediate representations and weight matrices.
- About W3: What is the reason for the differences in the results reported in this work and in in solo-learn (https://github.com/vturrisi/solo-learn)?
- About W4: Can the authors demonstrate that OR improves even well-tuned baselines like BYOL with 1000 epochs (74.3 in BYOL paper) or MoCo-v2 with 800 epochs (71.1% in https://arxiv.org/pdf/2003.04297), or MoCo-v3 (74.6 for RN50 see: https://github.com/facebookresearch/moco-v3). Any of them would be good.
limitations: limitations: Yes, they are discussed.
paper_summary: paper_summary: In order to combat the problem of dimensionality collapse in self-supervised learning, the authors of this work introduce orthogonal regularization on encoders during pretraining. Taking inspiration from previous work on supervised orthogonality regularization, the authors study the impact of OR on eigenvalues of weights when considering self-supervised learning and find that OR learned weights/features contain more meaningful features and avoid dimensional collapse better. The recommended result seems to be effective for both contrastive and non-contrastive SSL methods.
strength_weakness: strength_weakness: 1) I would like to point out the improvements of OR on SSL methods is not as major on the simple datasets such as Cifar100 set according to Table 1. However, the changes in larger datasets with BYOL are more noticeable but I would like to ask why results for other methods were not included for larger datasets. 
2)  Particularly with regards to Fig 3, to me it doesn’t look as if the feature eigenvalue changes much with the OR except for the representations. Perhaps a combination of other OR methods could help ? Or adding orthogonality regularization with the features. 
3) With regards to classification accuracy on Table 5, higher batch size and longer training somewhat diminish the impact of OR improvement. Perhaps this result could be used to encourage OR when training with smaller batches or less epochs is required ?
questions: questions: 1) For section 3.2 under what case would we have input > output ? 
2) Could the authors provide some explanation for what entails whitening of the features ?
limitations: limitations: None
paper_summary: paper_summary: Driven by the dimensional collapse phenomenon observed in self-supervised learning, this research adopts a methodology designed to prevent the reduction in dimensionality of weight matrices. The empirical findings indicate a reduction in dimensional collapse in certain instances.
strength_weakness: strength_weakness: The proposed method lacks novelty, essentially applying the loss formulation from Barlow Twins to the weight matrix.

Moreover, the experimental results are insufficient to substantiate the efficacy of the method due to the following reasons:

1. Many of the SSL methodologies utilized to assess the OR are no longer considered state-of-the-art. Therefore, it is recommended to evaluate the OR using state-of-the-arts methods.

2. Table 1 reveals that the effectiveness of SO or SRIP appears to exhibit considerable variability, rendering the proposed methods unstable.

3. Tables 3 and 5 exclusively demonstrate the evaluation of OR for the BYOL method, omitting the efficacy assessments of other SSL methods. Additionally, in Table 4, the datasets are characterized as small-scale, which contradicts the claims made by the authors.
questions: questions: 1. What are the advantages of regularizing weights over regularization of representation?

2. What is the performance of SRIP in Table 2 and 5?
limitations: limitations: Please refer to the weakness.

Comments: title: Reply to the rebuttal
Reply for rebuttal
comment: Thank you for your kind response and suggestions.

>(A) 

We are very happy that we have solved this part of your concerns and it has greatly improved the quality of our work.

>(B)

Indeed, as suggested by Reviewer KCGP, we intend in the future to make changes to the form of the OR to avoid the adjustment of the OR hyperparameters by the increase in model size. 

>(C)

Thanks again for your recognition! 

>(D)

DINO on Solo-learn results are really puzzling. For this reason, we performed DINO ResNet50 300 epoch experiments on the LIGHTLY framework. However, the results are still far from those in the original DINO article, and we found that this is due to the fact that **LIGHTLY does not perform multi-crop**, which has a fatal impact on the performance. Without multi-crop, as shown in the following table (results from the paper of DINO), we believe that LIGHTLY's results are consistent with the original DINO article.

|  | KNN Top1 | Linear Top1 |
| :---- | :---- | :---- |
| DINO  (vit/s)  | 72.8 | 76.1 |
| DINO (vit/s) w/o multi-crop | 67.9 | 72.5 |
| DINO  (ResNet 50\) | 65.6 | 74.5 |
| DINO (ResNet 50\) w/o multi-crop LIGHTLY | 59.8 | 71.9 |
| DINO (ResNet 50\) w/o multi-crop  \+SO LIGHTLY | 60.4 | 72.2 |

In the future, to be more convincing, we need to add multi-crop to LIGHTLY or conduct experiments in the DINO repository.
Here are some post-rebuttal comments regarding how the authors addressed my concerns:

(A) Overall, the authors have satisfactorily addressed my concerns W1, W3, and W5.

(B) Regarding W2: Demonstrating the benefits of adding the OR regularization to SSL methods is indeed a notable contribution. However, OR is applied directly, in a plug-and-play manner, without any modifications proposed in this work to "turn it into a plug-and-play method for SSL," as the authors stated. Nevertheless, I still consider the contribution of showing the advantages of OR in the context of SSL methods to be significant.

(C) The authors provided many more experiments in the rebuttal, showing that OR consistently improves any baseline SSL method to which it is applied. As I mentioned in my initial review, this versatility of OR is the strongest aspect of this work!

(D) However, it remains unclear to me whether OR could improve the performance of a well-tuned baseline. For instance, the reproduced DINO results with the ResNet50 backbone provided by the authors are still notably worse than the ResNet50 results reported in the DINO paper. This is the main remaining weakness of the paper.

In conclusion, considering the above comments, I am going to increase my score. I am between "weak accept" and "borderline accept," leaning more towards "borderline accept" due to comment (D): it is unclear whether OR can enhance the performance of well-tuned baselines.
Thanks again for your suggestion and reply. 
Our next experiments with backbone replacement will try to adjust the hyperparameters of the corresponding SSL to optimize the original method and then add the OR. This setup will definitely be more convincing.
My point was that consistent results do not directly render an experiment useful, as there is too much room for improvement when the tuning is not done correctly. 

I want to clarify that I did not want to critique the new experiments, which are small scale, but look solid and cover a lot of methods.
Thank you for your response and further suggestions.

> DINO results on solo-learn

Indeed, all of our methods follow the config file provided by solo-learn, where the hyperparameters are inconsistent with the checkpoints, leading to rather puzzling results for DINO on solo-learn, which is why we supplemented our experiments with the performance of DINO on other frameworks. **Our additional experiments show that DINO on CIFAR10 and ImageNet1k under the LIGHTLY framework is more in line with expectations, and OR improves their results**. We will next replace the backbones (WideResNet and VITs) in the LIGHTLY framework.

>  Memory overhead

Yes, OR brings only a very small memory overhead and is not affected by batch size, and is one of its major advantages over other feature whitening based methods, we will add these discussions to the manuscript!

>  One the one hand, the introduction of a new hyperparameter and the increased effort to tune it should lead to a significant increase in a well tuned baseline that is trained for sufficiently long time. The current work does not convince me that this is the case. On the other hand, the improvements with short training schedules and the improved transfer capabilities are noteworthy

Due to limited time and GPU resources, we only tested the effect of OR in DINO 300 epochs on ImageNet1K during the rebuttal period, more epochs of the experiment are catching up to run, and your suggestion is very reasonable. We will not only test the performance under the original dataset, but its performance in transfer learning will also be reported, considering that we experimentally found that although the improvement is not significant under the original data, OR can still bring significant improvement in the transfer learning scenario.

>   A study how to the gamma scales with model size and training schedule that might reduce the necessary tuning of the introduced hyperparameter would be of interest.

We totally agree with you that less tuning of the OR hyperparameters can make the OR more plug-and-play. In the future, we will try not to add up the OR loss for each weight matrix, but average it. This way, the model size won't affect the magnitude of OR loss much.

> BarlowTwins/VicReg to specifically measure the benefits of improved efficiency regarding training time or training data as well as robustness to transfer learning could improve the work

Thanks again for your kind suggestions. In additional experiments, BarlowTwins gained boosts from OR on both CIFAR10 and ImageNet100. We will test the results of BarlowTwins and VICREG on ImageNet1K further with more training epochs and transfer learning scenarios.
Thanks to the authors for the thorough response and the additional experiments.

(A) Regarding experiments. It is not a fair comparison if one uses suboptimal hyperparameters for a baseline and then introduces a new hyperparameter that can be tuned. It does not provide a lot of insights of the usefulness of the method to a well tuned baseline. It only shows that the proposed regularization has some kind of effect on the training. The experiments with WideResnet and Dino in its current form are simply a distraction. Either the dataset is not large enough to apply a WideResNet or the hyperparameters do not work at all. For Dino, it is true that Dino ViT ourperforms Dino ResNet. But all other methods outperform the best Dino ViT result even with ResNet18. Therefore, the used hyperparameters or training schedule of the Dino setting do not work at all.

(B) Memory overhead. Thank you, the fact that memory overhead is feasible and does not scale with the used batchsize is great.

(C) Usefulness of the proposed method. One the one hand, the introduction of a new hyperparameter and the increased effort to tune it should lead to a significant increase in a well tuned baseline that is trained for sufficiently long time. The current work does not convince me that this is the case. On the other hand, the improvements with short training schedules and the improved transfer capabilities are noteworthy. To me this is very similar to the strengths and weaknesses of Barlow Twins (sensitive to the number of features in the projector space). The inductive bias of the independence regularization speeds up learning and creates a representation that is robust to shifts, but can also be detrimental to fit a function in domain, given enough samples and training time. A study how to the gamma scales with model size and training schedule that might reduce the necessary tuning of the introduced hyperparameter would be of interest. Futhermore a direct comparison to BarlowTwins/VicReg to specifically measure the benefits of improved efficiency regarding training time or training data as well as robustness to transfer learning could improve the work in my opinion.

I will keep my score, but I am not opposed to the acceptance of the work given the predominantly positive scores by other reviewers. EDIT: In expectation that the authors incorporate the feedback into their final version, I will raise my score.
Thank you so much for recognizing our work! Because of your detailed and kind suggestions, we have greatly improved the quality and novelty of our articles, and we will definitely add these to the manuscript.
Thank you for conducting the additional experiments and providing the discussions in response to the review. The authors' rebuttal has addressed most of the previously raised concerns, leading me to lean towards a positive review score. If the authors emphasize their responses regarding the novelty of the paper and the advantages of regularizing weights over regularizing representations in the introduction, it will likely enhance the paper's novelty and motivation. Therefore, I hope these responses will be incorporated into the manuscript.
Thank you very much for your kind reply and recognition of our work. Regarding empirical results, we also hope that we can address the concerns of other reviewers during the discussion phase.
Thank you for your response. Your answers clarified my questions. Other reviewers seem to have concerns regarding empirical results, of which I don't have a strong opinion, since I do not have much hands-on experience with SSL. In general, I believe the paper is well-structured and easy to understand. Hope the concerns regarding the experiments are resolved well during the discussion.