Title: Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Self-supervised learning;  Orthogonality Regularization; Dimensional collapse
Abstract: Self-supervised learning (SSL) has rapidly advanced in recent years, approaching the performance of its supervised counterparts through the extraction of representations from unlabeled data. However, dimensional collapse, where a few large eigenvalues dominate the eigenspace, poses a significant obstacle for SSL. When dimensional collapse occurs on features (e.g. hidden features and representations), it prevents features from representing the full information of the data; when dimensional collapse occurs on weight matrices, their filters are self-related and redundant, limiting their expressive power.
Existing studies have predominantly concentrated on the dimensional collapse of representations, neglecting whether this can sufficiently prevent the dimensional collapse of the weight matrices and hidden features. 
To this end, we first time propose a mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matrices, thus safeguarding against the dimensional collapse of weight matrices, hidden features, and representations. Our empirical investigations demonstrate that OR significantly enhances the performance of SSL methods across diverse benchmarks, yielding consistent gains with both CNNs and Transformer-based architectures.

Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization

Abstract

Self-supervised learning (SSL) has rapidly advanced in recent years, approaching the performance of its supervised counterparts through the extraction of representations from unlabeled data. However, dimensional collapse, where a few large eigenvalues dominate the eigenspace, poses a significant obstacle for SSL. When dimensional collapse occurs on features (e.g. hidden features and representations), it prevents features from representing the full information of the data; when dimensional collapse occurs on weight matrices, their filters are self-related and redundant, limiting their expressive power. Existing studies have predominantly concentrated on the dimensional collapse of representations, neglecting whether this can sufficiently prevent the dimensional collapse of the weight matrices and hidden features. To this end, we first time propose a mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matrices, thus safeguarding against the dimensional collapse of weight matrices, hidden features, and representations. Our empirical investigations demonstrate that OR significantly enhances the performance of SSL methods across diverse benchmarks, yielding consistent gains with both CNNs and Transformer-based architectures. Our code will be released at https://github.com/Umaruchain/OR_in_SSL.git.

1 Introduction

Self-supervised learning (SSL) has established itself as an indispensable paradigm in machine learning, motivated by the expensive costs of human annotation and the abundant quantities of unlabeled data. SSL endeavors to produce meaningful representations without the guidance of labels. Recent developments have witnessed joint-embedding SSL methods achieving, or even exceeding the supervised counterparts (Misra & Maaten 2020, Bardes et al. 2022, Caron et al. 2020, Chen, Fan, Girshick & He 2020, Chen, Kornblith, Norouzi & Hinton 2020, Chen & He 2021, Dwibedi et al. 2021, HaoChen et al. 2021, He et al. 2020, He & Ozay 2022, Jing et al. 2021, Li, Zhou, Xiong & Hoi 2020, Jing et al. 2020, Balestriero et al. 2023, Grill et al. 2020, Zbontar et al. 2021, Chen et al. 2021). The efficacy of these methods hinges on two pivotal principles: 1) the ability to learn augmentation-invariant representations, and 2) the prevention of complete collapse, where all inputs are encoded to a constant vector.

Efforts to forestall complete collapse have been diverse, including contrastive methods with both positive and negative pairs (He et al. 2020, Chen, Kornblith, Norouzi & Hinton 2020, Chen et al. 2021) and non-contrastive methods utilizing techniques such as self-distillation (Caron et al. 2021, Grill et al. 2020, Chen & He 2021), clustering (Caron et al. 2018, 2020, Pang et al. 2022) and feature whitening (Bardes et al. 2022, Zbontar et al. 2021, Weng et al. 2022, 2023). Notwithstanding, these methods are prone to dimensional collapse, a phenomenon where a few large eigenvalues dominate the eigenspace . Dimensional collapse can occur on both features (e.g. hidden features and representations) and weight matrices.

Encoder

Augmented

Input

Block1

Block2

Representations

W

Hidden

Haug1

features

Representations

Weight matrices

Existing

Orthogonality

Zaug1

methods

Regularization

W1

Hiddenfeatures

Hiddenfeatures

Hidden

Hidden

features

features

....Uncontrolled

Guaranteed

Weightmatrices

by theory

Representations

W

Zaugi

To prevent dimensional collapse of representations, as depicted in Figure 1, existing methods include modifying representations in downstream tasks (He & Ozay 2022), whitening representations directly (i.e. removing the projector) (Jing et al. 2021), incorporating regularizers on representations during pretraining (Huang et al. 2024, Hua et al. 2021). However, whether they sufficiently prevent the dimensional collapse of weight matrices and hidden features remains unknown (i.e., no theoretical guarantee) (Pasand et al. 2024). In Appendix A.1, we further demonstrate that whitening representations directly to eliminate the dimensional collapse of representations cannot adequately remove the dimensional collapse of weight matrices.

To address these challenges, we first time propose a mitigation approach employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. It is natural that OR prevents the dimensional collapse of weight matrices as it ensures weight matrices orthogonality, keeps the correlation between its filters as low as possible, and lets each filter have a norm of 1. For features (e.g. hidden features and representations), orthogonal weight matrices can promote uniform eigenvalue distributions and thus prevent the domination of eigenspaces by a

limited number of large eigenvalues, as theoretically substantiated by Huang et al. (2018), Yoshida & Miyato (2017), Rodríguez et al. (2016).

In our study, we introduce and assess the effects of two leading orthogonality regularizers, Soft Orthogonality (SO) and Spectral Restricted Isometry Property Regularization (SRIP), on SSL methods. We examine their integration with 13 modern SSL methods from Solo-learn and LightSSL, spanning both contrastive and non-contrastive methods (Chen, Fan, Girshick & He 2020, Chen et al. 2021, Grill et al. 2020, Caron et al. 2021, Dwibedi et al. 2021). Our findings indicate a consistent enhancement in linear probe accuracy on CIFAR-100 using both CNNs and Transformer-based architectures and OR exhibits a good scaling law at the model scale. Furthermore, when applied to BYOL trained on IMAGENET-1k, OR significantly improves the downstream performance on both classification and object detection tasks, suggesting its applicability to large-scale SSL settings. Remarkably, OR achieves these enhancements without necessitating modifications to existing SSL architectures or hyperparameters.

In summary, we present three major contributions:

We systematically study the phenomenon of dimensional collapse in SSL, including how feature whitening and network depth affect the dimensional collapse of weight matrices and hidden features.

We first time introduce orthogonal regularization (OR) as a solution to prevent the dimensional collapse of weight matrices, hidden features, and representations during SSL pretraining.

Our extensive experimental analysis demonstrates OR's substantial role in enhancing the performance of state-of-the-art joint-embedding SSL methods with a wide spectrum of backbones.

2 Related Work

2.1 Self-Supervised Learning

Self-supervised Learning (SSL) aims to learn meaningful representations from unlabeled data. Existing SSL methods can be broadly classified into two categories: generative and joint embedding methods. This paper concentrates on joint-embedding methods, which learn representations by aligning the embeddings of different augmented views of the same instance. Joint-embedding methods further subdivide into contrastive and non-contrastive methods. Contrastive methods, such as those proposed by He et al. (2020), Chen, Kornblith, Norouzi & Hinton (2020), Chen et al. (2021), treat each sample as a distinct class and leverage the InfoNCE loss (Oord et al. 2018) to bring representations of positive pairs closer together while distancing those of negative pairs in the feature space. These methods generally require a substantial number of negative samples for effective learning. In contrast, non-contrastive methods eschew the use of negative samples. They instead employ various techniques such as self-distillation (Caron et al. 2021, Grill et al. 2020, Chen & He 2021), clustering (Caron et al. 2018, 2020, Pang et al. 2022) and feature whitening (Bardes et al. 2022, Zbontar et al. 2021, Weng et al. 2022, 2023). Our empirical findings indicate that incorporating OR enhances the performance of both contrastive and non-contrastive SSL methods. The exploration of its effects on generative methods remains for future work.

2.2 Dimensional Collapse in SSL

Dimensional collapse plagues both generative and joint embedding SSL methods (Zhang et al. 2022, Jing et al. 2021, Zhang et al. 2021, Tian et al. 2021). To prevent the dimensional collapse of representations, existing work has typically focused on imposing constraints on the covariance matrix of the representations, including modifying representations in downstream tasks (He & Ozay 2022), removing the projector (Jing et al. 2021), incorporating regularizers on representations during pretraining (Huang et al. 2024, Hua et al. 2021). However, these strategies face challenges such as performance degradation upon removing the projector, not addressing collapse during pre-training, and failing to prevent dimensional collapse in hidden features and weight matrices within the encoder (referred to Appendix A.1). This motivates us to regularize the weight matrices of the DNNs directly in SSL.

2.3 Orthogonality Regularization

Orthonormality regularization, which is applied in linear transformations, can improve the generalization and training stability of DNNs (Xie et al. 2017, Huang et al. 2018, Saxe et al. 2013). OR has demonstrated its effects on tasks including supervised/semi-supervised image classification, image retrieval, unsupervised inpainting, image generation, and adversarial training (Bansal et al. 2018, Balestriero et al. 2018, Balestriero & Baraniuk 2020, Xie et al. 2017, Huang et al. 2018). Efforts to utilize orthogonality in network training have included penalizing the deviation of the gram matrix of each weight matrice from the identity matrix (Xie et al. 2017, Bansal et al. 2018, Balestriero et al. 2018, Kim & Yun 2022) and employing orthogonal initialization (Xie et al. 2017, Saxe et al. 2013). For more stringent norm preservation, some studies transform the convolutional layer into a doubly block-Toeplitz (DBT) matrix and enforce orthogonality (Qi et al. 2020, Wang et al. 2020).

In this work, we first time investigate the efficacy of two orthogonality regularizers, Soft Orthogonality (SO) and Spectral Restricted Isometry Property (SRIP) in SSL (Bansal et al. 2018). These regularizers aim to minimize the distance between the gram matrix of each weight matrix and the identity matrix-measured in Frobenius and spectral norms, respectively.

3 Preliminaries

3.1 Settings of Self-supervised Learning

In this section, we present the general settings for joint-embedding SSL methods. We consider a large unlabelled dataset X ∈ R N × D , comprising N samples each of dimensionality D . The objective of SSL methods is to construct an effective encoder f that transforms raw data into meaningful representations Z = f ( X ) , where Z ∈ R N × M and M denotes the representation dimensionality. The learning process of SSL methods is visually represented in Figure 2, where data augmentations transform X into two augmented views X aug1 , X aug2 ∈ R D × N . A typical joint-embedding SSL architecture encompasses an encoder f and a projector p . These components yield encoder features Z aug1 = f ( X aug1 ) and Z aug2 = f ( X aug2 ) , as well as projection features H aug1 = p ( Z aug1 ) and H aug2 = p ( Z aug2 ) . During training, the parameters of f and p are optimized via backpropagation to minimize the discrepancy between H aug1 and H aug2. To prevent the encoder f from producing a constant feature vector, contrastive methods utilize negative samples, and non-contrastive methods employ strategies such as the self-distillation technique.

The efficacy of the encoder f is usually assessed by the performance of the C -class classification as downstream tasks. Specifically, given a labeled dataset containing samples X s ∈ R S × D and their corresponding labels Y s ∈ R S × C , where S is the sample number. Then, a linear layer g parameterized by W c ∈ R C × M is appended on top of the learned representations Z s = f ( X s ) , and thus the classification task can be fulfilled by minimizing the cross-entropy between softmax ( g ( Z s )) and Y s . There are two strategies for the fine-tuning: 1) non-linear fine-tuning, which trains both g and f in the downstream tasks, and 2) linear evaluation, which freezes f and only trains g (referred to as the linear probe).

Augmented

Projector

Inputs

Encoder

Representations

Projector

features

f

Zaugi

p

Haug1

Inputs

X

Augmentation

SSLObjective

Xaug2

Zaug2

Haug2

f

p

Augmented

Encoder

Representations

Projector

Projector

Inputs

features

3.2 Orthogonality Regularizers

We introduce two orthogonality regularizers: Soft Orthogonality (SO) and Spectral Restricted Isometry Property Regularization (SRIP), which are seamlessly integrable with linear and convolutional layers.

Consider a weight matrix W ∈ R input × output in a linear layer, where input and output denote the number of input and output features, respectively. In line with Bansal et al. (2018), Xie et al. (2017), Huang et al. (2018), we reshape the convolutional filter to a two-dimensional weight matrix W ∈ R input × output , while we still use the same notation W for consistency. To be specific, input = S × H × C in and output = C out , with C in and C out being the number of input and output channels, and S and H representing the width and height of the filter, respectively.

The SO regularizer encourages the weight matrix W to approximate orthogonality by minimizing the distance between its Gram matrix and the identity matrix. This is quantified by the Frobenius norm as follows:

where I is the identity matrix of appropriate size.

The SRIP regularizer employs the spectral norm to measure the deviation from orthogonality, which is defined as:

where σ ( · ) denotes the spectral norm operator. Due to the high computational cost posed by the spectral norm, the power iteration method (Yoshida & Miyato 2017, Bansal et al. 2018) with two iterations is used for the estimation. The process for estimating σ ( W T W -I ) is:

where v ∈ R input is a vector initialized randomly from a normal distribution.

4 Incorporating OR into SSL

This section details the integration of OR with SSL methods. To be specific, we employ OR across the encoder, targeting both convolutional and linear layers during pretraining. We represent the SSL method's loss function as Loss SSL . Our overall optimization objective is the minimization of the combined loss equation:

where Loss OR is defined as ∑ W ∈ f SO ( W ) or ∑ W ∈ f SRIP ( W ) , depending on the selected orthogonality regularizers. The term γ serves as a hyperparameter that balances the SSL objective and OR loss. Notably, we only perform OR on the weight matrices located within the linear and convolutional layers of the encoder f . OR provides a versatile regularization strategy for the encoder f , facilitating its application across various SSL methods without necessitating modifications to the network designs or existing training protocols.

5 Analysis of Dimensional Collapse and the Effects of OR in SSL

In this section, we show that dimensional collapse happens not only to the representations (i.e. output of the encoder), but also to weight matrices and hidden features of the encoder. We also compare the feature whitening technique used by one previous method, VICREG (Bardes et al. 2022), with OR. Migrating to BYOL, we find that the feature whitening technique only solves the dimensional collapse at the feature level, but instead accelerates the collapse of the weight matrices, and it even leads to lower performance of the downstream tasks as shown in Table 1. In contrast, OR can

eliminate the dimensional collapse of weight matrices and thus the dimensional collapse of hidden features and representations.

In Appendix A.1, we further reveal that the original VICREG has a dimensional collapse problem with its weight matrices, which could not be solved by removing its projector. Adding OR to VICREG eliminates this problem and boosts the performance.

To study the eigenspace of a matrix, we utilize the normalized eigenvalues defined in He & Ozay (2022):

Definition 1 (Normalized eigenvalues) Given a specific matrix T ∈ R N × D , where N is the sample number and D is the feature dimension, we first obtain its covariance matrix Σ T ∈ R D × D . Then we perform an eigenvalue decomposition on Σ T to obtain its eigenvalues { λ T i } D i =1 = { λ T 1 , · · · , λ T i , · · · , λ T D } in descending order. And we obtain the normalized eigenvalues by dividing all eigenvalues by the max eigenvalue λ T 1 , denoted as { λ T 1 /λ T 1 , · · · , λ T i /λ T 1 , · · · , λ T D /λ T 1 } . To simplify the denotation, we reuse { λ T i } D i =1 to denote normalized eigenvalues of T .

These normalized eigenvalues are less than or equal to 1, where a larger value in one dimension indicates more information contained, and vice versa. We argue that if normalized eigenvalues drop very quickly, this means that only a few dimensions in the eigenspace contain meaningful information and also means that dimensional collapse has occurred.

We train three BYOL (Grill et al. 2020) models, without OR, with the feature whitening technique (e.g. Variance and Covariance regularization) from VICREG and with OR, respectively. We choose randomly initialized ResNet18 as the backbone (i.e. encoder) and train the three models on CIFAR-10 for 1,000 epochs, following the same recipe of Da Costa et al. (2022). Importantly, SO is selected as the orthogonality regularize, and γ is set to 1 e -6 . For the feature whitening technique, we impose the Variance and Covariance regularization from VICREG on the output of the predictor in BYOL as two additional loss terms, the former to ensure the informativeness of individual dimensions and the latter to reduce the correlation between dimensions. Following the solo-learn settings, we set the two loss term hyperparameters to γ vic and γ vic ∗ 0 . 004 , and then tune the γ vic from 1 e -3 to 1 e -5 .

After training, we calculate the normalized eigenvalues of both weight matrices and features (e.g. input features, hidden features, representations). ResNet18 contains four basic blocks, each containing four convolutional layers, and we visualize the normalized eigenvalues of the last convolutional layer in each block. Hidden features are the outputs of four basic blocks in ResNet18. We use the first batch in the test set as input features with batchsize 4,096 and feature dimension 3072 (32*32*3). Results are analyzed in the following sections.

5.1 Dimensional Collapse of Weight Matrices

We first examine the weight matrices of the encoder. Similar to 5.1, all the weight matrices are viewed as two-dimensional matrices, and on top of them, we can calculate their normalized eigenvalues. For a specific weight matrix W ∈ R input × output in a neural network layer, we denote { λ i W } output i = i as the normalized eigenvalues of this layer.

As shown in Figure 3, X and Y axis is the i index and i -th values of { λ i W } output i = i , respectively. It is clear that with OR, the eigenvalues of the convolutional layers of different depths decay more slowly, which means that their filters are less redundant and more diverse. In particular, we note that the deepest convolutional layer (i.e. layer4_512) has a much faster decay rate of the eigenvalues compared to the other convolutional layers in the absence of OR. Notably, the feature whitening technique does not alleviate this phenomenon. OR could significantly improve this. OR requires the weight matrix to be as orthogonal as possible, which means that the diagonal elements of its covariance matrix are as identical as possible, and the off-diagonal elements will be close to 0. Then

(a) Eigenvalues of weights (without OR)

(b) Eigenvalues of features (without OR)

(c) Eigenvalues of weights (with feature whitening)

(d) Eigenvalues of features (with feature whitening)

(e) Eigenvalues of weights (with OR)

(f) Eigenvalues of features (with OR)

cifar10_image_3072

eigenvalue

eigenvalue

representations_512

hidden_feats(layer1)_73984

hidden_feats(layer2)_36992

0.1

0.1

hidden_feats(layer3)_20736

hidden_feats(layer4)_12800

Normalised

Normalised

y=1/x

0.01

layer1_64

0.01

layer2_128

layer3_256

layer4_512

y=1/x

10

100

512

10

100

1000

73984

Eigenvaluerank

Eigenvaluerank

cifar10_image_3072

Normalised eigenvalue

eigenvalue

representations_512

hidden_feats(layer1)_73984

hidden_feats(layer2)_36992

0.1

0.1

hidden_feats(layer3)_20736

hidden_feats(layer4)_12800

Normalised

y=1/x

0.01

layer1_64

0.01

layer2_128

layer3_256

layer4_512

y=1/x

0.001

1

10

100

512

10

100

1000

73984

Eigenvaluerank

Eigenvaluerank

cifar10_image_3072

Normalised eigenvalue

Ieigenvalue

representations_512

hidden_feats(layer1)_73984

hidden_feats(layer2)_36992

0.1

0.1

hidden_feats(layer3)_20736

hidden_feats(layer4)_12800

Normalised

y=1/x

0.01

layer1_64

0.01

layer2_128

layer3_256

layer4_512

y=1/x

10

100

512

10

100

1000

73984

Eigenvaluerank

Eigenvaluerank

the eigenvalue decomposition of such a covariance matrix will have elements in { λ i W } output i = i that are all close to 1 (verified in Appendix A.2).

5.2 Dimensional Collapse of Features

As the weight matrices become orthogonal, the distribution of their outputs stabilizes, thereby preventing the dimensional collapse of hidden features and representations. This property has been demonstrated in vector form by Huang et al. (2018) and is now presented in matrix form:

Proposition 1 For a specific weight matrix W ∈ R input × output and X ∈ R N × input , comprising N samples each of dimensionality input . We denote ¯ X and ¯ S as the sample means of X and S , respectively. Let S = XW , where W T W = I . The covariance matrix of X is Σ X = ( X -¯ X ) T · ( X -¯ X ) N -1 . (1) If ¯ X = 0 and Σ X = σ 2 I , then ¯ S = 0 and Σ S = σ 2 I . (2) If input = output , we have ∥ S ∥ 2 F = ∥ X ∥ 2 F . (3) Given the back-propagated gradient ∂L ∂S , we have ∥ ∥ ∂L ∂S ∥ ∥ 2 F = ∥ ∥ ∂L ∂X ∥ ∥ 2 F .

The first point of Proposition 1 illustrates that in each layer of DNNs, the orthogonal weight matrix preserves the normalization and de-correlation of the output S , assuming the input is whitened. This reveals that as the network gets deeper, the hidden features of each layer and the final representations do not tend to collapse. Moreover, orthogonal filters maintain the norm of both the output and the back-propagated gradient information in DNNs, as demonstrated by the second and third points of Proposition 1.

To verify that the dimensional collapse of features can be eliminated by OR, we visualize the normalized eigenvalues of features (input features, hidden features, and representations) as shown in Figure 3. Without the OR constraint, features located in deeper layers (i.e. representations) will have the fastest eigenvalues decay rate, and the distributions of eigenvalues of hidden features vary considerably at different depths. After adding the OR, it can be seen that the decay rates of hidden features at layers 3 and 4 are almost the same, while the eigenvalues of representations decay much more slowly. We also visualize the representations in Appendix A.3, which also verifies that the dimensional collapse of the representations is mitigated. Interestingly, the effect of OR on the feature level is similar to that of the feature whitening technique, however, the latter is unable to eliminate the dimensional collapse in the weight matrices.

6 Numerical Experiments

We study the effects of OR on SSL methods through extensive experiments. we first demonstrate that OR improves the classification accuracy on CIFAR-10, CIFAR-100, and IMAGENET100, and the improvement is consistent across different backbones and SSL methods. On the large-scale dataset IMAGENET-1k (Deng et al. 2009), OR boosts the classification accuracy on both in-distribution and out-distribution datasets (i.e. transfer learning datasets), demonstrating consistent improvement. Moreover, OR also enhances the performance in downstream tasks(e.g. object detection).

Baseline methods and datasets. We evaluated the effect of adding OR to 13 modern SSL methods, including 6 methods implemented by solo-learn (MOCOv2plus, MOCOv3, DINO, NNBYOL, BYOL, VICREG) (Chen & He 2021, Chen et al. 2021, Grill et al. 2020, Dwibedi et al. 2021, Caron et al. 2021) and 10 methods implemented by LightlySSL (BarlowTwins, BYOL, DCL, DCLW, DINO, Moco, NNCLR, SimCLR, SimSiam, SwaV) (Zbontar et al. 2021, Yeh et al. 2022, Caron et al. 2020, Chen & He 2021). We pretrain SSL methods on CIFAR-10, CIFAR-100, IMAGENET-100 and IMAGENET-1k and evaluate transfer learning scenarios on datasets including CIFAR-100, CIFA-10 (Krizhevsky et al. 2009), Food-101 (Bossard et al. 2014), Flowers-102 (Xia et al. 2017), DTD(Sharan et al. 2014), GTSRB (Haloi 2015). We evaluate the objection detection task on PASCAL VOC2007 and VOC2012 (Everingham et al. 2010). Detailed descriptions of datasets and baseline SSL methods are shown in Appendix A.4 and A.5, respectively.

Training and evaluation settings. For each SSL method, we use the original settings in sololearn (Da Costa et al. 2022) and LightlySSL. These settings include the network structure, loss function, training policy (training epochs, optimizers, and learning rate schedulers) and data augmentation policy. The splits of the training and test set follow torchvision Marcel & Rodriguez (2010). For all the classification tasks, we report the linear probe or KNN accuracy; for the objection detection task, we perform non-linear fine-tuning. Details of training, parameter tuning, and evaluation are presented in Appendix A.6. It is worth noting that the Solo-learn and LightlySSL setups are not the same as the official implementation of the SSL methods, e.g., there is no use of multi-crop augmentation in DINO, and there is no exceptionally long training epoch. We leave experiments on migrating OR to the official implementation for future work.

Recipe of adding OR. For OR, γ of SRIP is tuned from { 1 e -3 , 1 e -4 , 1 e -5 } and γ of SO is tuned from { 1 e -5 , 1 e -6 , 1 e -7 } on a validation set. When you want to add OR to your SSL pre-training, you simply pass the encoder into the loss function, and then you just need to set γ of the OR according to the backbone and regularizer you use as shown in Table 9 of Appendix A.6.

6.1 OR is Suitable for Different Backbones and SSL Methods

After pretraining on CIFRA-100, for each SSL method, we report the corresponding classification accuracy as shown in Table 2. Both two orthogonality regularizers consistently improve the linear classification accuracy. Note that OR boosts the performance of both constrastive (MoCov2plus,

Mocov3) and non-contrastive methods(BYOL, NNBYOL, DINO) as they are all susceptible to dimensional collapse (Zhang et al. 2022, Jing et al. 2021, Zhang et al. 2021, Tian et al. 2021). Non-contrastive methods gain more improvements in contrast to contrastive methods. When we use ResNet18, MOCOv3 improves 3% on Top-1 accuracy while DINO and NNBYOL improve 6% and 5% , respectively. OR can also boost the performance of the Sota method like BYOL by 1% . When we scale to ResNet 50 and WideResnet28w2, OR consistently boosts their performance. Moreover, the additional time overhead of adding OR to SSL is low compared to the original training time (referred to A.7).

In addition to CNN backbones, OR is also able to improve SSL performance on Transformer-based backbones (e.g., VIT) (Han et al. 2022, Dosovitskiy et al. 2020, Zhou et al. 2021). We pretrain DINO on CIFAR-100 with different depths of VITs. As shown in Table 3, with the increasing depth of the VIT, the original DINO performance is increasing, and OR is able to increase their performance even further, which exhibits a good scaling law. Interestingly, under the Transformer-based architecture, ORis able to improve performance more (up to 12% ) compared to CNN backbones. This is consistent with some existing studies (RoyChowdhury et al. 2017) that linear layers are more likely to have redundant filters than convolutional layers in DNNs, i.e., more prone to dimensional collapse.

To evaluate OR on more SSL methods, under the LightSSL framework, we test SO on CIFAR-10, IMAGENET-100, and IMAGENET-1k. OR still consistently improves the performance of various SSL methods as shown in Table 4.

6.2 OR Works on Large-scale Dataset

Wedemonstrate the effects of OR on the large-scale dataset IMAGENET-1k. Specifically, we pre-train three BYOL models- BYOL without OR, BYOL with SO, and BYOL with SRIP on IMAGENET-1k

(with ResNet50). For each testing classification dataset, we report the accuracy as shown in Table 5 and 6.

We can observe that OR not only improves the accuracy on IMAGENET-1k but also on all the transfer learning datasets. OR improves TOP-1 accuracy by 3% in IMAGENET-1k and by 3% to 9% in each transfer learning datasets. The transfer learning task evaluates the generality of the encoder as it has to encode samples from various out-of-distribution domains with categories that it may not have seen during pretraining. OR also significantly improves SSL's performance in the objection detection task by 20% on AP. The above results are close to Chen et al. (2021), Da Costa et al. (2022), Weng et al. (2023) where also training only 100 epochs.

Considering that training epoch and batchsize during pretraining significantly impact the performance Chen et al. (2021), Huang et al. (2024), Lavoie et al. (2022), we further increase the training epoch(200) and batchsize(256) and scale up the learning rate accordingly. As shown in Table 7, OR consistently improves the performance.

7 Conclusions

The existing studies focus on the dimensional collapse of representations and overlook whether weight matrices and hidden features also undergo dimensional collapse. We first time propose a mitigation approach to employing orthogonal regularization (OR) across the encoder, targeting both convolutional and linear layers during pretraining. OR promotes orthogonality within weight matrices, thus safeguarding against the dimensional collapse of weights, hidden features, and representations. Our empirical investigations demonstrate that OR significantly enhances SSL method performance across diverse benchmarks, yielding consistent gains with both CNNs and Transformer-based architectures as the backbones. Importantly, the time complexity and required efforts on fine-tuning are low and the performance improvement is significant, enabling it to become a useful plug-in in various SSL methods.

In terms of future research, we wish to examine the effect of OR on other pre-training foundation models, such as vision generative SSL models such as MAE (He et al. 2022), auto-regression models like GPTs and LLaMAs (Radford et al. 2018, 2019, Brown et al. 2020, Touvron et al. 2023), and Contrastive Language-Image Pre-training models (Radford et al. 2021, Li et al. 2022). We believe OR is a pluggable and useful module to boost the performance of vision and language foundation models. In fact, this paper is the first to test the effectiveness of OR in a Transformer-based architecture and it is reasonable to believe that it will perform well in these domains.

Acknowledgments

The work described in this paper was supported by the National Natural Science Foundation of China (No. 52102385), grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU/25209221 and PolyU/15206322), and grants from the Otto Poon Charitable Foundation Smart Cities Research Institute (SCRI) at the Hong Kong Polytechnic University (Project No. P0043552). The contents of this article reflect the views of the authors, who are responsible for the facts and accuracy of the information presented herein.

References

Balestriero, R. & Baraniuk, R. G. (2020), 'Mad max: Affine spline insights into deep learning', Proceedings of the IEEE 109 (5), 704-727.

Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y. et al. (2023), 'A cookbook of self-supervised learning', arXiv preprint arXiv:2304.12210 .

Balestriero, R. et al. (2018), A spline theory of deep learning, in 'International Conference on Machine Learning', PMLR, pp. 374-383.

Bansal, N., Chen, X. & Wang, Z. (2018), 'Can we gain more from orthogonality regularizations in training deep networks?', Advances in Neural Information Processing Systems 31 .

Bardes, A., Ponce, J. & LeCun, Y. (2022), 'Variance-invariance-covariance regularization for selfsupervised learning', ICLR, Vicreg 1 , 2.

Bossard, L., Guillaumin, M. & Van Gool, L. (2014), Food-101-mining discriminative components with random forests, in 'Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13', Springer, pp. 446-461.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020), 'Language models are few-shot learners', Advances in neural information processing systems 33 , 1877-1901.

Caron, M., Bojanowski, P., Joulin, A. & Douze, M. (2018), Deep clustering for unsupervised learning of visual features, in 'Proceedings of the European conference on computer vision (ECCV)', pp. 132-149.

Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P. & Joulin, A. (2020), 'Unsupervised learning of visual features by contrasting cluster assignments', Advances in neural information processing systems 33 , 9912-9924.

Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P. & Joulin, A. (2021), Emerging properties in self-supervised vision transformers, in 'Proceedings of the IEEE/CVF international conference on computer vision', pp. 9650-9660.

Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. (2020), A simple framework for contrastive learning of visual representations, in 'International conference on machine learning', PMLR, pp. 1597-1607.

Chen, X., Fan, H., Girshick, R. & He, K. (2020), 'Improved baselines with momentum contrastive learning', arXiv preprint arXiv:2003.04297 .

Chen, X. & He, K. (2021), Exploring simple siamese representation learning, in 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 15750-15758.

Chen, X., Xie, S. & He, K. (2021), An empirical study of training self-supervised vision transformers, in 'Proceedings of the IEEE/CVF international conference on computer vision', pp. 9640-9649.

Da Costa, V. G. T., Fini, E., Nabi, M., Sebe, N. & Ricci, E. (2022), 'solo-learn: A library of selfsupervised methods for visual representation learning', Journal of Machine Learning Research 23 (56), 1-6.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K. & Fei-Fei, L. (2009), Imagenet: A large-scale hierarchical image database, in '2009 IEEE conference on computer vision and pattern recognition', Ieee, pp. 248-255.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. et al. (2020), 'An image is worth 16x16 words: Transformers for image recognition at scale', arXiv preprint arXiv:2010.11929 .

Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P. & Zisserman, A. (2021), With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, in 'Proceedings of the IEEE/CVF International Conference on Computer Vision', pp. 9588-9597.

Everingham, M., Van Gool, L., Williams, C. K., Winn, J. & Zisserman, A. (2010), 'The pascal visual object classes (voc) challenge', International journal of computer vision 88 , 303-338.

Girshick, R., Donahue, J., Darrell, T. & Malik, J. (2014), Rich feature hierarchies for accurate object detection and semantic segmentation, in 'Proceedings of the IEEE conference on computer vision and pattern recognition', pp. 580-587.

Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M. et al. (2020), 'Bootstrap your own latent-a new approach to self-supervised learning', Advances in neural information processing systems 33 , 21271-21284.

Haloi, M. (2015), 'Traffic sign classification using deep inception based convolutional networks', arXiv preprint arXiv:1511.02992 .

Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y. et al. (2022), 'A survey on vision transformer', IEEE transactions on pattern analysis and machine intelligence 45 (1), 87-110.

HaoChen, J. Z., Wei, C., Gaidon, A. & Ma, T. (2021), 'Provable guarantees for self-supervised deep learning with spectral contrastive loss', Advances in Neural Information Processing Systems 34 , 5000-5011.

He, B. & Ozay, M. (2022), Exploring the gap between collapsed & whitened features in selfsupervised learning, in 'International Conference on Machine Learning', PMLR, pp. 8613-8634.

He, K., Chen, X., Xie, S., Li, Y ., Dollár, P. & Girshick, R. (2022), Masked autoencoders are scalable vision learners, in 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 16000-16009.

He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. (2020), Momentum contrast for unsupervised visual representation learning, in 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 9729-9738.

Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y. & Zhao, H. (2021), On feature decorrelation in self-supervised learning, in 'Proceedings of the IEEE/CVF International Conference on Computer Vision', pp. 9598-9608.

Huang, H., Campello, R. J., Erfani, S. M., Ma, X., Houle, M. E. & Bailey, J. (2024), 'Ldreg: Local dimensionality regularized self-supervised learning', arXiv preprint arXiv:2401.10474 .

Huang, L., Liu, X., Lang, B., Yu, A., Wang, Y. & Li, B. (2018), Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, in 'Proceedings of the AAAI Conference on Artificial Intelligence', Vol. 32.

Jing, L., Vincent, P., LeCun, Y . & Tian, Y . (2021), 'Understanding dimensional collapse in contrastive self-supervised learning', arXiv preprint arXiv:2110.09348 .

Jing, L., Zbontar, J. et al. (2020), 'Implicit rank-minimizing autoencoder', Advances in Neural Information Processing Systems 33 , 14736-14746.

Kim, T. & Yun, S.-Y. (2022), 'Revisiting orthogonality regularization: a study for convolutional neural networks in image classification', IEEE Access 10 , 69741-69749.

Krizhevsky, A., Hinton, G. et al. (2009), 'Learning multiple layers of features from tiny images'.

Lavoie, S., Tsirigotis, C., Schwarzer, M., Vani, A., Noukhovitch, M., Kawaguchi, K. & Courville, A. (2022), 'Simplicial embeddings in self-supervised learning and downstream classification', arXiv preprint arXiv:2204.00616 .

Li, J., Li, D., Xiong, C. & Hoi, S. (2022), Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, in 'International conference on machine learning', PMLR, pp. 12888-12900.

Li, J., Zhou, P., Xiong, C. & Hoi, S. C. (2020), 'Prototypical contrastive learning of unsupervised representations', arXiv preprint arXiv:2005.04966 .

Li, X., Chen, S. & Yang, J. (2020), Understanding the disharmony between weight normalization family and weight decay, in 'Proceedings of the AAAI Conference on Artificial Intelligence', Vol. 34, pp. 4715-4722.

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P. & Zitnick, C. L. (2014), Microsoft coco: Common objects in context, in 'Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13', Springer, pp. 740-755.

Marcel, S. & Rodriguez, Y. (2010), Torchvision the machine-vision package of torch, in 'Proceedings of the 18th ACM international conference on Multimedia', pp. 1485-1488.

McInnes, L., Healy, J. & Melville, J. (2018), 'Umap: Uniform manifold approximation and projection for dimension reduction', arXiv preprint arXiv:1802.03426 .

Misra, I. & Maaten, L. v. d. (2020), Self-supervised learning of pretext-invariant representations, in 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 67076717.

Oord, A. v. d., Li, Y. & Vinyals, O. (2018), 'Representation learning with contrastive predictive coding', arXiv preprint arXiv:1807.03748 .

Pang, B., Zhang, Y., Li, Y., Cai, J. & Lu, C. (2022), Unsupervised visual representation learning by synchronous momentum grouping, in 'European Conference on Computer Vision', Springer, pp. 265-282.

Pasand, A. S., Moravej, R., Biparva, M. & Ghodsi, A. (2024), 'Werank: Towards rank degradation prevention for self-supervised learning using weight regularization', arXiv preprint arXiv:2402.09586 .

Qi, H., You, C., Wang, X., Ma, Y. & Malik, J. (2020), Deep isometric learning for visual recognition, in 'International conference on machine learning', PMLR, pp. 7824-7835.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. et al. (2021), 'Clip: Learning transferable visual models from natural language supervision', arXiv preprint arXiv:2103.00020 .

Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. et al. (2018), 'Improving language understanding by generative pre-training'.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. et al. (2019), 'Language models are unsupervised multitask learners', OpenAI blog 1 (8), 9.

Rodríguez, P., Gonzalez, J., Cucurull, G., Gonfaus, J. M. & Roca, X. (2016), 'Regularizing cnns with locally constrained decorrelations', arXiv preprint arXiv:1611.01967 .

RoyChowdhury, A., Sharma, P., Learned-Miller, E. & Roy, A. (2017), Reducing duplicate filters in deep neural networks, in 'NIPS workshop on deep learning: Bridging theory and practice', Vol. 1, p. 6.

Saxe, A. M., McClelland, J. L. & Ganguli, S. (2013), 'Exact solutions to the nonlinear dynamics of learning in deep linear neural networks', arXiv preprint arXiv:1312.6120 .

Sharan, L., Rosenholtz, R. & Adelson, E. H. (2014), 'Accuracy and speed of material categorization in real-world images', Journal of vision 14 (9), 12-12.

Tian, Y., Chen, X. & Ganguli, S. (2021), Understanding self-supervised learning dynamics without contrastive pairs, in 'International Conference on Machine Learning', PMLR, pp. 10268-10278.

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F. et al. (2023), 'Llama: Open and efficient foundation language models', arXiv preprint arXiv:2302.13971 .

Wang, J., Chen, Y., Chakraborty, R. & Yu, S. X. (2020), Orthogonal convolutional neural networks, in 'Proceedings of the IEEE/CVF conference on computer vision and pattern recognition', pp. 1150511515.

Weng, X., Huang, L., Zhao, L., Anwer, R., Khan, S. H. & Shahbaz Khan, F. (2022), 'An investigation into whitening loss for self-supervised learning', Advances in Neural Information Processing Systems 35 , 29748-29760.

Weng, X., Ni, Y., Song, T., Luo, J., Anwer, R. M., Khan, S., Khan, F. S. & Huang, L. (2023), 'Modulate your spectrum in self-supervised learning', arXiv preprint arXiv:2305.16789 .

Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y. & Girshick, R. (2019), 'Detectron2', https://github. com/facebookresearch/detectron2 .

Xia, X., Xu, C. & Nan, B. (2017), Inception-v3 for flower classification, in '2017 2nd international conference on image, vision and computing (ICIVC)', IEEE, pp. 783-787.

Xie, D., Xiong, J. & Pu, S. (2017), All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation, in 'Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition', pp. 61766185.

Yeh, C.-H., Hong, C.-Y., Hsu, Y.-C., Liu, T.-L., Chen, Y. & LeCun, Y. (2022), Decoupled contrastive learning, in 'European conference on computer vision', Springer, pp. 668-684.

Yoshida, Y. & Miyato, T. (2017), 'Spectral norm regularization for improving the generalizability of deep learning', arXiv preprint arXiv:1705.10941 .

Zbontar, J., Jing, L., Misra, I., LeCun, Y . & Deny, S. (2021), Barlow twins: Self-supervised learning via redundancy reduction, in 'International conference on machine learning', PMLR, pp. 1231012320.

Zhang, Q., Wang, Y. & Wang, Y. (2022), 'How mask matters: Towards theoretical understandings of masked autoencoders', Advances in Neural Information Processing Systems 35 , 27127-27139.

Zhang, S., Zhu, F., Yan, J., Zhao, R. & Yang, X. (2021), Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning, in 'International Conference on Learning Representations'.

Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q. & Feng, J. (2021), 'Deepvit: Towards deeper vision transformer', arXiv preprint arXiv:2103.11886 .

A Appendix / supplemental material

A.1 Effects of Representation Whitening on the Encoder

In this section, we explore the effect of whitening representations on hidden features and weight matrices in the encoder. To be specific, similar to the settings of 5, we train three VICREG Bardes et al. (2022) models: original VICREG, VICREG without projector (Li, Chen & Yang 2020), and VICREG with OR.

Original VICREG adds two regularization terms (variance and covariance regularization) to whiten the projector features. We use X aug 1 as an example to introduce them.

Variance regularization. The variance regularization term ensures that each dimension of the learned representation Z maintains a non-trivial variance. This is critical to prevent the collapse of dimensions, where a model might ignore certain informative variations in the data. Mathematically, the variance regularization can be expressed as follows:

where D is the dimensionality of Z aug 1 = f ( X aug 1 ) , z d represents the d -th dimension of Z aug 1 , S ( z d , ϵ ) = √ Var ( z d )) + ϵ is the regularized standard deviation of z d across different samples, and γ is a threshold parameter that dictates the minimum desired standard deviation for each dimension.

Covariance regularization. The covariance regularization term is designed to decorrelate the different dimensions of Z aug 1 . By minimizing the off-diagonal elements of the covariance matrix of Z aug 1 , this term helps ensure that different dimensions capture distinct aspects of the data, thereby preventing redundancy in the representation. The covariance regularization is defined as:

̸

where Cov ( z i , z j ) denotes the covariance between the i -th and j -th dimensions of Z aug 1 . This term effectively encourages the representation to have orthogonal dimensions, which is beneficial for learning independent features.

As for the VICREG without projector, we discard the projector and apply the SSL objective directly to the representations, which ensures that the representations are whitened (no dimensional collapse in representations), i.e., minimize the correlation among dimensions and make each dimension rich in information. For OR, we choose SO as the regularizer and set γ as 1 e -6 . We then experimentally observe that guaranteeing that dimensional collapses do not occur in representations or projector features (i.e., VICREG without projector and projector features) does not guarantee that dimensional collapses do not occur in weight matrices in the encoder. Moreover, discarding the projector even damages the performance of the original VICREG, while OR still boosts the performance as shown in Table 8.

As shown in Figure 4, the weight matrices of the original VICREG suffer from dimensional collapse and whitening representations directly (i.e., getting rid of the projector) even makes the eigenvalue decay faster for the weight matrices. OR can alleviate this phenomenon.

A.2 Visualization of Weight Matrices

In this section, layer4 of ResNet18 pretrained with BYOL on CIFAR-10 is visualized. To be specific, we calculate the correlation coefficient matrix of the weight matrix and then plot the HeatMap of the correlation coefficient matrix and the results of Spectral Biclustering. As shown in Figure 5,

(a) Eigenvalues of weights(VICREG)

(b) Eigenvalues of features(VICREG)

(e) Eigenvalues of weights(VICREG with SO)

(f) Eigenvalues of features(VICREG with SO)

Normalisedeigenvalue

eigenvalue

0.1

0.1

Normalised

cifar10_image_3072

representations_512

0.01

layer1_64

0.01

hidden_feats(layer1)_73984

layer2_128

hidden_feats(layer2)_36992

layer3_256

hidden_feats(layer3)_20736

layer4_512

hidden_feats(layer4)_12800

y=1/x

y=1/x

10

100

512

10

100

1000

73984

Eigenvaluerank

Eigenvaluerank

cifar10_image_3072

Normalised eigenvalue

eigenvalue

representations_512

hidden_feats(layer1)_73984

hidden_feats(layer2)_36992

0.1

0.1

hidden_feats(layer3)_20736

hidden_feats(layer4)_12800

y=1/x

Normalised

0.01

layer1_64

0.01

layer2_128

layer3_256

layer4_512

y=1/x

10

100

512

10

100

1000

73984

Eigenvaluerank

Eigenvaluerank

cifar10_image_3072

Normalised eigenvalue

Ieigenvalue

representations_512

hidden_feats(layer1)_73984

hidden_feats(layer2)_36992

0.1

0.1

hidden_feats(layer3)_20736

hidden_feats(layer4)_12800

Normalised

y=1/x

0.01

layer1_64

0.01

layer2_128

layer3_256

layer4_512

y=1/x

10

100

512

10

100

1000

73984

Eigenvaluerank

Eigenvaluerank

HeatMap can intuitively indicate that the correlation of non-diagonal elements is constrained to 0 by OR. The Biclustering results show an obvious blocky structure, which means that there is clustering between filters, and the weight matrix is low-rank and redundant.

A.3 Visualization of Representations

We used BYOL (ResNet18) for pretraining on CIFAR-10. After pretraining, we perform dimension reduction and visualization of learned representations using UMAP (McInnes et al. 2018). As shown in Figure 6, in the absence of OR, there is a tendency for the cluster centers of each category to move closer together and more outliers appear. This is due to the fact that in the absence of OR, BYOL produces representations dominated by some extremely large eigenvalues (i.e. dimensional collapse), which is consistent with results in Section 5.2.

(a) HeatMap (without OR)

(b) HeatMap (with OR)

(c) Biclustering (without OR)

(d) Biclustering (with OR)

CorrelationMatrixHeatmap

CorrelationMatrixHeatmap

1.0

1.0

100

0.8

100

0.8

200

0.6

200

0.6

300

0.4

300

0.4

400

400

0.2

0.2

500

500

0

100

200

300

400

500

0.0

0

100

200

300

400

500

0.0

Biclustered Matrix

Biclustered Matrix

0

100

200

300

400

500

0

100

200

300

400

500

fo

0+

100

100

200

200

300

300

400

400

500

500

(a) BYOL without OR

(b) BYOL with OR (SO)

A.4 Datasets

We utilized several datasets for pretraining and evaluating SSL methods. Below we provide a detailed description of these datasets:

IMAGENET-1k (Deng et al. 2009): A large dataset contains 1,281,167 training images, 50,000 validation images, and 100,000 test images, which spans 1000 object classes.

IMAGENET-100 (Deng et al. 2009): A subdataset of IMAGENET-1K, containing 100 classes with 1000 training data and 300 test data per class.

CIFAR-10 (Krizhevsky et al. 2009): Comprising 60,000 images in 10 classes, with each class containing 6,000 images. The split includes 50,000 training images and 10,000 test images.

CIFAR-100 (Krizhevsky et al. 2009): This dataset consists of 60,000 images divided into 100 classes, with 600 images per class. The dataset is split into 50,000 training images and 10,000 test images.

Food-101 (Bossard et al. 2014): This dataset includes 101,000 images of food dishes categorized into 101 classes, with each class having approximately 1,000 images.

Flowers-102 (Xia et al. 2017): Contains 8,189 images of flowers from 102 different categories. Each class consists of between 40 and 258 images.

DTD (Sharan et al. 2014): The Describable Textures Dataset (DTD) includes 5,640 images categorized into 47 different texture categories.

GTSRB (Haloi 2015): The German Traffic Sign Recognition Benchmark (GTSRB) dataset consists of over 50,000 images of traffic signs across 43 categories.

PASCAL VOC2007 and VOC2012 (Lin et al. 2014): Used for evaluating objection tasks, this dataset includes complex everyday scenes with annotated objects in their natural context. The objection detection task contains 20 categories. We use the VOC2007 and VOC2012 train-val (16551 images) as the training set and then report the performance on the VOC2007 test set (4952 images).

Each dataset was carefully curated to support the training and validation of our models, ensuring a comprehensive evaluation across various image classification and segmentation tasks.

A.5 Joint-embedding SSL methods

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations without the need for labeled data. This appendix provides a concise overview of several SSL methods used in this paper.

MOCOv2, introduced by Chen & He (2021), on top of MOCO's momentum encoder and the use of the dynamic dictionary with a queue to store negative samples (He et al. 2020), adds the MLP projection head and more data augmentation. Compared to MOCOv2, MOCOv2plus uses a symmetric similarity loss.

MoCov3 (Chen et al. 2021) makes some improvements on the basis of v1/2, firstly, because the batchsize is large enough when training V3, the memory queue is removed, and the negative samples are sampled directly from the batch. Secondly, symmetric contrastive loss is used, and finally, an extra prediction head is added to the original encoder, which is a two-layer fully connected layer.

Bootstrap Your Own Latent (BYOL), proposed by Grill et al. (2020), introduces a novel approach to SSL that does not rely on negative pairs. Instead, BYOL employs a dual-network architecture where the encoder learns to predict the representations of the momentum encoder. Through a series of updates (i.e. EMA), where the momentum encoder gradually assimilates the encoder's weights, BYOL effectively learns robust representations. The success of BYOL depends not only on the EMA, but also on its additional projector and the BN in the projector to avoid a complete collapse of the encoder. This method challenges the conventional wisdom that contrastive learning requires negative pairs, opening new avenues for SSL research.

Expanding on the ideas of BYOL, NNBYOL (Dwibedi et al. 2021) introduces the concept of using nearest neighbors to augment the learning process. By leveraging the similarities between different instances in the dataset, NNBYOL aims to refine the quality of the learned representations further. This approach underscores the potential of incorporating instancelevel information into the SSL framework, enhancing the discriminability and robustness of the resulting models.

DINO (Caron et al. 2021) uses a self-distilling architecture. The outputs of the teacher networks (i.e. the momentum encoder) are subjected to a centering operation by averaging over a batch, and each network outputs a K-dimensional feature that is normalized using Softmax. The similarity between the student model (i.e. the encoder) and the teacher model is then computed using cross-entropy loss as the objective function. A stop-gradient operator is used on the teacher model to block the propagation of the gradient, and only the gradient is passed to the student model to make it update its parameters. The teacher model is updated using the weights of the student model (i.e. EMA).

Barlow Twins computes the correlation matrix between the embeddings of two different views of the same sample and avoids collapse by making it as close as possible to the unit matrix. This approach makes the embeddings between the two views of the sample as similar as possible while minimizing the redundancy between vector components.

VICREG avoids the complete collapse problem with variance and covariance regularization.

DCL and DCLW removes the NPC effect of infoNCE loss by getting rid of the positive term from the denominator and thus significantly improves the learning efficiency

SimSiam achieves a very strong baseline without using large batchsize, negative samples, or momentum encoder using only the stop-gradient operation.

SwaV is trained by predicting the clustering assignment of another view and also introduces multi-crop, which increases the number of views by reducing the image size without increasing the extra memory and computational requirements.

SimCLR establishes a simple and effective architecture for contrastive learning by increasing the batchsize, augmenting the data and adding a nonlinear projector after the representation.

A.6 Hyper-parameters of Pretraining and Evaluation

For each SSL method, we use the original settings of Solo-learn and LightlySSL (Da Costa et al. 2022). These settings include the network structure, loss function, training policy, and data augmentation policy. Considering that we use numerous SSL methods and that our setup is exactly the same as them, please go to their official implementation.

For OR, the appropriate regularization term γ generally depends only on the backbone used by SSL and the orthogonality regularizer (SRIP or SO) chosen. As shown in Table 9, when you want to add OR to your SSL pre-training, you simply pass the encoder into the loss function, and then you just need to set γ of the OR according to the backbone and regularizer you use.

For the classification tasks, due to computational constraints, we do not perform non-linear fine-tuning in classification tasks. Instead, we perform a linear probe or KNN to evaluate the quality of obtained representations as typically done in the literature (Huang et al. 2024, Li, Chen & Yang 2020, Lavoie

et al. 2022). To be specific, for each SSL method and dataset, after pretraining, We train a linear classifier on top of frozen representations of the training set. Then we report the Top-1 and Top-5 linear classification accuracy on the test set. When training the linear classifier, we use 100 epochs, weight decay to 0.0005, learning rate 0.1 (we divide the learning rate by a factor of 10 on Epoch 60 and 100), batchsize 256, and SGD with Nesterov momentum as optimizer (In IMAGENET-1k, we use batchsize 128 and learning rate 0.2).

For the object detection task, we perform nonlinear fine-tuning on ResNet50 in RCNN-C4 (Girshick et al. 2014) with batchsize 9 and base learning rate 0.01. We use the detectron2 (Wu et al. 2019), following the MOCO-v1 (He et al. 2020) official implementation exactly.

A.7 Time Cost of OR

Implementing OR requires computing the OR loss in the backbone at each gradient update, we count the time overhead required by the different backbones to compute OR at one time, and we have averaged over 10 times as shown in Table A.7. In the pre-training phase, the time overhead of OR is only related to the backbone and the steps that need to be updated, IMAGENET-1k (100 epochs, batchsize 128) has a total of 62599 steps, and CIFAR-100 (1000 epochs, batchsize 256) has a total of 194999 steps. As you can see, compared to the original pre-training overhead of dozens and hundreds of hours, the additional time added by OR is very small, steadily improving SSL's performance. Notably, if we use a larger batchsize such as 4096, our time overhead will be reduced by 64 on IMAGENET-1k and 16 on CIFAR-100.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we explained that dimensional collapse would exist in weight matrices and features, which would damage the performance of SSL. We used OR to eliminate the dimensional collapse in SSL. In addition, consistent improvement results were achieved under each benchmark.In the introduction, we list our 3 contributions in detail.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our approach requires the introduction of additional computational overhead, which is discussed in Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We just expressed the existing theory in the vector form to the matrice form.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer:[Yes]

Justification: All of our experimental codes including how to divide the generated data, training and testing are all the same with open-sourced Solo-learn. And we've also attached them in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include instructions in the appendix, including how to download and divide the data set, how to execute training and test scripts, and how to visualize the results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have explained data division and hyperparameters in detail in both Appendix and code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We only conducted multiple averaging operations in the statistical OR time overhead, and the experiments in the main paper did not carry out multiple averaging operations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We compared the time cost of OR in Appendix, and explained that our experiments were all completed on 4 3090 GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The data sets we use are all public data sets, and we strictly follow existing studies, which will not have a bad impact on the society.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our goal is to improve the performance of the vision foundation model, which will have a positive impact on various downstream tasks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our approach addresses the potential dimensional collapse in SSL without raising their risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We referenced the framework we used and the data set we used. In the code, we listed the URL of each data set and the code environment we needed.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
Driven by the dimensional collapse phenomenon observed in self-supervised learning, this research adopts a methodology designed to prevent the reduction in dimensionality of weight matrices. The empirical findings indicate a reduction in dimensional collapse in certain instances.

STRENGTHS:
The proposed method seamlessly integrates with existing SSL methods.

WEAKNESSES:
The proposed method lacks novelty, essentially applying the loss formulation from Barlow Twins to the weight matrix.

Moreover, the experimental results are insufficient to substantiate the efficacy of the method due to the following reasons:

1. Many of the SSL methodologies utilized to assess the OR are no longer considered state-of-the-art. Therefore, it is recommended to evaluate the OR using state-of-the-arts methods.

2. Table 1 reveals that the effectiveness of SO or SRIP appears to exhibit considerable variability, rendering the proposed methods unstable.

3. Tables 3 and 5 exclusively demonstrate the evaluation of OR for the BYOL method, omitting the efficacy assessments of other SSL methods. Additionally, in Table 4, the datasets are characterized as small-scale, which contradicts the claims made by the authors.

QUESTIONS:
1. What are the advantages of regularizing weights over regularization of representation?

2. What is the performance of SRIP in Table 2 and 5?

LIMITATIONS:
Please refer to the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn order to combat the problem of dimensionality collapse in self-supervised learning, the authors of this work introduce orthogonal regularization on encoders during pretraining. Taking inspiration from previous work on supervised orthogonality regularization, the authors study the impact of OR on eigenvalues of weights when considering self-supervised learning and find that OR learned weights/features contain more meaningful features and avoid dimensional collapse better. The recommended result seems to be effective for both contrastive and non-contrastive SSL methods.\n\nSTRENGTHS:\nOverall I think the idea is simple and the paper is well written. Particularly the observations regarding the eigenvalues of SSL features and weights and the impact of OR are informative. In addition, I believe the OR improvements on transfer learning and downstream tasks are considerable and could become a staple part of SSL.\n\nWEAKNESSES:\n1) I would like to point out the improvements of OR on SSL methods is not as major on the simple datasets such as Cifar100 set according to Table 1. However, the changes in larger datasets with BYOL are more noticeable but I would like to ask why results for other methods were not included for larger datasets. \n2)  Particularly with regards to Fig 3, to me it doesn’t look as if the feature eigenvalue changes much with the OR except for the representations. Perhaps a combination of other OR methods could help ? Or adding orthogonality regularization with the features. \n3) With regards to classification accuracy on Table 5, higher batch size and longer training somewhat diminish the impact of OR improvement. Perhaps this result could be used to encourage OR when training with smaller batches or less epochs is required ?\n\nQUESTIONS:\n1) For section 3.2 under what case would we have input > output ? \n2) Could the authors provide some explanation for what entails whitening of the features ?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method to reduce the dimensional collapse issue in self-supervised pretrained networks. Dimensional collapse occurs when a few large eigenvalues dominate the eigenspace. While previous research focused solely on the output representations of the pretrained networks, this work shows that dimensional collapse also occurs on the intermediate representations and weight matrices.\n\nTo address this issue, the authors use orthogonal regularization (OR) objectives for the weight matrices in linear layers (dense or convolutional) within the network. The OR objective promotes orthogonality in the weight matrices by minimizing the distance between the Gram matrix of each weight matrix and the identity matrix. They demonstrate that using the OR regularization objective leads to a smoother decay of eigenvalues in the (intermediate) representations and weight matrices of a self-supervised pretrained ResNet18.\n\nThe authors evaluate the OR regularization objective by pretraining various backbones (RN18, RN50, WideResNet28w2, and ViT-tiny/small/base) with multiple joint-embedding SSL methods (MoCov2, MoCov3, DINO, BYOL, NNBYOL) on CIFAR-100. They then train linear classification models on the frozen features, observing improved accuracy in nearly all cases with OR. Finally, they pretrain a RN50 backbone with BYOL on ImageNet-1k and evaluate it through linear probing on ImageNet and fine-tuning on Pascal object detection, showing improved results with OR.\n\nSTRENGTHS:\n- The used method of reducing dimensional collapse of SSL methods is simple and can be easily integrated on various SSL methods and backbones.\n- The strongest aspect of this work is that it shows that OR consistently improves various joint-embedding SSL methods and backbones. However, the improvements (in linear accuracy) are small in the case of well-tuned baselines.\n\nWEAKNESSES:\n- **(W1)** The introduction's claimed contribution, \"We **systematically study** the phenomenon of dimensional collapse in SSL, including how whitening representations and network depth affect the dimensional collapse of weight matrices and hidden features,\" seems to me to be an exaggeration. In the main paper, there is only one experiment pretraining ResNet18 with BYOL w/ and w/o OR on CIFAR-10, and another experiment in the supplementary material using VICReg, which are not directly comparable. Given the emphasis on this contribution, a more appropriate approach would be to include an analysis of whitening's impact on dimensional collapse in the main paper (e.g., adding another column with whitening-based results in Figure 3). This analysis should be conducted under the same conditions as the experiments in Figure 3, for example, using BYOL with a projector and whitening. This setup would enable a straightforward comparison of vanilla BYOL, BYOL with OR, and BYOL with whitening in terms of their effects on the eigenvalues of intermediate representations and weight matrices.\n- **(W2)**: **Limited technical contribution**: as discussed in the related work sections of this work, orthogonality regularization of the weight matrices within neural network has been extensively studied before (related work subsection 2.3). In particular, the OR objectives studied here are directly taken from Bansal et al. 2018, and straight-forwardly applied to the examined self-supervised approaches, without any modification at all. So, the main novelty of this work is to apply in a plug-and-play way the OR regularization objectives of Bansal et al. 2018 on self-supervised methods.\n- **(W3)** About the CIFAR100 results: in the results provided in solo-learn (https://github.com/vturrisi/solo-learn), which the authors also use for their experiments, some baselines (i.e., SSL methods that do not use OR) have higher results than what is reported in this work and often higher even to the results reported in this work with the OR regularization objective.  In particular, this is the case for MoCov2+ (69.89% in solo-learn vs 69.51% here), MoCov3 (68.83% in solo-learn vs 67.13% here), DINO (66.76% in solo-learn vs 59.13% in here). For BYOL however, is the other way around (70.46% in solo-learn vs 71.15% in here). What is the reason for the differences in the results?\n- **(W4)** The improvement that OR offers over a well-tuned BYOL baseline on ImageNet (200 pretraining epochs and 256 batch-size) is marginal: from 69.76 to 70.16. It seems that it offers significant performance improvements in mostly un-tuned baselines, like the results when pretraining BYOL for 100 and for 64 batch-size. In reality, even 200 epochs are not adequate for pretraining BYOL that, its performance reaches 74.3 for 1000 epochs (see BYOL paper). So, it is unclear whether OR could offer any improvement for such long pretraining settings that are typically used in SSL. \n- **(W5)** Not my primary concern, but although OR is shown to provide a small improvement on the linear accuracy of some SSL methods (e.g., going from 69.76 to 70.16 for BYOL pretrained with 200 epochs on ImageNet-1k), no evidence is shown that OR actually avoids dimension collapse in settings where otherwise (i.e., without OR) the SSL method will completely collapse and fail to learn any meaningful representation.\n\nQUESTIONS:\nI would like the authors to address my concerns (W1), (W2), (W3) and (W4), as described in the weaknesses section. Some more specific questions:\n- About W1: can the authors provide a comparison of vanilla BYOL, BYOL with OR, and BYOL with whitening in terms of their effects on the eigenvalues of intermediate representations and weight matrices.\n- About W3: What is the reason for the differences in the results reported in this work and in in solo-learn (https://github.com/vturrisi/solo-learn)?\n- About W4: Can the authors demonstrate that OR improves even well-tuned baselines like BYOL with 1000 epochs (74.3 in BYOL paper) or MoCo-v2 with 800 epochs (71.1% in https://arxiv.org/pdf/2003.04297), or MoCo-v3 (74.6 for RN50 see: https://github.com/facebookresearch/moco-v3). Any of them would be good.\n\nLIMITATIONS:\nYes, they are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper aims to improve self-supervised learning (SSL) by introducing orthogonality regularization (OR) during pretraining to prevent dimensional collapse. OR improses orthogonality constraint on weight matrices, not only preventing dimensional collapse of representation but also regularizing the weights from collapsing. The experimental results show that the benefit of the proposed method is consistent and widely applicable across datasets and architecture choices.

STRENGTHS:
* This paper effectively communicates both the problem it addresses and the solution it presents. The motivation and solutions are compelling and presented in an easily understandable manner.
* The benefit of the proposed method appears to be consistent and significant.

WEAKNESSES:
* More discussion on the difference between the proposed orthogonal regularization methods would have been helpful. On the surface, the two methods are almost equally good.

QUESTIONS:
* In practice, a practitioner has to choose between SO and SRIP. What would be the authors' recommendation? 
* Do you expect that applying dropout on the encoder's hidden neurons would result in a similar effect, as dropout is also meant to prevent feature co-adaptation? It is quite likely that the existing encoders have already employed dropout, so I wonder if dropout is insufficient to prevent the collapse.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDimensional collapse, characterized by the dominance of a few large eigenvalues, is a phenomenon in self-supervised learning (SSL) that can lead to redundant features and weight matrices. To address this, the authors propose orthogonal regularization (OR) during pretraining to promote orthogonality in weight matrices, rather than using common approaches that regularize features via the objective (e.g., Barlow Twins, VicReg). The authors conduct experiments with five SSL methods from the SoloLearn framework to demonstrate the benefit of OR.\n\nSTRENGTHS:\nThe paper is well-written and provides a good introduction to the topic of representation collapse. The motivation for the approach is clearly stated, and the experimental setting is well documented. With small architectures like ResNet18, the effect of orthogonal regularization (OR) appears impressive at first glance. Although OR introduces some computational overhead, it is relatively small and may diminish further with larger batch sizes.\n\nWEAKNESSES:\nThe soundness of some experiments seems doubtful. Results for WideResNet28x2 are consistently below those for ResNet18. Additionally, the results using Dino are generally poor, leading to all ViT experiments being below any ResNet baseline.\n\nThe performance of BYOL with OR, when trained for only 100 epochs with a batch size of 64, is even below the reported accuracy in SoloLearn. Considering the optimal training duration for BYOL is approximately 1000 epochs with a batch size of 4096, this comparison appears to be in a severely undertrained regime. Slightly longer training, 200 epochs with a batch size of 256, shows that the benefit of the OR approach diminishes rapidly.\n\nMinor issue: Line 44 states, \"We evaluate the effect of adding OR in seven modern SSL methods during pretraining,\" but results are shown for only five methods.\n\nAdditionally, since SSL approaches like Barlow Twins and VicReg aim to achieve orthogonality as well, there should be an experiment evaluating the effect of OR on these methods.\n\nUsing one learning rate (LR) for all methods in linear probing might not be optimal. It is cheap to attach multiple linear probing heads and find the most suitable LR for each method to ensure fair comparisons.\n\nQUESTIONS:\nMemory Overhead of OR: How much memory overhead does orthogonal regularization (OR) introduce? \n\nExplanation for Lower Gamma in Larger Models: The paper uses a lower gamma for larger models, but why? One might think that larger models would require more regularization. \n\nResults for Barlow Twins and VicReg: Why were results for Barlow Twins or VicReg not reported? Is the OR approach redundant when combined with these methods, which also aim to achieve orthogonality?\n\nBalanced Eigenvalues and Performance Discrepancy: The eigenvalues of VicReg without a predictor (Appendix Fig 4) appear more balanced than those of BYOL OR (Fig 3), yet VicReg's performance is far worse.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}