An Investigation into Whitening Loss for Self-supervised Learning
Authors: Xi Weng
Lei Huang
Lei Zhao
Rao Muhammad Anwer
Salman Khan
Fahad Khan
Venue: NeurIPS 2022 Accept (2022)
Keywords: whitening
self-supervised learning
normalization
feature decorrelation

Abstract: A desirable objective in self-supervised learning (SSL) is to avoid feature collapse.  Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size.  Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations. The code is available at https://github.com/winci-ai/CW-RGP.

An Investigation into Whitening Loss for Self-supervised Learning
Xi Weng"1, Lei Huang 1.2, Lei Zhao
Rao Muhammad Anwer², Salman Khan", Fahad Shahbaz Khan SKLSDE, Institute of Artificial Intelligence, Beihang University, Beijing, China Mohamed bin Zayed University of Artificial Intelligence, UAE
Abstract
A desirable objective in self-supervised learning (SSL) to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose framework with an informative indicator analyze whitening loss, which provides clue demystify several interesting phenomena as well as pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding be full-rank. This full-rank constraint is also sufficient avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses promising potential for learning good representations. The code is available https://iituu..com// wttpt///wwwwiiii////eeeeee.
Introduction
Self-supervised learning (SSL) has made significant progress over the last several years I 6 () almost reaching the performance of supervised baselines many downstream tasks 33 24 35
16
Several recent approaches rely on joint embedding architecture in which a dual pair of networks are trained produce similar embeddings for different views of the same image ( Such methods aim to learn representations that are invariant to transformation of the same input. One main challenge with the joint embedding architectures is how to prevent collapse of representation, in which the two branches ignore the inputs and produce identical and constant output representations [6] g.
One line of work uses contrastive learning methods that attract different views from the same image (positive pairs) while pull apart different images (negative pairs), which can prevent constant outputs from the solution space [43] While the concept is simple, these methods need large batch size to 37]. Another line of work tries to directly match the positive targets without introducing negative pairs. A seminal approach, B BYOL| [16], shows that an extra predictor and momentum is essential for representation learning. SimSiam [8 further generalizes [16] by empirically showing that stop- -radddidtent is essential for preventing trivial solutions. Recent works generalize the collapse problem into dimensional collapse [1] 25 where the embedding vectors only span a dimeeniimniional subspace and would be highly correlated. Therefore, the embedding vector dimensions would vary together and contain redundant information. To prevent the dimensional
obtain a good performance
"equal contribution correspponingg author (huangleiAle) A alllbaa.eeee cduaa.edu.cn). This work partially done while Lei Huang was visiting scholar Mohamed hin Zayed University of Artificial Intelligence, UAF. 2This collapse is also referred to informational collapse in 2
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

collapse, whiteming loss is proposed by only minimizing the distance between embeddings of positive
pairs under the condition that embeddings from different views are whitened [2 21. A typical way
is using batch whitening (BW) and imposing the loss on the whitened output 12 21 which obtains
promising results.
Although whitening loss has theoretical guarantee in avoiding collapse. we experimentally observe that this guarantee depends on which kind of whitening transformation [26] is used in practice (see Section 3.2 for details). This interesting observation challenges the motivations of whitening loss for SSL. Besides, the motivation of whiteming loss is that the whitening operation can remove the correlation among axes [2] and a whitened representation ensures the examples scattered in spherical distribution 122 Based on this argument, one can use the whitened output as the representation for downstream tasks, but is not used in practice. To this end, this paper investigates whitening loss and tries to demystify these interesting observations. Our contributions are as follows:
We decompose the symmetric formulation of whitening loss into two asymmetric losses, where each asymmetric loss requires online network to match whitened target. This mechanism provides a pivoting point connecting to other methods, and a way to understand why certain whitening transformation fails to avoid dimensional collapse.
Our analysis shows that BW based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse.
We propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based method in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results ImageNet classification and COCO object detection show that CW-RGP has promising potential in learning good representation.
2 Related Work
A desirable objective in self-supervised learning is avoid feature collapse.
Contrastive learning prevents collapse by attracting positive samples closer, and spreading negative
44). In these methods, negative samples play an important role and need to be well designed 34 20). One typical mechanism is building memory bank with a momentum encoder to provide consistent negative samples, proposed in MoCos [19], yielding promising results
samples apart 43
[99 [ 30 Other works include SimCLR (6) addresses that more negative samples in a batch with strong data augmentations perform better. Contrastive methods require large batch sizes or memory banks, which tends to be costly, promoting the questions whether negative pairs is necessary.
Non-contrastive methods aim to accomplish SSL without introducing negative pairs explicitly 3
4 16, 8). One typical way to avoid representational collapse is the introduction of asymmetric network architecture. BYOL [66] appends a predictor after the online network and introduce momentum into the target network. SimSiam [ further simplifies BYOL. by removing the momentum mechanism, and shows that stop-gradient to target network serves as an alternative approximation to the momentum encoder. Other progress includes an asymmetric pipeline with self-distillation loss for Vision Transformers [5]. It remains not clear how the asymmetric network avoids collapse without negative pairs, leaving the debates on batch normalization (BN) [14 41 [36] and stop-gradie stop-gradient [8] (46, even though preliminary works have attempted to analyze the training dynamics theoretical with certain assumptions [40] and build a connection between asymmetric network with contrastive learning methods 33). Our work provides a pivoting point connecting asymmetric network to
profound whitening loss in avoiding collapse.
Whitening loss has theoretical guarantee in avoiding collapse by minimizing the distance of positive pairs under the conditioning that the embeddings from different views are whitened 45 12 21 2
embedding the so-called soft whitening, which is proposed in Barlow Twins [45), VICReg (2) and CCA-SSG [47). Another way is using batch whitening (BW) (221 the so-called hard whitening, which is used in W-MSE [2] and Shuffle-DBNN 211]. We propose a different hard whitening method channel whitening (CW) that has the same function that ensures all the singular values of transformed output being one for avoiding collapse. But CW is more numerical stable and works better when batch size is small, compared to BW. Furthermore, our CW with random group partition (CW-RGP) can effectively control the extent of constraint on embedding and obtain better performance in practice. We note that recent work ICL [88 proposes to decorrelate instances, like CW but having several significant
One way to obtain whitened output is imposing whitening penalty regularization

---

differences in technical details. ICL uses soop-gradient" for the whitening matrix, while CW requires back propaaattn through the whitening transformation. Besides, ICL uses extra pre-conditioning on the covariance and whitening matrices, which is essential for the numerical stability, while CW does not use extra pre-conditioning and can work well since it encourages the embedding to be full-rank.
3 Exploring Whitening Loss for SSL
3.1 Preliminaries
H: encoding
Z: embeding
whitened output
Let x denote the input sampled uniformly from set of images D. and T denote the set of data transformations available for augmentation. We consider the Siamese network fo(-) parameterized by 9. takes input two randomly augmented views, X1 = (x) and X2 T2(x),, where T1,2 T. The network fe(-) is trained with an objective function that minimizes the distance between embeddings obtained from different views of the same image:
X
H
Zz
Encoder
Pro
W Whitening
2,
3XUPPX
x
Encoder
H2
Pro
Zz
Whitening
2,
Siamese network
Figure 1: The basic notations for SSL used in this paper.
C(x, 0) = Ex T2TTT fe((T(()))
(I)
where .(:-) a loss function. partieular, the Siamese network usually consists of an encoder Fo, (.) and projector Ga, (.). Their outputs h = Eg, (T(x)) and Go, (h) are referred to as encoding and embedding, respectively. summarize the notations and use the corresponding capital letters denoting mini-batch data in Figure [ Under this notation, we have fo(-) Go, (Eo, (-)) with learnable parameters 0 {0., 0g )- The encoding h is usually used as representation for evaluation by either training a linear classifier [99] transferring downstream tasks. This is due that h is shown to obtain significantly better performance than the embeulding'
66 g]
The mean square error (MSE) of L2 normmalzeze vectors is usually used as the loss function g
Z1
Z2
E 22) =
7
1z2122
where || ||2 denotes the L2 norm. This loss is also equivalent to the negative cosine similarity, up to scale of? and optimization irrelevant constant.
Collapse and Whitening Loss. While minimizing Eqn. [ trivial solution known collapse could occur such that fo(x) = c, Yx E D. The state of collapse will provide no gradients for learning and offer no information for discrimination. Moreover, weaker collapse condition called dimensional collapse can be easily arrived, for which the projected features collapse into a fow-dimensional manifold. As illustrated in (21). dimensional collapse is associated with strong correlations between axes, which motivates the use of whitening method in avoiding the dimensional collapse. The general idea of whitening loss [12] is to minimize Eqn. [] under the condition that embeddings from different views are whitened, which can be formulated as
min L(x; 0) EX TI [(21, 22),
s.t. cov (z,, 2,) I, E (1,2).
Whitening loss provides thcoretical guarantee in avoiding (dimensional) collapse, since the embedding is whitened with all axes decorrelated [12 21). While it is difficult to directly solve the problem
of Eqn. 3 Ermolov et al. [12) propose to whiten the mini-batch embedding E RRdxxm using
batch whitening (BW) [22 38 and impose the loss on the whitened output Z E RR xm given the mini-batch inputs X with size of m, as follows:
min (( Ex TI 12 22
with 2, + (Z.), ) i (1,2), where (b--) denotes the whitening transformation over mini-batch data.
'The dual view formulation can be extended to different views, shown in 12

---

60
400
Moe lom
40
RunkiEn
2CA
IN
-
200
PCA
Itam
20
0.0
90
100)
1.50
0
50
100
150
50
1000
150
1.00
150
Epoch
Epoch
Epoch
Epoch
(b)
(c)
(d)
Figure 2: Effects of different whitening transformations for SSL. We use the ResNet- 18 as the encoder (dimension of representation 512.), two layer MLP with ReLU and BN appended as the projector
(dimension of embedding is 64). The model is trained on CIFAR-10 for 200 epochs with batch size of 256 and standard data argumentation, using Adam optimizer 2r (more details of experimental setup please see supplementary materials). We show (a) the linear evaluation accuracy; (b) the training loss; (c) the rank of embedding; (d) the rank of encoding.
Whitening Transformations. There are an infinite number of possible whitening matrices, as shown in 26 [2]], since any whitened data with a rotation is still whitened. For simplifying notation, we assume Z is centered by := Z m 111)). Ermolov et al.
[2] propose W-MSE that uses Cholesky decomposition (CD) whitening: (c) = L L- in Equ. where is a lower triangular matrix from the Cholesky decomposition, with LL" E. Here Σ
Zz" is the covariance matrix of the embedding. Hua et al. [21] use zero-phann vero-phase component analysis (ZCA) whiteming [22 in Equ. 4
71
$2CA = UA UT where A = diag( A1, .d,) and U = [u ..... Ud, are the eigenvalues and associated eigenvectors of Σ, i.e., UAU = Σ. Another famous whitening is principal components analysis (PCA) whitening: PCA = A UT [26 22
3.2 Empirical Investigation Whitening Loss
In this section, we conduct experiments investigate the effects of different whitening transformations (-) used in Eqn. for SSL. Besides, we investigate the performances of different features (including encoding H, embedding Z and the whitened output Z) used as representation for evaluation. For illustration, we first define the rank and stable-rank [42] of a matrix as follows:
Definition 1. Given matrix A E ddxm d S m, denote (X1...., "d). the singular values of A in a descent order with convention A1 > 0. The rank of A is the number of its non-zero singular values, denoted as Rank(A (A)
>i i=1 T(A, > 0), where I(.) is the indicator function. function. The stable-rank of A
is denoted as r rAA)
By definition, Rank(A) can be a good indicator to evaluate the extent of dimensional collapse of A, and r (A) can be indicator to evaluate the extent of whitening of A. It can be demonstrated that r(A) < Rank(A) < d 422 Note that if A is fully whitened with covariance matrix AAT mI, we have r(A) Rank(A) A) = d. We also define normalized rank as Rank Rank( and normalized stable-rank F(A) = for comparing the extent of dimensional collapse and whitening of matrices with different dimensions, respectively.
d
PCA Whitening Fails to Avoid Dimensional Collapse. We compare the effects of ZCA, CD, PCA transformations for whitening loss, evaluated on CIFAR-10 using the standard setup for SSL (see Section 4.1 for details). Besides, we also provide the result of batch normalization (BN) that only performs standardization without decorrelating the axes, and the 'Plain' method that imposes the loss directly on embedding. From Figure 2 we observe that naively training a Siamese network ('Plain') results in collapse both the embedding (Figure c)) c)) and encoding (Figure 2 d)), which significantly hampers the performance (Figure Ia)), although its training loss becomes close to zero (Figure ()). We also observe that extra BN imposed the embedding prevents collapse to point. However, it suffers from the dimensional collapse where the rank embedding and encoding significantly low, which also hampers the performance. ZCA and CD whitening both maintain high rank embedding and encoding by decorrelating the axes, ensuring high linear evaluation accuracy. However, note that PCA whitening shows significantly different behaviors: PCA whitening cannot decrease the loss and even cannot avoid the dimensional collapse, which also leads to significantly downgraded performance. This interesting observation challenges the motivations of whitening loss for SSL. We defer the analyses and illustration in Section 3.3]
Whitened Output is not a Good Representation. As introduced before, the motivation of whiten- ing loss for SSL is that the whitening operation can remove the correlation among axes 21 and

---

Nornslized stable ranner
Top-1 mocomnyy (16)
SANN sccumeyy
-0-
800
300
H
75
Z
2
70
0
100
150
130
Epoch
Epoch
Epoch
(a)
(b)
Figure 3: Comparisons of features when using encoding H. embedding Z and whitened output Ž respectively, We follow the same experimental setup Figure 2 We show (a) the linear evaluation accuracy; (b) the kNN accuracy: (c) the normalized stable-rank comparing the extent of whitening (note that the normalized stable-rank Z is always 100% during training and we omit it for clarity). The results are averaged by five random seeds, with standard deviation shown using shaded region.
whitened representation ensures that the examples scattered in spherical distribution 12 which sufficient to avoid collapse. Based on this argument, one should use the whitened output 2 as the representation for downstream tasks, rather than the encoding H that is commonly used. This raises questions that whether H is well whitened and whether the whitened output is good feature. We conduct experiments to compare the performances of whitening loss, when using H, Z and Z as representations for evaluation respectively. The results are shown in Figure We observe that using whitened output Z as representation has significantly worse performance than using H. Furthermore, we find that the normalized stable rank H is significantly smaller than 100%. which suggests that H is not well whitened. These results show that the whitened output could not be good representation.
3.3 Analysing Decomposition of Whitening Loss
For clarity, use the mini-batch input with size of m. Given mini-batch input X with two augmented views, Eqn. 4 can be formulated as:
((X) i
(5)
Let us consider proxy loss described as:
c((x) 1 1 (Z2) m
se 11.
- m
(Zz)st
Zzel
C2
where (.)s indicates the gradiett operation. is easy to demonstrate that dc ac'
de 80 (see
supplementary materials for proof), That is, the optimization dynamics of L is equivalent to C' By looking into the first term Eqn. 6 we have:
1 m
G
llo(Z) al.)
Here, can view (Zii predictor that depends Z during forward propagation, and Zo as whitened target with r rZZ)) Rank dg. In this way, we find that minimizing E only requires the embedding Z, being full-rank with Rank(Z,) = dg. as stated by following proposition.
Proposition 1. Let = arg minz, (Z,). We have that à is not an empty set, and AZ E A, Z₁ is full-rank. Furthermore, for any {o, with 0ı 2 2,... od, > O, we construct = {Z, IZi U2 diag( O2,..., dd,) VT, where U2 E RRa, Xd. and V, E IRmxa, are from the singular value decomposition of Zz, i.e.. U2(yml)VV 22. When use ZCA whitening, we have A. C A.
The proof shown in supplementary materials. Proposition states that there are infinity matrix with full-rank that is the optimum when minimizing C w.r.t. Z₁. Therefore, minimizing C only requires the embedding Z, being full-rank with Rank(Z,) d., and does not necessarily impose the constraints Z to be whitened with r(Z,) dg. Similar analysis also applies to C and
5

---

minimizing C2 requires Zy being full-rank. Therefore, BW-based methods shown in Equ. 4 do not impose whitening constraints on the embedding as formulated in Eqn. El but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse for embedding, even though it is a weaker constraint than whitening.
Our analysis further implies that whitening loss in its symmetric formulation (Eqn. 5 can be decomposed into two asymmetric losses (Eqn. ( where each asymmetric loss requires an online network to match a whitened target. This mechanism provides a pivot connecting to other methods, and clue to understand why PCA whitening fails to avoid dimensional collapse for SSL.
Connection to Asymmetric Methods. The asymmetric formulation of whitening loss shown in Eqn. bears resemblance to those asymmetry methods without negative pairs, e.g., SimSiam [8] In these methods, extra predictor is incorporated and the moop-gaddeet is essential for avoid collapse. In particular, SimSiam uses the objective as:
1 PP (+) o Z1 - (Z2) m
((X)
[P,, Z2 (1))
where (o) is the predictor with learnable parameters 8g. By contrasting Eqn. and the first term of Eqn. 8 we find that: 1) BW-based whitening loss ensures a whitened target Zg, while SimSiam does not put constraint on the target Zz: 2) SimSiam uses a learnable predictor Pa, (-), which is shown to empirically avoid collapse by matching the rank of the covariance matrix by back propapropngttion [40), while BW-based whitening loss has an implicit predictor d(Zi) depending on the input itself, which is a full-rank matrix by design. Based on this analysis, we find that BW-based whitening loss can surely avoid collapse if the loss converges well, while Simsian can not provide such a guarantee in avoiding collapse. Similar analysis also applies to BYOL [16], except that BYOL uses a momentum target network for providing target signal.
Connection to Soft Whitening. VICReg (2 also encourages whitened embedding produced from different views, but by imposing a whitening penalty as a regularization on the embedding, which is called soft whitening. In particular, given mini-batch input, the objective of VICReg is as follows") R
((X) 7
Z212 +
Σ
Ea- m
ZZ
AII
where a 2 is the penalty factor. Similarly, can use proxy loss for VICReg and considering its term corresponding to optimizing Z, only (similar to Eqn. 7 we have:
EVICReD X
1 7 (Z2)
2 AT
(10)
Based on this formulation, we observe that VICReg requires embedding Z, to be whitened by, 1) the additional whitening penalty, and 2) fitting the (expected) whitened targets Zz. By contrasting Eqns. 7 TO] we highlight that the so-called hard whitening methods, like W-MSE [12), only impose full-
rank constraints on the embedding, while soft whitening methods indeed impose whitening constraints. Similar analysis also applies to Barlow Twins 45 except that the whiteninggdecoorrllttion penalty is imposed on the cross-covariance matrix of embedding from different views.
Connection to Other Non-contrastive Methods. SwAV 4. a clustering bbased method, uses a "swapped" prediction mechanism where the cluster assignment (code) of a view is predicted from the representation of another view, by minimizing the following objective:
((X) = ZQ))aa))) + ((C
(Q1)st).
(11)
Here, Cs the prototype matrix learned by back-propagatio Q is the predicted code with equal- partition and high-entropy constraints, and SWAV uses cross-entropy loss as E(.-) to match the distributions. The constraints on Q; are approximately satisfied during optimization, by using the iterative Sinkhorn- Knopp algorithm conditioned on the input C' Note that SwAV explicitly uses stop-gradient when it calculates the target Q; By contrasting Eqn. 7 and the first term of Eqn. [I] we find that: 1) SwAV can be viewed an online network match a target with constraints, like B BW-based whitening loss, even thought the constraints imposed on the targets between them are
*Note the slight difference where VICReg uses margin loss on the diagonal of covariance, while our notation
uses MSE loss.

---

different; 2) From the perspective of asymmetric structure, SwAV indeed uses linear predictor C7 that is also learned by propapaaon like SimSiam, while BW-based whitening loss has an implicit predictor o(z1 depending on the input itself. Similar analysis also applies to DINO [ which further simplifies the formulation of SwAV by removing the prototype matrix and directly matching the output of another view, from the view of knowledge distillation. DINO uses centering and sharpening operations to impose the constraints on the larget (output of another view). One significant difference between DINO and whitening loss is that DINO uses population statistics of centering calculated by moving average, while whitening loss uses the mini-batch statistics of whitening.
Max
Why PCA Whitening Fails to Avoid Di- mensional Collapse? Based on Eqn. 7 we note that whitening loss can favor- ably provide full-rank constraints embedding under the condition that the online network can match the whitened targets well. We experimentally find that PCA-based whitening loss provides volatile sequence of whitened targets dur- ing training, as shown in aa) is difficult for the online network to match such target signal with significant vari- ation, resulting in minimal decrease in the whitening loss (see Figure 2). Fur- thermore, we observe that PCA-based whitening loss has also significantly vary- ing whitening matrix sequences (-) (Figure 4b)) even given the same input data. This coincides with the observation in [6 8). where an unstable predictor results in significant degenerate performance. Our observations are also accordance with the arguments in [22] [33 that PCA-based BW shows significantly large stochasticity. We note that ZCA whitening can provide relatively stable sequences of whitened targets and whitening matrix during training (Figure 4 which ensures stable training for SSL. This likely
1500
Proquzaay
2000
PCA ZCA
the
2001
100 10.0 3000
(a)
Satıpling
(b)
Vaniaoco
Figure 4: Illustration of PCA-based whitening loss suf- fering from training instability. We follow the same ex- perimental setup as Figure 2 Given a certain mini-batch input input (m 2048), monitor its whitened output Ž' and whitening matrix E' for each epoch I. We calculate the variance along the training epochs for each element of Z and 4 We show (a) the mean and maximum of vari- ances of Ž. noting that PCAA-100 110000 indicates the variance of PCA whitened output is calculated along the first 100 epochs; and (b) the histogram of variance of s.
due to the property of ZCA-based whitening that minimizes the total squared distance between the original and whitened variables [26] [22].
Why Whitened Output is not Good Representation? A whitened output removes the corre- lation among axes [2 and ensures the examples scattered in a spherical distribution [2], which bears resemblance to contrastive learning where different examples are pulled away. We conduct experiments to compare SimCLR (6, BYOL [66,, VICReg 2 and W-MSE [2), and monitor the cosine similarity for all negative pairs, stable-rank and rank during training. From Figure we find that all methods can achieve a high rank on the encoding. This is driven by the improved extent of whitening on the embedding. Furthermore, we observe that the negatives cosine similarity decreases during the training, while the extent of stable-rank increases, for all methods. This observation suggests that representation with stronger extent of whitening is more likely to have less similarity among different examples. We further conduct experiments to validate this argument, using VICReg
Nagatbes sims baityy (%6
BYOL
4n
75
SimCLR
(22)
Rank(t))
400
VJCRsg
50-
W-MSE
20-
300
0.5
30
100
150
200
50
16n Epoch
150
512
(00)
1.0
Spoch
Epoch
Penulry factorr
(a)
(b)
(c)
Figure 5: Comparison of different SSL methods. We follow the same experimental setup as Figure 2 We show (a) the normalized stable-rank of embedding; (b) the rank of encoding; (c) the negatives cosine similarity, calculated on the embeddings from all negative pairs (different examples). We also train VICReg with varying penalty factor a to show the relationship between the normalized stable-rank and negatives cosine similarity in (d). Here, we use embedding dimension of 64. We have similar observations when using the embedding dimension of other numbers (e.g., 128 and 256).
7

---

500
F12)
85
500
84
CW
GP2
<iPa
RGP3
RGP4
CW
GP
GP4
RGP2
RCP
CW
GP4
RGP2
RGP4
(c)
Figure 6: Illustration of CW with random group partition. We follow the same experimental setup as Figure 2 except that we set dimension of embedding as 2048 tailored for CW. We use 'GP2" ('RGP2") to indicate CW using group partition (random group partition), with group number of 2. (a) The linear and k-NN accuracies; (b) The normalized stable-rank of embedding: (c) The rank of embedding. All experiments are repeated five times, with standard deviation shown as error bars.
with varying penalty factor (Eqn. [0p adjust the extent whitening on embedding (Figure d()). Therefore, whitened output leads to the state that all examples have dissimilar features. This state can break the potential manifold the examples in the same class belong to. which makes the learning more difficult [17]. Similar analysis for contrastive learning is also shown in [6]. where classes represented by the projected output (embedding) are well separated, compared encoding.
4 Channel Whitening with Random Group Partition
One main weakness BW-based whitening loss is that the whitening operation requires the number of examples (mini-batch size) m to be larger than the size of channels d, to avoid numerical instability This requirement limits its usage scenarios where large batch training data cannot be fit into the memory. Based on previous analysís, the whitening loss can be viewed as online learner to match a whitened target with all singular values being one. We note the key of whitening loss is that conducts a transformation é + Ż, ensuring that the singular values of Ž are one. We thus propose channel whitening (CW) that ensures the examples in mini-batch are orthogonal:
Centering Z = (T- 1Z)
Whitening 2 Z
(12)
where E Rmxx is the 'whitening matrix' that is derived from the corresponding 'covariance ... In our implementation, we use ZCA whitening to obtain $ CW ensures the examples in a mini-batch are orthogonal each other, with ŻT Iركة This means CW has the same ability as BW for SSL in avoiding the dimensional collapse, by providing target Z whose singular values are one. More importantly, one significant advantage of CW is that it can obtain numerical stability when the batch size is small, since the condition that d > m be obtained by design (e.g., we can set the channel number of embedding d to be larger than the batch size m). Besides, find that CW amplify the full-rank constraints the embedding by dividing the channels/neurons into random groups, as we will illustrate.
1 matrix': E' 2-I ze.
Random Group Partition. Given the embedding Z IRdxm d > m. we divide it into 2 1 groups (Z)
R 5 xmmgg Ii=1 where we assume that d is divisible by g and ensure > m. We then perform CW each Z Z() l,...... Note that the ranks of Z and (t) all at most Therefore, m embedding. compared to CW without group partition that only one constraint with Rank(Z) (Z) = m. Although CW with group partition can provide more full-rank constraints for mini-batch data, find that it can also make the population data correlated, if group partition is all the same during training, which decreases the rank and does not improve the performance in accuracy by our experiments (Figure 6). We find random group partition, which randomly divide the channels/neuron. into group for each iteration (mini-bet batch data), alleviate this issue and obtain improved performance, from Figure We call our method as channel whitening with random group partition (CW-RGP) and provide the full algorithm and Py Torch-style code in supplementary materials.
CW with group partition provides g constraints with (a)
We note that Hua et al. [2 use a similar idea for BW, called Shuffted-DBN. However Shuffled-DBN cannot well amplify the full-rank constraints by using more groups, since BW-based methods require m > to avoid numerical instability. We further show that CW-RGP works remarkably better than
An empirical setting is 2d that can obtain good performance shown in 12
21

---

Table 1: Classification accuracy (top 1) of linear classifier and 5-nearest neighbors classifier for different loss functions and datasets with a ResNet-18 encoder.
CIFAR-10
CIFAR-100
STL-10
Tiny- Liny-ImaeeNet
Method
linear
5-nn
linear
5-nn
linear
5-nn
linear
5-nn
SimCLR 6
91.80
88.42 66.83 56.56
90.51
85.68
48.84
32.86
BYOL [6
91.73 89.45
66.60
56.82
91.99
88.64
51.00
36.24
SimSiam 8 (rерro.)
90.51
86.82 66.04
55.79
88.91
84.84
48.29
34.21
ShuffledDBBN (repro.)
90,45
88.15
66.07
56.97
89.20
84.51
48.60
32.14
Barlow Twins
45
(repro.)
88,51
86.53
65.78
55.76
88.36
83.71
47.44
32.65
VICReg 2 (repro.)
90,32
88.41
66.45
56.78
90.78
85.72
48.71
33.35
Zero-ICL 48
(repro.)
88.12
86.64
61.91
53,47
86.35
82.51
46.25
32.74
W-MSE 2
12
91.55
89.69
66.10
56.69
90.36
87.10
48.20
34.16
W-MSE 4 12
91.99
89.87
67.64
56,45
91.75
88.59
49.22
35.44
CW-RGP (ours)
91.92
89.54
67.51
57.35
90.76
87.34
49.23
34.04
CW-RGP 4 (ours)
92.47
90.74
68.26
58.67
92.04
88.95
50.24
35.99
Shuffle-DBNN in the subsequent experiments. We attribute this results to the ability of CW-RGP in amplifying the full-rank constraints by using groups.
4.1 Experiments for Empirical Study
In this section, we conduct experiments to validate the effectiveness of our proposed CW-RGP. We eval-
Table 2: Comparisons on ImageNet linear clas- sification. All are based ResNet-50 encoder. The table is mostly inherited from [8]
uate the performances of CW-RGP for classifica-
tion on CIFAR-10, CIFAR-100 22 STL-10 [10]. TinyImageNee (29) and ImageNet [T. We also evaluate the effectiveness in transfer learning, for pre-trained model using CW-RGP. We run the ex- periments on one workstation with 4 GPUs. For more details of implementation and training proto- col, please refer to supplementary materials.
Method
Batch size
100 eps
200 eps
SimCLR 6
4096
66.5
68.3
MoCo v2
256
67.4
69.9
BYOL 16
4096
66.5
70.6
SwAV 4
4096
66.5
69.1
SimSiam
256
68.1
70.0
W-MSE
4096
69.4
Zero-CL
48
1024
68.9
Evaluation for Classification We first conduct experiments on small and medium size datasets (in-
BYOL
16 (repro.)
512
66.1
69.2
SWAV 4
(repro.)
512
65.8
67.9
cluding CIFAR-10, CIFAR-100, STL-10 and Tiny-
W-MSE
12 (repro.)
512
66.7
67.9
ImageNet), strictly following the setup of W-MSE paper [12]. Our CW-RGP inherits the advantages of W-MSE in exploiting different views. CW-RGP and CW-RGP 4 indicate our methods with s = and = 4 positive views extracted per image respectively, similar to W-MSE [12]. The results of baselines shown in Table. are partly inherited in [12] except that we reproduce certain baselines under the same training and evaluation settings as in [12 (some different dyper-parameter settings are shown supplementary materials). We observe that CW-RGP obtains the highest accuracy on almost all the datasets except than 2, similar to W-MSE. These results show that CW-RGP is a competitive SSL method. We also confirm that CW with random group partition could obtain a higher performance than BW (and with random group partition), comparing CW-RGP to W-MSE and Shuffled-DBB
CW-RGP 4 (ours)
512
69.7
71.0
Besides, CW-RGP with 4 views are generally better
We then conduct experiments on lcrge-scale ImageNet, strictly following the setup of SimSiam paper (8). The results of baselines shown in Table 2 are mostly reported in [8]. except that the result of W-MSE 4 is from the W-MSE paper [12 and we reproduce BYOL [16). SWAV 4 and W-MSE 4 [1] under batch size of 512 based the same training and evaluation settings in [8] for fairness. CW-RGP 4 is trained with a batch size of 512 and gets the highest accuracy among all methods under both 100 and 200 epochs training. We find that our CW-RGP can also work well when combined with the whitening penalty used VICReg. Note that we also try batch size of 256 under 100-epoch training, which gets the top-1 accuracy of 69.5%.
Transfer to downstream tasks We examine the representation quality by transferring our model to other tasks, including VOC [3] object detection. COCO [2) object detection and instance segmentation. We use the baseline (except for the pre-training model, the others are exactly the same) of the detection codebase from MoCo [19] for CW-RGP to produce the results. The results of baselines shown in Table are mostly inherited from [8) We clearly observe that CW-RGP performs better than or on par with these state-of-the-an approaches on COCO object detection and instance segmentation, which shows the great potential of CW-RGP in transferring to downstream tasks.

---

Table 3: Transfer Learning. All competitive unsupervised methods are based on 200-epoch pre- training in ImageNet (IN). The table mostly inherited from [ Our CW-RGP performed with random seeds, with mean and standard deviation reported.
VOC 07+12 detection
COCO detection
COCO instance seg.
Method
AP
AP
AP75
AP50
AP
AP75
APso
AP
AP7s
Scratch
60.2
33.8
33.1
44.0
26.4
27.8
46.9
29.3
30.8
INsupeevisee
81.3
53.5
58.8
58.2
38.2
41.2
54.7
33.3
35.2
SimCLR [6
81.8
55.5
61.4
57.7
37.9
40,9
54.6
33.3
35.3
MoCo v2
82.3
57.0
63.3
58.8
39.2
42.5
55.5
34.3
36.6
BYOL [16
81.4
55.3
61.1
57.8
37.9
40.9
54.3
33.2
35.0
SwAV 4
81.5
55.4
61.4
57.6
37.6
40.3
54.2
33.1
35.1
SimSiam
82.0
56.4
62.8
57.5
37.9
40.9
54.2
33.2
35.2
CW-RGP (ours)82.22+o.or 10 63.8
60.5 20.26 =0.7
20.1
57.3
20.12 37.9 37.9=0.14
Ablation for Random Group Partition.
Table 4: Results of ablation for random group partition.
We also conduct experiments to show the advantages of random group partition for channel whitening. We use 'CW', 'CW- GP' and "CW-RGP' to indicate channel whitening without group partition, with group partition and with random group partition, respectively. We further con- 2 and
CIFAR-10
CIFAR-100
Method
linear
5-nn
linear
5-nn
CW 2
91,66
88.99
66.26
56.36
CW-GP
91.61
88.89
66.17
56.53
CW-RGP
91.92
89.54
67.51
57.35
CW 4
92.10
90.12
66.90
57.12
4W-GP4
92.08
90.06
67.34
57.28
sider the setup with positive views. We use the same setup as in Table and show the results in Table 4 We have similar observation as in Figure 6 improver
4
CW-RGP 4
92.47
90.74
68.26
58.67
that CW with
random group partition improves the performance.
Ablation for Batch Size. Here, we conduct experiments empirically show the advantages of CW over BW, in terms of the stability using different batch size. We train CW and BW on ImageNet- 100, using batch size ranging in {32, 64, 128, 256]. Figure shows the results. We can find that CW is more robust for small batch size training.
Decrass dhbor scuncy
Decrease of AN acoumcy
BW
BW
Batch (a)
Figure 7: Decrease in top-1 and 5-nn accuracy (in % points) of CW and BW at 100 epochs on ImageNet-100.
Conclusion and Limitation
In this paper, we invested whitening loss for SSL, and observed several interesting phenomena with further clarification based on our analysis framework. We showed that batch whitening (BW) based methods only require the embedding be full-rank, which is also sufficient condition for collapse avoidance. We proposed channel whitening with random group partition (CW-RGP) that well motivated theoretically in avoiding collapse and has been validated empirically in learning good representation.
Limitation. Our work only shows how to avoid collapse by using whitening loss, but does not explicitly show what should be the extent of whitening of a good representation. We note that concurrent work addresses this problem by connecting the eigenspectrum of representation to a power law [15]. and shows the coefficient of the power law is strong indicator for the effects of representation. We believe our work can be further extended when combined with the analyses from [15]. Besides, between encoding and embedding [18), which is important to answer why encoding is usually used as representation for evaluation, rather than the whitened output embedding. Our attempts, shown in supplementary materials, provide preliminary results, but does not offer an answer to this question.
work does not answer how the projector affects the extents of whitening
Acknowledgement This work was partially supported by the National Key Research and Develop- ment Plan of China under Grant 202 IZD01 12901, National Natural Science Foundation of China (Grant No. 62106012), the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Bachman, P., Hjelm, R.D., Buchwalter, W.: Learning representations by maximizing mutual information actoss views. In: NeurIPS (2019)
[2] Bardes, A., Ponce, J., LeCun, Y:: Vicreg: Varianceiimarrinee-ccvvaaaae regularization for self-supervised learning. In: ICLR (2022)
[3] Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: ECCV (2018)
[4] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski. P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: NeurIPS (2020)
[5] Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supe vision transformers. In: ICCV (2021)
[6] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML (2020)
[7] Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04297 (2020)
[8] Chen, X., He, K.: Exploring simple siamese representation learning. In: CVPR (2021)
[9] Chen, X., Xie, S., He, K.; An empirical study of training self-supervised vision transformers. In: CVPR (2021)
[10] Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks unsupervised feature learning. In: Proceedings of the fourteenth international conference artificial intelligence and statistics (2011)
[11] Deng, J., Dong, W. Socher, R., Li, LJJ,, Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hicrarchical Image Database. In: CVPR (2009)
[12] Ermolov, A., Siarohin, A., Sangineto, E., Sebe, N.: Whitening for self-supervised represetationo learning. In: ICML (2021)
[13] Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88, 303-308 (September 2009)
[14] Fetterman, A., Albrecht, J.: Understanding self-supervised and contrastive learning with "boot- strap your own latent" (byol). Technical Report (2020)
[15] Ghosh, A., Mondal, A.K., Agrawal, K.K., Richards, B.A.: Investigating power laws in deep representation learning. arXiv preprint arXiv:2202.05808 (2022)
[16] Grill, J.B., Strub, F,, Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent a new approach to self-supervised learning. In: NeuraIPS (2020)
[17] HaoChen, J.Z., Wei, C., Gaidon, A., Ma. T.: Provable guarantees for self-supervised deep learning with spectral contrastive loss. In: NeurlPS (2021)
[18] He, B., Ozay, M.: Exploring the gap between collapsed and whitened features in self-supervised learning. In: ICML. (2022)
[19] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning, In: CVPR (2020)
[20] Henaff, O.: Data-efficient image recognition with contrastive predictive coding. In: ICML (2020)
[21] Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., Zhao, H.: On feature decorrelation in self- supervised learning. In: ICCV (2021)

---

[22] Huang, L., Yang, D., Lang, B., Deng, J.: Decorrelated batch normalization. In: CVPR (2018)
[23] Huang, L.., Zhao, L.., Zhou, Y., Zhu, F., Liu, L.,, Shao, L.: An investigation into the stochasticity of batch whitening. In: CVPR (2020)
[24] Jaiswal, A., Babu, A.R., Zadeh, M.Z., Banerjee, D., Makedon, F.: A survey on contrastive self-supervised learning. arXiv preprint arXiv:2011.00330 (2020)
[25] Jing, L., Vincent, P., LeCun, Y., Tian, Y:: Understanding dimensional collapse in contrastive self-supervised learnng... In: ICLR (2022)
[26] Kessy, A., Lewin, A., Strimmer, K.: Optimal whitening and decorrelation. The American Statistician (2018)
[27] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014)
[28] Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep. (2009)
[29] Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N (2015)
[30] Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., Gao, Efficient self-supervised vision transformers for representation learning. In: ICLR (2022)
[31] Li, J., Zhou, P., Xiong, C., Hoi, S.C.: Prototypical contrastive learning of unsupervised repre- sentations. In: ICLR (2021)
[32] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, L.: Microsoft coco: Common objects in context. In: ECCV (2014)
[33] Liu, Y., Pan, S., Jin, M., Zhou, C., Xia, E., Yu, P.S.: Graph sulf-supe learning: A survey. arXiv e-prints pp. arXiv-2103 (2021)
[34] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint
(2018)
[35] Ranasinghe, K., Naseer, M., Khan, S., Khan, E.S., Ryoo, M.: Self-supervised video transformer. In: CVPR (2022)
[36] Richemond, P.H., Grill, J.B., Altché, F, Tallec, C., Strub, F., Brock, A., Smith, S., De, S., Pascanu, R., Piot, B., et al.: Byol works even without batch statistics. arXiv preprint arXiv:2010.100 10241 (2020)
[37] Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., Khandeparkar, H.: A theoretical analysis of contrastive unsupervised representation learning. In: ICML (2019)
[38] Siarohin, A., Sangineto. E., Sebe, N.: Whitening and coloring batch transform for gans. In: ICLR (2019)
[39] Tao, C., Wang, H., Zhu, X., Dong, J., Song, S., Huang, G., Dai, J.: Exploring the equivalence of siamese supervipe learning via A unified gradient framework. In: CVPR (2022)
[40] Tian, Y., Chen, X., Ganguli, S.: Understanding self-supervised learning dynamics without contrastive pairs. In: ICML (2021)
[41] Tian, Y., Yu, L., Chen, X., Ganguli, S.: Understanding self-supervised learning with dual deep networks. CoRR abs/2010.00578 (2020)
[42] Vershynin, R.: High-dimensional probability: An introduction with applications data science. Cambridge university press (2018)
[43] Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non parametric instance discrimination. In: CVPR (2018)
[44] Ye, M., Zhang, X., Yuen, P.C., Chang, S.F.: Unsupervised embedding learning via invariant and spreading instance feature. In: CVPR (2019)
12

---

[45] Zbontar, J., Jing, L., Misra, I. Lecun, Y.. Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: ICML (2021)
[46] Zhang. C., Zhang, K., Zhang, C., Pham, T.X., Yoo, C.D., Kweon, LSS.: How does simsiam avoid
collapse without negative samples? unified understanding with self-superviser contrastive learning. In: ICLR (2022)
[47] Zhang, H., Wu, Q. Yan, J., Wipf, D., Yu, P.S.: From canonical correlation analysis to self- supervised graph neural networks. In: NeurlPS (2021)
[48] Zhang, S., Zhu, F., Yan, J., Zhao, R., Yang, X.: Zero-CI:: Instance and feature decorrelation for negative-free symmetric contrastive learning. In: ICLR (2022)
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] We include the code https://github.. https://ithub..om/iin---ae wwt.///iciai///-RRG..... and instructions needed to reproduce the main experimental results in supplemental material, and the data is standard dataset in public.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] We specify the training details in supplemental material
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See the corresponding figures and tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4. and
supplemental material.
(b) Did you mention the license of the assets? [Yes] See supplemental material.
(c) Did you include any new assets either the supplemental material or as a URL? [Yss] We include new code in https://gttubb.cmm www.wwai/nCiiwaa..
(d) Did you discuss whether and how consent obtained from people whose data you're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "rSugjUExorR", "reviewer": "Reviewer_h596", "summary": "- Self-supervised learning approaches need to avoid collapse to a trivial representation. This has been tackled by various approaches in literature including use of negatives, use of asymmetric networks etc. Use of whitening loss has been explored by some recent works. \n- This paper studies this whitening loss and various variants of the whitening transformation used in practice. \n- The paper investigates some issues with the transformations used and a new random group partition based channel whitening approach to prevent collapse. The approach works well for large batch sizes which is shown through experiments on datasets like ImageNet and transfer to COCO. \n", "questions": "Please refer to the section on \"Strengths And Weaknesses\" for questions and comments. ", "limitations": "- The authors have adequately discussed limitations of the proposed approach. \n- While the work is more at a fundamental level, I do urge the authors to include some discussion on potential negative impacts. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper seeks to analyze various approaches used for whitening of feature representations used for SSL. The paper is very well written. \n- The paper does a good job at explaining some of the preliminaries.\n- Previous approaches have been analyzed extensively through experiments based on public repositories. \n- The paper decomposes the whitening loss and connects it to common SSL approaches. \n- Obtains state-of-the-art results on standard benchmark datasets. \n- I think the paper will be of significance to the wider ML/SSL community. \n\nMinor Weaknesses, suggestions and questions :\n- Since one the Section 3.3 is one of the most important contributions of this paper, I would suggest having some more details & intuitions behind the explanations. Especially : L185-196. This could perhaps be included in the supplementary material. \n- Can the centering+sharpening operation used in DINO & equipartion constraint used in SwAV be looked at through a similar analysis ?\n- L280: To ensure $\\frac{d}{g} > m$, do you modify the last layer of the projection head ? \n- How does the proposed approach compare to state-of-the-art on training time ? \n- Do you have some thoughts on why the gap in performance decreases at 100 ep vs 200 ep (Table 2) ? What about W-MSE 4 @ 200 ?\n- I agree with the motivation in L263-266. I think from a practical standpoint, it is be a better idea to report required GPU memory & GPUs per approach in Table 2 in addition to the batch size. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VVaP_YTVg4j", "reviewer": "Reviewer_j1ii", "summary": "The paper provides an analysis of different whitening losses used in Self-supervised learning, seeking to interpret some empirical observations, e.g. the connection between whitening losses and the asymmetric methods (BYOL/SimSiam), why PCA does not work, and why whitened outputs are not good representations. The paper also proposes channel whitening with random group partition (CW-RGP), which is shown to be an effective whitening loss. CW-RGP is evaluated on image recognition (ImageNet1k and 4 other benchmarks), object detection (VOC/COCO), and instance segmentation (COCO).", "questions": "i) When evaluating the performance of the whitened representation (L167, Fig.3) I wonder if the whitening matrix (\\Phi(Z)) is estimated per mini-batch or over the whole training set?\n\nii) L308-309, are “d=2” and “d=4” here referring to “g=2” and “g=4”?\n", "limitations": "Limitations are discussed in the main paper. Potential negative societal impacts are not discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\ni) the proposed random group partition is technically sound;\n\nii) the performance of the proposed CW-RGP method looks promising on some benchmarks e.g. COCO object detection/ instance segmentation;\n\niii) an ablation study on the batch size is provided;\n\niv) the writing is clear in general.\n\n\nWeaknesses\n\ni)  The interpretations about “why PCA does not work” and “why the whitened output is not good” are not convincing. To me, the explanation could be much simpler and more intuitive: the batch whitened outputs rely on the batch statistics: an image may have different whitened representations when computed in different mini-batches.\n\nWhen using PCA, the descriptor of an image relies on the eigenvectors (U in L136), which may change dramatically across mini-batches. This explains why BW-based approaches prefer large batch sizes. It explains the experimental results shown in Fig. 4. It also explains why whitened outputs are not good representations, i.e. experiments in Fig. 3.\n\nNote that the predictors in the asymmetric methods (L197), on the other hand, do not rely on batch statistics, which I believe is a key difference.\n\nii) The comparisons in Table 2 may not show the full picture, e.g. baselines like BYOL/SWAV may be significantly under-trained. Here, the batch size for BYOL/SWAV is 4096. When trained for fewer epochs (e.g. 200 epochs), a large batch size may hurt the performance as it leads to significantly fewer training iterations. It would be better if baselines like BYOL/SWAV-batch-size-512-epoch-100/200 are also included.\n\n\niv) Channel whitening has been proposed before in [47] for the same task. As [47] has been published in ICLR 2022. I’m not sure if [47] could be considered a concurrent work. Compared to [47], the new content is the random group partition. This extra design may not be enough for NeurIPS. Overall, I believe [47] should at least be included as a baseline, and the ablation on the random group partition should be included.\n\nMinor issues\n\ni) L18-19 “two networks are trained …[8]”, I think there is only one network \n\nii) L286: “rand” → “random”\n\niii) Table 1, Simsim → SimSiam\n\niv) Table 1, references are included for some baselines (SimCLR, BYOL), but not all (e.g. Shuffled-DBN, W-MSE)\n\nv) Table 2 & 3, it would be better if references are included.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9qN8891QCin", "reviewer": "Reviewer_qKzP", "summary": "This paper gives out a thorough investigation into whitening loss applied in self-supervised learning. Based on their analysis, they proposed a channel whitening method named CW-RGP. From their experiments on variuos datasets, CW-RGP gets the SOTA in most cases.", "questions": "I would wonder whether the Barlow Twins method can be better. \n\nThen grouping the experiments based on their techniques is easier to show the contribution, especially this paper belongs to the whitening loss branch. \n\nFinally, the comparison methods from Table 1,2,3 are a bit different, actually I was finding the W-MSE results in Table 3. Can you explain a bit?", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1, The writing is nicely done and the paper is organized well for understanding.\n\n2, The experiments are quite sufficient, including the analysis experiments and comparison experiments.\n\n3, The understanding for whitening loss is convincing and there are also some new indicators proposed for future study.\n\nWeakness:\n\nOverall, I didn't see much disadvantages in this paper, I'm just curious about several points below.\n\n1, The barlow twins and VICReg methods are not present in all tables. In the related work, the authors have classified them as soft whitening, so I think some comparison with them should not be neglected. \n\n2, Although, Barlow Twins might get even better results than CW-RGP (from my experiments before), I don't think it can be very important. This paper mainly studies the whitening loss, thus I think the comparison with whitening loss based method (such as W-MSE) is more important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept