Whitening for Self-Supervised Representation Learning
Authors: Aleksandr Ermolov
Aliaksandr Siarohin
Enver Sangineto
Nicu Sebe
Venue:  (2021)
Keywords: self-supervised learning
unsupervised learning
contrastive loss
triplet loss
whitening

Abstract: Most of the self-supervised representation learning methods are based on the contrastive loss and the instance-discrimination task, where augmented versions of the same image instance  ("positives") are contrasted with instances extracted from other images ("negatives"). For the learning to be effective, a lot of negatives should be compared with a positive pair, which is computationally demanding. In this paper, we propose a different direction and a new loss function for self-supervised representation learning which is based on the whitening of the latent-space features. The whitening operation has a "scattering" effect on the batch samples, which compensates the use of negatives, avoiding degenerate solutions where all the sample representations collapse to a single point. Our Whitening MSE (W-MSE) loss does not require special heuristics (e.g. additional networks) and it is conceptually simple. Since negatives are not needed, we can extract multiple positive pairs from the same image instance. We  empirically show that W-MSE is competitive with respect to popular, more complex self-supervised methods. The source code of the method and all the experiments is included in the Supplementary Material.

Under review as a conference paper at ICLR 2021
WHITENING FOR SELF-SUPERVISED REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Most of the self-supervised representation learning methods are based on the con- trastive loss and the cattane-ddisrimmnatiie task, where augmented versions of the same image instance ("positives"") are contrasted with instances extracted from other images ("negatives"). For the learning to be effective, lot of negatives should be compared with a positive pair, which computationally demanding. In this paper, we propose different direction and a new loss function for self- supervised representation learning which is based on the whitening of the latent- space features. The whitening operation has "scattering" effect on the batch samples, which compensates the use of negatives, avoiding degenerate solutions where all the sample representations collapse to single point. Our Whitening MSE (W-MSE) loss does not require special heuristics (e.g. additional networks) and it is conceptually simple. Since negatives are not needed, can extract mul- tiple positive pairs from the same image instance. We empirically show that W- MSE competitive with respect to popular, more complex self-supervised meth- ods. The source code of the method and all the experiments is included in the Supplementary Material.
INTRODUCTION
One of the current main bottlenecks in deep network training is the dependence on large annotated training datasets, and this motivates the recent surge of interest in unsupervised methods. Specif- ically, self-supervised representation learning. a network is (pre-)trained without any form of manual annotation, thus providing a means to extract information from unlabeled-date sources (c.g., text corpora, videos, images from the Internet, etc.). In self-supervision, label information re- placed by prediction problem using some form of context or using a pretext task. Pioneering work in this direction was done in Natural Language Processing (NLP), in which the co-occurrence of words in a sentence is used to learn language model (Mikolov et al. 2013a b) Devlin et al. 2019).
In Computer Vision, typical contexts or pretext tasks are based on: (1) the temporal consistency
in videos (Wang& & Gupta| 2015 Misra et al. 2016 Dwibedi al.) 2019). ). (2) the spatial order of
patches in still images (Noroozi& & Favaro 2 Misra van der Maaten [2199 Hénaff et al. 2019)) or (3) simple image transformation techniques (Ji et al. | 2019 He et al. 2019 Wu al. [018)) The intuitive idea behind most of these methods is to collect pairs of positive and negative samples: two positive samples should share the same semantics, while negatives should be perceptually different. A triplet loss (Sohn) 2016 Schrooff et al. 2015) Hermans et al. 2017: Wang & Gupta| 20015; Misra al. 2016) can then be used learn a metric space which should represent the human perceptual similarit. However, most of the recent studies use a contrastive loss (Hadsell et al. 2006) or one
of its variants (Gutmann & Hyvärinen 2010 van den Oord al.) 2018 Hielm et al. 2019). while
Tschannen et al. (2019) show the relation between the triplet loss and the contrastive loss.
worth noticing that the success of both kinds of losses is strongly affected by the number and the quality of the negative samples. For instance, in the case of the triplet loss, common practice is to select hard/semi-hare negatives (Schroff et al. 20155 Hermans et al.) (017)). On the other hand, Hjelm et al. (2019) have shown that the contrastive loss needs large number of negatives to be competitive. This implies using batches with a large size, which is computationally demanding, especially with high- nigh-resolution images. In order alleviate this problem. Ww et al. (2018) use memory bank of negatives, which is composed of feature-vector representations of all the training samples. He et al. (2019) conjecture that the of large and fixed-representatiion vocabularies is

---

Under review as a conference paper at ICLR 2021
one of the keys to the success of self- self-supervision in NLP. The solution proposed by He et al. (2019)
extends Wu et al. 2018) using a memory- efficient queue of the last visited negatives, together with momentum encoder which preserves the intra intra-queue representation consistency, [Chen et al. (2020)
have performed large-scale experiments confirming that a large numher of negatives (and therefore a large batch size) is required for the contrastive loss to be efficient. Concurrently with work, Grill et al. (2020) have suggested that it is not necessary to rely on the contrastive scheme, introducing nigh-performing alternative based bootstrapping.
In this paper we propose a new self-supervised loss function which first scatters all the sample representations in a spherical distribution and then penalizes the positive pairs which are far from each other. In more detail, given set of samples V = {v.}, corresponding to the current mini- batch of images = { we first project the elements of V onto spherical distribution using a whitening transform (Siarohin et al. 2019). The whitened representations [zi). corresponding to V, are normalized and then used to compute a Mean Squared Error (MSE) loss which accumulates the emror taking into account only positive pairs (z,, Zj). We do not need to contrast positives against negatives as the contrastive loss or in the triplet loss because the optimization process leads to shrinking the distance between positive pairs and, indirectly, scatters the other samples to satisfy the overall spherical-distie ddistribution constraint.
In summary, contributions are the following:
We propose a new loss function, Whitening MSE (W-MSE), for self-supervised training. W-MSE constrains the batch samples to lie in spherical distribution and it is an alternative to positive-negative instance contrasting methods.
Our loss does not rely on negatives, thus including more positive samples in the batch can be beneficial; we indeed demonstrate that multiple positive pairs extracted from one image improve the performance.
We empirically show that our W-MSE loss outperforms the commonly adopted contrastive loss when measured using different standard classification protocols. We show that W-MSE is competitive with respect to state-of-the-an self-supervised methods.
2 BACKGROUND AND RELATED WORK
A typical self-supervised method is composed of two main components: a pretext task, which ex- ploits some a-priori knowledge about the domain to automatically extract supervision from data, and loss function. In this section we briefly review both aspects, and we additionally analyse the recent literature concerning feature whitening.
Pretext Tasks. The temporal consistency in video provides an intuitive form of self seff-supervision: temporally-close frames usually contain a similar semantic content (Wang Guptal 2015 van den Oord et al. 2018). Misra et al. (2016) extended this idea using the relative temporal order of frames, while Dwibedi et al. (2019) used a temporal cycle consistency for self-supervision, which based comparing two videos sharing the same semantics and computing inter-video frame-to- frame nearest neighbour assignments.
When dealing with still images, the most common pretext task is instance discrimination (Wu al. (2018)): from a training image x, composition of data-augmentation techniques are used to extract two different views of x (r; and s.y). Commonly adopted transformations are: image cropping. rotation, color jittering, Sobel filtering, etc.. The learner is then required to discriminate (æ;, from other views extracted from other samples (Wu et 2018 Ji et al. 2019 He et al. 2019 Chen
et al. 2020
Denoising auto-encoders (Vincent et al. 2008) add random noise the input image and try to recover the original image. More sophisticated pretext tasks consist in predicting the spatial order of image patches (Norozzi & Favaro 2016; Misra & van der Maaten 2019) or reconstructing large masked regions of the image (Pathak et al. (016). Hjelm et al. (2019); Bachman et al. (2019) compare the holistic representation of an input image with patch of the same image. Héénfff et al.)
Here and in the following, with "spherical distribution" mean distribution with a zero-mean and an maarrxx covariance.
2

---

Under review as a conference paper at ICLR 2021
2019) use similar idea, where the comparison depends on the patch order: the appearance of given patch should be predicted given the appearance of the patches which lie above it in the image.
standard data augmentation techniques on still images to obtain positive pairs, which is a simple method to get self-supervision (Chen et al.) (020) and does not require a pretext- task specific network architecture (Hjelm al., 2019: Bachman al. 2019: Hénaff et al. 2019).
In this paper
Loss functions. Denoising auto-encoders use a reconstruction loss which compares the generated image with the imput image before adding noise. Other generative methods use adversarial loss in which a discriminator provides supervisory information to the generator (Donahue et al. 2017
Donahue & Simonyan 2019).
Early supe-sse (deep) discriminative methods used a triplet loss (Wang & Gupta 2015 Misra al. 2016):: given two positive images I,, 2y and a negative Iy (Sec. D together with their corre- sponding latent-space representations Z,, Zj, Zg. this loss penalizes those cases in which z; and z are closer to each other than z; and zj plus a margin m.
I Trriplet = max (x. 25 2. 33 m, 0).
Most of the recent self-supervised discriminative methods are based on some contrastive loss (Had- sell et al. 2006) variant, in which and 2, are contrasted against set of negative pairs. Following the common formulation proposed by van den Oord et al. (2018):
exp (2) ZT/T)
LContrastion log
E L exp (z Zu/T)
where T is lemperature hyperparameter which should be manually set and the sum in the denom- inator is over a set of K - negative samples. Usually K is the size of the current batch, i.c., K 2., being N the number of the positive pairs. However, as shown by Hjellm al.) (2019). the contrastive loss (2) requires large number of negative samples to be competitive. Wu et al. (018); al. (2019) use set of negatives much larger than the current batch, by pre-computing latent-ppace representations of old samples. SimCLR (Chen et al.) (2020)) uses a simpler, but com- putationally very demanding, solution based on large batches.
While recent works (van den Oord et al., 2018 Hénaff et al. 2019, Hjelm et al. 2019: Bachman jet al. 2019 Ravanelli |
draw a relation between the contrastive loss and an estimate of the mutual information between the latent-space image representations, Tschannen et al. (2019) showed that the success of this loss is likely related to learning a metric space, similarly to what happens with triplet loss. On the other hand, Wang & Isola (2020) showed that the L2 normalized contrastive loss asymptotically converges to the minimization of two desirable characteristics of the latent-space representations
the surface of the unit hypersphere: uniformity and semantic align- ment. In the same paper, the authors propose two new losses (Luniform and Lalign) which explicitly deal with these characteristics.
Concurrently with our work, BYOL (Grill et al. (2020)) proposes a "bootstrapping" scheme which is alternative to the positive-negative contrastive learning. In BOO, an "online" network is optimised to predict the output of a "target" network, whose parameters are a running average of the online network. The predictions of the two networks are compared using an additional prediction network and an MSE loss. However, very recently, /Ftttermnn & Albrecht (2020) and Tian et al. (2020) have empirically shown that BYOL can avoid a collapsed solution through the use of the Batch Norm (BN) (Toffe& & Szegedy 2015) which avoids constant representations. Our work can be seen generalization of this finding with a much simpler network architecture (more details in Sec. 3.I
In this paper we propose different loss which is competitive with respect other alternatives. Our loss formulation simpler because it does not require a proper setting of the t hyperparameter in equation 2 in equation T. or additional networks with specific weight update schemes in BYOL.
Feature Whitening. We adopt the efficient and stable Cholesky decomposition (Dereniowski & Marek] 2004) based whitening transform proposed by (Siarohin et al. (2019) to project our latent- space vectors into spherical distribution (more details in Sec. 3). Note that Huang al. (2018);
3

---

Under review as a conference paper at ICLR 2021
representaion space
Whitened representation space
5) Normalized representation on hypersphere
attract each other with MIS
Intermediate iteration, scattering is preserved
optimisation over the positive samples are clustered together
Figure 1: schematic representation of the W-MSE based optimization process. Positive pairs are indicated with the same shapes and colors. (1) A representation of the feature batch V when training starts. (2, 3) The distribution of the elements after whitening and L2 normalization. (4) The MSE computed over the normalized z features encourages the network to move the positive pair representations closer each other. (5) The subsequent iterations move closer and closer the positive pairs, while the relative layout of the other samples is forced to lie in a spherical distribution.
Siarohin et al. (2019) use whitening transforms in the intermediate layers of the network for com- pletely different task: extending BN to multivariate batch normalization.
THE WHITENING MSE Loss
Given image 2, extract an embedding parametrized with 8 (more details below). We require that: (1) the image embeddings are drawn from non-degenerate distribution (the latter being distribution where, e.g., all the representations collapse to a single point), and (2) positive image pairs xz), which share a similar semantics, should be clustered close each other. We formulate this problem follows:
f(x; 0) using encoder network f(:0)
ming E dist 24 2z); s.l. coo(z,, 2,) cov(2, (. 2,)
(3)
I,
(4)
where dist(-) is a distance between vectors, / is the identity matrix and (z,, Zz) corresponds to positive pair of images 23). With equation 4 we constrain the distribution of the z values to be non-degenerate, hence avoiding that all the probability mass is concentrated in single point. Moreover, equation 4 makes the components of to linearly independent from each other, which encourages the different dimensions of represent different semantic content. We define the distance with the cosine similarity, implemented with MSE between normalized vectors:
2j
(z Zj)
disl(zi, 2j)
2
1 12
| 11
|
|2,
In Appendix we we also include other experiments in which the cosine similarity replaced by the Euclidean distance. We provide below the details on how positive image samples are collected, how they are encoded and how the above optimization is implemented.
First, similarly to Chen et al. (2020), obtain positive samples sharing the same semantics from a single image x and using standard image transformation techniques. Specifically, we use a com- position of image cropping, grayscaling and color jittering transformations T(: p). The parameters
4

---

Under review as a conference paper at ICLR 2021
E(+
g()
Whitening
Norm.
MSE
Figure 2: A scheme of our training procedure. First, d (d = 4 in this case) positive samples are generated using augmentations. These images are transformed into vectors with the encoder E(-) Next, they are projected onto a lower dimensional space with projection head g Then, Whiten- ing projects these vectors onto a spherical distribution, followed by an optional L2 normalization. Finally, the dashed curves show all the d(d 1)/2 6 comparisons used in our W-MSE loss.
(p) are selected uniformly at random and independently for each positive sample extracted from the same image: 2; Ta;; P:). We concisely indicate with pos(i, the fact that D; and 2 (Ia, 2y E B, B the current batch) have been extracted from the same image.
The number of positive samples per image d may vary, trading off diversity in the batch and the amount of the training signal. Favoring more negatives, most of the methods use one positive pair (d 2). However, Ji et al. (2019) have demonstrated improved performance with samples, while Caron et al. (2020) use samples. In our MSE-based loss (see below), use all the possible did - 1)/2 combinations of positive samples. We include experiments for = 2 (1 positive pair) and d = (6 positive pairs).
For representation learning. we use a backbone encoder network E(-). E(-). trained without human supervision, will be used in Sec. for evaluation using standard protocols. We standard ResNet-18 (He et al., 2016) as the encoder, and E(x) is the output of the average-pooling layer. This choice has the advantage which use encoder architectures specific for a given pretext task (see Sec. 2 Since h E R5 is high-dimensional vector, following Chen et (2020) we use nonlinear projection head g()) to project h in a lower dimensional space: v g(h, (h). where g(-) is implemented with MLP with one hidden layer and BN layer. The whole network f()) is given by the composition of g(-) with E(-) (see Fig. 2)
be simple and easily reproducible, in contrast other methods
Given N original images and batch of samples = (II,,... where K Nd, let V [VI,...Vrk be the corresponding batch of features obtained as described above. In the proposed W-MSE loss we compute the MSE over all Nd(d-1)) 1)/2 positive pairs, where constraint|4] is satisfied using the reparameterization of the v variables with the whitened variables
Lw-MsE(V)
2
Σ
dist(z;, 2z),
(6)
Nd(d-1)
(VoN))EV
where z = Whilening(v), and:
Whitening(v) = -vvlv (v
In equation μv the mean of the elements in V: pv * Ex Vk. while the matrix Wy is such that: WT Wv Ey! being Ey the covariance matrix of V:
Σv K-1 Σ (vk pV)(VE -T)
(8)
For more details how Wy is computed, refer to Appendix B Equation 7 performs the full
whitening each v, E V and the resulting set vectors Z {z).... Zx lies in a zero-centered distribution with covariance matrix equal to the identity matrix (Fig.)
5

---

Under review as a conference paper at ICLR 2021
prodiie pair
pertition
partiniio 2
V
maddom pemmstation
Vo
Vi
V:
V;
Wi. Wi. W., pey
W.. j
Figure 3: Batch slicing. V is first partitioned in parts (d 2 in this example). We randomly permute the first part and we apply the same permutation to the other d - 1 parts. Then, we further split all the partitions and we create sub-batches (V;). Each V; is independently used compute the sub-batch specific whitening matrix Wi and centroid pyy.
The intuition behind the proposed loss is that equation 6 penalizes positives which are far apart from each other, thus leading g(E()) ) to shrink the inter-positive distances. On the other hand, since Z must lie in spherical distribution, the other samples should be "moved" and rearranged in order to satisfy constraint (see Fig.
Batch Slicing. The estimation of the Mean Square Error in equation 6 depends on the whitening matrix Wv, which may have a high variance over consecutive iteration batches V,, Vi+1.... For this reason, inspired by the resampling methods (Efron 1982), given batch V, we slice V in different non-overlapping sub-hatches and we compute a whitening matrix independently for each sub-batch. In more details, we first partition the batch in parts, being the number of positives extracted from one image. In this way, each partition contains elements extracted from different original images (i.e., no pair of positives is included the elements of the each partition with the same permutation. Next, each partition is further split in aub-batches, using the heuristic that the size of each sub-batch (V;) should be equal to the size of embedding (v) times 2 (this prevents instability issues when computing the covariance matrices). Next, for each V,, we use only its elements compute a corresponding whitening matrix Wi. which is used to whiten the elements of Vi only (Fig. 3) In the loss computation (equation (6). all the elements of all the sub-batches are used, thus implicitly alleviating the differences among the different whitening matrices. Finally, it is possible repeat the whole operation several times and to average the result to get a more robust estimate of equation 6
single partition, see Fig. B). Then, we randomly permute
3.1 DISCUSSION
In a common instance-disccrimnnatio task (Sec. 2 e.g., solved using equation 2 the similarity of a positive pair (z. 22)) contrasted with the similarity computed with respect to all the other samples (zx) in the batch (z,'2a, 1 < K, + i). However, 2 and 2,, extracted from different image instances, can occasionally share the same semantics (c.g.. and Iy are two different image instances of the unknown "cat" class). Conversely, the proposed W-MSE loss does not force all the instance samples to lie far from each other, but it only imposes soft constraint (cquation 4 which avoids degenerate distributions.
Note that previous work (He et al. 2019 Hénaff et al. 2019 Chen et al., 2020 highlighted that BN may be harmful for learning semantically meaningful representations because the network can "cheat" and exploit the batch statistics in order to find a trivial solution to equation 2
However, our whitening transform (equation 7 applied only to the very last layer the network f(() (see Fig. 2) and it is not used in the intermediate layers, which is instead the case of BN. Hence, our f() cannot learn to exploit subtle inter-sample dependencies introduced by batch-statistics because of the lack of other learnable layers top of the features.
Similarly to equation 6 in BYOL Grill et al. 2020) an MSE loss is used to compare the latent rep- resentations of two positives computed by slightly different networks without contrasting positives with negatives (Sec. 2). However, the MSE loss alone inclined to collapse the representations of all the images to constant value, which would make the MSE computation equal to zero. In BYOL,

---

Under review as a conference paper at ICLR 2021
both the projection and the prediction sub-networks have BN layers, and, very recently, (Fetterman & Albrechtt 2020; Tian et al. 2020) have empirically shown that BYOL, without these BN layers, generates collapsed latent-space representations with close-to-chance level classification accuracy. The reason of this behaviour seems to depend on the fact that the feature standardization in BN scatters the z values in batch and avoids constant representations. Our W-MSH can be seen as a generalization of this implicit property of BYOL, in which the z values of the current batch are full-whitened, so preventing possible collapsing effects of the MSE loss. Importantly, we reach this result without the need of a target network or sophisticated training protocols.
Finally, note that using BN alone without whitening, as W-MSE, and without additional networks, as in BYOL, is not sufficient. Indeed, if we just minimize an MSE after feature standardization, the network can easily find a solution where all the dimensions of the embedding represent the same feature. We have empirically verified this behaviour in preliminary experiments based on standardization, in which the network converges zero loss value after a few epochs but with a low classification accuracy.
EXPERIMENTS
We test our loss and its competitors on the following datasets.
CIFAR-I0 and CIFAR-100 (Krizhevsky & Hinton 2009), two small-scale datasets com- posed of 32> x 32 images with 10 and 100 classes, respectively.
Tiny ImageNet (Le & Yang 2015), a reduced version of ImageNet, composed of 200 classes with images scaled down 64 x 64. The total number of images is: 100K (training) and 10K (testing).
STL-10 (Coates al. 2011), also derived from ImageNet, with 96 x 96 resolution images. While CIFAR-TO, CIFAR-TO0 and Tiny ImageNet are fully-labeled, STL-10 is composed of 5K labeled training samples (500 per class) and 100K unlabeled training examples from a similar but broader distribution of images. There are additional 8K labeled testing images.
ImageNett100.. a random 100-class subset of ImageNet (the list of the 100 classes is pub- lished in (Wang & Isola 2020)), consisting of unaltered ImageNet images.
Setting. The goal of our experiments is to compare W-MSE with ttateoof-the-at losses, isolating the effects of other settings, such as the architectural choices. For this reason, we use the same encoder E()) ResNet-18 18 for all the experiments. We independently select the hest hyperparameter values for every method and every dataset. Each method uses L2 feature normalization unless otherwise stated. Contrastive refers to our implementation of the contrastive loss (equation 2 following the details in (Chen et al., 2020), with temperature т = 0.5. BYOL is our reproduction of (Grill et al. 2020), introduced concurrently with our work. For this method we the exponential moving average with cosine increasing, starting from 0.99. W-MSE 2 and W-MSE 4 correspond to our method with d 2 and d 4 positives extracted per image, respectively. For CIFAR-10 and CIFAR-100, the slicing sub-batch size is 128, for Tiny ImageNet and STL-10, it is 256. For experiments W-MSE 2 for Tiny ImageNet and STL-10 we use iterations of batch slicing, for all other experiments we use iteration.
In all the experiments, we use the Adam optimizer (Kingma & Ba 2014). For all the tested methods (including ours), we use the same number of epochs and the same learning rate schedule. Specif- ically, for CIFAR-10 and CIFAR-100, we use 1,000 epochs with learning rate 3 x 10 3, for Tiny ImageNet, 1,000 epochs with learning rate 2 x 10-3; for ST1.-10, 2,000 epochs with learning rate 10-3 We use learning rate warm-up for the first 500 iterations of the optimizer, and 0.2 learning rate drop 50 and 25 epochs before the end. We use mini-batch size of K = 1024 samples. The dimension of the hidden layer of the projection head g()) is 1024. The weight decay is 10-6, 6 Finally, we use embedding size of 64 for CIFAR-10 and CIFAR-100, and an embedding of size of 128 for STL-10 and Tiny ImageNet. For ImageNet- 100 we use a configuration similar to the Tiny ImageNet experiments, and 240 epochs of training.
As a common practice when using ResNet-like architectures for small-size image resolutions, in all the experiments, except ImageNet-100 we have a first convolutional layer with kernel size 3,
7

---

Under review as a conference paper at ICLR 2021
Table 1: Classification accuracy (top 1) of linear classifier and 5-nearest neighbors classifier for different loss functions and datasets with ResNet-18 18 encoder.
Method
CIFAR-10 10
CIFAR-100
STL-10
Tiny ImageNet
linear
5-nn
linear
5-nn
linear
5-nn
linear
5-nn
Contrastive
91.80
88.42
66.83
56.56
90.51
85.68
48.84
32.86
BYOL
91.73
89.45
66.60
56.82
91.99
88.64
51.00
36.24
W-MSE 2
91.55
89.69
66.10
56.69
90.36
87.10
48.20
34.16
W-MSE 4
91.99
89.87
67.64
56,45
91.75
88.59
49.22
35.44
Table 2: Classification accuracy on ImageNet-100. W-MSE (2 and 4) are based on a ResNet-18 encoder. indicates that the results are based on ResNer-50 encoder and the values are reported from (Wang & Isola 2020).
Method
linear (top 1)
linear (top 5)
5-nn
MoCo
72.80
91.64
Lalign and Lunitorm f
74.60
92.74
W-MSE 2
76.00
93.14
67.04
W-MSE 4
79.02
94.46
71.32
stride and padding Additionally, in case of CIFAR-10 and CIFAR-100, we remove the first max pooling layer.
Image Transformation Details. We extract crops with a random size from 0.2 to 1.0 of the original area and a random aspect ratio from 3/4 to 4/3 of the original aspect ratio, which is a commonly used data-augmentation technique. We also apply horizontal mirroring with probability 0.5. Finally, apply color jittering with configuration (0.4, 0.4, 0.4, 0.1) with probability 0.8 and grayscaling
with probability 0.1. For ImageNet-100 we follow details in (Chen et al. 2020): crop size from 0.08 1.0, stronger jittering (0.8, 0.8, 0.8, 0.2), grayscaling probability 0.2, and Gaussian blurring with 0.5 probability.
Evaluation Protocol. The most common evaluation protocol for unsupervised feature learning is based on freezing the network encoder (E(-), in our case) after unsupervised pre-training, and then train a supervised linear classifier on top of it. Specifically, the linear classifier is fully-connected layer followed by softmax, which is placed on top of F(-) after removing the projection head g(-). In all the experiments we train the linear classifier for 500 epochs using the Adam optimizer and the labeled training set of each specific dataset, without data augmentation. The learning rate is exponentially decayed from 10-2 to 10 The weight decay is 5 10-6
In our experiments, we also include the accuracy of k-nearest neighbors classifier (k-nn, k: 5). The advantage of using this classifier is that does not require additional parameters and training, and it is deterministic.
4.1 COMPARISON WITH THE STATE OF THE ART
Tab. shows the results of the experiments on small and medium size datasets. For W-MSE, samples are generally better than The contrastive loss performs the worst in most cases, The W-MSE 4 accuracy is the best on CIFAR-10 and CIFAR-100, while BYOL leads on STL-10 and Tiny ImageNet, although the gap between the two methods is minor. Appendix A we plot the linear classification accuracy during training for the STL-10 dataset. The plot shows that W-MSE and BYOL have a similar performance during most the training. However, in the first 120 epochs, BYOL significantly underperforms W-MSE (e.g., the accuracy after 20 epochs: W-MSE 4, 79.98%: BYOL, 73.24%). indicating that YOL requires a "warmup" period. On the other hand, W-MSE performs well from the beginning. This property is useful in those domains which require rapid adaptation of the encoder, e.g., due to the change of the data distribution in continual learning or in reinforcement learning.
8

---

Under review as a conference paper at ICLR 2021
Tab. shows the results a larger dataset (ImageNet-100) 100). In that table, MoCo is the contrastive- loss based method proposed in (He et al. 2019). where a momentum encoder and a large queue of negatives are used to improve the contrast of the positive pairs with respect to the other samples (see Sec. 2). Lalign and Luniform are the two losses proposed in (Wang & Isola 2020) (Sec. 2 Note that, while W-MSE (2 and 4) in Tab. refer our method with ResNet-18 encoder, the other results reported from (Wang & Isola, 2020), where a much larger-capacity network (i.e., a ResNet- is used the encoder. Despite this large difference in the encoder capacity, both versions of W-MSE significantly outperform the other two compared methods in this dataset.
4.2 TRAINING TIME COMPLEXITY
Following (Siarohin et al. 2019), the complexity of the whitening transform is +Mk++)),,, where k is the embedding dimension and M is the size of the sub-batch used in the batch slicing process. Since < M (see Sec. 33. the whitening transform is O(Mk²), which is basically equivalent to the forward pass of M activations in a fully-connected layer connecting two layers of k neurons each. In fact, the training time is dominated by other architectural choices which are usually more computationally demanding than the loss computation. For instance, BYOL (Grill et al.) 2020) needs 4 forward passes through 2 networks for each pair of positives. Hence, to evaluate the wall- clock time, we measure the time spent for one mini-batch iteration by all the methods compared in Tab. [ We use the STL-10 dataset, ResNet-18 encoder and a server with one Nvidia Titan Xp GPU. Time one iteration: Contrastive - 459ms, BYOL- 602ms, W-MSE
478ms, W-MSE 4- 493ms. The 19ms difference between Contrastive and W-MSE 2 is due to the whitening transform. Since the factual time is mostly related to the sample forward and backward passes, the d(d 1) positive comparisons in equation 6 do not significantly increase the wall-clock time of W-MSE with respect to W-MSE 2.
4.3 CONTRASTIVE LOSS WITH WHITENING
Table Accuracy of the whitened contrastive loss on CIFAR-10 trained for 200 epochs.
Method
linear 5-nn
Contrastive
89.66
86.55
Contrastive with Whitening
diverged
Contrastive, unnormalized features
79.48
76.60
Contrastive with Whitening, unnormalized features
77.39
74.14
In this section, we analyse the effect of the whitening transform in combination with the contrastive loss. Tab. 3 shows the results. The first row refers to the standard contrastive loss. Note that the difference with respect to Tab[[iis due to the use of only 200 training epochs. The second row refers to equation where the features (z) are computed using equation and then Le normalized, while in the last two rows, z is not normalized. If the features are whitened and then normalized, we observed an unstable training, with divergence after few epochs. The unnormalized version with whitening converged, but its accuracy is worse than the standard contrastive loss (both normalized and unnormalized). This experiments show that whitening itself does not improve the performance. but it only allows to satisfy the constraint 4
5 CONCLUSION
In this paper, we have proposed a new self-supervised representation learning loss, W-MSE, which alternative to common loss functions used in the field. Differently from the triplet loss and the contrastive loss, both of which are based on comparing samples, W-MSE computes only the intra-positive distances, while using a whitening transform to avoid degenerate solutions. Despite W-MSE is very simple, its classification accuracy is comparable with state-of-the-ars are methods, achieving results significantly higher than MoCo, which requires an additional momentum encoder and a large queue of past samples. W-MSE is also comparable with B BYO, which needs an additional target network and a specific training protocol. We believe that the use of whitening to avoid collapsing effects can inspire other self-supervised methods.
instance-level similarity against other

---

petrrassive-lerriingg... Under review as a conference paper at ICLR 2021
REFERENCES
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In NeurIPS, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments, 2020.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In AISTATS, 2011.
Dariusz Dereniowski and Kubale Marek. Cholesky factorization of matrices in parallel and ranking of graphs. In 5th Int. Conference Parallel Processing and Applied Mathematicx, 2004.
Jacob Devlin, Ming-Way Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurlPS, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In /CLR, 2017.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem- poral cycle-consistency learning. CVPR, 2019.
Bradley Efron. The jackknife, the bootstrap. and other resampling plans, volume 38. Siam, 1982.
Abe Fetterman and Josh Albrecht. Understanding self-supervised and contrastive learning with bootstrap your latent (BY).). https:/huntitleddd www.ttt...ounnereeeetoeee seaiinng-ss-----eeeeee self-vu-er 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A approach to self-supervised learning, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. unsupervised visual representation learning, arXiv:1911.05722 2019.
Momentum contrast for
Olivier Hénaff, Ali Razavi, Carl Doersch, M. Ali Eslami, and Aäron van den Oord. Data- efficient image recognition with contrastive predictive coding. arXiv: /905.099000 2019.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person identification. arXiv: / /003 2017.
R. Devon Hjelm, Alex Fedorov, Samuel aavoie-Marchildee Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In /CLR, 2019.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. CVPR, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
10

---

Under review as a conference paper at ICLR 2021
Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. 100.33130 2013a.
Tomas Mikolov, Ily Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013b.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. arXiv: /9/2v/////.///// /9/2.0191, 2019.
Ishan Misra, C. Lawrence Zitmick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. CVPR, 2016.
Mirco Ravanelli and Yoshua Bengio. Learning speaker representations with mutual information. arXiv:1812.00277, 2018.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face recognition and clustering. In CVPR, 2015.
Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening and coloring transform for GANs. In International Conference on Learning Representations, 2019.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. NIPS, 2016.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv:2010.00577. 2020.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. 1rXiv:9907.1625,, 2019.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. 1807 2018
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, 2020.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. arXiv: 800.00197, 2018.
11

---

Under review as a conference paper at ICLR 2021
A
TRAINING DYNAMICS
Fig.
and show the training dynamics for each of the considered losses. Charts are smoothed with 0.3 moving average for readability (curves before smoothing shown semi-transparent)
accuraey
0.9
BYOL W-MSE W-MSE 2
0.85
Contrastive
0.8
0.75
epoch
1k
1.5k
Figure 4: Training dynamics on STL-10 dataset for linear classifier
accuracy
0.85
BYOL W-MSE 4
W-MSE Contrastive
0.7
epoch
500
1.5k
Figure 5: Training dynamics on STL-10 dataset for 5-nn classifier
B CHOLESKY WHITENING AND BACKPROGATION
We compute Wy (equation K) following SSarohhin et al. 2019 and using the Cholesky decomposi- tion. The Cholesky decomposition is based the factorisation of the covariance symmetric matrix using two triangular matrices: Σv = LL where a lower triangular matrix. Once we get L, compute the inverse of L, and we get: Wy - Note that Cholesky decomposition is fully diferentiable and it is implemented in all of the major frameworks, such as PyTorch and TensorFlow. However, for the sake of completeness, provide below the gradient computation.
12

---

Under review as a conference paper at ICLR 2021
B.1 GRADIENT COMPUTATION
We provide here the equations for whitening differentiation. Let Z be the whitened version of the batch V, i.e., Z Wv(V (v) (equation
The gradient 0T. can be computed by: 0V
ØL 2
0L + W
OL 0z
(9)
Qv K
10s
where the partial derivative at is backpropogated, while aL is computed as follows: oz
al 1 ---2-? w7 Po のΣ
AL dwv
w?) ) Wv w + po Owr
aL
(10)
In equation 10 is Hadamard product, while 0L JWv
is:
0L
0L VT.
(11)
dwv
az
and P is:
0
2
1
0
P
2
1
0
1
C EUCLIDEAN DISTANCE
Table 4: Classification accuracy (top 1) using the Euclidean distance (unnormalized embeddings) on STL-100
Method
linear
5-nn
Contrastive
78.00
71.07
BYOL
80.83
74.94
W-MSE 2
89.91
85.56
W-MSE 4
90.40
87.09
The cosine similarity is crucial component in most of the current self-supervised learning ap- proaches. This is usually implemented with an L2 normalization of the latent representations, which corresponds to projecting the features the surface of the unit hypersphere. However, in W- MSE, the whitening transform projects the representation onto a spherical distribution (intuitively, say the whole unit hypersphere). Preserving the module of the features before the Ls normalization may be useful in some applications, e.g., clustering the features after the projection head using Gaussian mixture model. Tab. shows an experiment on the STL-10 dataset where we use unnormalized embeddings for all the methods (and = 1 for the contrastive loss). Comparing Tab. with Tab. I the accuracy decrease of W-MSE significantly smaller than the other methods.
13


Reviews:
[{"review_id": "zjeBkkuNAcT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an MSE loss function with whitening (W-MSE) for self-supervised representation learning. The motivation is to reduce the demand of negative examples in contrastive representation learning. The proposed W-MSE loss function is compared with popular contrastive loss on a few benchmarks.\n\nContrastive learning is a popular topic in the self-supervised learning domain. Most of the existing methods rely on negative samps to avoid trivial solutions. This paper proposed a simple and clean solution to tackle this problem, by using whitening in the loss term. This paper is also well explained and illustrated.\n\nThe main weakness of the paper is the experiments. Based on the results, I am not convinced that the proposed W-MSE is effective. Here are more comments:\n\n(1) The experimental results are pretty close to the existing contrastive/BYOL baselines. On CIFAR-10 and STL-10, results are saturated thus the diff is very minor. On more challenging CIFAR-100 and Tiny ImageNet, the results are mixed when compared to BYOL. \n\n(2) Given that the results are very close, what are the other benefits of using W-MSE loss? For example, is the training speed faster than the other methods (contrastive, BYOL) with negative sampling? It would be nice to include such results to demonstrate the effectiveness of W-MSE.\n\n(3) This paper claims that without negative sampling, W-MSE loss encourages the use of more positive pairs in the batch. I'm wondering if the authors have tried more positive pairs beyond 4 in the paper.\n\n(4) I cannot understand the motivation of ablating the popular methods with Euclidean distance. Cosine similarity is just simple and it costs nothing compared to Euclidean distance. I think ablating this from existing contrastive loss is not sufficient to show the effectiveness of BYOL. \n \n(5) It would be nice to include a few comparable numbers from literature directly, instead of reproducing their methods. For example, this paper used ResNet18 while the BYOL paper used ResNet50. This paper reports results on Tiny ImageNet while the existing methods report on ImageNet. Note that I do not penalize this paper for this point, as computational cost is high for using larger architecture and larger dataset. But it would be good to have a comparable baseline in the middle ground, e.g. ResNet50 architecture on CIFAR-100 dataset, where BYOL reports 78.4% top-1 accuracy with linear eval. \n\n======Post Rebuttal Update======\n\nI would like to thank the authors for their rebuttal, which has addressed part of my concerns. After reading the authors' rebuttal and other reviewers' comments, I'm still concerned on the weak baselines and mixed results in this paper. Unfortunately, I will keep my rating.\n\nConcrete suggestions to improve this paper in the future:\n\n(1) Strongly recommended: use ResNet50 instead of ResNet18 for the small scale experiments, in this way you get directly the numbers from the literature (e.g. BYOL on CIFAR-100);\n\n(2) Nice to have: for the expensive ImageNet experiments, it would be nice to get comparable results using the smallest comparable architecture (e.g. ResNet50) from the literature. SimCLR claims that \"With 128 TPU v3 cores, it takes ∼1.5 hours to train our ResNet-50 with a batch size of 4096 for 100 epochs\" and MoCo claims that \"For IN-1M, we use a mini-batch size of 256 in 8 GPUs, ..., train for 200 epochs ..., taking ∼53 hours training ResNet-50.\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "svow4IP1Cl0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the mean square error loss with whitening operation to project positive pairs closely to each others while projecting the different positive pairs far away from each other on a unit sphere. This way, similar to BYOL, this paper removes the construction of negative pairs while improving the MoCo-V2 slightly on not very challenging benchmarks. The authors can find my questions/comments in the list below.\n\n1. What does gray and blue squares represent in figure 3? I believe figure 3 can be revisited for the sake of better understanding the method. For example, I was not able to understand why v_i and v_i+8 represent the same image. In this current shape, it creates confusion rather than helping for clarity.\n\n2. In the Batch Slicing section, the authors mention that the size each of sub-batch should be close to the size of the embedding x 2. Does this make sense? Is the size of embedding 512? If yes, do you have 1024 samples in a sub-batch? I would be happy if the authors can comment on this.\n\n3. I think my biggest concern with the paper is the lack of extensive experiments. All the experiments are done on small-scale datasets which is highly questionable when they are used for unsupervised learning. It would be much more convincing to have experiments on ImageNet which is a standard experiment for unsupervised learning.\n\n4. The improvement on relatively less challenging benchmarks are very marginal. Even in some cases, CIFAR100, contrastive loss does better than the proposed one. And the proposed method only outperforms BYOL in CIFAR10.\n\n5. My impression is that the contrastive method in table 1 and 2 represent the MoCo-v2. I would replace it with MoCo-v2 in these tables as there are many other methods that uses contrastive loss functions.\n\n6. Another problem with the experiments is that, they lack experiments on different downstream tasks such as object detection. \n\n7. The authors try the Euclidean and MSE distance to project positives closely. Did they consider the cosine similarity loss? I did not understand the point of using Euclidean distance experiments without normalization? What do they exactly prove? And we can always normalize the embeddings as it is done in the other methods. I would be happy to receive some comments on this from the authors.\n\n8. Finally, it would be nice to the advantages of the proposed method (removal of negatives) in terms of training complexity. Does it, as expected, reduce the complexity in the training time? If yes, quantifying it would increase the strength of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TvReA43uQrQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to first do representation \"whitening\", so that the representations are scattered in the space and not collapsing to a single data point; then compute distance metric on top of that (e.g. Euclidean, cosine similarity). A nice thing about explicit scattering is that it does not require large numbers of negative examples to pull the features apart. Experiments are done on several toy datasets like CIFAR.\n\n+ The paper is a nice, alternative attempt to remove negative examples in contrastive learning. Indeed, large number of negative examples is annoying and this direction is both exciting and significant.\n+ The approach proposed in this paper intuitively makes sense. There are several works already explaining that contrastive learning is essentially doing some kind of scattering in the space. \n+ The proposed approach seems pretty simple. The whitening code is only a dozen lines in PyTorch. I haven't run the code to verify the results though.\n\nI think the experiments are not too satisfying though. \n\n- The comparison is less fair between BYOL and multi-crop version of W-MSE. In SwAV, it shows that with multiple crops, the performance can be boosted quite a bit. I haven't tried on BYOL but I believe it could also be helping there. So the most apple-to-apple comparison between BYOL and W-MSE is the 2-crop version (d=2). In this case, BYOL is outperforming in most entries. Though it can be viewed as concurrent work (I think W-MSE is actually even earlier than BYOL), but the experiment session in the paper is not clear about this.\n- Overall running experiments on these toy datasets are less satisfying, not only because it lacks comparison to other major approaches (like MoCo on ImageNet), but also because the signal we get from smaller datasets may not transfer well to more real-world images. \n- I would like to see a comparison in terms of timing -- maybe BYOL (because of its momentum encoder and it needs 4 forward pass of the network to compute a single pair of losses) is running much smaller in training. W-MSE can run much faster because it only needs 4 (or even 1?) forward pass. This is a potential advantage that W-MSE has, but it is not clear from the paper.\n\nOther than experiments, I am also not too satisfied with the writing. The paper mentions another paper when talking about the key method (how to do whitening, e.g. which is back propagated, which is not) when describing the central technique of the paper. I would like to see the paper more self-contained in the next version. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "1h488jkojc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper provides a simple self-supervision loss function which is computed using only positive samples. In particular, the proposed loss function is based on the whitening of a latent space-features. Results show competitive but not better results than other methods. \n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I vote for accepting. The paper proposes a simple loss function based on whitening and the MSE loss. The method does not need negative samples to contrast with the positives since the whitening process shrinks the overall samples to satisfy the spherical-distribution while the MSE loss attracts the positive ones. Although I have some concerns with the validation I really like the method.\n\n \n##########################################################################\n\nPros: \n\n \n1. Interesting and simple method.\n\n2. Clarity of the paper.\n \n \n##########################################################################\n\nCons: \n\n1. The validation \n \n-- The evaluation has been carried with small and simple datasets (CIFAR,STL and Tiny ImageNet). BYOL authors presented an extensive experiment setup with several datasets. Why not use the same (at least some) databases and compare them with their reported results on the paper (and not use your own implemented version of BYOL)? In fact, some of those datasets are already used in this paper but as you use ResNet-18 instead of ResNet-50 (used in BYOL paper) the results are not comparable. \n\n-- Why not ResNet-50 instead of ResNet-18. Validation would be easier with other published methods\n\n-- W-MSE with d=4 seems to work better than d=2. What about larger d?\n\n-- Although the authors point out that the proposed MSE loss does not need negative samples, negative samples are needed in the whitening transformation. \n\n-- It would be great to decouple the system performance. What is the system performance using Withening with the classical contrastive loss? Which is the improvement provided by the MSE loss over the classical contrastive loss?\n\n-- The authors say that Contrastive Loss needs large numbers of negative samples. How many negative samples were used in the experimental setup? The proposed method is evaluated with d=2 and d=4. Which is the positive/negative proportion of samples in the batches using the contrastive loss? How was this proportion selected?\n\nMinor Comments:\n- Please check: \"On the other hand, Hjelm et al. (2019) have shown that the contrastive loss needs a large number of negatives to be competitive\"\n\n\n \n##########################################################################\n\nQuestions during the rebuttal period: \n\n \nPlease address and clarify the cons above \n\n \n#########################################################################\n\nSome typos: \n\n(1) Page 4: matrix and (z_i,z_j) correspond -> corresponds\n(2) Page 7: jitterering --> jittering\n\n\nI think that the paper\n\n\nUPDATE AFTER REBUTTAL:\nThe authors have covered most of my concerns about the paper and I think that the paper has been substantially improved. However, my biggest concern was about the experiment results and  I think the paper still lacks on validation comparison with other methods. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received borderline recommendations (5, 5, 6, 7) but even the two slightly more positive reviewers were lukewarm (R1 and R2). While the reviewers acknowledged the heavy computational requirements to do an apples-to-apples comparison with existing baselines, they remain underwhelmed with the lack of experiments. I agree with their criticism; even though the proposed idea seems promising, without comprehensive experiments, it is difficult to judge the significance of this work. R1 commented after the discussion period that an earlier version of this paper actually had ImageNet results. R4 made excellent suggestions to improve the paper further. The authors are strongly encouraged to incorporate them into their future submission.

(I am copying R1's comment below in case it is invisible to the authors after the notification.)

Sorry for the late update -- I have read the rebuttal earlier. I would like to keep my acceptance rating but after the rebuttal I am fine either way. The paper first appeared in March on ArXiv, so indeed it is a concurrent work (actually an earlier work compared to BYOL or SwAV). We have actually tried to reproduce the results in the paper a while back but it did not go well (could not reproduce it), but this time the submission also includes the code. While I haven't run it, I trust the results are reproducible (maybe there are some tricks that I am not aware of).

Regarding running experiments on toy examples -- I can understand that this research is resource-constrained for ImageNet, but the earlier draft actually had some results on ImageNet (60+ top-1 accuracy) (see appendix of https://arxiv.org/pdf/2007.06346v1.pdf), and for some reason this submission removed that. So this is not a positive sign. Overall, my experience for CIFAR vs ImageNet is that it is easier to make things work on CIFAR, while it is much harder to do so on ImageNet. So maybe some trials are indeed done by the authors, but they choose to not report it in the submission for some reason. On the other hand, one can argue that results on toy datasets are good enough contributions for an early develop of something and they are just not ready for larger and more challenging datasets yet.

Therefore, this paper is quite a struggle. I hoped to see a better-than-this submission as this paper actually had all the time from March to October to improve its quality of experiments (actually even for ImageNet, one can to dozens of cycles on it during this time), but it did not for some reason.
Decision: Reject