Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
Authors: Grégoire Mialon
Randall Balestriero
Yann LeCun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised learning
VICReg
Barlow Twins
HSIC

Abstract: Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance matrix of their projector’s output. This study highlights important properties of such strategy, which we coin Variance-Covariance regularization (VCReg). More precisely, we show that VCReg enforces pairwise independence between the features of the learned representation. This result emerges by bridging VCReg applied on the projector’s output to kernel independence criteria applied on the projector’s input. This provides the first theoretical motivations and explanations of VCReg. We empirically validate our findings where (i) we put in evidence which projector’s characteristics favor pairwise independence, (ii) we use these findings to obtain nontrivial performance gains for VICReg, (iii) we demonstrate that the scope of VCReg goes beyond SSL by using it to solve Independent Component Analysis. We hope that our findings will support the adoption of VCReg in SSL and beyond.

Under review as a conference paper at ICLR 2023
VARIANCE COVARIANCE REGULARIZATION ENFORCES PAIRWISE INDEPENDENCE IN SELF-SUPERVISER REP- RESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W- MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance matrix of their projector's output. This study highlights important properties of such strategy, which we coin Mariance-Covarian show that VCReg combined to MLP
regu-
larization (VCReg). More precisely, projector enforces pairwise independence between the features of the learned repre- sentation. This result emerges by bridging VCReg applied on the projector's output to kernel independence criteria applied on the projector's s input. This provides the first theoretical motivations and explanations of MLP projectors in SSL. We empirically validate our findings where (i) we put in evidence which projector's characteristics favor pairwise independence, (ii) we use these findings obtain nontrivial performance gains for VICReg, (iii) we demonstrate that the scope of VCReg goes beyond SSL by using it to solve Independent Component Analysis. We hope that our findings will support the adoption of VCReg in SSL and beyond.
INTRODUCTION
Seff-Supervised Learning (SSL) via joint embedding architectures has risen learn visual represen- tations outperforming their supervised counterpart. This paradigm enforces similar embeddings for two augmented versions of the same sample, thus allowing an encoder learn representation for a given modality without labels. Importantly, f could solve the learning task by predicting the same embedding for every input, a failure mode known as collapse. To avoid this, various mechanisms have been proposed hence the diversity of SSL methods (e.g.. Grill et al. (2020); Caron et al. (2020; 2021); Chen & He (2021)). Most of these are a composition g o f of the encoder with a projector neural network g. Only f is retained after training, in opposition supervised training that never introduces g. The projector was proposed by Chen et al. (2020) and significantly improved the quality of the learned representation in terms of test accuracy Although some works (Appalaraju et al., 2020; Bordes et al., 2022) provide empirical knowledge on the projector, none provide theoretical analysis of MLP projectors in practical SSL (Jing et al., 2021; Huang et al., 2021; Wang & Isola, 2020; HaoChen et al., 2021; Tian et al., 2020; Wang & Liu, 2021; Cosentino al., 2022).
ImageNet and other downstream tasks.
This study sheds a new light on the role of the projector via the lens of Variance-Covariance Regular- ization (VCReg), strategy introduced in recent SSL methods (Bardes et al., 2022; Zbontar et al., 2021; Ermolov et al., 2021) to cope with collapse by constraining or regularizing the covariance or cross-correlation of the projector g output be identity. More precisely, we demonstrate that VC regu- larization of the projector's output precisely enforces pairwise independence between the components of the projector's 's input i.e. the encoder's 's output, and connects this property to projector's character- istics such as width and depth. This provides the first theoretical motivations and explanations of MLP projector in SSL. Fully or partially pairwise independent representations are generally sought for, e.g to disentangle factors of variation (Li et al., 2019; Träuble et al., 2021). Our experimental analysis suggests that different levels of pairwise independence of the features in the representation emerge from a variety of SSL criteria along with mutual independence. However, opposed to other frameworks, VCReg allows for theoretical study and explicit control of the learned independence amount. We prove and experimentally validate this property for random projectors, study how it

---

Under review as a conference paper at ICLR 2023
emerges in learned projectors, and use our results to obtain new and significant performance gains VICReg over Bardes et al. (2022). We then ablate the SSL context and lean on our findings to show that VCReg of a SSL projector solves Independent Component Analysis (ICA). Beyond providing a novel theoretical understanding of the projector, we believe that this work also leads to a better understanding of VICReg. The scope of VCReg is not limited SSL.: our experiments ICA open the way to other applications where some degree of independence is needed.
2 BACKGROUND
2.1 MEASURING STATISTICAL. INDEPENDENCE USING KERNEL METHODS
(X]...,, the independence between two sets of realizations X2 } E IR IM,X;; E RM is fundamental task that has long history in statistics e.g. through the Mutual Information (MI) of the two random variables X1 and X₂ from which those two sets are independently drawn from (Cover, 1999). Those variables are said independent the realization of one does not affect the probability distribution of the other. Computing the MI in practice is known to be challenging (Goebel et al., 2005), which has led to considerable interest in using alternative criteria e.g. based on functions in Reproducing Kernel Hilbert Spaces (RKHS) (Bach & Jordan, 2002), a special case of what is known functional covariance or correlation (Rényi, 1959). It consists in computing those statistics after nonlinear transformation (Leurgans et al., 1993) in
Measuring
sup /.FFF
where f1, fa are constrained to lie within some designed functional space, and the Cov can be used instead of the Corr. If Eq. (1) is small enough, then Xı and X2 are independent in regard to the functional spaces Fi, F2. For example, Fi and F22 are unit balls in their respective vector spaces, then Eq. (1) is just the norm of the usual correlation/covariance operator (Mourier, 1953) which would be enough for independence under joint Gaussian distributions (Melnick & Tenenbein, 1982).
HSIC and pairwise independence. More recently, Gretton et al. (2005a) introduced a pairwise independence criterion known as the Hilbert-Schmide Independence Criterion (HSIC) which can be estimated given empirical samples X; and X2 E RXXM as
HSIC(X, x2) HSSICH,, = (N 1 1)2 Tr( HK,H),
(2)
with H the centering matrix I 11 1. (Ki)a, ki (Xi, x and (K2)sJ = k2 (X X2) the two kereel matrices of X1 and X2 respectively, and k1, ly of F2 universal kernels such as the Gaussian kernel (see Steinwart (2001); Micchelli al. (2006) for other examples). Crucially, since
N.
HSIC(X,, X2) 2 sup f1EF
Cov (X1)
HSIC can be used to test for (pairwise) independence as formalized below.
Theorem 1 (Thm. 4 from (Gretton et al., 2005a)). HSIC(X,, X2) = 0 and only if X, and X2 are independent.
Gretton et al. (2005a) also provide a statistical test for pairwise independence based on HSIC. Further quantities such as upper bounds on the MI can be found in a similar way. e.g. see Thm. 16 in Gretton et al. (2005b). In our experiments, we will rely on HSIC under the Gaussian kernel scaled by the median of the distribution of pairwise euclidean distances between samples.
dHSIC and mutual independence. Mutual independence of a set of D M M-dimensional random variables XI,..., XD is a stronger property than independence between all pairs of random variables the set. To evaluate it, Pfister et al. (2018) introduce dHSIC, multivariate extension of HSIC. In
short, dHSIC measures the distance between mean embeddings a under RKHS F (Smola et al., 2007) of the product of distributions and the joint distribution:
dHSIC(X,,.... ..p)......)))) == ( @ ... & xND) - ( (.......... XD)ILE
(4)
Similarly to HSIC, Pfister et al. (2018) establish the equivalence between dHSIC = 0 and mutual independence along with statistical test. We provide estimator of dHSIC given empirical samples X,..... Xn of the above random variables in IR Nx each as well as implementations of HSIC and dHSIC in Appendix D.
2

---

Under review as a conference paper at ICLR 2023
Complexities. In what follows, consider the D features of batch of representations X RNx as D scalar random variables (M = 1) with N samples each. The subsequent complexities of HSIC for one pair of features and dHSIC are (NN2) and (DD22) respectively. Testing pairwise independence between all variables in a D-set with HSIC is O O() while dHSIC requires N 2D (Pfister et al., 2018). Since competitive visual representations typically have D 2048, resort to surrogates to estimate independence of the representations in practice. These surrogates are detailed in Section 5.
2.2
CORIANCC-COVARIANEE REGUL
SELF-SUPERVISER LEARNING
SSL with joint embeddings learn visual representations by producing two different augmented views of an input batch of images S E R Rx denoted by Sleft and Sright- Each view is fed to an encoder f. typically ResNet50 (He et al., 2016), producing representations Xien and righht E RNxD which are passed through a projector g to output embeddings Zief and Zright E RNPP. Finally, an invariance term encouraging Zieft and Zright be similar is applied. While most SSL methods require architectural or training strategies to avoid collapse (Grill et al., 2020; He et al., 2021), Bardes et al. (2022) and Zhontar et al. (2021) only require to modify the loss. After training, only f is retained to be used in downstream tasks. We will denote the (2N, P) matrix Zrotal [ 1
VICReg. In Bardes et al. (2022), an anti-collapse term Lvc, which we coin VC regularization (VCReg), is added an invariance loss form Lvic:
(0, VCov( Zrotal)k& + a Σ Cov Cov(Zrotal)jn
Lvc
max
0,
j=1
LVIC
(1 (Zen)) (Zright) )n.
LvC,
(6)
N
The leftmost term in Lve corresponds to regularizing the variance each feature in Ziotal to be at least unit, while the second term seeks to minimize the covariance between each pair of features in Ziotal- Note that Lvc applies to each view separately. In Bardes et al. (2022), the weight of each term in Lvic can be tuned. Since the authors find that best results are obtained with equal weights for the invariance and the variance terms, we only vary a. Note that Bardes et al. (2022); Zbontar al. (2021) observe that wider projectors further improve the representation learned by the encoder f. yet neither further study this intriguing phenomenon. Although this work focuses on VCReg as formulated in VICReg, we provide background on Barlow Twins and W-MSE in the Appendix A, while next section establishes the similarity between VICReg, Barlow Twins and W-MSE as VCReg optimizers.
VC REGULARIZATION OF SSL PROJECTOR'S S OUTPUT ENFORCES PAIRWISE INDEPENDENT FEATURES AT THE ENCODER'S S OUTPUT
In this section, we demonstrate that for random Multi-Layer Perceptrons (MLPs) projectors, minimiz- ing VC of the projector's 's output amounts minimizing HSIC -a a measure of pairwise dependence (see Section 2.1)- between all pairs of features in the learned representation, i.e the projector's 's input. We then justify how this reasoning extends to learned projectors. Our claims are experimentally verified in Section 5.
Notations. In our setting, X Nx is the D dimensional output of the encoder for batch of size N. X is then fed to the projector g to form embeddings Z = = (X) [g(X)) ..... g( X r) xNP on which SSL. criteria are generally applied. Note that Z would be Zieft or Zright in the previous subsection. The acronym MLP refers to projectors typically used in SSL, neural network with three layers of same width, unless stated otherwise.
Our proof strategy consists in proving that VCReg of each MLP layer (Linear Batch Normalization (BN) + ReLU) output enforces pairwise independence of its input, before composing these results. We first study nonlinear elementwise projectors g : IR + I L. which belong to the wider class of DeepSets (Zaheer et al., 2017), and of which BN followed by ReLU can be seen as an instance. We denote the mapping of such projectors Z x(X) {g( 1) - g( ())
3

---

Under review as a conference paper at ICLR 2023
Lemma (Nonlinear elementwise projectors minimize HSIC of their input). Let R R be nonlinear elementwise projector; then, minimizing the covariance of Z with respect to the encoder f amounts minimizing HSIC on all feature pairs in X with kernels K; )((()),)))((
Proof. Let us consider the N values of the ith data feature (x.,,) as realizations of a random variable. Recalling Eq. (2), independence of two random variables can be estimated via HSIC. Considering the
arbitrarily complicated network g and Z (g(X,,1) ................................ ((x,,)) E RN
we have:
1
HSIC((X)) (.6: (X):))
(T(s((()..))ll ).))" Hg(( (x)))))(()))) ) HH)
H)
(N 1)a
1
|l Hg(()))))))
(N - 1)3
||Cov (o((X) 5
|oov (Z)
[HSIC((X) Di (X):a) = ||Cov (Z) @ ((1 In) @ 1,.1 IT. in
and, in the case 1 we have 1, = 1 leading to Eijy HSIC((X))ss) (X):))
Eigg Cov (Z)..) concluding the proof.
To rigorously obtain independence, the K,'s must be universal kernels. To satisfy this assumption, we may randomize Batch Normalization by drawing the mean and variance from some distribution. Combining this operation with a ReLU, we obtain random elementwise nonlinearities approaching random features (Rahimi & Recht, 2007) of universal kernel (Sun et al., 2018). Increasing L improves the approximation of such kernel (Chen & Phillips, 2017) i.c. the larger L. the better approximation of HSIC the covariance term in VICReg is; see Appendix 1) for the randomized Batch Normalization implementation, Figure 8 for its justification, and Section 5 for empirical validation.
Remark 2 (Necessity of variance regularization). Although the variance regularization term does not explicitly appear when minimizing HSIC all pairs, it is necessary when optimizing X to prevent the degenerate solution of X being constant, a common collapse mode of SSL.
Lemma 2 (Random linear projectors minimize HSIC of their input). Let g be random linear projector with weights W, and X has same variance for each column. Then, for large projectors, minimizing the covariance of Z = g( minimizing HSIC with linear kernel for each pair of features in X. (Proof in Appendix B.I.)
XW with respect to the encoder f amounts to
Since the corresponding kernel in lemma 2 is linear, only decorrelation can be achieved for such projector's 's input. This however differs from PCA as we optimize over X and not g's parameters (W). Proving lemma 2 requires a projector with orthogonal weights, i.e. W W I, which gets more and more accurate with random weights (1///D, 1/VD) as P (the output dimension of g) increases. This follows since the central limit theorem states that the dot-product between two P-dimensional weight vectors tends to 0 with rate 00/VP). Weight initialization in neural nets roughly follows u(//D, I/VB), which will be used to implement random projectors.
Theorem 3 (MLP projectors with random weights enforce pairwise independence.). Let us consider a MLP composed of alternating random linear layers and elementwise nonlinearities. Then, for large projectors, minimizing the variance and covariance of the output Z enforces pairwise independence between all pairs of features in the input X.
Proof. Let us consider the last block of such MLP, iee a fully-connected linear layer followed by an elementwise nonlinearity. According to lemma 1, applying VCReg to the MLP, ie.. at the output of the nonlinearity will enforce pairwise independence under corresponding kernel for the input of the nonlinearity, which is also the output of the last linear layer. If the latter is wide enough that it can be considered orthogonal, Theorem 11 in Comon (1 (1994) ensures that pairwise independence is preserved for the input of the layer. We then recursively extend the result backward to the whole MLP. If the last MLP layer is a fully-connected linear, then lemma applies and we go back to the preceding elementwise nonlinearity.
4

---

Under review as a conference paper at ICLR 2023
Fig. 6 in Appendix C.1 shows that each layer in the random MLP is recursively VC-regularized from VCReg being applied only at the projector's output. Following Theorem 3, we can expect that wider projectors are beneficial to enforee pairwise independence while adding layers or learning the projector is not necessary; see Section 5 for empirical validation.
Extension to BarlowTwins and W-MSE, and generality of VCReg. Our results focus on VCReg as formulated in VICReg but can in fact be extended methods that constrain the covariance of Z explicitly, namely BarlowTwiie and W-MSE (Balestriero & LeCun, 2022). Indeed, the objective in W-MSE (Equations 3-4 in Ermolov et al. (2021)) is VICReg with explicit constraint the variance and covariance. Increasing the variance and covariance hyper-parameters in VICReg produces W- MSE hence our results extend seamlessly. The objective from Eq. (7) in BarlowTwins is also similar to VICReg. The derivation, deferred to Appendix B.2, shows that minimizing the constrained form of BarlowTwins objective from Eq. (7) is equivalent to minimizing VICReg's 's invariance term whilst explicitly constraining the variance covariance terms as in W-MSE: hence our results also hold for BarlowTwins: see Section 5 for empirical validation. In particular we will see that as BarlowTwing explicitly enforces minimum VCReg, it better optimizes HSIC compared to VICReg with standard hyper-paramenn parame-araeetrrs.. Finally, as opposed to BarlowTwins loss and most SSL. methods, VCReg can be used and be beneficial within single branch architectures. We provide such use case in Section 5.
Learned projectors. In state-of tfttt-rrrttt SSL representations, the projector is learned, which is not rigorously covered by Theorem 3. Complementing the study of Bordes et al. (2022), we argue that learning the projector is only crucial to satisfy the invariance criterion since random projectors are sufficient to obtain pairwise independent features, as demonstrated in Section 5. In fact, our experiments show that i) using VICReg, keeping the projector random generally produces lower HSIC of their input i.e. better enforces pairwise independence than learning the projector's weights, ii) using VICReg, learning the projector to optimize VCReg more strongly than the invariance term reduces performances, and iii) using VCReg, learning the projector, e.g. in the later studied ICA setting, creates degenerate representation that does not enforce pairwise independence. We thus conjecture that learning the projector's parameters to mainly minimize VICReg's 's invariance term leaves the parameters close enough to their random initialization (Jacot al., 2018) to maintain an accurate estimate of HSIC. In fact, we will see that the wider the projector, the less far away from initialization the parameters have to move, and the better optimized HSIC.
4 RELATED WORK
Feature decorrelation. Also known as whitening, feature decorrelation ensures that correlation between each pair of different features in batch of feature vectors is zero, and that each feature has unit variance. It was originally used as data pre-processing technique, see e.g. Hyvärinen & Oja (2000), before being extended to deep networks (Cogswell et al., 2015) as regularizer. In the context of SSL, Hua et al. (2021); Ermolov et al. (2021) find that feature decorrelation helps solving the collapse issue. The former avoids collapse via appropriate Batch Normalization and its decorrelating variant (Huang et al., 2018) in the projector. This variant of Batch Normalization can be seen as the hard constraint counterpart of Barlow learning fully-connected layer (Husain & Bober, 2019) recovering Principal Component Analysis; as mentioned earlier (Section 3), this differs from VCReg which keeps the layer's parameters random and optimizes its input.
Practically, whitening can be implemented by
Independence criterion for learning features. Enforeing mutual independence to learn a rep- resentation has been proposed for example by Schmidhuber (1992). More recently, and in the context of supervised learning, Chen et al. (2019) demonstrated improved training of ResNets by reducing the pairwise dependence in the features at each layer via combination of Dropout and Batch Normalization. By opposition, VCReg is applied to the projector's s output and common SSL projectors do not rely on Dropout; we also show that Batch Normalization is not necessary to reduce pairwise dependence.
HSIC in supervised and self-supervised learning. HSIC-based losses have been employed in supervised learning e.g. see Mooij et al. (2009); Greenfeld & Shalit (2020) to ensure independence between the residual errors and the labels of a task at hand. Korbblith et al. (2019) measure similarity between two neural representations via HSIC. In SSL, Tsai al. (2021) showed that a modified
5

---

Under review as a conference paper at ICLR 2023
version of BarlowTwins loss maximizes HSIC under linear kernel between the embeddings of two augmented versions of the same sample. In a similar fashion, Li et al. (2021) proposed a SSL framework based on maximizing the dependence between the embeddings of transformations of an image and the image identity via HSIC. Both works differ from ours, which connects VCReg to the minimization of HSIC between pairs of features in the representation.
EXPERIMENTS
We first put in evidence pairwise independence properties emerging in learned visual representations. before validating and exploiting our theoretical findings. Finally, we ablate the SSL context and demonstrate that the composition of a SSL--like random projector with VCReg induces enough independence to perform ICA. Importantly, none of the experiments with VCReg use Dropout, and Batch Normalization is not used when the projector is random. Hence, pairwise independence cannot be attributed to those two techniques as opposed to Chen al. (2019); Hua et al. (2021).
5.1 PAIRWISE INDEPENDENCE EMERGES IN MOST VISUAL REPRESENTATIONS
Setup. In these experiments, track two metrics for statistical independence of the components in the learned representation during ResNet50 training on ImageNet with popular SSL frameworks, as well as supervised baseline. The first metrie, HSIC (Gretton et al., 2005a), tracks pairwise independence. The second metric, dHSIC (Pfister et al., 2018), tracks mutual independence. As explained in Section 1. neither HSIC nor dHSIC scale to the full representation (2048 components). Therefore, we instead rely on proxys: for pairwise independence, we compute and average HSIC on all pairs of the first n components of the representation. For mutual independence, we sample sets of components for which we display dHSIC. We set n = 10 and compute these statistics on the whole ImageNet validation set. We expect both metrics decrease during the training, which would suggest decreasing dependence among the representations. Although statistical tests are available for both metrics, we track bare HSIC and dHSIC as they are continuous. However, we perform HSIC tests in this first serie of experiments so that the bare values can be linked to concrete portions of independent pairs. The width of the Gaussian kernel HSIC is scaled for each representation (see 2. to allow for comparison between methods. See Appendix E for the detailed setup.
Results. Figure shows that different learning frameworks implicitly optimize pairwise indepen- dence of the features in the representation throughout the training (left), and at some point mutual independence (right). As expected from Section 3, HSIC is continuously optimized in Barlow Twins and VICReg (it increases for DINO)*, and Barlow Twins better enforces pairwise independence since the covariance matrix is constrained. Precise values along with HSIC independence tests at level a = 0.5 and corresponding test accuracies can be found in Table 1. Lower HSIC values do not necessarily entail more successful tests: it is possible to allocate low HSIC to sufficiently many pairs while having an overall larger HSIC. For example, Supervised has less independent pairs in spite of an overall smaller HSIC. Table also suggests that pairwise independence is not predictive of test accuracy. Although it seems to be a desirable property, it is not sufficient to yield a good representation: for example, DINO achieves better test accuracy than BarlowTwinn while having higher HSIC. We study this trade-off in the next subsection. Although we observe decreasing dHSIC, i.e. improvement of mutual independence in the representations, all methods quickly reach floor and do not improve further although it is lower for Barlow Twins and VICReg. Hence, we cannot conclude that this property is properly optimized and do not study it in subsection 5.2, but conduct. experiments to better evaluate mutual independence obtained by VCReg in subsection 5.3.
5.2 PROJECTOR CHARACTERISTIE FOSTERING OR HURTING PAIRWISE INDEPENDENCE
Setup. These experiments validate the results from Section 3 by studying which projector charac- teristics help or hurt HSIC optimization when learning visual representations. The setup is the same as above except that dHSIC is not used anymore and the projector may be replaced by projectors introduced in Section 3: an elementwise nonlinear projector from lemma 1, random linear projectors of various width from lemma 2, and MLP taking the representation (D 2048) as input, and outputting an embedding of size P. with two hidden layers of width P and ReLUs. The MLP can be
HSIC is continuously optimized for SimCLR as well: this could be explained in light of Garrido et al. (2022), which connect VICReg and SimCLR, and is omitted here for concisencss.
As opposed to other methods here, DINO relies on multicrop, which partly explains the performance gap.

---

Under review as a conference paper at ICLR 2023
ImageNet Training
ImageNet Training
0.6
0.0100
DINO
DINO
0.7
SimCLR
SImCLR
16
Barlow/wits
0.0095
Barlow/Toiie
0.6
VICitng
VICBag
0.0090
HSIC
d#SIc
0.4
0.0085
0.3-
0.2-
0.0060
0.1
0.0073
20
40
60
80
20
4.
60
80
Epoch
Epoch
Figure 1: Overall, HSIC and dHSIC of the encoder output improves for all methods during training. VCReg based methods have smoother and/or further optimization of HSIC and dHSIC.
Table I: Measure of pairwise dependence (HSIC), pairwise independence testing, and test accuracy for popular SSL. supervised representations averaged on multiple subsets of features.
Method (100 ep.)
HSIC + % of indep. pairs
ImageNet Top1
BarlowTwins (Zbontar et al., 2021)
0,30 + 0.03
38 =
67.8
VICReg (Bardes et al., 2022)
0.39 + 0.03
34 + 2
68.4
SimCLR (Chen et al., 2020)
0.40 + 0.03
36 +
67.7
DINO (Caron et al., 2021)
0.38 + 0.02
+
70.42
Supervised (Wightman al., 2021)
0.20 + 0.05
+
78.1
random as in Theorem 3 or learned as in Section 5.1. We apply an invariance loss along with VCReg on the output of the projectors and scale the covariance coefficient with the size of the projector for fair comparison. See Appendix E for the detailed setup.
Results. Figure 2 (top left) demonstrates that nonlinear elementwise projectors, random linear projectors and random MLPs with one layer achieve lower HSIC than learned MI.Ps, in particular when resampled as to get better HSIC estimates. Hence, learning the projector is not necessary to obtain pairwise independence. As expected in Section 3, for both theoretical and classical projectors, increasing width yields lower HSIC while increasing depth hurts HSIC, see Figure 2. Table 5 in Appendix C.1 shows that increasing the weight of VCReg does not improve HSIC: a possible explanation is that the projector would otherwise start to optimize too much for VCReg. This intuition will be backed by Experiments 5.3. Modifying Batch Normalization to get closer to a universal kernel in lemma 1 yields significant test accuracy gains for VICReg right). Generally, low HSIC is beneficial provided that the test accuracy on ImageNet is reasonable: since HSIC does not guarantee that information is retained from the data, optimizing for it only results in performance loss, see inflexions in Figure 2 (top right) and Figure 5. We conjecture that HSIC combined to e.g. test accuracy on ImageNet could be used for model selection.
Bardes et al. (2022) (Figure 2, top
Discussion. As expected from Section 3, Batch Normalization the projector is not required to enforce pairwise independence: Figure 2 (top right) shows that removing BN improves HSIC while maintaining test accuracy, which could be explained by the fact that each activation in the projector is implicitly regularized by VCReg (Figure 6), alleviating the need for BN. This could also explain why adding layers is detrimental HSIC: implicit VCReg of the activations, an assumption of Theorem will be more and more loosely enforced. Although projectors from Section 3 learn representations with better pairwise independence properties than classical SSL methods, most of them are not as competitive in terms of test accuracy (Fig. 2, top right). This complements the view of Appalaraju et al. (2020); Bordes et al. (2022), which argue that the projector requires some learning capacity to filter out information that is irrelevant to the invariance criterion. Our experiments suggest that the projector capacity should rather be increased via width than via depth: adding layers can be detrimental HSIC seen above but also to accuracy (Appalaraju et al., 2020; Chen al., 2021).
5.3 ABLATION: INDEPENDENT COMPONENT ANALYSIS (ICA) WITH VCREG
The following experiments aim to demonstrate that our findings hold outside of SSL. To this end, we show that VCReg of SSL-like projector's output induces enough pairwise independence solve
7

---

Under review as a conference paper at ICLR 2023
Pairwise indep. for various projectors
Independence-test accuracy trade-off
70
VJCiley random BN (69.1 aec)
0.7
VICTey (68
Test Ancuracy
Mean HSIC
0.5
To projector
Increasing width
60
0.4
VICReg (60.4 acc.)
0.3
0.2
50
projector
512
1024
1096 Projector width P
2040
8192
16384
0.1
0.2
0.3
0.4
0.6
0.7
Mean HSIC
Rundum linear
Random MLP (I-5) cosmpling
Leamned MLF (l-1)
Learned MLP (I-3) no BN
Hondom liner recamping
Learned MLP (le3)
landom MLP [m1) ressmpiig
Nockner elementwine
Pairwise indep. for various projectors
Linear ICA on synthotic data
1.00
0.6
Moximum correlation
0.95
Mean HSIC
0.90
0.4
0.00
0.2
resamplin
Random
0.75
zesampling
Leur zed
Resampling
64
128
256
512 1024 2048 4096 8192
Projector depth!
Projector width
Figure 2: Top left: wider projectors and resampling both yield more pairwise independence for the encoder output. Bottom left: increasing depth hurts HSIC for random or learned projectors. Top right: trade-off between independence of the representation and test accuracy (width D) 8192). Bottom right: ICA experiments. Resampling and increased width improve the quality of the reconstruction.
Loss
Learned
g
VC- (egulllarrzzaiion((
Random
Figure 3: Linear ICA model expressed from VCReg of the projector's output.
linear ICA, i.e. recovering independent sources S NxD from mixture Y SA BND. See Appendix A for a primer on ICA.
VC regularized projectors solve linear ICA. In this setting, finding M enforcing pairwise independence in the components of YM is generally sufficient to recover S (Comon (1994), Theorem 11). VCReg of a random projector's output should therefore be able to recover S. Our model can be seen on Figure 3: whitened batches of mixtures Y fed an encoder Here, is linear transformation M described above. The output X = s(Y) is then fed to a projector and VCReg applied the output randomly resampled at each gradient step to get better HSIC estimates. This setting corresponds to one branch of a VICReg network i) with a linear projection M encoder instead of a neural network, ii) with random projector, and iii) without invariance criterion. We perform ICA on two datasets, synthetic one (Brakel & Bengio, 2017) with sources among which 2 are noise, and a real audio dataset (Kabal, 2002) with sources among which one is noise, also used in Brakel & Bengio (2017). The evaluation metric is the maximum correlation between the true and reconstructed sources. As this metric is not available without ground truth, select the model with lowest dHSIC for X. which can be perfectly evaluated here since the number of sources is small. Our model recovers the sources, and is even competitive with methods specifically designed for linear ICA such as Fast
g(X) covariance matrix. As in Experiments 5.2. the projector
8

---

Under review as a conference paper at ICLR 2023
Table 2: Maximum correlation (between true and reconstructed sources (the higher the better t). The projector performs competitive reconstruction in the linear setting (left), where pairwise independence is sufficient, but not in PNL (right) where it is not.
Linear mixture
Post Non Linear mixture
Method
Synthetic data
Audio data
Method
Synthetic data
Audio data
Whitening
0.8074
0.9876
Whitening FastICA
0.7981
0.9046
FastCCA
0, 0,9998
1.0
0,8311
0.8989
Anica
0.9987:16.5e-
0.9996=t-..9-4 0.9936:1644-44
Anica
0.9-
18e
VCReg
0.9986:18.2e-4
VCReg
0.8465##122-44
0.8706=:376e-4
ICA (Hyvärinen & Oja, 2000) or Anica (Brakel & Bengio, 2017), see Table 2. Both increasing the width of the projector and resampling it each step (especially for smaller projectors) improve the reconstruction as can be seen in Figure 2 (bottom right). This is in line with our findings in Section 3 and Experiment 5.2.
VC regularized projectors do not solve nonlinear ICA. Experiment 5.1 suggests that mutual independence also improves during training. although not as clearly as pairwise independence. Hence, one could ask whether VCReg also enforces the former enough to solve nonlinear ICA. To test this hypothesis, we apply VCReg to particular case of nonlinear ICA which allows identifiability but does not have equivalence between pairwise and mutual independence: the post-nonlinear mixture (PNL) (Taleb & Jutten, 1999). In PNL, the sources are linearly mixed before being fed to elementwise nonlinear functions. For these experiments, and following Brakel & Bengio (2017), our encoder is MLP. During our first experiments, we observed an informational collapse of the encoder, which produces seemingly mutually independent variables with very poor reconstruction of the sources. To alleviate this issue, add decoder taking X as input and reconstructing Y. Figure 9 in Appendix E shows our modified setup. We compare VCReg to FastCCA and Anica. Although FastCCA is not meant solve the nonlinear case, it remains an interesting baseline. Table 2 shows that our model fails to recover the sources, as it does only slightly better than FastICA in the synthetic case. Hence, although mutual independence increases during training, VCReg does not optimize it enough to solve nonlinear ICA. We propose simple explanation for this limitation. Indeed, each feature in Z is a nonlinear function of all features X. Hence, is not possible to completely decorrelate two components of Z as they both contain the same set of features. It is still possible to improve mutual independence to some extent since, in practice, only parts of the inputs are considered at once by nonlinear mappings such as neural networks (Erhan et al., 2009; Adebayo et al., 2018).
Learning the projector does not enforce pairwise independence. ICA experiments done with learned projector fail to recover independent sources: this further demonstrates that learning the projector to specifically optimize the VCReg is counter-productive and that in the context of SSL, learning the projector is rather useful for satisfying the invariance criterion, which is absent in the ICA experiments. VCReg is rather optimized by the encoder f.
6 CONCLUSION: WHAT THE INTEREST OF PAIRWISE INDEPENDENCE?
This work claims that SSL projectors enforce pairwise independence. But, is it desirable property in a representation? For example, Wang & Jordan (2021) claim that mutual independence can be detrimental learn disentangled representations. Pairwise independence may nevertheless be beneficial for representations to be linearly probed: Table 3 suggests that given two representations with similar test accuracy ImageNet, the one with significantly lower HSIC performs better on some downstream tasks on linear evaluation, a common requirement for SSL representations. Showing the projector be responsible for it, and demonstrating HSIC be useful for model selection would be important results we leave for future work. Finally, this work focuses on random weights yet opens the way to weight distributions that are closer to training parameters in practice which, for example, have been characterized for overparameterized networks (Jacot et al., 2018). 8).
Table 3: Test accuracy of SSL representations common downstream tasks (linear evaluation).
HSIC
ImageNet
Places205
iNat
CO3D
CO3D video
VICReg VICReg, no BN in proj.
0.337
68.0
46.5 + 0.2
27.6 0.1
97.7 0.1
98.1 + 0.1
0.239
67.6
47.1 + 0.1
29.9 + 0.2
98.0 + 0.1
98.53 + 0.03
9

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
For theoretical results, explanations on assumptions are provided Section and proofs are displayed either in the main paper in Appendix B.1. The results verified experimentally in Section 5.
Implementations of the algorithms used (HSIC, dHSIC, Randomized Batch Normalization, Linear ICA, and PNL ICA) are provided in Appendix D, and our code will be released if the paper is accepted.
Our experimental setup is detailed either in the main paper in Appendix E. The latter include architectural details, augmentations, optimizer with learning rate and batch size, and grid search for hyper-parameters. The numbers used produce the Figures in 5 provided in C.I.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Srikar Appalaraju, Yi Zhu, Yusheng Xie, and István Fehérvári. Towards good practices in self- supervised representation learning. arXiv preprint arXiv:2012.00868, 2020.
Francis R Bach and Michael Jordan. Kernel independent component analysis. Journal of Machine
Learning Research (JMLR), 2002.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arxiv preprint arXiv:2205. /108, 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivarrance-ccvvaaaae regularization for self-supervised learning. International Conference on Learning Representations (ICLR), 2022.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Improving deep networks generalization by removing their head. arXiv preprint arXiv:2206. 221i132,00.,3378, 2022.
Philemon Brakel and Yoshua Bengio. Learning independent features with adversarial nets for non-linear ica. arXiv preprint arXiv: 2017.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Di Chen and Jeff M Phillips. Relative eπror embeddings of the gaussian kernel distance. In International Conference on Algorithmic Learning Theory, pp. 560-576. PMLR, 2017.
Guangyong Chen, Pengfei Chen, Yujun Shi, Chang Yu Hsieh, Benben Liao, and Shengyu Zhang. Rethinking the usage of batch normalization and dropout in the training of deep neural networks. arXiv preprint arXiv: (00::/995009922 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances Neural Information Processing Systems (NeurIPS), 2021.
Xinlci Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfit- ting in deep networks by decorrelating representations. arXiv preprint arXiv: 15 1,06688, 2015.
10

---

Under review as a conference paper at ICLR 2023
Pierre Comon. Independent component analysis, concept? Signal processing, 36(3):287-314, 1994.
Romain Cosentino, Anirvan Sengupta, Salman Avestimehr, Mahdi Soltanolkotabi, Antonio Ortega, Ted Willke, and Mariano Tepper. Toward geometrical understanding of self-supervised contrastive learning. arXiv preprint arXiv:2205. 06926,, 2022.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations (ICLR), 2020.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning (ICML),
2021.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning, arXiv preprini arXiv:2206.02574, 2022.
Bernhard Goebel, Zaher Dawy, Joachim Hagenauer, and Jakob C Mueller. An approximation to the distribution of finite sample size mutual information estimates. In IEEE International Conference on Communications, 2005. ICC 2005. 2005, volume 2, pp. 110--1106. IEEE, 2005.
Daniel Greenfeld and Uri Shalit. Robust learning with the hilbert-schmidt independence criterion. In International Conference on Machine Learning (ICML), 2020.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005a.
Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, Bernhard Schölkopf, et al. Kernel methods for measuring independence. Journal of Machine Learning Research (JMLR), 2005b.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-super learning. Advances Neural Information Processing Systems (NeurIPS). 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self supesvuee deep learning with spectral contrastive loss. Advances Neural Information Processing Systems (NeurIPS), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decor- relation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9 9608,,90,,00000 2021.
Lei Huang, Dawei Yang. Bo Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 791-800, 2018
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization contrastive self- supervised learning. arXiv preprint arXiv:2111.00745 2021.

---

Under review as a conference paper at ICLR 2023
Syed Sameed Husain and Miroslaw Bober. Remap: Multi-layer entropy- pooling of dense enn features for image retrieval. IEEE Transactions on Image Processing,
2019.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications.
Neural networks, 13(4-5):411 14010,1400000 2000.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09544, 2021.
Peter Kabal. Tsp speech database, 2002. McGill University, Database Version, 1(0):09-02.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML), 2019.
Sue E Leurgans, Rana A Moyeed, and Bernard Silverman. Canonical correlation analysis when the data are curves. Journal of the Royal Statistical Sociely: Series B (Methodological), 55(3): 725-740, 1993.
Yazhe Li, Roman Pogodin, Danica Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Zejian Li, Yongchuan Tang, Wei Li, and Yongxing He. Learning disentangled representation with pairwise independence. In Proceedings of the AAAI conference on artificial intelligence, 2019.
Edward L Melnick and Aaron Tenenbein. Misspecifications of the normal distribution. The American Statistician, 36(4))372-373, 1982.
Charles Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal kernels. Journal of Machine Learning Research (JMLR), 2006.
Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Joris Mooij, Dominik Janzing, Jonas Peters, and Bernhard Schölkopf. Regression by dependence minimization and its application to causal inference in additive noise models. In International Conference on Machine Learning (ICML), 2009.
Edith Mourier. Eléments aléatoires dans un espace de banach. In Annales de l'institut Henri Poincaré, volume 13, pp. 161-244, 1953.
Niklas Pfister, Peter Bühlmann, Bernhard Schölkopf, and Jonas Peters. Kernel-based tests for joint independence. Journal of the Royal Statistical Society; Series B (Statistical Methodology), 80(1): 5-31, 2018.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems (NeurIPS), 2007.
Alfréd Rényi. On measures of dependence. Acta mathematica hungarica, 10(3-4):441-455 1959.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural computation, (6):863-879 1992.
Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In International Conference on Algorithmic Learning Theory, pp. 13-31. Springer, 2007.
Ingo Steinwart. On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research (JMLR), 2001.
12

---

Under review as a conference paper at ICLR 2023
Yitong Sun, Anna Gilbert, and Ambuj Tewari. On the approximation properties of random relu features. arXiv preprint arXiv:/810.04377,000 2018.
Anisse Taleb and Christian Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing, 47(10):2807-2200 1999.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems (NeurIPS), 2020.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning (ICML), 2021.
Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with pegative-sammle-frre contrastive learning, 2021.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. Proceedings of the Conference Computer Vision and Patlern Recognition (CVPR), 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- PMLR, 2020.
Yixin Wang and Michael Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109.03799 2021.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2110.00476, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny, Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032.00, 2021.
13

---

Under review as a conference paper at ICLR 2023
A FURTHER BACKGROUND
A.1 VCREG METHODS
Learning visual representations has long history in machine learning. Shortly after the ad- vent of convolutional neural networks (CNN), it was common to train model on supervised task such as ImageNet before removing the classifier and use the remaining model to produce fea- tures for downstream tasks (e.g., classification, segmentation), technique coined transfer learning. Then, strategies to Iearn representations without labelled dataset by enforeing invariant embeddings emerged (Misra Maaten, 2020: Chen et al., 2020). Joint embedding methods can be divided in two categories. Contrastive learning (c.g., SimCLR (Chen et al., 2020)) pulls together representations of two augmented versions of the same image while pushing away the representation of this image from the representation of different images. Non-contrastive learning (e.g., DINO (Caron et al., 2021)) pulls together representations of two augmented versions the same image while avoiding collapse using different techniques. The frameworks considered in this work belong to the latter category. Non-contrastive self-supervised representation learning of images has a few peculiarities:
Data augmentations are central, at least when it comes to learn from ImageNet, and must be carefully chosen.
Most methods can be used either with Vision Transformers (Dosovitskiy al., 2020), or with CNNs such as ResNets (He et al., 2016). In this work, we focus on ResNets.
Model selection is usually performed via the Topl accuracy on the validation of ImageNet using a linear classifier on top of the learned representation. Outside of joint embedding, methods such as Masked-Aute anto-Encoderr (He et al., 2021) perform poorly linear evaluation while delivering excellent performance when fine-tuned on the downstream task.
The crucial role of the projector. was found by Chen et al. (2020) that adding a MLP on top of the encoder (removed after the training) significantly improved the quality of the learned representation. For example, in 100 epochs, VICReg without projector would achieve 48% validation accuracy on ImageNet instead of 68 and SimCLR 50% instead of 68%. Since then, a few theoretical work attempted to explain SSL with joint embeddings. Jing et al. (2021) study the role of linear projectors with restricted augmentations, while other works such as (Huang al., 2021; Wang & Isola, 2020; HaoChen et al., 2021; Tian et al., 2020; Wang & Liu, 2021) only consider an encoder without projector in their theoretical analysis. Finally, Cosentino et al. (2022) consider MLP projector in a very restricted context where data augmentations are Lie group transformations.
BarlowTwins. Zbontar et al. (2021) propose slightly different approach based on regularizing C, the P x P cross-correlation matrix between Zien and Zright. by optimizing
LBT -(OO)A + a Σ (C))
Here, the leftmost term corresponds to regularizing the cross-correlation of the same feature in the two views to be unit, while the rightmost term regularizes the correlattnn between pairs of different features in the two views. Importantly, (C),, falls back to measuring the cosine similarity
between the th column of Ziet and the jth column of Zright i.e. (C),.) [[(Z)n) al (i.Z) Hence, the leftmost term is also an invariance term: all features must be similar for both views.
((Zee)) (..n),,. (Zngat)) (Z)
W-MSE. Ermolov et al. (2021) use the following loss min 2 - (Zieft Zright)
| Z || Zrighe l|2
s.t. Cov Ziekt Td, Cov (Zrigtt) Td.
where the cosine similarity can be replaced by the Fuclidean distance.
14

---

Under review as a conference paper at ICLR 2023
A.2 INDEPENDENT COMPONENT ANALYSIS
The goal of ICA (Comon, 1994) is to find a transformation of random vector Y which minimizes the statistical dependence of its components, Its simplest instance is linear ICA, and is motivated by problems such as the cocktail-party, where Y E IRN typically results from a linear transformation of independent sources S (e.g overlapping voices) one wants to recover. Formally, Y = SA. with A E IR DxD an unknown mixing matrix that can therefore not be inverted. Instead, the linear ICA approach searches for M IR. DxD such that YM- S by maximizing the statistical mutual independence between the components of Y.. Being able to recover S is property known as identifiability.
B PROOFS
B.1 PROOF OF LEMMA 2: RANDOM LINEAR PROJECTORS MAXIMIZE PAIRWISE INDEPENDENCE
Proof. Let's consider the linear regime for which ((x),,,) (x).. " with orthogonal weights Le. (w, wj) = ((=j)) This assumption is realistic since we assume that E R with K very large, and that those are randomly sampled. From that, we then have
HSIC((X).s (X) (N 1)2
Σ (IT i
()((()))))((())))
Σ
(.(())
(N - 1)2
*
we will now push the sum inside the norm by considering the following equality:
Σ (0,D)
Σ |((i)))) +
Σ Σ
Tr (r(())))) s(k, = Σ (1
$
# (L.)(()))))))))))
since in our case
[(i,j) (xs))=s((())) ) H wx H (X) J
w:
((1,5)" w0 H (X) Tr(f(,))) f(k,t)) = ((i=kAj= (X) (X), HX)) ):e0
(X )
leading to
Σ HSIC((X) 1.d (X),))
Σ ((X)) Ho(())))
(N 1)2
itj
E
((()),))) Eo((x) (X((()..)))
(N 1)2
N (( H
1
Σ ,((()),))) diag( Cov (X (X))
= =lCo((XW) - diag Cov ( (W))
= (')
since we assume that all columns of X have same variance and that Wi is orthogonal we have for the lre-last equality
1
Hg(()),))))))))) = Var((X)..1 Σ w.w: = Var((X)..).
N
and for the last equality we use the fact that since g is linear, E.((()).)) = Ex() = XW
with W = [w) wx]²".
For simplicity, Y directly denotes N empirical D-dimensional observations.
15

---

Under review as a conference paper at ICLR 2023
B.2 ON THE EQUIVALENCE BETWEEN VICREG AND BARLOWTWINS OBJECTIVES
We can express Barlow Twins objective as:
K
min
(Cov(Zien. Zright) - 1)2 + a E R'
Cov(Ziet: Zright)
#=1
s.t. Cov (oftt) Cov(Zright) =1.
Assuming Zen Ziet = I, 2 terms, we have
right Zright i.c. perfect minimization of the variance and covariance
(Ziuft).. ((et))..))
Ois ([ (Ziot):.i12
2 ((Ziet)) (Zright)ss) = [Zizo) ).. (Zright) 1,
and thus
E 11
Σ [Ziot)):s - (Zright Zieh
Zrighelle x (Ziett Zright);
so we recover the invariance loss exactly with the diagonal terms of BwilowTwins. We now show this is actually enough to minimize this quantity to also minimize the off-diagonal terms:
Σ i
Σ (1 [[ (Zeet): (Zright 1())
i
= Σ (1 [[((ae)):s - (Ziet)iss + (Ziet):s - (Zright) ):.s 1B,2
ifj
Σ (1 マ/1 (iott); - (Zien) (is 2 (Ziot):ss - (Zright): ):o1
(( (Zeert).s - (Zien)iss (Zlett Fright)).)) )²
Σ 1 (Zioot) (Zright) ((Ziet)) (Ziofe )is (Zien - (Zrigha ):.)) )²
Zioft )us (Zright) 203 + (Ioof)).. (Zieft (Zietti.s (Zrighe)
(1 (Ziet):s (Zright) 112 + (Zeen)) (2 (Zright)
12
25 K 1)
Ziees Zrighell
4
hence minimizing the BarlowTwins loss with the explicit whitening constraint is equivalent to minimizing VICReg with explicit whitening constraint.
16

---

Under review as a conference paper at ICLR 2023
ImageNet Training
0.8
Elementwise nonlinear
0.7
Random linear resampling
Random (D=8192, 1=1)
0.6
Learned (D=8192, 1=3)
0.5
HSIC
0.4
0.3
0.2
0.1
0.0
20
40
60
80
Epoch
Figure The projectors proposed in Section better optimize HSIC than their classical counterpart.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I HSIC OF REPRESENTATIONS LEARNED FROM IMAGENET
Table 4: Measure of pairwise dependence (the lower the better) and test accuracy for the projectors studied in Section 3. I, is the number of nonlinear projections in Lemma 1, / the width of the projector and I is the number of layers.
Resampling
No resampling
Projector type
Mean HSIC
ImageNet Topl
Mean HSIC
ImageNet Topl
No projector
N.A.
N.A.
0.48
48.6
Elementwise nonlinear (L 8)
N.A.
N.A.
0.34
$1.0
Random linear (D 16384)
0.16
54.3
0.13
54.2
Random linear (D -8192)
0.14
53.6
0.30
52.9
Random linear (D -4096)
0.16
52.3
0.24
52,9
Random linear (D = 2048)
0.18
51.7
0.38
52.3
Random linear (D 1024)
0.27
51.1
0.39
52.0
Random linear (D 512)
0.49
50.0
0.54
51.3
Random (D 8192, 1)
0.06
50.9
0.48
53.6
Random 8192, 2)
0.63
53.3
0.52
53.4
Random 8192, 3)
0.56
52.8
0.62
54.9
Learned 8192, [ 1)
0.17
58.4
N.A.
N.A.
Learned (D 8192. / 2)
0.36
65.1
N.A.
N.A.
Learned (D 8192, 1 3)
0.30
68.4
N.A.
N.A.
Same without BN
0.22
68.1
N.A.
N.A.
Same with Random BN
0.31
69.1
N.A.
N.A.
17

---

Under review as a conference paper at ICLR 2023
Independence-test. accuray trade-off
0.6
0.5
0.4
HSIC
0.3
0.2
0.1
0.0
0.2
0.4
0.6
0.8
1.0
Acc
ImageNet
CO3D
Eurosat
iNat
Figure 5: Minimizing HSIC is beneficial up some extent, since HSIC does not guarantee that useful information is present in the representation. Beyond this limit, the performance decreases on ImageNet and on downstream tasks.
Table VICReg with (048-2048-2044 projector and varying covariance coefficient. Increasing this coefficient does not necessarily decreases HSIC. Indeed, the projector may start to optimize too much for the VC criterion thus getting trapped in bad solutions.
Width
Mean HSIC +
ImageNet Top1 t
2
0.33
66.6
4
0.42
66.7
0.37
8
0.29
66.3
10
0.28
66.5
12
0.37
66.2
Table VICReg with learned MLP projector and varying sizes. The wider the projector, the lower HSIC, which conforms to our theoretical results in Section 3.
Covariance coeff
Mean HSIC
ImageNet Top1
512
0.42
63.5
1024
0.37
65.4
2048
0.33
66.6
8192
0.31
68.4
18

---

Under review as a conference paper at ICLR 2023
Linear
10
10
10
15
20
25
15
25
10
20
15
20
Re
Linear
Dut
10
10
10
20
25
15
25
10
20
10
20
25
Figure The covariance matrices for some features of each activation in a random VICReg projector (same in Bardes et al. (2022) with width 8192) before corresponding hidden layer are close to diagonal, suggesting that each hidden layer output is implicitly VC-regularized.
10
10
10
15
20
25
1.5
1.0
20
20
Re
Linear
Out
10
10
10-
15
30
25
15
25
10
20
30
15
20
Figure The covariance matrices for some features of each activation learned VICReg projector (same as in Bardes et al. (2022) with width 8192) before corresponding hidden layer are close to diagonal, suggesting that each hidden layer output is implicitly VC-regularized.
19

---

Under review as a conference paper at ICLR 2023
D IMPLEMENTATIONS
D.1
HSIC EQ. (2)
def IM (X, sigma):
pairwise_distances torch.cdist return torch.expl -pairwise distances (2 sigma*
def HSIC 1, x_2, sigma_1, sigma_2]
N s. batch sizc, should be the same for X_1 and X_2 K_ CauusslannemmllllttiiX..X. sigma_1) Gaussian kernel matrix
X_1 with bandwidth sigma_1
(X_2, sigma_2) Gaussian kernel matrix of X_2 with bandwidth si sigma_2
K_2 aaaasiianneneeMaatte
torch.eye (N) 1.0
# centering matrix 2 H) / (N-1) .*2)
10
HSIC torch.trace (K_
11
return HSIC
In our study, the bandwidth of the Gaussian kernel is determined by the median of the distribution of pairwise euclidean distances between samples.
D.2 DHSIC EQ. (4)
dHSIC can be estimated given empirical samples Xi...., Xa with respective kernel matrices Ki,..., Ka as:
dHSIC(X) X
Ki)
II
KR)i.
2 No+1
ΣOX
Σ (K)
N
Ned
1.j
k=1
When d = 2, the first term corresponds to biased HSTC estimator. The implementation of dHSIC given by:
import Gaussiankerne * defined above
def dMSIc sigma):
length shapap (0)
term_l 1.0
term_2 2 1.0
term_3 2.0 / length
for in range (D)
K_j CaussianKermeLMttriiitt.
10
term_ torch. mul (term_1, K.j)
11
term_2 1.0 / length length / term_2 + torch. sum (K_j)
12
term_3 1.0 / length term_3 K_j. sum (axis-0)
term (1.0 length) term 3 torch, sum (term_3)
torch. sum (term_
15
16
return term term 2 term_ 3 # the three terms of the estimator
As these evaluations already take few minutes, scaling it significantly larger portions of compo- nents would be impractical.
D.3 RANDOMIZED BATCH NORMALIZATION
class RandBatchhNrmll (torch, nn BatchNormld) def forward (input):
# the mean and variance term in classical batch norm are
# randomized
torch. randn likedn__iiedie (m) self self.estimator_mean.s
self.csstimtorrmmeat mean
std torch. C Ganma
self wsw.inttoosssttmmme..e meat.memenn..mmmmmmmm square eelf.eetinator_std.
self ieett.estt / self.es est iattimat imaoor imator_stdd sad..aaddd
).sampe()
20

---

Under review as a conference paper at ICLR 2023
self. weight (input m) torch. sqrt (std. square_ self.eps) self.bias
D.4 LINEAR ICA MODEL 3
For in projector.paramtorrs. (): # freeze the projector p.requires_grad False
For y Icader:
x encoder (y) matrix multiplication with M
projector (x) $ compute embedding
C terch.cov (2) I covariance matrix
10
loss - terch.MSE(C (C, forch.identityy loss backward optimizer.step
VCReg loss
13
14
projector. init_() () to resample the projector for p in paramttrss freeze the projector p.requires_graa = False
15
D.5 NONLINEAR ICA MODEL 9
The implementation of our nonlinear ICA model 9 differs from 3 by the addition of reconstruction constraint:
for p in p.requires_grad False
(): freeze the projector
for in Loader:
encoder (y) MIP encoder
2 projector (x) # compute embedding
rec reconstructor (x) MLP reconstructor
11
torch. (2) # covariance matrix loss = torch, MSE (C, torch.identity.. id)ntity()) Imda torch.MSE MSE y_rec)
12
VCReg loss with reconstruction
loss bacckarrd
optimizer.step())
15
16
projecter._ init_ () to resample the projector
for in projector.purammterss// # freeze the projector p.requires grad False
21

---

Under review as a conference paper at ICLR 2023
0.495
0.982
1.465
-1.592
-0.100
1.377
-1.004
-0.999
-09995
0.590
0.981
1.368
1.855
2.984
4.100
0.006
0.010
0.013
Figure 8: Distributions of f (top row) and c.k (bottom row), which are the BN centering and scaling statistics computed at each mini-batch for three units in DN layer with w'2f1 (([11, 0, -], diag([1, 3, 0.1])) iE not relevant in this context). The empirical distributions in black are obtained by repeatedly sampling mini-batches of size 64 from a training set of size 1000. The analytical distributions in green can be obtained by analysis, and its empirical distributions in red closely match the training one.
E EXPERIMENTAL DETAILS
E. DETAILS ON IMAGENET EXPERIMENTS
Architectures and hyper-parameters. We use Resnet50 bottleneck, optimized during 100 epochs with LARS and an initial learning rate 0.3 for all methods. The batch size is 1024 for all methods except SimCLR. All methods have projector 8192 8192 8192 except DINO and Supervised. Throughout the training, we evaluate the learned representation using an online linear classifier. Details specific each method:
Barlow Twins: for Experiment 5. the off-diagonal coefficient 0.0051.
VICReg: for Experiment 5.1, the projector size is 8192 8192 8192, Invariance coeff is 25, Variance coeff is 25 and Covariance coeff 1. For the rest of the experiments, and following Bardes et al. (2022), we only size of the projector. We scale in the square root of the output size of the projector in order have similar magnitude for the covariance term for all projector sizes.
the covariance coefficient when changing the
SimCLR: for Experiment 5.1, we choose higher batch size of 2048 SimCLR is sensitive to this parameter (Chen et al., 2020). The temperature 0.15.
DINO: use crops. The head has 4 layers of widths 2048 2048 2048 256 65536.
22

---

Under review as a conference paper at ICLR 2023
Augmentations. At train time, the resolution is 160 and
FandonlorizonttaF1ipp)) ()
ColorJitter (0.8, 8, 0.4, C.4, 0.2, 0.1).
Greyscale(O... (0.2).
NormalizeImage with ImageNet mean and standard deviation.
GaussianBlur () with kernel size (5, 9) and sigma (0.1,
At validation time, the resolution is 224 and the images are cropped and normalized with ImageNet mean and standard deviation.
E.2 ICA SETUP
Detailed setup. In these experiments, keep the same optimizer, train 100 epochs and choose a batch size of 64 for both datasets. We tune the learning rate according to a logarithmic grid [1.0, 10.0, 100.0] (then rescaled according batehsasee
256 ). The MLP can either be SSL-like projector or simply a fully-connected layer followed by a ReLU. The wider the MLP, the better the result hence it does not require selection. For the synthetic dataset, the MLP has width 1024, and 8192 for the audio one.
Linear ICA: We tune the standard and covariance coefficients according to a logarithmic grid [1, 10, 100]-
ICA: Our architecture for the nonlinear ICA experiments is presented in Figure 9. The encoder f is MLP with layers and width 128. The decoder a learnable nonlinearity (a MLP with 3 hidden layers and depth 16) followed by fully-connected layer. We tune the standard, covariance and reconstruction coefficients according to a larger logarithmic grid [1,3, 10, 30, 100).
VC-Reeullarzationn22
h
Loss
Leamed
Y'
Random
MSFLY, Y")
Figure 9: Nonlinear ICA model.
Reconstructed sources for ICA. In this paragraph, we provide ground truth sources for the synthetic data along with examples of reconstructed sources.
23

---

Under review as a conference paper at ICLR 2023
True sources
-1
-1
/
MM An
25
75
100
125
150
175
200
Figure 10: Data before mixing.
Mixed sources
0.0
M
0.0
-2.5
WM
mM
1
2
2.5
wMW ww
-2.5
W W
25
50
75
100
125
150
175
200
Figure 11: Data after mixing, before feeding to the various ICA models.
24

---

Under review as a conference paper at ICLR 2023
Reconstructed sources (max correlation=O.999))
-5
-5
5
-5
10
Mull
-10
5
Moltth
0
M
-5
-5
25
50
75
100
125
150
175
200
Figure 12: This level of max correlation is typically achieved by FastICA, VCReg or Anica.
Poorly reconstructed sources (max correlation 0.8)
M
0
m Wy
-10
www.mm
-5
0
-10
10
Ma mn / Wylll
WA Mpma
0
-10
-5
10
-10
25
50
75
100
125
150
175
200
Figure 13: This level of max correlation is typically achieved by Whitening.
25


Reviews:
[{"review_id": "oDFJgNN8JM", "reviewer": "Reviewer_PbKY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to connect an effective regularization technique used in deep learning for visions with a special kind of measuring independence technique through kernel method. This connection would show that the regularization's goal is to suppress correlation of learned representations. Then the authors proceed to validate their claim via experiments. \n\nThe central idea is quite interesting (though I doubt whether that's very new): a random projection would intuitively give you a set of good basis and if the features are not correlated along these basis, then they should not be correlated. I am aware that Andoni, Panigraphy, Valiant, and Zhang's work utilizes a similar idea (and I believe this is well known enough so that probably there are other works doing similar things). The authors are aware of some weakness associated with this approach, e.g., it needs to assume the projection is random and not learned, and it appears that there does not exist any technique to circumvent the problem. \n\nOverall, it feels like a paper that can appear in top venues but it is at the borderline side. ", "summary_of_the_review": "1. it builds a quite interesting connection between deep learning and independence test. \n2. it is around the borderline papers accepted to neurips/icml/iclr (calibration). \n3. while the specific connection is new, the idea of taking advantages of random projections seems to be around for a while. \n4. the projection has to be random to make the analysis work. The authors seem to be aware of the weakness. ", "strength_and_weaknesses": "Strength: The direct connection the authors made between kernel-based independence test and the regularization seems to be quite interesting. Non-trivial amount of analysis is needed to make the connection rigorous. \n\nWeakness: \nIt seems that the analysis relies on the projection is random and there was a remark on that. \n\nAlso, there was a discussion on the fully independence (Eq. 4): I guess I am a bit lost: do we have fully independence if we have pairwise independence under the specialized test (usually the answer is no? but I feel less sure because I dont have too much intuition on the kernel-based test). If we need fully independence, does that mean we need exp(d) samples, which is impossible in practice? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "naj1OwIeZ0", "reviewer": "Reviewer_UspR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the VCRed regularization in self-supervised learning algorithms. The authors show that VCReg enforces pairwise independence between the features of the learned representation. Detailed analysis is provided. ", "summary_of_the_review": "This paper studies the VCRed regularization in self-supervised learning algorithms. However, the motivation is not clear and the results are not interesting.", "strength_and_weaknesses": "Strengths:\n\n1: This paper studies the VCRed regularization in self-supervised learning algorithms. This problem is interesting and could have a high impact on the community.\n\n2: This paper is well-written and well-organized. The analysis is comprehensive.\n\n\nWeaknesses:\n1. The motivation is unclear to me. Can the study on independence explain why self-supervised learning works or help improve the existing SSL methods?\n\n2. The contributions are limited. The results in this paper do not provide enough useful intuition or knowledge for improving SSL.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i7Dyvd8d9m", "reviewer": "Reviewer_MKGJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies VCREG, a particular regularization function used for contrastive representation learning. The purpose of this term is to prevent embedding collapse, where all inputs map to the same embedding (a trivial way to satisfy that embeddings corresponding to different views of the same object should map to the same embedding). In this paper, they model the embedding procedure as operating in two stages: first, an encoder produces a representation X for each input, then X is passed to a neural network g to produce a higher dimensional embedding Z=g(X). The VCREG term aims to enforce 0 covariance between the elements of Z. The main observation of the paper is that 0 covariance between elements of Z can be viewed (approximately) as enforcing independence between the elements of X. ", "summary_of_the_review": "This paper is not yet ready for publication. ", "strength_and_weaknesses": "Strengths: the basic observation is fairly clean and, I believe, correct (though I have not checked the proofs).\n\nWeaknesses:\n\nThe main weakness I see is a lack of motivation. It seems pretty clear a priori that the regularization term is pushing towards a relaxed notion of independence between elements of X. The main contribution of the paper is then an argument that, loosely, this relaxation is not too severe and we can view the regularizer as targeting independence between elements of the representation.\n\nWhat's not clear to me is: why is this independence desirable? What might go wrong with a weaker notion of independence? Does this somehow help us understand when VCREG should or should not be preferred to other techniques for preventing embedding collapse? Without some answer to these questions, it's not clear to me why the technical results here are insightful.\n\nThese questions seem particularly salient given that 1. the experiments show that the learned representations don't actually have elementwise independence (e.g., in figure 1 measured HSIC never gets close to 0), and 2. the existence of arguments that elementwise independence of representations is anyways undesirable---see e.g., https://arxiv.org/abs/2109.03795\n\nClosely related: I was generally unclear about what questions the experiments were meant to answer.\n\nMore minor: the results only hold for a variant of the encoder learning where the function g taking X to Z is random (i.e., not learned). This is clear in the relevant section, but not appropriately flagged in the introduction. Indeed, the authors present empirical results showing that training this map does indeed cause the independence to fail.\n\nI also have some relatively minor problems with the writing, which is not yet fully baked.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "bOhDGnO5Mr", "reviewer": "Reviewer_EpBg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a proposed VCReg loss and its relation to pairwise independence of features.\n\nTheoretically, the paper equates the covariance term in VCReg loss to HSIC, for both arbitrary nonlinear point-wise function and sufficiently wide random linear transformation.\n\nEmpirically, the paper studies how properties of projection head affect pairwise independence. The results suggest that shallower and wider projectors should be used, and that a random projector suffices if we are only concerned about pairwise independence. Moreover, since VCReg enforces pairwise independence which is sufficient for linear ICA, the paper finds that VCReg is competitive with specialized algorithms such as Fast ICA.\n\nA major concern though is that it is unclear why pairwise independence is an interesting property to study, since the paper finds that it does not correlates well with downstream performance.", "summary_of_the_review": "This paper is centered around pairwise independence, however, currently it's unclear to me why this is an interesting property to study for representation learning, especially since the empirical results already show that there is no clear correlation between HSIC and downstream accuracy.\n\nThe paper is clearly structured overall, but many details could be improved for better readability.", "strength_and_weaknesses": "The paper is centered around pairwise independence of the features. I find the paper clearly structured overall, and there is sufficient discussion on related works.\n\nHowever, I'm unconvinced why this is a property of interests or desirable. My current impression is that fully collapsing is certainly not good, but neither is full independence:\n- Empirical findings show that more independence does not always mean better accuracy.\n       - For example, random projectors usually get lower HSIC than learned projectors, but learned projectors get better accuracy.\n- dHSIC plateaus during training, which suggests that there may be a mismatch between the SSL objective and pairwise independence.\n\nThat said, pairwise independence may be useful for model selection, if there is a clear correlation between HSIC and test accuracy.\n- The empirical evidence is Figure 2 (top right), which I currently find questionable: how many replicates/runs are in each dot of the plot? If there's only a single run, then the trends may be due to variances in random seeds.\n\nA clarification question about Lem 1: There seems to be a discrepancy of taking $L=1$ in the proof, and taking $L$ to be large when approximating the random kernel. How does the proof handle $L \\neq 1$? I also find the notations confusing: what does $1-I_D$ mean (i.e. subtracting a matrix from a number)? Is $\\otimes$ here denoting the tensor product?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WG2bkx4dBgU", "reviewer": "Reviewer_NhDr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nSummary:\nThis paper provides some explanations for regularization in self-supervised learning algorithms and, to some extent, other phenomena beyond self-supervised learning. Its analysis is reasonable and comprehensive. However, this analysis does not yield very useful insights into the improvement of self-supervised learning or supervised learning. It can neither fundamentally explain why existing self-supervised learning works nor provide effective guidance on how to improve the efficiency of self-supervised learning in the future. Overall, this is an article on the borderline, leaning toward being accepted.\n", "summary_of_the_review": "\nSee \"Summary Of The Paper.\" This article is an above-average article and deserves a weak acceptance.\n", "strength_and_weaknesses": "\n\n(Positive) The success or failure of self-supervised learning is still a mystery. This article attempts to provide some explanation for the regularization involved, which is highly encouraged.\n\n\n(Positive) The paper demonstrates that regularizing the projector's output helps to encourage independence between the projector's input features. This is very interesting.\n\n\n(Positive) The article also analyzes that the depth and width of the projector also affect the independence of its input features. This is also included as part of the theoretical framework presented in this paper.\n\n(Positive) The method in this paper is very simple and clean.\n\n(Negative) The theory in this paper (including the central limit law) relies on many random projections. But this does not fit perfectly with both self-supervised learning and supervised learning, whose parameters are to be learned. This creates a huge gap between its theoretical and practical needs.\n\n\n(Negative) The most worrying thing is that low HSIC does not bring high test accuracy. For example, it is demonstrated in the paper that both random projection and resampling reduce HSIC (Figure 2). But what's the use of this? We cannot improve test accuracy with these strategies. Although the authors say, \"the correlation is true within the same method, see Figure 2 (top right)\", this is misleading. The x-axis is obtained by increasing the width. It is already a consensus that the test accuracy can be improved by increasing the width, which was not first discovered by this article. So, we did not learn much useful knowledge from this paper to help self-supervised learning.\n\n(Negative) The analysis in this paper does not yield very useful insights into the improvement of self-supervised learning or supervised learning. It can neither fundamentally explain why existing self-supervised learning works nor provide effective guidance on how to improve the efficiency of self-supervised learning in the future. \n\n(Negative) From an HSIC perspective, the results in Figure 1 are good. But this is only for HSIC.\n\n(Negative) Learning the projector is not necessary to obtain pairwise independence and vice versa.\n\n(Positive) I am happy to see that modifying Batch Normalization to get closer to a universal kernel in lemma one yields significant test accuracy gains for VICReg over Bardes et al. (2022).\n\n\n(Negative)  The authors said that projector capacity should rather be increased via width than via depth: adding layers can be detrimental to HSIC as seen above but also to test accuracy. Where is the evidence of \"detrimental to the test accuracy\"?\n\n\n(Positive) I am happy to see that \"Learning the projector does not enforce pairwise independence.\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject