An Information Theory Perspective on Variance-Invariance-Covariance Regularization
Authors: Ravid Shwartz-Ziv
Randall Balestriero
Kenji Kawaguchi
Tim G. J. Rudner
Yann LeCun
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-Supervised Learning
Generalization Bounds
Information-Theory
Deep Neural Networks
TL;DR: This paper provides an information-theoretic analysis of the VICReg method in self-supervised learning, introduces a family of improved SSL techniques based on these principles, and sets the stage for enhanced transfer learning research.

Abstract: Variance-Invariance-Covariance Regularization (VICReg) is a self-supervised learning (SSL) method that has shown promising results on a variety of tasks. However, the fundamental mechanisms underlying VICReg remain unexplored. In this paper, we present an information-theoretic perspective on the VICReg objective. We begin by deriving information-theoretic quantities for deterministic networks as an alternative to unrealistic stochastic network assumptions. We then relate the optimization of the VICReg objective to mutual information optimization, highlighting underlying assumptions and facilitating a constructive comparison with other SSL algorithms and derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. Building on these results, we introduce a family of SSL methods derived from information-theoretic principles that outperform existing SSL techniques.

An Information-Theoreiic Perspective on Varianc-llvarinne--Cvaainee Regularization
Ravid Shwartz-Ziv' New York University
Randall Balestriero Meta AI, FAIR
Kenji Kawaguchi
National University of Singapore
Tim G. J. Rudner New York University
Yann LeCun
New York University & Meta AI, FAIR
Abstract
Setvariance-Covariinnn Regularization (VICReg) is sulf-supervised learn- ing (SSL) method that has shown promising results variety of tasks. However, the fundamental mechanisms underlying VICReg remain unexplored. In this paper, present
coormatiio-thhorettic perspective the VICReg objective. We begin by deriving information-theorts quantities for deterministic networks as an alternative unrealistic stochastic network assumptions. We then relate the opti- mization of the VICReg objective to mutual information optimization, highlighting underlying assumptions and facilitating a constructive comparison with other SSL algorithms and derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. Building on these results. we introduce a family of SSL methods derived from information-theort principles that outperform existing SSL techniques.
Introduction
Self-supervised learning (SSL) is a promising approach to extracting meaningful representations by optimizing a surrogate objective between inputs and self- self-generated signals. For example, Variance- Invariance-Covate Regularization (VICReg) [7]. widely-used SSL algorithm employing de-correlation mechanism, circumvents learning trivial solutions by applying variance and covariance regularization.
Once the surrogate objective is optimized, the pre-trained model be used feature extractor for a variety of downstream supervised tasks such as image classification, object detection, instance segmentation, or pose estimation [15, 16, 48, 67]. Despite the promising results demonstrated by SSL methods, the theoretical underpinnings explaining their efficacy continue to be the subject of investigation [5, 42].
Information theory has proved a useful tool for improving our understanding of deep neural networks (DNNs), having a significant impact on both applications in representation learning [3] and theoretical explorations [60, 72]. However, applications of information-theorett principles to SSL have made unrealistic assumptions, making many existing Eefommation-thoretti approaches to SSL of limited use. One such assumption is to assume that the DNN to be optimized is stochastic- an assumption that is violated for the vast majority DNNs used in practice. For comprehensive review on this topic, refer to the work by Shwartz-Ziv and LeCun [62].
"Correspondence ravid. shwartz. thwarrz..ivv@yy.. www.nyy..veeeeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this paper, we examine developed for deterministic DNNs, from an information-theonee theoretic perspective. We propose an approach that addresses the challenge of mutual information estimation in deterministic networks by transitioning the randomness from the networks to the input data-a more plausible assumption. This shift allows us to apply an cetemaatioonthoorettee analysis to deterministic networks. To establish connection between the VICReg objective and information maximization, we identify and empirically validate the necessary assumptions. Building on this analysis, we describe differences between different SSL algorithms from an information-theoreti perspective and propose a new family of plug-in methods for SSL. This new family of methods leverages existing information estimators and achieves state-oftthe-ae predictive performance across several benchmarking tasks. Finally, we derive generalization bound that links information optimization and the VICReg objective to downstream task performance, underscoring the advantages of VICReg.
Regularization (VICReg), an SSL method
Our key contributions are summarized as follows:
1. We introduce novel approach for studying deterministic deep neural networks from an ceermation-thhorreee perspective by shifting the stochasticity from the networks to the in- puts using the Data Distribution Hypothesis (Section 3)
2. We establish a connection between the VICReg objective and information-theoredd optimization, using this relationship to elucidate the underlying assumptions of the objective and compare to other SSL methods (Section 4).
3. We propose family of information-theorett SSL methods, grounded in our analysis, that achieve state-of-the-an performance (Section 5).
4. We derive a generalization bound that directly links VICReg to downstream task generalization, further emphasizing its practical advantages over other SSL methods (Section 6).
2 Background & Preliminaries
We first introduce the necessary technical background for our analysis.
2.1 Continuous Piecewise Affine (CPA) Mappings.
rich class of functions emerges from piecewise polynomials: spline operators. In short, given a partition Ω of a domain DR spline of order k is a mapping defined by polynomial of order k on each region w E f2 with continuity constraints on the entire domain for the derivatives of order O,... k 1. As will focus affine splines (k = 1), only define this case for clarity. A K -idimennioona affine spline f produces its output via fz) = Ewen (A, z - b..) (=ew), with input E I D and Aw gK bw E RK Yw E Ω the per-region slope and offset parameters respectively, with the key constraint that the entire mapping is continuous over the domain (TR)).
Deep Neural Networks as CPA Mappings. A deep neural network (DNN) is a (non-linear) operator fe with parameters θ that map a input æ E RR to a prediction y E R The precise definitions of DNN operators can be found in Goodfellow et al. [27]. To avoid cluttering notation, will omit unless needed for clarity. For our analysis, we only assume that the non- lincaritiis in the DNN are CPA mappings-as is the case with (leaky-) ReLU, absolute value, and max-pooling operators. The entire input-output mapping then becomes a CPA spline with an implicit partition S2 the function of the weights and architecture of the network [51, 6]. For smooth nonlinearities, results hold using a first-order Taylor approximation argument.
2.2 Sell-Supervised Learning.
SSL is set of techniques that learn representation functions from unlabeled data, which can then be adapted to various downstream tasks. While supervised learning relies on labeled data, SSL formulates a proxy objective using self-generated signals. The challenge in SSL is to learn useful representations without labels. It aims to avoid trivial solutions where the model maps all inputs to constant output [11, 36). To address this, SSL utilizes several strategies. Contrastive methods like SimCLR and its InfoNCE criterion learn representations by distinguishing positive and negative examples [16, 54]. In contrast, non-contrastive methods apply regularization techniques to prevent the collapse [14, 17, 28).

---

Regularization (VICReg). A widely used SSL method for training joint embedding architectures [7]. Its loss objective is composed of three terms: the invariance loss, the variance loss, and the covariance loss:
The invariance loss is given by the mean-squared Euclidean distance between pairs of embedding and ensures consistency between the representation of the original and augmented inputs.
Invariance loss:
Regularization:
The regularization term consists of two loss terms: the variance loss hinge loss to maintain the standard deviation (over a batch) of each variable of the embedding and the covariance loss, which penalizes off-diagonal coefficients of the covariance matrix of the embeddings foster decorrelation among features.
VICReg generates two batches of embeddings. Z [f(z)........ f(æ))] and Z' [f(x))...., f(')))))))).()) each of size (B x K). Here, æ, and of are two distinct random augmen- tations of a sample 4, The covariance matrix C E 12 xxx is obtained from [Z, Z'I. The VICReg loss can thus be expressed follows:
K
C
a max (0.7 - VCL [(CR,))
+ + z Z' Z'I//N
(1)
K
WEE
Inveriance
Regularization
2.3 Deep Neural Networks and Information Theory
Recently, information thhooretic methods have played an essential role in advancing deep learning by developing and applying cooomationnthhorelee estimators and learning principles DNN training [3, 9, 34, 56, 64, 65, 69, 72]. However,
-hheorri objectives for deterministic DNNs often
face common issue: the mutual information between the input and the DNN representation infinite. This leads to ill ill-posed optimization problems.
Several strategies have been suggested to address this challenge. One involves using stochastic DNNs with variational bounds, where the output of the deterministic network is used as the parameters of the conditional distribution [43, 61]. Another approach, as suggested by Dubois et al. [22]. assumes that the randomness of data augmentation among the two views is the primary source of stochasticity in the network. However, these methods assume that randomness comes from the DNN, contrary to common practice. Other research has presumed random input but has made no assumptions about the network's representation distribution properties. Instead, relies on general lower bounds to analyze the objective [71, 75].
3 Self-Supervised Learning in DNNs: An Information-Theoree Perspective
To analyze information within deterministic networks, we first need establish an information- theoretic perspective on SSL (Section 3.1), Subsequently, we utilize the Data Distribution Hypothesis (Section 3.2) to demonstrate its applicability deterministic SSL networks.
3.1 suff-Supervieed Learning from Information-Theone Viewpoint
Our discussion begins with the MultiView InfoMax principle, which aims to maximize the mutual information T[Z; X') between view X' and the second representation Z. As demonstrated in Federici et al. [24], we can optimize this information by employing the following lower bound:
T'2,X') H(Z) HZ) 2 (() + E. [log ((=[=')).
Here, (Z) represents the entropy of Z. In supervised learning, the labels Y remain fixed, making the entropy term HYY) a constant. Consequently, the optimization is solely focused on the log-loss, Eg [log (2]=()))")) which could be either cross-entropy square loss.
However, for joint embedding networks, degenerate solution can emerge, where all outputs "collapse" into an undesired value [16]. Upon examining Equation (2), we observe that the entropies are not fixed and can optimized. As result, minimizing the log loss alone can lead the representations to collapse into trivial solution and must be regularized.
3

---

3.2 Understanding the Data Distribution Hypothesis
Previously, we mentioned that naive analysis might suggest that the information in deterministic DNNs is infinite. To address this point, we investigate whether assuming dataset is mixture of Gaussians with overrlaplg support can provide a manageable distribution over the neural network outputs. This assumption is less restrictive compared to assuming that the neural network itself stochastic, as concerns the generative process of the data, not the model and training process. For detailed discussion about the limitations of assuming stochastic networks and comparison between stochastic networks vs stochastic input, see Appendix N. In Section 4.2, we verify that this assumption holds for real-world datasets.
The so-called manifold hypothesis allows us treat any point as a Gaussian random variable with low-rank covariance matrix aligned with the data's is manifold tangent space [25], which enables us to examine the conditioning
latent representation with respect the mean of the observation, i.e., || ~ >N x", Σx-) ). Here, the cigenvectors of Eg. align with the tangent space of the data manifold at at" which varies with the position of x² space. this setting, a dataset is considered a collection of distinct points F; n = I,.... N, and the full data distribution is expressed as a sum of Gaussian densities with low-rank covariance, defined as:
X Σ Na Exz (rr-n) with T~ Cai(N).
Here, T is uniform Categorical random variable. For simplicity, we assume that the effective support of N N(x;; Ex;) do not overlap (for empirical validation this assumption see Section 4.2). The effective support is defined as {ar E TR. :p(x) > c). We can then approximate the density function follows:
p(a) N (d)")(())) Dwi..)) /N,
where (z) is the Gaussian density at x and nxx) = arg min,, (x - Din (x x;)).
3.3 Data Distribution Under the Deep Neural Network Transformation
Let us consider an affine spline operator f. as illustrated in Section 2.1, which maps space of dimension D to space of dimension K, where K 2 D. The image or the span of this mapping is expressed as follows:
Im(f) A [f(x) : IR - WEn Aff(w; Au,
(5)
In this equation, Aff(w; Aw; {Awer bw
w) denotes the affine transformation of region w by the per-region parameters Aw, b.. Ω denotes the partition of the input space where æ resides. To practically compute the per-region affine mapping, we set A., the Jacobian matrix of the network at the corresponding input x, and 6 to be defined as f(x) - Awa. Therefore, the DNN mapping composed of alfine transformations on each input space partition region Ω based on the coordinate change induced by A,, and the shift induced by b,..
When the input space is associated with density distribution, this density is transformed by the mapping f. Calculating the density of f(X) is generally intractable. However, under the disjoint arbitrarily increase the density's 's representation power
support assumption from Section 3.2, by raising the number of prototypes N. As a result, each Gaussian's 's support is contained within the region where its means lie, leading to the following theorem:
Theorem 1. Given the setting of Equation (4), the unconditional DNN output density, Z, can be approximated as a mixture of the affinely transformed distributions x x (mm)"
N A "h + bu(a)t) AT Am(z)) Eon; Aw Au(a))
1(T=m)
2
where wxx;) = w E Ω x; E is the partition region in which the prototype x;, lives in. Proof. See Appendix B.
In other words, Theorem 1 implies that when the input noise is small, we can simplify the conditional output density to single Gaussian: X In) N ()( E(rn)), where u(zn) An bw(ee) and (zx)) = A (( Li.,, Aw(an)

---

SimCLR VICReg
375
More Gaussian
SWAV
p-value
0.20
25
0.01
0.0
0.4 Normalized Input Noise (o)
0.8
Gaetto
Plaernt od
Foustei
Figure 1: Left: The network output for SSL training is more Gaussian for small input noise. The p-value of the normality test for different SSL models trained on ImageNet for different input noise levels. The dashed line represents the point at which the null hypothesis (Gaussian distribution) can be rejected with 99% confidence. Right: The Gaussians around each point are not overlapping. The plots show the 12 distances between raw images for different datasets. As can be seen, the distances are largest for more complex real-wor world datasets.
4 Information Optimization and the VICReg Optimization Objective
Building on our earlier discussion, used the Data Distribution Hypothesis to model the conditional output in deterministic networks as Gaussian mixture. This allowed us to frame the SSL training objective as maximizing the mutual information, X() and [(2
However, in general, this mutual information intractable. Therefore, we will use our derivation for the network's representation to obtain tractable variational approximation using the expected loss, which we can optimize.
The computation of expected loss requires to marginalize the stochasticity in the output. We can employ maximum likelihood estimation with a Gaussian observation model. For computing the expected loss over z' samples, must marginalize the stochasticity in Z'. This procedure implies that the conditional decoder adheres to a Gaussian distribution: (Z z,) ~ N (p(x,), I E(r,)).
However, calculating the expected log loss over samples of Z is challenging. We thus focus lower bound the expected log loss over Z' samples. Utilizing Jensen's inequality, we derive the following lower bound:
(dllog Egr [log (<2|x))] 2 Eg/ [log ((122))] (dlog 2T - (2 (2-M(z)) ((x))= Trlog E(x'))
(6)
2
Taking the expectation over Z, we get
Ezja Eg'| log (=|l=))]] = (d 2n (1(x) - 1xx1))² log ((E(x) (E(x'))).
(7)
Combining all of the above, we obtain
I(2; 2 H(2) + d
log 2n En. (((x)) =²²'))² + log([E(r)) [E(')))
(8)
2
The full derivations are presented in Appendix A. To optimize this objective in practice, we approxi- mate xx)xx') using the empirical data distribution
N
L(x1 IN, z'x)
H(Z) log [(x)) (1(-)
2 (m(z+) - 2(
N
(9)
1=1
Regularizer
5

---

4.1 Variance- Sitraaece--aaraane--CCaaaanne Regularization: An Information-Theone Theoretic Perspective
objective. The "invariance term" in Equation (9), which pushes augmentations from the same image closer together, the same term used in the VICReg objective. However, the computation of the regularization term poses a significant challenge. Entropy estimation is well-established problem within information theory, with Gaussian mixture densities often used for representation. Yet, the differential entropy of Gaussian mixtures lacks closed form solution [55].
Next, we connect the VICReg to our information-theorrtii--asss
straightforward method for approximating entropy involves capturing the distribution's 's first two moments, which provides an upper bound the entropy. However, minimizing upper bound doesn't necessarily optimize the original objective. Despite reported success from minimizing an upper bound [46, 53], this approach may induce instability during the training process.
Let Σg denote the covariance matrix of Z. We utilize the first two moments to approximate the entropy we aim to maximize. Because the invariance term appears in the same form the original VICReg objective, we will look only at the regularizer. Consequently, we get the approximation
N
Ex ... xN) E{r,)] -
IN. xx
log
(10)
Theorem Assuming that the eigenvalues of (zx)) and "aa)), along with the differences between the Gaussian means juz;)) and u(xx), are bounded, the solution to the maximization problem
Ez(21 ... IN) [(x)) (E()))
max Σz
log
(11)
involves setting Z to diagonal matrix.
Proof. See Appendix J.
According to Theorem 2, we can maximize Equation (10) by diagonalizing the covariance matrix and increasing its diagonal elements. This goal can be achieved by minimizing the off off-diagonal elements of Ehe covariance criterion of VICReg-and and by maximizing the sum of the log of its diagonal elements. While this approach is straightforward and efficient, it does have a drawback: the diagonal values could tend towards zero, potentially causing instability during logarithm computations. A solution this issue is resulting in the variance term of VICReg. This establishes the link between our coommation-thhorcde objective and the three key components of VICReg.
an upper bound and directly compute the of the diagonal elements,
4.2 Empirical Validation of Assumptions About Data Distributions
Validating our theory, we tested if the conditional output density P(Z becomes Gaussian as input noise lessens. We used a ResNet-50 model trained with SimCLR or VICReg objectives on CIFAR-10, CIFAR-100, and ImageNet datasets. We sampled 512 Gaussian samples for each image from the test dataset, examining whether each sample remained Gaussian the DNN's penultimate layer. We applied the D' Agostino and Pearson's 's test to ascertain the validity of this assumption [19].
Figure (left) displays the p-value as a function of the normalized std. For low noise levels, we reject that the network's s conditional output density is non-Gaussian with an 85% probability when using VICReg. However, the network output deviates from Gaussian as the input noise increases.
Next, we verified our assumption of non-overlapping effective support in the model's 's data distribution. We calculate the distribution of pairwise l2 distances between images across seven datasets: MNIST
[41], CIFARIO, CIFARI 10 [40], Flowers 102 [52], Food101 [12], and FGVAircaft [45]. Figure 1 (right) reveals that the pairwise distances are far from zero, even for raw pixels. This implies that we can use small Gaussian around each point without overlap, validating our assumption as realistic.

---

Self-Supervised Learning Models through Information Maximization
The practical application of Equation (8) involves several key "design choices". We begin by comparing how existing SSL models have implemented it, investigating the estimators used, and discussing the implications of their assumptions. Subsequently, we introduce new methods for SSL that incorporate sophisticated estimators from the field of information theory, which outperform current approaches.
5.1 VICReg vs. SimCLR
In order to evaluate their underlying assumptions and strategies for information maximization, we compare VICReg to contrastive SSL methods such as SimCLR along with non-contrastive methods like BYOL and SimSiam.
Contrastive Learning with SimCLR. In their work, Lee al. [43] drew a connection between the SimCLR objective and the variational bound on information regarding representations by employing the von Mises-Fisher distribution. By applying our analysis for information deterministic networks with their work, we compare the main differences between SimCLR and VICReg:
(i) Conditional distribution: SimCLR assumes a von Mises-Fisher distribution for the encoder, while VICReg assumes Gaussian distribu- tion. (ii) Entropy estimation: SimCLR approx- imated based the finite sum of the input samples. In contrast, VICReg estimates the en- tropy Z solely based on the second moment. Developing SSL methods that integrate these two distinctions form an intriguing direction for future research.
SiR
VICReg
BYO1.
Empirical comparison. We trained ResNet- 18 on CIFAR-10 for VICReg, SimCLR, and BYOL and compared their entropies directly using the pairwise distances entropy estimator. (For more details, see Appendix K.) This esti- mator was not directly optimized by any method and was an independent validation. The results (Figure 2), showed that entropy increased for all methods during training, with SimCLR having the lowest and VICReg the highest entropy.
VICReg+
VICReg+ PairDist
1e0
1el
1e2
1e3
Steps
Figure 2: VICReg has higher Entropy during training. The entropy along the training for dif- ferent SSL methods. Experiments with ResNet-18 18 on CIFAR-10. 10. Error bars represent standard error over 5 trials.
conducted
5.2 Family of alternative Entropy Estimators
Next, we suggest integrating the invariance term of current SSL methods with plug-in methods that optimize entropy.
Entropy estimators. The VICReg objective seeks to approximate the log determinant of the empirical covariance matrix through its diagonal terms. As discussed in Section 4.1, this approach has its drawbacks. An alternative to employ different entropy estimators. The LogDet Entropy Estimator [74] is one such option, offering tighter upper bound. This estimator employs the differential entropy of a order with scaled noise and has been previously shown be a tight estimator for high-dimen nigh-dimensional features, proving robust to random noise. However, since this estimator provides an upper bound on entropy, maximizing this bound doesn't guarantee optimization of the original objective. To counteract this, also introduce lower bound estimator based the pairwise distances of individual mixture components [38]. In this family, a function determining pairwise distances between component densities is designated for each member. These estimators computationally efficient and typically straightforward to optimize. For additional entropy estimators, Appendix F. Beyond VICReg. these methods serve plug-in estimators for numerous SSL algorithms. Apart from VICReg, we also conducted experiments integrating these estimators with the BYO algorithm.
7

---

Table The proposed entropy estimators outperform previous methods. CIFAR-IO, CIFAR- -00, and Iny-ImageNet top-1 accuracy under linear evaluation using ResNet-18, 18, ConvNetX and VIT as backbones. Error bars correspond to one standard error over three trials.
Method
CIFAR-10
-mag-emageNet
CIFAR-I00
ResNet-18
ConyNetX
VIT
ConvNetX
VIT
SimCLR
89.72 0.05
50.86 0.13
51.16 + 0.13
67.21 0.24
67.31 0.18
Barlow Twins
88.81 + 0.10
51.34 = 0.10
51.40 + 0.16
68.54 + 0.15
68.02 + 0.12
SwAV
89.12 0.13
50.76 0.14
51.54 + 0.20
68.93 0.14
67.89 0.21
MoCo
89.46 + 0.08
52.36 0.21
53.06 + 0.21
70.32 - 0.15
69.89 + 0..14
VICReg
89.32 0.09
51.02 0.26
52.12 + 0.25
70.09 0.20
70.12 0.17
BYOL
89.21 + 0.11
52.24 0.17
53.44 + 0.20
70.01 = 0.27
69.59 + 0.22
VICReg + PairDist (ours)
90.37 0.09
52.61 0.15
53.70 + 0.13
71.10 = 0.16
70.50 + 0.19
VICReg + LogDet (ours)
90.27 0.08
52.91 0.17
54.89 + 0.20
71.23 0.18
70.61 + 0.17
BYOL PairDist (ours)
90.19 + 0.14
53.47 + 0.22
54.33 + 0.21
71.39 + 0.25
71.09 + 0.24
BYOL LogDet (ours)
90.11 0.16
53.19 = 0.25
54.67 + 0.27
71.20 0.21
70.79 = 0.26
Setup. Experiments were conducted three image datasets: CIFAR-10. CIFAR-100 [39], and Tiny-ImageNet [20]. For CIFAR-1O, ResNet- 18 [31] was used. In contrast, both ConvNeXt [44] and Vision Transformer [21] were used for CIFAR-100 and Tiny-ImageNer. For comparison, examined the following SSL methods: VICReg, SimCLR, BYOL, SwAV [14), Barlow Twins [73],
and MoCo [33]. The quality of representation was assessed through linear evaluation. A detailed description of different methods can be found in Appendix H.
Results. As evidenced by Table 1, the proposed entropy estimators surpass the original SSL methods. Using a more precise entropy estimator enhances the performance of both VICReg and BYOI,, compared their initial implementations. Notably, the pairwise distance estimator, being a lower bound, achieves superior results, resonating with the theoretical preference for maximizing true entropy's lower bound. Our findings suggest that the astute choice entropy estimators, guided by our framework, paves the way for enhanced performance.
A Generalization Bound for Downstream Tasks
In earlier sections, we linked information theory principles with the VICReg objective. Now, we aim to extend this link to downstream generalization via a generalization bound. This connection further aligns VICReg's generalization with information maximization and implicit regularization.
Notation.
Consider input points x, outputs y E R", labeled training data ((2,, y:)) of size and unlabeled training data Š ((a) x >> of size m, where x and 2;+ share the same (unknown) label. With the unlabeled training data, we define the invariance loss
Ts(So) = [ So(a
where fe is the trained representation on the unlabeled data S. We define a labeled loss lx,y (w) = ||W fx) - yll where w vec[W] E Rdd is the vectorization of the matrix W E Brxd Let ws vec [Ws] be the minimum norm solution as Ws minimizew: [[W"|] such that
W' E arg min W
(ffo(a.) -v.11
TI
We also define the representation matrices
2 = [f (xx)) E Rdx= and Z3 (f(xt).... ..... ..,.f(mm) E Rdxm

---

and the projection matrices
Pz. = - 1-==s= (ZsZs T) and P2g - - Zg (ZgZs T)
We define the label matrix Ys [M/....,, .9] E Rexr and the unknown label matrix Ys [1 :.... ] E Rmxxr, where Vi is the unknown label of xe Let F be a hypothesis space of fa- For given hypothesis space F, we define the normalized Rademacher complexity
Rm
1 Vm
S.E
Σ E.ll(()))) -((tt)))(()))))]]) SEF
sup
where .I....; : Sm are independent uniform random variables in 1-, It normalized such that Rm (F) = (1) as m + 0C.
6.1 A Generalization Bound for Varfanne-Invariancc-Cvvaaa Regularization
Now we will show that the VICReg objective improves generalization on supervised downstream tasks. More specifically, minimizing the unlabeled invariance loss while controlling the covariance ZgZg and the complexity of representations (F) minimizes the expected labeled loss:
Theorem 3. (Informal version). For any 5 > O, with probability at least 1 8-
2
2R (F) vm
Ex, Ex.ykay((s)) I (f(o)
P Yslle + P223
+ Qm,ns
(12)
where 2m,n= 1nt//// / (1/10///n) + as m, + co. In 2m,, the value of G for the term decaying at the rate 1/vm depends on the hypothesis space of fe and w whereas the term decaying at the rate 1/vn is independent of any hypothesis space.
Proof. The complete version of Theorem 3 and its proof are presented in Appendix I.
The term [P Ysli in Theorem 3 contains the unobservable label matrix Yg. However, we minimize this term by using [P2g Yslle I [P ||sll and by minimizing P ll The factor [|22 |l is minimized when the rank of the covariance Zz is maximized. This can be enforced by maximizing the diagonal entries while minimizing the off-diagonal entries, as done VICReg.
The term |P Ysllr contains only observable variables, and we can directly measure the value of this term using training data. In addition, the term || Ysll is also minimized when the rank of the covariance ZgZs is maximized. Since the covariances ZsZs and ZgZg concen- trate to each other via concentration inequalities with the error the order of O (ln(1/1)))//)) R., (F) (F)//F((I)))) (In(1/8))/m), we can also minimize the upper bound on || Zg Ys || by maximizing the diagonal entries of 7g while minimizing its dif-diagonal entries, as is done in VICReg.
Thus, VICReg can be understood
method minimize the generalization bound in Theorem by minimizing the invariance loss while controlling the covariance to minimize the label-agnostic upper bounds on |P2 2g Ysll P and P 2: Yslle. If we know partial information about the label Ys of the unlabeled data, we can use it to minimize |P 2g Ysllf and |P Yslle directly.
6.2 Comparison of Generalization Bounds
The SimCLR generalization bound [58] requires the number of labeled classes go infinity to close the generalization gap, whereas the VICReg bound in Theorem 3 does not require the number of label classes to approach infinity for the generalization gap to go to zero. This reflects that, unlike SimCLR, VICReg does not use negative pairs and thus does not use loss function based on the implicit expectation that the labels negative pair are different. Another difference is that our VICReg bound improves on n. This is because Saunshi et [58] assumes partial access to the true distribution per class for setting, which removes the importance of labeled data size and is not assumed in our study.
increases, while the previous bound of SimCLR [58] does not depend

---

Consequently, the generalization bound in Theorem 3 provides a new insight for VICReg regarding the ratio of the effects of m v.s. through G G/In(116)/m + /in(1/8)/n. illuminates the advantages of VICReg over standard supervised training. That is, with standard training, the generalization bound via the Rademacher complexity requires the complexities of hypothesis spaces, R., (W)) and R., (FF/ /n, with respect to the size of labeled data n, instead of the size of unlabeled data m. Thus, Theorem 3 shows that using SSL,, we can replace the complexities of hypothesis spaces in terms of with those in terms of m. Since the number of unlabeled data points is typically much larger than the number labeled data points, this illuminates the benefit of SSL.
Finally, Theorem also
6.3 Understanding Theorem 2 via Mutual Information Maximization
Theorem 3, together with the result of the previous section, shows that, for generalization in the downstream task, it is helpful to maximize the mutual information 7 T(Z; X') in SSL via minimizing the invariance loss (s(fe) while controlling the covariance ZgZs The term 2mm (FF) /mm captures the importance of controlling the complexity of the representations fa. To understand this term further, let consider discretization of the parameter space of JF to have finite [J] < co. Then, by Massart's Finite Class Lemma, we have that (F) Cv/n [F] for
constant C > 0. Moreover, Shwartz-Ziv [60] shows that we can approximate in |F by 2T(2))) Thus, in Theorem 3, the term (f) + 2
Vi v [P2g Ygll + 1 PP2 Yslle corresponds to I (Z; X') which we want to maximize while compressing the term of 2Rm which corresponds to I(; [23, 63, 66].
Although we can explicitly add regularization on the information to control 2Rm (F)/ Vm, it is possible that [(2;X] X') and (F)/vm are implicitly regularized via implicit bias through design choises [29, 68, 30]. Thus, Theorem connects the information-thhorre understanding of VICReg with the probabilistic guarantee on downstream generalization.
7 Limitations
In our paper, we proposed novel methods for SSL premised on information maximization. Although methods demonstrated superior performance some datasets, computational constraints pre- cluded us from testing them on larger datasets. Furthermore, our study hinges on certain assumptions that, despite rigorous validation efforts, may not hold universally. While strive for meticulous testing and validation, it's crucial to note that some assumptions might not be applicable in all scenarios conditions. These limitations should be taken into account when interpreting study's results.
8 Conclusions
We analyzed the Variancelnvarrance-Covvaana Regularization for self-supervised learning through an information-theoredr lens. By transferring the stochasticity required for an information-thhoreee analysis the input distribution, we showed how the VICReg objective can be derived from information-thhoreee principles, used this perspective to highlight assumptions implicit in the VI- CReg objective, derived a VICReg generalization bound for downstream tasks, and related it to information maximization.
Building on these findings, we introduced a new JICReg-inspired SSL objective. Our probabilistic guarantee suggests that VICReg can be further improved for the settings of partial label information by aligning the covariance matrix with the partially observable label matrix, which opens up several avenues for future work, including the design of improved estimators for information-theoretic quan- tities and investigations into the suitability of different SSL methods for specific data characteristics.
10

---

References
[1] Alessandro Achille, Giovanni Paolini, and Stefano Soatto. Where is the information in a deep neural network? arXiv preprint arXiv: 1tt.//995./2213,,. 2019.
[2] N.A. Ahmed and D.V. Gokhale. Entropy expressions and their estimators for multivariate distributions. IEEE Transactions on Information Theory, 35(3):688-692) 1989, doi: 10.1109/ 18.30 18.30996.
[3] Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00441 2016.
[4] Rana Ali Amjad and Bernhard Claus Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence, 2019.
[5] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: 2019.
[6] Randall Balestriero and Richard Baraniuk. A spline theory of deep networks. In Proc. ICML, volume 80, pages 374-383, Jul. 2018
[7] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varince-immrrinne-coaaiiee regular- ization for self-supervised learning, arXiv preprint arXiv:2105.04400 2021.
[8] Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 4822 2002.
[9] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv: arXiv:/801.044 08060 2018.
[10] Itamar Ben-Ari and Ravid Shwartz-Ziv. Attentioned convolutional Istm inpaintingnetworkk for anomaly detection in videos. arXiv preprint arXiv: I. 10222, 2018.
[11] Ido Ben-Shaul, Ravid Ahwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineering self-supervised learning. arXiv preprint 2.0.::23005.5661. 2023.
[12] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 100-iing discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[13] Brendon J Brewer. Computing entropies with nested sampling. Entropy, 19(8):422, 2017.
[14] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 13:9412-9924, 2020.
[15] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[16] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[17] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[18] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-85) 828, 2012.
11

---

[19] Ralph B. D' A'Aostinno An omnibus test of normality for moderate and large size samples. Biometrika, 58(2):341334, 1971. ISSN 00063444. URL
http://www...fooooog//bbb/22222
[20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. leee, 2009.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
[22] Yann Dubois, Benjamin Bloem-Reddy. Karen Ullrich, and Chris J Maddison, Lossy compression for lossless prediction. Advances in Neural Information Processing Systems, 34, 2021.
[23] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2019.
[24] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017 2020.
[25] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049 2016.
[26] Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. arXiv preprint arXiv: 1810.05728, 2018.
[27] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning, volume 1. MIT Press, 2016.
[28] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to ooff-suerrvied learning, Advances in neural information processing systems, 33:21271-21284, 2020.
[29] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems, pages 6151 6151- 2017.
[30] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pages 9461-9471, 2018.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR,
2015.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[33] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[34] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: /808. 068.06670 2018.
[35] Marco Huber, Tim Bailey, Hugh Durrant- Durrant-Whyte, and Uwe Hanebeck. On entropy approximation for gaussian mixture random vectors. In IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181 188, 09 2008. doi: 10. 100.00000...00
4648062.
12

---

[36] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representa- lions; 2022. URL.
[37] Kenji Kawaguchi, Zhun Deng, Kyle Luh, and Jiaoyang Huang. Robustness Implies General- ization via Data-Dependent Generalization Bounds. In International Conference on Machine Learning (ICML). 2022.
[38] Artemy Kolchinsky and Brendan D Tracey. Estimating mixture entropy with pairwise distances. Entropy, 19(7):361, 2017.
[39] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[40] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
[41] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(11))2278-23:) 2324, 1998.
[42] Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning, Advances in Neural Information Processing Systems, 34, 2021.
[43] Kuang-Huck Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances Neural Information Processing Systems, 34, 2021.
[44] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11976- 1986, 2022.
[45] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.51524 2013.
[46] Julieta Martinez, Jashan Shewakramani, Ting Wei Liu, Ioan Andrei Bårsan, Wenyuan Zeng, and Raquel Urtasun. Permute, quantize, and fine-tune: Efficient compression neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15699-157008, 2021.
[47] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual infor- mation. In International Conference on Artificial Intelligence and Statistics, pages 875-884. PMLR, 2020.
[48] Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invarian rep- resentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[49] Neeraj Misra, Harshinder Singh, and Eugene Demchuk. Estimation of the entropy of a mul- tivariate normal distribution. Journal of Multivariate Analysis, 92(2):324-342, 2005. ISSN 0047-259X. doi: https://doi.orr/I 10. jmm..
[50] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2012.
[51] Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Proc. NeurIPS, pages 2924-2932, 2014.
[52] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[53] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing systems, pages 271-279, 2016.
13

---

[54] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.0388, 2018.
[55] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6): 1191-1253, 2003.
[56] Zoe Piran, Ravid Shwartz-Ziv, and Naftali Tishby. The dual information bottleneck. arXiv preprint 20l.:2000.04441,, 2020.
[57] Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information. CoRR, abs/1905.06922, 2019. URL
http://arxiv.oggaass119933 1905.0922..
[58] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In Interna- lional Conference on Machine Learning, pages 5628-5637. PMLR, 2019.
[59] Shai Shalev-Shwarzz and Shai Ben-David. Understanding machine learning: From theory to algorithmss Cambridge university press, 2014.
[60] Ravid Shwartz-Ziv. Information flow in deep neural networks. arXiv preprint arXiv:2202.06749 2022.
[61] Ravid Shwartz-Ziv and Alexander A Alemi. Information in infinite ensembles of infinitely-wide neural networks. In Symposium on Advances in Approximate Bayesian Inference, pages 1-17. PMLR, 2020.
[62] Ravid Shwartz-Ziv and Yann LeCun. To compress or not to cemppees--ell--suervvii learning and information theory: A review. arXiv preprint arXiv:2304, 20550533 2023.
[63] Ravid Shwartz-Ziv and Naftali Tishby. Compression of deep neural networks via information, (2017). arXiv preprint arXiv://03.0000
2017.
[64] Ravid Shwartz-Zii and Naftali Tishby, Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00812 2017.
[65] Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation compression and generalization in deep neural networks, 2018
[66] Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What do maximize in self- supervised learning? arXiv preprint arXiv:2207.10088/ 2022.
[67] Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann Le- Cun, and Andrew Gordon Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. arXiv preprint 2rXiv:2205.10279, 2022.
[68] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 19(1):2822-285 2018
[69] Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory, pages 3437-3452. PMLR, 2020.
[70] Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907 13625, 2019.
[71] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pages 9929-99399 PMLR, 2020.
[72] Aolin Xu and Maxim Raginsky. Information-theoreticc analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, 30, 2017.
14

---

[73] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 12310-12320. PMLR, 2021.
[74] Zhanghao Zhouyin and Ding Liu. Understanding neural networks with logarithm determinant entropy estimator. arXiv preprint arXiv:2105.03705, 2021.
[75] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990, PMLR, 2021.
15

---

Appendix
Table of Contents
This appendix organized as follows:
Appendix A, we provide a detailed derivation of the lower bound of Equation (8).
In Appendix B, provide full proof of our Theorem on the network's representation distribu- tion.
Appendix C, we provide additional empirical validations. Specifically, we empirically check if the optimal solution to the information maximization problem in Section 4.1 is the diagonal matrix.
Appendix show the collapse phenomenon under Gaussian Mixture Model (GMM) using Expectation Maximization (EM) and demonstrate how is related to SSL and how we can prevent
Appendix E provides additional details the SimCLR method.
Appendix F provides detailed review of entropy estimators, their implications, assumptions, and limitations.
Appendix G, we provide proofs for known lemmas that we are using throughout our paper.
In Appendix H, provide detailed information on the hyperparameters, datasets, and architec- tures used in our experiments in Section 5.2.
Appendix I, we provide full proof our generalization bound for downstream tasks from Section 6.
In Appendix J. we provide full proof of the theorems for Section 4.1 on the connection between information optimization and the VICReg objective.
Appendix K provides experimental details experiments conducted in Section 5.1 for entropy comparison between different SSL. methods.
Appendix I., we provide detailed information on the reproducibility of our study.
Appendix O, we discuss the broader impact of our work. This section explores the implications. significance, and potential applications of our findings beyond the scope of the immediate study.
16

---

Appendix A Lower bounds on Ey [log (z|2²)]
In this section of the supplementary material, we present the full derivation of the lower bound on E, log ((2[x')] Because X is a Gaussian, we can write it as Z' =uxx') + L(x')e where c (0, 1) and L(x')" E(r)). Now, setting Σ, = I, will give us:
By [log (2|1x²)]
(13)
BBay [log (=[122)]
dd
(2 (2 2/)² (1)) (2 2)
=Bal
log 2n
(14)
2
d
log 2T
E 2/aa" (2- 2/)9|
(15)
log 2T
E.
(2 (2-1(z²) u(x)) -(x')e)²
(16)
(s (=-) )) - ( mlx²) [(x'))) + (((6)))(((())))
log 2T
E,
(17)
E. (2=)((a)) )) + (z (((z))((z")) B.Id 1 E. L())
log 2T
(18)
2
1
log 2T
2 (z u(x'))
rr (1x'
(19)
2
where Eg [log [(2|2')] Es [log Bzy [=(2|2)]] 2 E. [[og((222)] by Jensen's inequality.
E. [e] = O and E. [e (L(x')"((x'')) = Tr log (2') by the Hutchinson's estimator.
d log 2TT-
1
Esje [2 Dlog [=]
-Baja
(2 (z-1(z²))²
Tr log I(x')
(20)
2
2
2
d
log 2m
1
E//a
- -((')))))
Tr I(x')
(21)
2
2
d
1
(m(x) + L(x)e - m(x'))²] - log Mix')
log 2n
E.
(22)
2
2
d
log 2m
1
E.
(m(x)) m(('))²] E. [(((x)) - m(x1)) ((x)c)
2
2
E, [e_ L(x)* L(x)e] Tr log Tr Elx')
2
(23)
d
1 (u(x)) =2=a))²²
1 Tr Tog E(x)
log 2T
Tr -ol E(x')
2
2
2
2
(24)
d
1 (u(x)) - u(x'))²
log 2T
log (|E(²) -E(2'))))
(25)
2
2
2
17

---

Appendix B Data Distribution after Deep Network Transformation
Theorem Given the setting of Equation (4), the unconditional DNN output density denoted as Z approximates (given the truncation of the Gaussian its effective support that is included within single region w of the DN's input space partition) a mixture of the affinely transformed distributions ([=)))))) e.g. for the Gaussian case
2~
N
M (Aaaa)) + bw(a;)) AT Daz w(mtt) Sw;;
Autz;))
T=n
n=1
where (xx,)) = w EQ F
2n E w is the partition region in which the prototype v,, lives in.
Proof. We know that If Sa p(alaa )da as then f is linear within the effective support of p. Therefore, any sample from p will almost surely lic within single region w Ω2 and therefore the entire mapping be considered linear with respect p. Thus, the output distribution is linear transformation the input distribution based on the per-region affine mapping.
Appendix C Additional Empirical Validation
To validate empirically Theorem 2, we checke empirically if the optimal solution for
Ezl
Σ log
Σ2Ix
a diagonal matrix. We trained VICReg on ResNet18 on CIFAR-10 and did random perturbations (with different scales) for Ez. Then, for each perturbation, we calculated the average distance of this perturbed matrix from diagonal matrix and the actual value of the term
|2||Ezix] Ezpx
Σ log
In Figure 3, we plot the difference from the optimal value of this term function of the distance from the diagonal matrix. As we can sec, we get an optimal solution where we are close to the diagonal matrix. This observation gives empirical validation of Theorem 2.
The distance from the optimal value of Σ, ReppellRietrs
0.16
0.12
0,88
0.36
Distance from diagonal
Figure 3: The optimal solution for the optimization problem a diagonal matrix. The average distance from diagonal matrix for different perturbation scales. Experiments conducted CIFAR- 10 with the ResNet-13 18 network.
18

---

4(Z)=45
H(Z)=40
H(Z)=34
2 data
t
data
data
centroids
centroids
centroids
Figure 4: Evolution of GMM training when enforcing a one-to-one mapping between the data and centroids akin to K-means i.e. using a small and fixed covariance matrix. We see that collapse does not occur. Left In the presence of fixed input samples, observe that there is no collapsing and that the entropy of the centers is high. Right when we make the input samples trainable and optimize their location, all the points collapse into single point, resulting in sharp decrease entropy.
Appendix D EM and GMM
Let examine toy dataset the pattern of two intertwining moons illustrate the collapse phenomenon under GMM (Figure 1, right). We begin by training classical GMM with maximum likelihood, where the means initialized based on random samples, and the covariance is used the identity matrix. red dot represents the Gaussian's 's mean after training, while blue dot represents the data points. In the presence of fixed input samples, that the entropy of the centers high (Figure 4, left). However, when we make the input samples trainable and optimize their location, all the points collapse into single point, resulting in sharp decrease in entropy (Figure 4, right).
observe that there is collapsing and
To prevent collapse, follow the K-means algorithm in enforcing sparse posterions, i.e. using small initial standard deviations and learning only the mean. This forces a one-to-one mapping which leads all points be closest to the mean without collapsing. resulting in high entropy (Figure 4 middle, the Appendix). Another option to prevent collapse is to use different learning rates for input and parameters. Using this setting. the collapsing of the parameters does not maximize the likelihood. Figure 1 (right) shows the results of GMM with different learning rates for learned inputs and parameters. When the parameter learning rate is sufficiently high in comparison the input learning rate, the entropy decreases much more slowly and no collapse occurs.
Appendix E SimCLR
In contrastive learning, different augmented views of the same image are attracted (positive pairs), while different augmented views are repelled (negative pairs). MoCo [33] and SimCLR [16] recent examples of self-supervised visual representation learning that reduce the gap between self- supervised and fully-supervised learning. SimCLR applies randomized augmentations to create two different views, x and and encodes both of them with shared encoder, producing representations Fz and Ny. Both T. and ry are /2- 12-normalized. The SimCLR version of the InfoNCE objective is:
image
1
1 To
e
E
log
Lk=1
where 7) is temperature term and K is the number of views in minibatch.
19

---

Appendix F Entropy Estimators
Entropy estimation is of the classical problems in information theory, where Gaussian mixture density is one of the most popular representations. With a sufficient number of components, they can approximate any smooth function with arbitrary accuracy. For Gaussian mixtures, there is, however, no closed-form solution to differential entropy. There exist several approximations the literature, including loose upper and lower bounds [35]. Monte Carlo (MC) sampling is way approximate Gaussian mixture entropy. With sufficient MC samples, an unbiased estimate of entropy with an arbitrarily accurate can be obtained. Unfortunately, MC sampling is very computationally expensive and typically requires a large number of samples, especially in high dimensions [13]. Using the first two moments of the empirical distribution, VIGCreg used one of the most straightforward approaches for approximating the entropy. Despite this, previous studies have found that this method is a poor approximation of the entropy many cases [35]. Another option is use the LogDet function. Several estimators have been proposed to implement it, including uniformly minimum variance unbiased (UMVU) [2], and Bayesian methods [49]. These methods, however, often require complex optimizations. The LogDet estimator presented in [74] used the differential entropy order entropy using scaled noise. They demonstrated that it can be applied to high-dimensional features and is robust to random noise. Based Taylor-series expansions, [35] presented lower bound for the entropy of Gaussian mixture random vectors. They use Taylor-series expansions the logarithm of each Gaussian mixture component to get analytical evaluation of the entropy measure. In addition, they present a technique for splitting Gaussian densities to avoid components with high variance, which would require computationally expensive calculations. Kolchinsky and Tracey [38] introduce a novel family estimators for the mixture entropy. For this family, a pairwise- -dittincd function between component densities is defined for each member. These estimators are computationally efficient as long as the pairvise-distance function and the entropy of each component distribution easy compute. Moreover, the estimator is continuous and smooth and is therefore useful for optimization problems. In addition, they presented both a lower bound (using Chernoff distance) and an upper bound (using the KL divergence) the entropy, which are exact when the component distributions are grouped into well-separated clusters,
Appendix G Known Lemmas
We use the following well-known theorems as lemmas in our proofs. We put these below for completeness. These are classical results and not our results.
Lemma G.1. (Hoeffding's inequality) Let Xn,...Xn be independent random variables such that < X4 < b almost surely. Consider the average of these random variables,
S. 1 -Xx, ... + Xn). Then, for all t 0,
(n(11/)
IPs
E. [[Sa] - S. 2 (b 2b-a)/ a)p
1 6,
2n
and
In(1/8) 2n
Ps Sn [[S]] (b V)v
< 6.
Proof. By using Hoeffding's inequality, we have that for all t 0,
(- (-70- 11 a)2
2nt²
Ps E [S.] - Sn 2 1) s exp
and
2nt²
Ps (Sn - E[(S]] 2) $ exp (- - a)2
20

---

Setting 8 exp (xp(-1ent)
2n12 (b-a)]
and solving for >>
2nt²
1/8 = exp
(b - a)²
2n12
In(1/6) (b a)2
(b - a)² lm(116) 1?2
2n
= (b v)v In(1/5)
2n
It has been shown that generalization bounds can be obtained via Rademacher complexity 50, 59]. The following is trivial modification of [50, Theorem 3.1] for one-sided bound on the nonnegative general loss functions:
Lemma G.2. Let G be a set of functions with the codomain [0, M). Then, for any 5 > O, with probability at least 5 over an i.i.d. draw of samples 5 (q. (q))= the following holds for all
3 E G:
Eg((((g)] S
m Σ xbai)) 22 + Mv
1
In(1/8)
(26)
2m
where Rm (9) == Es.elsup m ETh (x)(4))) and E Em are independent uniform random variables taking values {-1,1}.
Proof. Let S = mm and S' (g)) Define
(S) = sup Ez. (v)) PEG
- m Σ (gg).
(27)
To apply McDiarmid's inequality to (S)). we compute an upper bound on (s(( - (SS) where S and 5' be two test datasets differing by exactly point of an arbitrary index in: i... S, S for all i * in and S., + S% Then,
((4))) - =b(l))
M
((S") ((S) S sup Peg
(28)
m
m
Similarly, (p() ((8') < M Thus, by McDiarmid's inequality, for any > O, with probability at least 1 6,
In(1/0)
((S) < (s(p(() + M
(29)
2m
Moreover,
1 m
1 m
Es(p(S)] Es
sup Es WEg
((gh)
(qt)
(30)
m
m
1 m
< Es,s
sup WeG m
(v(4)) v(q:))
(31)
m (1((()))) - -((a))
< Bs,s,s'
sup weg m
(32)
m
< 2Eg
sup WES m
(4)) = 2R (9),
(33)
where the first line follows the definitions of each term, the second line Jensen's inequality and the convexity of the supremum, and the third line follows that for each & E (-1,+11, the distribution of each term ((((())) 8/) ((f()(() yt)) is the distribution of (f(f(=)), {(f v4) since S and
S' are drawn iid with the same distribution. The fourth line uses the subadditivity of supremum. O
21

---

Appendix H Implentation Details for Maximizing Entropy Estimators
In this section, we will provide more details on the implantation of the experiments conducted in Section 5.2.
Setup Our experiments are conducted on CIFAR-10 [40]. We use ResNet-18 [32] as backbone.
The experimental process organized into two sequential stages: unsupervised pretraining followed by linear evaluation. Initially, the unsupervised pretraining phase is executed, during which the encoder network is trained. Upon its completion, we transition to the linear evaluation phase, which serves as an assessment tool for the quality of the representation produced by the pretrained encoder.
Training Procedure:
Once the pretraining phase is concluded, we adhere to the fine-tuning procedures used in established baseline methods, as described by [14].
During the linear evaluation stage, we start by performing supervised training of the linear classifier. This is achieved by using the representations derived from the encoder network while keeping the network's is coefficients frozen, and applying the same training dataset. Subsequently, we measure the test accuracy of the trained linear classifier using separate validation dataset. This approach allows us to evaluate the performance of our model in a robust and systematic manner.
The training process for each model unfolds over 800 epochs, employing batch size of 512. We utilize the Stochastic Gradient Descent (SGD) optimizer, characterized by a momentum of 0.9 and a weight decay of 1e" 4. The learning rate is initiated at 0.5 and is adjusted according to cosine decay schedule complemented by a linear warmup phase.
During the data augmentation process, two enhanced versions of every input image are generated. This involves cropping each image randomly and resizing it back to the original resolution. The images are then subject to random horizontal flipping, color jittering, grayscale conversion, Gaussian blurring, and polarization for further augmentation.
For the linear evaluation phase, the linear classifier is trained for 100 epochs with a batch size of 256. The SGD optimizer again employed, this time with momentum of 0.9 and weight decay. The learning rate is managed using a cosine decay schedule, starting at 0.2 and reaching a minimum of 2e
Appendix I A Generalization Bound for Downstream Tasks
In this Appendix, we present the complete version of Theorem 3 along with its proof and additional discussions.
L1 Additional Notation and details
We start to introduce additional notation and details. We use the notation of a E X for an input and y E y C R- for an output. Define p(y) P( = y) be the probability of getting label y and p(y) = Σラ 1 [3; = y} to be the empirical estimate of p(y). Let Ç be an upper bound on the norm of the label as yyl|= S for all y E y. Define the minimum norm solution Ws of the unlabeled data as Ws minimizew ||W'|| |F s.t. W' E arg minw m E [[ (at - 9 (a Ill Let ks be a data-dependent upper bound on the per-sample Euclidian norm loss with the trained model as [[s v < Kg for all (x, y) E X x XV. Similarly, let Kg be data-dependent upper bound on the per-sample Euclidian norm loss as [W)) - yl < ng for all (I. y) E X' x y Define the difference between Ws and W3 by == ||s W8ll2- Let W be hypothesis space of W such that V E3, (supwew.ser E & (x - W Wf(x;) II] the normalized Rademacher complexity of the set f + (xg*(x+) Wf(x+) I|| W E W. E F). we denote by ς a upper bound on the per-sample Euclidian norm loss as || W f(x) - yll s 6 for all (z. y, W. f) E X x y x W x F.
W3 E W. We denote by R., FW = _L
We adopt the following data aata-generating process model that was used in previous paper on analyzing contrastive learning 158, 10]. For the labeled data, first, y is drawn from the distribution p on ) and then x drawn from the conditional distribution D, conditioned on the label y. That is, we have the join distribution D(z, y) (z)x)()) with ((I), b)) D". For the unlabeled data,
22

---

first, each of the unknown labels y² and y is drawn from the distritbuion p. and then each of the positive examples x+ and x++ is drawn from the conditional distribution Du+ while the negative example x' is drawn from the D, Unlike the analysis of contrastive learning, we do not require negative samples. Let T3 be data-dependent upper bound on the invariance loss with the trained representation as ( - fo( || I g for all ( ~ D2 and y E y. Let T be a data- -ntaeinndeen upper bound on the invariance loss with the trained representation as|| f()
f())]l _ for all (z, ~ Dy E y, and f E F. For simplicity, we assume that there exists function 9 such that y g" (x) E IR' for all (z, y) E X' x V Discarding this assumption adds the average of label noises to the final result, which goes to zero as the sample sizes n and m increase, assuming that the mean of the label noise is zero.
0.2 Proof of Theorem
Proof of Theorem 3. Let W -s where Ws is the the minimum norm solution as Ws minimizew [|W'||F s.t. W" E arg minw 1 - [ [o(;)) - will. Let W = Ws where W3 is the minimum norm solution as W = Wg = minimizew: [["]]e s.t. W e arg minw 1 Σ [W - g" (12) Since y y= gx), m
y= W
fo(x) (g W" fo())) = W fo(o())+ + ((()
where p(x)= - (x) W fz). Define Ls(w)= = h/ E. I fo(z)) 3i1l- Using these,
Ls(w)=
[ vill
[W - W' =(xx:)]
[W W fo(rill 1
Σ
|xx((
n
72
a Σ
[o
||((=))
where W- = W- - W*. We now consider new fresh samples H. Dy, for 1,..., n to rewrite the above further as:
Ls(w)=
Wfl(z) - Σ |x((x))
- (V - 1
Σ
||+((x))1
[W - 1 Σ ( WS((a))ll
||(((x))|
72
1=1
1
fo
(fo fo(x:))
[xx((;))|
This implies that
n
W (a.))l S Ls(w) + 1 n
Σ
(fo..) - fo(x:)))
72
n i=1
23

---

Furthermore, since y = W fz) + ((x), by writing is - fe(.) + (TE:) (where y; = since I; D,, for n)),
W fo( 1 n1 1
Σ
-
71
(o(,)) 81 ((
n
(e(.)) 3il - Σ ||,()
Combining these, we have that
|| so(a,) Dell S Ls(w)
1 +
Σ -
-
(34)
T1
n
|((x))|
+ n
Σ |((())))
To bound the left-hand side of equation 34, now analyze the following random variable:
Ex,y (I YII (sso(a.) Tsll.
(35)
where 34 since T. Du. for I...., Importantly, this means that as Ws depends ys Ws depends on Vi- Thus, the collection random variables fe (āz) Fi li.l..... s fe(,)) Tn is not independent. Accordingly, we cannot apply standard concentration inequality to bound equation 35. A standard approach in learning theory is to first bound equation 35 by foo yll - 1 Ex= fo
Tall < supwew Ex, [ fo (z) - yl 1 Σ W fe.) 7 for some hypothesis space W (that is independent of S) and realize that the right- haanthhndd side contains Dall for which
the collection of independent random variables || (ā)) - 51 Il. [/ can utilize standard concentration inequalities. This reasoning leads the Rademacher complexity of the hypothesis space W. However, the complexity of the hypothesis space W can be very large, resulting in loose bound. In this proof, show that can avoid the dependency on hypothesis space W by using a very different approach with conditional expectations to take care the dependent random variables W Sa (z1) Ti .l...., [Ws WSsfa(n, (nn,) I Intuitively, utilize the fact that for these dependent random variables, there is a structure of conditional independence, conditioned on each y E y.
We first write the expected loss as the sum of the conditional expected loss:
Ex,y||WsJo(X) YII]=
Σ
Ex,y [Ilsso(X) YII | Y = y) =
vEY
Σ
Ex, (llssf(,,)) - vI|lPYY = v),
vEY
where X, is the random variable for the conditional with Y y- Using this, decompose
equation 35 into two terms:
Ex,r YII 1 n
W -7.1
gill
(36)
[Tg
Σ
Ex, [[Wsso(X,) - w/1l
Σ
Till
n
n
DEY
Σ
- wlll (P(Y-
|zyl
n
NEy
24

---

where
= [n] y y
The first term in the right-hand side of equation 36 further simplified by using
Till - Σ n VEV VEI
Wsfo(a.)
[Wsfo(E;) yll.
as
Ex, (l -lll [ul 1 [Wso((. Dell
Σ
n n
DEY
Σ
[yl [x,1IWsso(XX)) vlll
Σ
(Isso(T;) y/l
I iEZ,
yE>
where D = {y E/ v [zyl # 0}. Substituting these into equation equation 36 yields
[xx YII
Σ (WsSo(E)) - Till
(37)
Zyl
Ex,,IIWsSo(x,) - y/ll
(Wso(T:)
yyl
n
[Tg
VEy
IEI
Σ yEY
Ex, (s.o((x)) - vlll (P(Y =y)
[Tyl
Importantly, while ( ".l......IIsfff(( Mr on the right-hand side equation 37 vll are independent random
dependent random variables, (o U11 ........ variables since Ws and e; are independent and is fixed here. Thus, by using Hoeffding ' inequality (Lemma G.I), and taking union bounds over y E V. we have that with probability at least the following holds for all y E V::
Ex, [I w/ll
1
Σ
[[s ml1 S EKs
In(()1/8)
Iyl
2|Inr1
iET
This implies that with probability at least 1 - 8,
Σ Iyy
Ex, [o wlll
Σ [Tyl iET
[Wsfo(E.) =11)
BE
< K5S Σ
In( (5166)
2|In1
vε
In(())/6)
Σ
=Ks
2m
VEY
Substituting this bound into equation 37, we have that with probability at least 1 8,
Ex, - YI]
Σ
[WsJo(E.) 7ill
(38)
In [n(())/0)
Ex, [I] =vll (P(Y =y) Inn
S Ks Σ Vplyi V
+
2n
WEY
vEy
25

---

where
p(y)
n
Moreover, for the second term the right-hand side of equation 38, by using Lemma of [37], have that with probability at least 1 - 6,
Ex, (WWss(XX))) ylll (PCY y) [
Σ BEy
Ex,))xx,lllssff - w/l z/l)
22 )
Σ
$
2n
NEY
S Ks
Σ NEY
vly)
2
21n(111/0) 2n
where p(u) = P(Y y). Substituting this bound into equation 38 with the union bound, we have that with probability at least 8,
72
Ex,y (lF[(l/ss(()) III
Wo - Dell
(39)
72
In 1n(2)51/0)
In (2
S Ks
Σ Voli
+ Kis Σ
ply)
2n
2n
NEY
DEY
(2 2211888
In (2)))/8)
x Σ
p(y)
KS
Σ
ply)
Kg
2n
2n
VEY
VEJy
2 ln 2n((2
S
KS
p(y) + vplw)
2n
DEY
Combining equation 34 and equation 39 implies that with probability least 1 5,
E Ex.r YII
(40)
2 In (2D//8
[WsSo(F.) - Dell + KS\
VV(y) + vP(w)
2n
DEY
1 Σ ([(((()) fo(x)))l + < (ws)
n
n ||+(=:)) + 1
(1n(2()11/8)
|((x)))|
(Vbls) VVle)).
n
2n
DEY
We will now analyze the term
E || + + [T:(())) on the right-hand side of equa-
tion 40. Since W
Ws. g.
||(())) 1 110 (z.) Wgfo(x:))l
By using Hoeffding's inequality (Lemma G.1), we have that for any > O, with probability least
1 8,
[(((x)) < 1 Σ 119 Wxfo(a))) S Ex+ (1lg((x)) - Wafo(xt)) [11 + Kg
In
2n
26

---

Moreover, by using [50, Theorem 3.1] with the loss function x+ + (1g) - W Wf((t))l (i... Lemma G.2), we have that for any 8 > with probability at least 1-
2R (W (Wo
In In(1/0)
Ex [Ilg Wsff( IIl
(x (sff(x)))1
xiV
m
vm
2m
(41)
Ei= & (1( is the normalized Rademacher complexity of the set fa+ + (x+) Wf(x+) II W W. E FF (it is normalized such that (F) O(1) m for typical choices of F), and Çı..... E. independent uniform random variables taking values in 11, Takinng union bounds, we have that for any 5 > 0, with probability at least -
where (W o F)
m
In(2/5)
In(2/5)
m Σ (=g) W5fo(
+ #RV
n
mm
2m
2n
Similarly, for any > O, with probability at least 1 5,
2Rm (W F)
In(2/6)
In(2/5)
||( 11g" (x Wsfo(x;))l
+KV
+
+KsV
Vm
2m
2n
Thus, by taking union bounds, have that for any 5 0, with probability at least 6,
a
[x(x))| Σ (xx()
(42)
2
4Rm (WoF) + 2ng
In(4/5) 2m
In(4/5)
(x Wsff(xt) +
+ 2x5
vm
2n
To analyze the first term on the right-hand side of equation 42, recall that
1 | [
3 minimize [ F s.t. W
arg min W
(43)
m
Here, since W fo(x)) RT, we have that
W Woo(=)) (o(lW [T @ [.] vec[W] E ER",
where I, e grxr is the identity matrix, and [fo(=))] & LJ xr is the Kronecker product of the two matrices, and vec[W] E dR the vectorization of the matrix W E grx Thus, by defining A, [ @ I E Brrdr and using the notation of w = vec[W] and its inverse W vec-i(w) [w] (i.e., the inverse of the vectorization from R to Rdd with fixed ordering). we can rewrite equation 43 by
Wg = vec vec [wgl where w3 minimize | s.t. w' E arg min Σ llg: Awll2
2
with = gxx") E RR. Since the function w + >> i- 11g: A 2 is convex, a necessary and sufficient condition of the minimizer of this function obtained by
m
ma
0
19
A
AT (o AQw) E Rd
This implies that
m A A - Σ
m
AT gi.
i=1
i=1
27

---

In other words,
A1
91
A2
92
A Aw A'g
where
E R
mr de
and =
E mr
:
Am_
gm.
Thus,
w' E arg min
99 - Awwl2 = ((AA((AT))) A +0:00 Null(A))
where (AT A)) is the Moore-Penrose inverse of the matrix A T A and Null(A) is the null space of the matrix A. Thus, the minimum norm solution obtained by
vec[Ws] wg (AT A) A
Thus, by using this Ws, we have that
Σ (x]
m
Wafe(a 1
Σ
m1
(g: A2
m
m
((g - )
m
(o-- )
mm
Vm ||g - Awslle
1 I |1(1 A(A A(ATA)tA') A)tAT)gll
lgg A)'AA gll2 = Vm
m
where the inequality follows from the Jensen's inequality and the concavity the square root function. Thus, have that
1x((;))
||,)).))|
(44)
vm |[[I A(A A )glla + 4R,.. (W
In(4/5)
In(4/5)
+ 2
2x55 V
Vm
2m
2n
By combining equation 40 and equation 44 with union bound, we have that
E W IIll
(45)
Ls(ws)= + Σ [[o()))) fo(x:))
Paglle
mm
4Rm (W
In(8/8)
In(8/8)
+ 2
+ 2K5 V
Vm
2m
2n
2ln(4))1/8)
(Vp(y) vp(s)).
+ KS\
2n
DEy
where W Ws W* and PA =
A(A A(ATA)) AT
We will now analyze the second term on the right-hand side of equation 45:
1
1 Σ (fo(z.) - so(z:)l
((fo(z)- - fo(x:))ll < [|]]
(46)
28

---

where || is the spectral norm of W. Since E; shares the same label with 2; as E; Dy. (and T; ~ D. and because fe is trained with the unlabeled data S, using Hoeffding's inequality (Lemma G.1) implies that with probability least 1-d d,
In(1/6)
Σ IIfo(T.) So (2x,)) S By lll - fo(x)]]] + T9
(47)
2n
Moreover, by using [50, Theorem 3.1] with the loss function (2. [1fo(z) fo(x)) (i.e., Lemma G.2), we have that with probability at least 1 - 6,
m
2Rm (F)
In(1/6)
Bx Dg [llfo())) fo(x)]l] - m Σ
1 - fo(a (0(x)))]
+ VV
m
2m
(48)
where (F) Vim -. /sup ser & f(l IIl is the normalized Rademacher complexity of the set {(at xx+) + |1s(x)) [ [(++)|| f f F} (it is normalized such that (F) = (1) as m + oo for typical choices of F), and fi...., fm are independent uniform random variables taking values in (-1,1). Thus, taking union bound, we have that for any 5 > with probability at least -5, 5,
((o(F))) fa(x))))
(49)
m m
2R, (F)
ln(2/0)
In(2/0)
[
fo(a II|
+ +TS/
Vm
2m
By combining equation 45 and equation 49 using the union bound, we have that with probability at least 1 1 5,
Ex, [ - -Y]]
(50)
2Rm (F)
In(4/0)
In(4/5)
$ [ss(ws)= + +11i112
[lo(x))) So(x; )||
+ +
TST
Vm
2m
2n
2
4mm FW
In(16/0)
In(16/8)
+ mm
| +
+ +2K1
+ 2x3 V
m
2m
2n
21n(8)2118)
p(v) + VPlvi)
+ +
2n
yEY'
m
2
Ls(ws)) + ||1||2
[ so( |||
P
Pagll2 + Qm...
m
mm
where
2Rm
In(3/8)
ln(3/5)
Qm, = 11
+TV
+ TB
Vm
2m
2n
2In(6[D11/8)
Σ
(Vb()) VP(u))
+ KS
2n
MEV
4Rm (Wo F)
In(4/8)
In(4/8)
2x
+ 2xg V
m
2m
2n
Define Zg =f((x)),...... ... f
Rdxm Then, we have A 17g Tr]. Thus,
PA = 1 - 1Zg T @ 13114
2 1. @ 4. = 1 [g (Zg T) @ 1,] = P22 @ 11
where Pz₃ Im Z3] ZZ
T)tZ3 R By defining Ys lg* ), ....... . (xtt)
gmmrr since g veclY
P P |1[Pz, 23 @ I ve 1l vecly P 2 llle = |Pzz Ysllf
(51)
29

---

On the other hand, recall that Ws is the minimum norm solution
1 Σ
Ws minimize W
[ W
arg min
[S -vill?
W
W 7
By solving this, have
Ws = Y Zg T (ZgZs 1)
where Zs 1f(2)), (xx)) E Ædxx and Ys 1/,,..., Vn! T
E Ruxr Then,
Ls(ws) = m Σ
[Wsfo(x;) will
(WsSo(ri)) - vc)a)²
71
- -
k=1
[Ws - -Y
Vn ||YT (Zs T (Z5Zs ()t Zs T)]lr
|1(T Zs (2s 2s
Vn
Thus,
1
Ls(ws)
P2g
(52)
Vni
where Pz. -1 (ZsZs T)
By combining equation 50-equation 52 and using 1 S V2, we have that with probability at least
1-6, 6,
Exy (Xxl()Ws)(X)) YII & cls(So) 2 Vm
1
PP 12g YsllF
P2s
Vslle +
Qm...
(53)
Vn
where
2R (F)
In(3/8)
13/3/0)
Qm,n
+ T3
Vm
2m
2n
2 21n(6()118))
+ Kg\
v p(()
v v/(())
2n
4Rm 4Rn [ (Wo
In(4/8) 2m
In(4/8)
2x
+ 2xg V
Vm
2n
Now,
Appendix J
Information Optimization and the VICReg Objective
The eigenvalues of E(r,) are in some range S A(E(((,))))))))>)>>>>>>>>>>>)))) S Assumption 2. The differences between the means of the Gaussians are bounded
Assumption
M = max (X,, (xX,))12
Lemma J.1. The maximum eigenvalue of each pl p(X))u(X,) is at most M.
30

---

Proof. The term ((Xx) (XX,) is outer product of the mean vector ((x)), which is symmetric matrix. The eigenvalues of symmetric matrix are equal the squares of the singular values of the original matrix. Since the singular values of vector are equal to its absolute values, the maximum
eigenvalue of (((X))n(( 1 (², is equal to the square of the maximum absolute value of ju( (())). By the second assumption, this is at most M.
L
Lemma J.2. The maximum eigenvalue of -uzuz is non-positive and its absolute value is at most M.
Proof. The term -nzu2 is negative outer product of the overall symmetric matrix. Its eigenvalues are non non-positive and equal to the negative squares of the singular values of pzz. Since the singular values of vector are equal its absolute values, the absolute value of the maximum eigenvalue of -nzuz is equal to the square of the maximum absolute value of μaz. which is also bounded by M by the second assumption.
vector μzz. which is
Lemma J.3. The sum of the eigenvalues of Σz is bounded
(1 s (b M) x K
Proof. Given a Gaussian mixture model where each component Zazz has mean μ(X,) and covariance matrix E(r,), the mixture can be written as:
Z = Σ Z
where pj are the mixing coefficients. The covariance matrix of the mixture, Σz, is then given by:
Ez Σ ((X))))+(XX))x(X)))))) - =zzzz
where μzz is the mean of the mixture distribution.
By Lemmas 1.1, 1.2, and assumptions and 2, the maximum eigenvalues of (E(zy), μ (Xx)) and zzzz- are at most b, M,, and M, respectively. Therefore, by Weyl's 'sy inequality for the sum of two symmetric matrices, the maximum eigenvalue Σz is at most + M.
K
Amaa Σ2) S
(max (X(((X)) + M) <b+ M
K
means that we can bound the sum of the eigenvalues of Σz with
Σ x(Ez) (b M) K
Lemma J.4. Let Ez be a positive semidefinite matrix of size x N. Consider the optimization problem given by:
maximize log det((iz)
such that:
(x Sc
1=1
Σz 0
where >x(E.z) denotes the i-th eigenvalue of E.z and c. a constant. The solution to this problem diagonal matrix with equal diagonal elements.
31

---

Proof. The determinant of a matrix is the product of its eigenvalues, so the objective function log det(Ziz) can be rewritten E (Ez() Our problem is then to maximize this sum under the constraints that the sum of the cigenvalues does not exceed c and that D is positive semi-definite.
Applying Jensen's ' inequality to the concave function log(z) with weights 1/N, we find that N EE (og(A((EE)) & log( & E Ax(z)) Equality holds if and only if all Ax(E2) are equal.
Setting Ax(E2)= = x for all i, we see that the constraint En A((Ez) S e: becomes No S c, leading to the optimal cigenvalue
c/N under the constraint.
Since 2 is positive semi-definite, it can be diagonalized via an orthogonal transformation without changing the sum of its eigenvalues or its determinant. Therefore, the solution to the problem is diagonal matrix with all diagonal entries equal to c/N.
This completes the proof.
Theorem J.S. Given a Gaussian mixture model where each component X has covariance matrix [X,)), under the assumptions above, the solution to the optimization problem
Ez
maximize Σ log
[X(2,
is a diagonal matrix Σz with equal diagonal elements.
Proof. The objective function can decomposed as follows:
Σzl
Σ log
Σ (log Ez1 log (x()))))
[(XI)
K log EΣz|
Σ log [ X
where K is the number of components in the Gaussian mixture model.
In this optimization problem, we are optimizing over Σz. The term Σ, log [E(X,) constant with respect to Σz, therefore we can focus on maximizing K log [Ez|-
As the determinant of matrix is the product of its eigenvalues, log Ez| is the of the logs of the eigenvalues of Σz. Thus, maximizing log Σx| corresponds to maximizing the sum of the logarithms of the eigenvalues of Σz.
According to Lemma 1.4, when we have a constraint on the sum of the eigenvalues, the solution to the problem of maximizing the of the logarithms of the eigenvalues of a positive semidefinite matrix z is a diagonal matrix with equal diagonal elements.
From Lemma 1.3, we know that the sum of the eigenvalues of Σz is bounded by (b + M) x K. Therefore, when we maximize K log [Ezl under these constraints, the solution will be diagonal matrix with equal diagonal elements. This completes the proof of the theorem.
Appendix K Entropy Comparison -Experimentl Details
We use ResNet-18 [32] as our backbone. Each model is trained with 512 batch size for 800 epochs. We use the SGD optimizer with a momentum of 0.9 and a weight decay of 1e-4 The initial learning rate is 0.5. This learning rate follows the cosine decay with a linear warmup schedule. For augmentation,
augmented versions of each input image generated. During this process, each image is cropped with random size, and resized to the original resolution, followed by random applications of horizontal mirroring, color jittering, grayscale conversion, Gaussian blurring and solarization. For the entropy estimation, we use the same method as [38], which uses a lower bound of the entropy using the distances of the representations under the assumption of mixture of the Gaussians around the representations with constant variance.
H(Z) H(ZCC)
Σ
c In Σ Ge D(pilles)
(54)
32

---

. 5SL Bound
Sugeewisdd Boand
ReaNet56
Generaliza Bound
VICReg Bound
0.8
Palynomai
ReaNetL10
0.6
0.4
1.3
0.2
1.00
0.2
0.4
0.6
0.8
1.0
1.2
Number Samples (m)
Generalization Gap Loss
Figure 5: Our generalization bound predicts more accurately the generalization gap in the loss. (left) Our SSL VICReg generalization bound outperforms state-of-the-an -fr supervised generalization bounds. (right) Strong correlation between the generalization gap and our generalization bound for VICReg. Pearson correlation 0.9633. Conducted CIFAR-10.
where Pt and pj are the distributions the representation of the i-th and j-th examples, and (pp IP;) represents the divergence between these distributions. Also, c indicates the weight of component (c; 2 0. Σ, c. 1), and C is a discrete random variable where P(C i) =
Appendix L
Reproducibility Statement
All of the methods in our study are based existing methods and their open-source implementations. provide a detailed implementation setup for both the pre-training and downstream experiments. Additionally, we are committed to ensuring reproducibility and open science. Therefore, after publication, we will provide pretrained checkpoints and make the code openly available on a public repository. This will enable other researchers reproduce results and build upon our work.
Appendix M Expriemtns on the generalization bound
In this section we have more empirical results on the connection between our generalization to the generalization gap.
Appendix N Limitations of assuming inherent network randomness
First, we provide a brief overview of the challenges associated with estimating mutual information (MI) in DNNs. This context will set the stage a detailed discussion on the advantages of our method
Estimating MI in DNNs poses various practical challenges. For example, deterministic DNNs with strict monotonic nonlinearities yield infinite or constant MI, complicating optimization [4] or making optimization ill-posed. One solution is to assume unknown distribution over the representation and estimating MI directly [9, 57]. However, this is challenging theoretically and requires large sample sizes [55, 47] and is prone sigmificant estimation errors [70].
To address these issues, prior works have proposed various assumptions about randomness in neural networks. For example, [1] posits noise in DNNs parameters, while [1] assumes noise in the representations. These assumptions are valid they outperform deterministic DNNs or yield similar representations to those trained in practice.
Our empirical evidence suggests that noise in the input has much better performance and represen- tations similar to the deterministic network compared to noise in the network itself. We compared our model, which introduces noise at the input level, with models that add noise network repre- sentations [26], using CIFAR-100 and ImageNe and VICReg with ConyNeXt. The results,
33

---

tabulated below, indicate that the noise injected into the representation has much worse performance degradation compared to our input noise model.
Table 2: Comparison of Noisy Network and Noisy Input across different datasets for different noise levels
3 (Noise Level)
Tiny- Imageet
CIFAR 100
Noisy Network
Noisy Input (ours)
Noisy Network
Noisy Input (ours)
(no noise)
53.1
53.1
70.1
70.1
B = 0.05
51.7
53.0
69.7
70.0
8 0.1
50.2
52.8
68.8
69.6
3 = 0.2
48.1
52.3
67.1
68.9
Our approach, focusing on input noise, does necessitate explicit noise injection during training- unlike methods assuming inherent network noise.
Lastly, we validate that the assumption of input noise leads to better alignment with deterministic networks. Following [14], we used Centered Kernel Alignment (CKA) [18] to compare deterministic DNNs with our noise model (input noise) to DNNs with noise in the representations [26]. The results, also tabulated below, confirm that assuming input noise aligns more closely with deterministic DNN representations than assuming noise in the DNN parameters.
Table Performance comparison of different methods at varying noise levels
Noise Level/Method Deterministic Network
Noisy Network
Noisy Input (our method)
B 0.05
0.97
0.82
0.93
B 0.1
0.97
0,69
0.85
0.2
0.97
0.54
0.77
3 = 0.3
0.97
0.32
0.69
In summary, our input noise model more effectively captures deterministic DNN behavior and is more robust than assuming inherent network randomness.
Appendix O Boader Impact
In the landscape machine learning, SSL algorithms have found a broad array of practical applica- tions, ranging from image recognition to natural language processing. With the proliferation of data in recent years, their utility has grown exponentially, often serving as key components numerous real-world cases. This paper introduces superior performance. Given the widespread use of SSL. algorithms, the advancements proposed in this paper have the potential to considerably enhance the effectiveness of systems that rely these techniques. As result, the impact of these improved algorithms could be far-reaching, potentially influencing multitude of applications and sectors where machine learning is currently employed. With this potential for value also comes the potential that proposed methods make false promises that will not benefit real-world practitioners and may, in fact, cause harm when deployed in sensitive applications. For this reason, we release our numerical results and implementation details for the sake of transparency and reproducibility. As with all new state-of- of-the-art methods, our improvements may also improve models used for malicious intentions.
family of SSL algorithms that have demonstrated
34


Reviews:
[{"review_id": "1rlNoSfbBM", "reviewer": "Reviewer_tvXs", "summary": "This paper explained the effect of VICReg learning method using information theoretic perspective under some assumptions on the distribution of the input data. Based on the assumption, it connected the VICReg to mutual information minimization, and proposed new self-supervised method that performed better than existing methods on some simple tasks and networks.\n\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nThank you for the response. I will keep the score.", "questions": "1. In Theorem 1, what is the exact meaning of \"approximated\"? Could you explain which parts are ignored to derive this approximation?\n2. Could explain more on how Figure 1 verified the assumption on input data? What is the meaning of \"sampled 512 Gaussian samples for each image\"? What is \"Gaussian samples\"?", "limitations": "This paper did not fully discuss the limitations of the proposed theory and methods. At least, two limitations should be addressed: 1) it should tell when the assumption on randomness of input data will not be satisfied; and 2) the theory is restricted to ReLU network.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposed to transition the randomness from the stochastic network to the input data in order to applying information theory, which is a novel technique.\n2. It established the explanation for VICReg objective and proposed new self-supervised methods based on that.", "weaknesses": "1. The presentation of this paper is not good, e.g., the logic is not clear by following the title of each section and sub-section.\n2. The theory is based on assumption on input data and ReLU-type network which limits its application.\n3. The experiments are toy as self-supervised methods are more often to applied in large-scale pre-training tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer tvXs,\n\nWe appreciate your constructive comments and suggestions. We believe your feedback will significantly help us improve our paper's quality and readability. Here, we address each of your concerns.\n\n## Weaknesses:\n\n1. **Presentation of the Paper** - We value your feedback on improving the paper's presentation. Specific suggestions to enhance clarity would be greatly appreciated. We're committed to ensuring the flow of ideas is coherent and easily understood in our revisions.\n\n2. **On the Assumptions We Made:**\n   - (i) **(Leaky) ReLU-type Network** - Our theory accommodates the most common nonlinearities, such as ReLU, leaky ReLU, max pooling, and absolute values. Our results extend to other smooth nonlinearities through a first-order Taylor approximation argument. We believe these cover the majority of modern deep networks. If you feel our approach is limited, please specify any other common network architectures we should address.\n   - (ii) **The Input Data** - We have conducted rigorous experiments to validate our assumptions about the input data across various datasets. To further substantiate our findings, we replicated Figure 1 on ImageNet samples, affirming that the samples do not overlap and the network output for SSL training is more Gaussian for small input noise (See Figure 1 in the attached page). Unlike more common assumptions that include stochastic networks, ours focuses on the generative process of the data, making it less restrictive. If you still have reservations, please point out any additional assumptions you think we need to address.\n3. **Large-scale Tasks** - Our primary goal was to provide a theoretical understanding of the VICREG objective, where we believe there is distinct value in understanding the theoretical advantages and optimization process of SSL in general and VICReg in particular. However, we agree that the analysis would be enriched by considering more datasets and SSL methods. Prompted by your feedback, we have now added additional experiments on larger datasets:\n    - (i) **Validation of Our Assumptions** - We validate our assumptions on the ImageNet dataset to ensure the robustness of our findings (see Figure 1 in the attached page).\n    - (ii) **Large Datasets and Architectures** - We train our proposed entropy estimators on both the tiny-ImageNet [1] and CIFAR-100 datasets with ConvNetX[2] and VIT [3] as the backbones.\n    - (iii) **Inclusion of Various SSL Methods** - We extend our experiments to include multiple SSL methods, such as MoCo[4] and SwAV [5]. This allows for a more comprehensive comparison and demonstrates the adaptability of our approach.\n\nOur newly conducted experiments, as detailed on the attached page, illustrate the superiority of our plug-in estimators across various methods and datasets, reinforcing our theoretical findings and underlining the effectiveness of our proposed entropy estimators.\n\n\n\n## Questions:\n\n### The Approximations in Theorem 1\n There are two approximations done in Theorem 1 when considering its validity for real-world data. First, we assume the data distribution to be of the form given by Equation (2), i.e., a mixture of Gaussians. That alone can be made an arbitrarily good approximation by increasing the number of components in the mixture. Second, and more importantly, we constrain that density so that each of the Gaussian components (i) does not overlap with any other component and that (ii) does not overlap with the boundaries of the partition of the DNN. Recall that the DNN is an affine spline and thus has a partition of the input space and per-region affine mappings on each region. In practice, however, the overlap has to be understood as having a negligible probability, albeit not exactly 0. We will clarify those limitations in the revised manuscript and around Theorem 1.\n\n###  Validation of Our Assumptions\nTo **validate the non-overlapping support assumption**, we measured the L2 distance between the images. If the images had overlapping effective support, their distances would be expected to be small. However, Figure 1 (right) shows that the distances between different images are significant and increase as the dataset size grows, reinforcing the non-overlapping support.\n    To validate our assumption about the **Gaussian nature at the output layer of the DNN**, we checked the final layer representations of 512 Gaussian samples with the mean around each image at varying noise levels. Figure 1 (left) shows that, at lower noise, the output stays Gaussian. This supports our assumption of a noise region where the input and conditional output remain Gaussian around the mean.\n\n## Limitations \nWe would like to thank you for discussing these limitations. See our above answers. \n\n\nThank you again for your thoughtful review. We made an effort to address your feedback, including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.\n\n## References \n[1] Le, Y. et al., \"Tiny ImageNet Visual Recognition Challenge\", 2015.\n\n[2] Dosovitskiy, A. et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\", 2020.\n\n[3] Liu, Z. et al, \"A ConvNet for the 2020s\", 2022.\n\n[4] He, K. et al., \"Momentum Contrast for Unsupervised Visual Representation Learning\", 2020.\n\n[5] Caron, M. et al., \"Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\", 2022."}, {"review_id": "ESkij3otd4", "reviewer": "Reviewer_VQBf", "summary": "This paper presents two theoretical results on the VICReg self-supervised learning technique. Taking an information theoretic perspective, it shows that the VICReg objective can be seen as maximizing a lower bound on the mutual information between Z and X' where X' is an augmented version of data point X and Z is the encoded X. Second result presents a generalization bound relating the VICReg objective to performance on a downstream task (where the augmented versions of a data point share the same label).\n\nIn order to show the first result, the authors make several assumptions. First to be able to treat a deterministic network probabilistically, they assume the data distribution is a mixture of Gaussians with non-overlapping Gaussians centered at each data point (they confirm that this is a valid assumption for the datasets they look at). Building on earlier work formalizing neural networks as affine spline operator, this allows them to show that the output distribution of the network q(z|x) is Gaussian. Using this fact, they derive a lower bound on mutual information I(Z;X') which closely resembles the VICReg objective. This lower bound contains an entropy term H(Z) and an invariance term (the same with VICReg objective). Then they show that the variance-covariance regularization term in VICReg can be seen as an approximation of H(Z). This motivates replacing the regularization term in VICReg with alternative entropy estimators. They present results with two estimators: logdet and pairwise distance based. They find that using these alternative estimators lead to slight improvement in downstream task performance.\n\nTheir second results shows a link between the VICReg objective (hence mutual information) and downstream task performance. Here they assume the data augmentations do not change the label of a data point (wrt to the downstream task) and show that downstream task performance can be bounded from above with an expression that aligns closely with VICReg objective.", "questions": "- Why are there no results for Barlow Twins in Figure 2?\n- Why are there no results for BYOL in Table 1?\n- VICReg + LogDet does better but in the text the authors say VICReg + PairDist does better. Why?\n- Line 243: The results seem to show that entropy *increases* during training not decrease.\n- 118: should be q(z|x')?\n- Eqn 8. is not equal to L, right? It should be H(Z)?\n- Line 265: Looking at Table 1, LogDet achieves best results not PairDist.\n- What are the logdet and pairdist estimators? Can you write the equations for these?\n- Line 340, doesn't the term in Theorem 3 correspond to -I(Z;X')?", "limitations": "No concerns about potential negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall a pretty good paper. Understanding self-supervised learning techniques better (and improving them) should be of interest to many researchers.\n\nI don't know the literature they build on really well so it is hard for me to judge originality but the results they present seem important.\nThis was perhaps intuitively obvious but showing that the regularization term in VICReg approximates entropy is a nice result, since it makes it more clear how better entropy estimators should (and does) help as they show in their empirical evaluation.\nThe second result on downstream task performance is again interesting but how this result is useful was not really clear. \nIt was also nice to see that the authors checked if the assumptions they make on non-overlapping support and Gaussian output distribution held in practice.", "weaknesses": "Overall I think the paper would benefit from more empirical evaluations and more clarity in certain places in the text.\n\nThe empirical results in Table 1 are not very strong. Why does using better entropy estimators lead to such small improvements? Is this because of the dataset or perhaps the size of the network? It'd be nice to see if these alternative entropy estimators lead to more significant improvements. This would confirm the value of the theoretical analysis much more strongly.\n\nThis lack of empirical evaluation is even more of a problem for the second generalization bound result. As it stands, it is not very clear why a practitioner should care about this result. Can the authors provide some empirical/practical implications of this result and perhaps explore some of these? Currently, it almost feels like this result should go into a separate paper, where it is developed more and investigated more deeply (perhaps following the idea mentioned in Line 353).\n\nOn clarity, there are a couple of sections where it is not quite clear how and why it fits with the rest of the text. For example, what is the purpose of section Entropy Estimation (line 237)? As it stands it doesn't really add much. I'd suggest either adding more detail about how SimCLR's objective and entropy estimator is different and what that means or perhaps remove this section. Currently, I'm not sure what I should take away from this section.\n\nAnother example is line 327, where authors say Theorem 3 illuminates the benefit of SSL. What does this mean more precisely? How does this illuminate the benefit of SSL? I understand that in Theorem 3 you get a scaling with m (number of unlabeled data points) instead of n (number of labeled points), and m > n usually in practice, But this doesn't mean the generalization performance of SSL will be better than supervised learning with n labeled points. To be able to make that claim, you would need to compare the bound in Theorem 3 with a similar bound for supervised learning, right? Can the authors show how their results explains the benefit of SSL empirically or with some simulations?\n\nSimilarly for section 6.3, it is not clear what is the point of this section. What are the implications of this observations? And why should we be minimizing I(Z;X) (while maximizing I(Z;X'))? This should either be expanded and explained in more detail or removed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed review and the constructive feedback you've provided. Your insights are invaluable and will significantly contribute to the refinement of our paper. Allow us to address each of your concerns in detail:\n\n## Weaknesses:\n\n### Empirical Evaluations\n\nOur primary goal was to offer a theoretical understanding of the VICReg objective. We believe in comprehending the theoretical advantages and optimization process of SSL in general and VICReg in particular. However, we concur with your suggestion that a broader empirical analysis would enrich our work. Responding to your feedback, we've expanded our experiments to include larger datasets such as Tiny-ImageNet[1] and CIFAR-100. We utilized models such as ViT[2] and ConvNetX[3] as backbones and compared them to more SSL methods like MoCo[4] and SwAV[5]. Our results demonstrated substantial improvements of our estimators over the different methods (See Table 1 in the attached page). These findings affirm our theoretical analysis and showcase the scalability of our method.\n\n### Generalization Bound\n\nThe generalization bound aims to explain why VICReg works, its theoretical benefits over supervised learning, and other SSL methods. Generalization bounds are instrumental in studying learning algorithms, as they provide insights into how well a model will perform on unseen data. The paper shows that our bound extends previous works and offers many advantages over previous SSL bounds. However, in response to your comments, we empirically analyzed our bound by calculating all the relevant terms, including the Rademacher complexity and covariance matrices [6], for different pre-trained SSL models. Our results (Figure 2) demonstrate a positive correlation between the generalization gap and our bound. Our findings reveal that our bound captures the \"true\" generalization performance, even though it is calculated using only the training samples. This insight underscores our ability to predict the model's performance on unseen data, relying solely on the training set.\n\n## On Clarity:\n\n### Entropy Estimation Section \nOur intention in Section 5.1 was to compare VICReg and SimCLR based on our analysis. We highlighted that considering these methods from an information theory perspective reveals differences in estimating entropy. This perspective allows us to analyze the models' differences, improving our understanding. \n\n### Theorem 3 \n\nSupervised generalization bounds scale with $1/\\sqrt{n}$, considering only supervised examples. In contrast, our bound incorporates unsupervised examples, lowering the generalization bound as their number increases. We agree with your point about comparing our bound with supervised learning. To do so, we trained our models on CIFAR10 with VICReg and calculated the supervised generalization bound from [2]. Our bound improves as the number of unlabeled training examples increases and outperforms the supervised one only with 400 unlabeled examples (Figure 2).\n\n### Section 6.3\n\nThis section discusses how our bound relates to the compression of MI, which has recently demonstrated superior performance in SSL. For the first time, our bound provides a theoretical justification for why representation compression benefits SSL. We will revise this section to articulate our argument better.\n\n## Questions:\n\n- **Figure 2 and Table 1**: We omitted Barlow Twins from Figure 2 due to its similarity to VICReg. However, we'll include it in the revised appendix. Additionally, we will include BYOL's accuracy in Table 1, which is performed worse than SimCLR.\n\n- **Discrepancy between the table and the text** - We apologize for the typo about VICReg + LogDet.\n- **Lines 243 and 118:** We apologize for the typos. We will fix it in the revised version.\n- **Equation 8:** Given that we used $log(|\\Sigma_Z| )$ to estimate the entropy of Z, we removed the invariance term in the transition from Eqn 7 to 8 because it appears in the same form in the original VICReg objective. Eqn 8 is not the full objective function to optimize. In the rest of the section, we focused on dealing with the differences between the two objectives. We will clarify this in the revised version.\n- **LogDet and PairDist estimators:** We apologize for the oversight in not providing their equations. Due to a lack of space in this response, we will include them only in the revised version. You can find the PairDist in [7] and the LogDet in [8].\n- **The term in Line 340  in Theorem 3:** No, the Rademacher complexity term represents the complexity of our model. This term includes all the information in the representation—both the relevant and irrelevant—and can be bounded by the information  $I(X;Z)$, which encompasses all the information from one of the views about its corresponding representation. The term $I(Z;X\\prime)$ contains only the relevant information, which we don't want to compress and is necessary for our downstream task prediction.\n\nThank you again for your thoughtful review. We made an effort to address your feedback, including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.\n\n## References \n\n[1] Le, Y. et al., \"Tiny ImageNet Visual Recognition Challenge\", 2015\n\n[2] Dosovitskiy, A. et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\", 2020\n\n[3] Liu, Z. et al, \"A ConvNet for the 2020s\", 2022\n\n[4] He, K. et al., \"Momentum Contrast for Unsupervised Visual Representation Learning\", 2020\n\n[5] Caron, M. et al., \"Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\", 2022\n\n[6] Kawaguchi, K. et al., \"How Does Information Bottleneck Help Deep Learning?\" , 2023\n\n[7] Kolchinsk A. et al.,  \"Estimating Mixture Entropy with Pairwise Distances\", 2017\n\n[8] Zhouyin Z. et al., \"Understanding Neural Networks with Logarithm Determinant Entropy Estimator\" , 2021\n"}, {"review_id": "O0DkSyt50M", "reviewer": "Reviewer_sVT5", "summary": "This paper analyzes the SSL methods from an information-theoretic perspective. Unlike previous theoretical analyses, the paper focuses on deterministic networks more suitable for real-world scenarios. Assuming a data distribution as a mixture of Gaussians (MoG) and a deep neural network (DNN) as a conditional probability approximation (CPA), the study demonstrates the equivalence between the variational lower bound of mutual information (MI) and the learning objective of VICReg (with some approximations). The theoretical and experimental differences between VICReg and SimCLR are presented, highlighting the superiority of VICReg. The study also shows that using a better entropy estimator can enhance the performance of VICReg. Generalization bounds are provided, and their practical implications are explained.", "questions": "The questions are included in the Weakness section.", "limitations": "There are no explicit discussions on limitations in the paper. It would be recommended to highlight any limitations arising from the assumptions or approximations made in this paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-organized and well-written.\n- To the best of my knowledge, the theoretical analysis in this paper is novel.\n- The authors provide empirical validation for the assumptions used in their paper and suggest possible remedies for some special cases. \n- This paper provides a clear intuition behind their contribution beyond the theoretical analysis.", "weaknesses": "- There is a lack of empirical evidence demonstrating the limitations of assuming randomness from DNNs.\n- In Figure 2, a comparison is made with BYOL, and in Table 1, a comparison is made with Barlow Twins. Why?\n- Additionally, SimCLR, BYOL, and Barlow Twins are relatively classic (and basic) methods. Why weren't they compared with more recent approaches?\n- ResNet-18 + CIFAR-10 is a very basic setup in the vision domain. Would it be possible to try ResNet-50 + Tiny ImageNet at least? How about exploring ViT as well?\n- (Minor) There are typos in the submitted version: L130 (. .), L156 (Missing .), L199 (appendix .1.), L227 (Missing ) ).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer sVT5,\n\nThank you for your comprehensive review and the constructive feedback you've provided. Your insights will significantly enhance our paper. We want to address your comments in detail:\n\n## Weaknesses:\n\n### Limitations of assuming randomness from DNNs\n\nNumerous methods propose using randomness in DNNs [1-3]. However, many conventional methods, especially SSL methods like VICReg, operate deterministically. This paper aims to shed light on the theoretical underpinnings of standard SSL methods. Transforming deterministic methods to random ones often demands intricate tuning and hyperparameter optimization, with no guarantee of the same performance.\n\n\n### Extending the Empirical Evaluations\n\nOur main objective was to delve into a theoretical understanding of the VICReg objective, emphasizing its significance in the broader context of SSL. However, responding to your feedback, we have enriched our empirical analysis by:\n\n- **Exploring Larger Datasets**: We executed our models on the \nTiny-ImageNet [4] and CIFAR-100 datasets.\n- **Integrating Recent Architectures**: We incorporated modern architectures, including the Vision Transformer (VIT) [5] and ConvNetX [6].\n- **Incorporating Additional SSL Methods**: Our analysis was expanded to encompass methods like MoCo[7] and SwAV[8].\n\nOur updated results (refer to Table 1 in the attached page) underline substantial enhancements of our plug-in estimators across diverse methods, reaffirming our theoretical assertions and underscoring the adaptability of our approach.\n\n### Figure 2 and Table 1\n\nIn Figure 2, we chose not to feature Barlow Twins due to its resemblance to VICReg in entropy estimation. Nonetheless, it will be included in the appendix of our revised version for completeness. Additionally, in Table 1, we will display the accuracy results of BYOL, which lagged behind SimCLR.\n\n### Typos\n\nWe acknowledge the oversight and will rectify the identified typographical errors in our subsequent version.\n\n## Limitations:\n\nWe concur that a candid discussion of the study's limitations is essential. While our current version addresses these in the appendix, we will transition this section to the main content in our revised manuscript in light of your feedback.\n\nThank you again for your thoughtful review. We made an effort to address your feedback, including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.\n\n\n\n\n## References\n[1] Alemi, A.A. et al., \"Deep Variational Information Bottleneck,\" 2016\n\n[2] Dubois Y. et al., \"Lossy compression for lossless prediction\", 2023.\n\n[3] Lee K et al., \"Compressive visual representations\", 2022\n\n[4] Le, Y. et al., \"Tiny ImageNet Visual Recognition Challenge\", 2015.\n\n[5] Dosovitskiy, A. et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\", 2020.\n\n[6] Liu, Z. et al, \"A ConvNet for the 2020s\", 2022.\n\n[7] He, K. et al., \"Momentum Contrast for Unsupervised Visual Representation Learning\", 2020.\n\n[8] Caron, M. et al., \"Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\", 2022.\n"}, {"review_id": "0a8eiYY6Uk", "reviewer": "Reviewer_2fL7", "summary": "This paper presents an information-theoretic analysis of the self supervised learning algorithm VICReg. The authors first introduce an approach to study neural networks using an information-theoretic perspective by shifting the stochasticity from the networks to the input data. Therefore they derive an information-theoretic objective (equation 7) that is linked (through a series of approximations) to the VICReg objective. The assumptions of the proposed theory are validated empirically using ResNet-18 across various datasets.\n\nThe paper also proposes a plug-in entropy estimator that can be merged with the objective term of other SSL methods for entropy optimization. The authors shows empirically that doing so with VICReg and training a ResNet-18 architecture on CIFAR-10 dataset leads to an improvement on the test accuracy compared to the original VICReg method. \n\nLastly the paper derives a generalization bound for VICReg that provides a probabilistic guarantee on the downstream generalization ability of this SSL method. ", "questions": "As mentioned above the biggest weaknesses are related to some theoretical parts that are not clear, and the experimental evaluation which is not sufficiently convincing. Here are some opportunities to make the paper stronger (and potentially the rating higher).\n\n**Clarifications**\nProviding more information here could help improve the Soundness and Presentation rating.\n\n- In Equation 4, could the authors clarify how the inequality holds? Even in Appendix A this first step is taken for granted but it might be helpful to explain it, at least in the Appendix.\n- In Section 4.1 it is not clear to me what happened to the invariance term when going from equation 7 to equation 8. That term seems to have been dropped. Could the authors clarify this?\n- “We sampled 512 Gaussian samples for each image” I suppose this means that given 1 image noise is added by sampling 512 different noise that is added to the image? Could this be clarified?\n- “However, these methods (reviewer: these methods being [61, 65]) assume that randomness comes from the DNN, contrary to common practice.” Could the author clarify where in 61 and 65 this is assumed?\n- “the mutual information between the input and the DNN representation is infinite”. This statement is presented as always true but I do not understand how this would be always guaranteed. None of the papers cited in this section ([8,17,28,37,52,61,…]) seem to support this statement. Do the authors refer to the difficulties of computing MI in continuous domain or could the authors clarify this statement? \n- It seems that the loss function of VICReg reported in this paper has some inaccuracies. For example, there is a \\gamma parameter that appears twice (in the regularization and the invariance term) but odes not match the loss in [5]. Since in [5] \\gamma was fixed to 1 it might not make much difference but it would be better to report the correct equation. \n- In section 5.1 “The results showed that entropy decreased for all methods during training, with SimCLR having the lowest and VICReg the highest entropy.” From Figure 2 it seems the entropy is increasing not decreasing. Could the authors clarify?\n- “The entire input–output mapping then becomes a CPA spline “. I wonder if this statement is correct. According to [4] DNNs can be seen as a composition of max-affine spline operators, which I am not sure is the same as stated in the paper under review. Could the authors provide more information?\n- How many trial were used to compute the error bars in Table 1?\n- “the invariance term of current SSL methods with plug-in methods for optimizing the entropy.” Is it merged with the invariance or with the regularizer term?\n\n**Experimental Evaluation**\n- The downstream task results are only presented on CIFAR-10 and ResNet-18. Unfortunately, in my experience not everything that works on CIFAR-10 (and on a small architecture) transfers to more realistic datasets (ImageNet would be ideal, TinyImageNet or CIFAR-100 would at least be better than CIFAR-10). The paper would be stronger with these experiments. \n- Since the authors claim that the plug-in estimators could be beneficial for “other SSL methods“ it would be great to show how it affects other SSL methods beyond VICReg.\n- Other works, e.g., [A], have found that entropy maximization also leads to other beneficial aspects beyond downstream task performance. Specifically, it seems to lead to easier and more robust training with respect to smaller batch size or different EMA parameters (where EMA is used). A similar analysis would also improve and strengthen the practical implication of this work.\n- Section 5.1 seems only marginally related to the rest of the paper. However, it would be interesting (and would connect more tightly this section to the rest of the paper) to show in Figure 2 the evolution of the entropy of VICReg (and other algorithms, see points below) with the proposed entropy plug-in estimator to show that indeed it leads to higher entropy. \n\n[A] “The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning” Galvez et al. ICML2023  \n\n\n", "limitations": "Limitations are not clearly described in the main paper. They are present in the appendix but I believe they would be more useful in the main part of the manuscript. Additionally in the limitation section of the appendix the authors state “Furthermore, our study hinges on certain assumptions that, despite rigorous validation efforts, may not hold universally” which is fair but it would be even better and more precise to clarify which are such assumptions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The perspective offered by the new analysis is interesting and could have practical implications.\n", "weaknesses": "- I find some of the explanation (example Section 4) is not immediately clear. There are some statements that should be verified and could be clarified in the manuscript in order to make the paper more accessible (see questions below on clarifications).\n- I find the empirical analysis of the practical implications weak: experiments are limited to CIFAR 10 on ResNet18. I believe there could be an opportunity to strengthen the paper by providing a more thorough evaluation on larger architectures and more challenging datasets (see questions below on experimental evaluation for some suggestions).\n- The claim that the proposed plug-in entropy estimator could help other SSL methods (beyond VicReg) is not supported by empirical evaluation. \n- The paper contains some significant errors: punctuation missing, use of different styles to refer to the appendix, missing figure citations, a few grammar errors... See Minor Notes below for a non-exhaustive list.\n\n**Minor Notes**\n- Line 156: a full stop seems to be missing “of prototypes N As”.\n- Line 227: missing full stop at the end of the page.\n- Line 242: missing closed parenthesis.\n- Notation is not consistent, Appendix A, B, C... vs appendix 1,2,3....\n- Figure 2 is not referenced in the text.\n- Table 1 missing closed parenthesis on the last line.\n- Line 314 Grammar: “label classes to go infinity”.\n- Line 347: Grammar: “By transferring the required stochasticity required “ .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2fL7,\n\nWe appreciate the comprehensive review and your valuable insights. Your insights will significantly enhance our paper. We want to address your comments in detail:\n\n\n## Extending the Empirical Evaluations\n\nOur paper emphasizes VICReg's theoretical aspects and advantages over other methods. Nevertheless, we acknowledge that additional empirical studies could amplify our findings. Therefore, we have enriched our experiments by:\n\n- **Exploring Larger Datasets**: We executed our models on the \nTiny-ImageNet [1] and CIFAR-100 datasets.\n- **Integrating Recent Architectures**: We incorporated modern architectures, including the Vision Transformer (VIT) [2] and ConvNetX [3].\n- **Incorporating Additional SSL Methods**: Our analysis was expanded to encompass methods like MoCo[4] and SwAV[5].\n\nOur updated results (Table 1 in the attached page) underline substantial enhancements of our plug-in estimators across diverse methods, reaffirming our theoretical assertions and underscoring the adaptability of our approach.\n\nIn agreement with your suggestion, we have included **the evolution of entropy using our proposed estimator** in the attached page,  which shows increased entropy that enables better downstream task classification.\n\n\n## Minor Notes:\nWe will fix these suggestions in our revised version.\n\n## Clarifications\n1. **Equation 4:**  We agree that the inequality warrants an explicit explanation. The inequality arises from the decomposition of p(z|x'), where the expectation is over p(X'), to p(z|z')p(z'|x'), where we first sample from p(X') and then sample from p(Z'|x').\n2. **Invariance term in Equations 7 and 8:** This term is included in Equation 8 because it is also a component of the original VICReg objective. Equation 8 is not equal to Equation 7; it is only part of the objective we optimize. In Section 4.1, we aimed to analyze the remaining terms and compare them to the VICReg objective, given that the invariance term is the same in both objectives.\n3. **Gaussian samples for each image:** You are correct. \n4. **Randomness in DNNs:** We apologize for the typo. \"However, these methods...\" should precede the sentence \"Other research has presumed...\". The methods that assume randomness originates from the DNN are referenced as [17], [37], and [52], considering a deterministic input and a variational DNN.\n5. **Infinite Mutual Information (MI)**. This concept was explored in depth in [1]. Two main issues arise when discussing MI in DNNs:\n(i) Estimating MI directly can be challenging, even with an underlying distribution [2,3].\n(ii) MI can often be infinite or a piecewise constant function of the network's parameters. Theorem 1 in [1] showed that MI is infinite for continuous inputs with almost all non-trivial parameter choices, and it becomes a piecewise constant function for discrete inputs, complicating optimization.\n\n6. **VICReg loss function:** We apologize for the confusion. The \\gamma in the variance loss differs from the invariance term's coefficient.\n\n7. **Entropy in Figure 2:** We apologize for the typo. The entropy in Figure 2 is indeed increasing, not decreasing, during training, as our analysis suggested.\n\n8. **DNNs as CPA splines:**  Lets first consider a single layer mapping made of an affine transformation followed by a nonlinearity such as ReLU or Leaky-ReLU. That nonlinearity is a continuous piecewise affine mapping (a spline of order 1). Taking ReLU for illustration, the mapping is 0 from - infinity to 0, and then 1 from 0 to infinity, making the mapping continuous and affine on each of its two regions ((-infinity,0], (0, infinity)). Composing that affine spline (activation) with the preceding affine mapping preserves that property, i.e., the entire layer is an affine spline. Max-affine splines (as named in [8]) are simply a particular type of affine splines where the activation can be expressed using a pointwise maximum (recall that ReLU(x) = max(0,x)), that naming, however, does not impact the fact that a layer is in fact an affine spline. Moving to the entire input-output mapping of a DNN, i.e., a composition of layers, we can use the previous statement. We know that a DNN is a composition of affine splines. As such, the entire mapping is itself an affine spline. Detailed derivations are provided in [8].\n\n9. **Trials in Table 1:** Each experiment was conducted 3 times, and we reported the average and standard deviation. \n\n10. **Plug-in estimators and the invariance term:** We retained the invariance term and combined it with our plug-in entropy estimators. \n\n## Limitations:\n\nIn our revised version, we will include the limitations section in the main body. In terms of elaborating on the universality of the assumptions, while we have tried to validate our assumption that the dataset points do not overlap, it may not hold true for other datasets. Moreover, determining the appropriate amount of Gaussian noise in the dataset remains unclear.\n\nThank you again for your thoughtful review. We made an effort to address your feedback, including multiple new experiments and paper edits, and we would appreciate you raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n## References \n[1] Amjad, R.A. et al., \"Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle,\" 2019.\n\n[2] Poole, B. et al., \"On Variational Bounds of Mutual Information,\" 2019.\n\n[3] Alemi, A.A. et al., \"Deep Variational Information Bottleneck,\" 2016.\n\n[4] Le, Y. et al., \"Tiny ImageNet Visual Recognition Challenge,\" 2015.\n\n[5] Dosovitskiy, A. et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,\" 2020.\n\n[6] Liu, Z. et al, \"A ConvNet for the 2020s,\" 2022.\n\n[7] Caron, M. et al., \"Unsupervised Learning of Visual Features by Contrasting Cluster Assignments,\" 2022.\n\n[8] Balestriero et al.,  \"A spline theory of deep networks\", 2018."}]

Author rebuttal:
Dear Reviewers,

Thank you for your thoughtful and detailed reviews of our work. We appreciate your time and the constructive feedback you have provided. We have carefully considered your comments and concerns and would like to address them in a unified response:

While our empirical explorations have been amplified, the heart of our paper remains its theoretical insights into VICReg. Our theoretical findings provide a fresh lens through which we can understand VICReg and SSL, and our empirical endeavors serve to validate and extend these theoretical observations. Our manuscript primarily delves into the theoretical nuances of VICReg, elucidating its advantages from an information theory standpoint. While this remains the bedrock of our contribution, your comments highlighted the importance of further empirical evaluations to substantiate and complement our theoretical insights.

## Extending the Empirical Evaluations

Heeding your feedback, we've undertaken a series of rigorous empirical evaluations:

- **Larger Datasets**: We ventured into larger datasets like Tiny ImageNet [1] and CIFAR-100, ensuring broader applicability and robustness of our findings.
- **Integration with Recent Architectures as Backbones**: Modern architectures such as Vision Transformer (VIT) [2], ConvNetX [3], and ResNet-50 were incorporated, aligning our results with state-of-the-art techniques.
- **Broadened SSL Methods**: Our analysis was enriched by including diverse SSL methods like MoCo [4] and SwAV [5], offering a more holistic comparison and demonstrating our approach's versatility.

Our newly conducted experiments, as detailed on the attached page, underscore marked improvements of our plug-in estimators across various methods and datasets, reinforcing our theoretical findings. Moreover, we use these models and datasets to validate our assumptions further. 

### Empirical Validation of the Generalization Bound

One of the cornerstones of our research is the derived VICReg generalization bound. Generalization bounds offer crucial insights into how effectively learning algorithms will perform on unseen data. Our generalization bound not only extends previous works but also showcases distinct advantages over existing SSL bounds. In response to reviewers' suggestions, we have empirically validated our bound and compared it to supervised generalization bounds  [6]. Our empirical results (Figure 2 in the attached page) manifest a clear correlation between the generalization gap in the loss and our proposed bound and its superior over supervised bounds. These findings suggest that our bound aptly captures the generalization performance — a noteworthy observation given that the bound is derived solely from training samples. This reaffirms our capacity to predict model performance on unseen data based only on training data.

## Overview and Unique Contributions

We addressed concerns raised about the originality of our paper by emphasizing our unique approach. To encapsulate our contributions:

- **Information-Theoretic Perspective**: We offer a fresh take on VICReg through an information theory lens, unraveling the deeper intricacies of its operation.
- **Illuminating VICReg's Assumptions**: We've elucidated VICReg's latent assumptions, fostering clearer understanding and allowing comparison to other SSL methods.
- **Pioneering VICReg Generalization Bound**: We've introduced the maiden generalization bound for VICReg, spotlighting its theoretical prowess and practical ramifications.
- **Innovative Information-Based SSL Objective**: Insights from our analysis led us to propose a novel information-based objective, which showed improved performance over existing SSL methods.
- **Robust Empirical Validation**: Our empirical analyses robustly validate our theoretical postulates, showcasing the practical potency of our propositions, including our assumptions.


## Addressing Concerns and Questions:
We addressed each reviewer's individual weaknesses and questions in detail. Please refer to the individual responses below for detailed explanations of how we addressed each concern.

**In conclusion**, we sincerely believe that the revisions and clarifications made in response to the reviewers' feedback have significantly strengthened our manuscript.


We have put forth significant effort to address all your feedback and feel that your suggestions have improved our work. We welcome further discussion, and we appreciate your valuable input.

## References:
[1] Le, Y. et al., "Tiny ImageNet Visual Recognition Challenge", 2015.

[2] Dosovitskiy, A. et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", 2020.

[3] Liu, Z. et al, "A ConvNet for the 2020s", 2022.

[4] He, K. et al., "Momentum Contrast for Unsupervised Visual Representation Learning", 2020. 

[5] Caron, M. et al., "Unsupervised Learning of Visual Features by Contrasting Cluster Assignments", 2022.

[6] Kawaguchi, K. et al., "How Does Information Bottleneck Help Deep Learning?", 2023

Meta-review: The authors connect variance-invariance-covariance regularization (VICReg) objective optimization to optimizing the lower bound on some information-theoretic quantities. They then derive generalization bounds for VICReg, relating this objective to the performance on downstream tasks. To be able to use information-theoretic approaches, they assumed randomness in the data, allowing them to give bounds for deterministic networks.
The authors also present some experiments justifying their assumptions about the data distribution, etc.. The reviewers found those to be limited in scope, and the authors expanded a bit during the rebuttal period. In addition, during the rebuttal the authors also provided additional experiments showing that their derived bounds correlate with the observed generalization gap. The latter I found to be quite limited (see, e.g., “In Search of Robust Measures of Generalization” Dziugaite et al and “Fantastic Generalization Measures and Where to Find Them” Jiang et al for some guidance on what more extensive evaluations should look like). Overall, the key contributions are theoretical and the weakness of the empirical results was addressed to some extent during the rebuttal period, thus I recommend acceptance.
Decision: Accept (poster)