Title: An Information Theory Perspective on Variance-Invariance-Covariance Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self-Supervised Learning

Generalization Bounds

Information-Theory

Deep Neural Networks
Abstract: Variance-Invariance-Covariance Regularization (VICReg) is a self-supervised learning (SSL) method that has shown promising results on a variety of tasks. However, the fundamental mechanisms underlying VICReg remain unexplored. In this paper, we present an information-theoretic perspective on the VICReg objective. We begin by deriving information-theoretic quantities for deterministic networks as an alternative to unrealistic stochastic network assumptions. We then relate the optimization of the VICReg objective to mutual information optimization, highlighting underlying assumptions and facilitating a constructive comparison with other SSL algorithms and derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. Building on these results, we introduce a family of SSL methods derived from information-theoretic principles that outperform existing SSL techniques.

An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization

Abstract

Variance-Invariance-Covariance Regularization (VICReg) is a self-supervised learning (SSL) method that has shown promising results on a variety of tasks. However, the fundamental mechanisms underlying VICReg remain unexplored. In this paper, we present an information-theoretic perspective on the VICReg objective. We begin by deriving information-theoretic quantities for deterministic networks as an alternative to unrealistic stochastic network assumptions. We then relate the optimization of the VICReg objective to mutual information optimization, highlighting underlying assumptions and facilitating a constructive comparison with other SSL algorithms and derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. Building on these results, we introduce a family of SSL methods derived from information-theoretic principles that outperform existing SSL techniques.

1 Introduction

Self-supervised learning (SSL) is a promising approach to extracting meaningful representations by optimizing a surrogate objective between inputs and self-generated signals. For example, VarianceInvariance-Covariance Regularization (VICReg) [7], a widely-used SSL algorithm employing a de-correlation mechanism, circumvents learning trivial solutions by applying variance and covariance regularization.

Once the surrogate objective is optimized, the pre-trained model can be used as a feature extractor for a variety of downstream supervised tasks such as image classification, object detection, instance segmentation, or pose estimation [15, 16, 48, 67]. Despite the promising results demonstrated by SSL methods, the theoretical underpinnings explaining their efficacy continue to be the subject of investigation [5, 42].

Information theory has proved a useful tool for improving our understanding of deep neural networks (DNNs), having a significant impact on both applications in representation learning [3] and theoretical explorations [60, 72]. However, applications of information-theoretic principles to SSL have made unrealistic assumptions, making many existing information-theoretic approaches to SSL of limited use. One such assumption is to assume that the DNN to be optimized is stochastic-an assumption that is violated for the vast majority DNNs used in practice. For a comprehensive review on this topic, refer to the work by Shwartz-Ziv and LeCun [62].

In this paper, we examine Variance-Invariance-Covariance Regularization (VICReg), an SSL method developed for deterministic DNNs, from an information-theoretic perspective. We propose an approach that addresses the challenge of mutual information estimation in deterministic networks by transitioning the randomness from the networks to the input data-a more plausible assumption. This shift allows us to apply an information-theoretic analysis to deterministic networks. To establish a connection between the VICReg objective and information maximization, we identify and empirically validate the necessary assumptions. Building on this analysis, we describe differences between different SSL algorithms from an information-theoretic perspective and propose a new family of plug-in methods for SSL. This new family of methods leverages existing information estimators and achieves state-of-the-art predictive performance across several benchmarking tasks. Finally, we derive a generalization bound that links information optimization and the VICReg objective to downstream task performance, underscoring the advantages of VICReg.

Our key contributions are summarized as follows:

We introduce a novel approach for studying deterministic deep neural networks from an information-theoretic perspective by shifting the stochasticity from the networks to the inputs using the Data Distribution Hypothesis (Section 3)

We establish a connection between the VICReg objective and information-theoretic optimization, using this relationship to elucidate the underlying assumptions of the objective and compare it to other SSL methods (Section 4).

We propose a family of information-theoretic SSL methods, grounded in our analysis, that achieve state-of-the-art performance (Section 5).

We derive a generalization bound that directly links VICReg to downstream task generalization, further emphasizing its practical advantages over other SSL methods (Section 6).

2 Background & Preliminaries

We first introduce the necessary technical background for our analysis.

2.1 Continuous Piecewise Affine (CPA) Mappings.

A rich class of functions emerges from piecewise polynomials: spline operators. In short, given a partition Ω of a domain R D , a spline of order k is a mapping defined by a polynomial of order k on each region ω ∈ Ω with continuity constraints on the entire domain for the derivatives of order 0 ,. . . , k -1 . As we will focus on affine splines ( k = 1 ), we only define this case for clarity. A K -dimensional affine spline f produces its output via f ( z ) = ∑ ω ∈ Ω ( A ω z + b ω ) ✶ { z ∈ ω } , with input z ∈ R D and A ω ∈ R K × D , b ω ∈ R K , ∀ ω ∈ Ω the per-region slope and offset parameters respectively, with the key constraint that the entire mapping is continuous over the domain f ∈ C 0 ( R D ) .

Deep Neural Networks as CPA Mappings. A deep neural network (DNN) is a (non-linear) operator f Θ with parameters Θ that map a input x ∈ R D to a prediction y ∈ R K . The precise definitions of DNN operators can be found in Goodfellow et al. [27]. To avoid cluttering notation, we will omit Θ unless needed for clarity. For our analysis, we only assume that the non-linearities in the DNN are CPA mappings-as is the case with (leaky-) ReLU, absolute value, and max-pooling operators. The entire input-output mapping then becomes a CPA spline with an implicit partition Ω , the function of the weights and architecture of the network [51, 6]. For smooth nonlinearities, our results hold using a first-order Taylor approximation argument.

2.2 Self-Supervised Learning.

SSL is a set of techniques that learn representation functions from unlabeled data, which can then be adapted to various downstream tasks. While supervised learning relies on labeled data, SSL formulates a proxy objective using self-generated signals. The challenge in SSL is to learn useful representations without labels. It aims to avoid trivial solutions where the model maps all inputs to a constant output [11, 36]. To address this, SSL utilizes several strategies. Contrastive methods like SimCLR and its InfoNCE criterion learn representations by distinguishing positive and negative examples [16, 54]. In contrast, non-contrastive methods apply regularization techniques to prevent the collapse [14, 17, 28].

Variance-Invariance-Covariance Regularization (VICReg). A widely used SSL method for training joint embedding architectures [7]. Its loss objective is composed of three terms: the invariance loss, the variance loss, and the covariance loss:

Invariance loss: The invariance loss is given by the mean-squared Euclidean distance between pairs of embedding and ensures consistency between the representation of the original and augmented inputs.

Regularization: The regularization term consists of two loss terms: the variance loss -a hinge loss to maintain the standard deviation (over a batch) of each variable of the embedding-and the covariance loss , which penalizes off-diagonal coefficients of the covariance matrix of the embeddings to foster decorrelation among features.

VICReg generates two batches of embeddings, Z = [ f ( x 1 ) , . . . , f ( x B )] and Z ′ = [ f ( x ′ 1 ) , . . . , f ( x ′ B )] , each of size ( B × K ) . Here, x i and x ′ i are two distinct random augmentations of a sample I i . The covariance matrix C ∈ R K × K is obtained from [ Z , Z ′ ] . The VICReg loss can thus be expressed as follows:

̸

2.3 Deep Neural Networks and Information Theory

Recently, information-theoretic methods have played an essential role in advancing deep learning by developing and applying information-theoretic estimators and learning principles to DNN training [3, 9, 34, 56, 64, 65, 69, 72]. However, information-theoretic objectives for deterministic DNNs often face a common issue: the mutual information between the input and the DNN representation is infinite. This leads to ill-posed optimization problems.

Several strategies have been suggested to address this challenge. One involves using stochastic DNNs with variational bounds, where the output of the deterministic network is used as the parameters of the conditional distribution [43, 61]. Another approach, as suggested by Dubois et al. [22], assumes that the randomness of data augmentation among the two views is the primary source of stochasticity in the network. However, these methods assume that randomness comes from the DNN, contrary to common practice. Other research has presumed a random input but has made no assumptions about the network's representation distribution properties. Instead, it relies on general lower bounds to analyze the objective [71, 75].

3 Self-Supervised Learning in DNNs: An Information-Theoretic Perspective

To analyze information within deterministic networks, we first need to establish an informationtheoretic perspective on SSL (Section 3.1). Subsequently, we utilize the Data Distribution Hypothesis (Section 3.2) to demonstrate its applicability to deterministic SSL networks.

3.1 Self-Supervised Learning from an Information-Theoretic Viewpoint

Our discussion begins with the MultiView InfoMax principle , which aims to maximize the mutual information I ( Z ; X ′ ) between a view X ′ and the second representation Z . As demonstrated in Federici et al. [24], we can optimize this information by employing the following lower bound:

Here, H ( Z ) represents the entropy of Z . In supervised learning, the labels Y remain fixed, making the entropy term H ( Y ) a constant. Consequently, the optimization is solely focused on the log-loss, E x ′ [log q ( z | x ′ )] , which could be either cross-entropy or square loss.

However, for joint embedding networks, a degenerate solution can emerge, where all outputs 'collapse' into an undesired value [16]. Upon examining Equation (2), we observe that the entropies are not fixed and can be optimized. As a result, minimizing the log loss alone can lead the representations to collapse into a trivial solution and must be regularized.

3.2 Understanding the Data Distribution Hypothesis

Previously, we mentioned that a naive analysis might suggest that the information in deterministic DNNs is infinite. To address this point, we investigate whether assuming a dataset is a mixture of Gaussians with non-overlapping support can provide a manageable distribution over the neural network outputs. This assumption is less restrictive compared to assuming that the neural network itself is stochastic, as it concerns the generative process of the data, not the model and training process. For a detailed discussion about the limitations of assuming stochastic networks and a comparison between stochastic networks vs stochastic input, see Appendix N. In Section 4.2, we verify that this assumption holds for real-world datasets.

The so-called manifold hypothesis allows us to treat any point as a Gaussian random variable with a low-rank covariance matrix aligned with the data's manifold tangent space [25], which enables us to examine the conditioning of a latent representation with respect to the mean of the observation, i.e., X | x ∗ ∼ N ( x ; x ∗ , Σ x ∗ ) . Here, the eigenvectors of Σ x ∗ align with the tangent space of the data manifold at x ∗ , which varies with the position of x ∗ in space. In this setting, a dataset is considered a collection of distinct points x ∗ n , n = 1 , ..., N , and the full data distribution is expressed as a sum of Gaussian densities with low-rank covariance, defined as:

Here, T is a uniform Categorical random variable. For simplicity, we assume that the effective support of N ( x ∗ i , Σ x ∗ i ) do not overlap (for empirical validation of this assumption see Section 4.2). The effective support is defined as { x ∈ R D : p ( x ) > ϵ } . We can then approximate the density function as follows:

where N ( x ; ., . ) is the Gaussian density at x and n ( x ) = arg min n ( x -x ∗ n ) T Σ x ∗ n ( x -x ∗ n ) .

3.3 Data Distribution Under the Deep Neural Network Transformation

Let us consider an affine spline operator f , as illustrated in Section 2.1, which maps a space of dimension D to a space of dimension K , where K ≥ D . The image or the span of this mapping is expressed as follows:

In this equation, Aff ( ω ; A ω , b ω ) = { A ω x + b ω : x ∈ ω } denotes the affine transformation of region ω by the per-region parameters A ω , b ω . Ω denotes the partition of the input space where x resides. To practically compute the per-region affine mapping, we set A ω to the Jacobian matrix of the network at the corresponding input x , and b to be defined as f ( x ) -A ω x . Therefore, the DNN mapping composed of affine transformations on each input space partition region ω ∈ Ω based on the coordinate change induced by A ω and the shift induced by b ω .

When the input space is associated with a density distribution, this density is transformed by the mapping f . Calculating the density of f ( X ) is generally intractable. However, under the disjoint support assumption from Section 3.2, we can arbitrarily increase the density's representation power by raising the number of prototypes N . As a result, each Gaussian's support is contained within the region ω where its means lie, leading to the following theorem:

Theorem 1 . Given the setting of Equation (4), the unconditional DNN output density, Z , can be approximated as a mixture of the affinely transformed distributions x | x ∗ n ( x ) :

where ω ( x ∗ n ) = ω ∈ Ω ⇐⇒ x ∗ n ∈ ω is the partition region in which the prototype x ∗ n lives in. Proof. See Appendix B.

In other words, Theorem 1 implies that when the input noise is small, we can simplify the conditional output density to a single Gaussian: ( Z ′ | X ′ = x n ) ∼ N ( µ ( x n ) , Σ( x n )) , where µ ( x n ) = A ω ( x n ) x n + b ω ( x n ) and Σ( x n ) = A T ω ( x n ) Σ x n A ω ( x n ) .

0.0

0.2

0.4

0.6

0.8

σ

)

0.01

0.10

0.20

0.30

p-value

→

More Gaussian

SimCLR

VICReg

SwAV

MNIST

CIFAR10

CIFAR100

Flowers102

Food101

FGVCAircraft

0

25

50

75

100

125

150

175

l2 Distances

4 Information Optimization and the VICReg Optimization Objective

Building on our earlier discussion, we used the Data Distribution Hypothesis to model the conditional output in deterministic networks as a Gaussian mixture. This allowed us to frame the SSL training objective as maximizing the mutual information, I ( Z ; X ′ ) and I ( Z ′ ; X ) .

However, in general, this mutual information is intractable. Therefore, we will use our derivation for the network's representation to obtain a tractable variational approximation using the expected loss, which we can optimize.

The computation of expected loss requires us to marginalize the stochasticity in the output. We can employ maximum likelihood estimation with a Gaussian observation model. For computing the expected loss over x ′ samples, we must marginalize the stochasticity in Z ′ . This procedure implies that the conditional decoder adheres to a Gaussian distribution: ( Z | X ′ = x n ) ∼ N ( µ ( x n ) , I +Σ( x n )) .

However, calculating the expected log loss over samples of Z is challenging. We thus focus on a lower bound - the expected log loss over Z ′ samples. Utilizing Jensen's inequality, we derive the following lower bound:

Taking the expectation over Z , we get

Combining all of the above, we obtain

The full derivations are presented in Appendix A. To optimize this objective in practice, we approximate p ( x, x ′ ) using the empirical data distribution

4.1 Variance-Invariance-Covariance Regularization: An Information-Theoretic Perspective

Next, we connect the VICReg to our information-theoretic-based objective. The 'invariance term' in Equation (9), which pushes augmentations from the same image closer together, is the same term used in the VICReg objective. However, the computation of the regularization term poses a significant challenge. Entropy estimation is a well-established problem within information theory, with Gaussian mixture densities often used for representation. Yet, the differential entropy of Gaussian mixtures lacks a closed-form solution [55].

A straightforward method for approximating entropy involves capturing the distribution's first two moments, which provides an upper bound on the entropy. However, minimizing an upper bound doesn't necessarily optimize the original objective. Despite reported success from minimizing an upper bound [46, 53], this approach may induce instability during the training process.

Let Σ Z denote the covariance matrix of Z . We utilize the first two moments to approximate the entropy we aim to maximize. Because the invariance term appears in the same form as the original VICReg objective, we will look only at the regularizer. Consequently, we get the approximation

Theorem 2 . Assuming that the eigenvalues of Σ( x i ) and Σ( x ′ i ) , along with the differences between the Gaussian means µ ( x i ) and µ ( x ′ i ) , are bounded, the solution to the maximization problem

involves setting Σ Z to a diagonal matrix.

According to Theorem 2, we can maximize Equation (10) by diagonalizing the covariance matrix and increasing its diagonal elements. This goal can be achieved by minimizing the off-diagonal elements of Σ Z -the covariance criterion of VICReg-and by maximizing the sum of the log of its diagonal elements. While this approach is straightforward and efficient, it does have a drawback: the diagonal values could tend towards zero, potentially causing instability during logarithm computations. A solution to this issue is to use an upper bound and directly compute the sum of the diagonal elements, resulting in the variance term of VICReg. This establishes the link between our information-theoretic objective and the three key components of VICReg.

4.2 Empirical Validation of Assumptions About Data Distributions

Validating our theory, we tested if the conditional output density P ( Z | X ) becomes a Gaussian as input noise lessens. We used a ResNet-50 model trained with SimCLR or VICReg objectives on CIFAR-10, CIFAR-100, and ImageNet datasets. We sampled 512 Gaussian samples for each image from the test dataset, examining whether each sample remained Gaussian in the DNN's penultimate layer. We applied the D'Agostino and Pearson's test to ascertain the validity of this assumption [19].

Figure 1 (left) displays the p -value as a function of the normalized std. For low noise levels, we reject that the network's conditional output density is non-Gaussian with an 85% probability when using VICReg. However, the network output deviates from Gaussian as the input noise increases.

Next, we verified our assumption of non-overlapping effective support in the model's data distribution. We calculate the distribution of pairwise l 2 distances between images across seven datasets: MNIST [41], CIFAR10, CIFAR100 [40], Flowers102 [52], Food101 [12], and FGVAircaft [45]. Figure 1 (right) reveals that the pairwise distances are far from zero, even for raw pixels. This implies that we can use a small Gaussian around each point without overlap, validating our assumption as realistic.

5 Self-Supervised Learning Models through Information Maximization

The practical application of Equation (8) involves several key 'design choices'. We begin by comparing how existing SSL models have implemented it, investigating the estimators used, and discussing the implications of their assumptions. Subsequently, we introduce new methods for SSL that incorporate sophisticated estimators from the field of information theory, which outperform current approaches.

5.1 VICReg vs. SimCLR

In order to evaluate their underlying assumptions and strategies for information maximization, we compare VICReg to contrastive SSL methods such as SimCLR along with non-contrastive methods like BYOL and SimSiam.

Contrastive Learning with SimCLR. In their work, Lee et al. [43] drew a connection between the SimCLR objective and the variational bound on information regarding representations by employing the von Mises-Fisher distribution. By applying our analysis for information in deterministic networks with their work, we compare the main differences between SimCLR and VICReg:

(i) Conditional distribution: SimCLR assumes a von Mises-Fisher distribution for the encoder, while VICReg assumes a Gaussian distribution. (ii) Entropy estimation: SimCLR approximated it based on the finite sum of the input samples. In contrast, VICReg estimates the entropy of Z solely based on the second moment. Developing SSL methods that integrate these two distinctions form an intriguing direction for future research.

Empirical comparison. We trained ResNet18 on CIFAR-10 for VICReg, SimCLR, and BYOL and compared their entropies directly using the pairwise distances entropy estimator. (For more details, see Appendix K.) This estimator was not directly optimized by any method and was an independent validation. The results (Figure 2), showed that entropy increased for all methods during training, with SimCLR having the lowest and VICReg the highest entropy.

5.2 Family of alternative Entropy Estimators

Next, we suggest integrating the invariance term of current SSL methods with plug-in methods that optimize entropy.

Entropy estimators. The VICReg objective seeks to approximate the log determinant of the empirical covariance matrix through its diagonal terms. As discussed in Section 4.1, this approach has its drawbacks. An alternative is to employ different entropy estimators. The LogDet Entropy Estimator [74] is one such option, offering a tighter upper bound. This estimator employs the differential entropy of α order with scaled noise and has been previously shown to be a tight estimator for high-dimensional features, proving robust to random noise. However, since this estimator provides an upper bound on entropy, maximizing this bound doesn't guarantee optimization of the original objective. To counteract this, we also introduce a lower bound estimator based on the pairwise distances of individual mixture components [38]. In this family, a function determining pairwise distances between component densities is designated for each member. These estimators are computationally efficient and typically straightforward to optimize. For additional entropy estimators, see Appendix F. Beyond VICReg, these methods can serve as plug-in estimators for numerous SSL algorithms. Apart from VICReg, we also conducted experiments integrating these estimators with the BYOL algorithm.

1e0

1e1

1e2

1e3

Steps

3

6

9

H

(

Z

)

SimCLR

VICReg

BYOL

VICReg+ LogDet

VICReg+ PairDist

Setup. Experiments were conducted on three image datasets: CIFAR-10, CIFAR-100 [39], and Tiny-ImageNet [20]. For CIFAR-10, ResNet-18 [31] was used. In contrast, both ConvNeXt [44] and Vision Transformer [21] were used for CIFAR-100 and Tiny-ImageNet. For comparison, we examined the following SSL methods: VICReg, SimCLR, BYOL, SwAV [14], Barlow Twins [73], and MoCo [33]. The quality of representation was assessed through linear evaluation. A detailed description of different methods can be found in Appendix H.

Results. As evidenced by Table 1, the proposed entropy estimators surpass the original SSL methods. Using a more precise entropy estimator enhances the performance of both VICReg and BYOL, compared to their initial implementations. Notably, the pairwise distance estimator, being a lower bound, achieves superior results, resonating with the theoretical preference for maximizing a true entropy's lower bound. Our findings suggest that the astute choice of entropy estimators, guided by our framework, paves the way for enhanced performance.

6 A Generalization Bound for Downstream Tasks

In earlier sections, we linked information theory principles with the VICReg objective. Now, we aim to extend this link to downstream generalization via a generalization bound. This connection further aligns VICReg's generalization with information maximization and implicit regularization.

Notation. Consider input points x , outputs y ∈ R r , labeled training data S = (( x i , y i )) n i =1 of size n and unlabeled training data ¯ S = (( x + i , x ++ i )) i m =1 of size m , where x + i and x ++ i share the same (unknown) label. With the unlabeled training data, we define the invariance loss

where f θ is the trained representation on the unlabeled data ¯ S . We define a labeled loss ℓ x,y ( w ) = ∥ Wf θ ( x ) -y ∥ where w = vec[ W ] ∈ R dr is the vectorization of the matrix W ∈ R r × d . Let w S = vec[ W S ] be the minimum norm solution as W S = minimize W ′ ∥ W ′ ∥ F such that

We also define the representation matrices

and the projection matrices

We define the label matrix Y S = [ y 1 , . . . , y n ] ⊤ ∈ R n × r and the unknown label matrix Y ¯ S = [ y + 1 , . . . , y + m ] ⊤ ∈ R m × r , where y + i is the unknown label of x + i . Let F be a hypothesis space of f θ . For a given hypothesis space F , we define the normalized Rademacher complexity

where ξ 1 , . . . , ξ m are independent uniform random variables in {-1 , 1 } . It is normalized such that ˜ R m ( F ) = O (1) as m →∞ .

6.1 AGeneralizationBoundforVariance-Invariance-CovarianceRegularization

Now we will show that the VICReg objective improves generalization on supervised downstream tasks. More specifically, minimizing the unlabeled invariance loss while controlling the covariance Z ¯ S Z ¯ S ⊤ and the complexity of representations ˜ R m ( F ) minimizes the expected labeled loss :

Theorem 3 . (Informal version). For any δ > 0 , with probability at least 1 -δ ,

where Q m,n = O ( G √ ln(1 /δ ) /m + √ ln(1 /δ ) /n ) → 0 as m,n →∞ . In Q m,n , the value of G for the term decaying at the rate 1 / √ m depends on the hypothesis space of f θ and w whereas the term decaying at the rate 1 / √ n is independent of any hypothesis space.

Proof. The complete version of Theorem 3 and its proof are presented in Appendix I.

The term ∥ P Z ¯ S Y ¯ S ∥ F in Theorem 3 contains the unobservable label matrix Y ¯ S . However, we can minimize this term by using ∥ P Z ¯ S Y ¯ S ∥ F ≤ ∥ P Z ¯ S ∥ F ∥ Y ¯ S ∥ F and by minimizing ∥ P Z ¯ S ∥ F . The factor ∥ P Z ¯ S ∥ F is minimized when the rank of the covariance Z ¯ S Z ¯ S ⊤ is maximized. This can be enforced by maximizing the diagonal entries while minimizing the off-diagonal entries, as is done in VICReg.

The term ∥ P Z S Y S ∥ F contains only observable variables, and we can directly measure the value of this term using training data. In addition, the term ∥ P Z S Y S ∥ F is also minimized when the rank of the covariance Z S Z S ⊤ is maximized. Since the covariances Z S Z S ⊤ and Z ¯ S Z ¯ S ⊤ concentrate to each other via concentration inequalities with the error in the order of O ( √ (ln(1 /δ )) /n + ˜ R m ( F ) √ (ln(1 /δ )) /m ) , we can also minimize the upper bound on ∥ P Z S Y S ∥ F by maximizing the diagonal entries of Z ¯ S Z ¯ S ⊤ while minimizing its off-diagonal entries, as is done in VICReg.

Thus, VICReg can be understood as a method to minimize the generalization bound in Theorem 3 by minimizing the invariance loss while controlling the covariance to minimize the label-agnostic upper bounds on ∥ P Z ¯ S Y ¯ S ∥ F and ∥ P Z S Y S ∥ F . If we know partial information about the label Y ¯ S of the unlabeled data, we can use it to minimize ∥ P Z ¯ S Y ¯ S ∥ F and ∥ P Z S Y S ∥ F directly.

6.2 Comparison of Generalization Bounds

The SimCLR generalization bound [58] requires the number of labeled classes to go infinity to close the generalization gap, whereas the VICReg bound in Theorem 3 does not require the number of label classes to approach infinity for the generalization gap to go to zero. This reflects that, unlike SimCLR, VICReg does not use negative pairs and thus does not use a loss function based on the implicit expectation that the labels of a negative pair are different. Another difference is that our VICReg bound improves as n increases, while the previous bound of SimCLR [58] does not depend on n . This is because Saunshi et al. [58] assumes partial access to the true distribution per class for setting, which removes the importance of labeled data size n and is not assumed in our study.

Consequently, the generalization bound in Theorem 3 provides a new insight for VICReg regarding the ratio of the effects of m v.s. n through G √ ln(1 /δ ) /m + √ ln(1 /δ ) /n . Finally, Theorem 3 also illuminates the advantages of VICReg over standard supervised training. That is, with standard training, the generalization bound via the Rademacher complexity requires the complexities of hypothesis spaces, ˜ R n ( W ) / √ n and ˜ R n ( F ) / √ n , with respect to the size of labeled data n , instead of the size of unlabeled data m . Thus, Theorem 3 shows that using SSL, we can replace the complexities of hypothesis spaces in terms of n with those in terms of m . Since the number of unlabeled data points is typically much larger than the number of labeled data points, this illuminates the benefit of SSL.

6.3 Understanding Theorem 2 via Mutual Information Maximization

Theorem 3, together with the result of the previous section, shows that, for generalization in the downstream task, it is helpful to maximize the mutual information I ( Z ; X ′ ) in SSL via minimizing the invariance loss I ¯ S ( f θ ) while controlling the covariance Z ¯ S Z ¯ S ⊤ . The term 2 ˜ R m ( F ) / √ m captures the importance of controlling the complexity of the representations f θ . To understand this term further, let us consider a discretization of the parameter space of F to have finite |F| < ∞ . Then, by Massart's Finite Class Lemma, we have that ˜ R m ( F ) ≤ C √ ln |F| for some constant C > 0 . Moreover, Shwartz-Ziv [60] shows that we can approximate ln |F| by 2 I ( Z ; X ) . Thus, in Theorem 3, the term I ¯ S ( f θ ) + 2 √ m ∥ P Z ¯ S Y ¯ S ∥ F + 1 √ n ∥ P Z S Y S ∥ F corresponds to I ( Z ; X ′ ) which we want to maximize while compressing the term of 2 ˜ R m ( F ) / √ m which corresponds to I ( Z ; X ) [23, 63, 66].

Although we can explicitly add regularization on the information to control 2 ˜ R m ( F ) / √ m , it is possible that I ( Z ; X | X ′ ) and 2 ˜ R m ( F ) / √ m are implicitly regularized via implicit bias through design choises [29, 68, 30]. Thus, Theorem 3 connects the information-theoretic understanding of VICReg with the probabilistic guarantee on downstream generalization.

7 Limitations

In our paper, we proposed novel methods for SSL premised on information maximization. Although our methods demonstrated superior performance on some datasets, computational constraints precluded us from testing them on larger datasets. Furthermore, our study hinges on certain assumptions that, despite rigorous validation efforts, may not hold universally. While we strive for meticulous testing and validation, it's crucial to note that some assumptions might not be applicable in all scenarios or conditions. These limitations should be taken into account when interpreting our study's results.

8 Conclusions

We analyzed the Variance-Invariance-Covariance Regularization for self-supervised learning through an information-theoretic lens. By transferring the stochasticity required for an information-theoretic analysis to the input distribution, we showed how the VICReg objective can be derived from information-theoretic principles, used this perspective to highlight assumptions implicit in the VICReg objective, derived a VICReg generalization bound for downstream tasks, and related it to information maximization.

Building on these findings, we introduced a new VICReg-inspired SSL objective. Our probabilistic guarantee suggests that VICReg can be further improved for the settings of partial label information by aligning the covariance matrix with the partially observable label matrix, which opens up several avenues for future work, including the design of improved estimators for information-theoretic quantities and investigations into the suitability of different SSL methods for specific data characteristics.

References

Alessandro Achille, Giovanni Paolini, and Stefano Soatto. Where is the information in a deep neural network? arXiv preprint arXiv:1905.12213 , 2019.

N.A. Ahmed and D.V. Gokhale. Entropy expressions and their estimators for multivariate distributions. IEEE Transactions on Information Theory , 35(3):688-692, 1989. doi: 10.1109/ 18.30996.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

Rana Ali Amjad and Bernhard Claus Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence , 2019.

Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 , 2019.

Randall Balestriero and Richard Baraniuk. A spline theory of deep networks. In Proc. ICML , volume 80, pages 374-383, Jul. 2018.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 , 2018.

Itamar Ben-Ari and Ravid Shwartz-Ziv. Attentioned convolutional lstm inpaintingnetwork for anomaly detection in videos. arXiv preprint arXiv:1811.10228 , 2018.

Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineering self-supervised learning. arXiv preprint arXiv:2305.15614 , 2023.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014.

Brendon J Brewer. Computing entropies with nested sampling. Entropy , 19(8):422, 2017.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems , 33:9912-9924, 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9650-9660, 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15750-15758, 2021.

Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research , 13(1):795-828, 2012.

Ralph B. D'Agostino. An omnibus test of normality for moderate and large size samples. Biometrika , 58(2):341-348, 1971. ISSN 00063444. URL http://www.jstor.org/stable/2334522 .

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yann Dubois, Benjamin Bloem-Reddy, Karen Ullrich, and Chris J Maddison. Lossy compression for lossless prediction. Advances in Neural Information Processing Systems , 34, 2021.

Marco Federici, Anjan Dutta, Patrick Forr´ e, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations , 2019.

Marco Federici, Anjan Dutta, Patrick Forr´ e, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017 , 2020.

Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society , 29(4):983-1049, 2016.

Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. arXiv preprint arXiv:1810.05728 , 2018.

I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning , volume 1. MIT Press, 2016.

Jean-Bastien Grill, Florian Strub, Florent Altch´ e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems , pages 6151-6159, 2017.

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems , pages 9461-9471, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018.

Marco Huber, Tim Bailey, Hugh Durrant-Whyte, and Uwe Hanebeck. On entropy approximation for gaussian mixture random vectors. In IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems , pages 181 - 188, 09 2008. doi: 10.1109/MFI.2008. 4648062.

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=YevsQ05DEN7 .

Kenji Kawaguchi, Zhun Deng, Kyle Luh, and Jiaoyang Huang. Robustness Implies Generalization via Data-Dependent Generalization Bounds. In International Conference on Machine Learning (ICML) , 2022.

Artemy Kolchinsky and Brendan D Tracey. Estimating mixture entropy with pairwise distances. Entropy , 19(7):361, 2017.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable self-supervised learning. Advances in Neural Information Processing Systems , 34, 2021.

Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances in Neural Information Processing Systems , 34, 2021.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11976-11986, 2022.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Julieta Martinez, Jashan Shewakramani, Ting Wei Liu, Ioan Andrei Bˆ arsan, Wenyuan Zeng, and Raquel Urtasun. Permute, quantize, and fine-tune: Efficient compression of neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15699-15708, 2021.

David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics , pages 875-884. PMLR, 2020.

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6707-6717, 2020.

Neeraj Misra, Harshinder Singh, and Eugene Demchuk. Estimation of the entropy of a multivariate normal distribution. Journal of Multivariate Analysis , 92(2):324-342, 2005. ISSN 0047-259X. doi: https://doi.org/10.1016/j.jmva.2003.10.003.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2012.

Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Proc. NeurIPS , pages 2924-2932, 2014.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing systems , pages 271-279, 2016.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Liam Paninski. Estimation of entropy and mutual information. Neural computation , 15(6): 1191-1253, 2003.

Zoe Piran, Ravid Shwartz-Ziv, and Naftali Tishby. The dual information bottleneck. arXiv preprint arXiv:2006.04641 , 2020.

Ben Poole, Sherjil Ozair, A¨ aron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information. CoRR , abs/1905.06922, 2019. URL http://arxiv.org/abs/1905.06922 .

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning , pages 5628-5637. PMLR, 2019.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Ravid Shwartz-Ziv. Information flow in deep neural networks. arXiv preprint arXiv:2202.06749 , 2022.

Ravid Shwartz-Ziv and Alexander A Alemi. Information in infinite ensembles of infinitely-wide neural networks. In Symposium on Advances in Approximate Bayesian Inference , pages 1-17. PMLR, 2020.

Ravid Shwartz-Ziv and Yann LeCun. To compress or not to compress-self-supervised learning and information theory: A review. arXiv preprint arXiv:2304.09355 , 2023.

Ravid Shwartz-Ziv and Naftali Tishby. Compression of deep neural networks via information, (2017). arXiv preprint arXiv:1703.00810 , 2017.

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 , 2017.

Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation compression and generalization in deep neural networks, 2018.

Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What do we maximize in selfsupervised learning? arXiv preprint arXiv:2207.10081 , 2022.

Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew Gordon Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. arXiv preprint arXiv:2205.10279 , 2022.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1):2822-2878, 2018.

Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory , pages 3437-3452. PMLR, 2020.

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625 , 2019.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 9929-9939. PMLR, 2020.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems , 30, 2017.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ ephane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Zhanghao Zhouyin and Ding Liu. Understanding neural networks with logarithm determinant entropy estimator. arXiv preprint arXiv:2105.03705 , 2021.

Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning , pages 12979-12990. PMLR, 2021.

Table of Contents

This appendix is organized as follows:

In Appendix A, we provide a detailed derivation of the lower bound of Equation (8) .

In Appendix B, we provide full proof of our Theorem 1 on the network's representation distribution.

In Appendix C, we provide additional empirical validations. Specifically, we empirically check if the optimal solution to the information maximization problem in Section 4.1 is the diagonal matrix.

In Appendix D, we show the collapse phenomenon under Gaussian Mixture Model (GMM) using Expectation Maximization (EM) and demonstrate how it is related to SSL and how we can prevent it.

Appendix E provides additional details on the SimCLR method.

Appendix F provides a detailed review of entropy estimators, their implications, assumptions, and limitations.

In Appendix G, we provide proofs for known lemmas that we are using throughout our paper.

In Appendix H, we provide detailed information on the hyperparameters, datasets, and architectures used in our experiments in Section 5.2.

In Appendix I, we provide full proof of our generalization bound for downstream tasks from Section 6.

In Appendix J, we provide full proof of the theorems for Section 4.1 on the connection between information optimization and the VICReg objective.

Appendix K provides experimental details on experiments conducted in Section 5.1 for entropy comparison between different SSL methods.

In Appendix L, we provide detailed information on the reproducibility of our study.

In Appendix O, we discuss the broader impact of our work. This section explores the implications, significance, and potential applications of our findings beyond the scope of the immediate study.

Appendix

Appendix A Lower bounds on E x ′ [log q ( z | x ′ )]

In this section of the supplementary material, we present the full derivation of the lower bound on E x ′ [log q ( z | x ′ )] . Because Z ′ | X ′ is a Gaussian, we can write it as Z ′ = µ ( x ′ ) + L ( x ′ ) ϵ where ϵ ∼ N (0 , 1) and L ( x ′ ) T L ( x ′ ) = Σ( x ′ ) . Now, setting Σ r = I , will give us:

where E x ′ [log q ( z | x ′ )] = E x ′ [ log E z ′ | x ′ [ q ( z | z ′ )] ] ≥ E z ′ [log q ( z | z ′ )] by Jensen's inequality, E ϵ [ ϵ ] = 0 and E ϵ [ ϵ ( L ( x ′ ) T L ( x ′ ) ϵ ] = Tr log Σ( x ′ ) by the Hutchinson's estimator.

Appendix B Data Distribution after Deep Network Transformation

Theorem 4 . Given the setting of Equation (4), the unconditional DNN output density denoted as Z approximates (given the truncation of the Gaussian on its effective support that is included within a single region ω of the DN's input space partition) a mixture of the affinely transformed distributions x | x ∗ n ( x ) e.g. for the Gaussian case

where ω ( x ∗ n ) = ω ∈ Ω ⇐⇒ x ∗ n ∈ ω is the partition region in which the prototype x ∗ n lives in.

Proof. We know that If ∫ ω p ( x | x ∗ n ( x ) ) d x ≈ 1 then f is linear within the effective support of p . Therefore, any sample from p will almost surely lie within a single region ω ∈ Ω , and therefore the entire mapping can be considered linear with respect to p . Thus, the output distribution is a linear transformation of the input distribution based on the per-region affine mapping.

Appendix C Additional Empirical Validation

To validate empirically Theorem 2, we checke empirically if the optimal solution for

is a diagonal matrix. We trained VICReg on ResNet18 on CIFAR-10 and did random perturbations (with different scales) for Σ Z . Then, for each perturbation, we calculated the average distance of this perturbed matrix from a diagonal matrix and the actual value of the term

. In Figure 3, we plot the difference from the optimal value of this term as a function of the distance from the diagonal matrix. As we can see, we get an optimal solution where we are close to the diagonal matrix. This observation gives us an empirical validation of Theorem 2.

0.16

0.12

E

△

0.08

0.04

0.00

-2

-1

0

1

2

Distance from diagonal

H(Z)=45

data

centroids

H(Z)=40

data

centroids

H(Z)=34

data

centroids

Appendix D EMand GMM

Let us examine a toy dataset on the pattern of two intertwining moons to illustrate the collapse phenomenon under GMM (Figure 1, right). We begin by training a classical GMM with maximum likelihood, where the means are initialized based on random samples, and the covariance is used as the identity matrix. A red dot represents the Gaussian's mean after training, while a blue dot represents the data points. In the presence of fixed input samples, we observe that there is no collapsing and that the entropy of the centers is high (Figure 4, left). However, when we make the input samples trainable and optimize their location, all the points collapse into a single point, resulting in a sharp decrease in entropy (Figure 4, right).

To prevent collapse, we follow the K-means algorithm in enforcing sparse posteriors, i.e. using small initial standard deviations and learning only the mean. This forces a one-to-one mapping which leads all points to be closest to the mean without collapsing, resulting in high entropy (Figure 4 middle, in the Appendix). Another option to prevent collapse is to use different learning rates for input and parameters. Using this setting, the collapsing of the parameters does not maximize the likelihood. Figure 1 (right) shows the results of GMM with different learning rates for learned inputs and parameters. When the parameter learning rate is sufficiently high in comparison to the input learning rate, the entropy decreases much more slowly and no collapse occurs.

Appendix E SimCLR

In contrastive learning, different augmented views of the same image are attracted (positive pairs), while different augmented views are repelled (negative pairs). MoCo [33] and SimCLR [16] are recent examples of self-supervised visual representation learning that reduce the gap between selfsupervised and fully-supervised learning. SimCLR applies randomized augmentations to an image to create two different views, x and y , and encodes both of them with a shared encoder, producing representations r x and r y . Both r x and r y are l 2 -normalized. The SimCLR version of the InfoNCE objective is:

where η is a temperature term and K is the number of views in a minibatch.

Appendix F Entropy Estimators

Entropy estimation is one of the classical problems in information theory, where Gaussian mixture density is one of the most popular representations. With a sufficient number of components, they can approximate any smooth function with arbitrary accuracy. For Gaussian mixtures, there is, however, no closed-form solution to differential entropy. There exist several approximations in the literature, including loose upper and lower bounds [35]. Monte Carlo (MC) sampling is one way to approximate Gaussian mixture entropy. With sufficient MC samples, an unbiased estimate of entropy with an arbitrarily accurate can be obtained. Unfortunately, MC sampling is very computationally expensive and typically requires a large number of samples, especially in high dimensions [13]. Using the first two moments of the empirical distribution, VIGCreg used one of the most straightforward approaches for approximating the entropy. Despite this, previous studies have found that this method is a poor approximation of the entropy in many cases [35]. Another option is to use the LogDet function. Several estimators have been proposed to implement it, including uniformly minimum variance unbiased (UMVU) [2], and Bayesian methods [49]. These methods, however, often require complex optimizations. The LogDet estimator presented in [74] used the differential entropy α order entropy using scaled noise. They demonstrated that it can be applied to high-dimensional features and is robust to random noise. Based on Taylor-series expansions, [35] presented a lower bound for the entropy of Gaussian mixture random vectors. They use Taylor-series expansions of the logarithm of each Gaussian mixture component to get an analytical evaluation of the entropy measure. In addition, they present a technique for splitting Gaussian densities to avoid components with high variance, which would require computationally expensive calculations. Kolchinsky and Tracey [38] introduce a novel family of estimators for the mixture entropy. For this family, a pairwise-distance function between component densities is defined for each member. These estimators are computationally efficient as long as the pairwise-distance function and the entropy of each component distribution are easy to compute. Moreover, the estimator is continuous and smooth and is therefore useful for optimization problems. In addition, they presented both a lower bound (using Chernoff distance) and an upper bound (using the KL divergence) on the entropy, which are exact when the component distributions are grouped into well-separated clusters,

Appendix G Known Lemmas

We use the following well-known theorems as lemmas in our proofs. We put these below for completeness. These are classical results and not our results.

Lemma G.1. (Hoeffding's inequality) Let X 1 , ..., X n be independent random variables such that a ≤ X i ≤ b almost surely. Consider the average of these random variables, S n = 1 n ( X 1 + · · · + X n ) . Then, for all t > 0 ,

Proof. By using Hoeffding's inequality, we have that for all t > 0 ,

and and

Setting δ = exp ( -2 nt 2 ( b -a ) 2 ) and solving for t > 0 , 1 /δ = exp ( 2 nt 2 ( b -a ) 2 ) = ⇒ ln(1 /δ ) = 2 nt 2 ( b -a ) 2 = ⇒ ( b -a ) 2 ln(1 /δ ) 2 n = t 2 = ⇒ t = ( b -a ) √ ln(1 /δ ) 2 n

It has been shown that generalization bounds can be obtained via Rademacher complexity [8, 50, 59]. The following is a trivial modification of [50, Theorem 3.1] for a one-sided bound on the nonnegative general loss functions:

Lemma G.2. Let G be a set of functions with the codomain [0 , M ] . Then, for any δ > 0 , with probability at least 1 -δ over an i.i.d. draw of m samples S = ( q i ) i m =1 , the following holds for all ψ ∈ G :

where R m ( G ) := E S,ξ [sup ψ ∈G 1 m ∑ m i =1 ξ i ψ ( q i )] and ξ 1 , . . . , ξ m are independent uniform random variables taking values in {-1 , 1 } .

Proof. Let S = ( q i ) i m =1 and S ′ = ( q ′ i ) i m =1 . Define

To apply McDiarmid's inequality to φ ( S ) , we compute an upper bound on | φ ( S ) -φ ( S ′ ) | where S and S ′ be two test datasets differing by exactly one point of an arbitrary index i 0 ; i.e., S i = S ′ i for all i = i 0 and S i 0 = S ′ i 0 . Then,

Similarly, φ ( S ) -φ ( S ′ ) ≤ M m . Thus, by McDiarmid's inequality, for any δ > 0 , with probability at least 1 -δ ,

where the first line follows the definitions of each term, the second line uses Jensen's inequality and the convexity of the supremum, and the third line follows that for each ξ i ∈ {-1 , +1 } , the distribution of each term ξ i ( ℓ ( f ( x ′ i ) , y ′ i ) -ℓ ( f ( x i ) , y i )) is the distribution of ( ℓ ( f ( x ′ i ) , y ′ i ) -ℓ ( f ( x i ) , y i )) since S and S ′ are drawn iid with the same distribution. The fourth line uses the subadditivity of supremum.

̸

Moreover,

̸

Appendix H Implentation Details for Maximizing Entropy Estimators

In this section, we will provide more details on the implantation of the experiments conducted in Section 5.2.

Setup Our experiments are conducted on CIFAR-10 [40]. We use ResNet-18 [32] as our backbone.

Training Procedure : The experimental process is organized into two sequential stages: unsupervised pretraining followed by linear evaluation. Initially, the unsupervised pretraining phase is executed, during which the encoder network is trained. Upon its completion, we transition to the linear evaluation phase, which serves as an assessment tool for the quality of the representation produced by the pretrained encoder.

Once the pretraining phase is concluded, we adhere to the fine-tuning procedures used in established baseline methods, as described by [14].

During the linear evaluation stage, we start by performing supervised training of the linear classifier. This is achieved by using the representations derived from the encoder network while keeping the network's coefficients frozen, and applying the same training dataset. Subsequently, we measure the test accuracy of the trained linear classifier using a separate validation dataset. This approach allows us to evaluate the performance of our model in a robust and systematic manner.

The training process for each model unfolds over 800 epochs, employing a batch size of 512. We utilize the Stochastic Gradient Descent (SGD) optimizer, characterized by a momentum of 0.9 and a weight decay of 1 e -4 . The learning rate is initiated at 0.5 and is adjusted according to a cosine decay schedule complemented by a linear warmup phase.

During the data augmentation process, two enhanced versions of every input image are generated. This involves cropping each image randomly and resizing it back to the original resolution. The images are then subject to random horizontal flipping, color jittering, grayscale conversion, Gaussian blurring, and polarization for further augmentation.

For the linear evaluation phase, the linear classifier is trained for 100 epochs with a batch size of 256. The SGD optimizer is again employed, this time with a momentum of 0.9 and no weight decay. The learning rate is managed using a cosine decay schedule, starting at 0.2 and reaching a minimum of 2 e -4 .

Appendix I A Generalization Bound for Downstream Tasks

In this Appendix, we present the complete version of Theorem 3 along with its proof and additional discussions.

I.1 Additional Notation and details

We start to introduce additional notation and details. We use the notation of x ∈ X for an input and y ∈ Y ⊆ R r for an output. Define p ( y ) = P ( Y = y ) to be the probability of getting label y and ˆ p ( y ) = 1 n ∑ n i =1 ✶ { y i = y } to be the empirical estimate of p ( y ) . Let ζ be an upper bound on the norm of the label as ∥ y ∥ 2 ≤ ζ for all y ∈ Y . Define the minimum norm solution W ¯ S of the unlabeled data as W ¯ S = minimize W ′ ∥ W ′ ∥ F s.t. W ′ ∈ arg min W 1 m ∑ m i =1 ∥ Wf θ ( x + i ) -g ∗ ( x + i ) ∥ 2 . Let κ S be a data-dependent upper bound on the per-sample Euclidian norm loss with the trained model as ∥ W S f θ ( x ) -y ∥ ≤ κ S for all ( x, y ) ∈ X × Y . Similarly, let κ ¯ S be a data-dependent upper bound on the per-sample Euclidian norm loss as ∥ W ¯ S f θ ( x ) -y ∥ ≤ κ ¯ S for all ( x, y ) ∈ X × Y . Define the difference between W S and W ¯ S by c = ∥ W S -W ¯ S ∥ 2 . Let W be a hypothesis space of W such that W ¯ S ∈ W . We denote by ˜ R m ( W◦F ) = 1 √ m E ¯ S,ξ [sup W ∈W ,f ∈F ∑ m i =1 ξ i ∥ g ∗ ( x + i ) -Wf ( x + i ) ∥ ] the normalized Rademacher complexity of the set { x + ↦→ ∥ g ∗ ( x + ) -Wf ( x + ) ∥ : W ∈ W , f ∈ F} . we denote by κ a upper bound on the per-sample Euclidian norm loss as ∥ Wf ( x ) -y ∥ ≤ κ for all ( x, y, W, f ) ∈ X × Y × W × F .

We adopt the following data-generating process model that was used in a previous paper on analyzing contrastive learning [58, 10]. For the labeled data, first, y is drawn from the distribution ρ on Y , and then x is drawn from the conditional distribution D y conditioned on the label y . That is, we have the join distribution D ( x, y ) = D y ( x ) ρ ( y ) with (( x i , y i )) n i =1 ∼ D n . For the unlabeled data,

first, each of the unknown labels y + and y -is drawn from the distritbuion ρ , and then each of the positive examples x + and x ++ is drawn from the conditional distribution D y + while the negative example x -is drawn from the D y -. Unlike the analysis of contrastive learning, we do not require negative samples. Let τ ¯ S be a data-dependent upper bound on the invariance loss with the trained representation as ∥ f θ (¯ x ) -f θ ( x ) ∥ ≤ τ ¯ S for all (¯ x, x ) ∼ D 2 y and y ∈ Y . Let τ be a data-independent upper bound on the invariance loss with the trained representation as ∥ f (¯ x ) -f ( x ) ∥ ≤ τ for all (¯ x, x ) ∼ D 2 y , y ∈ Y , and f ∈ F . For simplicity, we assume that there exists a function g ∗ such that y = g ∗ ( x ) ∈ R r for all ( x, y ) ∈ X × Y . Discarding this assumption adds the average of label noises to the final result, which goes to zero as the sample sizes n and m increase, assuming that the mean of the label noise is zero.

0.2 Proof of Theorem 3

Proof of Theorem 3. Let W = W S where W S is the the minimum norm solution as W S = minimize W ′ ∥ W ′ ∥ F s.t. W ′ ∈ arg min W 1 n ∑ n i =1 ∥ Wf θ ( x i ) -y i ∥ 2 . Let W ∗ = W ¯ S where W ¯ S is the minimum norm solution as W ∗ = W ¯ S = minimize W ′ ∥ W ′ ∥ F s.t. W ′ ∈ arg min W 1 m ∑ m i =1 ∥ Wf θ ( x + i ) -g ∗ ( x + i ) ∥ 2 . Since y = g ∗ ( x ) ,

where φ ( x ) = g ∗ ( x ) -W ∗ f θ ( x ) . Define L S ( w ) = 1 n ∑ n i =1 ∥ Wf θ ( x i ) -y i ∥ . Using these,

where ˜ W = W -W ∗ . We now consider new fresh samples ¯ x i ∼ D y i for i = 1 , . . . , n to rewrite the above further as:

This implies that

Furthermore, since y = W ∗ f θ ( x ) + φ ( x ) , by writing ¯ y i = W ∗ f θ (¯ x i ) + φ (¯ x i ) (where ¯ y i = y i since ¯ x i ∼ D y i for i = 1 , . . . , n ),

Combining these, we have that

To bound the left-hand side of equation 34, we now analyze the following random variable:

where ¯ y i = y i since ¯ x i ∼ D y i for i = 1 , . . . , n . Importantly, this means that as W S depends on y i , W S depends on ¯ y i . Thus, the collection of random variables ∥ W S f θ (¯ x 1 ) -¯ y 1 ∥ , . . . , ∥ W S f θ ( n n ) -¯ y n ∥ is not independent. Accordingly, we cannot apply standard concentration inequality to bound equation 35. A standard approach in learning theory is to first bound equation 35 by E x,y ∥ W S f θ ( x ) -y ∥ -1 n ∑ n i =1 ∥ W S f θ (¯ x i ) -¯ y i ∥ ≤ sup W ∈W E x,y ∥ Wf θ ( x ) -y ∥ -1 n ∑ n i =1 ∥ Wf θ (¯ x i ) -¯ y i ∥ for some hypothesis space W (that is independent of S ) and realize that the right-hand side now contains the collection of independent random variables ∥ Wf θ (¯ x 1 ) -¯ y 1 ∥ , . . . , ∥ Wf θ ( n n ) -¯ y n ∥ , for which we can utilize standard concentration inequalities. This reasoning leads to the Rademacher complexity of the hypothesis space W . However, the complexity of the hypothesis space W can be very large, resulting in a loose bound. In this proof, we show that we can avoid the dependency on hypothesis space W by using a very different approach with conditional expectations to take care the dependent random variables ∥ W S f θ (¯ x 1 ) -¯ y 1 ∥ , . . . , ∥ W S f θ ( n n ) -¯ y n ∥ . Intuitively, we utilize the fact that for these dependent random variables, there is a structure of conditional independence, conditioned on each y ∈ Y .

We first write the expected loss as the sum of the conditional expected loss:

where X y is the random variable for the conditional with Y = y . Using this, we decompose equation 35 into two terms:

where

The first term in the right-hand side of equation 36 is further simplified by using

as

̸

where ˜ Y = { y ∈ Y : |I y | = 0 } . Substituting these into equation equation 36 yields

Importantly, while ∥ W S f θ (¯ x 1 ) -¯ y 1 ∥ , . . . , ∥ W S f θ (¯ x n ) -¯ y n ∥ on the right-hand side of equation 37 are dependent random variables, ∥ W S f θ (¯ x 1 ) -y ∥ , . . . , ∥ W S f θ (¯ x n ) -y ∥ are independent random variables since W S and ¯ x i are independent and y is fixed here. Thus, by using Hoeffding's inequality (Lemma G.1), and taking union bounds over y ∈ ˜ Y , we have that with probability at least 1 -δ , the following holds for all y ∈ ˜ Y :

This implies that with probability at least 1 -δ ,

Substituting this bound into equation 37, we have that with probability at least 1 -δ ,

where

Moreover, for the second term on the right-hand side of equation 38, by using Lemma 1 of [37], we have that with probability at least 1 -δ ,

where p ( y ) = P ( Y = y ) . Substituting this bound into equation 38 with the union bound, we have that with probability at least 1 -δ ,

Combining equation 34 and equation 39 implies that with probability at least 1 -δ ,

We will now analyze the term 1 n ∑ n i =1 ∥ φ ( x i ) ∥ + 1 n ∑ n i =1 ∥ φ (¯ x i ) ∥ on the right-hand side of equation 40. Since W ∗ = W ¯ S ,

By using Hoeffding's inequality (Lemma G.1), we have that for any δ > 0 , with probability at least 1 -δ ,

Moreover, by using [50, Theorem 3.1] with the loss function x + ↦→ ∥ g ∗ ( x + ) -Wf ( x + ) ∥ (i.e., Lemma G.2), we have that for any δ > 0 , with probability at least 1 -δ ,

where ˜ R m ( W ◦ F ) = 1 √ m E ¯ S,ξ [sup W ∈W ,f ∈F ∑ m i =1 ξ i ∥ g ∗ ( x + i ) -Wf ( x + i ) ∥ ] is the normalized Rademacher complexity of the set { x + ↦→∥ g ∗ ( x + ) -Wf ( x + ) ∥ : W ∈ W , f ∈ F} (it is normalized such that ˜ R m ( F ) = O (1) as m → ∞ for typical choices of F ), and ξ 1 , . . . , ξ m are independent uniform random variables taking values in {-1 , 1 } . Takinng union bounds, we have that for any δ > 0 , with probability at least 1 -δ ,

Similarly, for any δ > 0 , with probability at least 1 -δ ,

Thus, by taking union bounds, we have that for any δ > 0 , with probability at least 1 -δ ,

To analyze the first term on the right-hand side of equation 42, recall that

Here, since Wf θ ( x + i ) ∈ R r , we have that

where I r ∈ R r × r is the identity matrix, and [ f θ ( x + i ) ⊤ ⊗ I r ] ∈ R r × dr is the Kronecker product of the two matrices, and vec[ W ] ∈ R dr is the vectorization of the matrix W ∈ R r × d . Thus, by defining A i = [ f θ ( x + i ) ⊤ ⊗ I r ] ∈ R r × dr and using the notation of w = vec[ W ] and its inverse W = vec -1 [ w ] (i.e., the inverse of the vectorization from R r × d to R dr with a fixed ordering), we can rewrite equation 43 by

with g i = g ∗ ( x + i ) ∈ R r . Since the function w ↦→ ∑ m i =1 ∥ g i -A i w ∥ 2 is convex, a necessary and sufficient condition of the minimizer of this function is obtained by

This implies that

In other words,

Thus,

where ( A ⊤ A ) † is the Moore-Penrose inverse of the matrix A ⊤ A and Null( A ) is the null space of the matrix A . Thus, the minimum norm solution is obtained by

Thus, by using this W ¯ S , we have that

where the inequality follows from the Jensen's inequality and the concavity of the square root function. Thus, we have that

By combining equation 40 and equation 44 with union bound, we have that

where ˜ W = W S -W ∗ and P A = I -A ( A ⊤ A ) † A ⊤ .

We will now analyze the second term on the right-hand side of equation 45:

where ∥ ˜ W ∥ 2 is the spectral norm of ˜ W . Since ¯ x i shares the same label with x i as ¯ x i ∼ D y i (and x i ∼ D y i ), and because f θ is trained with the unlabeled data ¯ S , using Hoeffding's inequality (Lemma G.1) implies that with probability at least 1 -δ ,

Moreover, by using [50, Theorem 3.1] with the loss function ( x, ¯ x ) ↦→∥ f θ (¯ x ) -f θ ( x ) ∥ (i.e., Lemma G.2), we have that with probability at least 1 -δ ,

where ˜ R m ( F ) = 1 √ m E ¯ S,ξ [sup f ∈F ∑ m i =1 ξ i ∥ f ( x + i ) -f ( x ++ i ) ∥ ] is the normalized Rademacher complexity of the set { ( x + , x ++ ) ↦→ ∥ f ( x + ) -f ( x ++ ) ∥ : f ∈ F} (it is normalized such that ˜ R m ( F ) = O (1) as m → ∞ for typical choices of F ), and ξ 1 , . . . , ξ m are independent uniform random variables taking values in {-1 , 1 } . Thus, taking union bound, we have that for any δ > 0 , with probability at least 1 -δ ,

By combining equation 45 and equation 49 using the union bound, we have that with probability at least 1 -δ ,

where

Define Z ¯ S = [ f ( x + 1 ) , . . . , f ( x + m )] ∈ R d × m . Then, we have A = [ Z ¯ S ⊤ ⊗ I r ] . Thus, P A = I -[ Z ¯ S ⊤ ⊗ I r ][ Z ¯ S Z ¯ S ⊤ ⊗ I r ] † [ Z ¯ S ⊗ I r ] = I -[ Z ¯ S ⊤ ( Z ¯ S Z ¯ S ⊤ ) † Z ¯ S ⊗ I r ] = [ P Z ¯ S ⊗ I r ] where P Z ¯ S = I m -Z ¯ S ⊤ ( Z ¯ S Z ¯ S ⊤ ) † Z ¯ S ∈ R m × m . By defining Y ¯ S = [ g ∗ ( x + 1 ) , . . . , g ∗ ( x + m )] ⊤ ∈ R m × r , since g = vec[ Y ⊤ ¯ S ] ,

On the other hand, recall that W S is the minimum norm solution as

By solving this, we have

where Z S = [ f ( x 1 ) , . . . , f ( x n )] ∈ R d × n and Y S = [ y 1 , . . . , y n ] ⊤ ∈ R n × r . Then,

Thus,

where P Z S = I -Z S ⊤ ( Z S Z S ⊤ ) † Z S .

By combining equation 50-equation 52 and using 1 ≤ √ 2 , we have that with probability at least 1 -δ ,

where

Now,

Appendix J Information Optimization and the VICReg Objective

Assumption 1 . The eigenvalues of Σ( x j ) are in some range a ≤ λ (Σ( x j )) ≤ b

.

Assumption 2 . The differences between the means of the Gaussians are bounded

Lemma J.1. The maximum eigenvalue of each µ ( X j ) µ ( X j ) T is at most M .

Proof. The term µ ( X j ) µ ( X j ) T is an outer product of the mean vector µ ( X j ) , which is a symmetric matrix. The eigenvalues of a symmetric matrix are equal to the squares of the singular values of the original matrix. Since the singular values of a vector are equal to its absolute values, the maximum eigenvalue of µ ( X j ) µ ( X j ) T is equal to the square of the maximum absolute value of µ ( X j ) . By the second assumption, this is at most M .

Lemma J.2. The maximum eigenvalue of -µ Z µ T Z is non-positive and its absolute value is at most M .

Proof. The term -µ Z µ T Z is a negative outer product of the overall mean vector µ Z , which is a symmetric matrix. Its eigenvalues are non-positive and equal to the negative squares of the singular values of µ Z . Since the singular values of a vector are equal to its absolute values, the absolute value of the maximum eigenvalue of -µ Z µ T Z is equal to the square of the maximum absolute value of µ Z , which is also bounded by M by the second assumption.

Lemma J.3. The sum of the eigenvalues of Σ Z is bounded

Proof. Given a Gaussian mixture model where each component Z | x j has mean µ ( X j ) and covariance matrix Σ( x j ) , the mixture can be written as:

where p j are the mixing coefficients. The covariance matrix of the mixture, Σ Z , is then given by:

where µ Z is the mean of the mixture distribution.

By Lemmas 1.1, 1.2, and assumptions 1 and 2, the maximum eigenvalues of (Σ( x j ) , µ ( X j ) µ ( X j ) T and µ Z µ T Z . are at most b , M , and M , respectively. Therefore, by Weyl's inequality for the sum of two symmetric matrices, the maximum eigenvalue of Σ Z is at most b + M .

It means that we can bound the sum of the eigenvalues of Σ Z with

Lemma J.4. Let Σ Z be a positive semidefinite matrix of size N × N . Consider the optimization problem given by:

where λ i (Σ Z ) denotes the i -th eigenvalue of Σ Z and c is a constant. The solution to this problem is a diagonal matrix with equal diagonal elements.

such that:

Proof. The determinant of a matrix is the product of its eigenvalues, so the objective function log det(Σ Z ) can be rewritten as ∑ N i =1 log( λ i (Σ Z )) . Our problem is then to maximize this sum under the constraints that the sum of the eigenvalues does not exceed c and that Σ Z is positive semi-definite.

Applying Jensen's inequality to the concave function log( x ) with weights 1 /N , we find that 1 N ∑ N i =1 log( λ i (Σ Z )) ≤ log( 1 N ∑ N i =1 λ i (Σ Z )) . Equality holds if and only if all λ i (Σ Z ) are equal.

Setting λ i (Σ Z ) = x for all i , we see that the constraint ∑ N i =1 λ i (Σ Z ) ≤ c becomes Nx ≤ c , leading to the optimal eigenvalue x = c/N under the constraint.

Since Σ Z is positive semi-definite, it can be diagonalized via an orthogonal transformation without changing the sum of its eigenvalues or its determinant. Therefore, the solution to the problem is a diagonal matrix with all diagonal entries equal to c/N .

This completes the proof.

Theorem J.5. Given a Gaussian mixture model where each component Z | X i has covariance matrix Σ( X i ) , under the assumptions above, the solution to the optimization problem

is a diagonal matrix Σ Z with equal diagonal elements.

Proof. The objective function can be decomposed as follows:

where K is the number of components in the Gaussian mixture model.

In this optimization problem, we are optimizing over Σ Z . The term ∑ i log | Σ( X i ) | is constant with respect to Σ Z , therefore we can focus on maximizing K log | Σ Z | .

As the determinant of a matrix is the product of its eigenvalues, log | Σ Z | is the sum of the logs of the eigenvalues of Σ Z . Thus, maximizing log | Σ Z | corresponds to maximizing the sum of the logarithms of the eigenvalues of Σ Z .

According to Lemma 1.4, when we have a constraint on the sum of the eigenvalues, the solution to the problem of maximizing the sum of the logarithms of the eigenvalues of a positive semidefinite matrix Σ Z is a diagonal matrix with equal diagonal elements.

From Lemma 1.3, we know that the sum of the eigenvalues of Σ Z is bounded by ( b + M ) × K . Therefore, when we maximize K log | Σ Z | under these constraints, the solution will be a diagonal matrix with equal diagonal elements. This completes the proof of the theorem.

Appendix K Entropy Comparison - Experimental Details

We use ResNet-18 [32] as our backbone. Each model is trained with 512 batch size for 800 epochs. We use the SGD optimizer with a momentum of 0.9 and a weight decay of 1 e -4 . The initial learning rate is 0 . 5 . This learning rate follows the cosine decay with a linear warmup schedule. For augmentation, two augmented versions of each input image are generated. During this process, each image is cropped with random size, and resized to the original resolution, followed by random applications of horizontal mirroring, color jittering, grayscale conversion, Gaussian blurring and solarization. For the entropy estimation, we use the same method as in [38], which uses a lower bound of the entropy using the distances of the representations under the assumption of a mixture of the Gaussians around the representations with constant variance.

10 1

10 2

10 3

10 4

Number of SSL Samples (m)

1.00

1.1

1.2

1.3

1.4

1.5

Generalization Bound

SSL Bound

Supervised Bound

0.2

0.4

0.6

0.8

1.0

1.2

Generalization Gap in Loss

0.0

0.2

0.4

0.6

0.8

1.0

VICReg Bound

ResNet18

ResNet30

ResNet56

ResNet110

Polynomial Fit

where p i and p j are the distributions of the representation of the i-th and j-th examples, and D ( p i || p j ) represents the divergence between these distributions. Also, c i indicates the weight of component i ( c i ≥ 0 , ∑ i c i = 1 ), and C is a discrete random variable where P ( C = i ) = c i .

Appendix L Reproducibility Statement

All of the methods in our study are based on existing methods and their open-source implementations. We provide a detailed implementation setup for both the pre-training and downstream experiments. Additionally, we are committed to ensuring reproducibility and open science. Therefore, after publication, we will provide pretrained checkpoints and make the code openly available on a public repository. This will enable other researchers to reproduce our results and build upon our work.

Appendix M Expriemtns on the generalization bound

In this section we have more empirical results on the connection between our generalization to the generalization gap.

Appendix N Limitations of assuming inherent network randomness

First, we provide a brief overview of the challenges associated with estimating mutual information (MI) in DNNs. This context will set the stage for a detailed discussion on the advantages of our method

Estimating MI in DNNs poses various practical challenges. For example, deterministic DNNs with strict monotonic nonlinearities yield infinite or constant MI, complicating optimization [4] or making optimization ill-posed. One solution is to assume an unknown distribution over the representation and estimating MI directly [9, 57]. However, this is challenging theoretically and requires large sample sizes [55, 47] and is prone to significant estimation errors [70].

To address these issues, prior works have proposed various assumptions about randomness in neural networks. For example, [1] posits noise in DNNs' parameters, while [1] assumes noise in the representations. These assumptions are valid if they outperform deterministic DNNs or yield similar representations to those trained in practice.

Our empirical evidence suggests that noise in the input has much better performance and representations similar to the deterministic network compared to noise in the network itself. We compared our model, which introduces noise at the input level, with models that add noise to network representations [26], using CIFAR-100 and Tiny-ImageNet and VICReg with ConvNeXt. The results,

tabulated below, indicate that the noise injected into the representation has much worse performance degradation compared to our input noise model.

Our approach, focusing on input noise, does not necessitate explicit noise injection during trainingunlike methods assuming inherent network noise.

Lastly, we validate that the assumption of input noise leads to better alignment with deterministic networks. Following [14], we used Centered Kernel Alignment (CKA) [18] to compare deterministic DNNs with our noise model (input noise) to DNNs with noise in the representations [26]. The results, also tabulated below, confirm that assuming input noise aligns more closely with deterministic DNN representations than assuming noise in the DNN parameters.

In summary, our input noise model more effectively captures deterministic DNN behavior and is more robust than assuming inherent network randomness.

Appendix O Boader Impact

In the landscape of machine learning, SSL algorithms have found a broad array of practical applications, ranging from image recognition to natural language processing. With the proliferation of data in recent years, their utility has grown exponentially, often serving as key components in numerous real-world use cases. This paper introduces a new family of SSL algorithms that have demonstrated superior performance. Given the widespread use of SSL algorithms, the advancements proposed in this paper have the potential to considerably enhance the effectiveness of systems that rely on these techniques. As a result, the impact of these improved algorithms could be far-reaching, potentially influencing a multitude of applications and sectors where machine learning is currently employed. With this potential for value also comes the potential that proposed methods make false promises that will not benefit real-world practitioners and may, in fact, cause harm when deployed in sensitive applications. For this reason, we release our numerical results and implementation details for the sake of transparency and reproducibility. As with all new state-of-the-art methods, our improvements may also improve models used for malicious intentions.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an information-theoretic analysis of the self supervised learning algorithm VICReg. The authors first introduce an approach to study neural networks using an information-theoretic perspective by shifting the stochasticity from the networks to the input data. Therefore they derive an information-theoretic objective (equation 7) that is linked (through a series of approximations) to the VICReg objective. The assumptions of the proposed theory are validated empirically using ResNet-18 across various datasets.\n\nThe paper also proposes a plug-in entropy estimator that can be merged with the objective term of other SSL methods for entropy optimization. The authors shows empirically that doing so with VICReg and training a ResNet-18 architecture on CIFAR-10 dataset leads to an improvement on the test accuracy compared to the original VICReg method. \n\nLastly the paper derives a generalization bound for VICReg that provides a probabilistic guarantee on the downstream generalization ability of this SSL method.\n\nSTRENGTHS:\n- The perspective offered by the new analysis is interesting and could have practical implications.\n\nWEAKNESSES:\n- I find some of the explanation (example Section 4) is not immediately clear. There are some statements that should be verified and could be clarified in the manuscript in order to make the paper more accessible (see questions below on clarifications).\n- I find the empirical analysis of the practical implications weak: experiments are limited to CIFAR 10 on ResNet18. I believe there could be an opportunity to strengthen the paper by providing a more thorough evaluation on larger architectures and more challenging datasets (see questions below on experimental evaluation for some suggestions).\n- The claim that the proposed plug-in entropy estimator could help other SSL methods (beyond VicReg) is not supported by empirical evaluation. \n- The paper contains some significant errors: punctuation missing, use of different styles to refer to the appendix, missing figure citations, a few grammar errors... See Minor Notes below for a non-exhaustive list.\n\n**Minor Notes**\n- Line 156: a full stop seems to be missing “of prototypes N As”.\n- Line 227: missing full stop at the end of the page.\n- Line 242: missing closed parenthesis.\n- Notation is not consistent, Appendix A, B, C... vs appendix 1,2,3....\n- Figure 2 is not referenced in the text.\n- Table 1 missing closed parenthesis on the last line.\n- Line 314 Grammar: “label classes to go infinity”.\n- Line 347: Grammar: “By transferring the required stochasticity required “ .\n\nQUESTIONS:\nAs mentioned above the biggest weaknesses are related to some theoretical parts that are not clear, and the experimental evaluation which is not sufficiently convincing. Here are some opportunities to make the paper stronger (and potentially the rating higher).\n\n**Clarifications**\nProviding more information here could help improve the Soundness and Presentation rating.\n\n- In Equation 4, could the authors clarify how the inequality holds? Even in Appendix A this first step is taken for granted but it might be helpful to explain it, at least in the Appendix.\n- In Section 4.1 it is not clear to me what happened to the invariance term when going from equation 7 to equation 8. That term seems to have been dropped. Could the authors clarify this?\n- “We sampled 512 Gaussian samples for each image” I suppose this means that given 1 image noise is added by sampling 512 different noise that is added to the image? Could this be clarified?\n- “However, these methods (reviewer: these methods being [61, 65]) assume that randomness comes from the DNN, contrary to common practice.” Could the author clarify where in 61 and 65 this is assumed?\n- “the mutual information between the input and the DNN representation is infinite”. This statement is presented as always true but I do not understand how this would be always guaranteed. None of the papers cited in this section ([8,17,28,37,52,61,…]) seem to support this statement. Do the authors refer to the difficulties of computing MI in continuous domain or could the authors clarify this statement? \n- It seems that the loss function of VICReg reported in this paper has some inaccuracies. For example, there is a \\gamma parameter that appears twice (in the regularization and the invariance term) but odes not match the loss in [5]. Since in [5] \\gamma was fixed to 1 it might not make much difference but it would be better to report the correct equation. \n- In section 5.1 “The results showed that entropy decreased for all methods during training, with SimCLR having the lowest and VICReg the highest entropy.” From Figure 2 it seems the entropy is increasing not decreasing. Could the authors clarify?\n- “The entire input–output mapping then becomes a CPA spline “. I wonder if this statement is correct. According to [4] DNNs can be seen as a composition of max-affine spline operators, which I am not sure is the same as stated in the paper under review. Could the authors provide more information?\n- How many trial were used to compute the error bars in Table 1?\n- “the invariance term of current SSL methods with plug-in methods for optimizing the entropy.” Is it merged with the invariance or with the regularizer term?\n\n**Experimental Evaluation**\n- The downstream task results are only presented on CIFAR-10 and ResNet-18. Unfortunately, in my experience not everything that works on CIFAR-10 (and on a small architecture) transfers to more realistic datasets (ImageNet would be ideal, TinyImageNet or CIFAR-100 would at least be better than CIFAR-10). The paper would be stronger with these experiments. \n- Since the authors claim that the plug-in estimators could be beneficial for “other SSL methods“ it would be great to show how it affects other SSL methods beyond VICReg.\n- Other works, e.g., [A], have found that entropy maximization also leads to other beneficial aspects beyond downstream task performance. Specifically, it seems to lead to easier and more robust training with respect to smaller batch size or different EMA parameters (where EMA is used). A similar analysis would also improve and strengthen the practical implication of this work.\n- Section 5.1 seems only marginally related to the rest of the paper. However, it would be interesting (and would connect more tightly this section to the rest of the paper) to show in Figure 2 the evolution of the entropy of VICReg (and other algorithms, see points below) with the proposed entropy plug-in estimator to show that indeed it leads to higher entropy. \n\n[A] “The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning” Galvez et al. ICML2023\n\nLIMITATIONS:\nLimitations are not clearly described in the main paper. They are present in the appendix but I believe they would be more useful in the main part of the manuscript. Additionally in the limitation section of the appendix the authors state “Furthermore, our study hinges on certain assumptions that, despite rigorous validation efforts, may not hold universally” which is fair but it would be even better and more precise to clarify which are such assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the SSL methods from an information-theoretic perspective. Unlike previous theoretical analyses, the paper focuses on deterministic networks more suitable for real-world scenarios. Assuming a data distribution as a mixture of Gaussians (MoG) and a deep neural network (DNN) as a conditional probability approximation (CPA), the study demonstrates the equivalence between the variational lower bound of mutual information (MI) and the learning objective of VICReg (with some approximations). The theoretical and experimental differences between VICReg and SimCLR are presented, highlighting the superiority of VICReg. The study also shows that using a better entropy estimator can enhance the performance of VICReg. Generalization bounds are provided, and their practical implications are explained.\n\nSTRENGTHS:\n- The paper is well-organized and well-written.\n- To the best of my knowledge, the theoretical analysis in this paper is novel.\n- The authors provide empirical validation for the assumptions used in their paper and suggest possible remedies for some special cases. \n- This paper provides a clear intuition behind their contribution beyond the theoretical analysis.\n\nWEAKNESSES:\n- There is a lack of empirical evidence demonstrating the limitations of assuming randomness from DNNs.\n- In Figure 2, a comparison is made with BYOL, and in Table 1, a comparison is made with Barlow Twins. Why?\n- Additionally, SimCLR, BYOL, and Barlow Twins are relatively classic (and basic) methods. Why weren't they compared with more recent approaches?\n- ResNet-18 + CIFAR-10 is a very basic setup in the vision domain. Would it be possible to try ResNet-50 + Tiny ImageNet at least? How about exploring ViT as well?\n- (Minor) There are typos in the submitted version: L130 (. .), L156 (Missing .), L199 (appendix .1.), L227 (Missing ) ).\n\nQUESTIONS:\nThe questions are included in the Weakness section.\n\nLIMITATIONS:\nThere are no explicit discussions on limitations in the paper. It would be recommended to highlight any limitations arising from the assumptions or approximations made in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents two theoretical results on the VICReg self-supervised learning technique. Taking an information theoretic perspective, it shows that the VICReg objective can be seen as maximizing a lower bound on the mutual information between Z and X' where X' is an augmented version of data point X and Z is the encoded X. Second result presents a generalization bound relating the VICReg objective to performance on a downstream task (where the augmented versions of a data point share the same label).\n\nIn order to show the first result, the authors make several assumptions. First to be able to treat a deterministic network probabilistically, they assume the data distribution is a mixture of Gaussians with non-overlapping Gaussians centered at each data point (they confirm that this is a valid assumption for the datasets they look at). Building on earlier work formalizing neural networks as affine spline operator, this allows them to show that the output distribution of the network q(z|x) is Gaussian. Using this fact, they derive a lower bound on mutual information I(Z;X') which closely resembles the VICReg objective. This lower bound contains an entropy term H(Z) and an invariance term (the same with VICReg objective). Then they show that the variance-covariance regularization term in VICReg can be seen as an approximation of H(Z). This motivates replacing the regularization term in VICReg with alternative entropy estimators. They present results with two estimators: logdet and pairwise distance based. They find that using these alternative estimators lead to slight improvement in downstream task performance.\n\nTheir second results shows a link between the VICReg objective (hence mutual information) and downstream task performance. Here they assume the data augmentations do not change the label of a data point (wrt to the downstream task) and show that downstream task performance can be bounded from above with an expression that aligns closely with VICReg objective.\n\nSTRENGTHS:\nOverall a pretty good paper. Understanding self-supervised learning techniques better (and improving them) should be of interest to many researchers.\n\nI don't know the literature they build on really well so it is hard for me to judge originality but the results they present seem important.\nThis was perhaps intuitively obvious but showing that the regularization term in VICReg approximates entropy is a nice result, since it makes it more clear how better entropy estimators should (and does) help as they show in their empirical evaluation.\nThe second result on downstream task performance is again interesting but how this result is useful was not really clear. \nIt was also nice to see that the authors checked if the assumptions they make on non-overlapping support and Gaussian output distribution held in practice.\n\nWEAKNESSES:\nOverall I think the paper would benefit from more empirical evaluations and more clarity in certain places in the text.\n\nThe empirical results in Table 1 are not very strong. Why does using better entropy estimators lead to such small improvements? Is this because of the dataset or perhaps the size of the network? It'd be nice to see if these alternative entropy estimators lead to more significant improvements. This would confirm the value of the theoretical analysis much more strongly.\n\nThis lack of empirical evaluation is even more of a problem for the second generalization bound result. As it stands, it is not very clear why a practitioner should care about this result. Can the authors provide some empirical/practical implications of this result and perhaps explore some of these? Currently, it almost feels like this result should go into a separate paper, where it is developed more and investigated more deeply (perhaps following the idea mentioned in Line 353).\n\nOn clarity, there are a couple of sections where it is not quite clear how and why it fits with the rest of the text. For example, what is the purpose of section Entropy Estimation (line 237)? As it stands it doesn't really add much. I'd suggest either adding more detail about how SimCLR's objective and entropy estimator is different and what that means or perhaps remove this section. Currently, I'm not sure what I should take away from this section.\n\nAnother example is line 327, where authors say Theorem 3 illuminates the benefit of SSL. What does this mean more precisely? How does this illuminate the benefit of SSL? I understand that in Theorem 3 you get a scaling with m (number of unlabeled data points) instead of n (number of labeled points), and m > n usually in practice, But this doesn't mean the generalization performance of SSL will be better than supervised learning with n labeled points. To be able to make that claim, you would need to compare the bound in Theorem 3 with a similar bound for supervised learning, right? Can the authors show how their results explains the benefit of SSL empirically or with some simulations?\n\nSimilarly for section 6.3, it is not clear what is the point of this section. What are the implications of this observations? And why should we be minimizing I(Z;X) (while maximizing I(Z;X'))? This should either be expanded and explained in more detail or removed.\n\nQUESTIONS:\n- Why are there no results for Barlow Twins in Figure 2?\n- Why are there no results for BYOL in Table 1?\n- VICReg + LogDet does better but in the text the authors say VICReg + PairDist does better. Why?\n- Line 243: The results seem to show that entropy *increases* during training not decrease.\n- 118: should be q(z|x')?\n- Eqn 8. is not equal to L, right? It should be H(Z)?\n- Line 265: Looking at Table 1, LogDet achieves best results not PairDist.\n- What are the logdet and pairdist estimators? Can you write the equations for these?\n- Line 340, doesn't the term in Theorem 3 correspond to -I(Z;X')?\n\nLIMITATIONS:\nNo concerns about potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explained the effect of VICReg learning method using information theoretic perspective under some assumptions on the distribution of the input data. Based on the assumption, it connected the VICReg to mutual information minimization, and proposed new self-supervised method that performed better than existing methods on some simple tasks and networks.\n\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nThank you for the response. I will keep the score.\n\nSTRENGTHS:\n1. This paper proposed to transition the randomness from the stochastic network to the input data in order to applying information theory, which is a novel technique.\n2. It established the explanation for VICReg objective and proposed new self-supervised methods based on that.\n\nWEAKNESSES:\n1. The presentation of this paper is not good, e.g., the logic is not clear by following the title of each section and sub-section.\n2. The theory is based on assumption on input data and ReLU-type network which limits its application.\n3. The experiments are toy as self-supervised methods are more often to applied in large-scale pre-training tasks.\n\nQUESTIONS:\n1. In Theorem 1, what is the exact meaning of \"approximated\"? Could you explain which parts are ignored to derive this approximation?\n2. Could explain more on how Figure 1 verified the assumption on input data? What is the meaning of \"sampled 512 Gaussian samples for each image\"? What is \"Gaussian samples\"?\n\nLIMITATIONS:\nThis paper did not fully discuss the limitations of the proposed theory and methods. At least, two limitations should be addressed: 1) it should tell when the assumption on randomness of input data will not be satisfied; and 2) the theory is restricted to ReLU network.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}