Title: Understanding the Role of Equivariance in Self-supervised Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Self-supervised Learning

Unsupervised Learning

Representation Learning

Equivariance

Theory

Downstream Generalization
Abstract: Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features to be augmentation-aware. However, even for the simplest rotation prediction method, there is a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information-theoretic perspective to understand the generalization ability of E-SSL. In particular, we identify a critical explaining-away effect in E-SSL that creates a synergy between the equivariant and classification tasks. This synergy effect encourages models to extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light on new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. Code is available at 
 https://github.com/kaotty/Understanding-ESSL.

Abstract

Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features ( e.g., colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features to be augmentation-aware. However, even for the simplest rotation prediction method, there is a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information-theoretic perspective to understand the generalization ability of E-SSL. In particular, we identify a critical explaining-away effect in E-SSL that creates a synergy between the equivariant and classification tasks. This synergy effect encourages models to extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light on new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. Code is available at https://github.com/kaotty/Understanding-ESSL .

1 Introduction

Self-supervised learning (SSL) of data representations has made remarkable progress. Existing SSL methods can be categorized into two types: invariant SSL (I-SSL) and equivariant SSL (ESSL). The idea of I-SSL is to encourage the representation to be invariant to input augmentations ( e.g., color jittering). Contrastive learning that pulls positive samples closer and pushes negative samples apart is widely believed to be a prominent I-SSL paradigm, leading to rapid progress in recent years [13, 50, 5, 80, 46, 30, 3, 32, 33, 60, 49, 14, 37, 15, 77, 64, 81]. Nevertheless, since invariant representations lose augmentation-related information ( e.g., color information), their performance on downstream tasks can be hindered, as frequently observed in practice [47, 17, 34].

In view of these limitations of I-SSL, there has been a growing interest in revisiting E-SSL. Contrary to I-SSL, E-SSL learns representations that are sensitive to (or aware of) the applied transformation. 4 For instance, RotNet [31] is an early exemplar of E-SSL that learns discriminative features by predicting the rotation angles from randomly rotated images [43]. It has also been exploited in recent works and achieves promising improvements in conjunction with I-SSL [73, 67, 17, 18, 25, 54, 34]. Recently, E-SSL has shown potential for serving as the foundation for building visual world models [26].

Despite this intriguing progress in practice, compared to invariant SSL methods with a vast literature of theoretical analyses [58, 66, 48, 35, 68, 59, 78], there is little theoretical understanding of equivariant SSL methods. A particular difficulty lies in the understanding of the pretraining tasks, which may seem quite irrelevant to downstream classification. Taking RotNet as an example, the random rotation angle is independent of the image class, so it is unclear how rotation-equivariant representations are helpful for image classification. Generally speaking, it is unclear why, when, and how equivariant representations can generalize to downstream tasks .

In view of this situation, the primary goal of this paper is not to design a new E-SSL variant, but to revisit the basic E-SSL methods and understand their essential working mechanisms. We fulfil this goal by proposing a simple yet theoretically grounded explanation for understanding general E-SSL from an information-theoretic perspective. We show that the effectiveness of E-SSL can be understood via the 'explaining-away' effect in statistics, which implies an intriguing synergy effect between the image class C and the equivariant transformation A ( e.g., rotation) such that almost surely, they have strictly positive mutual information when given the input X , i.e., I ( C ; A | X ) > 0 that explains the effectiveness of E-SSL. This understanding also provides valuable guidelines for practical E-SSL design with three principles to pursue a large synergy effect I ( C ; A | X ) : lossy transformations, class relevance, and shortcut pruning, as been validated on practical datasets. Theoretically, we also quantitatively analyze the influence of data transformation on the synergy effect with a theory model.

Equipped with these theoretical findings, we further revisit advanced E-SSL methods in the recent literature [73, 67, 17, 18, 25, 54, 34] and find that many of these empirical successes can be well explained in our framework from two aspects: finer equivariance and multivariate equivariance. Besides, motivated by our theory, we also discover an under-explored aspect of E-SSL, model equivariance, where we show that adopting equivariant neural networks can yield strong improvements for certain E-SSL methods. These fruitful theoretical and practical merits suggest that our E-SSL theory provides a general and practically useful explanation for understanding and designing E-SSL methods that have the potential to guide future E-SSL designs.

2 Related Work

Invariant and Equivariant SSL. Without access to labels, SSL methods design various surrogate tasks that create self-supervision for representation learning. Early SSL methods, often in the form of predictive learning, learn from predicting the transformation of randomly transformed images, such as, RotNet [31], Jigsaw [51], Relative Patch Location [20]. Later, discriminating instances in the latent space with contrastive learning demonstrates prominent performance [21, 72, 52, 40, 13, 38, 56], with variants including non-contrastive methods [33], clustering methods [10-12], regularization methods [77]. However, data augmentations used in contrastive learning to avoid shortcuts often come at the cost of information lost for downstream tasks ( e.g., color for flower classification). To address this issue, there is a surge of interest in E-SSL that learns features to be sensitive to the applied transformations. Among them, Xiao et al. [73] use separate embeddings for each augmentation. Wang et al. [67] apply equivariant prediction on residual vectors between positive views. Dangovski et al. [17] combine contrastive learning and rotation prediction. Devillers and Lefort [18], Garrido et al. [24] utilize conditional predictors with augmentation parameters. Park et al. [54], Gupta et al. [34] model latent equivariant transformations explicitly.

Theory of SSL. Most existing theories of SSL methods focus on contrastive learning (CL) and its variants from different perspectives: information maximization [52, 40, 65], downstream generalization [58, 66, 48, 35, 68, 59], feature dynamics [66, 69], asymmetric designs [63, 82], feature identifiability [79], etc. But for general E-SSL methods, there is little, if any, theoretical understanding

on how they learn meaningful features for downstream tasks (in particular, image classification). Our work fills this gap by establishing a general information-theoretic framework for understanding E-SSL.

Equivariance in Deep Learning. Invariance and equivariance represent data symmetries that can be exploited during learning. There are two approaches to utilize invariance and equivariance. One is equivariant learning (to which E-SSL belongs) that uses equivariant training regularization such that features are approximately equivariant; the other is equivariant models that obey exact equivariance by design w.r.t. groups like rotation and scaling [16, 29, 8]. Equivariant models find wide applications in graph, manifold, and molecular domains [8, 29], but are rarely explored for equivariant SSL. In this work, we find that model equivariance can be particularly helpful for equivariant learning in terms of both training and generalization, which opens an interesting direction to explore on the interplay between equivariant learning and equivariant models for future research.

3 The Challenges of Understanding Equivariant SSL

Notations. Weintroduce existing SSL methods from a probabilistic perspective. Generally, we denote a random variable by a capital letter such as X , its sample space as X , and its outcome as x . We learn a representation ( Z/ Z /z x ) from the input ( X/ X /x ) through a deterministic encoder function F : X → Z . The general goal of SSL is to learn discriminative representations that are predictive of the image classes (labels) without actual access to label information. For ease of discussion, we mainly adopt the common Shannon information, where the entropy of X is H ( X ) = -E P ( X ) log P ( X ) and the mutual information between X and Y is I ( X ; Y ) = H ( X ) -H ( X | Y ) . It is also tempting to adopt V -information [75] that is analogous to Shannon's notion but aligns better with practice by taking into account computational constraints. For ease of understanding, we adopt Shannon's information in the main paper and extend the main results to V -information in Appendix C.

Equivariant SSL (E-SSL). For each raw input ¯ X sampled from the training set D , we independently draw a random augmentation A and get the augmented sample X = T ( ¯ X,A ) with a transformation mapping T : ¯ X × A → X . The general objective of Equivariant SSL (E-SSL) is to learn representations Z = F ( X ) that are sensitive to the applied transformation A . For example, RotNet [31] utilizes random four-fold rotation A = { 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ } for data augmentation, and learns feature equivariance by predicting the rotation angles from the representation Z . Therefore, E-SSL is driven by maximizing the following mutual information between the augmentation A and the representation Z :

Note again that equivariance studied in this paper, as in many E-SSL works [17, 18, 24, 34], is a relaxed notion of exact equivariance defined in a group-theoretical sense (with invertibility and compositionality) as in equivariant networks [16]. In E-SSL, equivariance generally means augmentation sensitivity, and the mutual information I ( A ; Z ) measures the degree of equivariance of Z to A .

Contrastive Learning as E-SSL. Contrary to E-SSL, I-SSL enforces features to be invariant to the applied augmentation A . CL is widely believed to be an example of invariant learning [17]. In CL, we apply two random data augmentations, A 1 , A 2 to the same input ¯ X and get two positive samples X 1 , X 2 and their representations Z 1 , Z 2 respectively. Since CL is driven by pulling Z 1 , Z 2 together, their mutual information objective is often formalized as max Z 1 = F ( ¯ X,A 1 ) ,Z 2 = F ( ¯ X,Z 2 ) I ( Z 1 ; Z 2 ) [1]. However, it is easy to observe that the constant outputs Z = const are also optimal with maximal I ( Z 1 ; Z 2 ) , suggesting that invariance alone is sufficient for SSL. In fact, contrastive learning can mitigate feature collapse with the help of pushing away from the representation of the other instances ( i.e., negative samples), making it essentially an equivariant learning task w.r.t. the instance, known as instance discrimination [21, 72]. Indeed, contrastive objectives are essentially nonparametric formulations of instance classification [72], and under similar designs, parametric instance classification achieves similar performance [9]. Non-contrastive variants with only positive samples are also shown to have inherent connection to contrastive methods in recent studies [63, 82, 23].

Equivariance is Not All You Need. A common intuition among E-SSL methods is that better downstream performance comes from better feature equivariance [18, 25, 54, 34]. Here, we begin our discussion by showing a counterexample in the following proposition. All proofs in this paper can be found in Appendix A.

0

50

100

150

200

Epoch

0.0

0.5

1.0

1.5

2.0

2.5

E-SSL Training Loss

(a) E-SSL Training Loss

0

50

100

150

200

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

Test Linear Accuracy

horizontal flip

grayscale

four-fold rotation

vertical flip

jigsaw

four-fold blur

color inversion

random encoder

(b) Test Linear Accuracy

Proposition 1 (Useless equivariance) . Assume that the original input ¯ X ∈ R d and the augmentation A ∈ R d ′ are independent, and X = [ ¯ X,A ] ∈ R d + d ′ is obtained with direct concatenation (DC). Then, there exists a simple linear encoder that has perfect equivariance to A , but yields random guessing on downstream classification.

Proposition 1 shows an extreme case when perfect equivariance is unhelpful for feature learning at all. Inspired by this finding, we further examine common image transformations for E-SSL: horizontal flip, grayscale, four-fold rotation, vertical flip, jigsaw, four-fold blur and color inversion (details in Appendix B). Prior to ours, Dangovski et al. [17] show that when merged with contrastive learning, additional equivariance to some augmentations ( e.g., four-fold rotation and vertical flip) can bring benefits while some are even harmful ( e.g., horizontal flip). It is not fully explored how much E-SSL alone depends on the chosen augmentation. Below, we study seven common transformations for E-SSL on CIFAR-10 [45] with ResNet-18: horizontal flip, four-fold rotation, grayscale, vertical flip, jigsaw, four-fold blur, and color inversion. We only apply random crops to avoid cross-interactions between different transformations. So the numbers reflect the relative strengths of different methods, instead of their optimal performance that can be attained.

Figure 1 reveals big differences between different choices of transformations: with linear probing, four-fold rotation and vertical flip perform the best and attain more than 60% accuracy, while the others do not even attain significant gains over random initialization (34%). This distinction cannot be simply understood via feature usefulness , since color information imposed by learning grayscale and color inversion is known to be important for classification [74]. Meanwhile, in Figure 1a, we find that the degree of equivariance (measured by the training loss of E-SSL) does not explain the difference either, since among ineffective ones, some with large training loss have very low equivariance ( e.g., horizontal flip), while some have very high equivariance with nearly zero equivariant loss ( e.g., grayscale). These phenomena show that equivariance alone, either strong or weak, does not have a good or bad indication of downstream performance, which motivates us to provide a more general understanding of E-SSL.

4 A Theory of Equivariant SSL

In Section 3, we have shown that feature equivariance alone does not guarantee effective downstream performance, which makes it even unclear how equivariant learning extracts useful features. To resolve these puzzles, we provide an information-theoretic analysis for E-SSL that serves as a natural explanation for the phenomena above.

4.1 Explaining E-SSL via Explaining-away

We start by establishing a causal diagram of the data generation process of E-SSL, where we assume that the original input ¯ X is generated from its class variable C (relevant to input semantics, e.g., shape), intrinsic equivariance variable ¯ A (relevant to semantics, e.g., the intrinsic orientation of an object) and style variable S (features irrelevant to semantics and targeted equivariance, e.g., color

and texture) through some unknown processes. Then, we apply an independently drawn augmentation variable A ( e.g., a random rotation angle), and get the transformed input X . Afterwards, we obtain its representation Z through a neural network.

Collider structure in E-SSL. The causal diagram shows that the class variable C and the augmentation variable A are independent. However, there exists a so-called collider structure where the augmented sample X is a common child of C and A . A well-known fact from statistics called the explaining-away effect ( a.k.a. selection bias) [55, 44] says that in a collider block, when conditioning on the collider X or its descendent like Z , the parents C and A are no longer independent. For example, the weather ( A ) and the road condition ( C ) are independent factors that can contribute to car accidents ( X ) .

However, given that an accident happens ( X is known), if we know that it rains today, it would be less likely that the road is broken, and vice versa. In this case, we say that the weather A explains away the possibility of road conditions C . The theorem below formally characterises the explain-effect effect in the E-SSL process and its information-theoretic implication. A caveat is that Lemma 1 guarantees that explaining-away happens in most, but not all cases ( e.g., Proposition 1), and we explain these exceptions in Section 4.2.

Lemma 1 (Explaining-away in E-SSL) . If the data generation process obeys the diagram in Figure 2, then almost surely, A and C are not independent given X or Z , i.e., A ̸ ⊥ ⊥ C | X and A ̸ ⊥ ⊥ C | Z . It implies that I ( A ; C | X ) > 0 and I ( A ; C | Z ) > 0 hold almost surely.

C

¯

X

S

¯

A

X

A

Z

Explaining-away helps E-SSL. In statistics, explaining-away often appears as the selection bias in observational data that misleads causal inference ( e.g., the Berkson's paradox [6]) and demands careful treatment [76, 7]. In contrast, explaining-away plays a critical positive role in E-SSL. In particular, the fact I ( A ; C | Z ) > 0 implies an important synergy effect between A and C during equivariant learning, as shown below:

Eq. (2) implies that for the same feature Z , using class information C gives a better prediction of A (lower uncertainty H ( A | Z, C ) ) than without using class features. Intuitively, given a rotated image, recognizing the object class C in the first place makes it easier to determine the rotation angle A . Driven by this synergy effect, the encoder will learn to encode class information C in the representation to assist the equivariant prediction of A . 5 We formally characterize this intuition in the following theorem.

Theorem 1 (Class features improve equivariant prediction) . Under the data generation process in Figure 2, consider an E-SSL task with input X , its class C X , and its representation Z . Assume a class representation Z C = ϕ ( C X ) that can perfectly predict the label C X ( ϕ is an invertible mapping). Then, almost surely, the combined feature ˜ Z = [ Z, Z C ] obtained by appending Z C to Z will strictly improve the equivariant prediction with larger mutual information I ( A ; ˜ Z ) > I ( A ; Z ) . Also, we have I ( C ; ˜ Z ) ≥ I ( C ; Z ) , so the classification performance improves in the meantime.

As an implication of Theorem 1, to achieve better equivariant prediction, during E-SSL, the model will try to extract more class features, which will jointly improve downstream classification. This explains why during E-SSL with rotation prediction, the classification accuracy also rises along the process, outperforming the random encoder (Figure 1b).

0

50

100

150

200

Epoch

0.4

0.6

0.8

1.0

Train Rotation ACC

rotation

rotation + cls

rotation - cls

(a) Rotation Accuracy on CIFAR-

10, ResNet18

0

50

100

150

200

Epoch

0.4

0.6

0.8

Train Rotation ACC

rotation

rotation + cls

rotation - cls

(b) Rotation Accuracy on CIFAR-

100, ResNet18

0

25

50

75

100

Epoch

0.3

0.4

0.5

0.6

0.7

Train ACC

rotation

rotation + cls

rotation - cls

(c) Rotation Accuracy on Tiny-

ImageNet-200, Resnet18

0

25

50

75

100

Epoch

0.4

0.6

0.8

1.0

Train ACC

rotation

rotation + cls

rotation - cls

0

50

100

150

200

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Training ACC

horizontal flip

horizontal flip + cls

0

50

100

150

200

Epoch

0.3

0.4

0.5

0.6

Training ACC

four-fold blur

four-fold blur + cls

(d) Rotation Accuracy on CIFAR-

10, ResNet50

(e) Horizontal Flip Accuracy on

CIFAR-10, ResNet18

(f) Four-fold Blur Accuracy on

CIFAR-10, ResNet18

Remark: Extension to V -information . In the discussion above, we mainly adopt Shannon's entropy measures for simplicity, which ignores computational and modeling constraints. Computation-aware notions like V -information [75] would align our theory better with practice. Notably, if we replace Shannon information with V -information, the analyses above still hold (see Appendix C). A nice property of V -information is that feature extraction steps can increase the information by making prediction computationally easier (which only decreases information in Shannon's notion instead), providing better justification for the benefit of representation learning. Thus, from the perspective of V -information, in ESSL, class features increase the equivariance w.r.t. A with easier computation.

Verification of synergy effects via controlled experiments. To validate the above analysis in practice, we further carry out a controlled experiment to study how class information affects the equivariant pretraining task. Specifically, taking the rotation prediction task as an example, we add or substitute a class prediction loss with an additional linear head in the pretraining objective. In the former case, we explicitly inject class information into the presentation by joint training with the classification loss; in the latter, we explicitly eliminate class information from the representation by adversarially maximizing the classification loss [22] (see Appendix B). As shown in Figure 3, we get slightly better rotation prediction accuracy when explicitly incorporating the class information, while getting worse performance (with a larger margin) when discouraging class information, which agrees well with Theorem 1. Note that there is still nontrivial training accuracy because the class is not the only factor that can explain equivariant prediction (style features S can also play a role).

4.2 Maximizing the Synergy Effect: Principles for Practical Designs of E-SSL

Our theoretical understanding above not only establishes theoretical explanations for downstream performance, but also provides principled guidelines for E-SSL design. The overall principle is to maximize the synergy I ( A ; C | X ) = H ( A | X ) -H ( A | X,C ) , which can be understood from the following aspects that explain various E-SSL behaviors that we observe in Section 3.

Principle I: 'Lossy' Transformations. First, let us look at H ( A | X ) , which determines the upper bound of the explaining-away effect. A higher H ( A | X ) means that the equivariant prediction task

is inherently harder. Revisiting Proposition 1, our theory gives a natural and rigorous explanation for why direct concatenation (DC) fails for E-SSL. Essentially, the DC output X = [ A, ¯ X ] admits a simple linear encoder such that A can be perfectly recovered from X , implying H ( A | X ) = 0 , which leads to I ( A ; C | X ) = 0 , i.e., no explaining-away effect. This implies an intriguing property of E-SSL, that in order to attain nontrivial performance on downstream tasks, the chosen transformation T must be lossy -in the sense that one cannot perfectly infer A after the transformation, i.e., H ( A | X ) > 0 . 6 Considering computational and model constraints in practical scenarios, this task should be at least hard for the chosen training configuration ( i.e., H V ( C | X ) > 0 ). Only when the transformation is hard enough, neural networks will strive to learn class information to assist its prediction. Indeed, Figure 1 shows that the transformations whose training loss decreases very quickly ( e.g., grayscale and jigsaw) indeed have relatively poor test accuracy, which further verifies our theory.

Principle II: Class Relevance. Aside from task hardness, we also need to ensure H ( A | X,C ) is low enough; i.e., extracting class information can effectively improve equivariant prediction . As a negative example, with a direct concatenation X = [ C, A ] as in Proposition 1, even if we add noise to A such that H ( A | X ) > 0 , extracting the class C is still unhelpful for predicting A . From an information-theoretic perspective, it satisfies H ( A | X ) = H ( A | X,C ) , so we always have I ( A ; C | X ) = 0 . In Figure 1, horizontal flip and four-fold blur have large training losses until the end of the training, even if we deliberately inject class features (see Figures 3e & 3f). This suggests that these equivariant tasks are intrinsically hard and class information does not contribute much to equivariant prediction. Instead, rotation prediction and vertical flip are hard at the beginning, but the uncertainty can be decreased significantly via learning class information. These transformations thus have a large synergy effect that benefits downstream performance. We conjecture that it is because these transformations are global ( i.e., changing pixel positions) instead of local changes ( i.e., only modifying pixel values) like grayscale and color inversion, so class information as global image semantics are more helpful for such tasks. Another important implication is that the transformation should be class-preserving so as to make class features helpful for the equivariant task. This rule has been verified extensively in contrastive learning [58, 62, 35, 68].

Principle III: Shortcut Pruning. Note that in the causal diagram (Figure 2), class C and style S features jointly determine the raw input ¯ X . According to our theory, style features may also explain the equivariant target A ( i.e., I ( A ; S | X ) > 0 ). Since style features are often easier for NN learning [28], they can become shortcuts for equivariant prediction such that class features are suppressed [28, 57]. Therefore, to ensure the learning of class-related

semantic features, it is important to avoid these shortcuts. One effective approach to corrupt these style features (to some extent) through aggressive data augmentation, e.g., color jitter, cropping, and blurring commonly adopted in contrastive learning, without corrupting class features a lot. Indeed, Chen et al. [13] show that the choice of data augmentations plays a vital rule in the success of contrastive learning, and Tian et al. [62] point out its goal is to prune class-irrelevant features. Here, we generalize this principle to E-SSL as well through our explaining-away framework. As shown in Table 1, the aggressive data augmentations from SimCLR also bring much better performance for E-SSL methods, bringing RotNet (82.26%) close to SimCLR (89.49%). It demonstrates that instead of merging with contrastive learning as in all recent E-SSL works [67, 17, 18, 25, 54, 34], learning from equivariance alone can potentially achieve competitive performance.

4.3 Analysis on the Influence of Transformation

The theory in Section 4.1 guarantees that E-SSL will learn class features almost surely under general conditions. Yet, without further knowledge, it is generally hard to derive more quantitative results for downstream performance. For a concrete discussion, we consider a simplified data generation process as an exemplar. Note that simplified data models are frequently adopted in the literature of self-supervised learning theory [63, 71] to gain insights for their real-world behaviors.

Setup. We consider a simple combination of the class C and the augmentation A as a weighted sum,

where λ ∈ R is the mixing coefficient. Here, we assume a balanced class setting, where C ∼ Cat( N C ) follows a uniform categorical variable over N C classes: 0 , 1 , . . . , N C -1 . Similarly, we assume that the augmentation A ∼ Cat( N A ) is an independent uniform categorical variable over N A classes: 0 , 1 , . . . , N A -1 . In this simple setting, it is easy to see that given X , when C is known, we will have a perfect knowledge of A as A = X -λC , indicating H ( A | X,C ) = 0 . Therefore, we have I ( A ; C | X ) = H ( A | X ) . In other words, transformations only influence the explaining-away effect through the uncertainty of predicting A . This is an extreme case for the ease of theoretical analysis. Nevertheless, the following theorem shows that under this setup, we can have a quantitative characterization of the optimal choice of N A and λ that sheds light on the design of E-SSL methods.

Theorem 2. The following results hold for the additive problem in Eq. (3):

Balanced Mixing is Optimal. With N C and N A held constant, I ( A ; C | X ) is maximized under balanced mixing with λ = 1 .

Large Action Space is Beneficial. With N C held constant and λ = 1 , we have a lower bound of the mutual information I ( A ; C | X ) ≥ ln N C -1 N A [ ( N C -1) ln N C -( N C -1) 2 N C ln ( N C -1) + N C -2 2 ] , which is monotonically increasing with respect to N A .

Theorem 2 has two important implications. First, it suggests that a balanced mixing of A and C gives the optimal synergy effect, since it can maximize the uncertainty of using X for predicting A alone (agreeing with Principle I). Second, it shows that a large action space ( |A| ) is preferred, making it harder to use spurious features (e.g., the boundary values of C and A ) as a shortcut to determine A (agreeing with Principles II and III). These theoretical results illustrate our analyses above and provide insights for understanding advanced designs in E-SSL methods, as elaborated below.

5 Understanding Advanced E-SSL Designs

In Section 4, we have established a theoretical understanding of basic E-SSL through the explainingaway effect. However, basic E-SSL (like rotation prediction) often fails to achieve satisfactory performance, and many advanced designs have been proposed to enhance E-SSL performance [67, 17, 18, 25, 54, 34]. In this section, we further explain how these advanced designs improve performance by enhancing the synergy effect between class information and equivariant prediction.

5.1 Fine-grained Equivariance

A conclusion from Theorem 2 is that a larger action space of the transformation A benefits the explaining-away effect by increasing the task difficulty H ( A | X ) . Guided by this principle, one way to improve E-SSL is through learning from more fine-grained equivariance variables with a larger action space ( |A| ) , which encourages models to learn diverse features and avoid feature collapse for specific augmentations. For example, four-fold rotation is a 4 -way classification task while CIFAR100 has 100 classes. When the neural networks are expressive enough such that it clusters samples with the same augmentation to (almost) the same representation (known as neural collapse [53]), the class features also degrade or vanish, which hinders downstream classification. For example, Table 1 shows that for rotation prediction, stronger augmentations suffer from less feature collapse (lower training accuracy), while enjoying better classification accuracy. Indeed, we show that the advantages of state-of-art SSL methods can be understood through this information-theoretic perspective.

Information-theoretic Understanding of Instance Discrimination. As disclosed in Section 3, contrastive learning is essentially an E-SSL task with equivariance prediction of instances. Specifically, each raw example ¯ x i serves as an instance-wise class, forming an action space I , where all augmented samples of ¯ x i belong to the class i . Therefore, the instance classification task has an action space of |I| = N , where N is the number of training dataset that is much larger than rotation prediction with |A| = 4 , making instance discrimination a harder task, especially under strong data augmentations [72, 21]. Since the instance index I is also independent of the class variable C , it is not

fully clear why it is helpful for learning class-relevant features. 7 Instead, our explaining-away theory gives a natural explanation from the instance classification perspective. In this way, our explanation of E-SSL can be regarded as a unified understanding of existing SSL variants.

Equivariance Beyond Instance. Although contrastive learning already adopts a very large action space with |I| = N , there is recent evidence showing that it can still learn shortcuts [57, 73] and lack feature diversity [70]. Therefore, it is natural to consider even finer-grained equivariance, such as learning to predict patch-level or pixel-level features [2], inputs [39], or tokenized patches [4], which comprises many variants of SSL methods, ranging from MAE [39], BERT [19], to diffusion models [41, 61]. Here, either random mask [19] or Gaussian noise [41] can be viewed as random variables (similar to the rotated angle in rotation prediction) and is independent of the class semantics, so they fit into our theory as well. Features learned from these tasks do show more diversity in practice and benefit downstream tasks requiring fine-grained semantics [39, 42]. Therefore, our theory provides a principled way to understanding the benefits of fine-grained supervision in SSL.

5.2 Multivariate Equivariance

As discussed in Section 4.2, equivariant prediction may have class-irrelevant features as shortcuts, while corrupting these features ( e.g., color) with data augmentation might affect certain downstream tasks ( e.g., flower classification that requires color information too). A more principled way that has been explored recently is through joint prediction of multiple equivariance variables [67, 17, 18, 24, 54, 34], which we refer to as multivariate equivariance. In the following theorem, we show that multivariate equivariance is provably beneficial since it monotonically increases the synergy effect between class information and equivariant prediction, as shown in the following theorem.

Theorem 3. For two transformation variables A 1 , A 2 , we will always have I ( A 1 , A 2 ; C | Z ) ≥ max { I ( A 1 ; C | Z ) , I ( A 2 ; C | Z ) } . In other words, multivariate equivariance brings strengthens the explaining-away effect, with a gain of g = max { I ( A 2 ; C | Z, A 1 ) , I ( A 1 ; C | Z, A 2 ) } .

Theorem 3 can also be easily extended to more equivariant variables. Note that the gains of multivariate equivariance I ( A 2 ; C | Z, A 1 ) reflects the amount of additional information that the class information C can explain away A 2 under the same value of A 1 ; therefore, more diverse augmentations provide a large gain in the synergy effect. Recent works on image world model show that equivariance to multiple transformation delivers better downstream performance and outperforms invariant learning [26].

5.3 Model Equivariance

Apart from the design of transformations that is the main focus of E-SSL methods, an often overlooked part is the equivariance of the backbone models, which we call model equivariance. Intriguingly, we find that equivariant networks can be very helpful for E-SSL when the transformation equivariance aligns well with model equivariance .

Setup. We compare a standard non-equivariant ResNet18 [36] and an equivariant ResNet18 (EqResNet18) w.r.t. the p 4 group (consisting of all compositions of translations and 90 -degree rotations) [16] of similar parameter sizes. The models are pretrained on CIFAR-10 and CIFAR-100 for 200 epochs

with rotation prediction, and then the learned representations are evaluated with a linear probing (LP) head for downstream classification (details in Appendix B). Note that a rotation-equivariant model does not necessarily predict rotation angles perfectly, since in E-SSL, the model only has access to the transformed input but not the ground-truth transformation.

As shown in Table 2 and 4 (see Appendix B), we find that equivariant models bring significant gains for rotation prediction by more than 20% on CIFAR-10 and CIFAR-100. Under aggressive data augmentations ( e.g., SimCLR ones), equivariant models provide better equivariant prediction of rotation with high accuracy (91.98% v.s. 83.26% on CIFAR-10 and 82.69% v.s. 68.29% on CIFAR100), which also yields better performance on downstream classification with 23.20% and 26.46% higher accuracy respectively. Even more surprisingly, with mild augmentations (no or crop&flip), both models achieve perfect rotation prediction, while equivariant models can still improve classification accuracy a lot.

Therefore, we find that under compatible equivariance, equivariant models have significant advantages for E-SSL in terms of both self-supervised pretraining (better pretraining accuracy) and downstream generalization (best classification accuracy). The following theorem justifies this point by showing that the mutual information w.r.t. the transformation A lower bounds the mutual information w.r.t. the classification C . Therefore, given the same equivariant task ( e.g., same data augmentations), features with better equivariant prediction (larger lower bound) will also have more class information.

Theorem 4. For any representation Z , its mutual information with the equivariant learning target A lower bounds its mutual information with the downstream task C as follows:

Here, a small gap I ( X ; A | C ) means a better generalization between these two tasks. Because I ( X ; A | C ) = H ( A | X,C ) is a lower bound of I ( A ; C | X ) that indicates class relevance, it further justifies our Principle II (Section 4.2) that better class relevance brings better E-SSL performance.

5.4 Strict Equivariant Objectives

Mathematically, an exact definition of equivariance requires that for each transformation a in the input space, there is a corresponding transformation T a in the representation space so that f ( a ( x )) ≈ T a f ( x ) . Common rotation prediction objectives do not satisfy this property. Other works also study the use of exact equivariant objectives. Here, we take CARE [34] as an example and compare it against rotation prediction with cross-entropy (CE) loss.

Table 3 shows that training with exact equivariant objective (CARE) leads to further improvement in test accuracy (57.32% → 64.50%), which aligns with our observation of model equivariance in Section 5.3. Altogether they suggest that enforcing exact feature equivariance (either through model architecture or feature regularization) can bring considerable benefits in downstream generalization.

6 Conclusion

In this paper, we have provided a general theoretical understanding of how learning from seemingly irrelevant equivariance (such as, random rotations, masks and instance indices) can benefit downstream generalization in self-supervised learning. Leveraging the causal structure of data generation, we have discovered the explaining-away effect in equivariant learning. Based on this finding, we have established theoretical guarantees on how E-SSL extracts class-relevant features from an informationtheoretic perspective. We also identify several key factors that influence the explaining-away. Since this work is theory-oriented to fill the gap between practice and theory by investigating how E-SSL works, we do not explore extensively for a better E-SSL design. Nevertheless, the fruitful insights developed in this work could inspire more principled designs of E-SSL methods in future research.

References

Laurence Aitchison and Stoil Krasimirov Ganev. InfoNCE is variational inference in a recognition parameterised model. Transactions on Machine Learning Research , 2024. ISSN 2835-8856. 3

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a jointembedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15619-15629, 2023. 9

Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems , 32, 2019. 1

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR , 2022. 9

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021. 1

Joseph Berkson. Limitations of the application of fourfold table analysis to hospital data. Biometrics Bulletin , 2(3):47-53, 1946. 5

Dylan Brewer and Alyssa Carlson. Addressing sample selection bias for machine learning methods. Journal of Applied Econometrics , 39(3):383-400, 2024. 5

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv: 2104.13478 , 2021. 3

Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classification for unsupervised visual feature learning. Advances in neural information processing systems , 33:15614-15624, 2020. 3

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020. 2

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021. 2

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020. 1, 2, 7

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020. 1

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021. 1

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016. 3, 9

Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging equivariance in representations. In International Conference on Learning Representations , 2021. 1, 2, 3, 4, 7, 8, 9

Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve self-supervised learning. In iclr , 2023. 2, 3, 7, 8, 9

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2019. 9

Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV , 2015. 2

Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems , 27, 2014. 2, 3, 8

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. arXiv preprint arXiv: 1505.07818 , 2015. 6

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations , 2023. 3

Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. In Proceedings of the 40th International Conference on Machine Learning , Proceedings of Machine Learning Research. PMLR, 2023. 2, 3, 9

Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. preprint arXiv:2302.10283 , 2023. 2, 3, 7, 8

Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.00504 , 2024. 2, 9

Dan Geiger, Thomas Verma, and Judea Pearl. Identifying independence in bayesian networks. Networks , 20(5):507-534, 1990. 16

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020. 7

Jan E Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review , 56(12):14605-14662, 2023. 3

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 , 2018. 1

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 , 2018. 2, 3

Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Obow: Online bag-of-visual-words generation for self-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6830-6840, 2021. 1

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020. 1, 2

Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structuring representation geometry with rotationally equivariant contrastive learning. arXiv preprint arXiv:2306.13924 , 2023. 1, 2, 3, 7, 8, 9, 10

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. In NeurIPS , 2021. 2, 7

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. 9

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020. 1

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020. 2

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16000-16009, June 2022. 9

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018. 2

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020. 9

Drew A Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. arXiv preprint arXiv:2311.17901 , 2023. 9

Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1920-1929, 2019. 2

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009. 5, 16

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 4

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 577-593. Springer, 2016. 1

Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-aware self-supervision. Advances in Neural Information Processing Systems , 34:17710-17722, 2021. 1

Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008.01064 , 2020. 2

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6707-6717, 2020. 1

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In CVPR , 2020. 1

Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision , pages 69-84. Springer, 2016. 2

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. 2

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117 (40):24652-24663, 2020. 8

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204.11371 , 2022. 2, 3, 7, 8, 9

Judea Pearl. Causality . Cambridge university press, 2009. 5

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 2

Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems , 34:4974-4986, 2021. 7, 9

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML , 2019. 2, 7, 9

Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. arXiv preprint arXiv:2202.14037 , 2022. 2

Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew G Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. Advances in Neural Information Processing Systems , 35:27706-27715, 2022. 1

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR , 2021. 9

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems , 33:6827-6839, 2020. 7

Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning , pages 10268-10278. PMLR, 2021. 2, 3, 7

Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119 , 2022. 1

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In ICLR , 2020. 2

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , 2020. 2

Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. Advances in Neural Information Processing Systems , 34:12104-12115, 2021. 2, 7, 8, 9

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR , 2022. 2, 7, 9

Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR , 2023. 2

Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv: 2205.14141 , 2022. 9

Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning , pages 11112-11122. PMLR, 2021. 7

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3733-3742, 2018. 2, 3, 8

Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.05659 , 2020. 2, 9

Yuyang Xie, Jianhong Wen, Kin Wai Lau, Yasar Abbas Ur Rehman, and Jiajun Shen. What should be equivariant in self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4111-4120, 2022. 4

Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference on Learning Representations , 2020. 3, 6, 20, 21

Alice C Yu and John Eng. One algorithm may not fit all: how selection bias affects machine learning performance. Radiographics , 40(7):1932-1937, 2020. 5

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021. 1, 2

Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML , 2023. 2

Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable contrastive learning with automatic feature importance discovery. In NeurIPS , 2023. 2

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 , 2021. 1

Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi-granular self-supervised learning framework. arXiv preprint arXiv:2203.14415 , 2022. 1

Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. In International Conference on Learning Representations , 2023. 2, 3

A Omitted Proofs

A.1 Proof of Proposition 1

Proof. It is easy to see that the linear encoder that takes the last d ′ dimension of the input does not rely on any class information while giving a perfect prediction of A , i.e., f ( X ) = X [ d +1: d + d ′ ] = A. Therefore, it gives random guess prediction on downstream classification.

A.2 Proof of Lemma 1

Proof. We begin by restating an important result in probabilistic graphical models (PGMs) for conditional independence.

Lemma 2 (Theorem 3.5 (rephrased) [44]) . For almost all distributions P that factorize over the causal diagram G , that is, for all distributions except for a set of measure zero in the space of CPD (conditional probability distributions) parameterizations, we have that I ( P ) = I ( G ) , where I ( G ) denotes the set of independencies that correspond to d -separation:

Lemma 2 shows that almost all distributions that factorize over the causal diagram G obey the d -separation rules. According to d -separation [27, 44], the collider structure in Figure 2 implies that A ̸ ⊥ ⊥ C | X and A ̸ ⊥ ⊥ C | Z . Further, recall that for any three random variables A,B,C , the mutual information I ( A ; B | C ) ≥ 0 , and I ( A ; B | C ) = 0 iff they are conditionally independent, i.e., A ⊥ B | C . Combined the facts above, we will have I ( A ; C | X ) > 0 and I ( A ; C | Z ) > 0 almost surely.

A.3 Proof of Theorem 1

Proof. Leveraging Lemma 1, we know that I ( A ; C | Z ) holds almost surely, which implies that

Since Z C is a reparameterization of C , we have H ( A | Z, C ) = H ( A | Z, Z C ) = H ( A | ˜ Z ) . Subtracting H ( A ) on both sides of Eq. (5) gives

which is equivalent to I ( A ; ˜ Z ) > I ( A ; Z ) . Besides, we also have I ( C ; ˜ Z ) = I ( C ; Z, Z C ) ≥ I ( C ; Z ) , which completes the proof.

A.4 Proof of Theorem 2

Proof. First, let us consider the first proposition of Theorem 2. We have

Since H ( λC ) = H ( C ) and H ( A ) and H ( C ) are determined by N A and N C , the goal is then transformed into minimizing H ( A + λC ) . In order to address this problem, we first draw a table below to enumerate the possible outcome of A + λC . To determine H ( A + λC ) , we can divide the elements in the table into groups, where they are distributed to the same group if and only if they have the same value. Let us assume that N A ≤ N C because the opposite condition can be solved in a similar way.

Note that the number of elements in a single group cannot be greater than N A , for at most one element in each column can be distributed to that group. Therefore, we denote x k as the number of groups that consist of k elements, k ∈ { 1 , 2 , ..., N A } . Our target is H ( A + λC ) = ∑ N A t =1 x t t N A N C ln N A N C t . For simplicity, we further denote y k = 1 N A N C ln N A N C k , then H ( A + λC ) = ∑ N A t =1 tx t y t .

Considering the total number of elements, we have

We now pay attention to the top row and the rightmost column, where the elements are bound to be in different groups. Under closer observation, we find that the element 0 forms a group alone, as does the element N A -1 + ( N C -1) λ . The element 1 is in a group made up of at most two elements, as are the elements λ , ( N A -1) + ( N C -2) λ , ( N A -2) + ( N C -1) λ . Based on similar analysis, the following relationship can be deduced:

We also have

The equality condition for this inequality is

This indicates that every secondary diagonal (from upper right to lower left) in the aforementioned table forms a group of elements, which means λ = 1 . This completes the proof of the first claim.

Let us further look at the second proposition. From the discussion above, we know that λ = 1 is the optimal value, and we want to maximize I ( A,C | X ) = H ( A ) + H ( C ) -H ( A + C ) . Let us assume

that N A ≥ N C . In order to calculate in detail, we first list the probability distribution of A + C .

Now, we have

Thus,

Note that ( N C -1) ln N C -( N C -1) 2 N C ln ( N C -1) + N C -2 2 is greater than 0 . Therefore, the lower bound of I ( A,C | X ) is monotonically non-decreasing with respect to N A . If N A is adequately large, I ( A,C | X ) approximates ln N C .

A.5 Proof of Theorem 4

Proof. The lower bound can be easily derived by taking the difference between the two quantities:

where the last line comes from the information processing inequality.

B Experiment Details

In this section, we detail the setting of each individual experiment in this work. All experiments are conducted with a single NVIDIA RTX 3090 GPU.

B.1 Experiment Details of Different Equivariant Pretraining Tasks

In this experiment, we conduct equivariant pretraining tasks based on seven different types of transformations. In order to maintain fairness and avoid cross-interactions, we only apply random crops to the raw images before we move on to these tasks. We adopt ResNet-18 as the backbone with a two-layer MLP that has a hidden dimension of 2048 and an output dimension corresponding to the pretraining tasks. Under each transformation, we train the model for 200 epochs on CIFAR-10, with batch size 512 and weight decay 10 -6 . The detailed pretraining tasks are listed as follows.

Horizontal Flip & Vertical Flip & Color Inversion & Grayscale. We randomly ( i.e., with probability 0.5) apply the specific transformation to images and require the model to predict whether or not we have really done the transformation. In these cases, the output dimension is 2.

Four-fold Rotation. We rotate the images with equal probability ( i.e., with probability 0.25) by 0°, 90°, 180°, and 270° and require the model to predict which rotation angle we have actually adopted. In this case, the output dimension is 4.

Four-fold Blur. We apply Gaussian blurs to the images using kernel sizes of 0, 5, 9, and 15, where kernel size 0 refers to not applying Gaussian blurs. We then require the model to predict the kernel size. In this case, the output dimension is 4.

Jigsaw. We divide the images into 2 × 2 patches, randomly shuffle their order, and then require the model to predict the original arrangement. In this case, the output dimension is 24, since there are 4! = 24 possible permutations for the shuffled arrangements.

During the pretraining tasks, we simultaneously train a classifier, which is a single-layer linear head and is trained without affecting the rest of the network. Apart from the seven tasks, we also conduct a baseline experiment, where we fix a random encoder and optimize the classifier alone in order to assess the effectiveness of these pretraining tasks.

B.2 Experiment Details of How Class Information Affects Equivariant Pretraining Tasks

In this experiment, our goal is to figure out how class information affects rotation prediction. Figure 4 demonstrates the outline of the model we use to conduct this experiment. We apply random crops with size 32 and horizontal flips with probability 0.5 to the raw images.

Training objectives. As for the experiment process, we first use rotation prediction as the pretraining task with a cross-entropy loss between our predicted angles and the actual angles, defined as

where N is the image number, the one-hot vector p ij refers to the true rotation angle of the i th image, and ˆ p ij refers to the prediction of the model. In the case where class information is incorporated, we simply add to the original loss function the cross-entropy between the classes predicted by the classifier and their corresponding ground truth labels, defined as

where N is the image number, C is the class number, the one-hot vector y ij refers to the true class of the i th image, and ˆ y ij refers to the prediction of the model. In other words, when class information is injected, the loss function is L rot + λ 1 L cls , where the mixing coefficient λ 1 is a hyper-parameter. Furthermore, to eliminate class information from the first setting, we minimize the classifier loss with L cls , trying to probe class information in the representation; in the meantime, we optimize the encoder to maximize the classification loss, aiming to eliminate any class information that can be found by the classifier. In particular, we adopt a joint training objective for the encoder as L rot -λ 2 L cls , where the mixing coefficient λ 2 is also a hyper-parameter. This leads to a min-max optimization between the encoder and the linear classifier. We choose λ 1 = 0 . 5 and λ 2 = 9 , under which the class features can be shown to benefit or harm rotation prediction.

In the pretraining process, we mainly use Resnet-18 as the backbone with a two-layer MLP that has a hidden dimension of 2048 and an output dimension of 4, and a single-layer linear head as

X

encoder

Z

projector

classifier

R

C

-

a classifier. For each setting, we train the model for 200 epochs on CIFAR-10 and CIFAR-100 respectively with batch size 512 and weight decay 10 -6 . Additionally, we train the model for 100 epochs on Tiny-ImageNet-200 with batch size 256 and weight decay 10 -6 . To further explore different backbones, we also use Resnet-50 and train the model for 200 epochs on CIFAR-10 with batch size 256 and weight decay 10 -6 .

Furthermore, we are interested the effects of class information on the accuracy of the pretraining tasks based on other transformations such as horizontal flips and four-fold blurs that are regarded as intrinsically hard. We simply inject class information into the representation by adding a classification loss to the original loss as well. Slightly different from the operation in rotation prediction, we apply random resized crops before conducting the pretraining tasks to avoid interfering with the prediction in these tasks. The other details and parameters are the same as those in rotation prediction.

B.3 Experiment Details in the study of model equivariance

In order to compare the performance of Resnet and EqResnet, we use rotation prediction as our pretraining task and obtain the linear probing results. We apply various augmentations to the raw images, such as no augmentation, a combination of random crop with size 32 and horizontal flip, and SimCLR augmentation with output size 32. To be more specific, SimCLR augmentation refers to a sequence of transformations, including random resized crop with size 32, horizontal flip with probability 0.5, color jitter with probability 0.8, and finally grayscale with probability 0.2.

In these experiments, we predict rotation angles with a two-layer MLP that has a hidden dimension of 2048 and an output dimension of 4, and a single-layer linear head as a classifier. For each setting, we train the model for 200 epochs on CIFAR-10 and CIFAR-100 with batch size 128 and weight decay 5 × 10 -4 . The results on CIFAR-100 are displayed in Table 4.

C V -information: Background and Extensions

In this section, we introduce V -information [75], which is a computation-aware and model-aware extension of Shannon's notation that is more suitable for modeling neural representation learning. Then, we extend our theory and show that the main results still hold under V -information.

C.1 Definitions and Properties of V -information

V -information is proposed by Xu et al. [75] under the consideration of computational constraints, which happens to be one of the drawbacks of traditional Shannon information theory. An additional merit of V -information is that it can be estimated from high-dimensional data. The formal definition of V -information is derived as follows. Denote Y as the target random variable that the model is trying to predict and X as another random variable that provides side information for the prediction of Y . Let X and Y be the sample spaces of X and Y . Define Ω := { f : X ∪ ∅ → P ( Y ) } as a set of the functions that maps X to a family of probability distributions over Y .

Definition 1 (Predictive Family) . V ⊆ Ω is called a predictive family if ∀ f ∈ V , ∀ P ∈ range ( f ) , ∃ f ′ ∈ V that satisfies ∀ x ∈ X , f ′ [ x ] = P, f ′ [ ∅ ] = P .

In other words, a predictive family is a set of probability measures that are allowed to be used under computational constraints. The existence of f ′ indicates that the agent can optionally ignore the side information.

Next, we introduce predictive conditional V -entropy and predictive V -information.

Definition 2 (Predictive Conditional V -entropy) . H V ( Y | X ) = inf f ∈V E x,y ∼ X,Y [ -log f [ x ]( y )] . Specifically, H V ( Y |∅ ) = inf f ∈V E y ∼ Y [ -log f [ ∅ ]( y )] .

Definition 3 (Predictive V -information) . I V ( X → Y ) = H V ( Y |∅ ) -H V ( Y | X ) .

Apart from the definitions, we have to highlight an important property of predictive V -information.

Lemma 3 (Xu et al. [75]) . I V ( A → B ) = 0 iff A and B are independent variables.

C.2 Extension to V -information

First, we present the V -information version of Lemma 1.

Theorem 5 (Explaining-away in E-SSL) . If the data generation process obeys the diagram in Figure 2, then almost surely, A and C is no dependent given X or Z , i.e., A ̸ ⊥ ⊥ C | X and A ̸ ⊥ ⊥ C | Z . It implies that I V ( C → A | X ) > 0 and I V ( C → A | Z ) > 0 hold almost surely.

Proof. Lemma 3 indicates that for any three random variables A,B,C , the inequality I V ( A → B | C ) ≥ 0 always holds and that I V ( A → B | C ) = 0 iff A ⊥ B | C . Based on the analysis of the collider structure in Appendix A.2, we know that A and C are not independent given either X or Z . Thus, we have I V ( C → A | X ) > 0 and I V ( C → A | Z ) > 0 almost surely.

Then, we present the V -information version of Theorem 1.

Theorem 6. Assume that the representation Z consists of two parts Z = [ Z I , Z C ] , where Z I is classirrelevant, and Z C = ϕ ( C ) is a representation of the class C with an invertible mapping ϕ . If there is a positive synegy effect I V ( C → A | Z I ) > 0 , we will have I V ( Z I → A ) < I V ( Z → A ) , showing that with class features Z C we can attain strictly better equivariant prediction. As a consequence, the optimal features of equivairant learning will contain class features.

Proof. We have the assumption I V ( C → A | Z I ) = H V ( A | Z I ) -H V ( A | C, Z I ) > 0 . Given that the function ϕ is invertible and Z C = ϕ ( C ) , we have H V ( A | C, Z I ) = H V ( A | Z C , Z I ) = H V ( A | Z ) < H V ( A | Z I ) . Subtracting H V ( A ) from both sides and rewriting the inequality, we finally obtain I V ( Z → A ) > I V ( Z I → A ) .

D Detailed Elaboration on Computing I ( A ; C | X )

The computation of I ( A ; C | X ) requires knowledge of both the pretraining label A and the class label C . When both are available, we can compute it following its decomposition I ( A ; C | X ) = H ( A | X ) -H ( A | X,C ) . Estimating entropy in high-dimensional space is generally hard, and a common approach is to leverage variational estimates for the two entropy terms H ( A | X ) and H ( A | X,C ) using neural networks.

Variational estimates of entropy . Take H ( A | X ) as an example. We can learn a NN classifier P θ ( A | X ) optimized by minimizing the cross-entropy L ( θ ) = -E P d ( X,Y ) P θ ( A | X ) . The following

inequality shows that CE loss is the variational upper bound of H ( A | X ) . When θ is minimized at the minimum with P θ ( A | X ) = P d ( A | X ) , we have L ( θ ) = H ( A | X ) as the perfect estimate. Since NNs are generally expressive approximators, we believe that the (converged) CE loss can serve as a good estimate for H ( A | X ) .

Variational estimates of I ( A ; C | X ) . Combing these two, we notice that a good estimate for I ( A ; C | X ) is the difference between the cross entropy losses of the predictor P θ ( A | X ) and P θ ( A | X,C ) , which is exactly what we studied in the verification experiment in Section 4.1. In other words, the gap of the CE losses (or similarly the accuracies) between 'rotation' and 'rotation+cls', can serve as a quantitative measure of the synergy effect I ( A ; C | X ) . Amore computationally friendly way is to select only a small subset of samples and to train it shortly. As shown in Figure 3, the gap is already exhibited at the early stage of training.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We support each point by either theory or experiment.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided a URL to our code in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The differences are often significantly large and the exact performance is not the primary concern of this work.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We obey all aspects of the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theory-oriented paper with no societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not involve such models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the authors of the models and the dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We introduce no new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not use crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have such studies.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical understanding of equivariant self-supervised learning (E-SSL) methods and their effectiveness. The authors propose an information-theoretic analysis that explains the synergy effect between class information and equivariant transformations, leading to improved downstream performance. They also identify three principles for E-SSL design: lossy transformations, class relevance, and shortcut pruning. The paper further discusses the importance of model equivariance and demonstrates its advantages in E-SSL. Overall, the contributions of this paper lie in the theoretical explanation of E-SSL and the guidance it provides for designing effective E-SSL methods.\n\nSTRENGTHS:\n1. The article is well-structured, and the arguments are clear.\n2. The author analyzes how equivariance affects self-supervised learning from an information theory perspective and proposes three principles for designing E-SSL.\n3. The theoretical validity is demonstrated through theoretical proofs and some small experiments.\n\nWEAKNESSES:\n1. The experiments are conducted on few datasets which is not very convincing. \n2. There exist some typos in the paper, such as the “xIn” in line 352.\n3. It is better for the authors to provide a design enhancement example and conduct experiments to prove the effectiveness of the example, as it only explains the success of previous E-SSL works.\n\nQUESTIONS:\n1. Is 89.49% mentioned in line 279 the ACC of the SimCLR framework during testing? Even with the enhancement of SimCLR, the task of predicting rotation only achieved a maximum effect of 59.06 in downstream tasks. These two values are not very close.\n2. In the paragraph starting from line 328, it is believed that the action space of the instance discrimination task is N, but I think it is actually a combination of many binary classification tasks with an action space of 2. So I think contrastive learning is not a special case to equivariant tasks.\n3. Principle 2 states that it is best to be category related, does it mean that category label information needs to be known during pre-training stage in SSL?\n4. Why can predictive patches like MAE be considered as an equivariant SSL task?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to reduce the gap between theory and practice for equivariant SSL, which refers here to the sub-family of discriminative SSL methods that do not enforce the invariance of representations to augmentations. The authors propose an explanation based on the \"explain-away\" (_“Explaining away” is a common pattern of reasoning in which the confirmation of one cause of an observed or believed event reduces the need to invoke alternative causes._) and conclude that this principle justifies strictly positive mutual information between the pre-task labels (e.g., the rotation angle) and the downstream task label (i.e., semantic class label) thereby explaining why the pretext and downstream tasks are not systematically misaligned.\n\nSTRENGTHS:\n- Relevance of the research question: while an intuition as to why E-SSL performs better than a random encoder might be easy to formulate (see below in weaknesses), a theoretical formulation supporting these intuitions eludes. A principled explanation would help the community in the design of better and more principles E-SSL approaches.\n\nWEAKNESSES:\nMy main concern regards the soundness of the contribution, the reasoning leading to this concern follows:\n- in this work, the authors use the \"explaining-away\" principle which relies on the proposed latent variable model shown in Figure 2 to justify E-SSL performance. Authors consider a partitioning of the latent information in $C$ and $S$, the former encapsulates semantic information, the latter the non-semantic information (should hence include the object position & orientation, textures). \n- I believe a reasonable intuition as to why E-SSL (e.g., predicting the rotation angle of an image as a pretext task and then performing semantic classification) can perform better than a random encoder - as shown by authors in Figure 1 -, is because of a selection bias in images used for training: each semantic class/object is more likely to be depicted in a specific orientation (e.g., a person standing) than in any other orientation (i.e., a person upside down). This means that the pretext and downstream tasks are not completely misaligned: being aware of the class information allows one to make a better guess about the rotation angle than a random guess. If an object is as likely to be depicted in one orientation than any other orientation than the class label is uninformative to make a guess regarding the rotation angle. The difference in results between horizontal and vertical flip in Figure 1 supports that intuition. The explanation proposed by the authors does not support that intuitive explanation of E-SSL but instead seems to propose a overly-simplified \n- The explaining-away principle justify a strictly positive mutual information between $\\bar{X}$ and $A$ which authors extend to a strictly positive mutual information between $\\bar{C}$ and $A$. It is not obvious from the get-go how authors can theoretically justify this extension. \n\nTo summarize, the two aforementioned points should be further discussed by the authors to confirm the soundness of the proposed explanation which seems to take shortcuts both from an intuitive and theoretical perspective. I would suggest discussing these points more thoroughly. \n\nMinor comments:\n- Relevance of certain results: Theorem 2 is tied to an over-simplified mixing model ($ X = A + C$) \n- Rigor:\n     - Notation: notations should be consistent throughout the paper (e.g., line 250 and 256, line 236/237 and line 240) and assumptions should be made clear (e.g., eq (3) and line 293, lambda=1 is not stated)\n     - Principle III: refers to theoretical results - \"style features may also explain the equivariant target A\" - , which have not been explicitly shown (in section 4.1 or in the appendix)\n    - Contrastive learning vs. Instance discrimination: while instance discrimination performs classification with a number of classes equivalent to the number of training instances, contrastive learning like SimCLR usually discriminates at a batch level hence I believe line 339 does not accurately describe contrastive learning but rather instance discrimination.  \n- References for claims: \"style features are often easier for NN learning\" - line 260,\n- Typos: line 352, 68\n\nQUESTIONS:\n- Can you please clarify what is meant by \"global\" vs. \"local\" transformations (principle II)?\n- Can authors rigorously justify the extension of the explaining-away principle from X to C?\n\nLIMITATIONS:\nLimitations are not explicitly discussed but do mention the use of simplified models (e.g., section 4.3) for theoretical purposes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

comments: SUMMARY:
This study contributes theoretical insights that enhance our understanding of conventional practices in SSL training. Building on these theoretical foundations and supported by experimental evidence, the study puts forward several principles for the practical implementation of equivariant self-supervised learning designs.

STRENGTHS:
This study presents theoretical findings corroborated by experimental results. Specifically, the theory enhances our understanding of the design of augmentation functions. These findings are crucial as previous designs of augmentation functions have largely relied on empirical exploration.

WEAKNESSES:
Experiments are conducted using smaller-scale models.

Typo in line 352.

QUESTIONS:
Could the principles 1,2,3 be applied to other domains beyond vision-based SSL?

LIMITATIONS:
Please refer to the question.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theoretical and empirical study of the role of invariant and equivariant representations in self-supervised learning. While a number of works have been focused on learning equivariant representations, it remains unclear whether or not equivariance is beneficial in specific tasks. By studying how applying/predicting transformations in latent space is a task that requires class information (through the explaining away effect) the authors derive insights on the behaviour of existing methods and provide directions for future work, notably that invariance may not be necessary at all to obtain good representations.\n\nSTRENGTHS:\nThe theoretical analysis confirms previous empirical findings, for example that the considered augmentations must be complex/lossy [1] as well as diverse [1] to improve how much class information is present in the representations.\n\nThe collider structure and explaining away phenomenon are an elegant way to understand the interplay between the original data and its transformations, showing that C and A cannot be considered to be independent. \n\nWhile invariance loss are commonly used to learn equivariant representations[2,3] (in conjunction with an equivariant objective), previous work had found that even with a purely predictive objective, competitive performance can be obtained[1,4]. This is in line with the conjecture line 281 \"learning from equivariance alone can achieve competitive performance\", and raises an important future line of work.\n\n\n[1]Garrido, Quentin, et al. \"Learning and leveraging world models in visual representation learning.\" arXiv preprint arXiv:2403.00504 (2024).\n\n[2] Gupta, Sharut, et al. \"Structuring representation geometry with rotationally equivariant contrastive learning.\" arXiv preprint arXiv:2306.13924 (2023).\n\n[3]Devillers, Alexandre, and Mathieu Lefort. \"Equimod: An equivariance module to improve self-supervised learning.\" arXiv preprint arXiv:2211.01244 (2022).\n\n[4] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" arXiv preprint arXiv:2302.10283 (2023).\n\nWEAKNESSES:\nThe paper seems to focus on two distinct families of methods. Methods which are augmentation aware (e.g. RotNet) and methods which are truly equivariant, which are explicitly designed to learn the transformation between transformed data in latent space, bringing more structured representations. While this distinction may not affect the general ideas and theoretical arguments, it may impact empirical results and practical considerations. It would be good to have a discussion/comparison of these distinct approaches.\n\nMost of the practical insights stemming from this work are qualitative. A conclusion such as \"knowing C is beneficial to predicting A\" is a useful general guideline, but doesn't directly translate to practical guidance. The results would be stronger with more quantitative assessments of class relevance (perhaps in another, more computationally friendly domain), i.e. estimating I(A;C|X). This should be seen more as an avenue for future work though.\n\nQUESTIONS:\nIn experiments such as figure 1, what is the training loss/method considered ?\n\nTheorem 1 puts forward an argument which seems key to the success or not of equivariant SSL (when used without an invariant objective), \"How much information about C do you need to know to predict A\". Bringing this point (and class relevance in general) forward in section 3.1 would make the motivation from figure 1 clearer.\n\nLIMITATIONS:\nLimitations are addressed in section 6, focusing on the theoretical nature of the work and how it doesn't directly translate into improved methods. It would be good to elaborate further on the limitation of the analysis, such as how different instantiations of an equivariant object may lead to different behaviors, or the difficulty of characterizing quantities such as I(A;C|X) for various transformations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}