Understanding the Role of Equivariance in Self-supervised Learning
Authors: Yifei Wang
Kaiwen Hu
Sharut Gupta
Ziyu Ye
Yisen Wang
Stefanie Jegelka
Venue: NeurIPS 2024 poster (2024)
Keywords: Self-supervised Learning
Unsupervised Learning
Representation Learning
Equivariance
Theory
Downstream Generalization
Primary area: learning_theory

Abstract: Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features to be augmentation-aware. However, even for the simplest rotation prediction method, there is a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information-theoretic perspective to understand the generalization ability of E-SSL. In particular, we identify a critical explaining-away effect in E-SSL that creates a synergy between the equivariant and classification tasks. This synergy effect encourages models to extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light on new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. Code is available at 
 https://github.com/kaotty/Understanding-ESSL.

Understanding the Role of Equivariance in Self-supervised Learning
Yifei Wang" MIT
Kaiwen Hu' Peking University
Sharut Gupta MIT
Ziyu Ye The University of Chicago
arXiv:2411.06508W [cs.LG] 10 Nov 2024
Yisen Wang Peking University
Stefanie Jegelka TUM and MIT
Abstract
Contrastive learning has been leading paradigm for self-supervised learning, but it is widely observed that comes at the price of sacrificing useful features (e.g., colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features be augmentation-awart rotation prediction method, there a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information -itmatttoo-thooetti perspective to understand the generalization ability of E-SSL. particular, we identify critical explaining-awee awwy effect in E-SSL that creates synergy between the equivariant and classification tasks. This synergy effect encourages models extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. com/kaott/etoooooooooo kaotty /Undderstanddiiii ESSL.
However, even for the simplest
Code is available at https: //ggtttttb./////////////g com
1 Introduction
Self-supervised learning (SSL) of data representations has made remarkable progress. Existing SSL methods can be categorized into two types: invariant SSL (I-SSL) and equivariant SSL (E- SSL). The idea of I-SSL is to encourage the representation to be invariant input augmentations (e.g., color jittering). Contrastive learning that pulls positive samples closer and pushes negative samples apart widely believed to be prominent I-SSL paradigm, leading to rapid progress in recent years [13, 50, 5, 80, 46, 30, 3, 32, 33, 60, 49, 14, 37, 15, 77, 64, 81]. Nevertheless, since invariant representations lose augmentation-related information (e.g., color information), their performance downstream tasks can he hindered, as frequently observed in practice [47, 17, 34].
"Equal Contribution. Correspondence: Yifei Wang <yifei_ w w@mit.edur I CIT, MCML, MDSI EECS,, CSAIL
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In view of these limitations of I-SSL, there has been growing interest in revisiting E-SSL. Contrary to I-SSL, E-SSL learns representations that are sensitive to (or aware of) the applied transformation." 4 For instance, RotNet [31] is an early exemplar of E-SSLL that learns discriminative features by predicting the rotation angles from randomly rotated images [43]. It has also been exploited in recent works and achieves promising improvements in conjunction with I-SSL. [73, 67, 17, 18, 25, 54, 34). Recently, E-SSL has shown potential for serving as the foundation for building visual world models [26].
Despite this intriguing progress in practice, compared to invariant SSL methods with a vast literature of theoretical analyses [58, 66, 48, 35, 68, 59, 78], there is little theoretical understanding of equivariant SSL methods. A particular difficulty lies in the understanding of the pretraining tasks, which may seem quite irrelevant to downstream classification. Taking RotNet as an example, the random rotation angle is independent of the image class, so it is unclear how ootatio-equivariann representations are helpful for image classification. Generally speaking, is unclear why, when, and how equivariant representations can generalize to downstream tasks.
In view of this situation, the primary goal of this paper is not to design new E-SSL variant, but to revisit the basic E-SSL methods and understand their essential working mechanisms. We fulfil this goal by proposing a simple yet theoretically grounded explanation for understanding general E-SSL from an information-theorehi perspective. We show that the effectiveness of E-SSL can be understood via the "explaining-awar away" effect in statisties, which implies an intriguing synergy effect between the image class C' and the equivariant transformation A (e.g.. rotation) such that almost surely, they have strictly positive mutual information when given the input X, i.e. I(C': A|x) > effectiveness of E-SSL.. This understanding also provides valuable guidelines for practical E-SSL. design with three principles to pursue a large synergy effect I(C; A|X): lossy transformations, class relevance, and shortcut pruning, as been validated on practical datasets. Theoretically, we also quantitatively analyze the influence of data transformation on the synergy effect with a theory model.
that explains the
Equipped with these theoretical findings, we further revisit advanced E-SSL methods in the recent literature [73, 67, 17, 18, 25, 54, 34] and find that many of these empirical successes can be well explained in our framework from two aspects: finer equivariance and multivariate equivariance. Besides, motivated by our theory, we also discover an under-explored aspect of E-SSL, model equivariance, where we show that adopting equivariant neural networks can yield strong improvements for certain F-SSIL methods. These fruitful theoretical and practical merits suggest that our E-SSI. theory provides a general and practically useful explanation for understanding and designing E-SSL methods that have the potential to guide future E-SSL designs.
2 Related Work
Invariant and Equivariant SSL. Without access to labels, SSL methods design various surrogate tasks that create self-supervision for representation learning. Early SSL methods, often in the form of predictive learning, learn from predicting the transformation of randomly transformed images, such as, RotNet [31], Jigsaw [51]. Relative Patch Location [20]. Later, discriminating instances in the latent space with contrastive learning demonstrates prominent performance [21, 72, 52, 40, 13, 38, 56), with variants including non-contrastive methods [33], clustering methods [10-12], regularization methods [77]. However, data augmentations used in contrastive learning to avoid shortcuts often come at the cost of information lost for downstream tasks (e.g., color for flower classification). To address this issue, there is a surge of interest in E-SSL that learns features to be sensitive to the applied transformations. Among them, Xiao al. [73] use separate embeddings for each augmentation. Wang et al. [67] apply equivariant prediction on residual vectors between positive views. Dangovski et al. [17] combine contrastive learning and rotation prediction. Devillers and Lefort [18], Garrido et al. [24] utilize conditional predictors with augmentation parameters. Park et al. [54]. Gupta et al. [34] model latent equivariant transformations explicitly.
Theory of SSL. Most existing theories of SSL methods focus on contrastive learning (CL.) and its variants from different perspectives: information maximization [52, 40, 65], downstream gener- alization [58. 66, 48, 35, 68, 59], feature dynamics [66, 69], asymmetric designs [63, 82], feature identifiability [79], etc. But for general E-SSL methods, there is little, if any, theoretical understanding
'A rigorous detinition of feature equivariance requires more restrictions; for example, the transformations must be invertible. In existing SSL literature, equivariance is (loasely) referred to as the property that features are aware of general input transformations (not necessarily invertible). We follow this convention in this paper.

---

on how they learn meaningful features for downstream tasks (in particular, image classification). Our work fills this gap by establishing a general information-theoree framework for understanding E-SSL.
Equivariance in Deep Learning. Invariance and equivariance represent data symmetries that can be exploited during learning. There are two approaches to utilize invariance and equivariance. One is equivariant learning (to which E-SSL belongs) that uses equivariant training regularization such that features are approximately equivariant; the other is equivariant models that obey exact equivariance by design w.r.t. groups like rotation and scaling [16, 29, 8]. Equivariant models find wide applications in graph, manifold. and molecular domains [8, 29]. but are rarely explored for equivariant SSL. In this work, we find that model equivariance can be particularly helpful for equivariant learning in terms of both training and generalization, which opens an interesting direction to explore on the interplay between equivariant learning and equivariant models for future research.
3 The Challenges of Understanding Equivariant SSL
Notations. We introduce existing SSL methods from a probabilistic perspective. Generally, we denote a random variable by a capital letter such as X, its sample space as X, and its outcome as æ. We learn a representation (2/ // from the input (X/X/2) through a deterministic encoder function F: X' Z. The general goal of SSL is to learn discriminative representations that are predictive of the image classes (labels) without actual access to label information. For ease of discussion, we mainly adopt the common Shannon information, where the entropy of X is (X) = -(()) (x) log P(X) and the mutual information between X and Y is T(X;Y) = (()) YY))) It is also tempting to adopt V-information [75] that is analogous to Shannon's 's notion but aligns better with practice by taking into account computational constraints. For ease of understanding, we adopt Shannon's information in the main paper and extend the main results to V-information in Appendix C.
Equivariant SSL (E-SSL). For each raw input X sampled from the training set D, we independently draw a random augmentation A and get the augmented sample X = T(X. A) with a transformation mapping T: X x A + X. The general objective of Equivariant SSL (E-SSL) is to learn represen- tations Z = F()) that are sensitive to the applied transformation A. For example, RotNet [31] utilizes random four-fold rotation A {00, 90%, 180 ) for data augmentation, and learns feature equivariance by predicting the rotation angles from the representation Z. Therefore, E-SSL is driven by maximizing the following mutual information between the augmentation A and the representation Z:
max [(A; 2).
Note again that equivariance studied in this paper, as in many E-SSL works [17, 18, 24, 34), is relaxed notion of exact equivariance defined in a group-theoretical sense (with invertibility and com- positionality) in equivariant networks [16]. In E-SSL, equivariance generally means augmentation sensitivity, and the mutual information I (A; 2) measures the degree of cquivariance of Z to A.
Contrastive Learning as E-SSL. Contrary to E-SSL, I-SSL enforces features to be invariant to the applied augmentation A. CL is widely believed to be an example of invariant learning [17]. In CL, we apply two random data augmentations, A1, A2 to the same input X and get two positive samples X2 and their representations Z1, Ze respectively. Since CL driven by pulling Z1, Zz together, their mutual information objective is often formalized as max 200X ((),- A1g. F(X (2)2) I(21; Z2) [1]. However, it easy to observe that the constant outputs = consl are also optimal with maximal T(Z1; suggesting that invariance alone is sufficient for SSL. In fact, contrastive learning can mitigate feature collapse with the help of pushing away from the representation of the other instances (i.e., negative samples), making it essentially known as instance discrimination [21, 72]. Indeed, contrastive objectives are essentially non- parametric formulations of instance classification [72], and under similar designs, parametric instance classification achieves similar performance [9]. con-contrastive variants with only positive samples also shown have inherent connection to contrastive methods in recent studies [63, 82, 23).
equivariant learning task w.r.f. the instance,
Equivariance is Not All You Need. common intuition among E-SSL methods is that better downstream performance comes from better feature equivariance [18, 25, 54, 34]. Here, we begin our discussion by showing a counterexample in the following proposition. All proofs this paper can be found in Appendix A.
3

---

0.6
herizontal flip
2.5
Test Linear Accuracy ay
grayscale faur-fold rotation vertical flip jigsaw
0.5
2.0
E:551 Training
1.5
0.3
four-fold blur
1.0
color inversion
0.2
encoder
0.1
0
50
100
150
200
0
50
100
150
200
Epoch
Epoch
E-SSL Training Loss
(b) Test Linear Accuracy
Comparison between different transformations for E-SSL on CIFAR-10 with ResNet- Note that different pretraining tasks may have different classes (e.g. for rotation and for horizontal flip). The baseline is random initialized encoder with 34% test accuracy under linear probing.
Figure 1:
Proposition (Useless equivariance). Assume that the original input X E gd and the augmentation A E Wd are independent, and X [X, E Rd+d' is obtained with direct concatenation (DC). Then, there exists a simple linear encoder that has perfect equivariance to A, but yields random guessing on downstream classification.
Proposition shows an extreme case when perfect equivariance unhelpful for feature learning at all. Inspired by this finding, we further examine common image transformations for E-SSL: horizontal flip. grayscale, four-fold rotation, vertical flip. jigsaw, four-fold blur and color inversion (details in Appendix B). Prior to ours, Dangovski al. [17] show that when merged with contrastive learning, additional equivariance to some augmentations (e.g., four-fold rotation and vertical flip) can bring benefits while some even harmful (e.g.. horizontal flip). It is not fully explored how much E-SSL alone depends on the chosen augmentation. Below, we study seven common transformations for E-SSL CIFAR-10 10 [45] with ResNet-18 18: horizontal flip, four-fold rotation, grayscale, vertical flip, jigsaw, four-fold blur, and color inversion. We only apply random crops to avoid cross-interactions between different transformations. So the numbers reflect the relative strengths of different methods, instead of their optimal performance that can be attained.
Figure reveals big differences between different choices of transformations: with linear probing, four-fold rotation and vertical flip perform the best and attain more than 60% accuracy, while the others do not even attain significant gains over random initialization (34%). This distinction cannot be simply understood via feature usefulness, since color information imposed by learning grayscale and color inversion is known be important for classification [74]. Meanwhile, in Figure 1a, find that the degree of equivariance (measured by the training loss of E-SSL) does not explain the difference either, since among ineffective ones, some with large training loss have very low equivariance (e.g., horizontal flip), while some have very high equivariance with nearly zero equivariant loss (e.g., grayscale). These phenomena show that equivariance alone, either strong weak, does not have a good or bad indication of downstream performance, which motivates us to provide a more general understanding of E-SSI.
4 A Theory of Equivariant SSL
In Section 3, we have shown that feature equivariance alone does not guarantee effective downstream performance, which makes it even unclear how equivariant learning extracts useful features. To resolve these puzzles, we provide an information-theeretic analysis for E-SSL that serves as a natural explanation for the phenomena above.
4.1 Explaining E-SSI. via Explaining-away
We start by establishing a causal diagram of the data generation process of E-SSL, where we assume that the original input X is generated from its class variable C' (relevant to input semantics, egg, shape), intrinsic equivariance variable A (relevant semantics, e.g., the intrinsic orientation of an object) and style variable S (features irrelevant to semantics and targeted equivariance, e.g., color

---

and texture) through some unknown processes. Then, we apply an independently drawn augmentation variable A (e.g.. a random rotation angle), and get the transformed input X. Afterwards, we obtain its representation Z through a neural network.
Collider structure in E-SSL. The causal diagram shows that the class variable C and the augmentation variable A are independent. However, there exists a so-called collider struc- ture where the augmented sample X is common child of C and A. A well-known fact from statistics called the explaining-away effect (a.ka. selection bias) [55, 44] says that in collider block, when conditioning on the collider X or its descendent like Z. the parents C and A are no longer independent. For example, the weather (A) and the road condition (C) are independent factors that can contribute car accidents (X). However, given that an accident happens (X is known), if we know that rains today, would be less likely that the road is broken, and vice versa. In this case, we say that the weather A ex- plains away the possibility of road conditions C. The theorem below formally characterises the explain-effect effect in the E-SSL process and its information thhoorettchc implication. A caveat is that Lemma guarantees that explaining- -wway happens in most, but not all cases (e.g., Propo- sition 1), and explain these exceptions in Section 4.2.
S
X
X
Z
A
A
Figure 2: The causal diagram of equivariant self- supervised learning. The observed variables are in grey. C: class; S: style; A: intrinsic equivariance variable; X: raw input; A: augmentation; X: aug- mented input; Z: representation.
Lemma 1 (Explaining away in E-SSL). If the data generation process obeys the diagram in Figure 2, then almost surely, and are not independent given X or Z, i.e., A.K. C|X and A A. C Z. It implies that [(A; CI > and [(A: Z)Z) hold almost surely.
Explaining-awai helps E-SSL. In statistics, explainin--waa often appears as the selection bias in observational data that misleads causal inference (e.g., the Berkson's paradox [6]) and demands careful treatment [76, 7]. In contrast, explaining-awe away plays critical positive role in E-SSL. In particular, the fact [AA; Z)Z) > implies an important synergy effect between and C' during equivariant learning, as shown below:
[(A; [|A = HAA)) - H(A
= H(AlZ) > H(A
(2)
Eq. (2) implies that for the same feature Z, using class information C' gives a better prediction of A (lower uncertainty H (AA (A L2 C)) than without using class features. Intuitively, given a rotated image, recognizing the object class C in the first place makes it easier to determine the rotation angle A. Driven by this synergy effect, the encoder will learn to encode class information C' in the representation to assist the equivariant prediction of A. We formally characterize this intuition in the following theorem.
Theorem (Class features improve equivariant prediction). Under the data generation process in Figure 2, consider an E-S task with input X, its class Cx, and its representation Z. Assume class representation Ze = ((Cx) that can perfectly predict the label Cx (Ø is an invertible mapping). Then, almost surely, the combined feature 7 = [2, Zc) obtained by appending Ze to Z will strictly improve the equivariant prediction with larger mutual information I(A; Z) > [(A; Z). Also, we have TCC; 7) 2 T(C; Z), so the classification performance improves in the meantime.
As an implication of Theorem to achieve better equivariant prediction, during E-SSL, the model will try to extract more class features, which will jointly improve downstream classification. This explains why during E-SSL with rotation prediction, the classification accuracy also rises along the process, outperforming the random encoder (Figure 1b).
*When taking a closer look, there could be multiple explaining-avy away effects Figure 2. For example, we have A + X + A. In practice, it could mean that knowing the intrinsic rotation of an image helps predict the rotated angle A. Also, we have A + X + C. meaning that knowing A is also helpful for predicting the class C. This is more nuanced analysis of how predicting rotated angles A could finally help predict the class C.
5

---

1.0
Trin Rofation ACC
Irain Ratation ACF
0.8
0.7
0.8
Train ACC
0.6
0.6
0.6
0.5
0.4
rotation + cls
0.4
rotation dis rotation ds
rotation dB
0.3
100 Epoch
150
200
100 Epoch
150
50
Epoch
(a) Rotation Accuracy CIFAR-(b) (b) Rotation Accuracy on CIFAR-(c) Rotation Accuracy on Tiny-
10, ResNet18
100, ResNet18
ImageNet- Resnet18
1.0
rotatian
herizontal flip herizontal
0.6
0.8
Train ACC.
0.5
0.6
Training
0.6
training
0.4
0.4
0.2
0.3
four-fold blur
four-fold blur ds
0
25
50
75
100
0.0-6
50
100
150
200
50
100
150
200
Epoch
Epoch
Epoch
(d) Rotation Accuracy on CIFAR-(c) Horizontal Flip Accuracy on (f) Four-fold Blur Accuracy on CIFAR-10, ResNet18
10, ResNet50
CIFAR-10, ResNet18
Figure 3: controlled experiment on the influence of class information on equivariant prediction. We include three methods: 1) equivariant prediction (baseline); 2) jointly minimizing equivariant and classification losses ("+cls"):; 3) minimizing the equivariant loss while adversarially maximizing the classification loss [22] (*- cls"). We study rotation prediction for (a), (b), (c) and (d), horizontal flip for (e), and four-fold blur for (f).
Remark: Extension to V-information. In the discussion above, we mainly adopt Shamnon's entropy measures for simplicity, which ignores computational and modeling constraints. Computation--ue awarre notions like V-information [75] would align theory better with practice. Notably, if we replace Shannon information with V-information, the analyses above still hold (see Appendix C). A nice property of V-information is that feature extraction steps can increase the information by making prediction computationally easier (which only decreases information in Shannon's 's notion instead), providing better justification for the benefit of representation learning. Thus, from the perspective of V-information, in ESSL, class features increase the equivariance w.r.f. A with easier computation.
Verification of synergy effects via controlled experiments. To validate the above analysis in practice, we further carry out a controlled experiment to study how class information affects the equivariant pretraining task. Specifically, taking the rotation prediction task as example, we add or substitute a class prediction loss with an additional linear head in the pretraining objective. In the former case, explicitly inject class information into the presentation by joint training with the classification loss; in the latter, we explicitly eliminate class information from the representation by adversarially maximizing the classification loss [22] (see Appendix B). As shown in Figure 3, we get slightly better rotation prediction accuracy when explicitly incorporating the class information, while getting worse performance (with larger margin) when discouraging class information, which agrees well with Theorem I. Note that there is still nontrivial training accuracy because the class is not the only factor that can explain equivariant prediction (style features can also play role).
4.2 Maximizing the Synergy Effect: Principles for Practical Designs of E-SSL
Our theoretical understanding above not only establishes theoretical explanations for downstream performance, but also provides principled guidelines for E-SSL design. The overall principle is to maximize the synergy I(A; C H (AAX) H (AA) X, C), which can be understood from the following aspects that explain various E-SSL behaviors that we observe in Section 3.
Principle I: "Lossy" Transformations. First, let us look at (AAX)) which determines the upper bound of the explaining-an effect. A higher H (A(X) means that the equivariant prediction task

---

is inherently harder. Revisiting Proposition 1, our theory gives a natural and rigorous explanation for why direct concatenation (DC) fails for E-SSL. Essentially, the DC output X = [A, X admits a simple linear encoder such that A can he perfectly recovered from X, implying {A) O. which leads to T [A; X = 0, i.e., no explaining-awer
effect. This implies an intriguing property of E-SSL, that in order to attain nontrivial performance on downstream tasks, the chosen transformation T must be lossy- - in the sense that one cannot perfectly infer A after the transformation, i.e., H (A]X) > 0. Considering computational and model constraints in practical scenarios, this task should be at least hard for the chosen training configuration (i.e., Hy > 0), Only when the transformation is hard enough, neural networks will strive to learn class information to assist its prediction. Indeed, Figure shows that the transformations whose training loss decreases very quickly (e.g., grayscale and jigsaw) indeed have relatively poor test accuracy, which further verifies theory.
Principle II: Class Relevance. Aside from task hardness, we also need to ensure H(A)X,C) is low enough; i.e., extracting class information can effectively improve equivariant prediction. As a negative example, with a direct concatenation X [C, A] as in Proposition 1, even if we add noise to A such that X)) > 0, extracting the class C' is still unhelpful for predicting A. From an information-thhorree
perspective, it satisfies H (A = H (A[X, C), so we always have [(A; C = O. In Figure I, horizontal flip and four-fold blur have large training losses until the end of the training, even if we deliberately inject class features (see Figures 3e & 3f). This suggests that these equivariant tasks are intrinsically hard and class information does not contribute much to equivariant prediction. Instead, rotation prediction and vertical flip are hard at the beginning, but the uncertainty can be decreased significantly via learning class information. These transformations thus have a large synergy effect that benefits downstream performance. We conjecture that is because these transformations modifying pixel values) like grayscale and color inversion, so class information as global image semantics are more helpful for such tasks. Another important implication is that the transformation should be class-preserving so as to make class features helpful for the equivariant task. This rule has been verified extensively contrastive learning [58, 62, 35, 68].
global (i.e., changing pixel positions) instead of local changes (iee, only
Principle III: Shortcut Pruning. Note that in the causal diagram (Figure 2), class C and style S features jointly determine the raw in- put X. According our theory, style fea- tures may also explain the equivariant target A (i.e., T|A: > 0). Since style fea- tures are often easier for NN learning [28], they can become shorteuts for equivariant prediction such that class features are suppressed [28, 57]. Therefore, to ensure the learning of class-related semantic features, it is important to avoid these shortcuts. One effective approach to corrupt these style features (to some extent) through aggressive data augmentation, e.g., color jitter, cropping, and blurring commonly adopted in contrastive learning, without corrupting class features a lot. Indeed, Chen et al. [13] show that the choice of data augmentations plays a vital rule in the success of contrastive learning, and Tian et al. [62] point out its goal is to prune class-irrelevant features. Here, we generalize this principle to E-SSL as well through our explaining- away framework. As shown in Table 1, the aggressive data augmentations from SimCLR also bring much better performance for E-SSL methods, bringing RotNet (82.26%) close to SimCLR (89.49%). demonstrates that instead of merging with contrastive learning as in all recent E-SSL works [67, 17, 18, 25, 54, 34), learning from equivariance alone can potentially achieve competitive performance.
Table I: Comparison of rotation prediction under different augmentations (CIFAR-10, 10, ResNet18).
Augmentation
Train Rot ACC
Test Cls ACC
None
99.98
56.92
Crop+flip
97.71
57.32
SimCLR [13]
83.26
59.06
4.3 Analysis on the Influence of Transformation
The theory in Section 4.1 guarantees that E-SSL will learn class features almost surely under general conditions. Yet, without further knowledge, it is generally hard to derive more quantitative results for downstream performance. For conerete discussion, we consider simplified data generation process as an exemplar. Note that simplified data models are frequently adopted in the literature of self-super learning theory [63, 71] to gain insights for their real-word real-world behaviors.
'For rotation prediction, there exist samples whose rotated angle cannot be uniquely determined, such as. frogs and airplane. Thus, rotation is also lossy transformation in this sense.
7

---

Setup. We consider a simple combination of the class C' and the augmentation A as weighted sum,
X = A AC,
(3)
where E is the mixing coefficient. Here, we assume a balanced class setting, where C ~ Cal(NN)) follows a uniform categorical variable over Nc classes: 0, o,....., NCc assume that the augmentation A ~ Cat(NA) is an independent uniform categorical variable over NA classes: 0, l...... NA 1. In this simple setting, it is casy to that given X, when C' is known, we will have a perfect knowledge
Similarly, we
A as = X XC, indicating H (A C*) = O. Therefore, we have (A; CIX) = H (. In other words, transformations only influence the explaining-away effect through the uncertainty of predicting A. This is an extreme case for the ease of theoretical analysis. Nevertheless, the following theorem shows that under this setup, we have quantitative characterization of the optimal choice of NA and À that sheds light on the design of E-SSL methods.
Theorem 2. The following results hold the additive problem in Eq. (3):
1) Balanced Mixing is Optimal. With Ne and NA held constant, I [(AC is maximized under balanced mixing with =
2) Large Action Space is Beneficial. With No held constant and X = we have lower bound of the mutual information I(A; C|X) z In In
Na [Ne - 1) NC (Ne-1)² In (Nc No- 2 which is monotonically increas- ing with respect to NA-
I
Theorem has two important implications. First, it suggests that balanced mixing of A and C gives the optimal synergy effect, since it can maximize the uncertainty of using X for predicting A alone (agreeing with Principle I). Second, it shows that a large action space ([-A) is preferred, making harder to spurious features (c.g.. the boundary values of C' and A) shorteut to determine A (agrecing with Principles II and III). These thcoretical results illustrate our analyses above and provide insights for understanding advanced designs in E-SSL methods, elaborated below.
5 Understanding Advanced E-SSL Designs
In Section 4, we have established a theoretical understanding of basic E-SSL through the explaining- away effect. However, basic E-SSL (like rotation prediction) often fails to achieve satisfactory performance, and many advanced designs have been proposed to enhance E-SSL performmfeee [67, 17, 18, 25, 54, 34]. In this section, we further explain how these advanced designs improve performance by enhancing the synergy effect between class information and equivariant prediction.
5.1 Fine-grained Equivariance
conclusion from Theorem is that larger action space of the transformation A benefits the explaining-awe awaay effect by increasing the task difficulty H (A] Guided by this principle, one way to improve E-SSL is through learning from more fine-grained equivariance variables with a larger action space (lA(), which encourages models to learn diverse features and avoid feature collapse for specific augmentations. For example, four-fold rotation is 4-way classification task while CIFAR- 100 has 100 classes. When the neural networks are expressive enough such that it clusters samples with the same augmentation to (almost) the same representation (known as neural collapse [53]), class features also degrade vanish, which hinders downstream classification. For example, Table 1 shows that for rotation prediction, stronger augmentations suffer from less feature collapse (lower training accuracy), while enjoying better classification accuracy. Indeed, show that the advantages of state-of-ar SSL methods can be understood through this
perspective.
ceeemaaiionthhoreee Understanding of Instance Discrimination. As disclosed in Section 3, contrastive learning is essentially an E-SSL task with equivariance prediction of instances. Specifi- cally, each raw example Σ; serves as an instance-wiss class, forming an action space I, where all augmented samples of F; belong to class Therefore, the instance classification task has an action space of [Z] N, where N is the number of training dataset that is much larger than rotation prediction with |A| 4, making instance discrimination a harder task, especially under strong data augmentations [72, 21J. Since the instance index I is also independent of the class variable C², it is not
8

---

Table Training rotation prediction accuracy and test linear classification accuracy under different base augmentations (CIFAR-10) 10, ResNet18).
Dataset
Augmentation
Network
Train Rotation ACC
Test Classification ACC
Gain
ResNer18
99.98
56.92
None
EqResNet18
100.00
72.32
+16.40
ResNet18
97.71
57.32
CIFAR-10 Crop&Flip
EqResNet18
99.97
82.54
+25.22
ResNet18
83.26
59.06
SimCLR
EqResNet18
91.98
82.26
+23.20
fully clear why it is helpful for learning class-relevant features." ? Instead, our explaining-awe away theory gives natural explanation from the instance classification perspective. In this way, our explanation of E-SSL can be regarded as unified understanding of existing SSL variants.
Equivariance Beyond Instance. Although contrastive learning already adopts a very large action space with [Z] = N, there is recent evidence showing that it can still learn shortcuts [57, 73] and lack feature diversity [70]. Therefore, it natural consider even finer-grained equivariance, such as learning to predict patch-level or pixel-level features [2], inputs [39]. or tokenized patches [4], which comprises many variants SSL methods, ranging from MAE [39], BERT [19], diffusion models [41, 61]. Here, either random mask [19] or Gaussian noise [41] can be viewed as random variables (similar the rotated angle in rotation prediction) and is independent of the class semantics, so they fit into our theory as well. Features learned from these tasks do show more diversity in practice and benefit downstream tasks requiring fineggrined semantics [39, 42]. Therefore, our theory provides a principled way to understanding the benefits of fine-grane fine-grained supervision in SSL.
5.2 Multivariate Equivariance
As discussed Section 4.2, equivariant prediction may have glass-nrrelevant features as shortcuts, while corrupting these features (e.g., color) with data augmentation might affect certain downstream tasks (e.g., flower classification that requires color information too). A more principled way that has been explored recently is through joint prediction of multiple equivariance variables [67, 17, 18, 24, 54, 34], which we refer as multivariate equivariance. In the following theorem, we show that multivariate equivariance is provably beneficial since it monotonically increases the synergy effect between class information and equivariant prediction, shown in the following theorem.
Theorem 3. For two transformation variables A2, we will always have I(A1, A2; CIZ) > (eax(((Ai) C (),), [A2; C' Z) In other words, multivariate equivariance brings strengthens the explaining-awar effect, with gain of max (A2; C (2 I I(A1; C|Z, A2))
Theorem 3 can also be easily extended to more equivariant variables. Note that the gains of mul- tivariate equivariance [(A;; C* ZZ A1) reflects the amount of additional information that the class information C' can explain away Ag under the same value of A1; therefore, more diverse augmen- tations provide large gain the synergy effect. Recent works on image world model show that equivariance to multiple transformation delivers better downstream performance and outperforms invariant learning [26].
5.3
Model Equivariance
Apart from the design of transformations that is the main focus of E-SSL methods, an often overlooked part is the equivariance of the backbone models, which we call model equivariance. Intriguingly, we find that equivariant networks can be very helpful for E-SSL when the transformation equivariance aligns well with model equivariance.
Setup. We compare standard non-equivariant ResNet18 [36] and an equivariant ResNet18 (EqRes- Net18) w.e.t the p4 group (consisting of all compositions of translations and 90-degree rotations) [16] of similar parameter sizes. The models are pretrained on CIFAR-10 and CIFAR-100 for 200 epochs
"Existing theories rely on strong assumptions between data augmentation (such as, the augmentation does not change image classes), which is often violated in practice. [58, 68].

---

with rotation prediction, and then the learned representations are evaluated with linear probing (LP) head for downstream classification (details in Appendix B). Note that aatation-equivariann model does not necessarily predict rotation angles perfectly, since in E-SSL, the model only has access to the transformed input but not the ground-truth transformation.
As shown in Table 2 and 4 (see Appendix B), we find that equivariant models bring significant gains for rotation prediction by more than 20% on CIFAR-10 and CIFAR-100. Under aggressive data augmentations (e.g., SimCLR ones), equivariant models provide better equivariant prediction of rotation with high accuracy (91.98% v.s. 83.26% on CIFAR-I0 and 82.69% v.s. 68. 68.29% on CIFAR- 100), which also yields better performance on downstream classification with 23.20% and 26.46% higher accuracy respectively. Even more surprisingly, with mild augmentations (no or crop&flip), both models achieve perfect rotation prediction, while equivariant models can still improve classification accuracy lot.
Therefore, we find that under compatible equivariance, equivariant models have significant advantages for E-SSL in terms of both self-supervised pretraining (better pretraining accuracy) and downstream generalization (best classification accuracy). The following theorem justifies this point by showing that the mutual information w.E.f the transformation A lower bounds the mutual information w.r.t the classification C. Therefore, given the same equivariant task (e.g,, same data augmentations), features with better equivariant prediction (larger lower bound) will also have more class information.
Theorem For any representation its mutual information with the equivariant learning target A lower bounds its mutual information with the downstream task C' as follows:
I(Z S 1(X;A)))
Here, small gap T(X; ([XxAAAC AACC) means better generalization between these two tasks. Because T(;; A|C)= = (A|X, C) lower bound of [(A; CI that indicates class relevance, it further justifies our Principle II (Section 4.2) that better class relevance brings better E-SSL performance.
5.4 Striet Equivariant Objectives
Mathematically, an exact definition of equivari- ance requires that for each transformation a in the input space, there is a corresponding trans- formation T, in the representation space so that [((())) x T. (x). Common rotation predic- tion objectives do not satisfy this property. Other of exact equivariant objectives. Here, we take CARE [34] as an ex- ample and compare it against rotation prediction with cross-entropy (CE) loss.
Table 3: Comparison of augmentation-aware and
truly equivariant methods (CIFAR-10, ResNet18)
Equivariant Loss
Train Rot ACC
Test Cls ACC
CE loss
97.71
57,32 64.50
works also study the
CARE loss
99.95
Table 3 shows that training with exact equivariant objective (CARE) leads to further improvement in test accuracy (57.32% > 64.50%) ), which aligns with our observation of model equivariance in Section 5.3. Altogether they suggest that enforcing exact feature equivariance (either through model architecture or feature regularization) can bring considerable benefits downstream generalization.
6 Conclusion
In this paper, we have provided a general theoretical understanding of how learning from seemingly irrelevant equivariance (such as, random rotations, masks and instance indices) can benefit down- stream generalization in self- self-supervised learning. Leveraging the causal structure of data generation, we have discovered the explaining-ave away effect in equivariant learning. Based on this finding, we have established theoretical guarantees on how E-SSL extracts class-relevant features from an information- theoretic perspective. We also identify several key factors that influence the explaining-away Since this work is theory-oriented to fill the gap between practice and theory by investigating how E-SSL works, we do not explore extensively for a better F-SSI. design. Nevertheless, the fruitful insights developed in this work could inspire more principled designs of E-SSL methods in future research.
10

---

Acknowledgement
This research was supported part by NSF AI Institute TILOS (NSF CCF-21 126665). NSF award 2134108, and the Alexander von Humboldt Foundation. Yisen Wang was supported by National National Natural Science Foundation of China
Key R&D Program of China (2022Z0016030)) (92370129, 62376010), and Beijing Nova Program (20230484344, 20240484642).
References
[1] Laurence Aitchison and Stoil Krasimirov Ganev. InfoNCE is variational inference in a recogni- tion parameterised model. Transactions Machine Learning Research, 2024. ISSN 2835-8856.
3
[2] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Supe-supe learning from images with a joint- embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15619-15229, 2023.
[3] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019. 1
[4] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022. 9
[5] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariacce-ovvai regular- ization for self-supervised learning. arXiv preprint arXiv:2105.049000 2021. 1
[6] Joseph Berkson. Limitations of the application of fourfold table analysis to hospital data. Biometrics Bulletin, 2(3):47-53, 1946. 5
[7] Dylan Brewer and Alyssa Carlson. Addressing sample selection bias for machine learning methods. Journal of Applied Econometrics, 39(3):383-400, 2024. 5
[8] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv: 2/04. 13478, 2021.
[9] Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance clas- sification for unsupervised visual feature learning. Advances in neural information processing systems, 33:15333:11111 15114-1546444 2020.
[10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning visual features by contrasting cluster assignments. In NeurIPS, 2020.
2
[11] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self superviedd vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020. 1, 2,
[14] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances neural information processing systems, 33:22243-2233 22255,, 2020.
[15] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.

---

[16] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMLR, 2016. 3,
[17] Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging equiv- ariance in representations. In International Conference on Learning Representations, 2021. 1,
2,3, 4, 7, 8,
[18] Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve self-supervised learning. In iclr, 2023. 2, 7, 8,9
[19] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[20] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. 2
[21] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27, 2014. 3,
[22] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. arXiv preprint arXiv: 1505.07818, 2015. 6
[23] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations, 2023. 3
[24] Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2023, 3,9
[25] Quentin Garrido, Laurent Najman, and Yann Lecun. Self- learning of split invariant equivariant representations. preprint arXiv:2302. 10283, 2023. 2, 3, 3,7,8
[26] Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.005504 2024. 2,9
[27] Dan Geiger, Thomas Verma, and Judea Pearl. Identifying independence in bayesian networks. Networks, 2065):507-534, 1990. 16
[28] Robert Geirhos, Körn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(11):665-677 2020. 7
[29] Jan E Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohisson, Christoffer Petersson, and Daniel Persson. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review, 16012):4405-1446,, 2023. 3
[30] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803 2018.
[31] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803 2018. 2,
[32] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Obow: Online bag-of-visual-worr words generation for self-supervised learning. In Proceedings of the JEEF/CVF conference on computer vision and pattern recognition, pages 6830-6840, 2021.
12

---

[33] Jean-Base Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch. Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-211
2020. 1,2
[34] Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structuring representation geometry with rotationally equivariant contrastive learning. arXiv preprint WrXiv:2306.13924 2023. 1, 2, 3. 7. 8, 9, 10
[35] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In NeurIPS, 2021. 2,7
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[37] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[38] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[39] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, June 2022. 9
[40] Devon Hjelm, Alex Fedorov, Samuel Lavoie Marchilonn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018. 2
[41] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 9
[42] Drew A Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K Lampinen, Andrew Jaegle, James McClelland, Loic Matthey, Felix Hill, and Alexander Lerehner. Soda: Bottleneck diffusion models for representation learning. arXiv preprint arXiv:2311.179011 2023. 9
[43] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1920-1929, 2019. 2
[44] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques.
MIT press, 2009. 5, 16
[45] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 4
[46] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for au- tomatic colorization. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 577-593. Springer, 2016.
[47] Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-awarr self-supervision. Advances in Neural Information 2021.
Processing Systems, 34:17710-1777
[48] Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supe learning. arXiv preprint (xXiv:0008.01066 2020.
[49] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian rep- resentations. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6707-67 17, 2020.
[50] Ishan Misra and I Laurens van der Maaten, Self-supervised learning of pretext- repre- sentations. In CVPR, 2020.
13

---

[51] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69-84. Springer, 2016. 2
[52] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 08,444348, 2018. 2
[53] Vardan Papyan, XY Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40):24652-24663, 2020. 8
[54] Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Wal- ters. Learning symmetric embeddings for equivariant world models. arXiv preprini arXiv:2204. 1137,, 2022. 2,8.7.8,9
[55] Judea Pearl. Causality. Cambridge university press, 2009.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[57] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shorteut solutions? Advances in neural information processing systems, 34:4974-49866 2021. 7,9
[58] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019. 2, 7,9
[59] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. arXiv preprint arXiv:2202.14007, 2022.
[60] Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew G Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. Advances in Neural Information Processing Systems, 35:27706 2022.
[61] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[62] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-688 2020. 7
[63] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021. 2, 3.7
[64] Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resmets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119, 2022.
[65] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In ICLR, 2020. 2
[66] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[67] Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. Advances in Neural Information Processing Systems,
2021. 2, 7, 8,
[68] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
2, 7,
14

---

[69] Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023. 2
[70] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv: 2205.14141, 2022.
[71] Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference Machine Learning, pages 11112-11122.
PMLR, 2021." 7
[72] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer
vision and pattern recognition, pages 3733-3742, 2018. 2,3, 3 8
[73] Tete Xiao, Xiaolong Wang. Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008. 05659, 2020. 2,9
[74] Yuyang Xie, Jianhong Wen, Kin Wai Lau, Yasar Abbas Ur Rehman, and Jiajun Shen. What should be equivariant in self- learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4111-4120, 2022. 4
[75] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In Infernational Conference on Learning Representations, 2020. 3, 6, 20, 21
[76] Alice C Yu and John Eng. One algorithm may not fit all: how selection bias affects machine learning performance. Radiographies, 40(7):1932 1937, 2020.
[77] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 12310-12320. PMLR, 2021. 1,2
[78] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML,, 2023." 2
[79] Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable contrastive learning with automatic feature importance discovery. In NeurlPS, 2023. 2
[80] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.0788 2021.
[81] Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs:
A multi-granular self-supervised learning framework. arXiv preprint 2rXiv:2203.14411 2022.
[82] Zhijian Zhuo Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified theoretical un- derstanding of con-contrastive learning via rank differential mechanism. In International Conference on Learning Representations, 2023. 2, 3
15

---

A Omitted Proofs
A.1 Proof of Proposition
Proof. It is easy to see that the linear encoder that takes the last d' dimension of the input does not rely on any class information while giving a perfect prediction A, i.e., f(X) X[441:d+d1 A. Therefore, it gives random guess prediction on downstream classification.
A.2 Proof of Lemma
Proof. We begin by restating an important result in probabilistic graphical models (PGMs) for conditional independence.
Lemma 2 (Theorem 3.5 (rephrased) [44]). For almost all distributions P that factorize causal diagram G. that is, for all distributions except for a set of measure zero in the space of CPD (conditional probability distributions) parameterizations, denotes the set of independencies that correspond to d-separation:
the
have that I(P) I(G), where I(G)
I(9) = {x 1 LYY Z) d-sepg (X; 2)}
Lemma' shows that almost all distributions that factorize over the causal diagram obey the d-separation rules. According to d-separation [27, 44), the collider structure in Figure 2 implies that A C|X and A V. Z Further, recall that for any three random variables A, B, C, the mutual information T(A; BJC) 2 0, and [(A; B(C) 0 iff they are conditionally independent, i.e., A BIC. Combined the facts above, will have (A; CIX) > 0 and / (A; C(Z) > 0 almost surely.
A.3 Proof of Theorem
Proof.
Leveraging Lemma we know that [[A; C(Z) holds almost surely, which implies that H(AlZ): > H(A]Z,C).
Since Zo is reparameterization of C, we have H (A]ZZ HIA Zc) (A) Subtracting (A) both sides of Eq. (5) gives
H ((A (AA) H(A H(A);
which equivalent to T(A: 2) > T(4; 2). Besides, we also have (C; = I(C;2,7c) >
[O; Z), which completes the proof.
A.4 Proof of Theorem
Proof. First, let us consider the first proposition of Theorem 2. We have
1(A;C X) = - X(A;X)
T(A:C) +
(6)
-[A;
= =H(A) - ([A+ H(A + XC A)
= =H(A) (IA +C) +
Since H(XC) HCC) and H(A) and H(C) are determined by NA and Nc, the goal then transformed into minimizing H(A + AC). In order to address this problem, we first draw table possible outcome of A + XC. To determine H(A + XC), we can divide the elements in the table into groups, where they are distributed to the same group if and only if they have the same value. Let us assume that NA No because the opposite condition can solved in similar way.
below to enumerate
16

---

0
1
2
- -2
NA-1
1+1
2
-
NA -
21
1
24 2A
NA NA-2 22-2222
-A-1+ 1+21
" NA - 11-22NNc -2 - 2
(Nc 2
1+ I+ (Nc 2
2+ (No 2/A
(Nc -0-1 1)A
1+ (No 1)A
2+ (No 1)
NA -22 (Nc - -1)
NA - - 1NN - 1]A
Note that the number of elements in single group cannot be greater than NA. for most element in each column can be distributed to that group. Therefore, we denote 2 as the number of groups that consist of k elements, & E {1, ..., NaJ. Our target is [ - AC) = Em Tt NaNc In No For simplicity, we further denote y/ NaNo In NaNo then (AA + AC) = ΣM tz,9e-
Considering the total number of elements, we have
I1 2x² +... NAANA NANo.
We now pay attention the top and the rightmost column, where the elements bound to be in different groups. Under closer observation, we find that the element ( forms a group alone, as does the element NA (Ne 1)A. The element 1 is in group made up of most elements, as are the elements X, (NA- - 1) + (No 2) (NA - 2) + (Nc 1)X. Based on similar analysis, the following relationship can be deduced:
I1 2 2, æ1 + 2r² 2 6, 21 + 2r² +... + (NA 1) 2 NA(NA 1).
(8)
We also have
> 1/2 YN
(9)
NA
H(A+AC) Σ
Lelht
NA -1
NA 1
= -NANC- Σ
LEE)NNA +
Lagth
NA-1
= =NANCUNA +
Σ
(xr yN.)
NA-1
-NANCUNA (UNA-1--A0
txxt
(10)
Na-2
(UNA-2 - -1
9NA-1 Σ
txt + + (y/2 - -y))(11 + -202) + (311 - )
1=1
NANCONA (N 1)(MNA11 NNN)+ )+
(NA 1) - -2) UNA-1) + + 6(y2 - yys) + 2(81 y2)
N4-1
NN NA 1) +2 Σ ty:-
The equality condition for this inequality is
-NA-1 2.
(11)
This indicates that every secondary diagonal (from upper right lower left) in the aforementioned table forms group of elements, which means = 1. This completes the proof the first claim.
Let us further look the second proposition. From the discussion above, we know that À = the optimal value, and want to maximize [(A, C|X) H(A) + H(C) H(A + C)., Let us assume
17

---

that NA 2 No. In order to calculate in detail, we first list the probability distribution of A + C.
1 P(A+ C[A+
2 P Nc 2) No -1 NaNc
P(A+C= = NANc
NA
P(A+C= = Ne - 1) = NA
P(A+ C(A+C=Nc) - NA P(A+C= N - 1) NA (12)
P(A+C=NA) NA) = No-1
1
NANo ..... P(A+C= No + NA - 2) NANc
Now, we have
Ne-1 H(A Σ
NANo NANG ) (NA - No 1) 1 1
In
In
NA
NA
i=1
No-1
NA No +
= 2
Σ
(In (In + Int]]
In NA
NANe
NA
(13)
Nc
Ne-1
NA Nc +1
(In + Nc))
(ilnt) +
In
NA
NANc
NA
2 No-1
Nc -1 1
= =InNA +
In
(i
NA
NANC
Thus,
И(A) + (CC) (H(C))((AOC)))
Ne-1
NA - No +1
2
INNe + NANc (ilmi)
NA
NA No
2
[ Ne-1
2
Nn + NANo
x xlndd
(14)
NA
NA No +1
1
[[No - 1)2 In (No - 1) (Nc - 1)2 -1
In + NANc
NA
2
(Nc 1)2
In (Nc - 1)+ Nc 2 Nc
Nn
[[Nc In
Nc
NA
Note that (Nc 1) No
(Ne-1)² In (No - 1) + - is greater than O. Therefore, the lower bound of I(A, C|X) is monotonically non-decreasing with respect NA. If NA is adequately large, I(A, CIX) approximates In No.
No
A.5 Proof of Theorem
Proof.
The lower bound can be easily derived by taking the difference between the two quantities:
I(Z:C) I(Z:A) 21(Z;C;A) I(Z: A)
(15)
I(Z:A(C)
(16)
2 - A(X;
(17)
where the last line comes from the information processing inequality.
B Experiment Details
In this section, we detail the setting of each individual experiment in this work. All experiments are conducted with single NVIDIA RTX 3090 GPU.
18

---

B.1 Experiment Details of Different Equivariant Pretraining Tasks
In this experiment, we conduct equivariant pretraining tasks based on seven different types of transformations. In order to maintain fairess and avoid cross tross-interactions, we only apply random crops to the raw images before we move on these tasks. We adopt ResNet-18 as the backbone with two-layer MLP that has hidden dimension of 2048 and an output dimension corresponding to the pretraining tasks. Under each transformation, we train the model for 200 epochs on 11FAR-10, with batch size 512 and weight decay 10-6. The detailed pretraining tasks are listed as follows.
Horizontal Flip & Vertical Flip & Color Inversion & Grayscale. We randomly (i.e., with probability 0.5) apply the specific transformation to images and require the model to predict whether or not we have really done the transformation. In these cases, the output dimension is 2.
Four-fold Rotation. We rotate the images with equal probability (i.e. with probability 0.25) by 0" 90". 180%, and 270 and require the model to predict which rotation angle we have actually adopted. In this case, the output dimension is 4.
Four-fold Blur. We apply Gaussian blurs to the images using kernel sizes of 0, 5, 9, and 15, where kernel size 0 refers to not applying Gaussian blurs. We then require the model to predict the kernel size. In this case, the output dimension is 4.
Jigsaw. We divide the images into 2 patches, randomly shuffle their order, and then require the model to predict the original arrangement. In this case, the output dimension is 24, since there are 24 possible permutations for the shuffled arrangements.
During the pretraining tasks, we simultaneously train a classifier, which is a single-layer linear head and is trained without affecting the rest of the network. Apart from the seven tasks, also conduct a baseline experiment, where we fix a random encoder and optimize the classifier alone in order to assess the effectiveness of these pretraining tasks.
B.2 Experiment Details of How Class Information Affects Equivariant Pretraining Tasks
In this experiment, our goal is to figure out how class information affects rotation prediction. Figure demonstrates the outline of the model we use to conduct this experiment. We apply random crops with size 32 and horizontal flips with probability 0.5 to the raw images.
Training objectives. As for the experiment process, we first use rotation prediction as the pretraining task with a cross-entropy loss between our predicted angles and the actual angles, defined as
N
Lrot
Pij log(d:)),
(18)
N
ビ=1=11
where N is the image number, the one-hot vector Pij refers to the true rotation angle of the 7th image, and Pig refers to the prediction of the model. In the case where class information is incorporated, simply add the original loss function the cross-entropy between the classes predicted by the classifier and their corresponding ground truth labels, defined as
Cols = Σ Vig log(ū))) j=
(19)
where N is the image number, C is the class number, the one-hot vector Yij refers to the true class of the 5th image, and hiy refers to the prediction of the model. In other words, when class information is injected, the loss function is Lrot + Cote. where the mixing coefficient aı is a prper-parameter. Furthermore, to eliminate class information from the first setting, we minimize the classifier loss with Loie trying to probe class information in the representation; in the meantime, we optimize the encoder to maximize the classification loss, aiming to eliminate any class information that can be found by the classifier. In particular, we adopt joint training objective for the encoder as Lrot - Ayfelss where This leads to min max optimization between
the mixing coefficient X is also dyper-paramedd the encoder and the linear classifier. We choose X; = 0.5 and X2 = 9, under which the class features can be shown to benefit or harm rotation prediction.
In the pretraining process, we mainly use Resnet-18 as the backbone with two-layer MLP that has hidden dimension of 2048 and output dimension of 4, and single-layer linear head as
19

---

projector
R
X
encoder
classifier
Figure 4: The model of this experiment. X: raw input; Z: representation; R: rotation prediction; C: class prediction. For rotation prediction, unless specified, the gradient flowing from the classifier to the encoder is detached.
Table Training rotation prediction accuracy and test linear classification accuracy under different base augmentations (CIFAR-100, ResNet18).
Dataset
Augmentation
Network
Train Rotation ACC
Test Classification ACC
Gain
ResNet18
99.83
11.31
None
EqResNet18
100.00
32.38
+21.07
ResNet18
90.94
13.19
CIFAR-100
Crop&Flip
EqResNet18
99.88
49.47
+36.28
ResNet18
68.29
10.65
SimCLR
EqResNet18
82.69
37.11
+26.46
a classifier. For each setting, we train the model for 200 epochs on CIFAR-10 and CIFAR-100 train the model for 100
respectively with batch size 512 and weight decay 10-6 Additionally, epochs on LinyIImageNet-200 with batch size 256 and weight decay 10-6, 6 To further explore different backbones, we also Resnet-50 and train the model for 200 epochs CIFAR-10 with batch size 256 and weight decay 10-6
interested the effects of class information the accuracy of the pretraining tasks based on other transformations such as horizontal flips and four-fold blurs that are regarded as intrinsically hard. We simply inject class information into the representation by adding classification loss to the original loss as well. Slightly different from the operation rotation prediction, we apply random resized crops before conducting the pretraining tasks to avoid interfering with the prediction in these tasks. The other details and parameters are the same as those rotation prediction.
Furthermore,
B.3 Experiment Details in the study of model equivariance
In order compare the performance of Resnet and EqResnet, use rotation prediction as our pretraining task and obtain the linear probing results. We apply various augmentations to the raw images, such as no augmentation, combination of random crop with size 32 and horizontal flip, and SimCLR augmentation with output size 32. To be more specific, SimCLR augmentation refers sequence of transformations, including random resized crop with size 32, horizontal flip with probability 0.5, color jitter with probability 0.8, and finally grayscale with probability 0.2.
In these experiments, of 2048 and an output dimension of 4, and single-layer linear head as a classifier. For each setting, train the model for 200 epochs CIFAR- 10 and CIFAR-100 with batch size 128 and weight decay 5 x 10 The results on CIFAR-100 are displayed in Table 4.
predict rotation angles with two-layer MLP that has hidden dimension
C Y-information: Background and Extensions
In this section, we introduce V-information [75], which is a computation-awar and model-aware extension of Shannon's 's notation that more suitable for modeling neural representation learning. Then, extend our theory and show that the main results still hold under V-information.
20

---

C.1 Definitions and Properties of V-information
V-information is proposed by Xu al. [75] under the consideration of computational constraints, which happens to be one of the drawbacks of traditional Shannon information theory. An additional merit of V-information that it can be estimated from high- eigh-dienssionl data. The formal definition of V-information is derived as follows. Denote Y as the target random variable that the model is trying to predict and X as another random variable that provides side information for the prediction of Y. Let X' and Y/ be the sample spaces of X and Y. Define Ω == {f X U U + P())) as a set of the functions that maps X to a family of probability distributions over Y.
Definition 1 (Predictive Family), V C Ω is called a predictive family if E V, VP range(f), 3f E V that satisfies Va E X, ['xx] = P =
In other words, predictive family is set of probability measures that are allowed to be used under computational constraints. The existence of f indicates that the agent can optionally ignore the side information.
Next, we introduce predictive conditional V-entropy and predictive V-information.
Definition 2 (Predictive Conditional V-entropy), [Y) infrevEe..-x..t log [()] Specifically, Hy (0) = infrevyyy-y( [- log [0] (w)]
Definition 3 (Predictive \--nformatiinn. Tv(X + Y) Hy (Y10) Hv(YIx).
Apart from the definitions, we have to highlight an important property of predictive V-information. Lemma 3 (Xu al. [75]). Iv(A + B) 0 iff A and are independent variables.
C.2 Extension to y-information
First, we present the V-information version of Lemma
Theorem 5 (Fxplaining-away in E-SSL). If the data generation process obeys the diagram in Figure 2, then almost surely, A and C is no dependent given X or Z, Le,, A V. C|X and V. CVZ. It implies that [v A|X) > 0 and [v AlZ) 0 hold almost surely.
Proof. Lemma 3 indicates that for any three random variables A, B,C, C, the inequality (v - BIC) 2 always holds and that (A BIC) 0 iff A 1 BJC. Based on the analysis of the collider structure in Appendix A.2, we know that A and C are not independent given either X or Z. Thus, we have (v A(X) and [v(C- + AlZ)= >> 0 almost surely.
Then, we present the V-information version of Theorem 1.
Theorem Assume that the representation Z consists of two parts = [Z,, Zc), where Z₁ is class- irrelevant, and Zc = ((C)) is representation of the class C with an invertible mapping &. If there a positive synegy effect [C + A ZT) > O, we will have ly(Zi - A) < ly - A), showing that with class features Ze we attain strictly better equivariant prediction. As consequence, the optimal features of equivairant learning will contain class features.
Proof. We have the assumption Lv(C AlZ1) Hv(A Hv Z1) > 0. Given that the function o is invertible and Ze = Φ(C), we have Hy(ACC, Zr) = Hy Zi) = Hy(A[Z) < (A)))) Subtracting (A( from both sides and rewriting the inequality, we finally obtain
[v - A) > [v(Z,
D Detailed Elaboration on Computing C(A;
The computation of (A; C(X) requires knowledge of both the pretraining label A and the class label C. When both are available, we can compute following its decomposition I(A; C|X)= = X)A H (A) C). Estimating entropy in high-dimensional space is generally hard, and common approach is to leverage variational estimates for the two entropy terms HAA and H(A X, C)) using neural networks.
Variational estimates of entropy. Take HAA as an example. We can learn a NN classifier Po (A(X) optimized by minimizing the cross-entropy L(0) = PP (A)X). The following
21

---

inequality shows that CE loss is the variational upper bound of (A) When 0 is minimized at the minimum with Pa(A(X) Pa(A we have L(0) (A) as the perfect estimate. Since NNs are generally expressive approximators, we believe that the (converged) CE loss can serve good estimate for (A
Variational estimates of ((A) Combing these two, we notice that good estimate for [(A; C) is the difference between the cross entropy losses of the predictor (A(A and Po(A|X, C)), which is exactly what we studied in the verification experiment in Section 4.1. In other words, the gap of the CE losses (or similarly the accuracies) between "rotation" and "rotation+cls", can serve as a quantitative measure of the synergy effect I [A; CIX).. A more computationally friendly way select only small subset of samples and train shortly. As shown in Figure 3, the gap already exhibited at the early stage of training.
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We support each point by either theory or experiment.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
See Section A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
We have provided a URL to our code in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppccceecccccccccccccclt S ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The differences often significantly large and the exact performance is not the primary concern of this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http:///eenr/eeeeeeee http://uurrp.sccpulllaaaa htte//ulliccceeeee Caublic/EthcsSuiiee ctt.iiiicceeeeleeeccc ines?
Answer: [Yes]
Justification:
We obey all aspects of the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is theory-oriented paper with no societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not involve such models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the authors of the models and the dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We introduce no new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have such studies.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "P9v7SKDnRn", "reviewer": "Reviewer_a2cS", "summary": "This paper proposes a theoretical and empirical study of the role of invariant and equivariant representations in self-supervised learning. While a number of works have been focused on learning equivariant representations, it remains unclear whether or not equivariance is beneficial in specific tasks. By studying how applying/predicting transformations in latent space is a task that requires class information (through the explaining away effect) the authors derive insights on the behaviour of existing methods and provide directions for future work, notably that invariance may not be necessary at all to obtain good representations.", "questions": "In experiments such as figure 1, what is the training loss/method considered ?\n\nTheorem 1 puts forward an argument which seems key to the success or not of equivariant SSL (when used without an invariant objective), \"How much information about C do you need to know to predict A\". Bringing this point (and class relevance in general) forward in section 3.1 would make the motivation from figure 1 clearer.", "limitations": "Limitations are addressed in section 6, focusing on the theoretical nature of the work and how it doesn't directly translate into improved methods. It would be good to elaborate further on the limitation of the analysis, such as how different instantiations of an equivariant object may lead to different behaviors, or the difficulty of characterizing quantities such as I(A;C|X) for various transformations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The theoretical analysis confirms previous empirical findings, for example that the considered augmentations must be complex/lossy [1] as well as diverse [1] to improve how much class information is present in the representations.\n\nThe collider structure and explaining away phenomenon are an elegant way to understand the interplay between the original data and its transformations, showing that C and A cannot be considered to be independent. \n\nWhile invariance loss are commonly used to learn equivariant representations[2,3] (in conjunction with an equivariant objective), previous work had found that even with a purely predictive objective, competitive performance can be obtained[1,4]. This is in line with the conjecture line 281 \"learning from equivariance alone can achieve competitive performance\", and raises an important future line of work.\n\n\n[1]Garrido, Quentin, et al. \"Learning and leveraging world models in visual representation learning.\" arXiv preprint arXiv:2403.00504 (2024).\n\n[2] Gupta, Sharut, et al. \"Structuring representation geometry with rotationally equivariant contrastive learning.\" arXiv preprint arXiv:2306.13924 (2023).\n\n[3]Devillers, Alexandre, and Mathieu Lefort. \"Equimod: An equivariance module to improve self-supervised learning.\" arXiv preprint arXiv:2211.01244 (2022).\n\n[4] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" arXiv preprint arXiv:2302.10283 (2023).", "weaknesses": "The paper seems to focus on two distinct families of methods. Methods which are augmentation aware (e.g. RotNet) and methods which are truly equivariant, which are explicitly designed to learn the transformation between transformed data in latent space, bringing more structured representations. While this distinction may not affect the general ideas and theoretical arguments, it may impact empirical results and practical considerations. It would be good to have a discussion/comparison of these distinct approaches.\n\nMost of the practical insights stemming from this work are qualitative. A conclusion such as \"knowing C is beneficial to predicting A\" is a useful general guideline, but doesn't directly translate to practical guidance. The results would be stronger with more quantitative assessments of class relevance (perhaps in another, more computationally friendly domain), i.e. estimating I(A;C|X). This should be seen more as an avenue for future work though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer a2cS for acknowledging our theoretical contributions. Below we address your main concerns.\n\n---\n\n**Q1**. Comparing augmentation aware methods (e.g. RotNet) and truly equivariant methods (eg CARE).\n\n> While this distinction may not affect the general ideas and theoretical arguments, it may impact **empirical results and practical considerations**\n> \n\n**A1**. Thank you for your insightful question. For a controlled study on the role of true equivariance, we add the equivariant objective proposed in CARE [1], which is known to enforce true equivariance when it attains an optimum. The results are shown below. \n\nWe find that although the two have similar rotation loss, training with the equivariance regularization leads to further improvement in test accuracy (57.32% → 64.50%), which is quite similar to our findings on the model equivariance.  As discussed in Sec 5.3, these pieces of evidence suggest that **enforcing feature equivariance (either through model architecture or feature regularization) does have considerable benefits compared to vanilla predictive loss**. This is a valuable insight and we will add it in the revision. Thanks for bringing it up!\n\n|  | Rotation ACC | Equi loss (Equivariance) | Test Classification ACC |\n| --- | --- | --- | --- |\n| RotNet | 97.71 | 18.8444 | 57.32 |\n| RotNet + 0.1*Equi Loss | 99.95 | 0 | 64.50 |\n\n---\n\n**Q2**. Quantitative estimate of I(A;C|X).\n\n**A2**. Indeed, in our analysis, the mutual information I(A;C|X) is the key factor that upper bounds the utility of class features during pretraining and thus determines the downstream transferability of E-SSL. In fact, the performance gap between “rotation” and “rotation+cls” shown in the controlled experiments in Sec 4.1 can serve as a surrogate metric for the MI as a variational estimate. \n\nDue to the limit of space, we put the detailed derivation in a following comment that you may read if interested. Please let us know if there is more to clarify.\n\n---\n\n**Q3.** In experiments such as figure 1, what is the training loss/method considered?\n\n**A3**. For a fair comparison, we follow the standard RotNet training setting and adopt CE loss for predicting discrete variables. We adopt the same training hyperparameters for all settings. More details are described in Appendix B.1. We will make it more clear in revision.\n\n---\n\n**Q4**. Theorem 1 puts forward an argument which seems key to the success or not of equivariant SSL (when used without an invariant objective), \"How much information about C do you need to know to predict A\". Bringing this point (and class relevance in general) forward in section 3.1 would make the motivation from figure 1 clearer.\n\n**A4**. Thank you for your suggestion. During our writing, we felt that although important, it was hard to bring out the “how C helps to predict A” perspective earlier in Sec 3.1 without explaining the explaining-away effect (that appears later in Sec 4.1). Though it may sound natural after understanding the explaining-away effect, before that, people might wonder, “why do you care about how C helps to predict A” (since this retrospective understanding is also new). Bearing this in mind, we did not want to introduce puzzles here. We will try to re-organize it for a better balance. Thanks again for sharing your thoughts!\n\n---\n\nThank you again for your insightful questions, which helps make this work more complete. Please do not hesitate to let us know if you have additional concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y6dtMDtocn", "reviewer": "Reviewer_yP9X", "summary": "This study contributes theoretical insights that enhance our understanding of conventional practices in SSL training. Building on these theoretical foundations and supported by experimental evidence, the study puts forward several principles for the practical implementation of equivariant self-supervised learning designs.", "questions": "Could the principles 1,2,3 be applied to other domains beyond vision-based SSL?", "limitations": "Please refer to the question.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This study presents theoretical findings corroborated by experimental results. Specifically, the theory enhances our understanding of the design of augmentation functions. These findings are crucial as previous designs of augmentation functions have largely relied on empirical exploration.", "weaknesses": "Experiments are conducted using smaller-scale models.\n\nTypo in line 352.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer yP9X for acknowledging our contributions to theoretical understandings. Below, we further address your concerns on the empirical side.\n\n---\n\n**Q1.** Experiments are conducted using smaller-scale models.\n\n**A1.** Thank for your advice! The experiments are mainly designed to validate our theoretical insights, and as shown in previous works, E-SSL methods generalize well across different model sizes. Following your suggestions, we further validate our findings on a larger model ResNet-50 and a more complex dataset CIFAR-100. \n\nAs shown below, the results agree well with our findings in the main paper, that 1) better rotation accuracy under a larger model brings better test classification accuracy; and 2) model equivariance brings significant gains. Notably, the improvement of model equivariance is even more significant on CIFAR-100, considering that CIFAR-100 has 100 classes whose accuracy is harder to improve.\n\n*Results on CIFAR-100 (akin to Table 2 on CIFAR-10).*\n\n| Augmentation | Network | Train Rotation ACC | Test Classification ACC | Gain |\n| --- | --- | --- | --- | --- |\n| none | ResNet18 | 99.83 | 11.31 |  |\n|  | EqResNet18 | 100 | 32.38 | + 21.07 |\n| crop+flip | ResNet18 | 90.94 | 13.19 |  |\n|  | EqResNet18 | 99.88 | 49.47 | +36.26 |\n| simclr  | ResNet18 | 68.29 | 10.65 |  |\n|  | EqResNet18 | 82.69 | 37.11 | + 26.46 |\n\n*Results on ResNet-50 (with crop+flip) on CIFAR-100.*\n\n| Network | Train Rotation ACC | Test Classification ACC |\n| --- | --- | --- |\n| ResNet18 | 90.94 | 13.19 |\n| ResNet50 | 99.30 | 14.53 |\n\n---\n\n**Q2**. Could the principles 1,2,3 be applied to other domains beyond vision-based SSL?\n\n**A2**. Yes! Our principles are based on a theoretical analysis of the **general methodology of E-SSL**, not tailored to vision-based SSL. In other domains (text, graph, speech, time series etc), there are also many similar SSL methods that can be understood as E-SSL in the general sense (see e.g., the discussion of MAE and BERT in Sec 5.1). Taking the **BERT model in NLP** as example, our theory can understand its success from these the three principles: \n\n- **Principle I:** First, BERT learns representations by predicting the masked words (task $A$), and it is shown that a large masking ratio is important for learning useful features (for task $C$). As analysed in Principle I, this is also because we need to corrupt the input enough in order for the high-semantic class-related features to be utilized for masked prediction.\n- **Principle II:** Meanwhile, class information (high-level text semantics) is indeed helpful for masked prediction, thus demonstrating the importance of class relevance (Principle II).\n- **Principle III:** At last, masking prediction as a token-level dense prediction task helps prevent the shortcut features (Principle III).\n\nThus, the insights of our principles are general and not limited to a certain modality. As the first theoretical work in the field of E-SSL, it can help explain many existing E-SSL-like approaches as well as inspire new designs in multiple domains.\n\n---\n\nHope the explanation above address your concerns! Please let us know if there is more to clarify.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLGzGiZqwx", "reviewer": "Reviewer_TYZN", "summary": "This paper aims to reduce the gap between theory and practice for equivariant SSL, which refers here to the sub-family of discriminative SSL methods that do not enforce the invariance of representations to augmentations. The authors propose an explanation based on the \"explain-away\" (_“Explaining away” is a common pattern of reasoning in which the confirmation of one cause of an observed or believed event reduces the need to invoke alternative causes._) and conclude that this principle justifies strictly positive mutual information between the pre-task labels (e.g., the rotation angle) and the downstream task label (i.e., semantic class label) thereby explaining why the pretext and downstream tasks are not systematically misaligned.", "questions": "- Can you please clarify what is meant by \"global\" vs. \"local\" transformations (principle II)?\n- Can authors rigorously justify the extension of the explaining-away principle from X to C?", "limitations": "Limitations are not explicitly discussed but do mention the use of simplified models (e.g., section 4.3) for theoretical purposes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Relevance of the research question: while an intuition as to why E-SSL performs better than a random encoder might be easy to formulate (see below in weaknesses), a theoretical formulation supporting these intuitions eludes. A principled explanation would help the community in the design of better and more principles E-SSL approaches.", "weaknesses": "My main concern regards the soundness of the contribution, the reasoning leading to this concern follows:\n- in this work, the authors use the \"explaining-away\" principle which relies on the proposed latent variable model shown in Figure 2 to justify E-SSL performance. Authors consider a partitioning of the latent information in $C$ and $S$, the former encapsulates semantic information, the latter the non-semantic information (should hence include the object position & orientation, textures). \n- I believe a reasonable intuition as to why E-SSL (e.g., predicting the rotation angle of an image as a pretext task and then performing semantic classification) can perform better than a random encoder - as shown by authors in Figure 1 -, is because of a selection bias in images used for training: each semantic class/object is more likely to be depicted in a specific orientation (e.g., a person standing) than in any other orientation (i.e., a person upside down). This means that the pretext and downstream tasks are not completely misaligned: being aware of the class information allows one to make a better guess about the rotation angle than a random guess. If an object is as likely to be depicted in one orientation than any other orientation than the class label is uninformative to make a guess regarding the rotation angle. The difference in results between horizontal and vertical flip in Figure 1 supports that intuition. The explanation proposed by the authors does not support that intuitive explanation of E-SSL but instead seems to propose a overly-simplified \n- The explaining-away principle justify a strictly positive mutual information between $\\bar{X}$ and $A$ which authors extend to a strictly positive mutual information between $\\bar{C}$ and $A$. It is not obvious from the get-go how authors can theoretically justify this extension. \n\nTo summarize, the two aforementioned points should be further discussed by the authors to confirm the soundness of the proposed explanation which seems to take shortcuts both from an intuitive and theoretical perspective. I would suggest discussing these points more thoroughly. \n\nMinor comments:\n- Relevance of certain results: Theorem 2 is tied to an over-simplified mixing model ($ X = A + C$) \n- Rigor:\n     - Notation: notations should be consistent throughout the paper (e.g., line 250 and 256, line 236/237 and line 240) and assumptions should be made clear (e.g., eq (3) and line 293, lambda=1 is not stated)\n     - Principle III: refers to theoretical results - \"style features may also explain the equivariant target A\" - , which have not been explicitly shown (in section 4.1 or in the appendix)\n    - Contrastive learning vs. Instance discrimination: while instance discrimination performs classification with a number of classes equivalent to the number of training instances, contrastive learning like SimCLR usually discriminates at a batch level hence I believe line 339 does not accurately describe contrastive learning but rather instance discrimination.  \n- References for claims: \"style features are often easier for NN learning\" - line 260,\n- Typos: line 352, 68", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer TYZN for a critical reading of our work. We carefully examine the intuition you propose, and find that it actually fits well into our theory.\n\n---\n\n**Q1.** Discussion on the intuitive understanding of why E-SSL works and how it fits into our theory.\n\n**A1**. We resonate with your insight that for rotation prediction to be effective, the class/object itself must exhibit a rotational bias (i.e., it is a **necessary** condition). Nevertheless, this intuition lacks a solid theoretical explanation. We find it **naturally fits as a piece of our explaining-away framework, where it can have a rigorous explanation**. \n\nTo extend our current analysis, besides the causal diagram in Fig 2, we further consider the inherent rotation angle $\\bar A$ and an edge $\\bar A\\to \\bar X$, i.e., the original object is also generated from its intrinsic angle. Notice that now there is a new collider structure $C\\to \\bar X\\leftarrow \\bar A$, which means that $I(\\bar A;C|\\bar X)>0$, which rigorously explains why class information $C$ is helpful for predicting the intrinsic rotation $\\bar A$ from the original object $\\bar X$.\n\nSince the intrinsic rotation $\\bar A$ is unknown, RotNet further applies a (known) random rotation $A$ to $\\bar X$ and get $X$, and the rotated image has an angle $A'=\\bar A+A$. Apparently, figuring out  $\\bar A$ is directly helpful for predicting the random $A$ (the E-SSL target). Formally, we have a collider structure $\\bar A\\to X\\leftarrow A$ and have a positive synergy $I(A;\\bar A|X)>0$.\n\nThus, our explaining-away analysis justifies that 1) knowing $C$ is also helpful for predicting $\\bar A$, and 2) knowing $\\bar A$ is helpful for predicting $A$ as well. In this way, introducing $\\bar A$ does not invalidate our previous theory but provide **more fine-grained analysis on the way of how class information helps rotation prediction**. \n\nThank you again for this valuable perspective. We will definitely incorporate it and acknowledge your insights.\n\n---\n\n**Q2**. Theorem 2 is tied to an over-simplified mixing model ($X=A+\\lambda C$).\n\n**A2**. We note that Theorem 1 already gives us **a general characterization that is agnostic of the data distribution or the model class**. Therefore, our framework and analysis does apply to general real-world scenarios. Due to its generality, it is also hard to obtain a very fine-grained quantitative analysis. Thus, the purpose of Thm 2 is to gain more quantitative insights with a linear model. Note that linear data assumptions are also commonly adopted in SSL theory, eg [1]. We leave more complex cases if left for future work.\n\n**Ref:**\n\n[1] Wen, Zixin, and Yuanzhi Li. \"Toward understanding the feature learning process of self-supervised contrastive learning.\" ICML. PMLR, 2021.\n\n---\n\n**Q3**. Rigor:\n\n> Notation: notations should be consistent throughout the paper (e.g., line 250 and 256, line 236/237 and line 240) and assumptions should be made clear (e.g., eq (3) and line 293, lambda=1 is not stated)\n\n\nThanks. We will fix it.\n\n> Principle III: refers to theoretical results - \"style features may also explain the equivariant target A\" - , which have not been explicitly shown (in section 4.1 or in the appendix)\n\n\nIndeed, we didn’t elaborate the style features here. Similar to the intrinsic rotation variable $\\bar A$, the feature variable $S$ also constitutes the object $\\bar X$ (with a causal edge $S\\to \\bar X$). Due to the explaining-away effect in the collider structure $S\\to X\\leftarrow A$, $S$ can also explain and help predict $A$ with $I(S;A|X)>0$. Therefore, style features, as an easy-to-learn shortcut, can lead the model to learn fewer class features. We will elaborate this relationship rigorously in the revision.\n\n> Contrastive learning vs. Instance discrimination\n\n\nWe note that contrastive learning can be seen as a minibatch approximation to instance classification by random subsampling a few instances (e.g., 2048) at each time. In expectation, the objective is still distinguishing the positive instance from all the other negative instances in the dataset. **This negative class sampling technique is widely used in standard classification tasks, e.g., negative sampling in word2vec** [1]. Thus, we think it is plausible to regard contrastive learning essentially as an instance classification task.\n\nReferences:\n\n[1] Mikolov et al. Distributed Representations of Words and Phrases and their Compositionality. NeurIPS 2013.\n\n---\n\n**Q4**. Reference for claims and Typos.\n\n**A4**. Thanks. We will add the references and fix the typos.\n\n---\n\n**Q5**. Can you please clarify what is meant by \"global\" vs. \"local\" transformations (principle II)?\n\n**A5**. Here, local changes refer to those that only modify pixel values but do not modify the pixel’s positions (eg color inversion). Therefore, to predict these operations, the NNs only need to look at each pixel’s values without relying on global context. Instead, global operations change pixel positions (e.g., rotation, flip), and in these cases, figuring out the global content can facilitate the prediction of these global changes. We will clarify it in the revision.\n\n---\n\n**Q6**. Can authors rigorously justify the extension of the explaining-away principle from X to C?\n\n**A6**. Of course. In fact, according to probabilistic graphical models, **as long as there is a collider structure between the variables**, there will be an explaining-away effect. In the causal structure $C\\to \\bar X\\to X$, we can also omit the intermediate node $\\bar X$ (which also holds since $X$ is causally dependent on $C$) and write the collider structure $C\\to X\\leftarrow A$, which implies the explaining-away effect with $I(A;C|X)>0$. \n\n---\n\nThank you for your careful reading and for sharing your insights. If you would find our explanations satisfactory, we respectfully ask you to consider re-evaluating our work based on the revisions. We are also happy to address any remaining concerns during the discussion stage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CKsYsS7Mw6", "reviewer": "Reviewer_RgKg", "summary": "This paper provides a theoretical understanding of equivariant self-supervised learning (E-SSL) methods and their effectiveness. The authors propose an information-theoretic analysis that explains the synergy effect between class information and equivariant transformations, leading to improved downstream performance. They also identify three principles for E-SSL design: lossy transformations, class relevance, and shortcut pruning. The paper further discusses the importance of model equivariance and demonstrates its advantages in E-SSL. Overall, the contributions of this paper lie in the theoretical explanation of E-SSL and the guidance it provides for designing effective E-SSL methods.", "questions": "1. Is 89.49% mentioned in line 279 the ACC of the SimCLR framework during testing? Even with the enhancement of SimCLR, the task of predicting rotation only achieved a maximum effect of 59.06 in downstream tasks. These two values are not very close.\n2. In the paragraph starting from line 328, it is believed that the action space of the instance discrimination task is N, but I think it is actually a combination of many binary classification tasks with an action space of 2. So I think contrastive learning is not a special case to equivariant tasks.\n3. Principle 2 states that it is best to be category related, does it mean that category label information needs to be known during pre-training stage in SSL?\n4. Why can predictive patches like MAE be considered as an equivariant SSL task?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The article is well-structured, and the arguments are clear.\n2. The author analyzes how equivariance affects self-supervised learning from an information theory perspective and proposes three principles for designing E-SSL.\n3. The theoretical validity is demonstrated through theoretical proofs and some small experiments.", "weaknesses": "1. The experiments are conducted on few datasets which is not very convincing. \n2. There exist some typos in the paper, such as the “xIn” in line 352.\n3. It is better for the authors to provide a design enhancement example and conduct experiments to prove the effectiveness of the example, as it only explains the success of previous E-SSL works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## \n\nWe thank Reviewer PgKg for appreciating our writing and theoretical contributions. Below we address each of your concerns.  \n\n---\n\n**Q1.** The experiments are conducted on few datasets which is not very convincing.\n\n**A1**. Following your suggestion, we further validate our findings on ResNet-50 and CIFAR-100. The results all agree with our findings in the main paper, that 1) better rotation prediction brings better test error, and 2) model equivariance contributes to better test classification. Notably, model equivariance contributes even more on CIFAR-100.\n\n*Results on CIFAR-100.*\n\n| Augmentation | Network | Train Rotation ACC | Test Classification ACC | Gain |\n| --- | --- | --- | --- | --- |\n| none | ResNet18 | 99.83 | 11.31 |  |\n|  | EqResNet18 | 100 | 32.38 | + 21.07 |\n| crop+flip | ResNet18 | 90.94 | 13.19 |  |\n|  | EqResNet18 | 99.88 | 49.47 | +36.26 |\n| simclr  | ResNet18 | 68.29 | 10.65 |  |\n|  | EqResNet18 | 82.69 | 37.11 | + 26.46 |\n\n*Results on ResNet-50 (crop+flip) on CIFAR-100.*\n\n| Network | Train Rotation ACC | Test Classification ACC |\n| --- | --- | --- |\n| ResNet18 | 90.94 | 13.19 |\n| ResNet50 | 99.30 | 14.53 |\n\n---\n\n**Q2.** There exist some typos in the paper, such as the “xIn” in line 352.\n\n**A2**. Thanks for pointing out. We will fix them in the revision.\n\n---\n\n**Q3.** It is better for the authors to provide a design enhancement example and conduct experiments to prove the effectiveness of the example, as it only explains the success of previous E-SSL works.\n\n**A3**. We note that the key focus of this work is to establish the first theoretical understanding of E-SSL, instead of devising a new variant. The former may have a larger significance because it is more fundamental to this field. Driven by our theoretical insights, we also first study **combining model equivariance with training equivariance for E-SSL**, and it improves RotNet by **25.22% on CIFAR-10 and 36.26% accuracy on CIFAR-100**. Remarkably, we are the first to attain comparable performance **by predicting augmentations alone** (while previous methods rely on merging with contrastive learning). This illustrates that our analysis can open doors for new paths for E-SSL designs.\n\n---\n\n**Q4.** Is 89.49% mentioned in line 279 the ACC of the SimCLR framework during testing?\n\n**A4**. Indeed, we made a typo here. Later, when merging with the model equivariance (Sec 5.3), its performance (82.26%) can be close to SimCLR. We will fix this confusion in the revision.\n\n---\n\n**Q5**. In the paragraph starting from line 328, it is believed that the action space of the instance discrimination task is N, but I think it is actually a combination of many binary classification tasks with an action space of 2. So I think contrastive learning is not a special case to equivariant tasks.\n\n**A5**. We understand that contrastive learning (CL) can be understood as many binary classification tasks between positives and negatives. Nevertheless, this understanding defines **infinitely many** binary tasks and **every task has been seen only once** during training, a setting rarely studied in the ML literature. The deviation from common learning tasks makes it hard for us to have a solid understanding of how contrastive learning actually behaves. \n\nInstead, it is a common and natural way to understand CL as instance classification (IC). In fact, in the seminal work by Wu et al. [1] (prior to InfoNCE) that **first proposed contrastive learning** (their loss is the same as InfoNCE), they do regard CL as **a non-parametric approach to IC**, as the title suggested. We refer to [1] for a detailed explanation of this connection.\n\nWe further note that for a very large number of classes like IC, it is common to subsample the classes in the denominator of CE loss. A well-known example is the **negative sampling technique proposed in word2vec [2]** for learning on a very large vocabulary. In instance classification, existing works also subsample the negative classes [1,3], leading to a mini-batch number of negative samples during training. \n\nHope this elaboration address your concern! We will clarify this part in the revision.\n\nRef:\n\n[1] Wu, Zhirong, et al. \"Unsupervised feature learning via non-parametric instance discrimination.\" CVPR. 2018.\n\n[2] Mikolov, Tomas, et al. \"Distributed representations of words and phrases and their compositionality.\" NeurIPS 2013.\n\n[3] Cao, Yue, et al. \"Parametric instance classification for unsupervised visual feature learning.\" NeurIPS 2020.\n\n---\n\n**Q6**. Principle 2 states that it is best to be category related, does it mean that category label information needs to be known during pre-training stage in SSL?\n\n**A6**. No. For SSL, the labels are not supposed to be known during pretraining. What we are discussing in Principle 2 is that the choice of the augmentation type (eg the rotation or the flip) should be related to the class labels, in the sense that extracting class-relevant information from the input is helpful for predicting these transformations (no class label required) during E-SSL pretraining. We will make it clear in the revision.\n\n---\n\n**Q7**. Why can predictive patches like MAE be considered as an equivariant SSL task?\n\n**A7**. In this paper, we consider a general notion of equivariant SSL, in the sense that the representations is learned to be aware of the pretext, eg aug parameters. MAE augments by masking patches at random **positions $p$ (the aug parameter in MAE)**. Then, MAE takes the position $p$ as input, and learns to reconstruct images at these locations, thus its representations can adapt to the input position variables $p$, in contrast with contrastive learning whose representation is global and invariant to augmentations. Thus, we regard MAE more as an equivariant method. We will explain it in the revision.\n\n---\n\nWe hope this clarifies your questions. If you would find it satisfactory, we respectfully hope that you may consider re-evaluating our work based on the revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their positive comments and constructive critics on our work. Taking these valuable insights into consideration, we address these problems carefully in each response. We will also do the following:

- **A series of validation experiments on more datasets and models.** We will add a series of new experiments that reproduce our verification experiments on more datasets (CIFAR-100 and Tiny-ImageNet-200) and larger networks (ResNet-50), including *the pretext comparison (Fig 1),  the controlled study on class features (Fig 3), the base augmentation comparison (Tab 1) and model equivariance (Tab 2)*. See Figs A & B and Tabs A & B in the **rebuttal PDF** for details.
- **Fix typos.** We will improve readability by fixing the typos pointed out by the reviewers.
- **Discuss the Role of rotational bias.** Inspired by Reviewer TYZN on the role of intrinsic rotational bias, we will further incorporate the intrinsic equivariance variable $\bar A$ into the causal graph and elaborate on the explaining-away effect of $\bar A$ (elaborated below) for the prediction of class $C$. Notably, **this is a natural extension of our theory and does not change the overall framework, understanding, or message of E-SSL, but makes this understanding more complete**. We sincerely acknowledge Reviewer TYZN for sharing the insight.
- **Compare equivariant to augmentation-aware methods**. Inspired by Reviewer a2cS, we will add the comparison between truly equivariant and augmentation-aware methods, showing that true feature equivariance has additional benefits.

##

Meta-review: This paper explores why and how equivariant self-supervised learning (e.g., rotation prediction) learns useful representations from an information-theoretic perspective. The authors effectively addressed the reviewers' concerns during the author-reviewer discussion phase, and after the discussion, all the reviewers unanimously support the acceptance of this paper. Although AC has some concern about the limited empirical support, AC thinks the merits of this paper outweigh the concern: this paper can be a useful guideline for researchers considering equivariant self-supervised learning techniques. Therefore, AC recommends acceptance.
Decision: Accept (poster)