Understanding the Role of Equivariance in Self-supervised Learning

Authors: yifei  wang
kaiwen  mit
  hu
sharut  gupta
yisen  wang
stefanie  jegelka
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:48:35

Abstract: Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features to be augmentation-aware. However, even for the simplest rotation prediction method, there is a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information-theoretic perspective to understand the generalization ability of E-SSL. In particular, we identify a critical explaining-away effect in E-SSL that creates a synergy between the equivariant and classification tasks. This synergy effect encourages models to extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light on new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. Code is available at 
 https://github.com/kaotty/Understanding-ESSL.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) of data representations has made remarkable progress. Existing SSL methods can be categorized into two types: invariant SSL (I-SSL) and equivariant SSL (E-SSL). The idea of I-SSL is to encourage the representation to be invariant to input augmentations (e.g., color jittering). Contrastive learning that pulls positive samples closer and pushes negative samples apart is widely believed to be a prominent I-SSL paradigm, leading to rapid progress in recent years [13, 50, 5, 80, 46, 30, 3, 32, 33, 60, 49, 14, 37, 15, 77, 64, 81] . Nevertheless, since invariant representations lose augmentation-related information (e.g., color information), their performance on downstream tasks can be hindered, as frequently observed in practice [47, 17, 34] .In view of these limitations of I-SSL, there has been a growing interest in revisiting E-SSL. Contrary to I-SSL, E-SSL learns representations that are sensitive to (or aware of) the applied transformation. 4 For instance, RotNet [31] is an early exemplar of E-SSL that learns discriminative features by predicting the rotation angles from randomly rotated images [43] . It has also been exploited in recent works and achieves promising improvements in conjunction with I-SSL [73, 67, 17, 18, 25, 54, 34] . Recently, E-SSL has shown potential for serving as the foundation for building visual world models [26] .Despite this intriguing progress in practice, compared to invariant SSL methods with a vast literature of theoretical analyses [58, 66, 48, 35, 68, 59, 78] , there is little theoretical understanding of equivariant SSL methods. A particular difficulty lies in the understanding of the pretraining tasks, which may seem quite irrelevant to downstream classification. Taking RotNet as an example, the random rotation angle is independent of the image class, so it is unclear how rotation-equivariant representations are helpful for image classification. Generally speaking, it is unclear why, when, and how equivariant representations can generalize to downstream tasks.In view of this situation, the primary goal of this paper is not to design a new E-SSL variant, but to revisit the basic E-SSL methods and understand their essential working mechanisms. We fulfil this goal by proposing a simple yet theoretically grounded explanation for understanding general E-SSL from an information-theoretic perspective. We show that the effectiveness of E-SSL can be understood via the "explaining-away" effect in statistics, which implies an intriguing synergy effect between the image class C and the equivariant transformation A (e.g., rotation) such that almost surely, they have strictly positive mutual information when given the input X, i.e., I(C; A|X) > 0 that explains the effectiveness of E-SSL. This understanding also provides valuable guidelines for practical E-SSL design with three principles to pursue a large synergy effect I(C; A|X): lossy transformations, class relevance, and shortcut pruning, as been validated on practical datasets. Theoretically, we also quantitatively analyze the influence of data transformation on the synergy effect with a theory model.Equipped with these theoretical findings, we further revisit advanced E-SSL methods in the recent literature [73, 67, 17, 18, 25, 54, 34] and find that many of these empirical successes can be well explained in our framework from two aspects: finer equivariance and multivariate equivariance. Besides, motivated by our theory, we also discover an under-explored aspect of E-SSL, model equivariance, where we show that adopting equivariant neural networks can yield strong improvements for certain E-SSL methods. These fruitful theoretical and practical merits suggest that our E-SSL theory provides a general and practically useful explanation for understanding and designing E-SSL methods that have the potential to guide future E-SSL designs.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We support each point by either theory or experiment.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Invariant and Equivariant SSL. Without access to labels, SSL methods design various surrogate tasks that create self-supervision for representation learning. Early SSL methods, often in the form of predictive learning, learn from predicting the transformation of randomly transformed images, such as, RotNet [31] , Jigsaw [51] , Relative Patch Location [20] . Later, discriminating instances in the latent space with contrastive learning demonstrates prominent performance [21, 72, 52, 40, 13, 38, 56] , with variants including non-contrastive methods [33] , clustering methods [10] [11] [12] , regularization methods [77] . However, data augmentations used in contrastive learning to avoid shortcuts often come at the cost of information lost for downstream tasks (e.g., color for flower classification). To address this issue, there is a surge of interest in E-SSL that learns features to be sensitive to the applied transformations. Among them, Xiao et al. [73] use separate embeddings for each augmentation. Wang et al. [67] apply equivariant prediction on residual vectors between positive views. Dangovski et al. [17] combine contrastive learning and rotation prediction. Devillers and Lefort [18] , Garrido et al. [24] utilize conditional predictors with augmentation parameters. Park et al. [54] , Gupta et al. [34] model latent equivariant transformations explicitly.Theory of SSL. Most existing theories of SSL methods focus on contrastive learning (CL) and its variants from different perspectives: information maximization [52, 40, 65] , downstream generalization [58, 66, 48, 35, 68, 59] , feature dynamics [66, 69] , asymmetric designs [63, 82] , feature identifiability [79] , etc. But for general E-SSL methods, there is little, if any, theoretical understanding on how they learn meaningful features for downstream tasks (in particular, image classification). Our work fills this gap by establishing a general information-theoretic framework for understanding E-SSL.Equivariance in Deep Learning. Invariance and equivariance represent data symmetries that can be exploited during learning. There are two approaches to utilize invariance and equivariance. One is equivariant learning (to which E-SSL belongs) that uses equivariant training regularization such that features are approximately equivariant; the other is equivariant models that obey exact equivariance by design w.r.t. groups like rotation and scaling [16, 29, 8] . Equivariant models find wide applications in graph, manifold, and molecular domains [8, 29] , but are rarely explored for equivariant SSL. In this work, we find that model equivariance can be particularly helpful for equivariant learning in terms of both training and generalization, which opens an interesting direction to explore on the interplay between equivariant learning and equivariant models for future research.3 The Challenges of Understanding Equivariant SSL Notations. We introduce existing SSL methods from a probabilistic perspective. Generally, we denote a random variable by a capital letter such as X, its sample space as X , and its outcome as x. We learn a representation (Z/Z/z x ) from the input (X/X /x) through a deterministic encoder function F : X → Z. The general goal of SSL is to learn discriminative representations that are predictive of the image classes (labels) without actual access to label information. For ease of discussion, we mainly adopt the common Shannon information, where the entropy of X is H(X) = -E P (X) log P (X) and the mutual information between X and Y is I(X; Y ) = H(X) -H(X|Y ). It is also tempting to adopt V-information [75] that is analogous to Shannon's notion but aligns better with practice by taking into account computational constraints. For ease of understanding, we adopt Shannon's information in the main paper and extend the main results to V-information in Appendix C.For each raw input X sampled from the training set D, we independently draw a random augmentation A and get the augmented sample X = T ( X, A) with a transformation mapping T : X × A → X . The general objective of Equivariant SSL (E-SSL) is to learn representations Z = F (X) that are sensitive to the applied transformation A. For example, RotNet [31] utilizes random four-fold rotation A = {0 • , 90 • , 180 • , 270 • } for data augmentation, and learns feature equivariance by predicting the rotation angles from the representation Z. Therefore, E-SSL is driven by maximizing the following mutual information between the augmentation A and the representation Z:EQUATIONNote again that equivariance studied in this paper, as in many E-SSL works [17, 18, 24, 34] , is a relaxed notion of exact equivariance defined in a group-theoretical sense (with invertibility and compositionality) as in equivariant networks [16] . In E-SSL, equivariance generally means augmentation sensitivity, and the mutual information I(A; Z) measures the degree of equivariance of Z to A.Contrastive Learning as E-SSL. Contrary to E-SSL, I-SSL enforces features to be invariant to the applied augmentation A. CL is widely believed to be an example of invariant learning [17] . In CL, we apply two random data augmentations, A 1 , A 2 to the same input X and get two positive samples X 1 , X 2 and their representations Z 1 , Z 2 respectively. Since CL is driven by pulling Z 1 , Z 2 together, their mutual information objective is often formalized as max Z1=F ( X,A1),Z2=F ( X,Z2)I(Z 1 ; Z 2 ) [1].However, it is easy to observe that the constant outputs Z = const are also optimal with maximal I(Z 1 ; Z 2 ), suggesting that invariance alone is sufficient for SSL. In fact, contrastive learning can mitigate feature collapse with the help of pushing away from the representation of the other instances (i.e., negative samples), making it essentially an equivariant learning task w.r.t. the instance, known as instance discrimination [21, 72] . Indeed, contrastive objectives are essentially nonparametric formulations of instance classification [72] , and under similar designs, parametric instance classification achieves similar performance [9] . Non-contrastive variants with only positive samples are also shown to have inherent connection to contrastive methods in recent studies [63, 82, 23] .Equivariance is Not All You Need. A common intuition among E-SSL methods is that better downstream performance comes from better feature equivariance [18, 25, 54, 34] . Here, we begin our discussion by showing a counterexample in the following proposition. All proofs in this paper can be found in Appendix A. Note that different pretraining tasks may have different classes (e.g., 4 for rotation and 2 for horizontal flip). The baseline is a random initialized encoder with 34% test accuracy under linear probing.Proposition 1 (Useless equivariance). Assume that the original input X ∈ R d and the augmentation A ∈ R d ′ are independent, and X = [ X, A] ∈ R d+d ′ is obtained with direct concatenation (DC). Then, there exists a simple linear encoder that has perfect equivariance to A, but yields random guessing on downstream classification.Proposition 1 shows an extreme case when perfect equivariance is unhelpful for feature learning at all. Inspired by this finding, we further examine common image transformations for E-SSL: horizontal flip, grayscale, four-fold rotation, vertical flip, jigsaw, four-fold blur and color inversion (details in Appendix B). Prior to ours, Dangovski et al. [17] show that when merged with contrastive learning, additional equivariance to some augmentations (e.g., four-fold rotation and vertical flip) can bring benefits while some are even harmful (e.g., horizontal flip). It is not fully explored how much E-SSL alone depends on the chosen augmentation. Below, we study seven common transformations for E-SSL on CIFAR-10 [45] with ResNet-18: horizontal flip, four-fold rotation, grayscale, vertical flip, jigsaw, four-fold blur, and color inversion. We only apply random crops to avoid cross-interactions between different transformations. So the numbers reflect the relative strengths of different methods, instead of their optimal performance that can be attained.Figure 1 reveals big differences between different choices of transformations: with linear probing, four-fold rotation and vertical flip perform the best and attain more than 60% accuracy, while the others do not even attain significant gains over random initialization (34%). This distinction cannot be simply understood via feature usefulness, since color information imposed by learning grayscale and color inversion is known to be important for classification [74] . Meanwhile, in Figure 1a , we find that the degree of equivariance (measured by the training loss of E-SSL) does not explain the difference either, since among ineffective ones, some with large training loss have very low equivariance (e.g., horizontal flip), while some have very high equivariance with nearly zero equivariant loss (e.g., grayscale). These phenomena show that equivariance alone, either strong or weak, does not have a good or bad indication of downstream performance, which motivates us to provide a more general understanding of E-SSL.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: We have provided a URL to our code in the abstract.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
a theory of equivariant ssl: 
 In Section 3, we have shown that feature equivariance alone does not guarantee effective downstream performance, which makes it even unclear how equivariant learning extracts useful features. To resolve these puzzles, we provide an information-theoretic analysis for E-SSL that serves as a natural explanation for the phenomena above.We start by establishing a causal diagram of the data generation process of E-SSL, where we assume that the original input X is generated from its class variable C (relevant to input semantics, e.g., shape), intrinsic equivariance variable Ā (relevant to semantics, e.g., the intrinsic orientation of an object) and style variable S (features irrelevant to semantics and targeted equivariance, e.g., color and texture) through some unknown processes. Then, we apply an independently drawn augmentation variable A (e.g., a random rotation angle), and get the transformed input X. Afterwards, we obtain its representation Z through a neural network.Collider structure in E-SSL. The causal diagram shows that the class variable C and the augmentation variable A are independent. However, there exists a so-called collider structure where the augmented sample X is a common child of C and A. A well-known fact from statistics called the explaining-away effect (a.k.a. selection bias) [55, 44] says that in a collider block, when conditioning on the collider X or its descendent like Z, the parents C and A are no longer independent. For example, the weather (A) and the road condition (C) are independent factors that can contribute to car accidents (X). However, given that an accident happens (X is known), if we know that it rains today, it would be less likely that the road is broken, and vice versa. In this case, we say that the weather A explains away the possibility of road conditions C.The theorem below formally characterises the explain-effect effect in the E-SSL process and its information-theoretic implication. A caveat is that Lemma 1 guarantees that explaining-away happens in most, but not all cases (e.g., Proposition 1), and we explain these exceptions in Section 4.2. Lemma 1 (Explaining-away in E-SSL). If the data generation process obeys the diagram in Figure 2 , then almost surely, A and C are not independent given X or Z, i.e., A ̸⊥ ⊥ C|X and A ̸⊥ ⊥ C|Z. It implies that I(A; C|X) > 0 and I(A; C|Z) > 0 hold almost surely.Explaining-away helps E-SSL. In statistics, explaining-away often appears as the selection bias in observational data that misleads causal inference (e.g., the Berkson's paradox [6] ) and demands careful treatment [76, 7] . In contrast, explaining-away plays a critical positive role in E-SSL. In particular, the fact I(A; C|Z) > 0 implies an important synergy effect between A and C during equivariant learning, as shown below:EQUATIONEq. ( 2) implies that for the same feature Z, using class information C gives a better prediction of A (lower uncertainty H(A|Z, C)) than without using class features. Intuitively, given a rotated image, recognizing the object class C in the first place makes it easier to determine the rotation angle A. Driven by this synergy effect, the encoder will learn to encode class information C in the representation to assist the equivariant prediction of A. 5 We formally characterize this intuition in the following theorem. Theorem 1 (Class features improve equivariant prediction). Under the data generation process in Figure 2 , consider an E-SSL task with input X, its class C X , and its representation Z. Assume a class representation Z C = ϕ(C X ) that can perfectly predict the label C X (ϕ is an invertible mapping). Then, almost surely, the combined feature Z = [Z, Z C ] obtained by appending Z C to Z will strictly improve the equivariant prediction with larger mutual information I(A; Z) > I(A; Z). Also, we have I(C; Z) ≥ I(C; Z), so the classification performance improves in the meantime.As an implication of Theorem 1, to achieve better equivariant prediction, during E-SSL, the model will try to extract more class features, which will jointly improve downstream classification. This explains why during E-SSL with rotation prediction, the classification accuracy also rises along the process, outperforming the random encoder (Figure 1b ). Figure 3 : A controlled experiment on the influence of class information on equivariant prediction. We include three methods: 1) equivariant prediction (baseline); 2) jointly minimizing equivariant and classification losses ("+cls"); 3) minimizing the equivariant loss while adversarially maximizing the classification loss [22] ("-cls"). We study rotation prediction for (a), (b), (c) and (d), horizontal flip for (e), and four-fold blur for (f).Remark: Extension to V-information. In the discussion above, we mainly adopt Shannon's entropy measures for simplicity, which ignores computational and modeling constraints. Computation-aware notions like V-information [75] would align our theory better with practice. Notably, if we replace Shannon information with V-information, the analyses above still hold (see Appendix C). A nice property of V-information is that feature extraction steps can increase the information by making prediction computationally easier (which only decreases information in Shannon's notion instead), providing better justification for the benefit of representation learning. Thus, from the perspective of V-information, in ESSL, class features increase the equivariance w.r.t. A with easier computation.Verification of synergy effects via controlled experiments. To validate the above analysis in practice, we further carry out a controlled experiment to study how class information affects the equivariant pretraining task. Specifically, taking the rotation prediction task as an example, we add or substitute a class prediction loss with an additional linear head in the pretraining objective. In the former case, we explicitly inject class information into the presentation by joint training with the classification loss; in the latter, we explicitly eliminate class information from the representation by adversarially maximizing the classification loss [22] (see Appendix B). As shown in Figure 3 , we get slightly better rotation prediction accuracy when explicitly incorporating the class information, while getting worse performance (with a larger margin) when discouraging class information, which agrees well with Theorem 1. Note that there is still nontrivial training accuracy because the class is not the only factor that can explain equivariant prediction (style features S can also play a role).Our theoretical understanding above not only establishes theoretical explanations for downstream performance, but also provides principled guidelines for E-SSL design. The overall principle is to maximize the synergy I(A; C|X) = H(A|X) -H(A|X, C), which can be understood from the following aspects that explain various E-SSL behaviors that we observe in Section ??. Principle I: "Lossy" Transformations. First, let us look at H(A|X), which determines the upper bound of the explaining-away effect. A higher H(A|X) means that the equivariant prediction task is inherently harder. Revisiting Proposition 1, our theory gives a natural and rigorous explanation for why direct concatenation (DC) fails for E-SSL. Essentially, the DC output X = [A, X] admits a simple linear encoder such that A can be perfectly recovered from X, implying H(A|X) = 0, which leads to I(A; C|X) = 0, i.e., no explaining-away effect. This implies an intriguing property of E-SSL, that in order to attain nontrivial performance on downstream tasks, the chosen transformation T must be lossy -in the sense that one cannot perfectly infer A after the transformation, i.e., H(A|X) > 0. 6Considering computational and model constraints in practical scenarios, this task should be at least hard for the chosen training configuration (i.e., H V (C|X) > 0). Only when the transformation is hard enough, neural networks will strive to learn class information to assist its prediction. Indeed, Figure 1 shows that the transformations whose training loss decreases very quickly (e.g., grayscale and jigsaw) indeed have relatively poor test accuracy, which further verifies our theory.Principle II: Class Relevance. Aside from task hardness, we also need to ensure H(A|X, C) is low enough; i.e., extracting class information can effectively improve equivariant prediction. As a negative example, with a direct concatenation X = [C, A] as in Proposition 1, even if we add noise to A such that H(A|X) > 0, extracting the class C is still unhelpful for predicting A. From an information-theoretic perspective, it satisfies H(A|X) = H(A|X, C), so we always have I(A; C|X) = 0. In Figure 1 , horizontal flip and four-fold blur have large training losses until the end of the training, even if we deliberately inject class features (see Figures 3e & 3f ). This suggests that these equivariant tasks are intrinsically hard and class information does not contribute much to equivariant prediction. Instead, rotation prediction and vertical flip are hard at the beginning, but the uncertainty can be decreased significantly via learning class information. These transformations thus have a large synergy effect that benefits downstream performance. We conjecture that it is because these transformations are global (i.e., changing pixel positions) instead of local changes (i.e., only modifying pixel values) like grayscale and color inversion, so class information as global image semantics are more helpful for such tasks. Another important implication is that the transformation should be class-preserving so as to make class features helpful for the equivariant task. This rule has been verified extensively in contrastive learning [58, 62, 35, 68] . 2 ), class C and style S features jointly determine the raw input X. According to our theory, style features may also explain the equivariant target A (i.e., I(A; S|X) > 0). Since style features are often easier for NN learning [28] , they can become shortcuts for equivariant prediction such that class features are suppressed [28, 57] . Therefore, to ensure the learning of class-related semantic features, it is important to avoid these shortcuts. One effective approach to corrupt these style features (to some extent) through aggressive data augmentation, e.g., color jitter, cropping, and blurring commonly adopted in contrastive learning, without corrupting class features a lot. Indeed, Chen et al. [13] show that the choice of data augmentations plays a vital rule in the success of contrastive learning, and Tian et al. [62] point out its goal is to prune class-irrelevant features. Here, we generalize this principle to E-SSL as well through our explaining-away framework. As shown in Table 1 , the aggressive data augmentations from SimCLR also bring much better performance for E-SSL methods, bringing RotNet (82.26%) close to SimCLR (89.49%). It demonstrates that instead of merging with contrastive learning as in all recent E-SSL works [67, 17, 18, 25, 54, 34] , learning from equivariance alone can potentially achieve competitive performance.The theory in Section 4.1 guarantees that E-SSL will learn class features almost surely under general conditions. Yet, without further knowledge, it is generally hard to derive more quantitative results for downstream performance. For a concrete discussion, we consider a simplified data generation process as an exemplar. Note that simplified data models are frequently adopted in the literature of self-supervised learning theory [63, 71] to gain insights for their real-world behaviors.Setup. We consider a simple combination of the class C and the augmentation A as a weighted sum,EQUATIONwhere λ ∈ R is the mixing coefficient. Here, we assume a balanced class setting, where C ∼ Cat(N C ) follows a uniform categorical variable over N C classes: 0, 1, . . . , N C -1. Similarly, we assume that the augmentation A ∼ Cat(N A ) is an independent uniform categorical variable over N A classes: 0, 1, . . . , N A -1. In this simple setting, it is easy to see that given X, when C is known, we will have a perfect knowledge of A as A = X -λC, indicating H(A|X, C) = 0. Therefore, we have I(A; C|X) = H(A|X). In other words, transformations only influence the explaining-away effect through the uncertainty of predicting A. This is an extreme case for the ease of theoretical analysis. Nevertheless, the following theorem shows that under this setup, we can have a quantitative characterization of the optimal choice of N A and λ that sheds light on the design of E-SSL methods. Theorem 2. The following results hold for the additive problem in Eq. ( 3):1) Balanced Mixing is Optimal. With N C and N A held constant, I(A; C|X) is maximized under balanced mixing with λ = 1.2) Large Action Space is Beneficial.With N C held constant and λ = 1, we have a lower bound of the mutual informationI(A; C|X) ≥ ln N C - 1 N A (N C -1) ln N C -(N C -1) 2 N C ln (N C -1) + N C -2 2, which is monotonically increasing with respect to N A .Theorem 2 has two important implications. First, it suggests that a balanced mixing of A and C gives the optimal synergy effect, since it can maximize the uncertainty of using X for predicting A alone (agreeing with Principle I). Second, it shows that a large action space (|A|) is preferred, making it harder to use spurious features (e.g., the boundary values of C and A) as a shortcut to determine A (agreeing with Principles II and III). These theoretical results illustrate our analyses above and provide insights for understanding advanced designs in E-SSL methods, as elaborated below. 
understanding advanced e-ssl designs: 
 In Section 4, we have established a theoretical understanding of basic E-SSL through the explainingaway effect. However, basic E-SSL (like rotation prediction) often fails to achieve satisfactory performance, and many advanced designs have been proposed to enhance E-SSL performance [67, 17, 18, 25, 54, 34] . In this section, we further explain how these advanced designs improve performance by enhancing the synergy effect between class information and equivariant prediction.A conclusion from Theorem 2 is that a larger action space of the transformation A benefits the explaining-away effect by increasing the task difficulty H(A|X). Guided by this principle, one way to improve E-SSL is through learning from more fine-grained equivariance variables with a larger action space (|A|), which encourages models to learn diverse features and avoid feature collapse for specific augmentations. For example, four-fold rotation is a 4-way classification task while CIFAR-100 has 100 classes. When the neural networks are expressive enough such that it clusters samples with the same augmentation to (almost) the same representation (known as neural collapse [53] ), the class features also degrade or vanish, which hinders downstream classification. For example, Table 1 shows that for rotation prediction, stronger augmentations suffer from less feature collapse (lower training accuracy), while enjoying better classification accuracy. Indeed, we show that the advantages of state-of-art SSL methods can be understood through this information-theoretic perspective.Information-theoretic Understanding of Instance Discrimination. As disclosed in Section 3, contrastive learning is essentially an E-SSL task with equivariance prediction of instances. Specifically, each raw example xi serves as an instance-wise class, forming an action space I, where all augmented samples of xi belong to the class i. Therefore, the instance classification task has an action space of |I| = N , where N is the number of training dataset that is much larger than rotation prediction with |A| = 4, making instance discrimination a harder task, especially under strong data augmentations [72, 21] . Since the instance index I is also independent of the class variable C, it is not fully clear why it is helpful for learning class-relevant features. 7 Instead, our explaining-away theory gives a natural explanation from the instance classification perspective. In this way, our explanation of E-SSL can be regarded as a unified understanding of existing SSL variants.Equivariance Beyond Instance. Although contrastive learning already adopts a very large action space with |I| = N , there is recent evidence showing that it can still learn shortcuts [57, 73] and lack feature diversity [70] . Therefore, it is natural to consider even finer-grained equivariance, such as learning to predict patch-level or pixel-level features [2] , inputs [39] , or tokenized patches [4] , which comprises many variants of SSL methods, ranging from MAE [39] , BERT [19] , to diffusion models [41, 61] . Here, either random mask [19] or Gaussian noise [41] can be viewed as random variables (similar to the rotated angle in rotation prediction) and is independent of the class semantics, so they fit into our theory as well. Features learned from these tasks do show more diversity in practice and benefit downstream tasks requiring fine-grained semantics [39, 42] . Therefore, our theory provides a principled way to understanding the benefits of fine-grained supervision in SSL.As discussed in Section 4.2, equivariant prediction may have class-irrelevant features as shortcuts, while corrupting these features (e.g., color) with data augmentation might affect certain downstream tasks (e.g., flower classification that requires color information too). A more principled way that has been explored recently is through joint prediction of multiple equivariance variables [67, 17, 18, 24, 54, 34] , which we refer to as multivariate equivariance. In the following theorem, we show that multivariate equivariance is provably beneficial since it monotonically increases the synergy effect between class information and equivariant prediction, as shown in the following theorem. Theorem 3. For two transformation variables A 1 , A 2 , we will always have I(A 1 , A 2 ; C|Z) ≥ max{I(A 1 ; C|Z), I(A 2 ; C|Z)}. In other words, multivariate equivariance brings strengthens the explaining-away effect, with a gain of g = max{I(A 2 ; C|Z, A 1 ), I(A 1 ; C|Z, A 2 )}.Theorem 3 can also be easily extended to more equivariant variables. Note that the gains of multivariate equivariance I(A 2 ; C|Z, A 1 ) reflects the amount of additional information that the class information C can explain away A 2 under the same value of A 1 ; therefore, more diverse augmentations provide a large gain in the synergy effect. Recent works on image world model show that equivariance to multiple transformation delivers better downstream performance and outperforms invariant learning [26] .Apart from the design of transformations that is the main focus of E-SSL methods, an often overlooked part is the equivariance of the backbone models, which we call model equivariance. Intriguingly, we find that equivariant networks can be very helpful for E-SSL when the transformation equivariance aligns well with model equivariance.Setup. We compare a standard non-equivariant ResNet18 [36] and an equivariant ResNet18 (EqRes-Net18) w.r.t. the p4 group (consisting of all compositions of translations and 90-degree rotations) [16] of similar parameter sizes. The models are pretrained on CIFAR-10 and CIFAR-100 for 200 epochs with rotation prediction, and then the learned representations are evaluated with a linear probing (LP) head for downstream classification (details in Appendix B). Note that a rotation-equivariant model does not necessarily predict rotation angles perfectly, since in E-SSL, the model only has access to the transformed input but not the ground-truth transformation.As shown in Table 2 and 4 (see Appendix B), we find that equivariant models bring significant gains for rotation prediction by more than 20% on CIFAR-10 and CIFAR-100. Under aggressive data augmentations (e.g., SimCLR ones), equivariant models provide better equivariant prediction of rotation with high accuracy (91.98% v.s. 83.26% on CIFAR-10 and 82.69% v.s. 68.29% on CIFAR-100), which also yields better performance on downstream classification with 23.20% and 26.46% higher accuracy respectively. Even more surprisingly, with mild augmentations (no or crop&flip), both models achieve perfect rotation prediction, while equivariant models can still improve classification accuracy a lot.Therefore, we find that under compatible equivariance, equivariant models have significant advantages for E-SSL in terms of both self-supervised pretraining (better pretraining accuracy) and downstream generalization (best classification accuracy). The following theorem justifies this point by showing that the mutual information w.r.t. the transformation A lower bounds the mutual information w.r.t. the classification C. Therefore, given the same equivariant task (e.g., same data augmentations), features with better equivariant prediction (larger lower bound) will also have more class information.Theorem 4. For any representation Z, its mutual information with the equivariant learning target A lower bounds its mutual information with the downstream task C as follows:EQUATIONHere, a small gap I(X; A|C) means a better generalization between these two tasks. Because I(X; A|C) = H(A|X, C) is a lower bound of I(A; C|X) that indicates class relevance, it further justifies our Principle II (Section 4.2) that better class relevance brings better E-SSL performance. Mathematically, an exact definition of equivariance requires that for each transformation a in the input space, there is a corresponding transformation T a in the representation space so that f (a(x)) ≈ T a f (x). Common rotation prediction objectives do not satisfy this property. Other works also study the use of exact equivariant objectives. Here, we take CARE [34] as an example and compare it against rotation prediction with cross-entropy (CE) loss.  
conclusion: 
 In this paper, we have provided a general theoretical understanding of how learning from seemingly irrelevant equivariance (such as, random rotations, masks and instance indices) can benefit downstream generalization in self-supervised learning. Leveraging the causal structure of data generation, we have discovered the explaining-away effect in equivariant learning. Based on this finding, we have established theoretical guarantees on how E-SSL extracts class-relevant features from an informationtheoretic perspective. We also identify several key factors that influence the explaining-away. Since this work is theory-oriented to fill the gap between practice and theory by investigating how E-SSL works, we do not explore extensively for a better E-SSL design. Nevertheless, the fruitful insights developed in this work could inspire more principled designs of E-SSL methods in future research. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The differences are often significantly large and the exact performance is not the primary concern of this work.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This is a theory-oriented paper with no societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not involve such models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the authors of the models and the dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Proof of Proposition 1Proof. It is easy to see that the linear encoder that takes the last d ′ dimension of the input does not rely on any class information while giving a perfect prediction of A, i.e., f (X) = X [d+1:d+d ′ ] = A. Therefore, it gives random guess prediction on downstream classification.Proof. We begin by restating an important result in probabilistic graphical models (PGMs) for conditional independence.Lemma 2 (Theorem 3.5 (rephrased) [44] ). For almost all distributions P that factorize over the causal diagram G, that is, for all distributions except for a set of measure zero in the space of CPD (conditional probability distributions) parameterizations, we have that I(P ) = I(G), where I(G) denotes the set of independencies that correspond to d-separation:I(G) = (X ⊥ Y | Z) : d-sep G (X; Y | Z) .Lemma 2 shows that almost all distributions that factorize over the causal diagram G obey the d-separation rules. According to d-separation [27, 44] , the collider structure in Figure 2 implies that A ̸⊥ ⊥ C|X and A ̸⊥ ⊥ C|Z. Further, recall that for any three random variables A, B, C, the mutual information I(A; B|C) ≥ 0, and I(A; B|C) = 0 iff they are conditionally independent, i.e., A ⊥ B|C. Combined the facts above, we will have I(A; C|X) > 0 and I(A; C|Z) > 0 almost surely.Proof. Leveraging Lemma 1, we know that I(A; C|Z) holds almost surely, which implies thatEQUATION)Since Z C is a reparameterization of C, we have H(A|Z, C) = H(A|Z, Z C ) = H(A| Z). Subtracting H(A) on both sides of Eq. ( 5) givesH(A|Z) -H(A) < H(A|Z I ) -H(A),which is equivalent to I(A; Z) > I(A; Z). Besides, we also have I(C; Z) = I(C; Z, Z C ) ≥ I(C; Z), which completes the proof.Proof. First, let us consider the first proposition of Theorem 2. We haveEQUATIONSince H(λC) = H(C) and H(A) and H(C) are determined by N A and N C , the goal is then transformed into minimizing H(A + λC). In order to address this problem, we first draw a table below to enumerate the possible outcome of A + λC. To determine H(A + λC), we can divide the elements in the table into groups, where they are distributed to the same group if and only if they have the same value. Let us assume that N A ≤ N C because the opposite condition can be solved in a similar way.0 1 2 ... N A -2 N A -1 λ 1 + λ 2 + λ ... N A -2 + λ N A -1 + λ 2λ 1 + 2λ 2 + 2λ ... N A -2 + 2λ N A -1 + 2λ ... ... ... ... ... ... (N C -2)λ 1 + (N C -2)λ 2 + (N C -2)λ ... N A -2 + (N C -2)λ N A -1 + (N C -2)λ (N C -1)λ 1 + (N C -1)λ 2 + (N C -1)λ ... N A -2 + (N C -1)λ N A -1 + (N C -1)λNote that the number of elements in a single group cannot be greater than N A , for at most one element in each column can be distributed to that group. Therefore, we denote x k as the number of groups that consist of k elements, k ∈ {1, 2, ..., N A }. Our target isH(A + λC) = N A t=1 x t t N A N C ln N A N C t .For simplicity, we further denotey k = 1 N A N C ln N A N C k , then H(A + λC) = N A t=1 tx t y t .Considering the total number of elements, we haveEQUATION)We now pay attention to the top row and the rightmost column, where the elements are bound to be in different groups. Under closer observation, we find that the element 0 forms a group alone, as does the element N A -1 + (N C -1)λ. The element 1 is in a group made up of at most two elements, as are the elements λ,(N A -1) + (N C -2)λ, (N A -2) + (N C -1)λ.Based on similar analysis, the following relationship can be deduced:EQUATION)We also haveEQUATIONEQUATIONThe equality condition for this inequality isEQUATION)This indicates that every secondary diagonal (from upper right to lower left) in the aforementioned table forms a group of elements, which means λ = 1. This completes the proof of the first claim.Let us further look at the second proposition. From the discussion above, we know that λ = 1 is the optimal value, and we want to maximizeI(A, C|X) = H(A) + H(C) -H(A + C). Let us assume that N A ≥ N C .In order to calculate in detail, we first list the probability distribution of A + C.EQUATIONNow, we haveEQUATIONThus,EQUATIONNote that(N C -1) ln N C -(N C -1) 2 N C ln (N C -1) + N C -2 2is greater than 0. Therefore, the lower bound of I(A, C|X) is monotonically non-decreasing with respect to N A . If N A is adequately large, I(A, C|X) approximates ln N C .Proof. The lower bound can be easily derived by taking the difference between the two quantities:EQUATIONwhere the last line comes from the information processing inequality.In this section, we detail the setting of each individual experiment in this work. All experiments are conducted with a single NVIDIA RTX 3090 GPU.In this experiment, we conduct equivariant pretraining tasks based on seven different types of transformations. In order to maintain fairness and avoid cross-interactions, we only apply random crops to the raw images before we move on to these tasks. We adopt ResNet-18 as the backbone with a two-layer MLP that has a hidden dimension of 2048 and an output dimension corresponding to the pretraining tasks. Under each transformation, we train the model for 200 epochs on CIFAR-10, with batch size 512 and weight decay 10 -6 . The detailed pretraining tasks are listed as follows.Horizontal Flip & Vertical Flip & Color Inversion & Grayscale. We randomly (i.e., with probability 0.5) apply the specific transformation to images and require the model to predict whether or not we have really done the transformation. In these cases, the output dimension is 2.Four-fold Rotation. We the images with equal probability (i.e., with probability 0.25) by 0°, 90°, 180°, and 270°and require the model to predict which rotation angle we have actually adopted.In this case, the output dimension is 4.Four-fold Blur. We apply Gaussian blurs to the images using kernel sizes of 0, 5, 9, and 15, where kernel size 0 refers to not applying Gaussian blurs. We then require the model to predict the kernel size. In this case, the output dimension is 4.Jigsaw. We divide the images into 2 × 2 patches, randomly shuffle their order, and then require the model to predict the original arrangement. In this case, the output dimension is 24, since there are 4! = 24 possible permutations for the shuffled arrangements.During the pretraining tasks, we simultaneously train a classifier, which is a single-layer linear head and is trained without affecting the rest of the network. Apart from the seven tasks, we also conduct a baseline experiment, where we fix a random encoder and optimize the classifier alone in order to assess the effectiveness of these pretraining tasks.In this experiment, our goal is to figure out how class information affects rotation prediction. Figure 4 demonstrates the outline of the model we use to conduct this experiment. We apply random crops with size 32 and horizontal flips with probability 0.5 to the raw images.Training objectives. As for the experiment process, we first use rotation prediction as the pretraining task with a cross-entropy loss between our predicted angles and the actual angles, defined asEQUATIONwhere N is the image number, the one-hot vector p ij refers to the true rotation angle of the i th image, and pij refers to the prediction of the model. In the case where class information is incorporated, we simply add to the original loss function the cross-entropy between the classes predicted by the classifier and their corresponding ground truth labels, defined asEQUATIONwhere N is the image number, C is the class number, the one-hot vector y ij refers to the true class of the i th image, and ŷij refers to the prediction of the model. In other words, when class information is injected, the loss function is L rot + λ 1 L cls , where the mixing coefficient λ 1 is a hyper-parameter. Furthermore, to eliminate class information from the first setting, we minimize the classifier loss with L cls , trying to probe class information in the representation; in the meantime, we optimize the encoder to maximize the classification loss, aiming to eliminate any class information that can be found by the classifier. In particular, we adopt a joint training objective for the encoder as L rot -λ 2 L cls , where the mixing coefficient λ 2 is also a hyper-parameter. This leads to a min-max optimization between the encoder and the linear classifier. We choose λ 1 = 0.5 and λ 2 = 9, under which the class features can be shown to benefit or harm rotation prediction.In the pretraining process, we mainly use Resnet-18 as the backbone with a two-layer MLP that has a hidden dimension of 2048 and an output dimension of 4, and a single-layer linear head as Furthermore, we are interested the effects of class information on the accuracy of the pretraining tasks based on other transformations such as horizontal flips and four-fold blurs that are regarded as intrinsically hard. We simply inject class information into the representation by adding a classification loss to the original loss as well. Slightly different from the operation in rotation prediction, we apply random resized crops before conducting the pretraining tasks to avoid interfering with the prediction in these tasks. The other details and parameters are the same as those in rotation prediction.In order to compare the performance of Resnet and EqResnet, we use rotation prediction as our pretraining task and obtain the linear probing results. We apply various augmentations to the raw images, such as no augmentation, a combination of random crop with size 32 and horizontal flip, and SimCLR augmentation with output size 32. To be more specific, SimCLR augmentation refers to a sequence of transformations, including random resized crop with size 32, horizontal flip with probability 0.5, color jitter with probability 0.8, and finally grayscale with probability 0.2.In these experiments, we predict rotation angles with a two-layer MLP that has a hidden dimension of 2048 and an output dimension of 4, and a single-layer linear head as a classifier. For each setting, we train the model for 200 epochs on CIFAR-10 and CIFAR-100 with batch size 128 and weight decay 5 × 10 -4 . The results on CIFAR-100 are displayed in Table 4 .In this section, we introduce V-information [75] , which is a computation-aware and model-aware extension of Shannon's notation that is more suitable for modeling neural representation learning. Then, we extend our theory and show that the main results still hold under V-information.inequality shows that CE loss is the variational upper bound of H(A|X). When θ is minimized at the minimum with P θ (A|X) = P d (A|X), we have L(θ) = H(A|X) as the perfect estimate. Since NNs are generally expressive approximators, we believe that the (converged) CE loss can serve as a good estimate for H(A|X).Variational estimates of I(A; C|X). Combing these two, we notice that a good estimate for I(A; C|X) is the difference between the cross entropy losses of the predictor P θ (A|X) and P θ (A|X, C), which is exactly what we studied in the verification experiment in Section 4.1. In other words, the gap of the CE losses (or similarly the accuracies) between "rotation" and "rotation+cls", can serve as a quantitative measure of the synergy effect I(A; C|X). A more computationally friendly way is to select only a small subset of samples and to train it shortly. As shown in Figure 3 , the gap is already exhibited at the early stage of training.NeurIPS Paper ChecklistJustification: We obey all aspects of the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).A rigorous definition of feature equivariance requires more restrictions; for example, the transformations must be invertible. In existing SSL literature, equivariance is (loosely) referred to as the property that features are aware of general input transformations (not necessarily invertible). We follow this convention in this paper.When taking a closer look, there could be multiple explaining-away effects in Figure2. For example, we have Ā → X ← A. In practice, it could mean that knowing the intrinsic rotation of an image helps predict the rotated angle A. Also, we have Ā → X ← C, meaning that knowing Ā is also helpful for predicting the class C. This is a more nuanced analysis of how predicting rotated angles A could finally help predict the class C.For rotation prediction, there exist samples whose rotated angle cannot be uniquely determined, such as, frogs and airplane. Thus, rotation is also a lossy transformation in this sense.Existing theories rely on strong assumptions between on data augmentation (such as, the augmentation does not change image classes), which is often violated in practice.[58,68]. 


Decision: This paper explores why and how equivariant self-supervised learning (e.g., rotation prediction) learns useful representations from an information-theoretic perspective. The authors effectively addressed the reviewers' concerns during the author-reviewer discussion phase, and after the discussion, all the reviewers unanimously support the acceptance of this paper. Although AC has some concern about the limited empirical support, AC thinks the merits of this paper outweigh the concern: this paper can be a useful guideline for researchers considering equivariant self-supervised learning techniques. Therefore, AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a theoretical and empirical study of the role of invariant and equivariant representations in self-supervised learning. While a number of works have been focused on learning equivariant representations, it remains unclear whether or not equivariance is beneficial in specific tasks. By studying how applying/predicting transformations in latent space is a task that requires class information (through the explaining away effect) the authors derive insights on the behaviour of existing methods and provide directions for future work, notably that invariance may not be necessary at all to obtain good representations.
strength_weakness: strength_weakness: The paper seems to focus on two distinct families of methods. Methods which are augmentation aware (e.g. RotNet) and methods which are truly equivariant, which are explicitly designed to learn the transformation between transformed data in latent space, bringing more structured representations. While this distinction may not affect the general ideas and theoretical arguments, it may impact empirical results and practical considerations. It would be good to have a discussion/comparison of these distinct approaches.

Most of the practical insights stemming from this work are qualitative. A conclusion such as "knowing C is beneficial to predicting A" is a useful general guideline, but doesn't directly translate to practical guidance. The results would be stronger with more quantitative assessments of class relevance (perhaps in another, more computationally friendly domain), i.e. estimating I(A;C|X). This should be seen more as an avenue for future work though.
questions: questions: In experiments such as figure 1, what is the training loss/method considered ?

Theorem 1 puts forward an argument which seems key to the success or not of equivariant SSL (when used without an invariant objective), "How much information about C do you need to know to predict A". Bringing this point (and class relevance in general) forward in section 3.1 would make the motivation from figure 1 clearer.
limitations: limitations: Limitations are addressed in section 6, focusing on the theoretical nature of the work and how it doesn't directly translate into improved methods. It would be good to elaborate further on the limitation of the analysis, such as how different instantiations of an equivariant object may lead to different behaviors, or the difficulty of characterizing quantities such as I(A;C|X) for various transformations.
paper_summary: paper_summary: This study contributes theoretical insights that enhance our understanding of conventional practices in SSL training. Building on these theoretical foundations and supported by experimental evidence, the study puts forward several principles for the practical implementation of equivariant self-supervised learning designs.
strength_weakness: strength_weakness: Experiments are conducted using smaller-scale models.

Typo in line 352.
questions: questions: Could the principles 1,2,3 be applied to other domains beyond vision-based SSL?
limitations: limitations: Please refer to the question.
paper_summary: paper_summary: This paper aims to reduce the gap between theory and practice for equivariant SSL, which refers here to the sub-family of discriminative SSL methods that do not enforce the invariance of representations to augmentations. The authors propose an explanation based on the "explain-away" (_“Explaining away” is a common pattern of reasoning in which the confirmation of one cause of an observed or believed event reduces the need to invoke alternative causes._) and conclude that this principle justifies strictly positive mutual information between the pre-task labels (e.g., the rotation angle) and the downstream task label (i.e., semantic class label) thereby explaining why the pretext and downstream tasks are not systematically misaligned.
strength_weakness: strength_weakness: My main concern regards the soundness of the contribution, the reasoning leading to this concern follows:
- in this work, the authors use the "explaining-away" principle which relies on the proposed latent variable model shown in Figure 2 to justify E-SSL performance. Authors consider a partitioning of the latent information in $C$ and $S$, the former encapsulates semantic information, the latter the non-semantic information (should hence include the object position & orientation, textures). 
- I believe a reasonable intuition as to why E-SSL (e.g., predicting the rotation angle of an image as a pretext task and then performing semantic classification) can perform better than a random encoder - as shown by authors in Figure 1 -, is because of a selection bias in images used for training: each semantic class/object is more likely to be depicted in a specific orientation (e.g., a person standing) than in any other orientation (i.e., a person upside down). This means that the pretext and downstream tasks are not completely misaligned: being aware of the class information allows one to make a better guess about the rotation angle than a random guess. If an object is as likely to be depicted in one orientation than any other orientation than the class label is uninformative to make a guess regarding the rotation angle. The difference in results between horizontal and vertical flip in Figure 1 supports that intuition. The explanation proposed by the authors does not support that intuitive explanation of E-SSL but instead seems to propose a overly-simplified 
- The explaining-away principle justify a strictly positive mutual information between $\bar{X}$ and $A$ which authors extend to a strictly positive mutual information between $\bar{C}$ and $A$. It is not obvious from the get-go how authors can theoretically justify this extension. 

To summarize, the two aforementioned points should be further discussed by the authors to confirm the soundness of the proposed explanation which seems to take shortcuts both from an intuitive and theoretical perspective. I would suggest discussing these points more thoroughly. 

Minor comments:
- Relevance of certain results: Theorem 2 is tied to an over-simplified mixing model ($ X = A + C$) 
- Rigor:
     - Notation: notations should be consistent throughout the paper (e.g., line 250 and 256, line 236/237 and line 240) and assumptions should be made clear (e.g., eq (3) and line 293, lambda=1 is not stated)
     - Principle III: refers to theoretical results - "style features may also explain the equivariant target A" - , which have not been explicitly shown (in section 4.1 or in the appendix)
    - Contrastive learning vs. Instance discrimination: while instance discrimination performs classification with a number of classes equivalent to the number of training instances, contrastive learning like SimCLR usually discriminates at a batch level hence I believe line 339 does not accurately describe contrastive learning but rather instance discrimination.  
- References for claims: "style features are often easier for NN learning" - line 260,
- Typos: line 352, 68
questions: questions: - Can you please clarify what is meant by "global" vs. "local" transformations (principle II)?
- Can authors rigorously justify the extension of the explaining-away principle from X to C?
limitations: limitations: Limitations are not explicitly discussed but do mention the use of simplified models (e.g., section 4.3) for theoretical purposes.
paper_summary: paper_summary: This paper provides a theoretical understanding of equivariant self-supervised learning (E-SSL) methods and their effectiveness. The authors propose an information-theoretic analysis that explains the synergy effect between class information and equivariant transformations, leading to improved downstream performance. They also identify three principles for E-SSL design: lossy transformations, class relevance, and shortcut pruning. The paper further discusses the importance of model equivariance and demonstrates its advantages in E-SSL. Overall, the contributions of this paper lie in the theoretical explanation of E-SSL and the guidance it provides for designing effective E-SSL methods.
strength_weakness: strength_weakness: 1. The experiments are conducted on few datasets which is not very convincing. 
2. There exist some typos in the paper, such as the “xIn” in line 352.
3. It is better for the authors to provide a design enhancement example and conduct experiments to prove the effectiveness of the example, as it only explains the success of previous E-SSL works.
questions: questions: 1. Is 89.49% mentioned in line 279 the ACC of the SimCLR framework during testing? Even with the enhancement of SimCLR, the task of predicting rotation only achieved a maximum effect of 59.06 in downstream tasks. These two values are not very close.
2. In the paragraph starting from line 328, it is believed that the action space of the instance discrimination task is N, but I think it is actually a combination of many binary classification tasks with an action space of 2. So I think contrastive learning is not a special case to equivariant tasks.
3. Principle 2 states that it is best to be category related, does it mean that category label information needs to be known during pre-training stage in SSL?
4. Why can predictive patches like MAE be considered as an equivariant SSL task?
limitations: limitations: Yes.

Comments: title: Answer to Rebuttal
Thanks
Thanks
Detailed elaboration on computing $I(A;C|X)$
comment: Thank you for your response. I have checked the rebuttal and tend to keep my score.
Thank you to the authors for their efforts in addressing my comments. 
I have re-read the paper and read the other reviews/rebuttals. 
I am happy to increase for the following reasons: 
- authors have cleared my concerns regarding the soundness of their theoretical results (i.e., use of explaining away effect)
- going through the material once again, I see better how my intuition can be explained by the author's work, I believe connecting the author's contributions with a higher level intuition the reader might have regarding E-SSL and why it works would benefit the paper and hope the authors will adjust the manuscript accordingly. 

Overall, I believe the paper contributes to a better understanding of E-SSL methods. 
I believe the manuscript should be updated to incorporate the distinction between CL and ID made by the authors in the rebuttal for completeness.
Thank you for the prompt response! We are very glad to hear that your concerns are now resolved in our rebuttal. We will be sure to incorporate these discussions in our revision.
Thank you for the prompt response and for appreciating our new experiments and discussions. We concur with you that these results would enhance the theoretical and empirical insights of the proposed understanding. We will be sure to incorporate these results in our revision.
Thank you for conducting the additional experiments and providing the detailed discussions in response to the question.
The authors' rebuttal has addressed and resolved previously raised concerns.
Thank you for the clarifications and experiments with CARE.
The experiments with CARE as well as the approximation of $I(A;C | X)$ are welcome as they help relate the theoretical insights to practical E-SSL methods as well as help in designing them.

I hope that all discussions can be added in a revised version of the manuscript.
The computation of I(A;C|X) requires knowledge of both the pretraining label $A$ and the class label $C$. When both are available, we can compute it following its decomposition $I(A;C|X)=H(A|X)-H(A|X,C)$. Estimating entropy in high-dimensional space is generally hard, and a common approach is to leverage variational estimates for the two entropy terms $H(A|X)$ and $H(A|X,C)$ using neural networks.

**Variational estimates of entropy**. Take $H(A|X)$ as an example. We can learn a  NN classifier $P_\theta(A|X)$ optimized by minimizing the cross-entropy $L(\theta)=-E_{P_d(X,Y)} P_{\theta}(A|X)$. The following inequality shows that CE loss is the variational upper bound of $H(A|X)$. When $\theta$ is minimized at the minimum with $P_\theta(A|X)=P_d(A|X)$, we have $L(\theta)=H(A|X)$ as the perfect estimate. Since NNs are generally expressive approximators, we believe that the (converged) CE loss can serve as a good estimate for $H(A|X)$.

$L(\theta)=H(A|X)+KL(P_d(A|X)\|P_\theta(A|X))\geq H(A|X)$ 

**Variational estimates of** $I(A;C|X)$. Combing these two, we notice that a good estimate for $I(A;C|X)$ is the difference between the cross entropy losses of the predictor $P_\theta(A|X)$ and $P_\theta(A|X,C)$, which is exactly what we studied in the verification experiment in Sec 4.1. In other words, the gap between the CE losses (or similarly the accuracies) between “rotation” and “rotation+cls”, can serve as a quantitative measure of the synergy effect $I(A;C|X)$. A more computationally friendly way is to select only a small subset of samples and to train it shortly. As shown in Figure 3, the gap is already exhibited at the early stage of training.