Title: Self-supervised Transformation Learning for Equivariant Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Equivariant Learning

Transformation Representation

Self-supervised Transformation Learning
Abstract: Unsupervised representation learning has significantly advanced various machine learning tasks. In the computer vision domain, state-of-the-art approaches utilize transformations like random crop and color jitter to achieve invariant representations, embedding semantically the same inputs despite transformations. However, this can degrade performance in tasks requiring precise features, such as localization or flower classification. To address this, recent research incorporates equivariant representation learning, which captures transformation-sensitive information. However, current methods depend on transformation labels and thus struggle with interdependency and complex transformations. We propose Self-supervised Transformation Learning (STL), replacing transformation labels with transformation representations derived from image pairs. The proposed method ensures transformation representation is image-invariant and learns corresponding equivariant transformations, enhancing performance without increased batch complexity. We demonstrate the approach’s effectiveness across diverse classification and detection tasks, outperforming existing methods in 7 out of 11 benchmarks and excelling in detection. By integrating complex transformations like AugMix, unusable by prior equivariant methods, this approach enhances performance across tasks, underscoring its adaptability and resilience. Additionally, its compatibility with various base models highlights its flexibility and broad applicability. The code is available at https://github.com/jaemyung-u/stl.

Self-supervised Transformation Learning for Equivariant Representations

Abstract

Unsupervised representation learning has significantly advanced various machine learning tasks. In the computer vision domain, state-of-the-art approaches utilize transformations like random crop and color jitter to achieve invariant representations, embedding semantically the same inputs despite transformations. However, this can degrade performance in tasks requiring precise features, such as localization or flower classification. To address this, recent research incorporates equivariant representation learning, which captures transformation-sensitive information. However, current methods depend on transformation labels and thus struggle with interdependency and complex transformations. We propose Selfsupervised Transformation Learning (STL), replacing transformation labels with transformation representations derived from image pairs. The proposed method ensures transformation representation is image-invariant and learns corresponding equivariant transformations, enhancing performance without increased batch complexity. We demonstrate the approach's effectiveness across diverse classification and detection tasks, outperforming existing methods in 7 out of 11 benchmarks and excelling in detection. By integrating complex transformations like AugMix, unusable by prior equivariant methods, this approach enhances performance across tasks, underscoring its adaptability and resilience. Additionally, its compatibility with various base models highlights its flexibility and broad applicability. The code is available at https://github.com/jaemyung-u/stl .

Keyword

Equivariant Learning, Transformation Representation, Self-supervised Transformation Learning

1 Introduction

Recently, unsupervised representation learning [15, 10, 34] has made remarkable strides in various machine learning tasks and is actively employed as the ground model. In particular to the state-ofthe-art computer vision models [4, 18, 3, 5, 48, 1], invariant representation learning utilizes various augmentations, hereinafter transformations , including but not limited to random crop, horizontal flip, color jitter, and Gaussian blur. Their objective is to embed the semantically same inputs obtained through transformation, based on the notion that semantic differences due to the various transformations are inconsequential. Unfortunately, although effective most of the time, it does not always guarantee performance gain in the target downstream tasks. For instance, in tasks such as localization or flower classification, applying random crop or color jitter for invariant learning degrades performance by diluting discriminative features regarding the position of the object or the color of the flower, respectively.

To circumvent such a problem while harnessing the benefits of invariant representation learning, the research has transitioned towards incorporating equivariant representation learning [8, 28, 36, 9, 13] alongside invariant representation learning. As learning an equivariant representation in parallel with an invariant representation requires the representation to be responsive to transformations, the objective of equivariant representation learning is to capture transformation-sensitive information. This can be achieved either by training the model to predict transformations through representation pairs as in E-SSL [8] and AugSelf [28], referred to as implicit equivariant learning , or by learning the corresponding equivariant transformations in the representation space as in SEN [36], EquiMod [9], and SIE [13], referred to as explicit equivariant learning . However, these methods face a major issue due to the requirement of transformation labels. Firstly, optimizing with a transformation label, each transformation is treated independently, disregarding interdependency among transformations. Consequently, each component in color jitter transformation is treated distinctively although they are related to each other in the sense that they are applying transformation in the color as shown in Figure 1 (a). Secondly, due to the limited expressiveness of transformation labels, it fails to encompass complex transformations such as AugMix [23], where the sequences of transformations are randomly applied with varying weights. After all, the reliance on the transformation label limits the performance gain in equivariant representation learning.

To address the aforementioned limitations, we propose Self-supervised Transformation Learning (STL) for learning representation of transformation, which enhances the potential of equivariant learning. In STL, the transformation label is replaced by the transformation representation derived from the representation pairs of original and transformed image. To ensure the transformation representation captures the information of transformation, we train it to be invariant to the same transformation applied to various images, similar to how contrastive learning trains image representation to be invariant to various transformations. Based on the derived transformation representations, we aim to learn the corresponding equivariant transformations in the representation space. To avoid the trivial solution when learning the equivariant transformation, we apply the transformation representation obtained from another image but with the same transformation. Through optimization, STL is able to learn the transformation representation with the same batch complexity as previous methods.

We demonstrate the effectiveness of STL in transfer learning across various datasets and tasks, including both classification and detection. Additionally, by incorporating AugMix, a complex transformation that is not feasible with existing equivariant learning, STL enhances performance across all tasks, showcasing its broad applicability. STL's compatibility with diverse base models, further highlights its versatility, as it achieves the highest average accuracy across foundational models. Extensive experiments and ablation studies further validate STL's ability to capture interdependencies among transformations in an unsupervised manner. This is evident not only in transformation representations and equivariant transformation clustering, which reveal nuanced relationships between transformations (see Figure 1), but also in its superior performance in transformation prediction, surpassing existing equivariant learning methods.

2 Preliminaries: Transformation Invariant and Equivariant Learning

Transformation as Group Action. A group G consists of elements and an operation that satisfies closure, associativity, the existence of an identity element e , and the existence of inverses for all elements. The group action of G on a set X is defined as a function · : G × X → X , which ensures the identity operation e · x = x for all x ∈ X and maintaining the compatibility of the operations ( g · h ) · x = g · ( h · x ) for all g, h ∈ G . In the context of image processing, transformations can be viewed as a group action where the group T consists of transformations, and the set X is the collection of images. For example, if R ∈ T is a rotation and F ∈ T is a flip transformation, applying R followed by F to an image x ∈ X is represented as F · ( R · x ) = F ( R ( x )) . This structure ensures that image transformations are consistent and reversible.

Transformation Invariant Representation. In the context of transformation as group action, the transformation invariance of an image representation obtained through an encoder f : X → Y , which maps the set of images X to representation space Y , is defined as follows: Let T be a group of transformations applied to an input image x ∈ X , with t ( x ) representing the image after transformation t ∈ T . The representation f ( x ) is invariant to all transformations in T if f ( x ) of an input image x remains unchanged even after applying any transformation t to x .

In self-supervised learning, leveraging transformation invariance is crucial for learning representations, as it aligns transformed input images within the representation space. The objective of transformation invariant representation learning L inv is formalized with a dissimilarity loss L as follows:

where L represents the dissimilarity metric between representations. In contrastive learning, L is instantiated as metrics like the InfoNCE [35] loss, which reduces the distance between representations of semantically similar inputs while increasing the gap between those of dissimilar inputs.

Transformation Equivariant Representation. Extending the concept of invariance, transformation equivariance ensures that an image representation changes predictably according to the applied transformation. The representation f ( x ) is equivariant to all transformations in T if there exists a group action ϕ : T × Y → Y on the representation space Y , ensuring that the application of t to an image x leads to a corresponding transformation ϕ ( t, f ( x )) in its representation.

The function ϕ ( t, · ) : Y → Y is referred to as the equivariant transformation on the representation space corresponding to the transformation t . When the equivariant transformation ϕ ( t, · ) becomes the identity, it implies transformation invariance, showcasing the model's insensitivity to transformations. In EquiMod [9] and SIE [13], equivariant transformation ϕ ( ˆ t, · ) is represented by a network parameterized by the corresponding transformation label ˆ t . This network is trained to associate each transformation t with its corresponding equivariant transformation ϕ ( ˆ t, · ) in the representation space. The objective of transformation equivariant representation learning L equi is formalized as follows:

3 Equivariant Learning with Self-supervised Transformation Learning

3.1 Equivariant Learning without Transformation Label

Transformation Representation. Instead of relying on transformation labels that require knowledge of the transformation group structure, we propose leveraging pairs of representations derived from original images and their transformed counterparts to implicitly represent transformations. We introduce an auxiliary encoder f T : Y × Y → Y T designed to process pairs of representations ( f ( x ) , f ( t ( x ))) , where f ( x ) and f ( t ( x )) are the representation of the original image and of the transformed image respectively. This encoder outputs a transformation representation y x t ∈ Y T , capturing the inherent transformation between the original and transformed images without explicit transformation labels.

TransformationInvariantLearning

fT

yt

TransformationEquivariantLearning

(Φ(y',f(x))，f(t(x))

p(yt, )

fT

Self-supervised Transformation Learning

Ltrans (c,αx',t) = L (

yyt

yt

t(c)

f

YT

Ft(x')

C

Equivariant Learning with Transformation Representation. In contrast to methods directly utilizing transformation labels to learn equivariant transformations in the representation space, our proposed approach substitutes labels with transformation representations derived from pairs of images. However, there is a risk of encountering a trivial solution when using the transformation representation derived from the representation pair of the same image for which the equivariant transformation is applied. Specifically, the equivariant transformation might simply output the same representation f ( t ( x )) = ϕ ( f T ( f ( x ) , f ( t ( x ))) , f ( x )) that was used to obtain the transformation representation. To address this issue, we propose using transformation representations y x ′ t derived from pairs of a different image x ′ to apply equivariant transformations. Therefore, the transformed representation of the image through equivariant transformation in the representation space can be expressed as follows:

̸

Leveraging this approach, it is possible to learn transformation equivariant representations without explicit transformation labels, through the following objective:

̸

3.2 Self-supervised Transformation Learning (STL)

We hypothesize that transformation representations, y x t , derived from an input image x and its transformed image t ( x ) , encode the transformation t independently of the input image x . Nonetheless, ensuring image-invariant encoding of t is not trivial.

̸

To address this, we introduce Self-supervised Transformation Learning (STL), which adapts contrastive learning for transformation representation. Contrary to contrastive learning, which aims to align representations f ( x ) of the same image under different transformations to promote transformation invariance, STL instead focuses on aligning transformation representations, y x t and y x ′ t , derived from different images x = x ′ subjected to the identical transformation t . The objective of STL is formalized as follows:

̸

̸

Hereinafter, equivariant learning using transformation representations learned through self-supervised transformation learning will simply be referred to as STL.

Aligned transfon

between differer

Images

3.3 Implementation Details

Dissimilarity Metric. We use the InfoNCE loss from SimCLR [4] for the formulation and implementation of STL. Our methodology is not limited to this model, as demonstrated by ablation study, which shows the feasibility of applying our approach across various self-supervised learning models, such as BYOL [18], SimSiam [5], and Barlow Twins [48], through straightforward extensions (see Appendix A). Like EquiMod [9], we employ specialized projectors g inv , g equi, and g trans to map representations into distinct embedding spaces Z inv , Z equi, and Z trans, aligned with the objectives of invariant, equivariant, and transformation representation learning. We adopt the InfoNCE loss as the dissimilarity metric across these spaces, similar to the approach in SimCLR. The InfoNCE loss function is defined as follows:

̸

where y represents representation of an input image, y + denotes the corresponding representation to align, sim ( · ) indicates a similarity function, and τ is a temperature scaling parameter. For simplicity, batch { y } i are omitted in the subsequent loss functions. We define three specific loss functions for invariant, equivariant, and transformation representation learning, each building on the InfoNCE loss:

Aligned Transformed Batch. To implement STL, we need transformation representations obtained from different images. Unlike typical batch configurations in contrastive learning, where different transformations are applied to each input image, we construct batches by applying identical transformations to image pairs, as illustrated in Figure 3. In our approach, each image undergoes two distinct transformations, denoted t and t ′ , but for simplicity, we consider only a single direction of transformation, treating t as equivalent to t ′ · t -1 . This aligned transformed batch configuration maintains the same computational complexity as typical contrastive

learning setups while preserving input diversity. It also increases the count of identical transformations applied across different images, which is essential for transformation learning, without diminishing input diversity. To assess computational costs, we measured forward-backward time over 1000 iterations following a 1000-iteration warm-up. With an auxiliary network and loss calculation, our approach required only about 10% more time per iteration than SimCLR, which focuses solely on invariant learning, as shown in Table 1.

Overall Objective. The overall framework of STL is shown in Figure 2. Using aligned transformed batch, along with the InfoNCE loss, we define the overall objective with the hyperparameters λ inv , λ equi and λ trans for balancing the respective losses as follows:

̸

4 Experiments

Baselines. We compare STL with implicit and explicit equivariant learning methods, using SimCLR [4] as the base invariant model. Implicit methods (E-SSL [8] and AugSelf [28]) learn equivariant representations via transformation prediction tasks. Explicit methods (SEN [36], EquiMod [9], and SIE [13]) use transformation labels for equivariant learning. All methods, including STL, are trained and evaluated with SimCLR as the base model. Experiments with other base models are included in the ablation study.

Datasets. We pretrain on STL10 [7] with ResNet-18 and ImageNet100 [39, 42] with ResNet-50, following the split in [42]. Evaluation spans 11 downstream classification tasks (CIFAR10/100 [27], Food [2], MIT67 [38], Pets [37], Flowers [33], Caltech101 [12], Cars [26], Aircraft [31], DTD [6], SUN397 [47]) with a linear transfer protocol [41]. For detection, VOC07+12 dataset [11] and the protocol from [48] are used. Dataset and protocol details are in Appendix B and C.

Setup. For STL and explicit baselines (SEN, EquiMod, and SIE), we use an equivariant transformation network with a hypernetwork based on SIE. In SEN, EquiMod, and SIE, the hypernetwork uses transformation labels; in STL, it leverages transformation representations. STL also includes a 3-layer MLP with a 512-dimensional hidden layer to encode 128-dimensional transformation representations from input pairs. Equivariant transformations include random crop and color jitter, with other transformations applied randomly, consistent with typical contrastive learning. The transformation prediction loss weight is set to 0.5 for implicit baselines, and the equivariant learning weight is set to 1 for explicit baselines. STL uses weights of 1, 1, and 0.2 for invariant, equivariant, and transformation learning losses, respectively. We apply AugMix [23] to evaluate STL's adaptability to complex transformations, incompatible with other methods. Details on transformation labels in standard equivariant learning are in Appendix D. All analyses and ablations, except main experiments, use STL10-pretrained models. Additional setup details are in Appendix E.

4.1 Main Results

Image Classification. To assess generalizability, we apply the linear evaluation protocol on various downstream tasks. As shown in Table 2, STL outperforms existing methods on 7 out of 11 datasets. With AugMix, a complex transformation combination, STL achieves the highest performance across all datasets, underscoring its ability to generalize across diverse transformations, even those without explicit labels, to improve generalization. In Table 3, STL shows a minimal trade-off on in-domain tasks compared to SimCLR, with only a slight decrease from 81.20% to 81.10%, a smaller trade-off than other explicit equivariant models. Combined with AugMix, STL reaches 81.64%, showing adaptability to complex transformations and further enhancing in-domain performance. Results on STL10-pretrained models are provided in the Appendix F.

Object Detection. We evaluate STL on the VOC07+12 object detection task. As shown in Table 4, STL outperforms the invariant learning model SimCLR, as well as AugSelf, a representative model for implicit equivariant learning, and EquiMod, a representative for explicit equivariant learning, across all metrics: AP all , AP 50 and AP 75 . STL achieves 51.95 in AP all , and with AugMix, further improves to 52.70 in AP all , 78.81 in AP 50 , and 57.76 in AP 75 , demonstrating robust adaptability to complex transformations and enhanced precision in localization.

4.2 Analysis

Transformation Representation. To evaluate learned transformation representations of STL, we assess both parameter prediction and type classification on test images with transformations used during pretraining, specifically crop and color jitter. Parameter prediction uses MSE loss to measure accuracy in predicting transformation specifics, while type classification assesses the model's ability to distinguish among transformation categories. As shown in Table 5, STL achieves the lowest MSE for crop, color jitter, and the combined metric, indicating precise capture of transformation details. In classification, STL attains 93.67% accuracy, surpassing other models, even though AugSelf incorporates parameter regression into its learning. This demonstrates STL's robust generalization in transformation learning without direct parameter supervision.

Additionally, a qualitative UMAP visualization of transformation representations from test image pairs reveals STL's understanding of transformation relationships. Figure 1(b) shows distinct clusters for each transformation type, including crop, brightness, contrast, saturation, and hue, with color-related transformations grouped closely, reflecting STL's capture of inter-relationships among similar transformations. Figure 4 shows that transformations with similar intensity levels are

positioned closer in the representation space, forming continuous representations that capture the intrinsic order of intensity within each transformation type. This structure indicates STL's effective learning of transformation intra-relationships, demonstrating its nuanced understanding of both type and intensity.

Transformation Equivariance. We evaluate the accuracy of STL's learned equivariant transformations in reflecting real transformations in image space. This evaluation involves applying 60 transformations per image from the STL10 test dataset, including standalone transformations like crop and color jitter, as well as the standard combinations used during training. For each transformed representation, we rank other representations by similarity, ordering them from closest to furthest. Metrics include Mean Reciprocal Rank (MRR), Hit@k (H@k), and Precision (PRE). MRR is defined as the mean of reciprocal ranks 1 /r , where r is the rank of the nearest representation corresponding to the designated transformation. Hit@k calculates the probability P ( r ≤ k ) that the correct transformation rank r is within the top k . PRE is the mean squared error between the representation from the top-ranked transformation and the actual transformation's parameter vector.

As shown in Table 6, STL outperforms previous methods in most metrics, indicating closer alignment of its equivariant transformations with real image transformations in representation space. Notable exceptions are crop H@5 and PRE, where other methods perform slightly better. Overall, STL captures both individual and combined transformations more faithfully, achieving robust alignment between representation and image transformations. Self-supervised transformation learning, by maintaining image-invariance, enhances input consistency for equivariant transformation learning and significantly improves alignment accuracy. Without this component, STL's ability to capture transformation nuances declines, underscoring the role of each component in the STL framework.

4.3 Ablation Studies

Loss Functions. We conduct an ablation study to analyze the impact of each loss function on STL's performance across image classification and transformation prediction tasks. Specifically, we examine the contributions of invariant learning ( L inv), equivariant learning ( L equi), and self-supervised transformation learning ( L trans) by selectively removing each loss term. Table 7 illustrates a clear tradeoff between invariance and equivariance in STL's performance. The Only Invariance configuration achieves high in-domain accuracy of 84.74% but suffers from low out-domain accuracy of 43.11% and limited transformation prediction capabilities, highlighting restricted generalizability. In contrast, Only Equivariance improves out-domain accuracy to 49.99% and achieves strong transformation prediction with an MSE of 0.02, indicating enhanced generalization and transformation awareness, albeit with a slight reduction in in-domain performance.

When L trans is omitted, the model maintains high in-domain accuracy but exhibits weak out-domain and transformation performance, suggesting that the absence of transformation representation learning leads to a focus on invariance. Excluding L inv improves out-domain accuracy to 48.62% and enhances transformation alignment by preventing collapse into pure invariance, although this comes at a moderate cost to in-domain accuracy. Without L equi, the in-domain performance decreases further as transformation learning alone lacks sufficient structure for alignment. Finally, the full STL, which incorporates all three losses, achieves the best balance, with superior performance across in-domain and out-domain tasks and optimal transformation prediction results. This configuration minimizes in-domain trade-offs while capturing a comprehensive view of transformations, ensuring alignment between representation and image transformations across both in-domain and out-domain tasks.

Transformations. Table 8 shows STL's flexibility and effectiveness across various transformations. STL achieves the highest mean accuracy across all settings, outperforming AugSelf and EquiMod in single transformations such as crop and color, as well as in combined transformations including all transformations, reaching mean accuracy scores of 47.70%, 48.51%, and 48.48%, respectively. These results highlight STL's robust equivariant learning across diverse transformation types, enabling strong generalization without constraints on specific transformations.

Base Invariant Learning Models. Table 9 demonstrates STL's compatibility with various base models such as SimCLR, BYOL, SimSiam, and Barlow Twins. STL consistently improves the mean accuracy across all base models, outperforming both AugSelf and EquiMod, with the highest overall mean accuracy achieved with BYOL at 54.44%. These results demonstrate that STL is compatible with different invariant learning frameworks, confirming its adaptability and effectiveness regardless of the underlying representation learning method.

5 Related Works

Transformation equivariant learning captures transformation-sensitive features by embedding transformation directly into the learning process, categorized into implicit and explicit approaches. Implicit learning predicts transformations by observing changes in representations, allowing models to infer transformation without directly modeling the functions. In contrast, explicit equivariant learning encodes transformations within the representation space, enforcing behaviors in learned representations that mirror input transformations. These approaches can be reviewed in detail in the Appendix G.

Implicit Equivariant Learning. In implicit equivariant learning, models learn to recognize applied transformations by observing changes in representations, enabling the representations to capture transformation-sensitive information. Notable methods include InfoMin [43], selecting optimal views to maintain relevant task information, and Prelax [45], aligning residual vectors for robust multi-view alignment. Similarly, E-SSL [8] and AugSelf [28] leverage transformation-aware auxiliary tasks to train models to preserve transformation-sensitive details, enhancing robustness by maintaining sensitivity to transformation variance.

Explicit Equivariant Learning. On the other hand, explicit equivariant learning directly encodes input transformation into the representation space, building equivariant transformations that operate consistently in representation space. AEAE [19] leverages group actions to embed transformation effects explicitly in the representation space, while SymReg [40] enhances transformation consistency by selecting optimal loss terms based on transformation group information. CARE [20] introduces rotational symmetry by aligning embeddings directly with input rotations, providing robust transformation encoding. Similarly, SEN [36] applies symmetric embedding networks to synchronize transformations in the input space with learned representations. In a more flexible approach, EquiMod [9] models equivariant transformations by conditioning transformation labels as inputs and dynamically adapting representations through a neural network. Building on these approaches, SIE [13] separates invariant and equivariant representations, using dedicated networks to distinctly capture transformation-sensitive and invariant aspects.

Applications of Equivariant Learning. The applicability of equivariant learning extends across various domains, including robotics, medical imaging, molecular modeling, and multimodal representation learning. By leveraging inherent symmetries within data, it enhances sample efficiency in robotic manipulation [44], improves accuracy in medical image processing [21], and boosts predictive performance in molecular data analysis [46]. In multimodal contexts, it achieves finer alignment by considering transformations within each modality, facilitating robust cross-modal understanding [29, 14]. These applications underscore the broad utility of equivariant learning in aligning model representations with domain-specific transformations.

6 Discussion and Conclusion

The Self-supervised Transformation Learning (STL) approach introduces a novel method for deriving transformation representations that capture equivariance without relying on predefined transformation labels. This framework optimizes representational versatility through a synergy of three key loss functions: invariant learning, equivariant learning, and self-supervised transformation learning. These loss functions allow representations to adapt to complex transformation dynamics, including interdependencies among transformations, thereby greatly enhancing model generalization. STL consistently outperforms existing methods across 7 out of 11 classification tasks and demonstrates exceptional performance in object detection, proving its strong ability to generalize across diverse transformations. Additionally, integrating STL with AugMix yields robust performance improvements across all tasks, demonstrating enhanced resilience. STL's adaptability and consistent performance across various transformations and base models underscore its versatility for broad applications.

Limitations. While STL significantly advances equivariant learning, it encounters challenges with transformations that extend beyond single image pairs. Complex transformations, such as those involving combinations or mixtures of multiple images (e.g., mixup [49]), fall outside STL's current capacity as it relies on pairwise transformation representations. Further research could explore ways to adapt STL to accommodate more complex, multi-image transformations and better capture their inherent structure.

Broader Impacts. STL holds promise for applications requiring precise and interpretable transformations, such as in medical imaging analysis and autonomous driving. However, as STL learns transformation representations from the data, it may inherit biases embedded in the training data, raising fairness concerns, especially in sensitive domains. Implementing fairness-aware training techniques and thorough validation processes could help mitigate these risks. Additionally, while STL advances model robustness and generalization, its computational demands may have environmental implications. Efficiency improvements, such as model distillation, could reduce the model's energy footprint, supporting sustainable deployment.

Acknowledgments

This work was supported by Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (RS-2024-00439020, Developing Sustainable, Real-Time Generative AI for Multimodal Interaction, SW Starlab).

References

A. Bardes, J. Ponce, and Y. LeCun. Variance-invariance-covariance regularization for self-supervised learning. ICLR, Vicreg , 1:2, 2022.

L. Bossard, M. Guillaumin, and L. Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014.

M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33: 9912-9924, 2020.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

R. Dangovski, L. Jing, C. Loh, S. Han, A. Srivastava, B. Cheung, P. Agrawal, and M. Soljaˇ ci´ c. Equivariant contrastive learning. arXiv preprint arXiv:2111.00899 , 2021.

A. Devillers and M. Lefort. Equimod: An equivariance module to improve self-supervised learning. arXiv preprint arXiv:2211.01244 , 2022.

C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision , pages 1422-1430, 2015.

M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-338, 2010.

L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop , pages 178-178. IEEE, 2004.

Q. Garrido, L. Najman, and Y. Lecun. Self-supervised learning of split invariant equivariant representations. arXiv preprint arXiv:2302.10283 , 2023.

T. Geng, T. Wang, Y. Zhang, J. Duan, W. Guan, and F. Zheng. Uniav: Unified audio-visual perception for multi-task video localization. arXiv preprint arXiv:2404.03179 , 2024.

S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 , 2018.

R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 1440-1448, 2015.

P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

X. Guo, E. Zhu, X. Liu, and J. Yin. Affine equivariant autoencoder. In IJCAI , pages 2413-2419, 2019.

S. Gupta, J. Robinson, D. Lim, S. Villar, and S. Jegelka. Structuring representation geometry with rotationally equivariant contrastive learning. arXiv preprint arXiv:2306.13924 , 2023.

A. Hashemi, Y. Feng, and H. Sabet. Spherical cnn for medical imaging applications: Importance of equivariance in image reconstruction and denoising. ArXiv , 2023.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 , 2019.

S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

S. Kornblith, J. Shlens, and Q. V. Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2661-2671, 2019.

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

H. Lee, K. Lee, K. Lee, H. Lee, and J. Shin. Improving transferability of representations via augmentationaware self-supervision. Advances in Neural Information Processing Systems , 34:17710-17722, 2021.

J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and J. Kim. Uniclip: Unified framework for contrastive language-image pre-training. Advances in Neural Information Processing Systems , 35:1008-1019, 2022.

I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing , pages 722-729. IEEE, 2008.

M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision , pages 69-84. Springer, 2016.

A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

J. Y. Park, O. Biza, L. Zhao, J. W. van de Meent, and R. Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204.11371 , 2022.

O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

A. Quattoni and A. Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition , pages 413-420. IEEE, 2009.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

M. Shakerinava, A. K. Mondal, and S. Ravanbakhsh. Structuring representations using group invariants. Advances in Neural Information Processing Systems , 35:34162-34174, 2022.

C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision , pages 843-852, 2017.

Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 776-794. Springer, 2020.

Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? Advances in neural information processing systems , 33:6827-6839, 2020.

D. Wang, M. Jia, X. Zhu, R. Walters, and R. Platt. On-robot learning with equivariant models. arXiv preprint arXiv:2203.04923 , 2022.

Y. Wang, Z. Geng, F. Jiang, C. Li, Y. Wang, J. Yang, and Z. Lin. Residual relaxation for multi-view representation learning. Advances in Neural Information Processing Systems , 34:12104-12115, 2021.

R. Winter, M. Bertolini, T. Le, F. Noé, and D.-A. Clevert. Unsupervised learning of group invariant and equivariant representations. Advances in Neural Information Processing Systems , 35:31942-31956, 2022.

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010.

J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

H. Zhang. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

A STL Formulations for Various Base Invariant Models

A.1 STL Extension on BYOL

In adapting STL to BYOL [18], we utilize dissimilarity loss of BYOL to define the invariant, equivariant, and transformation losses. BYOL's dissimilarity metric is:

where g θ denotes the projection network parameterized by θ , q θ is the prediction network also parameterized by θ , and g ξ represents the target network parameterized by ξ . The terms q θ and g ξ refer to the normalized outputs of q θ and g ξ , respectively. Using this, we define the STL objectives as follows:

A.2 STL Extension on SimSiam

For SimSiam [5], STL uses SimSiam's dissimilarity loss:

where g denotes the projection network, h is the prediction network, stopgrad indicates an operation that halts gradient backpropagation, and D represents cosine similarity. This enables us to structure the STL losses as:

A.3 STL Extension on Barlow Twins

For Barlow Twins [48], STL applies Barlow Twins' dissimilarity loss:

̸

where C denotes the cross-correlation matrix between embeddings of transformed views Y and Y + , and λ is a regularization parameter that controls the weight of off-diagonal terms in C , penalizing redundancy in the representations. Define X = { x i } i , X ′ = { x ′ i } i , and T = { t i } i , representing the sets of input images, paired images, and transformations, respectively. Then, the STL losses for Barlow Twins are:

B Datasets

Table 10 presents detailed descriptions of (a) pre-taining dataset, (b) linear evaluation datasets, and (c) object detection and instance segmentation dataset. For linear evaluation dataset, validation samples are randomly selected from the training split if an official validation split is not provided. For the object detection and instance segmentation dataset, we use trainval set for training VOC07+12 [11] while only using test set for testing.

C Evaluation Protocols

Linear Evaluation. Following established protocols [25, 4, 18], we train linear classifiers on frozen representations from center-cropped images resized to 224 × 224 (or 96 × 96 for STL10). No data augmentation is applied. Each image is resized along its shorter side to 224 and then centercropped to 224 × 224 . The ℓ 2 -regularized cross-entropy objective is minimized using L-BFGS, with regularization selected from 45 logarithmically spaced values between 10 -6 and 10 5 on the validation set. The optimal model is then retrained on both training and validation splits, with test accuracy reported. L-BFGS is capped at 5,000 iterations, with each step initialized from the previous solution.

Object Detection. We use the VOC2007+2012 trainval set with 16,551 images to train a Faster R-CNN [16] with a C-4 backbone. Training spans 24,000 iterations with a batch size of 16 and SyncBatchNorm. The learning rate starts at 0.1, decreasing by a factor of 10 at 18,000 and 22,000 iterations. A linear warmup [17] is applied for the first 1,000 iterations with a 0.333 slope.

D Transformation Labels

In this section, we describe the labels of the transformations utilized in AugSelf [28] (random crop, horizontal flip, color jitter, grayscale, and Gaussian blur) in the following. The labels are designed by the parameters in each transformation.

RandomResizedCrop . The labels are constructed with the center points, H center and W center , of the crop and the height H and width W values of the cropping area.

RandomHorizontalFlip . As flip transformation is a simple operation, the label is 0 or 1.

ColorJitter . The four parameters of color jitter are brightness, contrast, saturation, and hue. Each parameter has its own range, with brightness, contrast, and saturation ranging from 0.0 to 1.0, while hue ranges from 0.0 to 0.5.

RandomGrayscale . This transformation applies the grayscale to the image. In line with the flip, the label is 0 or 1.

GaussianBlur . The standard deviation is utilized for the Gaussian blur label ranging from 0.1 to 2.0.

E Pretraining Setups

For the pretraining experiments, we use NVIDIA RTX4090.

E.1 ImageNet100 Pretraining

We conduct pretraining on the ResNet-50 architecture [22] using ImageNet100, a subset of ImageNet containing 100 categories [39], with dataset splits consistent with those in [42]. All methods are trained for 500 epochs with a batch size of 256, using a cosine learning rate schedule without restarts [30]. The initial learning rate is set at 0.03, with a weight decay of 0.0005. The model includes a 3-layer projection MLP head, g ( · ) , with a hidden dimension of 2048 and an output dimension of 128. Batch normalization [24] is excluded from the last layer.

E.2 STL10 Pretraining

For pretraining on the STL10 dataset [7], we use the standard ResNet-18 architecture [22]. All methods utilize stochastic gradient descent (SGD) with a learning rate of 0.03, a batch size of 256, a weight decay of 0.0005, and a momentum of 0.9. The learning rate follows a cosine decay schedule without restarts [30].

SimCLR [4]. A 3-layer projection MLP head g ( · ) with a hidden dimension of 512 and an output dimension of 128 is used, with batch normalization excluded from the final layer. In contrastive learning, we apply a temperature scaling parameter of 0.2.

Barlow Twins [48]. A 2-layer projection MLP head g ( · ) is employed, with a hidden dimension of 512 and an output dimension of 2048. Batch normalization is excluded from the last layer.

BYOL [18]. The model uses a 2-layer projection MLP head g ( · ) , with a hidden dimension of 4096 and an output dimension of 256, omitting batch normalization in the final layer.

SimSiam [5]. We employ a 2-layer projection MLP head g ( · ) with both hidden and output dimensions of 2048, with batch normalization excluded from the final layer.

F Image Classification Results of STL10-pretrained Models

Table 11: Image Classification. Evaluation of representation on in-domain and 11 downstream out-domain classification task: Linear evaluation accuracy (%) of ResNet-18 [22] pretrained on STL10 [7] averaged over three random seeds (mean ± std).

G Explicit and Implicit Equivariant Learning

Self-supervised Transformation Learning

Transformation Prediction

Transformation Invariant Learning

Self-supervised Transformation Learning

Transformation Equivariant Learning

Transformation Invariant Learning

(a) Explicit Equivariant Learning

(b) Implicit Equivariant Learning

Linv(x,t）=C(f(x)，f(t（x)))

Linv(x,t)=C(f(x),f(t（x))

Lequi(x,t)=L(Φ(t,f(x))，f(t(x))

Cequi(x,t) =C(fr(f(x),f(t(x).0)

o(i,-)

（y)

Ltrans(z,x',t)= L (yyt

Ctrans (x,a',t)= C(yy

Figure 5 illustrates the framework of explicit and implicit equivariant learning. Explicit methods like SEN [36], EquiMod [9], and SIE [13] apply a direct equivariant transformation network to representations, leveraging transformation labels for alignment. In contrast, implicit methods, such as E-SSL [8] and AugSelf [28], infer transformation effects without directly applying transformations by utilizing auxiliary tasks to deduce transformation states.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claim in the abstract and introduction is dealt with in the background, method, and experiment sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We deal with this in the Discussion and conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the hyperparameters, training, and evaluation protocols with details in the paper and supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: All training and evaluation are conducted based on publicly available codes, and we provide the details of our implementations.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We report all the details in the paper and supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The experiment merely contains negligible statistical effects.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe detailed information about the experiment in the paper and supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We wrote a paper in compliance with the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We denote the negative societal impacts that the proposed method may have.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is concerned with the learning of expressive representations in a self-supervised fashion. In particular, it aims at learning representations that\n\nSTRENGTHS:\n- Presentation: The paper is clearly written, well-structured, easy to follow, and overall is in a good state \n- Proposed approach: the idea proposed by the authors is not ground-breaking but simple, makes sense, and is efficient. It applies the same reasoning used in invariant self-supervised learning to equivariant representation learning.\n- Evaluation pipeline: the authors evaluate their method quite thoroughly, evaluate semantic classification across a large number of datasets, evaluate object detection, and check the sensitivity to hyper-parameters and transformation prediction. \n- Experimental results: results show that the approach effectively learns representations that perform well on both semantic and non-semantic information in comparison to baselines. Sensitivity to hyperparameters seems limited.\n\nWEAKNESSES:\n- My main concern regards the baselines chosen for evaluation. Authors explicitly mention that SIE and SEN are excluded from the evaluation due to the need for these methods to have access to transformation labels which STL does not require. While I see how these methods require more \"supervision\" than the proposed method, STL still requires weak supervision (i.e., access to pairs of images that have undergone the same transformation). Cases, where one would have access to such weak supervision but not access to transformation labels, are quite rare (e.g., AugMix as mentioned by authors), as a consequence I think the evaluation should include these baselines, and results should be analyzed considering this light difference in supervision level. My score will be increased if this concern is addressed.\n- Experimental results are computed for a single seed, for object detection results, the performance gap between EquiMod and STL is narrow hence additional seeds would make the evaluation more robust. \n- In Table 1, STL+AugMix can be compared to SimCLR *with* AugMix augmentations which I do not believe is reported in this table.\n\nMinor:\n- line 40: typo\n\nQUESTIONS:\nNo questions but a suggestion for improvement, I think section 4.3, first part, is intriguing and it is not clear to me why equivariance and transformation learning are _complementary_. Discussing further the intuition behind these results would I think improve the paper.\n\nLIMITATIONS:\nLimitations are discussed and in particular, the main limitation I see to this work (i.e., the framework requires access to a pair of images that have undergone the same transformation, this weak form of supervision is not always accessible in the wild), is mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Self-supervised Transformation Learning (STL) to learn equivariant representations. The core of the method is to not use augmentation information but instead use transformation representations obtained from pairs of data. In this sense, instead of knowing the transformation information, it is necessary to have pairs of data with the same transformation applied to them. This allows STL to leverage more complex augmentation schemes such as AugMix to reach higher performance on classification and detection benchmarks.\n\nSTRENGTHS:\nThe non reliance on knowledge of the group elements leads to a method with different assumptions than previous works which can thus be used in different scenarios. Assuming knowledge of pairs of data with the same transformation can be a weaker or stronger assumption than knowledge of the group element depending on the domain, which makes STL complementary to existing works.\n\nAssuming that images are sampled randomly, using a pair of images to compute the transformation representation of another one ensures that information does not leak between the semantics of the image and the transformation representation. At inference time, applying the transformation does not require knowledge of pairs.\n\nThe performance gained using STL over existing methods is convincingly shown, across a wide range of tasks. We can however notice that aiming for equivariance over all augmentations does not lead to the best performance (Table 6),reinforcing previous findings that the benefits of invariance/equivariance can be task dependant\n\nWEAKNESSES:\n1\\) An important reference to [1] is missing from the paper. It was proposed in this paper to use pairs of data transformed by the same (but unknown) group element (see equation 4 for example) to learn equivariant representation. Although the considered experimental setting is different, and the way to leverage the pairs differs, it remains a very related method.\n\n2\\) With how the loss is designed, the representations are both aimed at being invariant and equivariant at the same time (albeit with a projector in between the loss and representations). As invariance is a perfectly valid solution when wanting equivariant solutions ($\\theta = Id$), it is possible that the learned representations are invariant to augmentations in the end, as illustrated in figure S2 of [2] for example. Currently, the metric defined in equation 15 doesn't provide enough information as to whether or not the predictor indeed applied transformation well (and thus whether or not the representations are equivariant).\n\nIntuitively (correct me if my understanding is wrong), the proposed metric looks at how much closer the predicted representation is to the target compared to its starting point. A value higher than 1 thus means that the prediction is closer to its target than its starting point. But here the values are barely around 1.1 which would suggest that while the predictor is indeed doing something positive, it's far from applying the transformations perfectly.\n\nCompleting this analysis with other common metrics such as the ones used in EquiMod[3] or metrics such as Hit at rank K, MRR, PRE[2,4,5] for example would provide more compelling evidence of the equivariance or not of the representations.\n\n3\\) Performance seems to not be reported on the pretraining dataset but only on other downstream tasks. It is important to report those numbers to understand the behaviour of the models both in and out of domain.\n\n\n\n\n\n[1] Shakerinava, Mehran, Arnab Kumar Mondal, and Siamak Ravanbakhsh. \"Structuring representations using group invariants.\" Advances in Neural Information Processing Systems 35 (2022): 34162-34174.\n\n[2] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" arXiv preprint arXiv:2302.10283 (2023).\n\n[3]Devillers, Alexandre, and Mathieu Lefort. \"Equimod: An equivariance module to improve self-supervised learning.\" arXiv preprint arXiv:2211.01244 (2022).\n\n[4]Kipf, Thomas, Elise Van der Pol, and Max Welling. \"Contrastive learning of structured world models.\" arXiv preprint arXiv:1911.12247 (2019).\n\n[5]Park, Jung Yeon, et al. \"Learning symmetric embeddings for equivariant world models.\" arXiv preprint arXiv:2204.11371 (2022).\n\nQUESTIONS:\nLine 42-43 \"with a transformation label, each transformation is treated independently, disregarding interdependency [...] each component in color jitter is treated distinctively although they are related to each other.\" I am not sure to see the point here, as this argument can be made for any image augmentation. Giving all transformation parameters (as well as their order if it isn't fixed) gives full information about the transformation. The transformations are fully independent of each other and at best they do not commute.\n\nLine 47-48 \"After all, the reliance on transformation labels limits the performance gain in equivariant representation learning\". This claim seems unsubstantiated, are there any references or concrete evidence to support this claim ?\n\nIn Equation 2 the notation lacks preciseness. If $\\mathcal{L}$ is an InfoNCE criterion, it cannot consider samples in isolation as is written in Equation 2, but instead needs knowledge of the full batch to be computed.\n\nWhen using Color Jitter for AugSelf and Equimod, is the order of the transformations fixed or randomly selected for each call, as is default in torchvision for example ? If it is not then the same labels can be associated with different transformations which will hinder their learning.\n\nWhen considering a large set of transformations, it is possible that the transformation representations only represent some of the transformations and not all of them, or that the predictor only uses partial information (e.g. applies hue change but not brightness change). Did you perform analyses on individual transformations to study the equivariance properties on individual transformations ?\n\nLIMITATIONS:\nThe authors address the limitations in section 6, notably how using pairs of transformed data is not a panacea, and can have issues to be extended to even larger transformation sets than considered with STL.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces STL, a method for learning self-supervised equivariant representations. It suggests replacing transformation labels with representations derived from data pairs. The proposed pretext task promotes learning invariant and equivariant representations alongside transformation-related information. The method's effectiveness is showcased through various classification and object detection tasks.\n\nSTRENGTHS:\n- The paper argues that existing methods treat each transformation independently as they require transformation labels, which disregards interdependency among transformations. The main strength of the paper lies in their method, which does not require transformation labels.\n- The complexity of transformations does not constrain the proposed method STL, as they demonstrate learning equivariance representations with complex transformations like AugMix.\n- Authors show empirically that the learned representations are competitive for classification and detection. They also show that STL learns equivariant representations with relational information between transformations.\n\nWEAKNESSES:\n- Experimental comparisons with some key existing equivariant SSL methods are missing [1, 2] and datasets (3DIEBench). \n\n- Missing several related works [3-6]. The authors should thoroughly review the relevant literature.\n\n- The approach seems computationally demanding since there are multiple (three) InfoNCE losses for each iteration. Additionally, the authors haven’t provided results/discussions on computational costs.\n\n- Information for reproducibility is limited, especially when extending STL to BarlowTwins, BYOL, and SimSiam.\n\n[1] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" *arXiv preprint arXiv:2302.10283* (2023). \n\n[2] Park, Jung Yeon, et al. \"Learning symmetric embeddings for equivariant world models.\" *arXiv preprint arXiv:2204.11371* (2022).\n\n[3] Gupta, Sharut, et al. \"Structuring representation geometry with rotationally equivariant contrastive learning.\" *arXiv preprint arXiv:2306.13924* (2023).\n\n[4] Xie, Yuyang, et al. \"What should be equivariant in self-supervised learning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.\n\n[5] Guo, Xifeng, et al. \"Affine Equivariant Autoencoder.\" *IJCAI*. 2019.\n\n[6] Gupta, Sharut, et al. \"Learning Structured Representations with Equivariant Contrastive Learning.\"\n\nQUESTIONS:\n- It’s known that InfoNCE loss can be prone to dimensional collapse [7]. Having multiple InfoNCE losses (especially when invariance and equivariance losses can be contradictory) in your method, have you observed dimensional collapse happening?\n- What influence does the size of the hypernetwork $f_T$ has over the downstream tasks? \n- How does different hyperparameter configurations of $\\lambda_{inv}(\\cdot)$,  $\\lambda_{equi}(\\cdot)$, and  $\\lambda_{trans}(\\cdot)$ affect the rate of convergence while pre-training?\n- How do you extend STL to asymmetric methods like BYOL and SimSiam? Are the InfoNCE criteria still kept for equivariance and transformation-related losses?\n- Any insights into how much contribution $\\lambda_{equi}$ and $\\lambda_{trans}$ have for learning the relational information between transformations?\n\n[7] Jing, Li, et al. \"Understanding dimensional collapse in contrastive self-supervised learning.\" *arXiv preprint arXiv:2110.09348* (2021).\n\nLIMITATIONS:\nThe authors have provided some limitations. It would be beneficial to include a discussion on computational costs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new way to learn equivariant representations by directly learns the transformation representation. They enforce that different transformations have their own input-agnostic representations. To obtain this, they learn an encoder that takes pairwise representations of the same image to extract the transformation representation. To avoid trivial solutions, they use different examples in transformation representation extraction and later transformation module, which helps disentangle the sample and transformation features. They further add an regularization on the sample consistency to encourage it to be sample-invariant. They combine it with invariant learning loss for training, and demonstrates its benefits on several tasks.\n\nSTRENGTHS:\n- I generally like the idea of disentangling sample and transformation representations in equivariant learning, and the paper proposes a clever patching-based objective to enforce it. \n- And the transformation representation they obtain in Figure 1 does make sense intuitively since it reflects the relative distance between different augmentations. \n- The evaluation demonstrates its advantages over previous works on linear probing and object detection tasks. The propose method is particularly superior on the transformation prediction task. I think it's because they successfully disentangle the features.\n\nWEAKNESSES:\n- The evaluation is not complete. I didn't find linear probing results (often the most important ones) on ImageNet-100 or STL-10.\n- Lack the comparison to a few relevant E-SSL baselines, such as [1,2].\n- Lack more in-depth analysis of the learned transformation representations. I have some doubts about it. Since there are many variations in each augmentation (eg cropping with different positions, ratios), what is the relationship of their representations in the latent space? Is there an Arithmetic relationship?\n\n\n[1] [Equivariant Contrastive Learning. ICLR 2022.](https://arxiv.org/abs/2111.00899)\n\n[2][ Residual Relaxation for Multi-view Representation Learning. NeurIPS 2021.](https://arxiv.org/abs/2110.15348)\n\nQUESTIONS:\nno\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}