Self-supervised Transformation Learning for Equivariant Representations
Authors: Jaemyung Yu
Jaehyun Choi
Dong-Jae Lee
HyeongGwon Hong
Junmo Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Equivariant Learning
Transformation Representation
Self-supervised Transformation Learning
Primary area: machine_vision
TL;DR: Self-Supervised Transformation Learning (STL) uses transformation representations from image pairs instead of labels to learn equivariant transformations, enhancing performance efficiently.

Abstract: Unsupervised representation learning has significantly advanced various machine learning tasks. In the computer vision domain, state-of-the-art approaches utilize transformations like random crop and color jitter to achieve invariant representations, embedding semantically the same inputs despite transformations. However, this can degrade performance in tasks requiring precise features, such as localization or flower classification. To address this, recent research incorporates equivariant representation learning, which captures transformation-sensitive information. However, current methods depend on transformation labels and thus struggle with interdependency and complex transformations. We propose Self-supervised Transformation Learning (STL), replacing transformation labels with transformation representations derived from image pairs. The proposed method ensures transformation representation is image-invariant and learns corresponding equivariant transformations, enhancing performance without increased batch complexity. We demonstrate the approach’s effectiveness across diverse classification and detection tasks, outperforming existing methods in 7 out of 11 benchmarks and excelling in detection. By integrating complex transformations like AugMix, unusable by prior equivariant methods, this approach enhances performance across tasks, underscoring its adaptability and resilience. Additionally, its compatibility with various base models highlights its flexibility and broad applicability. The code is available at https://github.com/jaemyung-u/stl.

Self-supervised Transformation Learning for Equivariant Representations
Jaemyung Yu!
Jaehyun Choi'
Dong-Jae Lee'
HyeongGwon Hong' Junmo Kim
Korea Advanced Institute of Science and Technology (KAIST)
(aaemyung, jhtwosun, hongguurnn junmo kim} (ain)0kaist.cc ac.
Abstract
Unsupervised representation learning has significantly advanced various machine learning tasks. In the computer vision domain, statc-of-the-as approaches utilize transformations like random crop and color jitter achieve invariant represen- tations, embedding semantically the same inputs despite transformations. How- ever, this degrade performance in tasks requiring precise features, such localization or flower classification. To address this, recent research incorporates equivariant representation learning, which captures transformation-seesiivve in- formation. However, current methods depend on transformation labels and thus struggle with interdependency and complex transformations. We propose Self- supervised Transformation Learning (STL), replacing transformation labels with transformation representations derived from image pairs. The proposed method ensures transformation representation is image-invariant and learns corresponding equivariant transformations, enhancing performance without increased batch com- plexity. We demonstrate the approach's 's effectiveness across diverse classification and detection tasks, outperforming existing methods out of benchmarks and excelling in detection. By integrating complex transformations like AugMix, unusable by prior equivariant methods, this approach enhances performance across tasks, underscoring its adaptability and resilience. Additionally, its compatibility with various base models highlights its flcxibility and broad applicability. The code is available at lttttt:///// :ttgps://giee giggttt..b....m com/ jammyugn u/stl
Keyword
Equivariant Learning, Transformation Representation, Self-supervised Transformation Learning
Introduction
Recently, unsupervised representation learning machine learning tasks and is actively employed as the ground model. In particular to the st 48 M. invariant representation learning utilizes various
10 34 has made remarkable strides in various [55
the-art computer vision models 18 3 augmentations, hereinafter transformations, including but not limited to random crop. horizontal flip. color jitter, and Gaussian blur. Their objective is embed the semantically same inputs obtained through transformation, based on the notion that semantic differences due to the various transformations are inconsequential. Unfortunately, although effective most of the time, does not always guarantee performance gain in the target downstream tasks. For instance, in tasks such as localization flower classification, applying random crop or color jitter for invariant learning degrades performance by diluting discriminative features regarding the position of the object or the color of the flower, respectively.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

EquiMod
STL (Ours)
STL (Ours)
Crop
Satur.
Color Jatter
Bright.
Hue
Conte.
(a) Equivariant Transformation
(b) Transtormation Representation
Figure 1: Visualization of Equivariant Transformation and Transformation Representation. (I.eft) UMAP
[32 visualizations of functional weights from equivariant transformations implemented with hypernetwrk. EquiMod uses transformation labels to generate these weights, while STI. derives them from the representation pairs of transformed and original image. (Right) UMAP visualizations of transformation representations obtained from representation pairs of original input image and transformed input image.
To circumvent such a problem while harnessing the benefits of invariant representation learning, the research has transitioned towards incorporating equivariant representation learning 8
28
36
9 [3]
alongside invariant representation learning. As learning equivariant representation in parallel with an invariant representation requires the representation to be responsive to transformations, the objective of equivariant representation learning is capture vaanformation-sensitvvve information. This can be achieved either by training the model to predict transformations through representation pairs in E-SSL [ and AugSelf (28), referred to as implicit equivariant learning, by learning the corresponding equivariant transformations in the representation space as in SEN (36), EquiMod (9], and SIE (13, referred to explicit equivariant learning. However, these methods face a major issue due to the requirement of transformation labels. Firstly, optimizing with a transformation label, each transformation is treated independently, disregarding interdependency among transformations. Consequently, each component in color jitter transformation is treated distinctively although they are related each other in the sense that they are applying transformation in the color shown in Figure (a). Secondly, due to the limited expressiveness of transformation labels, it fails to encompass complex transformations such as AugMix [23], where the sequences of transformations are randomly applied with varying weights. After all, the reliance on the transformation label limits performance gain in equivariant representation learning.
To address the aforementioned limitations, we propose Self-supervised Transformation Learning (STL) for learning representation of transformation, which enhances the potential of equivariant Iearn- ing. In STL, the transformation label is replaced by the transformation representation derived from the representation pairs of original and transformed image. To ensure the transformation representation captures the information of transformation, we train it to be invariant to the same transformation applied various images, similar to how contrastive learning trains image representation to be invariant to various transformations. Based on the derived transformation representations, we aim to learn the corresponding equivariant transformations in the representation space. To avoid the trivial solution when learning the equivariant transformation we apply the transformation representation obtained from another image but with the same transformation. Through optimization, STL is able to learn the transformation representation with the same batch complexity as previous methods.
We demonstrate the effectiveness of STL in transfer learning across various datasets and tasks, including both classification and detection. Additionally, by incorporating AugMix, a complex transformation that not feasible with existing equivariant learning. STL enhances performance across all tasks, showcasing its broad applicability. STL's compatibility with diverse base models, further highlights its versatility, as achieves the highest average accuracy across foundational models. Extensive experiments and ablation studies further validate STL's ability to capture interdependencies among transformations an unsupervised manner. This is evident not only transformation representations and equivariant transformation clustering, which reveal nuanced relationships between transformations (see Figure [] but also in its superior performance in transformation prediction, surpassing existing equivariant learning methods.

---

2 Preliminaries: Transformation Invariant and Equivariant Learning
Transformation Group Action. A group G consists of elements and an operation that satisfies closure, associativity, the existence of an identity element e, and the existence of inverses for all elements. The group action of G on set X is defined as function G X + X, which ensures the identity operation e - = x for all E X and maintaining the compatibility of the operations (g- h) x 9. (h-2)) for all g, h E G. In the context of image processing, transformations can be viewed as a group action where the group T consists of transformations, and the set X is the collection of images. For example, if R E T is rotation and F T is a flip transformation, applying R followed by F to an image I E X is represented as F (R - x) = F ([(x)). This structure ensures that image transformations are consistent and reversible.
Transformation Invariant Representation. In the context of transformation as group action, the transformation invariance of an image representation obtained through encoder X Y, which maps the set of images X to representation space Y, is defined as follows: Let T be a group of transformations applied an input image I E X, with t(x) representing the image after transformation / E T. The representation f(x) is invariant to all transformations in T if f(x) of an input image remains unchanged even after applying any transformation / to σ.
f(x) f(t())) E.
(1)
In self-supervised learning, leveraging transformation invariance is crucial for learning representations, as it aligns transformed input images within the representation space. The objective of transformation invariant representation learning Lim is formalized with a dissimilarity loss L as follows:
min Ez,r [Cim (z,
s.t. Line (2, t) ((f(x), f((z))))
(2)
where L represents the dissimilarity metric between representations. In contrastive learning, C is instantiated as metries like the InfoNCE (35) loss, which reduces the distance between representations of semantically similar inputs while increasing the gap between those of dissimilar inputs.
Transformation Equivariant Representation. Extending the concept of invariance, transformation equivariance ensures that an image representation changes predictably according to the applied transformation. The representation fzz is equivariant to all transformations in T if there exists group action Φ T x Y + Y on the representation space Y, ensuring that the application of / to an image x leads corresponding transformation ((t, f=)) in its representation.
f(t(z)) o(t, f()) e T.
The function -t + is referred to as the equivariant transformation on the representation space corresponding to the transformation f. When the equivariant transformation Φ(t," .) becomes the identity, it implies transformation invariance, showcasing the model's insensitivity to transforma- tions. In EquiMod [ and SIE [33 equivariant transformation o(E, -) is represented by a network parameterized by the corresponding transformation label i. This network is trained to associate each transformation with its corresponding equivariant transformation o(f, :) in the representation space. The objective of transformation equivariant representation learning Lequi is formalized follows:
min E.z, (fequu (Ir, 1) f
s.t.
Lequi (x. 1) = ((((f, [((x))))
3 Equivariant Learning with Self-supervised Transformation Learning
3.1 Equivariant Learning without Transformation Label
Transformation Representation. Instead of relying on transformation labels that require knowledge of the transformation group structure, we propose leveraging pairs of representations derived from original images and their transformed counterparts to implicitly represent transformations. We introduce auxiliary encoder fr Y Y Yr designed to process pairs of representations (f(=), f(t()))) where f(x) and f((())) are the representation of the original image and of the transformed image respectively. This encoder outputs transformation representation yf Yr. capturing the inherent transformation between the original and transformed images without explicit transformation labels.
vr = fr((()), f(t(=))) e Yr fort E T and E X.
(5)
3

---

Transformation Invariant Learning
(z, f((())
Transformation Equivariant Learning Lauulzz (v f((()))
(0 3
V1
Self-supervised Transformation Learning Erras (z, (Vi.
fr
Figure 2: Transformation Equivariant I Learning with Self-supervised Transformation Learning. (Left) The overall framework of STL. For given image and transformations, it demonstrates: 1) transformation invariant learning, which aligns the representations of image and transformed image; 2) transformation equivariant learning, where the representation of image transformed by an equivariant transformation (obtained from the transformation representation of different image with the same applied transformation) aligns with transformed image's representation; 3) self-supervised transformation learming, which aligns the transformation representations obtained from different image pairs. (Right) It illustrates the transformations of each representation and the equivariant transformations within the representation spacc.
Equivariant Learning with Transformation Representation. In contrast methods directly utilizing transformation labels to learn equivariant transformations in the representation space. our proposed approach substitutes labels with transformation representations derived from pairs of images. However, there is risk of encountering trivial solution when using the transformation representation derived from the representation pair the same image for which the equivariant transformation applied. Specifically, the equivariant transformation might simply output the same representation f((())) = p(fr (f) f((()))((())))) (t(()))) (x:) that was used obtain the transformation representation. To address this issue, we propose using transformation representations y derived from pairs of a different image x' apply equivariant transformations. Therefore, the transformed representation of the image through equivariant transformation in the representation space can be expressed as follows:
Vi' f(x)) = (fr (x') (t(z'))), f(x)) for x + x' e.
(6)
Leveraging this approach, it possible learn transformation equivariant representations without explicit transformation labels, through the following objective:
min Exter at (equa x', f.fr.0
s.t. Lequi (z, x',t) = L (0 f(x)). f(t(x))).
(7)
3.2 Self-supervised Transformation Learning (STL)
We hypothesize that transformation representations, vf derived from input image and its transformed image (xx), encode the transformation t independently of the input image x. Nonetheless, ensuring image-invares invariant encoding of t is not trivial.
y yi
Vx t
(8)
To address this, we introduce Self-supervised Transformation Learning (STL), which adapts con- trastive learning for transformation representation. Contrary to contrastive learning, which aims to align representations f(xx) of the same image under different transformations to promote transforma- tion invariance, STL instead focuses on aligning transformation representations, si and yf derived from different images + x' subjected the identical transformation t. The objective of STL is formalized as follows:
min E Engar Curan (x, s.t. Ciaas (x, L (VVL (Vr
99)
fJf
Hereinafter, equivariant learning using transformation representations learned through eepe-uuee transformation learning will simply be referred to STL.

---

Aligned transformations
between different images
Images
Images
Applied Transformations
Applied Transformations
Figure 3: Aligned Transformed Batch. (Left) In sclf-super learning methods, batch compositions typically involve applying two different transformations to each input image. (Right) In STL, batches composed by pairing two images together, and applying the same transformation pair.
3.3
Implementation Details
Dissimilarity Metric. We use the InfoNCE loss from SimCLR [4] for the formulation and imple- mentation of STL. Our methodology is not limited to this model, as demonstrated by ablation study, which shows the feasibility of applying our approach across various self-supervised learning models, [5), and Barlow Twins [48), through straightforward extensions (see Appendix A). Like EquiMod 19]. we employ specialized projectors 9ivv, Jequis and Jwans to map representations into distinct embedding spaces Zun Zequi- and Zuans aligned with the objectives of invariant, equivariant, and transformation representation learning. We adopt the InfoNCE loss as the dissimilarity metric across these spaces, similar the approach in SimCLR. The InfoNCE loss function is defined as follows:
such B BOLL [18]. SimSiam
exp (sim (g(v), gly /))
LINtONCE (y. :9. (v):) - log Ev. exp (sim
(10)
/T)
where y represents representation of an input image, y denotes the corresponding representation to align, sim(-) indicates a similarity function, and 7 is temperature scaling parameter. For simplicity, batch } are omitted in the subsequent loss functions. We define three specific loss functions for invariant, equivariant, and transformation representation learning, each building on the InfoNCE loss:
(z, t) Lequi (I, x LIntONCE
LInIONCE
9lav, Tuw)
(11)
(orccc((((() (0(v,' f(x)), f(t(x)); Jequi Tequi).
(12)
Luass (I, ,t, LIntONC (V,, vi': Itrans: Turans)
(13)
Aligned Transformed Batch. To implement STL, we need trans- formation representations obtained from different images. Unlike typical batch configurations in contrastive learning, where differ- ent transformations are applied to each input image, we construct batches by applying identical transformations to image pairs. as illustrated in Figure 3 In our approach, each image undergoes two distinct transformations, denoted and but for simplicity, we consider only single direction of transformation, treating as equivalent to 1-1 This aligned transformed batch configuration maintains the same computational complexity as typical contrastive learning setups while preserving input diversity. It also increases the count of identical transfor- mations applied across different images, which is essential for transformation learning, without diminishing input diversity. To assess computational costs, we measured 1000 iterations following 1000-iteration warm-up. With auxiliary network and loss calculation, our approach required only about 10% more time per iteration than SimCLR, which focuses solely on invariant learning, as shown in Table I
Table Computational Cost. Forward-backwart time per itera- tion on NVIDIA 3090 GPU with ResNet-50 and batch size 256.
Method
Time
Ratio
SimCLR AugSelf
0.51
1.00
0.51
EquiMed
0.53
1.01
STL (Ours)
0.56
1.11
time over
Overall Objective. The overall framework of STL shown in Figure 2 batch, along with the InfoNCE loss, we define the overall objective with the hyperparameters X Asqui and Ammos for balancing the respective losses as follows:
Using aligned transformed
Civ t) + Aequi Coqui (I, x', + Aerans (I, x' ,',
min f.rr.0
Ex
(14)
5

---

Table 2: Out-domain Classification. Evaluation of representation generalizability on the out- down- stream classification tasks. Linear evaluation accuracy (%) reported for ResNet-50 pretrained ImageNer100.
Method
CIFARIO
CIFARI00
Food
MIT67
Pets
Flowers
Caltech101
Cars
Aireralt
DTD
SUN997
Mean
Invariant Learning:
SimCLR
84.24
64.15
59.00
54.78
58.95
91.58
79.32
27.07
36.00
66.01
42.77
60.33
with AagMix
62.90
57.24
52.37
43.17
Implicit Equivartant Learning:
E-SSL
61.75
AugSelf
K5.55
66.09
62.63
57.16
62.61
93:41
82.33
30.71
40.35
68.51
45.24
63.14
Explict Equivariant Leurning:
SEN
W1.68
56.53
52.50
46.79
45.27
79.24
73.42
14.41
27.51
57.45
33.51
51.57
EquiMad
82.89
61.36
56.38
52.84
52.68
87.42
79.17
22.02
34,62
64.10
39.86
57.58
SIE
81.72
58.49
54.04
49.70
47.21
84.37
74.39
16.71
31.68
59.20
35.29
53.89
STL (Ours)
86.55
66.84
64.32
56.64
65.00
94.51
81.83
35.44
45.42
64.68
44.68
64.18
with AugMin (Ours)
87.19
66.12
59.70
67.10
84.61
38.48
46.14
69.57
45.75
66.11
4 Experiments
Baselines. We compare STL with implicit and explicit equivari- ant learning methods, using SimCLR as the base invariant model. Implicit methods (E-SSL (8] and AugSelf [28) learn
Table 3: In-domain Classification. Evaluation representation on in- domain classification task. Linear eval- uation accuracy (%) is reported for ResNet-50 pretrained ImageNet100.
cquivariant representations via transformation prediction tasks. Explicit methods (SEN (36), EquiMod [9], and SIE (13) use transformation labels for equivariant learning. All methods, including STL, are trained and evaluated with SimCLR the base model. Experiments with other base models are included in the ablation study.
Method
Accuracy
Invoriant Learning
SimCLR
81,20
SimCLR with AugMix
80.54
Datasets. We pretrain on STLIO 7 with ResNet- 18 and
Implicú Equivariant Learning E-SSL
ImageNet100 [39 42 with ResNet-50, following the split 42.. Evaluation spans downstream classification tasks
82.10
AugSelf
$1.08
(CIFAR10/100 27 Food [2]. MIT67 [38]. Pets (37]. Flow- ers (33), Caltech101 [12), Cars (26). Aircraft [31], DTD [6].
Explicir Equivariant Learning
SEN
76.32
SUN397 [7]) with linear transfer protocol 41. For detec-
EquiMod
80.70
SIE
79.41)
tion, VOCO7+12 dataset [1] and the protocol from 48 are
STL (Ours)
81.10
used. Dataset and protocol details are in Appendix and
STL with AugMix (Ours)
81.64
Setup. For STL and explicit baselines (SEN, EquiMod, and SIE), we use an equivariant transformation network with hypernetwork based on SIE, In SEN, EquiMod, and SIE, the hypernetwork uses transformation labels; in STI., leverages transformation representations. STI also includes 3-layer MI.P with a 51 512-dimensional hidden layer to encode 128-dimensional transformation representations from input pairs. Equivariant transformations include random crop and color jitter, with other transformations applied randomly, consistent with typical contrastive learning. The transformation prediction loss weight is set to 0.5 for implicit baselines, and the equivariant learning weight is set to 1 for explicit baselines. STL uses weights of 1, and 0.2 for invariant, equivariant, and transformation learning losses, respectively. We apply AugMix 23 to evaluate STL's adaptability to complex transformations, incompatible with other methods. Details on transformation labels standard equivariant learning are in Appendix [D] All analyses and ablations. except main experiments, use STL. 1TT..0--rerrained models. Additional setup details are in Appendix
4.1 Main Results
Image Classification. To assess generalizability, we apply the linear evaluation protocol on various downstream tasks. As shown in Table STL outperforms existing methods on of datasets. With AugMix, complex transformation combination, STL achieves the highest performance across all datasets, underscoring its ability to generalize across diverse transformations, even those without explicit labels, to improve generalization. In Table 3 STL shows a minimal trade-off on in-domain tasks compared to SimCLR, with only a slight decrease from 81. 11.00% to 81. 10%, smaller trade-off than other explicit equivariant models. Combined with AugMix, STL reaches 81.64%, showing adaptability to complex transformations and further enhancing in-domain performance. Results on STL10-pretee pretrained models provided in the Appendix F

---

Table 4: Object Defection. Evaluation of representa- tion generalizaaility a downstream object detection task. Average precision reported for ImageNet100- pretrained ResNet-50 fine-tuned on VOCO7+12 12.
Table 5: Transformation Prediction. Evaluation of transformation representation from learned repre- setation pairs. Regression tasks use MSE loss, and transformation type classification uses accuracy (%).
Method
alll
AP50
AP75
Regression
Classification (t)
SimCLR
45.67
72.50
Method
Crop
Color
All
Trans. Type
AugSelf
45.99
72.46
49.23
SimCLR
0.02
0.08
68.54
EquiMod
51.55
78.03
56.17
AugSelf
0.01
0.04
0.03
88.49
EquiMod
0.01
0.07
0.04
82.20
STL (Ours)
51.95
78.34
56,96
with AugMix (Ours)
52.70
78.81
57.76
(Ours)
0.03
0.02
93.67
Object Detection. We evaluate STL on the VOC07+12 object detection task. As shown in Table 4 STL outperforms the invariant learning model SimCLR, as well as AugSelf, a representative model for implicit equivariant learning, and EquiMod, representative for explicit equivariant learning, across all metrics: Al Ap and AP75. STL achieves 51.95 in APall, and with AugMix, further improves 52.70 in AP 78.81 in AP and 57.76 in AP75, demonstrating robust adaptability to complex transformations and enhanced precision in localization.
4.2 Analysis
Transformation Representation. To evaluate learned trans- formation representations of STL., we assess both parameter prediction and type classification on test images with transfor- mations used during pretraining, specifically crop and color jitter. Parameter prediction uses MSE loss measure accuracy in predicting transformation specifics, while type classification assesses the model's ability to distinguish among transforma- tion categories. As shown in Table 5) STL achieves the lowest MSE for crop, color jitter, and the combined metric, indicating precise capture of transformation details. In classification, STL attains 93.67% accuracy, surpassing other models, even though AugSelf incorporates parameter regression into its learning. This demonstrates STL's robust generalization in transforma- tion learning without direct parameter supervision.
Chop
Sabur,
Sttter
Bright..
1200
Coake
Figure 4: Visualization of Transfor- mation Representations by Intensity. UMAP visualization of transformation representations organized by intensity levels for each transformation type, in- cluding random crop and color jitter variations in brightness, contrast, satura- tion, and hue. Parameter ranges for each transformation are divided into four seg- ments to apply varying intensities, with darker colors representing higher inten- sities. Representations are captured by ResNet- 18 model pretrained on STL10 with transformation backbone.
Additionally, a qualitative UMAP visualization of transforma- tion representations from test image pairs reveals STL's under- standing of transformation relationships. Figure l[b) shows distinct clusters for each transformation type. including crop, brightness, contrast, saturation, and hue, with color-related transformations grouped closely, reflecting STL's capture of inter-relan relationotiss among similar transformations. Figure 4 shows that transformations with similar intensity levels are positioned closer in the representation space, forming continuous intrinsic order of intensity within each transformation type. This structure indicates STL's effective learning of transformation intra-relationshiis, demonstrating its nuanced understanding of both type and intensity.
representations that capture the
Transformation Equivariance. We evaluate the accuracy of STL's Iearned equivariant transfor- mations in reflecting real transformations in image space. This evaluation involves applying 60 transformations per image from the STLI0 test dataset, including standalone transformations like crop and color jitter, as well as the standard combinations used during training. For each transformed representation, we rank other representations by similarity, ordering them from closest to furthest. Metrics include Mean Reciprocal Rank (MRR), Hit@I (H@k), k), and Precision (PRE). MRR is defined as the mean of reciprocal ranks 1/r, where r is the rank of the nearest representation corresponding to the designated transformation. Hit@k k calculates the probability P(r S k) that the correct transfor- mation rank r is within the top k. PRE is the mean squared error between the representation from the top-ranked transformation and the actual transformation's 's parameter vector.
7

---

Table 6: Transformation Equivariance. Evaluation of the equivariant transformation. Mean Reciprocal Rank
(MRR), Hit@k (H@k), and Precision (PRE) metrics various transformations (crop and color jitter).
Crop
Color
Method
MRR())
H@1((t)
H@s(()
PRE()
MRR(1)
H()(()
H(ek1)
PRE())
MRR(1)
H0:((7)
PRE()
SEN
0.34
0.15
0.58
0.14
0.18
0.05
0.31
3.69
0.22
0.08
0.37
2.70
EquiMed
0.37
0.17
0.60
0.13
0.16
0.05
0.28
3.72
0.22
0.09
0.36
2.72
SIE
0.33
0.14
0.55
0.33
0.17
0.05
0.28
3.70
0.21
0.08
0.35
2.74
w/o Cinas (Ours)
0.31
0.18
0.46
0.69
0.27
0.13
0.40
3.37
0.29
0.16
0.43
2.50
STL (Ours)
0.37
0.22
0.51
0.64
0.33
0.18
0.52
2.76
0.36
0.21
0.53
2,07
Table 7: Loss Function Ablation Study. Image classification and transformation prediction results ResNet- 18 pretrained on STL10 with selective inclusion of loss terms for invariant learning (Cim). equivariant learning (Lewi), and self-supervised transformation learning (Cramm). ). For image classification, in-domain accuracy (%) and the average accuracy (%) across multiple out-domain datasets are shown. For transformation prediction, MSE is used regression crop and color transformations, and accuracy (%) is used transformation type classification.
Loss Functions
Image Classification
Transformation Prediction
Method
Lin Cegu
Curas
In-domain (1)
Out-domain (f)
Regression
Classification (+)
Only Invariance
84.74
43.11
0.08
68.54
Only Equivariance
83.53
49,99
0.02
93.54
Line
/
$1.86
48.62
0.02
STL w/o Lequi
5
80.99
47,30
0,02
93,92
STL w/o Lians
v
85.11
48.49
0.08
69.57
/ 84.83
0.02
As shown in Table o STL outperforms previous methods in most metries, indicating closer alignment of its equivariant transformations with real image transformations in representation space. Notable exceptions crop H@5 and PRE, where other methods perform slightly better. Overall, STL captures both individual and combined transformations more faithfully, achieving robust alignment between representation and image transformations. Self-supervised transformation learning, by enhances input consistency for equivariant transformation learning and significantly improves alignment accuracy. Without this component, STL's ability capture transformation nuances declines, underscoring the role of each component in the STL framework.
maintaining
4.3
Ablation Studies
Loss Functions. We conduct an ablation study to analyze the impact of each loss function on STL's performance across image classification and transformation prediction tasks. Specifically, we examine the contributions of invariant learning (Cin)), equivariant learning (Cegui), and sulf-supe transformation learning (Ciram)) by selectively removing each loss term. Zillustras clear trade- off between invariance and equivariance in STL's performance. The Only Invariance configuration achieves high in in-domain accuracy of 84.74% but suffers from low out-domain accuracy of 43. and limited transformation prediction capabilities, highlighting restricted generalizability. In contrast, Only Equivariance improves out-domain accuracy to 49.99% and achieves strong transformation prediction with an MSE of 0.02, indicating enhanced generalization and transformation awareness, albeit with slight reduction in in-domain performance.
When Crans omitted, the model maintains high in-domain accuracy but exhibits weak out-domain and transformation performance, suggesting that the absence of transformation representation learning leads to focus on invariance. Excluding Lim improves out-domain accuracy to 48.62% and enhances transformation alignment by preventing collapse into pure invariance, although this comes at moderate cost to in-domain accuracy. Without Loqui the in-domain performance decreases further as transformation learning alone lacks sufficient structure for alignment. Finally, the full STL, which incorporates all three losses, achieves the best balance, with superior performance across in-domain and out-domain tasks and optimal transformation prediction results. This configuration minimizes in-domain trade-offs while capturing comprehensive view of transformations, ensuring alignment between representation and image transformations across both in-domain and out-domain tasks.
8

---

Table 8: Transformation Ablation Study. Linear evaluation accuracy (%) of ResNet-18 pretrained on STL10
with various transformations used equivariance targets.
Trans.
Method
CIFARIO
CIFARI00
Food
MTT67
Pets
Flowers
Callech101
Cars
Aircraft
DTD
SUN397
Mean
AugSell'
82,89
54.92
33.19
39.70
44.40
64.96
67.63
15.58
25.38
41.86
27,89
45.31
EquiMod
83.76
55.33
32.01
37.76
11.65
63.00
66.28
14.18
24.96
11.54
26.46
44.27
STL
84.94
59.12
36.15
39.40
45.35
68.38
70.78
17.96
33,00
41.86
28.71
47,70
AugSelf
84.33
57.47
36.57
39.40
46.50
71.18
67.91
17.03
27.12
43.83
29.37
47.36
color
EquiMod
82.22
51.77
31.21
31,18
39.57
61.17
62.07
12.5]
21.36
39.52
23,18
41.73
STL
84,16
58.71
38.49
41.34
45.90
74.36
68.48
17.31
27.12
46.54
31,17
48.51
crop
AugSelf
84.26
57.78
36.82
40.30
45.
73.38
68.11
17.22
27.63
45.96
30.38
47.94
EquiMod
81.35
51.86
33.91
37.76
11.92
66,
67.38
15:22
25,80
12.50
26.70
44.60
STL
85.37
61.05
39.41
41.27
46.58
76.43
71.47
19.04
30.75
46.17
32.13
49,97
AugSelf
81,76
54,90
36.51
40,90
46.
71.43
70.14
18.63
30,96
45.21
30.40
47.91
EquiMod
81,42
56.65
34.23
37.99
12.98
67.16
68.41
15.18
26.91
43.94
26.97
45.89
STL
84.96
58.91
36.71
42,09
46.25
72.41
71,01
17.72
28.44
43.83
30.99
488.48
Table 9: Base Invariant Learning Model Ablation Study. Linear evaluation accuracy (%) of ResNet-18 pretrained on STLI0 with various base models for invariant learning.
Base
Method
CIFARIO
CIFAR100
Food
MIT67
Pets
Flowers
Calrech1011
Cars
Aireraft
ITID
SUN397
Mean
85.55
59.80
37.54
42.61
50.61
73.50
72.16
23.02
31.71
41.95
31.63
50.31
BYOL
AugSelf EquiMod
87.01
64.84
43.14
47.24
52.49
78.88
75:42
25.47
37,02
48.03
34,94
54.04
81.64
56.55
32.74
39.18
14.64
66.54
68.37
15:17
24.27
12.71
26.96
45.64
STL
86.88
65.63
42.98
46.42
52.33
79.61
76.04
28.68
39.21
46.44
34.57
54.44
83.26
55.69
34.32
40.52
46.52
66.06
69.13
17.15
27.99
41.91
28.97
46.50
SimSiam AugSelf
85.44
62.20
39.78
43/13
16.77
77.90
71.72
18.67
33.30
45.53
32.65
50.67
EquiMod
81.20
51.23
31.21
37.99
40.53
63.98
64.19
12:22
22.11
40.69
25.76
42.83
STL
85,20
62,58
40.15
44.03
48.65
76.68
71,37
22.42
32,37
45.39
32,19
51,02
81.67
$1.68
27.79
33.13
39.60
57.63
62.17
11.33
19.47
37.13
23.43
40.48
Barlow Twins
AugSelf
82.46
51.71
27.83
35.75
39.33
58.24
61.87
11.88
19.77
37.29
23.31
40.86
EquiMod
81.57
52.15
30.00
36.79
38.70
62.64
63.22
11.80
20.55
10.21
21.92
42.05
STL
83.74
56.73
32.69
38.36
42.65
67.28
68.09
16.24
24.33
41.97
28.53
45.51
Transformations. Table shows STL's flexibility and effectiveness across various transformations. STL achieves the highest mean accuracy across all settings, outperforming AugSelf and EquiMod in single transformations such as crop and color, as well as in combined transformations including all transformations, reaching mean accuracy scores of 47. 37.70%% 48.51% %, and 48.48%, respectively. These results highlight STL's robust equivariant learning across diverse transformation types, enabling strong generalization without constraints specific transformations.
Base Invariant Learning Models. Table demonstrates STL's compatibility with various base models such as SimCLR, BYO,, SimSiam, and Barlow Twins. STL consistently improves the mean accuracy across all base models, outperforming both AugSelf and EquiMod, with the highest overall mean accuracy achieved with BYOL at 54.44%. These results demonstrate that STL is compatible with different invariant learning frameworks, confirming its adaptability and effectiveness regardless of the underlying representation learning method.
5 Related Works
Transformation equivariant learning captures transformation-sensiiive features by embedding transfor- mation directly into the learning process, categorized into implicit and explicit approaches. Implicit learning predicts transformations by observing changes in representations, allowing models to infer transformation without directly modeling the functions. In contrast, explicit cquivariant learning en- codes transformations within the representation space, enforcing behaviors in learned representations that mirror input transformations. These approaches can be reviewed in detail in the Appendix G
Implicit Equivariant Learning. In implicit equivariant learning. models learn to recognize applied transformations by observing changes in representations, enabling the representations capture transformation-sensee information. Notable methods include InfoMin [43], selecting optimal views to maintain relevant task information, and Prelax 455 aligning residual vectors for robust multi-view
alignment. Similarly, E-SSL. g and AugSelf 28 leverage transformation-awaree auxiliary tasks
to train models to preserve maunforrmtiion-sessitii details, enhancing robustness by maintaining
sensitivity to transformation variance.

---

Explicit Equivariant Learning. On the other hand, explicit equivariant learning directly encodes input transformation into the representation space, building equivariant transformations that operate consistently in representation space. AEAF [9] leverages group actions to embed transformation effects explicitly in the representation space, while SymReg [40] enhances transformation con- sistency by selecting optimal loss terms based on transformation group information. CARE [20] introduces rotational symmetry by aligning embeddings directly with input rotations, providing robust transformation encoding. Similarly, SEN (36) applies symmetric embedding networks to synchronize transformations in the input space with learned representations. In a more flexible approach, EquiMod [9] models equivariant transformations by conditioning transformation labels as inputs and dy dynamically adapting representations through a neural network. Building on these approaches, SIE [3] separates invariant and equivariant representations, using dedicated networks to distinctly capture waanformattion-sesstti and invariant aspects.
Applications of Equivariant Learning. The applicability of equivariant learning extends across various domains, including roboties, medical imaging, molecular modeling, and multimodal repre- sentation learning. By leveraging inherent symmetries within data, it enhances sample efficiency in robotic manipulation [44], improves accuracy in medical image processing [21], and boosts predictive performance in molecular data analysis [46.. In multimodal contexts, it achieves finer alignment by considering transformations within each modality, facilitating robust cross- cross-modal understand- ing (29 [14]. These applications underscore the broad utility of equivariant learning in aligning model representations with commin--ppecffie transformations.
Discussion and Conclusion
The Self-supervised Transformation Learning (STL) approach introduces a novel method for deriving transformation representations that capture equivariance without relying on predefined transformation labels. This framework optimizes representational versatility through a synergy of three key loss functions: invariant learning, equivariant learning, and self-supervised transformation learning. These loss functions allow representations to adapt to complex transformation dynamics, including interdependencies among transformations, thereby greatl enhancing model generalization. STL consistently outperforms existing methods across 7 out of 11 classification tasks and demonstrates exceptional performance in object detection, proving its strong ability to generalize across diverse transformations. Additionally, integrating STL with AugMix yields robust performance improvements across all tasks, demonstrating enhanced resilience. STL's adaptability and consistent performance across various transformations and base models underscore its versatility for broad applications.
Limitations. While STL significantly advances equivariant learning, encounters challenges with transformations that extend beyond single image pairs. Complex transformations, such as those involving combinations or mixtures of multiple images (c.g., mixup (4)),, fall outside STL's current capacity as it relies on pairwise transformation representations. Further research could explore ways to adapt STL accommodate more complex, multi-image transformations and better capture their inherent structure.
Broader Impacts. STL holds promise for applications requiring precise and interpretable trans- formations, such as in medical imaging analysis and autonomous driving. However, as STL learns transformation representations from the data, it may inherit biases embedded in the training data, raising fairness concerns, especially in sensitive domains. Implementing fairness-aware training techniques and thorough validation processes could help mitigate these risks. Additionally, while STL advances model robustness and generalization, its computational demands may have environmental implications. Efficiency improvements, such as model distillation, could reduce the model's energy footprint, supporting sustainable deployment.
Acknowledgments
This work was supported by Institute of Information & Communications Technology Planning & (valuation(ITTP) grant funded by the Korea goV)) (RS-2024-00439020, Developing Sustainable, Real-Time Generative AI for Multimodal Interaction, SW Starlab).
10

---

References
[1] A. Bardes, J. Ponce, and Y. LeCun. Varinaee-iivarnnne--vvvvaaanee regularization for self-supervised learning. ICLR, Vicreg, 1:2, 2022.
[2] L.. Bossard, M. Guillaumin, and L. Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461, 461. Springer, 2014.
[3] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33: 9912-9924, 2020.
[4] T. Chen, Korblithh, M. Norouzi, and G. Hinton. A simple framework contrastive learning of visual representations. In International conference on machine learning, pages 1597-16600 PMLR, 2020.
[5] X. Chen and K. He. Exploring simple siamese representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[6] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3606-3613, 2014.
[7] A. Coates, A. Ng, and H. Lec. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223, JMLR Workshop and Conference Proceedings, 2011.
[8] R. Dangovski, L.. Jing, Loh, S. Han, A. Srivastava, Cheung, P. Agrawal, and M. Soljačié. Equivariant contrastive learning. arXiv preprint arXiv:2111.00899 2021.
191 A. Devillers and M. Lefort. Equimod: An cquivariance module to improve self-supervised learning. arXiv preprint 2rXtv:221..01444 2022.
[10] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
[11] M. Everingham, I.. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-336 2010.
[12] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[13] Q Garrido, L. Najman, and Y. Lecun. Sule-supe learning of split invariant equivariant representations. arXiv preprint 2rXiv:2302.10288 2023.
2023.
[14] T. Geng, T. Wang, Y. Zhang, Duan, W. Guan, and F. Zheng. Uniav: Unified audio-visual perception for multi-task video localization. arXiv preprint arXiv:2404.03179, 2024.
[15] S. Gidaris. P. Singh, and Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07722 2018.
[16] R. Girshick. Fast r-cnn, In Proceedings of the IEEE international conference on computer vision, pages 1440-144 1448, 2015.
[17] P. Goyal, P. Dollár. R. Girshick, P. Noordhuis. L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint 1rXiv:1706.02677, 2017.
[18] J.-B. Grill, E. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271 12224444, 2020.
Z. Guo, M. Gheshlaghi Azar,
[19] X. Guo, E. Zhu, X. Liu, and J. Yin. Affine equivariant autoencoder. In IJCAI, pages
2019.
[20] S. Gupta, J. Robinson, D. Lim, Villar, and S. Jegelka. Structuring representation geometry with rotationally cquivariant contrastive learning. arXiv preprint arXiv:2306. 2023.
[21] A. Hashemi, Y. Feng, and H. Sabet. Spherical enn for medical imaging applications: Importance of equivariance in image reconstruction and denoising. ArXiv, 2023.

---

[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 770-778, 2016.
[23] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint www.:vv111..000. 027717 2019.
[24] S. loffe and C. Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456, pmlr, 2015.
[25] S. Korbliihh, J. Shlens, and Q. L. Do better imagenet models transfer better? In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 2661-2671. 2019.
[26] J. Krause, M. Stark, Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554 561, 2013,
[27] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] H. Lee, K. Lee, K. Lee, H. Lee, and Shin. Improving transferability of representations via augmentation- aware self-supervision. Advances in Neural Information Processing Systems, 34:17110-17724 2021.
[29] J. Lee, J. Kim, H. Shon, B. Kim, S. H. Kim, H. Lee, and Kim. Unielip: Unified framework for contrastive language-image pre-training. Advances Neural Information Processing Systems, 35: 1008-191,,,, 2022.
[30] I. Loshchilov and Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0 03983, 2016.
[31] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 306.5151, 2013.
[32] L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rXiv 2018.
[33] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pages 722-729. IEEE, 2008
[34] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pages 69-84. Springer, 2016.
[35] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arviv 807.00.3 03748, 2018.
[36] J. Y. Park, O. Biza, L. Zhao, J. W. van de Meent, and R. Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204. 11371, 2022.
[37] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[38] A. Quattoni and A. Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pages 413 420. IEEE, 2009.
[39] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bersstcin, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-251 2015.
[40] M. Shakerinava, A. K. Mondal, and S. Ravanbakhsh. Structuring representations using group invariants. Advances in Neural Information Processing Systems, 15:34162-34174, 2022.
[41] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843 852, 2017.
[42] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In Computer Vision ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[43] Y. Tian, C. Sun, Poole. D. Krishman, C. Schmid, and P. Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
[44] D. Wang, M. Jia, X. Zhu, R. Walters, and R. Platt. On-robot learning with equivariant models. arXiv preprint 220: 04923, 2022.
12

---

[45] Y. Wang, Z. Geng, F. Jiang, C. Li, Y. Wang, J. Yang. and Z. Lin. Residual relaxation for multi-view representation learning. Advances in Neural Information Processing Systems, 34:12104-12115, 2021.
[46] R. Winter, M. Bertolini, Le, F. Noé, and D.-A. Clevert. Unsupervised learning of group invariant and equivariant representations. Advances Neural Information Processing Systems, 35:31942-31999 2022.
[47] J. Xiao, J. Hays. K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey pages 3485 3492. IEEE, 2010.
2010 IEEE computer society conference on computer vision and pattern recognition,
[48] J. Zbontar, L. Jing, Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning, pages 12310-12320 PMLR, 2021.
[49] H. Zhang. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: (710:1171 2017.
13

---

A STL Formulations for Various Base Invariant Models
A.1 STL Extension on BYOL
In adapting STL to BYOL (18), we utilize dissimilarity loss of BYOL to define the invariant, cquivariant, and transformation losses. BYOL's dissimilarity metric is:
LBYOL(U,
99,9, :9.ll,.""".9.,...
I (t)) De (1
(15)
where Do denotes the projection network parameterized by 0, is the prediction network also parameterized by 0, and de represents the target network parameterized by g. The terms qa and Je refer to the normalized outputs of qo and 9G, respectively. Using this, we define the STL objectives as follows:
Lim (2,, (((r)) Dio Guv, 0,E),
(16)
Loqui (2, LovoL(O(VE f(x)), [t((x)); Goqui Joquis 0, E)
(17)
Craans (2,, x', LBYOL (2) vi' Awass: Arans 0, E)
(18)
A.2 STL Extension SimSiam
For SimSiam
5) STL. uses SimSiam's dissimilarity loss:
CSimSium (v. (u. h) = (h(o(())) stopgrad(g(x"))) D(h(g(ot)) "(((()))))))))))))))) stopgrad(g())))
(19)
where y denotes the projection network, h is the prediction network, stopgrad indicates an operation that halts gradient backpropagation, and 'D represents cosine similarity. This enables us to structure the STL losses as:
(In t) LSimSiam (z)), [(((x)); Finv: hinv)
(20)
Lequi (Z,, ,t,t) = SimStaa (0(1) of f(x)), f(t(x)); Gequis hequi);
(21)
Luans (2, a',t) LSimSiam (v, Ve Iuaass bwans).
(22)
A.3
STL Extension on Barlow Twins
For Barlow Twins 48 STL applies Barlow Twins' dissimilarity loss:
LaartowTwins (Y {ur), y
( hig. (1-Cx)) (1- + 179
CO
(23)
where C denotes the corssccorellatin matrix between embeddings of transformed views Y and Y+,
and À is regularization parameter that controls the weight of off-diagonal terms in C, penalizing redundancy in the representations. Define X = {zi)i. X' = {zi),. and T {4:}, representing the sets of input images, paired images, and transformations, respectively. Then, the STL losses for Barlow Twins are:
Lime X, T' LBarlow/Tins (((())))))..L(((())))((()()((()) Sinv, Ainv),
(24)
Leequl (X,X* X', = TwartowTwins 2/ ( (xx))) (1((((x))) Jequi Aequi),
(25)
Lians (aann(XX, X'TT) LBaroowTwine (1 }i, {3/. Jos Granss Avans) )
(26)
14

---

B Datasets
Table 10: Dataset Information. Overview of dataset composition and evaluation metrics. Each dataset specifies the number classes, training/validatiionnest splits, and the corresponding evaluation metric.
Category
Dataset
of classes
Training
Validation
Test
Metric
(a) Pretraining
STL10
?
10
105,000
ImageNet100
1,000
126,689
CIFARIO 27
45,000
5,000
10,000
Top-1 accuracy
CIFAR100 27
100
45,000
5,000
10,000
Top-1 accuracy
Food 2
101
68,175
7,575
25,250
Top-1 accuracy
MIT67
67
4,690
670
1,340
Top-1 accuracy
Pets
37
2,940
740
3,669
Mcan per-elass accuracy
Flowers 8
102
1,020
1,020
6.149
Mean per-class accuracy
(h) Linear Evaluation
Caltech10 12:
101
2,525
505
5,647
Meun Per-class accuracy
Cars 26
6,494
1,650
8,041
Top-1 accuracy
Aireraft 31
100
3,334
3,333
3,333
Mean Per-class accuracy
DTD (split I) 0
47
1,880
1,880
1,880
Top-1 accuracy
SUN397 (split 1)
397
15,880
3,970
19,850
Top-1 accuracy
(c) Object Detection
VOC2007+2012
20
16,551
4,952
Average Precision
Table 10 presents detailed descriptions of (a) pre-taining dataset, (b) linear evaluation datasets, and (c)
object detection and instance segmentation dataset. For linear evaluation dataset, validation samples are randomly selected from the training split if an official validation split is not provided. For the
object detection and instance segmentation dataset, we use trainval set for training VOC07+12 [1]
while only using test set for testing.
C Evaluation Protocols
Linear Evaluation. Following established protocols 25 4 18 train linear classifiers frozen representations from center-cropped images resized to 224 x 224 (or 96 x 96 for STL10). No data augmentation is applied. Each image is resized along its shorter side to 224 and then center- cropped to 224 x 224. The éz-regularized cross-entropy objective is minimized using L-BFGS, with regularization selected from 45 logarithmically spaced values between 10-6 and 10 on the validation set. The optimal model is then retrained on both training and validation splits, with test accuracy reported. L-BFGS is capped 5,000 iterations, with each step initialized from the previous solution.
Object Detection. We use the VOC2007+2012 trainval set with 16,551 images to train a Faster R-CNN [16] with C-4 backbone. Training spans 24,000 iterations with batch size of 16 and SyneBatchNorm. The learning rate starts at 0. decreasing by a factor of 10 at 18,000 and 22,000 iterations. A linear warmup [17] is applied for the first 1,000 iterations with 0.333 slope.
D Transformation Labels
In this section, we describe the labels of the transformations utilized in AugSelf (28) (random crop, horizontal flip, color jätter, grayscale, and Gaussian blur) in the following. The labels are designed by the parameters in each transformation.
The labels are constructed with the center points, Heenter and W Weetter, of the crop and the height H and width W values of the cropping area.
RandomRessizedCrop.
RandomfoorizontalFiig As flip transformation is simple operation, the label is 0 or 1.
ColorJitter. The four parameters of color jitter are brightness, contrast, saturation, and hue. Each parameter has its own range, with brightness, contrast, and saturation ranging from 0.0 to 1.0, while hue ranges from 0.0 to 0.5.
RandomGrayscale. This transformation applies the grayscale to the image. In line with the flip, the label is or I.
GaussianBlur, The standard deviation is utilized for the Gaussian blur label ranging from 0.1 to 2.0.
15

---

E
Pretraining Setups
For the pretraining experiments, use NVIDIA RTX4090.
E.1
ImageNet100 Pretraining
We conduct pretraining on the ResNet-50 architecture [22 using ImageNer100. a subset of ImageNet containing 100 categories [39], with dataset splits consistent with those in 42]. All methods
trained for 500 epochs with a batch size of 256, using a cosine learning rate schedule without
restarts [30]. The initial learning rate is set 0.03, with a weight decay of 0.0005. The model
includes a 3-layer projection MLP head, g(),, with a hidden dimension of 2048 and an output
dimension of 128. Batch normalization
2 is excluded from the last layer.
E.2 STL10 Pretraining
For pretraining on the STL10 dataset 7. we use the standard ResNet- 18 architecture [22]. All methods utilize stochastic gradient descent (SGD) with learning rate of 0.03, batch size of 256, weight decay of 0.0005, and a momentum of 0.9. The learning rate follows a cosine decay schedule
without restarts [30].
SimCLR 4]. A 3-layer projection MLP head g(.) with a hidden dimension 512 and an output dimension of 128 is used, with batch normalization excluded from the final layer. In contrastive learning, we apply a temperature scaling parameter 0.2.
Barlow Twins 48). A 2- 2-layer projection MLP head g(-) is employed, with a hidden dimension of 512 and an output dimension of 2048. Batch normalization excluded from the last layer.
BYOL I8] The model uses a 2-layer projection MLP head g(:), with a hidden dimension of 4096 and output dimension of 256, omitting batch normalization in the final layer.
SimSiam (5). We employ a 2-layer projection MLP head g(-) with both hidden and output dimensions of 2048, with batch normalization excluded from the final layer.
F Image Classification Results of STL
Models
Table 11: Image Classification. Evaluation representation on in-domain and downstream out-domain classification task: Linear evaluation accuracy (%) of ResNet-18 222 pretrained on STL10 7 averaged over three random seeds (mean + std).
Mirieed
V11.8
CHARI
CHARIE
Fool
SITR?
Flavers
SUNART
Mine
Doneeemmeee Peansen Loaning::
81328635
02425635
Apylss TwweSrmettrrree Aw Seonnn
Suckck Zontemanio Apah umacc
2014
14
91211111
SURETT
MITIET 11
17
1417220
31/022111
01972011
1T 1223199
65.52 65512110 mn
1001231 M4
15912210
15/122111
44 241122914
NTT (ler)
77230-1
91681111 11
21112211
11 14:21111
11
311121100
16

---

G
Explicit and Implicit Equivariant Learning
(e f164))
Trandoentzioon Eqpivarinnt Learines Eapebt.)) = 010.. f(a)). (l0
Tranferanatien Predictio Cuuilii CUNUSn Nrrma
Spncch by ao
(1.7.0 Cof (g
(st
EajEspleh
Implist Equirartant
Explicit and Implicit Equivariant Learning. Transformation equivariant learning with transforma- tion lahels is divided into (Left)) explicit and (Right) implicit equivariant learning.
Figure 5:
Figure 5 illlustrate the framework of explicit and implicit equivariant learning. Explicit methods like SEN [6), EquiMod 19]. and SIE [13 apply direct equivariant transformation network to representations, leveraging transformation labels for aligmment. In contrast, implicit methods, such as E-SSL g and AugSelf [28]. infer transformation effects without directly applying transformations by utilizing auxiliary tasks deduce transformation states.
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claim in the abstract and introduction is dealt with in the background, method, and experiment sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We deal with this in the Discussion and conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide the hyperparameters, training, and evaluation protocols with details in the paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All training and evaluation are conducted based on publicly available codes, and we provide the details of our implementations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pbble wwwppppgpuuiiieeeep /otic///uuiee//oeeuuuiiiiiii
(CodeSubiisss wwtauubmmsssooni ssiiiioiiccc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We report all the details in the paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
The experiment merely contains negligible statistical effects.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe detailed information about the experiment in the paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: bectpa://neuripeee http://wwwwraaeeccoo/olllcccccaa
Answer: [Yes]
Justification:
We wrote paper in compliance with the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We denote the negative societal impacts that the proposed method may have.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "EmGMLtRDqZ", "reviewer": "Reviewer_r3iR", "summary": "This paper proposes a new way to learn equivariant representations by directly learns the transformation representation. They enforce that different transformations have their own input-agnostic representations. To obtain this, they learn an encoder that takes pairwise representations of the same image to extract the transformation representation. To avoid trivial solutions, they use different examples in transformation representation extraction and later transformation module, which helps disentangle the sample and transformation features. They further add an regularization on the sample consistency to encourage it to be sample-invariant. They combine it with invariant learning loss for training, and demonstrates its benefits on several tasks.", "questions": "no", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I generally like the idea of disentangling sample and transformation representations in equivariant learning, and the paper proposes a clever patching-based objective to enforce it. \n- And the transformation representation they obtain in Figure 1 does make sense intuitively since it reflects the relative distance between different augmentations. \n- The evaluation demonstrates its advantages over previous works on linear probing and object detection tasks. The propose method is particularly superior on the transformation prediction task. I think it's because they successfully disentangle the features.", "weaknesses": "- The evaluation is not complete. I didn't find linear probing results (often the most important ones) on ImageNet-100 or STL-10.\n- Lack the comparison to a few relevant E-SSL baselines, such as [1,2].\n- Lack more in-depth analysis of the learned transformation representations. I have some doubts about it. Since there are many variations in each augmentation (eg cropping with different positions, ratios), what is the relationship of their representations in the latent space? Is there an Arithmetic relationship?\n\n\n[1] [Equivariant Contrastive Learning. ICLR 2022.](https://arxiv.org/abs/2111.00899)\n\n[2][ Residual Relaxation for Multi-view Representation Learning. NeurIPS 2021.](https://arxiv.org/abs/2110.15348)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoroughly review our manuscript.\n\nIn response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments.\n\nWe would greatly appreciate it if you could review our responses to your comments and the submitted rebuttal PDF.\n\nSincere thanks in advance for your time and efforts.\n\n---\n\n## **[W1] In-domain Evaluation on STL-10 and ImageNet100**\n\nTo address the concern regarding the in-domain performance evaluation, we have included the results of the linear evaluation conducted on the pretraining dataset in Table A3 in the rebuttal PDF. Our findings demonstrate that, except for the SEN method which focuses solely on equivariant learning, all other approaches including the proposed method (STL) effectively maintain the performance level of the base invariant learning model within the in-domain setting. This evidence underscores the robustness of our approach in preserving performance both in-domain and out-of-domain.\n\n## **[W2] Comparison to Equivariant Contrastive Learning (E-SSL)**\n\nFrom the suggested baselines, E-SSL[1] and Prelax[2], we conducted experiments on E-SSL as the code is publicly available while Prelax does not. For the comparison with E-SSL, we re-implemented it by applying crop and color transformations to the original image and predicting their parameters. The results including E-SSL can be seen in Table A1 in the rebuttal PDF.\n\n## **[W3] Intra-relationship between Transformations**\n\nFigure 1 (c), in the main manuscript, demonstrates the inter-relationship between different transformation types. To further explore the intra-relationship within transformation representations, we applied various augmentations, specifically focusing on different aspects of cropping and color adjustments. For cropping, we considered variations in the box's center position and scale. For color transformations, we varied parameters such as brightness, contrast, saturation, and hue.\n\nWe employed UMAP visualizations to represent these transformations in the transformation representation space, as shown in Figure A3 in the rebuttal PDF. The results reveal that in the space, crops align according to the movement of the center position of the box along the x-axis, while color transformations align according to the degree of parameter adjustment. This arrangement suggests that the representations are sensitive to the specific parameters of the transformations.\n\nWe believe these findings support the view that transformation representations are organized in a manner that reflects their inherent properties, thereby capturing both inter- and intra-relationships effectively. This structured representation allows for a more nuanced understanding of transformation effects, supporting our hypothesis on transformation sensitivity and contributing to the broader discourse on representation learning.\n\n---\n\n**References**\n\n1. Dangovski, Rumen, et al. \"Equivariant contrastive learning.\", ICLR, 2022.\n2. Wang, Yifei, et al. \"Residual relaxation for multi-view representation learning.\", NIPS, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VnFpuNG9c4", "reviewer": "Reviewer_gZBh", "summary": "The paper introduces STL, a method for learning self-supervised equivariant representations. It suggests replacing transformation labels with representations derived from data pairs. The proposed pretext task promotes learning invariant and equivariant representations alongside transformation-related information. The method's effectiveness is showcased through various classification and object detection tasks.", "questions": "- It’s known that InfoNCE loss can be prone to dimensional collapse [7]. Having multiple InfoNCE losses (especially when invariance and equivariance losses can be contradictory) in your method, have you observed dimensional collapse happening?\n- What influence does the size of the hypernetwork $f_T$ has over the downstream tasks? \n- How does different hyperparameter configurations of $\\lambda_{inv}(\\cdot)$,  $\\lambda_{equi}(\\cdot)$, and  $\\lambda_{trans}(\\cdot)$ affect the rate of convergence while pre-training?\n- How do you extend STL to asymmetric methods like BYOL and SimSiam? Are the InfoNCE criteria still kept for equivariance and transformation-related losses?\n- Any insights into how much contribution $\\lambda_{equi}$ and $\\lambda_{trans}$ have for learning the relational information between transformations?\n\n[7] Jing, Li, et al. \"Understanding dimensional collapse in contrastive self-supervised learning.\" *arXiv preprint arXiv:2110.09348* (2021).", "limitations": "The authors have provided some limitations. It would be beneficial to include a discussion on computational costs.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper argues that existing methods treat each transformation independently as they require transformation labels, which disregards interdependency among transformations. The main strength of the paper lies in their method, which does not require transformation labels.\n- The complexity of transformations does not constrain the proposed method STL, as they demonstrate learning equivariance representations with complex transformations like AugMix.\n- Authors show empirically that the learned representations are competitive for classification and detection. They also show that STL learns equivariant representations with relational information between transformations.", "weaknesses": "- Experimental comparisons with some key existing equivariant SSL methods are missing [1, 2] and datasets (3DIEBench). \n\n- Missing several related works [3-6]. The authors should thoroughly review the relevant literature.\n\n- The approach seems computationally demanding since there are multiple (three) InfoNCE losses for each iteration. Additionally, the authors haven’t provided results/discussions on computational costs.\n\n- Information for reproducibility is limited, especially when extending STL to BarlowTwins, BYOL, and SimSiam.\n\n[1] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" *arXiv preprint arXiv:2302.10283* (2023). \n\n[2] Park, Jung Yeon, et al. \"Learning symmetric embeddings for equivariant world models.\" *arXiv preprint arXiv:2204.11371* (2022).\n\n[3] Gupta, Sharut, et al. \"Structuring representation geometry with rotationally equivariant contrastive learning.\" *arXiv preprint arXiv:2306.13924* (2023).\n\n[4] Xie, Yuyang, et al. \"What should be equivariant in self-supervised learning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.\n\n[5] Guo, Xifeng, et al. \"Affine Equivariant Autoencoder.\" *IJCAI*. 2019.\n\n[6] Gupta, Sharut, et al. \"Learning Structured Representations with Equivariant Contrastive Learning.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoroughly review our manuscript.\n\nIn response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments.\n\nWe would greatly appreciate it if you could review our responses to your comments and the submitted rebuttal PDF.\n\nSincere thanks in advance for your time and efforts.\n\n---\n\n## **[W1-1] Comparison to SIE and SEN**\n\nIn response to the reviewer's comments on the lack of comparisons with existing equivariant SSL methods, we compared our approach with SIE [1] and SEN [2]. Following the SIE paper, we used InfoNCE loss for SEN to reduce hyperparameter tuning. SIE splits representation dimensions for invariant and equivariant features. All methods, including SEN and SIE, used SimCLR as the base model. Table A1 in the rebuttal PDF shows our method is competitive with SEN and SIE, demonstrating its robustness and effectiveness.\n\n## **[W1-2] Evaluation on 3DIEBench**\n\nThe 3DIEBench dataset, while valuable in equivariance learning by pre-applying transformation with transformation label, poses a challenge in evaluating with STL due to its static nature. As transformations in 3DIEBench are pre-defined and fixed, we cannot re-apply the same transformation to different samples during training as STL requires dynamic application of transformation during training to learn the interdependency of each transformation. To utilize 3DIEBench effectively, we would not only need full access to the simulator used for generating the 3DIEBench dataset but also real-time generation to apply the pair-wise transformation during training.\n\n## **[W2] Related Work Recommendation**\n\nThank you for highlighting the missing references. We will incorporate these references in the final version.\n\n## **[W3] Computational Costs**\n\nTo empirically evaluate computational costs, we conducted an experiment using ResNet-50 with a batch size of 256. We utilize a single NVIDIA 3090 GPU, measuring the average training time per iteration over 1000 iterations following a 1000-iteration warm-up phase. As presented in Table A6 in the rebuttal PDF, the additional computational cost due to our network design is approximately 1.1 times that of SimCLR. In comparison to EquiMod, the computational cost remains nearly equivalent.\n\n## **[W4, Q4] STL Extension on Various Base Models**\n\n**STL extension on BYOL**\n\nIn our STL extension to BYOL, we utilize the dissimilarity loss function intrinsic to BYOL to define the invariant, equivariant, and transformation losses. The dissimilarity loss in BYOL is: $$\\\\mathcal{L}\\_\\\\text{BYOL} (y, y^+;g, q, \\theta, \\xi) = \\|\\overline{q}\\_\\theta (g_\\theta (y)) - \\overline{g}_\\xi (y^+) \\|_2^2$$\n\n**STL extension on SimSiam**\n\nFor the SimSiam model, the dissimilarity loss is defined as follows: $$\\\\mathcal{L}\\_\\\\text{SimSiam} (y, y^+; g, h) = \\frac{1}{2} \\\\mathcal{D}(h(g(y)), \\\\text{stopgrad}(g(y^+))) + \\frac{1}{2} \\\\mathcal{D}(h(g(y^+)), \\\\text{stopgrad}(g(y)))$$ \nwhere $\\\\mathcal{D}(a, b) = -\\frac{a}{\\\\|a\\\\|\\_2} \\\\cdot \\frac{b}{\\\\|b\\\\|\\_2}$\n\n**STL extension on BarlowTwins**\n\nIn the case of BarlowTwins, the dissimilarity loss is given by:\n$$\\\\mathcal{L}\\_\\\\text{BarlowTwins} (Y=\\\\{y\\_i\\\\}\\_i, Y^+=\\\\{y^+\\_i\\\\}\\_i; g, \\\\lambda) = \\\\sum\\_i (1 - \\\\mathcal{C}\\_{ii})^2 + \\\\lambda \\\\sum\\_i \\\\sum\\_{j \\\\neq i} \\\\mathcal{C}\\_{ij}^2$$\n\nwhere\n$\\\\mathcal{C}\\_{ij} = \\frac{\\\\sum\\_b g(y\\_b)\\_i g(y^+\\_b)\\_j}{\\\\sqrt{\\\\sum\\_b (g(y\\_b)\\_i)^2} \\\\sqrt{\\\\sum\\_b (g(y^+\\_b)\\_i)^2}}$ and $g(y)\\_i$ is the i-th dimension of $g(y)$.\n\n## **[Q1] Dimensional collapse caused by multiple infoNCE losses**\n\nThroughout our experiments, we did not observe dimensional collapse, likely due to the design and implementation of STL. Each loss function is crafted to enhance discrimination between samples, reducing collapse risk. Additionally, using a non-linear projector and appropriate batch size helped maintain dimensionality in learned representations. To ensure proper batch size for transformations, we used the aligned batch configuration proposed in this paper, preserving batch complexity by matching the number of transformation types to sample types.\n\n## **[Q2] Ablation study on auxiliary transformation backbone network**\n\nIn Table A5 in the rebuttal PDF, we present empirical results that demonstrate the impact of varying the number of layers in the auxiliary transformation backbone on the average accuracy across different downstream classification tasks. Our findings indicate that change in the number of layers in result in only marginal variations in performance. This suggests the size of hypernetwork is not a primary factor influencing the accuracy in these tasks.\n\n## **[Q3] Convergence Speed**\n\nIn Figure A2 in the rebuttal PDF, we have provided the linear evaluation results across different $\\\\mathcal{L}\\_\\\\text{equi}$ \\: $\\\\mathcal{L}\\_\\\\text{trans}$ weight ratios while fixing $\\\\mathcal{L}\\_\\\\text{inv}$ to 1. Our observations indicate that there is no significant difference in convergence speed attributable to the varying ratios.\n\n## **[Q5] Transformation representation**\n\nTable A4 in the rebuttal PDF shows the results of downstream classification and transformation prediction tasks on STL, based on different weights of equivariant and transformation losses. The best performance for both tasks is observed with weight ratios of 1:0.2 or 1:0.5. When the transformation loss ratio is reduced to 1:0.1, performance in the transformation prediction task decreases, indicating insufficient learning of transformation representations. Increasing the ratio to 1:1 prioritizes transformation learning but disrupts the balance with other components, leading to performance drops in both tasks. At a higher ratio of 1:2, transformation prediction improves significantly, but classification performance declines sharply. While a higher ratio benefits transformation learning, maintaining a lower ratio enhances the generalizability of image representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KwqCYgys2k", "reviewer": "Reviewer_18TY", "summary": "The authors propose Self-supervised Transformation Learning (STL) to learn equivariant representations. The core of the method is to not use augmentation information but instead use transformation representations obtained from pairs of data. In this sense, instead of knowing the transformation information, it is necessary to have pairs of data with the same transformation applied to them. This allows STL to leverage more complex augmentation schemes such as AugMix to reach higher performance on classification and detection benchmarks.", "questions": "Line 42-43 \"with a transformation label, each transformation is treated independently, disregarding interdependency [...] each component in color jitter is treated distinctively although they are related to each other.\" I am not sure to see the point here, as this argument can be made for any image augmentation. Giving all transformation parameters (as well as their order if it isn't fixed) gives full information about the transformation. The transformations are fully independent of each other and at best they do not commute.\n\nLine 47-48 \"After all, the reliance on transformation labels limits the performance gain in equivariant representation learning\". This claim seems unsubstantiated, are there any references or concrete evidence to support this claim ?\n\nIn Equation 2 the notation lacks preciseness. If $\\mathcal{L}$ is an InfoNCE criterion, it cannot consider samples in isolation as is written in Equation 2, but instead needs knowledge of the full batch to be computed.\n\nWhen using Color Jitter for AugSelf and Equimod, is the order of the transformations fixed or randomly selected for each call, as is default in torchvision for example ? If it is not then the same labels can be associated with different transformations which will hinder their learning.\n\nWhen considering a large set of transformations, it is possible that the transformation representations only represent some of the transformations and not all of them, or that the predictor only uses partial information (e.g. applies hue change but not brightness change). Did you perform analyses on individual transformations to study the equivariance properties on individual transformations ?", "limitations": "The authors address the limitations in section 6, notably how using pairs of transformed data is not a panacea, and can have issues to be extended to even larger transformation sets than considered with STL.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The non reliance on knowledge of the group elements leads to a method with different assumptions than previous works which can thus be used in different scenarios. Assuming knowledge of pairs of data with the same transformation can be a weaker or stronger assumption than knowledge of the group element depending on the domain, which makes STL complementary to existing works.\n\nAssuming that images are sampled randomly, using a pair of images to compute the transformation representation of another one ensures that information does not leak between the semantics of the image and the transformation representation. At inference time, applying the transformation does not require knowledge of pairs.\n\nThe performance gained using STL over existing methods is convincingly shown, across a wide range of tasks. We can however notice that aiming for equivariance over all augmentations does not lead to the best performance (Table 6),reinforcing previous findings that the benefits of invariance/equivariance can be task dependant", "weaknesses": "1\\) An important reference to [1] is missing from the paper. It was proposed in this paper to use pairs of data transformed by the same (but unknown) group element (see equation 4 for example) to learn equivariant representation. Although the considered experimental setting is different, and the way to leverage the pairs differs, it remains a very related method.\n\n2\\) With how the loss is designed, the representations are both aimed at being invariant and equivariant at the same time (albeit with a projector in between the loss and representations). As invariance is a perfectly valid solution when wanting equivariant solutions ($\\theta = Id$), it is possible that the learned representations are invariant to augmentations in the end, as illustrated in figure S2 of [2] for example. Currently, the metric defined in equation 15 doesn't provide enough information as to whether or not the predictor indeed applied transformation well (and thus whether or not the representations are equivariant).\n\nIntuitively (correct me if my understanding is wrong), the proposed metric looks at how much closer the predicted representation is to the target compared to its starting point. A value higher than 1 thus means that the prediction is closer to its target than its starting point. But here the values are barely around 1.1 which would suggest that while the predictor is indeed doing something positive, it's far from applying the transformations perfectly.\n\nCompleting this analysis with other common metrics such as the ones used in EquiMod[3] or metrics such as Hit at rank K, MRR, PRE[2,4,5] for example would provide more compelling evidence of the equivariance or not of the representations.\n\n3\\) Performance seems to not be reported on the pretraining dataset but only on other downstream tasks. It is important to report those numbers to understand the behaviour of the models both in and out of domain.\n\n\n\n\n\n[1] Shakerinava, Mehran, Arnab Kumar Mondal, and Siamak Ravanbakhsh. \"Structuring representations using group invariants.\" Advances in Neural Information Processing Systems 35 (2022): 34162-34174.\n\n[2] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" arXiv preprint arXiv:2302.10283 (2023).\n\n[3]Devillers, Alexandre, and Mathieu Lefort. \"Equimod: An equivariance module to improve self-supervised learning.\" arXiv preprint arXiv:2211.01244 (2022).\n\n[4]Kipf, Thomas, Elise Van der Pol, and Max Welling. \"Contrastive learning of structured world models.\" arXiv preprint arXiv:1911.12247 (2019).\n\n[5]Park, Jung Yeon, et al. \"Learning symmetric embeddings for equivariant world models.\" arXiv preprint arXiv:2204.11371 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoroughly review our manuscript.\nIn response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments.\nWe would greatly appreciate it if you could review our responses to your comments and the submitted PDF.\nSincere thanks in advance for your time and efforts.\n\n---\n\n## **[W1] Related Work Recommendation**\n\nAs mentioned in the review, the suggested related work [1] is similar to STL in that [1], too, utilizes the difference in the pair of images, and we will update the reference accordingly. The main difference between [1] and STL lies in utilizing the transformation group. As [1] utilizes the transformation group information for selecting the appropriate loss term utilized for training. Acknowledging this limitation [1] suggested a way to bypass this limitation, however, as STL does not require the transformation group in addition to the transformation label,  STL could be considered as a more general methodology for learning representations with pairs of images.\n\n## **[W2] Transformation Equivariance**\n\nYou are correct in noting that our metric is designed to measure the proximity of the transformed representation to the target representation relative to the original. The intention was to assess how well the equivariant transformation aligns with its image space counterpart compared to an identity transformation. However, we recognize that comparing solely to the identity transformation does not fully capture the nuances of equivariance across a variety of transformations.\nTherefore, we employed suggested metrics (MRR, H@k, and PRE) using a pretrained STL-10 model to extend our analysis to include metrics that evaluate the relative alignment of transformations in the image space. We tested on STL-10 test data subjected to transformations like individual transformations such as cropping and color jitter, as well as the standard combination of transformations used during training.\nOur results, detailed in Table A2 in the rebuttal PDF, show that our method surpasses existing techniques in most metrics, except for crop H@5 and PRE. This improvement suggests that the equivariant transformations learned by our approach more accurately reflect actual transformations in the image space compared to prior methods.\n## **[W3] In-domain Evaluation on STL-10 and ImageNet100**\nTo address the concern regarding the in-domain performance evaluation, we have included the results of the linear evaluation conducted on the pretraining dataset in Table A3 in the rebuttal PDF. Our findings demonstrate that, except for the SEN method, which focuses solely on equivariant learning, all other approaches, including STL, effectively maintain the performance level of the base invariant learning model within the in-domain setting. This evidence underscores the robustness of our approach in preserving performance both in-domain and out-of-domain.\n\n## **[Q1] Interdependencies between Transformations**\n\nWhile it is true that individual augmentations can be applied independently in the image space, STL focuses on the interdependencies between transformations in the representation space, particularly concerning equivariant transformations. Due to the length limitation, please refer to the global response for further discussion.\n\n## **[Q2] Leveraging Complex Augmentation**\n\nFrom \"[...] transformation labels limits the performance gain [...]\", we highlight that while complex augmentation like AugMix significantly enhance performance (shown in Table 1 in the main manuscript denoted as STL with AugMix), prior equivariance learning methods cannot leverage such complex augmentations due to inaccessbility of the corresonding transformation labels thereby bounding their performance gain.\n\n## **[Q3] InfoNCE with Batch Information**\n\nEquation 2 was simplified to focus on the core concept. However, as we acknowledge that the InfoNCE loss requires consideration of the full batch, we will update Equation 10 to incorporate batch interactions as follows:\n$$\n\\mathcal{L}_\\text{InfoNCE} (y, y^+; g, \\tau, \\{y_i\\}_i) = -\\log \\frac{\\exp\\left(\\text{sim}\\left(g(y), g(y^+)\\right)/\\tau\\right)}{\\sum{y_i \\neq y} \\exp\\left(\\text{sim}\\left(g(y), g(y_i)\\right)/\\tau\\right)}\n$$\nThis reflects the necessary batch-level computation. Similarly, Equations 11-13 should be conditioned on the batch samples, aligning the formulation with the InfoNCE criterion’s requirements which will updated in the final version accordingly.\n\n## **[Q4] The order of the transformations**\n\nIn our implementation for AugSelf and Equimod, the order of transformations, including Color Jitter, is fixed rather than randomized. As mentioned in your review, this consistency must hold to prevent any association of different transformations with the same labels, thereby supporting effective learning and representation alignment.\n\n## **[Q5] Analysis on individual transformations**\n\nWe have conducted a detailed analysis of individual transformations to examine their equivariance properties in Table 5 in the main manuscript. Furthermore, we analyzed the similarity between equivariant transformations and their corresponding transformations using metrics such as MRR, Hit@k, and PRE. For this analysis, we employed an STL-10 pre-trained model, transforming the STL-10 test data using crop, color jitter, and combinations of augmentations. Table A2 in the rebuttal PDF shows that our method surpasses existing approaches in all metrics except for H@5 and PRE in the crop transformation. This superiority demonstrates that equivariant transformations learned through STL effectively capture and reflect the actual transformations applied.\n\n---\n\n**References**\n1. Shakerinava et al., \"Structuring representations using group invariants.\", NIPS 2022.\n2. Garridov et al., \"Self-supervised learning of split invariant equivariant representations.\", ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oGDD8QpQPR", "reviewer": "Reviewer_KJt3", "summary": "This paper is concerned with the learning of expressive representations in a self-supervised fashion. In particular, it aims at learning representations that", "questions": "No questions but a suggestion for improvement, I think section 4.3, first part, is intriguing and it is not clear to me why equivariance and transformation learning are _complementary_. Discussing further the intuition behind these results would I think improve the paper.", "limitations": "Limitations are discussed and in particular, the main limitation I see to this work (i.e., the framework requires access to a pair of images that have undergone the same transformation, this weak form of supervision is not always accessible in the wild), is mentioned.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Presentation: The paper is clearly written, well-structured, easy to follow, and overall is in a good state \n- Proposed approach: the idea proposed by the authors is not ground-breaking but simple, makes sense, and is efficient. It applies the same reasoning used in invariant self-supervised learning to equivariant representation learning.\n- Evaluation pipeline: the authors evaluate their method quite thoroughly, evaluate semantic classification across a large number of datasets, evaluate object detection, and check the sensitivity to hyper-parameters and transformation prediction. \n- Experimental results: results show that the approach effectively learns representations that perform well on both semantic and non-semantic information in comparison to baselines. Sensitivity to hyperparameters seems limited.", "weaknesses": "- My main concern regards the baselines chosen for evaluation. Authors explicitly mention that SIE and SEN are excluded from the evaluation due to the need for these methods to have access to transformation labels which STL does not require. While I see how these methods require more \"supervision\" than the proposed method, STL still requires weak supervision (i.e., access to pairs of images that have undergone the same transformation). Cases, where one would have access to such weak supervision but not access to transformation labels, are quite rare (e.g., AugMix as mentioned by authors), as a consequence I think the evaluation should include these baselines, and results should be analyzed considering this light difference in supervision level. My score will be increased if this concern is addressed.\n- Experimental results are computed for a single seed, for object detection results, the performance gap between EquiMod and STL is narrow hence additional seeds would make the evaluation more robust. \n- In Table 1, STL+AugMix can be compared to SimCLR *with* AugMix augmentations which I do not believe is reported in this table.\n\nMinor:\n- line 40: typo", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoroughly review our manuscript.\n\nIn response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments.\n\nWe would greatly appreciate it if you could review our responses to your comments and the submitted rebuttal PDF.\n\nSincere thanks in advance for your time and efforts.\n\n---\n\n## **[W1] Comparison to SEN and SIE**\n\nIn response to the reviewer's comments on the absence of experimental comparisons with existing equivariant self-supervised learning (SSL) methods, we have conducted comparisons with the SIE[1] and SEN[2] methods.\n\nAs in the SIE paper, our implementation of the SEN method adopts the InfoNCE loss instead of the triplet loss to minimize the need for extensive hyperparameter tuning. Additionally, SIE employs a strategy where the representation dimension is divided to separately address invariant and equivariant representations. For consistency, all methods, including SEN and SIE, were trained using SimCLR as the base model.\n\nTable A1 in the rebuttal PDF presents the results, demonstrating that our method performs competitively against both SEN and SIE, supporting the robustness and effectiveness of our approach.\n\n## **[W2] Various Seeds**\n\nWe appreciate the reviewer’s feedback and have addressed the concern regarding the robustness of our results. In response, we conducted additional experiments using different random seeds for pretraining to ensure that the observed improvements were statistically significant. The results of these additional experiments are presented in Table A1 in the rebuttal PDF, which illustrates that our method, STL, consistently outperforms baseline approaches across various downstream tasks, achieving improvements ranging from an average of 3% to nearly 10%. These results demonstrate that STL significantly enhances performance compared to existing methods, confirming the robustness and generalizability of our approach.\n\n## **[W3] STL with AugMix**\n\nWe have indeed conducted experiments applying AugMix to SimCLR, and the results are documented in Table A1 in the rebuttal PDF. Our findings indicate that while incorporating AugMix with SimCLR yields a performance improvement compared to using standard augmentations, the model using STL with AugMix demonstrates superior performance. This suggests that our STL approach effectively leverages the AugMix augmentations to enhance representation learning beyond the capabilities of SimCLR with AugMix.\n\n## **[W4] Detailed Comments**\n\nThank you for bringing the typo on line 40 to our attention. We will correct the error in the final manuscript. We appreciate your meticulous review.\n\n## **[Q] Complementary Relation**\n\nTable 5 in the main manuscript illustrates the performance of three models: the Only Equivariance model(employing $\\\\mathcal{L}\\_\\\\text{inv}$ and $\\\\mathcal{L}\\_\\\\text{equi}$), the Only Transformation model (employing $\\\\mathcal{L}\\_\\\\text{inv}$ and $\\\\mathcal{L}\\_\\\\text{trans}$), and the STL model (incorporating $\\\\mathcal{L}\\_\\\\text{inv}$, $\\\\mathcal{L}\\_\\\\text{equi}$, and $\\\\mathcal{L}\\_\\\\text{trans}$). The results include the average accuracy in downstream classification tasks, and the regression and classification outcomes for transformation prediction tasks.\n\nThe Only Equivariance model significantly improves downstream task performance over SimCLR, but its transformation prediction capabilities are limited. Conversely, the Only Transformation model excels in transformation prediction compared to SimCLR but shows limited improvement in downstream tasks. In contrast, the STL model, which integrates both $\\\\mathcal{L}\\_\\\\text{equi}$ and $\\\\mathcal{L}\\_\\\\text{trans}$, demonstrates enhanced performance in both downstream tasks and transformation prediction, empirically validating their complementary nature.\n\nThe mechanism of this complementarity lies in the ability of the transformation representations, learned through $\\\\mathcal{L}\\_\\\\text{trans}$, to facilitate the effective learning of equivariant transformations in the representation space. Specifically, the transformation representations allow $\\\\mathcal{L}\\_\\\\text{equi}$ to ensure that transformations in image space correspond accurately to transformations in representation space. This correspondence is demonstrated by the improved performance metrics, such as MRR, H@k, and PRE, for STL's equivariant transformations compared to the Only Equivariance model, as evidenced by the results in Table A2 in the rebuttal PDF.\n\n---\n\n**References**\n1. Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\", ICML, 2023.\n2, Park, Jung Yeon, et al. \"Learning symmetric embeddings for equivariant world models.\", ICML, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks for 

Dear Reviewers and Area Chairs,

We thank the reviewers for their constructive feedback. We are glad to take various helpful reviewer comments to clarify and complete our work. Reviewers agreed on the originality, motivation, soundness, and significance of the paper. In here, we breifely recap the gloal of our paper and the proposed methods, **Self-supervised Transformation Learning(STL).**

Previous methods using transformation labels are divided into explicit equivariant learning, which learns equivariant transformations, and implicit equivariant learning, which focuses on transformation prediction, as shown in Figure A1 in rebuttal PDF. Our approach differs by learning transformation representations without transformation labels, enabling explicit equivariant learning. This method allows STL to effectively capture transformation-sensitive information without relying on predefined labels.

The primary goal of STL is learning better equivariant representation, which learns transformation-sensitive information. Transformations that induce similar semantic changes in the image space should correspondingly yield similar changes in the representation space. For instance, transformations affecting color information differ in their semantic impact compared to spatial transformations like cropping, which alter relative spatial information and proportions within an image.

Figures 1(a) and (b) of our paper illustrate the visualizations of learned equivariant transformation's functional weights for corresponding transformations. In these visualizations, previous approaches, such as EquiMod, treats transformations like crop and color changes as independent mappings. In contrast, STL demonstrates that color-related equivariant transformations, which share similar semantic changes, are learned with similar mappings in the functional space. This highlights STL's ability to capture the nuanced interdependencies between transformations more effectively. Together, the experimental results firmly demonstrate the represtntation ability of STL, which outperforms existing methods in 7 out of 11 classification tasks and shows superior average performance.  

Our approach underscores the importance of recognizing these interdependencies to enhance semantic understanding within the representation space, thus improving the effectiveness of equivariant representation learning. Additionally, STL effectively learns intra-relationships within transformations (Figure A3 in the rebuttal PDF), and experiments show that the equivariant transformations learned by STL reflect actual transformations more accurately than existing methods, as demonstrated by equivariance metric measurements (Table 5 of our paper and Table A4 in the rebuttal PDF). Unlike previous methods, STL can leverage complex transformations such as AugMix that were previously inaccessible due to the lack of transformation labels (Table 1 of our paper).

Meta-review: This paper received mixed reviews of 4,4,6,6. Overall the paper proposes a self-supervised representation learning technique for learning equivariant representations that can be applied to various settings such as BYOL, SimCLR, Barlow Twins and SimSiam. The method obtains strong performances and is presented clearly. While at the beginning of the discussion phase reviewers pointed out missing experiments, and discussions of related works, the authors provided these to a satisfying degree such that the engaged reviewers upgraded their scores. Overall, this work is of interest to a large part of the vision community of NeurIPS and the AC recommends acceptance.
Decision: Accept (poster)