Soft Equivariance Regularization for Invariant Self-Supervised Learning
Authors: Joohyung Lee
Changhun Kim
Hyunsu Kim
Kwanhyung Lee
Juho Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Soft Equivariance
Self-Supervised Learning
Invariant Representation
Vision Transformer
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: SER adds a soft geometric equivariance loss on an intermediate ViT token map while keeping the base SSL loss on the final embedding, improving ImageNet-1k/robustness and COCO with ~1% extra FLOPs.

Abstract: Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations (e.g., random crops and photometric jitter). While effective for recognition, enforcing strong invariance can suppress transformation-dependent structure that is useful for robustness to geometric perturbations and spatially sensitive transfer. A growing body of work, therefore, augments invariance-based SSL with equivariance objectives, but these objectives are often imposed on the same **final** (typically spatially-collapsed) representation. We empirically observe a trade-off in this coupled setting: pushing equivariance regularization toward deeper layers improves equivariance scores but degrades ImageNet-1k linear evaluation, motivating a layer-decoupled design. Motivated by this trade-off, we propose **Soft Equivariance Regularization (SER)**, a plug-in regularizer that decouples where invariance and equivariance are enforced: we keep the base SSL objective unchanged on the final embedding, while softly encouraging equivariance on an \emph{intermediate spatial token map} via analytically specified group actions $\rho_g$ (e.g., $90^{\circ}$ rotations, flips, and scaling) applied directly in feature space. SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only **1.008$\times$** training FLOPs. On ImageNet-1k ViT-S/16 pretraining, SER improves MoCo-v3 by **+0.84** Top-1 in linear evaluation under a strictly matched 2-view setting and consistently improves DINO and Barlow Twins; under matched view counts, SER achieves the best ImageNet-1k linear-eval Top-1 among the compared invariance+equivariance add-ons. SER further improves ImageNet-C/P by **+1.11/+1.22** Top-1 and frozen-backbone COCO detection by **+1.7** mAP. Finally, applying the same **layer-decoupling** recipe to existing invariance+equivariance baselines (e.g., EquiMod and AugSelf) improves their accuracy, suggesting layer decoupling as a general design principle for combining invariance and equivariance. Code is available at https://github.com/aitrics-chris/SER.

Published as a conference paper at ICLR 2026
SOFT EQUIVARIANCE REGULARIZATION FOR INVARIANT SELF-SUPERVISED LEARNING
Joohyung Lee'
Changhun Kim
Hyunsu Kim?
Kwanhyung Lee
Juho Lee
AITRICS
2 Voronoi Inc.
Korea Advanced Institute of Science and Technology (KAIST)
(chris, changhun. kim, destin) wwwteeeeetittiiittoeeeoeeeeeeeee
hk@voronoi. io, juholeeOkaist. ac.kr
ABSTRACT
Self-supervised learning (SSL. typically learns representations invariant to semantic-preserving augmentations (e.g., random crops and photometric jit- ter). While effective for recognition, enforcing strong invariance can suppress cesstormaationddeeedeett structure that is useful for robustness to geometric per- turbations and spatially sensitive transfer. A growing body of work, therefore, augments invariance-brr based SSL with equivariance objectives, but these objectives are often imposed the same final (typically spatially-collapsee) representation. We empirically observe trade-off in this coupled setting: pushing equivariance regularization toward deeper layers improves equivariance scores but degrades ImageNet-1k Ik linear evaluation, motivating a layer- design. Motivated by this trade-off, propose Soft Equivariance Regularization (SER). plug- in regularizer that decouples where invariance and equivariance are enforced: we keep the base SSL objective unchanged the final embedding. while softly encouraging equivariance on an intermediate spatial token map via analytically specified group actions Pg (e.g., 90" rotations, flips, and scaling) applied directly in feature space. SER learns/predicts requires no auxiliary transformation prreticctionnniiiinnnn head, and adds only 1.008 train- ing FLOPs. On ImageNet- 1k ViT-S/16 pretraining, SER improves McCo-v3 by +0.84 Top-1 in linear evaluation under strictly matched 2-view setting and con- sistently improves DINO and Barlow Twins; under matched view counts, SER achieves the best ImageNet-Ik linear-eval Top-1 among the compared invari- ance+equivariance add-ons. SER further improves ImageNet-C/P by +1.11/+1.22 Top-1 and frozen- backbone COCO detection by +1.7 mAP. Finally, applying the same layer-decoupling recipe to existing invariance+equiiariancc baselines (e.g.. EquiMod and AugSelf) improves their accuracy, suggesting layer decoupling as a general design principle for combining invariance and equivariance. Code is avail- able at httpt:::///// http::://githbb... com wwmmm/aitiisssoo chris/ SER
per-sample transformation codes/labels,
INTRODUCTION
Self-supervised learning (SSL) is a central paradigm for visual representation learning (Chen al. 2020 Caron et al. 2021 Wang et al. 2023). enabling strong features to be learned from large- scale unlabeled data. Many successful SSL methods follow a simple principle: learn representations that are invariant to a distribution of semantic-preserving augmentations, such as random crops and photometric jitter. This invariance highly effective for recognition, but can also suppress mannsfoomationddeeeddeee structure (e.g.. orientation, reflection, or scale cues) useful for geometric robustness and spatially sensitive transfer. A complementary principle is equivariance: rather than discarding transformation information, equivariant representation changes in predictable and structured way when the input is transformed (Dangovski et al.) 2021 Marchetti et al. 2023).
Prior work that incorporates equivariance into SSL be broadly grouped into two families Yu ct al. 2024p. Implicit approaches encourage transformation awareness through auxiliary objectives (e.g., predicting contrasting transformations) (Dangovski et al. 2021
Lee al. 2021). Explicit

---

Published as a conference paper at ICLR 2026
approaches aim to model how features transform in latent space, often introducing auxiliary trans- formation/action modules or learning/predicting per-sample transformation codes/labels as training targets (Devillers & Lefort 2023 Park et all, 2022 Garrido al., 2023 Yu et al., 2024). At Im- ageNet scale, equivariant SSL has also been explored in hetpiiiive/worldmmmaeee settings (Garrido) et al. 2024). In this work, we study complementary problem: how incorporate an explicit equív- ariance regularizer on top of strong invariance- Twins) for ViTs, without changing the architecture or the baseline invariance objective.
SSL backbones (MoCo-v3, DINO, Barlow
A common design choice in prior invariant-equivariint SSL methods is to impose both objectives on the final representation. However, the final representation is typically spatially collapsed, and thus poorly aligned with spatial group actions. More importantly, we empirically observe a trade- off when equivariance regularization is pushed toward the output: equivariance scores increase, but ImageNet- kk einlur-evaluation accuracy consistently decreases (Figure 3 and table 4. This moti- vates decoupled design in which invariance and equivariance are encouraged at different layers.
We propose Soft Equivariance Regularization (SER), simple yet scalable regularizer that de- couples where invariance and equivariance are enforced. SER trains the final embedding with an invariant baseline SSL objective, while softly regularizing sputially struclured intermediate repre- sentations toward equivariance via analytically known feature-space space group actions Pg. Since some common SSL augmentations (notably random cropping) SER uses a simple batch-pan aatch-pariitio strategy: the full augmentation pipeline contributes to the invari- ance objective, while equivariance is enforced only for an invertible geometric subset.
non-invertible and do not form group,
Transformation labels. Following the terminology of STL. Yuu al. 2024), by "no transforma- tion labels" we mean that SER does not learn or predict per-sample transformation labels/codes as additional supervision. SER still assumes the standard (known) augmentation pipeline and uses the resulting relative transform g 929 but does not introduce a transformation prediction head.
cranffrmmatoon-prcditttoon
We evaluate SER on ImageNet-Ik 1k pretraining of ViT-S/16 and a suite of robustness and transfer benchmarks. Because the number of views significantly affect SSL performance, empha- size matched-view comparisons and report results under strictly matched settings. Across strong invariance-based SSL methods (MoCo-v3, DINO, Barlow Twins), SER consistently improves ImageNettscale classification and robustness (e.g.. ImageNet-C/P), and yields larger gains on spa- tially sensitive transfer such as fozee-backboon COCO detection. Furthermore, we show that the same layer-decoupling modification can improve existing invariancetequivariinn baselines, sug- gesting general recipe for combining invariance and equivariance in practice.
Contributions. Our contribution is fourfold:
Empirical trade-off at the final layer. We show that imposing invariance and equivariance on the same final representation can be suboptimal: pushing equivariance regularization toward later layers increases equivariance scores but reduces ImageNet- 1k lvnear-evaluation accuracy (Figure 3 and table 4).
Layer-decoupled soft equivariance regularization. We propose SER, which promotes equivariance on an intermediate spatial representation while leaving the baseline SSL loss unchanged the final embedding.
Analytic feature-space group actions without extra modules. SER applies analytically specified actions Pg intermediate feature maps, avoiding auxiliary transformation/accioon modules and avoiding learne//reedictdd per-sample transformation codes/labels.
Layer decoupling upgrades prior inv+equiv methods. Beyond our loss design, we show that moving the equivariance objective of prior invarianttequivarian methods (e.g., EquiMod, AugSelf) from the final layer an intermediate layer improves their accuracy as well, indicating that layer decoupling is broadly beneficial.
2 BACKGROUNDS
2.1 SELF-SUPERVISED LEARNING
self-supervised learning (SSL) leverages supervisory signals derived from the data itself, reducing reliance on human-annotated labels. Early SSL methods used pretext tasks such as rotation predic-
2

---

Published as a conference paper at ICLR 2026
tion (Gidaris et al. 2018) solving jigsaw puzzles (Noroozi & Favaro 2016). Modern SSL in vision predominantly learns representations by enforcing invariance across augmented views of the same image, often via instance discrimination redundancy-reduucion objectives (Chen al. 2020 He et al. 2020 Grill et al. 2020; Zbontar et al. 2021: Bardes et al.] 2022). Concretely, a training sam- ple is transformed into multiple views by independently sampling augmentations from predefined distribution (e.g., crops and photometric jitter), and the model is optimized so that representations of these views agree.
Different SSL frameworks instantiate this invariance principle with different objectives. Contrastive approaches such as SimCLR and MoCo (including MoCo-v3) use noise-contrastive estimation losses (Chen et al.] 2020; He et al., 2020) [Chen & He. (02T), while non-con con-contrastive methods such as BYOL and SimSiam minimize a similarity loss between predicted and target embeddings (Grill et al. 2020 Chen & He 2021). Redundancy-reduucionn methods such as Barlow Twins and VICReg enforce agreement while discouraging collapse through covariance-basse
regularization (Zhontar et al. 2021 I Bardes al.) 2022). Many frameworks use two global views by default, while increas- ing the number of views (e.g., multi-crop training) can improve performance at the cost additional compute and memory (Caron et al. 2020 2021
SER is designed to be complementary to these invariance-base objectives: we keep the base SSL loss unchanged on the final embedding, and add an auxiliary equivariance regularizer applied to an intermediate spatial representation (Section 3.3)
2.2 EQUIVARIANT REPRESENTATION LEARNING
Invariant SSL encourages representations to ignore sugmentatiooninndues variability, which can be beneficial for recognition but may suppress sentation learning instead aims encode how representations should change under transformations in structured way, complementing invariance (Dangovski et al. 2021 Marchetti et al. (023). Re- cent approaches that incorporate equivariance into SSL can be broadly grouped into implicit and explicir families (Yu et al., 2024).
cetprmmattiodddeedeeeee structure. Equivariant repre-
Implicit methods. Implicit approaches encourage transformation awareness via auxiliary objectives, for example by predicting transformation parameters/labels or contrasting transformations across views (Dangovski et al., |02T] Lee al., 202I). While effective for certain transformations, such objectives can be challenging to scale to complex compositions or to disentangle multiple simulta- neously applied augmentations.
Explicit methods. Explicit approaches attempt to model how features transform in latent space, often by introducing auxiliary transformation/accionn modules or by learnin//predicting per-sample transformation codes/labels training targets (Devillers & Lefort, 2023 Garrido et al. 2023 Yu ct al. 2024). A common practical design is to impose equivariance constraints on a spatially col- lapsed representation (e.g.,pooled CNN features or VIT [CLS] token), which limit sensitivity to fine-grained geometric changes and makes spatial group actions less directly applicable.
SER. In this work, focus simple and scalable alternative for ViTs:
impose soft equiv- ariance regularizer on an intermediate, spatially structured representation (a token map), while keep- ing the final embedding optimized purely by the original invariance-basee based SSL objective. This de- coupling avoids applying equivariance to a collapsed representation and avoids learning/predicting transformation codes/labels or training auxiliary transformation-predicche heads.
2.3 SYMMETRY, GROUPS, AND EQUIVARIANCE
Symmetry refers to a transformation that preserves relevant structure of an object (Bronstein et al., 2021]. For example, rotating a perfect circle around its center does not alter its appearance. The set of such transformations can form group & under a binary operation (here, composition), satisfying closure, associativity, an identity element, and inverses.
To formalize transformation behavior, we consider group representations. A (linear) representation a homomorphism p: G - GL(n);; we denote p(g) by Pg. A function f: + y G-equivariant if, for all E G and x E X,
s(p(())) = Po (s))))
3

---

Published as a conference paper at ICLR 2026
((1
(ft2) (())
311
E g
fiz)
Forward Equivariance
g g
P (o()))
Cine
Invariance
Aequir
fl
((3)
92 EG
Pg(x1)
(p)(a)))
(1() ( (P
Figure 1: Overview of SER. For each image in b2, we sample two views from the equivariant- view policy Teq. We decompose each sampled transform into geometric component y and photometric component (e.g., color jitter), and denote by 91. G the geometric parts of the two views. We use the relative transform g 9291 to align intermediate token maps feature space. For clarity, the independently sampled photometric components in Teg are omitted in the diagram.
For simplicity, overload Pg denote the induced actions both the input and output spaces; in practice these actions may differ. Note that invariance is a special case of equivariance where Pg is the identity map for all g.
In practice, many models are only approximately equivariant, Hard equivariance enforced by architecture design (e.g., group-equivariant CNNs (Cohen & Welling 2016)), whereas soft equiv- ariance is encouraged by a regularization term weighted by coefficient (Finzi et al., [02 Kim al., )023). CNNs provide a canonical example: convolutional layers are (approximately) equivari- ant to translations, up to discretization and boundary effects. Motivated by such symmetries, many architectures encode broader transformation groups, including group-equivariant CNNs (Cohen & Welling 2016 2016) (017) and equivariant graph networks (Keriven & Peyré 2019), often improving data efficiency and generalization.
3 SOFT EQUIVARIANCE REGULARRIZATIN FOR INVARIANT SELF-SUPERVISED LEARNING
Previous methods for introducing equivariance into invariant SSL often impose both invariance and equivariance objectives the final representation. However, the final representation is typically spatially collapsed, which well-suited for invariance but poorly aligned with spatial group actions. SER instead encourages equivariance at intermediate spatial representation that retains lattice structure, where analytic feature-space actions are naturally defined.
3.1 SOFT EQUIVARIANCE INTERMEDIATE FEATURES
A straightforward way to introduce equivariance is impose it directly on the final representation, as explored in prior work. However, these final representations are spatially collapsed and do not admit natural spatial action. We therefore regularize non- [CLS] patch tokens where spatial structure is still explicit (e.g.. the gray feature maps in Figure T.
For ViTs, to preserve clean spatial token map for equivariance, decompose the encoder as
f = (2) o (()
where (() is structure-preserviin feature extractor that outputs spatial token map, and f(2 is an invariance-oriented head that produces the final embedding. We introduce the [CLS] token only the input of f(2), so does not affect the spatial structure of the intermediate representation produced by (1) 2p.
4

---

Published as a conference paper at ICLR 2026
Geometric group for equivariance. Let G denote the set of invertible geometric transformations used by SER (anisotropic scaling without cropping, horizontal flips, and 90" rotations). SER defines the input-space and feature-space actions Py only for g E g.
We adopt the standard two-view SSL protocol (Gupta et al. 2023). Given an image æ and two sampled geometric elements J1, 92 G we form two views
31 Pm (x),
222
Paz (z).
where Py denotes applying the corresponding geometric transform in image space. (We omit pho- tometric augmentations in notation: in practice, both views are also independently subjected to the standard photometric jitter from the base SSL pipeline, but we do not define a feature-space action for these non-group transforms. Consequently, Lequiv enforces geometric equivariance while being evaluated under potentially different photometric perturbations.) Let 929 denote the relative transform such that Z2 = Py (11) SER enforces equivariance on the intermediate spatial token map produced by the prefix encoder (-) by minimizing the equivariance error
Lequiv = En
f (12))
g = 9291
where -(..-) measures discrepancy between spatial token maps; in this work we instantiate d with a patch-wise contrastive loss (Sec. (.3). This form of equivariance objective has appeared before g.... Eq. 4 in Yu et al. 2004));; our key difference that we apply it to an intermediate, spatially structured representation and use analytically specified feature-space action Pg directly, without training an additional cunsformation-prediiee prediction module or latent action network. Minimizing Cequiv does not enforce exact equivariance, but rather encourages soft equivariance at that layer.
Feature-space action. Since (-) is defined on regular patch grid, we implement Ps directly on the token map: discrete rotations and flips correspond to token permulations on the grid, while scaling is implemented by deterministic grid resampling with the same resize operator as the input- space transform. This lets us apply the known geometric action in feature space without learning an additional action network.
Because Lequiv alone does not provide an instance-diseriminatio signal for the final embedding, we jointly train (2) with a standard invariance- SSL loss on the final representation (c.g., the [CLS] token), in MoCo-v3, DINO, and Barlow Twins (Chen al. 2020) He et al. 2020 Grill et al., 2020) (Zbontar et al.] 2021). Importantly, SER adds no extra transformation/actioo module (unlike, c.g., EquiMod or STL (Devillers & Lefort 2023
Yu al. 2024
3.2 AUGMENTATION POLICY AND BATCH PARTITIONING
Typical augmentation policies used in invariance- SSL include RandomRes' i zedCrop, RandomHor www.tolllll.oo and photometric modifications such as color jittering and grayscale. However, RandomResion ierbommeeiiedroooog zedCrop with cropping does not form group: the discarded region can- not be recovered by an inverse transform. More importantly, cropping changes the spatial support of the image; result, the relative transform g 929 and the corresponding action on spatial token map are not well defined.
Therefore, SER splits each mini-batch into two sub-baue aub-batches (Figure 2 The first sub-batch b fol- lows the baseline augmentation policy T used by the underlying invariant SSL method. The second sub-batch by follows modified equivariant-view policy Teq that disables cropping while retaining photometric jitter and sampling geometric transforms from the invertible group G:
b T,
ba Teq
Concretely, we define
Toq T (Random Crop) (Rotation 90%)
implemented by replacing RandomResi MandomResizzeCrop with its invertible geometric component (anisotropic scaling without cropping) and adding discrete 90" rotations. Thus, Teg retains the full photometric pipeline from T (color jitter, grayscale, blur, solarization) but samples its invertible geometric transforms from & (anisotropic scaling without cropping. horizontal flips, and 90º rota- tions). Equivalently, each draw from Teg can be decomposed into geometric transform g E G and
5

---

Published as a conference paper at ICLR 2026
mini-batch partition b
Image Encoder, e.g., VIT
LET
CLS
Equivariant
Invariant
Linv
Fog Tea
Lequiv
mini-butch partition bg
Figure 2: overview of the training pipeline. The mini-batch is split into bj and by: b uses the baseline SSL augmentation policy TT (including cropping), while b2 uses an equivariant-vier policy Toq that disables cropping and adds discrete 90" rotations while retaining the baseline photometric jitter. Both bi and b2 contribute to the baseline invariance loss. SER additionally applies equivari- ance regularizer to an intermediate spatial token map using only the invertible geometric component of the b2 transforms to define the feature-spar cetture-ppcce action Pg-
photometric transform p. SER uses only g to define its feature-space action Pg. meaning photomet- ric transformations action. Consequently, fequiv enforces only geometric equivariance, though optimizing under inde- pendently sampled photometric jitter between the views can additionally encourage photometric robustness at the regularized layer. Both 6, and by contribute to the baseline invariant SSL loss; b additionally contributes the equivariance regularizer.
elements of the geometric group G and have no associated feature-space
Scaling implementation. Scaling in Pg reuses the same resize operator (interpolation kernel and padding behavior) as the base SSL data pipeline keep comparisons fair. For ViT-S/16, we restrict scaling factors so that the transformed image remains aligned to the 16-pixel patch grid, ensuring the resulting token map lies on a regular lattice. Rotations by 909 and horizontal flips are exact token permutations and require no interpolation.
3.3 TRAINING OBJECTIVE FOR SOFT EQUIVARIANCE REGULARIZATION
We instantiate equivariance regularizer in Equation (2) using patch-wise NT-Xent (noise- contrastive) loss applied on the sub-batch by (Chen et al. 2020 In practice, reuse the same paired views (2) sampled from Teg to compute both the baseline invariant loss Linv and the equivariance regularizer Lequivi SER therefore does not require additional crops beyond the standard two-view protocol per image.
Let H,, W,, Dy denote the height, width, and channel dimension of the intermediate feature maps. For each æ E b2, we sample two augmented views from Ten and denote their associated geometric components by 91.92 G; for brevity write æ Pas (x) and x22 Paz (x) and omit the independent photometric jitter applied to each view. We then compute
z Pg (f())
z' 2(=)(
=
with z' z' E xWD We index images in by by i and spatial locations by j E (O...., HyW, 1}. and denote by žij and z/y the corresponding feature vectors from z and z', respectively. Each vector is projected by 2-layer MLP with GELU into 312-dimensional space (Caron et al. 2021 The equivariance contrastive loss for anchor (i.j)
exp(s(z4p) 2/1²))
Cij Cquiv = -lo exp(s(zij) (4))) + Emgii En exp(s ((25 Zmn)) Empre En exp(s(22,,,1,) 2/mn))

---

Published as a conference paper at ICLR 2026
where s2x y) denotes cumperature-scaled cosine similarity, s(x, v) = 1 x y ( We set T 0.3 for MoCo-v3 and Barlow Twins, and T = 0.5 for DINO. Following O Pinheiro et al. 2020,, negatives are sampled only from other images in the batch.
The overall equivariance loss averages this quantity over all images and spatial locations in b2 and contributes the full training objective, where À controls the strength of equivariance regularization.
C = Linel + Linv2 + X Lequiv,
Lequiv
b W ΣΣ
Ci. equiv
(3)
Both Linvi and Linv2 use exactly the baseline invariance-based SSL. loss (e.g. MoCo-v3, DINO, Barlow Twins) applied to sub-batches 61 and b2, respectively. Thus, SER is agnostic to the choice of base SSL algorithm and can be seamlessly integrated with different invariance- methods.
4
EXPERIMENTS
We empirically evaluate Soft Equivariance Regularization (SER) against representative equivariant SSL baselines and strong invariant SSL. backbones. We first describe the experimental setup (Sec- and then address the following questions:
tion 4.1
Does SER improve VIT representations compared purely invariant SSL and prior equiv- ariant SSL add-ons?
Does SER scale to ImageNet-Ik pretraining, and how does it impact robustness and spa- tially sensitive transfer?
Which design choices are responsible for the gains (layer placement, [CLS] insertion, and the equivariance loss itself)?
4.1 EXPERIMENTAL SETUP
Baselines. We compare SER with representative equivariant SSL approaches, including im- plicit methods (E-SSL (Dangovski et al. 2021). AugSelf (Lee et al. 202I)) and explicit methods (EquiMod (Devillers & 2023) STL Yu al. 2024)). Because several baselines use different numbers of views (e.g.. EquiMod: 3 global crops; E-SSL: global 4 local crops), direct compar- isons can be confounded by view count. Since increasing the number of views typically improves SSL performance but increases compute and memory (Caron et al.) 2020 2021). we emphasize matched-view comparisons. addition to the standard 2-view setting, we also report SER under a matched 2+4 scheme for fair comparison with methods that use local crops.
Datasets and evaluation benchmarks. We pretrain and evaluate on ImageNet-Ik (Deng et al. 2009). following standard SSL protocols (Chen et al.) 2020; Caron et al. 2021). We addition ally evaluate generalization under natural distribution shifts (ImageNet-V V2 (Recht al. 2019 ImageNet-R (Hendrycks et al., 2021), ImageNet-Sketch (Wang et al. 2019)) and robustness under corrupiions/peettrrationss (ImageNet-C and ImageNet-P (Hendrycks Dietterich) 2019)) ImageNet-P emphasizes geometric perturbations (c.g., translation, rotation, and scaling). whereas ImageNet-C primarily focuses on appearance- We also evaluate out-of-domain transfer on 3DIEBench (Garido et al. 2023) via linear probing on its 2D renderings (we do not exploit or supervise 3D transformation parameters), and evaluate spatial transfer with frozen-backbone COCO object detection.
corruptions (c.g.. blur, noise, brightness/fog).
Implementation details. Unless otherwise noted, we pretrain VIT-S/16 on ImageNet- 1k. SER is implemented as an add-on to invariant SSL backbones (MoCo-v3 (Chen et al. (021). DINO (Caron) al. 2021), Barlow Twins (Zbontar et al. 202 h)); preserve their architectures and default hy- perparameters. Our modifications are limited to: (i) partitioning each mini-batch into b, (baseline augmentations) and by token after the aquivariance-regulatii layer to preserve a spatial token map; and (iii) adding the equivariance regularizer and its projection MLP.
augmentations for equivariance); (ii) inserting the [CLS]

---

Published as a conference paper at ICLR 2026
Table 1: Top-1 and top-5 accuracy (%) under linear evaluation. Note that all equivariant repre- sentation learning methods use MoCo (He et al. 2020) as their baseline, which outprrormedd DINO and BarlowTTinn
our setting (see Table 2). Concatenated [CLS] tokens from the last 4 layers were used as an input to the linear classifier, tollowing the feature-based evaluations in (Devlin et al.)
2019 Caron al., 2021). 'View' refers to the number of crops sampled per image (see Section for more detail). ImageNet-Ik Ik scores are averaged over 3 runs.
View
Algorithm
Param (M)
ImageNNt--1k
ImageNNetSketch Top
ImageNet-V2 Top
ImageNet-R
3DIEBench
Top
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
MoCo-v3
42,9
68.44
#0.07
88.02
10.04
31.87
56.54
78.68
AugSelf
43.7
67.55
87.62
25.35
53.74
76.68
2 view
+0.05
+0.05
65.49
85,91
STL
62.2
15.40
28.96
55.43
78.02
17.22
28.49
#0.12
+0.08
69.28
88.79
SER
43.4
#0.01
+0.02
17.68
32.54
56.95
79.29
18.95
30.72
70.17
92.78
view
EquiMed
43.3
+0.02
+0.01
14.81
28.11
56.31
79.93
16.54
27.32
67.97
91.97
70.6
89.85
E-SSL
43.3
19.23
34.77
58.33
80.93
19.86
32.36
view
+0.04
#00.2
71.56
90.04
SER
43.4
+0.03
#0.01
19.76
34.81
59.50
80.72
20.27
32.54
70.91
93.15
Table Top-1 and top-5 accuracy (%) under linear evaluation with different baseline invari- ant self-supervised learning (SSL) methods. All methods use 2-view augmentation policy, and ImageNet- Ik scores are averaged over runs.
ImageNet-Ik
ImageNet-Sketch
ImageNet-V2
ImageNet-R
Algorithm
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
68.44
88.02
MoCo-v3
17.65
31.87
56.54
78.68
18.59
30.08
#0.07
#0.04
69.28
88.79
SER
17.68
32.54
56.95
79.29
18.95
30.72
20.01
#0.02
67.37
87.55
DINO
17.13
32.09
55.00
77.38
18.28
30.38
#0.02
#0.01
67.63
87.56
SER
18.07
34.03
55.19
77.84
18.96
31.55
20.01
+0.01
63.34
84.3
Barlow Twins
10.90
21.17
47.69
70.73
12.30
20.94
40.03
#0.04
64.02
84.73
SER
12,39
24.39
50.89
74.20
13.90
23.99
#0.03
#0.01
For all studies, pretrain with AdamW using batch size 2048 for 100 epochs, with 10-epoch warmup and cosine learning-rate decay. We use a scaling range of [0.7, 1.3) in the geometric pipeline. For linear evaluation, we concatenate [CLS] tokens from the last 4 layers as input to linear classi- fier (Devlin et al.) 22199] Caron et al. 2021) and train for 50 epochs with cosine decay (no warmup), following common VIT SSL practice. (Where reported, we use multi-seed evaluation and provide mean + std.)
4.2
MAIN RESULTS
Linear evaluation on ImageNet-1k. We evaluate representation quality via linear probing on ImageNet- 1k. As shown in Table SER improves over the invariant baselines and compares favor- ably to prior equivariant add-ons. Because equivariant baselines use different numbers of views, we emphasize matched- view comparisons. In the strictly matched 2-view regime, several equivariant add-ons reduce ImageNet- 1k accuracy relative MoCo-v3, whereas SER improves MoCo-v3 by +0.84 Top- 1. We additionally report SER in a matched 2ie-view setting for fair comparison with methods that use local crops.
8

---

Published as a conference paper at ICLR 2026
Top-1 Accuracy
Top 5 Accuracy
Top Accuracy
Top Accuracy
69.5
69.0-
70
99.0
Aocurttyy N00
68.5
Aocurnyy NO
63.5
64.0
Insertion
Figure 3: Ablation study on the location to regularize towards equivariance (left) and to insert the [CLS] token the ViT encoder with fixed equivariance regularization layer the 3rd layer (right). Both Top-1 (left) and Top-5 (right) accuracies peak when the equivariance loss and [CLS] introduced near the middle of the network.
Table 3: Nonlinear evaluations using ImageNet-lk with different equivariant representation learning methods. Note that all equivariant representation learning methods use MoCo their baseline. View refers to the number of crops sampled per image.
View
Algorithm
MLP
20-NN
Fine-tune Top-1
Top-1
Top-5
Top-1
Top-5
MoCo-v3
67.84
88.37
61.56
73.83
91.91
+ AugSelf
63.24
85.86
60.63
73.50
91.67
view
+ STL
65.74
86.32
57.34
73.90
91.49
SER
68.04
88.37
61.64
74.33
91.79
3 view
+ EquiMod
68.33
88,50
58.28
74.08
91.75
+ E-SSL
68.45
88.31
64.56
75.00
92.36
view
+ SER
70.99
89.72
65.32
75.02
92.12
Robustness and spatial transfer. On MoCo-v3 (ViT-S/16, matched 2-view), SER improves erage Top-1 on ImageNet-C/PP by +1.11/+1.22 and frozen-backbone COCO detection by +1.7 mAP (Appendix Sections A.3 and A.5
Tables to 13).
Generalizability across invariant SSL methods. Although MoCo-v3 is our main backbone, we also evaluate SER on DINO (Caron et 2021) and Barlow Twins (Zbontar et al. 2021). As shown in Table SER consistently improves all baselines without architecture changes.
Nonlinear evaluation and fine-tuning. Following (Garrido et al. 2023). we evaluate learned rep- resentations under nonlinear probing (3-layer MLP), k-NN evaluation, and full fine-tuning, Results are reported in Table
4.3 ABLATION AND ANALYSIS
key claim of SER is that equivariance should be enforced on an intermediate spatial representa- tion, whereas many prior approaches impose both invariance and equivariance on the final repre- sentation (Lee et al. 202 Dangovski al. 202T Devillers & Lefort 2023 Yu et 2024). We also insert the [CLS] token after the equivariance-regulee regularized layer to preserve spatial structure (Figure 2). Figure shows an intermediate "sweet spot" for both the equivariance- layer and the [CLS] insertion location.
We further examine the relationship between equivariance strength and discriminative quality when moving the equivariance regularizer closer to the output. As shown in Table [ deeper equivari- ance regularization increases equivariance scores measured at the final layer. but steadily reduces ImageNet- 1k linear-eval accuracy.
Control: = 0 (isolating the equivariance loss). To isolate the effect of Lequiv from batch partitioning and augmentation changes, we train control model with identical settings but set X

---

Published as a conference paper at ICLR 2026
Table Layer-wise ablation for the effect of equivariance regularization. Moving Lequiv toward deeper layers increases the equivariance score of the final representation but reduces ImageNet- Ik linear-eval accuracy, indicating trade-off when equivariance enforced closer to the final repre- sentation. Higher f indicates greater equivariance.
Algorithm
Equivariance Loss Layer
ImageNet Rotation f
Hilip
Scale
MoCo-v3
68.44
0.804
0.870
0.945
MoCo + SER
Layer 3
69.28
0.840
0.881
0.945
MoCo + SER
Layer
68.72
0.888
0.886
0.957
MoCo SER
Layer 12
68. 18.18
0.924
0.892
0.957
Table Control experiment isolating the effect the equivariance regularizer (3 sceds).
Method
ImageNet-Ik Top-1 (%)
MoCo-v3 (baselinc)
68.44 = 0.07
+SER, À (control)
68.82 + 0.01
+ SER, > (full)
69.28 + 0.01
Table 6: Effect of applying laycr-decoupling recipe to prior invariant--cquivariatt baselines using
MoCo-v3 by moving equivariance objective from the final to an intermediate layer (3-seeds average).
Method
Equiv. loss layer
ImageNet- Ik Top-1 (%)
EquiMod (Devillers & Lefort 2023)
12 (final)
68.95 + 0.02
EquiMod (Devillers Lefort
2023
3 (intermediate)
69.51 = 0.02
AugSelf (Lee el al.
2021
12 (final)
67.55 1 0.05
AugSelf Lee el al.
2021
(intermediate)
68.23 0.06
0. As shown in Table 5 batch/augmentationn changes account for modest gain, while enabling Loquiv yields an additional, statistically stable improvement.
Layer decoupling upgrades prior invariant-equivariant methods. To test whether aawarrinne--t-----memmeediee
is a general design principle, re-implement EquiMod and AugSelf with their equivariance losses moved from the final layer to the same intermediate layer used by SER, keeping the rest of each method unchanged. Table [ shows that both methods improve, suggesting that layer decoupling can strengthen existing invariant-equivariann approaches.
Additional ablations. We report sensitivity to the partition ratio r and loss weight X, group- element removal, and applying Cequiv all layers in the appendix (Sec.
5 CONCLUSION
We introduced Soft Equivariance Regularization (SER), simple and scalable way to incorporate equivariance into invariant self-supervised learning. SER decouples invariance and equivariance across layers: the final representation is trained with an unchanged baseline SSL objective, while an intermediate spatial token map is softly regularized toward equivariance via analytically speci- fied group actions.
Empirically, we show that coupling invariance and equivariance on the same final (often spatially collapsed) representation can be suboptimal, and that intermediate-layer regularization yields a bet- ter trade-off. SER avoids auxiliary transformation//acionn modules and does not require learning or predicting per-sample transformation labels/codes. Across multiple invariant SSL baselines for ViTs pretrained on ImageNet- 1k, SER consistently improves downstream accuracy and robustness, with minimal additional compute. Finally, results suggest that layer decoupling is useful de- sign principle for combining invariance and equivariance, and can also strengthen existing invari- ant+equivariant methods when applied their equivariance objectives.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was partly supported by Institute of Information & communications Technology Plan- ning & Evaluation (ITPP) grant funded by the Korea governmett (MSIT) No.RR-2019--119007 Artificial Intelligence Graduate School Program (KAIST)) and Institute of Information commu- nications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2024-005502279, Global AI Frontier Lab).
ETHICS STATEMENT
All authors acknowledge and commit complying with the ICLR Code of Ethics. Our work pro- poses Soft Equivariance Regularization (SER), methodological contribution to self-supervised visual representation learning, and does not involve human-subject experimentation, user stud- ies, the collection of new personal data. All experiments use only publicly available datasets (ImageNet-Ik, MS COCO, 3DIEBench, ImageNet-R, ImageNet-Skettee C, and ImageNet-P) under their respective licenses/terms. We recognize that widely used vision benchmarks can contain societal and demographic biases and may include images of individuals; consequently, models trained with SER may inherit amplify such biases when deployed down- stream. We therefore encourage practitioners to perform domain-appropriate bias/fairness, privacy, and misuse-risk evaluations (e.g., surveillance-related applications) prior to deployment, and fol- low applicable legal and licensing constraints. To support research integrity and transparency, we report matched-view comparisons, document limitations, and provide code for independent verifi- cation.
ImageNet-V V2, ImageNet-
REPRODUCIBILITY STATEMENT
To support reproducibility, all experimental results reported in this paper are produced from the public implementation at pattps://giihii fully specified the main paper (including the training pipeline, augmentatiion/batc partitioning, analytic feature-space group actions, and the objective; Section and Algorithm []. while the experimental protocols and key hyperparameters are described in Section 4.1 with additional ab- lations and benchmark-specne and sensitivity to À and batch partition ratio) provided in the Section A Multi-seed performance numbers are computed by averaging results over random seeds (and we report variability where applicable). All evaluations use standard public benchmarks with established splits and metrics, enabling results be reproduced directly by running the released training/evaluation scripts and configurations.
com
/SER, The SER method
details (e.g., matched-view settings, robustness/transfer evaluations,
REFERENCES
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianee-invariance-oovaa regulariza- tion for self-supervised learning. In The Tenth International Conference Learning Represen- tations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: //openzeview. wwwtteeuummmee www.dm22
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 134788 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
11

---

Published as a conference paper at ICLR 2026
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen,
Saining Xie, and Kaiming He. empirical study of training self-supervieed vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp- 9640-9649, 2021.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria-Florina Bal- can and Kilian Q Weinberger (eds.). Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2990-2999, JMLR.org, 2016. URL URLhttp: //proceedings mlx. wwl..press/v49. www.///cchecceece html
Taco S. Cohen and Max Welling. Steerable cnns. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htthsttss:/// /owp://ieee...eeeeeeeeeo hwtp:ennntoooooeeeeeee www.ooommmddddd-ee-eeee
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić. Equivariant contrastive learning. arXiv preprint arXiv:2 11.0299, 2021.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve visual instance discrimination. In International Conference on Learning Representations (ICLR), 2023.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
M. Finzi, G. Benton, and A. G. Wilson. Residual pathway priors for soft equivariance constraints. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Q. Garrido, L. Najman, and Y. LeCun. supe-supe learning of split invariant equivariant repre- sentations. arXiv preprint arXhv:23022110883 2023.
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun, Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.00504, 2024.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRe- view.net, 2018. URL htttps:://ppeeeeeeeee
net. forrm?frum?? 1oo.ummmdd111422110
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach self-supervised learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020, URL https:tttss.. http:::::s///ppppeeeeeccee www.inuurr.omm cc/ Abst .html
Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structur- ing representation geometry with rotationally equivariant contrastive learning. arXiv preprint arXiv:2306. 13924, 2023.
12

---

Published as a conference paper at ICLR 2026
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seartle, WA. USA, June 13-19, 2020, pp. 9726- 9735. Computer Vision Foundation / IEEE, 2020. doi: N0.10999V244220002000000000
URL.
htt.p:s/700..orrr....111///722 wwT.TT0000//00O0000O 2020. 00975
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint 1IXhv::903.12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distributtin generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. $340-8349, 2021.
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neu- ral networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' A'Alhhé-Biie Buch Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7090-7099, 2019. URL ea 22526001155a11222116631122222222
neuripss CC/ cc/papprr2219/haas. html
H. Kim, H. Lee, H. Yang, and J. Lee. Regularizing towards soft equivariance under mixed symme- tries. In International Conference on Machine Learning (ICML), 2023.
Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-awarc self-supervision. In Conference Neural Information Processing Systems (NeurIPS), 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Giovanni Luca Marchetti, Gustaf Tegnér, Anastasiia Varava, and Danica Kragic. Equivariant rep- resentation learning via class-pose decomposition. In Francisco J. R. Ruiz, Jennifer G. Dy, and van de Meent (eds.), International Conference on Artificial Intelligence and Statis- tics, 25-27 April 2023, Palau de Congressos, Valencia, Spain, volume 206 of Proceedings of Machine Learning Research, pp. 4745 4756. PMLR, 2023. URL httptt::s/pppeeeeeeeeeeeeee html
m wttess//206mmmreeee
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jig- saw puzzles. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vi- sion ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part VI. volume 9910 of Lecture Notes in Computer Science, pp. 69-84. Springer, 2016. doi: 00079978-3-31-446666444 978-3-311 46466-4_5
URL attps: /
Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. Advances in neural information processing systems, 33:4489-4500 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint axXh::2004.07193, 2023.
Y. Park, O. Biza, L. Zhao, J. W. van de Meent, and Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204.112 11371, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classi- fiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400 PMLR, 2019.
13

---

Published as a conference paper at ICLR 2026
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae V2: scaling video masked autoencoders with dual masking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 14549-14560. IEEE, 2023. doi: 10.1 10.110/CYYR3777:22300338 //doi.org// 1tt.111...1.99990 1109/CVPR52729 2023. 01398
URL https:
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference machine learning, pp- 9929-99399 PMLR, 2020.
Jaemyung Yu, Jachyun Choi, DongJae Lee, HyeongGwon Hong, and Junmo Kim. Self-supervised transformation learning for equivariant representations. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self Self-super learning via redundancy reduction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 13100-1320. PMLR, 2021. URL zbontar2la, bttt::/ppee http:://proceecingg .mlr. p html
Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pp. 7324-7334, PMLR, 2019.
14

---

Published as a conference paper at ICLR 2026
Algorithm Soft Equivariance Regularization (SER) for invariant self-supervised learning
Input: batch B. partition ratio r. baseline augmentation policy T. equivariant-view policy Teg (crop disabled; photometric jitter retained), geometric group Ç (scale, HFlip, Rot90%), encoder
f f o ( base SSL loss Lsss.. (.), weight
2: Partition B into b and b2 where |b2 r B and |b,| (1 ([1-r)1 r)
Sample paired views for the baseline SSL loss
for each
do
4: Sample two views (I), ~ T
baseline 2-view pipeline
5: end for
6: for each E b2 do
7: Sample two views x2) ~ Teq and record geometric parts 9 E jitter in Teq is sampled independently across the two views
B photometric
8: end for
// Invariance losses (on the final embedding, unchanged)
9: Linvl &- L.sss.bb T)
10: Linv2 Lasss(br (b2; Teq)
/1 Equivariance regularizer on intermediate spatial features (reusing the same by views)
11: Compute intermediate token maps {hi (0)
f(1((=10))) and { + f())(=22))) for all
(1)
(1()
E Eb
12: for each with (1( E b2 do
13: Relative geometric transform: g( + 92 )
Align features: (1))) Pgtt: (1))
15:
end for
Loquiv + PatchhNTent ((h) Jiens: (12 Fiet))
D negatives from other images/tokens
I/ Total objective
17: C+ + Livv + X Lequiv
18:
Update encoder parameters by minimizing L 19: Output: pre-trained encoder f
A FURTHER DISCUSSIONS AND EXPERIMENTS
A.1 NUMBER OF VIEWS AND MATCHED- VIEW COMPARISONS
It is well established that increasing the number of global/local views (augmentations) can improve SSL representation quality, at the cost of additional compute and memory (Caron al. 2020 2021). Therefore, comparing methods that use different numbers of views can be confounded by view count. In particular, E-SSL uses a 2+4-view strategy (2 global and 4 local views), while EquiMod uses global views.
To ensure fair comparisons, we also implement SER under matched 2ie-view setting. Following the local-to-global design in DINO (Caron et al., 202T), we do not pass all four local views through the MoCo momentum encoder, which avoids computing losses among local views. For the equiv- ariance regularizer, we form one global pair and two local pairs, and compute the equivariance loss within each pair. Results of this 2+4-view variant are reported separately under the "2+4-view" rows in the tables.
A.2 ADDITIONAL ABLATIONS AND ANALYSIS
Equivariance-loss layer and [CLS] insertion. In Section we evaluate the common prac- tice imposing equivariance and invariance objectives at the encoder's final representation. Our results show that applying the equivariance regularizer too early or too late is suboptimal, while the best downstream performance achieved by regularizing an intermediate layer (in our main VIT- S/16 setting, the third transformer block), where spatial structure is preserved. We also find that the [CLS] insertion point matters: inserting [CLS] too early can hinder equivariance regularization on spatial tokens, while inserting it too late can degrade invariance learning (Figure
15

---

Published as a conference paper at ICLR 2026
Table Transformation prediction. Evaluation transformation label prediction from the learned representation of different layers (see Section A for more detail)
Tasks
Methods
Layer
Layer
Layer
Layer 9
Layer 12
MoCo + AugSelf
81.55
96.71
99.71
99.13
68.74
Rotation Prediction (%)
MoCo + STI.
79.56
92.8
99.46
99.76
98.08
MoCo + SER
79.97
93.34
99.52
99.73
96.59
MoCo + AugSelf
63,24
85,66
96.48
91.51
62,68
HFlip Prediction (%)
MoCo + STL
62,91
82,09
88.83
97.22
83,27
MoCo + SER
63.02
78.68
92.87
96.59
75.95
Table Sensitivity to the batch partition ratio and equivariance loss weight (ImageNet- k linear eval, 3 seeds).
(MoCo-v3)
0.005
0.01
0.02
Top-1 (%)
68.44 0.07
69.08 + 0.03
69.28
0.01
68.30 = 0.05
(control)
0.5
1.0
5.0
10.0
Top-1 (%)
68.82
0.01
69.28
0.01 68.92
0.02 68.73
0.01 68.67 = 0.03
Equivariance score. In Table we quantify how equivariance measured the final representation changes when shifting the equivariance-loss layer. Following (Zhang) 2019), we sample geometric transforms from rotation 90%, horizontal flip, and scaling, and measure an equivariance score using cosine similarity:
Equivariance Ex, (o) .2)~ cos (P)())))), M()(=)))]
9-91
where f()) denotes the final representation (thus evaluate equivariance at the last layer). Higher values indicate stronger equivariance.
Transformation prediction probes. In Table 7 evaluate whether frozen features support pre- dicting rotation/flip labels, following (Garrido et al. 2023). This is a classification accuracy (not an R² regression score). Although SER regularizes equivariance at intermediate layer, deeper layers (including the final layer) retain strong transformation information.
Sensitivity to r and X. We evaluate sensitivity the batch partition ratio and the equivariance loss weight A (ImageNet-Il Ik linear eval, 3 seeds). SER remains stable across reasonable range of values and does not rely on finely tuned hyperparameters.
Ablating geometric group elements. To assess sensitivity to the choice of invertible geometric transformations, we ablate elements of the geometric set used by SER. Performance remains com- petitive and above the MoCo-v3 baseline as long as reasonable geometric group is available.
Applying Lequiv on all layers. We compare applying the equivariance regularizer single in- termediate layer versus all transformer blocks. Regularizing all layers provides gains, but is slightly worse than the best single-layer choice and increases compute.
A.3
ОBJECT DETECTION COCO
Equivariance is expected to be particularly helpful for tasks that require spatial sensitivity beyond image-level classification. To further examine transfer, we evaluate frozen-encoder object detection on Lin et al. (2014). As shown in Table I1, SER achieves the strongest detection performance among the compared pretrained encoders under the same detection setup. Following Oquab et al. (023), freeze encoder weights and train the detection head for 45k iterations with batch size 32 on COC02017 train, reporting performance on COC02017 val. All methods share the same detection pipeline; only the pretrained encoder differs.
16

---

Published as a conference paper at ICLR 2026
Table 9: Ablating geometric group elements used in SER (3 seeds).
Variant
ImageNet- Ik Top-1 (%)
SER (full)
69.28 + 0.01
SER Rot90
69.17 + 0.01
SER Rot90
HFlip
69.12 = 0.12
SER Rot90 Scale
68.67 = 0.01
Table 10: Applying Lequiy to all transformer blocks vs. single intermediate layer (3 seeds).
Variant
Top-1 (%)
Top-5 (%)
MoCo-v3
68.44 + 0.07
88.02 + 0.04
+SER (layer
69.28 + 0.01
88.79 = 0.02
+SER (all layers)
69.21 1 0.05
88.78 0.12
Table 11: COCO object-detection results with frozen backbone (higher better).
Metric
MoCo
MoCo SER
MoCo STL
MoCo AugSelf
mAP
0.225
0.242
0.221
0.197
mAP@550
0.404
0.428
0.400
0.359
m5
0.222
0.244
0.218
0.192
Table 12: Top-1 accuracy comparison ImageNet-C, including 15 types of common corruptions, for our method and other equivariant representation learning methods built upon the invariant repre- sentation learning baseline MoCo (He al. 2020).
Noise
Blur
Weather
Digital
Algorithm
Avg.
Gauss.
Shot
Impal.
Deño.
Glass
Motion
Zoom
Snow
Frost
Fog
Bright.
Cont.
Elas.
Pixel
JPEG
MoCo-v3
39.18
37.81
36.09
33.51
13.85
31.49
25.
30.61
30.03
35.0
62.81
$2.00
54.63
55,78
53,37
39.47
+ AugSelf
34,91
31.81
31.44
35.06
17.12
34.28
27.67
31.99
28.50
35.01
61.99
50.64
55.48
54.88
53.17
38.93
STL
17.78
16.26
14.65
29.51
15.13
27.33
25.77
29.50
27.60
34.40
61.81
48.84
54.31
46.40
50.63
33.33
+ SER
39.42
38.30
36.85
36.23
15.91
34.90
27.35
30.71
30.12
36.04
63.60
52.81
55.85
56.63
54.03
40.58
EquiMod!
34.33
32.59
31.76
25.73
31.38
52.40
37.45
SSSS
43.80
42.59
40.80
38.44
16.40
36.73
28.20
34.22
32.24
37.93
65.50
55.89
56.12
55.80
55.21
42.66
SER:
39.88
39.27
37.18
36.21
19.58
34.16
31.90
36.07
35.87
40.74
66.76
54.90
57.87
56.42
56.79
42.91
A.4 WHY SER DOES NOT COLLAPSE TO A TRIVIAL SOLUTION
We briefly explain why SER does not collapse to a trivial intermediate representation. Minimizing Lequiv corresponds to minimizing (f()) (x)), f (P)(p))))
First, trivially invariant intermedi- ate representation does not generally yield zero loss: if (p)()) (x)) then the loss becomes d(p, (f)) (z)), which is nonzero unless (x))(x) is itself invariant under Pa (c.g.. spatially constant map). Second, our contrastive Lequiv discourages collapse by promoting separation from negatives sampled from other images and spatial locations, which encourages diversity rather than spatial constancy (cf. (Wang & Isola] 20200). Third, joint optimization with the baseline invariance objective Linv (e.g., MoCo) further promotes non-trivial, instance-discrmmiattie features.
A.5 ADDITIONAL. ROBUSTNESS RESULTS ON IMAGENET-C AND IMAGENET-P
This section reports the ImageNet-C and ImageNet-P results referenced in the abstract and the main text (Sec. 4.. For MoCo-v3 (ViT-S/16, matched 2- 2-view), SER improves average Top- 1 on ImageNet- C from 39.47 to 40.58 (+1.11, Table (2) and on ImageNet-P from 63.91 to 65.13 (+1.22, Table 13).
We additionally report ImageNet-P results when applying SER to other invariant SSL backbones in
Table 14
17

---

Published as a conference paper at ICLR 2026
Top-1 accuracy comparison on ImageNet-P, including 14 perturbation types, for our method and other equivariant representation learning methods built upon the invariant represen- tation learning baseline MoCo (He et al. 2020).
Table 13:
Noise
Blur
Weather
Digital
Algorithm
Avg.
Gau. N.
Shot
Spock.
Motion
Zoom
Gau. B.
Snow
Spatter
Bright.
Trans.
Rot.
Tilt
Scale
Shear
MoCo-v3
67.85
67.97
57.31
68,30
56.57
66.57
63.43
68.00
64.55
67.58
45.18
65.32
63.91
+ AugSelr
67.44
67.47
67.47
57.51
67.76
67.81
56.77
66.15
60.77
67.39
64.41
67.10
47.15
64.96
63.58
SSL SER
66.15
54.97
66.29
66.32
64.71
61.16
66.00
62.07
65.86
42.53
63.17
61.85
68.97
69.01
69.00
59.09
69.46
69.35
58,02
67.86
64.58
69.04
65.76
68.60
46.78
66.34
65.13
+ EquiMod"
68.60
68.70
68.78
57.08
69.17
69.13
56.97
67.51
60.88
68.75
65.18
68.27
47.04
66.
64.44
+ B--SSL'
70.26
70.19
70.22
61.49
70.65
70.54
60.60
68.87
65.82
70.27
66.92
69.82
48.86
67.59
66.58
SER*
71.68
71,65
71.67
60.36
71.87
71.87
61.92
70.47
67.18
71.62
68.62
71.34
52.74
68.99
68.00
Table 14: Experiments with various SSI. algorithms. Top-1 accuracy (%) on ImageNet-P. All mod- els are trained with the setting addressed in Section 4.1 See Table 13 for the results from MoCo.
Noise
Blur
Weather
Digital/ / Geometric
Algorithm
Avg.
G.Nsc
Shot
Spk]
Mot.
Zoom
G.Blr
Snow
Spat
Brt.
Tran
Rot
Tilt
Scal
Shear
DINO
66.69
66.67
66.70 52.74
67.02
66 94
55,83
65.60
61.27
66.70
62.73
66.49
42.96
63.51
62.27
SER
67.39
67.31
67.41
54.41
67.69
67.66
56.19
66.06
62.13
67.22
63.33
67.02
42.76
64.31
62.92
Barlow Twins SER
60.09
60.08
60.12
42.25
60.53
60.53
46,37
58.84
52.92
60.22
55,47
59.37
33.15
56.79
54.77
63,85
63.93
63.91
49.18
64.29
64.19
50.09
62.34
57.93
63.89
59.63
63.38
37.29
60.56
58.89
Table 15: Top-5 accuracy comparison on ImageNet-C, including 15 types of common corruptions, for our method and other equivariant representation learning methods built upon the invariant repre- sentation learning baseline MoCo (He et al. 2020).
Noise
Blur
Weather
Digital
Algorithm
Avg.
Gauss.
Shot
Impul.
Deli.
Glass
Motion
Zoum
Snow
Frost
Fog
Bright.
Coni.
Elas.
Pixel
JPEG
MoCo-v3
63.30
61.68
59.76
56.22
28.46
53.52
45.98
52.52
50:54
58 64
84.47
76.74
77.38
79.44
77.76
61.76
AugSelr
58.96
55.21
54.61
59.06
34.52
57.55
48.71
55.05
49.4
59.30
84.39
76.26
78.25
79.15
78.02
61.90
+ STL
37.10
34.36
32.04
52.71
31.08
48.45
46.35
50.96
47.60
59.05
84.07
74.74
76.96
71.37
75.66
54.83
SER
64.16
62,70
61.21
59.92
32,28
58.17
48.22
53.23
51.26
60.49
85.68
77.94
78,55
80.52
78.58
63.53
EquiMod!
58.63
56.28
55.34
55.28
31.48
54.45
48.20
51.11
45.88
55.49
84.63
73.66
78.51
78.46
77.62
60033
+ E-SSL:
68.70
67.42
65.64
63,00
33.09
60.35
49,83
57.64
53.72
62.83
86.80
80.33
79.05
80.32
80.05
65.92
SER"
64.00
62,89
37.02
56.65
58811
57.43
65.06
87.36
79.14
79.82
80.42
65.48
Table 16:
Top-5 accuracy comparison on ImageNet-P, including 14 perturbation types, for our method and other equivariant representation learning methods built upon the invariant represen- tation learning baseline MoCo (He et al.) 2020).
Noise
Blur
Weather
Digital
Algorithm
Avg.
Gau, N.
Shot
Speck.
Mation
Zoom
Gan.
Snow
Spatter
Bright.
Trans.
Rot.
Tilt
Scale
Shear
v3
87.75
87.81
87,82
80.34
87.91
87.94
79.22
86.75
84,54
87.72
85.16
88.48
69.08
85.84
84.67
AugSeir
87.59
80.81
87.73
86.50
83.07
87.66
85.25
87.31
71.44
85.83
84.69
+ STL
86.67
86.65
86.66
78.33
86.82
86.77
77.62
85.59
83.21
86.50
83.81
86.50
66.31
84.75
83.30
SER
88.60
82.01
88.66
88.68
80.56
87.58
85.59
88.52
86.01
88.17
71.25
86.78
85.69
EquiMMa!
88.78
88.79
88.75
80.86
88.93
88.98
80.16
87.95
83.41
88.68
86.38
88.51
71.89
87.13
85.66
E-SSL:
89.60
89.58
89.64
83.94
89.78
89.72
82.95
88.
86.88
89.47
87.25
89.31
73.39
87.79
87.01
+ STERA
89.96
89.93
90.00
82,50
90.10
90.08
83.23
89.15
87.30
89.93
87.80
89.68
75.80
88.29
87.41
A.6 LATENT SPACE VISUALIZATION
Beyond quantitative metrics, we provide a qualitative visualization of latent features extracted from MoCo (invariance only) and MoCo+SER. Due to ImageNet's 's large class count (1000), we randomly sample 20 classes for visualization. As shown in Figures 4 and [ the resulting embeddings show clearer qualitative separation for several classes under SER, consistent with improved linear evalua- tion and robustness.
18

---

Published as a conference paper at ICLR 2026
Table 17: Computation overhead. Measured FLOPs includes both forward and backward pass with a 2-view augmentation policy, and "Relative overhead" is the relative FLOPs to vanilla MoCo-v3. FLOPs for SER were measured for the overall mini-batch computation and divided by the mini- batch sample number (including both b1 and b2 as illustrated in Figure
Method Per-image FLOPs Relative overhead
MoCo-v3
18.48G
1.0x
+ SER
18.63G
1.008x
B LIMITATIONS
SER relies on structured geometric transformations (rotations, scaling, flips), limiting applicability modalities tasks where such transformations meaningful. Because scaling is implemented via discrete resamplig/inerpooattonn on finite grid, its invertibility holds only up to discretization effects. Extending the approach domains without clearly defined group actions (e.g., text, audio, graphs) is non-trivial, While SER is designed to be lightweight, it introducss additional computation; our main setting the measured overhead is small (Table 17). but any added cost can matter in resource- limited regimes.
C USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) to provide writing assistance during the preparation of this manuscript. The LLMs were used in the following ways:
Polishing and rephrasing sentences for clarity and readability, including parts of the intro- duction, background, and experiments.
Condensing text to meet page limits.
used for research ideation, experimental design, implementation, or result generation. All conceptual contributions, algorithm development, theoretical analysis, and experimental work were conceived, conducted, and verified entirely by the authors.
Importantly, the LLMs
19

---

Published as a conference paper at ICLR 2026
(a) MoCo-v3
(b) MoCo-v3 Ours
Class Labels
Class Labels
Figure 4: t-SNE visualization of latent space features from 20 randomly sampled ImageNet- Ik
classes, comparing (a) MoCo-v3 (trained with invariance loss alone) and (b) MoCo-v3 + Ours. Our method promotes better class clustering, demonstrating that incorporating equivariance benefits downstream tasks requiring invariance.
(a) MoCo-v3
(b) MoCo-v3 Ours
Class Labels
Class Labels
4675 4883
-513 4626 #761
-513
Figure 5: t-SNE visualization of latent space features from 20 randomly sampled ImageNet-C classes under defocus blur corruption, comparing (a) MoCo-v3 (trained with invariance loss alone) and (b) MoCo-v3 Ours. Our method maintains better class clustering under corrpption, demon- strating robustness benefits of incorporating equivariance.
20


Reviews:
[{"review_id": "cwLlFq892B", "reviewer": "Reviewer_EYXE", "summary": "This paper proposes using soft equivariance regularization (SER), where output features use an invariant loss while intermediate features use equivariant regularization in the context of SSL. For transformers, the output features are spatially collapsed and thus in order to use both invariance and equivariance objectives commonly used in SSL, the authors encourage equivariance only in the intermediate representations, which are then passed to an invariant block that also takes [CLS] token as input. The authors show results on ViT-S on ImageNet-1k with linear and nonlinear evaluations, and compare against various SSL baselines.", "questions": "- The method seems to hinge heavily on $\\lambda$ to balance $L_{inv}$ and $L_{equiv}$ and also the batch partition ratio. Is the method sensitive to these parameters?\n- It's not clear if regularizing intermediate representations do in fact encourage soft equivariance on the output representations. Is there any way to verify this on the output representations?\n- I feel that many SSL methods heavily use non-invertible transformations (e.g. crop, color jitter, random grayscale, mixup/cutmix, etc). Partitioning the batch is reasonable, but does the method rely on using a balance of invertible and noninvertible transformations?\n- How does SER fit in with MAE style pretraining? Would you say SER is an alternative to masking for learning representations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Practical way to use transformers for invariance and equivariance objectives in SSL, considers spatial collapse and distinguishes between invertible and noninvertible augmentations.\n- Experiments are done at ImageNet scale.\n- The evaluation is pretty comprehensive: considers linear and nonlinear evals, robustness (ImageNet-P), and also detection transfer. They consider all the SOTA SSL baselines I'm aware of and also do extensive ablations on the intermediate layer position.\n- Does not require transformation labels.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QPRa0h7Eml", "reviewer": "Reviewer_t2Vy", "summary": "The authors tackle two aspects of equivariant representation learning. First, is scaling to imagenet pretraining, where most works tend to focus on smaller datasets. The second is to not enforce equivariance at a representational level, but partially through the network. The rest of the network is then trained with the final goal being the usual invariance, leading to invariant representations at the end of the network. To enforce equivariance partially through the network, the authors leverage the spatial structure that is kept to apply the same transformation as done on the input.", "questions": "1) Since $f^{(1)}$ is equivariant, it could be interesting to visualize the feature maps to see how the spatial structure is preserved compared to other approaches/a fully invariant baseline.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The proposed approach can be integrated with off the shelf SSL methods and architectures, making it widely usable.\n\n- The goal of learning invariant representations, but still enforcing intermediate representations to be equivariant is an interesting take on equivariant SSL in general.\n\n- The scale of the experiments is appreciated, making it closer to state of the art setups.\n\n- The authors demonstrate performance gains over baseline methods and other equivariant methods, with the caveat that here the representations are ultimately invariant.\n\n- Performance gains are observed over the baseline methods across datasets, with some caveats (see weaknesses)", "weaknesses": "1) Line 94: “Our approach requires no explicit transformation labels”. If my understanding is correct, having access to the exact transformation (generated from the labels) is required, which while a different constraint can be even harder to obtain in practice. Perhaps [1] (cited in the paper) should be discussed more, as the authors use a similar idea by applying the transformation to few dimensions of the representations, as defined by the dataset/transformation.\n\n2) Claims of being the first to scale equivariant SSL to ImageNet (abstract, lines 61-63) are incorrect (and unnecessary) as this was already done in [2], training ViT-B models on ImageNet and studying the impact of equivariance on downstream tasks.\n\n3) Unclear loss definitions. $L_{inv}$ is never properly defined, and the choice of negative samples for both  $L_{inv}$ and $L_{equi}$ is unclear. Lines 295-296 indicate that negative pairs are samples from $z$, but then line 300 indicates that “we omit negative pairs sample from the same image as the anchor”. I assume that the latter is correct, but the notation must be made clearer. For $L_{equi}$, how are the negative pairs chosen ? Across the same image or across the batch ? While appendix B.4 clarifies a bit, it should not take this amount of work to understand the loss, which is the core of the paper.\n\n4) The authors mention that cropping is removed from the equivariant sub-batch as it does not form a group, but other commonly used augmentations do not either. The standard protocol of BYOL[3] (used in MoCov3 that is the base of experiments in the paper) includes grayscale, blurring and solarization, all of which definitely do not have an inverse. This would suggest that experiments are performed with suboptimal data augmentations in the invariant batch, which makes the comparison to the baselines more complex/unclear.\n\n[1] Marchetti, Giovanni Luca, et al. \"Equivariant representation learning via class-pose decomposition.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\n[2] Garrido, Quentin, et al. \"Learning and leveraging world models in visual representation learning.\" arXiv preprint arXiv:2403.00504 (2024).\n\n[3] Grill, Jean-Bastien, et al. \"Bootstrap your own latent-a new approach to self-supervised learning.\" Advances in neural information processing systems 33 (2020): 21271-21284.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yEZNhGcJuc", "reviewer": "Reviewer_XBxz", "summary": "- Introduces SER, which decouples invariance and equivariance by applying an equivariance loss to intermediate, spatially structured features while learning invariance on the final head. \n- Demonstrates ViT-S/16 pretraining on ImageNet-1k with linear/non-linear evals and robustness suites (IN-Sketch, IN-V2, IN-R, IN-C/P) and an OOD 3D dataset (3DIEBench). Shows consistent gains over MoCo-v3 and positive deltas when plugged into DINO and Barlow Twins; also improves frozen-backbone object detection on COCO.", "questions": "See weaknesses above", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The paper is well written and easy to follow\n- SER does not require transformation labels or predictors; it leverages known group actions on feature maps with a simple NT-Xent loss, keeping the method lightweight and scalable.\n- By excluding crop from set of groups $G$ and keeping it in the invariance path, the method respects group theory while preserving the benefits of strong augmentations for representation quality. \n- Improves linear/nonlinear evaluation on ImageNet-1k across MoCo-v3/DINO/Barlow Twins and shows better robustness on ImageNet-C/P and 3DIEBench", "weaknesses": "- The paper applies rotations, flips, and anisotropic scaling at intermediate feature maps but does not detail how resampling/interpolation, padding patch-grid alignment are handled. On discrete token lattices, 90° rotations are exact but scales are not. The chosen interpolation kernel can materially affect equivariance error. A detailed description and sensitivity study are missing. \n- SER avoids predicting labels, yet it relies on knowing $g_2g_1^{-1}$ from the augmentation pipeline and on having a known group action $\\rho_g$ at feature level. This limits applicability when transforms are stochastic/unknown (e.g., random photometric transforms, camera jitter).\n- ImageNet Top-1 gains over MoCo-v3 are quite modest, especially in 2-view setting. No confidence intervals or multi-seed statistics are provided, so practical significance remains uncertain.\n- By construction, SER cannot handle non-group transforms (e.g., crop, typical color jitter) except via the invariance path, so transformation-aware tasks for photometric changes or crops remain unaddressed.\n- The $b_2$ path introduces 90° rotations and removes crop. It is possible that the gains partially stem from the augmentation distribution shift rather than the equivariance loss. One should include the same set of augmentations for MoCo-v3 but without \n the equivariant loss i.e. equivalently run SER with $\\lambda=0$. Its important to disentangle the effect of augmentations and the equivariant loss. \n- There are related papers that put the equivariant objective “earlier” (pre-projector / on spatial features) and keep the invariant objective “later” (post-projector) [1, 2] . The contributions of this work don't seem to be significantly novel compared to existing literature. \n\n[1] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" arXiv preprint arXiv:2302.10283 (2023).\n\n[2] Devillers, Alexandre, and Mathieu Lefort. \"Equimod: An equivariance module to improve self-supervised learning.\" arXiv preprint arXiv:2211.01244 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1iVY026s6d", "reviewer": "Reviewer_BZu2", "summary": "The authors present a simple yet effective method for introducing an equivariant regularisation constraint into self-supervised ViTs. The goal is to better decouple the traditional approach of simultaneous invariant and equivariant learning on the output embedding space by enforcing intermetiate layers to be equivairant to the input transformations by preserving the spatial structure, while the output vector remains invariant. The approach demonstrates strong invariant performance on downstream linear evaluation tasks, and scales to imagenet level datasets. However, its primary weakness is the lack of a direct evaluation to confirm that the intermediate layers actually learn the desired equivariant properties, and its motivation for avoiding transformation labels is partially contradicted by its own methodology.", "questions": "1. In practice how do you determine which layers to apply the soft-equivariant objective to? For some transformation groups the layers which have the most significant effect may change, therefore in practice this is a heuristic / empirical pre-processing step to identify? How does the layer performance change under different architectures and datasets?\n2. You do not want to operate the equivariant objective on the pooled feature vector of the output. Hence the question is, why not just omit the final pooling/mean layer? Also prior works such as Konstantinou, et al., (2025) EquiCaps: Predictor-Free Pose-Aware Pre-Trained Capsule Networks. perform on the feature embedding prior to pooling hence maintaining the spatial component. \n3. How does the model perform if you apply the equivariant regularisation on all layers of the equivariant network? Why perform on just one when you can enforce all features?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The analysis and subsequent conclusion that enforcing equivariant at the output embeddings is a significant and interesting contribution to the equivariant self-supervised literature. This provides new insights and research directions that the community will find valuable. \n- The method presented is simple yet effective at improving downstream invariant performance. The approach is generalisable, and enforces the desired properties at defined layers within the network. The enforcement of equivariant feature maps at intermediate layers is seemingly correct. \n- The empirical results demonstrate the effectiveness of the method and its ability to scale, where prior works have not yet managed to achieve competitive performance on large-scale datasets. \n- Limitations are presented and discussed\n- Code and implementation details are provided for validation and reproduction.", "weaknesses": "**Major:**\n- It is mentioned in the introduction one of the drawbacks is that knowing the transformation labels is an inherent drawback, however, the proposed method also requires the transformation group to be known. While not necessarily a weakness in my opinion, this contradicts the authors motivation.\n- No evaluation of the equivariant property is performed. For example, how well do the intermediate representations which the regularisation has been applied to and the invariant representations encode the translation information. It cannot be claimed that equivariance has been enforced without some analysis of the inversion property at a minimum. \n\n**Minor:**\n- I would argue that the regularisation is not necessarily soft. The authors are specifically enforcing features to preserve the transformation in the latent space of the intermediate layers via a direct distance metric. This seems like a specific hard constraint.\n- The additional computational overhead is presented as a limitation but not quantified.\n- It is not clear how the 3DIEBench 3D tranformations are encoded into the equivariance regularisation. Are the these object transformations encoded in the g parameter or are the standard crops and rotations applied?\n- See questions for additional minor weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)