Soft Task-Aware Routing of Experts for Equivariant Representation Learning
Authors: Jaebyeong Jeon
Hyeonseo Jang
Jy-yong Sohn
Kibok Lee
Venue: NeurIPS 2025 poster ()
Keywords: equivariant representation learning
self-supervised learning
contrastive learning
Primary area: deep_learning

Abstract: Equivariant representation learning aims to capture variations induced by input transformations in the representation space, whereas invariant representation learning encodes semantic information by disregarding such transformations. Recent studies have shown that jointly learning both types of representations is often beneficial for downstream tasks, typically by employing separate projection heads. However, this design overlooks information shared between invariant and equivariant learning, which leads to redundant feature learning and inefficient use of model capacity. To address this, we introduce \textbf{S}oft \textbf{T}ask-\textbf{A}ware \textbf{R}outing (STAR), a routing strategy for projection heads that models them as experts. STAR induces the experts to specialize in capturing either shared or task-specific information, thereby reducing redundant feature learning. We validate this effect by observing lower canonical correlations between invariant and equivariant embeddings. Experimental results show consistent improvements across diverse transfer learning tasks. The code is available at \url{https://github.com/YonseiML/star}.

Soft Task-Aware Routing of Experts for Equivariant Representation Learning
Hyeonseo Jang Department of Statistics and Data Science, Yonsei University (jasbyeong98, jhyeonseo715, jysohn1108, kibok] (aokki@ynneeiccoom ac.
Jaebyeong Jeon
Jy-yong Sohn Kibok Lee
Abstract
Equivariant representation learning aims capture variations induced by input transformations in the representation space, whereas invariant representation learn- ing encodes semantic information by disregarding such transformations. Recent studies have shown that jointly learning both types of representations is often beneficial for downstream tasks, typically by employing separate projection heads. However, this design overlooks information shared between invariant and equiv- ariant learning, which leads to redundant feature learning and inefficient use model capacity. To address this, we introduce Soft Task-Awart ware Routing (STAR), routing strategy for projection heads that models them as experts. STAR induces the experts to specialize in capturing either shared or task-specific information, thereby reducing redundant feature learning. We validate this effect by observing lower canonical correlations between invariant and equivariant embeddings. Ex- perimental results show consistent improvements across diverse transfer learning tasks. The code is available at attttps: /tttts://githube com com/YonseiML/star
Introduction
Self-supervised learning (SSL) has emerged as promi- nent paradigm for learning representations from large- scale unlabeled data 20 [7]. Among various ap- proaches, invariant representation learning methods gen- erate multiple views the same instance through trans- formations such data augmentations, and encourage these views map to the same representation, thereby preserving semantic content regardless of the transforma- tions applied. However, enforcing strict invariance can discard nugmentation-awaree information, such color and spatial location, which may degrade performance in downstream tasks that depend such information representation learning has emerged as a complementary paradigm that captures structured variations in the representation space corresponding to transformations applied in the input space [4 [8 51).
Figure 1: Crater Illusion. A lunar im- age that appear as a dome (left) or crater (right) depending on orientation [46).
(27]
To address this limitation, equivariant
By preserving augmentation-avye aware information, equivariant representation learning yields richer representations, leading to improved performance downstream tasks.
Recent approaches such as EquiMod [9] employ a shared encoder followed by two projection heads, which independently produce invariant and equivariant embeddings from shared representation. This design implicitly assumes that the invariant and equivariant learning tasks are independent, encouraging each projection head to specialize in its respective objective. However, this assumption might not be true in practice; rather, these two tasks are inherently interdependent. This interdepen- dence can be intuitively illustrated by the well-known crater illusion, where the same lunar surface may appear either as crater or a dome depending on the light conditions, as shown in Figure [] For instance, recognizing whether the object is crater or dome-a semantic category typically
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

captured through invariant learning helps infer which side is illuminated property expected to be captured through equivariant learning. Conversely, understanding the orientation of the scene helps recognize the semantic category. This bidirectional dependency highlights the interdependent nature of invariant and equivariant learning I Consequently, when two independent projection heads are employed for invariant and equivariant learning, they tend to redundantly capture shared information across branches- phenomenon we refer to as redundant feature learning.
To mitigate redundant feature learning, we introduce Soft Task-Aware Routing (STAR)-a strategy that explicitly coordinates shared and task-specific information. We instantiate STAR two forms: (i) by adding shared projection head that provides common embedding bridging the two learning objectives and capturing information beneficial to both, and (ii) by adapting the Multi-gate Mixture- of-Experts (MMoE) [31 as the projection module, which dynamically allocates experts based on the input for each task. These designs allow the model to more effectively disentangle shared and task- specific information. For the latter implementation,
employ MMoE solely during self- self-supervised pretraining and transfer only the encoder for downstream tasks, unlike its conventional use for joint training and inference in supervised multi-task learning [3.]
Our contributions are summarized as follows:
We reveal that invariant and equivariant objectives are inherently interdependent, showing that conventional two-branch approaches with separate projection heads independently encode shared information across tasks, leading to redundant feature learning-
We propose Soft Task-Awant ware Routing (STAR) for invariant-equivariant representation learning that explicitly coordinates shared and task-specific information, thereby mitigating redundant feature learning and improving transfer performance across various downstream tasks.
We validate the effectiveness of STAR by demonstrating substantial reduction in canonical corre- lation between projection modules, along with positive correlation between reduced redundant feature learning and improved generalization performance.
2 Related Work
Equivariant Representation Learning. Farly approaches to SSL. enforce invariance to transforma- tions by encouraging augmented views of the same image to produce similar embeddings 3 20 17 4 but such constraints may discard informative features like color or spatial location, which could be important for downstream tasks (27). To address this limitation, recent research explores equivariant representation learning, aiming to learn representations that transform consistently with input trans- formations. E-SSL
g introduces an auxiliary task of predicting the applied augmentation, while AugSelf 27 predicts the difference between applied augmentation parameters. Both approaches implicitly learn equivariant representations by predicting input transformations through embedding space differences without explicitly modeling the transformation functions. CARE [18] shows that rotational symmetry in the representation space, aligned with input augmentations, can induce equiv- ariance without modeling the transformations. In contrast, recent methods such as EquiMod [9] SIE [1), and STL. [51] explicitly model transformation effects by predicting commentation-inducee shifts in the embedding space. EquiMod achieves this through learnable predictor conditioned on the applied augmentation. SIE splits the representation into invariant and equivariant parts, and generates coamenttaiion-oonttionnne predictor using a hypernetwork. STL learns transformation representations directly from unlabeled image pairs. Both STL and CE-SSL (50) are trained with explicit augmentation parameters, while CE-SSL focuses on preserving structured variability in the representation space rather than explicitly modeling the transformations. Our proposed approach, STAR, advances this line of work by emphasizing the interplay between invariant and equivariant learning. Rather than treating the two objectives as independent, STAR explicitly models the shared information essential to both while mitigating redundant feature learning, thereby enabling the model to more effectively capture features distinctive to each objective.
Mixture of Experts. The Mixture of Experts (MoE) framework was first introduced by 23) where multiple experts are trained jointly along with single gating module. This gate performs soft routing by assigning input-dependent weights to each expert's output. The router adaptively combines these outputs, enhancing both flexibility and performance on downstream tasks. Recently, MoE has been
In Figure 3 we empirically verify the existence of shared information between invariant and equivariant learning
tasks using benchmark dataset.
2

---

extended to multi-task and other representation learning settings. One such example is MMoE [3]] which introduces a multi-gan -ate architecture into the MoE framework for multi-task learning. this design, experts are shared across tasks, while each task is equipped with its own gating network. This setup enables the model to learn task relationships directly from data. In computer vision, V-MoE [42 introduces a sparse Vision Transformer that achieves comparable accuracy to large dense models with about half the inference cost. Neural Experts []] further extend MoE to implicit representations by dividing the input space among MLP experts, enabling local, piecewise function learning. Unlike conventional MoE models that require routing during both training and inference- making them difficult to transfer across tasks-STie
confines the .MME structure to the projection head during pretraining. Since projection heads are discarded after self-supervised pretraining, the expert specialization achieved within them does not need to be retained during transfer. This design preserves the benefits of expert routing while completely eliminating the transferability limitation of conventional MoF, enabling efficient fine-tuning and seamless adaptation to downstream tasks.
3 Preliminaries: Invariant and Equivariant Representation Learning
Let X' be the image space and A be a set of augmentation parameters that induce transformations on X' Given image E X' and augmentation parameter E A, we define the transformation function T: X' x A + X' that produces the augmented view T(x; a). An encoder f :X' + y maps the input image its latent representation y E
Invariant Representation Learning. Invariant representation learning aims to learn encoder that is invariant to transformations applied to the input. Formally, invariance on the representation space is defined as:
Va E A, A relaxed formulation, which requires consistency among transformed views of the same image, is given by:
f f(T(r;
Va,d' E A, -
If the parameter of an identity transformation is included in A. then Eq. implies Eq.
In practice, this invariance is achieved by training the encoder to produce similar representations across augmented views of the same image. This commonly achieved by minimizing dissimilarity loss of the form:
cinv = ((;((( a)), ((T(;;; af
where a, d' E A. In contrastive learning, a common choice for the loss function L the InfoNCE loss [47). which aims minimize the distance between representations of augmented views of the same image, while pushing them away from representations of other images. This leads the encoder to learn representations that are consistent across views, thereby preserving semantic content while discarding information that is specific to the applied transformations.
Equivariant Representation Learning. Equivariant representation learning focuses on learning representations that reflect the transformations applied to the input. Unlike invariance, which seeks to map all augmented views to the same representation, equivariance preserves structured changes that correspond to the transformation in the input space. Formally, equivariance the representation space defined as:
Va E A. f(T(z;)) = (f()))))))
where Ør : // x A -> y denotes a transformation function the representation space. The function Φr is learned to reflect how the input transformation T(::a) affects the latent representation. When dr reduces to the identity function for all a, Eq. 44) simplifies to the invariance condition in Eq. ① In this case, invariance can be viewed as special case of equivariance.
To encourage equivariance during training, the model learns to predict the transformed representation of transformed input. This leads to a learning objective of the form:
_ L(oT(J o), ((T(x;))))
where the loss L minimizes the discrepancy between the predicted and target representations of the transformed input. This formulation is designed to capture features that are sensitive to trans- formations, such pose and color. It is often incorporated as auxiliary loss in addition to
3

---

Single Shared
2
Equivariant Learning
Projection
MMoE
2
Projection
InfoNCE
2
PT
2
Finv
Esh
Eeq
Rinv
E1
Eg
E3
F
Req
MMoE
MMoE
Router
Expert
Equiv.
T(2: (()
Tha:
Figure 2: Overview of Proposed Routing Strategy: Soft Task-Aware Routing (STAR) of Experts. Given augmented views ((;; a) and T(x; a'), the encoder extracts features, which are then projected by the single shared projection (with three experts) or the MMoE projection module into invariant (z"")) and equivariant (zº9) embeddings. For equivariant learning, the projected augmentation parameter (a) and the equivariant embedding ² are fed into a predictor Φr to predict the target embedding 24 In practice, implement & as a single-layer MLP, and oT as 3-layer MLP.
invariant learning objective in recent works on equivariant learning 9 14 5T.
The overall
saariant--quivariann loss is formulated weighted sum of the invariant and equivariant losses:
== L XL
(6)
where is balancing coefficient that controls the contribution of the equivariant loss.
4 Method
In this section, we present Soft Task-Aware Routing (STAR) for learning invariant and equivariant representations. Motivated by redundant feature learning that arises when using separate projection heads for the invariant and equivariant objectives, STAR disentangles shared and task-specific information and adaptively weights their contributions based on the task and input image. For clarity, we refer to each projection head as expert throughout the paper.
4.1 Soft Task-Aware Routing
Setup. Given a batch of input images (x,) B we generate two augmented views per image by applying transformations a;):;)) and a.)))))) where a, af are augmentation parameters sampled from distribution over A. Specifically, for the i-th image z;, obtain two augmented views:
U: =Taa;a;;
= T(z;:))..
Thus, the resulting batch consists of 2B augmented views, where the first B correspond to { and the remaining to { 1 Each augmented view 0; is processed by the encoder f to extract a latent representation. The features then passed into our proposed task-aware projection modules, as illustrated in Figure to produce the invariant embedding =!!h and the equivariant embedding et
Single Shared Projection. As discussed in Section using two separate experts for invariant and equivariant learning lead to redundant feature learning experts. To alleviate this issue, is crucial to model the shared information between the two tasks. A straightforward yet effective approach is to introduce single expert shared across invariant and equivariant learning tasks, thereby allowing the model to capture information essential to both. To formalize this, we define three experts taking the representation input: invariant expert FioN an equivariant expert E.9, and shared expert E*, The embeddings are then computed as follows:
inv 2
= E (f(e:) (f(e:))
2 = = ===(((((:)) Ef(e)))
(8)

---

An overview of this computation is illustrated in the left panel of Figure 2 naturally encourages the shared expert to capture information relevant both tasks, since its output contributes directly to both invariant and equivariant objectives. However, the shared expert's output always weighted equally regardless of tasks and input images, making the model unable to account for the varying importance of shared information across them. While effective in reducing redundancy, this approach is inflexible and cannot adjust how shared and task-specific information are weighted.
This additive formulation
MMoE Projection. To extend the single shared projection to a more flexible and adaptive design, we introduce the MMoE projection module, which adaptively selects relevant experts from shared set based on the task and input. This mechanism is illustrated in the center panel of Figure 2 Specifically, the MMoE module contains the shared set of experts {E& [==]=== routers Riny and R The routers compute assignment weights as:
and two task-specific
Si softmax (R (f())) S1 softmax (R*
29 where !ay and denote the assignment weights of the k-th expert for the i-th view in the batch, i,k i, used in the invariant and equivariant projections, respectively. Here, the k-th component of the softmax activation applied to vector [wi w2...., WN] E ER is defined as: exp(wk)
softmax = Σド= exp(w;)
(10)
In our case, w the router output, and S denotes the score assigned the k-th expert. Unlike hard assignment strategies, our method employs soft routing, where each expert contributes proportionally its routing score. Accordingly, the invariant and equivariant embeddings are computed as weighted sums:
Thh (f(ov))
S Fe (f(()))))
(11)
The equivariant embedding of the original input 2;, denoted as 2f. obtained in the same manner:
N
S (x((x))
(12)
This design enables the model to adaptively route experts depending on the learning objective (i.e., invariant or equivariant) and the input image, resulting in a natural division into shared and task- specific roles. Notably, the single shared projection introduced earlier can be regarded degenerate case of MMoE projection, where all experts are equally weighted rather than adaptively determined.
4.2 Equivariant Learning
To explicitly model the shift induced by augmentations in the equivariant embedding space, we compute the predicted equivariant embedding by adding shift that reflects the effect of the applied augmentation to the original equivariant embedding 2.. This shift in the equivariant embedding space predicted by an equivariant predictor oT that takes as input both of and projected augmentation parameter, obtained via projection function y/:
2 +
(aa)))
(13)
The residual connection in Eq. (L3 ensures that the semantic content of the original equivariant embedding is preserved while effectively modeling the shift in the embedding space caused by transformations.
Following the standard formulation of invariant learning, we define the equivariant loss in a similar manner using the InfoNCE loss [47]. For each predicted embedding 2 the corresponding target embedding 244 serves as a positive sample, while the remaining equivariant embeddings in the batch serve negatives. The overall equivariant loss is defined as:
1
2B
exp (sim(2 09 2, 1/T)
L
log
(14)
2B
2B Σ exp (sim (2,2, , //T))
1
where .im(.,) denotes cosine similarity and T is temperature parameter. We use the invariant- equivariant loss described Eq. (6) for our method employing SimCLR 3 for invariant loss unless otherwise specified. We set 0.2 and x = 1 for all experiments.
5

---

Table Out-of-Domain Classification. Linear evaluation accuracy (%) of ResNet-50 pretrained on ImageNet100 for 500 epochs and ResNet-18 18 pretrained on STL10 for 200 epochs, respectively. Bold entries indicate the best performance among methods, while underlined entries denote the second best. Standard deviations are reported in Table B.2
Method
CIFARIO
CIFAR1O0
Food
MIT67
Pets
Flowars
Caltech101
Aircraft
DTD
SUN397
Mean
Avg. Rank
ImageNer/00 preetraiid
76.71
69.17
AugSelf
88.61
69.68
65.37
67.51
77.24
89.70
85.09
47.45
48.65
69.31
53.00
69.24
3.82
EquiMod
88.99
70.22
64.43
67.54
77.78
90.33
86.62
48.94
49.91
69.33
52.79
69.72
3.18
CARE
82.81
58.97
55.78
56.39
59.89
75.84
73.14
29.12
36.00
62.22
42.48
57.51
6.00
70.57
STAR-MMOE
S71.
SYL.IU-ppetrriiedd RexNee-18
SimCLR
83.56
55.19
33.75
39.01
46.15
60.27
66.85
17.38
27.17
43.12
28, 58
45.55
5.73
AugSelf
84.03
58.65
38.11
41.74
47.80
68.54
69.33
20.23
31.55
45.68
32.51
EquiMod
STAR-SS
35.64
61.10
40.17
44.50
50,02
72.59
23.39
21.89
33. 33.90
48.
34.25
31.46
2.55
STAR MMoE
87.45
64.78
31.28
46.82
51,10
73.99
74.76
22.74
35.61
89.75
35.50
$3.07
1.36
Table In-Domain Classification. Linear evalua- tion accuracy (%) of ResNet-50 and ResNet-18 18 pre- trained on ImageNet100 and STL10, respectively.
Table 3: Object Detection. Evaluation of learned representations on the object detection task using Faster R-CNN with a frozen ResNet- 50-C4 backbone on VOCO7+12, Reported met- ries include AP, AP50, and AP75.
Method
STL10
ImageNet100
SimCLR
85.24
83,43
Method
AP
AP50
AP75
AugSelf
85.99
83.95
SimCLR
47.96=0.19
76.35+0.14
51. 51.62:0.46
EquiMod
87.01
84.81
AugSelf
48.00=0.21
1420.8
51.8340.34
CARE
79.86
80.38
EquiMod
48.52=0.15
76.55+0.01
52.8240.22
STAR-SS
86.75
84.82
CARE
48.41=0.28
76.1640.20
52.06+0.55
STAR-SS
48.77=0.
STAR-MMoF
86.74
84.83
STAR STAR-MMoE
48.85=0.20
76.81#0.07
53.01+0.23
Experiments
Setup. We pretrain ResNet-18 19 on STL10 7 for 200 epochs and ResNet-50 on Ima- geNet100 [4] [5] for 500 epochs, both with batch size of 256. For our proposed methods, we denote the single shared projection variant as STAR-SS and the MMoE projection variant as STAR-MMoE. We compare methods against existing equivariant representation learning ap- proaches, including AugSelf 277, EquiMod [9) and CARE [88. For all approaches, SimCLR is used invariant representation learning baseline.
For STL10, we use 16 experts as the default configurations, while 8 experts are used for ImageNet100. For analysis, use the STL10-pretraine model with 8 experts for better interpretability. All results are averaged over three runs, reporting mean and standard deviation, and we reproduce all compared methods ensure fair comparisons. See Appendix B for detailed configurations.
5.1 Main Results
Image Classification. We conduct transfer learning experiments on downstream datasets: CIFAR10/100 (26), Food (2). MIT67 (39]. Pets [37]. Flowers [34], Caltech101 [13), Cars [25]. Aircraft (32), DTD 6), and SUN397 49 For evaluation,
follow the linear evaluation protocol used in 224 Table shows the transfer learning results across various downstream tasks. Out of the 11 downstream datasets, our method achieves the best performance in when pretrained on STL10, and 10 when pretrained on ImageNet100, consistently surpassing previous approaches. As shown in Table 2 our method also achieves strong in-domain performance, suggesting that improvements in out-of-domain performance are achieved without degrading, even potentially enhancing in-domain performance.
Object Detection. We evaluate our method on the object detection task using the Pascal VOCO7+12 dataset [1]]. According to [16], performance obtained after full fine-tuning reflects not only the quality the learned representations, but also the effects of initialization and optimization strategies. Therefore, linear evaluation with frozen backbone is more appropriate for assessing the quality of

---

Table 4: Few-Shot Classification. Few-shot classificiation accuracy (%) with 95% confidence intervals averaged over 2000 episodes. (N, K) denotes N-way Khshot tasks. Bold entries indicate the best performance among methods, while underlined entries denote the second best.
FC100
CUB200
Plant Discase
Method
1)
(6,5)
(5.
(5.6)
(5. 1)
(5,5)
ImageNett/00
1ewp-eeeeOO00Pereinne ResNet-50
SimCLR
35.72=0.34
50.27=0.38
45.65+0.49
61.1340.44
71.55+00.47
87.88+0.32
AugSelf
36.12=0.37
51.02=0.40
45, 45.9620.49
61.94+0.48
12.1840.47
88,48+0.32
EquiMod
35.21
49.4920.39
46.
61.72a0.49
72.4810.47
38.7120.32
CARE
27.09
36.4420.35
41.01
53.4010.46
50.9340.46
70.7440.8 40
STAR-SS
35.8020.37
51.33=0.39
47.11
62. 1270.47
72.48+0.45
89,4640.22
STAR-MMoE
38.2620.37
53,57
47.2620047
63,46200447
73.86210.66
90.0410.29
STL.IO-reetrineed ResNet-18
SimCLR
36.49=0.35
41.4640.44
35.55+0.36
50.1 180..37
56.89+0.44
75.86+0.39
AugSelf
37.99=0.37
52.81=0.39
38.34+0.38
54, 54.02+0.39
59,76+0.49
78.31+0.36
EquiMod
36.4220.36
50.6420.36
39. 39.0820.43
52. 82.8840.44
58.2540048
78.01=0.37
CARE
32.93=0.39
43.04=0.40
36.93+0.42
51.28+0.43
58. 58.62+0.49
79 60.6670..6
STAR-SS
38.97=0.39
53.36=0.40
40.70+0.43
55.37+0.42
60.11#0.46 48
79.83+0,36
STAR-MMoE
39,3520.39
54.8720.40
41.4620.42
57.31:0045
61.70220447
81.1420.36
the learned representations. Following this principle, we adopt a representation evaluation protocol analogous to that used in image classification, where the backbone is frozen to assess the quality of the learned representations. Specifically, we transfer pretrained weights to Faster R-CNN [41 architecture with R50-C4 backbone, freeze all convolutional layers from CI to C4, and fine-tune only the region proposal network (RPN) and the object classification head (C5). As shown Table our method outperforms all baselines in AP, APS0, and AP75, demonstrating that it produces stronger and more transferable representations for object-level understanding.
Few-Shot Classification. We evaluate the generalizability of learned representations under limited data conditions via few-shot classification, following the linear evaluation protocol for few-shot learning in [27]. Specifically, we report classification accuracies and 95% confidence intervals for 5-way 1-shot and 5-way 5-shot tasks across 2000 episodes on FC100 [36). CUB200 [48, and Plant Disease datasets [33]. As shown in Table our method consistently achieves the best performance across all few-shot classification settings, outperforming other equivariant representation learning methods in every case.
5.2 Analysis
Shared Information between Invariant and Equivariant Learning. To verify the existence of shared information between invariant and equiv- ariant learning, we conduct k-NN retrieval on STL10 test set augmented with flipped images using invariance-only and equivariance-only mod- els, where we employ SimCLR (3 as invariance- only model and EquiMod [9] without the invariant loss as equivariance-ong equivariance-onl model retrieves semantically similar neighbors even without invariant learning. Specifically, in the results of the equivariance-only
Query
Invariance-only
Equivariance-only
Figure 3: k-NN Retrieval on STL10. Query image (left); retrievals from the invariance- only model (top-right) and the equivariance-only model (bottom-right).
model. Notably, the
model, we observe three patterns: (i) it consistently retrieves samples belonging the same semantic object class, (ii) it consistently matches samples with the same color, and (iii) alternates between samples of the same orientation with query and their flipped counterparts. These observations indicate that the equivariance-onl) model encodes not only sugmentation-awar aspects such as orientation and color but also semantic cues like object identity, highlighting the shared information between invariant and equivariant learning.
Expert Specialization. We investigate how the MMoE projection module allocates experts to the invariant and equivariant objectives by analyzing the average routing weight distributions, measured
7

---

Query
I-NN
2-NN
N-NN
4-NW
0.5
Invarience
0033
Shared (Expert 1)
Routing Weight
Equivariance
0.30
Min/Max Ratio
0.25
0.3
Invarfance (Expert
0.2
0.15
0.10
dirplasa
airgions
ship
ship
bruck
0.05
Equivariance (Expert 7)
0.0
2
3
5
0.00
Expert Index (Reordered)
dirplane
bruck
airplone
(a) Routing weight distribution across experts.
(b) k-NN retrieval on STL10 test dataset.
Figure 4: Analysis of Expert Specialization. (a) Routing weights averaged over test data in STL10, with experts reordered based on their roles. The min/max ratio (purple) measures the balance between how much each expert is utilized by the invariant and equivariant objectives. (b) k-NN retrieval results using the output embeddings of individual experts.
0.60
0.60
Canonical Correlation
STAR-55
.34
0.55
53
EquiMed
37
Mean Accuracy
16
0.45
12
Mean Canonical
0.45
52
.22
.3
.30
Expert
0.30
0.40
STAR-MINLS
80
0.25
STAR-G5
48
16
32
Expert Index (Reordered)
# of Experts
of Experts
(a) Canonical correlation matrix.
(b) Mean canonical correlation.
(c) Mean accuracy.
Figure 5: Analysis of Redundant Feature Learning. (a) Pairwise canonical correlation between expert outputs. (b) Mean canonical correlation and (c) mean classification accuracy on 11 out-of- domain datasets across different numbers of experts in our proposed method. For (a), the numerical values of the diagonal elements are omitted for better visualization.
on the STL10 test dataset, as shown in Figure 4a Expert receives relatively balanced weights from both routers, implying that it learns information shared between the invariant and equivariant objectives. In contrast, Experts 2 to 6 are mainly used for the invariant objective, and Experts and 8 are mainly used for the equivariant objective, indicating the task-specific specialization of experts. This distinction is further quantified by the min/max ratio, which computes the proportion of the smaller routing weight value to the larger one for each expert; higher values imply more balanced usage across tasks. To confirm these observations, we conduct k-NN retrieval on the STL10 test set, shown in Figure 4b Expert 3 and Expert 7 retrieve samples that emphasize invariance- and equivariance-speer speciifc features, respectively, while Expert retrieves semantically consistent neighbors, indicating that it captures shared information beneficial to both objectives. These results demonstrate that the MMoE architecture supports meaningful expert specialization aligned with the learning objectives.
Redundant Feature Learning. We assess redundant feature learning among experts by measuring canonical correlation between their outputs, where higher correlation indicates that experts capture similar, potentially redundant information. In Figure Sa the matrix shows that experts assigned to the same objective (e-g., Experts and 8) tend exhibit higher mutual similarity, whereas the similarity between experts specialized in different objectives (e.g. Experts and 7) remains relatively low. Notably, Expert exhibits moderate similarity with both groups, indicating that it primarily encodes information shared across invariant and equivariant objectives.
To further quantify how redundant feature learning affects generalization, we vary the number of experts to control the degree of redundant feature learning in the model. As shown in Figure [5b] increasing the number of experts tends to reduce redundant feature learning, as reflected in the lower mean canonical correlation. This reduction in redundant feature learning is accompanied by consistent improvements the mean accuracy, as shown in Figure 5c Compared to baselines such EquiMod, our method consistently achieves both reduced redundant feature learning and higher

---

Norm of Weight Update
Pequivariance
Invoriance
STAR-SS (0.34)
EquiMod (0.59)
0.6
0.5
0.4
0.3
50
75 125
150
Export Index (Roordered)
Figure 6: Convergence of Experts. Frobenius norm of weight update for experts over training epochs. All models are initialized and optimized using the same scheme to ensure a fair comparison of weight updates.
Figure 7: Equivariance of Expert Embeddings. P-equivariance and invariance of experts embed- dings in our method pretrained STL100.
accuracy, suggesting the effectiveness of dynamic expert allocation. These results highlight that reducing redundant feature learning through adaptive expert assignment plays key role in promoting specialization and improving generalization.
Impact of Redundant Feature Learning for Representations. To understand how redundant feature learning affects the backbone representations, we analyze its impact on expert convergence and the resulting gradient quality. As shown Figure experts our method with the MMoE projection converge faster than those in EquiMod, consistent with prior findings that MoE architectures achieve
faster convergence than dense models under the same number of training steps 22 12 40 Since
convergence of expert directly shapes the gradient signals to the backbone, faster convergence implies that our experts provide higher-quality and task-fpecific gradients the backbone early in training, when the learning rate is high. In contrast, experts in EquiMod converge more slowly and yield suboptimal gradients, as redundant feature learning hinders task specialization and weakens task- specific updates. We empirically validate this by measuring the cosine similarity between gradients from different experts with respect to the backbone, which decreases from 0.59 in EquiMod to 0.30 in our method with MMoE projection. This result indicates reduced redundant feature learning and improved task specialization.
Evaluation of Equivariance. We evaluate the equivariance of representations and experts us- ing R-equivariance and P-equivariance, as pro- posed in [38] R-equivariance, measured with cosine similarity, quantifies how well the trans- formed embedding be predicted from the original embedding and the transformation pa- rameters, whereas P-equivariance, measured with mean squared error, quantifies how accurately the transformation parameters be recovered from the original and transformed embeddings. As shown in Table 5 representations from method achieve higher R-equivariance and lower
Table 5: Equivariance Representations. Com- parison of R-equivariance and P-equivariance of representations learned by different methods pre- trained on STL10.
Method
R-equiv. f
P-equiv.
SimCLR AugSelf
0.74
0.72
0.92
0.32
EquiMod CARE
0.91
0.38
0.97
0.51
STAR-SS
0.93
0.27
STAR-MMoE
0.98
0.27
P-equivariance than those from other methods, indicating stronger equivariance.
Figure 7 shows the P-equivariance and invariance scores across experts in our method. Invariance is measured as the cosine similarity between embeddings of differently augmented views. Equivariant experts exhibit lower P-equivariance and lower invariance, indicating that they specialize in the equiv- ariant learning task and capture transformation--elatee information essential for learning equivariance. In contrast, invariant experts show higher P-equivariance and higher invariance, demonstrating they specialize in the invariant learning task. addition, the shared expert shows moderate P-equivariance together with an invariance score comparable to that of invariant experts, reflecting its role in encoding shared information between invariant and equivariant learning. These results further confirm that our method achieves task specialization.

---

Training Efficiency. Figure 8 compares the mean accuracy of different methods over train- ing time in the out-of-domain classification set- ting, with wall-clock time measured on a single RTX 4090 GPU. While most methods steadily improve, SimCLR and AugSelf show saturation even decline after 300 epochs, indicating ten- dency to overfit the training distribution and limited ability to generalize to unseen domains. This observation is consistent with Figure Aa. la where their in in-oomain accuracy continues in- crease beyond 300 epochs. In contrast, our pro- posed method consistently achieves higher racy and maintains this advantage throughout the entire training process. For example, our method surpasses other approaches trained for longer du- rations. Although our method may incur a higher cost per epoch, its ability to efficiently achieve strong generalization in less wall-clock time highlights its advantage in scenarios where early stopping is desirable.
STAR-MMME
EquiMod
52
Augšelf
Mean Accuracy (%)
SimCLR,
50
48
46
44
1.0
1.5
2.0
2.5
3.0
3.5
Training Time (h)
Figure 8: Training Efficiency in Out-of-Domali Classification. Mean accuracy (%) of STLI0- pretrained models in out-of-domain classification. Markers are shown every 100 epochs.
6 Discussion and Conclusion
This work revisits equivariant representation learning and demonstrates that treating the invariant and cquivariant objectives as fully independent, as commonly done in wo-bbanch architectures, leads to redundant feature learning experts. To address this, propose STAR, which dynamically routes experts to each task. This reduces redundant feature learning and enhances generalization across downstream tasks such image classification, few-shot learning, and object detection.
Linitations. A limitation of our method is the of soft routing, which prevents the of sparse routing strategies such as top-k k routing. While sparse routing improves computational efficiency in conventional MoF models 44 [2 it causes unstable training in setting due the presence of batch normalization each expert. Activating only a subset of experts per sample leads to unreliable batch statistics, especially problematic in SSL where stable statistics we adopt soft routing to ensure that all experts receive inputs in every batch. Although this reduces computational efficiency and scalability, it enables stable training dynamics.
essential. To avoid this,
Acknowledgements
This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Ministry of Science and ICT (MSIT) of the Korean government (RS-2024-00341144 RS- 2024-00345351, 00-2022-000008003), and Institute of Information & Communications Technology Planning & Evaluation (IIP) grant funded by MSIT (KS-2023-00259934, RS- RS-2025-02283044)
References
[1] Yizhak Ben-Shabat, Chamin Hewa Koneputugodage, Sameera Ramasinghe, and Stephen Gould. Neural experts: Mixture of experts for implicit neural representations. In NeurIPS, 2024.
[2] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 -mining discriminative components with random forests. In ECCV, 2014.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[4] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR. 2021.
[5] Xinlei Chen Saining Xie* and Kaiming He. empirical study of training self-supervised vision transformers. In ICCV, 2021.
10

---

[6] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures the wild. In CVPR, 2014.
[7] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In AISTATS, 2011.
[8] Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić. Equivariant contrastive learning. In ICLR, 2022.
[9] Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve
self-supervised learning. In ICLR, 2023.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[11] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 2010.
[12] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv:2101.0300 2022.
[13] Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop, 2004.
[14] Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. In ICML, 2023.
[15] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in I hour. arXiv preprint arXiv:1706.02677 2017.
[16] Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking sule-supe visual representation learning. In ICCV, 2019.
[17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach self-supervised learning. In NeurIPS, 2020.
[18] Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structuring representation geometry with rotationally equivariant contrastive learning. In ICLR, 2024.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[21] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
[22] Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[23] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 1991.
[24] Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better imagenet models transfer better? In CVPR, 2019.

---

[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In CVPR Workshops. 2013.
[26] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[27] Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-aware self- selfssupervision In NeurIPS, 2021.
[28] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In ICLR, 2021.
[29] Nya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. In ICLR, 2016.
[30] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[31] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of- ef-eeerss.. In KDD, 2018.
[32] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. Technical report, Visual Geometry Group, University of Oxford, 2013.
[33] Sharada P. Mohanty, David P. Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in Plant Science, 2016.
[34] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number classes. In ICVGIP, 2008
[35] Jeongheon Oh and Kibok Lec. On the effectiveness of supervision in asymmetric non-contrastive learning. In ICML, 2024.
[36] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, 2018.
[37] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
[38] Christos Plachouras, Julien Guinot, George Fazekas, Elio Quinton, Emmanouil Benetos, and Johan Pauwels. Towards unified representation evaluation framework beyond downstream tasks. In IJCN,, 2025.
[39] Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In CVPR, 2009.
[40] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation scale. In ICML, 2022.
[41] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[42] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In NeurlPS, 2021.
[43] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
[44] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In ICLR, 2017.
12

---

[45] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
[46] NASA/GSFC/Arizona State University. Recent impact crater on the lunar surface show- ing crater illusion. https:://comone www.wwwia..iiooommmmmmm wttpw//ndwww....e//e/oo/oooo.wam http:///mmcexxpppppprroe////4///
2023. Public Domain.
[47] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1r07v:1807.033 2018.
[48] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
[49] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. CVPR, 2010.
[50] Thomas Yerxa, Jenelle Feather, Eero P. Simoncelli, and Yeon Chung. Contrastive-equivvrraat sulf-supe learning improves alignment with primate visual area it. In NeurIPS, 2024.
[51] Jaemyung Yu, Jachyun Choi, Dong-Jae Lee, HyeongGwon Hong, and Junmo Kim. Self- supervised transformation learning for equivariant representations. NewrIPS, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: The paper provides open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //nips.cc/ppl wwwppbblccgguiicuceccccccoeooopp hetggudddsse/eebbbinninn htt.SSabmiiiiooooiiioiooooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports standard deviations of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources, including computational time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The proposed method for jointly learning invariant and equivariant repre- sentations using task-aware routing mechanism has the potential advance the field of self-supervised learning and improve the transferability of learned features across domains. By explicitly modeling shared and task-specific components, our approach enables more expressive and generalizable representations, which may benefit applications in downstream tasks such as classification, detection, and few-shot learning.
While our work is foundational and not directly tied to deployment in sensitive domains, stronger representation learning techniques could be misused in areas such surveillance or user profiling, especially if used without transparency or accountability. To encourage responsible use and promote reproducibility, we plan to publicly release our code and pretrained models upon acceptance. This release will support transparency and allow the community to evaluate, adapt, and monitor potential societal impacts.
17

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not contain any data or models that have high risk for misuse. Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators or original owners of assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets introduced in the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There are no crowdsourcing experiments or research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not contain any potential risks iccured by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development the paper does not involve L.MSs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:e///eerrrppccc/oooffeereeee /ttp://///oeeeeeec////////.... for what should or should not be described.
20

---

Soft Task-Aware Routing of Experts for Equivariant Representation Learning
Supplementary Material
A Additional Experiments
A.1
Comparison with STL
Table A.1 compares the linear evaluation performance of method against STL [51 and STL combined with AugMix [21] across 11 downstream classification benchmarks. We report results under two pretraining settings: STL10 with ResNet-18 and ImageNet100 with ResNet-50. Following the experimental setting of the STL paper, we evaluate on the AWS Flowers dataset, which differs from the Oxford Flowers (34 in its train/test split. Except for this replacement, all other evaluation protocols are kept identical.
Table A.I: Image Classification. Linear evaluation accuracy (%) across 11 datasets. Models pretrained on STL10 using ResNet-18 or on ImageNet100 using ResNet-50. Symbol denotes performance reported in [5]]. based on 200 epochs for STLI0 and 500 epochs for ImageNet100. For Flowers, we use AWS Flowers instead of the Oxford Flowers.
Pretnining
Michoal
CIFARIO
CIFAR100
Food
MIT67
Pax
Flassen
Chltadh11
Can
Airendi
DTD
SUNY99
Mran
STL"
$1.10
56.35
66:34
64.32
36.64
65.00
94.51
$1.83
35.42
45.42
64.68
48.69
64.18
ImagoNee100
AugMis* STAR-MME
67.1
79.27
93.64
$7.76
70.80
73.50
77.37
51.10
78.93
74.76
53.52
Across both pretraining regimes, our method consistently outperforms STL and its AugMix variant on the majority of datasets. These results demonstrate the effectiveness of our method in producing more generalizable representations.
A.2 Training Efficiency
83.5
83.0
Accuracy
84
81.5
03
EquiMadi
81.0
EquiMMd
AugSelf
AugSelf
02
SimCLR
SinCLi
1.0
2.5
5.0
Training Time (h)
Traiiing mg Time (h)
(a) STLLOTTT-eeeeee pretrri-eeeinedd model
(b) Image preerriind model
Figure A.1: Training Efficiency in In-Domain Classification. Mean accuracy (%) of (a) STLIO- pretrained and (b) ImageNet1 poreetriind models in in-domain classification. For STL10 pretraining. all markers are shown every 100 epochs; for ImageNet100 pretraining, markers are shown every 100 epochs for SimCLR and AugSelf (from 200 epochs) and every 50 epochs for EquiMod and STAR-M MMoE (from 100 epochs).
21

---

Figure A. shows that all methods follow similar convergence trends in the in-domain classification setting, with most reaching comparable accuracy levels training progresses. Notably. E achieves performance par with AugSelf, the best method under ImageNer100 pretraining, and EquiMod, the best under STL10 pretraining.
56
60.5
55
5 STAR MiMoE
EquiMod
AugSolf
63
SimCLR
$9.0
Accurac
62
58.0
57.5
STAR-MMoE
50
EquiMod
AugSalt
SiNCLR
56.5
1.0
2.5
3.0
Training Time (h)
Training ng Time (h)
(a) STett-prerrainedd model
(b) 1mageNer100-pretriine model
Figure A.2: Training Efficiency in Few-Shot Classification. Mean accuracy (%) of (a) STL10- pretrained and (b)
models in few-shot classification. STL10 pretraining, all markers are shown every 100 epochs; for ImageNet100 pretraining, markers are shown every 100 epochs for SimCLR and AugSelf (from 200 epochs) and every 50 epochs for EquiMod and STAR-MMoE (from 100 epochs).
However, the distinction becomes more pronounced in the few-shot classification setting, as shown in Figure A.2 our method consistently outperforms all other methods, including those trained for longer durations. Figure A.2a further illustrates that, unlike STAR-MMEE, the methods pretrained with STL10 become saturated even with additional training.
70.0
69.5
89.0
68.5
STAR- MMaE
EquiMod
AugSait
SimCLR
Training Time
Figure A.3: Training Efficiency in Out-of-Domain Classification. Mean accuracy (%) of ImageNet100-ppe preettrraied models in out-of-domain classification. Markers are shown every 100 epochs for SimCLR and AugSelf (from 200 epochs) and every 50 epochs for EquiMod and STAR- MMoE (from 100 epochs).
Consistent with the results in Figure Figure A.3 exhibits the same trend, further underscoring the strength of STAR-MMon with ImageNet100 pretraining. In most cases, our method surpasses others even when they are trained for longer durations. This demonstrates that the proposed approach not only achieves superior transfer accuracy but also attains it more efficiently, making it particularly well suited for scenarios that demand strong generalization to unseen domains and high training efficiency.
A.3
VIT Backbone
To assess the backbone independence of our proposed method, we conduct additional experiments using the Vision Transformer (ViT) [0 architecture. Specifically, we adopt MoCo-v3 [5), which utilizes ViT as its backbone, and modify SimCLR by replacing the original ResNet18 ResNet50
22

---

with VIT. Following the experimental setup introduced in [3]. ViT-Small is pretrained on Ima- geNet100 for 200 epochs with a batch size of 256. The training setup commonly employs the Adamw optimizer [30], with linear warm-up of the learning rate during the first 40 epochs, a momentum of 0.9, and a weight decay of 0.1. cosine learning rate schedule [29] used for both the encoder and the projector, while in the case of MoCo-v3, the same schedule is also applied to the predictor.
MoCo-v3. We adopt the original parameter settings, with a learning rate of 1.5e-4 and a temperature of 0.2 for both contrastive and equivariant learning. The exponential moving average (EMA) coefficient is initialized 0.99 and gradually increased 1. We use a 3-layer MLP for each expert, with hidden and output dimensions set to 4096 and 256, respectively. The equivariant predictor is single-layer MLP with an input dimension of 512 and an output dimension of 256.
SimCLR. The learning rate is set to 1.5e-3, and the temperature for contrastive and equivariant learning also fixed at 0.2. We use a 3-layer MLP for each expert, with hidden and output dimensions set to 4096 and 256, respectively. The equivariant predictor is a single-layer MLP with an input dimension of 512 and an output dimension of 256.
Table A.2: Backbone Ablation Study. Linear evaluation accuracy (%) of ViT-S/16 pretrained on Ie0
CHFARIO:
85.19am:::
65, Tatu0
$6.678349
57.21a1:31
65.34a1442
84.39a5:22
75.04an:1
30.67a0.44 d6
35.66an.33
61.15at095
44.91a0:22
60.25
SinCLR
EquiMod
AugSelt
35 34
86.81n22
07.47
65 48-844
51 1645
60.00a1137
57 9040
62.86a8155
65 M4-n
86 86.906332
54 54.89-57
78.122
33.7189.5
10.740 m
16.00-0 14
38.773021
61 01.46-113
62.8760:32
45.4340 15
47.082908
62.54
60.64
Chrr
M6.884111
68.2468.15
19.75u1..9
68.494638
38.644m0
62914835
46.440775
MoCoss
FiuiModd
85 99-413
65.41.9/24
33-0
00.6 00.694370
39.7540 35
58.21an:33
67.88-0335
50 12-00-111
85.5740177
76.58an15
30.57a0.72
10.000
17:06-1
60.672044
63.21-6.31
45.71s0
46.330 11
60.94
60.21
Ours
86.723088
07.8968.34
00.3869:44
20.40.0.19
07.572033
87.3446.22
79.649033
35.276930
39.303053
62.90e118
47.7369.16
63.23
In Table A.2 method achieves better performance than existing equivariant representation learning approaches across most datasets. This highlights the effectiveness of the proposed MMoE projection module, which upplicable the ViT backbone.
A.4 Ablation Study Components
We conduct an ablation study to assess the contribution of each component in our method. As shown in Table A.3] substituting the projection head in EquiMod with the MMoE projection module leads to clear improvements in both in- and oft-oi-domain settings, highlighting its effectiveness in reducing redundant feature learning through dynamic expert allocation. both EquiMod and our proposed method, adding the residual connection (RC) and increasing the depth of the equivariant predictor (PD) contribute to improved out-of-domain generalization. However, the deeper predictor slightly reduces the in-domain performance. Combining all three components results in the best out-of-domain performance, demonstrating their complementary contributions to both expressivity and generalization.
Table A.3: Ablation Study on Components. MMoE: MMoE projection module; RC: residual connection; PD: deeper equivariant predictor.
Method
MMoE
RC
PD
In-domain
Out-domain
87.01
49.54
87.19
49.78
EquiMod
86.03
50.85
87.01
50.99
X
X
87.39
51.43
J
/
87.41
52.07
STAR-MMoE
X
86.57
52.53
86.74
53.07
23

---

A.5 Ablation Study on Baseline Invariant Learning Methods
Although the main results are presented based on SimCLR, as shown in Table A.4 our method generalizes well when combined with other invariant learning objectives, consistently outperforming all methods across various baseline invariant learning methods, including MoCo, SimSiam, and BYOL. Notably, EquiMod performs worse than AugSelf in non-contrastive frameworks such as SimSiam and BYO,, whereas our method maintains strong performance across both contrastive and non-contrastive frameworks.
Table A.4: Ablation Study on Baseline Invariant Learning Methods. Linear evaluation accuracy (%) of ResNet-18 18 pretrained on STL10 with various methods across SSL frameworks.
KI Miei-:0
$3.040
No SoansT
14.74w0 33
41 Teet
63
16 16.57004
41.5a0
2M 20500055
12
AugSal
60.0002)1
16.45-015
65:45
35.07a0
$1.15
STAR-MMet
37
39
11.254.33
12424855
11,354.35
XURMOU
54.354166
51.73
SinSian
EqriMed
Argter
22.10a133
86.29-8:33
35
62.774224
60 014064333
41:714465
35-39u037
45332041
05414934
114
15-0-40
11
10414436
12344000
12 1b
3030
STAR-MMEE
87.854331
353943912
12.331127
41.36u031
46392000
13.72334
33992824
35.56u116
34.728117
$2.42
57 JosuLS
$6.39.3.11
00 S0
37. 1.38ds 15
41 4177ane
30 Mati
67422033
7044:1
13. Ma.
52 18a0/1
44 44,79o2
31,59000
49.73
BVOL
AsgSar
13n.s
$6.022.29
64 Manuis
43.38a.35
11.552241
13.86:211
35.0mm
35.04:: 09
0331
6980.99
16:05:333
06-256021
43:240337
47.234600
52 13000099
76:224831
76766833
36.34633
5336
A.6 Hyperparameter Optimization
We study how variation of hyperparameters can influence our model. For that purpose, we train ResNet-18 18 STL10 with 16 experts and ResNet-50 ImageNet100 for each factor modification, and present results for both in-domain and out-domain scenarios. We mainly inspect the influence of X, the balancing coefficient of the equivariant loss (see Eq. (6)), and т, the temperature parameter used in the equivariant learning objective (see Eq. (4)). All All hyperparameters are tuned based on out-of-domain performance ensure generalization. As shown in Table A.5 both parameters have notable impact on performance.
Table A.5: Hyperparameter Tuning for À and is the loss balancing coefficient, and is the temperature parameter used the equivariant learning objective. We report performance on in-domain and out-domain settings under varying values of each.
Pretraining
A
In-domain
Out-of-domain
0.1
84.02
70.32
Pretraining
d
In-domain
Out-of-domain
0.2
84.20
70.82
0.5
84.66
70.94
0.05
84,12
70.53
IeageNet100
84.83
71.23
0.1
84,70
70,86
2
84.79
70.61
ImageNet100
0.2
84.83
71.23
5
84.68
70.44
0.5
84,20
70.98
10
83.82
69.14
1
84,10
70.74
0.1
86.47
50.37
0.05
85.84
51.74
0.2
86.46
$1.27
0.1
86.98
52.59
0.5
86.65
52.25
ST1.10
0.2
86.74
53.07
STL10
1
86.74
53.07
0.5
86.66
52.09
2 86.17
52.52
86.45
50.66
5
84.95
50.88
82.64
48.87
(b) Equivariant temperature parameter
(a) A: Loss balancing cocfficient
We observe that À controls the relative strength of the equivariant learning signal. Increasing improves performance up to certain point, with the best results observed at X = 1, beyond which performance drops due to the overemphasis on equivariance. This highlights the importance of balancing equivariant and invariant objectives to prevent one from dominating the learning process. For т, which modulates the sharpness of the similarity distribution in the equivariant contrastive loss, we find that 0.2 achieves optimal results. Smaller values such as 0.05 may cause representational
24

---

collapse due to overly confident similarity distributions, whereas larger values such as reduce the discriminative power of the model. These trends are consistent across both in-domain and out-domain evaluations, emphasizing the necessity of careful calibration of hyperparameters for generalizable representation learning. Moreover, the similar tendencies for both À and 7 are observed across STL10 and ImageNet 00 pretraining settings, and we find that EquiMod exhibits the same optimal hyperparameter configuration, indicating that these settings generalize well across related architectures.
A.7
Qualitative Analysis of Learned Representations
Query
2 VIN
2-HM
3-MN
4-MH
Query
1-NN
2-4MM
3-MN
4-MH
AugSelf
AugSeff
wolercrew
welorrss
EquiMed
Equi
Tower
wotercrma
Sartatee Batr
Teadily
gamaa
leadiity
Ieugeintiee
Seadd
leadily
leaditty
Seard Fiy
Figure A.4: k-NN Retrieval on Flowers Test Set. Results of k-NN retrieval using backbone features learned by Equivariant SSL methods.
presents qualitative results of k-NN retrieval on the Flowers test set using backbone features from equivariant SSL methods. All models are based ResNet-18 pretrained STL10. We observe that most methods exhibit strong sensitivity to color information, often retrieving visually similar but semantically incorrect samples. For example, methods such as AugSelf and EquiMod frequently return instances from different classes that share similar color distributions with the query. STI. also tends favor samples with matching low-level visual cues, rather than consistently retrieving semantically correct instances. This indicates that their learned representations may not fully capture semantic object-level features.
Figure A.4
In contrast, our method consistently retrieves samples from the same class as the query while still preserving sensitivity to fine-grained visual details such as color and texture. This suggests that our proposed method enables the model to encode features that are consistent with the class more effectively. Quantitatively, this advantage is reflected in performance improvement of approximately 5 percentage points compared to the method that performs best on the Flowers dataset.
A.8 Retrieval Visualization of Individual Experts
Figure A.5 provides full k-NN retrieval results using the output embeddings of all individual experts. While Figure 4b in the main paper focuses on three specific experts, namely the shared expert (Expert 1), the invariant expert (Expert 3), and the equivariant expert (Expert 7), this extended visualization enables a more comprehensive examination of the specialization exhibited by all experts.
We observe diverse retrieval patterns across experts. Expert consistently retrieves semantically similar instances, indicating focus object-level meaning. Experts 2 through 6 often retrieve samples that differ in color from the query, suggesting that these experts have learned to ignore color and instead emphasize more abstract features. In contrast, Expert 7 and Expert 8 tend to retrieve samples with similar color characteristics, indicating that these experts have learned to capture information related to the augmentations.
This qualitative evidence supports the notion that the MMoE architecture induces meaningful divi- sion of roles among experts, with some specializing in robust semantic consistency and others in transformation sensitivity. Such functional diversity contributes to the model's capacity to generalize across tasks and domains.
25

---

Query
1-NN
2-NN
3-NN
4-NN
Expert 1
airplane
airplane
airplane
airplane
airpiane
Expert 2
airpiane
truck
truck
truck
Expert
airpiane
airplane
ship
ship
truck
Expert
airplane
airplane
ship
airpiane
airplane
Expert
airplane
airplane
airplane
airplane
Expert 6
airplane
airplane
airplane
Expert 7
airplane
airplane
Expert 8
airplane
airplane
Figure A.5: k-NN Retrieval on STL10 Test Set. k-NN retrieval results using the output embeddings of all individual experts.
A.9 Expert Specialization in ImageNet100 Pretraining
Invariance
Routing Welght
0.4
Equivariance
Min/Max Ratio
0.25
0.3
0.20
0.2
0.15
0.10
0.1
Expert Index (Reordered)
Figure A.6: Routing Weight Distribution Across Experts. Routing weights averaged over test data in ImageNet100, with experts reordered based their roles.
investigate expert specialization ImageNet100 pretraining following the same procedure as in Figure Figure A.6 illustrates that Expert 1 receives relatively balanced routing weights from both invariant and equivariant branches, whereas Experts 2-6 are predominantly used for invariant learning and Experts 7-8 for equivariant learning. This observation indicates that the experts are well
26

---

specialized for their respective learning objectives. Furthermore, Figure A.7 results on a subset the ImageNet100 set, where we randomly select 10 classes for visualization. Consistent with Figure A.5 each expert retrieves samples aligned with its designated role, confirming that our method effectively promotes expert specialization the ImageNer100 pretraining setting as well.
presents K--N retrieval
Query
1-NN
2-NN
3-NN
4-NN
Expert 1
tripod
tripod
tripod
Expert 2
tripod
harmonica
Doberman
tripod
Expert 3
tripod
tripod
Dutch oven
throne
keyboard
Expert 4
tripod
tripod
lorikeet
tripod
lori
Expert 5
tripod
Dutch even
tripod
bannister
tripod
Expert 6
tripod
tripod
tripod
Doberman
tripod
Expert 7
tripod
tripod
harmonica
reel
reel
Expert 8
tripod
Doberman
throne
Great Dane
Doberman
Figure A.7: k-NN Retrieval ImageNet100 Test Set. k-NN retrieval results using the output embeddings of all individual experts.
B
Experimental Setup
B.1 Datasets
Table B.I summarizes the datasets used in our experiments. Category (a) includes the pretraining datasets, STLIO and ImageNet100, which are exclusively used during the unsupervised pretraining phase. Category (b) covers the datasets used for linear evaluation, with each dataset annotated by the number classes and the number samples in the training, validation, and test splits. For datasets without an official validation split, validation samples are randomly selected from the training set. Category (c) consists of few-shot benchmarks, including the meta-lest split FC100 (36), as well as the full datasets of CUB200 48 and Plant Disease [33). Finally, Category (d) comprises object detection datasets, where we use the trainval split of VOC07+12 [T] for training and the test split for evaluation.
27

---

Table B.1: Dataset Information. Overview of the datasets used in the experiments. This table lists dataset names, the number of classes, and the counts for training, validation, and test samples, along with the evaluation metrics.
Category
Dataset
of classes
Training
Validation
Test
Metric
(a) Pretraining
STLI0
7
10
105,000
ImageNet100
45
1000
126,689
CIFARIO 25
10
45,000
5,000
10,000
Top-1 accuracy
CIFAR100 26
100
45,000
5,000
10,000
Top-1 accuracy
Food [2
101
68.175
7.575
25,250
Top-1 accuracy
MIT67 39
67
4,690
670
1,340
Top-1 accuracy
Pets
2,940
3,669
Mean per-class accuracy
Lineur Evaluation
Flowers
102
1,020
1,020
6.149
Mcan per-class accuracy
Ca 18:
101
2,525
505
5,647
Mean per-class accuracy
Cars 25
196
6,494
1,650
8,041
Top-1 accuracy
Aircraft 32
100
3,334
3,333
3,333
Mean per-class accuracy
DTD (spli
47
1,880
1,880
1,880
Top-1 accuracy
SUN397 (split 1)
49
397
15,880
3,970
19,850
Top-1 accuracy
FC100 [3
20
12,000
Average accuracy
(c) Few-shot
CUB200 48
200
11,780
Average accuracy
Plant Disease
38
54,305
Average accuracy
(d) Object Detection
VOC2007+2012
11
20
16,551
4,952
Averuge precision
B.2
Pretraining Setups
B.2.1
ImageNet100 Pretraining
We pretrain ResNet-50 backbone
19
ImageNet100, a 100-class subset of ImageNet 43
following the dataset splits in [45). The model is trained using the SimCLR framework 3] employing stochastic gradient descent (SGD) for 500 epochs with batch size of 256. A cosine annealing learning rate schedule [29] is used, initialized at 0.03 and without restarts, and a weight decay of 0.0005 is applied. The architecture includes 8 experts {E, each implemented with a hidden dimension of 2048 and an output dimension of 128. Batch normalization [2) is excluded from the final layer of each expert. The equivariant predictor Φr consists of layers, each with hidden dimension of 512. The routers R'n and R are implemented as single-layer MLPs that output 8- 8-dimensional vectors corresponding to the number of experts. followed by softmax activations to produce normalized weights over experts. Pretraining on ImageNet 00 is performed using 4 NVIDIA RTX 4090 GPUs
3-layer MLP
B.2.2 STL10 Pretraining
We pretrain a ResNet-18 18 backbone on STL10 using stochastic gradient descent (SGD). Training is conducted for 200 epochs with batch size of 256. A cosine annealing learning rate schedule without restarts used, with the initial learning rate set to 0.03, except for SimSiam where a learning rate of 0.05 is used. A weight decay of 0.0005 is applied. The routers are implemented in the same manner as those used for ImageNet100 pretraining. For STL10, pretraining is conducted on single NVIDIA RTX 4090 GPU.
SimCLR. We use 3-layer expert architecture with 16 experts (E, = 1,
each with hidden and output dimensions of 512 and 128, respectively. Batch normalization is excluded from the final layer of each expert. The equivariant predictor is 3-layer MLP with hidden dimension of 5 122. temperature parameter of 0.2 is used consistently for both contrastive and equivariant learning objectives.
MoCo. A 3-layer expert architecture with 8 experts is employed, where each expert has a hidden dimension of 512 and an output dimension of 128. Batch normalization is excluded from the final layer The equivariant predictor is single-layer MLP. A temperature parameter of 0.2 is used consistently for both contrastive and equivariant learning objectives.
SimSiam. We use 2-layer expert architecture with 4 experts, each having hidden and output dimensions of 2048. Batch normalization is excluded from the final layer. The equivariant predictor a single-layer MIP with input and output dimensions of 2048, temperature of 0.1 is used for equivariant learning.
28

---

BYOL. A 2- expert architecture is used, consisting of 4 experts with a hidden dimension of 4096 and an output dimension of 256. Batch normalization is excluded from the final layer. The equívariant predictor is 2-layer MIP with hidden dimension of 512. A temperature of 0.1 is used for equivariant learning.
B.3 Evaluation Protocol
Linear Evaluation. We adopt the standard linear evaluation protocol B 17 24], where linear classifier is trained on top of frozen features extracted from center-cropped images of size x (or 96 96 when pretrained STL.10), without any data augmentation. Specifically, each image is first resized so that its shorter side is 224 pixels, followed by a center crop of size 224 x 224. The classifier is optimized using fg-regularized cross-entropy objective with L--FGS. The regularization strength is selected based on validation accuracy from 45 logarithmically spaced values ranging from 10 10-6 to 10 and the final test accuracy is reported using the best model. We set the maximum number of L-BFGS iterations to 5000 and employ warm-start initialization by using the previous solution as the starting point for the next optimization step.
Few-Shot Classification. To evaluate representations in few-shot benchmarks, we perform logistic regression on top of frozen features using x support samples, without any fine-tuning or data augmentation, within each N-way K-shot episode.
Object Detection. We train Faster R-CNN 41 with R50-C4 backbone on the VOC2007+2012
trainval split containing 16551 images. To assess the quality of learned representations, we freeze all convolutional layers from C1 to C4 and train only the region proposal network (RPN) and the object classification head C5. The model is optimized for 24000 iterations with a batch size of 16 using synchronized batch normalization. The learning rate is set to 0.1 initially and decays by factor of 10 at 18000 and 22000 iterations. A linear warmup [15] is applied during the first 1000 iterations with slope 0.333.
R-equivariance and P-equivariance. To evaluate R-equivariance, we train a linear layer predict the embedding of an augmented image from the original image embedding and its corresponding augmentation parameters. The augmentation parameters are first projected to a 32-dimensional vector using a single-layer projector. We then concatenate the projected augmentation parameters with the original image embedding and feed the result into a one-layer predictor to generate the predicted embedding of augmented image. Finally, we compute the cosine similarity between the predicted and ground-truth embedding of augmented image.
For P-equivariance. we train a I-layer predictor to estimate the augmentation parameters from the embeddings of the original and augmented images. Specifically, we concatenate the embeddings of the original and augmented images and feed the resulting vector into the predictor. Finally, we compute the mean-squared error (MSE) between the predicted and ground-truth augmentation parameters.
B.4 Augmentations
In this section, we describe how augmentation parameters are defined based on the transformations used in AugSelf (27) including random crop. horizontal flip, color jitter, grayscale, and Gaussian blur. Each parameter set is defined according to the specific configuration of each transformation. In our method, all parameters are normalized using the empirical mean and standard deviation of each cunsformation-speciii variable before being projected into the embedding space. These normalized parameters are then projected into the same dimensional space as the equivariant embedding through single linear layer.
RandomResizedCrog
The parameter is defined by the center coordinates Heenter and W Weenter of the crop, along with the crop size given by height H and width W. The crop is applied to images resized to 96x96 for STL.10 and 224x224 for ImageNer100.
This transformation is applied with a probability of 0.5. Since the operation is binary, the parameter is defined as either 0 or 1.
BandomHorizzonallPI..
ColorJitter Color jitter includes four parameters: brightness, contrast, saturation, and hue. It is applied with probability of 0.8. The maximum strength set to 0.4 for brightness, contrast, and saturation, and 0.1 for huc. Each parameter is sampled independently from the ranges [0.6, 1.4] for
29

---

brightness, contrast, and saturation, and [-0.1, 0.1] for hue. The transformations are applied in random order rather than a fixed sequence. ColorJitter applied, a default parameter of [1, 1, I, O] is used.
RandomGrayScale. Grayscale conversion applied with probability of 0.2. Similar to flipping, the parameter is binary with values of 0 or 1.
Gaussi anBlur. The parameter consists of both the standard deviation of the blur, which ranges from 0.1 to 2.0, and binary flag indicating whether the transformation was applicd.
30

---

Table B.2: Out-of-Domee Domain Classification. Linear evaluation accuracy (%) of ResNet-50 and ResNer- 18 pretrained on Image/et 100 and STL10, respectively Bold entries indicate the best performance among methods, while underlined entries denote the second best.
Avg. Rank
5,00
3,82
3.18
6:00
1.82
5.73
4.18
3.82
3.36
2.55
68. 19.42221.30 .0
69 09.2440 22
69.72s0.17
$7.51#0-42
70 70.90He0:29
71.23+0.21
45. 45.55 55.55x0.2 27
48 08.92+0.26 28
49. 99.5420.3 33
08.68+4007
51 46:40.005
53.07
Mean
SUN397
52.0200.77
53.00.0.12 12
52.79
42 02.48=0.25
53.86-00.15
54.12+0.10
28.5 28.5820.03
32.51
32.15e0.10
35.78+MMM
34.2520.2
25.50+0.25
35.
69. 69.1740.28
69.31 83
09.33a0.17
62.22
70.5740115
70.80:0.20
43.1220.25 22.022
45.68s1 19
47 57.0020.66 60
50.00200000
58.50005 35
49 49.75+0.92
DTD
Aircraft
48.23
48.65=0.228
49.9) #0.22
36.00=0488
50. 16:16.2225
51.15=0.49
27. 27.17=0.84
31.53
33.03:0.56
33.77
33,90-041
35.61
47. 17.09
47 477488044
48 38.9440.20
29, 10.12220.50
51.01
51.54x0.56
17. 17.3820.31
20.2 20.23
19.9520.69
21 21.99+000
22.74+0.39
22.
ImageeNrr (00 pretrained ResNer-50
Caltech
85.0240.25 20
85 $5.09e0.1 14
86.62a0.08 09
73.1 73.1440.10 16
53-78a0055
37.7640 25
STLIO PrL.l--peerraineedee RexNer-18
66.8 66.85e0.3 22
69.33s0.3 36
71 71.1730.10
61 11-7540.07
74.76.0.35
73.
Flowers
88.37 88.37
89.70=0.3 0..3
90.33s0.02
15.8420.14
07.1722::22
91.45e0.20
60.27e0.66
08.54a0.30
67.0720.29
05.8400.03
72. 22:000225
73.99x0.25
75
16.71
77.24e0.38
77.7860.13
59 19.8980.80 80
78. 58444242
79.27=0.27
46.1 66.1520.34
47.8 47.80s0.11 18
48 10
46 1940.2
50
51.10%0.44
Pets
d6.5721.0
67.51
67.5430.43
56.3920.37
67.9640.42
39.01 39.01=092
41.74x092
42. 12.4921.50
18
50.00065
46.8 66.82
MIT67
63.60a0.32
65.37=0.22
64.4320.15
55.78 $5.7840.10 10
66.8240.06
07.05=0.11
13.75:0.25
38. 11.222
37.4 37.4340.21
43.52:0.01
40. 17:0.20
11,24+0.15
Food
67.9240.19
69 19.6880.16 16
70.22:0.20 2000
58 98.97+0-43
71,4540228
72.31
55. 19.199..5
58.6 58.65$1.92 92
60:0620.68 68
51 11.32+0.01
61 10
64.78+0.35
CIFAR
CIFARIO
87. 07.88:0.27
88.6 08.61=0.12
88. 38.9920.18
82.81=0.19
89,8 100.0
90.09=0.12
33.5620.84
84.03=1.31
85. 45.73=0.43
77. 10=0004
64.20.11
87 777
STAR-MMoB MMoE
STAR MMOE
Method
SimCLR
AugSelf
EquiMod
STAR-SS
SimCLR
AugSolf
EquiMod
STAR-SS
CARE
CARF
31


Reviews:
[{"review_id": "nOY9yi4S3t", "reviewer": "Reviewer_TqVZ", "summary": "This paper introduces a new Invariant-Equivariant SSL paradigm that combines the invariant and equivariant losses through a mixture of experts’ approach. This allows the model to learn both components, but removes the redundant information across the two projector heads. Some of the learned projector heads become specialized for one task while others are shared across the tasks. The authors present experimental results on STL10 and Imagenet100 pretrained models, testing a wide range of out-of-domain classification tasks, and find that their method improves out of domain classification.", "questions": "* I found Figure 1 confusing. Naively, I would think the L/R flip of the objects would also be preserved as an augmentation, but this stands out as something that the invariance-only model is doing better at preserving than the equivariant model.  Could this be explained, or could the figure be made clearer? For instance, what would hypothetical images be that visually contain the augmentation-aware information but do NOT contain semantic structure? \n* The paper should cite and discuss CE-SSL from last years’ NeurIPS, which also combines equivariant and invariant losses (this is the paper that also talks about ImageNet1000 vs ImageNet100 training for equivariant model evaluation) https://proceedings.neurips.cc/paper_files/paper/2024/file/ae28c7bc9414ffd8ffd2b3d454e6ef3e-Paper-Conference.pdf \n* The authors talk about “representational redundancy” and how it is reduced by including a shared projector in the invariant and equivariant projectors, however, to my knowledge, these projectors are typically discarded anyway and the “representation” that is typically used is the backbone (which is shared across the different projectors). How does this learned representation (the backbone representation) relate to the assumption of independence of the projectors? Naively, I am not sure why having completely independent projectors should encourage one to learn a better representation, and as the text is currently written, the more general motivation for benefits of this change are not clear.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* Incorporating MMoE into the projection model to select the “task” by using a gating network seems quite interesting as a way to mitigate the dependence on the lambda term for the two losses. \n* The paper presents results with other invariant learning objectives (Table 6) to show that the results generalize in other SSL frameworks. \n\n**Weaknesses**\n* The paper motivation states that the invariance and equivariant components are encoding “redundant” information, however this statement is not quantified. It would be nice to have seen this quantified in some way for the comparison models, and then show that it is improved for the presented models. The k-NN retrieval is qualitative and not quantitative. \n* The analysis of the “expert specialization” is performed only on the STL10 network, and the results seem somewhat qualitative and difficult to interpret. \n* Although the results for out-of-domain classification seem very reasonable and consistent, suggesting the model is doing something interesting, the margin of differences across models is very small for these OOD experiments (and has been reported to disappear for the equivariant models when trained on ImageNet1000, see below).", "was_revised": true, "final_justification": "During the rebuttal the authors detailed changes that would be made for improving the clarity of the submission and placing it in context of other work. The authors also detailed additional analyses to look at the equivariant and invariant heads of the models, and measured how the learning dynamics changed with the various variants of the loss. \n\nOverall, I think this is a solid submission and appropriate for the conference, but the number of changes required has me a little hesitant to give it a higher score without seeing an updated draft.", "rebuttal": "> W1. The paper motivation states that the invariance and equivariant components are encoding “redundant” information, however this statement is not quantified. It would be nice to have seen this quantified in some way for the comparison models, and then show that it is improved for the presented models. The k-NN retrieval is qualitative and not quantitative.\n\nWe already quantify representational redundancy using the **canonical correlation** between the heads, as shown  in **Figure 4**. This metric captures the linear dependency between representations produced by different heads and provides a direct measure of redundancy across projection spaces. The connection between head-level redundancy and representation is further discussed in our response to **Q3**.\n\nAs reported in **Figure 4(b)**, EquiMod [1], which uses two independent projectors for invariant and equivariant learning, exhibits high Canonical Correlation (0.5830), indicating substantial overlap in learned embeddings. In contrast, our methods, including Single Shared Projection (0.5383) and MMoE Projection with 16 experts (0.3948), show significantly lower Canonical Correlation. This corresponds to a **7.6%** and **32.3%** relative reduction, respectively, reflecting reduced redundancy and enhanced specialization across heads.\n\n> W2. The analysis of the “expert specialization” is performed only on the STL10 network, and the results seem somewhat qualitative and difficult to interpret.\n\nWe further confirmed expert specialization patterns in our analysis on ImageNet100, which also show clear and interpretable expert allocation. We will include the ImageNet100 results in the revised version.\n\n**Figure 3** provides both **quantitative** and **qualitative** evidence of expert specialization. \n\nSpecifically, **Figure 3(a)** visualizes the gating weight distribution across experts, showing that during training, the model naturally assigns distinct roles: 1 shared expert, 5 invariant experts, and 2 equivariant experts among the 8 experts. The Min/Max ratio shows how evenly each expert is used across invariant and equivariant learning tasks. For instance, Expert 1, identified as the shared expert, has a Min/Max ratio of **0.34**, indicating that it is assigned to both tasks relatively evenly. This balanced assignment encourages the expert to capture shared information beneficial to both objectives.\n\n**Figure 3(b)** and **Figure A.3** in Appendix visualize the k-NN retrieval results based on each expert’s embedding. Invariant experts retrieve samples with similar shapes, whereas equivariant experts tend to focus on color similarity. The shared expert, on the other hand, retrieves semantically consistent samples within the same class. These results indicate that different experts attend to distinct transformation sensitivity and semantic attributes, providing clear evidence of expert specialization.\n\n> W3. Although the results for out-of-domain classification seem very reasonable and consistent, suggesting the model is doing something interesting, the margin of differences across models is very small for these OOD experiments (and has been reported to disappear for the equivariant models when trained on ImageNet1000, see below). \n\nAs commonly reported in the SSL and Equivariant SSL literature, performance gains on OOD tasks are often  marginal.\n\nFor instance, SimSiam [2] reports similarly small margins (<1% absolute) over the prior best (e.g., MoCo v2) on VOC/COCO transfer learning tasks, yet these consistent gains are considered indicative of improved generalization.\n\n> Q2. The paper should cite and discuss CE-SSL from last years’ NeurIPS, which also combines equivariant and invariant losses (this is the paper that also talks about ImageNet1000 vs ImageNet100 training for equivariant model evaluation)\n\nWe thank the reviewer for highlighting CE-SSL [3]. CE-SSL introduces a contrastive equivariant SSL learning framework that explicitly adds equivariance-related objective to complement standard invariant SSL losses. Unlike our method, which proposes a new projection module design employing MMoE to reduce redundancy between projectors, CE-SSL [3] addresses the problem at the loss-design level.\n\nWe will revise the manuscript to cite and discuss this work. Regarding the findings on ImageNet1000 vs. ImageNet100, we were unable to include a verification in the main text due to resource limitations, but we will consider exploring whether similar trends hold in our setup if resources allow.\n\n> Q1. I found Figure 1 confusing. Naively, I would think the L/R flip of the objects would also be preserved as an augmentation, but this stands out as something that the invariance-only model is doing better at preserving than the equivariant model. Could this be explained, or could the figure be made clearer? For instance, what would hypothetical images be that visually contain the augmentation-aware information but do NOT contain semantic structure?\n\nWe first note that **Figure 1** illustrates the existence of shared information between representations learned **exclusively** with the invariant or equivariant objective. Both models retrieve samples from the same class, indicating the shared information (e.g., semantic structure), while the *equivariance-only* model additionally preserves augmentation-related information, suggesting that the two objectives rely on a common set of information while capturing complementary aspects.\n\nNonetheless, we agree that the LR flip shown in retrieval results may be confusing. We conjecture that this implies that the LR flip is considered less important when measuring similarities on learned representations. To confirm that the LR flip is also captured in learned representations, we extended the experiment with LR flipped STL10 test dataset, essentially doubling the number of test data.\n\nWe observed that the *equivariance-only* model retrieves samples in a pattern that highlights flip-aware behavior:\n\n- 1-NN: A flipped version of the query image.\n- 2-NN: A sample with a similar pose to the query.\n- 3-NN: A flipped version of the second image.\n- A similar trend is observed in the pair of 4- and 5-NN, and so on.\n\nIn addition, the *equivariance-only* model retrieves a greater number of neighbors that share both consistent color and similar pose compared to the *invariance-only* model.\n\nThis pattern suggests that the model captures flip-related variation in representation. In contrast, the invariance-only model tends to retrieve samples with random pose, highlighting its preference for transformation-agnostic similarity.\n\n> Q3. How does this learned representation (the backbone representation) relate to the assumption of independence of the projectors? Naively, I am not sure why having completely independent projectors should encourage one to learn a better representation, and as the text is currently written, the more general motivation for benefits of this change are not clear.\n\nWe would like to clarify how the independence or separation of the projection heads affects the quality of the learned backbone representation. Our analysis is based on two complementary perspectives:\n\n1. Redundant gradients from independent heads interfere with task-specific learning.\n\nWhen the invariant and equivariant heads are completely independent projectors, both must extract shared factors separately. This leads to redundant supervision on shared components, making the backbone repeatedly optimize for the same features. Consequently:\n\n- Model capacity is used inefficiently.\n- Task-specific gradients become weaker, as redundant gradients dominate.\n- The expressivity of the backbone is limited.\n\nWe empirically confirmed this by computing the cosine similarity between gradients from different heads with respect to the backbone: it decreased from **0.59** (EquiMod [1]) to **0.34** (Single Shared Projection) and **0.30** (MMoE Projection), indicating reduced redundancy and improved specialization.\n\n2. Explicit disentanglement leads to easier optimization and faster convergence.\n\nWhen each head is forced to handle both shared and task-specific information simultaneously, optimization becomes more difficult. By introducing a shared pathway (e.g., a shared projector or MMoE gating), each head can specialize more effectively, leading to better optimization signals. This improves:\n\n- Gradient quality early in training, when learning rate is high.\n- Convergence speed and stability of head parameters.\n\nTo support this, we measured the Frobenius norm of weight changes in projectors every 10 epochs. Our method showed smaller changes compared to EquiMod [1], indicating faster convergence of each head.\n\n| Method        | 20   | 30   | 40   | 50   | 60   | 70   | 80   | 90   | 100  | 110  | 120  | 130  | 140  | 150  | 160  | 170  | 180  | 190  | 200  |\n| :-: | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  |\n| EquiMod       | 7.16 | 5.05 | 4.19 | 3.74 | 3.47 | 3.27 | 3.07 | 2.88 | 2.66 | 2.43 | 2.18 | 1.91 | 1.63 | 1.33 | 1.02 | 0.72 | 0.44 | 0.19 | 0.04 |\n| Single Shared | 6.58 | 4.61 | 3.86 | 3.54 | 3.27 | 3.13 | 2.98 | 2.81 | 2.58 | 2.37 | 2.10 | 1.85 | 1.59 | 1.30 | 1.00 | 0.71 | 0.42 | 0.19 | 0.04 |\n| MMoE          | 5.28 | 3.13 | 2.21 | 1.88 | 1.68 | 1.55 | 1.45 | 1.35 | 1.25 | 1.15 | 1.03 | 0.91 | 0.78 | 0.64 | 0.50 | 0.36 | 0.22 | 0.10 | 0.02 |\n\nThese two analyses collectively demonstrate that our design not only reduces redundant supervision but also facilitates more effective and stable optimization, leading to improved representation.\n\n### Reference\n[1] Devillers and Lefort. EquiMod: An Equivariance Module to Improve Self-Supervised Learning. In *ICLR*. 2023.\n\n[2] Chen et al. Exploring Simple Siamese Representation Learning. In *CVPR*. 2021.\n\n[3] Yerxa et al. Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT. In *NeurIPS*. 2024."}, {"review_id": "mRCo8iaKCA", "reviewer": "Reviewer_inzh", "summary": "This works introduces the use of Mixture of Experts module in place of the standard MLP projector in a SSL framework. The authors present this framework as a mechanism to support equivariant representation learning as the MoE allows for specialisation to invariant and equivariant tasks either independently or as a shared space of the two. They introduce a series of constructive losses to enforce both representation properties, with the equivariant component enforced with a predictor network. The authors report strong invariant performance on a variety of benchmark tasks, and demonstrate robustness in and out of distribution. The ablation studies perform an investigation into the behaviour of the MoE and its impact on equivariant loss.", "questions": "- How does the k-nn retrieval perform on the output of each expert? Do you get specialised transformation encodings?\n- $\\psi$ from figure 2 doesn’t seem to be defined in the text, I assume this is some network to encoder the transformation parameters.\n- Is $\\phi_{T}$ a mlp predictor or a different type of network? This is not defined.\n- Why is there a drop in performance as a factor of training time with SimCLR (Figure 5), from experience with SimCLR this behaviour is unusual and instead the convergence behaviour of equimod is expected. While you state overfitting, the mean accuracy is low. Can you also confirm which dataset this is performed on?", "limitations": "Limitations are addressed, but could be expanded to include task specifics and broader implications.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n+ The investigation of equivariant SSL is timely and important to the computer vision community.\n+ The method presented is sensible, seemingly correct, and simple. It is presented as general purpose method that can be adapted to a variety of data domains, architectures and SSL transformations. \n+ The invariant representation quality is well explored with a wide variety of experimentation, tests for robustness and generalisation.\n+ The investigation into redundancy is a nice addition, and demonstrates well that the experts are able to specialise and improve performance without wasted resource.\n\n**Weaknesses**:\n- The main argument made at the outset of the paper is that using separate projection heads limits the expressivity of the network and limits shared information. However, your findings in Figure 3 (a) somewhat contradict your conjecture. In fact it can be seen that from the expert gating that experts are largely specialised to either invariant or equivariant, with only one being shared, but not balanced. Therefore, one could argue that the improved results are due to the increased capacity for the network to learn specialised equivariant and invariant features given MoE. It would be nice to prove that simply having a projector network for Transformation type in T does not reach the same performance, given that T is accessible during training.\n- The evaluation does not analyse or report any quantitative measures for equivariant representations. Therefore, there is no way to know if the objective function is actually learning equivariant representations. Some further analysis of the learnt representation space is needed before the claim of equivariance can be substantiated. For example, all experimentation evaluates the invariance of the representations, yet does not evaluate explicitly the transformation information in the representations.\n- The novelty of the approach is arguably limited, this method is essentially akin to EquiMod, or SEN but with a MoE as a projector. While I am keen for these findings to be distributed to the wider community, the novelty is narrow. \n- While it is nice to show faster convergence with wall time, it would be beneficial to see the memory consumption, iterations per second, FLOPs as a more quantitative evaluation of computer resources.\n- There are some omitted details in the manuscript that have been outlined in the questions, these is a minor concern that will improve the reproducibility of the work if included.", "was_revised": true, "final_justification": "I have increased my score as the authors have presented some additional results that address my concerns. I however, cannot comfortably give a higher score due to the concerns raised by other reviewers including limited improvement over baseline and large number of changes needed on the full manuscript.", "rebuttal": "> W1. The main argument is that separate projection heads limit expressivity of the network and limits shared, but Figure 3(a) seems to contradict this. One could argue the gains come from increased capacity to learn specialized features. It would be nice to show that a projector per transformation type T would not match the same performance.\n\nThe imbalance observed in **Figure 3(a)**, where only one expert is shared across tasks, does not contradict our conjecture. Our argument is not that experts must be balanced between task, but that having two independent projectors leads to redundant learning of shared information. In MMoE Projection, the observed specialization naturally emerges during optimization as the model allocates experts according to the invariant and equivariant objectives while avoiding repeated extraction of shared factors.\n\nTo address the possibility that the improved results simply stem from increased capacity, we conducted an ablation study using multi-projector baseline with six dedicated projection heads: one for invariance and the others for five transformation types. For fairness, the MMoE variant was also trained with six experts to match model capacity.\n\n| Method | Acc |\n| :-: | :-: |\n| EquiMod | 49.54 |\n| Multi-Projector  | 49.61 |\n| MMoE (6 Experts) | 52.35 |\n\nDespite identical capacity, MMoE Projection shows a relative improvement of **5.5%** out-domain classification performance than the transformation-specific multi-projector baseline (52.35 vs. 49.61). This confirms that the gain arises not merely from increased capacity, but from the structured inductive bias and task-aware routing introduced by MMoE.\n\n> W2. The evaluation does not analyse or report any quantitative measures for equivariant representations. Some further analysis of the learnt representation space is needed before the claim of equivariance can be substantiated.\n\nOur paper provides clear evidence that the learned features preserve augmentation-aware (i.e., equivariant) information, following the evaluation paradigm established in prior work such as AugSelf [3], where color consistency is used as a key qualitative indicator of equivariance. Based on this criterion, we provide two complementary pieces of evidence:\n\n- Figure A.2: Retrieves neighbors with similar color attributes in k-NN retrieval.\n- Table 1: Achieves the highest accuracy on the color-sensitive Flowers102 dataset.\n\nThese results indicate that our method effectively captures transformation-sensitive features.\n\nTo further strengthen this claim, we add explicit quantitative evaluations of transformation-aware information in the learned representations:\n\n1. Transformation Parameter Regression\nWe design a regression task using a frozen backbone and a single linear layer to predict augmentation parameters given the original and augmented images. We evaluate performance using Mean Squared Error (MSE).\n\n- For Crop, we predict scale and aspect ratio.\n- For Color, we predict brightness, contrast, saturation, and hue.\n- When both are applied, we predict all parameters jointly.\n\n| Method   | Crop ↓ | Color ↓ | All ↓  |\n| :-: | :-: | :-: | :-: |\n| SimCLR   | 0.016  | 0.026   | 0.025  |\n| AugSelf  | 0.006  | 0.017   | 0.016  |\n| EquiMod  | **0.005**  | 0.020   | 0.016  |\n| **MMoE (Ours)** | **0.005**  | **0.013**   | **0.013**  |\n\nMMoE achieves the lowest MSE across all transformation types, indicating that its representations more accurately encode transformation-aware information.\n\n2. Transformation Type Classification\nWe also perform a binary classification task to distinguish whether the applied transformation is Crop or Color, using the same frozen representation and a linear head.\n\n| Method   | Acc (%) ↑ |\n| :-: | :-: |\n| SimCLR   | 82.34      |\n| AugSelf  | 91.74      |\n| EquiMod  | 92.79      |\n| **MMoE (Ours)** | **95.76**  |\n\nMMoE attains the highest classification accuracy (**95.76%**), suggesting that it preserves clear discriminative signals between different transformation types in the learned representation.\n\nThese results collectively provide both **quantitative** and **qualitative** support for the claim that our method captures **transformation-aware** features in the learned representations.\n\n> W3. The novelty of the approach is arguably limited, this method is essentially akin to EquiMod, or SEN but with a MoE as a projector. While I am keen for these findings to be distributed to the wider community, the novelty is narrow.\n\nOur method proposes projection module designs that addresses the limitations of prior work such as EquiMod [1] and SEN [2] by explicitly handling redundancy and improving specialization:\n\n- Diagnosing the redundancy problem between two independent heads in Equivariant SSL.\n- Structuring the projection space to separate shared and task-specific components.\n- Leveraging multi-gate routing aligned with invariance and equivariance objectives.\n\nThese designs lead to better expert specialization and improved downstream performance, offering both conceptual and practical advances over prior work.\n\n> W4. While it is nice to show faster convergence with wall time, it would be beneficial to see the memory consumption, iterations per second, FLOPs as a more quantitative evaluation of computer resources.\n\nThank you for the suggestion. We report a detailed comparison of memory usage, FLOPs, and iteration speed in the table below. While MMoE Projection has marginally higher memory consumption and FLOPs than EquiMod, the values remain largely comparable.\n\n| Method      | Iter/s ↓  | Memory (MB) ↓  | GFLOPs ↓  | Acc  ↑  |\n| :-: | :-: | :-: | :-: | :-: |\n| SimCLR      | 17.03 | 3272       | 0.67         | 45.55 |\n| AugSelf     | 16.78     | 3303           | 0.67             | 48.92 |\n| EquiMod     | 12.62     | 4343           | 1.01             | 49.54 |\n| MMoE (Ours) | 9.69      | 4484           | 1.03             | 53.07 |\n\nImportantly, as shown in **Figure 5**, MMoE Projection achieves strong performance even with fewer training iterations, demonstrating that the model is not only effective but also sample-efficient. This confirms that its improved performance stems not from increased computational cost, but from architectural advantages.\n\n> W5. There are some omitted details in the manuscript that have been outlined in the questions, these is a minor concern that will improve the reproducibility of the work if included. (Q2 & Q3 included)\n\nWe are committed to ensuring reproducibility, and to that end, we have included not only the definitions of $\\psi$ and $\\phi_T$, but also all relevant hyperparameter settings and configuration details in **Appendix B**. To make the manuscript more self-contained, we will move key definitions such as those of $\\psi$ and $\\phi_T$ into the main text in the revised version.\n\nSpecifically, as defined in **Appendix B.2**:\n\n- $\\psi$ is a **single linear layer** that takes the augmentation parameter vector as input and expands it to match the output dimensionality of the expert representations. This design follows EquiMod and emphasizes the role of augmentation parameters in guiding the transformation.\n\n- $\\phi_T$ is an **MLP predictor**.\n\n> Q1. How does the k-nn retrieval perform on the output of each expert? Do you get specialised transformation encodings?\n\nOur analysis suggests that the experts are not specialized by transformation type, but rather align more closely with objective-level distinctions (i.e., shared vs. invariant vs. equivariant), as indicated by their k-NN retrieval results (**Figure A.3**).\n\nTo analyze this, we conducted k-NN retrievals within each expert’s embedding space on the STL10 test set. The results show that:\n\n- Expert 1 (shared) retrieves **semantically consistent images**, reflecting general-purpose representation.\n- Experts 2–6 retrieve samples **invariant to changes induced by transformation**, aligning with the invariant objective.\n- Experts 7–8 retrieve samples with **similar color tone**, suggesting sensitivity to color-based transformations.\n\nThis suggests that the experts specialize based on the nature of the objective, rather than being explicitly aligned with specific transformation types.\n\n> Q4. Why is there a drop in performance as a factor of training time with SimCLR (Figure 5), from experience with SimCLR this behaviour is unusual and instead the convergence behaviour of equimod is expected. While you state overfitting, the mean accuracy is low. Can you also confirm which dataset this is performed on?\n\nThe performance drop with longer training in **Figure 5** is observed because we evaluate **out-domain generalization**, not in-domain performance. As shown in **Figure A.1(a)** in Appendix, the in-domain accuracy consistently improves with longer training, while out-domain accuracy degrades. The out-domain results in **Figure 5** are evaluated on a **collection of 11 out-domain datasets** (details in **Appendix B.1**). This indicates that the drop is not a convergence problem but instead reflects a loss of generalization, likely stemming from the lack of explicit inductive bias in SimCLR for modeling augmentation-aware structures. \n\nWe will consider including the corresponding in-domain performance curve (currently in **Figure A.1(a)**) in the main text in the revised version for clarity.\n\n### Reference\n[1] Devillers and Lefort. EquiMod: An Equivariance Module to Improve Self-Supervised Learning. In *ICLR*. 2023.\n\n[2] Park et al. Learning Symmetric Embeddings for Equivariant World Models. In *ICML*. 2022.\n\n[3] Lee et al. Improving Transferability of Representations via Augmentation-Aware Self-Supervision. In *NeurIPS*. 2021."}, {"review_id": "K0GeM8pxhB", "reviewer": "Reviewer_5sNH", "summary": "The paper tackles the interesting problem of representational redundancy in self-supervised learning methods using both invariant and equivariant objective functions. The authors point out that predicting semantic category and predicting augmentation may require similar features. Training projection heads for each task independently may therefore cause similar features to be encoded multiple times in the representation, reducing overall model capacity. The authors propose a mixture-of-experts based gating mechanism in which several shared projection heads are trained simultaneously. Objective-based gating networks produce softmax distributions over the experts to appropriately weight the most fitting projections for each task. This, in addition to an equivariant learning mechanism using a residual connection, yields representations that produce superior empirical results to comparable equivariant learning methods. Additional analysis of the networks trained using their mechanism demonstrate some evidence that redundancy among projection heads is reduced can lead to poorer performance.", "questions": "- [W2] Are all encoders trained for 200 epochs (Section 5, setup)? Figure 5 potentially addresses this, but what about the other experiments in the paper?\n- [W4] Is the argument about representational redundancy concerning the encoder f(x) itself, or the outputs of the projection heads? Couldn’t it be the case that both the invariant and equivariant projection heads could use the same feature in the representation f(x) without that feature being encoded multiple times in the representation? Is the issue then concerning the capacity of the projection heads themselves?\n- [W3] Is there any justification for why the new method can hurt in-domain performance?", "limitations": "Pending the weaknesses mentioned previously, the authors adequately discuss the limitations of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper is well written, easy to follow, and has easy to interpret figures.\n- A large number of experiments are run, combining various tasks, datasets, evaluation types (in-domain, out-of-domain), and baselines.\n- The additional analyses presented in section 5.2 help the reader understand how different kinds of experts naturally emerge using the new mechanism.\n- The practicality of the method is considered, with Figure 5 demonstrating that the new method can produce stronger results in a shorter time.\n\nWeaknesses\n- [W1] The empirical gains from using the method are often quite small.\n- [W2] Not all comparisons may be fair. Are all encoders trained for 200 epochs (Section 5, setup)? If so, this would mean that the author’s method gets a significant computational advantage, since the method “incur(s) a higher cost per epoch”.\n- [W3] It is not clear why the method could actually hurt in-domain performance under some circumstances.\n- [W4] I’m not entirely convinced that redundancy/canonical correlation amongst the projection from the experts implies redundancy in the un-projected representation. See questions.", "was_revised": true, "final_justification": "Authors addressed each of the four listed weaknesses, providing additional experiments.", "rebuttal": ">  [W1] The empirical gains from using the method are often quite small.\n\nWe acknowledge the reviewer’s concern. Our method shows clear improvements across multiple benchmarks, and the performance gains are comparable to those of prior methods. For example, on STL10, we achieve a **7.1%** relative gain over the previous best (EquiMod) while attaining the highest overall accuracy. On ImageNet100, we observe **1.3%** gain over EquiMod.\n\nSimilar trends are observed in the few-shot classification and object detection tasks (see **Table 3-4**), where our method consistently outperforms the strongest baseline. These results collectively demonstrate the effectiveness of our approach across diverse downstream tasks.\n\n> [W2] Are all encoders trained for 200 epochs (Section 5, setup)? Figure 5 potentially addresses this, but what about the other experiments in the paper?\n\nUnless otherwise stated, all encoders are trained for 200 epochs. We chose this schedule because 200 epochs allows the baselines to reach their best performance and yields reasonable performance for our method as well.\n\nImportantly, as shown in **Figure 5**, **our methods already outperforms the strongest baseline even when trained for significantly fewer epochs, and its performance continues to improve with longer training**. \n\nA similar trend is observed in few-shot classification and object detection. In particular, under the few-shot classification setting, our method, trained with only 100 epochs, still outperforms the prior best method (AugSelf) on STL10-pretrained ResNet-18. Specifically, while AugSelf achieves an average accuracy of 53.54% (Table 2), our method reaches **55.27%**, yielding a relative improvement of **3.23%** despite fewer training iterations. This result **highlights the sample efficiency and strong transferability of our approach**.\n\n> [W3] Is there any justification for why the new method can hurt in-domain performance?\n\nWe attribute the slight drop in STL10 in-domain accuracy to the difference in dataset complexity and the effect of increased model capacity.\n\nCompared to EquiMod, our method adopts a **deeper predictor** to better model complex augmentation-induced shifts, which enhances out-domain generalization. However, in low-diversity datasets like STL10, where semantic variation is relatively limited, the added capacity may introduce excessive degrees of freedom, leading to unnecessary feature dispersion and marginal degradation in in-domain accuracy.\n\nTo support this explanation, **Table 5** presents an ablation using a shallower predictor (depth = 1). In this setting, the in-domain performance improves to **87.41%**, confirming that deeper predictor can be a factor in performance drop under simpler data conditions.\n\nImportantly, in the context of SSL, the quality of learned representations is typically assessed by their transferability rather than in-domain performance. From this perspective, our architectural choice prioritizes generalization, as evidenced by consistently strong out-domain classification results.\n\n> [W4] Is the argument about representational redundancy concerning the encoder f(x) itself, or the outputs of the projection heads? Couldn’t it be the case that both the invariant and equivariant projection heads could use the same feature in the representation f(x) without that feature being encoded multiple times in the representation? Is the issue then concerning the capacity of the projection heads themselves?\nWe clarify that the redundancy we discuss arises at the projection head output level. However, this head-level redundancy negatively impacts the encoder.\nThis may be harmless at inference, but during training, independent heads must each disentangle both shard and task-specific factors from entangled backbone features. This is inherently difficult and often results in suboptimal gradients being propagated back into the encoder, degrading representation quality.\n\nWe clarify that the redundancy we discuss arises at the projection head output level. However, this head-level redundancy negatively impacts the encoder.\n\nThis may be harmless at inference, but during training, independent heads must each disentangle both shard and task-specific factors from entangled backbone features. This is inherently difficult and often results in suboptimal gradients being propagated back into the encoder, degrading representation quality.\n\nIn contrast, explicitly separating shared and task-specific pathways (e.g., via a shared projector or MMoE gating) allows heads to specialize more easily. They converge faster and provide more reliable gradients. We confirmed this empirically: the Frobenius norm of differences in projector weights every 10 epochs decreased more quickly with our method than with EquiMod, indicating more stable optimization.\n\n| Method        | 20   | 30   | 40   | 50   | 60   | 70   | 80   | 90   | 100  | 110  | 120  | 130  | 140  | 150  | 160  | 170  | 180  | 190  | 200  |\n| :-: | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  | :-:  |\n| EquiMod       | 7.16 | 5.05 | 4.19 | 3.74 | 3.47 | 3.27 | 3.07 | 2.88 | 2.66 | 2.43 | 2.18 | 1.91 | 1.63 | 1.33 | 1.02 | 0.72 | 0.44 | 0.19 | 0.04 |\n| Single Shared | 6.58 | 4.61 | 3.86 | 3.54 | 3.27 | 3.13 | 2.98 | 2.81 | 2.58 | 2.37 | 2.10 | 1.85 | 1.59 | 1.30 | 1.00 | 0.71 | 0.42 | 0.19 | 0.04 |\n| MMoE          | 5.28 | 3.13 | 2.21 | 1.88 | 1.68 | 1.55 | 1.45 | 1.35 | 1.25 | 1.15 | 1.03 | 0.91 | 0.78 | 0.64 | 0.50 | 0.36 | 0.22 | 0.10 | 0.02 |\n\nFinally, this issue cannot be explained by projection head capacity alone. Even when we increased the number of projectors from 2 to 8 while reducing the hidden dimension (512 →180) to keep the parameter count approximately the same, applying MMoE Projection still improved the performance from **49.54** to **50.34**. This demonstrates that the performance gain comes from the inductive bias introduced by MMoE, not simply from increasing capacity."}, {"review_id": "voAllLSMoE", "reviewer": "Reviewer_tSPH", "summary": "This work addresses jointly learning invariant and equivariant representations in self-supervised learning. The authors argue that treating them as fully separate leads to redundancy. Instead of using independent projection heads, they introduce a task-aware routing mechanism that dynamically routes features through shared and task-specific paths. This allows the model to better capture both shared and unique aspects of each objective, improving generalization and reducing representational overlap. Their method shows performance improvements over classification, few-shot learning, and object detection tasks.", "questions": "- What would be the point of a separate equivariant learning mechanism? Shouldn’t that be part of the experts’ job? As in, the model should assign an expert to learn the observed shifts in the dataset as its role?\n\n- Since a topic of focus is the separation of heads by adding a shared projector, would you expect forcing disentanglement between representations of separate heads to cause minimal degradation due to the results in figure 3.a?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The experiments are comprehensive.\n- The experiments cover the claims made in the introduction.\n- Ablation studies show the effectiveness of the introduced components over previous work.\n\nWeaknesses:\n- The introduction section needs to be revised to clearly explain what this paper refers to as invariant and equivariant concepts. I initially assumed that based on figure 1 and L21, the invariance is with respect to a task but there is clear explanation and no buildup or flow in this section. One paragraph is not enough to explain your motivation and intuition before diving to specific examples and details in paragraph 2.\n\n- Following from the previous point, the second paragraph starts with a description of another work that is too technical for the introduction section. Same as before, the concept of invariance and equivariance are being referred to at the same time as invariant and equivariant tasks alongside features. So the introduction needs an overhaul to make it clear.\n\n- About statement in L37, “This not only results in inefficient use of model capacity, but also hinders each head’s ability to specialize in task-specific features.” Say each head learns both shared knowledge and task-specific knowledge. In that case, regardless of redundancy, you still have task-specific features. I’m assuming you’re making a point about heads not being able to only attend to task-specific knowledge and having to share their representational power with shared knowledge but L37 does not explain that clearly and it needs a revision.", "was_revised": true, "final_justification": "The proposed revisions to the text the authors have given address everything I was concerned about. The three ablation studies in response to Q1 and Q2 address my concerns about whether there are still some shared concepts left that the explicit mechanisms cannot get rid of. And the justification in the first part of the answer to Q1 appropriately explains why the proposed structure is necessary.\n\nI disagree with other reviewers about the magnitude of performance improvement. This improvement, despite being slightly small in some cases and significant in others, appears to come from the explicit formulation of the proposed method. Considering the extensive ablations that show the efficacy of each part of the proposed method, I’ve raised my score.", "rebuttal": "> W1 & W2. The introduction section needs to be revised to clearly explain what this paper refers to as invariant and equivariant concepts. I initially assumed that based on figure 1 and L21, the invariance is with respect to a task but there is clear explanation and no buildup or flow in this section. One paragraph is not enough to explain your motivation and intuition before diving to specific examples and details in paragraph 2. Following from the previous point, the second paragraph starts with a description of another work that is too technical for the introduction section. Same as before, the concept of invariance and equivariance are being referred to at the same time as invariant and equivariant tasks alongside features. So the introduction needs an overhaul to make it clear.\n\nRegarding the reviewer’s comment on the conceptual clarity of invariant and equivariant learning, we will clarify these concepts in the context of our framework as follows:\n\n- Invariant learning (or task) aims to extract features that remain unchanged under input transformations. It filters out transformation-specific signals while preserving semantic information.\n- Equivariant learning (or task), in contrast, seeks to retain transformation-induced variations in the representation. The goal is to ensure that the representation changes in a structured and predictable manner with respect to the transformation applied.\n\nWe clarify that the terms “invariant task” and “equivariant task” refer to **pretext tasks** that induce these respective properties in the learned representations, not to tasks that are themselves invariant or equivariant.  This clarification serves as a conceptual foundation that leads naturally into the second paragraph, where we introduce our modeling strategy and discuss baseline comparisons. We believe this revision significantly improves the clarity and coherence of the introduction.\n\n> W3. About statement in L37, “This not only results in inefficient use of model capacity, but also hinders each head’s ability to specialize in task-specific features.” Say each head learns both shared knowledge and task-specific knowledge. In that case, regardless of redundancy, you still have task-specific features. I’m assuming you’re making a point about heads not being able to only attend to task-specific knowledge and having to share their representational power with shared knowledge but L37 does not explain that clearly and it needs a revision.\n\nWe thank the reviewer for the accurate interpretation. We will revise the sentence to better reflect that redundancy reduces the effective capacity each head can dedicate to truly task-specific signals. As the reviewer pointed out, even when using two independent heads, each can still capture both task-specific features and shared knowledge. The core issue, however, lies in the redundancy: both heads must re-encode shared information, which leads to inefficient use of the representational space. Our intention in L37 was to emphasize that this redundancy reduces the effective capacity each head can dedicate to truly task-specific signals. We will revise the sentence to better reflect this point.\n\n> Q1. What would be the point of a separate equivariant learning mechanism? Shouldn’t that be part of the experts’ job? As in, the model should assign an expert to learn the observed shifts in the dataset as its role?\n\nWe understand the reviewer’s question as asking whether the explicit equivariant learning mechanism is necessary, given that the MoE gating could in principle assign experts to learn augmentation-induced shifts as part of their specialization. Based on this understanding, we provide the following response.\n\n1. Gating Alone Does Not Induce Equivariance\n\nWe agree that the mixture-of-experts (MoE) mechanism encourages each expert to focus on different portions of the data distribution. However, gating only selects which expert to use; it does not constrain the functional role of each expert. In particular, gating does not ensure that the selected expert encodes augmentation-specific transformations in a consistent and structured manner without equivariant objective.\n\n2. Why An Explicit Equivariant Objective Is Necessary\n\nThe invariant loss encourages the model to remove augmentation-induced variations to enforce semantic consistency. This removes the signals, such as color and pose, needed to learn augmentation-aware representations. In contrast, the equivariant objective provides an **explicit inductive bias** that encourages experts to preserve these signals in the embedding space by modeling transformation-consistent feature shifts.\n\n3. Gating Without Equivariant Loss\n\nTo isolate the effect of the equivariant objective, we performed an ablation where we removed the equivariant loss while keeping the MoE architecture with one gating network. The result shows that gating alone is insufficient for learning robust, generalizable features:\n\n| Method | In-domain | Out-domain | \n| :-: | :-: | :-: |\n|  MMoE (Ours) | 86.74 | 53.07 |\n| Invariance-only MoE | 85.93 | 49.27 |\n\nDespite identical architecture (except for gating network) and capacity, the performance drops significantly without the equivariant loss. This suggests that without an equivariant loss, the experts do not specialize in modeling augmentation-specific information, even when gating is present.\n\nWe hope this addresses the reviewer’s question. If we misunderstood the question, we kindly ask the reviewer to rephrase or elaborate so we may provide a more precise response.\n\n> Q2. Since a topic of focus is the separation of heads by adding a shared projector, would you expect forcing disentanglement between representations of separate heads to cause minimal degradation due to the results in figure 3.a?\n\nThe observed expert-wise separation in **Figure 3(a)** suggests that additional constraints enforcing disentanglement across heads would have limited effect, as our method already induces clear task-specific specialization via the MMoE gating mechanism.\n\nTo verify this, we conducted two complementary experiments:\n\n\n- Orthogonality regularization: We explicitly enforced orthogonality across expert outputs and observed only a small drop in out-domain classification accuracy from **53.07** to **52.75**, confirming that he additional orthogonality constraint introduces only minor adjustments to the parameters.\n- Hard assignment: We replaced soft gating with hard expert selection (1 shared + 5 invariant + 2 equivariant), matching the effective expert allocation observed in Figure 3(a). The resulting model showed only a minor performance drop from **52.37 (MMoE with 8 experts)** to **52.27**, indicating that the learned gating already approximates hard selection.\n\nThese results support the conclusion that our MMoE structure effectively promotes disentangled, task-specific representation learning, even without explicit constraints.\n\nWe hope our response aligns with the reviewer’s intent, and we would welcome any clarification if we have misunderstood the question."}]

Author rebuttal:

Meta-review: This paper studies the problem of representational self-supervised learning based on invariant and equivariant objective functions, for which the work introduces a task-aware routing method that is based on the Mixture of Experts framework. The reviewers have all expressed support for the paper, with both Reviewer TqVZ and inzh noting the timeliness and importance of the proposed MMoE approach. I therefore recommend acceptance, though I strongly encourage the authors to incorporate the suggestions that arose during the discussion period, since as pointed out by the reviewers, these greatly strengthen and clarify the results of the work.
Decision: Accept (poster)