Capsule Network Projectors are Equivariant and Invariant Learners
Authors: Miles Everett
Aiden Durrant
Mingjun Zhong
Georgios Leontidis
Venue: Accepted by TMLR ()

Abstract: Learning invariant representations has been the longstanding approach to self-supervised learning. However, recently progress has been made in preserving equivariant properties in representations, yet do so with highly prescribed architectures. In this work, we propose an
invariant-equivariant self-supervised architecture that employs Capsule Networks (CapsNets), which have been shown to capture equivariance with respect to novel viewpoints. We demonstrate that the use of CapsNets in equivariant self-supervised architectures achieves improved downstream performance on equivariant tasks with higher efficiency and fewer network parameters. To accommodate the architectural changes of CapsNets, we introduce a new objective function based on entropy minimisation. This approach, which we name CapsIE (Capsule Invariant Equivariant Network), achieves state-of-the-art performance on the equivariant rotation tasks on the 3DIEBench dataset compared to prior equivariant SSL methods, while performing competitively against supervised counterparts. Our results demonstrate the ability of CapsNets to learn complex and generalised representations for large-scale, multi-task datasets compared to previous CapsNet benchmarks.

Published in Transactions on Machine Learning Research (10/2025)
Capsule Network Projectors are Equivariant and Invariant Learners
Miles Everett
miles.euerett@aabdi
Deportment of Computing Science University of Aberdeen, UK
Aiden Durrant
aidem..doraantlaaadd.cccc.
Department of Computing Science University of Abendeen, UK
Mingjun Zhong
maingion..ceenng@@abdd..ccc
Department of Computing Science University of Aberdeen, UK
Georgios Leontidis
gerrgios.lontiii@@oddd.cc
Intendisciplinary Institute Department of Compuling Science University of Aberdeen, UK
Reviewed OpenReview: https: openreview. net/ forum? id= 7dWC0ggggg
Abstract
Learning invariant representations has been the long-standing approach to self-supervised learning. However, recently progress has been made in preserving equivariant properties in reprosontations, yet do so with highly prescribed architectures. In this work, we propose an invariant-equivariant self-supervised architecture that employs Capsule Networks (Caps.Nets), which have been shown to capture equivariance with respect to novel viewpoints. We demonstrate that the use of CapsNets in equivariant self-supervised architectures achieves improved downstream performance on equivariant tasks with higher efficiency and fewer network parameters. To accommodate the architectural changes of CapsNets, introduce new objective function based entropy minimisation. This approach, which name CapsIE (Capsule Invariant Equivariant Network), achieves state-of-the-art performance on the equivariant rotation tasks on the 3DIEBench dataset compared to prior equivariant SSL methods, while performing competitively against supervised counterparts. Our results demonstrate the ability CapsNets to learn complex and generalised representations for large-scale, multi-task datasets compared previous CapsNet benchmarks. Code available at https:/ htttps://githbb.co catee/ubb..om/Abbbeeeeeli.. CCpslE.
1 Introduction
Equivariance and invariance have become increasingly important properties and objectives of deep learning in recent times, with precedence being largely placed on the latter. The task of invariance, iee. being able to classify a specific object regardless of the camera perspective augmentation applied, has driven progress in modern self-superviser learning approaches, specifically those that follow joint embedding archilecture Assran et al., 2022 Bardes et al., 2022 Chen et al.) 2020). Equivariance, on the other hand, is the task of capturing embeddings which equally reflect the translations applied to the input space in the latent space. Equivariance thus has become an important property to capture to enable the learning of high-quality representations in the real world, where transformations such viewpoint essential.

---

Published in Transactions on Machine Learning Research (10/2025)
Fact
(Fact)
Routing Process
fo
Capaliet
Eprant
Arrryy Caooyyagea Coeaa
Aper
px
Pea
Crim
H( Saacr a
W.
4j
Pact
Lest
crl Coupiing Cocficients
H( act
x
fo
CayaNic,
2poso
Voses
(a) Schematic overview of the CapsIE architecture.
(b) Generalised visualisation of the CapsNet projector.
Figure 1: Left: Schematic overview of the proposed CapsIE architecture. Representations are fed into a CapsNet projector, and the output embeddings Zett and pose correspond to invariant and equivariant embeddings, respectively. Right: Generalised view of a Capsule projection head. CNN feature maps are transformed via the primary capsules into poses Ui, represented by cylinders, and activations a;, represented by circles. Poses are transformed to votes, which represent a lower-level capsule's prediction for each of the higher-level capsules. The routing process then determines how well these votes match the concept represented by the upper-level capsule, thereby creating the coupling coefficients. Coupling coefficients inform uj and aj, the output of the capsule projector head.
Self-supervised learning owes its success to invariant objectives, where all recent progress, whether that is by contrastive (Chen et al. 2020). information-maxiiistttoo (Bardes et al.) 2022 Zbontar et al.) 2021). or clustcring-basct methods (Caron et al., 2021 Assran et al. 2022) rely on ensuring invariance in their tasks; however, when employing the representations in alternative tasks, preserving information is essential to improve generalisation. To maintain properties of the transformation, one can predict the augmentations applied [Dangovski et al., 2022. Lee et al. 2021), yet this is typically not considered truly equivariant, given that the mapping of translormatiin tions is not represented in the latent. space. Methods that employ such a prediction methodology are typically considered equivariant, as the transformation in the input space is preserved in the latent space. Here, prediction networks are employed to reconstruct the view prior to transformation (Winter et al. 2022). learn symmetric representations (Park et al., 2022), or predict the latent representation of the transformed view from the representation of the original view given the transformation parameters (Garrido ct al.
representations under augmentation. This setting ensures performance in classification-
2023
The above methods, although promising, enforce equivariance via objective functions on vector representatious, yet these methods fail to employ architectural approaches that have been shown to be capable of better capturing these properties. Capsule Networks (CapsNets)), which utilise a process called routing (Sabour
2017 Hinton et al., 2018: Everell et al.] 2023 Hahn el al. 2019 De Sousa Ribeiro el al., 2020 Liu ee al. et al. 2024], are one such architecture, exhibiting desirable properties that other state-of-the-art (SOTA) architectures, such as Vision Transformers (ViTs) and CNNs, lack. Specifically, CapsNets have demonstratod
natural ability to possess strong viewpoint equivariance and viewpoint invariance properties. They achieve this through their ability to capture equivariance with respect to viewpoints in neural activities, and invariance in the weights. In addition, viewpoint changes have nonlinear effects on pixels but linear effects on object relationships (De Sousa Ribeiro et al.) 2020 Hinton et al., 2018). Idcally, these properties could lead to the development of more sample-efficient models that can exploit robust representations to better generalise to unscen cases and new samples.
However, common argument is that CapsNets have only shown these properties toy examples such the smalINORB dataset (LeCun et al. 2004), which many would consider irrelevant for modern architectures. Despite this, small CapsNets outperform much larger CNN and VIT counterparts (Everett et al. 2023). In this work, we propose a novel CapsNet formulation and corresponding objective function, achieving SOTA on equivariant viewpoint rotation tasks on the 3DIEBench dataset (Garrido et al.) 2023). which has been created to specifically benchmark equivariant and invariant properties of deep lcarning models. We prove that CapsNets retain their desirable properties on this dataset, which considcrably more difficult than what has been previously achieved with CapsNets, while also establishing new SOTA for this dataset.
2

---

Published in Transactions on Machine Learning Research (10/2025)
To summarise, contributions are:
We propose novel architecture (Figure based on Capsule Network projection head that utilises the key assumptions of capsule architectures to learn equivariant and invariant representations which does not require the explicit split of representations.
We design new objective function to accommodate the employment of a CapsNet projector, enforcing invariance through entropy minimisation.
We show state-of-the-arl performance 3DIEBench classification for equivariance benchmark tasks from our baseed architecture.
2 Problem Statement
Typically, self-supervised learning maximises the sim- ilarity between embeddings of two augmented views of an image such that they are invariant to augmen- tations, and instead capture semantically meaningful information of the original image. Views x and x' are each transformed from an image Rexxxx sampled from dataset D by image augmentations T,T' T sampled from set of augmentations T. Embeddings are obtained by feeding each view through an encoder fe, where the output represen- tations y. are fed through a projection head h& to produce embeddings z, whose similarity max- imised. However, it is detrimental in many settings that f is invariant to all transformations; instead, in this work, are focused ensuring that f is equivariant to viewpoint transformations. To train for and evaluate such properties, we base our study on the challenging 3DIEBench dataset and the cor- responding problem definition presented Garrido al. )023).
Invariant Objective
Minimise Similanty
Maximise Similarit
px(s) m1
Equivariant Objective
Rotalion
dt given gi
Figure 2: Visual depiction of the problem state- ment. Two images are represented by subscripts 0, while view under transformation g given by '. Ar- rows represent the construction of emboddings from an encoder network. Top the invuriance objective is to maximise the similarity between embeddings of views originating from the same image. Bottom the equivariance objective aim to learn the transformations Px(g) x applied to x.
As defined in Garrido et al. (2023) in which the dataset. was proposed the goal (visually depicted in Figure is to learn an encoder network f and predictor network P to construct representations that are equivariant to viewpoint transformations when the transformation group action p on the input is not known, but the group elements g that parameterise the transformations are known. Further details these transformations and the benchmark 3DIEBench dataset and the group theory defining the problem statement. are given in Scctions
A.1 and A.2 respectively.
3 Method
3.1
Architecture
Our method, which name CapsIE (Capsule Network Invariant Equivariant). follows the general joint embedding architecture previously described in Section 2 and extends those proposed by VICReg [ardes ct al. (2022 and Split Invariant Equivariant (SIE) Garrido et al. (2023) methods. Like previous methods, we employ a ResNet-18 He et al. (2016) encoder as the core feature extractor fo of our network. Yet, unlike SIE Garrido et al. (2023). we do not split the representations, and therefore do not require the use of separate invariant and cquivariant projection heads. Instcad, we employ a single CapsNet (described in Scction (3.2) which takes as input the full representation of the encoder in place of the multi-layer perceptron (MLP) the projection head h Given the architectural design of CapsNets, our projection head outputs both
3

---

Published in Transactions on Machine Learning Research (10/2025)
activation scalar, representing how active the capsule is, and 4 x 4 pose for each capsule. A simplified visual representation can be seen in Figure
To align with the above problem statement and Equation S we aim to simultaneously learn invariant and equivariant representations by optimising our network f with respect to the output activations and poses. In this case, we consider the vector of activations to capture the existence of semantic concepts/objent of the imput; thus, the invariant information is preserved by the transformation. The pose, on the other hand, is designed to encode positional information related to each corresponding capsule (De Sousa Ribeiro et al., 2024) (i.e. semantic concept), therefore, it contains equivariant information that was changed by the transformation. Akin to the STE method, we therefore consider two embedding vectors for each image view, Fact and 2pose, which correspond to the capsule activation and pose, and invariant and equivariant components, respectively.
To enforce our network learn oquivariant properties we utilisc prediction network Pv.g which takes as input the transformation ! and &pose to predict posee> and hence learn (g)) In our setting, g E I& corresponds to Tait-Bryan angles of the rotation applied 6 For prediction and the subsequent evaluation, quarterions are employed in place of the Tait-Bryan angles. In this work, we employ the hypernetwork approach proposed by (Garrido et al. 2023). which uses linear projector that takes as input the transformation parameters g to parameterise an MLP predictor. Such a network avoids the case where the transformation parameters are ignored and the predictor provides invariant solutions. We present more details of the predictor network in the appendix, and visually depict the full CapsIE architecture Figure 1
3.2 Capsule Network Projector
Image
Activation
Pose
Capsule Viewpoint Equivariance CapsNets are de- signed to handle spatial hierarchies and recognise objects regardless of their orientation or location, achieving equiv- ariance through their structure (Ribeiro et al. 2020). A capsule is a group of neurons vector- based represen- tations - representing instantiation parameters such as position, orientation, and size. Before any routing process begins, lower-level capsule poses u; are transformed n upper-level capsule poses Ujjz which align with concepts represented by higher-level capsules, preserving spatial relationships and hierarchical information. It is then de- termined through the routing process how well these trans- formed poses correspond with the concept represented by the upper-level capsule. CapsNets, unlike convolution, excel in achieving viewpoint invariance and viewpoint equivariance as they can capture equivariance with re- spect to viewpoints in neural activities, and invariance in the network's weights (De Sousa Ribeiro et al. 2024). Consequently, capsule routing aims to detect objects by looking for agreement between their parts, thereby per- forming equivariant inference.
0.2-0.7-0.6
0.2
0.9011
0
0.3 0.4 0.9
p(y)))
08.0400
0.9.4 0.20
0.3 0.8
p(ylat) ae)
199.3
0.9 0.9.0.10.
0.4
Figure 3: Simplified visual representation of CapsNet outputs. Voctor of activations outputs the probability of each capsule being activated, whereas the pose matrix corresponds to the object pose in relation to the frame.
Self Routing Capsule We use the Self Routing Cap-
sNet. (SRCaps) (Hahn al., 2019) based on the efficiency of its non-iterative routing algorithm. We consider the trade-off of small amount of classification accuracy to be acceptable when comparing the performance of SRCaps to other capsule architectures, which require significantly more resources to train. Based on the size of the 3DIEBench dataset, these other routing algorithms would be unsuitable.
SRCaps calculates the coupling coefficients between each capsule in lower layer i with each capsule upper layer j to produce the coupling coefficients Cjj- does so by using a learnable routing matrix Wroute multiplied with the lower capsule pose vector u; mimicking a single-layer perceptron produce routing coefficients by which when passed through softmax function produce coupling coefficients Gy- Additionally,
4

---

Published in Transactions on Machine Learning Research (10/2025)
determine the activation of upper-level capsules aj by first multiplying og by Cij to create votes and then dividing this by to create weighted votes.
Eien, Gi
Cjj (e
0g Eien, aj
(1)
The output pose of a capsule layer is calculated using a learnable weight matrix Wrose which when multiplied with u, provides a capsule pose each lower-level capsule for each upper-layer capsule ie UgJa. Following the same procedure as the activations, Uj is the weighted sum of these poses by ag.
Eign, Ciz0,úyi
Ug}s W Us Uj Eien, Cjya
(2)
3.3 Objective Functions
Invariant Criterion. To train our aforementioned architecture, we first introduce an invariant objective the cross entropy between activation probability vectors, H[ Zact) where Z refers to the matrix embeddings over batch. The aim is to enforce embedding probability pairs originating from the same image to be matched. To avoid trivial solutions and collapse to a single capsule, we employ the mean entropy maximisation regularisation (Assran et al. 2021 2022 on the same activation probability vectors to encourage the model to utilise the full set of capsules over a batch. This regularisation maximises the entropy of the mean probabilities HZaatt and H (Zat)) where Zact = E Zact and B the batch size.
Equivariant Criterion. As previously stated in Section 2 our goal is learn the predictor Pe.s to model (g) to enforce equivariant representations. This is achieved by minimising L2 (Euclidean Distance) objective between the output vector of the predictor Pa.o (Zpoee given translation parameters g and equivariant representation Zpose, and the augmented view's equivariant representation vector 2, To avoid collapse and improve training stability we also regularise the output of Pr:. (Zpoce) ) by ensuring the variance of the predicted equivariance representation is 1 to avoid collapse. Whereas, SIE (Garrido et al.) 2023 finds this to be an optional but recommended component, we found in practice, without such regularisation the predictor would consistently collapse to trivial solutions.
As with the activation vector, we employ variance variance-covariann covariance regularisation the pose prevent the representations from collapsing into trivial solutions. The variance objective V ensures that all dimensions d in the embedding vector are equally utilised while the covariance objective C decorrelates the dimensions to reduce redundancy across dimensions. The regularisation for equivariant vectors freg is given by
Greg = Ac C(Z) + Av V(Z),
where
(3)
1 d
-( - Σ Cou(Z) and V(Z) 7 Σ max (0,1-V - (0.1-VVar(z.)) (Vr(Z.s))
(4)
itj
The final objective function is given by the weighted sum of the individual objectives:
C(Zect, Zuct Zpose Zpooce Ainy (Zacr Zacct) (H (Z + (Zact)) +
(5)
N
1
Aequi
Ipppo (Zi.pna)) 2% Z..pose 11 +
(6)
N
1=1
Lreg Zpose) + Lreg (ppnac) + Av V (psig. (Zi
(7)
4 Experimentation
4.1 Training Protocol
To directly compare with prior works employing the 3DIEBench dataset, we follow an identical training protocol, defined in Garrido et al. 2023). All methods employ ResNet-18 encoder network (fo). For the
5

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Evaluation of invariant properties via downstream classification task. Representations are learnt under the invariance and rotation equivariant objective. We evaluate both the representations and the intermediate embeddings of the projection head under varying numbers of capsules. FLOPs and Parameters correspond to computation during training, '' refers to non-compatible experiments. For non-capsule models, 'All', 'Iuv., and 'Equi!' refer to the full vector representation, the 'left' half representation optimised under the invariant objective, and the 'right" half representation optimised under the equivariant objective, respectively. For capsule networks, 'Inv', and 'Equi!' refer to the activation vector and pose matrices, respectively.
Computational Load
Embedding Dims
Classification (Top-1
Method
Parameters
# FLOPs
Inv.
Equi.
All
Inv.
Equi.
Supervised
18
11.2M
3.09G
87.47
SR- ER-Caps- 16
11.0M
3.16G
73.85
SR- ER-Caps- 32
13.0M
4.27G
59.70
SR-Caps- 64
18.7M
8.22G
69.45
Encoder Representation
SIE
20.1M
13.07G
512
512
82.94
82.08
80.32
CapsIE 16
12.7M
3.49G
16
256
76.51
CapslE - -2
14.7M
4.57G
32
512
79.14
CapsTE: 64
20.4M
8.69G
64
1024
79.60
Projector 1st Intermediate Embedding
SIE
20.1M
13.07G
512
512
80.53
77.64
CapsIE 16
12.7M
3.49G
16
256
74.90
CapslE 32
14.7M
4.57G
32
512
78.60
CapsIE- 64
20.4M
8.69G
64
1024
79.24
projection head (ha); we compare various hyperparamelleissaioonn. which we describe in the following sections. For primary benchmarking we train our model for 2000 epochs using the Adam Kingma & Ba (2014) optimiser with default scttings, a fixed Icarning rate of 1c-3 and batch size of 1024. For ablations and sensitivity analyses we train for 500 epochs and employ a batch size of 512, with other settings remaining unchanged. We have found in practice that 500 epochs presents a strong correlation with performance. For all evaluations, pre-training was done with the equivariant criterion optimising for viewpoint rotation transformations. Full details on these transformation groups and the criteria are given in prior sections. By default the objective function weighting are as follows, Ainv = 0.1, Auqui = 5, xv = 10, Ao = =. Fach self-supervised 2000-epoch pretraining run took approximately 22 hours using three Nvidia A100 80GB GPUs for the 32 capule model, whereas the 64 capsule models, and required approximately comparison SIE training took approximately 26 hours using three Nvidia A100 80GB GPUs. All evaluation tasks are completed and 3 hours for classification. Given the computational overheads involved, all results are presented single run where the seed is set to "2224" with exception to Table. f where the mean and standard deviation of 5 seeds are reported.
hours using six Nvidia A100 S0GB GPUs. For
single Nvidia A100 80GB GPU and take approximately hours for angle prediction,
4.2 Downstream Evaluation
To evaluate the quality of representations learnt under the invariant and viewpoint rotation equivariant self-supervised criterion, we use the standard benchmark approach of learning downstream task-specific networks with frozen representations as input. In our case, we evaluate the representations in three distinct tasks to evaluate both invariant and equivariant properties. We use a linear evaluation training protocol of the frozen representations. Further details of the evaluation protocol are given in the appendix A.3.2
A.3.22
Invariant Evaluation. To evaluate invariant properties of the representation, we train a classifier on either the frozen representations output from the encoder network or the intermediate embeddings of the capsule

---

Published in Transactions on Machine Learning Research (10/2025)
network projector. Given our advocacy for CapsNets, we evaluate using both the standard linear classification and capsule layer whose number of output capsules is set to the number of classes. All methods are trained for 300 epochs by cross entropy.
Equivariant Evaluation. Evaluating equivariant properties is achieved through a rotation prediction task in which three layer ML. trained to predict the quaternions defining the rotation between two views of the same object. We train for 300 epochs using MSE loss. Similar rotation prediction, we evaluate the represcntation's equivariant properties by regressing the colour hue of an object view. We train single linear layer for 50 epochs using MSE. For evaluating the performance of predicting equivariance, use the metric Rº2 = E (Ne- (w)-0.)²
Σ ()--))² where {ui} are the ground truth (target values of the equivariance), y is the mean value of these targets, and {or} are the predictions. Higher R² indicates the model has better fit for predicting equivariant transformations.
Table 2: Evaluation of equivariant properties via downstream rotation prediction (left) and colour prediction (right) tasks. Representations are learnt under the invariance and rotation equivariant objective, we evaluate both the representations and the intermediate embeddings of the projection head under varying number of capsules. '' ''refers lo non-compalible experrments. For non-capsule models, 'All', 'Inv!, and 'Equi. refer to the full vector representation, the "left' half representation optimised under the invariant objective, and the "right" half reprosentation optimisod under the equivariant objective, respectively. For capsule networks, 'Inv', and 'Equi.' refer to the activation vector and pose matrices, respectively.
Rotation Prediction (R²)
Colour Prediction (R²)
Method
All
Inv.
Equi.
All
Inv.
Equi
Supervised
ResNet-18
0.76
0.99
SR-Caps- 16
0.83
0.99
SR-Caps 32
0.84
0.99
SR-Caps 64
0.80
0.99
Encoder Representation
SIE CapsIE 16
0.73
0.23
0.73
0.07
0.05
0.02
0.68
0.02
CapsIE 32
0.74
0.04
CapslE 64
0.72
0.01
Projector 1st Intermediate Embedding
SIE CapsIE 16
0.38
0.58
0.45
0.09
0.64
-0.01
CapsIE 32
0.71
-0.04
CapsIE 64
0.67
-0.09
The performance of CapsIE for both invariant and equivariant benchmark tasks is given in Tables andd 2 respectively. We evaluate both the representations produced by the ResNet-118 encoder and the intermediate embeddings of the capsule layer projection head given different values for the number of capsules. We observe that across all models that the invariant properties captured within the representations marginally suffer compared to the MLP projector of SIE. This observation expected given the significantly reduced number of embeddings employed in the invariant criterion compared to SIE. However, the evaluation of equivariant properties captured by the representations demonstrates that the use of capsule projector in place of an MLP can load to marginally improved performance in rotation prediction (f 0.01 I2²) advancing the prior state-of-the-ar whist also approaching the supervised baseline.
Representation Quality.
Additionally, we include a colour prediction task as a diagnostic test to determine whether the capsule projector additionally encodes colour changes, an additional group clement, without being explicitly optimisod do so. We hypothesise that poor colour prediction from CapsIE's pose is due to its optimisation being directly for angle and pose changes only. Since we observe these results with R² slightly below zero. However,
7

---

Published in Transactions on Machine Learning Research (10/2025)
given the negative score is very marginal, around the zero mark, we do not interpret this as significant negative correlation. Future investigations into the more significant negative correlation of the 64-capsule model, and whether the score negatively correlates with the number of capsules, are warranted. In Table 2 we can conclude that capsules do not inherently encode colour information under such strong objectives. This is beneficial as allows for more finegrained control for properties that are optimised for and does not. arbritarily encode additional group elements.
Intermediate Projector Embeddings. The role of the projector is primarily employed to decorrelate the embeddings which the objective function operates on from the representations employed downstream. The premise is to avoid representations that are over-fit to the self-supervised objective (Bordes et al.) 2023) However, it has been well studied that it can be beneficial to maintain a number of projector layers and instead utilise intermediate projector embeddings for downstream tasks. Specifically, the equivariant information that has been shown to be captured in the object pose (De Sousa Ribeiro et al. 2024
We evaluate the intermediate embeddings output from the primary capsule layer in the same manner as the representations; however, the activations and pose given over spatial region, we perform average pooling to return activation vector and 4 x 4 pose for each capsule, which we then flatten into vector. As with the representation evaluation, we report our invariant and equivariant task performance in Tables [ and 2 respectively. We find that evaluating the intermediate embeddings of the capsule projector consistently leads to improved performance on intation-equivaiian tasks compared to SIE intermediate embeddings across all settings. In our case, we demonstrate that the preservation of capsule layers for downstrean tasks results in significantly better evaluation performance of the representations, suggesting that capsule nets preserve such properties of interest compared to ML.Ps. Further investigations into this phenomenon are left to future research.
4.3 Quantitative Evaluation of Equivariance
To quantitatively evaluate the equivariant performance of our method and capsule projector, we provide evaluations in line with those proposed in Garrido et all (2023). We report the Mean Reciprocal Rank (MRR) and Hit Rate at k (H@k) on the multi-object setting. Given a source and target pose of an object, we first compute the embeddings of each image and pass the source embedding through the predictor. We then use the resulting vector to retrieve the nearest neighbours.
The MRR is the average reciprocal rank of the target embedding in the retrieved nearest neighbour graph. H@k this case computed to be if the target embodding is the k-NN graph of the predicted embedding, where we only look for nearest neighbours among the views of the same object.
The Prediction Retrieval Error (PRE) gives evaluation of predictor quality, and is given by the distance between its rotation Q1 E H and the target rotation 92 as d = 1- < 91.92* >² of the nearest neighbour of the predicted embedding averaged over the whole dataset.
All the results evaluated by the aforementioned metrics are given Table 3 Our CapsIE network outperforms EquiMod, Only Equivariance and SIE by considerable margin across all metrics and for all dataset splits. We achieve strong perfromance on PRE, reporting 0.21 PRE on the validation set compared to 0.48 for EquiMod and Only Equivariance and 0.29 for SIE. The same significant gains in equivariant performance are shown for MRR and H@1 and H@5. Note, random H@I results in performance of 2% (0.02), demonstrating that our method lies well above random.
4.4 Number of Capsules
Each capsule, in theory, should represent unique concept; thus when the number of capsules is increased, logically so should the network's 's representation ability to capture an increasing number of semantic concepts. Observing the invariant performance during training (Figure 4 and downstream evaluation in Table ] CapsIE gains a slight improvement with the addition of more capsules. Here, online training refers to a co-optimisation scheme where an evaluation network is trained in parallel during standard pretraining procodure, this allows for analysis and visualisation of training dynamics rather than relying on the losses of each term directly. Since this evaluation network has been trained for far longer and on much more diverse representation
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table Quantitative evaluation of the predictor when using a Capsule network projector, using PRE, MRR and H@k. The source dataset, for which embeddings are computed, and the dataset used for retrieval are given in the format source nutree-retieeal for PRE and source for MRR and H@k. Here source refers to the set from which emboddings are computed (train or val), while retricval corresponds to the set used for comparison/retrieew (train, val, all = train + val).
PRE (4)
MRR (t)
H@1 (t)
H (t)
Method
train-train
val-val
val-all
train
val
train
train
val
EquiMod
0.47
0.48
0.48
0.17
0.16
0.06
0.05
0.24
0.22
Only Equivariance
0.47
0.48
0.48
0.17
0.17
0.06
0.05
0.24
0.22
SIE
0.26
0.29
0.27
0.51
0.41
0.41
0.30
0.60
0.51
CapsIE
0.17
0.21
0.20
0.60
0.47
0.50
0.36
0.71
0.58
set during training, the performance is typically higher. The performance however, demonstrates that our model has better utilised the additional representational power to improve performance. This pattern is also observed when evaluating equivariant properties (shown in Figure 4 yet it is less pronounced. However, in Table we also show that increasing the mumber of capsules in a supervised SR-Caps model trained in a standard supervised fashion is not an indicator of increased performance, aligning with prior capsule research Everett al. 2023). This differentiation in behaviour provides an interesting direction for future research.
Invariance Evoluation During Thinnn.
Equivariani Exaluation During Training.
84
80
Rolation
2
76
Capsslee Capesles 32
Capecle +1
12
54
Capaaler -22
Capesies 15
Capsules 1
70
303 1000
1230
1400
1600
2820
2000
1990
1850
Iteratinn
1eraloo
Figure Invariant and equivariant performance of encoder representations during training when varying the number of capsules. (left) Classification evaluation performance (top-1 %) and (right) Rotation prediction online evaluation performance (R²) both learned encoder representations. This evaluation is co-optimised during training and hence differs from the downstream evaluation reported in the Table D and We observe that as capsule numbers increase so does performance; however, at low capsule numbers, later in training invariant objective takes precedence.
4.5
Additional Quantitative Results
4.5.1
Objaverse
Datasets such as 3DIEBench, derived from ShapeNet Chang et al. (2015), have some limitations, considering that they are synthetically generated, although photorealistic. However, datasets such as Objaverse (Deitke) et al. 2023) contain real-world scans, therefore providing the possibility of evaluating equivariant models on different settings. To further validate CapslE
randomly rotated multi-view dataset, we used subset of Objaverso-LVIS Deitke et al.] (2023) with six classes (airplanc, bench, car automobile, chair, coffee table, and gun), following the class selection of Wang et al.] (2024). We evaluated classification and rotation prediction using two approaches. Firstly, we perform transfer learning by fine-tuning the entire network pre-trained on 3DIEBench as described previously. Secondly, we take the model pre-trained on 3DIEBench, freeze the backbone encoder, and train only the task-specific heads from scratch following the same evaluation procedure described in A.3.2
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Evaluation on a subset of Objaverse-LVis LVIS using a ResNet-11 backbone. Representations are evaluated on an invariant task (classification) and an equivariant task (rotation prediction). Each model evaluated five times with different random seeds.
Pre-training (Frozen Backbone)
Transfer Learning (Fine-tuning)
Method
Classification (Top-1)
Rotation (R²)
Classification (Top-1)
Rotation (R²)
VICReg
80.43+1.11
0.29+00008
90.22++0.00
0.62#0.012
SimCLR
83.4440.88
0.29:00003
91.08=+0.72
0.63
AugSelf
83.87+0.38
0.30+000155
90.75+0.70
0.64+0.009
SEN
82.90
0.30+:0.011
90.86+0.38
0.64
EquiMod
83.76
0.30+0.0133
89.89#0.21
0.63#0.011
STE
75.27+1.22
0.29+0.005
89.78+0.66
0.62+0.007
CapsIE
7..58
0.4310.014
90.75=:0.4
0.61
Table 5: Transfer learning via DETR fine-tuning with frozen backbone on MOVi-E.
Method
Classification (Top-1)
Rotation (R²)
mAP
mAP 50
mAP75
SIE
73.7
0.20
26.47
41.83
28.26
CapsIE
72.5
0.23
28.26
43.41
31.38
Results shown in Table demonstrate that CapsIE performs competitively against other methods and, in some cases, out-erforms them by a large margin, e.g. rotation with frozen backbone. We do note that in the fine-tuning case our capsule architocture outperforms the STE invariant baseline demonstrating that the learned invariant representations may be generally well expressed but need a further small amount of supervised tuning to better structure the representations more effectively. This introduces promising direction for future work, and the applicability of pretrained capsule networks.
4.5.2 MOVi-E
To assess performance under more realistic and challenging conditions, we evaluate our approach the Multi-Object Video (MOVi-E) dataset Greff et al. (2022). MOVi-E is a synthetic benchmark containing scenes with up to 17 distinct objects placed in photorealistic environments, often with occlusions and complex backgrounds. Each scene is generated through a 2-second rigid-body simulation in which multiple objects fall and interact. While MOVi-E provides sequences with a linearly moving camera, in our experiments, we sample individual frames and process them independently rather than using the temporal dimension. More information on the datasct can be found in the original paper Greff et al. (2022) and the public repository
In this setting, the task involves detecting every object (via bounding boxes), classifying its type, and estimating its pose relative to the camera frame. For object detection, adopt the DETR architecture Carion et al. (2020), initialising the ResNet-50 backbone with weights pre-trained on 3DIEBench. The backbone is frozen, while the transformer encoder, decoder, and prediction heads are fine-tuned on MOVi-E. To enable pose estimation, we extend DETR with an additional MLP head that regresses rotation quaternions. This predictor mirrors the bounding box regression module, using a three-layer MLP with 256 hidden units. The quaternion regression loss is defined as mean squared error and added to the overall training objective, which a weighted sum of individual DETR losses Carion al. (2020).
Training is carried out for 200 epochs with a batch size of 64, starting from learning rate of 0.0001 that is reduced tenfold after epoch 100. The quaternion loss is assigned a weight of 2, and the Generalised Intersection over Union loss weight of 3, while all other hyperparameters remain unchanged from the original DETR setup.
10

---

Published in Transactions on Machine Learning Research (10/2025)
The results of classification, detection, and rotation regression are summarised in Table Our method outperforms STE in all tasks except classification, and it is the strongest. competing equivariant approach, indicating strong generalisation and robustness to complex multi-object scenarios. Although CapsIE performs competitively, its performance remains constrained by the difficulty of MOVi-E and the fact that the ResNet-50 backbone was trained only in single-object settings. We view this as a promising direction for future work, particularly through adapting the method exploit temporal vidco information.
5
Related Work
5.1
Equivariant Superviedd Learning
Self-supervised learning has seen the majority of its success in the invariant setting by contrastive (Chen ct al. 2020), information maximisation (Zbontar et al.] 2021 Bardes et al. 2022), or clustering methods (Caron et al. 2021 Assran et al. 2022). All families of approaches rely on training a network to be invariant to transformations by increasing the similarity between emboddings of the same image under augmentation. The differing approaches emerge from alternative methods to avoid collapse, a phenomenon where embeddings fall into lower-dimensional subspace rather than the entire available embedding space, resulting in a trivial solution (Hua et al., 2021). Although these methods differ, they all produce similarly performing representations, hence we employ information maximisation methods as the basis of this work due to their computational efficiency.
Learning to be invariant to transformations is typically useful for semantic discrimination tasks, yet preserving information about the transformations can be highly beneficial. Some approaches have attempted to capture specific information regarding transformations by predicting the applied augmentation parameters (Lee et al. 2021}, preserving the strength of augmentations (Xie al. 2022] and introducing rotational transformations (Dagovsski et al. 2022). However, stated in Garrido et al. (2023), these methods provide guarantee that a mapping is learnt in the latent space that reflects the transformations in the input space. Hence,
methods have been employed that address this limitation (Devillers & Lefort 2023 Park et al. 2022 Garrido et al. 2023). ). All of these methods employ predictor networks to predict displacement representations in the
latent space, given a single view representation and the transformation parameters. The latter, SIE Garrido
ct al. (2023), the basis of our work, which further extends prior methods by splitting representation vectors into invariant and equivariant parts better separate differing information.
5.2 Capsule Networks
CapsNets present an alternative architecture to CNNs, addressing their limitations by explicitly preserving hierarchical spatial relationships between features (Sabour et al. 2017). CapsNets replace scalar neurons with vector or matrix poses, representing specific concepts at different levels of parse tree as the network goes deeper. The first layer (primary capsules) corresponds to the most basic parts, while capsules in deeper layers represent more complex concepts composed of simpler concepts they get closer to the final layer, where each capsule corresponds to a specific class.
The key components of the CapsNet are the pose and the activation. The pose of a capsule is an embedding vector or matrix which provides a representation for the concept. The activation scalar is a value between 0 and which represents how certain the network is that the concept is present and can be calculated directly from the values of the posc or via other mcans via the routing mechanism.
The key novelty in CapsNets is the routing mechanism, which determines the contributions of lower-level capsules to higher-level capsules. Numerous routing algorithms, both iterative and non-iterative, have been proposed to address the efficiency and offectiveness of this process (Saoour et al.] 2017 Mazzia ct al. 2021
Feng et al. 2024 Ribeiro et al. 2020 Hinton et al.; 2018 De Sousa Ribeiro et al. 2020 Yang et al. 2021
Everett et al. 2024). Among these, SRCaps Hahn et al. (2019) introduces a non-iteratie iterativee routing mechanism.
This method retains all the desirable properties of CapsNets, such as equivariance, while largely mitigating the time cost of iterative methods at the expense of slight performance loss. However, SRCaps faces the same limitations as other CapsNets where high-resolution or high-class datasets are beyond the network's
11

---

Published in Transactions on Machine Learning Research (10/2025)
abilities when trained in standard fashion. For more detailed description of capsule routing mechanisms, please check this review De Sousa. Ribeiro et al. (2024).
5.3 Part-Whole Decomposition and Interpretability
Capsule networks naturally lend themselves to an explicit part-whole decomposition because each capsule encodes entity's instantiation parameters (pose, prosence and related attributes) and lower-level capsules cast "votes" for higher-level capsules so that coherent clusters of votes identify which parts belong to which whole - an idea formalised as muuting-by-agreene
in the original capsule proposals (Sabour et al.] 2017). This moding/routing mechanism produces structured, disentangled representations (pose matrices or vectors plus activation lengths) that are directly interpretable: pose parameters report geometric transformations while activation magnitudes reflect part/object presence, enabling inspection of what parts drove a decision (Hinton al. 2018 De Sousa Ribeiro et al. 2024).. Work on stacked/ backke/autooncodinng capsule models has further shown that object-level capsules can be learned (even unsupervised) from part poscs, strengthening the claim that capsules implement an intrinsic compositional (part-Pobject) inductive bias (Kosiorek et al. 2019). Other formulations that treat routing as posterior inference or introduce explicit routing uncertainty demonstrate how cetaabliisiievariatiionnn routing makes the part whole assignments and the model's confidence in them explicit, improving robustness and giving a principled way to read out uncertainty about which parts compose a given object (De Sousa Ribeiro al. 2020).
6 Conclusion
Our proposed method demonstrates how self-supervised CapsNets can be employed to better learn equivariant representations, leveraging architectural assumptions, removing the need to explicitly split representation vectors and train separate projector networks. The resulting solution, CapsIE, achieves state-of-the-at performance in equivariant downstream benchmarks with an improvement of 0.01 R² on prior sule-uuper rotation prediction tasks, Our results contribute significantly to the application of CapsNets in supervieee reprosentation learning, introducing desirable propertics with improved effectiveness over MLP projectors.
This work aims to learn higher quality and more applicable representations of images without human-generaled annotations; therefore, such methods can lead to positive societal impacts and the development of more accurate or informative models for a number of downstream tasks. However, as is the case with all vision systems, there is potential for exploitation and security concerns; therefore, one should consider AI misuse when extending our method.
As stated in the original dataset proposal and the problem setting, the methodology presented relies on the group elements being known. Hence, the applicability of the proposed method is only possible in settings where group elements are known. We additionally explore alternative equivariant tasks that were not explicitly optimised for as a diagnostic task to evaluate if the capsule projector additionally encodes additional group elements. Our findings determine that our method retains the fine-grained control of learning equivariant embeddings specified by the objective function, a beneficial property presented in prior methods to avoid learning of arbitrary group elements.
While our capsule networks perform well the equivariant tasks, the invariant performance has suffered as consequence. We attribute this to the reduced capacity of capsule networks, where the number of embeddings is significantly lower than that of the MLP counterpart. To improve performance, alternative capsule networks other than SRCaps could be employed, which have shown improved classification performance, or the number of capsules can be scaled up, where Figure demonstrated improved top-1 performance with greater capsule number. Lastly, a further limitation of our work is the absence of multiple soed runs in most experiments (with the exception of Table 4 consistent with (Bardes et al. 2022) and (Garrido et al., 2023), due to the disproportionately high computational costs.
12

---

Published in Transactions on Machine Learning Research (10/2025)
References
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
8443-8452, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for labcl-efficient learning. In European Conference on Compuler Vision, pp. 456 473. Springer, 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-iiaarianc-covvaaaa regularization for self-supervised learning. In International Conference on Learning Representations, 2022.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Improving deep networks generalization by removing their head. TMLR, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European conference on computer vision, pp. 213 229.
Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXio preprint or arviv:2104 14,22144.424, 2021.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information- Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive Icarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačíć. Equivariant supervud learning: Encouraging equivariance in representations, In International Conference on Learning Representations, 2022.
Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos Kollias. Introducing routing uncertainty in capsule networks. Advances Neural Information Processing Systems, 33:6490-6502, 2020.
Fabio De Sousa Ribeiro, Kevin Duarte, Miles Everett, Georgios Leontidis, and Mubarak Shah. Object-centric learning with capsule networks: A survey. ACM Compuling Surveys, 56(11):1-29) 2024.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13142 13153, 2023.
Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve visual instance discrimination. In International Conference on Learning Representations, 2023.
Miles Everett, Mingiun Zhong, and Georgios Leontidis. Masked capsule autoencoders. arXiv preprint arXiv:2403.04 2024.
Miles Authony Everett, Mingjun Zhong, and Georgios Leontidis. Protocaps: A fast and non-iterative capsule network routing method. Transactions on Machine Learning Research, 2023.
Yangbo Feng, Junyu Gao, and Changsheng Xu. Spatiotemporal orthogonal projection capsule network for incremental few-shot action recognition. IEFE Transactions on Multimedia, 2024.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Quentin Garrido, Laurent Najman, and Yann LeCun. Self-supervised learning of split invariant equivariant representations. Tn Proceedings of the 40th International Conference on Machine Learning, pp. 10975-10996, 2023.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pollern recognilion, pp. 3749 3761, 2022.
Taeyoung Hahn, Myeongjang Pyeon, and Gunhee Kim. Self-routing capsule networks. Advances in neural information processing systems, 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International conference on learning representations, 2018.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9598-9608, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 0rXiv:1412.698/ 2014.
Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey Hinton. Stacked capsule autoencoders. Advances in neural information processing systems, 32, 2019.
Yann LeCun, Fu Jie Huang, Leon Bottou, et al. Learning methods for generic object recognition with invariance to pose and lighting. In CVPR (2), pp. 97 104. Citeseer, 2004.
Hankook Lec, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representa- tions via angmentation self-supervision. Advances in Neural Information Processing Systems, 34: 17710-17722, 2021.
Yi Liu, De Cheng, Dingwen Zhang, Shoukun Xu, and Jungong Han. Capsule networks with residual pose routing. JEEE Transactions on Newral Networks and Learning Systems, 2024.
Vittorio Mazzia, Francesco Salvetti, and Marcello Chiaberge. Ffffeicnt-ccassne:: Capsule network with self-attention routing. Scientific reports, 11(1):14634, 2021.
Jung Ycon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXi0:2204.11371, 2022.
Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos Kollias. Capsule routing via variational bayes. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3749 3756, 2020.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
Jiayun Wang, Yubei Chen, and Stella Yu. Pose-aware self-supervised learning with viewpoint trajectory regularization. In European Conference on Computer Vision, pp. 19 19-37. Springer, 2024.
Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djórk-Amé Clevert. Unsupervised learning of group invariant and equivariant representations. Advonces in Neural Information Processing Systems, 35: 31942 31956, 2022.
Yuyang Xie, Jianhong Wen, Kin Wai Lau, Yasar Abbas Ur Rehman, and Jiajun Shen. What should be equivariant in self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4111-4120, 2022.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Jinyu Yang, Peilin Zhao, Yu Rong, Chaochao Yan, Chumyuan Li, Hehuan Ma, and Junzhou Huang. Hierarchical graph capsule network. In Proceedings of the A AAAT Conference on Arlificial Intelligence, volume 35, pp- 10603 10611, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. Xiviv preprint ar Arx:::2003. 032300 2021.
A Appendix
A.1 3DIEBench Dataset
Model
Figure The 3DIEBench dataset, one model used create 50 different views a synthetic environment, which saved as images along with the latent values by which they transformed.
Typical equivariant datasets generally handcrafted and simple, with small number of classes and instances within each class. This is due to the time needed in order image datasets do allow for testing invariance in the form of augmenting the same image in two different ways, they do not allow for the precise transformation of the subject. Thus, there is need for new, synthetic dataset.
correctness. While standard
We use the 3DIEBench (Garrido et al. 2023 datase which has been created specifically to be a hard yet controlled test-bed for invariant and equivariant methods. The dataset consists 52,472 3d objects across 55 classes of 3d objects from 2hapeNetCorev2 (Chang al., 2015) posed in 50 different views well as the latent information of the view. This be seen in figure For training then randomly seloct two views from each model in the training set. The parameters by which the model could have been augmented are listed in Table G
A.2 3DIEBench Problem Statement
First, we define equivariance by defining Group consisting of set G and binary operation G. G G + G such that are associative; there is identity which satisfies e-a for each a G' there exists an inverse a such that a = a- a. Group actions are concerned with how groups manipulate sets, where the left. group action can be defined as a function a of group and set S, a: G + S such that a(c, s) = s, Vs E S, and a(g, ((g,a(hha)) s)) = a'gh, s), Vs E S, and Yg, E G. our setting, we are concerned with group representation V. which we define p: G GL(V) where GL(V) the general linear group on V. Here p(g) describes the transformation applied to both the input data x and latent f(x given paramcters g
Va G; and
which are linear group actions acting on vector space
Park et al. 2022
The full dataset and splits employed found http::///sthhh..mm/eeeeerrrrrrrp
15

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: Values of the factors of variation used for the generation 3DIEBench. Each value is sampled uniformly from the given interval. Object rotation generated Tait-Bryan angles using extrinsic rotations. Light position expressed in spherical coordinates. This table is sourced from (Garrido al, 2023
Parameter
Minimum value Maximum value
Object rotation X
Object rotation Y
Object rotation Z
Floor hue
Light hue
0
Light 0
0
Light 6
0
2m
Garrido al. 2023). Transformations comprise colour scaling and shifting, and rotations around fixed point.
Following this, we can define the function f: X - Y as being equivariant with respect to a group G with representations px and py if Vx E X, and Vg E G,
[(x((g) = pr f(x).
(8)
The goal is to therefore learn I and py to construct representations that are equivariant to viewpoint transformations when Px is not known, but the group elements that parameterise the transformations are known.
A.3
Training protocols
A.3.1 CapsiE Pre-training
Our proposed CaplE model is comprised of a ResNet-19 18 encoder, SR-CapsNet comprised a primary capsule layer routed to socond capsule layer. The SR-CapsNet projector takes as input the activation map output of the ResNet prior to the final global average pooling. The predictor network employed is that described in Garridoo et al. 2023 All details of the architectural design given in the main paper.
Training of our CapsIE model is donc over 2000 epochs with batch size of 1024, optimised via the Adam optimiser with learning rale 0.001, and default parameters, 8 = 0.9, 32 = 0.999. By default the objective function weighting are as follows, Ainv = 0.1, Aequi = 5, xv = 10, Ac = 1 where we empirically found these optimal for our setting. Further performance gains could be achieved by the tuning of such parameters, however, we deemed this unnecessary.
For ablation studies, where we explicitly state, train for fewer epochs and with smaller batch size, 500 and 512, respectively. We find in practice that this setting is strong proxy for full training performance and significantly saves computational resources.
Training time for 2000 epochs with batch size of 1024, as previously stated, took approximately 22 hours using three Nvidia A100 80GB GPUs, with 64 capsule models taking approximately 25 hours using six Nvidia A100 S0GB GPUs.
A.3.2 Downstream Evaluation
our work, we perform evaluation on both the frozen ResNet-18 representations and the representations from the primary capsules layer, which are evaluated using either linear classifier or an additional capsule layer acting as class capsules, ie the number of capsules set the number of classes and activations are used the logits. Here, we detail the exact training protocols to ensure complete reproducability.
16

---

Published in Transactions on Machine Learning Research (10/2025)
For our evaluations we use two different depths of MLP heads, these are: 1. Deep MLP referring to MLP with layers containing in dim 1024 out, dim neurons, with intermediate ReL.U activations. 2. Shallow MLP referring to single MLP layer with in dim in neurons and out dim out neurons. When we evaluate our primary capsules for angle and colour prodiction, we average the 8x8 feature map so that we only have a single pose vector for the entire image. For our Capsule Classification lask, we do not have an dim as we do not use a MLP, but instead use a capsule layer which operates on the primary capsules pose and activations.
Table 7: Training settings for our evaluations. Settings are the same for all number of capsules. NC is used as shorthand for number of capsules. denotes that this element is not used. * denotes multiplication.
Representations Augle
Representations
Representations
Capsule Angle
Capsule Colour
Capsule
Colour Classification
Classification
Caps Head MLP Head
Yes
Deep
Shallow
Deep NC * 16
Shallow
dim
512
512
512
NC 16
N/A
out out_ddmm
4
2
55
4
2
55
Optimizer
Adamn
Adam
Adam
Adam
Adam
Adam
LR
0.001
0.001
0.001
0.001
0.001
0.001
31
0.9
0.9
0.9
0.9
0.9
0.9
82
0.999
0.999
0.999
0.999
0.999
0.999
Batch Size
256
256
64
256
256
256
Epochs
300
50
300
300
50
300
Objective
MSE
MSE
Cross Entropy
MSE
MSE
Cross Entropy
A.3.3 Supervised Training of SR-Caps In our work we train a Self Routing Capsule Network model in supervised fashion for the downstream tasks to evaluate whether our pretrained model improves the quality of downstream evaluations. The training setting of these runs can be found in table
Deep MLP refers to an MLP with layers containing number caps 16 * 2- 1024 neurons, with intermediate ReLU activations. Shallow MLP head refers single MLP layer with number caps 16 2 in neurons and cither 4 (for rotation prediction) or 2 (for colour prediction) out neurons.
nunber_caps
Table 8: Training settings for our supervised Self Routing Capsule Network model. Settings are the same for all numbers of capsules. denotes that this element. not used.
Angle
Colour
Classification
Caps Head
Yes
MLP Head
Deep Adam
Shallow
Optimizer
Adam
Adam
LR
0.001
0.001
0.001
31
0.9
0.9
0.9
32
0.999
0.999
0.999
Batch Size
256
256
64
Epochs
300
50
300
Objective
MSE
MSE
Cross Entropy
A.3.4 Invariant and Equivariant SSL Benchmarks
We report below the classification (invariant, Table 9. rotation prediction, and colour prediction (equivariant, Table 10) performance of bascline self-supervieed methods. The below bascline results are acquired from (Garrido al. 2023), with the exception of those denoted by ** which corresponds to our re-implementation.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Table 9: Evaluation of invariant properties on downstream classification task for baseline SSL methods. We evaluate both the representations and the intermediate embeddings of the projection head when different numbers of capsules in the projection head is used. < refers to non-compatible experiments.
Embedding Dims
Classification (Top-1%)
Method
Inv.
Equi
All Inv. Equi
Encoder Representation
VICReg
84.74
VICReg, 9 kept identical
72.81
SimCLR
86.73
SimCLR, g kept identical
71.21
SimCLR + AugSelf
85.11
EquiMod (Original predictor)
87.19
EquiMod (SIE predictor)
87.19
SIE Garrido et al. 2023
512
512
82.94
82.08
80.32
STE
512
512
82.54
82.11
80.74
CapsIE 16
16
256
76.51
CapsIE 32
32
512
79.14
CapslE 64
64
1024
79.60
Capsule Projector 1st Intermediate Embedding
STE
1024
1024
80.53
77.64
CapsIE 16
16
256
74.90
CapslE 32
32
512
78.60
CapslE 64
64
1024
79.24
Table 10: Evaluation of cquivariant propertics on downstream rotation prediction (left) and colour prediction (right) tasks for baseline SSL methods. We evaluate both the representations and the intermediate embeddings of the projection head when different numbers of capsules in the projection head used.
Rotation Prediction (R²)
Colour Prediction (R²)
Method
All
Inv.
Equi.
All
Inv.
Equi.
Encoder Representation
VICReg
0.41
0.06
VICReg, g kept identical
0.56
0.25
SimCLR
0.50
0.30
SimCRR, g kept identical
0.54
0.83
SimCLR AugSelf
0.75
0.12
EquiMod (Original predictor)
0.47
0.21
EquiMod (SIE predictor)
0.60
0.13
SIE Garrido et al. 2023
0.73
0.23
0.73
0.07
0.05
0.02
SIE
0.72
0.21
0.71
0.06
0.05
0.03
CapsIE 16
0.68
0.02
CapslE 32
0.74
0.04
CapslE 64
0.72
0.01
Projector 1st Intermediate Embedding
SIE
0.38
0.58
0.45
0.09
CapsIE 16
0.64
-0.01
CapslE 32
0.71
-0.04
CapsIE 64
0.67
-0.09
18


Reviews:
[{"review_id": "BSio2tEVT8", "reviewer": "Reviewer_eRzZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The authors claim that CapsIE achieves state-of-the-art performance on rotation tasks using the 3DIEBench dataset, showing a 0.01 R2 improvement over previous self-supervised rotation prediction methods. Their results are getting close to supervised baselines. The quantitative evaluation also confirms significant improvements in PRE, MRR and H@k metrics compared to earlier methods.\n* CapsIE is more efficient and uses fewer network parameters than equivalent SSL methods like SIE. The authors attribute this efficiency to their choice of SRCaps.\n* The study provides good insights into intermediate embeddings, which leads to better performance on rotation equivariant tasks compared to other SSL methods' intermediate embeddings. This finding further confirms that CapsNets are better at preserving important properties in their layers compared to MLPs.\n* Overall, the paper is well-written and easy to understand.\n\nWeaknesses:\n* CapsIE is limited by its reliance on known group elements, which restricts its applicability to real-world scenarios where these transformations are often unknown or implicit.\n* The results show that CapsIE's ability to capture invariant properties is slightly worse compared to SIE's MLP projector. The authors attribute this to using significantly fewer embeddings for the invariant criterion. While this paper focuses mainly on equivariance, a deeper analysis of this trade-off would be valuable - including its practical implications for downstream invariant tasks and potential solutions.\n* While SRCaps is efficient, it comes at the cost of slightly lower classification accuracy compared to other capsule architectures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces CapsIE, a novel SSL method that uses capsule networks to learn representations that stay consistent in some ways while changing appropriately in others.\nMost existing SSL methods use simple NN layers (MLPs) as projection heads, which can be limiting. Previous approaches like SIE had to explicitly split their representations and use separate projection heads to handle this challenge.\nCapsIE takes a different approach by using a capsule network as its projection head. This design naturally handles equivariance - claiming it can appropriately adjust representations when the input is viewed from different angles or perspectives, without needing the complex splitting mechanisms used before.\nThe paper also created a custom loss function designed specifically for their capsule network projector. The authors claimed that this loss function works in two ways: it maintains consistency (invariance) by minimizing entropy, and it ensures appropriate changes (equivariance) by maximizing the similarity between predicted representations and those from augmented views of the same data. They also include a regularization term on the pose embeddings to prevent the model from collapsing into trivial solutions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Recommendation to strengthen the work: \n\n* The paper mentioned that CapsNets might preserve additional group elements without prior knowledge, further elaboration on concrete steps to overcome this limitation would strengthen the work. \n\n* While the majority of focus of this paper is on equivariance, a deeper analysis of this trade-off, its practical implications for downstream invariant task and potential mitigation strategy can be good addition to the paper. How does this affect tasks that need invariant representations? What can be done to mitigate any negative effects?\n\n* The authors chose SRCaps due to resource constraints on the 3DIEBench dataset, but this choice hurts performance on invariant properties. The paper needs more discussion or experiments showing how this trade-off impacts overall performance.\n\n* Although the paper identified interesting direction for future research regarding capsule utilization behaviour, it can be more interesting of them offer even preliminary hypothesis or more-in depth discussion on these observations which could enhance the immediate impact of the current submission. \n\nTechnical issues for acceptance:\n\n* The paper is lack of objective function hyperparameter(HP) sensitivity analysis (λinv = 0.1, λequi = 5, λV = 10, λC = 1). While a thorough search of finding optimal parameters are resource-intensive, providing some sensitivity analysis or ablation studies on these key weighting parameters can show the robustness of the method and its performance across different HP settings. \n\n* The negative R2 values for the colour prediction Equi are presented without any discussion/explanation in the text\n\nWriting and Organizing issues:\n* Figures are placed far from where they're mentioned. Figure 1 is referenced two pages after it appears - either move it closer or provide better descriptions so readers don't have to flip back and forth.\n* Figure 3 is never referenced in the text at all.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DFQ6F9c2Ck", "reviewer": "Reviewer_FDL6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the description of the background and problem is clear; the work tackles an established problem in the field; it utilizes a reasonable existing architecture (Capsule Networks) and explores how it could help; the CapsIE models tend to have a lower computational cost\n\nWeakness: there is some clarity and detail lacking in the methods and results; the claimed benefits are not supported by the actual results; not entirely clear how to best use these findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to expand methods for self-supervised learning with an emphasis on creating invariant and equivariant representations. It does so by modifying a past architecture to use Capsule networks. This allows for the joint learning of invariant and equivariant representations without the need for entirely split representations. The CapsIE network performs worse on classification and color prediction than the prior split architecture (SIE). The CapsIE projector performs better than SIE's projector on rotation prediction and related tests of equivariance, while their encoders perform essentially the same.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarity: The authors should explain the training procedure for the supervised and SIE models better. They should explain the motivation for and interpretation of the color prediction results (were they expecting good color prediction and didn't get it? Or is color meant to be ignored?). Not clear what \"co-optimized\" means in Figure 4 caption or why the results re different from Table 2. SIE acronym is never actually spelled out. Several typos throughout.\n\nInsufficient evaluations: The results are not presented with standard deviations over multiple random seeds, so there is no way to know if the performance differences are significant. The authors also make this statement \"the use of a capsule projector in place of a MLP can lead to vastly improved performance in rotation prediction (↑ 0.01 R2) \". This is obviously false. .01 is by no means a \"vast improvement\" and it is not even clear if that .01 difference would remain if results from multiple seeds were reported. Any claims of superiority based on the this result should be removed. \n\nBroader goals:  The authors mention that this work can learn high quality representations without human annotation, yet it seems like this class of methodology can only be employed with a dataset containing well annotated pose descriptions (which can only be generated through simulated images or human annotations). And the equivarence learned only applies to exactly the feature the model was trained to be equivariant to (rotation). These facts combined make the problem look increasingly more like a supervised one. Is there a belief that this methodology could somehow be used to learn representations with beneficial features that aren't a direct result of what is trained into them with well-labeled data?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nHqYPlZ2aj", "reviewer": "Reviewer_N8YA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper gives a clear idea to learn both invariant and equivariant features in self-supervised learning.\n\n- It uses Capsule Networks as a projector, which helps to keep part-whole spatial information.\n\n- The method shows better results than some earlier equivariant self-supervised methods on a real dataset.\n\nWeaknesses:\n\n- The paper does not show strong theory to prove that the pose predictor will always learn a good mapping for different transformations. If the mapping is not unique, the training may fail.\n\n- It is not fully clear how the pose vector can really handle very complex or fine-grained changes, like when there are many parts or occlusion. The method may lose detail when objects have complex shapes.\n\n- The method works on a specific dataset with 3D rotations, but there is no test on larger or more diverse scenes. So it is hard to say if the same design can scale well to real-world data with multiple objects and noise.\n\n- The group theory words like “group action” are used, but the model does not really check or use full group properties like closure and inverse. So the theory part may look formal but not very useful in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents CapsIE, a self-supervised learning framework that employs Capsule Networks to jointly learn invariant activations and equivariant pose representations, and shows through experiments on the 3DIEBench dataset that this approach achieves better performance on equivariant rotation tasks than prior equivariant self-supervised baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please add more explanation or experiments to show when the pose predictor may fail or give non-unique mappings, especially for complex transformations.\n\n2. Clarify how much the use of group theory ideas really helps the final model, or remove extra group theory terms if they are not needed.\n\n3. In the abstract, it is said that \"Code is available at anonymised\". However, this text does not corresponds to any link to the code. Please modify it.", "broader_impact_concerns": "This paper is purely technological, and does not have ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)