Disentangled representation learning through unsupervised symmetry group discovery
Authors: Barthélémy Dang-Nhu
Louis Annabi
Sylvain ARGENTIERI
Venue: ICLR 2026 Poster (2026)
Keywords: Representation learning
Disentanglement
Group Theory
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Symmetry-based disentangled representation learning leverages the group structure of environment transformations to uncover the latent factors of variation. Prior approaches to symmetry-based disentanglement have required strong prior knowledge of the symmetry group's structure, or restrictive assumptions about the subgroup properties. In this work, we remove these constraints by proposing a method whereby an embodied agent autonomously discovers the group structure of its action space through unsupervised interaction with the environment. We prove the identifiability of the true action group decomposition under minimal assumptions, and derive two algorithms: one for discovering the group decomposition from interaction data, and another for learning Linear Symmetry-Based Disentangled (LSBD) representations without assuming specific subgroup properties. Our method is validated on three environments exhibiting different group decompositions, where it outperforms existing LSBD approaches.

Published as a conference paper at ICLR 2026
DISENTANGLED REPRESENTATION THROUGH UNSUPERVISED SYMMETRY GROUP DIS-
LEARNING
COVERY
Barthélémy Dang-Nhu, Louis Annabi & Sylvain Argentieri Sorbonne Université, CNRS, Institut des Systèmes Intelligents et de Robotique, ISIR. F-75005 Paris. France {dangnhu, annabi, argent upmc. fr
ABSTRACT
Symmetry-based disentangled representation learning leverages the group struc- ture of environment transformations uncover the latent factors of variation. Prior approaches to symmetry-basee disentanglement have required strong prior knowledge of the symmetry group's structure, or restrictive assumptions about the subgroup properties. In this work, we remove these constraints by proposing a method whereby an embodied agent autonomously discovers the group structure of its action space through unsupervised interaction with the environment. We prove the identifiability of the true symmetry group decomposition under minimal assumptions, and derive two algorithms: one for discovering the group decom- position from interaction data, and another for learning Linear Symmetry-Barte -Based Disentangled (LSBD) representations without assuming specific subgroup prop- erties. Our method is validated on three environments exhibiting different group decompositions, where outperforms existing LSBD approaches.
INTRODUCTION
An important property of representation is its disentanglement, as it enables a form of inter-
2017). fairness (Locatello et al., [019a), improved transferability (Lee pretability Higgins al.)
et al., 2021 Bengio et al. (020)). and the ability to directly manipulate the latent space (Kim &
Mnih 2018 Chen et al.j 2016). For this reason, many unsupervised disentangled representation
learning methods have been proposed, initially relying Variational Autoencoders (VAEs) (Kim)
& Mnih 2018 Kumar et al. 2018]] Higgins et al. 2017) or Generative Adversarial Networks (GANs) CCen et al., (0166). Locatello et al. (2019b) showed that unsupervised disentanglement requires additional prior knowledge or inductive biases. Thus, several approaches, relying on different additional assumptions, address the question of unsupervised disentangled representation learning.
Learning disentangled representations relies on the assumption that there exist true underlying factors of variation in the environment (Bengio et al. 2013)), and aims to infer them from available observations. The symmetry-based approach of Higgins et al. (2018) proposes to achieve such disentanglement by exploiting the subgroup decomposition of the group of environment transfor- mations, called symmetries. Each subgroup is associated with a specific part of the representation. When symmetry from a particular subgroup is applied, only the corresponding part of the repre- sentation varies. (Caselles-Dupré is only possible when access is granted to transitions (initial observation, transformation, resulting observation). Several notable works follow this approach (Quessard et al., 20200; Tonnaer et al. 2022 Keurti al. 2023). However, they rely restrictive assumptions regarding the nature of the symmetry group or prior knowledge about its structure. This paper aims to overcome these limitations by designing algorithms and providing proofs for the autonomous discovery of the symmetry group structure, and its exploitation for disentangled representation learning.
al. (2019) demonstrated that symmetry- based disentanglement

---

Published as a conference paper at ICLR 2026
Our contributions are as follows:
We prove, under certain assumptions, the identifiability of the ground group decomposi- tion of the symmetry group from a dataset transitions.
We derive from this theorem an algorithm for the discovery of the symmetry group decompo- sition.
We introduce novel method for learning LSBD representation directly from group de- composition, without imposing any structural assumptions on the subgroups, and we provide theoretical guarantees of disentanglement under specific assumptions.
We combine these two algorithms and show experimentally that the full method outperforms other LSBD methods on several datasets with different group structures.
PRELIMINARIES
W
y
c
x
W
W
gy
ide
9x
Figure 1: Colored Flatland environment. The group of symme- tries can be decomposed as G G2 x G,, x G. corresponding respectively to the cyclic groups of translations axis/vertical axis, and in a list of predefined colors. The agent has access to several symmetries (or actions) Gr = fxx= . Ga.
G
2
the horizontal
Figure 2: Equivariance prop- erty.
Gy = (y } C Gy, and Ge = {at, e- Ge.
We consider the framework of Linear Symmetry- -ased Disentanglement (LSBD) (Higgins al., 2018), which provides a formal definition of disentanglement suitable for deriving identifiability results and guiding the design of representation learning algorithms. Let V denote the set of possible environmental states, We define generative process 6 W - X' that maps a state to an observation, and an encoder X > Z that maps observations into latent representation. The overall mapping is then given by f h e. b W - Z. We assume that is an intrinsic (and unknown) property of the environment, while h agent-specific and can learned.
We further assume the existence of symmetry group G acting on W. A key assumption is that G satisfies the standard group axioms: the existence of an identity element, closure under composition, and the existence of inverses. This group structure enables the definition of group action 'w G W - W, which maps each pair (g, w) E G x W to transformed world state w/ E W resulting from the application of g. The agent is endowed with an action set G G that contains only subset of the full group. Crucially, 9 is not required to form a group itself, in particular, the agent's 's actions may not be reversible, and the identity element of G may not be included in G.
We also assume that the group G admits a decomposition into a direct product of subgroups, i.e., G = G2 x... x Gk. For example, in the Flatland environment illustrated in Figure the symmetry group G can be decomposed into three subgroups corresponding to cyclic groups of horizontal translations, vertical translations, and color shifts.
Definition (Linear Symmetry Based Disentanglementt, A representation h is said be symmetry- based disentangled (SBD) with respect to <W, b, Il G+) if:
1. There exists a group action : G x Z,
2. Equivariance holds: Vg E G, w E W, we have g [(w) [[g ww w)).
3. There exists a decomposition Z = $ Zx and group actions Gs x 2x Zr such that
(g).-... (91.-...94) .9K) (21, (g) -1.-... .9 K 2k),
2

---

Published as a conference paper at ICLR 2026
4. The function h is injective
Moreover, the representation is said be linearly disentangled (LSBD) .2 is linear, i.e. there exists a representation p p:C. G - GL( such that g 2 p(g)2.
The original definition provided by Higgins et al. (2018) does not explicitly state the fourth condition requiring the encoder h to be injective. However, this constraint implicitly assumed within the LSBD framework; without it, any constant mapping would trivially satisfy the LSBD criteria. Daselles-Dupré
al. (2019, Theorem 1) demonstrated that learning an LSBD represen- tation is impossible from observations alone, they proved that multiple distinct world sets and group actions can produce the same set of observations. However, these environments differ in their transitions. Consequently, they proposed leveraging transitions of the form (x, 9,,") rather than relying solely on passive observations x. This perspective naturally aligns with the reinforcement learning setting, where agents actively interact with the environment by perform- ing actions that induce state transitions. Accordingly, in the remainder of this work, we refer to symmetries g actions. Mathematical background and a symbol table can be found in Appendix A
RELATED WORK
Several methods have been proposed to learn LSBD representations from transitions, all relying on auto-encoder architectures. Forward- VAE (Caselles- Dupré et al. 2019) augments the evidence lower bound (ELBO) of VAE with latent-space action loss. Disentanglement is encouraged by constraining the matrices p(g) to follow a predefined structure, which requires prior knowledge of the subgroup decomposition of the symmetry group, as well as the minimal number of latent dimen- sions assigned to each subgroup. Another method, proposed by Quessard et al. (2020), referred as SO-Based Disentangled Representation Learning (SOBDRL), aims to learn representations with a prediction loss that aim to infer the next observation x' from (x. g). The action matrices are parame- terized as elements of the special orthogonal group (O)d), the disentanglement is encouraged with a regularization term that minimizes the number of latent dimensions involved each transforma- tion, encouraging transformations constrained to SO(2). LSBD-VAE, introduced by Tonnaer et al. (2022), relies on the A-VAE architecture (Rey et al. 2019). which supports latent spaces defined arbitrary manifolds. In this framework, both the group decomposition G G1 and its representation are assumed to be known a priori. This prior knowledge allows the model to align the latent geometry with the group structure and to incorporate an action-aligned loss term, in the spirit of Forward- VAF. Homomorphism AutoEncoder (HAF), proposed by Keurti et al. (2023). assumes that G a Lie group and that the agent has access (gg). where (p an unknown non- linear mapping. The action representation is learned from this mapping by jointly predicting both current and future states in the observation and latent spaces. Disentanglement is encouraged by enforcing block-diagonal structure on the action matrices.
Some other methods aim to learn LSBD representation solely from the observation based on the metric of the ground-truth Riemannian manifold (Pfau et al.) (020), the cardinality of each cyclic group (Yang al. 2022) or the commutativity of the whole symmetry group (Zhu et al. 2021).
We observe that all state-of-the-ee methods based on transitions rely on assumptions regarding the structure of the symmetry group or its subgroups. In contrast, the goal of this work is relax these assumptions by introducing symmetry- disentangled representation learning approach that does not require any prior knowledge of the group decomposition.
METHODS
We suppose that the available actions G are a subset of the whole group action and that there is a dataset D of transitions (x, g, x²) where E G are the indices of the actions taken by the agent. We aim to learn an LSBD representation h. Our method consists of three steps:
1. We learn an entangled representation í.e. a representation satisfying only points 1, and 4 of Definition to learn an action representation p p:C G GL(22) and encoder h X Z.
3

---

Published as a conference paper at ICLR 2026
2. From p and h compute decomposition G G1 ... Gk by regrouping actions using a custom pseudo-distame based on group theory.
3. From this decomposition we learn disentangled representation.
4.1 (STEP 1) LEARN AN ENTANGLED REPRESENTATION
Our objective is to learn encoder h X - 2 R2 and an action representation Pv 9 Rdddd satisfying the equivariance property defined in Definition | As there is prior knowledge about th action matrices, each matrix Pv(g) is directly parameterized by de learnable scalars, resulting in total of [g] x de parameters. To perform this step, we introduce a method referred as Action-based VAE (A-V.E), which builds upon the varia- tional autoencoder (VAE) framework (Kingma & Welling 2014 Rezende et al. 2014). The goal to map each observation x E X to a latent representation in
R Let T = (ar, 9. x²) denote transition, and let and 2' be the corresponding latent representations of and x', respectively. The model architecture is illustrated Figure and defined as follows:
Peva(= 2,9)
po(X'l2') = (No(()))) Diag(rr(22)))) (igoat)))
(1)
pola' 12)
(o.l ((p)lo)oo())) Id), (o(Z'(=') = N (us(zz), Diag(as(>)))))
(2) 90(2' axt)
(3) Figure 3: Graphical model
In contrast to the standard VAE, condition the prior distribution over Z' both the past obser- vation x and the action g. To maximize the expected log-likelihood B[log p(T)] with respect to the model parameters σi and vh, we derive the corresponding evidence lower bound (ELBO) for graphical model. As shown in Appendix B
we obtain (up to an additive constant):
1 p(T) > 2 Il (o) us(x)) 2
1l0022))1 + Σ log (ogo)(zz),
}
action part
(= Elogoo(<))+ x' - (=Ha(z)) o0(2!)
reconstruction part
(4)
Analogously to 3-VAE (Higgins et al. 2017). we introduce a weighting coefficient to balance the two components of the objective, resulting the loss function L LREC + ÅACTLACT- Each of the three conditional distributions in the model is implemented using deep neural networks trained via backpropagation. The model parameters 0, é, and &b are optimized to maximize the ELBO. As in standard VAEs, apply the reparameterization trick to enable gradient-based optimization through the reconstruction term. In practice, the standard deviations dg and 0g are fixed. Details of the neural network architectures provided in Appendix 1.2
4.2 (STEP 2) LEARN THE GROUP STRUCTURE
Once the action representation P3 and the encoder h = have been learned, we aim to leverage them to recover the group decomposition G G2 Gk. By abuse of notation, will treat the direct factors G, as subgroups of G.
4.2.1 ASSUMPTIONS
Assumption 1. The environment is fully observable i.e. the observation function b W X is injective.
It is a strong assumption, however it is necessary as we have the following result: Theorem 1. For a SBD representation to exist, it is necessary for the observation function functiooonn to be injective (up an interaction equivalence class).
The definition the interaction equivalence class and the proof of the theorem are provided in Appendix G The key idea that components of the world state that do not influence the agent's
4

---

Published as a conference paper at ICLR 2026
interaction can be discarded, yielding equivalent environment from the agent's perspective. In this reduced environment, the observation function must be injective for SBD representation to exist. Although this assumption is not always stated explicitly, it is in fact necessary condition for all SBD representation learning algorithms and is not specific to our method.
The next two assumptions assumptions specific to the proposed algorithm, and are intended to replace the stronger prior assumption commonly made in the SBD literature consisting in providing prior knowledge of the group decomposition. We first assume that each action belongs to a unique subgroup G3 We refer to this property as disentanglement of the action set with respect to 7 Gk- is a strong assumption but we demonstrate empirically Appendix H.2 that related SBD methods make a similar implicit assumption.
Assumption 2. G is disentangled with respect to G Th Gy- That is, G 91 .... 9K with yk, G3 C Gk-
We argue that this assumption alone not sufficient to recover the correct decomposition. To illustrate this point, consider two distinct environments analogous to Flatland shown Figure 4 (a) 2 3 cyclic grid i.e. Ga
(8)
Z/2Z 2/3Z with actions g = 3 u } and (b) a 6 x 1 cyclic grid i.e. G 2/62 with actions G (2x+ 3r }. Both environments satisfy As- sumption and can share the same representation, as there exists an isomorphism from GH to G that maps each element of G" to corresponding element in G From the agent's 's perspective, these two situations are indistinguishable in the absence of additional assumptions. Ideally, seek assumption that both covers wide range of practical scenarios, i.e. action sets G, and enables computationally tractable procedure for recovering the group decomposition. Among the various options considered, we adopt the following assumption, as offers favorable trade-off between situation coverage and computational feasibility:
3x +
2x
Figure 4:
Two isomorphic
group actions satisfying As- sumption
Assumption 3. For all g.g' E 9 with g + g', if they belong to the same subgroup then there exists E G and m E M] such that we have either un g', 9 = g y'u", g." ggu or g' = u g.
Combined with Assumption 2 it is straightforward to show that the implication of Assumption 3i iiss fact an equivalence. As result, we obtain a simple and practical criterion for determining whether two actions belong to the same subgroup. In terms of situation coverage, as soon as M > 2, Assumption 3 holds in common cases such as when G; contains an action and its inverse, when GR Gs. when G. G In practice, the action sets considered in the experimental sections of state-of-thh-an SBDRL algorithms typically fall into one these categories. In the scenario illustrated in Figure 4 Assumption 3 allows to assume that situation (b) will never occur, method will thus assume that the environment corresponds to case (a).
4.2.2 ALGORITHM
We now introduce method to recover the group decomposition i.e. cluster the available actions into subgroups. Given an encoding function h X' - IRd and matrix A E Rdxd, we define the following semi-norm:
|| = E, [1]4h())]]
From this and Assumption 3 we define the following pseudo-distance to determine whether two actions belong to the same subgroup. We write A, instead of pr(g) for simplicity and readability:
da(g.g)) = min GeG m=[ M]]L,M]
min {1Ag- - A Aglloi |I - Ag ATllas Ag AT Aglls; Ag - Ag A ll
(6)
Theorem 2. If the Assumption to 3 are satisfied, the dataset contains all the possible transitions,
W is finite and the A-VAE loss converges toward its minimum, then some point of the training, two available actions will belong to the same subgroup if and only if their distance with respect to de is below specific threshold yi computed from h and pa-
5

---

Published as a conference paper at ICLR 2026
Based on Theorem ever (g(g.g') g') < 1). The choice of the threshold 77, the details of the algorithm, and the proof are provided in Appendix C
we design clustering algorithm that groups together actions g and g' when-
Once the group decomposition has been recovered, a suitable disentangled representation learning algorithm be applied. For example, if 9 (91. 9 u (92, 92 ", then G is isomorphic to a subgroup of S0(2) x SO(2), and Forward- VAE (Caselles-Dupré et al. 2019) can be employed with an appropriate parameterization. However, discussed in Section 3 existing LSBD methods still rely on some form of prior knowledge about the group structure. In the following section, we
address this limitation by introducing new disentangled representation learning algorithm that does not require such prior information.
4.3 (STEP 3) LEARN A DISENTANGLED REPRESENTATION
Now that we have the symmetry group decomposition, we aim to find a linear disentangled repre- sentation i.e. decomposition Z 1 @ @ Zx and an action representation p == P1 @ PK such that for each action 9 = (g)
9K) E G and latent factor x (Z1. , 2K) we have 2k) (pr (0i) 2....., PK )=k)
This definition allows the Z, to be any sub-vector spaces of 2 as long as they form a direct sum, our method, we additionally choose to search for represen-
they are not required to be orthogonal. tations where the disentanglement aligns with Cartesian axes of the latent space Z= x x...xx x Zx- This choice is motivated by the fact that, under most widely accepted definitions of disentangle- ment, each latent dimension is expected to encode information about at most one ground-truth factor variation (Wang et al. 2024p. Consequently plg...... ..o.......K......... (2). zk) = p(g)z with concat( (1. =K) and p(g) diag(pi (g)))...,P((g)))
Thanks to Assumption 2 each action is known to belong to a unique sub- group. Consequently, for any g G. and V' # k have p (g) equal to the identity transformation. In matrix terms, this implies that each ac- tion is represented by the identity matrix, except for single block along the diagonal, as illustrated in 5 with the matrix À (in practice, the indices of the matrix may be permuted, however for the sake of clarity, we illustrate only the case in which the active dimensions are adjacent). Learning the structure of these matrices amounts to assigning each latent dimension to a unique subgroup Gs. Let Ti E (0, 1)d denote the bi- nary indicator vector encoding the set of dimensions assigned to the k--th subgroup. such that Σx TT. 1.
Ä
TT
TROT
Figure 5: Masking used to build disentangled ac- tion matrices
To enforce the desired block structure in the action matrices, we apply the mask ग to unstruc- tured action matrices A as illustrated in Figure 5 Let k(g) denote the index of the subgroup to which the action g belongs, and let denote the element-wise product. The structured action matrix À, is then defined as:
Ag = T ( @ Ag + (1 - Tk(g) k 1
To learn the vectors TRe we employ a continuous relaxation. Specifically, we use d softmax op- erations to ensure that Thi E [0, 1] with Y, Thi 1. In order promote disentanglement, we introduce an additional term in the A-VAE loss function that encourages the vectors T to be close to be binary. A natural approach to minimize the entropy H(T)) Σ, H(T..)) However, empir- ical observations show that directly minimizing this entropy causes it to collapse to zero before the other loss components begin to decrease, leading to a random dimension assignment. To address this issue, we define the disentanglement loss LDIS Σ, [T Cl. where C' is a target entropy value that is gradually annealed from its maximum to zero during training. We refer to this method as the Group-Masked Action-based VAE (GMA-VAE). The following result, proven in Appendix D formalizes the disentanglement guarantee:
Theorem 3. If Assumptions ! 2 are satisfied, the dataset contains all the transitions and G is finite, then the encoders minimizing the GMA-VAL loss are LSBD representations with respect to (W, b, (9t)) with (Gx) representing the subgroup generated by GE-

---

Published as a conference paper at ICLR 2026
5 RESULTS
5.1 EXPERIMENTS
To evaluate the disentanglement, we use the Independence (Inde) metric (Painter et al.) 2020j that was specifically designed for the LSBD framework; we will also use classical disentan- glement metrics: 3-VAE (Higgins et al.] 2017), Mutual Information Gap (MIG) (Chen et al. 2018). DCI disentanglement metric (Eastwood & Williams 2018). Modularity (Mod) (Ridgeway & Mozer, 2018) and SAP (Kumar et al. (0188). All these metrics take values between 0 and and are meant to be maximized.
Metries:
Algorithms: We categorize the baseline methods into three classes: (1) Supervised methods where the action representation p is given. The only supervised method is LSBD- VAE (Tonnaer et al. 2022). (2) Self-supervised methods where p is learned We introduce a modified LSBD-VAE in which the action representation p is learned rather than provided, refer to this variant method LSBD- VAE*. As also reported in Tonnaer et al. (2022). we were unable to obtain satisfactory results with Forward- VAE (Caselles-Dupré et al. 2019) on our datasets, therefore it is not included among the baselines. An other method is HAE (Keurti et al. 2023) that is specifically designed for Lie groups and therefore is only used in Section 5.6 (3) Unsupervised methods which rely solely on observations rather than transitions. This category includes classical disentanglement approaches: 3-VAE (Higgins et al. 2017), Factor-VAII (Kim & Mnih 2018) and DIP-VAE I/II (Kumar et al. 2018).
in SOBDRL (Qessard et al.] (020)
Latent dimension: For A- VAE we arbitrarily chose latent dimension of 13, for the LSBD methods we chose the minimal dimension depending on the method and the symmetry group. Those minimal dimensions are discussed in Appendix [
Similarly to Flatland (Caselles- Duuré et al. 2018), our first enviromment consists of disk moving along the and y axes over black background as illustrated Figure I Additionaly to the groups acting on the position of the disk, a third group acts on the color feature and can be either cyclic shift of the RGB channels, corresponding to Gc Z/3Z with Gc (c=,ct) or a full permutation group over the RGB channels, i.e., Gc = G3 with Ge = G The second environment is based on the COII. dataset (Nene et al.) 1996), which contains images of objects captured from multiple viewpoints. Each observation consists of n adjacent objects. Each object i [1, ni can be rotated through k, discrete angles, forming a cyclic rotation group GR, = Z/k,Z, with the action set Gn. , In addition, the objects can be permuted via the symmetric group Ge 6,, The third environment use the 3DShapes dataset (Burgess & Kim 2018), which consists of rendered images of a 3D object placed in colored room. The data is generated from six discrete ground-truth factors: wall hue, object hue, background hue, object scale, object shape, and viewing angle. For each factor i, we define cyclic symmetry group G; Z/k; and an action set consisting of two shifts, G, = {g, .g," corresponding to increments and decrements along the factor axis. The final dataset is MPI3D (Gondal et al.) 2019), which consists of realistic images of a robotic arm moving an object. Among the various factors of variation available in the dataset, we retain only the horizontal and vertical rotation angles of the arm. Both are modeled eyclic groups, GH = Z/40Z and Gy = Z 2/40Z, with corresponding action sets GH = G and GV- = G
Environments:
5.2 ACTION CLUSTERING
To evaluate the action clustering performance of Step 2, we use the Flatland environment with cyclic color shifts (FLC) and color permutations (FLP), as well as the COIL dataset with two (COIL2) and three (COIL3) objects. Our algorithm successfully recovers the ground-trut trrth group decomposition in 100% of runs. The average group distances across random seeds are reported in Appendix H.3 In these experiments, the datasets include all possible transitions, and the available actions are simple (e.g., an action and its inverse). To assess the robustness of our method, we consider more challeng- ing settings with both complex action sets and limited transition coverage. For this purpose, we use the COIL environment with three or four objects and random action sets G satisfying Assumptions and 3 those environments are given Appendix [.] In this setting, for each state w E W, we ran- domly sample na & [9] available actions to be used in the dataset. The results show that, soon as n, > 2, the method consistently recovers the correct group decomposition. Importantly, the same hyperparameters are used across all of these latter experiments.
7

---

Published as a conference paper at ICLR 2026
5.3
DISENTANGLEMENT
FL.
(b) FLP
COIL.
(d) COIL3
(c) 3DShapps
1.00
0.50
0.00
0.58
0.5
0.23
0.2
0:21
Lamptttts
Odi-io
6.8
0.00
sae
B- B-VEE
Inde
Mod
DCI
SAP
MIG
Figure 6: Median of disentanglement metrics. Vertical lines indicates the 25th to 75th percentile
To evaluate the disentanglement we use the same environments as before, the disentangled results are shown Figure [ For more clarity we do not present the disentanglement of unsupervised methods as they perform significantly worse than LSBD methods. Detailed results including unsupervised methods are available in Appendix H.4
The first observation is that all methods perform poorly in MIG and SAP as these two metrics require each ground truth factor of variation be encoded in unique dimension. However, linear disentanglement mostly requires features to be encoded in at least two dimensions as discussed in Appendix
F The only exception is COIL2 221222m as the permutation group 62 be encoded in only one dimension with our method. The second observation is that our method performs almost perfectly for the other metrics and yields a disentanglement comparable to the supervised method -SBD-VAE.
5.4 LONG-TERM PREDICTION
(a) COIL2
We aim to investigate the effect of disentanglement long-term prediction accuracy. To this end, we use the trained models to pre- dict a final observation given an initial observation and sequence of actions. For the COIL2 dataset, SOBDRL fails to consistently learn disentangled representation. We therefore separate the seeds into two groups: those where disentanglement is achieved and those where it is not. On the COIL3 dataset, SOBDRI. is unable to dis- entangle the representation at all, as the method is not suited to permutation-bant based symmetries. We also omit the results of LSBD- VAE* on COIL3, as it fails to consistently produce accurate predic- tions even for single-step transitions.
10-1
20
10-4
Sequence Jength
(b) COIL3
20-4
Figure 7 shows the prediction error function of the sequence length. We observe a drop in the prediction error of SOBDRL on COIL.2. This behavior comes from the fact that disentangled ac- tions of SOBDRL are SO(2) rotations. In one of the seeds, the action that swaps the two objects has larger angular error than the other actions, causing the corresponding latent dimensions to diverge first. However, due the cyclic nature of SO(2), the ac- cumulated angular eтrors eventually cancel out, completing full rotation and temporarily restoring the correct latent representation.
20-1
107
101
Sequence
LSBD-VAF
L.BD-VAF*
SOBDRL (entangled)
SOBDRL (disentangled)
GMA VAE (ours)
VAB (ouns)
Figure 7: Median of long- term prediction error, the shaded area indicates the 25th to 75th percentile
Overall, three types of behavior emerge from the results. First, en- tangled self-super methods (A-VAH and SOBDRL) achieve good short-term predictions but quickly diverge as the sequence length increases. In particular, the A-VAE curve ends early because the latent representations even- tually diverge to NaN values. Second, disentangled self-supervised methods (GMA-VAIE SOB- DRL and LSBD-VAE") achieve significantly better long-term predictions. Finally, the supervised method LSBD-VAL achieves perfect prediction performance regardless of sequence length. This is explained by the fact that, with access to ground action matrices, the model satisfies exactly AgAg' Agg's making multi-step prediction no more difficult than single-step prediction.
8

---

Published as a conference paper at ICLR 2026
5.5 GENERALIZATION
To assess how disentanglement impacts generalization, train each model COIL2 and COIL3 using restricted datasets. We first consider the independant and identicaly distributed (iid) setting, in which the training and test follow the same distribution. Specifically, for each state, uniformly sample n.a = |///2 actions to include in the training data. The second experiment assesses the out-of-distribution (ood) generalization capabilities of the models. In this setting, the training set is restricted to transitions in which only the right-most object is allowed to rotate. We evaluate the prediction eтrоr on both seen and unseen transitions, the results are reported in Table[]use Jusin the format seen unseen prediction error. We highlight in bold the methods for which the error increases by less than 5% between seen and unseen transitions. In both experiments, we observe that all disentangled methods generalize well, while most entangled methods exhibit poor generalization, particularly in the ood setting.
Table I: iid and ood prediction error, the format used is seen unseen prediction error
idd
ood
COIL2
COIL3
COIL2
COIL3
LSBD- VAE
1.8e-5/7.9es
1.le-4 / 1.1e-4
7.6e-5 / 7.6e-5
9.9e-5/ 9,9--5
LSBD-VAE
8.7e-5 / 8.8e-5
8.8e-5/8.8e-5
Disentangled
1.7c-4 1.7c-4
5.1c-5/
SOBDRL
Entangled
1.7e-4 / 3.7e-3
2.5e-4 2.5e-4
5.7e-5 47.02/0..22
2.5e-4 / 0,01
GMA-VAE
6.1c-5 / 6,2c-5
1.cc4/1.1e
6,2e-5 / 6,2e-5
1.c-4/1.1e4
A-VAE
7.7e-5 7.8e-5
2.9c-4/8.7e-4
6.7e-5
2.9c-4/0.0 0.05
5.6
LIE GROUPS
Here, aim investigate how proposed method can be ex- tended to continuous groups, such as Lie groups. The action clus- tering procedure of Step 2 cannot be applied to continuous groups since it relies on a finite clustering process. Nevertheless, when the group decomposition is assumed to be known, GMA-VAL still be used to learn a disentangled representation, as the proof of Theorem can be adapted continuous symmetry groups:
(a) MPI3D
1.00-
0.75
0.50
0,25
HAR
OMA-AA VAE (ourg)
Theorem 3', If Asssupttions Aand hold, the datasel contains all transitions, and the monoid generated by each Gy is G2, then the encoders that minimize the GMA-VAE loss are LSBD representa- tions with respect to (W,b, Gt).
(b) Noisy MPI3D
To empirically validate this claim, we consider the MP13D dataset, in which actions correspond continuous rotations of robotic arm. The symmetry group is modeled by SO) x (2),), each sub- an axis of rotation of the arm. The action representation p + p(g) is implemented as neural network, while the rest of the GMA- VAE architecture remains unchanged. As reported in Figure Bal GMA-VAE achieves performance com- parable to SOBDRL and outperforms HAE.
1.00
0,75-
each
0.50
group corresponding
0.25-
HAE
GMA VAR
BOBDRE
towrs)
To evaluate the robustness of GMA-VAE to noise, we introduce Gaussian noise with a standard deviation of 2T/115 to the actions. The results, presented in Figure 8b indicates increase in the SAP metric, which expected since low SAP value is required for a well-disentangled LSBD representation. For the remaining metrics, GMA- VAE performs at least as well as the other methods, suggest- ing that it exhibits greater resilience action noise.
9-VAE DCI
Inde SAP
Mod
MIG
Figure 8: Median disentan- glement metrics on MPI3D

---

Published as a conference paper at ICLR 2026
6 ALTERNATIVE REPRESENTATION LEARNING PARADIGMS
This section shortly reviews three other related but different paradigms for representation learning. First, the causal representation learning approach (Schölkopf et al. 202I) proposes to ground the latent variables in the causal generative processes of the environment and seeks representations that correspond to underlying causal factors and their relations. Several identiiability results have been derived in this framework, relying on different assumptions, for instance regarding the available actions (or interventions) (Brehmer et al., (022), the structure of the causal graph (Lippe et al.) 2023), prior knowledge of the intervention targets (Lippe et al.) 2022). or other inductive biases such as the sparsity of the causal graph (Lachapelle et al. 2022), or the transferability of causal representations (Bengio et al. 2020). Symmetry- based and causality-basee disentanglement share some similarities (mathematically grounded, exploit interventions actions) but have very different assumptions, justifying our choice not to include this framework in our comparisons.
Another line of work is object-centric representation learning, where the goal is to learn a factorized representation of objects and optionally their underlying dynamics. While promising, these methods often rely either explicit structural assumptions strong inductive biases that closely tied to the nature of the observations, most notably images. In particular, they frequently exploits the spatial organization of visual scenes, encouraging representations that capture localized and compositional object structure (Zhu et al. 2018 Greff et al. [2199] Locatello et al. 2020 Kipf et al., 2022). More- over, the question of disentangling the different features representing each object often left aside. This field of research is not as focused on idenfiability proofs, but has shown strong empirical results in more complex and realistic environments.
A third line of work leverages group-equivarian neural networks, which exploit underlying sym- metries. Classical approaches impose symmetries explicitly by designing architectures that are equivariant by construction (Cohen & Welling, 2016 Worrall et al.) 2017 Weiler & Cesal 2019 Dehmamy et al. 202T). However, these architectures are typically tailored to geometric transfor- mations in the observation space (e.g., rotations translations). More recent methods aim to learn cquivariant neural networks with respect to more general symmetries, primarily by incorporating an additional equivariant loss term. In most cases, however, least part of the group structure is assumed to be known (Mondal et al. 2022, Park et al.] [202]] Shakerinava et al. 2022] Jin et al.) 2024). While such symmetry-based approaches enhance the structure of the latent space and often improve sample efficiency and generalization, they do not explicitly focus on learning disentangled representations.
7 CONCLUSION
We introduced two independent algorithms: action clustering method based A-VAE, which provably recovers the ground-truth symmetry group structure, and a symmetry-basee disentangled representation learning method, GMA-VAE, which achieves performance comparable LSBD- VAF, even though the latter assumes prior knowledge of the action representations. Both of our methods rely on a strong assumption which requires the available actions to be disentangled. How- ever, to the best of our knowledge, related state-of-the-art LSBD approaches also implicitly de- pend on this assumption to consistently learn disentangled representation. While this restricts the applicability of the method to certain environments, it enables theoretical guarantees for both the action clustering and the disentanglement process. We further evaluate LSBD representations on downstream tasks and show that disentangled representations lead to significantly better long-term prediction performance and generalization, particularly out-of-distributtio scenarios.
A limitation of our approach compared to existing methods is that the full pipeline requires train- ing two neural networks from scratch. A future work would be to initialize GMA-VAL with the pretrained encoder from A-VAE, or develop and representation learning steps into a single optimization process. Another limitation is the lack of more realistic experiments that explicitly challenge the assumptions of the LSBD framework and our own hypotheses. Investigating the behavior of our method under such conditions constitutes an important direction for future work.
end-to-end method that unifies the action clustering
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All the previous results are reproducible using the code on GitHut It includes all necessary com- ponents to generate the datasets, the training procedures with the same hyperparameters and initialization seeds, and reproduce the figures.
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning; A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3798):1988-1828. 2013.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Sebastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. meta-transfer objective for learning to disentan- gle causal mechanisms. In International Conference on Learning Representations, 2020.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal repre- sentation learning. Advances in Neural Information Processing Systems, 35:38319-38333 2022.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https///ghhlllccomm//eeppn//llhh 2018.
Hugo Caselles-Dupré, Louis Annabi, Oksana Hagen, Michael Garcia-Ortizz and David Filliat. Flat- land: a lightweight first-person 2-d environment for reinforcement learning. arXiv preprint arXiv: 10, 2018.
Hugo Caselles-Dupré, Michael Garcia Ortiz, and David Filliat. Symmetry-based disentangled rep- resentation learning requires interaction with environments. Advances in Neural Information Processing Systems, 32, 2019.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, volume 29, 2016.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- on machine learning, pp. 2990-2999. PMLR, 2016
Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems, 34:2503-2515 2021.
Sunny Duan, Loic Matthey, Andre Saraiva, Nicholas Watters, Christopher Burgess, Alexander Lerchner, and Irina Higgins. Unsupervised model selection for variational disentangled represen- tation learning. arXiv preprint arXiv: 1905. 12614, 2019.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation disen- tangled representations. In 6th International Conference Learning Representations, 2018.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International conference on machine learning, pp. 2424 2433. PMLR, 2019.
https: thub. www.maa.aaa.cccccoo.oooooa.eeeaa htt.m//rtthn///em gmavae

---

Published as a conference paper at ICLR 2026
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey. Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:/8/2.0220 2018.
Chester Holtz, Gal Mishne, and Alexander Cloninger. Evaluating disentanglement in generative models without knowledge of latent factors. In Topological, Algebraic and Geometric Learning Workshops 2022, pp. 161-171. PMLR, 2022.
Lawrence J. Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2:193-218, 1985.
Yinzhu Jin, Aman Shrivastava, and P Thomas Fletcher. Learning group actions on latent represen- tations. Advances in Neural Information Processing Systems, 37:127273-12729 2024.
Hamza Keurti, Hsiao-Ru Ru Pan, Michel Besserve, Benjamin Grewe, and Bernhard Schölkopf. Ho-
momorphism autoencoder-leammigg group structured representations from observed transitions. In International Conference on Machine Learning, pp. 16190-16215. PMLR, 2023.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conference on ma- chine learning, pp. 2649-2658. PMLR, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 3rd Inter- national Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference Learning Representations, 2022.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pp. 428-484. PMLR, 2022.
Seunghun Lee, Sunghyun Cho, and Sunghoon Im. Dranet: Disentangling representation and adap- tation networks for unsupervised cross-domain adaptation. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 15252-15261. 2021.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference Machine Learning, pp. 13557-13603, PMLR, 2022.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence. pp. 1263-1273. PMLR, 2023.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. Advances in neural information processing systems, 32, 2019a.
12

---

Published as a conference paper at ICLR 2026
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference machine learning, pp- 4114-4124. PMIR, 2019b.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in neural information processing systems, 13:11525-1153, 2020.
Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi, and Siamak Ravanbakhsh. Eqr: Equivariant representations for data-efficient reinforcement learning. In International Conference on Machine PMLR, 2022.
Learning, pp.
Sameer A Nene, Shree Nayar, Hiroshi Murase, et al. Columbia object image library (coil-100). Technical report, Technical report CUCS- CUCS-006-96, 1996.
Matthew Painter, Adam Prugel-Bennett and Jonathon Hare. Linear disentangled representations and unsupervised action estimation. Advances in Neurul Information Processing Systems, 33: 13297-13307, 2020.
Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. In International Conference on Machine Learning, 2022.
David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffu- sion. Advances Neural Information Processing Systems, 33:17403- 17415, 2020.
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33:19727-19737. 2020.
Richard Rasala. On the minimal degrees of characters of sn. Journal of Algebra, 45(1):132-181, 1977.
Luis A Pérez Rey, Vlado Menkovski, and Jacobus W Portegies. Diffusion variational autoencoders. arXiv preprint arXiv: 1001vv9901.08000 108991, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286 PMLR, 2014.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems, 31, 2018
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)x61--63, 2021.
Mchran Shakerinava, Arna Kumar Mondal, and Siamak Ravanbakhsh. Structuring representations using group invariants. Advances in Neural Information Processing Systems, 35:34162-344 2022.
Loek Tonnaer, Luis Armando Perez Rey, Vlado Menkovski, Mike Holenderski, and Jim Portegies. Quantifying and learning linear symmetry-based disentanglement. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 21584-21608, 2022.
Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learn- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9677-9666 2024.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in neural
information processing systems, 32, 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 5028-5037 2017.
13

---

Published as a conference paper at ICLR 2026
Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards building group-based unsupervised representation disentanglement framework. In International Confer- ence on Learning Representations, 2022.
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y. Ng. Gunnar E. Carlsson, and Stefano Ermon. Evaluating the disentanglement of deep generative models through manifold topology. In Inter- national Conference on Learning Representations, 2021.
Guangxiang Zhu, Zhiao Huang, and Chongjie Zhang. Object-oriented dynamics predictor. Advances in Neural Information Processing Systems, 31, 2018.
Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group vae for disentanglement learning. In International Conference on Machine Learning, pp. 12924-1222 12934. PMLR, 2021.
A MATHEMATICAL BACKGROUND
Vector Subspaces. Let V be vector space over field R. A subset W S V is called vector subspace if it is closed under vector addition and scalar multiplication; that is, for all u, v E W and Å E R, we have u u V E W and Au E W.
Direct Sums of Subspaces. Let be vector space and let Wi, W2 s V subspaces. We say that V is the direct sum of W1 and W2, denoted V = Wi @ W2, if every v E V can be uniquely written as u = w1 + W2 with W1 E W1 and W2 E W2, and W1 o W2 = {0).
Eigenvalues and Eigenspaces. Let T - V be a linear operator on a vector space V. A scalar IR is eigenvalue of T if there exists non-zero vector V such that T(v) Av. The corresponding set of vectors {u E V T(b) = λu) is called the eigenspace associated with X, and is a subspace of V. The of eignvalues is called the spectrum.
Groups. A group is set equipped with a binary operation (æ, v) - xy satisfying the following axioms:
(Associativity) (zy) æyyz) for all x, y, G;
(Identity element) There exists an element e G such that re x for all G; (Inverse element) For every x E G, there exists w e G such that xx = x t e.
We often denote by G (e) the set non-identity elements of G.
Examples of Groups.
The cyclic group Z/nZ of integers modulo n.
The symmetric group G,, of permutations of elements.
The general linear group GL(V) of invertible linear transformations vector space V.
The special orthogonal group SO(n) of n x n orthogonal matrices with determinant 1.
Direct Product of Groups. Given two groups G1 and G2, their direct product is the group G1 G2 with the operation defined componentwise:
92) (h,, b2) = (g1h1,92h2).
Each group G, is referred to as a direct factor of G. By abuse of notation, G is often identified with the subgroup G {e2) S G, where e2 is the identity element of G2-
14

---

Published as a conference paper at ICLR 2026
Subgroup and submonoid generated. For subset S G. call the subgroup gen- erated by S noted (5) the smallest subgroup of G that contains S. We have (5) {s 22....... ... sk k N, E S {+1]}. We call the submonoid generated by S the set of all finite products of elements of S, and note it (5), = (8182 8k l N, 8, S}. If is finite, then for all 5 G we have (S) (S)
Group Representations.
A representation of a group G on a vector space V is map
P: p:GxV-V
such that for all g, h E G and v E V, by denoting g u for p(g, v) we have
and g. (h u) (gh)
Equivalently, a representation can be described as group homomorphism
p: G - GL(V), where p(g)(v) = p(g. v).
The representation p:G. GL(V) injective and only kernel, defined as ker(p) ;= {g G' p(g) Idy]. ). is reduced to the identity element {e). In this case, write p G GL(V) and refer to it as a faithful representation.
Direct Sum of Representations. Let (pi, Vi) and (02, 1/2) be two representations of group G. Their direct sum is the representation
P1 + P2 : GL(V
defined by
(ei ① (2) == (2)( The space Vi @ 1/2 is then said to carry the direct sum representation of G.
Symbol
Meaning
Symbol
Meaning
W
World space
2:
+ Z2
Direct sum vector spaces Injective map from A to B Spectral norm matrix A Cyclic group of order Symmetric group on elements
X
Observation space
A
B
X
Observation random variable
|
z
Latent space
2/nZ
Z
Latent random variable
Gn
G
Action Group
GL(V)
General linear group on V Special orthogomal group on V
GEG G
Available action set
SO(V)
b:W
Generative function
(S)
Group generated by S Monoid generated by set S without its identity element
h:X-2
Encoder function
(5)
f=hob:W-1 -
Latent generative function
G
p:G- GL(Z)
Action representation
'2: 2
Group action on latent space
"W:GxW-N W
Group action world space
Table 2: Table of symbols
B ELBO DERIVATION
We focus on the transition r (x., g, 2²). We have:
Po(X'|>') (Ho(2)), Diag(a)(22))3) (
Po.o(Z'(2x.) = ((pp(o))s(z)) Id)
(Z'('') (Nllaa")~((((sff (Ho((a)) Diagglasas""")))
We will use several times the fact that if Y N(p, Diag(a)?) then logp(y) - E, log a; ? cste with the element-wise division in the norm.
15

---

Published as a conference paper at ICLR 2026
Our initial goal is to optimize the log- seelikklihood log p(T) log P 9. log ( + est. Indeed, we consider that the model has no prior (or constant prior) on (x.g) and therefore focus optimizing the log- likelike log Pv.e (2' 1z, g):
log Pei x. g) 90(2' 1
log P xz x. = E (o=='()=)
20l2' x2)
log pp(x' |2', po (=
(x,g)
95(2' (²)) )
g0(=' |xd)
p(z' x² 2. g)
According to Bayes formula
log (= (') p(=' x², x1, log Purole' x.
= Ex--g((111))
+ ==/-90(='1==)
g0lzz (x))
+ (x
|logp(1' 2 22
Drz (gol=' x I') ,2, 00
DxL (9o(2' [d') x.))
+ Exrrqo(=1 |logpo(a' 22)]
According to Figure 3
Dxxt (go(2' (')))22 P 2z,
+ Exr~qu(=1)a)) [log |²)
The lower bound we have derived is composed of two lines. The first line corresponds to the KI. divergence between two multivariate normal distributions and thus has an analytical expression. We have (up an additive constant):
1 - (t) uo (x)) (2())]]++ log (x('))
-DEL (g0(z' xx') (F x,
The second line is equal to (up to an additive constant):
(a Σ log logooo=z)), x (=(22) 12
x0(2')
Putting everything together, we obtain:
logp(T) -
-==000000) Σ log (x)))
}
action part
x' - -o()) 2
Σ Elogoo(s)) -=l-go(=7p
reconstruction part
a0(2')
+C
C ACTION CLUSTERING ALGORITHM AND PROOF OF THEOREM 2
Let G G1 .... u GK G with Ak G. G denote the ground-truth decomposition of the available action set. Our objective to design an algorithm that recovers this decomposition based on da-
16

---

Published as a conference paper at ICLR 2026
C.1 ALGORITHM
Let A E RRdxd we denote by || = m || following algorithm:
the spectral norm. We chose the
Compute the variables
r max AlA the maximal spectral norm,
& (w., w") Agf(w) ))- f(w")l the action loss upperbound,
(1+ AM ri) the threshold
Start with unitary clusters: K |g] and &, {9:}
Iteratively merge the clusters and minimizing their distance
d(), 9,)
max 2
dc(g 94)
Stop whenever the distance is above the threshold 1).
What if the identity action belong the available action set G ? After a succesful convergence of the method, for all
we have de (e, g) R since g = g' e. As result, merged with another element at the first iteration and will not influence the following computations d(),, Gý). As be assigned to any subgroup, its presence does not impact the performance correctness of the overall method.
C.2 PROOF OF CONVERGENCE
We aim show that if Assumption are satisfied, the dataset contains all the transitions, W is finite and A- A-V loss converge toward its minimum, then clustering algorithm will necessarily find the ground-truth decomposition at some point of the training.
For clarity, we assume that every composition of de in Assumption 3 is of the form u""g'. The proves can easily be adapted for the other forms.
Proposition 1. Under Assumptions 2 and subgroup if and only if there exists m E [1, M] and u E G such that g = w""g.
different actions 9.g' G belong the same
Proof. The forward implication is given by Assumption 3 For the backward implication we distin- guish two cases:
If one of element is the identity action e, then e belong the same subgroup every action
2. If both elements differ from e, then according to Assumption 2 there exists a, and e such that G g' Gø and u" Ge. Therefore 9 Ge Mee and then G, n (Ge x Gb) * {c). we had a * b, this would contradict the direct decomposition of G.
the standard deviation of noises of A- A-VAE are fixed, then the loss for a transition (x,, g, x') is equal
>= (polc)llls -o(('') ? Eg-gg()> [1Ia -=0(')) [']
Unlike 3-VAE, which requires trade-off between the regularization and the reconstruction. this loss can have both the action loss and the reconstruction loss converging toward 0.
When the action loss converges toward zero, max/weg [[af(()) fg w)ll - as it is its upper-bound. Additionally, since the coefficients of the action matrices bounded in our implementation, is also bounded. As result, the term converges toward zero during training.
straightforwardly have
We also have the following result:
17

---

Published as a conference paper at ICLR 2026
Proposition 2. Under Assumption 7 if the reconstruction loss converges toward zero, then
= min
w/E [f(w) - f(w'') +00
Proof. Let h X Z be the encoder and d - X be the decoder. Let w1 * u2 e W with I; b(w;) and = h(x;), as b is injective we have 21 # 22. As the reconstruction loss converges toward 0 and the dataset contains all the transitions and therefore all the observations, we have
E Eg-N(2,,021) [| [/1;=d(==])) -
Let denote = (21 22)/2, A || 22/1, B(2 1) the ball of radius 1 centered = and V(d) its volume. Let denote p(R) (v2rt)j exp R the minimum value of the Gaussian density over a ball of radius R. We aim to show that if the reconstruction loss sufficiently low, then A must be sufficiently large for the contribution of the reconstruction over the ball to become negligible.
*
A
21 K
* 22
B
Since convergence in 2 in probability implies convergence in L in probability, for any > 0, at some point of the training there is
[ J2RRd
((2:2,,011 x; d(z) ee
We have
<
N( d(z)]ldz
N(zz; 1||2; - d(=)]ld=
JCB
A
+
||, d(=)]ldz
2
JEB
by definition of since for any B, || 2/ & ||z = +11++11 1| =|| < A/2+1
Therefore, by summing the two equations, we obtain:
2c 2 p
+ 1
(1 - d(())l + ||22 d(z)) dz
2
JZBB
[
> p
+1 1
I1 -22l
with triangular inequality
2
2CB
> p
+1
V(d) xi - x2ll
2
Using the definition of p and isolating A || 22|1, we obtain
18

---

Published as a conference paper at ICLR 2026
1 V(d)lkz x2
) -2
||=1 22 30 20 /
2 2
2( (22TT))
+o00
Therefore ||21- - 22 +00 as the reconstruction loss converges toward O. Finally, as W is finite, we have minwwt" w/ [[[(w) f(w")l +00
Consequently, at some point of the training, the inequality 8 2ny holds.
Proposition 3. Under Assumptions 2 and and only do ('.lgg))) <
3 if > 2n then g and g' belong to the same subgroup if
Proof. Suppose that g.g6 E G with * g' belong to the same subgroup, therefore, according to Proposition |
there exists 9 and e [1, MJ such that g = u""g g'. Let w W
[Ag - A Ag f(w)
< (w) - f(g - w)ll
+ If(" g' w) - Aaf(u g w]]
+ A 'S(ug' w) - A "W((g' wg]
+ A J(g' . w) AT Ag s(will
< (g) - f[g- w)
+ If(u g' w) Auf (o'("""" am 10gg/ w)]
+
+ III Amllll 11- 11 II( w) - Agg w)ll
+ ||| Aalll m2 -I w) Ag ((w)
r's
1=0
<
After applying expectation over w we find de (g (.
AM =
Let us now suppose that g e G do not belong to same subgroup, therefore, according to Proposi- tion ! for all u and m [1, M]
we have
E [g w) ff(um g' w)l
< flg w) f()l
+ [[) A Ag f(w]]
+ 11 Ag. - fu" g' w)ll
< E + +11AAf(ww)) AT Ag fl (w)
< AAg](w) A Ag [(w)] + Therefore [g,((w) - A Ag f(w) and m we get dalg,
> 8 - 1l > n, after applying expectation over w and min over
Therefore, at some point of the training, two actions belong the same subgroup if and only if their distance with repect to de is below the threshold 7). As a consequence, the clustering algorithm de- scribed previously successfully recovers the ground-truth -ruth decomposition, hence proving Theorem
In practice we found that using = a the fixed latent noise standard deviation as a threshold yielded better empirical results. Consequently, we this value for y in all our experiments.
19

---

Published as a conference paper at ICLR 2026
D PROOF OF THEOREM 3
the standard deviation of noises of GMA-VAE are fixed, then the loss for a transition (x, 9, x²) equals:
(xx,9,x') = Aprs ( + AAAT MacTlloo0))o000 mkp(x') (o( 1 Be'ngolt) |1le' mo(22))
) |e]
Let us suppose that G' is finite, that the dataset contains all possible transitions, and that the losses of EMMAVAE have converged to their global minimum of O. We aim to prove that the encoder is LSBD representation.
We cannot directly prove that disentangled with respect WW,b l Gε), as we cannot build a representation of an action that is not generated by the available action set. Instead, we aim to prove that the encoder disentangled with respect to (W, b, II.<g<>> with (Gx, being the subgroup generated by Gs. Similarly, we cannot prove that it is disentangled over all Z. Therefore, as done by Keurti et al. 20033 we restrict the latent space to V = span(f(W)) c Z
We proceed by proving that the learned representation satisfies all the criteria listed in Definition I
(1) There exists a group action : G x V
(g). Let g (9). as G is finite, we can write g g()).. ... g(() with Vi, gl e G. the group action is given by:
First, note that [l<)
9 ==== = p(g)= with p(g) = II Ā Agen
Note that the definition of p(g) depends on the decomposition of 9. which is not necessarily unique. Since any decomposition would be satisfying and G is finite, we can arbitrarily chose one decomposition for each g.
The fact that z a group action given thanks to the equivariance and is proven below.
(2) Equivariance holds: Vgg. w) (g) x W, z f(w) = f(g w w)
As the action loss is equal to zero we have:
W, 9. f(w) f(g w w)
Let g = [T, E (G) with (f E G and w E W. We apply recursively the previous equivariance to get g z f(w) f(g ww w), hence proving the equivariance over all (g) W.
We now prove that zz is indeed a group action thanks to the equivariance property. For all w E W and g.g" (G):
9 f(w) = f(g w EV
f(w) = slew w) = f(w)
2 (g.2 f(w)) = g 22 f(g w w)
f(g' 'w (g (g.ww)) w))
[((g'g) w www) w))
(g'g) .z (ww)
As ((W) generates V. for = E and g.g' E (9) we have:
g'ZZEV
E'z2=2
22 (g'22) 2) (g'g) z
20

---

Published as a conference paper at ICLR 2026
(3) There exists decomposition V 1/1 + VK and group actions "k Gk x Vj Vk such that:
(g1...... .... 9K) ' (21, : 2x) (g) 1 ....... '9 K 2k)
As the disentanglement loss is equal to zero, the masks are binary i.e. T E {0, 11d. Therefore the matrices A, for E 9 satisfy the block structure illustrated in Figure Additionally if E <(*) it can be decomposed into Ik: 9k ... 9k (nx) with 9k same block structure. Finally, for each g (g). -.1 9K) (9), Il p(gx) ) block diagonal up to permutation of the indices.
(6) 96 and therefore p(gx) π.. Ā of share the
Let us first find decomposition and group actions over Z. We take 2, spanfe; Tk. 1} with e the standard basis vectors, this choice reflects the objective of achieving disentanglement along Z and Eµ T O
the Cartesian axes. Hence have Z 21
Additionally we take 9k " 2 p(g we have p(gx)zx Zz as p(gh) is the identity on the complement subspace of Zy. We therefore have for each g = (g1..... . (9) and E Z:
9 2 = p(g)z
p(9)
Σ 2k
Z
Σ p(g)=K
pkg)
R1
p(gs)
21
Σ
p(gk)2k
Figure 9: Matrix representation of p(g)= p(gx) Zt. here the two first dimensions corresponds to En and the re- maining two to Zz.
Σ Ok "k 2/
(0) 21. 9K KK 2x)
Let us prove that the decomposition and group actions restricted on V are satisfying. Let's take Ve span{
w W) V Ze with f'(r) the projection of f (ww Zg, therefore the Ve are in direct sum. Moreover we have for all to E W, f(w) = -x ((w) with f(w) E V and therefore V V1 + VK.
Additionally, for all g E (9) and f(w) E V we have g f(w) = [[g w) meaning that for all component have 9k 'k 24 f(g 'w w)k E Vk. Finally, as previously done for zz it can be shown that : G& x V/_ V4 are group actions.
(4) The representation h is injective:
As the reconstruction loss equal to zero and Assumption Tholds, the distance between the encod- ing of two world states has lower bound as shown in the proof of Proposition 2
(5) The group action is linear:
As highlighted by Keurti morphism and we have g 2 Pv g This can be proven similarly to the argument used to show that z is group action. Hence proving the representation is linear V.
(2023). the restriction of p on V written py G GL(V) is
E MINIMAL LINEAR REPRESENTATION DIMENSION
Let G group, seek the minimal dimension such that there exists injective morphism into invertible matrices 5(G) = min/d 3p : G + GL( We also seek (G) the minimal dimension for special orthogonal matrices (S) min(d 3p - SO(d)).
21

---

Published as a conference paper at ICLR 2026
We get the following results:
G
8
8s0
Z
2/22
1
Z./nZ,
2
82
G3
2
64
3
G n 2
5
72 -
1
71
or m
E.1 FINITE CYCLIC GROUP IE ROTATION GROUP
Proof. Let G a finite cyclic group of cardinal n > 2 i.e. G = 8/nZ. There are two cases to consider:
2: Then G {e, g} with g² = e. Therefore 8(G) because the isomorphism p
(1); + (-1)} is satisfying. Furthermore 6so(G) > as SO(1) = {()} can only express the
trivial group.
n 2 Suppose o(G) = 1, then for each element g E G there exists a scalar Ag E R such that p(g) (Ag). Therefore for all and k > O we have g E G and then X E {gg 1 E G} which is finite set. Therefore Ag 1). consequently for all g G. p(g2) (22) = (1) and then g e.
This is contradictory, therefore 8(G) >> 2.
We show that 5(G) 2: let h be a generator of G iee for all g E G, there exists kg Io, nI such that h g. The following rotation matrix a satisfying morphism p(g) cos(2rkk/
sin sin(27kg//1) sin(2nkg//)) cos(2 ): Finally 5(G) and therefore 8so
E.2 INFINITE CYCLIC GROUP
Proof. Let G be infinite cyclic group Le. G = Z. Let h be generator G, therefore for all g G, there exists ky such that = ht For any x > 0, x * 1, the representation p E G + (a*a)
satisfying. Therefore 5(G)= (G)==l
As previously, we have (so(G) > and for any $ 2nQ the representation p(g) SO(2) satisfying. Finally dso(G) =
(kg( -in(k sin(kig)) ()s(kig)) )
E.3 PERMUTATION GROUP
Proof. Let = Gn with n > the permutation group.
We know that there exists injective morphism p Gn O(n 1) thanks to its standard representation, therefore 8(G,)) & n n 1. Furthermore we can inject O(n - 1) into SO(n) det pl ). And then (so( n.
Generalities:
p(g) 0
using g
2: We have G2 Z/2Z, previously 5(62) 1 and 65o 2.
n = 3: Similarly to previously 5(G²) > and (G3) > 1. Moreover there is no injection G3 - SO(2) as G3 would be commutative. Therefore d(Gs) 2 and 6so(Ga) = 3
n = 4: G4 is isomorphic to a subgroup of SO(3) as can be seen as the symmetry group of the tetrahedron. Moreover their is no injection G, - GL( G4 would be cyclic dihedral. Then
5(Ga) = 65o(Ga) = 3.
5: The minimal dimension for linear representation in C of G, is n Rasala 1977
Therefore the minimal dimension in R is at least n 1 i.e. 8(G) 2 n Consequently 0 ((G,)
1 and 65o(G, (..) (n 1,
22

---

Published as a conference paper at ICLR 2026
F BLOCK-DIAGONAL PARAMETERIZATION IS NOT ENOUGH FOR LINEAR DISENTANGLEMENT
Here, aim to show that using structured action parameterization. as done in SOBDRL Quessard ct al., 2020). LSBD-VAL and HAE (Keurti et al. 2023 is not sufficient to obtain disentangled representation, even when applying the regularization used in SOBDRL. Assume that the underlying group action is composed of two cyclic direct factors, i.e., G G1 x G2, and that G is isomorphic to subgroup of (2) S2( Our goal is to learn a disentangled representation of this subgroup. Moreover, we require the learned representation to be injective as any constant morphism would be disentangled. A block-diagonal-based method may use this prior knowledge by parameterizing
the action matrices with g G - R(0g, 0²) R2(0g) 0
with R2(0) being the R2((,)
rotation matrix of angle 0.
Suppose that the unknown ground-truth group decomposition consists of two cyclic direct factors with n and m elements, i.., G Z/nZ x Z/mZ. Let 91 and 92 denote generators of each respec- tive factor. For the action representation to be disentangled, p(g)) = (2m/n, 0) and p(g2) = (0, 2r/m), so that each generator only affects single latent sub- space. However, when using this type of parameterization as a disentanglement criterion, even with the regularization term introduced in SOBDRL, it remains impossible to guarantee disentanglement of the action representation. Below, we present examples of entangled representations that satisfy the imposed parameterization but fail to be disentangled.
would ideally want something like
Case 1: The representation such that p(91) R(27/n, (12 0) and p(g2) R(22//n /m, /rmmm is an injective bomomorphism, it was encountered during our LSBD- "ED experiments. This represen- tation entangled with respect the LSBD framework.
Proof. Suppose there exists a decomposition I Z, @ Za satisfying the disentangled definition, then p(g2) would be the identity function on Z1 i.e. for all 21 E Z1. p(g2) 1 21. If 2 * {0) then is an eigenvalue p(g2) which is impossible as its spectrum is 2 2i//m). Therefore
21 {0), the same reasoning can be applied on Z2, therefore Z₁ + Z2 {0) R².
Case n If n and m are coprime numbers, then the representation such that p(g)) ) = R(2x/n.0) and p(g2) [(27//n //,, 0) is injective as
Z n.Z
2
Z
G
mZ
nmZ
This type of representation was encountered with SOBDRL as it minimizes its disentanglement criterion. However this representation is entangled with respect to the LSBD framework
Proof. Suppose there exists decomposition F 21 + Z2 respecting the definition, then p(g1) would be the identity on Ze i.e. for all 22 E Z2. p(g1)(() )22 = 22 and then Z2 a subspace of the eigenspace of p(g1) associated with the eigenvalue 1. This eigenspace corresponds to the two last
dimension: R44 i.e. Zg c {0] x {0) x R². Similarly for p(g2), we have Z₁ c {0) x {0) x R² and therefore R4 Z1 Zz.
Case n If n = m = pq composite number with p and coprimes, the representation
P(g1) R(2t/p. 2T/9) and p(g2) R(27/ q, 27 h/p) is injective. It is like switching the two Z/qZ components of each direct factor.
G1
G2
Z
Z
Z
G
2
9
pz
q2
As for case 11 this representation is entangled with respect the LSBD framework
23

---

Published as a conference paper at ICLR 2026
G PROOF OF THEOREM 1
Reminders:
W the world state set
X' the observation set
Z Rd latent space
b: W - X' the observation function
h: X' - the encoding function
f = h b: W + Z
h disentangled with respect to (W, b. I G&) if:
There exists 2 G x 2 - Z
Equivariance holds: Vg E G, w W. g 2 [(w) f w w)
There exists Z 2 @ ... + 3k and * Gk x 2x - 2k such that
(91 ....... 9K (2) 2 (91 1 21 9K K 2K)
is injective
Suppose there exists SBD representation with respect (W,b, II G&), aim prove that can build <W, B. 1I G+) from (W, b, Le G₂> such that (1) the SBD representations with respect to (W, F, Hh Gk) are exactly the SBD representations with respect <W, b, II G&), (2) the observa- tion function i is injective and (3) (W, b, 1I. G&) and (W, T, Ile Ga yield the same sensorimotor interaction with the agent, and thus indistinguishable.
G.1 INTERACTION EQUIVALENCE
We would like to reduce W to indistinguishable cases, we therefore define the following equivalence relation:
Definition 2. Two world states w1 and u2 are called interaction equivalent if and only if:
Vg G, b(g "w wi) big 'w W2)
We denote this relation between the two world states as w1
This definition implies that two states wi and Wy that yield the same observation but can be dis- tinguished by interacting with the environment are equivalent. Therefore, it does not cover situations such as object occlusions. Based on this interaction equivalence relation, we define new composed of the equivalence classes:
Definition 3. We call interaction equivalent world state set and denote by W the set of interaction equivalence classes.
W [[w] w E W)
where [w] denotes the equivalence class of according Definition
We now turn to defining appropriate functions using the interaction equivalent world state set. Proposition 4. If wi ~ W2, then b(w,) b(w2).
Proof. This can be easily derived from Definition 2 and taking = e the identity element.
This proposition allows us define new observation function on the interaction equivalent world state set W:
Definition 4. We call interaction equivalent observation function and denote by b the function b W X such that:
V[u] W, B([w])) B(w)
24

---

Published as a conference paper at ICLR 2026
Proposition 5. If w1 wy. then lgg 9 w w1 g w W2-
Proof. Let wi and w2 be two world states from W such that w1 12 Let E G be a symmetry. We aim prove that g "w w1 9 "W w2-
Let g' & G be a symmetry. Since G a group. it follows that the composition I'g G. According Definition
b(g's g ww w1) b(g'g w w2)
And, by definition of a group action:
b(g "W (g w w1)) b(g' ww (g w w2))
According to Definition 2 we thus have "w w1 ww W2.
This proposition allows us define new group action on the interaction equivalent world state set W:
Definition 5. We call interaction equivalent group action and denote by "w the function "wp : G x
W W such that:
Vg E G, G[w[ E W.g w [w]] [g w w]]
Proposition 6. "w is group action.
Proof. We need to prove two properties:
1. v[w] E W. e w [w] = [w]
2. V(g.g)) G, M[l W. ww w [w]) = (g'9) ww [w]
We start with the first property. Let [w] E W be an interaction equivalence class. We have:
[w]
[e-w w]
according Definition
w
[w]
since "w is group action itself
For the second property, suppose (g.g') and [w]] E W. We have:
g w (g
w [w]) g ' lg 'w w] [gg' "w (w-w w)]
according Definition 5
according Definition
[(g'g).w] (g'9) w [w]
since 'w a group action according to Definition
In summary, we have proposed an equivalence relation for world states that allows us define new world state set regrouping equivalent states, as well as an observation function and group action for this new world state set.
G.2 NECESSITY OF INJECTIVITY OF THE OBSERVATION FUNCTION
We now address the main point of this appendix, which is to show that if h is SBD representation with respect to (W, [T, Gi>. then has be injective. We start by first showing that SBD-ness is implied in the interaction equivalent world.
Proposition 7. If is SBD with respect to <W,b, [T, Gi), then is SBD with respect to (T.
Proof. According to Definition I SBD-ness with respect to (W, b, 11 Ga) requires four properties:
1. There exists group action 22 G x 2 2,
25

---

Published as a conference paper at ICLR 2026
2. Equivariance holds: Vg G, [w] W, have g z h b([w]) h b(gg w [w]),
3. There exists decomposition 2 21 x 2x and group actions Gk. Zz Zk such that
(g). . 9K) zz (Z1y..., zR) (g) 11 21: .1....... '9 KK zK),
4. h is injective.
Properties and are ensured by the fact that SBD with respect to (W,b, [ Gk). We thus just need to prove the second property. Suppose E G and [w] e W, we have:
2 h e b(w) 2. z h b([w]]
according to Definition since is SBD (cquivariance property)
h o b(g w)
ho b w w()
according to Definition
= o blg w [w])
according to Definition
Proposition 8. Reciprocally, if is SBD with respect (W b, I1 G+), then is SBD with respect to to, II. G.).
Proof. As in the previous proof, suffices to establish equivariance in order to prove disentangle- ment with respect to (W,b, 1.111.1000 G.>. Let g G and E W:
2 ho b(w) g h b([w])
according to Definition 4
= h o blg w [w])
since is SBD (equivariance property)
h o -w w])
according to Definition
= h o b(g-w w)
according to Definition
We can now introduce the main result of this appendix:
Theorem I If h is SBD with respect (W,b, I1e G1) then 6 is injective.
Proof. Suppose h is SBD with respect to (W,b, ITg G&). According to Proposition 7 h is also SBD with respect to (W, b. le G&).
Let us assume that 5 is not injective and show that it leads to contradiction.
Since is not injective, there exist [wi] # [w>] such that b[[w 1) = b([w>) Since [wi] * [wal then E G be the symmetry such that b(g ww wi) * big ww w2). On the one hand:
b(g w w1) * b(g w22 = b(lg w wil) * b(lg ww w2])
according to Definition 2
h b(lo ww wi]) h b(ly ww wa])
since h is injective
z h M([wi] # g zz h b( [wz])
equivariance property
On the other hand, since ([[w)]] } 7( ([w2]), then g z h b([wz]) contradiction and thus is injective.
9- ho b([wz]). We have
26

---

Published as a conference paper at ICLR 2026
From a prediction perspective this result is quite intuitive: for injective encoder to allow accurate prediction of future observations from I and g, it is necessary for the observation to be unam- biguous. From disentanglement perspective this is strong limitation, means that all the features to disentangle have to be constantly observed by the agent.
We use this result justify the introduction of Assumption to derive our algorithm for the group decomposition. Since have shown that LSB disentanglement is impossible when b is not injec- tive, it is a necessary assumption to assume.
H ADDITIONAL RESULTS
H.1 PREDICTION BASED MODEL SELECTION
For unsupervised disentanglement learning, choosing the model with the best disentanglement with- out knowing the ground-truth features is crucial and often difficult task. Several solutions have been discussed in the state of the art to adress this issue for purely unsupervised methods (Duan jtt al. 2019; Zhou et al. 2021 Hollz al. 2022). they mostly rely learning batch of models with different hyperparameters or initialisation and selecting the ones sharing some defined proper- ties. This model selection issue arises in our method at different stages; the action clustering resulting of Step 2 and the representation learned by GMA-VAE.
As we are sele-super framework, we investigate whether it possible to select the model achieving the best disentanglement (measured here with the independence metric (Painter et al., )020h) solely based on the prediction error. Similarly, we aim to determine whether it possible to identify which A-VAE model results in the correct action clustering in Step 2. To evaluate clustering quality, we use the Adjusted Rand Index (ARI) (Hubert & Arabie 1985). metric to be maximized that equals I if and only if the predicted clustering exactly matches the ground-truth partition (up to a permutation).
For multiple hyperparameter configurations and random seeds, we plot for the COII.2 experiment the metric of interest as a function the prediction error in Figure I0
(a) ARI of action clustering according to A- VAE pre- (b) Independence according prediction
diction error
1.0
A-VAE
0.8
0.8
Adjusted Rand
0.6
Independance
0.6
0.4
0.2
0.2
SOUDEL LABD-VAEE
0.0
GMA
10-3
10-4
10-3
10-2
Prediction Error
Prediction Error
Figure 10: Metric of interest as a function of the prediction error
For A- VAE (left), we see that for almost all of the models, either the prediction performs well and the action clustering retrieves the correct partition, either the model cannot predict correctly and the action clustering retrieves clustering only composed of singletons meaning that the ARI is equal to zero. However, few models out still has good prediction error with a low ARI. It is therefore possible to train batch of A VAE models with different hyperparameters and initialisation seeds and take the action clustering that is retrieved by the majority of models achieving the best predictions.
27

---

Published as a conference paper at ICLR 2026
For the disentangled representation learning algorithms (right), then notice that for the LSBD- VAE" and SOBDRL, they are few models that achieve a good prediction without being dis- entangled. However the model with the best prediction is always disentangled with a slight margin. For GMA-VAF, a good prediction always implies a disentanglement almost equal to 1.
H.2 ASSUMPTION 2 FOR OTHER ALGORITHMS
This section aims to demonstrate that, although not explicitly stated, related LSBD algorithms implicitly rely Assumption 2
(action disentanglement) some extent.
First, Forward- VAE directly requires disentangled actions, its action matrix parameterization is identical to GAE, with the vectors т {0, 1 provided as prior knowledge.
To evaluate the extent to which SOBDRL and LSBD-VAEF depend on this assumption, we modify the COIL2 experimental setup by progressively relaxing Assumption 2 In this modified setting, each action corresponds to an element of the product of k > distinct direct factors. We then measure the degree of disentanglement in the learned representation using the Independence score, as a function of the entanglement level k. The results, shown in Figure [I] indicate that the In- dependence score decreases as increases, and rapidly approaches score obtained by A- A-VAE with fully entangled actions (represented by the green dotted line). These findings support the claim that Assumption 2 i.c., h 1, in fact necessary for these algorithms to consistently learn disentangled representation.
A-VAE
1.0
SOBDRL
Independence
VABD
0.9
0.8
0.7
3
Action Entanglement
Figure 11: Disentanglement according to action entanglement
We also evaluated the disentanglement those methods the other metrics in Table 3 As can see the disentanglement strictly decreases as actions are more entangled except for the MIG metric.
Table Disentanglement with respect action entanglement
k
Beta-VAE
Inde
Mod
DCI
SAP
MIG
1.00 .00
'99 .00
1.00
.00
1.00 $ 00
.77 E .04
.06 E .03
LSBD-VAF
2
.99 +
.84
-03
+ .04
.82 .05
.14 + .02
+ .03
3
.78 + .05
.69 + .03
.59 + .05
.62 4 .05
.03 + .01
.04 + .01
1
99 + 01
94 002
.94 + .04
.8 .03
.39 00
.20 .07
SOBDRL.
2
.94 .03
.33 .02
.81 + .02
.70 + .06
.30 .06
3
.74 055
+ .05
.48 .05
.06
.19
28

---

Published as a conference paper at ICLR 2026
H.3 LATENT MANIPULATION
Here we aim to show that the learned disentangled representation allow for latent manipulation. In the discrete case, select images of the 3DShapes dataset that we encode with GMA-VAE, then we construct several latent representations by mixing the different latent factors of the two images. Finally, we decode those new latent representations obtain new images. The results are shown in Figure 12 We can see that by mixing the different latent factors, we can control the different features of the decoded images.
For the continuous case, we select one image of MPI3D that we encode with GMA-VAE, For each feature (vertical angle and horizontal angle). we successively apply rotation of fixed angle the corresponding latent factor while keeping the other latent factor unchanged. The decoded images are shown Figure 13 We can see that by rotating the latent factors, we can control the rotation of the object in the decoded images.
(b) Decoded images latent mix
(a) Original images
Figure 12: Decoding mixed latent representations of 3DShapes images
H.4 DISENTANGLEMENT METRICS
We show all the disentanglement and prediction error results of the different methods on the different datasets in Table 4 to 14 the values reported correspond to the five seeds with the 68% confidence interval. The random method corresponds to a randomly initialized neural network that not trained.
29

---

Published as a conference paper at ICLR 2026
(b) Decoded images from horizontal angle rotation
Original image
Decoded images from vertical angle rotation
Figure 13: Decoding rotated latent representations of MPI3D image
Table 4: Disentanglement for FlatLand with rotation colors
Beta- Beta-VAE VAE
Inde
Mod
DCI
SAP
MIG
Prediction
LSBD- VAE
meam
1.00 1 .00
.96
.00
1.00 I .00
.98 I .001
.53 I .05
.10 I .04
1.6e 09 4.8e 10
L
1,00 .00
_02
.90 1. .05
.87 L .07
40 1 .03
.07 .03
9.6e- 5.3e-10
hest
1.00
.97
1.00
.98
.02
3.3e-9
SOBDRL
03
.78 02
.85
.33 + 09
.28 + .04
.10 .02
2,7e-5
2.4e-5
best
99
84
J77
.6
44
.02
1.6e-8
GMA-VAE
mean
+ .00
1.00
+
1.6e-10
2.9c-11
best
1.00
1.00
1.00
1.00
.11
.00
1.30-10
A-VAE
mean
.89 1 .05
.84 .01
= .01
.44 1: .03
.11 1 .02
.05 1
6.6e-11
1.7e-11
.55
82
.77
.05
3.0e:11
B-VAE
688 i: 06
.77 011
.88
= .03
.22 1: .05
20 1 .04
.13 i 003
DIP-VAE
.66 +: .03
77 .01
.12 + .02
.11 + .01
14-t
DIP-VAL VAE
.84 .04
.62
77
.17 1: .01
.18 .04
.05 1
Random
E 133
04
.05 1 .01
M5 000
.44
Table 5: Disentanglement for FlatLand with permutation colors
Beta VAE
t Inde
t Mod
DC1
1 SAP
MIG
Prediction
LSBD-VAE
I
1
002
best
1.00
997
1.00
.88
.59
02
2.6e
LSBD-VAE
mecin
1.00 = .00
.98 00
= .000
.01
.4 i .06
.08 i .02
3.3e-6 i: 2.8e77
1.00
1.00
1.00
.00
2. 2.70-6
SOBDRL
menn
1.00
.00
.74 01
.79 t. 01
50 .02
.25 t .03
.08 t: .02
3.6e-3 t 8.30-7
GMA-VAF
meam
1.00
.00
.99 .00
1.00 :: .00
.99 001
.51 + .04
.08 +: .01
5.8e-6 3.
1.00
1,00
1.00
1.0c-6
.87 06
.99 .04
92 04
.47 .08
.15 + .05
.09 .01
1.le-6 1.9e-7
A-VAE
best
1.00
1.00
.47
.07
1.0c-6
VAE
Factor: VAE
.77 + .05
61 01
.79 + 02
13. m0
.17 04
.65 +
DIP-VAET
.0
.11
.10+
.06 16-+.0 .02
VAP-VAE II
.71 .01
01
111
.104 .02
.04 .01
Randon
.59 11
05
19 01
103
30

---

Published as a conference paper at ICLR 2026
Table 6: Disentanglement for COIL2
Beta-VAA
Inde
t Mod
1 DCT
SAP
MIG
Prediction
.00
011
8.2e.5
LSBD VAE
best
1,00
99
1,00
92
43
.00
7.4c-5
LSBD-VAE*
meea
1.00 1 .00
.9 1. 0000
1.00 I .00
97 1 00
.25 .05
.N .03
9.4e-5 3.1e-6
1,00
.9
98
91
.45
.00
8.9e-5
SOBDRL
mean
99 + .00
91 + 05
90 + 05
90 + 05
.52 00
31 + .06
7.2c-5 5.0e-6
1,00
GMA-VAB
mean
1.00 + .00
1.00 .00
1.00 + .00
1.00 + .00
.64 .04
.66 + .02
7.4e-5 5.5e-6
1,00
1,00
1,00
1,00
6,76-5
.79 1. .03
.79
.67 1.00
58 1 .03
.08
.99
1,0e4 1: 2.5e-5
best
.85
.75
.05
7.0e
.15
.01
.24
Factor VAE
81 1. .09
.78 .01
87 1 .01
.0 1 .05
.34 02
.33 1
DIP-VAE
24
011
.71
41 .02
32
.22
DIP VAE II
1:
.75
42 1
.16
Random
1 00b
.64
83
27
.11
02
.131
Table 7: Disentanglement for COIL3
Beta VAE
1
f DCI
LSBD-VAE
1,00 I .00
98 t 00
1.00 = .00
997 E .01
.22 .44
08 I .22
1.2-4 5.4c-6
best
1,00
98
1,00
40
.12
1.0e-04
LSBD-VAE
3.2e-3
best
1.00
96
88
86
28
13
2.6e-04
SOBDRL
mcan
.8 +
+ 00
.10 + 002
4.7c-4 1.30-4
best
96
91
.8
880
.42
.12
2.6e-04
GMA-VAI VAE
meun
1.00 1 .00
1.00 .00
1.00 .00
1.00 .00
.35
.03
.17 .02
1.1e.4 3.7e-6
1.00
1.00
1.00
1.00
34
.13
9.9c-05
VAE
mean
49 + .01
80 + 02
.61 + 001
.15 + .01
.03 .01
.04 +
2.8c-4 + 1.9c-5
B-VAE
44 1..00.0 .05
82 $2.1.
.66 .00
1 .03
.04
.01
08 1
Factor VAE
884 00.01
.2200
.11
.13 1
DIP-VAE
41.-
84
.04 .0
04
DIP-VAE II
49 05
82 42.4.4 01
35 + 01
.10
08 02
.66 +
Random
Table 8: Disentanglement for 3DShapes
Beta-VAE
Inde
t Mod
f DCI
1 SAP
MIG
Prediction
LSBD-VAL VAE
meca
.98 I .00
98 i 00
1.00 = _00
99 I .00
.43 03
07 : .01
7.6e-1 1: 2.2e-5
.99
99 .000
99 I
1.00 = .00
95 I .02
.99 004
39 I
1.le-3 I 2.6c-1
best
97
98
1.00
.03
8.3e-4
4 .05
.93
96
85 4 00
.41 .05
.22 .05
2.0e.3 8.7e.4
best
98
.97
1.00
99
31
26
9.26-4
GMA-VAR
.26
6.0e-4
best
98
1.00
1.00
1.00
69
19
1.14
A-VAB
mcan
+
.8 t:
.15 + 00
.01
.04 + 001
8.1e- + 3.60-5
best
42
900
.74
18
U07
.02
8.2e-4
B-VAB
52 04
.12 4
.74 + 00
22 .04
11 = .01
008 +
Factor
DIP-VAE
62 +
71
75++00
28
.15 01
.12
DIP-VAE
10.000 .03
.69 69.
35
.7
J4= .01
.2
Random
31 .44
64
20
04 00
00
34
Table 9: Disentanglement for MPI3D
Vet
Inde
1 Mod
1 DCI
t SAP
MIG
Prediction
SOBDRL
mecn
1.00
93
02
.04 + .02
8.5c-4 - 5.96-5
best
L.00
993
40
299
14
.02
6.8c-4
HAB
mecn
1.00
.71 .06
1 .07
.19 .0.
.07
.02 1 .0.0
4.6e-4 2.4e-5
1.00
40
54
2.6c-3
4.7c-1
1.00 1. .00
.97 .01
.9 i .08
.3 .04
21 =
044
.04 i:
1.6e-3 S.He-4
GMA-VAL
V-VAE
81 = 003
53 01
41 i
02 00
08 01
00 +
Randon
63 .03
32
05
8.le 2.6e
.02
31

---

Published as a conference paper at ICLR 2026
Table 10: Disentanglement for noisy MPI3D
f Beta-VAF
Inde
? Mnd
1 DCT
1 SAP
MIG
Prediction
.79
SOBDRI.
best
1.00
90
79
26
45
011
3.3e-3
HAB
maca
1.00
.79 06
05
.03 +
1.7e-3 3.1-8
best
1.00
90
40
27
.45
.01
1,70-3
GMA VAE
1.00 .00
.90 000
1
31 .02
.04 1 .01
1.6e-3 36e-5
best
1.00
90
84
31
64
1.6e-3
V-VAE
81 + 03
53 00
41 06
00 00
08
01
03 + .00
Random
63 = 003
52
.6
8.4-33 = 2.003
00 t
Table 11: Disentanglement for COIL2 iid setting
1 Beta- VAE
1 Inde
Mod
DCI
SAP
MIG
LSBDVAE
.99
.95
LSBD-VAE VAE
100
= .00
1.00
.01
SOBDRL
.97 1 .02
.90 .05
.94
.04
.88
.05
.44
.05
.27 1 .05
GMA-VAE
A-VAE
76 005
.75 002
.71
.53
.06
.11 .02
Table 12: Disentanglement for COIL2 in ood setting
Beta-VAF
Tnde
Mod
DCI
SAP
MIG
LSBDVAE
1.00 1 .00
99 .00
1.00 1. .00
.98
.01
.66 1. .05
.44 1 .03
LSBD-VAES
1.00 = .00
94
= .04
988
.02
.95 = .44
34 .03
.10 + .06
SOBDRL
.97
.92
VAE
.99 1: 00
1.00
.00
1.00
1 .00
.98
.02
.47 .07
.39 1. .09
A-VAE
.67
766
.92
.53
.03
.08 02
.66 + .02
Table 13: Disentanglement for COIL3 iid setting
Bein- VAF
? Inde
1 Mod
DCI
SAP
MIG
LSBDVAE
+
LSBD VAE
.28 .08
SOBDRL
.98 + .01
.3 + .01
.91 = 02
.76 .05
_66 + .05
.16 .04
GMA-VAE
1.00
1,00
+
A-VAE
.48 + 02
.80
.61 001
.15 02
.01 + .00
.02 000
Table 14: Disentanglement for COIL3 in ood setting
Beua-VAE
Inde
Mod
DCI
SAP
1
LSBDVAE
1.00 + .00
.99 .00
1.00
.00
.9 + .00
.43 .04
.05 .02
L
71 = .9
.85 = .02
.89 = .03
33 1: 12
20 = .03
.0 .03
.33
GMA-VAE A-VAB
1.00 + .00
.99 .00
1.00 .00
.9 +.00
.29 + 03
+ A3
55
24 .03
.78
.44 .01
.04
H.5 de MATRICES
Figure 14 presents the dg matrices for five different environments, Each matrix is normalized such that the clustering threshold corresponds to a distance 1. Each black square represents available actions GE associated with specific subgroups Gt. These blocks have pairwise distances strictly below whereas the distances between two actions belonging to different subgroups are strictly greater than 1, resulting in correct recovering of the ground-truth action partition.
I IMPLEMENTATION DETAILS
I.1 DATASETS
Here, we describe the datasets used in this paper. visual representation the observation and the available actions illustrated in Figure
32

---

Published as a conference paper at ICLR 2026
(a) FL.C
(b) FLP
(c) COIL2
(d) COIL3
(e) 3DShapes
10
Figure 14: de matrices averaged over 5 seeds, normalized so that the clustering threshold is 1.
Our implementation of Colored Flatland (Caselles-Dupree al. 2018) consists of RGB images of size 64 64 x 3 with a ball of radius 17 pixels. The ball can occupy 5 distinct positions along each axis, corresponding to Gx Gy 2/52. In the cyclic color experiment (FLC), the base color is [1, 0]. and the two available actions in Gc perform cyclic shifts of the active color channel in one direction. In the permutation color experiment (FLP), the base color is [1/3, 2/3, 1], and each available action corresponds to permutation of the RGB channels.
Our implementation of COIL (Nene al. 1996) consists of RGB images of size 64 64n x 3. where is the number of objects present in the scene. The group actions, and consequently the number of possible orientations for each object, are specified Table [5] In this table we denote by a permutation that permutes all objects, and by r; the rotation by one unit angle of the i-th object. The first two rows of Table [Table[5]corress corresspd to the datasets COIL2 and COIL3, which are used the most of the experiments. The remaining rows describe additional environments used for action clustering experiments in Section(5.2
Our implementation of 3DShapes (Burgess & Kim 2018) consists of RGB images of size 64 x 64 x 3, showing a 3D object placed at the center of a colored room. Each original generative factor was sub-sampled by factor of 2, resulting in the following cardinalities: wall hue: 5, object hue: 5, background hue: 5, object scale: 4, object shape: 2, and viewing angle: 7. For each generative factor i, we define a eyclic symmetry group G; Z/ k/K; over its possible values, along with two available actions G; = (9r .9.") corresponding to unit rotation in each direction.
Our implementation of MP13D (Gondal al. 20T9) consists of RGB images of size 64 64 3, showing a robotic arm manipulating colored object. Among the original generative factors, we only kept on two of them: horizontal and vertical object rotation angles, each having 40 different values. In order to approximate continuous symmetry groups, the actions are represented by g = (j.0) 0) with j {0, 1) the axis of rotation and E10, 2 the rotation angle. In the noisy version of MPI3D, we add Gaussian noise of standard deviation 2nr /15 on the rotation angle after applying the action.
Table 15: COIL Environments used
Gs
Gi
G2
Gs
G4 includes e
62
G
(-ri,m)
72, r2)
No
63 x
x
G
(-ri,m)
{ r2, 12)
{-r,rr3)
No
63
{a)
(rr
[3r2, Ar23
{rs]
Yes
6, x
{a)
(2ri, 6m
{2r2]
{ra)
Yes
64
{0,0
")
(rr
[3r2, 4r23
{rs]
(ra)
No
I.2 NETWORK ARCHITECTURES AND HYPERPARAMETERS
For all the experiments, dataset and method. we used Adam (Kingma & Ba 2015) with the default parameters given by pytorch, batch size of 16 and the auto-encoder architecture given Table 16
33

---

Published as a conference paper at ICLR 2026
(a) COIL
gs
0
g
Ri
R
gRs
(b) Colored Flatland
y
c
gy
gx
(c) 3DShapes
Bloor hue
object scale
wall buc
orientation
object hue
object shape
(d) MP13D
[0, 0]
110
Figure 15: Presentation of the environments
Table 16: Auto-encoder architecture used for all methods and all datasets
ENCODER
Input
Size (æ, M, c)
Conv
Channels: 32, Kernel size: 8, Stride: 4, Padding: 2, ReLU
Conv
Channels: 64, Kernel size: Stride: 4, Padding: 2. ReLU
Reshape
Flatten into 2/4 3/ x 64
Dense
Dimension: 256, ReLU
Dense
Dimension: depends on d and the noise parametrisation
DECODER
Input
Size d
Dense
Dimension: 256, ReLU
Dense
Dimension: x/4 x y/4 x 64, ReLU
Reshape
(x/4, 4, 64)
ConvT
Channels: 32, Kemel size: 8, Stride: 4, Padding: 2, ReLU
ConyT Channels: c, Kernel size; Stride: 4, Padding: Sigmoid
34

---

Published as a conference paper at ICLR 2026
For discrete action, an action matrix Ag of SOBDRL is parametrized by d(d 1) scalars &%,j" angle for each pair of plane (i,j), the action matrix is then constructed by multiplying all the rotations m = Tik,; R For LSBD-VAE, the block-diagonal actions matrices are already given as it is a supervised method. For LSBD-VAE* the only difference is that each block learned with SO parametrisation similarly to SOBDRL. For our method, the action matrices are parametrized directly with the matrices coefficient, we used a tanH activation to ensure stability.
For continuous action, each action representation are neural networks of two hidden layer of 256 units with ReLU activation functions. SOBDRL action representation has an output of size d(d-1) and use the same SO parametrisation as for discrete action to construct the action matrix. GMA- VAE action representation has an output of size d x d and directly parametrize the action matrix coefficients. HAE action representation has as output a block-diagonal matrix size d with the block shape given prior knowledge, matrix exponential is used on each block.
SOBDRL and HAE require multi-steps trajectories (Ty, It...., P1---1--------- 9:-T:;;-111- It-T) with T> 1. To have fair comparison, every experiment will use sequences with T 5, the other methods process independently each transition (#t+ks It+k; 1(+8+1).
For A-VAE and GMA-VAE, we used fixed latent noise of standard deviation a 0.1. For the a and M = For the GMA- VAE masking,
action clustering of Step we used the threshold force each sub-group to have at least one dimension assigned to it.
All hyperparameter details for each experiment are provided in the config folder the GitHub repository.
J USE OF LARGE LANGUAGE MODELS
LLMs moderately used help in literature reviewing and English writing.
35


Reviews:
[{"review_id": "hnYYE3TjXD", "reviewer": "Reviewer_9xAV", "summary": "This paper studied symmetry-based disentanglement learning. To remove assumption about the symmetry group’s structure,  this work introduces a novel, multi-step approach to learning disentangled representations by having an agent autonomously discover the underlying symmetry group structure of its environment. The proposed method consists of three main steps: 1) Learn an Entangled Representation (A-VAE). 2) Discover the Group Structure. 3) Learn a Disentangled Representation (GMA-VAE). The authors validate their method on three environments with different group structures (Flatland, COIL, and 3DShapes). The results show that their action clustering algorithm perfectly recovers the ground-truth group decomposition , and the final GMA-VAE model achieves disentanglement performance comparable to supervised methods that are given the group structure in advance, outperforming other self-supervised approaches.", "questions": "Why the MIG score is low on 3Dshapes? The discussion about failing cases is important to understand the drawbacks of proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Novelty:** The core contribution—learning an LSBD representation via three steps—is instructive for disentanglement learning. This shifts the paradigm from requiring prior knowledge to autonomously learning it from interaction.\n2. **Theoretical Grounding:** The paper is built on a solid theoretical foundation, providing formal proofs for its key claims. This guarantees the existence of actions belonging to the same subgroup (Theorem 2) and the  of learning a Linear Symmetry-Based Disentangled (LSBD) representation (Theorem 3).  This level of rigor adds significant weight to the proposed methods.\n3. **Strong Empirical Validation:** The method is thoroughly tested across multiple benchmarks. The final disentanglement performance is shown to be on par with a supervised method (LSBD-VAE), which is a very strong result.\n4. **Other properties:** The paper goes beyond standard disentanglement metrics to show the other properties of the learned representations. The experiments on long-term prediction clearly demonstrate that the entangled methods prefer short-term predictions, while self-supervised ones prefer long-term predicitons.", "weaknesses": "1. Multi-Stage Pipeline: The method is not end-to-end. It requires training two separate models sequentially: first the A-VAE to learn action matrices, and then the GMA-VAE to learn the final representation. The authors acknowledge this limitation, noting that a future direction would be to unify these steps into a single optimization process.  \n2. Limited Scope of Environments: The experiments are conducted on synthetic, visually simple datasets.  While these are well-suited for proving the group-theoretic concepts, it is unclear how the approach would scale to more complex, realistic, or stochastic environments where the underlying symmetries might be less perfect or harder to learn from raw pixels.\n3. No visualisation evaluation for  reconstruction quality and latent traversal to explicitly demonstrate the tradeoff. \n4. The number of MIG on 3Dshapes is low.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTvnL2q7TF", "reviewer": "Reviewer_B8Yx", "summary": "They propose an algorithm to learn and disentangle the group structure underlying an environment, through interactions with this environment. Unlike previous methods, their method does not assume specific subgoup properties.", "questions": "- The \"Geomancer\" should probably be cited as they somehow address a related question: https://arxiv.org/abs/2006.12982\n- How to understand the problem of separating subgroups in different dimensions, when it is not always possible with a continuous encoder, as shown in https://arxiv.org/abs/2102.05623?\n- How did Caselles-Dupre ́et al. (2019) demonstrate that symmetry-based disentanglement is only possible when access is granted to transitions? Consider explaining this in the paper. \n- \"Such an approach to disentanglement,\" => these alternative frameworks seem only tangentially relevant, and could be moved to appendix.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "My understanding of the framework is relatively superficial and I did not check the maths carefully. \nStrengths:\n- Careful explanations.\n- Thorough comparison with other methods and with different metrics of disentanglement, which show an advantage over other unsupervised methods.", "weaknesses": "- How is the method less supervised than LSBD-VAE? It seems pretty supervised to me, with access to actions and consequences of these actions? What does it mean that \"Both of our methods rely on a strong assumption which requires the available actions to be disentangled\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dhBuRe5UVt", "reviewer": "Reviewer_7wnh", "summary": "This paper tackles unsupervised symmetry group discovery for Linear Symmetry-Based Disentanglement (LSBD). It proposes a two-stage pipeline: (i) learn an “entangled” action representation and encoder via an Action-VAE trained on transitions, then (ii) cluster actions into subgroup factors using a group-theoretic pseudo-distance, and finally (iii) learn a block-masked GMA-VAE that enforces LSBD without assuming subgroup structure. The paper proves identifiability under injectivity and “disentangled action set” assumptions and evaluates on Flatland-style grids with color cycles/permutations, COIL rotations with permutations, and 3DShapes. Reported wins are on LSBD metrics and long-horizon prediction versus Forward-VAE/SOBDRL/LSBD-VAE variants.", "questions": "1. How does action clustering perform when the action set is not strictly disentangled (shared actuation across factors), or when a subset of transitions is missing/noisy? Please report clustering accuracy and downstream LSBD under controlled violations.\n\n2. Can the pseudo-distance and masking be extended to continuous Lie groups (SO(2), SE(2)/SE(3))? If not, what fails? Compare against Lie-symmetry discovery approaches. \n\n3. Can you demonstrate that discovered groups help in standard control tasks by instantiating ρ inside an equivariant policy/value network (as in group-equivariant RL)? Measure sample efficiency and generalization versus non-equivariant baselines.\n\n4. Partial observability: Your proofs assume injective observations. What happens in POMDPs (occlusions, distractors)? Any way to incorporate history (RNN/state-space models) while preserving identifiability?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear LSBD pipeline that separates (i) equivariant pretraining, (ii) group discovery, (iii) block-structured LSBD learning, with explicit assumptions and a clustering rule grounded in group algebra. \n\n2. Good baseline experiments within the LSBD family (Forward-VAE/SOBDRL/LSBD-VAE variants) and consistent reporting on multiple disentanglement metrics and multi-step prediction. \n\n3. Reproducibility: code, dataset generation, and hyperparameters are described and (per the authors) released.", "weaknesses": "1. **Lack of realistic interactive experiments.**\nThe environments are synthetic (Flatland, COIL with permutations, 3DShapes). There are no tests on widely used embodied/control suites (e.g., DeepMind Control/ProcGen/Habitat/ManiSkill) where continuous groups (SO(2), SE(2), SE(3)) and sensor noise dominate, and where symmetries are only approximate. By contrast, prior interactive symmetry/LSBD works motivate interaction explicitly and evaluate on non-trivial dynamics (e.g., SOBDRL/Forward-VAE), and the broader equivariant RL literature demonstrates robustness/sample-efficiency gains in control domains—precisely the use-case this paper claims to address.\n\n2. **Strong assumptions.**\nIdentifiability and clustering rely on (i) fully injective observations, (ii) a disentangled action set with each action belonging to exactly one subgroup, and (iii) the existence of short compositions mediating within-subgroup relations. These are rarely true in practical agents (redundant actuators, coupled controls, unmodelled dynamics). No robustness analyses are provided for violations (e.g., aliased actions, missing transitions, approximate commutativity). The empirical advantage may be specific to the LSBD formulation and the chosen metrics.\n\n3. **Limited group diversity and no continuous-group evaluation.**\nAll experiments use finite groups (cyclic, symmetric). There is no assessment on continuous groups or approximate symmetries (e.g., SE(2)/SE(3) rotations and translations, scaling), where many RL/control tasks live and where identifiability and clustering are harder.\n\n4. **Limited discussion of related work.** This paper is missing a lot of citations and related works from the group structured representation learning in interactive environments literature [1, 2]. Many of these work uses actions of the environment to disentangle or induce structure in the latent space and shows it's benefits in reinforcement learning etc. I would suggest the authors to do a more thorough literature review. \n\n[1] Learning Symmetric Embeddings for Equivariant World Models. JY Park et. al.\n[2] Structuring Representations Using Group Invariants. M Shakerinava et. al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QIx97dh1BB", "reviewer": "Reviewer_GrsJ", "summary": "This paper introduces a method to learn disentangled representations using symmetry groups without requiring prior knowledge of the group structure. The approach consists of three stages: learning an entangled representation (A-VAE), discovering the group decomposition via clustering, and learning a disentangled representation (GMA-VAE). This addresses a key limitation of existing LSBD methods that assume known group structure, and the work constitutes a valuable contribution to unsupervised representation learning.", "questions": "1. Could the authors elaborate on the tangible advantages of symmetry-based methods, particularly when augmented by the improvements proposed in this paper, over mainstream approaches like QLAE and FactorQVAE?\n2. How realistic are the underlying assumptions in practical settings? All experiments employ relatively simple synthetic datasets. Would the assumptions hold on more complex benchmarks such as MPI3D or Isaac3D?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a complete framework for automatically discovering group structure and utilizing it for disentanglement, addressing the limitation of prior LSBD methods that rely on known group structure. This contribution demonstrates considerable novelty and significance.\n2. This paper provides solid theoretical foundations for the proposed method while conducting systematic evaluation on multiple datasets, comparing not only standard disentanglement metrics but also evaluating long-term prediction and generalization capabilities.", "weaknesses": "1. The treatment of non-symmetry-based disentanglement methods is insufficient. While the paper situates itself within the LSBD literature, it provides minimal discussion of how symmetry-based approaches compare to traditional disentanglement methods.\n2. The experimental comparison is confined to symmetry-based methods, leaving the performance of the proposed approach against mainstream baselines such as QLAE and FactorQVAE unclear.\n3. The extensive mathematical notation throughout the manuscript hinders readability. A consolidated symbol table would significantly improve accessibility for readers less familiar with group theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)