CFASL: Composite Factor-Aligned Symmetry Learning for Disentanglement in Variational AutoEncoder
Authors: Hee-Jun Jung
Jaehyoung Jeong
Kangil Kim
Venue: Accepted by TMLR ()

Abstract: Symmetries of input and latent vectors have provided valuable insights for disentanglement learning in VAEs. However, only a few works were proposed as an unsupervised method, and even these works require known factor information in training data. We propose a
novel method, Composite Factor-Aligned Symmetry Learning (CFASL), which is integrated into VAEs for learning symmetry-based disentanglement in unsupervised learning without any knowledge of the dataset factor information. CFASL incorporates three novel features for learning symmetry-based disentanglement: 1) Injecting inductive bias to align latent vector dimensions to factor-aligned symmetries within an explicit learnable symmetry code-book 2) Learning a composite symmetry to express unknown factors change between two random samples by learning factor-aligned symmetries within the codebook 3) Inducing group equivariant encoder and decoder in training VAEs with the two conditions. In addition, we propose an extended evaluation metric for multi-factor changes in comparison to disentanglement evaluation in VAEs. In quantitative and in-depth qualitative analysis, CFASL demonstrates a significant improvement of disentanglement in single-factor change, and multi-factor change conditions compared to state-of-the-art methods.

Jeabyoung Jcong Published in Transactions on Machine Learning Research (11/2024)
CFASL: Composite Factor-Aligned Symmetry Learning for Disentanglement in Variational AutoEncoder
Hee-Jun Jung
jungheejjun388gmmiit..oo...
AT Graduate School Gwengja Institute of Science and Technology (G1ST)
AI Graduate School Gwengju Institute of Science and Technology (GIST)
Kangil Kim
kikim01 @gist. srist.oc..i oc.kr
A1 Graduate School
Gwangju Instätute of Science and Technology (GIST)
Reviewed on OpenReview: https: oppenreveeeeeeeeeuu net/ forun? mDGurH7l
Abstract
Symmetries of input and latent vectors have provided valuable insights for disentanglement learning in VAEs. However, only a few works were proposed as an unsupervised method, and even these works require known factor information the training data. We propose a novel method, Composite Factor-Aligned Symmetry Learning (CFASL), which is integrated disentanglement in unsupervised learning without
into VAËs for learning symmetry- any knowledge of the datasct factor information. CFASL incorporates three novel fcatures for learning symmeery-based disentanglement: 1) Injecting inductive bias to align latent vector dimensions to factor-alis aaitor- symmetries within an explicit learnable symmetry code- book 2) Learning a composite symmotry to express unknown factors change between two random samples by learning factor-aligned symmetries within the codebook 3) Inducing a group equivariant encoder and decoder training VAEs with the two conditions. In ad- dition, we propose an extended evaluation metric for multi-factor changes in comparison to disentanglement evaluation in VAEs. In quantitative and in-depth qualitative analysis, CFASL demonstrates significant improvement of disentanglement in single-facte factor change, and multi-factor change conditions compared to state-of-tho-at methods.
1 Introduction
Disentangling representations by intrinsic factors of datasets is a crucial issue in machine learning liter- ature (Bengio ct al.] 2013). In the Variational Autocncodor (VAE) frameworks, a prevalent method to
handle the issue is to factorize latent vector dimensions to encapsulate specific factor information (Kingma
& Welling 2013 Higgins al. 2017 Chen et al.) 2018 Kim & Mnih 2018 Jeong & Song 2019 Shao al.,
2020 2022). Although their effective disentanglement learning methods, Locatello et al. 20019 raises the serious difficulty of disentanglement without sufficient inductive bias.
In VAE literature, recent works using group theory offer possible solution to inject such inductive bias by decomposing group symmetries (Higgins et al. 2018) in the latent vector space. To implement group equivariant VAE, Winter et al. 2022 Nasiri & Bepler (2022) achieve the translation and rotation equivariant
VAE, The other branch implements the group equivariant function (Yang et al. 2022 Keller & Welling 2021b) over the pro-defined group elements. All of the methods effectively improve disentanglement by
CCorresoondngg author

---

Published in Transactions on Machine Learning Research (11/2024)
adjusting symmetries, but they focused on learning symmetries among observations to inject inductive bias rather than factorizing group elements to align them single factor and a single dimension changes, introduced in the definition provided in Higgins et al. (0118).
In recent works, unsupervised learning approaches of group equivariant models has been introduced. Miyato et al. (2022); Quessard et al. (2020) represent the symmetries on the latent vector space, which correspond the symmetries on the input space, by considering the sequential observations. Also, Winter al. (2022) proposes the group invariant and equivariant representations with different modules to learn the different groups of dataset structure. However, these approaches, despite being unsupervised learning, require the factor information of the dataset to construct the sequential input and to set different modules for learning symmetries.
This paper introduces a novel disentanglement method for Composite Factor-Aligned Symmetry Learning (CFASL) within VAE frameworks, aimed addressing the challenges encountered in unsupervisod Iearn- ing scenarios, particularly the absence of explicit knowledge about the factor structure in datasets. Our methodology follows as 1) network architecture to learn an explicit codebook of symmetries, responsible for each single factor change, called fuctor-aligned symmetriess 2) training losses to inject inductive bias into an explicit codebook where each factor-aligned symmetry only impacts single dimension value of the latent vectors for disentangled representations, 3) learning composite symmetries by predicting single fac- tor changes themselves without the information of factor labels for unsupervised learning, 4) implementing group equivariant encoder and decoder functions such that factor-aligned symmetry affects the latent vector space, 5) an extended metric (m-FVMs) to evaluate disentanglement in the multi-factor change condition. We conduct quantitative and qualitative analyses of our method common benchmarks of disentanglement in VAEs.
Table 1: Notation Table
Group
Codebook
Group
g
Codebook
GT,(n, IR)
General Lie group
g
ith section of codebook
Group element of Group G
0:
jth element of g
gl(n,
Lie algebra of GL(n, Lie algebra E gaxn group action
[]]
size of section
g
SS]
size of subsection
((s-)
jth symmetry of 2th section of codebook, equal to exp(g;) composite symmetry
9)
Set
x
dataset subset dataset Set of factors {fi,... fx ęth index value of f values of f except f Set of latent vectors latent vector
X
Others
R
Real number
a
standard deviation (scalar) standard deviation (vectio)
f-t
Z
μ
mean (vector)
X [
Random samples with [B] batch size dut product
(
Function
L2 norm
Gen(-)
F + X matrix exponential
orthogonal
exp
parallel
2 Preliminaries
2.1 Group Theory
Group: A group is set G together with binary operation o, that combines any two elements ga and gb in G, such that the following properties:
closure: Ja. Po E G 9a gb E G.
Associativity: Vga, Ic G. G, s.l. (ga gb) ge = Iu (g6 g.)-
2

---

Published in Transactions on Machine Learning Research (11/2024)
Identity element: There exists element G, s.t. Vg G.cog=
Inverse clement: Vg E G, Fg-1 E G: gog g c-=
Group action: Let (G,o) be group and set X, binary operation :: G X - X, then group action a a(g, - g. following properties:
Identity: e=x=2, where G,
Compatibility: Vga. 9b E G, EX, (ga o 9b)
(gb x)-
Equivariant map: Tet G be a group and both X, and X2 are G-set with corresponding group action of G in each sets, where g E G. Then function f f X1 - X2 is equivariant if flg X1) g f(X).
Lie Group and Lie algebra: Lie group is defined as a group that simultaneously functions as a differen- tiable manifold, with the operations of group multiplication and inversion being smooth and differentiable. In this paper, consider the matrix Lie group, which is Lie group realized as subgroup contained in GL(n, R), the group of n x invertible matrices over R.
Lie algebra gl(n, IR is a tangent space of the Lie group at the identity element. The Lie algebra covers a Lie group by the matrix exponential of elements of gl(n, R) as exp gl(n, R) - GL(n, R), where exp(X) I X2 X
2.2 Disentanglement Learning
Variational Auto- Enc.ddeeeooaae.... We take VAE Kingma & Welling 2013 as a base framework. VAE
optimizes the tractable evidence lower bound (ELBO) instead of performing intractable maximum likelihood estimation as:
ELBO (ale)) lo] Dku][g((\/))ll<<));
(1)
where (=(2(x) is the approximated posterior distribution, (>(x)² ~ Nius, ozl), the prior p(z) ~ _N I), the first term of Equation a reconstruction error between input and outputs from the decoder po(x))). and second term of Equation [ a Kullback-Leibler term to reduce the approximated posterior and prior distance.
Factor Metric (FVM) FVM metric (Kim & Mnih 2018) is of the disentanglement learning metrics, which evaluates the consistency of dimension variation single factor is fixed. First, the fixed factor ft is selected, and other factors f-i are randomly selected, then generate the subset of the datasct X. which corresponds to factor f, where X
&y = Gen(Jj..5") and Gen(-) is a generator. The second, latent vector from the encoder of VAE is normalized by the standard deviation (a) of the dataset (z, / a), then selects the th dimension as d = arg min, Var(2))) Var(.) is a variance of each dimension. The FMV metric is the accuracy of the majority vote classifier with data points (d,
2.3 Symmetries for Inductive Bias
Disentangled Representation Based on the Group Higgins et al. (2018) shows the definition of disentangled representation through the group. First, the generator Gen : F X and the inference process h X are defined. Then group G of symmetries acts on F via group action G F - F. Lastly, symmetries decompose as direct product G G1 x... Gn- Then the disentangled representation is defined with 1) group action G 7 + 7, 2) composition b F + Z is an equivariant map, and 3) 2, is fixed by the action of all Gj, and affected only by G,, where decomposition Z = 21 X... x Zn.
Implimentation of Equivariant Model To implement an equivariant map for learning symmetries, previous works utilize the Variational Auto-Encoder with an additional objective function defined as qs 2) g" (,(x,, where 9o + is an encoder, g².9* symmetry on input space X' and latent vector space 2, respectively. Yang et al. 20222 Winter al. 2022 represent the g² in the latent vector space
3

---

Published in Transactions on Machine Learning Research (11/2024)
instead 9 with following equation: g0(x2) g" (0(x)), where 22 g" t 21 to represent the o correspond to g². Symmetry 9" is defined as specific group such as 50(3), (n))) or GT(()) (Yang et al. 2022 Winter
et al. 2022 Zhu et al. 2021
(a) B-VAE
(b) 3-TCVAE
(c) Cont.-VAB
(d) CLG
(e) G-VAE
(f) Ours
Ideal
Figure 1: Distribution of latent vectors for dimensions responsible for Shape, X-pos, and Y-pos factors in the dSprites dataset. The groupified- VAE method applied 3-TCVAE because this model shows better evaluation score. The results show disentanglement for shape from the combination of the other two factors by coloring three shapes (square, ellipse, and heart) as red, blue, and green color, respectively. Each 3D plot shows the whole distribution. We fix Scale and Orientation factor values, and plot randomly sampled 640 inputs (20.8% of all possible observations (32 x 32 x = 3, 072)). We select the dimensions responsible for the factors by selecting the largest value the Kullback-Leible divergence between the prior and the posterior. Con is Control-VAE.
3 Related Work
Disentanglement Learning Diverse works for unsupervised disentanglement learning have elaborated in the machine learning field. The VAE based approaches have factorized latent vector dimensions with controllable weighted values to penalize Kullback-Liibler divergence (KL di-
weighted syper-parameters
vergence) (Higgins et al.] 2017 Shao et al.) 2020 2022). Extended works penalize total correlation for factorizing latent vector dimensions with divided KL divergence (Chen et al.] 2018) and discriminator (Kim) & Mnih 2018). Differently, we induce disentanglement learning with group equivariant VAE for inductive bias.
Group Theory-Based Approaches for Disentangled Representation In recent periods, various unsupervised disentanglement learning research proposes different approaches with another definition of disentanglement, which is based the group theory (Higgins et al. 2018). To learn the oquivariant function, Topographic VAE (Keller & Welling) 2021a proposes the sequentially permuted activations the latent vector space called shifting temporal coherence, and Groupified VAF (Yang et al. 2022) method proposes that imputs pass the encoder and decoder two times to implement permutation group equivariant VAE models. Also, Commutative Lic Group VAE (CLG-VAE) (Zhu et al. 2021 Mercatali et al.) 2022) maps latent vectors into Lie algebra with one-parameter subgroup decomposition group structure from abstract canonical point to inputs. Differently, we propose the trainable symmetries that are extracted between two samples directly on the latent space while maintaining the equivariance function between imput and latent vector space.
inductive bias to learn the
Symmetry Learning with Equivariant Model Lie group equivariant CNN (Dehmamy et al. 2021 Finzi et al. 2020 construct the Lie algebra convolutional network to discover the symmetries automatically. In the other literature, several works extract symmetries, which consist of matricos, between two inputs or objects. MMyato al. 2022)) extracts the symmetries between sequential sequentially augmented inputs by penalizing the transformation of difference of the same time interval. Other work extracts the symmetries by comparing two inputs, in which the differentiated factor is rotation or translation, and implements symmetrics with block diagonal matrices (Bouchacourt et al. 2021). Furthermore, Marchetti et al. (2023) decomposes the class and pose factor simultaneously by invariant and equivariant loss function with weakly supervised learning. The unsupervised learning work (Winter et al. 2022) achieves class invariant and group oquivariant function less constraint conditions. Differently, we generally extend the a class invariant and group equivariant model in the more complex disparity condition without any knowledge of the factors of datasets.
4

---

Published in Transactions on Machine Learning Research (11/2024)
Inductive Bins learnable
methad
dataset info.
orthogonality
symmetry
Higginn al. 2017 Chen al.
2018
Kim Mnih 2018)
Zhu et al. 2021 Miyao et al. 2022
Winter et al. 2022
X
Ours
Table 2: Summary of injected inductive bias of models for disentanglement learning. dataset info. represents that methods need the information of factors of the dataset. learnable symmetry represents the learnable parameters to represent the symmetries. orthogonality represents the orthogonality between the axes and latent vectors through the group action.
4 Limits of Disentanglement Learning of VAE
By the definition of disentangled representation Bengii et al. representations are distributed on the flattened surface as shown in Fi ig
2013 Higgins al. 2018). the disentangled
because each change of the factor
only affects a single dimension of latent vector. However, the previous methods (Higgins et al. 2017 Chen
cet al. 2018 Shao et al.) 2020 Zhu et al.] 2021
Yang et al. 2022 show the entangled reprosentations on
their latent vector space as shown in Fig. Ia Le Even though the group theory-based methods improve Yang ct al., 2022,, these stilll struggle with the same problem as shown in Fig. Id and [ee In addition, symnmetries are represented on the latent vector space for disentangled representations. In current works (Miyato et al. 2022 Keller & Weeling 2021b Quessard ltt al., 2020), the sequential observation is considered in unsupervised learning. However, these works need the knowledge of sequential changes of images to set up inputs manually, as summarized in Table _
the disentanglement performance (Zhu et al. 2021
To enhance these two problems of disentanglement learning of group theory-based methods, addressing two questions is crucial:
Do the explicitly defined symmetries impact the structuring of disentangled space as depicted in Fig. Ig
2. Can these symmetries be represented through unsupervised learning without any prior knowledge of factor information?
5 Methods
Our work is mainly focused on how to 1) define the inputs and symmetry, 2) optimize the symmetry, 3) represent the composite symmetry, and 4) inject the symmetry as an inductive bias. We first define the 1) inputs as a pair of two samples, 2) group, group action, and G-sct, and 3) codcbook in section 5.1 In the next, we optimize the codebook for disentangled representation in section 5.2 then extract the composite symmetry to represent transformation between two inputs in section 5.3 Lastly, we introduce the objective loss to inject an inductive bias for disentangled representations in section 5.4
5.1 Inputs and Symmetry
Input: A Pair of Two Samples To learn the symmetries between inputs with unknown factors changes, we randomly pair the two samples as an input. During the training, samples in the mini-batch B) are divided into two parts X|2 [ x, LIy and X = 12 .,, J. where [B] is a mini-batch size. In the next, model pairs the samples (x), (x), ........ (x) x and is used for learning symmetries between the elements of each pair.
2
Group, Group action and G-set We define the symmetry group as a matrix general Lie group GL(n), and set group action as a(g, 2) = Ig, where g E GT(n), z E R." and latent vector space Z is G-set.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Lde
Codebook
Two-Step
Step Attention
Zj,
Ep
Input
Decoder
pd
H1.H2
(144 2 N(H2, d2)
21 m
21.
Enunder
Decodda
72
01 02
N0,1))
Reconstruction CTT.
1st section
2nd section ... |S/th section
(a) Overall architecture
(b) Codebook
Two-Step Attention
MLP W
attm
MLP. .
M2
MLP
arêtn"
Codebook
Eq.
attn
Eq. (9)
MLF W. 51
SW(G()
Lee
22.22
MLF 2
MLP W32
T2
(c) Two-step attention
Figure 2: The overall architecture of the proposed method. ++ refers loss function. A pair of images (e-g-. differences between two images are in the x- and y-position) is given, and the goal of the model is to represent the differences on the latent vector space, called the composite symmetry ge for disentangled representations. 2) The codebook is designed to represent the composite symmetry ge- 3) Each section of the codebook is separated to affect single factor e.g., the :th section affects the x-position, and the jth section affects the y-position of images. 4) Each section consists of Lie algebra to provide diversity of symmetries. 5) As shown in (b), each loss optimizes the codebook to guarantee the 3) as follows: i) symmetries from the same section affect the same factor through the parallel loss Ept (e.g., symmetries from 5th section exp(g() only affects the x-position), fi) each section affects different factors by the perpendicular loss End (e.g., symmetry from 7th and jth section ge and ge affect x-position and y-position respectively). and iil) each section changes single dimension of latent vectors for disentangled representation by the sparsity loss L.. 6) atfn ensures diversity in symmetries representation, and predicts the activated section, in this case, th and jth sections for and y-position differences. 7) The model then represents the composite symmetry ge- 8) Lastly, model optimizes the fee to match 9621 (= 2//) and 22, and Ld to match the æ1 and po Po(ge go (a(2))) to inject the inductive bias.
Codebook: Explicit and Learnable Symmetry Representation for GL(n) To allow the direct injection of inductive bias into symmetrics, we implement an explicit and trainable codebook for symmetry representation. we consider the symmetry group on the latent vector space as subgroup of the general lic group GL(n) under a matrix multiplication. The codebook g = {G1 [ (2......*) is composed Gi, which affect to a different single factor, where k E {1,2, [S]], and |S] is the number of sections. The section gi is composod of Lie algebra (gi.g...... .2.....g)). where g) E R D PI, E 21, [SS]],
sections
) (SS] is the mumber of elements in each section, and D is dimension size of latent z. We assume that each Lie algebra 0,
0): g; = Σ, a 4.3 BB,,, where kl}. Then, the dimension of the element of the codebook is equal to [B], and the dimension of the Lie group composed of the codebook element is also [B]. To utilize previously studied effective expression of symmetry for disentanglement, we set the symmetry to be continnous (Higgins et al. 2022) and invertible
consists of lincarly independent bases Φ [B,/B; E RRxx Σ, B, +
E
via mairix exponential form [iaao Liu. 2020 as g e
-x J (g))* to construct the Lie group Hall
>k=n
2015).
5.2 Codebook: Aligge- Symmetry Learning with Inductive Bias (Orthogonality)
We define the alctor-aligned symmetry that represents corresponding factor change on the latent vector space and each independent factor only affects a single dimension value of the latent vector. For factor- aligned symmetry, compose the symmetry codebook and inject inductive bias via parallel loss Cpt and

---

Published in Transactions on Machine Learning Research (11/2024)
2/2
g/2
2 g/2
2 9
2
9
9
2 g/2
6/2
2 9
9/2
2 9
g/2
giz
9
2
(a) parallel loss
(b) (a) + spansity loss
(c) perpendicular loss
(d) (c) + spansity loss
Figure 3: Roles parallel, perpendieular, and sparsity on symmetrics in the codebook adjusting representation change. Parallel loss is for symmetries of the same section, and perpendicular loss is for different sections. Each axis (x and y) only affects single factor.
2) perpendicular loss Cpd that match each symmetry to single factor changes. Each loss optimizes the symmetries to affect the same and different factors, respectively. Then we add 3) sparsity loss L.s to the losses for disentangled representations shown in Fig. 3 It aligns single factor change to axis of latent vector space. Also, we implement the 4) commutative loss Le to reduce the computational costs for matrix exponential multiplication.
Inductive Bias: Group Elements the Same Section Impacts on the Same Factor Changes As we design the symmetries from the same section of the codebook affect the same factor shown in Fig. [ (example), add bias that the latent vector changed by symmetries of the same section to be parallel (z 23 giz for ith section) as shown in Fig. 3a
We define a loss function to make them parallel as:
(z 12-9 -32,2 g1
Spt
log
1 2/ 2/222
where 9f = e (.)) dot product, and 1/2 is a L, norm.
Inductive Bias: Group Elements of the Different Section Impacts on the Different Factor Changes As shown in Fig. 2 example of perpendicular loss to enforce each section affects different factor, we inject another bias that changes the latent vector through symmetries of the each sections to be orthogonal 9}z 1 z g z for different ith and kth sections) as shown in Fig. 3c
for different factors change impacts ( 2 1z-g)2 The loss for inducing the orthogonality is
19 Lpd Σ
189
(2 93 - 2²²z)
(3)
2/2/2 - 2/2
1,k=
Due to the expensive computational cost for Eq. 3 (O(IS)2 |SSP)), we randomly select a (j,1) pair of symmetries of each section. This random selection still holds the orthogonality, because if all elements in the same section satisfy Equation 2 and a pair of elements from a different section (g',g)) satisfies Equation 3
then any pair of the element (g' E E G)) satisfies the Equation 3 More details are in the Appendix B
Inductive Bias: Align Each Factor Changes to the Axis of Latent Space for Disentangled Representations Factorizing latent dimensions to represent the change of independent factors is an al- tribute of disentanglement defined in Bengio et al. (2013) and derived by ELBO term in VAE training frameworks (Chen et al.) 20188 Kin & Mnih 2018 However, the parallel and the perpendicular loss do not factorize latent dimension as shown in Fig. 3a 3c To guide symmetries hold the attribute, we enforce the change A1 و3²² be parallel shift to unit vector Fig. 3b
3d via sparsity loss defined as
[s8
C.
Σ
Σ
( z 12]2
max ((ax((A)z)))
(4)

---

Published in Transactions on Machine Learning Research (11/2024)
where Ajzz is 1thh dimension value.
Commutativity Loss for Computational Efficiency In the computation of the composite symmetry ge: the production 11 S| z is compulationally expensive because the Taylor series repeated for all (i,j) pairs.
To reduce the cost by repetition, we enforce all pairs of basis gf to be commutative to convert the production to eL., 82 (By the matrix exponential property: eA e
AB BA, where A, B E R=xx).
The loss for the commutativity is defined as:
18| Σ
Σ 4,
g
g) -
5.3 Composition of Factor-Aligned Symmetries via Two-Step Attention for Unsupervised Learning
In this section, we introduce how the model extracts the symmetries between two inputs. We propose the a two-steps process follows: 1) the model extracts the Lie algebra of each section g to represent the factor- aligned symmetries between two imputs in the first step, then 2) predicts which section of the codebook is activated in the second slep process shown in Fig. 2 example.
First Step: Select Factor-Aligned Symmetry In the first step, the model generates the factor-aligned symmetries of each section through the allention score, as shown in Fig.
ss
D.
Σ attn
(6)
where attm) = softman( (attmaaa[[MM33 3 W + bb) [M: [M o (2; W E RA|i|i|SS and by E IR SS] are learnable parameters,
.....|||], and softmar(-) is a softmax function.
Second Step: Section Selection In the second step of our proposed model, we enforce the prediction of factors that have undergone changes. We assumee that if some factor of two inputs is equal, then the variance of the corresponding latent vector dimension value is smaller compared to others. Based on this assumption, we define the target (I") for factor prediction: if æ1 22,
threshold, then we set Ti as 1 and 0 otherwise, where T; is a 8th dimension value of RIPI, zj,² an ith dimension value of 2j, and we set the threshold as hyper-parameter. For section prediction, we utilize the cross-entropy loss defined as follows:
[s| 17. c] log P(T: = c|[M; M1; [W bl),
C,
ΣΣ =E
(7)
W bb, W E 1 and by E R2² are learnable where P(T; = cMM W b:]) ps. p parameters, and E 1).
To infer the activated section of the symmetries codebook, we utilize the Gumbel softmax function to handle binary on-and-off scenarios, akin to switch operation:
G(P if pf 2 0.5 G(v())))
sw (G(())
(8)
il ph < 0.5
where ph,y is a jth dimension value of pj, and G(.) is the Gumbel softmax with temperature as 1c-4.
Integration for Composite Symmetry For the composite symmetry ge: we compute the product of weighted sums of switch function sw(p,) and prediction distribution attn as:
[s]
Je
g
(9)
where g ew(((l)))
8

---

Published in Transactions on Machine Learning Research (11/2024)
5.4 Equivariance Induction of Composite Symmetries
Motivated by the implementations of equivariant mapping in prior studies (Yang et al. 2022; Miyato el al. 2022) for disentanglement learning, we implement an equivariant encoder and decoder that satisfies (v. * 24) = 9c (2,) and po(ge o 24) = the (=();) respoctively, where do is an encoder, po is the docoder, de * 24 = ZJz and Ir o 2 = 2j. As shown in Fig. 2 example, we propose 1) the encoder equivariant loss Lee to match the 2; and Gr o 2j on the latent vector space, and 2) the decoder equivalent loss Lae to match the input 24 and generated image from the transformed vector pa(a) (e 22).
Equivariance Loss for Encoder and Decoder In the notation, de and Ii group elements of the group (V)*) and (G) respectively, and both groups are isomorphic. Each group acts on the input and latent vector space with group action and c, respectively. We specify the form of symmetry ge and o as an invertible matrix, and group action as matrix multiplication on the latent, vector space. Then, we optimize the encoder and decoder equivariant function as:
Ge o (1())) = (o 2i) 5 - +0 feee = MSF(g022, 2n)
(10)
(for encoder).
po o (o v/e (r({o(x:)) = polae qo(x,)) - 3-j + Late MSF(ppo9e qp(sx))) (. (for decoder).
(11)
Lequir Lee + eLat
(12)
For the equivariant encoder and decoder, we differently propose the single forward process by the encoder and decoder objective functions compared to previous work (Yang et al. 2022).
Objective and Base model Our method can be plugged into existing VAE frameworks, where the objective function is integrated additively as follows:
C(ø, x>) = LVAE + Loodebook + Ceguiv:
(13)
where AVAE is the loss function of VAE framework (Appendix A The other loss Loodebook Cpt + Lpd + C. + Le Ep and Lequiv Lee eLde, where is hyper-parameter to control the model performance scnsitivity.
5.5 Extended Evaluation Metric: m-FVM Metric for Disentanglement in Multi-Factor Change
We define the multi--octor change condition as simultaneously altering more than two factors in the transfor- mation between two samples or representations. To the best of our knowledge, there is no evaluation metric for disentanglement in multi-factor change, so we propase the extended version of the Factor-VAF metric (FVM) score called as multi-FVM score (m-FVMg), where factor F = Fi x... x Fn, E 32.3....,. 1}, and [F] is a number of factors. Similar to FVM, 1) we randomly choose the factor f f= (f. ((t...)))) 2) Then we fix the corresponding factor dimension value in the mini-bated 3) Subsequently, we estimate the 4t2,...) (221, in epoch. 4) We then count each pair of selected dimensions by std. values (the number of (#11 Z12, which are corresponded to fixed factors). 5) In the last, we add the maximum value of the number of (z11, #12,...) on all fixed factor cases, and divide with epoch.
standard deviation (std.) of each dimension to find the number of lowest std. dimension (211
6 Experiments
Device We set the below settings for all experiments in a single Galaxy 20801i GPU for 3D Cars and smallNORB, and a single Galaxy 3090 for dSprites 3D Shapes and CelebA. More details are in README. file.
The dSprites dataset Matthey et al. 2017) consists of 737,280 binary 64 x 64 images with five independent ground truth factors (number of values). i.e. x-position (32), y-position (32), orientation (40),
Datasets
9

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Disentanglement scores for single factor change (left 5 metrics) and multi-factor change (m-FVMs) with
random seeds.
3D Car
PVM
buta VAE
MEG
SAF
DC1
m--F VM2
PV
B-VAR
81.83(11..30
100.00, 10.00)
11.441
0.63 0.63,2011 24)
( 50)
61 28(
49.40
A-TCVAE
92.32(15.38)
100.00, 20.00)
17.191 (3.08)
1.13 00302.0.33 37)
.77
59.23 00.255:0
22..21550000 15.63)
Pactor- VAE
83.22(12.88)
100.00: 20.00)
10.84:20.000 93)
1.38(20000 68)
24.31 2000
50.43( 1 65)
Contzol-VAE
03.80(25.29)
100.000 100.00r-2 20.00000
0.73(42..44)
1.14 545
25. (5.66))4.63)
16.42:. + 101.344
CLG-VAB
21.5)(22.54)
100. oor 10.00
11,62(21.00 1565)
1.30(4-0.20 263
28 10(=1166)) 1.3
47.750 (5.83)
CEASL
08.70:+1.90)
100.00r 6,000
18.08( 11220
1.43(+0.18
84.81(13.80)
02:431
5,00
NORS
PV3
XIG
SAF
X-VAR
60.71(22.55
21-601 (0055a)
11:02(2:018)
24-43(133.30
4-TCVAE
09.30( 2.02)
60.401=10
21.64(=0.61)
11.31(00.37)
30.74[=0.20 0:20))
25.71(
Factor-VAL
61:03(
60.63(42.67 56.400=1.00
22.07(400.0
11,21(
24.941-0772)
4.33)
21-55020.00
11.19(20.08
25.07(10.44
16.
CEASL
02.78(+3.96)
(0.20)
22.23(#0.45)
11.42(
(4.68(=0.61)
27.00 3.00
daprites
FVM
bata VAB
MIG
SAP
DC1
u-FVMM
un Lm-FVM
m- FVMg
3-VAR
03.54(26647)
83 T.
13.19(
5.00
21 11:40066:300
63.80 393
50 60.13(2111 100E:111:00 (8)
48 (220 8888
8-TOVAE
79.10(25.57) 57)
88. 20(24.77 78)
23. 23.05( 10.110.01 13)
7. 7:20 66)
35. 33(20.007
61.75 6.7
57 07:8 39)
68 (1
Pactor-VAB
78.10(2 45)
84 24-400220000 55)
25 25.74 100.0000 58)
6.37111 52)
32. 30
58.39 1.5 18)
51 $1.43322 863
50 (22)22)
Control-VAR
69.61(17.67)
82. (012T-30 7-79)
5.931 1.778)
3.89 1 89)
12. 12-121-3 1
38 901 19.31)
28 00: 10:00011:700 15.
18 33--3355-0000 96)
CLG-VAR:
$2.98:+6.50)
86 -0.4053.445
28.96(+6.05)
+008)
51, 11.2826.882)
68.21( +-118)
48.68 0.00)
51 (0.1..11)
CEASI
82.30(13.88)
90 20020(13::00
15-533
33.82; 18. 183
7.28 110.0
46.52(
68.32 2 13)
66.25 T
71.351 112.00)
Mg
84.33(110.6G
41.20(=4400
20)
80)
06.03(=7.44)
70.26
01.52(4: 02)
70. 70.38[[1.03)
06)
63.61(133.90)
VAR
61.37(-1240
Conteol-VAE
1000+11..00
19.61 53)
4.76[+2.799
133.11
95)
65.58: 10-+-18:00
12.08)
CLG-VAE
83.10/
76)
0.36(+=1.08)
63.02 03.02(=3.80)
6- 50)
00.01117000 62)
Table 4: Disentanglement performance rank. Each dataset rank is an average of evaluation metrics, and Avg. is an average of all datasets.
smallNORB
3-TCVAE
4.29
2.58
-VA
2:71
4.00
Costrol-VAR
3.67
3.86
6.00
5.50
4.76
CLG-VAE
3.00
3.86
3.13
4.13
3.53
CPAEL
1.00
1.43
1.13
1.13
1.17
shape (3), and scale (6). Any composite transformation of x- and y-position, orientation (2D rotation), scale, and shape is commulative. The 3D Cars Reed el al. (2015) datasel consists of 17,568 RGB 64 x 64 x images with three independent ground truth factors: elevations ()), azimuth directions(24), and car models (183). Any composite transformation of clevations(x-xxis 3D rotation). azimuth directions (y-axis 3D rotation). and models are commutative. The smalINORB LeCun et al. (2004) dataset consists of total 96 x 96 24,300 grayscale images with four factors, which are calegory(10), e9evation(9), azimuth(18), (6) and resize the imput as 64 x 64. Any composite transformation of elevations(x-axis 3D rotation), azimuth (y-axis 3D rotation), light, and category is commutative. 4) The 3D Shapes datasct Burgess & Kim (2018) consists of 480,000 RGB 64 64 3 images with six independent ground truth factors: orientation shape(4) (4), floor color(10), scale(8), object color(10), and wall color(10), (10). 5) The CelebA dataset Liu al. (2015) consists of 202,599 images, and we crop the center 128 128 area and then, resize to 64 64 images.
Evaluation Settings We set prune dims. threshold as 0.06, 100 samples to evaluate global empirical variance in each dimension, and run it a total of 800 times to estimate the FVM score introduced in Kim & Mnih (0018). For the other metrics, we follow default values introduced in Michlo (2021), training and evaluation 10,000 and 5,000 times with 64 mini-batches, respectively Cao al. 2002).
Model Hyper-parameter Tuning We implement 8-VAE Higgins et al. (017)). 3-TCVAE Chen et al. 20188, control-VAL Shao et al. (2020). Commutative Lie Group VAE CLG-VAE) Zhu et al. (2021) and Groupified- VAE (G-VAE) Yang et al. (022) for baseline. For common settings to baselines, we set
Table 5: Comparison of disentanglement scores of plug-in methods in single factor change.
Dalaneto
U-VAR
PVX
CPASS
G-VAR
CPAST
G-VAR
BAP
CPASI
GEVAR
CIFASI
Japrites
09 00.004113.00 06
82.30
21 1000T 20)
33.02 18)
5.6(=2266
7.28(-6663
BL 87)
40.82(-0115)
3D Car
92 02(44(0)
06.70(=1.90)
11. 00(022111))
18.58 2.224
2.10
1.43(20.18)
26.01(+6.24)
34.81(=1.85)
NROB
26.64(11.45)
01.15(24.28)
20. 20)60(11.22)
22.28(
10.37(20.51)
11.12(20.45)
27.77(10.08)
24.50(20.61)
10

---

Published in Transactions on Machine Learning Research (11/2024)
batch size 64, learning rate 1c-4, and random seed from 21. 10) without weight decay. We train for 3 x 109 iterations on dSprites smallNORB and 3D Cars, 6 x 10 iterations on 3D Shapes, and 10 iterations on CelebA. We set hyper-parameter B E 2.0,, 6.0} for 3-VAE and 3-TCVAE, fix the a for B-TCVAE as 1 Chen et al. (2018). We follow the ControlVAE scttings Shao et al. (2020), the desired value C' E {10.0, 12.0, 14.0, 16.0), and fix the K = 0.01, K, 0.001. For CLGGVVB we also fol- low the scttings Zhu et al. (2021 as Ahession = 40.0, Adecomp = 20.0, p 0.2, and balancing parameter of lossrec group E {0.1, 0.5, 0.7}. For G-VAE, we follow the official settings Yang al. (2022) with 3-TCVAE (3 E {10, 20, 3, because applying this method to the 3-TCVAE model usually shows higher performance than other models Yang et al. (2022). Then we select the best case of models. We run the proposed model on the 3-VAE and 3-TCVAE because these methods have no inductive bias to symmetries. We set the same hyper-parameters of baselines with e E (0.1, threshold E (0.2,0.5), |S| = =|SS| = [D], where [D] is a latent vector dimension. More details for experimental settings.
6.1 Quantitative Analysis Results and Discussion
Disentanglement Performance in Single and Multi-Factor Change We evaluate four common disentanglementt metries: FVM (Kim & Mnih 2018), MIG (Chen et al. 2018), SAP (Kumar et al 2018), and DCT (Fastwood & Williams 2018). As shown in Table 3 our method gradually improves the disentanglement learning in dSprites, 3D Cars, 3D Shapes, and smalINORB datasets in most metrics. To show the quantitative score the disentanglement in multi-factor change, we evaluate the m-FVM;, where max(k) is 2, 3, and in 3D Cars, smallNORB, and dSprites datasets respectively. These results also show that our method positively affects single and multi-factor change conditions. As shown in Table f the proposed method shows a statistically significant improvement, as indicated by the higher average rank of dataset metrics compared to other approaches.
Table Ablation study for loss functions on 3D-Car and B. VAE with 10 random seeds.
2p
Le
Lequie
pd
MIG
SAF
m-FVKg
58. $8.19(21.00)
6.82( (2.9()
0.63 10.33
70-45(13.555
12.36(17.16)
88
58-57(26.65) 68)
7.19: 22.32)
1885 11.00)
18.39 1: (0)
48.231 25.51)
$8. 38.00(=7.75)
7.271 16)
1.31 70
10.58 24 45)
42.68 24.21
8-VAR
30.80022.000 (666
02-42227:600
7.301 1:3.
7.11(13.40)
3.35: 1.0 70)
1.00 1:0 40)
21 00
18.35 13.33)
70)
41.90( 17.90
41.90( 26.07)
89.34(-8.15)
9.44 +2.91)
1:20 4.400)
29.14
5.511
11.37(++9.20)
90.71(-8.75)
9.201
1.07 +0.0++0.00
22.74:
45.84|
Comparison of Plug-in Methods To compare our method with a wider range of approaches, we evaluate the disentanglement performance of the plug-in style method, G-VAE Yang et al. (2022) and apply both methods to B-TCVAF. As shown in Table our method shows statistically significant improvements in disentanglement learning although 3 hyper-parameter of CFASL is smaller than G-VAE
Ablation Study Table shows the ablation study evaluate the impact of each component of our method for disentanglement learning. In the case of w/o Ept, the extraction of the composite symmetry ge becomes challenging due to the lack of unified roles among individual sections. Also, the coverage of code w/o Epd is limited due to the absence of assurance that each section aligns with distinct factors. In the case of w/o L., each section assigns a different role and the elements of each section align on the same factor, and w/o C. case is better than w/o Lpt and w/o Lpd. These results imply that the no-differentiated role of each section struggles with constructing adequate composite symmetry 9c- Also, it shows that dividing the symmetry information in each section (Cnt, Lnd) is more important than L, for disentangled representation. To compare factor-aligned losses (w/o Lpt: w/o Epd, w/o C., and w/o E,t + End Lz), the best of among four cases is the w/o Cpt + Cpd - C. and it implies that these losses are interrelated. Also, constructing the symmetrics without the equivariant model is meaningless because the model docs not satisfy Equation I0 12 The w/o Lequiv naturally shows the lowest results compared to other cases except w/o Lpd and Lpt- Moreover, the w/o Cp case shows the impact of the section selection method (section 5.3 for unsupervised learning. Above all, cach group exhibits positive influence on disentanglement when compared to the base model (8-VAE) When combining all loss functions, our method consistently outperforms the others across the majority of evaluation metrics.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Table Additional experiments
(a)
tuning with 6 random seeds.
(b) Codebook size impact
Betn
10
0.01
87.38(+7787)
38)
8.96(-1.10) +.45(+0.01)
95.701 11000
uSt
7557724.00
82.21 94)
99.33
95,20
48.07(+5.00 38.87(#7.83)
18.08(+1.20
18.38(+13.88)
0.02(-0.45)
1.49(=0.18)
0:20(+0.84)
DCT
Divergence
HT: Encoder Equivariant Loss
HT: Decoder Equivariant Loss
KL Divergence
(a) kld (hyper)
(b) encoder (hyper)
[c) decoder (hyper)
(d) kld (w/o)
Figure 4: Loss curves: 1) HT: hyper-parameter tuning (e E (0.01,0.1,1.0)) with 3-TCVAE based CFASL. 2) AB:
ablation study with 3-VAE based CFASL.
Impact of Hyper- Hypar-Parameter Tuning We operate grid search of the hyper-paramcter As shown in Figure da] the Kullback-Leibler divergence convergences to the highest value, when is large (e = 1.0) and it shows less stable results. It implies that the CFASL with larger e struggles with disentanglement learming, as shown in Table 7a Also, the Lee in Figure 4b larger than other cases, which implics that the model struggles with extracting adequate composite symmetry because its encoder is far from the equivariant model and it also shown in Table 7a Even though
0.01 case shows the lowest value the most loss, Ldte Figure 4c is higher than others and it also implies the model struggles with learning symmetries, as shown in Table 7a because the model does not close to the equivariant model compare to 0.1 casc.
Posterior of CFASL The symmetry codebook and composite symmee- try are linear transformations of latent vectors. Intuitively, they enforce the posterior
be far from prior as (0(z(z) N[get,, geLig), where μ and Σ are close to zero vectors and identity matrix, respectively. However, as shown in Figure Ak] Kullbeck Leibler divergence is lower than VAF. Tt represents the ability of CFASL to preserve Gaussian normal distribution, which is similar to VAE.
P-TCVAE
Í-TCVÉ (Ours)
Figure 5: Heatmaps of Eigenvec- tors for latent vector representa- tions.
Impact of Factor- Aligned Symmetry Size We set the codebook size as 100, and 10 to validate the robustness of our method, In Table 7b the larger size shows better results than the smaller one and is more stable by showing a low standard deviation.
x;
x
9.0
0.
g'
Po (Z) (g 41) p 22)0 (z)
Figure 6: Generated images composite symmetry and its factor-aligned symmetries. Image ① and 2 are imputs, and image & is an output from image ( (ps(z))). Image ② is a output of group element ga acted on 21 (p(ga21)). Images ① outputs of decomposed composite symmetry ge acted z sequentially.
12

---

Published in Transactions on Machine Learning Research (11/2024)
xi
x?
Basc.
94
94
Select largest n KLD value dimensions
1
21
1
Ours
pe
Pe
P8
2i shape @: color @ azimuth none
(a) Overview and 3D Cars results.
4
2+
Base.
1
8
9
3 3 3
Base.
6
4
Ours 7
3 3 3 3 3 2
Ours
2.
Thinning 2: Thickening @: Swelling @: Fracture
① Shape @ Scale @: Rotation @ Object color
G: Slant @ number none
5: Wall color @: Floor color none
(b) Morpho-MNIST and 3D Shapes results.
Figure Generated images by dimension change. Red and blue colored squares represent the value of latent vector dimensions of 21 and 22. The images of baseline and CAFSL are the generated images from each latent vector.
6.2 Qualitative Analysis Results and Discussion
Is Latent Vector Space Close to Disentangled Space? The previous result shown in Figure ! is clear example of of whether the latent vector space closely approximates a disentangled space. The latent vector space of previous works (Figure la Iel are far from disentangled space (Figure Ig) but CFASL shows the closest disentangled space compare to other methods.
Alignment of Latent Dimensions to Factor-Aligned Symmetry In the principal component analysis of latent vectors shown in Figure 5. the eigenvectors V = [0i, 02. dj] are close to one-hot vectors compared to the bascline, and the dominating dimension of the one-hot vectors are all different. This result implies that the representation (factor) changes are aligned to latent dimensions.
Factor Aligned Symmetries To verify the representation of learnable codebook over composite symme- tries and factor-aligned symmetries, randomly select a sample pair as shown in Figure G The results imply that de generated from the codebook represents the composite symmetrics between two images (L) and (2) because the image 2 and the generated image @ by symmetry are similar (p(22) R p(gaz1)). Also, each aactor-aligned symmetry (gi) generated from codebook section affects a single factor changes as shown in images ( in Figure 6
Factor Aligned Latent Dimension To analyze each factor changes aligned to each dimension of latent vector space, we set the qualitative analysis as shown in Figure 7 We select the baseline and our model, which are the highest disentanglement performance scores. We select two random samples (æ1, x2), generate latent vectors 21 and 22, and select the largest Luibee divergence (KLD) value dimension from their posterior. Then, replacing the dimension value of 2 the value of 22 one by one sequentially. As a result, the shape and color factors changed when a single dimension value is replaced within the baseline
13

---

Published in Transactions on Machine Learning Research (11/2024)
x:
X2
Xx-1 X
X1
X2
Ak-1
X
9p.
99+
994
B(2.2)
Zg
-
B(1-11)
Ax
(p(())) @Ps(012))4) ... 3 Po(((x-1R)4A--)
Figure 8: Generalization over unseen pairs of images. We set pairs {(I)-1, < i ||||| - 13 then extract the symmetries between elements of each pair 9p ( -(a.3) 0(k-1,0)) in inference step, where 0(k-1,4) is symmelry between and The first row images are imputs (largets) and the second row images are the generated images by symmetry codebook.
the 3D Cars dataset, shown in Figure 7a However, method results show overlapped factor changes compared to baseline results. Also, the baseline results contain the changes of multiple factors in a dimension, but ours reduces the overlapped factors or contains only a single factor, as shown in Figure 7b. It shows that our model covers the diversity of datasets compared to the basclinc.
Unseen Change Prediction in Sequential Data The sequential observation as Miyato et al. (2022) is rarely observed in our methods, because of the random pairing during training (less pair observation). But their generated images trained symmetries of our method are similar target images as shown This result implies that our method is strongly regularized for unseen change.
Figure 8
7
Conclusion
This work tackles the difficulty of disentanglement learning of VAEs in unknown factors change conditions. We propose a novel framework to learn composite symmetries from explicit factor-aligned symmetries by codebook to directly represent the multi-factor change of a pair samples in unsupervised learning. The framework enhances disentanglement by learning an explicit symmetry codebook, injecting three inductive biases the symmetries aligned to unknown factors, and inducing group equivariant. VAF model. We quantitatively evaluate disentanglement in the condition by novel metric (m-FVM;) extended from a common metric for a single factor change condition. This method significantly improved in the multi- factor change and gradually improved in the single factor change condition compared to state-of-the disentanglement methods of VAEs. Also, training process does not need the knowledge of factor information of datasets. This work can be easily plugged into VAEs and extends disentanglement to more general factor conditions complex datasets.
8 Limitation and Future Work
In the real-world dataset, the variance of the factor is much more complex and has more combinations than the used datasets in this paper (the maximum number of factors is five). Although our method shows the advance disentanglement learning in multi-factor change conditions, remains the generalization real world datasets or larger dataset. We consider extending the learning of composite symmetries in general conditions. Another drawback use of six loss functions, which require more hyper-paramctor tuning. As the statistically learned group methods reduce hyper-parameters (Winter et al. 2022 we consider a more computationally efficient loss function.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) N0.2022R1A22201205. Development of AI for Canonicalized Expression of Trained Hypotheses by Resolving Ambiguity in Various Relation Levels of Representation Learning).
14

---

Published in Transactions on Machine Learning Research (11/2024)
References
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8))1798 1828, August 2013. ISSN 0162-8828. doi: 221109/tpami.0133 URL https://doi..org// 1111//TTPMII 2013.50
Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny. Addressing the topological defects of disentangle- ment via distributed operators, 2021.
Chris Burgess and Hyunjik Kim. 3d shapes dataset.
http://innnn/iiaaaassaasassseeee 2018..
2018.
Jinkun Cao, Ruiqian Nai, Qing Yang, Jialei Huang, and Yang Gao. An empirical study disentanglement of negative-free contrastive learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httts:///openrevier net/ forum?id=f.ll (i-fmmidkdJJ
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of dis- entanglement in variational autoencoders. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https://proceedings wwwpooceddinns..nuiss...... wwwwrip..cceepeeee hatprpsss.c/ppppp///////LL pdf
1ee3df seeaaffddddd22233377777722222222
Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. In A. Beygekzmer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, 2021. URL httphttt////oeereevie net.sewuuwee//rrr
1d=NPOWS ZLfC5
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference on Learning Representations, 2018. URL https: //openreview wwt.:/ooummdddd-y---aaa.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint arXiv:2002.12 12880, 2020.
B. Hall. Lie Groups, Lie Algebras, and Representations: An Elementary Introduction. Graduate Texts in Mathematics. Springer International Publishing, 2015. ISBN 9783319134673. URL https://books.. google.co httpp/o...../ooohheeecddddddd..
Trina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Trina Higgins, David Amos, David Pfan, Sébastien Racanière, Loic Matthey, Danilo J. Rezende, and Alexan- der Lerchner. Towards definition of disentangled representations. CoRR, hts//812.00230,, 2018. URL
wttpr//.rrr/bb//112...o http://atxiv.org// httg/bbs/18122 02200
Trina Higgins, Sébastien Racanière, and Danilo Rezende. Symmetry-based represttttissssss for artificial and biological general intelligence, 2022. URL httrp::/aaxxvv.or//b//2000
Yconwoo Jeong and Hyun Oh Song. Learning discrete and continuous factors of data via alternating disentan- glement. In Kamalika. Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 3091-3099) PMLR, 09 15 Jun 2019. URL httphhtt///ptt https://proceedingg. mlr http:/cccenngsla..ppppe////////p html
T. Anderson Keller and Max Welling. Topographic vaes learn equivariant capsules. CoRR, abs/ 2109.01344 2.21a. URL https: : arxiv owg/abs//////oooooooo..oooe.. httpaa/////19999..00 01394
15

---

Published in Transactions on Machine Learning Research (11/2024)
T. Anderson Keller and Max Welling. Topographic VAEs learn equivariant capsules. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Sys- tems, 2021b. URL
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.), Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.
PMLR, 10 15-15 Jul 2018. URL Utt.htttss//ppocceiinng mlr. wwt.mmlrpprr... mml..prees/vv..
kim18b. [
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL https://arxiv.org/ 11s//312.6114]
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. VARIATIONAL INFERENCE OF DISEN- TANGLED LATENT CONCEPTS FROM UNLABELED OBSERVATIONS. In International Conference on Learning Representations, 2018. URL https://openreviiet http://ppeaarrreeee.ett/oooo/o/ www.teeeetereee
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invari- ance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pultern Recognition, CVPR'04, pp. 97 104, USA, 2004. IEEE Computer Society.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3730-3738, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Muchine Learning Resewrch, pp. 4114-4122 html
PMLR, 09-15 Jun 2019. URL
mlr wttp:psees////seeeeeeeeeeeeeeee
Giovanni Luca Marchetti, Gustaf Tegnér, Anastasiaa Varava, and Danica Kragic, Equivariant representation learning via class-pose docomposition. In Francisco Ruiz, Jennifer Dy, and Jan-Willen van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statislics, volume 206 of Proceedings of Machine Learning Research, pp. 4745 44554776..60 PMLR, 25 27 Apr 2023. URL https: //proceedings. weeeeeddiinnnmm.......mmm wttpppres////..maaareeeeetttttte html
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexandor Lorchner. dsprites: Disentanglement testing sprites dataset.
hewpm///dddeeeiiieee/lleeaatseee 2017.
Giangiacomo Mercatali, Andre Freitas, and Vikas Garg. Symmetry-induced disentanglement graphs. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 31497-31511. Curran As- sociates, Inc., 2022. URL https://proceeddngg/ courips..c/paper filles/paperr/222//iiee
Nathan Juraj Michlo. Disent modular 2021. URL https////gituub..oommmiciieoo disent
hibemmt--ammolllaariiinnnnaagag disentangeed representation learning framework for pytorch. Github,
Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Alice H. Oh, Alckh Agarwal, Daniclle Belgrave, and Kyunghyun Cho (ods.), Advances in Neural Information Processing Systems, 2022. URL https::/openrevieer nett/forrummtdd77771kkk
Alireza Nasiri and Tristan Bepler. Unsupervised object representation learning using translation and rotation group equivariant VAE. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrevier: htwppppeereeeeieeeeeeeoorrriiiia net/forumn??d
gmm MMjMLL
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and
16

---

Published in Transactions on Machine Learning Research (11/2024)
H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19727-19737. Cur- ran Associates, Inc., 2020. URL https://proceediing http:p//ppoceeal.ss.ersp.....ppp wwt.seeecrppppppppppeeeeeeeeeeee e4665577aabb00aaa000223334422...
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In C. Cortes,
N. Lawrence, D. Lec, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL htttps:/pproceedingg weuripp.cc/pappe
20D/:111/001133000000000E0000000 Papere,aapprppppepa pdf
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarck Abdelzaher. ControlVAE: Controllable variational autoencoder. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.
PMLR, 13-18 Jul 2020. URL httpss//procceeiing mlr
press/v119/shao200 html
Huajie Shao, Yifei Yang, Haohong Lin, Longzhong Lin, Yizhuo Chen, Qinmin Yang, and Han Zhao, Re- thinking controllable variational autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognation (CVPR), pp. 19250-19259, June 2022.
Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Arné Clevert. Unsupervised learning of group invariant and equivariant representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:
/
Changyi Xiao and Ligang Liu. Generative flows with matrix exponential. In Hal Daumé 111 and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10452-10461 PMLR, 13-18 Jul 2020. URL htttps:/pproceednngs
mlr. press/v119/xiaa2000 html
Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards building group- based unsupervised representation disentanglement framework. In International Conference on Learning Representations, 2022. URL http:p/////pgpooooce.e.eo///e//e
Xinqi Zhu, Chang Xu, and Dacheng Tao, Commutative lie group VAE for disentanglement learning. CoRR, abs/2106.03375, 2021. URL https://arxiv http:///araa..rrr/bbb//o0..337
17

---

Published in Transactions on Machine Learning Research (11/2024)
A Loss Function of Baseline
As shown in Table 8 train the baselines with each objective function.
VAEs
3-VAE
Bngle log pa P2 (oa
Baaltle) log po(a
--TCVAE
II, a(x,)) (Pzz(
En DeL (o(e))))))))))
aet [log p(a [=]] P
Factor-VAL
PuL (n(=)]] II, (#5))
Control-VAF
(agl Po]])(=)2) ((A) ( (==))=)))))))
(e log p(=])) E (alla Pxz pp)(())) Byts loga(a)a)
CLG-VAE
Table Objective Function of the VAEs.
B Perpendicular and Parallel Loss Relationship
We define parallel loss Lp to set two vectors in the section of the symmetries codebook to be parallel: زj || 9)22 then,
# - - = olz g/.z)
(14)
(1 c)z (0) cg). )z
(15)
I) = 9} cgj or [(1-e] + ogg. 9/]== 0,
(16)
where I is an identity matrix and constant E R. However, all latent not eigenvector of ((1-c) (+gg) -9 Then, we generally define symmetry
1 o3
1-
- -3 c
(17)
where i, j, and j' are natural number 1 i | 1 S j.j S |SS), and k * j. Therefore, all symmetries in the same section are parallel then, any symmetry in the same section is defined by a specific symmetry the same section.
We define orthogonal loss C. between two vectors, which are different sections, be orthogonal: z-93z 1
z of 2, where i + k, 1 < i < |S), and 1 < j |SS]. By the Equation 17
0/2 1 - =-g²=
(18)
Co
Co
Co go +
I) - z 1
gb
I)z z
(19)
1
2)
1 (gt x - z),
(20)
Cb
where ca and 4 are any natural number, and 1 1 a. < [SS]. Therefore, two vectors from different
sections are orthoglnal and satisfied with Equation 17 then any pair of vectors from different sections is always orthogonal.
C Additional Experiment
C.1 Disentanglement Performance
Reconstruction vs. FVM We conduct the trade-off between the reconstruction and disentanglement performance as shown in Figure g We consider the results on the complex dataset because the trade-off is more distinct in the complex dataset such 3DShapes.
Complex Dataset. We show the model performance with MPI3D dataset.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Model Performance
0.90
FVM Score
0.85
1
4
6
B-VAE
0.80
B-TCVAE
CLG-VAE
2
0.75
0.7 0.2
CFASL
0.19
0.70
20
25
30
35
Reconstruction Error
Figure 9: Reconstruction loss vs. Factor VAE metric on 3D Shapes dataset. The numbers next each plot represent the value of los8rec group of CLGG-VAF and the others are the value of B parameter.
FVM
beta VAE
p-value
FVM
MIG
SAP
DCI
3-VAE 8-TCVAE CFASL
16.09(+0080)
36.40(+8..33
dSprites
0.011
0.005
0.016
0.001
16.95(+ 1.38)
51.75(+9.22) (4.80(14.54)
3D Cars
0.006
0.000
0.97
0.003
(7.43()3.83)
smalINORB
0.000
0.002
0.000
1.000
(a) Disentanglement performance on the MPI3D dataset.
(b) p-value estimation on cach datascts.
3D Shapes
3-VAE
3-TCVAE
Factor- VAE
Control-VAR
CLG-VAE 73.61
OURS
m- m-FVMs
80.26(13.78)
79.21(45.877)
76.69(15.08
73.31(16.51)
83.03
(c) m-FVMs results.
Table 9: Additional experiments
Statistically Significant Improvements As shown in Table 9b model significantly improves disen-
tanglement learning.
3D Shapes As shown in Table 9c CFAST also shows an advantage multi-factor change.
C.2 Ablation Studies
How Commutative Lie Group Improves Disetanglement Learning? The Lie group is not mutative, however most factors of the used datasets are commutative. For example, 3D Shapes dataset factors consist of the azimuth (x-axis), yaw (z-axis), coloring, scale, and shape. Their 3D rotations are all commutative. Also, other composite symmetries as coloring and scale are commutative. Even though we restrict the Lie group to be commutative, our model shows better results than baselines as shown Table
Impact of Commutative Loss on Computational Complexity As shown in Table I0 our methods reduce the composite symmetries computation. Matrix exponential based on the Taylor series and it needs high computation cost though its approximation is lighter than the Taylor series. We need one matrix exponential computation for composite symmetrics with commutative loss, in contrast, the other case needs the number of symmetry codebook elements |S| |SS] for the matrix exponential and also |S] |SS| - 1 time matrix multiplication.
6c
Ce
Table 10: Complexity.
19

---

Published in Transactions on Machine Learning Research (11/2024)
C.3 Additional Qualitative Analysis (Baseline CFASL)
Figure and
and 14a show the qualitative results on 3D Cars introduced in Figure G Figure and Figure 13
show the dSprites and smalINORB dataset results, respectively. Additionally, we describe Figure 12 and 4cc results over 3D Shapes datascts. We randomly sample the images in all cascs.
CFAST shows better results than the baseline. In the 1st and 2nd rows, the baseline changes shape and color factor when single dimension value is changed, but ours clearly disentangle the representations. Also in the 3rd row, the bascline struggles with separating color and azimuth but CFASL successfully separates the color and azimuth factors.
3D Cars As shown Figure 10c
1st row: our model disentangles the shape and color factors when the 2nd dimension value is changed.
2nd row: ours disentangles shape and color factors when the 1st dimension value is changed.
4 row: ours disentangles the color, and azimuth factors when the 2nd dimension value is changed.
dSprites As shown in Figure 11c the CFASL shows better results than the bascline. The CFASL signif- icantly improves the disentanglement learning as shown in the 4th and 5th rows. The baseline shows the multi-factor changes during a single dimension value is changed, while ours disentangles all factors.
1 row: ours disentangles the x- and y-pos factor when the 2nd dimension value is changed.
2nd row: ours disentangles the rotation and scale factor when the 2nd dimension value is changed.
3rd row: ours disentangles the x- and y-pos, and rotation factor when the 1st and 2nd dimension
values are changed.
4th row: ours disentangles the all factors when the 1st and 2nd dimension values are changed.
3D Shapes As shown in Figure 12c the CFASL shows better results than the baseline. In the 1st 3rd, and 5th rows, our model clearly disentangles the factors while the bascline struggles with discntangling multi- factors. Even though our model does not clearly disentangle the factors, compared to the baseline, which is too poor for disentanglement learning, ours improves the performance.
1s row: our model disentangles the object color and floor color factor when the 2nd and 3rd dimension values are changed.
2nd row: ours disentangles shope factor in 1st dimension, and object color and floor color factors at the 4th dimension value are changed.
3rd row: ours disentangles the object color and floor color factor when the 3rd dimension value is changed.
4th row: ours disentangles the scale, object color, wall color, and floor color factor when the 2nd and 3rd dimension values are changed.
5th row: ours disentangles the shape, object color, and floor color factor when the 1st and 2nd dimension values are changed.
smallNORB Even though our model does not elearly disentangle the multi-factor changes, ours shows beiter results than the baseline as shown in Figure 13c
s row: our model disentangles the category and light factor when the 2nd dimension value is changed.
3rd row: ours disentangles category factor and azimuth factors when the 5th dimension value is changed.
20

---

Published in Transactions on Machine Learning Research (11/2024)
(a) Generated images by composite symmetry on Cars dataset. The images the box are inputs.
images in the hlue box at odd column are same as 3 and even column are same as ? in Figure
Generated images by factor-aligned symmetries on Cars dataset. The images are same as
Fig
Baseline
CFASL
W
E
B
shape 2: color 3:
azimuth
none
(c) Generated images by dimension change 3D Cars dataset.
Figure 10: Figure 10a shows the generation quality composite symmetries results, Figure 10b shows the disentan- glement of symmetries by factors results, and Figure T0c shows the disentanglement of latent dimensions by factors results.
21

---

Published in Transactions on Machine Learning Research (11/2024)
(a) Generated images by composite symmetry dSprites dataset. The images in the red box are imputs.
The images in the blue box at odd column are same as @ and even column are same in Figure
Generated images by its factor-aligned symmetries dSprites datset.
The images
Baseline
CFASL
12
X
2
2.2 (3
1 x-pos 2: y-pos 3: rotation 4: scale 5
shape none
(c) Generated images by dimension change dSprites datasct.
Figure 11: Figure 1la shows the generation quality of composite symmetries results, Figure 11b shows the disentan- glement of symmetries by factors results, and Figure 11c shows the disentanglement of latent dimensions by factors results.
22

---

Published in Transactions on Machine Learning Research (11/2024)
(a) Generated images by composite symmetry on 3DShapes datasct. The images in the rod box are inputs.
The images in the blue box at odd column are same as 3 and even column are same as @ in Figure
(b) Generated images its factor-aligned symmetries 3DShapes datset. The images
Fig
Baseline
CFASL
5
4
0.255
245
B
6
2
2
2.0.
1: shape 2 scale 3: rotation 4: object color 5: wall color 6: floor color none
(c) Generated images by dimension change on 3DShapes datasct.
Figure 12: Figure 12a shows the generation quality of composite symmetriee results, Figure 12b shows the disentan-
glement of symmetries by factors results, and Figure 12c shows the disentanglement of latent dimensions by factors results.
23

---

Published in Transactions on Machine Learning Research (11/2024)
(a) Generated images by composite symmetry on smallNORB dataset. images in box are
inputs. The images in the blue box at column are same as i and even column are same as @
Figure B
(b) Generated images by its factor-aligned symmetries on SORB datset. The images are same
as 4 in Fig
Baseline
CFASL
E+
2
1
1.3
C.3
category 2 elevation 3: azimuth 4: light none
(c) Generated images by dimension change on smallNORB datasct.
Figure 13: Figure 13a shows the generation quality of composite symmetries results, Figure 13b shows the disentan-
glement of symmetries by factors results, and Figure 13c shows the disentanglement of latent dimensions by factors results.
24

---

Published in Transactions on Machine Learning Research (11/2024)
Target
Output
Elevatin
Target
Outgut pa()
Mode
Target
Output P
Azimath
Generalization over unseen pairs of images on 3D Car dataset.
langet
Output P-
Shape
Scale
Target x
Output P+)-)
Rotation
Targel Oulpul (-)
X-pos
Target
Output (
Y-pos
(b) Generalization unseen pairs of images on dSprites catasett
Target x
Ourput ())
Scale
Wall Color
Target
Output Po (+)
Shupe
Rotation
Target
Output Po(-)
Floor Color
Object Color
(c) Ceneralization over unseen pairs of images on 3DShapes dataset.
Figure Generalization over unseen pairs images. The proposed method represents the sequential symmetries over the whole factor of each dataset. Generated images follow the same process of Figure g
25


Reviews:
[{"review_id": "N1Y7D8VPvQ", "reviewer": "Reviewer_FuzW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not very familiar with this theory and lack domain knowledge regarding the experiments. \n\nMy current perspective is, the introduction of CFASL is a significant advancement in the field of disentanglement learning. By removing the dependency on known factors, the method opens new avenues for unsupervised learning.\nInnovative Features: The three components of CFASL—inductive bias for latent vector alignment, composite symmetry learning, and group equivariant training—are well-articulated and contribute meaningfully to the overall framework.\nEvaluation Metric: The proposed extended evaluation metric for multi-factor changes provides a more comprehensive assessment of disentanglement performance, which is a valuable addition to the existing literature.\nEmpirical Results: The quantitative results indicate that CFASL significantly outperforms state-of-the-art methods in both single-factor and multi-factor change conditions, supporting the authors' claims about the effectiveness of their approach.\nAreas for Improvement.\n\nWhile the paper introduces several novel concepts, some sections could benefit from greater clarity and detail. For instance, a more thorough explanation of the architecture and training process of the group equivariant encoder-decoder would enhance understanding.\n\nComparison with Baselines: Although the paper reports improved performance over existing methods, more extensive comparisons with a broader range of baseline models would strengthen the validation of CFASL. Including qualitative comparisons alongside quantitative metrics could provide deeper insights into the model's behavior.\nLimitations and Future Work: A discussion on the limitations of the proposed method and potential avenues for future research would be beneficial. Addressing scenarios where CFASL might struggle or fail to perform could provide a more balanced perspective.\nGeneralization: While the method shows promise, it would be helpful to discuss the generalizability of CFASL across different datasets and contexts. Including experiments on diverse datasets could bolster claims about the method's robustness and versatility.\n\nConclusion: Overall, the paper presents a compelling and innovative approach to unsupervised disentanglement learning in VAEs through CFASL. The method's novel features and strong empirical results indicate significant contributions to the field. With some improvements in clarity, comparative analysis, and discussions of limitations, this work has the potential to make a notable impact on future research in disentanglement learning.\n\nI recommend accepting this paper for publication, contingent upon addressing the aforementioned areas for improvement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I am not very familiar with this theory and lack domain knowledge regarding the experiments. \n\nThe paper presents a novel method, Composite Factor-Aligned Symmetry Learning (CFASL), aimed at enhancing disentanglement learning in Variational Autoencoders (VAEs) without requiring prior knowledge of dataset factor information. The authors argue that existing unsupervised methods often depend on known factors, which limits their applicability. CFASL introduces several innovative features, including an explicit learnable symmetry codebook, the concept of composite symmetries, and a group equivariant encoder-decoder architecture. Additionally, the authors propose an extended evaluation metric for assessing multi-factor changes, demonstrating the effectiveness of CFASL through quantitative and qualitative analyses.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Typo: latnet -> latent", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yxh2AqZaF7", "reviewer": "Reviewer_AYog", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper meets the evaluation criteria of TMLR. It is without doubt interesting for a relatively broad audience within the machine community, by the virtue of introducing a new qualitatively different solution for a commonly studied problem. The claims are well supported by empirical evidence and the mathematical reasoning is sound. However, I acknowledge that I have not been able to verify the correctness of the derivations and new method contributions in detail, as someone not particularly familiar with the formalisms used here.\n\nStrengths:\n - The work is well positioned within the related work, including also recent papers, and it includes clear novel elements.\n - The main contributions are clear.\n - The empirical evaluation is thorough, both in terms of comparison methods and the evaluation tasks. Qualitative comparisons are supported with clear illustrations, and already Fig. 1 helps understanding the key qualitative difference well.\n- The paper includes several illustrations that help understanding how the method works and the illustrations have been designed carefully to emphasize the main findings.\n\nWeaknesses:\n - The paper is understandable, but the language could be improved in many places. For instance, some sentences are missing a verb. Perhaps consider using proofreading service?\n - The paper is not completely self-contained, for instance not presenting the basic VAE principle and objective at all. The interested readers can be expected to be familiar with VAE, but it would still be better to introduce e.g. the notation used in this work properly. In Section 2.2 you cover previous works by saying they introduce \"an additional objective\" which is a bit confusing when the main objective has not even been mentioned.\n- The mathematical notation could be improved for clarity. At least write \"attn\" and \"softmax\" (on page 5) with something like \\text{attn}. What is the symbol in Eq. (4)?\n- The motivations for the key elements are brief and sometimes missing. For instance, in Section 4.3 you clearly describe what you did but not really why these things were done, or what is the effect of the choices you made (e.g. enforcing commutativeness). Explaining the reasons and their implications in a bit more detail would be useful for the readers.\n- The manuscript is somewhat crammed, clearly requiring both compact writing and tricks for figure placement and sizing to fit into the 12 pages. Many of the good illustrations are a bit lost due to being squeezed so small (even the illustrations in Appendix are a bit smallish) the tables are all using someone small font size, and as mentioned above the paper is lacking background and verbal explanations. I think the paper would work better if it was extended to 15-18 pages (without adding new technical content).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an unsupervised method for disentanglement in VAE learning, by leveraging symmetries and using tools from group theory. The new method, CFASL, includes several new components that are adequately motivated and whose effects are demonstrated in an ablation study. The authors additionally propose a new evaluation metric. The method is evaluated in several benchmark data sets and shown to outperform previous methods, and the paper includes numerous visualisations illustrating the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see the writing double-checked for correctness and clarity, but do not see need for any other substantial changes.\n\nSince the authors are anyway short on space, the remark about providing proper background and notation for standard VAE could be done in Supplement; just write in Section 2 that in this paper you assume the reader is well familiar with basic VAE but that you provide a brief summary in Appendix so that the overall objective etc are clear.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qeT2D6rk3e", "reviewer": "Reviewer_33dK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** I find the considered task (namely, learning a factor aligned VAE whose latent space holds in correspondence to symmetries in the data) very interesting. While I have some reservations regarding the proposed approach, I view the following as strengths of this work:\n\n1. The proposed method has several new components on top of existing approaches, which enable extracting transformations between inputs and defining symmetries on the latent space. This is achieved using an explicit codebook for symmetries and an injected inductive bias that matches each symmetry to a single factor change.\n\n2. The methodology is decomposed into steps, for which detailed explanations are provided across the subsections of Section 4. The reviewer appreciates the idea behind this presentation approach, which aims to split the overall cumbersome idea into smaller digestible pieces. \n\n3. The empirical performance is overall compelling, with CFASL presenting a clear improvement over competing approaches in many of the tested scenarios.\n\n4. Although not viewed as a major contribution, the extended evaluation metric to assess multi-factor changes proposed in this work is noted. This seems like a useful idea that may be employed for empirical evaluation in follow-up works. \n\n**Weaknesses:** While, as mentioned above, the considered task is interesting and important, I would like to start from a general comment on the state of the factor aligned VAE literature. The current approaches, including the one proposed in this work, are quite reductionist. They stitch up a model and loss from various components that are separately taking care of different aspects of the problem---in this case factor alignment for symmetries. A holistic and more interpretable approach that automatically adapts to those symmetries will be the ultimate solution, but I understand that this literature is not there yet. This is not held as criticism of the current submission. Rather, an observation of what I believe is an unsatisfactory aspect of current methodologies. \n\nBeyond that, given that this is a heuristic methodology paper, clarity in presentation and a valid justification of the proposed steps are crucial. However, as outlined below, I find that the current submission falls short in these areas: \n\n1.  The preliminaries section is massively lacking in background. Very basic ideas from group theory are presented, but central ideas for the proposed approach are not covered, making subsequent explanations hard to understand. This includes, for instance, background on the $\\beta$-VAE framework, specification of input/output/latent spaces, structural assumptions on the data (e.g., the symmetries it possesses), Lie groups/algebras, factor-VAE metric, etc. A notation section would also be good to add. In its current state, the paper is not stand-alone and will therefore be difficult to read for any researcher not directly in the field of study. \n\n2. The paper significantly lacks in the level of presentation. This is expressed not only in multiple typos, capitalization, and grammatical errors (e.g., \"onn\" above Fig.1, \"[...]. we\" and \"lie group\" in the paragraph below Fig. 2, \"not worth\" before Eq. 3, $||$ symbol before Eq. (1) in not defined, the first sentence in the last paragraph on page 7 start from half a sentence, broken indicator symbol in Eq. (4), and many others), but also in the explanations of the various steps in Section 4 being hard to follow. These explanations are dry and do not provide intuition for what is being done. They are also full of jargon, undefined notation, and typos, which makes then even harder to follow. This hampers the reproducibility and the rationale behind the proposed approach. \nFirst of all, the text requires significant polishing and a few rounds of English editing. Beyond that, I believe that incorporating a simple running example (e.g., a 2D image with rotations) to support these explanations would be helpful. I encourage the authors to consider doing so, and refer back to this example in each subsection, crystalizing each step and the relevant mathematical expressions in its context. \n\n3. Figure 2 is supposed to summarize the method and provide a comprehensive view. However, I think that it fails to do so because it is hard to parse, with neither the caption nor the text providing sufficient information to make sense of it. I suggest breaking it up into labeled subfigures and expanding the caption (or the relevant parts of the text) to provide a clearer description that addresses both the intuition and the mathematical expressions/formulae. For instance, in Section 4.3 it says: \"In the first step, the model generates the factor aligned symmetries of each section through the attention score as shown in Fig. 2\", but it is not explained which part of Fig. 2 the reader should refer to or what they should look for. This shortcoming repeats itself throughout the text and applies to other figures as well (often referred to by merely saying `as shown in Fig. x', without further context).\n\n4. It is unclear why there is a hyperparameter $\\epsilon$ weighting $L_{ee}$ and $L_{de}$ in $L_{equiv}$, but no such parameters are introduced within $L_{codebook}$. The authors should clarify this point and justify this design choice. \n\n5. The contribution in defining m-FVM$_k$ is hard to appreciate without proper background being provided for the vanilla FVM. \n\n6. Section 5 on related work seems misplaced and should appear as part of the introduction. If the goal is to review the competing methods before presenting the experiments, then the authors should include a dedicated paragraph summarizing the specific approach that CFASL will be compared to.\n\nOverall, I think that the paper presents a promising method with evident practical advantages. However, clarity of presentation and reproducibility are key aspects in which I think that the current submission significantly lacks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper titled proposes a new approach, dubbed CFASL, for unsupervised symmetry-based disentanglement in VAEs without requiring prior knowledge of dataset factors. The method integrates three key components:\n\n(i) Inductive bias for latent alignment: CFASL aligns latent vector dimensions with factor-aligned symmetries using a learnable symmetry codebook, which facilitates better disentanglement.\n(ii) Composite symmetry learning: Expresses changes between random samples by learning factor-aligned symmetries that capture complex transformations within the data.\n(iii) Group equivariant encoder and decoder: The model employs a group equivariant encoder and decoder to reinforce these symmetries during VAE training.\n\nThe paper then provides a rather broad empirical study that demonstrates the performance of CFASL on several benchmark datasets, comparing it with existing state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request that the authors address the weaknesses described above according to the provided suggestions.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes Composite Factor-Aligned Symmetry Learning, which introduces symmetry-based disentanglement into VAE. Although the method is slightly adhoc without theoretical justification, the experiments are broad and clear. 

All the reviewers believe this is an interesting piece of work and suggest for acceptance.
Decision: Accept (poster)