Disentangled Representation Learning via Modular Compositional Bias
Authors: Whie Jung
Dong Hoon Lee
Seunghoon Hong
Venue: NeurIPS 2025 poster ()
Keywords: Unsupervised Representation Learning
Disentangled Representation Learning
Compositionality Maximization
Primary area: general_machine_learning
TL;DR: We propose a modular inductive bias for disentangled representation learning, which we term a compositional bias, decoupled from both learning objectives and model architectures.

Abstract: Recent disentangled representation learning (DRL) methods heavily rely on factor-specific strategies—either learning objectives for attributes or model architectures for objects—to embed inductive biases. 
Such divergent approaches result in significant overhead when novel factors of variation do not align with prior assumptions, such as statistical independence or spatial exclusivity, or when multiple factors coexist, as practitioners must redesign architectures or objectives. 
To address this, we propose a compositional bias, a modular inductive bias decoupled from both objectives and architectures. 
Our key insight is that different factors obey distinct "recombination rules" in the data distribution: 
global attributes are mutually exclusive, *e.g.,* a face has one nose, while objects share a common support (any subset of objects can co-exist). We therefore randomly remix latents according to factor-specific rules, *i.e.,* a mixing strategy, and force the encoder to discover whichever factor structure the mixing strategy reflects through two complementary objectives: 
(i) a prior loss that ensures every remix decodes into a realistic image, and (ii) the compositional consistency loss introduced by Wiedemer et al., which aligns each composite image with its corresponding composite latent. 
Under this general framework, simply adjusting the mixing strategy enables disentanglement of attributes, objects, and even both, without modifying the objectives or architectures. Extensive experiments demonstrate that our method shows competitive performance in both attribute and object disentanglement, and uniquely achieves joint disentanglement of global style and objects.
Code is available at https://github.com/whieya/Compositional-DRL.

Disentangled Representation Learning via Modular Compositional Bias
Whie Jung
Dong Hoon Lee KAIST
Seunghoon Hong
(whieya, donghoonlee, seunghoon. hong) 0kaist. ac.kr
Abstract
Recent disentangled representation learning (DRL) methods heavily rely on factor- specific straegies-either learning objectives for attributes model architectures for objects-to to embed inductive biases. Such divergent approaches result in signif- icant overhead when novel factors of variation do not align with prior assumptions, such as statistical independence or spatial exclusivity, or when multiple factors coexist, practitioners must redesign architectures this, we propose a compositional bias, a modular inductive bias decoupled from both objectives and architectures. Our key insight is that different factors obey distinct "recombination rules" the data distribution: global attributes are mutually exclusive, e,g., a face has one nose, while objects share common support (any subset of objects can co-exist). We therefore randomly remix latents according factor-specific rules, i.e., mixing strategy, and force the encoder discover whichever factor structure the mixing strategy reflects through two complementary objectives: (i) prior loss that ensures every remix decodes into a realistic image, and (ii) the compositional consistency loss introduced by Wiedemer et al. [50], which aligns each composite image with its corresponding composite latent. Under this general framework, simply adjusting the mixing strategy enables disentan- glement of attributes, objects, and even both, without modifying the objectives or architectures. Extensive experiments demonstrate that our method shows com- petitive performance in both attribute and object disentanglement, and uniquely achieves joint disentanglement of global style and objects. Code is available at https::/githhii com/ http://heeaa/oooiiitieaaaaaa DRL.
objectives. To address
Introduction
Understanding the underlying structure of data has become increasingly important for building robust and interpretable machine learning models. key approach to addressing this challenge is unsupervised disentangled representation learning (DRL. 16). which aims to factorize data into its fundamental compositional concept representations. By interpreting the world through compositional concepts, it becomes feasible decompose unseen data into simpler, more interpretable components. Moreover, this approach dramatically improves the data efficiency of learning [24] (26). as unseen data can be explained combinations of already learned concepts.
As a key theoretical insight in DRL, Locatello et al. [30] show that disentangled representations aligned with underlying ground-trutt factors of variation cannot be reliably achieved without ap- propriate inductive biases or explicit supervision. Consequently, designing a method to embed suitable inductive biases, which impose constraints the model assumptions about the data, has become a central challenge in DRL. Specifically, attribute disentanglement methods often enforce cntrmattionthhoreett objectives 29 (22), while object disentanglement methods leverage spatial exclusivity through architectural components, such as slot attention encoders [19 or additive
decoders 2 25
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Attribute
Ippu Imago
Equiralaniy
Ammioro Micing
3 800n8 88003
Reproommttion
Compostinnal Strreer
distacttie suppoes
Image
Image
Corppoee Trage
Miding Stratigs
Object
Equiralanily
Misteg
Chicci
Xinued Represeration
...
Cenernted izage
Comprsiioonll Strattart wilk Common support
Trage
Iraage
Corsosoo Irrng
<Myjcc repersetatioon traringgttinngg
Figure 1: Overview of our method. derive a compositional bias, we analyze the compositional structure of attriibuee and object, and implement it as mixing strategy. Given a mixing strategy, we decode composite representations into an image and minimize our prior loss and consistency loss to ensure is both realistic and aligned with the latents. Note that the figure illustrates specific example for object mixing strategy.
Despite impressive progress in each domain, current approaches are constrained by divergent factor- specific strategies for embedding inductive biases, either through learning objectives or architectural components. These vastly different, factor-specific designs embedded within architectures or objective functions require substantial engineering when encountering a novel factor of variation that may violate prior assumptions, such as statistical independence or spatial exclusivity, or when different factors co-occur, as practitioners must redesign architectures or loss functions. Identifying suitable objective functions or architectural components for new factors is challenging, given the vast design space. This motivates the need for a general framework that simplifies inductive biases into modular component, applicable to various disentanglement factors.
As first step toward this goal, we propose compositional bias, a modular inductive bias decoupled from both learning objectives and architectures, enabling disentanglement of different factors within single framework. particular, we formulate DRL as process of maximizing compositionality, where the compositional bias characterizes factor-specific biases by defining valid ways to compose disentangled representations. Given two sets of latent representations from different images, we construct composite representation by exchanging random subsets of latents and maximizing the validity of the resulting composite image, measured by data likelihood and compositional consistency [50]. Analyzing the compositional structures of attributes and objects, derive specific compositional biases, or mixing strategies, that determine valid compositions for attribute and object disentanglemennt Incorporating factor-specific biases into this modular mixing strategy, rather than into the architecture or objective function, allows our method to disentangle objects and attributes under single framework by simply switching the mixing strategy. Our contributions are follows:
We propose a disentanglement framework that decouples factor-specific inductive biases from learning objectives and architectures, enabling both attribute and object disentangle- ment under a single set of objectives and architectures.
We derive mixing strategies as a compositional bias, embedding factor-specific biases in modular way and propose specific strategies for attribute, object, and joint disentanglement.
We compare our methods against baselines specifically designed for attribute or object disentanglement. For attribute disentanglement, our method achieves the best DCI (10), while on object-level tasks, it performs comparably to state-of-the-ar methods. Notably, our method demonstrates joint disentanglement of both factors within single framework.
2 Background
In this section, we review the two main streams of disentangled representation learning: attribute and object diieettanglement. In particular, we explain how current approaches incorporate factor- specific inductive biases into either learning objectives or model architectures, and why this tight coupling limits the ability to generalize these biases across different factors of variation. For more comprehensive review of the literature, we refer the reader to Appendix A.3
We refer to attribute factors as properties shared globally across an entire scene, e.g. color, style, while object factors distinct spatial entities within scene, such as individual objects.

---

Attribute disentanglement In attribute disentanglement 3 6 22 37 scenes are assumed to consist of fixed set of random variables [2), with methods typically enforcing statistical indepen- dence among latent dimensions through learning objectives to achieve attribute representations. For example, R 5 22 incorporate Total Correlation [48] into the VAE framework, while [29]3]]
37 use contrastive regularization to ensure that variations in each latent lead to distinct changes in the output space, More recently, [52] proposed minimizing an upper bound on mutual information among latent variables. These information-theoreti objectives work well when the data consist of a fixed set of fac- tors, with each latent variable corresponding to a specific factor. However, incorporating such biases directly into the learning objective is not straightforward in general. For instance, in object-centric scenes with varying numbers of objects and permutation invariance, defining objective becomes non-trivial. As result, practitioners must devise entirely new objectives models for novel scenarios, such as object-centric scenes. This underscores how embedding the bias directly into the objective function limits generalization new factors of variation.
coammatioo-thhoretti
31 typically models a scene as an unordered set of object representations sampled from shared generative process, such that permuting these representations does not affect the rendered image. Since measuring independence among object representations is challenging, existing methods often rely on architectural biases that enforce spatial exclusivity. Early approaches implement this by rendering each latent as pair of an image and
Object disentanglement Object-centric learning 4
11 14
mask, then blending these pairs to form the final output [4] 1 [4 28], with each mask corresponding to distinct region. Slot attention based methods [19 31 41 similarly impose spatial exclusivity in the encoder, assigning each latent to specific spatial locations in the input. Although these biases effectively disentangle objects, they fail to capture global attributes, such as global style, which are defined for the entire image, as their underlying assumptions no longer hold. Consequently, whenever these assumptions are violated, the architecture must be redesigned, which is non-trivial and costly.
3 Method
Our goal is to design a modular inductive bias for DRL, decoupled from learning objectives and architectures. In this section, we present overall formulation to maintain identical learning objectives and architectures across different factors (Sec. (.1..3.1). 3.11. how we derive modular inductive bias for different types of factors (Sec. 3.2) and detail specific learning objectives of our framework (See.)
3.1 DRL via Maximizing Compositionality
To modularize inductive bias for disentanglement, we construct a framework where learning objective and model architecture not factor-speciffe We formulate the learning objective of DRL maxi- mizing the validity of composite images generated by random recombination of latent representations, where validity is measured by data likelihood and consistency with given composite representations. This objective enforces recombination of disentangled factors to yield realistic images, without introducing any factor-specific biases into the model architecture or objective function..
Let x E I H XC be an input image, and z = {2,] be a corresponding set of K latent representations, where each z; E IR. capturing independent factors of variation. Building on an autoencoding framework, the encoder Eg maps into z and the decoder D, reconstructs x from z. To further regularize the compositionality of representations, we first generate composite images x by decoding concatenated random subsets of latents from two images x x² Formally, we define x = (T(T((EEETT) (Es (x'), Eg (x²)). where π(..-) : K x R K
+ E is a stochastic mixing operator between two sets of latents. We refer to specific designs of (.,-) -) for disentangling different factor types as the mixing strategy, which will be detailed in Sec. 3.2] Given x5, the compositionality of the representations is maximized through the prior loss and compositional consistency loss (eec. 3.3 Note that factor-specific biases will be embedded in mixing strategy, while the overall architectures and learning objectives remain unchanged regardless of the factors.
While Jung et al. [1 explored similar compositional objective, their work focused primarily on object disentanglement and relied on object-specific architectures, e.g., slot-attention encoder, thereby limiting its applicability to other factors such as attributes. In contrast, our approach imposes no factor-specific bias on either the architectures or the objectives. Instead, we focus on demonstrating
3

---

how factor-specine biases can be injected via mixing strategy, allowing disentanglement of different types of factors within a single framework.
3.2 Mixing Strategy
In this section, we illustrate how factor- sepcor-ppeccfic inductive biases can be implemented with mixing strategies. We first note that not all compositions of individual factors from different images produce valid images. For instance, when recombining the ground-truth (GT) factors of two face images, compositions containing two noses are invalid. This is because GT factors follow a particular compositional structure to form a complete image. Guided by this compositional structure, we formalize valid recombinations of factors using the factorized support assumption [39] It assumes that the support of disentangled factors' distribution factorizes over individual factors. Formally, let us denote the support of p(z) as (z)(() > 0}. Then z has a factorized support if S(p(z)) = S(p(z1)) x S(p(zz)) x .... S(((zx)), where x denotes the Cartesian product. implies that any random combination of z, extracted from distinct images corresponds to some valid image x. Based on this assumption, we analyze the different properties of each factor's 's support and derive mixing strategies between two images that define valid compositions.
Mixing Strategy for Attribute Disentanglement In attribute disentanglement, it is typically assumed that each scene consists of K unique factors [22 52]. For example, a ball consists of a fixed set of features such as color, texture, material, etc, with each factor being distinct and appearing only once. This implies that each latent z has a distinct support Spp(z.) * S(p(z,)) for all i + j. Therefore, when combining each latent z; from different images into composite representation z", each latent z; should be uniquely sampled from its own support. This suggests that the mixing strategy between z!_ x should guarantee mutual exclusiveness, i.e., each latent z, is drawn from one of the two images, but never from both, so that the resulting 2 always contains distinct factors. Formally, let I C (I,..., K} be randomly sampled subset of the index set. The mixing strategy Tattr for attribute disentanglement is defined as:
Tattr (2 (z',²²) x²) = [2) E E..
Mixing Strategy for Object Disentanglement Object-centrie learning often decomposes a scene into K interchangeable representations, each encoding single object instance. Since each object can exist independently of other objects, replacing an object in one image with any object from another image still yields a realistic composition. Formally, this means all z share the same support, i.e., S(p(z,) = S(p(zz (z))) for i,j E fl...., K.. Consequently, replacing z; with any z, from different images remains within the valid support of p(z). The mixing strategy for object disentanglement therefore involves randomly sampling K elements from the joint set of 2 and z² Unlike the mixing strategy for attributes, this approach permits unrestricted exchanges between x! and 2) at different
indices (see Figure 1. Specifically, let 1 12 C {I..... K be randomly sampled subsets of the index set. Then the corresponding mixing strategy Tobj defined as:
Tobj(z' ,²²) = {z]1 E r') (2) E [2), where 12| K
(2)
Mixing Strategy for Joint Disentanglement Unlike prior assumptions that consider either only attributes or objects, real-world scenes often contain both attribute factors and multiple objects, e.g., images with several objects rendered in varying artistic styles. While objective-leve! biases and model architectures in previous work require non-trivial modifications to capture both types of factors (Sec. 2). our framework naturally extends to this scenario. As attribute factors require mutual exclusiveness while object factors permit arbitrary exchange, we simply partition the K latents into the first M latents for attributes and the remaining K M latents for objects, and apply the corresponding mixing strategies each. Formally, let zi.M and z be the M attribute latents, M the K - . object latents. Then the mixing strategy Jooine is defined as: and z M+ and z
Tt Tjoint (z' x²) = Tattr (z).M- zi..)) U Taky (2)r 2 27141)
(3)
where Tattr and Tobj are defined in Eqs. I and ( respectively.
2 We demonstrate that mixing from two images is equivalent mixing from multiple images (Appendix A.4

---

3.3 Learning Objectives
Given a composite image x", we maximize its validity measured by likelihood p(x*) and consistency with z". Note that maximizing the likelihood p(x*) alone may lead to degenerate solutions that generate realistic images irrelevant to z². The compositional consistency objective prevents it by ensuring the alignment between x" and z². Below, we provide a detailed description of each objective.
Maximizing Likelihood of Composite Images To maximize the likelihood of x", we leverage pre-trained diffusion model Gy for its strong mode coverage [51] and compositional generalization capability 332 Since denoising loss in diffusion models serves an upper bound for the negative log-
likelihood 17 minimizing the denoising loss with respect to x is one way to increase the likelihood p(x"). However, due the expensive and noisy computation of gradients in propapanoggg
through a diffusion decoder, we upproximate the gradient to optimize p(x") as in [21 34]:
VoLrrior (0) = Er,, (G,, () - <) 0x
00
where ~ 24mii mmxx) is a timestep, wt is timestep -eneeeeppdeeeedennt function, NO. 1) is Gaussian noise. xi = V0,x" + FiE denotes a noised image of x and we is usually set to of [3].
We emphasize that we use a pre-trained, frozen, unconditional diffusion model for likelihood maximization. This is in contrast to L.2C [21] that employs a representation-ccndiiionn diffusion model G(x(z) for likelihood maximization. In L2C, the conditional score function is learned via a denoising loss conditioned on the encoded latent z of individual images. In this setup, the diffusion model encounters out- of-ooffriiibttiion (OOD) latents z from mixed representations during likelihood maximization, but there is no guarantee that the score estimation under these OOD conditions G( (x) will be valid. Instead, our method leverages pre-trained unconditional diffusion model to directly optimize likelihood and generate realistic images, while avoiding the OOD conditioning issue L2C. In Appendix A.9 we further compare our method to L2C's approach by additional experiments.
Compositional Consistency Loss Minimizing the prior loss alone could lead to degenerate solution, such as generating arbitrary realistic composite images x" regardless of the given z To prevent such degeneracy, we adopt compositional consistency loss from Wiedemer et al. [50] to ensure alignment between z" and the inverted latent 2 = Eg ( (z))) so that x" faithfully reflects the contents of z However, empirical observations reveal that minimizing the absolute distance directly is insufficient to prevent misalignment between x and x In practice, we find that the from all images tend to cluster closely together in the latent space, when directly minimizing the distance between zf and I This clustering keeps the distance between zf and 2' small even when the composite image x" does not faithfully reflect z². To address this, we instead minimize the relative distance between z and 2 i.e., their distance relative to negative samples, which are latents from other random images. This effectively strengthens the penalty on z' and z" even when their absolute distance is small, ensuring that z must not only match 2n but also remain distinguishable from negative samples. Formally, we define the compositional consistency loss, inspired by the InfoNCE loss [33]
exp(d(2",z*) //T)
Lcon (0) -log log EiE exp(d(ée, 2 )/7)
(5)
where 7 and d(.) denote temperature and cosine similarity, respectively, and B is a batch size. Note that we should consider the correspondence between z {zf,... 2xx and 2 . ,%. to compute the cosine distance. This can be problematic for object disentanglement, as object- disentangled representations can have permuted orders due to our mixing strategy. In this case, we first apply the Sinkhorn- Sinkhorn-Knopp algorithm [8] to compute a soft assignment between z" and 2", then use the briggmment-weighhee sum of the distances to compute the loss.
Overall Objectives Following the common practice in DRL 19
21
22
52 our framework is
built upon the auto-encoding framework. Specifically, instead of directly reconstructing the image, we minimize a denoising objective using diffusion decoder, following recent methods 21 52 as:
Reccom 6, +) = E., [w. [x 1, Ex()) - -c []'],
where x, = Vaix x + Vi o is noised image of x with timestep I,, ā, = I 3&) a schedule function, and w. is the weighting parameter. As use the diffusion decoder D&. we use iterative
5

---

Cars3D
3
MPP33
Method
FactorVAE
DC1
VFaciorVAE
DC1
FactorVAE
DCI
FactorVAE
22
0.906
0.161
0.840
0.611
0.152
0.240
8-TCVAE
0.873
0.179
InfoGAN-CR
29
0.411
0.020
0.587
0.478
0.439
0.241
LD
+7
0.852
0.216
0.805
0,380
0.391
0.196
GS
0.932
0.209
0.768
0.284
0.464
0.229
DisCo 37
0.855
0.271
0.877
0.708
0.371
0.292
Dis
0.976
0.232
0.902
0.723
0.617
Ours
0.877
0.365
0.975
0.837
0.708
0.458
Table I: Quantitative results on attribute disentanglement. Our method achieves state-of-the-an performance in almost all of the datasets, except FactorVAF score in Cars3D. We report mean of 10-repeated runs. Standard deviations are in Appendix A.14
Figure 2: Qualitative results on Shapes3D.
Our method identifies all GT factors.
decoding when generating the composite image x" from the diffusion decoder, but omit the expression for notational simplicity. The overall objective is given as:
Toaal +) CRconn (0, +) Xprior PPrior + AconCcon
where Aprior and ACon controls the relative importance of the objectives. Note that Lpior and LCon optimized only with respect to the encoder parameters (0), while keeping the decoder parameters (ø) fixed. This prevents unintended cooperation between the encoder and decoder that could generate realistic composite images from suboptimal latent representations.
Experiment
Implementation Details We implement the decoder D3 as latent diffusion model built on pre-trained VAE following 52 Since the diffusion decoder operates on VAE features, we design the image encoder to take these VAE features as input. For object disentanglement, same encoder architecture used in recent methods [9 but replace the slot attention module with a lightweight transformer (QFormer (27) to avoid the inherent inductive biases of slot attention. In attribute disentanglement, we follow the DifDiff 52 encoder design. When generating the composite image x" from z", cost of iterative decoding. Because backpropagating gradients through all decoding steps is often prohibitive, we follow recent work diffusion model fine-tuning 7 35 to truncate the gradient at the last decoding iteration. For the generative prior G.. we train an unconditional latent diffusion model each training dataset from scratch. See Appendix 6.6 for additional implementation details.
adopt the
few steps (1-4 steps) DDIM [44) sampling to reduce the computational
Datasets For attribute disentanglement, we evaluate our method on Shapes3 MPI3D 13 which are standard datasets in attribute DRL. Each data in these datasets is generated from a fixed number of GT factors. Following [37]] [2]. all experiments in attribute DRI. are conducted a 64x64 image resolution. For object disentanglement,
Cars3D 36
use three multi-object datasets, including CLEVR-Easy [42), CLEVR (20), and CLEVR- Tex [42). Each dataset consists of multiple instances of objects in different object properties, such colors and shapes. To evaluate joint disentanglement of attributes and objects, we introduce the CLEVR-Style dataset, a new variant of the CLEVR dataset augmented with four distinct artistic styles (see Appendix A.7p. A style is regarded attribute of the scene, as it is uniquely determined for each data. This construction yields a complex set of scenes in which both the style and the individual objects must be disentangled. For object- and joint disentanglement all experiments are conducted at a 128x 128 image resolution.
global
4.1 Attribute Disentanglement
We compare our method with: (1) VAE-based methods, including FactorVAE 22 and 3-TCVAE (
(2) GAN-based methods, including InfoGAN-CR (29), GANspace (GS) 15 LatentDiscovery
(LD) 47), and DisCo B77, and (3) the diffusion-barrd model DisDiff [2]. Note that ours and DisDiff use the same encoder and diffusion-decode disentanglement, we follow 99 (2 by applying PCA as post-processing step for evaluation. We use standard evaluation metrics for disentanglement: the FactorVAF. 22 score and the DCI (1) metric.
architecture. For methods with vistor-wise
We provide brief descriptions of these metries in Appendix A.5

---

Table 2: Comparisons on object property prediction. Ours achieves performance comparable to state-of-the methods, For the position" of CLEVREasY, the discrete labels in the dataset and reports the accuracy.
CLEvREasy
CLEVR
CLEVRTex
Method
Shape (1)
Color (t)
Position" (4) Shape(11 (1] Color(1) Material
Position| ()
Shape(1 (1)
Material( 1))
Position
72.25
72.33
79.4
0.064
30.44
7,890
0.482
SLASH
86.06
89.23
46.97
83.28
92.26
93.16
0.078
53.13
37.49
0.148
LSD
96.03
98.05
50.29
87.66
91.46
94.87
0.062
68.25
51.54
0.197
L2C
92.78
93.57
47.62
73.61
74.03
86.93
0.168
71.54
51.62
0.116
95.81
50.72
87.04
0.032
20.90
52.2
0.133
object
object
Remove
RATDOW
Figure 3: Qualitative results on object-wise manipulation. Objects marked by red arrows are replaced with those marked by green arrows. It demonstrates that our method effectively disentangles individual objects. We also find empty latent (depicted with Φ). which makes our approach capable of handling varying number of objects.
Main Results In Tab. 1 we compare our method to baselines for attribute disentanglement. Our method outperforms all baselines on the Shapes3D and MPI3D by clear margin, achieving 8% higher VAE score and 15.7-21.4% higher DCI metric than the second-best methods. On the Cars3D, our method achieves the highest DCI metric. Notably, our method outperforms the state- of of-the-art baseline DisDiff [52) by clear margin the Shapes3D and MPI3D. This demonstrates the effectiveness of our compositionality maximization approach, which directly enforces support factorization via mixing strategy. Meanwhile, method significantly outperforms Factor VAE (22), which similarly employs random mixing latents but is constrained by a VAF-based total correlation minimization. In contrast, our method restricted by specific model architecture such VAEs, enabling vector-wise disentanglement and an expressive diffusion decoder, which together lead to stronger performance.
Qualitative Results In Fig. 2 we analyze the quality of our disentangled representations by swapping latent representations between images. We encode one target image and six source images into K latent representations each. For each {1.... K), ) we construct swapped representations by replacing the k--th latent of the source images with the k-th latent of the target image, then decode these representations into the resulting images. The result shows our method's effectiveness attribute disentanglement and compositional image generation. On Shapes3I), our approach successfully identifies all six GT factors of variation. Additional results on Cars3l) (Appendix A0. show captures three independent factors, enabling controlled manipulation of each factor.
4.2 Object Disentanglement
We compare our method with state-of- taae--of-hhearrr object-centric approaches that use slot attention encoder: SA (3T], SLASH (23) LSD [9). and L2C [21] - LSD and L2C also employ diffusion decoders. We shared the same latent diffusion architecture for all diffusion-based approaches.
we evaluate the quality of object representations via object property prediction. this task, the correspondence between the object representation and its true label is determined through Hungarian matching using object masks. For the basclines,
Object property prediction Following 19 21
7

---

Image
GT
LSD
L2C
Ours
Figure 4: Object segmentation results on CLEVRTex. Despite lacking a built-in spatial clustering mechanism (e.g., slot-attention), our method combined with Spatial Broadcast Decoder consistently captures complete objects, whereas the baselines often split objects across multiple latents.
slot attention's attention weight produces these masks 19 Meanwhile, our framework lacks a slot attention module, we identify each object's region by averaging the difference in output images when composing each representation with others. More details can be found in the Appendix [A.8] Tab. demonstrates that method achieves competitive performance compared state-of-the of-tth-art art baselines, LSD, and L2C. Specifically, our method outperforms LSD on CLEVRTex and achieves comparable performance on CLEVR and CLEVR-Easy. It demonstrates the effectiveness of our mixing strategy as an inductive bias for object disentanglement, replacing slot-attention. In com- parison to L2C, which also maximizes compositionality, our method performs better on CLEVR and FLEVR-Easy while being competitive on CLEVRTex. We note that L2C degrades on CLEVR, likely due to undesirable positional biases. Overall, despite our method's broader generality both attribute and object disentanglement within a single framework, it achieves performance comparable to state-of ootte-of-the-rt methods tailored solely for object disentanglement.
Object-wise manipulation In Fig. we qualitatively explore the compositionality of our object representations. Given pairs of images, we encode each image into K latents, then create a mixed representation by swapping single latent between images and decode them into composite images. In the figure, we replace one object (red arrow) from the first column with the corresponding object from the first row (green arrow). The result demonstrates that our approach encodes individual objects in disentangled way, enabling object-wiss wise image manipulation. To be more specific, columns two through five show that each swapped object from the first row is successfully inserted into the first column's 's image, while the original object is removed from the scene. Moreover, in the fifth row and fifth column, we observe that our method allows the emergence of latent encoding of empty information.
Unsupervised object segmentation We evaluate object representations through unsupervised object segmentation. Unlike slot-attention based methods, which inherently cluster pixels spatially, our approach does not provide built-in mechanism to group pixels, so object segmentation masks cannot be extracted di- rectly. Nevertheless, evaluate the quality of object representations, we train a Spatial Broadcast Decoder (SBD) 49 on top of frozen object representations with a reconstruction loss in unsupervised manner. We followed [1] to extract explicit object masks for each representation using the SBD. We also apply this approach to the dott-attention-aaat based baselines, as it yields better results. We provide more details in the Appendix AI
Table 3: Comparisons object segmentation.
Method
CLEVR
CLEVRReX
FG-ARI
mloU
mBO
FO-ARI
mloU
mBO
LSD
91.74
25.59
25.84 84
71.64
56.26
56.75
L2C Ours
80.05
25.61
26.33
82.55
58.33
58.68
91.20
26.54
26.65
87.68
58.88
59.12
Tab. 3 shows our object segmentation results. On CLEVRTex, our method outperforms both LSD and I.2C in FG-ARI, mloU, and mBO (See Appendix A.5 for definitions of metries.) On CLEVR, ours achieves the best mloU and mBO scores, with FG-ARI comparable to LSD. We observe relatively

---

Table 4: Evaluation of joint disentanglement on CLEVR-Style dataset.
Style
Object
Method
Acctt)
GRAM(L)
Shapert)
Color(t)
Material(
Position( ()
DisDir
0,900
13.90
N/A
N/A
N/A
N/A
LSD
9.500
1,2C
12.20
12.86
85.02
95.35
95.35
0.046
Onrs
96.50
5.050
83.56
23.74
0.053
Source
Yarget
DisDiT
LSD
L.CC
Ours
GT
object
Remove
Figure 5: Qualitative comparisons on Style transfer in CLEVR-Style dataset.
Figure 6: Object manipulation of method in CLEVR-Style dataset.
lower mIoU and mBO on CLEVR, likely because including the constant background in each object's latent representation does not affect compositional generation, thus avoiding penalties from the compositional loss. Since these backgrounds carry no meaningful information, they do not impact the quality of object representations. Fig. showws qualitative segmentation results CLEVRTex. Our method consistently encodes complete objects into distinct latents, whereas LSD and L2C often split objects across multiple latents.
4.3 Joint Disentanglement of Attribute and Object
In this task, evaluate the disentanglement of both global attributes (style) and objects in the CLEVR-Style. We compare our method against DisDiff, LSD, and L2C which are state-of-the-ar methods in attribute and object disentanglement. For object disentanglement, object property prediction task. To assess style disentanglement, we sample 1K image pairs with identical content but different styles, swap the style latent of the first image with that of the second, and check whether the synthesized image matches the second. Quantitatively, we report GRAM loss [12] and style prediction accuracy synthesized images, where we trained ResNet classifier for style prediction in a supervised manner. While our method can specify the latent for encoding the attribute factor and applying the attribute mixing strategy, the baseline models do not have such mechanisms. To identify the style latent in baselines, we decode all K K possible latent exchanges between two images and choose the pair yielding the lowest style loss. To further assess the scalability and robustness of our method on more complex datasets, we also conduct experiments on the MSN-Style, an augmentation of the MultiShapeNet (MSN) (45 dataset, which includes over 10k unique, realistic furniture shapes. Further details and results on MSN-Style dataset can found in Appendix A.7 A.13
evaluate
style, i.e., allocating the first latent
Main Results Tab. our method significantly outperforms all baselines on style disentanglement. Our method achieves over 95% style prediction accuracy, confirming the disentanglemen of style information. As information-theorede varying number of permutable objects, DisDiff is trained suboptimally and fails to disentangle both style and objects. In object disentanglement, object-centric approaches (LSD, L2C) show reasonable performance thanks to the slot-attention module. However, they completely fail in disentangling style information (~10% accuracy.) This because slot-attention lacks motivation for the disentanglement of spatially non-exclusive factors. This clear contrast highlights the strength of our unified approach in jointly disentangling multiple factors simultaneously without altering objectives model architectures.
objectives of DisDiff cannot be naturally extended

---

Beeove
Shape30)
CLEVR
FactevAB
DCI Shapef Color Postion
Remove
Insert
Insert
abject
object
atribue
Losses
60.27
0.111
63.30
86.04
0.126
0.769
6421
80.23
0.116
1/000
87.04
93393
0.032
Mising straley
Inpest of Misine Strutayy
Attibuute
1,000
0.887
65.24
80.52
0.119
Ofject
0.634
0.127
87.04
43,93
0,833
Table 5: Ablation study confirms that ours works best with objectives and the proper mixing strategy.
method. It
OOD example
(h) OOD examlle
[c] Decnded images from diffemet mixing endlgy
Figure 7: Qualitative analysis on our method. verifics OOD
generalization (a).b) and importance of mixing-strategg (c).
Qualitative Results In Fig. 5 we examined style transfer by replacing the latent representation encoding the style of target images with that of source images. While our method successfully transfers the source images' style into the target images, none of the baselines correctly modifies the overall style of the images. Although DisDiff often alters the style (first and third rows), does not correctly reflect the original style and often changes the objects, indicating that the style is entangled with objects. We also present object-wise manipulation of our method Fig. onn the right. Successful insertion and removal of the objects verify that our approach encodes individual objects in disentangled way. is worth noting that ours is the only method that disentangles both factors at the same time, and it is done by simply adjusting mixing strategies without altering objectives or model architectures. Additional results and analysis are provided in Appendix 1. A.
4.4 Ablation Study
Impact of Losses In Tab. we conduct ablation study each term of our objectives. The result show that all three losses (fiift. Lpiorr- Lcon) are essential. In attribute disentanglement, adding each loss term sequentially improves performance, with the best results when all losses are combined. For object disentanglement, clear improvements occur only when all loss terms are used together.
Impact of Mixing Strategy In the bottom three rows of Tab. we investigate the importance of proper mixing strategy for attribute and object disentanglement. We apply object mixing to attribute disentanglement and attribute mixing to object disentanglement, respectively. The results show that the interchanged mixing strategy significantly degrades performance in both attribute and object disentanglement, highlighting the importance of using the correct mixing strategy our method.
More qualitative analysis In Fig (a, b), we observe that our method is capable of generating -it-----dittibutionn (OOD) examples that do not exist in the dataset, but can be created through composition. Notably, in the CLEVR-Easy dataset, which comprises images with 2-3 objects, our method can generate OOD images containing either a single object or objects through composition. by inserting or removing the representation that does not encode the object. In Fig. [f(c), we compare composite images from models trained with object mixing versus attribute mixing. Only the model trained with object mixing achieves object-wist wise manipulation, whereas the model trained with attribute mixing changes multiple objects simultaneously.
Conclusion
We introduced modular inductive bias for DRL that is decoupled from both learning objectives and model architectures. We formulated DRL as maximizing data likelihood together with the compositional consistency of composite images produced by latent mixing, and embed factor- specific biases via a modular mixing strategy. This design enables attribute, object, and joint disentanglement within a single framework by adjusting only mixing strategy, without requiring changes in architectures or objective functions. Extensive evaluations demonstrate that our method baselines in attribute disentanglement, while maintaining competitive
surpasses state-of-the-a performance in object disentanglement, More importantly, our method successfully disentangle both objects and global style at the same time in newly-introduced CLEVR-Style dataset, whereas current state-of-the-ar DRL approaches fails to disentangle both factors.
10

---

Acknowledgment This work was in part supported by the National Research Foundation of Korea RSS-2024-00351212 and RS-2024-004361655 and the Institute of Information & communica- tions Technology Planning & Evaluation (ITTP) IRS-2022-1122026, RS-2022-11220559, RS-2024- 00509279, and RSS-0019-11100755 funded by the Korea government (MSIT).
References
[1] Yoshua Bengio and Yann LeCun. Scaling learning algorithms toward ai. 2007.
[2] Jack Brady, Roland S Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius Von Kügelgen, and Wieland Brendel. Provably learning object-centric representations. In International Conference Machine Learning, pages 3038 3062. PMLR, 2023.
[3] Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in f-vae. In ICLR, 2017.
[4] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv: /900//91///// 1390, 2019.
[5] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[6] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[7] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models differentiable rewards. arXiv preprint arXiv:2309./7400 2023.
[8] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[9] Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34: 15620, 2021.
[10] Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International conference learning representations, 2018.
[11] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020.
[12] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pullern recognition, pages 2414-2423, 2016.
[13] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo. Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
[14] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In ICML, 2019.
(15) Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances newral information processing systems, 33:9841-9855 2020.
[16] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230. 2018.

---

[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[18] Youngjoon Jeong. Junha Chun, Soonwoo Cha, and Taesup Kim. Object-centric world model for language-guided manipulation. arXiv preprint urXiv:2503.06/77 2025.
[19] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In NeurIPS, 2023.
[20] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017.
[21] Whie Jung, Jachoon Yoo, Sungjin Ahn, and Seunghoon Hong. Learning to compose: Improving object centric learning by injecting compositionality. In ICLR, 2024.
[22] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conference on machine learning. pages 2649-2655 PMLR, 2018
[23] Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim. Shepherding slots to objects: Towards stable and robust object-centric learning. In CVPR, 2023.
[24] Yen- Yen-Ling Kuo, Boris Katz, and Andrei Barbu, Compositional networks enable systematic generalization for grounded language understanding. In EMNLP, 2021.
[25] Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julice Julien. Additive decoders for latent variables identification and partesian-produee extrapolation. Advances in Neural Information Processing Systems, 36, 2024.
[26] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 23-29 Jul 2023. URLhttps: httph::t///pppt http::/::////roeeennss....m mlr www.sees.w...pr....eppe.eeeeseee wwwpppssssssa.. wwwp/12221112 2tt.//////i3ggg/ html
[28] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriente, scene representation via spatial attention and decomposition. In ICLR, 2020.
(29) Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcen- trality: Self-supervised model training and selection for disentangling gans. In international conference on machine learning, pages 6127-6139. PMLR, 2020.
[30] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised lcarning of disentangled representations. In international conference machine learning, pages 41 14- 4124, PMLR, 2019.
[31] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
[32] Maya Okawa, Ekdeep S Lubana, Robert Dick, and Hidenori Tanaka. Compositional abilities emerge multiplicatively: Exploring diffusion models on synthetic task. Advances in Neural Information Processing Systems, 36, 2024.
[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03744 2018.
12

---

[34] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2022.
[35] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.037.99 2023.
[36] Scott E Reed, Yi Zhang. Yuting Zhang, and Honglak Lee. Deep visual analogy-making. Advances in neural information processing systems, 28, 2015.
[37] Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Learning disentangled representation by exploiting pretrained generative models: A contrastive learning view. In ICLR, 2022.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[39] Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentan- glement of correlated factors via hausdorff factorized support. In ICLR, 2023.
[40] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint wxXv:2202.00512 2022.
[41] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In ICLR, 2022.
[42] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh International Conference on Learning Representations, 2022.
[43] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. arXiv preprint arXiv:2211.01177, 2022.
[44] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.00500. 2020.
[45] Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint urXiv:2104.01145 2021.
[46] Yang Tao, Ren Xuanchi, Wang Yuwang, Zeng Wenjun, Zheng Nanning, and Ren Pengju. Towards building group-based unsupervised representation disentanglement framework. ICLR, 2022.
[47] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions the gan latent space. In International conference on machine learning, pages 9786-9796. PMLR, 2020.
[48] Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4()6668-2, 1960.
[49] Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv: 1901.00000000111 2019.
[50] Thaddaus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos, Matthias Bethge, and Wieland Brendel. Provable compositional generalization for object-centric learning. arXiv preprint arXiv:2310.05337, 2023.
[51] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint a1X00:211..07804, 2021.
[52] Tao Yang. Yuwang Wang, Yan Lu, and Nanning Zheng. Disdiff: Unsupervised disentanglement of diffusion probabilistic models. NewrIPS, 2023.
[53] Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group vae for disentanglement learning. In International Conference on Machine Learning, pages 12924-12934 PMLR, 2021.
13

---

A Appendix
A.1 Limitations and Future Work
In our work, as in most previous approaches, we assume the target factors of variation are known in advance, and our goal to learn how disentangle them. However, in real-world scenarios, the factors of variation within a dataset may be unknown or considerably more complex than the idealized cases of attribute or object disentanglement. Consequently, an important future direction is to automatically identify underlying factors of variation and determine the appropriate mixing strategy (potentially through learning) without relying on prior knowledge beyond the dataset itself. Mean- while, although our method shows strong performance in both attribute and object disentanglement, it does not provide theoretically grounded guarantees. Bridging the gap between methods that provide theoretical guarantees but only work on simple datasets, and methods like ours that demonstrate strong performance but lack such guarantees, is another important direction for future research. Moreover, although focus on learning disentangled representation in this work, exploring our framework on downstream tasks such as controllable image manipulation, e.g., attribute- or object-level edits, and object-centric world-model training that leverages our object-disentanleed representations, as exemplified by Jeong et al. [8].
A.2 Broader Impact
Our method can extract attribute or object components from existing images and use this extracted information to generate new images. This capability may raise privacy issues if applied to deepfake generation or unauthorized copying of digital content.
A.3 More Related Work
In this section, we discuss additional related work relevant to our method.
Identifiable DRL In object-level disentangled representation learning (or object-centric learning), a line of work [2] [50 leverages identifiability theory to derive conditions that provide theoretical guarantees for disentangled representations aligned with underlying factors of variation. Specifically, Brady et al. [2) shows that, under certain assumptions in object-centric scenes, an invertible composi- tional decoder can recover the ground-truth object latents (up to permutation). Lachapelle et al. (25) provides theoretical conditions for identifiability (up to permutation and blockwise invertible transfor- mations) when ground-truth latent variables are organized into specific blocks and an additive decoder is used. While these work can provide identifiability guarantees for object representations, it may not generally apply factors of variation such as attributes, which globally affect the image and do not necessarily satisfy the assumptions of additive decoders and compositionality definitions. Moreover, these approaches often impose strong restrictions on model design and expressiveness [4], making them applicable only to relatively simple datasets. Our method aims in a different direction from these works, seeking a modular inductive bias that is decoupled from both learning objectives and architectural constraints, such as an additive decoder, thereby enabling disentangled representation learning for various factors under a single objective and architecture.
Wiedemer et al. 50 Wiedemer et al. [50] demonstrates that autoencoders satisfying encoder- decoder consistency, in combination with an additive decoder, can yield object-centric representations that provably generalize compositionally. While this approach shares the similar conceptual goal of enabling generalization to novel compositions of factors disentangled representation learning as ours, there are fundamental differences in how we achieve valid generalization. As the common part, both methods recognize that valid generalization requires two key components: first, compositions of disentangled representations must yield valid data, e.g., realistic images or representation, and second, the composite representation z" must properly encode the information from its corresponding composite image x to satisfy the representation learning objective. To address the second require- ment, both our method and Wiedemer et al. (50) employ a compositional consistency loss with minor differences. However, the approaches diverge significantly for the first component-hon how to ensure the validity of composite representations.
Wiedemer et al. (50 employs an additive decoder to ensure valid compositions of disentangled representations (Sec. 3.1 in [50)). However, additive decoders are known to be unscalable for complex scenes, as their local decoding mechanism cannot capture complex interactions between
14

---

Table 6: Comparison to Wiedemer al. [SO] The additive decoder alone fails to achieve attribute, object, and joint disentanglement. While employing a slot attention module the encoder leads to reasonable performance on object disentanglement, it still fails in joint disentanglement.
Shapes31
CLEVE
CLyye-Style
FictorVAF
DC1
|
Shape
Color Miterial
Pesition [1)
GRAM (U)
Slupe
Color
Miterial
Position (3)
Churs Additive Decoder wio SA
0.975
87.04
94.11
0.032
96.50
5.05
83.56
90.48
0.000
0.031
33.87
15:24
$1.87
0.765
23.30
18.59
34,20
13.54
30.90
0.517
Additiv Decolerr SA
82,91
93.14
91.78
0.110
23.30
15.84
35.97
20.99
54.10
0,520
objects due to limited expressive power, More critically, additive decoders are designed based on the spatial exclusiveness bias (Def. 4 in (50). which assumes that each pixel should be affected by only a single latent variable. This limits their applicability to object-centric learning scenarios, preventing generalization to other disentanglement tasks.
In contrast, our method ensures validity through a prior loss (Eq. 4 that leverages SDS loss 34 to encourage composite image x to be realistic. As we do not require an additive decoder anymore, our approach allows us to utilize an expressive diffusion decoder for modeling complex scenes. Crucially, since our method does not embed factor-specific biases like spatial exclusiveness into the architecture, can be applied beyond object-centric learning to attribute disentanglement and joint disentanglement scenarios. More importantly, to the best of our knowledge, we are the first to demonstrate that different underlying factors of variation can be disentangled simply by adjusting the factor-specific mixing strategy, distinguishing our approach from previous methods that introduce factor-specific biases through architectures or objective functions.
Finally, we provide a quantitative comparison of Wiedemer et al. (50) with our method in attribute- object-, and joint disentanglement in Tab. 6 For fair comparison, we used the same encoder for [50] as ours and only changed the decoder to an additive decoder. As expected, [50] cannot disen- tangle attribute factors at all (Shapes3D, CLEVR-Style), since they violate the ppatial-xxclusiveess assumptions. Moreover, in object disentanglement, we found that an additive decoder alone cannot disentangle objects (in fact, Wiedemer al. [50] validated their method only on very simple 2D synthetic datasets). When we additionally use the slot- attentienno module in [50], it reasonably disen- tangles objects in the CLEVR dataset but still significantly inferior to our method in CLEVR-Style, possibly due to the limited expressive power of the additive decoder. Additionally, we observed that object- wise manipulation with [50) always leads to unrealistic images with transparently overlapping objects due to the lack of interactions between latents inside the additive decoder.
Group theory-based DRL Disentangled representation learning using Group theory is an actively researched area and is related to our work. Group theory-based DRL typically define disentangled
representation Z as follows: Given ground truth factors of variation W and decomposable group G X G,), the representation Z is disentangled w.r.t. G if (1) there exists a mapping f from W ZZ such that f(g W) = g- f(W) for all g E G and w E W, and (2) there is a decomposition Z = Z1 X x Zn such that each Z, is affected only by G. Despite this convincing principled definition, since the GT factors of variation W are infeasible to obtaín in unsupervised DRL, existing unsupervised methods often utilize necessary conditions for group actions of G applied to Z and disentangled representation Z [46] 53). For instance, Tao et al. [46 defines permutation group actions of element-wise addition on Z and introduces losses to enforce commutativity and cyclicity of group actions.
(i.c.. G= G1 x G2 X
Our method takes a similar approach, but with important distinctions. From a Group theory perspec- tive, unlike existing work, the group action in our method is defined on disentangled representation pairs (z x²) rather than a single latent 24. By defining the group action on pair (z x²), we can impose additional necessary conditions for how each underlying factor combines to generate observations, which cannot be induced by the group action defined on single latent 2j. For example, commutativity and cyclicity of group action are necessary conditions for both attributes and objects, but do not impose attribute or object-specific properties. Our main contribution here is that we define group action as factor-specific mixing, i.e., permutations, between two latents and demonstrate that this additional necessary condition imposes effective factor-specific inductive bias for attribute and object disentanglement without changing the overall learning objectives or model architectures. To the best of our knowledge, are the first to study differently learned disentangled representations of
15

---

different factors of variation through a mixing strategy (or a form of group action) without employing factor-specific architectures or learning objectives.
DRL with Factorized support Roth et al. [39] leverages a similar idea of combining latents to promote factorized support. While ours and prior work both leverage factorized support assumption. our main contributions are fundamentally different. First of all, we demonstrate that additional assumptions about the factorization structure of support, e.g., product of K distinct supports in attributes or repetition of a shared support in objects, lead to disentanglement of different factors of variation. By combining these two assumptions, we can even achieve joint disentanglement of attributes and objects, which the factorized support assumption alone cannot handle. Secondly, we show that these factorization structures can be encoded via simple mixing strategies (Fqs. I ا 3 replacing existing factor-specific biases. Those mixing strategies are decoupled from the architecture and objectives, so simply switching the mixing strategy enables us to disentangle attributes and objects within the single framework. Note that the prior method simply mixes the latent dimension-wist Our ablations (Tab. 5) empirically show that such a strategy is insufficient for disentangling objects.
Joint disentanglement of attribute and object Recently, SysBinder 43 introduced the Block- Slot representation, which models each object as slot formed by concatenating multiple multi- dimensional attribute (factor) representations called blocks, thereby enabling disentangled represen- tations of both objects and their attributes. This approach differs from ours by fully redesigning the model architecture to handle object-centric scenes, making it inapplicable to broader factors. In contrast, our method aims to support disentangled representation learning for various factors of variation within single framework with modular inductive bias. Nonetheless, extending our approach to disentangle multiple factors of variation using a modular inductive bias remains an important direction for future work.
A.4 Equivalence between mixing two and multiple images
Proof of equivalence this section, we explain why the random mixing between two images (i.e., z" = T(z!, 2²)) can replace the random composition of z, from K images. Formally, we will show that:
S(p(z)) = S(p(z)) then S(p(z)) = S* (p(z)),
(7)
where the factorized support Sx (p(()) = S(((z)) S(p(zz) -.. ... s(p(zk)) represents the random composition of each latent variable z, from K images.
Proof. Given S(p(z) S(p(z")), we can prove the followings:
1. If p(z, )2(2) > then p(z1, z2) >
Note that p(z1) 0 and p(z2) > 0 (++ p(z))((Z2) > 0) indicates the existence of z z² with z = 21. 2 = 22. By mixing z and z², we can compose z" where z = 21, z = Z2. Then, by the definition of the support that S(p(z)) {z|p(z) 0} and the given condition
z" E S((((z)) = (p(z)), p(2 22) 2 p(z") > O.
2. Assume that for some k 2 2. if IT &
- p(z))
- p(z1, Z2 zR)
then
[ p(z,) > - p(zz)(z11 Z2.... Zk. 2x+1) > 0. k+1 p(z,) > 0 implies p(zu+1) > 0 and IT p(z.) > 0. By the given assump- tion, p(z), 22..... Zx. > 0 and there exists z z² where z = for i E {l...., ,k] and 2 +1 = Zs+1- By mixing zt and x², we can compose zt where z, z, for E [1..... k + 1). As result, by the given condition z" E S(p(z")) S(p(z)).
Note that |
p(z1, 72, Ak, 2k+1) 2 p(z")= > O.
3. By mathematical induction, we conclude that if IT p(z,) > then p(z) > 0.
Note that (3) implies S(p(z)) Sx (p(z)), since Sx (p(z)) can be expressed as {z|p(z,) 0}. By using mathematical induction, we have proved that random mixing between two images can replace the random composition of multiple images to achieve disentanglement.
Empirical results In addition to proving equivalence, we compare our two image mixing strategies against mixing across multiple images (we use 64 here). We evaluate attribute disentanglement using
16

---

three random seeds and report the FactorVAE and DCI scores in Tab, 7 We observe no meaningful difference between mixing two images 64 images, supporting our theoretical result.
Table 7: Effects of the number of samples used in mixing strategy.
of samples for mixing FactorVAE
DCI
2
0.975=0.040 0.966#0.032
0.837=0. 105 0.802+0.0088
64
A.5
Evaluation metrics
In this section, provide brief descriptions and definitions of the metrics used in our experiments.
FG-ARI Foreground Adjusted Rand Index measures the agreement between predicted and ground- truth instance partitions, restricted to foreground pixels. Let I be the set of foreground pixels in ground-trui labels, and let y fl..... K] and ÿ E {1.... K} { denote ground-truth and predicted instance labels on I, respectively. Then, we define the contingency table n,j = Kip E I
Up i, Úp j} row sums a; E, nij. and column sums by Σn Rij. with Eij ml [Z]
The FG-ARI is defined as
E: (0) Σ, (7)
Eis
FG-ARI-
Σ. (2)
E,
Σ
which lies in [-1,1] and is higher when partitions agree better.
mloU Mean Intersection-overs Union averages class-wise IoU. For class let Po and Ge be
predicted and ground-truth masks. Then mloU defined as
Pe Gel
LoU,-
mloU
LoU
P..
CC|
CEC
where C is the set of evaluated classes.
mBO Mean Best Overlap measures, for each ground-truth object, how well it is covered by its single most overlapping prediction, and then averages these values. Let O the set of ground-truth instances with masks {Go}oEo. and let {M& }K be the set of predicted instance masks. For each
BO(o) = max loU Md), ISKCK
BO(O). mBO = O OEO
FactorVAE Score (2
The FactorVAE Score quantifies disentanglement of a representation with in which exactly one factor is fixed
respect generative factors. We repeatedly sample and all others vary. For each batch, we compute the empirical variance of each latent coordinate, normalize coordinate-wise, e.g., by the mean variance across coordinates, and select the index of the lowest-variance coordinate the fixed factor index from that feature. The classification accuracy on held-out batches reported as the FactorVAB Score (higher is better).
feature. A simple classifier, such as a majority vote, is used to predict
Disentanglement Score in DCI 10 Let R e Rd be an importance matrix obtained by training predictive model from latent coordinates z E Rd to factors {um}, where Rşm 2 0 measures the contribution of latent dimension j predicting factor m. Define normalized importances
Pjm = Rim/ Em Rim'. The per-dimension disentanglement is
Dj = (p,))/ log M,
where is the entropy. Weighting by total importance wy = Em Rym and normalizing, the overall DCI Disentanglemennts
DCI-D Σ DyD ij Σg wy
W
17

---

GRAM Loss A Gram loss measures feature correlations of neural network & at selected layers L. measures second-order feature statistics (style) between two images and y. For two images æ and let Fx(x) E C XH, be the feature at layer f, obtained by reshaping (b))). The Gram matrix is Ge(z) J.H.W. (())) The GRAM loss is measured as
LGRAM(E,Y)
Σ REC
||Gr(z) Gely)ll
A.6 Additional Implementation Details
When training our model, we use a fixed batch size of 64 and a learning rate of 0.0001 across all of the experiments. We use Aprior = 1 and Acoa 0.01 for all experiments. We set the number of latents to = 10 in attribute disentanglement and use K = 4, 11, 11, 12, 6 for CLEVREasy, CLEVR, CLEVRTex, CleyyTex-Style, MSN-Style datasets in object disentanglement, respectively. When training the diffusion model, we use v-prediction
40
loss to ensure reliable few-step generation.
Tab. summarizes the hyper-parameters of our encoder and decoder architectures used in the experiments. Following the diffusion-based baselines: DisDiff (52) and LSD [19], we employ pre- trained VQ-VAE and KL.rrgullarized auto-encoder model in attribute and object disentanglement, respectively. attribute disentanglement, the encoder first maps the input x into -d-mmeensional vector z E RKD and then uniformly divides it into K latents. In object disentanglement, the Q-former [7 of transformer blocks with attention heads and hidden dimension of 256. Specifically, have K learnable queries {q}
adopt
K RKxD D and those queries are updated via multiple self attention layers and cross attention layers, where the keys and values are linearly projected from the U-net encoder feature of x.
3
18

---

Conv 3 x x 128, strde=1 BatchNommdd
snide=1
ButchNomm2d
Inpui Resolution
Inpui Channels
3
Conv BanchNooodd
Rel.U
stide=]
Input Channels
4
ReLU
RutchNomm2d
Conv 3 x 128 stillel
Rel.U
Com x 128 x 128, stridenl
BatchNorm2d
Come 128 x 128, striden
ReLU
Res Blecks Layer
Heeass
Layer Atlention
Channlls
Yes
2
BatchNomm2d
ResBlock x.3 x 128 x 128, srikel
Table 9: ResBlock in the Encoder
Channel Multiplior:
Attention Resolution
[1.244,4]
[1.2,4.4]
ResBlock x.3 128 128, srikee BotchNomm2d
Table 10: Decoder Architecture used in attribute disentanglement
ReLU
FC 4006
Table 8: Encoder Architecture used in attribute disentanglement.
Sclf Attentive
Input Resolution
Output Resolition
Input
Channels
16 Middlle
Input Resolution
Inpui Chinnels
Res Blocks / Layer
scheduler
Layer
Mid Layer Atteation
Inpui Resolution
laput Channels
scheduler
Blockss
Lincar
Yes
16
Channel Multipliers
4 Heads
Res Blocks Layer
[1.1.244]
Attention Resolution
Heads
Channels
[1.2,44]
Channel Multipliers
Base Channels
Attentonn Reselution
11.2,4.4|
[1.2.4.4]
64
Channel Mulligliers
[1.2.44]
Table 11: Unet Encoder Architecture used in object disentanglement.
Table 13: Generative Prior Architecture used in attribute disentanglement.
Table 12: Decoder Architecture used in object disentanglement,
Cony 3 x 3.x x 128, strid=l
ReLJ
HatchNomm2d
stide=
RatchNomm2d
Coav x 128 x 128, stide11
ReLU
Toput Resolution
BatchNomm2d
Cone 3x3 128 128. sridel
Inpui Channels
ReLU
3 scheloler
Lincur
snide-1 BotchiNomi2d
Table 16: ResBlock in the Encoder
# Heads
Channel Multipliers
Base Channels
Res Blocks/ Layer
Resolution
[1.2,4.4]
RcLU
ReaBlock 3 x 128 strid=1
Rel.1
BatchNom2dd
ManchNomi2d
128, snide=1
Input Channels
Mid Layer Attention
Iuput Resolutxcn
Res Blocks Layer
schetluler
Lincar
Yes
2
Table 14: Generative Prior Architecture used object disentanglement.
Rel.l
BatchNomm2d
ResBlock 3 x 3 x 128 128, aride-1
Channel Multigliers
Attentoon Reselution
Base Channels
[1.2.44]
[1.2,4.4]
192
Table 17: Generative Prior Architecture used in object disentanglement.
Table 15: Encoder Architecture
in attribute disentanglement,
19

---

A.7 Details on Construction of CLEVR-Style and MSN-Style datasets
To construct the CLEVR-Style dataset, we first sample 25K images from the original CLEVR dataset and then augment each with three additional styles. Including the unmodified images, this produces total of 80K/10K/10K images for the train/val/test splits, respectively. All augmentations combine simple color-space adjustments with diffusion-based translation. The first style (second column of Fig. [8) applies an HSV shift (hue=0, saturation=8, value=2) followed by image translation using Stable Diffusion 38 with the prompt "an oil painting." The second style (third column of 8ig. applies an HSV shift (hue=0, saturation=1.5, value=2.5), converts to LAB color space, and reduces contrast by a factor of 0.15. The third style (fourth column of Fig. 8 applies an HSV shift (hue=5, saturation=3, value=l), converts to LAB color space for CLAHE (Contrast Limited Adaptive Histogram Equalization), and then segments the image into 480 superpixels. These combined transforms produce 4 different visual styles while preserving the underlying scene composition.
Similarly, we construct the MSN-Style dataset by first sampling 15k images from the original MSN dataset 45) and augmenting with three identical styles used CLEVR-Styye. produces total of 40k/10kk/0k images for the train/val/test splits, respectively. Fig. 9 presents the sample in MSN-Style with four different styles.
Figure 8: Example of CLEVV-Sttye dataset
Figure 9: Example of MSN-Style dataset
A.8 Experimental Details for Object Property Prediction
Matching Technique We developed technique to identify the specific region corresponding to an object's 's representation by analyzing images composed with that representation. For given target object latent representation, we first randomly sample multiple images and encode each into object representation. For each image, we then replace one object latent with the target latent, then decode the mixed representations. If the target object is properly encoded, it appears in the generated images. To determine the object region, we measure the RGB variance across these generated images and compare each generated image to the original one containing the target object representation. Finally, we combine these two metrics: variance and distance to the original image, to specify the object's region. We provide the pseudocode in Algorithm| 1
Evaluation protocol for object property prediction For each property, we follow 21 to train a two-layer MLP classifier with hidden dimension 256 on the frozen object representations.
Algorithm Matching Technique
Require: 2: an image; z enc(w)): object representation of a: n: number of latent vectors; zwrt randomly sampled reference B images 1: functioo MATCHING(2, sr, Fiar)
2:
+- eenende [x) for _n E #
3: Fmined replace htth__a_hhhhh latehllattttat Rat- 4, for 1...n]
4:
decode(_=) for
Smisss
5:
Jutt meun (sten dim 0)
6:
PniseL.cu meann (Faised;
7:
<- sollmaa(1 (1 - distance( Jniteed on) J.mann - 1)]
8:
softmax (ndimax(distance(Paies (faiveee Fiet .meann 1))
9:
return + d) arggmaa (dim 0)
P. Mask indicating marched region
end function
20

---

A.9 Comparison to Likelihood Maximization by Jung et al. [21
We conducted experiments on the CLEVRTex dataset by replacing our prior loss with the likelihood maximization loss proposed in L2C. Tab. T8 shows the results, which clearly demonstrate the superior performance of our likelihood maximization term compared L2C.
Table 18: Object property prediction results with L2C's likelihood maximization (prior) loss
shape (?)
material (1)
position (1)
Ours
70,90
52.20
0.133
Ours
L,2 prior loss
$4.58
27.18
0.165
A.10
Additional Results on Attribute Disentanglement
As in the Shapes3D dataset, our method captures three independent factors, i.e. direction, axis, and appearance, enabling controlled manipulation of each factor in the Cars3D dataset and presents the result in Fig. 10
attr
Target
Source
direction
axis
appearance
Figure 10: Qualitative results on attribute
disentanglement Cars3D dataset.
A.11 Additional Details and Results on Unsupervised Object Segmentation
As described the main paper, our method does not include built-in mechanism (e.g., slot-attention) to explicitly cluster pixels. To address this, we followed [31] to train the Spatial Broadcast Decoder (SBD) 49] on the frozen latent representations to predict object masks for each latent. Specifically, each frozen object representation is decoded individually by the SBD into an RGB image and alpha mask. We then normalize the alpha masks across all object representations using a softmax and use them as mixture weights to combine the RGB outputs into single image. We treat the normalized alpha masks as object-mask proxies for evaluation. The decoder is trained with a reconstruction loss to recover the original image for 30k iterations with learning rate of 0.001. Because the encoder is frozen and the SBD is shallow (see Tab. 19), this training process is relatively inexpensive.
While conventional methods often extract object masks from the attention weights of slot-attentions, we evaluate the baselines using both these alttatttn masks and masks obtained by training an SBD on their frozen latent representations, for fair comparison. The full results are reported in Tab. 20 Tab. 21 the main paper, we present only the SBD-baaed results, since they outperform those derived from slot-attention masks. In Fig. [T] we also present object segmentation results on CLEVR and CLEVRTex datasets.
21

---

Table 19: Decoder Architecture used in object segmentation
Deconv 5 x.5 x 64 64, stride22, padding=2, paddinga
Deconv 5 x 64 x 64, stride=2, padding-2, puppu__adddnne
ReLU
Deconv 5 5 64 64, stride=1. padding=22 output_ parkingg
ReL.U
Deconv 64 stride=2, padding=2. padding_
ReLL
Deconv
stride= paddiing-... outpul padding-
ReI.U
Deconv x 64 4, stride=1, padding=l, ourpor p
ReL.U
Table 20: Quantitative Results of unsupervised segmentation in the CLEVR dataset.
FG-ARL
mBO
Method
Slot-Artention
Mask
Slet-Atteation
SBD Mask
Slot-Attention
SBD Mask
LSD Ours
82,00
91.74 91,20
22.69
25.59
22.98
25 84
54.01
26.54
26.65
Table 21: Quantitative Results of unsupervised segmentation in CLEVRTex dataset.
RG-AR
mloU
mBO
Slot-Atteation
SBD Mask
Slot-Attcation
Mask
Slot-Atention
SBD Mask
LSD
46.54
71.64 82.55
45.87
$6.26
46.93 53.25
56.75
L.C
77.07
56.59
58.33
58.68
Ouez
87.68
$8.88
59.12
Imagge
GT
L.DD
1,CC
CLEVR
Imsgge
GT
CLEV RTex
Figure Unsupervised object segmentation results in the CLEVR and CLEVRTex dataset. In CLEVRTex, our method consistently encodes complete objects into distinct latents, resulting clean object segmentation results. On CLEVR, the constant background appears in the object regions of the segmentation results, leading to relatively lower mloU and mBO (Tab, 20). However, this does not affect the underlying quality of object representations, method still consistently captures each entire object in its segmentation region.
22

---

A.12 Additional Results on Joint Disentanglement of Attribute and Object
We present additional qualitative results on joint disentanglement of attribute (style) and object in Fig. 12 15 and Fig. [6] respectively. While all the baselines struggle to disentangle the style information into single latent representation, our method successfully disentangles the style and transfers it from source to target images. In addition to style disentanglement, our method also disentangles individual objects and enables object-wise manipulation shown in Fig. T6
Source
Target
DisDiff
LSD
L2C
Ours
GT
Figure 12: Style Transfer in LLEER--Styl dataset.
Source
Target
DisDiff
LSD
L2C
Ours
GT
Figure 13: Style Transfer in CLEVR- dataset.
23

---

Source
Target
Dis
LSD
L2C
Ours
GT
Figure 14: Style Transfer in CLEVR-Style dataset.
Source
Target
Disfiff
LSD
L2C
Ours
GT
Figure 15: Style Transfer in CLEVR-Style dataset.
24

---

object
Insert
Remove
Figure 16: Additional qualitative results on Object Manipulation in CLEVR-Style dataset. Objects marked by red arrows are replaced with those marked by blue arrows. It demonstrates that method effectively disentangles individual objects.
25

---

A.13
Additional Results of Joint Disentanglement on Complex Dataset
To further validate our method on more complex and realistic datasets, we augment the MultiShapeNet (MSN) dataset [45] with four different global styles as in our previous CLEVR-Sty experiments (See Appendix A.7 for details). MSN includes 11,733 unique furniture shapes with increased visual complexity compared to CLEVR. We compare our method with the strongest attribute- disentanglement baseline (DisDiff) and ohject-disentangleeen baselines (LSDD, I.2C), reporting quantitative results in Tab.
For style disentanglement, our method achieved the highest style prediction accuracy (Acc) and the lowest style loss (GRAM), demonstrating that it successfully isolates style information in a single latent and transfers it faithfully to the target image. Since baselines lack an internal mechanism to specify the latent representation for style information, we used a simple trick to identify the latent representation encoding style information. We decode all x K possible latent exchanges between two sets of K latents extracted from two images, and choose the pair yielding the lowest GRAM loss. Even with their best-case result, they fell short of our performance, confirming that they do not reliably capture style factor. Although I.2C achieves relatively high style prediction accuracy, style-swapping results in Fig. 17 [8] shows that swapping the single latent representation affects both the global style and objects, which indicates that L2C fails to isolate the global style but rather entangled with object information. DisDiff also fails to produce reasonable compositions, and we conjecture that this is due to their disentanglement objective (variant/invarian loss), which is not well-suited for multi-object scenes and destabilizes the overall training.
For object disentanglement, Tab. 22 shows that our method produces the highest mloU and mBO, indicating accurate localization and tight boundaries for each object mask, while FG-ARI was bit lower than that of LSD and L2C. This trade-off arises because our broadcast decoder generated tight, object-internal masks. 11-ARR-whiie measures membership between two pixels within the same group- penalizess such masks when GT segmentations are looser, whereas mloU and mBO reward the improved boundary precision. contrast, slot attention modules LSD and L2C often generate larger masks that bleed into the background or even include other objects, which leads to higher FG-ARI the expense of mloU and mBO. DisDiff, whose information-theet designed for a varying number of objects, fails to effectively separate either style or object factors.
objective is not
Table 22: Quantitative results on Joint Disentanglement in Atyletyye
Style
Object
Method
Acc(1)
GRAM(1)
FG-ART(1)
mlnUi (t)
mBO (1)
DisDiff
37.0
10.8
5.62
7.69
9.13
LSD
12.5
11.3
$2.7
23.3
23.9
L2C
Ours
97.5
7.16
42.3
44.1
44,2
26

---

Source
Target
Dis
LSD
L2C
Ours
GT
Figure 17: Style transfer results on MSN-Style dataset.
Source
Target
DisDiff
LSD
L2C
Ours
GT
Figure 18: Style transfer results MSN- ASN-Style dataset.
27

---

A.14 Effect of random seeds on performance
We repeat our attribute/object disentanglement experiments with ten/three different random seeds and present the results in Tab. 23 Tab. 24, respectively, showing that our method achieves competitive performance both tasks.
Table 23: Quantitative results on scene-level disentanglement. Our method achieves state-of-the-ar performance almost all of the datasets, except FactorVAE score in Cars3D.
Car3D
Shapes3D
MPI3D
Method
FactorVAE
DCT
FactorVAE
DCT
FactorVAE
DCT
0.906=0.052
0.840=0.066
0.15240.025
0.240=0.051
G-TCVAE
0.855a0.082
0.140=0.011
0.87320.074
0.61320.114
0.179
0,237a0.056
InfoGAN CR 29
0.411=0.013
0.020=0.011
0.587=0.058
0.478:0.055
0.43940.065
0.241+0.055
1.D
0.852+0.039
0.216+0.072
0.805=0.064
0.380+0.064
0.391 +0.039
0.196+0.033
GS
[5]
0.932=0.018
0.209=0.031
0.788+0.091
0,28440.034
0.464
0,229H0. 042
DisCo
0.85520.074
0.271:0.057
0.87720.031
0.70820.048
0.371100030
0.292+0.024
DisDill VQ 52
0.97620.018
0.232a0.019
0.902a0.043
0.723=0.013
0.61740.070
0.337a0.057
Ours
0.877=0.089
0.365#0.073
0.975=0.059
0.837
0.708+0.060
0,458+0.052
Table 24: Object property prediction results with 3 different runs for our model.
CLRRFayy
FVR
Mahhal
Position (E)
(1)
(?)
(1)
SA
72.25
72.33
41.08
79.6
91.30
93.18
0.064
30.41
7,890
0,682
SLASH
85.00
89.23
48.97
35:28
92.26
93.16
0.878
55.13
37.49
0.148
LS5
96.00
98.88
51.29
87.66
91.46
9487
0.062
08:25
51.54
0.197
1.3
92.78
93.57
47.62
73.61
74.03
86.93
0.168
71.54
51.42
0.116
Ours
03 79.7662.10 10
94.29:0
15.42e1
35.7240-37
93.79=9.22
94,33:0.77
0.0585900066
68 29.2.55
47.89c4189
0.14300009
A.15
Experiments on Correlated Factors
The datasets used in our main experiments handle only simple scenarios with statistically independent factors of variation (FoVs), unlike real-world cases. To verify our method more challenging scenarios with correlated factors, we followed Roth et al. [39] to generate correlated benchmarks. We introduced 0.1 correlation between 1/2/3 pairs of GT factors in Shapes3D. as in (39] then trained and evaluated our models using FactorVAL and DCI metrics across 3 seeds. Tab. 25 reports the results.
Our method showed only slight metric drops despite correlations. Unlike prior work that relies statistical independence between latent variables, e.g., Total Correlation, our compositional objective only encourages discovering atomic factors whose compositions valid under predefined mixing strategies. This allows successful disentanglement even with correlated factors, as the compositional requirement doesn't penalize correlations.
Table 25: Quantitative results on the datasct with correlated factors. Our method robustly disentangles the underlying ground-truth factors even with correlated factors.
Amount of Correlation
Factor VAE
DCI
No Correlation
0.975#0.08 059
0.837=0. 105
Pairs: a 0.1
0.930=0.006 061
0.798+0.082
Pairs: 2, a = 0.1
0.997=0.002
0.806+0.060
Pairs: 3, a = 0.1
0.965=0.030
0.801#0.024
A.16 Computing Resources
We conduct all our experiments on GPU Server that consists of an Intel Xeon Gold 6230 CPU, 256GB RAM, and 8 NVIDIA RTX 3090 GPUs (with 24GB VRAM), or 8 NVIDIA RTX 6000 GPUs (with 48GB VRAM). It takes about 24 GPU hours and from 36 48 GPU hours for the attribute and object disentanglement experiment, respectively.
28

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We states our motivation, contributions, scope of our work in abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
29

---

Justification:
We provide limitation of our work in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We do not claim for theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all the information needed to reproduce the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our code is not cleaned and prepared enough for sharing. Our dataset and codes will be released in future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/publcc/eee.c http:///uw...eeeeeeuuebieeeeeeee ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
31

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the details for experimental setting.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard deviation of our experiments for multiple runs with different seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We state information about computing resources in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
32

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurrps. httpp//sssccppub.tttttttttttseee
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide Broader impacts in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: Our paper possess no risk.
Guidelines:
The answer NA means that the paper poses such risks.
33

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the codes, data, papers, and pretrained models in paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed information about generating our new synthetic dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We do not conduct crowdsourcing research with human sugjects.
34

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not conduct crowdsourcing research with human sugjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
We do not involve LLMs any important, original, or non-standard compo- nents.
Justification:
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttppr/////rrrpp/ //oonferencess/2255/.. for what should or should be described.
35


Reviews:
[{"review_id": "2LtmjZacEm", "reviewer": "Reviewer_Z8su", "summary": "This paper is about disentangling attributes and objects in an image with a common framework. The idea is to optimize image reconstructions by mixing factors in the latent. The approach is assessed on public datasets and compared with existing approaches.", "questions": "In addition to or as a complement to what was reported in the weaknesses section, I would like the authors to clarify the following points:\n- How does the field of object-centric learning relate to object disentanglement? Somehow, they seem to be used as synonyms\n- How does the proposed method compare to weakly-supervised learning approaches? Or, in alternative, why have such methods not been considered?\n- The differences with [20] (ref. in the main paper) should be more clearly stated. For instance, they also use an encoding-decoding (a UNet)\n- Rows. 137-139: I find this example misleading. Isn't this more related to object disentanglement, since it reasons on object parts?\n- Row 162: Is the separation of attribute types in the latent forced at training time? In general, some more details on the training would be appreciated\n- Row 186: Is the final targeted downstream task image generation?\n- Eq. (5); Shouldn't we have an average on the bottom part of the fraction? As it is now, my impression is that the numerator and denominator are not numerically comparable\n- Sec. 4.1 and 4.2: I am not sure what version of the method we are talking about here. For instance, are we evaluating in Sec. 4.1 the relevant portion of the latent (i.e. the first M attributes) jointly learnt with the rest, or is it a \"partial\" learning of only that part?  \n- As already mentioned, although what is presented in the experiments on joint disentanglement is nice, the fact that only one dataset and only one style attribute are used should be justified, as it looks like a limitation", "limitations": "It is mentioned in Sec. 5 that limitations are discussed in the Appendix. However, there is no Appendix to the paper, while the information can be found in the Supplementary Material.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "This paper represents a nice reading, overall very clear (with few exceptions). The proposed idea shows a good level of novelty, and it may have a potential impact on different tasks where managing interpretable representations can be an add-on.\n\nOn the weaknesses, I want to mention the following points:\n- The introduction is not fully clear, and it fails to appropriately prepare the reader for the rest. Maybe, examples to give the intuitions behind attribute and object disentanglement could help to shape the story\n- On the related works, in my understanding, the class of disentanglement methods relying on weak-supervised (e.g. [1]) is not discussed. However, such methods may have the potential to address the challenges in this paper and might be of interest. \n- Again, on the related works, it would be useful to clarify the difference between the concepts of \"object disentanglement\" and \"object-centric learning\". To the unfamiliar reader, the impression is that the terms are sometimes used as synonyms\n- On the training, I am not sure I fully got the details. For instance, learning the disentanglement model adopted later for mixing is a step included in the training of the proposed approach, or is the idea to rely on pre-trained models?\n- The experiments are nice, but only refer to synthetic data, while in more general (possibly real) scenarios, conditions are more challenging  (for instance, we may have FoVs distributed on more than one component of the latent, or mixed). Some clarifications on the generalisation would help \n- Again, on the experiments, I think that weakly-supervised methods should be taken into account\n- Regarding the joint disentanglement, I have a couple of doubts. The first one is that while this is the core contribution of the paper, it is evaluated on one dataset only, and this is somehow a limitation. The second doubt refers to the fact that, as far as I understand, only one style attribute is considered, against multiple object-based properties. This leads me to another doubt that I have: in Shapes3D, why are we not considering floor and wall colours as style attributes? A clarification would help appreciate the difference\n- At different points, it is stated that the method can freely choose model architectures. If I'm corretly interpreting the statement, this would require showing experiments involving at least another architecture, to show this ability\n\n\n[1] https://arxiv.org/pdf/2002.02886", "was_revised": true, "final_justification": "Considering the reviews, the rebuttal and the discussion with the authors, I confirm my positive opinion on this paper. In the rebuttal, the authors successfully addressed my doubts, in particular they provided further experimental evidence of the joint disentanglement, which is a key contribution of this work.", "rebuttal": "We thank to the reviewer for their time and effort in providing constructive review.\n\n> W1. Clarification on the Introduction.\n\nFollowing the reviewer’s suggestion, we provide \na high-level intuition for how we derive compositional bias to disentangle attributes and objects as follows: \n\nAttributes and objects represent two distinct types of generative factors that differ in how they combine to form valid scenes (see Fig.1). Each attribute factor appears exactly once in a scene (e.g., a face has only one nose), whereas multiple instances of object factors can appear independently (e.g., varying numbers of cars on a road).\n\nInspired by this, we formulate those ‘valid’ compositions as mixing strategies and achieve disentanglement by ensuring that composite images generated through these mixing strategies appear realistic.\n\n\n> W2/W6/Q2. Comparison to weakly-supervised method.\n\nWe thank for pointing out the relevance of weakly-supervised (WS) disentanglement methods.\n\nIndeed, there is a line of work that addresses disentangled representation learning through weak supervision [1, 2]. A typical approach utilizes observation pairs that share a certain number of factors of variation (FoV). For instance, [2] assumes we are given paired observations for which some (but not all) FoV have the same values. Such additional knowledge from weak supervision has shown effectiveness in identifying the underlying Fov.\n\nMeanwhile, WS methods are not always useful, since such observation pairs are often hard to obtain, e.g., finding two people with identical eyes or nose. Thus, significant research efforts are focusing on the more challenging unsupervised setting of disentangled representation learning, which is our main scope. Note that, as [3] propose that purely unsupervised learning of disentangled representations is infeasible, many recent works design and utilize inductive biases for each underlying factor of variation(FoVs). While these inductive biases are mostly factor-specific and non-compatible with each other, our main contribution is a common framework that can handle various FoVs.\n\nNevertheless, the relationship between our method and WS approaches and the application of our method to the WS setting seems to be an interesting direction. In the current version, our method is not directly compatible with WS methods, since objective functions of those methods are mostly restricted to VAE formulations. Thus, designing a more general approach, in terms of model or loss, for the WS setting is needed. However, we think this is out of scope in the current paper and leave it as future work. \n\n[1] Weakly Supervised Disentanglement with Guarantees, ICLR20 \\\n[2] Weakly-Supervised Disentanglement Without Compromises, ICML20 \\\n[3] Common assumptions in the unsupervised learning of disentangled representations, ICML19\n\n\n> W3/Q1. Clarification on \"object-disentanglement\" and \"object-centric learning\".\n\nWe used those terms as synonyms. We will unify the terminology as 'object-centric learning' for clarity.\n\n> W4. Clarification on training.\n\nThe disentanglement model (encoder) is jointly trained with the reconstruction and compositional consistency objectives computed from composite images. \n\n> W5. More general scenarios\n\nWe agree with the reviewer that the datasets used handle only simple scenarios with statistically independent factors of variation (FoVs), unlike real-world cases.\n\nTo verify our method on more challenging scenarios with correlated factors, we followed [4] to generate correlated benchmarks. We introduced 0.1 correlation between 1/2/3 pairs of GT factors in Shapes3D, as in [4], then trained and evaluated our models using FactorVAE and DCI metrics across 3 seeds (see table below).\n\n|Amount of Correlation|FactorVAE|DCI|\n|:---:|:---:|:---:|\n|No Correlation|0.975±0.059|0.837±0.105|\n|Pairs:1, $\\sigma=0.1$|0.930±0.061|0.798±0.082|\n|Pairs:2, $\\sigma=0.1$|0.997±0.002|0.806±0.060|\n|Pairs:3, $\\sigma=0.1$|0.965±0.030|0.801±0.024|\n\nOur method showed only slight metric drops despite correlations. Unlike prior work that relies on statistical independence between latent variables (e.g., Total Correlation), our compositional objective only encourages discovering atomic factors whose compositions are valid under predefined mixing strategies. This allows successful disentanglement even with correlated factors, as the compositional requirement doesn't penalize correlations.\n\n[4] Disentanglement of correlated factors via hausdorff factorized support, NeurIPS23.\n\n\n> W7/Q9. Regarding the joint disentanglement.\n\nDue to the limited time during the rebuttal period, our additional experiment on joint disentanglement is not finished yet. We will provide a joint disentanglement experiment on MultiShapeNet dataset, which has increased complexity of objects with various furnitures, and results will be provided in the second discussion round.\n\nAs stated in our manuscript, we refer to attribute factors as globally-shared scene properties, e.g., color, style, whereas object factors are distinct spatial entities, such as individual objects. In Shapes3D, all of the data consist of unique floor and wall colors, so we classify them as attributes.\n\n> W8. Clarification on model architectures.\n\nLines 253-254 state our method can \"freely choose model architectures..\" but this doesn't mean it works well with arbitrary architectures.\n\nTo elaborate, while it has been shown that expressive power of the model architecture often becomes a bottleneck for disentanglement in complex scenes [6,7,8], conventional unsupervised disentangled representation learning methods are forced to use less expressive model architectures. Specifically, the VAE-based total correlation minimization loss [9,10] requires, by definition, factors to be encoded in single dimensions, making it difficult to use for vector-wise disentanglement. Additionally, object-centric learning methods [11] often rely on additive decoders to prevent interaction between latent variables. Those architectures work adequately on relatively simple synthetic datasets but such independent modeling cannot handle complex scenes involving interaction between objects [7]. These constraints enforce model architectures that often lead to lower performance, i.e., dimension-wise encoding or less expressive decoders. In contrast, our method is designed to be free from such architectural constraints that can bottleneck performance, allowing for sufficient expressive power, e.g., with vector-wise encoding and powerful diffusion decoder. \n\nThe intention in our paper was to use the term 'freely' in this context. However, we agree that the current writing could be misleading, and will clarify it in the revision.\n\n[6] Unsupervised Disentanglement of Diffusion Probabilistic Models, NeurIPS23 \\\n[7] Illiterate DALL-E Learns to Compose, ICLR22 \\\n[8] Object-Centric Slot Diffusion, NeurIPS22 \\\n[9] Disentangling by Factorising, ICML18 \\\n[10] Isolating Sources of Disentanglement in VAEs, NeurIPS18 \\\n[11] Provably Compositional Generalization, ICLR24\n\n\n> Q3. The differences from [20]\n\nAs stated in Lines 117-122, the main difference from our work is that [20] heavily relies on object-specific architecture such as slot-attention (SA) encoder and thereby cannot disentangle spatially non-exclusive factors, e.g., attributes. In contrast, our method does not impose factor-specific bias on the model architecture enabling disentangle both attributes and objects on the same model architectures (Sec 4.3). \n\nSpecifically, in terms of model architecture, [20] employs an auto-encoding framework where the encoder includes a SA module. In contrast, our method replaces the SA in QFormer to avoid spatial-exclusiveness biases within SA (Line 222), enabling application into attribute disentanglement. \n\nIn terms of learning objective, [20] uses unconditional diffusion model (DM) for maximizing the likelihood of the composite image, while we propose to use conditional DM for likelihood and modify the objective accordingly, as score estimation of unconditional DM is more robust to OOD compared to that of conditional DM. \n\nPlease refer to Lines 179-187 and Appendix A.9 for detailed discussion. We will revise our paper to more clearly state these distinctions.\n\n\n> Q4. About example in L137-139\n\nWe used eyes, a nose, a mouth within a face as an intuitive example for attributes, since those factors are uniquely determined in each face. However, we agree with the reviewer that it could be misleading as those components can be spatially decomposed. We will revise the example into spatially non-exclusive ones such as color, texture, material of the ball to clearly distinguish it from object factors.\n\n> Q5. The separation of attribute types at training\n\nYes, they are separated in the training stage. Given a pair of two images in the training stage, first M latent representations for each image are mixed with attribute mixing strategies (Eq.1) while the rest of K-M  latents are mixed with object mixing strategies (Eq.2). Then this composite representation is decoded into image via Decoder $\\phi$ and optimized with reconstruction, compositional consistency  and prior losses. We will clearly state details in the revised version of the paper. \n\n> Q6. The targeted downstream task\n\nIn the context of Lines 186, image generation is not our downstream task but part of our training objective. Our encoder is optimized to learn latent representations such that composite images generated by predefined mixing strategies to be valid. \n\n> Q7. About Eq.5\n\nNote that taking the average of the denominator will not affect the overall training as 1/N (averaging) will be decomposed into a constant w.r.t. $\\theta$.\n\n> Q8. Clarification on Sec 4.1 and 4.2\n\nWe used Eq.1 to randomly mix the entire set of latents for Sec. 4.1. Similarly, we use Eq.2 for Sec. 4.2. Only Sec. 4.3 uses both of the equations (Eq.1 for first M and Eq.2 for K-M latents) and all of the latents are jointly optimized."}, {"review_id": "3PbJZI47MC", "reviewer": "Reviewer_Yeku", "summary": "The paper proposes a disentanglement framework that decouples factor-specific inductive biases from learning objectives and architectures, enabling both attribute and object disentanglement under a single set of objectives and architectures. The framework derives mixing strategies as a compositional bias, embedding factor-specific biases in a modular way, and proposes specific strategies for attribute, object, and joint disentanglement.", "questions": "According to the weaknesses I previously highlighted, addressing the following points could potentially change my opinion:\n\tA thorough discussion clarifying the essential differences between the proposed “mixing strategy” and the commutative principle or Group-based DRL training methods.\n\tMore convincing visual comparisons with baseline models to substantiate the claims.\n\tExperimental results conducted on real-world multi-object datasets to demonstrate practical applicability.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths \n\tThe concept of separating attributes and object disentanglement is straightforward but significant for the DRL domain.\n\tThe Mixing Strategy is clearly articulated, with well-defined designs for losses and objectives.\nWeaknesses\n\tTraining a DRL model using the commutative principle within Group theory is a common approach. This method exchanges specific factors between two different images or objects for reconstruction, encouraging the model to learn the attributes of each specific factor. The authors are encouraged to clearly highlight the differences between this work and existing Group-based DRL methods to demonstrate the novelty of their approach.\n\tIn lines 90-93, the authors claim distinctions between their object disentanglement strategy and other object-centric DRL models. To support these claims, more visual qualitative comparisons with state-of-the-art object-centric DRL models should be provided in Section 4.\n\tAlthough quantitative results on attribute disentanglement are presented in Table 1, a visual comparison with baseline methods is still necessary to validate the superior performance of the proposed model. This could be included as an extension of Figure 2, showcasing comparisons with baselines. \n\tThe experiments are currently conducted only on toy datasets such as CLEVR and Shape3D, which limits the contribution of this work to the DRL community. Given that the study aims to disentangle attributes and objects simultaneously, the authors should consider evaluating their method on real-world multi-object datasets to demonstrate its practicality.", "was_revised": true, "final_justification": "Sorry for the delay due to some personal issues, and I have already read all comments and discussions about this paper and have finally decided to raise my initial score to 4.", "rebuttal": "We thank to the reviewer for their time and effort in providing constructive review.\n\n> W1/Q1. Comparison to existing Group-based DRL\n\nAs the reviewer pointed out, disentangled representation learning using Group theory is indeed an actively researched area and related to our work. Group theory-based DRL typically define disentangled representation $Z$ as follows: Given ground truth factors of variation $W$ and decomposable group $G$ (i.e., $G=G_1\\times G_2\\times …\\times G_n$), the representation $Z$ is disentangled w.r.t. $G$ if (1) there exists a mapping $f$ from $W$ to $Z$ such that $f(g\\cdot W)=g\\cdot f(W)$ for all $g\\in G$ and $w\\in W$, and (2) there is a decomposition $Z=Z_1\\times…\\times Z_n$ such that each $Z_i$ is affected only by $G_i$.\n\nDespite this convincing principled definition, since the GT factors of variation $W$ are infeasible to obtain in unsupervised DRL, existing unsupervised methods often utilize necessary conditions for group actions of $G$ applied to $Z$ and disentangled representation $Z$ [1,2]. For instance, [1], the related work closest to our method, defines permutation group actions of element-wise addition on $Z$ and introduces losses to enforce commutativity and cyclicity of group actions. \n\nOur method takes a similar approach, but with important distinctions. From a Group theory perspective, unlike existing work, the group action in our method is defined on disentangled representation pairs $(z^1, z^2)$ rather than a single latent $z_i$. By defining the group action on a pair $(z^1, z^2)$, we can impose additional necessary conditions for how each underlying factor combines to generate observations, which cannot be induced by group action defined on a single latent $z_i$. For example, commutativity and cyclicity of group action are necessary conditions for both attributes and objects but do not impose attribute or object-specific properties. \n\nOur main contribution here is that we define group action as factor-specific mixings (i.e., permutations) between two latents and demonstrate that this additional necessary condition imposes effective factor-specific inductive bias for attribute and object disentanglement without changing the overall learning objectives or model architectures. To the best of our knowledge, we are the first to study differently learned disentangled representations of different factors of variation through a mixing strategy (or a form of group action) without employing factor-specific architectures or learning objectives.\n\n[1] Towards building a group-based unsupervised representation disentanglement framework, ICLR22 \\\n[2] Commutative lie group vae for disentanglement learning, ICML21\n\n\n> W2/Q2. More visual qualitative comparisons with SoTA object-centric DRL methods\n\nWe thank the reviewer for the valuable comments. Due to the policy that prohibits providing any links during rebuttal, we cannot provide qualitative results during rebuttal. \nInstead, we describe the differences in qualitative results here and will provide visual qualitative comparisons to SOTA object-centric methods in the revised version of the paper. \n\n\nIn Lines 90-93, we claim that existing slot attention-based methods fail to disentangle spatially non-exclusive attributes such as global styles. In joint disentanglement experiment (Sec. 4.3), we observed that both LSD [3] and L2C [4] fail to disentangle style information into a single latent representation. When we decode all K × K possible latent exchanges between two images to identify the latent that encodes the style information, we found that exchange of single latent does not alter the global style, implying that style information is distributed along multiple latent representations. Therefore, the accuracy for both LSD and L2C are very low in Table 4. In contrast, our model successfully alters the global style of the scene without affecting the objects, which shows that ours successfully captures the style information into a single latent. \n\n[3] Object-Centric Slot Diffusion, NeurIPS23 \\\n[4] Learning to Compose: Improving Object Centric Learning by Injecting Compositionality, ICLR24\n\n\n\n> W3/Q2. Qualitative comparison of attribute-DRL with baselines\n\nAgain, due to the policy that prohibits providing any links during rebuttal, we will provide a description of qualitative results and further provide them in the revised version later. \n\n\nFor qualitative comparison, we choose the strongest baseline, DisDiff [5] and perform attribute-wise swapping as in Figure 2. \nFor the Car3D dataset, DisDiff often produces unrealistic samples as shown in the second column of Figure 5 in the DisDiff paper. While DisDiff shows reasonable manipulation along each attribute, we found that shapes are often entangled with car colors. \nMoreover, for the MPI3d dataset, we found that changing a single attribute, e.g., object shape in DisDiff often changes other attributes such as object colors or sizes. \n\nIn contrast, our method generally shows better disentanglement in terms of attribute swapping except in the extreme cases when the size of the object is too small so that it is hard to distinguish the attributes in the image. We found that both disdiff and our method perform well on the Shapes3D dataset. \n\n[5] Unsupervised Disentanglement of Diffusion Probabilistic Models, NeurIPS23\n\n\n> W4/Q3. Evaluation on real-world multi-object datasets\n\nTo address the reviewer's concern, we conducted additional experiments on the more realistic multi-object dataset, MultiShapeNet (MSN) dataset (Please refer to bottom rows in Table 3 of [6]). It includes 11733 unique shapes of furniture with increased overall complexity of shapes compared to CLEVR datasets. We compare our method with object-centric approaches using unsupervised segmentation. \n\nThe model architecture and hyperparameters were kept the same as in the previous object disentanglement experiments. We measure FG-ARI, mIoU, mBO on object masks following common practices in object-centric literature. As our method does not have a built-in mechanism to directly express group memberships between pixels, we additionally train a Spatial Broadcast Decoder on the frozen latent representations to predict explicit object masks for each latent representation as described in Section 4.2. The results are reported in the below table.\n\n\n| Model | FG-ARI | mIoU | mBO |\n|---|---|---|---|\n| SLATE | 70.44 | 15.55 | 15.64 |\n| LSD | 67.22 | 15.39 | 15.46 |\n| Ours | **76.34** | **36.69** | **36.90** |\n\nAmong the slot-attention based baselines, our method achieves the best performance across all of three metrics, even though ours does not employ any spatial clustering mechanism like slot attention. These results demonstrate the effectiveness of our framework in disentangling object representations in a complex dataset.\n\n\n[6] Stelzner el al., Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation, in ICLR 22."}, {"review_id": "ghBUnXP0ZJ", "reviewer": "Reviewer_1m9R", "summary": "This paper learns disentangled representations of attributes and objects through self-supervised methods, which is a highly meaningful and interesting research direction. By designing different mixing strategies and learning strategies, the method achieves highly effective spontaneous decoupling, and demonstrates relatively obvious advantages over existing methods in multiple metrics across various datasets. Previous Disentangled Representation Learning was mainly based on the Slot Attention framework. This work designs new learning objectives and architectures, achieving better performance in some aspects, which I consider highly innovative and appealing.", "questions": "Question: I would like to ask whether an object is represented by a single latent or multiple latents. My understanding is that the dimensions of the representation are grouped, and an object has multiple attribute representations. I wonder if my understanding is correct. If so, does it mean that specific channels of the representation encode specific attributes?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Propose attribute mixing strategy and object mixing strategy, which is the key to achieve attribute and object disentangle.\n\n2. Propose novel learning objectives and the experimental results show its effectiveness.\n\n3. Paper is well-organized, and the experiments are comprehensive.\n\n\nWeeknesses:\n1. Objective function is relatively complex, which may limit its potential and scaling to more data.", "was_revised": true, "final_justification": "This is an interesting work with good presentation. The authors' responses have well addressed my questions. I believe it meets the standards for publication and will maintain my rating.", "rebuttal": "We thank to the reviewer for their time and effort in providing constructive review.\n\n> W1. Objective function is relatively complex, which may limit its potential and scaling to more data.\n\nWe thank the reviewer for the valuable comment. Although our objective function consists of several components, we empirically observe that our method works well across three different tasks (object-, attribute-, joint-disentanglement) and seven different benchmarks, which implies the robustness of our objective functions. Moreover, our ablation study (Table 5), reveals that all objective functions are crucial for our method.\n\nStill, we agree that multiple objective functions of our method may require tuning of hyperparameters in larger and complex data. Improving our work into a more generalized version and stabilizing it for larger, complex data will be our important future work.\n\n\n> Q1. Whether an object is represented by a single latent or multiple latents.\n\nIn our implementation, both attributes and objects are represented by vector representations. Although objects are represented by multiple dimensions, each dimension does not have to encode specific attributes unless we impose additional constraints. As the reviewer suggested, objects could be regarded as a group of attributes so we would be able to extend our mixing strategies to disentangle those factors with hierarchy, i.e., scenes are decomposed into objects, and objects are further decomposed into attributes. We believe this is an interesting and important direction for our future work."}, {"review_id": "oPmygq0tBg", "reviewer": "Reviewer_XmPJ", "summary": "This paper proposes a novel, general method for disentangled representation learning (DRL), with the lens of (attribute and object) compositionality. The proposed compositional bias does not rely on leveraging specific attribute properties (such as, e.g., the geometry of the factors) and, thus, is very flexible. This is demonstrated by extensive experiments.", "questions": "1. Do you have an intuition why your method works worse on Cars3D?\n2. Could you please spell out with formulas/pseudocode your matching technique from A.7?\n3. For your baseline models, how much hyperparameter tuning did you do?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper is well written, it's high-level message easy to follow (thanks to Fig. 1 and the contribution list)\n- The experiments are extensive (and show superior performance or comparable to SOTA)\n- The authors address their works limitations (no theoretical guarantees, no handling of unknown factors)\n\n## Weaknesses\n\n### Major points\n**My main concern is the similarity (and even the same naming) to Wiedemer et al., 2024, which is not discussed in the paper (it is reference number 45, which is only briefly mentioned in the appendix, without a substantial discussion**. Especially that said paper has theoretical guarantees. I do notice that this paper has more extensive experimental evidence, and the formulation of the mixing strategy seems to be novel; however, that does not eliminate the need of the comparison to Wiedemer et al., 2024. I consider addressing this point as the necessary condition for the re-evaluation of my score. \n\n- **Abstract:** even though I work on compositional generalization, I could not really figure out the contributions from the abstract (particularly the mixing strategies part). After reading the paper, it became clear. I'd suggest rephrasing it (I am happy to iterate over suggestions during the rebuttal)\n- **Object and attribute latents:** please distinguish in the notation of the two types of latents better (I get that the super-/subscipts aim to indicate that). Particularly, I believe (2) might be incorrect: you cannot pick any object atributes (say, you pick the object color twice, but no shape). if this is excluded, please make it clearer.\n- (5): this formulation seems to be the same/very similar (depending on $d$) to the InfoNCE objective. Please refer to that to establish the connection\n\n### Minor points\n- L91: spatially non-exclusive factors could be a bt misleading, maybe consider \"global\" (my guess based on the exampels you provide)\n- L108: define $\\theta, \\phi$\n- **Limitations:** though the product space assumption is prevalent in the compositionality literature, all compositions might not imply valid configurations (e.g., hallucinations, or placing anachronistic objects in an image, e.g. a television into a medieval scene). I'd add this to either the limitations or to L52\n- L61: please reference the DCI paper\n- (4) - why do you calll it $\\mathcal{L}_{prior}?$ I believe it refers to the diffusion model, so maybe spell this out or call it the \"diffusion loss\" or something similar\n- L220 (typo?): do you mean the image decoder? I don't understand how an encoder would take features\n- Fig. 2: please add more information to the caption to make it self-contained (at least refer to your description in L255 and onward)\n- Fig. 3: the image shows $\\emptyset,$ but the caption shows $\\phi$ (al;so the green color is hard to see)\n- **Metrics**: could you please add a short description regarding how your metrics (FG-ARI mIoU, mBO, GRAM, etc) are defined? It's fine in the appendix\n- Could you plase add emphasis (bold/underline + explanation) to Tab. 4?", "was_revised": true, "final_justification": "Empirical and methodological contributions are useful and meaningful,  and the authors resolved my concerns regarding clarifying their contributions.", "rebuttal": "We thank to the reviewer for their time and effort in providing constructive review.\n\n> W1. Comparison to Wiedemer et al., 2024 [1]\n\nWe appreciate the valuable comments. We agree that the distinction between our method and Wiedemer et al. [1] should be more clearly stated. While both approaches share the conceptual goal of enabling generalization to novel compositions of factors in disentangled representation learning, there are fundamental differences in how we achieve valid generalization. \n\nAs the common part, both methods recognize that valid generalization requires two key components: first, compositions of disentangled representations must yield valid data (e.g., realistic images or representation $z^c \\in Z$), and second, the composite representation $z^c$ must properly encode the information from its corresponding composite image $x^c$ to satisfy the representation learning objective. To address the second requirement, both our method and [1] introduce a compositional consistency loss with minor differences but identical naming. However, the approaches diverge significantly for the first component–how to ensure the validity of composite representations.\n\nWiedemer et al. [1] employ an additive decoder to ensure valid compositions of disentangled representations (Sec. 3.1 in [1])). However, additive decoders are known to be unscalable for complex scenes, as their local decoding mechanism cannot capture complex interactions between objects due to limited expressive power [2]. More critically, additive decoders are designed based on the spatial exclusiveness bias (Def. 4 in [1]) , which assumes that each pixel should be affected by only a single latent variable. This limits their applicability to object-centric learning scenarios, preventing generalization to other disentanglement tasks.\n\nIn contrast, our method ensures validity through a prior loss (Eq.4) that leverages SDS loss [2] to encourage both $z^c$ and its corresponding composite image $x^c$ to be realistic. As we do not require additive decoder anymore, our approach allows us to utilize an expressive diffusion decoder for modeling complex scenes. Crucially, since our method does not embed factor-specific biases like spatial exclusiveness into the architecture, it can be applied beyond object-centric learning to attribute disentanglement or joint disentanglement scenarios. \n\nMore importantly, to the best of our knowledge, we are the first to demonstrate that different underlying factors of variation can be disentangled simply by adjusting the factor-specific mixing strategy, distinguishing our approach from previous methods that introduce factor-specific biases through architectures (e.g., spatial decoders in [1]) or loss functions (e.g., total correlation loss in FactorVAE). \n\nFinally, we provide a quantitative comparison of [1] with our method in attribute-, object-, and joint disentanglement. For a fair comparison, we used the same encoder for [1] as ours and only changed the decoder to an additive decoder. As expected, [1] cannot disentangle attribute factors at all (Shapes3D, CLEVR-Style), since they violate the spatial-exclusiveness assumptions. Moreover, in object disentanglement, we found that an additive decoder alone cannot disentangle objects (in fact, [1] validated their method only on very simple 2D synthetic datasets). When we additionally use the slot-attention module in [1], it reasonably disentangles objects in CLEVR dataset but still significantly inferior to our method in CLEVR-Style possibly due to the limited expressive power of the additive decoder. Additionally, we observed that object-wise manipulation with [1] always leads to unrealistic images with transparently overlapping objects due to the lack of interactions between latents inside the additive decoder.\n\n||Shapes3D||CLEVR||||CLEVR-Style||||||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n||FactorVAE|DCI|Shape|Color|Material|Position ($\\downarrow$)|Acc|GRAM ($\\downarrow$)|Shape|Color|Material|Position ($\\downarrow$)|\n| Ours | **0.975** | **0.837** | **87.04** | **93.93** | **94.81** | **0.032** | **96.5** | **5.05** | **83.56** | **90.48** | **93.74** | **0.053** |\n| Additive Decoder w/ SA | - | - | 33.86735 | 15.2355 | 51.8744 | 0.76527 | 23.3 | 18.585 | 34.20378 | 13.53733 | 50.89715 | 0.51651 |\n| Additive Decoder w/o SA | 0 | 0.031 | 82.90612 | 93.14322 | 91.78148 | 0.1095 | 23.3 | 15.84 | 35.96604 | 20.98686 | 54.10125 | 0.51978 |\n\n[1] Provable Compositional Generalization for Object-Centric Learning, ICLR24\n[2] Text-to-3D using 2D Diffusion, ICLR23\n\n> W2. Rephrasing the Abstract\n\nWe thank the reviewer for the valuable comments.\nWe will revise the Lines 8-16 to clarify the contribution of mixing strategies: \n“Our key insight is that different factors obey distinct “recombination rules” in the data distribution: global attributes are mutually exclusive (a face has one nose), whereas objects share a common support (any subset of objects can co‑exist). By training the model to maximize the likelihood and self‑consistency of images reconstructed from latents that are randomly remixed according to these rules, we force the encoder to discover whichever factor structure the mixing strategy reflects, without modifying the encoder, decoder, or loss terms.\"\n\n> W3. Better notation for objects and attributes\n\nSuperscript and subscripts denote image index and latent index, respectively. Both an attribute and an object latent are denoted with a single latent vector $z_i$, where which types of factors to be encoded in latent vector $z_i$ is determined by our mixing strategies (Eq. 1,2,3). That being said, if latent vector $z_i$ is mixed with other latents following Eq. 1 or 2 when training, the resulting $z_i$ will capture attribute or object, respectively. This is core motivation of Eq.3 that induces the joint training of attribute and objects. First M latents $z_{1:M}$ and rest of the K-M latents $z_{K-M:K}$ are mixed with Eq.1 and Eq.2, respectively, promoting the learning of attributes on first M latents and objects on rest of them. We empirically found that first latent which is mixed with Eq.1 always captures style attributes, while the rest of the slots encodes object concepts in Sec. 4.3. We will clearly state it in the main paper. \n\n> W4 / W8. Reference for the InfoNCE and the DCI paper\n\nWe will add references to the InfoNCE and DCI papers. For InfoNCE, we will further clarify its connection to our objective in the main paper.\n\n> W5. Misleading term in Line 91\n\nWe use the term spatially non-exclusive to contrast with spatial exclusivity. We will revise the term to 'global' for clarity. \n\n> W6. Missing definitions for $\\theta$ and $\\phi$\n\nWe will clearly define $\\theta$ and $\\phi$ as the learnable parameters of the encoder and decoder, respectively, in L108.\n\n> W7. Suggestions for adding limitation\n\nWe will add the discussion in the limitation section.\n\n> W9. Why do you call it $\\mathcal{L}_{prior}$?\n\nWe followed [20] for coherent terminology. As this loss is not for training the diffusion model, but rather optimizing the given data with a pretrained diffusion model that estimates a data prior. \n\n> W10. Clarification on the image decoder (L220)\n\nWe first encode the given image with pretrained vae model and we feed this vae latents into our encoder. It resembles Latent Diffusion model where input to the diffusion model is vae-features instead of RGB images. We will clearly state it in our main paper. \n\n> W11/W12. Suggestion for Fig.2 and Fig.3\n\nFor Fig.2, we will incorporate more information from the main paper to improve the caption so that it is self-contained, and we will refer to the descriptions in L255 and onward. For Fig.3, we will revise the caption and color for better readability.\n\n> W13. Adding descriptions for metrics.\n\nFollowing the reviewer's suggestion, we will add descriptions regarding our evaluation metrics (FG-ARI, MBO, GRAM, DCI, etc.) in the revised version. \n\n> W14. Adding emphasis to Tab. 4\n\nWe will add emphasis in our main paper. \n\n> Q1. Intuition for performance on Cars3D\n\nTo clarify, we argue that our method is not worse on Cars3D because it significantly outperforms the baselines on the DCI metric, despite being inferior on FactorVAE score. The major drop in score in our method is mainly due to the disentanglement of the rotation factor into horizontal axis and angle (Fig. 9 in Appendix), which might be penalized when a single factor is not correctly aligned with the GT factor (rotation). In contrast, the major performance of DisDiff seems to come from the entanglement of factors such as car shape and color, as shown in the last row of Fig.5 in the DisDiff paper.\n\n> Q2. Pseudocode for matching technique (A.7)\n\nHere is the pseudo code for our matching technique to identify object region:\n\n```\nx : an image\nz : Object representation of x, i.e., z = enc(x)\nn : the number of latent vectors\nx_ref : randomly sampled reference B images\ndef matching(z, x, x_ref):\n    # Generate mixed latent representations\n    z_ref = [encode(_x) for _x in x_ref]\n    z_mixed = [replace_ith_latent(z_ref, z, i) for i in range(n)]\n    x_mixed = [decode(_z) for _z in z_mixed]\n\n    # cm of imgs\n    x_ref_cm = x_ref.mean(dim=0)\n    x_mixed_cm = x_mixed.mean(dim=1)\n\n    # Metric 1\n    s = softmax([1 - distance(x, x_mixed_cm).mean(-1)])\n\n    # Metric 2\n    d = softmax([distance(x_mixed_cm, x_ref_cm)].mean(-1))\n\n    return (s + d).argmax(dim=0) # mask\n```\n\n> Q3. Baseline models hyperparameter tuning\n\nFor the attribute disentanglement experiment, we report the values from the DisDiff paper, so we do not train baseline models ourselves. \n\nFor object disentanglement and joint disentanglement, we aligned the hyperparameters (e.g., depth) for model architectures of LSD and L2C with ours. For all experiments, we used the best-performing learning rate, which turned out to be similar across all baselines and our methods."}]

Author rebuttal:

Meta-review: The paper makes a practical contribution to disentangled representation learning by introducing a modular compositional bias that supports attribute, object, and joint disentanglement under a unified framework. The approach is conceptually clear and empirically well-supported. While some concerns remain, including the lack of theoretical justification and overlap with prior work, these do not undermine the core practical contribution.
Decision: Accept (poster)