Interaction Asymmetry: A General Principle for Learning Composable Abstractions
Authors: Jack Brady
Julius von Kügelgen
Sebastien Lachapelle
Simon Buchholz
Thomas Kipf
Wieland Brendel
Venue: ICLR 2025 Poster (2025)
Keywords: disentanglement
compositional generalization
representation learning
object-centric learning
identifiability
unsupervised learning
out-of-domain generalization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose interaction asymmetry ("parts of the same concept have more complex interactions than parts of different concepts") as a general principle for provable disentanglement and compositional generalization of concepts without supervision.

Abstract: Learning disentangled representations of concepts and re-composing them in unseen ways is crucial for generalizing to out-of-domain situations. However, the underlying properties of concepts that enable such disentanglement and compositional generalization remain poorly understood. In this work, we propose the principle of interaction asymmetry which states: "Parts of the same concept have more complex interactions than parts of different concepts". We formalize this via block diagonality conditions on the $(n+1)$th order derivatives of the generator mapping concepts to observed data, where different orders of "complexity" correspond to different $n$. Using this formalism, we prove that interaction asymmetry enables both disentanglement and compositional generalization. Our results unify recent theoretical results for learning concepts of objects, which we show are recovered as special cases with $n=0$ or $1$. We provide results for up to $n=2$, thus extending these prior works to more flexible generator functions, and conjecture that the same proof strategies generalize to larger $n$. Practically, our theory suggests that, to disentangle concepts, an autoencoder should penalize its latent capacity and the interactions between concepts during decoding. We propose an implementation of these criteria using a flexible Transformer-based VAE, with a novel regularizer on the attention weights of the decoder. On synthetic image datasets consisting of objects, we provide evidence that this model can achieve comparable object disentanglement to existing models that use more explicit object-centric priors.

Published as a conference paper at ICLR 2025
INTERACTION ASYMMETRY: A GENERAL PRINCIPLE FOR LEARNING COMPOSABLE ABSTRACTIONS
Jack Brady Julius von Kügelgen
Sébastien Lachapelle
Simon Buchholz Thomas Kipfté
Wieland Brendelt1.2.6
Max Planck Institute for Intelligent Systems, Tübingen
Tübingen Al Center 3 ETH Zürich 'ELLIS Institute, Tübingen
Samsung- SAIT AI Lab, Montreal Google DeepMind
ABSTRACT
Learning disentangled representations of concepts and re-composing them unseen ways is crucial for generalizing to out-of-domain situations. However, the underlying properties of concepts that enable such disentanglement and composi- tional generalization remain poorly understood. In this work, we propose the prin- ciple of interaction asymmetry which states: "Parts of the same concept have more complex interactions than parts of different concepts". We formalize this via block diagonality conditions on the (n+1) order derivatives of the generator mapping concepts to observed data, where different orders of "complexity" correspond to different n. Using this formalism, we prove that interaction asymmetry enables both disentanglement and compositional generalization. Our results unify recent theoretical results for learning concepts of objects, which we show are recovered as special cases with n or 1. We provide results for up to n = 2, thus extend- ing these prior works to more flexible generator functions, and conjecture that the same proof strategies generalize to larger n. Practically, our theory suggests that, to disentangle concepts, an autoencoder should penalize its latent capacity and the interactions between concepts during decoding. We propose an implementation of these criteria using flexible Transformer-based VAE, with novel regularizer on the attention weights of the decoder. On synthetic image datasets consisting of objects, we provide evidence that this model can achieve comparable object disentanglement to existing models that use more explicit object-centric priors.
INTRODUCTION
A core feature of human cognition is the ability to abstract conceptual knowledge to generalize far beyond direct experience (Behrens et al., 2018; Mitchell, 2021; Murphy, 2004; Tenenbaum et al., 2011). For example, by applying abstract knowledge of the concept "chair", we can easily infer how to use "chair on a beach", even if we have not yet observed this combination of concepts. This feat is non-trivial and requires solving two key problems. Firstly, one must acquire abstract, internal model of different concepts in the world. This implies learning a separate internal representation of each concept from sensory observations. Secondly, these representations must remain valid when observations consist of novel compositions concepts, e.g., "chair" and "beach". In machine learn- ing, these two problems are commonly referred to as learning disentangled representations (Bengio al., 2013; Higgins et al., 2018; Schölkopf et al., 2021) and compositional generalization (Fodor and Pylyshyn, 1988; Goyal and Bengio, 2022; Greff et al., 2020; Lake et al., 2017).
Both problems are known to be challenging due the issue of non-identifiability (Hyvärinen al., 2023). Namely, many models can explain the same data equally well, but only some will learn representations of concepts which are disentangled and generalize compositionally. To guarantee identifiability with respect to (w.r.t) these criteria, it is necessary to incorporate suitable inductive biases into model (Hyvärinen and Pajunen, 1999; Lachapelle et al., 2023; Locatello et al., 2019). These inductive biases, in turn, must reflect some underlying properties of the concepts which give rise to observed data. This raises a fundamental question: What properties of concepts enable learning models which provably achieve disentanglement and compositional generalization?
*Correspondence to: jack. www.bbaaaayuu... www.eeppppp...... mpg. de. "Jint senior author.
Code available ggthhb.wom/aathatay//cccctooaano

---

Published as a conference paper at ICLR 2025
Slot
f
D
D f
2B1
at
Interaction Asymmetry
(1+1gp order interaction slots At most nt order interaction aorass alots
X
Figure 1: Illustratoon of Interaction Asymmetry. (Left) Observations result from generator f applied to latent slots 28, that represent separate concepts. As indicated by the reflection of the cylinder upon the cube, slots interact during generation. Our key assumption, interaction asymmetry, states that these interactions across slots must be less complex than interactions within the same slot. (Right) This is formalized by assuming block-diagonality across but not within slots for the (1+h)" order derivatives of the generator, i.e., D
Many works aim to answer this question by studying properties enabling either disentanglement or compositional generalization in isolation. This is insufficient, however, as disentanglement alone does not imply compositional generalization (Montero et al., 2022a; 2021; Schott et al., 2022), while compositional generalization requires first disentangling the concepts be composed. Only a few studies investigate propertics enabling both disentanglement and compositional generaliza- tion (Brady al., 2023; Lachapelle et al., 2023; Wiedemer et al., 2024a). Yet. the properties pro- posed in these works are rather restrictive and specific to objects in simple visual scenes. There growing evidence, however, that the principles humans use to learn conceptual knowledge are not concept-specific, but shared across different concepts (objects, attributes, events, etc.) (Behrens et al., 2018; Constantinescu et al., 60 Hawkins al., 2018). This suggests there exist some gen- eral properties of concepts which enable both disentanglement and compositional generalization.
In this work, we seek to formulate such general property for disentangling and composing con- cepts. We begin by aiming to deduce. from first principles, properties which are fundamental to concepts ($ 3). From this, we arrive at the guiding principle of interaction asymmetry (Principle 3.1) stating: "Parts of the same concept have more complex interactions than parts of different concepts". As illustrated in Fig. 1 (left), we define concepts as distinct groups, or slots, of latent variables which generate the observed data 2). Interaction asymmetry is then formalized as a block-diagonality condition across but not within slots 1 f. the tensor (n + 1)th order partial derivatives of the generator function (Asm. 3.5), where determines the complexity of interactions, Fig. 1 (right).
Theory. Using this formulation, we prove that interaction asymmetry dually enables both disentan- glement (Thm, 4.3) and compositional generalization (Thm. 4.4). We also show that our formalism provides a unifying framework for prior results of Brady et al. (2023) and Lachapelle et al. (2023), by proving that the properties studied in these works for visual objects are special cases of our assump- tions for n 0 and respectively. We provide results for up to n=2, thereby extending these prior works to more general function classes, and conjecture that our results generalize to arbitrary n 2 O.
Method. Our theory suggests that to disentangle concepts, model should (i) enforce invertibility, without using more latent dimensions than necessary, and (ii) penalize interactions across slots dur- ing decoding. To translate these insights into practical method, we leverage VAE loss (Kingma and Welling, 2014) for (i), and observe that the Transformer architecture (Vaswani et al., 2017) of- fers approximate means to achieve (ii) since interactions determined by the attention weights of the model. To this end, we introduce an inexpensive interaction regularizer for a cross- cross-attention mechanism, which we incorporate, with the VAE loss, into flexible Transformer based model ($ 5).
Empirical Results. We test this model's ability to disentangle concepts visual objects on a Sprites dataset (Watters et al., 2019a) and on CLEVR6 (Johnson et al., 2017). We find that the model re- liably learns disentangled representations of objects, improving performance over an unregularized Transformer ($ 6). Furthermore, we provide preliminary evidence that our regularized Transformer can achieve comparable performance to models with more explicit object-centric priors such as Slot Attention (Locatello et al., 2020b) and Spatial Broadcast Decoders (Watters et al., 2019b).
Notation. We write scalars in lowercase (z), vectors in lowercase bold (z), and matrices in capital bold (M). [K] stands for {1, 2.,, K). D; and D stand for the first- and second-order partial derivatives with respect to (w.r.t.) z; and (z,, respectively. If B C [r] and 2 E R", zn denotes the subvector (z))ee B indexed by B. A function is C if it is n-times continuously differentiable.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND
We begin with formalizing the core ideas of concepts, disentanglement, and compositional gen- eralization, mostly following the setup of Lachapelle et al. (2023). To begin, we assume that the observed data I X C dR results from applying a diffeomorphic generator f Z + X'' to latent vectors E Z == Rd= sampled from some distribution Pz. Concepts underlying (objects, attributes, events, etc.) are then modelled as K disjoint groups or slots of latents 2B, such that , 2xx ), where B& S [dz]. We assume that P= is only supported on a subset 2upp S Z which gives rise to observed data Kuupp = (uuup)) This generative process can be summarized as:
x = fz)),
Px
supp(p=) zupp
(2.1)
Next, consider a model f 2 Rd= trained to be invertible from Xsup to 2upp := f- whose inverse f maps to a representation & E Zaupp S Z. This model is said to learn a disentangled representation of = E Zuupp if each model slot ÆB, captures exactly one concept =B,
Definition 2.1 (Disentanglement). Let f 2 X be a diffeomorphism and 2 S Z. A model f disentangles on Z w.r.t f if there exist a permutation ι of [K] and slot-wise diffeomorphism (hi,..., hg) with 0g IR Bxaal Bel and [ [ such that for all z E Z:
f (h2 (ZBxa)) .....,............. hK (zBeux))
= f()).
(2.2)
In other words, a representation is disentangled if the model inverts the generator up to permutation and reparametrization of the slots. For compositional generalization, we would like this to hold not only on supp but also for arbitrary combinations of the slots therein. Namely, also on the set
Zcre = 2, x 2g x .... x 2k:
with
2 = (zB, Zuupp)
(2.3)
where 2x denote the marginal supports of ps and ZCpE the Cartesian-produce extension (Lachapelle et al., 2023) of zup- In general, zupp is a subset of ZcrE- Thus, to generalize compositionally, a model must also achieve disentanglement "out-of-domain" on novel compositions of slots in ZCPE.
Definition 2.2 (Compositional Generalization). Let f: : Z + X be a diffcomorphism. A model f that disentangles on Suup w.r.t. f (Defn. 2.1) generalizes compositionally if it also disentangles on ECPE w.r.t. f.
On the Necessity of Inductive Biases. It is well known that only small subset of invertible models achieve disentanglement on Zupp (Hyvärinen and Pajumen, 1999; Locatello et al., 2019) or generalize compositionally to ΣcpE (Lachapelle et al., 2023). To provably achieve these goals (without explicit supervision), we thus need to further restrict the space of permissible models, i.e., place additional assumptions on the generative process in Eq. (2.1). Such assumptions then translate into inductive biases on a model. To this end, the core challenge is formulating assumptions on pz or f which faithfully reflect properties of concepts, while sufficiently restricting the problem.
Assumptions on Pz. To guarantee disentanglement, several assumptions on Ps have been proposed, such as conditional independence of latents given an auxiliary variable (Hyvärinen et al., 2019; Khe- makhem et al., 2020); particular temporal (Hälvä and Hyvarinen, 2020; Hyvärinen and Morioka, 2016; 2017; Klindt et al., 2021), spatial (Hälvä et al., 2021; 2024), or other latent structures (Kivva et al., 2022; Kori al., 2024); multiple views (Ahuja et al., 2022; Brehmer et al., 2022; Gresele et al., 2020; Locatello et al., 2020a; von Kügelgen et al., 2021; Yao et al., 2024; Zimmermann et al., 2021); or interventional information (Buchholz et al., 2023; Lachapelle et al., 2022: 2024; Lippe et al., 2022: 2023; Varici et al., 2024; von Kügelgen et al., 2023). While sufficient for disentanglement, such assumptions do not guarantee compositional generalization. The latter requires that the behav- ior of the generator on ZCPE can be determined solely from its behavior on Zupp (see Defn, 2.2). In the most extreme case, where the values of each slot 2B, are seen only once, Zeupp will be a one-dimensional manifold embedded in Z, while Zere is always d_ d.-dimensional. This highlights that generalizing from Zxupp to ZCpE is only possible if the form of the generator ff is restricted.
Assumptions on f. Restrictions on f which enable compositional generalization have been pro- posed by Dong and Ma (2022); Lippl and Stachenfeld (2024); Wiedemer et al. (2024b). Yet, these results rely on quite limited function classes and do not address disentanglement, assuming it to be solved a priori. Conversely, several works explore restrictions on Df such as orthogonality (Buch- holz et al., 2022; Gresele et al., 2021; Horan et al., 2021) sparsity (Leemann et al., 2023; Moran
3

---

Published as a conference paper at ICLR 2025
et al., 2022; Zheng and Zhang, 2023) which address disentanglement but not compositional general- ization. More recently, Brady et al. (2023) and Lachapelle et al. (2023) proposed assumptions on f which enable both disentanglement and compositional generalization (Wiedemer et al., 2024a). Yet, these assumptions are overly restrictive such that f can only model limited types of concepts, e.g., non-interacting objects, and not more general concepts. We discuss these two works further in $ 4.3.
3 THE INTERACTION ASYMMETRY PRINCIPLE
In this section, we attempt to formulate assumptions that enable disentanglement and compositional generalization, while capturing more general properties of concepts. To approach this, we take step back and try to understand what are the defining properties of concepts. Specifically, we consider the question: Why are some structures in the world recognized as different concepts (e.g., apple vs. dog) and others as part of the same concept? We propose an answer this for concepts grounded in sensory data, such as objects (c.g.."car"), events (c.g., "making coffee"), or attributes (c.g., "color").
Sensory-groundee concepts correspond reoccurring visual or temporal patterns that follow an abstract template. They tend to be modular, such that independently changing one concept generally leaves the structure of other concepts intact (Greff et al., 2015, $ 4.1.1; Peters al., 2017). For example, a car can change position without affecting the structure of the street, buildings, or people around it. Thus, different concepts appear, in some sense, not interact.
On the other hand, parts of the same concept do not seem to possess this modularity. Namely, arbitrarily changing one part of a concept without adjusting other parts is generally not possible without destroying its inherent structure. For example, it not possible to change the position of the front half of a car, while maintaining something we would still consider car, without also changing the back half's position. Thus, parts of the same concept seem to interact.
This may then lead us to answer our initial question with: Parts of the same concept interact, while different concepts do not. However, this is an oversimplified view, as parts of different concepts can, in fact, interact. For example, in Fig. I we see the purple cylinder reflects upon and thus interacts with the golden cube. However, such interactions across concepts appear somehow simpler than interactions within a concept: whereas the latter can alter the concept's 's structure, the former gener- ally will not. In other words, the complexity of interaction within and across concepts appears to be asymmetric. We formulate this as the following principle (see Appx. G.1 for related principles).
Principle 3.1 (Interaction Asymmetry). Parts of the same concept have more complex interactions than parts of different concepts.
To investigate the implications of Principle 3.1 for disentanglement and compositional generaliza- tion, we must first give it precise formalization. To this end, we need a mathematical definition of the "complexity of interaction" between parts of concepts, i.e., groups of latents from the same or different slots. This can be formalized either through assumptions on the latent distribution p= or on the generator f. Since the latter are essential for compositional generalization, this is our focus.
Let us start by imagining what it would mean if two groups of latent components N and øg interact with no complexity, i.c., have no interaction within f. A natural way to formalize this is that A and zB affect distinct output components fi. Mathematically, this is captured as follows.
Definition 3.2 (At most 0thh order/No interaction). Let f Z + X be C²,, and let A, C [d.] be non-empty. ZA and ž have no interaction within of if for all z E 2, and all E A, E B:
D.f(z) 3 Dff(z) 0.
(3.1)
To define the next order of interaction complexity, we assume that n and z do interact, i.e., they affect the same output ft such that D ((x) and fz)((z) are non-zero for some i E A. j E B. This interaction, however, should have the lowest possible complexity. A natural way to capture this is to say that z; can affect the same output fi as zj but cannot affect the way in which fi depends on Zj- Since the latter is captured by D₁ this amounts to a question about the 2nd order derivative ft. We thus arrive at the following definition for the next order of interaction complexity.
Definition 3.3 (At most 1st order interaction). Let f : 2 > be C², and let A, B C [d.] be non- empty. z and zn have at most 1d order interaction within f if for all z E 2, and all i E A,j E B:
D2 f() = 0.
(3.2)
4

---

Published as a conference paper at ICLR 2025
Using the same line of reasoning, can continue to define interactions at increasing orders of complexity. For example, for at most 2nd order interaction, 24 can affect the derivative DjJi, such that Df fi()) + O, but cannot affect the way which Djfe depends on any other 2k: i.e., D ((()) = 0. This leads to general definition of interactions with at most n order complexity.
Definition 3.4 (At most nhth order interaction). Let n 2 1 be integer. Let f 2 X be C Let A, B S [d=] be non-empty. 2 and zB have at most n order interaction within f if for all = E 2, all i E A,j E B, and all multi-indices a E No. with |a|| = =n and S 03; Ot?:
D f(z)=
(3.3)
In other words, AA and ZB have at most n order interaction within f if all higher-than-n" th order cross partial derivatives w.r.t at least one component of 2, and of z are zero everywhere. Other- wise, if the statement in Defn. 3.4 does not hold for some 2 E Z, i E A, and j E B, we say that A (and similarly for 1st order interaction if Defn. 3.2 does
and z have (n+1) order interaction not hold). With these definitions, we can now provide a precise formalization of Principle 3.1.
Assumption 3.5 (Interaction asymmetry (formal)). There exists n e No such that (i) any two distinct slots BB, and BB, have at most n² order interaction within f; and (ii) for all z E Z, all slots 2B, and all non-empty A, B with By AUB, A and z B have (n+1) order interaction within f at z.
We emphasize that Asm. 3.5 (ii) does not state that all subsets of latents within a slot must have (n+11 order interaction, but only that slot cannot be split into two parts with most nh order interaction, see Fig. 1 (right). We also note that condition (ii) must hold uniformly over 2, which resembles the notion of "uniform statistical dependence" among latents introduced by Hyvärinen and Morioka (2017, Defn. 1). For further discussions of Asm. 3.5, see Appx. H.1.
THEORETICAL RESULTS
We now explore the theoretical implications of Asm. 3.5 for disentanglement on Supp and composi- tional generalization to ΣCPE- We provide our results for up to at most 2nd order interaction across slots. All results-i.e., at most 0th (no interaction), at most 1st, and at most 2nd order interaction use a unified proof strategy. Thus, we conjecture this strategy can also used to obtain results for n 2 3. This, however, would require taking (n + 1) 2 4 derivatives of compositions of multivariate func- tions, which becomes very tedious as n grows. General v order results are thus left for future work.
4.1 DISENTANGLEMENT
We start by proving disentanglement on Zuup for which we will need two additional assumptions.
Interactions. First, one issue we must address is that our formalization of interac- tion asymmetry (Asm. 3.5) not basis invariant. Specifically, it is possible that all splits of a slot BR have (n+1) order interactions while for M with M & slot-wise change of basis matrix, they have at most th order interactions. Since M2 need not affect interactions across slots, interac- tion asymmetry may no longer hold in the new basis. This makes it ambiguous whether interaction asymmetry truly satisfied, as B, and M M contain the same information. To address this, we assume interaction asymmetry holds for all slot-wise basis changes, or equivalent generators.
Definition 4.1 (Equivalent Generators). A function f Rdd - Rd is said to be equivalent to a generator f for all & E [K] there exists an invertible matrix M2 E RIBBIx|BA Bal such that
Vz E IRd=
J(M M MK f(zB, ZBx))
(4.1)
Sufficient Independence. We require one additional assumption on f which we call sufficient in- dependence. This assumption amounts to a linear independence condition blocks of higher-order derivatives of f. Its main purpose is to remove redundancy in the derivatives of f across slots, which can be interpreted as further constraining the interaction across slots during generation. In the case
A multi-index is ordered tuple (a1, (11,022 a',,... ...: (ta) of non-negative integers E No, with operations 0n 02
|a|] = 0n + ag + .... + ass za == 21 22 ... and D =
802 aod see Appx. B for details.
0z
5

---

Published as a conference paper at ICLR 2025
of n =0 (i.e., no interaction across slots), sufficient independence reduces to linear independence between slot-wise Jacobians of f (Defn. A.8). This is satisfied automatically since f a diffeomor- phism. When n2>0, we require an analogous linear independence condition on higher order deriva- tives of f. Below, we present this for the case n n= 2, while for n n= it is presented in Defn. A.9.
Definition 4.2 (Sufficient Independence (2nd Order)). A C function f : IRd. + Rd= with at most 2nd order interactions across slots is said to have sufficiently independent derivatives if V/2 E R
[Io.((t)]ee WR (=)leeB. [D (())((a.))))
rank
kE(K]
Σ RE|K]
raank ([ID.f(z))ken, (e)leB(e)))) + rank (Di (E))eee
With Defns. 4.1 and 4.2, can now state our theoretical results; see Appx. A for complete proofs.
Theorem 4.3 (Disentanglement on Zsup)). Let T E 2,). Let f Z X be a C dif- feomorphism satisfying interaction asymmetry (Asm, 3.5)sor all equivalent generators (Defn. 4.1) and sufficient independence (Appx. A.2). Let Zwup be regular closed (Defn. A.3), path-connected (Defn. A.14) and aligned-connected (Defn. A.16). A model f :2 + Rdd disentangles z on Zaup w.r.t. f (Defn. 2. if it is (i) a Cn+11 diffeomorphism between Zaup and Xsupp with (ii) at most n" order interactions across slots (Defn. 3.4) on Zupp-
Intuition. Assume for a contradiction that := f-1 o f entangles a ground-truth slot ZB, i.e., ((z) has multiple non-zero blocks. Because f and ff are invertible, must encode all of z,, in 2 : h(z). Further, because f satisfies interaction asymmetry, ZB, cannot be split into two parts with less than (n+1) order interaction. Taken together, this implies that if h entangles BB,, then there exist parts ZA and žB of ZB, with (n+1) order interaction, encoded in different model slots. Since the model ff is constrained to have at most nh order interactions across slots, it cannot capture this interaction. Thus, the only way that f can satisfy (i) and (ii) without achieving disentanglement if reparametrizingg z via h removed the interaction between A and zp. This situation is prevented by assuming sufficient independence and that Asm. 3.5 holds for all equivalent generators.
Conditions on Zsupp. The regular closed condition on Zuup in Thm. 4.3 ensures that equality be- tween two functions Zupp implies equality of their derivatives, while the path math-connectedness con- dition prevents the one-to-one correspondence hetween the slots of and those of á from changing across different (Lachapelle et al., 2023). The aligned--onnectednes condition is novel and allows one to take integrals go from local to global disentanglement (see Appx. A.3 for more details).
4.2 COMPOSITIONAL GENERALIZATION
We now show how Asm. 3.5 also enables learning a model that generalizes composition- ally (Defn. 2.2), i.e., that equality of f and f o h on supp also implies their equality on ZcpE- As discussed in $ 2. such generalization is non-trivial and requires specific restrictions on a function class. A key restriction imposed by interaction asymmetry is that interactions across slots are limited to at most th order. In Thm. 4.3, this prevents f h from modelling interactions between parts of the same fround-truthh slot in different model slots. We now aim to show that limiting the interac- tions across slots serves the dual role of making f and ff o "predictable", such that their behavior on ECPE can be determined from Ssupp- To do this, we will require characterization of the form of functions with at most th order interactions across slots, which we prove in Thm. C.2 to be:
f(z) = ΣK ph (zn.) + Easaa/sa a
(4.2)
where Ca Rdd In the first sum, slots are processed separately by functions f while in the second, they can interact more explicitly via polynomial functions of components from different slots, with degree determined by the order of interaction, n. With this, we can now state our result.
Theorem 4.4 (Compositional Generalization), Let n E (0, Let Zuup be regular closed (Defn. A.3). Let f Z X and f Z Rd be Ca diffeomorphisms with at most nhth order interactions across slots on Z. If f disentangles z on Zsupp w.r.t. f (Defn. 2.1), then it generalizes compositionally (Defi. 2.2).

---

Published as a conference paper at ICLR 2025
0f
Intuition. Consider the red dotted line in Fig. 2 (left) corresponding to {z E IR2 21 = ).). To generalize compositionally, the behavior of the Aft partial derivative de (2) on this line must be predictable from the behavior of f on Zupp. and similarly for f o h. Because f and, as we show, I h have at most Th order interactions across slots on Rd=, the form of this derivative constrained be a fixed-degree polynomial, see Eq. (4.2) and Fig. 2 (right). Thus, its global behavior on the dotted line in Rd. can be determined from its derivative locally region in zupp Applying this reasoning to all such line segments in- tersecting zupp- we can show that the behavior of f and foh on ΣcpR can be determined from Smpp-
2
(4) R2
0z
simas Ohils
ZCPF
Zz
Eupp
3 21
Figure 2: See intuition for Theorem 4.4.
4.3 UNIFYING AND EXTENDING PRIOR RESULTS
We now show that our theory also recovers the results of Brady al. (2023) and Lachapelle et al. (2023) as special cases for n = 0 and n 1, and extends them to more flexible generative processes.
At most 0th Order Interaction. Brady al. (2023) proposed two properties on f. which en- able disentanglement and compositional generalization (Wiedemer et al., 2024b): compositionality (Defn. 1) and irreducibility (Defn. E.2). Compositionality states that different slots affect distinct output components such that (z(=) has a block- structure. This is equivalent to f having at most 6th order interaction across slots (Defn. 3.2). Irreducibility is rank condition on DB, f(2 which Brady et al. (2023) interpreted as parts of the same object sharing information. In Thm. E.3, we show that irreducibility equivalent to f having 1st order interaction within slots for all equivalent generators. Thus, the assumptions in Brady et al. (2023) are equivalent to interaction asymmetry for all equivalent generators when n Further, proof strategy, unified with proofs for at most 1st / 2 order interaction across slots (Thm. A.20).
recover their disentanglement result using a novel
At most 1d Order Interaction. Lachapelle al. (2023) also proposed two assumptions on f for disentanglement and compositional generalization: additivity (Defn. E.4) and sufficient non- linearity (Defn. E.5). Additivity is equivalent to fi having a block-diagonal Hessian for all I E [dz] (Lachapelle et al., 2023). This is the same as f having at most 1st order interaction across slots (Defn. 3.4). Sufficient nonlinearity is a linear independence condition on columns of 1st and 2nd derivatives of f. In Thm. E.6, we show that sufficient nonlinearity implies that f satisfies sufficient independence for ther, we conjecture that the reverse implication does not hold. Thus, the assumptions of Lachapelle al. (2023) imply, and are conjectured be stronger than, our assumptions when n 1. We also recover their same disentanglement result using a unified proof strategy (Thm. A.22).
and has 2nd order interaction within slots for all equivalent generators. Fur-
Allowing More Complex Interactions. Our theory not only unifies but also extends these prior results to more general function classes. This is clear from considering the form of functions with at most th order interactions across slots in Eq. (4.2). For at most 0th (Brady et al., 2023) 1st order interactions (Lachapelle et al., 2023), the sum of polynomials on the RHS of (4.2) vanishes. Conse- quently, I reduces an additive function. Such generators can only model concepts with trivial in- teractions such as non-occluding objects. In contrast, we are able to go beyond additive interactions via the polynomial terms in (4.2). This formally corroborates the "generality" of interaction asym- metry, that it enables more flexible generative processes where concepts can explicitly interact.
5 METHOD:
TRANSFORMER VAE
We now explore how our theoretical results in Š can inform the design of a practical estimation
method. Our theory puts forth two key properties that a model should satisfy: (i) invertibility and (ii) limited interactions across slots of at most th order. To achieve disentanglement on Zupp- (i) and (ii)
must hold only "in-domain" on Soupp and Kaup (Thm. 4.3), while for compositional generalization. they must also hold out-of-domain, for achieving (i) and (ii) in-domain. Achieving them out-of-domain requires addressing separate practical challenges, which are out of the scope of this work. We discuss this in detail in Appx. H.3.
all of 2 and X' (Thm. 4.4). We will focus on approaches
7

---

Published as a conference paper at ICLR 2025
On Scalability, Approaches that enforce (i) and (ii) exactly will generally only be computationally tractable in low-dimensional settings. Such computational issues are typical when translating a disentanglement result into an empirical method, often resulting in methods which directly adhere to theory but cannot scale beyond toy data (e.g.. Brady et al., 2023; Gresele et al., 2021). Our core motivation, however, is learning representations of concepts underlying high-dimensional sensory data, such as images. Thus, to formulate a method which scales to such settings, we do not restrict ourselves to approaches which exactly enforce (i) and (ii) and also explore approximate approaches.
(i) Invertibility, Our theory requires invertibility between Xaupp S R and Zuup S 2 = R For most settings of interest, the observed dimension dy exceeds the ground-truth latent dimension dN- Thus, we generally cannot use models which are invertible by construction such as normalizing flows (Papamakarios et al., 2021). An alternative is to use an autoencoder in which f-1 and f are parameterized separately by an encoder 9 Rd - Rd and decoder f Rdd - Rd, which are trained to invert each other (on Zoupp and Kappp) by minimizing reconstruction loss Lre := Bli f((())) Minimizing fies alone, however, does not suffice unless the inferred latent dimension dN equals the ground-truth d=. Yet, in practice d2 is unknown. Moreover, choosing de > d2 is important for scalability (Sajjadi et al., 2022a). viable alternative is thus to employ a soft constraint where d= > dz. but the model is encouraged to encode x using minimal latent dimensions. To achieve this, we leverage the well known VAE loss (Kingma and Welling, 2014), which couples fise with a KL-divergence loss Lgi. between a factorized posterior ((* (z) and prior distribution p(²), i.e., Cxt= := Lig(a,] [dg] xx (q($(|=)]lp((2)) This loss encourages each & to be insensitive to changes in æ such that unnecessary dimensions should contain no information about (Rolinek et al., 2019).
(ii) At Most thth Order Interactions. One approach to enforce at most nhth order interactions across slots would be to parameterize the decoder f to match the form of such functions (see Thm. C.2) for some fixed n. However, this can result in an overly restrictive inductive bias and limit scalability. Moreover, n is generally unknown. Thus, a more promising approach is to regularize interactions to be minimal. Doing this naively though using gradient descent would require computing gradients of high-order derivatives, which is intractable beyond toy data. This leads to the question: Is there a scalable architecture which permits efficient regularization of the interactions across slots?
Transformers for Interaction Regularization. We make the observation that the Transformer architecture ( (Vaswani et al., 2017) provides an efficient means to approximately regularize interac- tions. In a Transformer, slots are only permitted to interact via an attention mechanism. We will focus on a cross-attention mechanism, which maps a latent vector & to output 2n (e.g.. pixel) via:
K W [2B,
EBKL:
V [
εBch
Q W ...b., (5.1)
exp (Q,,
A EiE[K] exp K.,)'
3 A. VT.
i1 = (.((
(5.2)
In Eq. (5.1), all slots are assumed to have equal size, and key K., and value V., vectors are com- puted for each slot le E [K] Query vectors are computed for output dimensions l E [dz] (e.g., pixel coordinates) and each I is assigned fixed vector o In Eq. (5.2), queries and keys are used to com- pute attention weights A These weights determine the slots pixel "attends" to when generating pixel token Rt, which is mapped to a pixel it by nonlinear function v/; see Appx. F for further details.
Within cross-attention, interactions across slots occur if the query vector for a pixel I attends to multiple slots, i.e., if A,, is non-zero for more than one k. Conversely, if At, is non-zero for only one k. then, intuitively, no interactions should occur. This intuition can be corroborated formally by computing the Jacobian of cross-attention w.r.t. each slot (see Appx. F.1). Thus, an approximate means to minimize interactions across slots is to regularize A towards having only one non-zero entry for each row A,... To this end, we propose to minimize the sum of all pairwise products Ar, AI, where j A (see Fig. 4). This quantity is non-negative and will only be zero when each row of A has exactly one non-zero entry. This resembles the compositional contrast of Brady et al. (2023), but computed on A, which can be efficiently optimized, as opposed to the Jacobian of f which is intractable to optimize. We refer to this regularizer as Linteracs see Eq. (F.9).
Model. Combining these different objectives leads us to the following weighted tiree-part-loss.
Laisen (f. 9. x) = Lree + OeLimtencct + BLLL,
(5.3)
8

---

Published as conference paper at ICLR 2025
Original
B
Sparial
Transforer Decoder regulaaiizei
Recorstruction
Redonatr uction
(nregulartred)
Transformer Decoder (eegularizd))
Figure 3: (A) Sprites Normalized slot-wise Jacobians for an unregularized (ax 0, = 0) and a regularized (a > 0, > 0) Transformer and a Spatial Broadcast Decoder (SBD). The unregularized model encodes objects across multiple slots, while the regularized model matches the disentanglement of the SBD. (B) CLEVR6 Slot- wise Jacobians for regularized Transformer and SBD objects CLEVR6 which interact via reflections. in reconstructions and Jacobians, the regularized Transformer models reflections, while mostly removing unnecessary interactions, while the SBD fails model reflections due restricted architecture.
As
We apply this loss to a flexible Transformer-basee autoencoder, similar to the models of Jabri et al. (2023); Jaegle et al. (2022); Sajjadi et al. (2022b). For the encoder ÿ. we first map data æ to features using the CNN Locatello al. (2020b). These features are processed by a Transformer, which has both self- and cross-attention at every layer. yielding a representation λ. Our decoder f then maps to output & using a cross-attention Transformer regularized with Linteract, see Appx. J for details.
Relationship to Models In Object-Centric Learning. Existing models for learning disentangled representations of concepts, particularly for disentangling objects without supervision, typically rely on architectural priors rather than regularization (Greff al., 2019; Locatello et al., 2020b; Seitzer et al., 2023; Singh et al., 2022a). While such priors promote disentanglement, they are often too restrictive. For example, Spatial Broadcast Decoders (Watters et al., 2019b) decode slots separately and only allow for weak interaction through a softmax function, which prevents modelling real- world data where objects exhibit more complex interactions (Singh al., 2022a). While have shown success in disentangling objects using more powerful Transformer decoders (Sajjadi al., 2022a; Singh et al., 2022a;b), they rely encoders that Slot Attention (Locatello al., 2020b) as an architectural component, which differs from current large-scale models, typically based on Transformers (Anil et al., 2023). In contrast, explore the flexible approach of starting with a very general Transformer-baat model and regularizing it towards a more constrained model.
works
EXPERIMENTS
We now apply our attention-reguulariedd Transformer- VAE 5) for learning representations of con- cepts. Since this model is designed to enforce the criteria outlined in Thm. 4.3 for disentanglement opposed compositional generalization. To
on suup- we focus evaluating disentanglement, this end, we focus on disentangling objects visual scenes, and leave an empirical study of a wider range of concepts (e.g., attributes, object-parts, events) for future work (see Appx. for details).
Data. We consider two multi-objeet datasets in our experiments. The first, which we refer to as Sprites (Brady et al., 2023; Watters et al., 2019a; Wiedemer et al., 2024b), consist of images with 2- objects set against black background. The second is the dataset (Johnson et al., 2017), consisting of images with 2-6 objects. In Sprites, objects do not have reflections and rarely occlude such that slots have essentially have no interaction. In CLEVR6, however, objects can cast shadows and reflect upon each other (see Fig. for an example), introducing more complex interactions.
Metries. common metric for object disentanglement is the Adjusted-Ran Index (ARI; Hubert and Arabie, 1985). The ARI measures the similarity between the set of pixels encoded by model slot, and the of ground-truth pixels for given object in scene, yielding optimal score if each slot corresponds to exactly one object. To assign pixel to unique model slot, prior works typically choose the slot with the largest attention score (from, e.g., Slot Attention) for that pixel (Seitzer et al., 2023). However, using attention scores can make model comparisons challenging and is also somewhat unprincipled (see Appx. J.2). We thus consider alternative and compute the ARI using the Jacobian decoder (J-ARI). Specifically, we assign a pixel to the slot with the largest L norm for the slot-wise Jacobian D DB, f(()) (see Fig. 3 for a visualization of these Jacobians).
While J-ARI indicates which slots most responsible for encoding each object, it does not indicate if additional slots affect the same object, i.e., 1,D f(()))l= * for more than one k. To measure

---

Published as a conference paper at ICLR 2025
Table I: Empirical Results. We show the mean + std. dev. for J-ARI and JIS (in %) over 3 seeds for different
choices of encoders and decoders and weights of loss terms Eq. (5.3) on Sprites and CLEVR6.
Model
Sprites
CLEVR6
Encoder
Decoder
Loss
J-ARI (?)
JIS (1)
J-ARI (1)
JIS (t)
Slot Attention
Spatial Broadcast
a = 0,8 =
89.3 1.5
91.4 + 0.8
97.0 0.2
95.3
0.7
Slot Attention
Transformer
0,3
90.1 + 1.4
73.6
95.5
63.1
1.0
Transformer
Transformer Transformer
= 0, = -0
80.5 11.5+:4.4 4.1
57.0
8.0
92.7 + 3.3
54.8 + 3.5
Transformer
a= 0.05,
82.8 4.
73.8 4.0
79.2 12.8
51.6 5.9
Transformer Transformer
0, = 0.05
9.0
92.8 0.9
96.6
+.3
80.3
0.4
Transformer
Transformer
0.05, = 0.05 (Ours)
03.7
9.7
96.5 0.4
83.8 2 1.1
this, we also introduce the Jacobian Interaction Score (JIS). JIS is computed by taking the maximum of [D,, f(())] [1 across slots after normalization, averaged over all pixels. If each pixel affected by only one slot, JIS is 1. For datasets where objects essentially do not interact like Sprites, JIS should be close to 1, whereas for CLEVR6, it should be as high as possible while maintaining invertibility.
6.1 RESULTS
Laisent Enables Object Disentanglement. In Tab. 1, we compare the J-ARI and JIS of regu- larized Transformer- model (a > 0, >> 0) trained with Laisen (Eq. (5.3)) to the same model trained without regularization (a 0, B == 0), i.c., with only Liee. On Sprites, the regularized model achieves notably higher scores for both J-ARI and JIS. This is corroborated by visualizing the slot- wise Jacobians in Fig. 3A, where we see the regularized model cleanly disentangles objects, whereas the unregularized model often encodes objects across multiple slots. Similarly, on CLEVR6, the reg- ularized model achieves superior disentanglement, as indicated by the higher values for both metries.
Comparison to Existing Object-Centric Autoencoders. In Tab. we also compare our model to existing models using encoders with Slot Attention and Spatial Broadcast Decoders (SBDs). On Sprites, our model achieves higher J-ARI and JIS than these models, despite using weaker architectural prior. On CLEVR6. our model outperforms Slot Attention with a Transformer decoder in terms of J-ART and JIS. Models using a SBD, however, achieve higher and nearly perfect JIS, i.., the learned slots essentially never affect the same pixel. In Fig 3B, we see this comes at the cost of SBDs failing model reflections between objects, while our model captures this interaction. This highlights that regularizing a flexible architecture with Lassent can enable a better balance between restricting interactions and model expressivity.
Ablation Over Losses. Lastly, in Tab. 1, we ablate the impact of the regularizers in Ldisent- Training without LxL. (a > 0, 3=0) 0) can in some cases give improvements in J-ARI and JIS over an unregularized model (a- 0, 3 0). However, across datasets this loss yields worse disentanglement than Ldisent (ar > 0, 3 > 0). This highlights that penalizing latent capacity via Lxt. is important for object disentanglement. Training without Limerac (a = 0, 3 > 0) generally yields a drop across both metrics compared to Ldisent, though on CLEVR6 this loss achieves comparable J-ARI. We found that training with LX can, some cases, implicitly minimize Limence explaining this result (Fig. 5).
More Complex Data. Tab. 2 in Appx. I presents additional results for the visually complex CLEVR- Tex dataset (Karazija al., 2021). For these experiments, we follow Seitzer al. (2023) and recon- struct image representations based on a pre-trained encoder rather than the original images. We find model to achieve superior J-ARI compared to unregularized Transformer and a Slot Attention baseline, but slot-wise MLP decoders yield higher JIS. For further details, see Appx. I.
7 CONCLUSION
In this work, we proposed interaction asymmetry as a general principle for learning disentangled and composable representations. Formalizing this idea led to constraint on the partial derivatives of the generator function, which unifies assumptions from prior efforts and extends their results to a more flexible class of generators that allow for non-trivial interactions. These theoretical insights inspired the development of flexible estimation method based on the Transformer architecture with a novel cross-attention regularizer, which can be efficiently implemented at scale, and which shows promising results on object-centric learning datasets. Future work should seek to further extend our theoretical results, address the empirical challenges for achieving compositional generalization, and test our method more large-scale data involving not only objects but also other types of concepts.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
JB would like to thank Nicoló Zottino for providing the initial idea (expressed in Fig. 2) that com- positional generalization could be extended to functions with at most n order interactions. The authors thank Luigi Gresele and the Robust Machine Learning Group for helpful discussions.
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen Al Center, FK7: 01 1039A, 011$18039B. WB acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. WB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. This work utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. Part of this work was done while JvK was affiliated with the Max Planck Institute for Intelligent Systems, Tübingen.
AUTHOR CONTRIBUTIONS
JB initiated and led the project. JB wrote the manuscript in close collaboration with JvK, and with feedback from all authors. JB conceived of the framework of interaction asymmetry and then formalized it together with SL and JvK. JB proved the disentanglement results in Appx. A in col- laboration with SL, who improved the clarity of several results and simplified some arguments. SL extended these results from local to global disentanglement in Appx. A.3. SB proved the character- ization of functions with at most 71" order interactions in Appx. C, based on an earlier weaker result for analytic functions by JvK. SB proved the compositional generalization results in Appx. discussions with JB. JB proved the results for unifying prior works in Appx. E. SB carried out the Jacobian computation in Appx. F.1. JB conceived of the regularizers and method in $ 5 and imple- mented and executed all experiments in $ 6 with advising from TK. WB created Fig. 1 and Fig. with insight from JB and feedback from JvK. SL created Fig. based on an outline provided by JB.
after
REFERENCES
K. Ahuja, J. S. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. In Advances in Neural Information Processing Systems, volume 35, pages 15516- 15528, 2022. [Cited on p. 3.]
K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In International Conference on Machine Learning, pages 372-407. PMLR, 2023. [Cited on p. 45.]
A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. arXiv preprint 1612. 00, 10, 2016. [Cited on p. 44.]
R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023. [Cited on p. 9.]
J. L. Ba. Layer normalization. arXiv preprint arXiv 2016. [Cited on p. 42.]
D. Baldwin, A. Andersson, J. Saffran, and M. Meyer. Segmenting dynamic human action via statis- (113)::182-14090 2008 [Cited on p. 44.]
tical structure. Cognition,
T. E. J. Behrens, T. H. Muller, J. C. R. Whittington. S. Mark, A. Baram, K. L. Stachenfeld, and Z. Kurth-Nelson. What is cognitive map? organizing knowledge for flexible behavior. Neuron, 100:490-509, 2018. [Cited on p. 1, 2, and 45.]
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3598) 2013. [Cited on p. 1.]
O. Biza, S. Van Steenkiste, M. S. Sajjadi, G. F. Elsayed, A. Mahendran, and T. Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. arXiv preprint arXiv:2302.04973, 2023. [Cited on p. 49 and 50.]
J. Brady, R. S. Zimmermann, Y. Sharma, B. Schölkopf, J. von Kügelgen, and W. Brendel. Provably learning object-centric representations. In International Conference on Machine Learning, pages PMLR, 2023. [Cited on p. 2, 4, 7, 8, 9, 40, 43, 44, and 47.]
11

---

Published as a conference paper at ICLR 2025
J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 38319-38331, 2022. [Cited on p. 3.]
T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instruc- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023. [Cited on 45.]
S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems, volume 35, pages 16946-16961, 2022. [Cited on p. 3.]
S. Buchholz, G. Rajendran, E. Rosenfeld, Aragam, B. Schölkopf, and P. Ravikumar, Learning linear causal representations from interventions under general nonlinear mixing, In Advances in Neural Information Processing Systems, 2023. [Cited on p. p.]
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVE international conference on computer vision, pages 9650-9660) 2021. [Cited on 49.1
M. Chang, T. Griffiths, and S. Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. Advances in Neural Information Processing Systems, 35:32694-32708, 2022. [Cited on p. 50.]
A. O. Constantinescu, J. X. O'Reilly, and T. E. Behrens. Organizing conceptual knowledge in humans with a gridlike code. Science, 1646629) 2016. [Cited on p. 2.]
A. Dittadi, S. S. Papa, M. De Vita, B. Schölkopf, O. Winther, and F. Locatello. Generalization and robustness implications in object-centric learning. In International Conference Machine Learning, pages 5221 -5285. PMLR, 2022. [Cited on p. 47.]
K. Dong and T. Ma. First steps toward understanding the extrapolation of nonlinear models to unseen domains. In International Conference on Learning Representations, 2022. [Cited on p. 3.]
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. [Cited on p. 49.]
B. Esmaeili, H. Wu, S. Jain, A. Bozkurt, N. Siddharth, B. Paige, D. Brooks, J. G. Dy, and J.-W. van de Meent. Structured disentangled representations. International Conference on Artificial Intelligence and Statistics, 2018. [Cited on p. 45.]
J. A. Fodor and Z. W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1) 1):3-71, 1988. [Cited on p. 1.]
S. Fortunato and D. Hric. Community detection in networks: A user guide. Physics reports, 659:
1-44, 2016. [Cited on p. 44.]
A. Gopalakrishnan, K. Irie, J. Schmidhuber, and S. van Steenkiste. Unsupervised learning of tempo- ral abstractions with slot-based transformers. Neural Computation, 35(4):593-626, 2023. [Cited p. 48.]
A. Goyal and Y. Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 478(2266) 2022. [Cited on p. 1.]
A. Goyal, A. Lamb, Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations, 2021. [Cited on p. 45.]
K. Greff, R. K. Srivastava, and J. Schmidhuber. Binding via reconstruction clustering. arXiv preprint arXiv: 1511.06418, 2015. [Cited on p. 4 and 44.]
K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. In International Conference Machine Learning, pages 2433. PMLR, 2019. [Cited p. 9.]
12

---

Published as a conference paper at ICLR 2025
K. Greff, S. Van Steenkiste, and J. Schmidhuber. On the binding problem in artificial neural net- works. arXiv preprint arXiv:2012.05220 202200000, 2020. [Cited on p. and 44.]
L. Gresele, P. Rubenstein, A. Mehrjou, Locatello, and B. Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ICA. In Uncertainty in Artificial Intelligence, pages 217-227. PMLR, 2020. [Cited on p. 3.]
I. Gresele, von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? In Advances in Neural Information Processing Systems, volume 34, pages 28233-2833 28248, 2021. [Cited on p. and 8.]
H. Hälvä and A. Hyvarinen. Hidden markov nonlinear ICA: Unsupervised learning from nonstation- ary time series. Conference on Uncertainty in Artificial Intelligence, pages 939-948. PMLR, 2020. [Cited on p. 3.]
H. Hälvä, S. Le Corff, L. Lehéricy, J. So, Y. Zhu, E. Gassiat, and A. Hyvarinen. Disentangling iden- tifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems, 34:1624-1633. 2021. [Cited on p. 3.]
H. Hälvä, J. So, R. E. Turner, and A. Hyvärinen. Identifiable feature learning for spatial data with nonlinear ICA. In International Conference Artificial Intelligence and Statistics, pages 3331- 3339. PMLR, 2024. [Cited on p. 3.]
Hawkins, M. Lewis, M. Klukas, S. Purdy, and Ahmad. A framework for intelligence and cortical function based on grid cells in the neocortex. Frontiers in Neural Circuits, 12, 2018. [Cited on p. 2.]
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-VAE. Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017. [Cited p. 45.]
I. Higgins, D. Amos, D. Pfau, S. Racanière, L. Matthey, D. J. Rezende, and A. Lerchner. Towards definition of disentangled representations. arXiv preprint arXiv: 10/200200,0 2018. [Cited on p. 1.]
S. Hochreiter and J. Schmidhuber. Feature extraction through lococode. Neural Computation, 11 (3):6797714, 1999. [Cited on p. 44.]
D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? In Advances in Neural Information Processing Systems, volume 34, 2021. [Cited on p. 3.]
L. J. Hubert and P. Arabie. Comparing partitions. Journal of Classification, 2: 2:199-218, 1985. [Cited on p. 9.J
A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems, pages 3765-3773, 2016. [Cited on p. 3.J
A. Hyvärinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR. 2017. [Cited on p. and 5.]
A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999. [Cited on p. 1 and 3.]
A. Hyvarinen and Perkio. Learning to segment any random vector. In IEEE International Joint Conference on Newral Network Proceedings, pages 4167-4172. IEFE, 2006. [Cited on 44.]
A. Hyvärinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statis- fics, pages 859-868. PMLR, 2019. [Cited on p. and 21.]
A. Hyvärinen, Khemakhem, and Morioka. Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10), 2023. [Cited on p. 1.]
A. Jabri, D. J. Fleet, and T. Chen. Scalable adaptive computation for iterative generation. In Inter- national Conference Machine Learning, pages 14569-14589. PMLR, 2023. [Cited on p. 9.]
13

---

Published as a conference paper at ICLR 2025
A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. loneseu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, al. Perceiver IO: A general architecture for structured inputs & outputs. In International Conference on Learning Representations, 2022. [Cited on p. 9 and 46.]
J. Jiang, F. Deng, G. Singh, and S. Ahn. Object-centric slot diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [Cited on 47.1
Y. Jiang, E. Liu, B. Eysenbach, J. Z. Kolter, and C. Finn. Learning options via compression. Ad- vances in Neural Information Processing Systems, 35:21 15:21184-22 181-11422111 2119 2022. [Cited p. 44.]
J. Johnson, B. Hariharan, I.. Van Der Maaten, L.. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901- 2910, 2017. [Cited p- 2, 9, and 50.]
L. Karazija, I. Laina, and C. Rupprecht. ClevrTex: A Texture-Rich Benchmark for UnsupervisedMulti-Objeee Segmentation. In Thirty-fifth Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2021. [Cited on p. 10 and 49.]
I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020. [Cited on p. 3.]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. [Cited on p. 49 and 50.]
D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014. [Cited on p. 2 and 8.]
T. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Compile: Compositional imitation learning and execution. In International Conference on Ma- chine Learning, pages 2418-3428 PMLR, 2019. [Cited on p. 48.]
E. Grefenstette, P. Kohli, and P. Battaglia.
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L.. Gustafson, Xiao, S. Whitehead, A. C. Berg, W.-Y Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 4015 4026, 2023. [Cited on p. 45.]
B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687- 15701, 2022. [Cited on p. 3.]
D. A. Klindt, L. Schott, Y. Sharma, Ustyuzhaninov, Brendel, M. Bethge, and D. Paiton. To- wards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations, 2021. [Cited on p. 3.]
A. Kori, F. Locatello, A. Santhirasckaram, F. Toni, B. Glocker, and F. D. S. Ribeiro. Iden- tifiable object-centric representation learning via probabilistic slot attention. arXiv preprint arXiv:2406.07141, 2024. [Cited on p. 3 and 45.J
A. Kraskov, H. Stögbauer, R. G. Andrzejak, and P. Grassberger. Hierarchical clustering using mutual informatio.. Europhysics Letters, 70(2):278, 2005. [Cited on p. 44.]
S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste-Julee Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In Conference on Causal Learning and Reasoning, pages 428 484. PMLR, 2022. [Cited on p. 3.]
S. Lachapelle, D. Mahajan, I. Mitliagkas, and S. Lacoste-Julien. Additive decoders for latent vari- ables identification and cartesian-product extrapolation. In Advances in Neural Information Pro- cessing Systems, volume 36, 2023. [Cited on p. 1. 2. 3, 4, 6, 7, 19, 21, 22, 23, 27, 35, 36, 39, 41, 42, 45, 46, and 48.]
S. Lachapelle, P. R. López, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julis Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint 205.::24000048990 2024. [Cited on 3.]
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017. [Cited on p. 1.]
14

---

Published as a conference paper at ICLR 2025
A. Lamb, D. He, A. Goyal, G. Ke, C.-F. Liao, M. Ravanelli, and Y. Bengio. Transformers with com- petitive ensembles of independent mechanisms. arXiv preprini urXiv:2003.00336, 2021. [Cited on p. 45.]
Y. LeCun. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. OpenRe- view, pages 1 1-62, 2022. [Cited on p. 45.]
T. Leemann, M. Kirchhof, Y. Rong, E. Kasneci, and G. Kasneci. When are post-hoc conceptual explanations identifiable? In Uncertainty in Artificial Intelligence, pages 1207-1218. PMLR, 2023. [Cited on p. 3.]
Lippe, S. Magliacane, Löwe, M. Asano, Cohen, and S. Gavves. Citris: Causal identifi- ability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557-13603. PMLR, 2022. [Cited on p. 3.]
P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In International Conference on Learning Representutions, 2023. [Cited on p. 3.]
S. Lippl and K. Stachenfeld. When does compositional structure yield compositional generalization? kernel theory. arXiv preprint arXiv:2405./639/ 2024. [Cited on p. 3.]
F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Interna- tional Conference on Machine Learning, pages 4114 141444. PMLR, 2019. [Cited on p. and 3.]
Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages PMLR, 2020a. [Cited on p. 3.]
F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovit- skiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, volume 33, pages 11525-11538, 2020b. [Cited on p. 2, 9, 47, 50, and 51.]
L. Mahon, L. Shah, and T. Lukasiewicz. Correcting flaws in common disentanglement metrics. arXiv preprint arXiv:2304.02333 2023. [Cited on 45.]
C. Meo, L. Mahon, A. Goyal, and Dauwels. S1 VAB:: On the relationship between disen- tanglement and diversity. In The Twelfth International Conference on Learning Representations, 2024. [Cited on p. 44.]
M. Mitchell. Abstraction and analogy-making artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):791001, 2021. [Cited on p. 1.]
M. Montero, J. Bowers, R. Ponte Costa, C. Ludwig, and G. Malhotra. Lost latent space: Ex- amining failures of disentangled models at combinatorial generalisation. In Advances in Neural Information Processing Systems, volume 35, pages 10136-10149, 2022a. [Cited on p. 2.]
M. L. Montero, J. Ludwig, R. P. Costa, G. Malhotra, and J. Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2021. [Cited on p. and 45.]
M. I. Montero, J. Bowers, R. P. Costa, C. J. Ludwig, and G. Malhotra. Lost in latent space: Exam- ining failures of disentangled models at combinatorial generalisation. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022h. [Cited on p. 45.]
G. E. Moran, D. Sridhar, Y. Wang, and D. Blei. Identifiable deep generative models via sparse decoding. Transactions Machine Learning Research, 2022. [Cited on p. 3.j
J. Munkres. Analysis On Manifolds. Basic Books, 1991. [Cited on 19.]
G. Murphy. The big book of concepts. The MIT Press, 2004. [Cited on p. 1.]
G. Papamakarios, E. Nalisnick, D. Rezende, S. Mohamed, and B. Lakshminarayanan. Normal- izing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22 (57):1-64, 2021. [Cited on p. 8.)
15

---

Published as a conference paper at ICLR 2025
J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017. [Cited on p. 4.]
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. Pmlr, 2021. [Cited on p. 45.]
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 1(2):3, 2022. [Cited on p. 45.]
N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.000 2024. [Cited on p. 45.]
J. R. Reynolds, J. M. Zacks, and T. S. Braver. A computational model of event segmentation from perceptual prediction. Cognitive science, 31(4):613 643, 2007. [Cited on p. 44.]
M. Rolinek, D. Zietlow, and G. Martius. Variational autoencoders pursue pea directions (by acci- dent). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12406-12415 2019. [Cited on p. 8.]
N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510 2023. [Cited on p. 4..
C. Saharia, W. Chan, Saxena, L. Li, Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic texl-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, volume 35, pages 36479-36494, 2022. [Cited on 45.]
M. S. Sajjadi, D. Duckworth, A. Mahendran, S. Van Steenkiste, F. Pavetic, M. Lucic, L. J. Guibas, K. Greff, and T. Kipf. Object scene representation transformer. In Advances in Neural Information Processing Systems, volume 35, pages 9524,, 2022a. [Cited on p. 8, 9, and 46.]
M. S. Sajjadi, H. Meyer, E. Pot, U. Bergmann, K. Greff, N. Radwan, Vora, M. Lučić, D. Duck- worth, A. Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view syn- thesis through set- set-latent scene representations. In Proceedings of the IEÉE/CVE Conference on Computer Vision and Pattern Recognition, pages 6229 -238,, 2022b. [Cited on 9 and 46.)
A. C. Schapiro. T. T. Rogers, N. I. Cordova, N. B. Turk- Brow-Br and M. M. Botvinick. Neural representations of events arise from temporal community structure. Nature neuroscience, 16(4): 486 496,492,220 2013. [Cited on p. 44.]
J. Schmidhuber. Towards compositional learning in dynamic networks. Technical University of Munich (Technical Report FKI-129-90), 1990. [Cited on p. 44.]
J. Schmidhuber. Learning factorial codes by predictability minimization. Neural computation, 4(6): 863-879, 1992. [Cited on p. 44.]
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612 634, 2021. [Cited p. 1.]
L. Schott, J. von Kügelgen, F. Träuble, P. V. Gehler, C. Russell, M. Bethge. B. Schölkopf, F, Lo- catello, and W. Brendel. Visual representation learning does not generalize strongly within the same domain. In International Conference Learning Representations, 2022. [Cited on p. and 45.]
M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-J. Simon-Gabriel, T. Hc, Z. Zhang, B. Schölkopf, T. Brox, and F. Locatello. Bridging the gap to real-world object-centric learning. In International Conference on Learning Representations, 2023. [Cited on p. 9, 10, 46, 47, 48 49, and 51.]
G. Singh, F. Deng, and S. Ahn. Illiterate DALL-E learns to compose. In International Conference Learning Representations, 2022a. [Cited on p. 46, 47, and 50.]
16

---

Published as a conference paper at ICLR 2025
G. Singh, Y.-F. Wu, and S. Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Advances in Neural Information Processing Systems, volume 35, pages 18181-18196, 2022b. [Cited on p. 9.]
C. Summerfield. Natural General Intelligence: How understanding the brain can help us build Al. Oxford university press, 2022. [Cited on p. 45.]
J. B. Tenenbaum, C. Kemp, T. I.. Griffiths, and N. D. Goodman. How to grow mind: Statistics, structure, and abstraction. Science, 33 (31(6222)
2011. [Cited p. 1.]
N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. arXiv preprint physics/0000057, 2000. [Cited on p. 44.]
E. C. Tolman. Cognitive maps in rats and men. Psychological review, 55 4:189-208, 1948. [Cited on p. 45.]
A. Vani, Nguyen, S. Lavoie, R. Krishna, and Courville. Sparo: Selective attention for robust and compositional transformer encodings for vision. arXiv preprint 2rXiv:2404115211, 2024. [Cited on p. 45.]
B. Varici, E. Acartürk, K. Shanmugam, and A. Tajer. General identifiability and achievability for causal representation learning. In International Conference on Artificial Intelligence and Statis- tics, pages 2312-2322. PMLR, 2024. [Cited on p. 3.]
A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems, vol- ume 30, 2017. [Cited on p. 8, and 42.]
von Küggelgen, Y. Sharma, I.. Gresele, W. Brendel, Schölkopf, M. Besserve, and Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Ad- vances in Neural Information Processing Systems, volume 34, pages 16451-16467, 2021. [Cited on p. 3.]
J. von Kügelgen, M. Besserve, L.. Wendong, L. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interven- tions. In Advances in Neural Information Processing Systems, 2023. [Cited on 3.]
Y. Wang, L. Liu, and J. Dauwels. Slot-vae: Object-centric scene generation with slot attention. In International Conference on Machine Learning, pages 36020-36035. PMLR, 2023. [Cited on p. 45.]
N. Watters, L. Matthey, S. Borgeaud, R. Kabra, and A. Lerchner. Spriteworld: A flexible, config- urable reinforcement learning environment, 2019a. [Cited on p. 2, 9, and 50.]
N. Watters, L. Matthey, C. Burgess, and A. Lerchner. Spatial broadcast decoder: A simple ar- chitecture for learning disentangled representations in VAEs. arXiv preprint 191.007901.07017. 2019b. [Cited on p. 2, 9, and 50.]
T. Wiedemer, J. Brady, A. Panfilov, A. Juhos, M. Bethge, and W. Brendel. Provable compositional generalization for object-centric learning. In International Conference Learning Representa- tions, 2024a. [Cited on p. 2, 4, 45, and 49.]
T. Wiedemer, P. Mayilvahanan, M. Bethge, and W. Brendel. Compositional generalization from first principles. In Advances in Neural Information Processing Systems, volume 36, 2024b. [Cited on p. 3, 7, and 9.]
D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F. Lo- catello. Multi-view causal representation learning with partial observability. In International Conference on Learning Representations, 2024. [Cited on p. 3.]
J. M. Zacks, C. A. Kurby, M. L. Eisenberg, and N. Haroutunian. Prediction error associated with the perceptual segmentation of naturalistic events. Journal of Cognitive Neuroscience, 23:4057- 4066, 2011. [Cited on p. 44.]
Y. Zheng and K. Zhang. Generalizing nonlinear ICA beyond structural sparsity. In Advances in Neural Information Processing Systems, volume 36, pages 13326-13355, 13355, 2023. [Cited on p. 4.]
R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 22979-12990. PMLR, 2021. [Cited on 3.]
17

---

Published as a conference paper at ICLR 2025
Appendices
Table of Contents
A Disentanglement Proofs
19
A.1 Additional Definitions and Lemmas
19
Sufficient Independence Assumptions
20
A.3 From Local Global Disentanglement
21
A.4 Disentanglement Most 0 Orden/No Interaction)
................................
23
A.5 Disentanglement (At Most 1 Order Interaction)
26
Disentanglement Most 2 Order interaction)
B Multi-Index Notation
34
C Characterization of Functions With At Most n Order Interactions
34
Compositional Generalization Proofs
37
E Unifying Assumptions from Prior Work
40
E.1 At Most 0t Order Interaction Across Slots
40
E,2 At Most 1a Order Interaction Across Slots
41
F Transformers for Interaction Regularization
F.1 Jacobian of Cross-Attention Mechanism
43
F2 Interaction Regularizer
43
G Extended Related Work
44
Theory
44
G.2 Method and Experiments
45
H Extended Discussion
46
H.1 Theoretical Assumptions
46
H.2 Method and Experiments
46
Enforcing Theoretical Criteria Oft-oo-Domain.
48
Experiments CLEVRTex
49
I.1
Experimental Setup
49
I.2
Results
49
Experimental Details
J.1 Data, Model, and Training Details
50
J.2 Metrics and Evaluation
51
J.3
Additional Figures
51
18

---

Published as a conference paper at ICLR 2025
A DISENTANGLEMENT PROOFS
A.1 ADDITIONAL DEFINITIONS AND LEMMAS
Definition A.1 (C*- diffeomorphism). Let A C Th and B C IR". A map f : - B is said to be a C* df-iiiffoooopiis it is bijective, C* and has a C* inverse.
Remark A.2. The property of being differentiable is usually defined only for functions with an open domain of TR= Note that, in the definition above, both A and B might not be open sets in their respective topologies. For an arbitrary domain C IR" we say that function f C* it can be extended to a C* function defined on an open set U' containing A. More precisely, f A + B is O* if there exists function g : - " such that 1) U is open containing A. g is C+:, and 3) g()) fa) for all a E A. See p9199 of Munkres (1991) for details about such constructions.
Definition A.3 (Regular closed sets). A set Zuup S RR is regular closed if Zupp is equal to the closure of its interior (in the standard topology of I.".
Süup i.e. if it
Lemma A.4 (Lachapelle et al. (2023)), Let A, C. R. and suppose there exists an homeomorphism f If A is regular closed R., we have that B C BP.
The way defined C* functions with arbitrary domain is such that function be differentiable without having a uniquely defined derivative everywhere on its domain. This happens when the derivative of two distinct extensions differ. The following Lemma states that the derivative of C* function is uniquely defined on the closure the interior of its domain.
Lemma A.5 (Lachapelle et al. (2023)). Let A R" and f A T. be a C function. Then, its k first derivatives uniquely defined on A in the sense that they do not depend the specific choice of C extension.
Notation For subset S [d2] and matrix A E RRxn As will denote the sub-matrix consist- ing of the columns A indexed by S i.e As = [A, .les- Similarly, for a vector 2, zg will denote the sub-vector of z consisting of components indexed by S ie. zs == (z)ies.
Lemma A.6. Let Rm and let be a partition of [n]. If
rank (A)
Σ rank (As)
(A.1)
SEB
Then Vz RR" s.t. A= 0, Asæg for any S B.
Proof. Assume for contradiction that there exist a E R", s.t. Az 0, and there exist S E B s.t. As, 2S, + O.
Now construct the matrix, denoted, A s, consisting of all columns in A except those indexed by S1, i.e.
A s, = [[..kees]] SEBYS,
(A.2)
By using (A.1) and the property that rank([ [B C1) S rank + rank (C), we get
(A)k(A)
Σ
rank (As) rank(As,)
(A.3)
SEB,S1
rank(
(A-s,) + rank(As)
(A.4)
rank(A)
(A.5)
Consequently, have that:
rank A) = (ank(Ass;) + rarka
(A.6)
This implies that the column spaces of both matrices denoted range( A range( (As,) respec- tively, do not intersect, except at the zero vector.
A simple example of such situation is the trivial function f: {0) + (0) which is differentiable at 0 but does not have well defined derivative because g(x) æ and h(x) both differentiable extensions of but have different derivatives
19

---

Published as a conference paper at ICLR 2025
Now we know that at
Az A- + As
(A.7)
(A.8)
Consequently,
A- 5- =s,
(A.9)
and by our assumed contradiction know that at
As, 2s. * 0
(A.10)
This implies that the column spaces of A-s,, As, must intersect at point other than the zero vector, which is contradiction.
Lemma A.7. Let A E Rdxd be an invertible matrix and [B Bx} be partition of] Assume there are k1, 22, & [K] such that:
AB,
* ABe.
(A.11)
Then there exists subset C [d] with cardinality [&]] that has the following properties:
1. The sub-block A is invertible.
2. S I for any W [K]
Proof. We first prove that there must exists an S satisfying point Since is invertible, each subset of rows is linearly independent and thus rank (B.)) = [Bg] This implies that there exist a set S [d] with cardinality [&&] such that Vi E S, A B,,5 linearly independent, and thus form basis of R RlB
If S Z Bg for all M E [K], we are done.
We consider the where there exists k such that S S Bg We will show that construct a different S from S which satisfies both conditions.
We know by (A.11) that there exist second block k* + k' such that for some j" B AB + 0. Since {ABrhhes forms basis of R BBl,, the vector AB can be represented uniquely
A Σ o.AA iES
(A.12)
where a; E for all i. Because AB * O, there exists S such that aj + 0. Because this representation is unique, we know that AB. is outside the span of (A,uiiiss
This means
that, by taking S* (S) (T}) U {j*), we have that {A is basis for R Baa in other words, Bt.S* S* invertible. Also, S not included in single block since ()) C Bg and j BA- with k' + k".
L
A.2 SUFFICIENT INDEPENDENCE ASSUMPTIONS
Definition A.8 (Sufficient Independence thth Order)). Let Rdd - Rdd be a C* function with thh order interactions between slots (Def. 3.2). The function f is said to have sufficiently independent derivatives if Vz d
rank (ID:f()))ie = Σ rank (IDef kE(K]
(A.13)
Definition A.9 (Sufficient Independence (1st Order)). Let f Rd= Rd. be C function with at most order interactions between slots (Def. 3.3). The function f said to have sufficiently independent derivatives if Vz E Rd::
rank ([D.ftt)) ieB, f ( kElK)
Σ kE[K]
Frank (ID: rank f())
20

---

Published as a conference paper at ICLR 2025
Definition 4.2 (Sufficient Independence (2nd Order)). A C function f :R - Rd with at most 2nd order interactions across slots said have sufficiently independent derivatives if Vz E Rd:;
[D f f( [D (()) [D.
rank
KE[K]
Σ KE
rank ((iD.f()))kEB [D f(z)) + rank (1L sn f
A.3 FROM LOCAL TO GLOBAL DISENTANGLEMENT
This section takes care of technical subtleties when one has to go from local to global disentan- glement. The disentanglement guarantee of this work is proven by first showing that Dh, i.e. the Jacobian of h := f-1 I f. has block
structure everywhere, and from there showing that can be written as (=) = (h1 (h1(2B=())) ( ..... hx (ZB+(k))) (see Defintion 2.1). Lachapelle et (2023) refers the first condition on Dh. local disentanglement and the sec- ond condition on h as global disentanglement, the latter of which corresponds to the definition of disentanglement employed in the present work. The authors also show that going from local to global disentanglement requires special care when considering very general supports Zupp, like we do in this work, as opposed to the more common assumption that Sspp := Rdd which makes this step more direct (e.g., see Hyvärinen al. (2019)). This section reuses definitions and lemmata taken from Lachapelle et al. (2023) and introduces a novel sufficient condition on the support of the latent factors, named adigned-comeeceedness, guarantee that the jump from local to global disentanglement can be made.
Definition A.10 (Partition-respectiin permutations). Let B == {B1, Bg.. Bg..... Bx} } be partition of (I....d}. A permutation π over (I., d} respects B if, for all B E B, TBB) E B.
Definition A.11 (B-block permutation matrices). A matrix A E Iddxd is B--lock permutation matrix if it is invertible and can be written A C.,, where Pa is the matrix representing the B-respecting permutation π (Definition A.10), i.e. Pae; = ea(t), and C' E Rddd is such that for all distinct blocks B,B B' E B, CB,B* 0.
Proposition A.12. The inverse of B-block permutation matrix is also B-block permutation ma- trix.
Proof. First note that must be invertible, otherwise A is not. Also, C-1 must also be such that (C-+) )B.B' 0 for all distinct blocks B, B' E B. This is because, without loss of generality, we assume the blocks of B are contiguous which implies that C is a block diagonal matrix that C-1 is also block diagonal. Since π preserves B, we have that P C-1 Pn is also block diagonal since, for all distinct B, B' E B, (P. (.)))BB) = (C- = O, where we used the fact that the blocks π (B) and π (B') are B, because π is B B-preserving, and are distinct, because m is a bijection. We can thus see that
1
CT
R
= C.P.
ČPP
-CP,1
where : ---PTPPPP is block diagonal and - is block-preserving.
Definition A.13 (Local disentanglement; Lachapelle et al. (2023)). A learned decoder f Rdd Rd= is said be locally disentangled w.r.t. the ground-truth decoder f when h(z) f(z) for all E Zamp where the mapping is diffeomorphism from Zupe onto its image satisfying
the following property: for all Zaupp= Dh(z) is block-permutation matrix respecting B [BI,.... }x).
Note that, in the above definition, the permutation of the blocks might change from another (see Example in Lachapelle et al. (2023)). To prevent this possibility, we will assume that Z is path-connected:
21

---

Published as a conference paper at ICLR 2025
Definition A.14 (Path-connected sets). A set sup S Rd is path-connected if for all pairs of points z", _ ε Zupp there exists a continuous map Φ [0. 1] -> Zxup such that Φb0) = = and ((1) = =z Such map is called path between 2" and z²!
The following Lemma from Lachapelle al. (2023) can be used show that when h is diffeo- morphism and Zupp is path-connected, the block structure cannot change. This is due to the fact that (2(2) is invertible everywhere and continuous function of We restate the Lemma without proof.
Lemma A.15 (Lachapelle et al. (2023)), Let C be path-connected topological space and let M + Raxd be a continuous function. Suppose that, for all a E C, M() is an invertible B-block permulation matrix (Definition A. 11). Then, there exists a B-respecting permulation such that for all E C and all distinct B, B' M (C)
It turns out that, in general. having that Dh has constant support suupp not enough to make the jump to global disentanglement. See Example 7 from Lachapelle et al. (2023). We now propose a novel condition on the support Zsupp and will show is sufficient guarantee global disentanglement in Lemma A.18.
structure across its
Definition A.16 (Aligned-connected sets), set A C Rd said to be aligned-connecter w.r.t. a partition {B1, B2...., Bx) } if, for all [K] and all a' A, the set {a A aB, d is path-connected.
Remark A.17 (Relation to path-comnectedness)) There exist sets that are path-connected but not aligned-connected and vice-versa. Example from Lachapelle et al. (2023) presents "U-shaped" support that path-connected but not aligned-connectee where 1 {a E I² aj Z a 1} and A(2) {a I2a a S -1,a2 < -1 is
Moreover, the set = (A) 1) (
aligned-connected w.r.t. the partition B = {{1), {2}} but not path-connected.
We now show how aligned-con connecedecnnnnteddesss combined with path path-conneetedness is enough to guaran- tee global disentanglement from local disentanglement.
Lemma A.18 (Local to global disentanglementt; Suppose h is diffeomorphism from Zuup S Rd= to its image and suppose Dh(=) is a B-block permulation matrix for all E Zuup (local disentanglement). If Zump path-connected (Defn. A.14) and aligned-comnetted set (Defn. A.16), then h(z) = (2 h..,,, (1 (global disentanglement).
for all E Bsapp where the h are diffeomorphisms
Proof. Since h is a diffeomorphism, Dh is continuous and D)(z) is invertible for all z E zupp-- Since also have that Zupp path-connected, we can apply Lemma A.15 to get that there exists a permutation [K] + [K] such that, for all z E Zsupp and all distinct k, K E [K], we have Dh(2) B B,B
0. We must now show that h (z) depends solely on = Consider another point z' Zuupp such that 2B = z'Bnnj) We will now show that hB. (z) = (x))(2)) i.e. changing zB does not influence (z).
O. In other words, DB_,,, hB, (z)
Because zaupp is aligned-connected, there exists a continuous path o : 00 1] + Z such that 6(0) = z', (11) z and @B (1) = ZBz() 2B, (M)
for all t E [0, 1]. By the fundamental
This lemma also holds C connected.
22

---

Published as a conference paper at ICLR 2025
theorem of calculus, have that B (z) -BB (z') )
L (hB o (t)dt
Dha, (o(t))
L.'
DB hB, (((t))oBBcn)) (t)
Σ
Dr hB B(rt) at
f
DB hB, (
Σ 0 0 (t)
dt
k'
0,
where we used the fact that (B (t) is constant function of and DB hB, (z) = 0 for distinct k, K'
We conclude that, for all k, we can write hB, (z) = hBB (#B.,))) which is the desired result.
Additionally, the functions B. (ZBzay) are diffeomorphisms because their Jacobians must be vertible otherwise the Jacobian of (which is block diagonal) would not be invertible (which would violate the fact that it is diffeomorphism).
Contrasting with Lachapelle et al. (2023). Instead of assuming aligned-comnettednes. Lachapelle et al. (2023) assumed that the block-specific decoders, which would correspond to the (z ) in (4.2), are injective which, when combined with path-connectedness, also enough to go from local to global disentanglement in the context of additive decoders (n 1). Whether similar strategy could be adapted for more general decoders with most rath order interactions left futuree work.
A.4 DISENTANGLEMENT (At MOST TTH ORDER/NO INTERACTION)
Lemma A.19. Let Zsupp S Z be a regular closed set (Defn. A.3). Let f : : 2 X be C' and supp Ssupp be a diffeomorphism. Let f == fo h. f has no interaction (Definition 3.4 with n 0), then, for all j.J E [ds] and z E 2supp- we have
Djf(z) Dy w (b(=))mmm' (ij)))
(A.14)
where
(²( = [W/( (=){=)]
w (z) [D,, f(z) D D (
m (² (j.5') (m] (i.j") ke]
mf(2, (j.j') [Dy (z) h. (z)le)
Proof. We have that
f Vz E Emup
Following the same line of argument as Lachapelle et al. (2023), we can use Lemma A.5 to say that the function fxx) = f o h(2) has well-defined derivatives on (2mmp) Since h -1 is a diffeomor- phism from Zuup (which is regular closed) to 2supy Lemma A.4 implies that 2xupp C (2.up)) This means that the function f(z) o h(2) has well-defined derivatives for all z E Zaupp.
By taking the derivative w.r.t. 2, on both sides of f(z) f f(z). we get
D)f(z)= Σ Σ D.f(M(2))DD,hu(())
(A.15)
kE[K] FEB
23

---

Published as a conference paper at ICLR 2025
We thus have that
(E
EE
D)f())D)f(()=
Σ
E
Di (((()))))))
E
D ( (2)
[LE(K] iE,
[ izE
Σ
E
Σ Dr,f(h(z)) @ Di,ff
ke|k] a B kk] fz
= Σ E
E
Duf(h(z)) @ Di,f (2)
k h E izE
where the last equality used the fact that f has no interaction (Definition 3.2). We conclude by noticing
)((
D. D..fhha)DD.fflli)gg.......... knE[K (a),
E E
(z(
X Theorem A.20. Let 2 2 be a C diffeomorphism satisfying interaction asymmetry
(Asm. 3.5) for all equivalent generators (Defn. 4.1) for 0. Let Zupe C 2 be regular closed (Defn. A.3), path-connected (Defn. A.14) and aligned-connected (Defn. A.16). A model f 2 + &Rd disentangles z on Zupp w.r.l f (Defn. 2.1) if is (i) a C difseomorphism between Zaup and Xaupp with (ii) at most 0 order interactions across slots (Defi, 3.4) on Zamp-
Proof. As mentioned in Section A.3, the proofs will proceed in two steps: First, we show local disentanglement (Definition A.13) and then we show (global) disentanglement via Lemma A.18. We first show local disentanglement.
Remark: We will use the following notation below:
D.f(z) R
(A.16)
We first define the function h: Suup + Zsupp relating the latent spaces of these functions on Zuup
h=
(A.17)
The function f can then be written terms of f and on Supp
f= h
(A.18)
Because f.f f are both C* diffeomorphism between zuup- Xsupp and Esups Kaupp. respectively, have that h is C² diffeomorphism.
By Lemma A.19, for all z E Emmpp j.j' [dz], have:
D,f(2) fz) (xx (z,(1)))
(A.19)
where uf and n are defined in Lemma A.19.
Define the sets
D
B2, kE[K]
D
D
(A.20)
Because f has no interaction (Definition 3.2), we have that, for all (i.j') D
0 = w (m (i,5))
Σ w/ ((
kE[K]
24

---

Published as a conference paper at ICLR 2025
Because f has no interaction, each row W (h(z)),, n. is non-zero for at most one k [K] (although this k can change for different values of n and 2). This implies that for all Zuup (iJ') D k E [
w (h( ( (j.j')
(A.21)
Case 1: |Bx Onne-Dimensional Slots)
When [& = 1, for all & E [dz]. the matrix W (h(z)) can be written as:
w (zf @ Dzf(=))
(A.22)
This matrix has single column, which must be non-zero since is C² diffeomorphism. Thus, We (h(z)) has full column rank and thus has null space equal to O. Using Eq. (A.21), conclude that for all (j,j") E DD, E [dz]:
0 = m (z, (j.j')
(A.23)
Applying the definition of m z, (f.j)), this implies that for all (J.J') E D k E [da]:
0 (z)
(A.24)
This means each row of the Jacobian matrix (z cannot have more than one nonzero value. Since the Jacobian is invertible, these nonzero values must all be different for different rows, otherwise whole column would be zero. Hence Dh(z) permutation-scaling matrix, i.e. have local disentanglement.
Case 2: [Bx] (Multi-Dimensional Slots)
Assume for contradiction that is not locally disentangled on Zuup w.r.t f. This implies that there exist a x" E Ssupps k. W, k" E [K] for K + k", such that:
DB (2))
DB hB. (z") 00
(A.25)
Because f are Ca diffeomorphisms, we know that h also a C diffeomorphism. Coupling this with Eq. (A.25), Lemma A.7 tells us that there exist S C [dz] with cardinality [Bg] such that:
VB B, S B, and Vi D,h,,
(z*) are linearly independent.
(A.26)
Now choose any B E B such that S = S B + 0. Furthermore, define the set S S1. Because S Z B, know that S2 is non- non-empty. Further, by construction S S₃ Sz. In other words, S1 and S2 are non-empty, form a partition of S, and do not contain any indices from the same slot.
Now construct the matrices, denoted As, and As, as follows:
As, == Ds, hB, (z"), As, == Ds, h..... (z")
(A.27)
And the matrix denoted A as:
A = [As,:
(A.28)
Note that because, Vi S, D.hB, (2*) are linearly independent (Eq. (A.26)), know that A₃ is invertible.
Now, define the following block diagonal matrix A E Rd,xd, as follows:
Aq 0
A
A =
(A.29)
Ak.
where Vi E [K] (k},A, is the identity matrix, and thus invertible, while Ag is defined according to Eq. (A.28).
Define := A Z, the function h Z + Z as h(x)) = Az and the function f Z - X' as f:= f h. By construction we have
V 2,
j(A
2B
A 2Bx) = ,2,
(A.30)
25

---

Published as a conference paper at ICLR 2025
Because all A, are invertible, then f is equivalent to in the sense of Def. (4.1).
We can now apply Lemma A.19 to f f h to obtain, for all j. E [d=]:
D,f(z) o Dj (z) = W (f()))mm (j.j")).
(A.31)
Choose & E 2 such that h(z) h(=)), which is possible because h(z²) E Z and h is bijection
from 2 to 2). We can then write
D, Dff(z) Dyf(z)== (h(2"")))))))) (E.()))))
(A.32)
Let J.J' S Bş be partition of Bş such that J is the set of columns of A corresponding to As. and J' be the set of columns of A corresponding As.. More formally, we have
A A = As, and AB = -sz
Since As, Ds, hB, (z*) and As, Ds (2*), we have that
A AB = Ds, (2") and AB Ds, BB
Since Dh(z)= = A, we have
DyhB (z) tB (z"). D,hB, (3) Ds, 1. (z") and
Choose some (j.J") E x J We know there must exist (s, (') E ESI S2 such that
D,h (2" and DyhB, (2) D. B2 (2*).
which implies
(=. = ( (s,
(A.33)
Moreover, since the Jacobian of h is block diagonal, we have that m (z, (i.j") = 0 for all k' + k (recall that j. Bş). This means we can rewrite (A.32) as
D,f(z) @ f w/ (h( ((((2*))))))) m (z, (J.J))
(A.34)
Plugging (A.33) into the above equation yields
D,f(z) o Dj f(z) W m (s,
(A.35)
Since (s,5') E St 82 C D we can apply (A.21) to get
D, = w (h(=z))m{( (s, s')) 0.
(A.36)
In other words, we found partition J' of the block B& such that D,f(z) o f()) D all (i,5") J x J'. This means that the blocks and J' have no interaction in f at z. This is contradiction with Assm. 3.5. Hence, we have local disentanglement.
0 for
Local to global disentanglement. We showed that Dh(2) is block-permutation matrix for all x 2uupe i.c. local disentanglement. Consider the inverse h, v := h-1 The Jaco- bian of is given by Dv(z) Dh (2) Dh(vv=))-1, by the inverse function theorem. By Proposition A.1 this means Dv(z) 2) is also a block permutation matrix for all z E Zupp- Since Zuup is aligned-connected (Definition A.1 Lemma A.18 guarantees that v²)) = (v) (ZB.a)),..... UK (2B())))) for all z E Zxupp where the Vg are diffeomorphisms. This implies that h(z) (=B, UK (ZB,-1kk)) )) for all E Zsup-- which concludes the proof.
write
A.S DISENTANGLEMENT (AT MOST ST ORDER INTERACTION)
Lemma A.21. Let Zsupp C Z be a regular closed set (Defn. A.3). Let f 2 X be C and Zxmpp Zupp be diffeomorphism. Let f == f h. If f has most 1st order interaction (Definition 3.4 with n 1), then, for all j.j E [dzz] and x E Zaup. we have
D3, f(z) = W (z, (i.5"))
(A.37)
26

---

Published as a conference paper at ICLR 2025
where
(²)2) = W [=]]]K[[K]
(x(x): = [P,, (2) f( (())
(² (f.j') = (z, (i.
(2, = [1] h (z))i ( ( (2)
Proof. The exact same argument as the one presented in Lemma A.19 (based on Lachapelle et al. (2023)) guarantees that, f and h have equal derivatives Zaupp. We leverage this fact next.
By taking the derivative w.r.t. 2j on both sides of f(z) = fo)( we get
D,f(z)= E E (.
(A.38)
iEB
Now take another derivative w.r.t. Zy for some j' E [dg] to get
D
f(=) - Σ Σ
Dr, (
Σ Σ hi (z)
LLE(K)GEBL,
k2E[K] S2E
Because we have at most first order interactions (Def. 3.4 with n = 1), the second sum over [K] drops, and we are left with:
D2 f(z)=
E
Σ
D (a
Σ R
D2 f (3) (z)
hs E Pi
Σ
Σ
D (b
D (n (2) (1,5))
Σ
k E|K]
w (h(=))m² (z, (j.j')),
which concludes the proof.
Theorem A.22. Let f 2 + X be a C² diffeomorphism satisfying interaction asymme- try (Asm. 3.5) for all equivalent generators (Defn. 4.1) for (Defn. A.9), Let Zaupp S Z be regular closed (Defn. A.3), path-connected (Defn. A.14) and aligned- connected (Defn. A.16). A model f Z > Re disentangles z Zsup w.r.. f (Defn. 2.1) if it is (i) a C2 diffeomorphism belween Zaup and Xapp with (ii) most 1st order interactions across slots (Defn. 3.4) on 2xapg
and sufficient independence
Proof. As mentioned in Section A.3, the proofs will proceed in two steps: First, we show local disentangiement (Definition A3) and then we show (global) disentanglement via Lemma A.18. We first show local disentanglement.
We first define the function h &supp + supp relating the latent spaces of these functions on Zsupp
h h:
(A.39)
The function f can then be written in terms of f and h on Eampp
f=fol h
(A.40)
Because f. f are both C diffeomorphism between Zuupp: Xsupp and Zupp Kaup. respectively, have that h is C diffeomorphism.
27

---

Published as a conference paper at ICLR 2025
Since f has most 1st order interactions, we apply Lemma A.21 to obtain, for all Zupp: J.J ) [da],
Dj (h(()))) (i,5"))
Since f has at most 1st order interaction, we have that, for all (j,j') E D
(z (i.(i"))
(A.41)
By defining
W res
(2) = [
#
w high
(z) Ph ( (())
h.resl %
2, (j,5') = [D}, hi; (z))iEBB.
high
(z, (j.j') = ID Dz), ( (z))(n))))B))
m.
we can restate the sufficiently independent derivative assumption (Def. A.9) as, for all 2 E 2 high rank (WZ rank w (i)))((())))))))))))))))) (x))
rank (W (z))
Σ
KE|K]
This condition allows to apply Lemma A.6 to go from (A.41) to, for all (j.j') E D k [K]:
w f high (h( (0(z)) ) high (z, (i.j'))
(A.42)
Case 1: |Bal (OOe--Dimmesiona Slots) By Assumption 3.5.ii, (with A B = (i)) f()) + 0. Note that W F.igh (h(=)) f2). Hence, (A.42) implies that mg h.hhgh (z, (i,j)) = 0 (which a scalar). This means m, htghhhh (z, (j,j") = (z) hx = Since this true for all and all distinct j. this means each row has at most one nonzero entry. Since Dh(z) is invertible, these nonzero entries must appear on different columns, otherwise a column will be filled with zeros. This means Dh(=) z) is permutation-scaling matrix, i.e. we have local disentanglement (Definition A.13).
Case 2: B&]
Mulli--Dieensiona Slots)
Assume for a contradiction that f is not locally disentangled on Zsup w.r.t. f. This implies that there exist x" E Zsupp= k, W K" E [K] with k' + k" such that:
DB,
#0, DB... hB, +0
(A.43)
Because f are C' diffeomorphisms, with Eq. (A.43), Lemma A.7 tells us that there exist an S C [d,] with cardinality |B,] such that:
know that h is also a C* diffeomorphism. Coupling this
VB E B, S B, and Vi E S, D.h,, (z") are linearly independent.
(A.44)
Now choose any B E such that S₁ == (Sn + 0. Furthermore, define the set S2 S/ S.. Because S Z B, we know that S2 is non-empty. Further, by construction S = Si U S2. In other words, S₁ and S2 are non-empty, form a partition of S, and do not contain any indices from the same slot.
Now construct the matrices, denoted As, and As, as follows:
As, Ds, hB, (2*), As Ds (z*)
(A.45)
And the matrix denoted A as:
A [As Asa)
(A.46)
Note that because, Vi E S, D,ha, (2*) are linearly independent (Eq. A.44), we know that A₁ is invertible.
Now, define the following block diagonal matrix A Rd. xd, follows:
A
0
0
A
A =
(A.47)
0
0
Ak.
28

---

Published as a conference paper at ICLR 2025
where Vi E [K] {K),4, to Eq. (A.46).
A, is the identity matrix, and thus invertible, while A₃ is defined according
Define _ == A A- the function h. 2 - 2 as h((z) == A= and the function f 2 - X as = f h. By construction we have
2z E 2,
f(A BB A 2Bx) f(B ,..., [Bx)
(A.48)
Because all A are invertible, then f is equivalent to f in the sense of Def. (4.1).
We can now apply Lemma A.21 to f = f o h to obtain, for all j.j' E [da]]
Dī, f(z) w ('((()))
(i,5))
(A.49)
Choose 2 E 2 such that h(z) h(="), which is possible because h(z²) E Z and h is bijection
from 2 to 2. We can then write
Dj, f(z) W (h(2= (2,
(A.50)
Let J' S B& be partition of Ba such that J is the set of columns of A corresponding to As, and be the set of columns of A corresponding to As.. More formally, we have
A B Asi and A Ase-
Since As, Ds, hag (z*) and As, Ds, (=*), we have that
A A = Ds. 1B. and AB Ds.hBB
Since Dh(3)= A, we have
Dyhn (3) hs, (z") and
Dyhn, (3) DDshn, (z").
For all (j.j') Jr, there must exist (s, s') E S1 S2 such that
D, Dyh (Z) DyhB, (=*) and hy (ž) DD (z*). This implies that, for all (j.j') E J x J', there exists (s, s') E S1 x S2 such that
h.high h. (.j')) = m hi. (8,88) 8)).
m
(A.51)
Moreover, since is block-wise function we have that, for all (j.j') E J C B₁ and k' [K] \ (k), m (z, (i-j)) O. We can thus write:
Di, w (m)))))))) (i.)"))
(A.52)
Since h is linear, we have that m heres (2, (j)))) = O, and thus
f(z)= w! high (le(2) 1m
high (z, (z,())
(A.53)
Plug the (A.51) into the above obtain that, for all (j.j') Jx
f(z) w wigh ( high (z", (8, s')) = 0
(A.54)
where the very last "- 0" is due to (A.42) (recall (s, E S S2 C D
In other words, we found partition J' of the block B₃ and value & such that DJ f()) for all (j.j") E J x J'. This means that the blocks J and J' have no second order interaction in f at E. This is contradiction with Assm. 3.5. Hence, we have local disentanglement.
From local global disentanglement. The same argument as in the proof of Theorem A.20 applies.
29

---

Published as a conference paper at ICLR 2025
A.6 DISENTANGLEMENT (AT MOST 2ND ORDER INTERACTION)
Lemma A.23. Let Zsup S Z be regular closed (Defn. A.3). Let f Z + X be C' and Zupp 7 supp be a diffeomorphism. Let f := f h. If f has at most 2nd order interaction (Definition 3.4 with n 2), then, for all j.j' E [dz] and x E Zuup. we have
of w (h(())))m (1.5'
(A.55)
where
w' ( (=))]KE[K]
(2)))== (z):=
[Da, f(z 1i)
ih (z))l
[ [ (112)))))))))
m² (i,'',"))"" = (z (i.J', ]
m3, (1.'',")) = [D] g (2))
[Dyhi, (z) gr + ( gun (2)+ (x) (2)
DDuitiis (2) (=))
As argued in Lemma A.21, differentiating f(z) h(2) w.r.t. 2j and zy on both sides yields
Proof.
D3 f(z)=
E Σ
Σ
Di, Pi
Σ
Σ
D2 (3 (2)
LE
kgE[K] 12EB22
Now take another derivative with respect to zj" compute Di or f(z). For the first term in the sum, have:
Σ
Σ
Σ
- (( jrr (z)
[LE[K] me
kxE[K] Bbg
And for the second term in the sum (the nested sum), we have:
Σ
D3 (n) (zD,,, (z) +
krE(K] Bx A2E[KK Bxg
LkyE[K]
ha (, (1
D² (D D
(=)
Because have second order interactions (Def. 3.4 with 2), this term can be rewritten
as:
Σ Σ
Σ
D bii. (z) +
kjE[K] Bi,
LizE BeE
Σ Σ
(z)
i
30

---

Published as a conference paper at ICLR 2025
Combining the first and second terms, we get:
Do oz f(z))
E
E
Di, ( (z) +
Σ
Σ
D i (3)))) ho + Dy (x) he (z) + (z)
(z) (z)) +
k2E[K] Ba
Σ
E
D D i2
Be Bb,
Σ
Σ Diff(h ( (z) +
An [|K] Li
E
Σ
D 11 f(((z)) [(h(z)) (Dyhi (2) + Dy (2) Di hi; (z) + ( hi, (z) +
hs,
Be 12E|ds]
Σ
D b
(w((h()))m (z. (1))
Theorem A.24. Let f 2 X be C diffeomorphism satisfying interaction asymme-- try (Asm. 3.5) for all equivalent generators (Defn. 4.1) for n = and sufficient independence (Defn. 4.2). Let zuup C Z be regular closed (Defn. A.3), path-connected (Defin. A.14) and aligned- connected (Defn. A.1 model f Z + Rda disentangles z on Zsupp w.r.t. f (Defn. 2.1) if is (i) a 33 diffeomorphism between Zzupp and Ksupp with (ii) at most 2nd order interactions across slots (Defn. 3.4) on 2
Proof. As mentioned in Section A.3, the proofs will proceed in two steps: First, we show local disentanglement (Definition A.1 and then we show (global) disentanglement via Lemma A.18. We first show local disentanglement.
We first define the function -xupp + Zupp relating the latent spaces of these functions on 2upp
h
(A.56)
The function f can then be written in terms of f and h on Zampp
f= f
(A.57)
Because f. f are both 2 diffeomorphism between Zaupp Ksupp and Zupp- Xamp respectively, we have that h is C diffeomorphism.
Since f has at most 2nd order interaction, we have that, for all E Zrugp (i.1',1") E D x [dg],
0 (<)
(A.58)
31

---

Published as a conference paper at ICLR 2025
By defining
W f rest (z)
[Di, f())linBB.
[D I (())(( EB.
fiig (z) == D
in 12 12.1 13 f (s) Sadse
m
h,
(z, (J.J',
[D], s hi, (z))
DDjh, (z) o hig (z) + (2)D) hi, (z) + Dyrhi, (2) h (x)
m
= "
high (z,
(2), D,
restate the sufficiently independent derivative assumption (Def. 4.2) as, for all Z
rank (W (=))
Σ
rank (We. (z) rank f (z)
kE[K]
This condition allows apply Lemma A.6 go from (A.58) to, for all (i.j',j"' DE [da],
k: E [K]:
w (h(2)) h.hhgh (z, j"")
(A.59)
Case 1: [Bul
Dine-Dimessiona Slots) By Assumption 3.5 (with A B {i)).
we have that (x) * Note that w high plies that mi h.high (z, (J.J', (")) O (which (h(z)) = D fz).. Hence, (A.42) im- Dy
f.igh
scalar). This means m he.high (z, (J+)',"")) Dz)()) 0 for all (i,i'j") E De x [dz]. In particular, we have
Dy (b) =0,
for all (i.j') D Since this is true for all k and all distinct j.j'. this means each row of Dh(z) has at most one nonzero entry. Since Dh(z) x)) is invertible, these nonzero entries must appear on different columns, otherwise column would be filled with zeros. This means Dh (x) is permutation-ssaling matrix, ie. we have local disentanglement (Definition A.13).
Case 2: B&] (Multi-Diman Dintesioamm Slots)
Assume for contradiction that f does not disentangled on Zuup w.r.t. f. This implies that there exist a z² E 2supps ks K, k" E [K] with 1e' # k²" such that:
D (2 00, D h.g (z') +
(A.60)
Because f are are C diffeomorphisms, know that h is also C diffeomorphism. Coupling this with Eq. (A.60), Lemma A.7 tells us that there exist an S C [dz] with cardinality [B&] such that:
VB E B, S B, and Vi S, S,D,B,, (z") are linearly independent.
(A.61)
Now choose any B E B such that S ( B) * (.. Furthermore, define the set S2 = S\ S. Because S Z B, we know that S2 is non-empty. Further, by construction S = S₁ U Sz In other words, S₁ and S2 are non-empty, form partition of S, and do not contain any indices from the same slot.
Now construct the matrices, denoted As, and As, as follows:
As, Ds, hB, (z"), As, Ds, Ds. (z*)
(A.62)
And the matrix denoted A as:
A = AAs
(A.63)
Note that because, Vi E S, D,hn, (2*) are linearly independent (Eq. (A.61)), we know that Ag is invertible.
32

---

Published as a conference paper at ICLR 2025
Now, define the following block diagonal matrix A Rd:xd: as follows:
A 0
0
...
A
A =
(A.64)
0
Ax.
where vi E [K] (, is the identity matrix, and thus invertible, while A, is defined according Eq. (A.63).
Define = A 2, the function h Σ + as h(z) = Az and the function f 2 - X' as f = f By construction we have
E Z, f [(A BB, A BBx (2,,....,,
(A.65)
Because all A," are invertible, then f is equivalent to in the sense of Def. (4.1).
We can now apply Lemma A.23 to f = f o h to obtain, for all j.''. E [da]]
Di f(z) (*(h(2)) ( (i,'',"" )""))
(A.66)
Choose & E 2 such that h(z) = h(z²), which is possible because h(2*) E 2 and h is bijection
from 2 to Z. We can then write
Di rr f(E) ('(h)=)))mm (2,
(A.67)
Let J,J' C. B& be partition of Bu such that J is the set of columns of A corresponding to As, and J' be the set of columns of A corresponding As,. More formally, we have
AB = Asi and AB As
Since As, Dss hB, (z*) and As, Ds, DB, (2*), we have that
A B
Ds, 1 (z') and AB D
Since Dh(z) A, have
D,hB. Ds, (2))))) and DyhB, (z) D9, (z").
For all (J E J x J x B& there must exist (s, 8, E S1 x S2 x 5 such that
DyhB, = D.hex Dy (z) = Dyh (z"), and D, (z) = (x"h,,
This implies that for all (i, 33" J B& there must exist (s, 8' s!) S1 x S2 such that
m highh
(z, (j.j'."")) = m (x (8,8',8"))
(A.68)
Moreover, since is a bisck-wise function, we have that, for all (j.j',j") E J x J' x Bx C B and all A [K] \ [k), we have m (z, (i. O. which allows us to rewrite (A.67) as
3 f(z) W (h(=")) (..'"""))
(A.69)
fe (z, (j. )"') 0, and thus
Since linear, we have that m k
Di f(2) fhigh
high (h (z") (²²)))m² h. (z, (E.'"""'"""))
(A.70)
Plug (A.68) into the above to obtain that for all (i. J x Br, there exists (8,8',8")
Si x S2 x S such that
D3 f( hhih (m hiigh (s,8's")) = =0,
(A.71)
where the very last "= O" is due to (A.59) (recall (8, (8,8'8")) S1 S2 5 D (d.)).
33

---

Published as a conference paper at ICLR 2025
In other words, found partition J' of the block B₁, and value such that D D5''"" f(z) for all (j.J', E J' x Bg- One can show that f as no 3rd order interaction across blocks because it is equivalent to f. which also has no 3rd order interactions across blocks. We thus have that D s" f())
for all (j.j'.j") E J x J' x [d=]. This means that the blocks J and J' have no third order interaction in f at ž. This is contradiction with Assm. 3.5.
From local to global disentanglement. The same argument in the proof of Theorem A.20 applics.
B MULTI-INDEX NOTATION
Multi-index notation is a convenient shorthand to denote higher order derivatives. A multi-index of dimension d is an ordered tuple a (a1,....,00 E N4. We introduce the shorthands
ad!
II og!
(B.1)
1=1
and we write a 2 if 8; for all and a + B denotes the element wise sum (difference) of the entries. We write
gm
gmd
D
(B.2)
0z
and
(B.3)
We will need the important property that
B-a
if 3 a otherwise.
D a))
(B.4)
Consider a partition of d. into slots Bi...., Bg. We define the set of interaction multi-indices of order n for n 2 2 by
Ln (a Nd= a n. 3. s. t. BR is E B&, with k # k2 and 0i Qi2 0}, (B.5)
i.e. the set of all multi-indices such that the non-zero components are contained at least two blocks. Clearly In depends on the block partition which we do not reflect in the notation. We also consider
1.
U Im 25m5n
(B.6)
Clearly, a Ioj and / is any multi-index,, then a B 111+118)
C CHARACTERIZATION OF FUNCTIONS WITH AT MOST n.' ORDER INTERACTIONS
In this section we characterize functions with interaction of most nhth order by proving Theo- rem C.2. Our characterization relies on the notion of aligned congeteeconesssdness introduced in Defini- tion 16 and the following topological notion.
Definition C.I. topological space X contractible if there is continuous function F: X
[0, 1] + X such that F(r, 0) = x and F(r,, 1) zo for point 2n E X. We call a subset of IRad contractible if it is contractible topological space with respect the induced subspace topology.
34

---

Published as a conference paper at ICLR 2025
Roughly, contractibility means that transform topological space continuously into point, which is possible if the space has no holes. Note that, e.g., all one dimensional connected sets and all convex sets are contractible. Sets that are not contractible are, e.g., spheres and disconnected sets. Note that the characterization in the following theorem generalizes Proposition in Lachapelle et al. (2023) by allowing higher order interactions and showing the result for more general domains. We denote, similar (2.3), for l2 C Rdd by S2 = {ZB; z E 52} the projections of l2 on the blocks.
Theorem C.2 (Characterization of functions with at most n order interactions across slots.). Let Ω be an open connected and aligned-connecte set such that Ω4 is contractible. Let f(z) 2Bg:..,......
be a C function on Q2 for an integer n. E Zzz-- Then any distinct slots zB, and BB have at most n order interaction within f (Defn. 3.4) if and only if. for some con- stants { (Ca E &dd
Jaelga and some C functions ft
S2k + Rd such that for all e z
f(z)
8h (ZB,) + Σ caza.
(C.1)
acIg
Remark C.3. To avoid unnecessary complications focus the where the ground truth f is defined on 2 = Rd= Then Ω = clearly satisfies the assumptions and actually the proof is slightly simpler. The more general result here would allows us handle also Z Rd in Appendix D with minor changes.
The proof can be essentially decomposed in two steps: We show how to reduce from interaction of at most order to interaction of at most order 1 and then establish the induction base for n 2.
Lemma C.A. Suppose f Ω + RR is a C function and open and connected. Assume that f has interaction of at most order n between any two different slots for some n. 2 2. Let E Ω be any point. Then the function
Da f
f(z) Σ
za
(C.2)
ad
aeI,
has interaction of order at most n
1.
Proof. First observe that f having interaction at most implies that D is constant in Q for a E In. Indeed, since a E In we conclude a + e, E In+1 where e, denotes the tuple with i-th entry 1 and all other entries O. Then, by definition of having interaction at most in n in Definition 3.4, we conclude that
f.Daf(z) Date: f(z) =0
(C.3)
This implies that the total derivative of Da vanishes on Q, which implies that Da constant because Ω is connected. Consider now any B In. Then we find using (B.4)
D f(zo) za D() Σ aET,
D f(2)
D
f(z)
31 =0
(C.4)
a!
Gl
where we used that f is constant and 28 = for a + 3 if a proof.
-13..
This ends the
We now establish the functional form for interaction of at most order 1. This essentially a sim- ilar statement as in Proposition 7 in Lachapelle et al. (2023) except that we consider more general domains so that their proof does not apply.
Lemma C.5. Assume Q an open connected and aligned-connecter connected set such that Dr is con- tractible. If f is a function such that different slots have interaction at most of order I then there are functions f such that
K
f(z) = Σ (zB.).
(C.5)
k=1
35

---

Published as a conference paper at ICLR 2025
Proof. Fix a k I K. With slight abuse of notation write (= xBg)) Fix now some
value Bgg We claim that for all Bgg ZBgg such that (* BBB)) (BBE) E !
DzD, f((zB, =BE)) DzsR (lssf(((2)))) f((zB. #'Be)).
(C.6)
By assumption we indeed know that
Dzmg Dzn, f f((=B2: BBe))
(C.7)
Moreover, by nigmeddcommectedness we know that the set Q2nn = {am (# xng) E Ω2} is connected so we conclude that the function
Azng + BB/xx,
z + DEEg ,(
(C.8)
indeed constant. This implies that there is function g depending on BB, such that
g" (zB.) Dze, f((=B. =B;))
(C.9)
for all (zB,, 2n²) E Ω. Locally g² is the gradient of function, but by assumption Ω4 is contractible and therefore, by the Poincaré-Lemma, there is a function f such that D g" Then we find
K Σ
DEpR [(Z znx)) g(zn)) D DEE D (ZB,) DEBL
S
(Z)W))
(C.
Thus the difference f Σk= f has vanishing derivative on S and since is connected we conclude that is constant. This implies (C.5) after shifting one f by this constant.
Based on these two lemmas the proof of Theorem C.2 is straightforward.
Proof of Theorem C.2. In the first step we show that the at most n-th order interaction condition holds then f can be written as in (C.1), i.e., '='. Applying inductively Lemma C.4 we conclude that there are constants Co R such that
f(z)
Σ
Σ
c
(C.11)
m= aE..
has interaction of order at most 1. Thus, we can apply Lemma C.5 which implies that represen- tation as (C.1) exists on Q. For the reverse direction 'z" we observe that clearly the functional form implies for B Ln-1 the relation
D = O.
(C.12)
Let show through examples that the topological conditions the set Ω neccessary. The following examples shows that the condition that Q2 is contractible cannot be dropped.
Example C.6. For every # IR? \ {0} we denote by 0(z) [0, 2T) the argument (i.e., the angle to the positive r-axis in radian) and by r(z) |=| the radius of z. We consider Ω C RR and
Bi (1, 2}. B2 = {3, 4] given by
Ω 2= (z r(zB,))))..) ((z))) E (1,2), (0(zn.) - -(zB.) mod 2T)) E (0,
(C.13)
and the function
f f(z) = (=z)=0(zB)) - 0(=B,) mod 2x.
(C.14)
Then Q is aligned-connected because the in questions are annular sectors and in particular path connected. Moreover, f is smooth because 0(2p,) - 0(zp,) mod 2n E (0, π) so it does not jump and D Dzng f because it is locally additive. However it is not globally additive as in (C.1).
The necessity of the aligned conncetedness condition can be shown by example that is similar to Example in Lachapelle et al. (2023). Example C.7. Consider = ([-1,0) x 2- ( 1]x [1, (() 1] 2-2, and f F + I given by
f(z) - {
21, 22 0 otherwise.
(C.15)
Then f is C², f has interaction of order at most but f cannot be written as in (C. Note that Ω not aligned-connected because (22 (1/2, 22) ) Ω) [-2,- 1) [1,2] is not connected.
36

---

Published as a conference paper at ICLR 2025
D COMPOSITIONAL GENERALIZATION PROOFS
In this appendix we prove extrapolation result Theorem 4.4. Based on the functional form derived in Theorem C.2 we relate two differenent disentangled representations.
Lemma D.1. Let f 2 -R- Rd be a C diffeomorphism of the form:
f(z) = [(=)=EF f (zB,) + Ca2 k=
(D.1)
aE
for some f in C Let f + Rd, be a diffeomorphism of the same functional form. Let zxupp + be such that f = f on supp- If is a slot-wise function, i.e. for all k e (z))) = (z(ZB,) and zupp is regularty closed then for all E zaupp
5"
(M) (hx(Z + T(2)
(D.2)
for some affine function L IRd. - Rdd
Remark D.2. We note that it is not possible to remove the affine function L from the state- ment. Indeed if all slots have dimension 1 and h1 (21) 21 1, h2(22) 22 + then is an additive function. Moreover, we cannot in general
h1 (21 (222 - 22 21 + prove that h itself is slotwise affine because the coefficients can be zero. In this case h can any slot-wise diffcomorphism.
Proof. First we remark that the polynomial part of the functional form in (D.1) contains all terms 2,2 where j are in different slots, thus can be equivalently written as
Σ Ca acl2
(ZB, @ 2B..) Ark
(D.3)
for some constant matrices {Ask E (R 1BPllBBeDx0 HKMEE(K] where @ denotes the Kronecker product (c.g.. [21, 22] [23, 21] = [2123, 21 24, 2223, 22
We assume that the permutation ı the identity. We know that f. f are diffcomorphisms between the same spaces and can thus be related by the function h via:
= h
(D.4)
Inserting the functional forms for f. f and leveraging that h is a slot-wise function and n is the identity, have for all Z
K
K
k
(zB
BB & zB.) ALk
K Σ
Σ jh (2n.)
Σ
(zm,) 2 h² (znw)) Awx.
k++
(D.5)
To prove the claim we now consider the expression
I(z) Σ 8h (zB.)
Σ th (hB.(ZB.)
(D.6)
K
K
K K
(1²())) 2 12 (EBw) A
Σ
Σ
(zB, & 2B,) A&a
i= #=
k=
k k==+1
37

---

Published as a conference paper at ICLR 2025
and prove that Lz) is affine function. To show this it is sufficient to prove that the second derivative D vanishes because Expp is path-connected Thus we consider all partial derivatives. Consider first the case where i BR and i' E By for k < k'. Then we find that
K E (ZBB)
K
DeDa = DrDo
- Σ fr (hB, (zB.)) 0.
(D.7)
R=1
remains to consider derivatives of the form DrDr where i, E B₁ for some slot i. Then we clearly have
D,Dr
(zB, @ xBx) Ask
(D.8)
k= 1+
because this is linear expression in ZB, Next, want show that
D,D: (zB,) @ h" (zB,)) A =
(D.9)
for all & k. To prove this we show the more general statement (that will be used in the proof of Theorem 4.4 below) that for any k of k' and any vector v E BR the functions
BB + (zBx) @ A
(D.10)
are affine on 2g or equivalently that
DrDr (h (zn,) v)) AR =
(D.11)
for every v E RPv, To prove this we consider any j E By and apply the derivative D, to (D.5) get
0 (D.D,, @ D,h" (zB²)) An
(D.12)
for every z E Zsupp- Now we use that by assumption h is a diffcomorphism. Using the block particular, this implies that for any
structure of h we find that also h² are diffeomorphisms. # E Zzupp the vectors (D,h' (=Bn)) B)B are lincarly independent vectors in R Byll and they thus generate Bl Therefore we can find coefficients aj (depending on zB such that
Σ a,D,h (zBB)= jEB
(D.13)
Then we get using (D.12)
D,Dy (h (zB,) @ v)) AWN D,Dy h²(2z)) 8 I (xBB)))
(1BB))
A
JEBy
(D.14)
Σ a, (D,D,h" @ Dj (ZBw)) AKK =0 JEB
So (D.10) holds and thus also (D.9) (we actually only get this for points 2 E Zμ such that there is =B, but by continuity and since zuupp is regularly closed this actually holds
E Zsupp with on Zz). The same reasoning shows that this also true if i, 1' Be (instead of i, Bx). We then find that for i, i E BB
K Σ FIW=KK1
K
D Σ
(2 @ k2 (ZBw)) A
(D.15)
K
D,D,y (ZBB) @ KK (IBw.) A
The last display together with (D.8) and (D.7) imply that D2 = 0 and thus L is affine. When и is not the identity the proof is similar.
38

---

Published as a conference paper at ICLR 2025
We also need the following simple lemma which states that we have unique Cartesian Cartesian-product tension of functions with interaction of order at most n between different slots.
Lemma D.3. Let f= - IR be a C diffeomorphism with interaction at most n between different
slots such that Zxup is regularly closed and for 2 E Zyup
K f(z) - - (zBL)
Σ
Σ
Caz
(D.16)
PSmSnaEly
for some S in C Then this relation holds on ΣCPF-
Proof. We know by Theorem C.2 that a representation as in (D.16) holds on 2 Rd and thus can be restricted ZOPE, however it might not be the same representation but involve functions f and constants Cx. Taking the difference and setting f f f and ća Ca ca we find that on Zupp
p (ZBz) ) +
Σ 25mSn am
Σ
caza
(D.17)
But by applying Da for E Im for down m we find 0 for all E Iga and thus the polynomial term vanishes. Next, apply D and find that ft is constant Z% (because Zuupp regularly closed). This implies that (D.17) holds ZcpE and thus (D.16) holds on ZCPE-
Using the previous lemmas can prove Theorem 4.4.
Theorem 4.4 (Compositional Generalization), Let n E (0, Let Zaup be regular closed (Defn. A.3). Let f Z > X and f 2 Rd be C diffeomorphisms with at most 1 order interactions across slots on If f disentangles z on Zaupp w.r.l. f (Defn. 2.1), then it generalizes compositionally (Defn. 2.2).
Proof of Theorem 4.4. Note that Corollary 3 in Lachapelle et al. (2023) already handles the case = 0, but the proof below is more general, and also covers the case of n 0, 1, since functions with most 1th and s order interactions are special cases of functions with most 2nd order interactions assuming f is a C diffeomorphism.
We can apply Theorem C.2 f which implies that can be written on Rd, in (C.1) and as explained in Lemma D.I an equivalent representation is
f(z) Σ
(zBr)
(ZB, 2Bxr) ARk
(D.18)
++
and we have similarly
f(z) Σ jk (zBr) + Σ (ZB, 2B,) ARk.
(D.19)
k-1k'st11
By assumption we have and the functions hx IR Brmol + IR Bal are diffeomorphisms. Our goal is to show that this relation actually holds on the Cartesian-product extensions ZCPE Let l be the set of points
h on Zsup where h(z) = (hi (ZBma)) ,..., hx (zn )
.,,....,,BB. E ú then z' (ZB....., 2 =BR) 2 for any 2'B, E Z, Let us define the map e² Z Z given by e² (2%))) = z'. We know by Lemma D.I that the function
such that f(z) = f h22)) for z E U. We claim that if
K
(Z
Σ
m (x) (ha(z) L(z)
(D.20)
1=1
1=1
39

---

Published as a conference paper at ICLR 2025
is affine on zupp Applying Lemma D.3 the same holds on ZCPE. Thus conclude that
BB (L))))
(1)
(z)()) T
(D.21)
is affine on 2t. Moreover.
K
BB Σ
Σ
(e'( @ (zh,, Aus
(D.22)
k=1W=k41
clearly affine on 2 and by (D.10) the same holds for
BB Σ
(z)) h² (c= AWN-
(D.23)
k=1k=k+1
The last three displays together imply that
z/8. - fole
(D.24)
affine on 2, and since it is zero in a neighhourhood of z'B, = 2n, (because 2 E (7) it is equal to zero Z₁. Since this is true for any slot B,
conclude that U Z. Indeed, pick any
open rectangle 2 x Zg X Z'x C Zuup U. We then infer that 2, x 2/2 ... 2'x U and by inducting over the slots and applying continuity at the boundary we obtain the claim.
E UNIFYING ASSUMPTIONS FROM PRIOR WORK
E.
AT MOST TTH ORDER INTERACTION ACROSS SLOTS
To prove that the assumptions in Brady et al. (2023) are special case of our assumptions for = 0, we first restate their assumptions formally. To this end, we first define the following set:
AA C [d.] Is(z) == {1 E [d=] (sfi(z) + }}.
(E.1)
The assumption of compositionality in Brady al. (2023) now be stated:
Definition E.1 (Compositionality). A differentiable function f Z + X, is said to be composi- lional if:
Vz E # [K] 1 =
(F.2)
We now state the second assumption in Brady al. (2023), deemed irreducibility. Definition E.2 (Inreducibility). A differentiable function f Z + X' is said to be irreducible if for all Z and [K] and any partition [=)=) S1 S2 (i.e., S1 S₂ 0 and S1, S2 * 9), we have:
rank (Dfs, (=)) rank (Dfs, (=)) rank (ff
(E.3)
We now prove that compositionality and irreducibility are equivalent to f having satisfying interac- tion asymmetry (3.5) for all equivalent generators (4.1) for 0.
Theorem E.3. A C diffeomorphism f Z + X' satisfies compositionaliity (Def. E.1) and irre- ducibility (Def. E.2) if and only if f has at most 0 order interaction across slots (Defn. 3.2) and satisfies interaction asymmetry (Assm. 3.5) for all equivalent generators (4.1).
Proof. We start by proving the forward direction, i.e,, that compositionality and irreducibility imply that if has at most 0th order interaction across slots and satisfies interaction asymmetry for all equivalent generators.
The definitions of compositionality and most 0hh order interaction across slots are pre- cisely equivalent, thus we only need to show that compositionality and irreducibility imply that satisfies interaction asymmetry for all equivalent generators. To show this will prove the following contraposition: that if f has at most t order interaction across slots and does not satisfy interaction asymmetry for all equivalent generators, then f is not irreducbble.
40

---

Published as a conference paper at ICLR 2025
Since has at most 0th order interaction across slots and does not satisfy interaction asymmetry for all equivalent generators, this implies that there exists matrix A E RR. xx B and a partition of Bg, into A, (AUB- Bg, A n B = =0) such that within the function f defined as:
Vz
,(A Bx = f(=n.......
2Bx)
(E.4)
where A, such that * is the identity matrix, the latents εn.ın have no interaction. This implies that under f. [A()) does not intersect with Iø(z). Further, because f is in- vertible, we know that Dfn. (2) is full column rank. Coupling these two properties, we conclude that rank(Dff, (2)) = rank(Dff(E)) + rank(Dfa())). Furthermore, the Jaco- bians (* and Df(z) will be related by an invertible linear map by construction. Thus, Dfs(ā) and Dzs(z) have equal rank for any subset S S [dz]. Therefore, conclude that DDf (2)fBB(()))))))))))))))))))))) (z)) = rank(Dfal (ff(()) + rank(Dff(z)) Because A and B form a partition of BN we conclude that is not irreducible.
We now prove the reverse direction if f has most 0th order interaction across slots and satisfies interaction asymmetry for all equivalent generators then f satisfies compositionality and irreducibility. As noted before, the definitions of compositionality and at most 0th order interaction across slots are precisely equivalent. Thus, we only need to show that if f has at most Oth order interaction across slots and satisfies interaction asymmetry then this implies f satisfies irreducibility. To show this, will prove the following contraposition: that if f does not satisfy irreducibility, then it does not satisfy interaction asymmetry for all equivalent generators with O.
Since f is not irreducible, we know that there exist a z, a slot & E [K], and a partition of Bg into A, B such that fB,, (=)) rank(Dff + rank(D.f column rank this implies that rank (Df = |A| and rank matrices Ms, E x and Ms, E [gd=x|B such that the column space of Ms, is the same as Df (z) and the columns space of Ms, is the same as Dfz(z). Now construct the following matrix M pd. IB, as follows:
Because DfB, (2) is full = B. Now take two
(E.5) Note that by construction this matrix has block structure such that rows for Ms, are never non- zero for the same rows Ms,. Because M and Dfg, (z) are both full column rank, then there exist a matrix A E IR BB such that:
M = [Ms, Ms,]
M = DfBx(2)A4
(E.6)
Now define the function as follows:
Vz 2, f f(A ...,..,, A A =Bx) f )Bx)
(E.7)
such that A, 1 is defined as above when = k. and otherwise it is the identity matrix.
Writing the derivative of D DB. (#) in terms of f we get DfB, (z) = M. Because M has a block structure we conclude that there exist partition of B& such that these latents have no interaction within f at æ. Because f is equivalent to f we conclude that the function does not satisfy interaction asymmetry for n 0.
E.2 AT MOST 1sT ORDER INTERACTION ACROSS SLUTS
We now prove that the assumptions in Lachapelle et al. (2023) are special case of our assumptions for n = 1. To this end, we first restate their assumptions. The first assumption in Lachapelle et al. (2023) is that the generator f is additive:
Definition E.4 (Additive decoder). A C² diffeomorphism f : Z + X' is said to be additive if:
f(z)= F where fh :R BBll + Edd= for any A [K] and Z.
(E.8)
KE[K]
Definition F.5 (Sufficient Nonlinearity), Let f 2 + X' be 2 diffeomorphism. For all k E [K]. let B& := B n {(11,62) i2) I 4.). f is said to satisfy sufficiently nonlinearity if Vz E Z the following matrix has full column-rank:
(()) = [D.(())leB [De [()) (() kE[K]
(E.9)
41

---

Published as a conference paper at ICLR 2025
We now state result.
Theorem E.6. Let : X be C2 diffeomorphism. If f satisfies additivity (Def. E.4) and sufficient nonlinearity (Def. E.5) then f has at most 1st order interactions across slots (Defin 3.3), satisfies sufficient independence (Defi. A.9), and satisfies interaction asymmetry (Asm. 3.5) for all equivalent generators (Defn. 4.1) 1.
Proof. We note that f having most first order interactions across slots is equivalent to having block-diagonal Hessian for every observed component. Such functions were proven to be equivalent to additive functions in Lachapelle
al. (2023). Furthermore, sufficient independence is clearly implied by sufficient nonlinearity as if all columns of the matrix (x)) are linearly independent, then blocks [D,f(2)) [De, f())) (t.))) will have non-intersecting columns spaces for all
k E [K] and will thus satisfy sufficient independence (Def. A.9. Consequently, the only thing we need to show is that sufficient nonlinearity (Def. E.5) implies interaction asymmetry (Assm. 3.5) for all equivalent generators 14.
Assume for a contradiction that sufficient nonlinearity (Def. E.5) did not imply interaction asymmetry (Assm. 3.5) for all equivalent generators (4.1) with = 1. This would imply that there exists an equivalent generator to f denoted f defined in terms of slot-wise linear function h:
f=f o h
(E.
such that f has at most first order interaction within some slot BR. In other words, leveraging Lemma A.21, there exist a (iJ E Be and a E s.t.
0 0= D.y
w W' (m( (iJ'))
(E.11)
Because W (h((z) assumed be full rank by sufficient nonlinearity (Def. E.5), then in order for this equation to hold m²(z, (i.))) must be zero. Note, however, that by construction h is defined slot-wise such that ž5 z'J map to the same slot hB.- By construction, if two 2j, 2 affect the same slot I.B then m (z, (i.j")), cannot be zero. Thus, we obtain a contradiction.
F TRANSFORMERS FOR INTERACTION REGULARIZATION
Each layer of Transformer (Vaswani et al., 2017) consist of two main components: an MLP sub- layer and an attention mechanism. Notably, in the MIP sub-layer, MLPs are applied separately to each slot or pixel query and their outputs are then concatenated. Further, additional layer normaliza- tion operations (Ba, 2016) are typically used in Transformers but are also separately applicd to each slot pixel query. Thus, the only opportunity for interaction between slots in Transformer curs through the attention mechanism. Our focus in this work is on the cross-attention mechanism, opposed to the alternative self-attention. As noted in 5, cross-attention takes the form:
K = W [&B, (k(K);
V W [&
Q = W [oolde[4))
(F.I)
exp (QToK.i..
Ad,
Sd = AA. T
is = c(za).
(F2)
ΣLE[K] exp Q.T K...
where K. K.,, V.N E Rdh, WK, WV E [d4l for query dimension dg. Further, Od E Rdd. Q. E Rd, W E dd, where do is the dimension of a pixel coordinate vector, and eb RRd + R.
Additional Details. In Eq. (F.2), we do not include the scaling factor d, for Ad,k, that is typically used as it does not affect our arguments below. We do, however, include it in our experiments. Further, when æ is an RGB image, ft will not be scalar but will instead be vector in RR since each pixel has 3 color channels. Additionally, in our experiments, multi-head attention is used. In this case, slot keys and values and pixel queries are partitioned into sub-vectors. Eqs. (F.1) and (F2) are then applied separately to each resulting sub-latent, and the resulting outputs are concatenated. When using multiple layers of cross-attention, as we do in our experiments, 3 only applied at the last layer and vectors o for a subsequent layer are defined as the vectors &a from the prior layer. Eqs. (F.1) and (F.2) is then repeated. We discuss how these additional caveats are dealt with empirically when implementing Limernet below in Appx. F.2, however, they do not affect our formal argument regarding regularizing interactions in Appx. F.1.
42

---

Published as a conference paper at ICLR 2025
F.1 JACOBIAN OF PROSS-ATTENTION MECHANISM
Our goal is to show that Ad, in equation is O, then partial derivative of Eqs. (F.1) and (F.2) w.r.t slot B i.c, Big will also be zero. This would then imply that if Ad,; is non-zero for at most one Ø5n,
slot & for every d [dz], and every 2 2uupp then slots do not interact in the of Defn. 3.2, since all such derivative products 0š. 0is for + k are zero. To this end, we are interested in S&n, 848;
computing the derivative:
0ā (x((T) D(2p....
a(aa)s
(FF3)
a(2B..),
where we here and from now on use the convention that we sum over every index that appears only on one side. To evaluate this decompose the terms
(aa); Ad, Vin Ad, Ai,.W.y
(F.44
We set Ma.- 0f (WQ) W so that
Q.1kk K (a.i(2a)..
(F.5)
This implies that
a B($B..;
(xp(Q.. = Ma exp( Q. K.
(F6)
where ò the Kronecker- Denlet (and here no summation over or is done). This implies using the product rule and the chain rule that
AAa
0 0(2B..): Ma
(F.7)
Plugging this together we get
Dia
(.)(()) O(ra);
0(2Bm)r
0(2Bm)r
0Aak 0,0())))))))))) O., A(2B.)r )r
Ad. W. W.,,
Adm W - (
A.k. Ad,
= ( + w (2 Ma.) 8.0 (a, Ms, Adm
(F8)
From this, that if Ad.m 0. then the partial derivative 88
will indeed be zero as 028m
Ad,m scales both terms in the last line of Eq. (F.8).
F.2 INTERACTION REGULARIZER
Based on Appx. F.1, we propose regularize the interaction in Transformer by minimizing the sum of all pairwise products AI,5 AI, where j # k. More specifically, we minimize the following loss:
Linteract E Σ AL,())A,,(33)
(F.9)
JE|d] k=j+1
where (z) is used indicate the input dependence of attention weights latents é. Linterect is a non-negative quantity which will be zero if and only if a matrix has at most one non-zero for each row (Brady et al., 2023).
Code to compute Linteract for a batch attention matrices can be seen in Fig. 4. note that when using multiple attention heads, we first sum the attention matrices over all heads to ensure consistent pixel assignments across different heads.
43

---

Published as a conference paper at ICLR 2025
When using multiple layers, also sum the attention matrices over each layer, for the same reason. Linteract is then computed on the result- ing attention matrix.
def L Linteract(aatt):
batch_sizee nun_slots, nun_pixels = attn, shape interaction
for i in range(nun_slotsi:
for j in range(i, nun_slots 1): interaction attn(:, i] attnl:, J+1] return interaction.neant)
Regularizing Higher Order Interactions. We note that while we motivate Cinteract regu- larizer for 1st order interactions, we do not ex- plicitly address regularizing for higher order in- teractions, i.e., for n 2 2. We conjecture there relationship between regularizing Limeract and higher order interactions but that is less direct than the 1st order case. We leave it to future work to explore these connections further, as well as alternative, computationally efficient regularizers which can more directly penalize higher order interactions.
Figure 4: PyTorch code to compute Linteract.
Computational Efficiency. We note that regularizing with Linteract adds minimal additional com- putational overhead since attention weights are already computed at each forward pass through the model, and, moreover can be easily optimized using gradient descent. This is in contrast to Brady et al. (2023) which required computing the Jacobian of the decoder f each forward pass and then optimizing it using gradient descent. This results in second-order optimization which is computa- tionally intractable for high-dimensional data such images (Brady et al., 2023).
G EXTENDED RELATED WORK
G.1 THEORY
Relationship Between Principle 3.1 and Other Principles. The principle of interaction asymme- try, "parts of the same concept have more complex interactions, than parts of different concepts" (3.1), is intuitively similar to several prior principles explored for learning concepts. For exam- ple, the prior works of Baldwin et al. (2008); Reynolds et al. (2007); Schmidhuber (1990); Zacks (e.g.. "making coffee", "driving to work"), Greff et al.
et al. (2011) on disentangling events/sub-utt (2015); Hyvarinen and Perkio (2006) on disentangling objects in an image, and Schmidhuber (1992) are all essentially based on the principle that parts same concept are more mutually predictable than parts of different concepts. Similarly, Hochreiter and Schmidhuber (1999); Jiang et al. (2022) implicitly use the idea that parts of same concept are more compressible than different concepts. Research on networks, use the idea that nodes from the same "community" interact more strongly than nodes from different communities (Fortunato and Hric, 2016), which also resembles ideas from clustering that points from the same cluster have higher mutual information than from different clus- ters (Kraskov et al., 2005). This network-based framework was applied by Schapiro al. (2013) as a model for grouping temporal events. Lastly, Greff et al. (2020) propose that objects do not interact much with their surroundings but internally have strong structure. While these different ideas intuitively similar to interaction asymmetry, they take on different formalizations. Moreover, these principles are generally used high-level heuristies for designing a learning algorithm, and their theoretical implications for disentanglement and compositional generalization are not explored.
Connection with Information Bottleneck Principle. Another notable principle for learning repre- sentations is the Information Bottleneck principle (Alemi et al., 2016; Tishby et al., 2000) which has also been applied in the context of learning disentangled representations (Mco et al., 2024). In the context of disentanglement, this principle suggest learning a representation which tries to balance a trade-off between minimizing the mutual information between a latent vector z and an observa- tion æ, and ensuring that z contains sufficient information to predict, i.e., reconstruct æ. From a theoretical standpoint, the Information Bottleneck principle differs from the principle of interaction asymmetry as defined in Asm. 3.5. Specifically, Asm. 3.5 an assumption on the generator f and does not place assumptions on the latent distribution pz. Consequently, our theory describes a setting in which disentanglement can be achieved without explicitly enforcing any additional prop- explicitlyyyl erties on Pz- We note, however, that despite this key difference, our theory does yield insights which resemble the Information Bottleneck principle. Specifically, as noted in $ 5, our theory suggest that a model uses an inferred latent dimension d: greater than the ground-truth dimension d,, then should aim to encode x using the minimal necessary latent dimension, i.e., the mutual information
44

---

Published as a conference paper at ICLR 2025
between x and unnecessary latent dimensions should be minimized, while ensuring that æ can be reconstructed from z.
On the Relationship Between Disentanglement and Compositional Generalization. A key premise motivating our theoretical study of compositional generalization is that, from a theoreti- cal perspective, disentanglement does not directly imply compositional generalization. Specifically, this would require that equality between f and f on Zupp (disentanglement Defn. 2.1) implies that these functions were also equal on all of (compositional generalization Defn. 2.2). As noted by Lachapelle et al. (2023); Wiedemer et al. (2024a), this will not be true for arbitrary functions and necessitates restrictions on the form of f. f on all of Z. While several works have provided empir- ical corroboration of this theoretical statement for concepts of objects (Wiedemer et al., 2024a) and object attributes (Montero et al., 2021; 2022b; Schott et al., 2022), prior works in disentanglement have suggested that disentanglement can some cases enable compositional generalization (Es- maeili et al., 2018: 88: Higgins et al., 2017; Mahon et al., 2023). We hypothesize that the compositional generalization abilities observed in the latter works are consequence of only leaving small num- ber of novel combinations out of the training set, such that compositional generalization becomes much easier compared to the more restricted training domains explored in (Montero et al., 2021; Wiedemer et al., 2024a). With this being said, it is possible that through hidden inductive biases in model, disentanglement can directly lead to compositional generalization, which would not be odds with our theoretical observation.
Polynomial Decoders. As noted in $ 4.2, Asm. 3.5 implies that the cross-partial derivatives of the generator f consisting of components from different slots will be finite-degree polynomials. This partially resembles the polynomial constraints f in Ahuja et al. (2023) for disentanglement. Im- portantly, however, Ahuja et al. (2023) assume that all cross-partial derivatives of f are polynomial such that the entire function is finite-degree polynomial. In contrast, Asm. 3.5 constrains the form of cross-partial derivatives acrosy slots to be polynomial, but does not constrain the form of cross- cross-partial derivatives within the same slot. In other words, Asm. 3.5 only constrains the interac- tions across slots, while Ahuja al. (2023) constrains all possible interactions. This is an important distinction since the former gives rise to much more flexible generators than the latter (see Eq. (4.2)).
G.2 METHOD AND EXPERIMENTS
VAE Losses in Object-Centric Models. Prior work in Wang et al. (2023) also apply a VAE loss to unsupervised object-centric learning setting, However, while minimize LKL directly in- ferred slots in & given by our Transformer encoder, Wang et al. (2023) minimize CKL on an interme- diate representation which is then further processed to yield é. Furthermore, the focus of Wang et al. (2023) is on scene generation an not penalizing the capacity of ž. Additionally, Kori et al. (2024) explore a loss for object-centric learning resembling a VAE loss, though their aim is to enforce a certain probabilistic structure on implied by their theoretical disentanglement result, opposed to penalize latent capacity.
Inductive Bias Through Explicit Supervision. Recently, many works have shown remarkable em- pirical success in disentangling (Kirillov et al., 2023; Ravi et al., 2024) and composing (Brooks al., 2023; Ramesh al., 2021: 2022; Ruiz et al., 2023; Saharia et al., 2022) visual concepts in images on web-scale data. These works achieve this through explicit supervision via segmentation masks natural language descriptions of each concept. opposed to constraints the generative pro- cess in Eq. (2.1). Notably, however, many species in human's 's evolutionary lineage disentangle and compose concepts in sensory data without using explicit supervision like natural language (Behrens et al., 2018; LeCun, 2022; Summerfield, 2022; Tolman, 1948). This suggest the existence of a self-supervised coding mechanism for disentanglement and compositional generalization that is still lacking in current machine learning models. The present work aims to make theoretical and empiri- cal progress towards such a mechanism.
Relation Between a Transformer Regularized with Linteract and Prior Works. Goyal et al. (2021) proposed RIMs which is Transformer-styy architecture aimed at enforcing "modular" structure. Contrary to our work, Goyal et al. (2021) do not regularize for modularity, but posit that it may emerge from "competition" induced by an attention mechanism. Similarly, Lamb et al. (2021) pro- pose an alternative Transformer architecture aimed enforcing modularity, which also tries to en- force competition using a mechanism similar to Goyal et al. (2021). More recently, Vani et al. (2024) propose a novel Transformer component which is aimed at yielding disentanglement by processing
45

---

Published as a conference paper at ICLR 2025
Transformer embedding into different slots using separate attention heads for each slot. While these works are similar to ours in that they aim to learn disentangled representations of concepts using a Transformer-style architecture, they are based on architectural changes to Transformer, whereas we use a standard cross-attention Transformer decoder and regularize it explicitly towards having a "modular" structure using Linteract-
H EXTENDED DISCUSSION
H.1 THEORETICAL ASSUMPTIONS
Non-Homogeneous Interactions. One potential limitation of our formulation of interaction asym- metry (Asm. 3.5) that the order of interaction, n, across slots, must be the same for latent vec- tors 2 E Σ and for any two slots 2B, 2,, This assumption will potentially be violated in practice. For example, for concepts of visual objects, it is likely that within each image. only a few objects, i..., slots, interact at time (e.g., see Fig. 1), such that different slots will have different orders of in- teraction within z. We conjecture that our theory can be extended handle such son-homogensous interactions, however, we leave this for future work. Furthermore, we note that despite this potential mismatch between theory and practice, our method still achieved robust object-lisentanglement on data in which the order of interaction appears to be non-homogeneous, e.g., CLEVR6.
Requirements on the Observed Dimension. We note that implication of sufficient independence for n > 0 (where n is the order of interaction across slots) is that the observed dimension dg must be greater than the latent dimension dz. Moreover, the required d2 will scale function of the number of latent slots K, the slot dimensions [B]], and the order of interaction across slots (n). For example, for functions with at most 1xt order interactions across slots, ensuring that the rank condition in sufficient independence (Defin A.9) met requires that de 2 E [B=((|B++1) +dz. Furthermore, for functions with at most 2nd order interactions, satisfying this condition (Defn. 4.2) d=(d,+1) + dg. We note that are interested in modelling high-dimensional sensory data, such as images, in which the observed dimension d, will be much greater than the latent dimension dz. Thus, for practical cases of interest, we expect these requirements dz be met.
requires dæ 2 ΣkE[K] [ + 2
6
Concepts Potentially not Captured by Interaction Asymmetry. For certain concepts, it is not obvious if interaction asymmetry will always hold. For example, consider object attributes such as the a-yy-position of an object, which can be modelled by one-dimensional slots. For such concepts, the interaction within a slot, i.e., the interaction of each latent component w.r.t. itself, should, intu- itively, be simple function. It thus not obvious if the interaction within each slot will necessarily be more complex than interactions across slots, such that f may not satisfy interaction asymme- try (Asm. 3.5). Additionally, is not immediately clear how interaction asymmetry can be applied to more abstract concepts which are not directly grounded in sensory data such as the concept of "democracy" or the concept of "function" in mathematics.
Restrictiveness of the Aligned-Connectter Assumption. Our theoretical results in $ 4 leverage the assumption that the latent space Zupp is aligned-connected Defn. A.16. To assess whether the liigned-connettednes. assumption is realistic, we believe it is helpful to look at concrete mathemat- ical examples of supports that satisfy it. For example, the whole space Rd is eligned-connected. More generally, any convex set is allgned--coneeee ball, and much more. Some aligned-come aligned-connected sets are not convex. For example, the "L-shaped" set (0, 2)2 \ [1, ( 2|2 is aligned-connected but not convex. This last example is useful to model con- crete settings where some combinations of latent factors are not observed at training time. This corresponds to the running example of Lachapelle et al. (2023) consisting of two balls moving up and down where the configurations where both balls appear in the top half of the image are never observed.
This include the hypercube [o, 1]d any closed
H.2 METHOD AND EXPERIMENTS
Self-Attention in Transformer Decoders. Our Transformer decoder in & 5 resembles the models from Jaegle et al. (2022); Sajjadi al. (2022a:b) which only rely on cross-attention mechanism. However, other works in object-centric learning leverage Transformer decoders which also include self-attention mechanism between queries at each layer (Seitzer et al., 2023; Singh et al., 2022a).
46

---

Published as a conference paper at ICLR 2025
When reconstructing individual pixels, e.g., Sprites and CLEVR6 in Š 6, applying self-attention between queries will not scale to high-dimensional images since it requires computing nxn attention weights, where is the number of pixels. However, when reconstructing image patches, as was done in our experiments on CLEVRTex, using self-attention is scalable since we have a significantly smaller number of queries, e.g., 16 x 16 on CLEVRTex. While we found that we could achieve strong disentanglement using only a cross-attention mechanism on CLEVRTex, it is possible that using self-attention could be advantageous when reconstructing image patches in even more complex settings. For such models, however, match our theory since adding self-attention between pixels will introduce additional interactions between slots. We leave it for future work investigate if our current training objective will still yield robust object-ddisentagllemeee regularized to be in line with our theory.
not immediately obvious how to regularize the decoder to
for such a model and, furthermore, if such a model can be
Trade-Offs with Slot Attention. On Sprites and CLEVR6 ($ 6) as well as CLEVRTex, we found that our regularized Transformer autoencoder achieved superior disentanglement, based on met- rics, to an unregularized variant with a Slot Attention encoder. Despite this, we emphasize that our goal is not to propose method as superior to Slot Attention-based methods. Instead, highlight that both methods offer different trade-offs. For example, training with our proposed loss (Eq. (5.3)) enables using a general Transformer encoder, thus potentially allowing our model to be applied more generally at scale compared to encoders with more explicit object-centric priors such as Slot Attention. This, however, comes at the cost of training with regularizers which require hyper- parameter selection. While our experiments did not require extensive hyperparameter tuning, is possible that certain datasets will exhibit increased sensitivity to these hyperparameters. Addition- ally, our interaction regularizer is based on decoders which only use a cross-attention mechanism. While this architecture yielded strong disentanglement in our experiments, Slot Attention encoders have been shown enable disentanglement using more expressive decoders which also use self- attention (Seitzer et al., 2023; Singh et al., 2022a).
Latent Prediction-Basse Disentanglement Metrics One potential issue with our dacobian-basee disentanglement metrics is that they may fail to measure whether multiple slots actually encode the same object. Specifically, if two slots affect the same object in pixel space, this could be due to both slots encoding the object in latent space, or it could be due to slot interactions modelled by the decoder. Definitively resolving this potential ambiguity would require measuring the information encoded in each slot directly in latent space. Along this line, prior works have considered latent prediction metrics in which the R2 score is computed from the predictions of model fit between each inferred slot and the best matching ground-trute slot (Dittadi et al., 2022; Jiang et al., 2023; Locatello et al., 2020b). While these metrics indicate if an inferred slot contains all information for a given object, they are insufficient for resolving the possible ambiguity of our current metries. This is because these metrics do not indicate if an inferred slot contains information about more than one object. This issue with latent prediction metries was pointed out by Brady et al. (2023) who aimed to address it by measuring the R2 score from an additional predictor fit to the second-best matching ground-truth slot. We found this metric to yield inconsistent results on CLEVR6, which we hypothesize was due to issues when determining the second-best matching ground-truth slot. This lead us to focus on decoder-based metrics which are more straightforward to compute. We leave it for future work to formulate a latent prediction metric which overcomes the aforementioned issues of prior works.
On Hyperparameter Selection. One potential limitation of using our regularized loss (Eq. (5.3)) in terms of scalability is that hyperparameter selection is required. In our experiments on Sprites and CLEVR6 ($ 6), extensive hyperparameter tuning was not required. Furthermore, we found the values of loss was weighted by a factor of on Sprites and 1 on CLEVR6. This indicates some level of ro- bustness of these hyperparameters across datasets which contain varying complexity of interaction. On CLEVRTex (Appx. 1), we found that these exact hyperparameter values did not transfer directly and a small amount of tuning was required to arrive at our values (.1 for all terms in the loss). We hypothesize that this is because, in our current implementation of our loss, the magnitude of the reconstruction loss scales with the dimension of the data. To this end, because the data dimension increased significantly on CLEVRTex (256 encoded image patches, each with 768 dimensions), the contribution of the reconstruction term to the loss needed to be slightly diminished. With all this being said, it is possible that more complex dataset could require more extensive hyperparameter tuning, however, we leave this for future work to investigate.
0.05, B = 0.05 to work robustly across both datasets, though the reconstruction
47

---

Published as a conference paper at ICLR 2025
Applying our Method to Other Types of Concepts. One important direction for future work is to apply our method to data consisting of different types of concepts such as object-atrribute or temporal events. For object-attributes, our same empirical framework can be applied, but with the additional caveat that the Transformer is permutation invariant, while object-atrributes do not posses the same permutation invariance as objects. To this end, methods such as adding a positional encoding to each slot, must be used to address this. Additionally, as noted by Gopalakrishnan et al. (2023); Kipf et al. (2019), the problem of disentangling temporal events in image sequences can also be modelled naturally using a slot-based framework. In this case, the "tokens" that a slot encoder, e.g., a Transformer or Slot Attention, operates on are not pixels processed by CNN, as in our current model. Instead, they would correspond to individual images in the sequence which are each mapped into representation "tokens". These tokens can then be mapped into slots by. e.g., a Transformer, and then decoded back to output space, where the queries for the Transformer decoder also would not correspond to individual pixels but instead to images in the temporal sequence.
Limitations of Ldisent- One potential issue with Lie is that for real-world data, reconstructing every pixel in an image exactly, may not be necessary and could lead to overly prioritizing tasks irrelevant information in & such as the background (Seitzer et al., 2023). It would thus be interesting to see if our theory and method could be extended to a self-supervised setting, as in Seitzer et al. (2023), in which exact invertibility is not strictly necessary. Regarding LKI. we first note that in addition to a model having inferred latent dimensionality dg equal to ground-truth dimension dz. our theory also requires that the inferred slot dimensions equals the ground-truth slot dimensions. While LKL explicitly regularizes for the former, it does not directly regularize for the latter. More specifically, LKL could, in principle, penalize latent capacity by putting information from all, e.g., objects, in one slot (assuming the slot size is large enough), opposed to distributing this information over components from different slots. Despite this, we found that this failure mode did not occur in our experiments. Another potential issue with LKt. is that it aims to enforce statistically independent latents which could lead to sub-optimal solutions if the ground-truth latents exhibit strong statistical dependencies. Lastly, regarding Linteract, a shortcoming of this regularizer is that, while it directly regularizes 1st order interactions (Appx. F. its connection to regularizing higher order interactions is not as direct. Future work should thus aim to investigate this point further both theoretically and empirically.
H.3 ENFORCING THEORETICAL CRITERIA OUT-OF-DOMAIN.
As noted in 5, enforcing (i) invertibility and (ii) most th order interactions across slots on f. out-of-domain, i.e., globally on all of 2, poses distinct practical challenges. We now discuss this in detail. To this end, we first discuss enforcing (ii) globally 2.
Restricting Interactions Globally. The easiest way to enforce that ff has most th order interac- tions across slots on Z is directly parameterize f match the form of such functions for some n (Eq. (4.2)). This is, for example, how at most 1st order interactions were enforced in Lachapelle et al. (2023), i.e., by defining f to be an additive function (Defn. E.4) on all of Z. We found for higher order interactions, parameterizing f directly to match the form of Eq. (4.2) leads to training difficulties on toy data. Moreover, even if we could easily train such a model, this explicit form would pose an overly restrictive inductive bias when scaling to more realistic data. This motivated us to consider how to regularize for (ii) opposed enforce it explicitly. The issue with this approach is that we only regularize the derivatives of f in-domain on Zuupe Yet, enforcing structure on the derivatives of f on Zuup does not imply that same structure will be enforced on all of Z. As noted in 4.2, however, by knowing the behavior of the derivatives of f on Zuup- we can infer their be- havior everywhere on Z. Thus, in principle, it should be possible to propagate the correct derivative structure learned by f locally on Zampp to all of Z. Practically, however, it is not obvious how this can be done in an effective manner. Thus, properly addressing this challenge would require further methodological and empirical contributions, which are not within the scope of the present work.
Enforcing Invertibility Globally. Additionally, even if f satisfies (ii) globally, we still must enforce (i) invertibilityy, globally. As noted in $ 5, it is not feasible to define ff such that it is an invertible function from 2 to X' by construction. This necessitated parameterizing the inverse of f with an encoder 9 which was trained to invert the decoder f via reconstruction loss. Assuming that decoder of satisfies (ii) globally, and is invertible on 2ump- it is possible to show that f will be
48

---

Published as a conference paper at ICLR 2025
invertible on all of Z and thus generalize compositionally. The issue, however, is that encoder only trained to invert f Zuup but not on unseen data from the rest of of 2. Consequently, even generalizes compositionally, an encoder 9 will not invert f out-of-domain, and can thus yield an arbitrary representations 2 on such data. This "encoder-decoder inconsisteney" was pointed out by Wiedemer et al. (2024a), which studied compositional generalization for decoders with at most O and 1st order interactions. They proposed a loss which addresses this problem by first generating out-of-domain samples using f. and then training the encoder 9 to invert on this "imagined" data. The implementation of this loss in Wiedemer et al. (2024a), deemed compositional consistency, was shown to be ineffective for images consisting of more than 2 objects, however (Wiedemer et al., 2024a). Consequently, scaling this loss, or exploring alternative losses for encoder-decoder consistency, remain open research question that require a deeper investigation to properly address.
For these reasons, the empirical aspects of this work focus on enforcing (i) and (ii) in-domain to achieve disentanglement on supp (Thm. 4.3). As highlighted above, however, our theory elucidates the core problems that need to solved empirically to also achieve compositional generalization, thus giving clear direction for future work.
I EXPERIMENTS ON CLEVRTEX
In this section, we conduct additional experiments on the CLEVRTex dataset (Karazija et al., 2021). dataset constitutes a significant step up in complexity from CLEVR6 and has been shown be highly challenging for existing object-centric models (Biza al., 2023; Karazija al., 2021). We outline our experimental setup and results below.
1.1 EXPERIMENTAL SETUP
Data. Each image in CLEVRTex consist of between 3 and 10 objects with rich textures, set against complex backgrounds (see Fig. 7 for example images). The dataset consists of 50,000 images. We use 40,000 images for training and 5,000 for validation and testing, respectively.
Models. We train models on this data. The first model is our regularized Transformer autoencoder from 5, for which we weight each term in the loss in Eq. (5.3) by hyperparameter value of .1. The second model is an unregularized Transformer autoencoder, and the third model is an unreg- ularized autoencoder which uses a Slot Attention encoder with both a Transformer and slot-wise MLP decoder. We train all models using the same setup as in Š 6, however, instead of reconstructing the original images, we reconstruct a representation of each image given by Vision Transformer (ViT) (Dosovitskiy et al., 2021), which pretrained using the DINO method (Caron et al., 2021). This approach, deemed DINOSAUR (Seitzer et al., 2023), was shown to help object-centric mod- els scale datasets with increased visual complexity. We thus replace the CNN backbone used in our experiments on Sprites and CLEVR6 with a pretrained ViT which operates on 8 x 8 patches of the original images. These patches mapped features which then processed by either Transformer or Slot Attention encoder. For all models, we use 11 slots with a slot dimension of 64.
Training and Evaluation Details We train all models across 3 random seeds using batches of 32. In all cases, we use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 5 x 10 which we warm-up for the first 10,000 training iterations and then decay by factor of 10 throughout training. We also warm-up the value of a for the first 25,000 training iterations. We report the J-ARI and JIS for each model after training for 300,000 iterations. To compute these scores, we bilinearly interpolate our normalized Jacobian maps to match the shape of the original image, since we are reconstructing image patches. When computing J-ARI and JIS for the siot MLP decoder, we rely on the alpha-mash maaha of the decoder opposed to its Jacobian due to computational issues when computing the Jacobian for this model.
1.2 RESULTS
We report our results in Tab. 2. As we can see, similar to on Sprites and CLEVR6, our regular- ized Transformer achieves strong object-dissntangleemnn outperforming both unregularized base- line methods in terms of J-ARI. Our model also achieve superior JIS compared to all baseline models with the exception being the slot-wise MIP decoder. This is not unexpected, however, as this decoder explicitly constrains interactions in the same way as the Spatial Broadcast Decdoer used
49

---

Published as a conference paper at ICLR 2025
Table Empirical Results, We show the mean std. dev. J-ARI and JIS (in %6) over seeds for different
choices of encnders and weights of the loss terms Fq. (5.3) on CLEVRTex.
Encoder
Decoder
Loss
J-ART (t)
JIS (t)
Transformer
Transformer
0, 0
81.4 3.7
50.9 + 2.8
Slot Attention
Transformer
= 0,3
94.2 0.2
54.4 + 0.3
Slot Attention
Slot-wise MLP
0, 0
92.8 + 0.2
84.3 + 0.4
Transformer
Transformer
a -01,, B = 0.1 (Ours)
95.9 + 0.06
65.4 + 0.6
in other experiments. We also visually corroborate these results by plotting normalized slot-wise cobians for each model which can be seen in Fig. 7.
J EXPERIMENTAL DETAILS
J.1 DATA, MODEL, AND TRAINING DETAILS
Data. The Sprites dataset used in & 6 was generated using the Spriteworld renderer (Watters et al., 201 9a) and consist of 100,000 images of size x 64 3 each with between and objects. The CLEVR6 dataset (Johnson et al., 2017; Locatello et al., 2020b) consist of 53,483 images of size 128 128 x 3 each with between 2 and 6 objects. For Sprites, we use 5,000 images for validation, 5,000 for testing, and the rest for training, while for CLEVR6, we use 2,000 images for validation and 2,000 for testing.
Encoders. All models use encoders which first process images using the same CNN of Locatello al. (2020b). When using Transformer encoder, these CNN features fed to 5 layer Trans- former which uses both self- and cross-attentic with 4 attention heads. When using a Slot Attention encoder, we use 3 Slot Attention iterations, and use the improved implicit differentiation proposed in Chang et al. (2022). Both the Transformer and Slot Attention encoders use learned query vectors opposed to randomly sample queries, which was shown by Biza al. (2023) to yield improved performance for Slot Attention. On Sprites, all models use 5 slots, each with 32 dimensions, while on CLEVR6, all models use 7 slots, each with 64 dimensions. When using a VAE loss, this slot dimension doubles since we must model the mean and variance each latent dimension.
Decoders. When using Spatial Broadcast decoder (Watters et al., 2019b), we use the same archi- tecture as (Locatello al., 2020b) across all experiments, using a channel dimension of 32 for both datasets. When using a Transformer decoder, we first upscale slots to 516 dimensions by processing them separately using 2 layer MLP, with a hidden dimension of 2064. We then apply 2 layer cross-attention Transformer to these features which uses 12 attention heads. To obtain the vectors or in Eq. (5.1), apply 2D positional encoding mapped by 2 layer MLP with a hidden dimension of 360 to yield O₁, which has dimension 180. The function & in Eq. (5.2) is implemented by 3 layer MLP with hidden dimension of 1 180, which outputs a 3 dimensional pixel ét for each pixel I. We additionally note that this architecture does not rely auto-regressive masking in Singh al. (2022a).
each pixel coordinate. This vector is then
Training Details. We train all models on Spriteworld across 3 random seeds using batches of 64 for 500,000 iterations. For CLEVR6, we use batches of 32 and train for 400,000 iterations. In all cases, we use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 5 10-4 which we warm- up for the first 30,000 training iterations and then decay by a factor of 10 throughout training. When training with [LL and aLiiteracts we use hyperparameter weights of 0.05, which well across both datasets. We found much larger values could lead to more training instability and, in some cases, insufficient optimization of Lree while smaller values often did not lead sufficient optimization of the regularizers. We warm-up the value of a for the first 30,000 training iterations.
found work
Additionally, when training with B, we drop the value of the learning rate after 30,000 training iterations to 1 x 10-4, which improved training stability. Lastly, on Sprites, we weight Liee by factor of 5, when training with or 3.
50

---

Published as a conference paper at ICLR 2025
J.2 METRICS AND EVALUATION
Computing ARI with Attention Scores. To compute the Adjusted Rand Index (ARI), each pixel must first be assigned to unique model slot. To this end, prior works typically choose the slot with the largest attention score from either Slot Attention or the alpha mask of a Spatial Broadcast decoder (Locatello et al., 2020b; Seitzer et al., 2023). This approach can be problematic since the attention scores used are model--epennent, making a direct comparison of ARI across models challenging. Further, the relationship between attention scores and the pixels encoded in model slot is somewhat indirect. As noted $ 6, we consider an alternative and compute the ARI using the Jacobian of a decoder (J-ART). Specifically, we assign a pixel to the slot with the largest L1 norm for the slot-wise Jacobian DR, (z). This can be done for any autoencoder and provides a more principled metric for object disentanglement since a decoder's Jacobian directly describes the pixels each slot encodes (assuming f. 9 invert each other).
Evaluation. We select models for testing which had the highest average values for J-ARI and JIS (each of which take values from 1) on the validation set. These models were then evaluated on the test set yielding the scores reported in Tab. 1.
J.3 ADDITIONAL FIGURES
In this subsection, we include additional experimental figures. Fig. 5, we compare the value of Linteraer throughout training for model with a Transformer encoder and decoder, trained using a our regularized loss Eq. (5.3), the VAE loss and a standard autoencoder loss on both Sprites and CLEVR6. We plot values over 3 random seeds; the shaded regions in the plots indicate one standard deviation from the mean. We find on Sprites (A) and CLEVR6 (B) that the VAE loss achieves much lower Linteract than the unregularized model. This provides possible explanation for the solid object disentanglement often achieved by the VAE loss in Tab. 1. We also observe, however, that using a > 0 leads to much lower values for Linteract compared to the implicit regularization from the VAE loss.
In Fig. 6, we compare slot-wise Jacobians for our model versus baseline models across both Sprites (A) and CLEVR6 (B). To create these plots, normalize the partial derivatives across slots such that they only take values between and The colors associated with partial derivative values can be interpreted using the color bar at the bottom of (A). We only compute partial derivatives on the foreground pixels and set the derivatives of background pixels w.r. each slot to 0. We see that when regularizing interactions via our model, slots rarely affect the same pixels (i.e., interact) unnecessarily, while for unregularized models, multiple slots often affect the same pixels even when no interactions should occur, e.g., for images in Sprites (A).
In Fig. we compare slot Jacobians on CLEVRTex was done in Fig. 6 and also observe here that the regularized Transformer achieves cleaner object decompositions compared to baseline models.
In Fig. 8, we compare decoder attention maps w.r.t. each slot for our model versus baseline models from 6,which also use Transformer decoder. These maps, which indicate the slots each pixel attends to, are plotted for both Sprites (A) and CLEVR6 (B). We compute these values by taking the mean attention weight over decoder layers. Similar Fig. 6, we see that, in model, pixels rarely unnecessarily attend to multiple slots. On the other hand, for unregularized models, pixels often attend to multiple slots in cases where no interactions between slots should occur.
51

---

Published as a conference paper at ICLR 2025
A
B
14
440
P-06-6
2-63
2-0056 0655
5-03--18
Tramin Beration
Traiiing Barabion
Figure 5: Analysis of Linteract when using a VAE loss. We plot Limeraet for the first 400,000 training iterations for Transformer autoencoder trained without regularization (a = 0, B = 0), with VAE loss which does not explicitly optimize Lintract (a = 0, B = 0.05), and with the loss in Eq. (5.3)
which regularizes both losses (a=0.05, 300.55.
52

---

Published as a conference paper at ICLR 2025
A
B
Senn
Siee
Sh12
Sior
Sierd Ster
Sh16
Sh117
Coneeeee Comtena Peeee and
Original Imagges
Ceseeaaee secleee Uringg berodd
See AERee Pee Preeitaae
Co Co Soog copeaa Dne
Figure 6: Normalized Slot-wise Jacobians. We plot the Jacobians w.r.t each slot (columns) for random test images (rows) from (A) Sprites and (B) CLEVR6 for our regularized Transformer model and the baseline models used in our experiments in 6.
53

---

Published as a conference paper at ICLR 2025
Slot 2
Slot 3
Siot
Transtommrr Enc/Den Reg ulbrited (Dung)
Original Images
Transformer Enc/Dec Unreqularized
Slot Actention Ent. Tra Dec
0.0
0.2
0.4
0.6
0.8
1.0
Figure 7: Normalized Slot-wise Jacobians (CLEVRTex). We plot the Jacobians w.r.t. each slot (columns) for random test images (rows) from CLEVRTex for our regularized Transformer model and the baseline models used in our experiments in Appx.
54

---

Published as a conference paper at ICLR 2025
A
B
Set1
Skat
Sot-5
Sioti
Stot.
Slat 5
Skot
Original
poo
200 FAREAA auguory Aaa
Jar Cot.en Coee Frc. Tra Trandfanta Dec
100 RoL SALlRpeR DM0 SE 396
Figure 8: Slot-wise Transformer Decoder Attention Maps. We plot decoder attention maps w.r.t. each slot (columns) for random test images (rows) from (A) Sprites and (B) CLEVR6 for our reg- ularized Transformer decoder and the baseline models in $ which also use a Transformer decoder.
55


Reviews:
[{"review_id": "fYVIWMRHb4", "reviewer": "Reviewer_K8sM", "summary": "This paper investigates the challenge of learning disentangled and composable representations, crucial for robust generalization in machine learning. It introduces the principle of \"interaction asymmetry,\" positing that parts of the same concept interact more complexly than parts of different concepts. This principle is formalized through block-diagonality conditions on higher-order derivatives of the generator function mapping concepts to observed data, with the order of the derivative reflecting the \"complexity\" of interaction. The authors theoretically demonstrate that interaction asymmetry enables both disentanglement and compositional generalization, unifying and extending prior work focused specifically on visual objects. They show that existing results for object-centric learning emerge as special cases of their framework, corresponding to lower-order interactions (0th and 1st order). The theory suggests practical criteria for disentanglement: enforcing invertibility with minimal latent dimensions and penalizing interactions across concept slots during decoding. The authors propose an attention-regularized Transformer-based VAE, leveraging the VAE framework for capacity control and incorporating a novel regularizer on the decoder's attention weights to minimize cross-slot interactions. Experiments on synthetic image datasets with objects demonstrate the effectiveness of this approach, showing that the proposed model learns disentangled object representations, achieving performance comparable to existing methods that rely on more explicit object-centric priors.", "questions": "Please see above comments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the principle of interaction asymmetry, offers a novel and insightful perspective on disentanglement and compositionality. While previous works have explored related ideas, the formalization through higher-order derivatives and the connection to the complexity of interaction is original and provides a unifying framework for prior results in object-centric learning. The proposed regularizer on the Transformer's attention weights is a creative application to a scalable architecture.\n- The theoretical results are rigorously presented and appear sound. The connection to existing works is carefully established, demonstrating the generality of the proposed framework. The experiments, while conducted on synthetic datasets, offer convincing evidence for the effectiveness of the proposed method and regularizer.\n- The paper is generally well-written and easy to follow. The authors clearly motivate their work, provide intuitive explanations of the core ideas, and present the theoretical results in a structured and accessible manner. The figures effectively illustrate the key concepts and experimental results.", "weaknesses": "- the aligned-connectedness condition on $Z_{supp}$ seems to be restrictive and its practical implications are not fully explored.\n- The proposed regularizer, while efficient, only approximately enforces minimal interactions. Analyzing the gap between this approximation and the theoretical ideal, and exploring alternative regularization strategies that more directly address higher-order interactions, could further enhance the method's effectiveness.\n- The experiments primarily focus on synthetic datasets and object-centric learning. While this provides a controlled setting for evaluating disentanglement, exploring the effectiveness of the proposed method on more complex, real-world datasets and diverse concept types (e.g., attributes, events) is crucial for demonstrating its broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wyCTt6KyeQ", "reviewer": "Reviewer_qqbq", "summary": "The paper titled \"Interaction Asymmetry: A General Principle for Learning Composable Abstractions\" presents the principle of \"interaction asymmetry\" to explain why some latent structures enable better generalization and disentanglement in representation learning. The core idea is that parts of the same concept interact more complexly compared to parts of different concepts, and the authors formalize this using block diagonality in higher-order derivatives of a generator function.\n\nThe main contributions of the paper include:\n\nTheory: A formal proof that the principle of interaction asymmetry enables both disentanglement and compositional generalization. The authors extend previous results to more general generator functions, generalizing beyond earlier specific cases.\nMethod: An autoencoder-based approach that uses a Transformer architecture with an attention-based regularizer to limit interactions across slots, following the interaction asymmetry principle.\nEmpirical Evaluation: Demonstration on synthetic datasets that the proposed model can achieve disentanglement comparable to other models while allowing for more flexible interactions among the latent representations.\nThe paper unifies prior work by presenting a more general theoretical framework and proposes a practical implementation that achieves strong results in object disentanglement tasks.", "questions": "1. Are there plans to extend the empirical evaluation to include more challenging, real-world datasets? This would help establish the practical utility of the model in diverse settings.\n\n2. Given the theoretical assumptions made in the paper, could you discuss how they might be relaxed to make the model more applicable to real-world scenarios? Are there any potential modifications to the theory that could broaden its applicability?\n\n3. How does the interaction asymmetry principle relate to the Information Bottleneck theory? A discussion on this connection could provide additional insights into the theoretical foundations of the proposed approach and highlight its relevance to existing literature on disentanglement.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Originality**:  \nThe paper introduces the principle of **interaction asymmetry** to advance disentanglement and compositional generalization, providing a unified theoretical foundation that applies to broader conditions. The use of block diagonality in higher-order derivatives is creative and extends representation learning. The combination of theoretical insights with a practical Transformer-based VAE implementation is also notable.\n\n**Quality**:  \nThe theoretical contributions are well-developed, extending previous findings to more complex settings. The regularized Transformer-based VAE is supported by empirical evidence, showing comparable performance to state-of-the-art methods. The use of multiple datasets (Sprites and CLEVR6) allows for a thorough assessment of the model's capacity for disentanglement.\n\n**Clarity**:  \nThe paper is well-written and accessible. The authors clearly define theoretical constructs like **interaction asymmetry** and **nth order interaction**, using visual aids to illustrate complex concepts effectively. The logical structure and provided diagrams help convey the ideas intuitively.\n\n**Significance**:  \nThe principle of interaction asymmetry offers a general framework for **disentanglement** and **compositional generalization**, extending the applicability of models to more complex generative processes. The flexible and scalable Transformer-based model positions this work as a significant step toward learning useful representations for real-world tasks.", "weaknesses": "1. **Limited Empirical Evaluation**: The empirical evaluation, while thorough for the presented datasets, is somewhat limited in scope. The paper only evaluates on synthetic datasets (Sprites and CLEVR6), which may not fully demonstrate the model's potential for real-world applications. Including experiments on more challenging, real-world datasets would strengthen the evidence for the model's practical utility and scalability.\n\n2. **Theoretical Assumptions**: The theoretical results rely on several assumptions (e.g., sufficient independence, specific conditions on the latent distribution). These assumptions may be restrictive in practical scenarios, and their implications are not fully explored in the context of real-world data. It would be beneficial to discuss how these assumptions could impact the applicability of the model and whether there are potential relaxations that could make the theory more practical.\n\n3. **Connection with Information Bottleneck**: The paper does not explicitly connect its disentanglement framework with the **Information Bottleneck** principle, which has been discussed in several works as an effective approach to disentanglement. Incorporating a discussion on how interaction asymmetry relates to information bottleneck theory could strengthen the theoretical underpinnings and offer insights into how limiting mutual information might aid in disentangling the latent representations. Including references to relevant literature, such as Tishby et al. (2000), Alemi et al. (2017)  and Meo et al. (2024), could provide additional context and depth to the discussion.\n\nhttps://arxiv.org/abs/physics/0004057\nhttps://arxiv.org/abs/1612.00410\nhttps://openreview.net/pdf?id=ptXo0epLQo", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6kzxm3XPUZ", "reviewer": "Reviewer_c6zh", "summary": "This paper introduces a novel principle for learning composable representations, termed “interaction asymmetry.”, stating that interactions between distinct concepts are less complex than those within sub-parts of each concept. By formalizing these interactions as high-order derivatives of each concept to output images, the authors theoretically demonstrate that interaction asymmetry facilitates disentanglement and compositional generalization (though the latter is not empirically validated). Notably, their theoretical findings encompass previous approaches for achieving compositional generalization as specific cases (e.g., assumptions of at most 0th or 1st order interactions) and extend to accommodate more complex scenarios with higher-order interactions.To implement these theoretical insights, the authors propose regularizing cross-attention layers in standard transformer architectures. Empirical evaluations on two synthetic datasets show that the proposed method effectively captures distinct objects. The main contributions of the paper are as follows:\n- The introduction of interaction asymmetry as a general principle for compositional representation learning, supported by a theoretical framework that links interaction asymmetry to disentanglement and compositional generalization.\n- A practical method to enforce interaction asymmetry within transformers by regularizing cross-attention layers, with empirical results on synthetic datasets that demonstrate the method’s ability to capture distinct objects.", "questions": "- In L519–521, can the trade-off between interaction and model expressivity be effectively controlled through the parameter $\\alpha$? For example, when increasing $\\alpha$, does it lead to increased J-ARI but prevent reflections in mirrored objects in output images?\n- As the Sprite dataset inherently exhibits 0th order interaction between objects, wouldn’t a spatial broadcast decoder (though the final alpha-blending layer does permit some interaction) serve as an architectural bias enforcing 0th order interaction? If so, should this setup represent the upper bound performance? Why does the proposed method outperform this approach?\n- Although not a critical question, why did the authors choose to use learned queries instead of randomly sampled queries, which are typically used in Slot Attention?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow, with a well-motivated presentation that effectively derives theorems and implementations.\n- The theoretical foundation of the paper is solid and sound, offering meaningful insights that contribute to our understanding of compositional representation learning.\n- It is interesting and valuable that the proposed approach unifies existing work while also extending to more complex scenarios, e.g., including high-order interactions between concepts, highlighting its versatility and potential impact.", "weaknesses": "- Although the theoretical foundation is solid, the experimental support is relatively limited. The empirical results are based solely on the simple Sprite and CLEVR6 datasets, raising questions about the generalizability of the findings to more complex scenarios.\n- Theorem 4.3 states that achieving disentanglement requires ensuring at most $n$-th order interaction between slots. \nWhile theoretically sound, implementing this in practice as minimizing interaction without knowing the ground truth value of $n$, could be a limitation. For instance, each dataset (such as Sprite or CLEVR6) may have interactions of different orders, implying that the regularizer's weight to control interaction strength should be carefully adjusted for each case. If the Sprite dataset exhibits 0th order interaction, a stronger regularizer may be necessary compared to that in CLEVR6. Additional interpretation or obervations from the authors on this aspect would make the empirical support for their theory more comprehensive. \n- To implement Theorem 4.3 in a feasible way, the authors propose to minimize interactions between distinct slots through an attention regularizer in the transformer. However, it is unclear if this regularizer provides any advantage over the existing spatial exclusiveness inductive bias in slot attention encoders, which already minimizes interaction through slot-wise competition. Clarifying the unique benefits of the proposed regularizer compared to slot attention’s inherent mechanism would enhance the paper's contributions.\n- Additionally, the authors claim that their generalized transformer structure with regularization is more flexible than slot attention-based methods (e.g., slot attention encoder + transformer decoder in L460–463). However, this claim is less convincing due to the following reasons: (1) To enforce invertibility, an additional KL term is required in the objective function, as shown in Table 1 (the value suggests that this term is crucial). \nTherefore, the proposed approach requires two additional terms (KL term and the regularizer term) and it may introduces additional burden on finding proper hyper-parameters. It is hard to tell if these inductive biases are favorable than architectural bias in slot attention methods.\n(2) If my understanding is correct, the transformer decoder lacks a self-attention layer, using only cross-attention to avoid slot interactions. This specific design differs from the conventional transformer architecture, potentially limiting scalability. In contrast, slot attention-based methods, they can resume off-the-shelf models such as off-the-shelf encoder + slot attention encoder + off-the-shelf decoder (as in DINOSAUR [1], SLATE [2], LSD [3]), which might be more flexible in real-world applications.\n- Finally, the proposed J-ARI metric may be misleading. The Jacobian of the decoder identifies regions most affected by each slot, but due to slot interactions, it may fail to segment objects accurately. For example, in Figure 3(B), the metallic sphere reflects the blue cuboid, causing the normalized slot-wise Jacobian to activate for the blue cuboid in the reflection area. However, this does not imply that the slot for the blue cuboid genuinely contains information about the metallic sphere. To assess whether composable abstractions are effectively captured, a downstream task like property prediction, as in LSD, may be necessary.\n\n[1] Seitzer et al., bridging the gap to real-world object- centric learning, in ICLR 23. \n\n[2] Singh et al., Illiterate dall-e learns to compose, in ICLR 22. \n\n[3] Jiang et al., Object-centric slot diffusion, in NeurIPS 23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lNZTsRl9Zc", "reviewer": "Reviewer_A7GP", "summary": "This paper introduces the interaction asymmetry principle as a way to measure disentanglement and compositional generalisation, based on decomposing the latent space into blocks, and requiring non-zero partial derivatives between elements of the same block, but not between those of different blocks. The authors prove, in the appendix, that functions meeting this condition are disentangled, and discuss how this framework generalises some existing characterisations of disentanglement. Then they describe how this idea can be implemented as an additional loss term in a vision transformer VAE, and present results quantifying the degree of disentanglement using two metrics they devise, and compared to a slot attention decoder and a spatial broadcast decoder.", "questions": "What does your work imply about the debate in the literature as to whether disentanglement and compositional generalisation are related?\n\nHow does  ‘sufficiently predictable’ imply compositional generalisation?\n\nWhat are the exact values of $\\alpha$ and $\\beta$ in Table 1?\n\nAlso, a general point: This formulation of the interaction principle is binary, either the required derivatives are zero everywhere or they are not. Disentanglement, on the other hand, is normally quantified to a matter of degree, giving a particular model on a particular dataset a score in [0,1]. How easy would it be to extend your formulation to provide a score in [0,1], rather than {0,1}?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The problem of measuring disentanglement and compositional generalisation is interesting.\n\nThe intro and high-level background is generally well-motivated.\n\nThe general principle of interaction asymmetry is, to my knowledge, creative in that it is unlike previous approaches to this problem. It is convincing and put to good use.\n\nEq 4.2 is a nice expression of how a function can, in some sense be decomposed into a disentangled and entangled portion.", "weaknesses": "The discussion on the relationship between disentanglement and compositional generalisation could be improved by engaging with further existing work. While Montero et al (2020, 2022) claim to find no relationship between disentanglement and compositional generalisation, Higgins et al. (2016) and Esmaili et al. (2019) claim that their disentangled methods also enabled compositional generalisation, and Mahon (2024) claims to find an empirical correlation between disentanglement and compositional generalisation across multiple models and datasets. In fact, the position of Montero seems somewhat incongruent with your argument that they both share an underlying mechanism of interaction asymmetry.\n\nI am slightly confused by some terminology: does ‘latent’ mean latent dimension or latent vector? e.g. on 197 \"group of latents from the same or different slots\" sounds like the former, but on line 409, it's in bold, suggesting a vector; does 'block' mean the same thing as 'slot'?\n\nIn Section 4.2, I don’t follow why $f$ and $\\hat{f} \\circ h$ being ‘sufficiently predictable’, in having all partial derivatives be fixed-degree polynomials, means that they are both equal on $Z_{cpe}$? It seems like there are three notions here, the interaction asymmetry condition, 'sufficiently predictable' and compositional generalisation.\n\nThe description of ARI on lines 478-9 is not the standard one as used e.g. in clustering (Vinh et al; 2009). Please state what the measure is exactly.\n\nComparing the different ablation settings in Table 1, it looks like having $\\beta > 0$ makes a bigger difference than $\\alpha > 0$, did you try a range of values for $\\beta$? it could be that if you have the correct value of $\\beta$, then $alpha>0$ makes no difference. It’s a bit odd to just report the value of these scaling parameters as > 0, it would be more insightful to report the exact values used.\n\nSome information from Appendix I should be in the main paper, at least the number of dimensions in each slot.\n\n**Refs**\n\nMontero et al. (2020) \"On the Role of Disentanglement in Generalisation\".\n\nMontero et al. (2022) \"Lost in Latent Space: Examining failures of disentangled models at combinatorial generalisation\"\n\nHiggins et al. (2016) \"beta-vae: Learning basic visual concepts with a constrained variational framework.\"\n\nEsmaeli et al. (2019) \"Structured Disentangled Representations\"\n\nMahon et al. (2024) \"Correcting Flaws in Common Disentanglement Metrics\"\n\nVinh et al (2009). \"Information Theoretic Measures for Clustering Comparison: Is a Correction for Chance Necessary?\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)