Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation
Authors: Sebastien Lachapelle
Divyat Mahajan
Ioannis Mitliagkas
Simon Lacoste-Julien
Venue: NeurIPS 2023 oral (2023)
Keywords: identifiability
nonlinear ICA
causal representation learning
disentanglement
object-centric representation learning
extrapolation
TL;DR: We show that additive decoders have an identifiable representation and allow to generate novel images never seen during training, an ability we refer to as Cartesian-product extrapolation.

Abstract: We tackle the problems of latent variables identification and "out-of-support'' image generation in representation learning. We show that both are possible for a class of decoders that we call additive, which are reminiscent of decoders used for object-centric representation learning (OCRL) and well suited for images that can be decomposed as a sum of object-specific images. We provide conditions under which exactly solving the reconstruction problem using an additive decoder is guaranteed to identify the blocks of latent variables up to permutation and block-wise invertible transformations. This guarantee relies only on very weak assumptions about the distribution of the latent factors, which might present statistical dependencies and have an almost arbitrarily shaped support. Our result provides a new setting where nonlinear independent component analysis (ICA) is possible and adds to our theoretical understanding of OCRL methods. We also show theoretically that additive decoders can generate novel images by recombining observed factors of variations in novel ways, an ability we refer to as Cartesian-product extrapolation. We show empirically that additivity is crucial for both identifiability and extrapolation on simulated data.

Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation
Sébastien Lachapelle
Divyat Mahajan'
Ioannis Mitliagkas'
Simon Lacoste-Julient Julient 1
Mila & DIRO, Université de Montréal
Samsung SAIT AI Lab, Montreal
Abstract
We tackle the problems of latent variables identification and "out-of-support" im- age generation representation learning. We show that both are possible for a class of decoders that we call additive, which reminiscent of decoders used for object-centric representation learning (OCRL) and well suited for images that can be decomposed sum of object-specific images. We provide conditions under which exactly solving the reconstruction problem using an additive decoder is guar- anteed identify the blocks of latent variables up permutation and block-wise invertible transformations. This guarantee relies only on very weak assumptions about the distribution of the latent factors, which might present statistical depen- dencies and have an almost arbitrarily shaped support. Our result provides a new setting where nonlinear independent component analysis (ICA) is possible and adds to our theoretical understanding of OCRL methods. We also show theoreti- cally that additive decoders can generate novel images by recombining observed factors of variations in novel ways, an ability we refer to as peatesian-produee extrapolation. We show empirically that additivity is crucial for both identifiability and extrapolation on simulated data.
Introduction
The integration of connectionist and symbolic approaches to artificial intelligence has been proposed as a solution to the lack of robustness, transferability. systematic generalization and interpretability of current deep learning algorithms [53, 4, 13, 25, 21] with justifications rooted in cognitive sciences [20, 28, 43] and causality [57, 63). However, the problem of extracting meaningful symbols grounded in low-level observations, e.g. images, is still open. This problem is sometime referred to as disentanglement [4, 48) or causal representation learning [63]. The question of identifiability in representation learning, which originated in works nonlinear independent component analysis (ICA) [65, 31, 33, 36), has been the focus of many recent efforts [49, 66, 26, 47, 3, 9, 41]. The mathematical results of these works provide rigorous explanations for when and why symbolic representations can be extracted from low-level observations. In a similar spirit, Object-centric representation learning (OCRL) aims learn representation different objects are encoded separately [19, 22, 11, 24, 18, 51, 14]. These approaches have shown impressive results empirically, but the exact reason why they can perform this form of segmentation without any supervision is poorly understood.
which the information about
Equal contribution. Canada CIFAR Al Chair.
Correspondence to: (lachaseb, divyat..ahaajan) la quebec
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(P
f P1)
(Pa
f
2B;
2Bg
2
B = [ 2). (3. 4]}
Zn.
20.
Figure Left: Additive decoders model the additive structure of scenes composed of multiple objects. Right: Additive decoders allow to generate novel images never seen during training via Cartesian-product extrapolation (Corollary 3). Purple regions correspond to latents/observations seen during training. The blue regions correspond to the crnttsiian-rroduee extension. The middle set is the manifold of images of balls. In this example, the learner never saw both balls high, but these can be generated nevertheless thanks to the additive nature of the scene. Details in Section 3.2.
1.1 Contributions
Our first contribution is an analysis of the identifiability of class of decoders we call additive (Definition 1). Essentially, a decoder ((z acting on a latent vector E Rds to produce an observation a said to be additive if it can be written as f(z) = =EBES ( (zn) where B is a partition of {I..... dz (zB) are "block-specific" decoders and the zB non-overlapping subvectors of z. This class decoder is particularly well suited for images æ that can be expressed as a sum of images corresponding
different objects (left of Figure 1). Unsurprisingly, this class of decoder bears similarity with the decoding architectures used OCRL (Section 2), which already showed important successes at disentangling objects without any supervision. Our identifiability results provide conditions under which exactly solving the reconstruction problem with an additive decoder identifies the latent blocks 2g up to permutation and block-wise transformations (Theorems & 2). We believe these results will be of interest to both the OCRL community, as they partly explain the empirical success of these approaches, and to the nonlinear ICA and disentanglement community, provides an important special case where identifiability holds. This result relies on the block-specific decoders being "sufficiently nonlinear" (Assumption 2) and requires only very weak assumptions on the distribution of the ground-truth latent factors of variations. In particular, these factors can be statistically dependent and their support can be (almost) arbitrary.
Our second contribution to show theoretically that additive decoders can generate images never seen during training by recombining observed factors of variations in novel ways (Corollary 3). To describe this ability, we coin the term "Cartesian-pre -proucctt extrapolation" (right of Figure 1). We believe the type of identifiability analysis laid out in this work to understand "futtoof-uupoor" generation is novel and could be applied to other function classes or learning algorithms such as DALLE-2 [59] and Stable Diffusion [61] to understand their apparent creativity and hopefully improve
Both latent variables identification and Cartesian-produce extrapolation are validated experimentally on simulated data (Section 4). More specifically, observe that additivity is crucial for both by comparing against a non- adn-dditivee decoder which fails to disentangle and extrapolate.
Scalars are denoted in lower-case and vectors in lower-case bold, e.g. IR and x E TR.. We maintain an analogous notation for valued and vector-valued functions, e.g. f and f. The ith coordinate of the vector æ is denoted by m; The set containing the first n integers excluding 0 is entries æ; for E S. Given function f (xx)) E T with input æs, the derivative of f w.r.t. is denoted by D&J (a)) E I and the second derivative w.r.t. æ; and By is D2 (xx) Rmm. See Table 2 in appendix for more.
Notation.
denoted by [n). Given a subset of indices S C [n]. 2g denotes the subvector consisting
Code: Our code repository can be found at this link.
2 Background & Literature review
Identifiability of latent variable models. The problem of latent variables identification can be best explained with simple example. Suppose observations a: E TRdz are generated i.i.d. by first sampling latent vector z e dd= from distribution Pz and feeding it into decoder function f Rd - 12

---

L.e. x = x= f()). By choosing an alternative model defined as f= f o v and == (z) where v 1Rd, + Rd is some bijective transformation, is easy to see that the distributions of ē = f()) and x are the same since f(=) = f for(u (x)) = f()). only the distribution over æ, it is impossible to distinguish between the two models (f. =) and (f. $). This is problematic when one wants to discover interpretable factors of variations since and 2 could be drastically different. There are essentially two strategies to go around this problem: (i) restricting the hypothesis class of decoders f [65. 26, 44, 54, 9, 73), and/or (ii) restricting/adding structure to the distribution of & [33, 50, 42, 47]. By doing so, the hope is that the only bijective mappings keeping f and 2 into their respective hypothesis classes will be trivial indeterminacies such as permutations and element-wise rescalings. Our contribution, which to restrict the decoder function f to be additive (Definition 1), falls into the first category. Other restricted function classes for f proposed in the literature include post-nonlinear mixtures [65], local isometries [16, 15, 29), conformal and orthogonal maps [26, 60, 9] as well as various restrictions on the sparsity of f [54, 73, 7, 71J. Methods that do not restrict the decoder must instead restrict/structure the distribution of the latent factors by assuming, e.g., sparse temporal dependencies [31, 38, 42, 40), conditionally independent latent variables given an observed auxiliary variable [33, 36), that interventions targeting the latent factors observed [42, 47, 46, 8, 2, 3, 64, 10, 67, 72, 34], that the support of the latents is a Cartesian- product [68, 62]. In contrast, our result makes very mild assumptions about the distribution of the latent factors, which can present statistical dependencies, have almost arbitrarily shaped support and does not require any interventions. Additionally, none of these works provide extrapolation guarantees do in Section 3.2.
The problem of identifiability is that, given
Relation nonlinear ICA. Hyvärinen and Pajunen [32] showed that the standard nonlinear ICA problem where the decoder f is nonlinear and the latent factors z; are statistically independent is unidentifiable. This motivated various extensions of nonlinear ICA where more structure on the factors is assumed [30, 31, 33, 36, 37, 27]. Our approach departs from the standard nonlinear ICA problem along three axes: (i) we restrict the mixing function to be additive, (ii) the factors do not have to be necessarily independent, and (iii) we can identify only the blocks zB opposed to each z; individually up to element-wins wise transformations, unless = {{1), {ds}} (see Section 3.1).
Object-centric representation learning (OCRL). Lin et al. [45] classified OCRL methods in two categories: scene mixture models [22, 23, 24, 51] & spatial-attentinn models [19, 12, 11, 18]. Additive decoders can be seen an approximation to the decoding architectures used in the former category, which typically consist of an object-specific decoder ((bb) acting on object-specific latent blocks zn and "mixed" together via a masking mechanism m (B)) which selects which pixel belongs which object. More precisely,
exp(ar(ZB))
f(z)
Σ m()) (2) @ f(obi) (ZB). where m (B) (z) = EB'S Evrgexp(ax BEB
and where B is a partition of [dæ] made of equal-size blocks B and a R!A + Ed outputs a score that normalized via softmax operation obtain the masks m (z). Many of these works also present some mechanism to select dynamically how many objects are present in the scene and thus have a variable-size representation z, an important technical aspect we omit in our analysis. Empirically, training these decoders based on some form of reconstruction objective, probabilistic or not, yields latent blocks zB that represent the information of individual objects separately. We believe our work constitutes step towards providing mathematically grounded explanation for why these approaches can perform this form of disentanglement without supervision (Theorems & 2). Many architectural innovations in scene mixture models concern the encoder, but our analysis focuses solely on the structure of the decoder f(z), which is shared aspect across multiple methods. Generalization capabilities of object-centric representations were studied empirically by Dittadi et al. [14] but did not cover Cartesian- -prducctt extrapolation (Corollary 3) on which we focus here.
Diagonal Hessian penalty [58]. Additive decoders are also closely related to the penalty introduced by Peebles et al. [58] which consists in regularizing the Hessian of the decoder to be diagonal. In Appendix A.2, we show that "additivity" and "diagonal Hessian" are equivalent properties. They showed empirically that this penalty can induce disentangiement on datasets such as CLEVR [35], which is a standard benchmark for OCRL, but did not provide any formal justification. Our work provides a rigorous explanation for these successes and highlights the link between the diagonal Hessian penalty and OCRL.
3

---

Compositional decoders [7]. Compositional decoders were recently introduced by Brady et al. [7] as a model for OCRL methods with identifiability guarantees. A decoder f is said be compositional when its Jacobian Df satisfies the following property everywhere: For all i E [d=] and B E B, * DBf((2)
O, where B == [d=] B. In other words, each ay can locally depend solely on one block z (this block can change for different z). In Appendix A.3, we show that compositional C² decoders are additive. Furthermore, Example shows a decoder that is additive but not compositional, which means that additive C decoders are strictly more expressive than compositional C decoders. Another important distinction with our work is that we consider more general supports for and provide novel extrapolation analysis. That being said, our identifiability result does not supersede theirs since they assume only C' decoders while our theory assumes C?.
= B= = f((z)
Extrapolation. Du and Mordatch [17] studied empirically how one can combine energy- -ased models for what they call compositional generalization, which is similar to our notion of Cartesian- product extrapolation, but suppose access to datasets in which only one latent factor varies and do not provide any theory. Webb al. [70] studied extrapolation empirically and proposed novel benchmark which does not have an additive structure. Besserve et al. [5] proposed a theoretical framework in which out-of-distribution samples are obtained by applying a transformation to a single hidden layer inside the decoder network. Krueger et al. [39] introduced domain generalization method which is trained to be robust to tasks falling outside the convex hull of training distributions. Extrapolation in text-conditioned image generation recently discussed by Wang al. [69].
3 Additive decoders for disentanglement & extrapolation
Our theoretical results assume the existence of some data matta-gnerating process describing how the observations are generated and, importantly, what are the "natural" factors of variations.
Assumption 1 (Data-generating process). The set of possible observations is given by a lower dimensional manifold f(z44) embedded in Rd= where zuss is an open set of Rd: and f : Z + Rd= is a CC-diffeomorphhisn onto its image. We will refer
f as the ground-truth decoder. At training time, the observations are ii.dd samples given by x = f(z) where z is distributed according to the probability measure Tptrain with support Zuain C Zisst Throughout, we assume that 2train is regularly closed (Definition 6).
Intuitively, the ground-truth decoder f is effectively relating the "natural factors of variations" z to the observations x in a one-to-one fashion. The map f is a ("-diffeomorphism onto its image, which means that it is C2 (has continuous second derivative) and that its inverse (restricted to the image of f) is also C²2 Analogous assumptions are very common in the literature on nonlinear ICA and disentanglementt [33, 36, 42, 1J. Mansouri et al. [52] pointed that the injectivity of f is violated when images show two objects that are indistinguishable, an important practical case that is not covered by our theory.
We emphasize the distinction between train, which corresponds to the observations seen during training, and Z which corresponds to the set of all possible images. The case where Zitain a Zlesi will be of particular interest when discussing extrapolation in Section 3.2. The "regularly closed" condition griin is mild, as it is satisfied as soon the distribution of z has density w.r.t. the Lebesgue measure on Rd= It is violated, for example, when z is a discrete random vector. Figure illustrates this assumption with simple examples.
Objective. Our analysis based on the simple objective of reconstructing the observations æ by learning an encoder 9 dR + dd and a decoder f IRd, + Rdd, Note that we assumed implicitly that the dimensionality of the learned representation matches the dimensionality of the ground-truthe We define the set of latent codes the encoder can output when evaluated on the training distribution:
Zuain == g (Z
(2)
When the images of the ground-truth and learned decoders match, i.e. f(Zurin) = f( ( which happens when the reconstruction task is solved exactly, one can define the map train 2 train as
: This function is going to be crucial throughout the work, especially to define B-disentanglement (Definition 3), as it relates the learned representation to the ground-truth representation.

---

Before introducing our formal definition of additive decoders, we introduce the following notation: Given a set Z S Rd and subset of indices B _ [dz], let us define Bn to be the projection of Z onto dimensions labelled by the index set B. More formally,
3B {
2} C R
Intuitively, we will say that a decoder is additive when its output is the summation of the outputs of "object-specific" decoders that depend only on each latent block 2B. This captures the idea that an image can be seen as the juxatoposition of multiple images which individually correspond objects in the scene or natural factors of variations (left of Figure 1).
Definition 1 (Additive functions). Let B be partition of [dz]'. function f Z + Rd is said to be additive if there exist functions f( ZB + Rda for all B E B such that
Vz E 2, f())
Σ
( (zn).
(5)
BEB
This additivity property will be central to our analysis as it will be the driving force of identifiability (Theorem 2) and Cartesian produccttttttteeeo extrapolation (Corollary 3).
Remark 1. Suppose we have
o <( I ( (ZB)) where is a known bijective function. For the decoder can be thought of as being multiplicative.
example, (y) = exp(y) fcomponent-win
Our results still apply since we can simply transform the data doing I == o -(x) to recover the additive form I = EBEB ( ((B)()))
Differences with OCRL in practice. We point out that, although the additive decoders make intuitive sense for OCRL, they are not expressive enough to represent the "masked decoders" typically used in practice (Equation (I)). The lack of additivity stems from the normalization in the masks m (z). We hypothesize that studying the simpler additive decoders might still reveal interesting phenomena present modern OCRL approaches due to their resemblance. Another difference that, in practice, the same object-speciff decoder (obi) is applied to every latent block øg. Our theory allows for these functions to be different, but also applies when functions are the same. Additionally, this paramcter sharing across f(B) enables modern methods to have a variable number of objects across samples, an important practical point our theory does not cover.
3.1 Identifiability analysis
We now study the identifiability of additive decoders and show how they can yield disentanglement. Our definition of disentanglement will rely on partition-respecing permutations:
Definition 2 (Partition-respeetiin permutations). Let B be partition of {1, dz A permutation over {1, ] respects B if. for all B E B, π T(B)= E B.
Essentially, permutation that respects B is one which can permute blocks of B and permute elements within a block, but cannot "mix" blocks together. We now introduce B-disentanglement.
Definition 3 d-diiennangglement).. A learned decoder f RRdd Rd is said to be B-disentangled w.r.t. the ground truth decoder f when f(ztti)) = f( (2 and the mapping v := f- o f is a diffeomorphism from Itrain to guai satisfying the following property: there exists a permutation π respecting B such that, for all B E B, there exists a function 0x(B)) - + zuain such that, for all z E iuain, (z) = (B(B) (zB). In other words, (=) depends only on zB.
Thus, B-disentanglement means that the blocks of latent dimensions z are disentangled from one another, but that variables within a given block might remain entangled. Note that, unless the partition is B {{1).... ... {dz}} this corresponds to weaker form of disentanglement than what is typically seeked in nonlinear ICA, i.e. recovering each variable individually.
Example 1. To illustrate B-ddisntanglement, imagine scene consisting of two balls moving around in 2D where the "ground-truth representation is given by z = (z) 1/2 , x² .²²"yy))) where 2B, = (x², y²) and 2B, (2²))²²) are the coordinates of each ball (here, B :- (11, {3, 4}}). In that case, learned representation is B-disentangled when the balls are disentangled from one another. However, the basis in which the position of each ball is represented might differ in both representations.
Without loss of generality, we assume that the partition B is contiguous, ie. each B E B can be written as B {i+ L,i 2, ...., ++| [B]..
5

---

Our first result (Theorem 1) shows a weaker form of disentanglement we call local B B-ddisettagllement This means the Jacobian matrix of v, De, has a pelock-permutationn structure everywhere.
Definition 4 (Local B-disentanglement) A learned decoder f IR - Rd is said to be locally B-disentangled w.r.t. the ground-trun truth decoder f when f = f( f(2man)) and the mapping = f-1 o f is a diffeomorphism from Quain Zuain with a mapping v : 2rriin - Zuain satisfying the following property: for all Iunin, there exists a permutation respecting 1 such that, for all B E B. the columns of (V=(B) E plB|xdz outside block B are zero.
In Appendix A.4, we provide three examples where local disentanglement holds but not global disen- tanglement. The first one illustrates how having a disconnected support can allow for a permutation π (from Definition 4) that changes between disconnected regions of the support. The last two examples show how, even if the permutation stays the same throughout the support, we can still violate global disentanglement, even with a connected support.
We now state the main identifiability result of this work which provides conditions to guarantee local disentanglement. We will then see how go from local global disentanglement the subsequent Theorem 2. For pedagogical reasons, we delay the formalization of the sufficient nonlinearity Assumption on which the result crucially relies.
Theorem 1 (Local disentanglement via additive decoders). Suppose that the data- geata-ggnrratngg process satisfies Assumption I, that the learned decoder f RR= - IRd is C'-diffeomorphhism that the encoder 9 gd. + 12d= is continuous, that both and f are additive (Definition 1) and that f sufficiently nonlinear as formalized by Assumption 2. Then, if and 9 solve the reconstruction problem on the training distribution, i.e. Rurin | (()())) 0. we have that f is locally B-disentangled w.r.t. f (Definition 4)
The proof of Theorem 1, which can be found in Appendix A.5, is inspired from Hyvärinen et al. [33]. The essential differences that (i) they leverage the additivity of the conditional log-density of given an auxiliary variable u (i.e. conditional independence) instead of the additivity of the decoder function f. (ii) we extend their proof techniques to allow for "block" disentanglement, i.e. when B is not the trivial partition {{1).... {da}), ). (iii) the asssumption "sufficient variability" of the prior p(z u) of Hyvärinen et al. [33] is replaced by an analogous assumption of "sufficient nonlinearity" of the decoder f (Assumption 2), and (iv) we consider much more general supports graai which makes the jump from local to global disentanglement less direct in our case.
The seenttifibbiiit--eeeesiiiiiiiiii trade-off between identifiability and expressivity: the finer the partition, the tighter the identifiability guarantee but the less expressive is the function class. The optimal level of granularity is going to dependent on the application hand. Whether B could be learned from data is left for future work.
trade-off. The level of gramularity of the partition B controls the
Sufficient nonlinearity. The following assumption is key in proving Theorem 2, as requires that the ground-truth decoder is "sufficiently nonlinear". This is reminiscent of the "sufficient variability" assumptions found in the nonlinear ICA litterature, which usually concerns the distribution of the latent variable z opposed to the decoder f [30. 31, 33. 36. 37, 42, 73]. We clarify this link in Appendix A.6 and provide intuitions why sufficient nonlincarity can be satisfied when dg >> dz.
Assumption 2 (Sufficient nonlinearity of f). Let == d2 + +E EBE [B[[B|+1)] For all 2umin f is such that the following matrix has linearly independent columns (i.e. full column-rank):
w(x) = [[. (B))(((()))
[D
(1))
(B) (zB)
E RRdexg_
(6)
NEB
(L
BEB
where B = B2 ((1,z) 1 ii) Note this implies d₂ 2
The following example shows that Theorem I does not apply if the ground-truth decoder f is linear. If that the case, it would contradict the well known fact that linear ICA with independent Gaussian factors is unidentifiable.
Example (Importance of Assumption 2). Suppose x = f(z) = Az where A E Rd. xd, is full rank. Take f(z) = AV= z and g(æ) == V-1 A x where V E Rddxxd, is invertible and Al is the left pseudo inverse of A. By construction, we have that E|x f(g(x)) = 0 and and f are

---

However, we still have that o=))== f(z) Vz where V does not necessarily have a block-permunn phoec-peemttation structure,
B-additive because f(z) EBEB A. and =
i.e. no disentanglement. The reason cannot apply Theorem I here is because Assumption 2 is not satisfied. Indeed, the second derivatives of (zn) ( == ARBB are zero and hence W (=) cannot have full column-rank.
Example 3 (A sufficiently nonlinear f). In Appendix A.7 we show numerically that the function
f(z) = [21, 2]. = z11 T + [[22 + 1), 1),( (z2 + 1)², (22 + 1)², 1)),, (z2 + 1)*JT
is diffeomorphism from the square 1- 0] [0,1] to its image that satisfies Assumption 2.
Example 4 (Smooth balls dataset is sufficiently nonlinear). In Appendix A.7 we present a simple synthetic dataset consisting of images of two colored balls moving up and down. We also verify mumerically that its underlying ground-truth decoder f is sufficiently nonlinear.
3.1.1 From local to global disentanglement
The following result provides additional assumptions guarantee global disentanglement (Defini- tion 3) as opposed to only local disentanglement (Definition 4). See Appendix A.8 for its proof.
Theorem (From local to global disentanglement), Suppose that all the assumptions of Theorem I hold. Additionally, assume Zuain is path-connected (Definition 8) and that the block-specific decoders f(8) and f( are injective for all blocks B E B. Then, if f and 9 solve the reconstruction problem the training distribution, i.e. Etrain x f()())) B- B-disentangled w.r.t. ff (Definition 3) and, for all B E B,
have that is (globally)
( = (B + (B) for all zB E 2train
(8)
where the functions (B)( are from Defintion 3 and the vectors clB) Rdd are constants such that Eneg cl 0. We also have that the functions Dx(B)= Zuain + xB) are C2-diffeomorphhims and have the following form:
3 )x(B)(EB) = (f
BB)),, for all z E Quain
(9)
Equation (8) in the above result shows that each block-specific learned decoder f(B) is "imitating" a block-specific ground-trute -ruud decoder f=( Indeed, the "object-specific" image outputted by the decoder f evaluated at some 2B E Etmin is the same as the image outputted by (B) evaluated at v(zn) E BB up to additive constant vector BB) These constants cancel each other out when taking the sum the bleck-specific decoders.
Path-connected
Not path-connected
Equation (9) provides an explicit form for the function U=(B). which essentially the learned block-specific de- coder composed with the inverse of the ground-truth block-
Regularly ddoeed
specific decoder.
Additional assumptions to go from local to global. As-
suming that the support of puann guain path-connected (see Definition in appendíx) is useful since it prevents the permutation of Definition 4 from changing between
Not regularty closed
two disconnected regions of itaiin See Figure 2 for an illustration. In Appendix A.9, we discuss the additional assumption that each (B) must be injective and show that, in general, it is not equivalent to the assumption that EBEB (B) is injective.
Figure 2: Illustrating regularly closed sets (Definition 6) and path-connected sets (Definition 8). Theorem 2 requires zinai to satisfy both properties.
3.2 Cartesian-product extrapolation
In this section, we show how a learned additive decoder can be used to generate images æ that are "out of support" in the sense that x é f(2uain), but that are still on the manifold of "reasonable" images, i.e. æ E (2 To characterize the set of images the learned decoder can generate, we will rely the notion of cantessaanproduee extension". which we define next.
7

---

Definition 5 (Cartesian-product extension). Given a set Z S &d and partition B of [dz), we define the Cartesian-producr extension of Z as
t
Zp,
2n
CPEg (
2
C2E :- πI
ZB where ZB = (zn E ZJ. BEB
Z8,
BB.
It is indeed an extension of Z since Z S BEB ZB.
Figure 3: Illustration of Definition 5,
Let define ū
CPE (2tain) + CPEg (Z to be the natural extension of the function
Zuain Zuain More explicitly, is the "concatenation" of the functions Un given in Definition 3:
b(z) = [üB (2n =(B.)) UB, ((B,))))
(10)
where is the number of blocks in B. This map is a diffeomorphism because each Ux(B) is diffeomorphism from Żrain to 2T0)) by Theorem
We already know that f(=) fo 0()) for all & E -imin The following result shows that this equality holds in fact on the larger set CPEs (uuain), the Cartesian-product extension of @rain See right of Figure I for illustration of the following corollary.
(PEx(2umin)
Corollary 3 (Cartesian-prodde
extrapolation). Suppose the assumptions of Theorem 2 holds. Then,
for all CPE (2tai))
Σ
f( (zB)
Σ
((((B)
(11)
BEB
BEB
Furthermore, if CPEss S Ziest then f(CPE& (2tmin)) S fZ
Equation (11) tells us that the learned decoder f "imitates" the ground-truth f not just over Stain but also over its Cartesian-product extension. This is important since it guarantees that we can generate observations never seen during training as follows: Choose latent vector new that is in the Cartesian Cartesian-product extension of imin but not in âirai itself, i.e. new E CPE
imin Then, evaluate the learned decoder nEw to get new f ). By Corollary 3, we know that i.e. it is the observation one would have obtain by evaluating the ground-truth decoder f on the point n(w) CPEg( train).. In addition, this x new has never been seen during Z train The experiment of Figure illustrates this procedure.
new = fo o =e*),, i.e.
training since o(2 ne"" & v Ztrain)
About the extra assumption "C S Z Recall that, in Assumption I, we interpreted f(2es) to be the set of "reasonable" observations of which we only observe a subset f ( Under this interpretation, giest is the set of reasonable values for the vector z and the additional assumption that CPEp( (2tmii) S giest in Corollary 3 requires that the Cartesian- parouuct-pructtttt extension of nin consists only of reasonable values of z. From this assumption, we can easily conclude that
f(CPEg (mmin) C f(z(e),, which can be interpreted as: "The novel observations new obtained via proddut extrapolation are reasonable" Appendix A. 11 describes an example where the assumption is violated, i.c. CPEp (Ztrain) g Ziess The practical implication of this is that the new observations new obtained via Cartesian- producc extrapolation might not always be reasonable.
Disentanglement is not enough for extrapolation. To the best of our knowledge, Corollary 3 is the first result that formalizes how disentanglement fact that disentanglement alone is not sufficient to enable extrapolation and that one needs to restrict the hypothesis class of decoders in some way. Indeed, given learned decoder f that is disentangled w.r.t. f on the training support Ztrinn one cannot guarantee both decoders will "agree" outside the training domain without further restricting and f. This work has focused on "additivity", but we believe other types of restriction could correspond to other types of extrapolation.
induce extrapolation. We believe it illustrates the
4 Experiments
We now present empirical validations of the theoretical results presented earlier. To achieve this, we compare the ability of additive and non-additive decoders to both identify ground-truth latent factors (Theorems & 2) and extrapolate (Corollary 3) when trained to solve the reconstruction task on simple images (64 x 64 3) consisting of two balls moving in space [2]. See Appendix B.
8

---

ScalarLatents
BlockLatents (independent =)
BlockLatents (dependent 2)
Decoders
RMSE
LMSspaa
RMSFO
L.S
Spear
RMSE
LMSTeee
RMSE
L.S
Non-add. .06 -00
70.6+
.18
73.7+4.04
.02+.001
53.9
.2++0001
78. +
Additive
.622.02 002
91.5
.11
89.5.15.2
.03
92.2.1.91
.1000002
99.9
Table 1: Reporting reconstruction mean squared error (RMSE .) and the Latent Matching Score (LMS t) for the three datasets considered: ScalarLatents and BlockLatents with independent and dependent latents. Runs were repeated with 10 random initializations. RMSE and LMS Spear are the same metric but evaluated out of support (see Appendix B.3 for details). While the standard error is high, the differences are still clear as can be seen in their box plot version in Appendix B.4.
for training details. We consider two datasets: one where the two ball positions only vary along the y-axis (ScalarLatents) and one where the positions can vary along both the x and y axes (BlockLateets).
ScalarLatentss The ground-truth latent vector z E R2 such that 21 and 22 corresponds to the height (y-coordinate) of the first and second ball, respectively. Thus the partition is simply B= {{1}: {2)} (cach object has only one latent factor). This simple setting is interesting to study since the low dimensionality of the latent space (d_ = 2) allows for exhaustive visualizations like Figure 4. To study partesian-produce extrapolation (Corollary 3), we sample z from a distribution with L-shaped support given by Ztrain = [0, 1] x [0, 1] [0.5, 1] x [0.5, 11, so that the training set does not contain images where both balls appear in the upper half of the image (see Appendix B.2).
BlockLatents: The ground-truth latent vector E I' is such that ((1,)} and 2(3.4) correspond to the x, position of the first and second ball, respectively (the partition is simply B = [[1,2), {3, 4)}. i.c. each object has two latent factors). Thus, this more challenging setting illustrates "block- disentanglement". The latent is sampled uniformly from the hypercube [0. 1 but the images presenting occlusion (when a ball is behind another) are rejected from the dataset. We discuss how additive decoders cannot model images presenting occlusion in Appendix A.12. We also present an additional version of this dataset where we sample from the hypercube [o, 1]a with dependencies. See Appendix B.2 for more details about data generation.
Evaluation metrics: To evaluate disentanglement, we compute a matrix of scores ($g,Br) E Rxxe where L is the number of blocks in B and $,,B' is score measuring how well we can predict the ground-truth block æB from the learned latent block g 9B (x) outputted by the encoder. The final Latent Matching Score (LMS) is computed as LMS = arg max EBEB $ (B)) where and the score used is the absolute value of the correlation, LMS is simply the mean correlation coefficient (MCC), which is widely used in the nonlinear ICA literature [30, 31, 33, 36, 42]. Because our theory guarantees recovery of the latents only up to invertible and potentially nonlinear transformations, we use the Spearman correlation, which can capture nonlinear relationships unlike the Pearson correlation. We denote this score by LMS and will use in the dataset ScalarLatents. For the BlockL
Gg is the set of permutations respecting B (Definition 2). When B =
dataset, we cannot use Spearman correlation (because zB are two dimensional). Instead, we take the score BB, to be the R² score of regression tree. We denote this score by L.S There are subtleties to take care of when one wants to evaluate LMS on non-additive model due to the fact that the learned representation does not have a natural partition B. We must thus search over partitions. We discuss this and provide further details on the metrics in Appendix B.3.
4.1 Results
Additivity is important for disentanglement. Table shows that the additive decoder obtains a much higher LMSspear & LMS than its non-additive counterpart on all three datasets considered, even if both decoders have very small reconstruction errors. This is corroborated by the visualizations of Figures 4 & 5. Appendix B.5 additionally shows object-specific reconstructions for the LtoekLatents dataset. We emphasize that disentanglement is possible even when the latent factors are dependent (or causally related), as shown on the Scalarl Latents dataset (L-shaped support implies dependencies) and on the MlckLattents dataset with dependencies (Table 1). Note that prior works have relied on interventions [3. 2, 8] Cartesian- Cartesian-pooduct supports [68, 62] to deal with dependencies.

---

Predicted Latent 2
Predicted Latent
11-13-1530 Predicted Latent
22 20
-11-18-2200
Predicted Latent
(a) Additive decoder
(b) Non-additive decoder
Figure 4:
Figure (a) shows latent representation outputted by the encoder ((x) over the training dataset, and the corresponding reconstructed images of the additive decoder with median SpS among runs performed on the ScalarLatents dataset. Figure (b) shows the same thing for the non-additive decoder. The color gradient corresponds to the value of one of the ground -ruth factor, the red dots correspond to factors used to generate the images and the yellow dashed square highlights extrapolated images.
Predicted Lakents
Predictedd Latents
0
0:25
0.30
0.75
0.25
030
0.75
0.25
0.30
0.75
0.25
0.50
0.75
moving along x axis
moving along axis
moving along
moving along
Predicted Latents
Predicted Latents
1
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Ball 1 moving along y axis
Ball moving along y axis
Ball 1 moving along y axis
Ball 2 moving alongy axis
(a) Additive Decoder
(b) Non-Additive Decoder
Figure 5: Latent responses for the case of independent latents in the BlockLatent dataset. In each plot, we report the latent factors predicted from multiple images where one ball moves along only one axis at a time. For the additive case, at most two latents change, as it should, while more than two latents change for the non-additive case. See Appendix B.5 for details.
Additivity is important for sareesian-producc extrapolation. Figure 4 illustrates that the additive decoder can generate images that are outside the training domain (both balls in upper half of the image) while its non-additive counterpart cannot. Furthermore, Table 1 also corroborates this showing that the -ost-o--upppot" (OOS) reconstruction MSE and LMS (evaluated only on the samples never seen during training) are significantly better for the additive than for the non-additive decoder.
Importance of connected support. Theorem 2 required that the support of the latent factors, g was conheconnected. Appendix 8.6 shows experiments where this assumption is violated, which yields lower LMS for the additive decoder, thus highlighting the importance of this assumption.
5 Conclusion
We provided an in-depth identifiability analysis of additive decoders, which bears resemblance to standard decoders used in OCRL, and introduced a novel thcoretical framework showing how this architecture can generate reasonable images never seen during training via "Cartesian-produs extrapolation". We validated empirically both of these results and confirmed that additivity was indeed crucial. By studying rigorously how disentanglement can induce extrapolation, our work highlighted the necessity of restricting the decoder to extrapolate and set the stage for future works to explore disentanglement and extrapolation other function classes such as masked decoders typically used in OCRL. We postulate that the type of identifiability analysis introduced in this work has the potential of expanding our understanding of creativity generative models, ultimately resulting in representations that generalize better.
10

---

Acknowledgements
This research was partially supported by the Canada CIFAR Al Chair Program, by an IVADO excellence PhD scholarship and by Samsung Electronics Co., Ldt. The experiments were in part enabled by computational resources provided by Calcul Québec (calculquebec. ca) and the Digital Research Alliance of Canada (alliancecal ca). Simon Lacoste-Julien is CIFAR Associate Fellow Learning in Machines Brains program.
References
[1] K. Ahuja, J. Hartford, and Y. Bengio. Properties from mechanisms: an equivariance per- spective identifiable representation learning. In International Conference Learning Representations, 2022.
[2] K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations, 2022.
[3] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. Proceedings of the 40th International Conference on Machine Learning, 2023.
[4] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 2013.
[5] M. Besserve, R. Sun, D. Janzing, and B. Schölkopf. A theory of independent mechanisms for extrapolation in generative models. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), 2021.
[6] J. Bradbury, R. Frostig. P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman- and Q. Zhang. JAX: composable transformations of Python+NumPPy programs, 2018. URL http: tttt:://githe /ttp//ggttt/.///////////eeeehhtt com/ google/jax.
[7] Brady, R. S. Zimmermann, Sharma, B. Schölkopf, J. von Kügelgen, and W. Brendel. Provably learning object-centric representations. In International Conference Machine Learning, 2023.
[8] J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In Advances Neural Information Processing Systems, 2022.
[9] S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems, 2022.
[10] S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar, Learning linear causal representations from interventions under general nonlinear mixing, 2023.
[11] C. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. Monet: Unsupervised scene decomposition and representation, 2019.
[12] E. Crawford and J. Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[13] A. d' Avila Garcez and L. Lamb. Neurosymbolic AI: The 3rd wave. ArXiv, abs/2012.05876. 2020.
[14] A. Dittadi, S. S. Papa, M. De Vita, B. Schölkopf, O. Winther, and F. Locatello. Generalization and robustness implications object-centric learning. In Proceedings of the 39th International Conference on Machine Learning, 2022.
[15] D. Donoho and C. Grimes. Image manifolds which are isometric to cuclidean space. Journal of Mathematical Imaging and Vision, 2003.
[16] D. I.. Donoho and Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 2003.
11

---

[17] Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. In Advances Neural Information Processing Systems, 2019.
[18) M. Engelcke, A. R. Kosiorek, O. P. Jones, and I. Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. International Conference on Learning Representations, 2020.
[19] S. M. A. Eslami, N. Heess, T. Weber, Tassa, D. Szepesvari, K. Kavukcuoglu, and G. Hinton, Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, 2016.
[20] J. A. Fodor and Z. W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 1988.
[21] A. Goyal and Y. Bengio. Inductive biases for deep learning of higher-level cognition. Proc. R. Soc. A 478: 20210068, 2022.
[22] K. Greff, A. Rasmus, M. Berglund, T. Hao, Valpola, and Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, 2016.
[23] K. Greff, S. van Steenkiste, and J. Schmidhuber. Neural expectation maximization. In Advances Neural Information Processing Systems, 2017.
[24] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. In Proceedings of the 36th International Conference Machine Learning, 2019.
[25] K. Greff, S. van Steenkiste, and J. Schmidhuber. On the binding problem in artificial neural networks. ArXiv, abs/2012.05208, 2020.
[26] L. Gresele, J. V. Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? In Advances in Neural Information Processing Systems, 2021.
[27] H. Hälvä, S. L. Corff, L. Lehéricy, J. So, Y. Zhu, E. Gassiat, and A. Hyvarinen. Disentan- gling identifiable features from noisy data with structured nonlinear ICA. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[28] S. Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 1990.
[29] D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? In Advances in Neural Information Processing Systems, 2021.
[30] A. Hyvärinen and H. Morioka. Unsupervised feature extraction by contraanne learning and nonlinear ica. In Advances in Neural Information Processing Systems, 2016.
[31] A. Hyvärinen and H. Morioka. Nonlinear ICA of Temporally Dependent Stationary Sources. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
[32] A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 1999.
[33] A. Hyvärinen, H. Sasaki, and R. E. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In AISTATS. PMLR. 2019.
[34] Y. Jiang and B. Aragam. Learning nonparametric latent causal graphs with unknown interven- tions, 2023.
[35] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. B. Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
12

---

[36] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ica: A unifying framework. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020.
[37] I. Khemakhem, R. Monti, D. Kingma, and A. Hyvärinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. In Advances in Neural Information Processing Systems, 2020.
[38] D. A. Klindt, L. Schott, Y. Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethge, and D. M. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In 9th International Conference on Learning Representations, 2021.
[39] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville. Out-of-distribution generalization via risk extrapolation (rex). In Proceedings of the 38th International Conference Machine Learning, 2021.
[40] S. Lachapelle and S. Lacoste-Julien. Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Learning, 2022.
[41] S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Y. Bengio. S. Lacoste-Julien, and Q. Bertrand. Synergies between disentanglement and sparsity: a multi-task learning perspective, 2022.
[42] S. Lachapelle, P. Rodriguez Lopez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In First Conference Causal Learning and Reasoning, 2022.
[43] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and Gershman. Building machines that learn and think like people. Behavioral and Bruin Sciences, 2017.
[44] Leeb, G. Lanzillotta, Y. Annadani, M. Besserve, S. Bauer, and Schölkopf. Structure by architecture: Disentangled representations without regularization, 2021.
[45] Z. Lin, Y. Wu, S. V. Peri, Sun, G. Singh, F. Deng, J. Jiang, and S. Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations, 2020.
[46] Lippe, S. Magliacane, S. Löwe, M. Asano, T. Cohen, and E. Gavves. iCITRIS: Causal
representation learning for instantaneous temporal effects. In UAI 2022 Workshop Causal Representation Learning, 2022.
[47] P. Lippe. S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. CITRIS: Causal identifiability from temporal intervened sequences, 2022.
[48] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[49] F. Locatello, B. Poole, G. Ractsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly- supervised disentanglement without compromises. In Proceedings of the 37th International Conference Machine Learning, 2020.
[50] Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2020.
[51] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, 2020.
[52] A. Mansouri, J. Hartford, K. Ahuja, and Y. Bengio. Object-centric causal representation learning. In NewrlPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022.
[53] G. F. Marcus. The algebraic mind: integrating connectionism and cognitive science, 2001.
13

---

[54] G. E. Moran. D. Sridhar, Y. Wang, and D. Blei. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research, 2022.
[55] Munkres, Analysis On Manifolds. Basic Books, 1991.
[56] R. Munkres, Topology. Prentice Hall, Inc., 2 edition, 2000.
[57] Pearl. The seven tools of causal inference, with reflections on machine learning. Commun. ACM, 2019.
[58] W. Peebles, J. Peebles, J.-Y. Zhu, A. A. Efros, and A. Torralba. The hessian penalty; A weak prior for unsupervised disentanglement. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[59] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204006/25, 2022.
[60] P. Reizinger, L.. Gresele, J. Brady, J. V. Kügelgen, D. Zietlow, B. Schölkopf, G. Martius, W. Brendel, and M. Besserve. Embrace the gap: VAEs perform independent mechanism analysis. In Advances in Neural Information Processing Systems, 2022.
[61] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[62] K. Roth, M. Ibrahim, Z. Akata, P. Vincent, and D. Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. The Eleventh International Conference on Learning Representations, 2023.
[63] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings the IEEE Advances in Machine Learning and Deep Neural Networks, 2021.
[64] C. Squires, A. Seigal, S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[65] A. Taleb and Jutten. Source separation in post-nonlinear mixtures. IEFE Transactions on Signal Processing, 1999.
[66] J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Lo- catello. self-supervised learning with data augmentations provably isolates content from style. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[67] J. von Kügelgen, M. Besserve, W. Liang, L. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions, 2023.
[68] Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective, 2022.
[69] Z. Wang, I. Gui, J. Negrea, and V. Veitch. Concept algebra for text-controlled vision models, 2023.
[70] T. W. Webb, Z. Dulberg, S. M. Frankland, A. A. Petrov, R. C. O'Reilly, and J. D. Cohen. Learning representations that support extrapolation. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[71] Q. Xi and B. Bloem-Reddy. Indeterminacy in generative models: Characterization and strong identifiability. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, 2023.
[72] J. Zhang, C. Squires, K. Greenewald, A. Srivastava, K. Shanmugam, and C. Uhler. Identifiability guarantees for causal disentanglement from soft interventions, 2023.
[73] Y. Zheng, Ng, and Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022.
14

---

Appendix
Table of Contents
A Identifiability and Extrapolation Analysis
16
A.1
Useful definitions and lemmas
16
Relationship between additive decoders and diagonal Hessian penalty
A.3 Additive decoders form a superset of compositional decoders [7]
19
A.4 Examples of local but non-global disentanglement
20
A.5 Proof of Theorem
22
A.6 Sufficient nonlinearity v.s. sufficient variability in nonlinear ICA with auxiliary variables
25
A.7 Examples of sufficiently nonlinear additive decoders
26
A.8 Proof of Theorem 2
27
A.9 Injectivity of object-specific decoders v.s. injectivity of their sum
30
A.10 Proof Corollary 3
31
A.1I Will extrapolated images make sense?
32
A.12 Additive decoders cannot model occlusion
32
B
Experiments
32
B.1 Training Details
32
B.2
Datasets Details
33
B.3
Evaluation Metrics
34
B.4
Boxplots for main experiments (Table 1)
34
Additional Results: BlockLatents Dataset
35
B.6
Disconnected Support Experiments
37
B.7
Additional Results: ScalarLatents Dataset
38
15

---

Observation Table 2: Table of Notation.
Calligraphic & indexing conventions
(1,2,...,7)
Scalar (random or not, depending on context)
Vector (random or not, depending on context)
Matrix
X
Set/Support
J
Scalar-valued function Vector-valued function
f
f
Restriction of f to the set A Jacobian of f and f Hessian of f Subset of indices Cardinality of the B Vector formed with the ith coordinates of æ, for all E B
Df.
D
D
B n
B
XB.B'
Matrix formed with the entries (i.j) E B' of X.
Given X' C R"" X'B
{EB x E X} (projection of X)
Recurrent notation
Rd,
2 E Rd
Vector of latent factors of variations
2 C Rd
Support of z
Ground-truth decoder function
Learned decoder function
13
A partition of [dz] (assumed contiguous w.l.o.g.)
B E B
block of the partition
Bi)) E B
The unique block of B that contains
T [d,]
[d.]
A permutation
SB
Ung5 8²
SS
[d.] SB
Rgg
d2 X
[M E xd (i.5) & SB = M., 0}
General topology
X
Closure of the subset X' C IR in the standard topology of R."
X
Interior of the subset X' C R" in the standard topology of IR""
A Identifiability and Extrapolation Analysis
A.1 Useful definitions and lemmas
We start by recalling some notions of general topology that are going to be used later on. For a proper introduction these concepts, see for example Munkres [56].
Definition (Regularly closed sets). A set Z C Rdd is regularly closed if Z 2 i.e. if is equal the closure of its interior (in the standard topology of )F").
Definition 7 (Connected sets). set C Rdd, is connected if it cannot be written as union of non-empty and disjoint open sets (in the subspace topology).
Definition 8 (Path-connected sets). A set Z C IRd is path-connected if for all puir of points 20 z1 E Z, there exists a continuous map d [0,1] Z such that ((0) 20 and Φ(1) z!.
Such map is called path between 20 and z!

---

Definition 9 (Homeomorphism)) Let A and B be subsets of IR" equipped with the subspace topology. A function f A - B an homeomorphissm if is bijective, continuous and its inverse continuous.
The following technical lemma will be useful in the proof of Theorem 1. For it, we will need additional notation: Let S S A C RR. We already saw that S refers to the closure S in the I&" topology. We will denote by (S) the closure of S in the subspace topology of A induced by R",
which is not necessarily the same as S. In fact, both can be related via ClA S A (see Munkres [56, Theorem 17.4, p.95]).
Lemma 4. Let A, B C R" and suppose there exists an homeomorphism f A - B. if regularly closed in R", we have that C B
Proof. Note that J is a continuous injective function from the open set A to ((A)). By the "invariance of domain" theorem [56, p.381], we have that f(A)) must be open in R". Of course, we have that f(A") _ B, and thus ( S 1 (the interior of B is the largest open set contained in B). Analogously, f-1 is a continuous injective function from the open set B to (B"). Again, by "invariance of domain"" f (BB) must be open in RR" and thus f (B)) C A We can conclude that f(A)) = Bº
We can conclude as follow:
cB B B B B = f(A)= ((A) - ((AP f(cl((A)) C Ecl((((A))
where the first inclusion holds by continuity of f [56, T1m.18.1 p.104).
This lemma is taken from [42].
Lemma 5 (Sparsity pattern of an invertible matrix contains a permutation). Let L E R be an invertible matrix. Then, there exists a permutation a such that Li # 0 for all
Proof. Since the matrix L is invertible, its determinant is non-zero, i.c.
det(L)=
Σ sign(T)) (T)
II L. + 0,
(12)
TEG
where 6m is the set of m-permutations. This equation implies that at least one term of the sum is non-zero, meaning there exists 6m such that for all i [m]. Li, # 0.
Definition 10 (Aligned subspaces of mmx)), Given subset S S {1, (1....,l) m] {1,..., n), define
IR mxx_ = (M& E Rm (1.j) ES = Mijj = 0)
(13)
Definition 11 (Useful sets). Given a partition B of [d], we define
SB = U B²
S : fl........................
d SB
(14)
Definition 12 (C* -iffeoomorphism. Let ACR" and B C Rm. A map f A B is said to be Ck c--diffeomorphism if it is bijective, C and has a C2 inverse.
Remark 2. Differentiability rypically defined for functions that have an open domain in IR... However, in the definition above, the set A might not be open in D" and B might not be open in R In the case of an arbitrary domain A, is customary to say that a function f C IR + I is Ck if there exists C² function g defined (i.e. g extends f). With this definition, we have that a composition of Chee functions is C. as usual. See for example p. 199 of Munkres [55].
open set U Th that contains A such that g 3-==
The following lemma allows us to unambiguously define the k first derivatives of a O function f : A + TT on the set A"
Lemma 6. Let A C R." and f : A - Ra be a Ck function. Then, its le first derivatives uniquely defined on A in the sense that they do not depend the specific choice of Ch extension.
17

---

Proof. Let g U - R and h V + IR" be two C* extensions of f to U S I=" and V S R both open R". By definition,
( = f(x) = h(r), Vx E A.
(15)
The usual derivative is uniquely defined on the interior of the domain, that Dg(x) (x) Dh(x), Væ A
(16)
Consider a point ao A.. By definition of closure, there exists sequence { C A s.t.
limp-poo 3k Zo. We thus have that
lim Dg(#+) lim Dh(x;)
(17)
R-000
1-+00
Dg( = Dh(
(18)
where we used the fact that the derivatives of g and h are continuous to go to the second line. Thus, all the O* extensions of f must have equal derivatives on A This means we can unambiguously define the derivative of f everywhere on A to equal to the derivative of one of its C extensions.
Since f is C* its derivative Df is C#-1, we can thus apply the same argument to get that the second derivative of f is uniquely defined on A² It can shown that A = AA. One can thus apply the same argument recursively to show that the first derivatives of f are uniquely defined on A C
Definition 13 (C* dC*-dfffoooo onto its image). Let A C R". A map f A + R is said
be a C difffeoomorphii onto its image if the restriction f to its image f A + f(A) Ch -ifffeomorpphsm.
Remark 3. If S CAC R" and f A 4 R. is C* di-fifffemoophiim on its image, then the restriction of to i.e. flg. also O diffeomorphism on its image. That is because f fls is clearly bijective, is C (simply take the C extension of f and so its inverse (simply take the C extension of f-)).
A.2 Relationship between additive decoders and the diagonal Hessian penalty
Proposition 7 (Equivalence between additivity and diagonal Hessian). Let f IRd - R. be a C function. Then,
Vz E Rdd, f(z) = (())=EBe)) ( where f(B) RI , R is C
Wk E [dz], = Rd, D²f is block diagonal with blocks B.
(19)
Proof. We start by showing the = direction. Let B and B' be two distinct blocks of B. Let i B and i' E B'. We can compute the derivative of ft w.r.t. 2;:
D=fa(z)=
D. ( (zB)
(20)
BEB
where the last equality holds because E B and not in any other block B. Furthermore,
Df D f BB) (ZB) = 0,
(21)
where the last equality holds because i' & B. This shows that D2 fz() is block diagonal.
We show the = direction. Fix l: E [dz], B E B. We know that DB. fx( 0 for
all z E Rd: Fix z E dRd Consider a continuously differentiable path & [00, 1] - RIB"1 such
that (00) 0 and (11) zn- As Di fx() is continuous function of z, fundamental theorem of calculus for line integrals to get that
use the
DBfA(zB. xB=) DBfx(EB,)) [/'' D D. ((t) 0,
(22)
(where Dz fa(2 (1)))(((1) denotes matrix-vector product) which implies that
(zf = (zff(zz..0)
(23)
18

---

And the above equality holds for all B E B and all E R
Choose arbitrary& Rdd, Consider continously differentiable path db [0. Rd such that
(0) = O and (1) = z. By applying the fundamental theorem of calculus for line integrals once more, have that
fx(z) fx((0= = =
(f
(24)
Dsfc(((()))
(25)
BEB
Σ
(26)
BEB
J.
=
Σ BEB
(27)
where the last equality holds by (23). We can further apply the fundamental theorem of calculus for
line integrals to each term ff DB Daff((bB(t (t) to get
fx(2) fa( = Σ (=, - fa(0,0))
(28)
BEB
fx(z) = fn(0) + E(Je(zB, 0) fo))
(fu(zB,
(29)
BEB
Σ
Sa(En, 0)
1
Sx((0)
(30)
B
BEB
(*B)==
and since z was arbitrary, the above holds for all E Rdd, Note that the functions f (B) (zB) must be C2 because fx is C This concludes the proof.
A.3 Additive decoders form a superset compositional decoders [7]
Compositional decoders were introduced by Brady et al. [7] suitable class of functions to perform object-centric representation learning with identifiability guarantees. They are also interested in Block-dienttanglmmee but, contrarily our work, they assume that the latent vector z is fully supported, i.e. = RRd:, We now rewrite the definition of compositional decoders in the notation used in this work:
Definition 14 (Compositional decoders, adapted from [7]). Given a partition B, a disferentiable decoder f RRd > RRd is said be compositional w.r.t. B whenever the Jucobian Df(z) such
that for all E [dz], B E B, Rdd, we have
0 = DB- fz=)((z)=0.
where B is the complement of B
In other words, each line of the Jacobian can have nonzero values only in one block B E B. Note that this nonzero block can change with different values of z.
The next result shows that additive decoders form superset of C2 compositional decoders (Brady al. [7] assumed only C.). Note that additive decoders are strictly more expressive than C compositional decoders because some additive functions are not compositional, like Example 3 for instance.
Proposition 8 (Compositional implies additive). Given partition B, if f RRd, Bd is composi- tional (Definition 14) and C2, then it is also additive (Definition 1).
Proof. Choose any i [dz]. Our strategy will be to show that D2 f. is block diagonal everywhere d and use Proposition 7 to conclude that f. is additive.
19

---

Choose an arbitrary& Zo E R By compositionality, there exists block B E B such that D (z O. We consider two cases separately:
Case Assume Da f((n)) o O. By continuity of DB fi there exists an open neighhorhood of 2o. U, S.t. for all E U, Da f(() A O. By compositionality. this means that, for all U. De f: 0. When a function is zero on an open set, its derivative must also be zero, hence DD fi = 0. Because C²2 the Hessian symmetric so that we also have f:=D)((z)) = We can thus conclude that the Hessian D fi fzo) is such that all entries are zero except possibly for D* fi(zo) Hence, D² f:( (z)) is block diagonal with blocks in B.
Case 2: Assume D=fo(z))= O. This means the whole row of the Jacobian is zero, i.e. Df((==) = 0. By continuity of Df:, we have that the set V == (D))))-((()) is closed. Thus this set decomposes as V = V U 0v where V and 0V are the interior and boundary of V. respectively.
Case 2.1: Suppose Zo E Vo. Then we can take a derivative so that D fz() O, which of course means that D2 f((()) is diagonal.
Case 2.2: Suppose 2u E OV. By the definition of boundary, for all open set U containing zu, U intersects with the complement of V. i.. (Dft) ) (R'. \ {0}). This means we can construct a sequence {= S V which converges Zg- By Case 1, we have that for all k 2 (zx) is block diagonal. This means that limg-+x D2 f. (zx) is block diagonal. Moreover, by continuity of D2 fi. we have that lim f(x()))) = f(()) Hence D2 (zo) is block diagonal.
We showed that for all 2u E Rd: D2 fzo) is block diagonal. Hence, is additive by Proposition 7.
A.4 Examples of local but non-global disentanglement
In this section, we provide examples of mapping v -itaii + Zuain that satisfy the local disentangle- ment property of Definition 4, but not the globul disentanglement property of Definition 3. Note that these notions are defined for pairs of decoders f and f. but here we construct directly the function which is usually defined as f-1 o f. However, given v we can always define f and f to be such that f-1 o f = v: Simply take f(z) := [z1....,
Ed.; : 0, ...,...... Oo E IRd, and := f o v. This construction however yields decoder f that is not sufficiently nonlinear (Assumption 2). Clearly the mappings a that we provide in the following examples cannot be written as compositions of decoders f f where f and f satisfy all assumptions of Theorem 2, as this would contradict the theorem. In Examples 5 & 6, the path-connected assumption of Theorem 2 is violated. In Example 7, it is less obvious to see which assumptions would be violated.
Example (Disconnected support with changing permutation). Let 2 +
BR2 R. 2 2 U 2(2) C 2 where 1(1) {z E IR22 21 < 0 (nndd= z2 < <0) and 212) {= R and 22 2
1}. Assume
i E 201) v(2)= = z2),
(31)
(1), 2 E 2()
Step 1: u is a diffeomorphism. Note that v is its own inverse. Indeed,
v(v(z)) 22) (2111 if E 211 ((22,21) (z), if 2(2)
Thus, v is bijective on its image. Clearly, v is C², thus w is also C², Hence, v is C (--iifeemoorphimm
Step 2: v is locally disentangled. The Jucobian of v is given by
1
07
if 2(1)
0 1
Do(2) =
(32)
17
if E 2(2)
1 0
20

---

1.0
0.5
20
-0.5
-1
0.5
3
0.5
1.0
21
Figure 6: Illustration of 3 = 110) U 2 in Example 7 where 20) is the blue region and g(0) is the orange region. The two black dots correspond (-1/2, -1/2) and (1/2, -1/2), where the function (2 Ø2) is evaluated to show that it is not constant in
which is everywhere a permutation matrix, hence u is locally disentangled.
Step 3: is not globally disentangled. That is because Uı (1 #2) depends on both 21 and ø2- Indeed, if 22 = 0, we have that vi v1(-1,0) = -1 -1 + 0 = v1 0). Also, if 21 = 1, we have that vi (1, 1) = 1 2 (1,
Example 6 (Disconnected support with fixed permutation). Let v 2 2(2) S I where 2 {z
R s.l. 2 = 2 U
RR z2 $ 03 and 2(2) {
R2
22 Z 13. Assume
022) = = + 1(= e Z(2)).
Step 1: is diffeomorphism.. The image of is the union of the following two sels: 2(1)
v(Z(1)) = 2() and z : v(((2) {= E RR #2 2 2}. Consider the map w z z
2 defined as w(2) = 2 - e== un(v(2)) = =0() 1(n(z) E 2(2))
E 2.))). We now show that w is the inverse of v:
(33)
= + 1(z E 2(2)) 1(2 1(2 2(2)) E Z(0)).
(34)
If x E 2()), have
w<((2))
1(z E 2(2)
(35)
= - 1( E 2(22) 2.
(36)
If 2 E 20,, we have
w(v(2)) = E 2(()) = =
(37)
v. A similar argument can be made to show that v(w(z)) = z. Thus w is the inverse v. Both v and its inverse are C2, thus a C2 di-iiffoomorphssm on its image.
Step 2: u is locally disentangled. This is clear since Dv((z) = I everywhere.
Step 3: v not globally disentangled. Indeed, the function v (21, x2) = 21 + 1(2 E Z(2) is not constant in 22-
s.t. 2 2b) u 2(00 where 2(0) and 2(0) are respectively the blue and orange regions of Figure 6. Both regions contain their boundaries. The function v is defined as follows:
Example 7 (Connected support). Let V 2 -
U1 (2) = 21
(38)
U2 = (22+1)++1 +1
ifz E 2(1)
(39)
i 200
Step 1: v is a diffeomorphism. Clearly, vn is C2. To show that v2 also is, we must verify that va (2())
is C at the frontier between 20) and 2(0), i.e. when E [144, 1] x {0).
(2(2) is continuous since
(22 + ++11++1
= g
(40)
2
===
=2=0
21

---

(2) is C' since
(22 + 1 +
= (22 + 11) +1)22==0 = = =x²
2-=0
(
====
(41)
(2) is C since
(22 (22+1)++1]
= = =1 2²2
(6
2===
(42)
2
===0
We will now find an explicit expression for the inverse of v. Define
w1 (1(2)= := 21
(43)
v 2 1 1, ifz v(2 log(z)).
W2 (2)
(44)
if E v v(2()))
is straightforward to see that w(v(2)) = z for all z E 2. One can also show that w is C² at the boundary between both regions vl v() and v b(2(1)), i.e. when z E [1/4, {1}.
Since both and its inverse Step 2: is locally disentangled. The Jacobian of v
C², is C² Ce-diffeomorphism.
0
ifz 2
0
22 +1
Dv(z)=
(45)
e if z E 2
0
which is permutation-scaling matrix everywhere on Ż. Thus local disentanglement holds.
Step 3: is not globally disentangled. However, U2(21, 22) is not constant in 21. Indeed,
1
(22 + 1)²
5 # e- 1
U2 2 1
(46)
2
8
x2=-1/2 1/2
Thus global disentanglement does not hold.
A.5 Proof of Theorem
Proposition 9, Suppose that the data- gentragnn process satisfies Assumption 1, that the learned
decoder f Rdd + Rdd is a C2 CF-diffeomorphism onto its image and that the encoder 9 Rdd - Rd, is continuous. Then, if f and ğ solve the reconstruction problem on the training distribution, i.e. Etrain|| |: f((()))1 0, we have that f(2uain) f(2rrinn) and the map = ff11 is C² de-diffeomorphism from 2iin to 2unin
Proof. First note that
grmin æ f((())) gtain [[(z) f(
(47)
which implies that, for -amost every E 2mmn
f(z) f()(f(z)))
But since the functions on both sides of the equations are continuous, the equality holds for all E guain This implies that f (uain) = -fo ğ f(zuan = f( (
By Remark 3, the restrictions f Z + f( ( and f irain + f(uuan are C2-
diffeomorphisms and, because f(zuain) f( 2uain, their composition = -- uuain guain is a well defined C2 de-diffeomorphis (since _ didifffoooosssss are closed under composi- tion).
22

---

Theorem 1 (Local disentanglement via additive decoders), Suppose that the data generating process satisfies Assumption I, that the learned decoder f TR - TR is di-diffeomorrhism, that the encoder 9 Rd + Rd= is continuous, that both and f are additive (Definition 1) and that f is sufficiently nonlinear formalized by Assumption 2. Then, if and 9 solve the reconstruction problem on the training distribution, i.e. [ f(](2))11 O, have that f is locally B-disentangled w.r.t. f (Definition 4)
Proof. We can apply Proposition 9 and have that the map v = f-r o f is Ci-difooomorphhis
from irain to Ztnain This allows one to write
fo fon(z) f(z) Vz E 2miin
(48)
Σ (B) (vB(()) = Σ f(B) (zB) Vz E Stmin BEB
(zB)
(49)
BEB
Since Zuain is regularly closed and is diffcomorphic to inain by Lemma 4, we must have that zirin S (2. Moreover, the left and right hand side of (49) are C²2 which means they have uniquely defined first and second derivatives on (Zurainoo by Lemma This means the derivatives are uniquely defined on irain
Let Êtrain Choose some E B and some j E J. Differentiate both sides of the above equation with respect to æj, which yields:
ΣΣ (on))
(50)
BEBİEB
Choose J' E EB (J} and E J' Differentiating the above w.r.f. Zy yields
Σ BEB B
D ( (m o(()+ +
Σ
D2. (1)
VEB
Σ BEB iiB
Σ
D. (00)0(((((()))000000100011
Σ
(B) (atr://hhhhhll0000m////0/0010100
=0, (51)
()
where B² == B² n
[(i, i' < i}. For the sake of notational conciseness, we are going to refer to SB and Sg as S and S (Definition 11). Also, define
Sz = U B2
(52)
BEB
Let us define the vectors
Vi {L, ax))) o
(53)
Vi Vie (E, (i(z) (D),,(()))) BB B, V(i,t') E B2, (z) This allows us to rewrite, for all k {1,...,du) dzf
(54)
(55)
Σ BEB
D.f. (v) (R (020)()))) + Σ
(
(B)
0.
LEB
(L,')
(56)
We define
w(2, k) = (DDff (B) (xB))eB () (D,
( ((")) (x8))(())))))) )8B
(57)
M(z) = [[](E))) [=))))) G. (s)
(58)
23

---

which allows us to write, for all & E {I....,d}}
M(z)w(v(2), k) = 0.
(59)
We now recognize that the matrix (v((z)) of Assumption 2 is given by
((o(2))* = [u(e(22,1) w(v(2),d.) dz)]
(60)
which allows to write
((z))(o(x) = 0 ((
(61)
(62)
Since (u)))) has full column-rank (by Assumption 2 and the fact that v(z) Z there exists rows that are linearly independent. Let K be the index set of these rows. This means (())))) an invertible matrix. We can thus write
W(v(z)) M({)) =0
(63)
(W(o(z)) (W(())))))W())))))oo)) M(z) (W(6()))k,)--0
(64)
M(2) =0,
(65)
which means, in particular, that, Vi (1,...., d.). (,(2) = i, Vi E (1.....d..}, V(.)") E 8 (r -0
(66)
Since the is diffeomorphism, its Jacobian matrix Dv(z) invertible everywhere. By Lemma this means there exists permutation such that, for all j. ())m()(2) + O. This and (66) imply that
v(j.j') ES, D)wxc)(2) Dy Dyux()(()) 00
(67)
40
v(.J') E (z)=()(z) =0.
(68)
To show that Dv(z) is B-block permutation matrix, the only thing left to show is that π respects B. For this, we use the fact that, VB E V(i, E B%, (z) = (recall (z) = 0). Because (2) = (2), we can write
S", v(i,t') S ( (2))
+ (z) 0.
(69)
We now show that if (i.j') E 5e (indices belong to different blocks), then (T(j), T(J') E S (they also belong to different blocks). Assume this false, i.e. there exists (jo. S such that (I(jo), क ((6))) E S. Then we can apply (69) (with i == T(jo) and i' == x(%)) and get
D VrO0) (z)) v (z) + (ro0) (T(z) (z) = 0,
(70)
40
where the left term in the sum is different of 0 because of the definition of w. This implies that
D3 (x (z) (x) (2) + 0,
(71)
otherwise (70) cannot hold. But (71) contradicts (68). Thus, we have that,
(j.j') S - (f()),(('))) S
(72)
The contraposed
(73)
(.J') - (T)),)))
(74)
From the above, is clear that म respects which implies that respects B (Lemma 10). Thus Dv(z) is a B-block permutation matrix.
D
24

---

Lemma 10 (B-rescecting permutations form a group). Let B be a partition of (1...., and R. be permustation of (L.....d, ..) that respect B. The following holds:
.......,d,, d,) and let
1. The identity permutation respects B.
2. The composition π T respects B.
3. The inverse permutation -- respects B.
B. The second statement follows since for all B E B, i (BB E B and thus r(*(B)) E B. We now prove the third statement. Let B B. Since π is surjective and respects B, there exists a B' E B such that क (B') B. Thus, -1 (B) - (mt')) B' B.
Proof. The first statement is trivial, since for all B B, e(B)
A.6 Sufficient nonlinearity v.s. sufficient variability in nonlinear ICA with auxiliary variables
In Section 3.1, we introduced the "sufficient nonlinearity" condition (Assumption 2) and highlighted its resemblance to the "sufficient variability" assumptions often found in the nonlinear ICA liter- ature [30, 31, 33, 36, 37, 42, 73J. We now clarify this connection. To make the discussion more concrete, we consider the sufficient variability assumption found in Hyvärinen et al. [33]. In this work, the latent variable z is assumed to be distributed according to
p(= u)
II pz(z, u)
(75)
In other words, the latent factors are mutually conditionally independent given an observed auxiliary variable u. Define
w(z u) = (0oz) log u)
22
log pi u) iE(d:)
E 2d,
(76)
sE[d,]
We now recall the assumption of sufficient variability of Hyvärinen et al. [33]:
Assumption 3 (Assumption of variability from Hyvärinen et al. [33, Theorem 1]). For any z E R4,,
there exists 2d= + values of u, denoted by (0) (1)). ...,uu( (2d.) such that the 2d, vectors
w(2,u (2)) - w(z (0)), )..)............... ...., w0z u (2d.)) w(2, u (0))
(77)
are linearly independent.
To emphasize the resemblance with assumption of sufficient nonlinearity, we rewrite it in the special case where the partition =
Note that, in that case, == d2
ΣBEB [B[(B)++) 2dz-
Assumption 4 (Sufficient nonlinearity (trivial partition)). For all z E Zunin f is such that the following matrix has independent columns (i.e. full column-ranky
(()) = [D. (z.)
[D&s (zi) [Eld,1]
Rdxx2dd
(78)
(Ejd,)
One can already see resemblance between Assumptions 3 & 4, e.g. both have something to do with first and second derivatives. To make the connection even more explicit, define w(z, k) to be the kth row of (x)) (do not conflate with w(2, u)). Also, recall the basic fact from linear algebra that the column-rank is always equal to the row-rank. This means that W(z) is full column-rank if and only if there exists k1, kid, E [dz] such that the vectors w kn) k1),-.., w kzd.) are linearly independent. It is then easy to see the correspondance between w 2, k) and w(z, u) w(2 ())
(from Assumption 3) and between the pixel index & E [d,] and the auxiliary variable
We now look at why Assumption is likely to satisfied when d, >> d.. Informally, one can see that when d, is much larger than 2dz, the matrix (z)) has much more rows than columns and thus becomes more likely that we will find 2d, rows that are linearly independent, thus satisfying Assumption 2.
25

---

14
1.0
00
04
140
02
Figure 7: Numerical verification that f 1-1, [0, + R from Example is injective (left), has full rank Jacobian (middle) and satisfies Assumption 2 (right). The left figure shows that f injective on the square [-1,0] x [0. 1] since one can recover uniquely by knowing the values of (z) and fz(2), i.e. knowing the level sets. The middle figure reports the det
Df(=)) (columns of the Jacobian are normalized to have norm 1) and shows that it is nonzero in the square [-1,0] [0, 1]. which means the Jacobian is full rank. The right figure shows the determinant of the matrix W(z) (from Assumption 2, but with normalized columns), we can see that it is nonzero everywhere on the square 1-1, 0] [0, 1]. We normalized the columns of Df and W so that the determinant is between 0 and 1.
A.7 Examples of sufficiently nonlinear additive decoders
Example 8 (A sufficiently nonlinear! f -Example 3 continued), Consider the additive function
21
(z2 + 1)
2/
(Z2 + 1)
f(z)
(79)
(Zz + 1)3 [
(22 + 1)
We will provide a numerical verification that this function a diffeomorphism from the square [-1,0] [0, to its image that satisfies Assumption 2.
The Jacobian of f is given by
1
2x
2(x2
+ 1)
= Df(z)=
3z 2
3(22
1)2
(80)
4(22 42
+ 1)3
and the matrix (2)) Assumption 2 is given by
1
2z
2(22 + 1)
(x))
(81)
3z
62
3(z2 + 1)2
6(22 + 1)
42
122
4(22 + 1)3
12( + 1)2
Figure presents numerical verification that f is injective, has a full rank Jacobian and satisfies Assumption 2. Injective f with full rank Jacobian is enough to conclude that f is diffeomorphism onto image.
Example 9 (Smooth balls dataset is sufficiently nonlinear Example 4 continued). We implemented a ground- truth additive decoder f 5) + 6464664- which maps to 64x64 RGB images consisting of two colored balls where 2 and 22 control their respective heights (Figure 8a). The analytical form of I can be found in our code base. The decoder f is implemented JAX [6] which allows for its automatic differentiation to compute Df and D2 (Figures 8h & 8c). This allows numerically that if is sufficiently nonlinear (Assumption 2). Recall that this assumption requires that (=)) (defined in Assumption has independent columns everywhere. To test this, we compute Vol(=) := =1 det((W<))) (z))) over grid of values of and verify that Vol(z) > everywhere (Figure 8d). Note that Vol(z) corresponds to the 4D volume of the parallelepiped embedded in 661661+3 spanned by the four columns of (z). This volume >> if and only if the columns linearly independent. Note that normalize the columns of (xz) that they have norm
verify
26

---

of one. follows that Vol(z) is between O and 1 where means the vectors are orthogonal, i.e. maximally independent. The minimal value of Vol(z) over the domain of f is 0.97, indicating that Assumption 2 holds.
(a)
(b)
(c)
1.04
seri (@W1))
270
1000
2673
3576
0.00
E.65
3176
0375
6981
147/
2980
0373
2976
2972
1171
2.00
0.26
6523
C200 8515 add0 0.625 0.50 0075
8330
Height uecond batl
He gri second bell 02)
(c)
(d)
Figure 8: Figure (a) shows an image the synthetic dataset of Example Figure (b) shows the derivative of the image w.r.t. 21 (the height of the left ball) where the color intensity of each pixel corresponds the Euclidean norm along the RGB axis. Figure (c) similarly shows the second derivative of the image w.r.t. 21. Figure (d) is contour plot of the function / det W (W(2) (2)) where (z) is defined in Assumption 2 (here columns are normalized to have unit norm). The smallest value of / dedetWWe (x)) W Wz))] across domain is z 0.97, indicating that Assumption 2 is satisfied. See Example 9 and code for details. Figure 8e is a higher resolution rendering of the red region of Figure 8d (to make sure there is no singularity there).
A.8 Proof of Theorem
We start with simple definition:
Definition 15 (B-block permutation matrices). A matrix A E Rdxd is B-block permutation matrix if is invertible and can be written as A C.. where P, the matrix representing the B-respecting permutation ß (Pa = Ex(s)) and C dxd (See Definitions & 11).
The following technical lemma leverages continuity and path-connectedness show that the block- permutation structure must remain the same across the whole domain. It can be skipped at first read.
Let be connected topological space and let M C Raxd be a continuous function. Suppose that, for all e C, M(c) is an invertible B-block permutation matrix (Definition 15). Then, there exists B-respecting permutation æ such that for all e C and all distinct B, B' E B. M (c)a(B)) O.
Lemma 11.
Proof. The reason this result is not trivial, is that, even if M(c) is B-block permutation for all the permutation might change for different c. The goal of this lemma is to show that, if C is connected and the map M.)) is continuous, then can find single permutation that works for all E C.
First, since C connected and M is continuous, its image, M(C), must be connected (by [56,
Theorem 23.5]).
27

---

Second, from the hypothesis of the lemma, we know that
M(C) CA
U
SH Px
(singular matrices):
(82)
(E)(8)
where ((B) is the of B-respecting permutations and RexdP. (MPx M Rdxd). We can rewrite the set A above as
A =
U TE((B)
(R Pr
{singular matrices))
(83)
We now define an equivalence relation over B-respecting permutation: x iff for all B B, и (B) = (B). In other words, two B-respecting permutations are equivalent if they send every block to the same block (note that they permute elements of given block differently). We notice that
π
ddxd dddd P- = SB
Pa'
(84)
Let ((B)/ ~ be the set of equivalence classes induce by ~ and let stand for one such equivalence class. Thanks to (84), we can define, for all 11 E G(B)/ the following set:
Vn == Rdxd Pa \ {singular matrices}, for some π E II,
(85)
where the specific choice of π e is arbitrary (any T' E would yield the same definition, by (84)). This construction allows us to write
A U Vn
(86)
(B)
We now show that (Vír) (B) forms a partition of A. Choose two distinct equivalence classes of permutations 1I and 11' and let π E and T' E 11' be representatives. We note that
P. RR
mddxdp P. (singulaa matrices],
(87)
since any matrix that both in Raxd P, and Raxd P. must have at least one row filled with zeros. This implies that
Vn VIT 0,
(88)
which shows that (Vn)nεe(B))>> indeed partition of A. Each Vn closed in A (wrt the relative topology) since
Vn
Radd Px
{singular matrices) An
Rdxd Pr
(89)
closed in dxd
Moreover, Vn is open in A, since
VIT = A\ U VIT 11'
(90)
closed in
Thus, for any E 6(B)/
~M the sets Vn and Unr Vir forms a separation (see [56, Section 23]). Since (C) a connected subset of A, must lie completely in Vn or Unr VIP, by [56, Lemma 23.2]. Since this is true for all IT, it must follow that there exists a IT" such that M(C) S Vnr, which completes the proof.
Theorem 2 (From local to global disentanglement). Suppose that all the assumptions of Theorem I hold. Additionally, assume Zuain path-connected (Definition 8) and that the block-specific decoders (BB) and ((B are injective for all blocks B E B. Then, if f and 9 solve the reconstruction problem on the training distribution, i.e. Eimin x f(g(x)) O, we have that f is (globally)
B- B-disentangled w.r.t. f (Definition 3) and, for all B B B.
(zB) =f ((a(B)
+ (B) for all zn E train
(8)
28

---

where the functions (n) are from Defintion 3 and the vectors c B are constants such that EBEB cB 0. We also have that the functions Dn(B)² 2gain Z are C2 d'-diffeomorphisms and have the following form:
(B)
(x(B)(ZB) = (f - BB)),, for all 2B E 2g
Proof. Step 1 Showing the permutation π does not change for different Theorem 1 showed local B-ddisentanleement, i.e. for all E grrai,, Dv(z) has B-block permutation structure. The first step towards showing global disentanglement is to show that this block structure is the same for all z E Erain (a priori, π could be different for different z). Since v is C2, its Jacobian Dv(2) is continuous. Since 2umin is path coth-comne Iman must also be since both sets are diffeomorphic. By Lemma 11, this means the B-block permutation structure of Dv(=) is the same for all z E grain (implicitly using the fact that path-connected implies connected). other words, there exists permutation π respecting B such that, for all z E graain and all distinct B, B' E B, (BU=(B') (z) O.
Step 2 - Linking object-specific decoders. We now show that, for all B E B, (zB)
((=()) + c(B) for all E Euain, To do this, we rewrite (50) as
Df()( = Σ (vn())))))))
(91)
BEB
but because B + TJJ) = Dyub(z) = peomut-permttttion structure), we get
(z)) = D ((())) (v)(
(92)
The above holds for all J E B. We simply change by B in the following equation.
D Df (zB) (zB) (f
(B)) (U=(m) (B))
(93)
Now notice that the r.h.s. of the above equation is equal to (f vr(B)). We can thus write
Df
D(F ((((((B))))))))))))()))))(())( Ux(B))() for all E 2triin
(94)
Now choose distinct 2, 00 E Zunin Since Zunin is path-connected, zuain also since they are diffeomorphic. Hence, there exists a continuously differentiable function & : [0, 1] + 2train such that (00)= = = and ((1) z. We can now use (94) together with the gradient theorem, a.k.a. the fundamental theorem of calculus for line integrals, show the following
S Df (f(()) @b(()dt= S/' D(F(M(B)) ())())(((z)) ((t)dt
(95)
f(B) - (1)) = ((m(B)) (x) - pla(B)) t. (2)) ((B) f(((B)) (x(B)(2) + (x f(r(B)) (x())
(96)
(97)
constant in
((B) f(T(B)) (x(B)(2) + (B)
(98)
which holds for all z E irain
We now show that Σ BEB (B)) -. Take some 20 E Stain, Equations (49) & (98) tell us that
Σ f (B) (B)(2()))) = Σ (B)) (z))
(99)
BEB
BEB
Σ ((x(
(U+(m((Z)) +
Σ (
(100)
BES
BEB
Σ (B))
(vn(())))) + Σ (B)
(101)
BEB
BEB
0 = Σ (B)
(102)
BEB
29

---

Step 3 - From local to global disentanglement. By assumption, the functions f( BB + R are injective. This will allow us show that (z( depends only on zB. We proceed by contradiction. Suppose there exists (zB, zB=) E 2tmin and 2 (x(B) > + (n(B) zbg)) This means
ze such that (zB, 2 zg=) ) 2train and
( Un(B) (zB, BB) + c = f(B) f ((((B) (Z(B) (g=) + (B))
|( ( (2B, zB))
(Ur(B) (" 2B))
which is contradiction with the fact that ((((B)) is injective. Hence, (x) depends only on zn. We also get an explicit form for (x)
( (f( (zn) - (B) Vx(B) for all E t
(103)
We define the map DR (zB) = (fx(B)) (f(B) - (!)) which is from 2mmi to *(B) This allows us to rewrite (98) as
(zn) = ((T(B1) (z(B)(Zn) + cB for all zn E train
(104)
Because f(B) is also injective, we must have that (BB train Zuain is injective as well. (B)
We now show that (B) is surjective. Choose some (B) E t
(B) We can always find (B)=
such that (Z=( Bny)) E grrain Because v : -train + zimin surjective (it is a diffeomorphism),
there exists _ ğtrain such that v(20) (B)))) By (103), have that
(x) = (xx
(105)
which means (=))) Zx(B).
We thus have that (x is bijective. is diffeomorphism because
det ( = det (BU+(H)(z) * Y 2tmin
(106)
where the first equality holds by (103) and the second holds because v is diffcomorphism and has Macck-permutation structure, which means it has nonzero determinant everywhere on Itrain and is equal to the product of the determinants of its blocks, which implies each block D (g must have nonzero determinant everywhere.
Since Ü=(B) ttain diffeomorphism.
a(B) bijective and has invertible Jacobian everywhere, it must be
A.9 Injectivity of object-specific decoders v.s. injectivity of their sum
We want to explore the relationship between the injectivity of individual object-specific decoders f ()) and the injectivity of their sum, i.e. EBEB (B)
We first show the simple fact that having each (BB injective is not sufficient have Σ BEB (B) injective. Take (zn) = (BB) nn where (BB) Rddx|Bl has full column-rank for all B E B. We have that
Σ (zB)
(zB)
Σ (B) BB)
ZB = [W (B
(107)
BEB
BEB
where it is clear that the matrix (B) each (B) is. This is the case, for instance, if all (BB) have the same image.
W E Xd is not necessarily injective even if
We now provide conditions such that Eses (B) injective implies each ((B injective. We start with simple lemma:
Lemma 12. If g h is injective, then h is injective.
Proof. By contradiction, assume that h is not injective, Then, there exists distinct æ1, E Dom (h) such that h(x1) h(x2). This implies go h(21) go h(z2), which violates injectivity of g h. L
30

---

The following Lemma provides condition on the domain of the function EBEB f( zuain so that its injectivity implies injectivity of the functions f(B)
Lemma 13. Assume that, for all B E B and for all distinct 2n, z'B E B there exists B such that (ZB, zBc ) (Z ZB=) E Zuai, Then, whenever EBEB f(B) is injective, each f(B) must be injective.
Proof.
Notice that f(z) = E { (zz) can be written as f = SumBlocks f(z) where (zB
f(z) =
and (um (Bi),
Bc)) = Σ
(
(108)
((B:)
BEB
Since is injective, by Lemma 12 f must be injective.
We now show that each f( must also be injective. Take ZB: 2/B E 2mth such that ((B) = f (x By assumption, we know there exists a ZB s.t. (zB, xge) and (z'g. xge) are in Z By construction, we have that f((BB, =B=)) = f((#// ZB=)). By injectivity of f. we have that (zg, > + (z'n >B)), which implies zB + n'n, i.e. f(B) is injective.
A.10 Proof of Corollary 3
Corollary (Cartesian-proddes extrapolation). Suppose the assumptions of Theorem 2 holds. Then,
for all E CPEs( Stmin)
Σ (( (zB) = Σ (((())
(11)
BEB
BEB
Furthermore, (PEalZitii)) C 2 then ((CPEg(2tmn) S )
Proof. Pick CPE(2ual) have that, for all B e B,
By definition, this means that, for all B B, zB 2mann We thus
((B) f(x(B)) (x(B)(ZB) + (B))
(109)
We thus sum over B to obtain
Σ
f LB (2B)
Σ f(x(B)) (x)BB( Σ (B))
(110)
BEB
=0
Since E CPE((2 (main) was arbitrary, we have
for all mai)))))n),
Σ (B))
(2B)
Σ
o (n((B) (x(B)(2B)
(111)
BEB
BEB
f(z) = f о 0(z).
(112)
where ö (2 + CPEg( ( (tain) is defined as
TB, (Zx-1(B.)
b(z) ==
(113)
Ün, (2x- (BB)
The map is a diffeomorphism since each (n(B) is a diffeomorphism from 2tain to Z By (112) we get
xBy)
f (CPPE (2tmin)) = o ü (CPE (
(114)
and since the map & is surjective we have (CCPE,s Ztmin) = CPE E (Zunii)) and thus
f(CPE& (2tmin) = (C
(115)
Hence if (PEg(Ztuin t then f(CPE f f(zto)).
31

---

A.11 Will all extrapolated images make sense?
Here a minimal example where the assumption CPEg( (Z (
- Z is violated.
Example 10 (Violation of CPEg( 2 g ). Imagine z = (z1, 22) where 21 and 22 are the r- positions of two distinct balls. It does not make sense to have two balls occupying the same location in space and thus whenever
22 we have (z1,Zz) & Z But if (1, and (2, 1) are both in Z implies that (1, 1) and (2, 2) are CPE( 2 guain) which is a violation of CPEp (Zuii)) g
A.12 Additive decoders cannot model occlusion
We now explain why additive decoders cannot model occlusion. Occlusion occurs when an object partially hidden behind another one. Intuitively, the issue is the following: Consider two images consisting of two objects, A and B (each image shows both objects). In both images, the position of object A is the same and in exactly one of the images, object B partially occludes object A. Since the position of object A did not change, its corresponding latent block = is also unchanged between both images. However, the pixels occupied by object do change between both images because of occlusion. The issue is that, because of additivity, 2, and cannot interact to make some pixels that belonged object "disappear"
be replaced by pixels of object In practice, object-centric representation learning methods rely masking mechanism which allows interactions between and z (See Equation in Section 2). This highlights the importance of studying this class of decoders in future work.
B Experiments
B.1 Training Details
Loss Function. We use the standard reconstruction objective of mean squared error loss between the ground truth data and the reconstructed/geenraaedd data.
Hyperparameters. For both the ScalarLatents and the Latenss dataset, we used the Adam optimizer with the hyperparameters defined below. Note that maintain consistent hyperparameters across both the Additive decoder and the Non-Additive decoder method.
ScalarLatents Dataset.
Batch Size: 64
Learning Rate: 1 10-3
Weight Decay: 5 x 10-4
Total Epochs: 4000
Blontt Dataset.
Batch Size: 1024
Learning Rate: 1 x 10
Weight Decay: 5 10
Total Epochs: 6000
We the following architectures for Encoder and Decoder across both the datasets (ScalarLatents. BlockLatents). Note that for the ScalarLatents dataset we train with latent dimension d, and for the Blockl Latents dataset train with latent dimension d, = 4, which corresponds to the dimensionalities of the ground-truth data generating process for both datasets.
Model Architecture.
Encoder Architecture:
RestNet-18 Architecture till the penultimate layer (512 dimensional feature output)
Stack of 5 fully-connected layer blocks, with each block consisting of Linear Layer ( dimensions: 512 512), Batch Normalization layer, and Leaky ReLU activation (negative
slope: 0.01).
32

---

Final Linear Layer (dimension: 512 x dz) followed by Batch Normalization Layer to output the latent representation.
Decoder Architecture (Non-additive);
Fully connected layer block with input latent representation, consisting of Linear Layer (dimension: dg x 512), Batch Normalization layer, and Leaky ReLU activation (negative slope: 0.01).
Stack of 5 fully-commn conneceddee layer blocks, with each block consisting of Linear Layer ( dimensions: 512 512), Batch Normalization layer, and Leaky ReLU activation (negative
slope: 0.01).
Series of DeConvolutional layers, where each DeConvolutional layer is follwed by Leaky
ReLU (negative slope: 0.01) activation.
DeConvolution Layer (Cou 64, Cout: 64, kernel: 4; stride: 2; padding: 1)
DeConvolution Layer (Cin 64, Cout: 32, kernel: 4; stride: 2; padding: 1)
DeConvolution Layer (Cini 32, Couti 32, kernel: 4; stride: 2; padding: 1)
DeConvolution Layer (Cin 32, Cout: 3, kernel: 4; stride: 2; padding: 1)
Decoder Architecture (Additive): Recall that an additive decoder has the form f(z) EBEB (zB). Each f(B) has the same architecture as the one presented above for the non- additive case, but the input has dimensionality [B] (which or depending on the dataset). Note that we do not share parameters among the functions (B)
B.2 Datasets Details
We use the moving balls environment from Ahuja et al. [2] with images dimension 64 x 64 x with latent vector (z) representing the position coordinates of each balls. We consider only two balls. The rendered images have pixels in the range [0. 255].
LcalarLatents Latents Dataset. We fix the x-coordinate of each ball to 0.25 and 0.75. The only factors
varying are the y-coordinates of both balls. Thus, 2 R²² and B = ((), [2)) where 2 and z2 designate the y-coordinates of both balls. We sample the y-coordinate of the first ball from continuous uniform distribution as follows: 21 Uniform(O, 1). Then we sample the y-coordinate of the second ball as per the following scheme:
22 ~ Uniform 1)
if 21 S 0.5 else
Uniform (0, 0.5)
Hence, this leads to the L-shaped latent support, i.c., train = 00, 1] x [0, 11 [0.5, 1] x [0.5, 1].
We 50k samples for the test dataset, while 20 samples for the train dataset along with 5 samples (25% of the train sample size) for the validation dataset.
Blockts Dataset. For this dataset, allow the balls to move in both the x, y directions, so that & E 4 and B {{1, 2}. (3, 4}}. For the case of independent latents, we sample each latent component independently and identically distributed according uniform distribution over (0, 1), ie. 2; ~ (O1 1). We rejected the images that present occlusion, i.c. when one ball hides another one.?
For the case of dependent latents, we sample the latents corresponding to the first ball similarly from the same continuous uniform distribution, i.e, 21, Uniform (0, 1). However, the latents of second ball are a function the latents of the first ball, as described in what follows:
Uniform (0, 0.5)
if 1.25
(2f x3 1.0
23
Uniform (0.5, 1)
if 1.25
(zî 23) 1.0
NNote that, in the independent latents case, the latents are not actually independent because of the rejection step which prevents occlusion from happening.
33

---

(0.5f if 1.25 x (z + 23) 1.0 Uniform (0, 0.5)
1.25 x (z) + <
Intuitively, this means the second ball will be placed in either the top-left or the bottom-right quadrant based on the position of the first ball. We also exclude from the dataset the images presenting occlusion.
Note that our dependent BlockLatent setup is same as the non-linear SCM case from Ahuja et al. [3].
We use 50k samples for both the train and the test dataset, along with 12.5k samples (25% of the train sample size) for the validation dataset.
Disconnected Support Dataset. For this dataset, we have setup similar to the ScalarLatents dataset; we fix the x-coordinates of both balls 0.25 and 0.75 and only vary the y-coordinates so that # E I22 We sample the y-coordinate of the first ball (z1) from Uniform(O, 1). Then we sample the y-coordinate of the second ball (zg) from either of the following continuous uniform distribution
with equal probability; Uniform(O, 0.25) and Uniform(0.77 (0.75. 1). This leads to a disconnected support given by gtrain = [0, 1] x [o. 1] [0.25, 0.75] x 0.75).
We use 50k samples for the test dataset, while we use 20k samples for the train dataset along with 5 samples (25% of the train sample size) for the validation dataset.
B.3 Evaluation Metrics
Recall that, to evaluate disentanglement, we compute a matrix of scores (#B.B' E Rext where & is the number of blocks in B and s is a score measuring how well we can predict the ground-truth block ω from the learned latent block y = 9r (ar) outputted by the encoder. The final Latent Matching Score (LMS) is computed LMS arg max 1 Σ BEB (B,) where Gg is the set of permutations respecting B (Definition 2). These scores are always computed on the test set.
Metric LMS As mentioned in the main paper, this metric is used for the ScalarI.aa Latenss dataset where each block is 1 I-dimensional. Hence, this metric is almost the same as the mean correlation coefficient (MCC), which is widely used in the nonlinear ICA literature [30, 31, 33, 36, 42], with the only difference that we use Spearman correlation instead of Pearson correlation as a score 8, The Spearman correlation can capture nonlinear monotonous relations, unlike Pearson which can only capture lincar dependencies. We favor Spearman over Pearson because our identifiability result (Theorem 2) guarantees we can recover the latents only up to permutation and element- wisse invertible transformations, which can be nonlinear.
Metric LMS This metric is used for the BlockL dataset. For this metric, we take s, to be the R² score of a Regression Tree with maximal depth of 10. For this, we used the class sklearn. tree. Decisi prrcisionTreeen TeeRReggesso from the sklearn library. We learn the parameters of the Decision Tree using the train dataset and then use it to evaluate LMS metric on the test dataset. For the additive decoder, it easy to compute this metric since the additive structure already gives a natural partition B which matches the ground-truth, However, for the non- non-additive decoder, there is no natural partition and thus we cannot compute LMS directly. To go around this problem, for the non-additive decoder, we compute LMS for all possible partitions of de latent variables into blocks of size B = 2 (assuming all blocks have the same dimension), and report the best LMS This procedure is tractable in our experiments due to the small dimensionality of the problem we consider.
B.4 Boxplots for main experiments (Table 1)
Since the standard error in the main results (Table was high, we provide boxplots in Figures 9 & 10 to have a better visibility on what is causing this. We observe that the high standard error for the Additive approach was due bad performance for a few bad random initializations for the ScalarLatents dataset; while we have nearly perfect latent identification for the others. Figure 14e shows the latent space learned by the worst case seed, which somehow learned a disconnected support even if the ground-truth support was connected. Similarly, for the case of Independent BlenkL, there are only couple of bad random initializations and the rest of the cases have perfect identification.
34

---

Reconstruction MSE (Log Scale)
LMS-Spearman (MCC)
100
x 10
75
10-1
6 x 10
50
Non-Add. Add. Non-Add. (In-Supp) (In-Supp) Sout-SupplOut-Supp)
Add.
Add.
Non-Add. Add. Non-Add.
(In-Supp) (In-Supp) (Out-SuppIOOttSupp
Figure 9: Reconstruction mean squared error (MSE) (1) and Latent Matching Score (LMS) (f) over
10 different random initializations for ScalarLatents dataset.
Reconstruction MSE (Log Scale)
LMS-Tree.
100
10-1
75
50
10
25
Additive
Non-Additive
Additive
Non-Additive
(a) Independent Latent Case
Reconstruction MSE (Log Scale)
LMS-Tree
100
2 x 10-2
80
10
60
Additive
Non-Additive
Additive
Non-Additive
(b) Dependent Latent Case
Figure 10: Reconstruction mean squared error (MSE) (L) and Latent Matching Score (LMS) (f) for
10 different initializations for BlockLatents dataset.
B,5 Additional Results: BlockLatents Dataset
To get a qualitative understanding latent identification BlockLatents dataset, we plot the response of each predicted latent change a particular ground-truth latent factor. We describe the following cases of changing the ground-truth latents:
Ball 1 moving along x-axis: We sample 10 equally spaced points for 21 from [0, 1]: while keeping other latents fixed follows: 22 0.25, 0.50, 0.75. We will never have occlusion since the balls are separated along the y-axis 24 Z2 > 0.
Ball 2 moving along x-axis: We sample 10 equally spaced points for 2y from [0, 1]: while keeping other latents fixed as follows: 21 = 0.50, have occlusion since the balls are separated along the y-axis 24 Z2 0.
0.25, = 0.75. We will never
Ball 1 moving along y-axis: We sample 10 equally spaced points for z2 from |0, 1]: while
keeping other latents fixed as follows: 21 = 0.25, 23 0.75, 24 = 0.50. We will never have occlusion since the balls are separated along the x-axis z3 2 0.
Ball 2 moving along y-axis: We sample 10 equally spaced points for from 00. 1]: while
keeping other latents fixed as follows: 21 0.25, Z2 0.50, 23 0.75. We will never have occlusion since the balls are separated along the x-axis ø3 0.
35

---

Predictedd Latents
Predicted Latens
0
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Bail 1 moving along x axis
Boll moving along axis
Ball 1 moving along axis
Ball 2 moving along axis
Predicted Lateris
Predicted Latents
8
0.25
0.50
0.75
0.25
0.50
Ball 1 moving alona y axis
Ball moving along axis
Boll 1 moving along axts
Ball 2 moving along axis
1atent
-de-
Latent
Latent3
- Latan:
-
Labent -dk- Latint
Latent3
* Latent4
(a) Additive Decoder (Best) (LMS irce (9.9)
(b) Non- Ton-Aidiiivee Decoder (Best) (LMS 83.9)
Predicted Latents
Predicted Latents
0,75 Ball moving along x axis
0.25
0.50
0.25
0.56
0.75
0.25
0.50
0,75
0.25
6.50
0.75
Ball moving along axis
Bail 1 moving along axis
Ball 2 moving along axis
Predicted Latents
2
Predicted Latents
0.25
0.75
0.50
0.75
0.50 Ball
0.25
Dall
Ball
Bell
moving
Labent
2
Latent
- Latant
-8
Labent
- Latent
Latent
- Latant4
(c) Additive Decoder (Median) (LMSTre 99.8)
(d) Non-Additive Decoder (Median) (LMSTee 58.6)
Predicted Latents
Predicted Leients
0.25
0.75 Ball moving along axts
0.50
0.25
0.50
Ball moving along axis
Ball moving along axis
Ball 2 moving alang axis
Predicted Latents
Predicted Latents
0.25
Ball
along
(atent
Iatent 2
Intent
- Latent-4
Labent
Labent
Intent 3
+ Extent
(e) Additive Decoder (Worst) (LMSTiee 54.1)
(1) Non-Additive Decoder (Worst) (LMS 24.6)
Figure Latent responses for the cases with best/median/worst LMST among runs performed on the BlockLatent dataset with independent latents. In each plot, we report the latent factors predicted from multiple images where one ball moves along only one axis time.
Figure 5 in the main paper presents the latent responses plot for the median LMS case among random initializations. In Figure 11, we provide the results for the case best and the worst LMS among random seeds. We find that Additive Decoder fails for only for the worst case random seed, while Mon-Additive Decoder fails for all the cases.
Additionally, we provide the object-specifir reconstructions for the Additive Decoder in Figure 12. This helps us better understand the failure Additive Decoder for the worst case random seed (Figure 12c), where the issue arises due to bad reconstruction error.
36

---

(a) Additive Decoder (Best)
(b) Additive Decoder (Median)
(c) Additive Decoder (Worst)
Figure 12: Object-specific renderings with the best/median/worss LMS among runs performed on the BlockLatents dataset with independent latents. In each plot, the first row is the original image, the second row is the reconstruction and the third and fourth rows the output of the object-specific decoders. In the best and median cases, each object-specific decoder corresponds to one and only one object, e.g. the third row of the best case always corresponds the red ball. However, in the worst case, there are issues with reconstruction as only one the balls generated. Note that the visual artefacts due to the additive constant indeterminacy other as is suggested by the absence of artefacts in the reconstruction.
in Theorem 2, which cancel each
B.6 Disconnected Support Experiments
Since path-connected latent support is an important assumption for latent identification with additive decoders (Theorem 2), provide results for the case where the assumption is not satisfied. We experiment with the Disconnected Support dataset (Section B.2) and find that we obtain much worse LMS Spear compared to the case of training with L-shaped support in the ScalarLatents dataset. Over 10 different random initializations, we find mean LMS performance of 69.5 with standard error of 6.69.
For better qualitative understanding, we provide visualization of the latent support and the extrapolated images for the median L.S among 10 random seeds in Figure 13. Somewhat surprisingly. the representation appears to aligned in the sense that the first predicted latent corresponds to the blue ball while the second predicted latent correspond the red ball. Also surprisingly, extrapolation
37

---

Predicted Latent 2
Predicted Latent 1
Figure 13: Learned latent space, Ztrin and the corresponding reconstructed images of the additive decoder with the median LMSspear among runs performed on the Disconnected Support dataset. The red dots correspond latent factors used to generate the images.
occurs (we images of both balls high). That being said, observe that the relationship between the predicted latent 2 (&2) and y-coordinate second (red) ball not monotonic, which explains why the Spearman correlation is so low (Spearman correlation scores are high when there is a monotonic relationship between both variables).
B.7 Additional Results: ScalarLatents Dataset
To get qualitative understanding of extrapolation, plot the latent support the test dataset and sample a grid of equally spaced points from the support of each predicted latent on the test dataset. The grid represents the crntesiinnprrddeeee of the support of predicted latents and would contain novel combinations latents that were unseen during training. We show the reconstructed images for each point from the cartesian-prosd product grid to whether the model is able to reconstruct well the novel latent combinations.
Figure in the main paper presents visualizations of the latent support and the extrapolated images for the median LMS case among random seeds. In Figure 14, we provide the results for the case of best and the worst LMSSppar among random seeds. We find that even for the best case (Figure 14b), Non-Additive Decoder does not generate good quality extrapolated images, while Additive Decoder generates extrapoalted images for the best and median case. The worst-case run for the Additive Decoder has disconnected support, which explains why is not able to extrapolate.
38

---

Predicted Latent 2
Predicted Latent
-4.0 000000--.0022 0000--0----0---22200 04-0.22.00 6.33
13
Predicted latent
Predicted Latent 1
(a) Additive Decoder (Best) (L.MS Spear 99.9)
(h) Non-Additive Decnder (Best) (LM Spear 99.9)
Predlcted Latent 2
Predicted Latent
-1-
-11-13-15:0
20
-15-10-0500 00
Predicted Latent
(c) Additive Decoder (Median) (LMSspea 99.9)
(d) Non-Additive Decoder (Median) (L.MSspear 76.1)
Predicted Latent 2
Predicted Latent 2
-0.50
-0.35
-1.20
-GE-44-02
Predicted Latent
Predicted Latent
(e) Additive Decoder (Worst) (LMS 69.5)
(f) Non-Additive Decoder (Worst) (LMS 39.8)
Figure 14: Figure (a, e) shows the learned latent space, Strain and the corresponding reconstructed images of the additive decoder with the best/median/worstt L.MS among runs performed on the ScalarLatents dataset. Figure (b, d, f) shows the same thing for the non-additive decoder. The red dots correspond to latent factors used to generate the images and the yellow square highlights extrapolated images.
39


Reviews:
[{"review_id": "XdtxiOJGgF", "reviewer": "Reviewer_7aiT", "summary": "Motivated by the problem of disentanglement in generative models, this paper proposes a novel decoder architecture, so-called additive decoders, based on the addition of block-wise latent variables, where blocks of latents correspond to semantic factors.\n\nAn extrapolation property of their decoder is demonstrated, which essentially consists of forming new products of latent blocks unseen at train time.\n\nA theoretical analysis of their decoder is carried out. Several new definitions are made, as well as few necessary assumptions. Two theorems are presented on \"local disentanglement\" and \"global disentanglement\".\n\nAn empirical investigation based on synthetic image data, consisting of two balls randomly placed, is carried out, showing a case where their decoder improves upon a baseline case.\n", "questions": "Can the authors further justify their statement that this work \"has the potential of expanding our creativity in generative model\" (line 368).\n\nCan assumption 2 be verified for the decoders in used in the Experiments (Section 4)?\n\nIs training on larger-scale data possible with this method? If not, why not?\n", "limitations": "Unfortunately I did not find any discussion of limitations in the main text. \n\nI don't think this paper requires a discussion on societal impact.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the problem of disentanglement in an interesting way: through blocks of latent variables which contribute additively to the overall decoding. This is a natural and reasonable idea.\n\nThe motivation for this paper is good: restrict the decoder to be additive to address the problem of latent variable identifiability.\n\nThe proof technique appears to build on the well-established results (Hyvärinen, AISTATS 2019).\n", "weaknesses": "The paper is quite technical. This makes it challenging to ensure all technical details are correct. I did not find any errors.\n\nThe authors write that this work may help explain the \"creativity\" of mainstream generative models like DALLE-2 and Stale Diffusion. It is not so clear to me that their analysis will be helpful.\n\nRegarding Assumption 2, which the authors write is \"key\" for Theorem 2, I am not clear on how realistic this assumption is in practice. I understand that it is motivated by similar assumptions made in the ICA literature. However I come away with the feeling that it may not have much practical relevance. The authors give a toy numerical example in Example 3, which is helpful. But for instance, has this assumption been verified for the additive decoders used in the Experiments (Section 4)?\n\nThe proposed additive decoder cannot handle occlusions, as noted by the authors in Section 4, and discussed in the appendix. This is a limitation, since real images may have occlusions. \n\nOnly small-scale synthetic data are included. It doesn't appear to me that training on larger scale data is feasible, but more discussion of this would be helpful. It's reasonable to develop theory on simple cases, perhaps even necessary in this cases, but we should be clear whether scaling the results can be expected.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Practical relevance of Assumption 2**\n\nAlthough we haven’t shown that Assumption 2 is necessary in the sense that it cannot be relaxed, we know that we cannot simply remove it and get the same guarantees since this would contradict the very well established non identifiability of linear ICA when latents are independent and non-gaussian. That’s because, without Assumption 2, the decoder would be allowed to be linear (See Example 2). So this assumption has at least some practical relevance. \n\n**Verifying Assumption 2 in the “ball dataset” (new experiments!)**\n\nThis is a very good suggestion which we decided to investigate. The image generator we are using to generate these images takes only integer values which, strictly speaking, prevents us from even talking about the derivative of this generator. That being said, we created a very similar dataset which takes continuous values as input and can thus be differentiated (in fact, this generator is infinitely differentiable). This allowed us to verify numerically that Assumption 2 is satisfied. To make this verification, we considered a grid of values of z and for each point on that grid, we computed the matrix W(z) from Assumption 2, normalized its columns to have unit length, and computed $\\sqrt{|det(W^t W)|}$ which gives the 4D volume of the parallelogram spanned by the 4 columns of W. This quantity should be 1 when columns are orthogonal (i.e. maximally linearly independent) and 0 when the columns are linearly dependent. We found the minimal value of $\\sqrt{|det(W^t W)|}$ to be ~0.97, which indicates that the columns of W are linearly independent. **Please check the pdf linked in the general response for a visualization of the new dataset as well as the contour plot of sqrt(|det(W^t W)|) as a function of z. The pdf also contains an experiment in which we trained both additive and non-additive decoders. It leads to very similar conclusion to what we found in the paper.** \n"}, {"review_id": "7Pr59onnOo", "reviewer": "Reviewer_CUR1", "summary": "This paper extends the recently popular approach of constraining the nonlinear function to achieve identifiable disentanglement. Here the idea is that $\\mathbf{f}$ is additive i.e. made of constituent functions that operate independently on non-overlapping partitions of the latent function. Identifiable disentanglement is achieved in this situation with very mild conditions on the latent distribution. These additive decoders can be seen as rudimentary version of the decoders used in object-centric representation learning and thus help explain their generalization performance. In particular, the paper shows that by exploring the full cartesian product of the latent symbols the model can generate images that are out of the training images' support (called cartesian-product extrapolation).", "questions": "Do you believe the block-wise unidentifiability/entanglement is undesirable? Or do you believe my concerns above are not a problem? I'm talking practically speaking. What do you think is the impact of this on CPE? What if the blocks could also be fully disentangled -- what would this change (e.g. in CPE?). Again I'm interested in your thoughts on practical implications rather than theory.\n\nYou say that: \"is “imitating” a block-specific ground-truth decoder\". Could you please explain what you exactly mean by imitation? I find that quite vague. \n\nYou write that \"We believe it illustrates the  fact that disentanglement alone is not sufficient to enable extrapolation and that one needs to restrict the hypothesis class of decoders in some way.\" These sound like very generic statements -- do you have confidence that they hold much beyond this setting?\n\n\"our in this work to understand “out-of-support” generation is a step towards understanding theoretically why modern generative models such as DALLE-2 [42] and 53 Stable Diffusion [43] can be creative\" Could you please explain this more specifically? Do you believe additiveness to be important to this? ", "limitations": "There is some good discussion of limitations in the paper e.g. \"additive decoders make intuitive sense for OCRL, they are not expressive enough to represent the “masked decoders” typically used in\", \"Additionally, this  parameter sharing across f (B) enables modern methods to have a variable number of objects across  samples, an important practical point our theory does not cover.\" Appendix also has nice examples of what happens when some assumptions are violated.\n\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- very well written paper with clear and intuitive examples despite the technical topic\n- a novel way in which we can understand disentanglement by restricting the nonlinearity from the point of view of OCRL is a nice new angle to this increasingly popular field\n- by making assumption about the structure of $f$ the authors are able make very mild assumptions about the distribution of the latent factors which is in contrast to the much more distribution\n- additive decoders and the relevant results here provide a nice simple baseline model upon which future works can build more realistic ocrl models\n- extrapolation guarantees is a nice addition, something previous works have been missing, and is something that hopefully will be adopted by the community (though it's unclear how that could be done; see below)\n- assumption of sufficient nonlinearity is an insightful result and its connection to previous literature is nicely illustrated (albeit in the appendix)", "weaknesses": "Since the latent variables have only very mild restrictions, the price is paid by having fairly strong restriction on the 'mixing' function i.e. block-wise additivity (likely problematic in many realistic situations such as images with occlusion) + requirements on nonlinearity. This is likely useful for OCRL (e.g. scene mixtures) but in general may be very restrictive in the more general nonlinear ICA/disentanglement and also abstracts away from the desired goal of traditional nonlinear ICA where the aim is to separate out sources from 'heavily mixed' signals. \n\nFurthermore, the additivity also leads to a slightly problematic level identifiability results in the sense that each partition-block $z_b$ is identified only up to arbitrary invertible nonlinear transformation (equation 8. & 9. and $v_B$ specifically) -- this can be interpreted as there being a nonlinear ICA / mixing problem completely *unsolved* and thus unidentified for each block. So even though we are no longer left with the generic unidentifiability problem of $x = f(z)$, we are still left with complete unidentifiability in each block $B$, $ x_B = f_B (z_B)$. It feels like the term \"partition-respecting permutation\" and \"B-disentanglement\" are thus very specific to this approach and not really corresponding to the commonly accepted definitions of disentanglement in literature. Indeed the authors write \"Thus, B-disentanglement means that the blocks of latent dimensions zB are disentangled from one 177 another, but that variables within a given block might remain entangled.\"\n\nRelated to this, for OCRL the model is quite simple, as admitted by the authors, and while they indeed may provide a good baseline (as mentioned above) it is hard to be sure whether the theorems fully explain their performance especially given these concerns about block-wise unidentifiability.\n \nIt would have been nice to see more extensive experiments and especially on more complex data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Strength of the assumptions**\n\nWe agree that the additive assumption is rather strong and won’t make sense in many applications other than OCRL. That being said, we believe our work is a nice starting point for other works to investigate more expressive function classes.\n\n**Issues raised by block-disentanglement**\n\nWe believe that extending existing proof techniques to block identifiability is actually a strength, since it is more general and it includes the trivial partition {{1}, {2}, ..., {d_z}}, which would yield full disentanglement. The level of granularity of the partition controls the trade-off between expressivity and identifiability: The finer the partition, the more constrained is the function class and the tighter the identifiability guarantee. The optimal level of granularity is going to be task-dependent. Regarding the impact on the CPE, we have that the finer the partition is, the larger the CPE is going to be. To see this, consider the case where the Z^train is a sphere in 3D (so that d_z = 3). If B = {{1}, {2}, {3}}, we have that CPE is the smallest cube containing the sphere. If B = {{1,2}, {3}}, the CPE is a cylinder, which is a proper subset of the cube. We plan on adding a discussion on these considerations in the next revision.\n\n**“it is hard to be sure whether the theorems fully explain their performance especially given these concerns about block-wise unidentifiability.”**\n\nNote that, in our experiments, we always test for B-disentanglement. For the ScalarLatents dataset, B = {{1}, {2}} whereas for the BlockLatents datasets, B = {{1,2}, {3,4}}. In the first case, B-disentanglement is simply “full/complete disentanglement”, but for the last case, our evaluation does not penalize entanglement within a block. At line 328, we provide details on our evaluation metrics. \n\n**“Could you please explain what you exactly mean by imitation?”**\n\nWe use “imitation” for lack of a better word at line 244 and 281. This terminology is borrowed from [Ahuja et al., 2021] which was used in a similar way for transition mechanisms. For example, in equation (8) which relates the f_hat^B with f^{\\pi(B)}, the two functions are not equal, but they are closely related to one another via the map v_bar and the additive constant c, so one can think about f_hat^B as imitating f^{\\pi(B)}. We provide a more explicit explanation at lines 245-248.\n\n[Ahuja et al., 2021] https://arxiv.org/abs/2110.15796 \n\n**Regarding necessity of restricting decoder in order to enable extrapolation:**\n\nWithout a restriction on the decoder class, there are no ways to guarantee what will happen outside the domain of the training set. And disentanglement is not enough since you might be disentangled on the training domain but crazy things might happen outside. Note that this restriction might be enforced via the choice of architecture (like we did in the present work) or by some implicit bias of the optimizer or the loss function etc… We use the phrase “restricted decoder” in its broadest sense here. \n"}, {"review_id": "khO9iDsyoY", "reviewer": "Reviewer_JkNs", "summary": "This paper presents the identification theory for the additive mixing function. Specifically, they transfer the existing nonlinear ICA conditions from the distribution (i.e., sufficient variability) to the nonlinear mixing function (i.e., sufficient nonlinearity). Under this model, they make the connection to extrapolation for generative models. Synthetic data experiments are designed to demonstrate their arguments.", "questions": "I would like to learn about the authors' response to the weaknesses listed above, which may give me a clearer perspective on the paper's contribution.", "limitations": "Please see the weakness section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well written — both the assumptions and the implications are adequately discussed and thus easy to understand.\n2. The block-wise identification result is novel as a result of the sufficient nonlinearity condition inspired by the sufficient variability condition in prior work.\n3. The connection to the extrapolation is interesting and yields a valuable understanding of current large models.", "weaknesses": "1. Some key assumptions, although discussed, are still evasive in their restrictiveness. The most notable is the sufficient nonlinearity assumption, which appears very restrictive. How to enforce this for the estimation model is challenging.\n2. The primary assumption, namely additivity, can be very stringent. It is hard to believe this would hold for any realistic data-generating process. This also somehow trivializes the significance of the extrapolation part. I would also like to learn about the relation to recent work [1].\n3. The experimental results lack detailed explanation. I struggle to make sense of the visualizations: what are the colored shades mean in Figure 4, and what does the color mean for the dots? Are the generating processes identical in the additive and the non-additive cases, i.e., does the only difference lie in the estimation model?\n\n[1]. https://arxiv.org/abs/2305.14229", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**On the restrictiveness of the sufficient nonlinearity assumption:**\n\nAt line 218, we refer to Appendix A.5 to provide intuition for why the sufficient nonlinearity assumption is likely to be satisfied when d_x >> d_z. However, we notice that this explanation is lacking from our submission. In the next revision, we will add this explanation to Appendix A.5: \n\nWe now look at why Assumption 2 is likely to be satisfied when d_x >> d_z. Informally, one can see that when d_x is much larger than d_z, the matrix W(z) has much more rows than columns and thus it becomes more likely that we will find enough rows that are linearly independent, thus satisfying Assumption 2.\n\n**Regarding how to enforce Assumption 2:**\n\nWe note that Theorem 1 and 2 never require to enforce Assumption 2 in the learned model. This is an assumption about the data generating process, not about the learned model, so there is no need to enforce it during training. \n\n**On the restrictiveness of the additivity assumption:**\n\nWe agree that the additivity assumption can be rather restrictive and is unlikely to hold for realistic data. Nevertheless we argued in the paper that standard OCRL decoders are “almost” additive and that our analysis shed light on why these methods work on realistic data. Of course, this is only a first step and more work is needed to understand these approaches theoretically. \n\n**Regarding [Brady et al., 2023]:**\n\nThis work came out almost exactly as we were submitting this work, and thus we did not include it in our related work section. We will add it to the next revision. Brady et al. studies “compositional decoders” which by definition have Jacobians that, for all z, have always at most one nonzero value per row (note that the nonzero elements can have different locations for different values of z). Example 3 in our paper is an example of function that is additive but not compositional, since the Jacobian has more than one nonzero element per row. Moreover, we suspect, although aren’t fully convinced yet, that compositional functions are additive, since we couldn’t come up with an example of a compositional function that is not additive. If that turns out to be the case, that would mean that additive decoders form a (strict) superset of compositional decoders. We will keep working on this and update you during the discussion period in case we find a definitive answer. \n\nOther important distinctions between our work and theirs is that (i) we consider very general domains for the latent vector z (Brady et al. study only fully supported latent vectors) and (ii) we prove additive decoders can extrapolate (their work has no discussion of extrapolation).\n\n**Some details lacking in experiments:** \n\nThanks for bringing this to our attention. The different shades of color correspond to the value of one of the ground-truth latent factors. This is useful to assess disentanglement visually. The red dots correspond to the latent factors used to generate the images. In figure 4, the only difference is the estimation model, the data is the same. We’ll make sure to add these clarifications in the next revision.\n"}, {"review_id": "LDq6zVfTKD", "reviewer": "Reviewer_C5F5", "summary": "The paper analyses the statistical identifiability of latent variables in an autoencoder with a so-called additive decoder. It is shown that under this class of decoders, the blocks of latent dimensions associated with the additive decoder can be identified. This result is further related to the ability of a model to extrapolate.", "questions": "1. It seems to me that additive decoders are effectively a form of mixture model with non-trivial components. Here, we know quite a bit about conditions under which components can be identified up to permutation. This seems quite similar to the presented results. Can you elaborate on this connection? Am I misunderstanding something?\n2. In definition 2, you write \"let $\\mathcal{B}$ be a partition...\" Should this have been $\\mathcal{B}$ be the set of partitions...\"? Otherwise, I struggle to understand what $B \\in \\mathcal{B}$ actually means. But perhaps I misunderstood something.\n3. Can we agree that $v := f^{-1} \\circ \\hat{f}$ is a diffeomorphism simply because assumption 1 states that $f$ is a diffeomorphism or is there more to this? (I ask as the statement about $v$ appears in several places in the paper)\n4. How do you ensure that $f$ actually is a diffeomorphism? Here I mainly very about $f$ self-intersecting. I can see how it is easy to ensure that $f$ is an immersion, but in my reading of the paper you seem to require that $f$ is an embedding. Did I get this part right?\n5. I did not understand Assumption 2 at all. Can you explain it to me?\n6. In Theorem 2 it is assumed that $f$ is injective, which seems like a rather strong assumption. Can this be loosened?\n7. In line 271 it is stated $Z$ is *typically* a subset of $CPE(Z)$. What is meant by \"typically\"?\n8. I didn't quite understand the motivation behind the Cartesian-product extension (CPE). According to the intuition of Fig. 3, the CPE makes an axis-aligned extension, but isn't the entire issue regarding identifiability that we cannot assign much meaning to the axes (the axes concern the parametrization and not the underlying support)? Then, why is it natural to extend along the axes? (To be clear, I do see the point of making an extension when studying extrapolation, so my question is more why the said extension should be axis-aligned).", "limitations": "There is no need for a discussion regarding societal impact, etc., in a purely theoretical paper such as the present.\n\nI wish the paper had had a greater discussion regarding the many assumptions made throughout the paper. Such a discussion would be akin to a \"limitations\" section often found in more empirical/methodological papers, and I don't see why a theoretical paper should not openly be discussing the limitations of the analysis (i.e. if the assumptions are appropriate).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(These may be subject to change depending on the answers to my questions below)\n\n1. The paper is generally very well written in terms of giving intuitions behind the presented math (see the counter view in Weakness 1 below).\n2. The paper touches on an important topic (identifiability), and the focus on additive decoders is both interesting, relevant, and novel.\n3. The paper does a nice job of providing proof sketches, which is helpful since space constraints prevent the authors from including actual proofs in the main text.\n4. The paper does a very nice job of connecting assumptions and results to existing work in various branches of the literature. This is very helpful.\n5. Finally, I want to emphasize that the theoretical findings are both novel and interesting.\n", "weaknesses": "(These may be subject to change depending on the answers to my questions below)\n\n1. The mathematics is often phrased sufficiently convoluted that the phrased intuitions are required (see Strength 1 above). E.g. I found definition 3 to be nearly unreadable, and I could not verify if the clearly phrased intuitions (lines 177-178) actually describe the math (I trust that it does, but it's a problem that it is so difficult to verify).\n2. The 'additive decoder' construction seems quite similar to mixture models for which decades of work exist regarding identifiability. I was surprised to not see this link even briefly touched upon.\n3. I found the extrapolation part of the paper to be less convincing than the identifiability part. Bluntly put, I got lost in the many assumptions made (Corollary 3 holds under the assumptions of Theorem 2, which hold under the assumptions of Theorem 1 which holds under Assumption 1) that I was unable to tell which were the important assumptions for the particular corollary. Thus, I lost my intuition, and the following discussion (Lines 381-304) seems rather speculative. Fere I struggle to determine what's what.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Connection to mixture models:**\n\nWe assume that by “mixture models” the reviewer refers to models of the form $p(x) = \\sum_k p(c=k)p(x | c=k)$ where $c$ is the random component index. After some thinking, we do not see a clear connection between mixture models and our additive decoders. A possible point of confusion is that a mixture model is a sum (more precisely a convex combination) of *distributions* while the additive decoder $x = \\sum_k f^{(k)}(z_k)$ is a sum of *random variables*. Both models are very different, since the distribution of a sum of random variables is the *convolution* of their respective distributions (assuming the RVs are independent), not the *convex combination*. Please let us know if the confusion persists or if we misunderstood your point.\n\n**Confusion around assumptions of Corollary 3:**\n\nEssentially, in each result, we always require the assumptions of the previous results. Each result essentially adds more assumptions to obtain stronger guarantees. We’ll add that comment in the next revision.\n\n**Lines 281 to 304 seems speculative:**\n\nLines 281-288: We believe these to be objective. We describe a procedure we actually implemented in the experiments of Figure 4.\n\nLines 289-297: See answer to Reviewer 2rwH under *“It appears that authors attempt to define the reasonableness”* for clarifications.\n\nLines 298-304: We believe it is clear that if a decoder f_hat is disentangled w.r.t. a ground-truth decoder f on the training domain, nothing prevents both decoders to have very different behavior outside the training domain, i.e. no extrapolation, unless these decoders are restricted in some way, either via architectural choice (like in this paper) or via some other means like implicit regularization of the optimizer or loss function. \n\n**Meaning of the term “partition”:**\n\nIn mathematics, a “partition of a set A” refers to a set of subsets of A that are mutually disjoint and covers A. This is a very common terminology, see e.g. https://en.wikipedia.org/wiki/Partition_of_a_set. An element of a partition is usually referred to as a “block”. So here, $\\mathcal{B}$ is a partition and $B \\in \\mathcal{B}$ is a block.\n\n**Why is v = f^{-1} \\circ f_hat a diffeomorphism?**\n\nWe need both f and f_hat to be diffeomorphisms (these assumptions can be found in Assumption 1 and the statement of Theorem 1, respectively). \n\n**How to ensure that f is a diffeomorphism?**\n\nWe would like to clarify that f is the ground-truth decoder so we cannot force it to be diffeomorphism, we have no choice but to assume that it is a diffeomorphism. That being said, Theorem 1 requires that f_hat, the learned decoder, is diffeomorphism. In practice, we do not enforce this even if our theory prescribes it. This doesn't seem to be a problem, but might be worth exploring in future works.  \n\n**“I did not understand Assumption 2 at all. Can you explain it to me?”**\n\nThis is a rather technical assumption which makes it difficult to formulate in words. But the main point is that it prevents f from being linear as Example 2 shows. Keep in mind that this is an assumption about the data-generating process. Relating this assumptions to notions of curvature in differential geometry as proposed by Reviewer 2rwH might help us understand this assumption better, but is left as future work.\n\n**“In Theorem 2 it is assumed that f is injective, which seems like a rather strong assumption. Can this be loosened?”**\n\nJust to clarify: the ground-truth decoder f was assumed to be a diffeomorphism (and hence injective) much earlier in Assumption 1. Loosening this common assumption is an important open question in the field. Part of the difficulty is in defining the relationship between the learned representation and the ground-truth representation which here is given by v. Without injectivity, many z could correspond to a single z_hat.\n\n**“In line 271 it is stated Z is typically a subset of CPE(Z). What is meant by \"typically\"?”**\n\nQuick correction: we wrote that “Z is typically a *proper* subset of CPE(Z)” (“proper” means Z != CPE(Z)). We simply meant that Z is going to be a subset of CPE(Z) unless Z = CPE(Z). One could argue that, in some sense, there are much more sets Z such that Z != CPE(Z) than there are sets such that Z = CPE(Z). That’s what we meant by “typically”. To avoid confusion, we replaced this sentence by “The Cartesian-product extension of Z, CPE(E), is indeed an extension of Z since Z is a subset of CPE(Z)”. \n\n**Why the extension should be axis-aligned?**\n\nOur identifiability result shows that the additive structure in the decoder induces natural axes (the axes that make the map from z to x additive). Given this, it is expected to have extrapolation that is aligned with these natural axes.\n"}, {"review_id": "Wh4xMhAup4", "reviewer": "Reviewer_2rwH", "summary": "This paper develops a theory to show how an additive decoder may be able to disentangle an image composed of several components. The proposed theory also shows how an additive decoder may produce novel images, possibly providing insights about the process performed by generative models.\n\nBesides the theory, paper also performs some experiments showing that additive decoders may have certain advantages in performing those tasks.", "questions": "Please see the questions under weaknesses, especially the questions about assumption 2. (Assumption 2 and what is built on it is my main concern about this work. I hope authors can be more clear and more convincing about their approach.)\n\nHave authors considered expanding their own experiments to something more sophisticated? For example, the shape of objects could be not just circles, but several other types: diamonds, triangles, rectangles, etc. The size of the objects could vary as well. \n\nDo authors think their extrapolation method can be applied to the VAEC dataset (or some modification of it)? The point is to connect this paper’s experiments to existing experiments in the literature. For example, when an object in the VAEC dataset is enlarged, the enlarged object can be considered an addition of two smaller objects which would fit the authors’ additive framework. Currently, this paper’s experiments seem to be isolated from the literature. If VAEC is not suitable, authors may want to consider some other datasets from the literature.\n", "limitations": "I did not see a discussion on limitations.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Paper is well written.\n\nThe topic and the experiments are interesting.\n\nIt has a broad and practical view. Literature review is relatively good.\n\nThe proposed theory might turn into a useful contribution for the research community.", "weaknesses": "Experiments are interesting but limited and disconnected from existing experiments in the literature, in my view.\n\n-----------------\n\nThere are a few publications that are relevant but not cited:\n\n–Li, N., Raza, M.A., Hu, W., Sun, Z. and Fisher, R, Object-centric representation learning with generative spatial-temporal factorization, NeurIPS 2021.\n\n–Yoon, J., Wu, Y.F., Bae, H. and Ahn, S., An investigation into pre-training object-centric representations for reinforcement learning, ICML 2023.\n\nBoth of the above papers have experiments on images that may be decomposed in the authors’ additive scenario and possibly be used as a baseline for comparison.\n\n-----------------\n\n“Reasonableness”, mentioned under section 3.2, seems to be an unclear definition underlying a significant portion of the theory. It appears that authors attempt to define the reasonableness, yet, it is not clear what is the difference between $Z^{test}$ and $Z^{train}$. Given the bijective assumption, the difference between $Z^{test}$ and $Z^{train}$ should refer to a specific region in the domain and range of the function. Yet, it is not clear what that difference is.\n\nI understand it is hard to define the limits of the underlying manifold of relevant images - that is exactly the heart of the difficulty in developing useful theory for deep learning. Could authors expand on their “reasonable” assumption? Perhaps identifying the boundaries of the “reasonable” manifold is hard, but it may be helpful to describe and contrast what is unreasonable. Perhaps providing a discussion and citing some previous studies on the underlying manifold of images would be helpful as well, e.g.:\n \nCohen, U., Chung, S., Lee, D.D. and Sompolinsky, H., 2020. Separability and geometry of object manifolds in deep neural networks. Nature Communications, 11(1), p.746.\n\n\n-----------------\n\nThere is an extrapolation study and a dataset called VAEC from the paper below.\n\nWebb, T., Dulberg, Z., Frankland, S., Petrov, A., O’Reilly, R. and Cohen, J., Learning representations that support extrapolation. ICML 2020.\n\nThe images in the VAEC dataset are designed as an extrapolation task. Do authors think this task can fit into their framework?\n\n\n-----------------\n\nIt seems that the notation D (for the Jacobian) is only defined in the appendix under Table 2. Since the notation is used in the main body of the paper, it would be useful to define it there. If instead of D, $\\nabla$ was used for the Jacobian, I would have inferred what authors mean by it. However, I was not sure about D until I found it in the appendix.\n\n-----------------\n\nFor assumption 2, it may be better to use “linearly independent” instead of “independent”.\n\nMoreover, it might be better to explain in words that: this assumption is requiring the … matrices, to have full column rank.\n\nOverall, the notion formalized in assumption 2 seems strange to me. Are authors familiar with the notion of curvature for functions and manifolds?\n\nIs there any precedence for the notion of nonlinearity defined under assumption 2 for any class of functions? I am not sure how authors’ notion of nonlinearity relates to known notions of nonlinearity/curvature, for example, the notions of curvature in differential geometry. Why should assumption 2 be satisfied over the entire manifold?\n\nIn its current form, assumption 2 stacks the first and second order derivatives together and then requires that the stacked matrix to have full column rank. It is not clear to me why stacking of these matrices is necessary. If the unrolled second derivatives have full column rank, would it be necessary for the first derivatives to have full column rank? If each of the first and second order derivatives, individually have full column rank, would that be sufficient?\n\n\n-----------------\n\nThis is not a weakness of the paper, but perhaps worth mentioning. It is common in the literature to use x as the input to a function/model, and use y or z as the output of a function/model. However, this paper uses z to denote the inputs and x to denote the output. This may sometimes be a bit confusing. But that is the authors’ choice, and this is just feedback.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Regarding uncited relevant works:**\n \nObject-centric representation learning is a vast field and our literature review did not cover all works. We believe many OCRL datasets have this nice additive structure and many OCRL approaches have the inductive bias of additive decoder, which underlines the relevance of our work. We will add these works to our review. \n\nWe want to reemphasize the point we addressed in our general response: we are not proposing any novel approach for object-centric learning; instead we analyze additivity as an inductive bias already present in many OCRL decoders. Hence, our analysis is applicable to several OCRL methods. The only baseline we considered was non-additive decoder as we are concerned with understanding the impact of additivity on latent identification and extrapolation. Hence, using the suggested papers by Nanbo et al. and Yoon et al. as baselines does not align with the goal of our analysis.\n\n**“The images in the VAEC dataset are designed as an extrapolation task. Do authors think this task can fit into their framework?”**\n\nThe VAEC dataset contains a single object, and we believe the attributes like size, location, and brightness cannot be disentangled/identified using additive decoders as the effect of these latent variables on the resulting image is not additive. For example, in our image dataset we could disentangle the latents (position) of a ball from those of the other ball, however, we cannot disentangle the x-y positions of each ball. Hence, the additive decoder inductive bias cannot help us to recover the position of the object in the VAEC dataset, similar to the non-identifiability of position of a particular ball in our dataset. This also implies the question of extrapolation in the VAEC dataset is out of scope from our analysis as we provide guarantees for extrapolation w.r.t latent factors that have additive relationship in the true data generation process.\n\nWe now attempt to demonstrate that the “scale” factor of variation is not additive, as suggested by the reviewer: Let z_1 = size of object and z_2 = position of object and let’s forget about the other factors for the sake of the argument. We now want to write the images as x = f1(z_1) + f2(z_2). If we want f1(z_1) to correspond to “adding mass to object”, the issue is that “where to add the mass in the image” will depend on “where the object is located”, which is given by z_2. This means f1(z_1) could not implement “adding mass” since, to do so, it would have to depend on z_2 (which would violate additivity). \n\n**“It appears that authors attempt to define the reasonableness”**\n\nClarification: We do not attempt to define reasonableness. Instead, we denote by Z^test the set of latent factors that are *reasonable* for a given application, without explicitly defining what it means to be reasonable. This is similar in spirit to the ground-truth decoder f which captures what are the \"natural factors of variations\" in the given application. Most works (including ours) do not try to define what makes some factors of variations \"natural\", they just denote by f the mapping between the \"natural factors\" and the images x, whatever these natural factors might be in a given application.\n\n**“It is not clear what is the difference between Z^test and Z^train”**\n\nZ^test is the set of all reasonable latent vectors. Z^train is the set of latent vectors observed during training. This distinction is made in Assumption 1.\n\n**“Perhaps identifying the boundaries of the “reasonable” manifold is hard, but it may be helpful to describe and contrast what is unreasonable.”**\n\nIn Appendix A.10, we give an example of an unreasonable z in the context of occlusion, i.e. z \\not\\in Z^test.\n\n**“It seems that the notation D (for the Jacobian) is only defined in the appendix under Table 2”**\n\nThe notation is introduced for the first time on page 2 in the paragraph titled “Notation”. We did not use \\nabla to avoid the confusion with the gradient of a scalar-valued function, since here we are concerned with the derivative of vector-valued functions. A solution to improve clarity would be to recall the meaning of D the first time it is used in the paper. We’ll make sure to add this clarification in the next revision.\n\n**“For assumption 2, it may be better to use “linearly independent” instead of “independent””**\n\nAgreed, we implemented the change in the text.\n\n**“Is there any precedence for the notion of nonlinearity defined under assumption 2 for any class of functions?”**\n\nNot that we know of. However this assumption is very closely related to previous works in nonlinear ICA (see discussion in Appendix A.5)\n\n**Assumption 2 and notions of curvature in differential geometry.**\n\nConnection between Assumption 2 and known notions of curvature in differential geometry is a very interesting question worth investigating in the future as it might lead to a better understanding of these types of assumptions in nonlinear ICA. However we consider this to be out of scope for this work.\n\n**“Why should assumption 2 be satisfied over the entire manifold?”**\n\nIt comes from the nature of the proof. The argument is local in the sense that we start by showing \"local disentanglement\" in Theorem 1 (Jacobian of v is a block-permutation scaling everywhere) and then use further assumptions to show global disentanglement in Theorem 2. For the local part of the proof, we need assumption 2 to hold for all z.\n\n**“If each of the first and second order derivatives, individually have full column rank, would that be sufficient?”**\n\nAgain this comes from the nature of our proof, which requires W(z) to have full column-rank everywhere. Having the first and second derivatives individually have full rank would not be sufficient in our current proof (although we do not exclude the possibility that a proof exists that requires only this weaker condition)."}]

Author rebuttal:
We would like to thank all reviewers for engaging with our work and asking important and thought-provoking questions.

We were glad to see that many reviewers appreciated the clarity, relevance and novelty of our work. For example, **Reviewer C5F5** said that “The paper touches on an important topic (identifiability), and the focus on additive decoders is both interesting, relevant, and novel.” We were happy to read that **Reviewer CUR1** thought that “extrapolation guarantees is a nice addition, something previous works have been missing, and is something that hopefully will be adopted by the community”. **Reviewer 2rwH** said that the paper had a “broad and practical view” while **Reviewer 7aiT** thought the paper was well motivated. **Reviewer JkNs** wrote “The paper is generally very well written in terms of giving intuitions behind the presented math”. **Reviewer CUR1** thought that our result was “insightful” and that “its connection to previous literature is nicely illustrated”.

Some concerns were raised by more than one reviewer, so we decided to address them here in this general response. The other concerns are addressed in individual responses. **Please not that we added additional data and experiments in response to Reviewer 7aiT (see PDF).**

**Absence of large-scale/more complex datasets (7aiT, CUR1, 2rwH)**

We would like to emphasize that the bulk of our contribution is not to propose a novel architecture with the goal to compare to SOTA methods in object-centric representation learning. Instead, we bring theoretical insights as to why current OCRL methods work in the first place (with the caveats that these OCRL decoders are not exactly additive, but close to it, as discussed in the paper). We note that Peebles et al. [41] already showed convincingly that the diagonal Hessian penalty (which is equivalent to additivity, as we showed in Appendix A.2) improves disentanglement on more realistic datasets, although without any theory for why this worked. Our work can be seen as filling this gap. In addition, our identifiability result is interesting in and of itself to the nonlinear ICA community since it shows yet another case where nonlinear ICA is possible. 

[41] W. Peebles, J. Peebles, J.-Y. Zhu, A. A. Efros, and A. Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In ECCV, 2020.


**Lack of discussion surrounding limitations. (7aiT, C5F5)**

We would like to point out that this is in opposition with Reviewer CUR1’s comment which pointed out to discussions of limitations in our work. For example, the paragraph on line 156 titled “Differences with OCRL in practice” discusses some limitations of our analysis like the fact that the standard masked decoder used in practice is not additive and the fact that we do not consider variable numbers of objects across images, like most OCRL methods can do. Also, on line 325, we refer to Appendix A.11 where we explain why additive decoders cannot represent occlusion. On line 357, we discuss the importance of the “connected support” assumption. 

If the reviewers have specific ideas of limitations to discuss, we would be happy to include them in our next revision.

**How is this analysis going to help us understand creativity in more modern generative model? (7aiT, CUR1)**

Thanks for raising this. We agree we did not develop this point sufficiently in the paper, so we add clarifications here and plan on adjusting the next revision accordingly. We believe our analysis illustrates how studying the identifiability of a function class can help us understand how it will extrapolate. We believe this type of analysis is novel and hope that it will motivate other researchers to apply similar analyses to other function classes/learning algorithms (e.g. DALLE-2) to understand and maybe improve extrapolation capabilities. Whether additivity is going to be central to these analyses is left to be seen. It might not be and this does not matter since our point that “identifiability is an interesting lens to study extrapolation” remains.

Meta-review: Motivated by the problem of disentanglement in generative models, 
This paper proposes a novel decoder architecture for latent variables identification and "out-of-support" image generation in representation learning. The model is based on the addition of block-wise latent variables, where blocks of latents correspond to semantic factors.
Identifiable disentanglement is achieved with very mild conditions on the latent distribution. Additionally, the paper shows that the model can generate images that are out of the training images' support (called cartesian-product extrapolation). 

I agree with the reviewers that the paper addresses an important problem in representation learning, which is learning disantangled representations, and that the proposed approach can have a significant impact in the community.
Decision: Accept (oral)