Title: Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation
Venue: NeurIPS 2023
Accepted: yes
Keywords: identifiability

nonlinear ICA

causal representation learning

disentanglement

object-centric representation learning

extrapolation
Abstract: We tackle the problems of latent variables identification and "out-of-support'' image generation in representation learning. We show that both are possible for a class of decoders that we call additive, which are reminiscent of decoders used for object-centric representation learning (OCRL) and well suited for images that can be decomposed as a sum of object-specific images. We provide conditions under which exactly solving the reconstruction problem using an additive decoder is guaranteed to identify the blocks of latent variables up to permutation and block-wise invertible transformations. This guarantee relies only on very weak assumptions about the distribution of the latent factors, which might present statistical dependencies and have an almost arbitrarily shaped support. Our result provides a new setting where nonlinear independent component analysis (ICA) is possible and adds to our theoretical understanding of OCRL methods. We also show theoretically that additive decoders can generate novel images by recombining observed factors of variations in novel ways, an ability we refer to as Cartesian-product extrapolation. We show empirically that additivity is crucial for both identifiability and extrapolation on simulated data.

Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

Abstract

We tackle the problems of latent variables identification and 'out-of-support' image generation in representation learning. We show that both are possible for a class of decoders that we call additive , which are reminiscent of decoders used for object-centric representation learning (OCRL) and well suited for images that can be decomposed as a sum of object-specific images. We provide conditions under which exactly solving the reconstruction problem using an additive decoder is guaranteed to identify the blocks of latent variables up to permutation and block-wise invertible transformations. This guarantee relies only on very weak assumptions about the distribution of the latent factors, which might present statistical dependencies and have an almost arbitrarily shaped support. Our result provides a new setting where nonlinear independent component analysis (ICA) is possible and adds to our theoretical understanding of OCRL methods. We also show theoretically that additive decoders can generate novel images by recombining observed factors of variations in novel ways, an ability we refer to as Cartesian-product extrapolation . We show empirically that additivity is crucial for both identifiability and extrapolation on simulated data.

1 Introduction

The integration of connectionist and symbolic approaches to artificial intelligence has been proposed as a solution to the lack of robustness, transferability, systematic generalization and interpretability of current deep learning algorithms [53, 4, 13, 25, 21] with justifications rooted in cognitive sciences [20, 28, 43] and causality [57, 63]. However, the problem of extracting meaningful symbols grounded in low-level observations, e.g. images, is still open. This problem is sometime referred to as disentanglement [4, 48] or causal representation learning [63]. The question of identifiability in representation learning, which originated in works on nonlinear independent component analysis (ICA) [65, 31, 33, 36], has been the focus of many recent efforts [49, 66, 26, 47, 3, 9, 41]. The mathematical results of these works provide rigorous explanations for when and why symbolic representations can be extracted from low-level observations. In a similar spirit, Object-centric representation learning (OCRL) aims to learn a representation in which the information about different objects are encoded separately [19, 22, 11, 24, 18, 51, 14]. These approaches have shown impressive results empirically, but the exact reason why they can perform this form of segmentation without any supervision is poorly understood.

+

=

%B2

CPEp(Ztrain)

f(CPEs(Ztrain))

%B2

CPEB(Ztrain)

f

f(B1)

f(B2)

f

ZB1

ZB2

B := {1,2},{3,4}}

2B

B

2B

1.1 Contributions

Our first contribution is an analysis of the identifiability of a class of decoders we call additive (Definition 1). Essentially, a decoder f ( z ) acting on a latent vector z ∈ R d z to produce an observation x is said to be additive if it can be written as f ( z ) = ∑ B ∈B f ( B ) ( z B ) where B is a partition of { 1 , . . . , d z } , f ( B ) ( z B ) are 'block-specific' decoders and the z B are non-overlapping subvectors of z . This class of decoder is particularly well suited for images x that can be expressed as a sum of images corresponding to different objects (left of Figure 1). Unsurprisingly, this class of decoder bears similarity with the decoding architectures used in OCRL (Section 2), which already showed important successes at disentangling objects without any supervision. Our identifiability results provide conditions under which exactly solving the reconstruction problem with an additive decoder identifies the latent blocks z B up to permutation and block-wise transformations (Theorems 1 & 2). We believe these results will be of interest to both the OCRL community, as they partly explain the empirical success of these approaches, and to the nonlinear ICA and disentanglement community, as it provides an important special case where identifiability holds. This result relies on the block-specific decoders being 'sufficiently nonlinear' (Assumption 2) and requires only very weak assumptions on the distribution of the ground-truth latent factors of variations. In particular, these factors can be statistically dependent and their support can be (almost) arbitrary.

Our second contribution is to show theoretically that additive decoders can generate images never seen during training by recombining observed factors of variations in novel ways (Corollary 3). To describe this ability, we coin the term 'Cartesian-product extrapolation' (right of Figure 1). We believe the type of identifiability analysis laid out in this work to understand 'out-of-support' generation is novel and could be applied to other function classes or learning algorithms such as DALLE-2 [59] and Stable Diffusion [61] to understand their apparent creativity and hopefully improve it.

Both latent variables identification and Cartesian-product extrapolation are validated experimentally on simulated data (Section 4). More specifically, we observe that additivity is crucial for both by comparing against a non-additive decoder which fails to disentangle and extrapolate.

Notation. Scalars are denoted in lower-case and vectors in lower-case bold, e.g. x ∈ R and x ∈ R n . We maintain an analogous notation for scalar-valued and vector-valued functions, e.g. f and f . The i th coordinate of the vector x is denoted by x i . The set containing the first n integers excluding 0 is denoted by [ n ] . Given a subset of indices S ⊆ [ n ] , x S denotes the subvector consisting of entries x i for i ∈ S . Given a function f ( x S ) ∈ R m with input x S , the derivative of f w.r.t. x i is denoted by D i f ( x S ) ∈ R m and the second derivative w.r.t. x i and x i ′ is D 2 i,i ′ f ( x S ) ∈ R m . See Table 2 in appendix for more.

Code: Our code repository can be found at this link.

2 Background & Literature review

Identifiability of latent variable models. The problem of latent variables identification can be best explained with a simple example. Suppose observations x ∈ R d x are generated i.i.d. by first sampling a latent vector z ∈ R d z from a distribution P z and feeding it into a decoder function f : R d z → R d x ,

i.e. x = f ( z ) . By choosing an alternative model defined as ˆ f := f ◦ v and ˆ z := v -1 ( z ) where v : R d z → R d z is some bijective transformation, it is easy to see that the distributions of ˆ x = ˆ f (ˆ z ) and x are the same since ˆ f (ˆ z ) = f ◦ v ( v -1 ( z )) = f ( z ) . The problem of identifiability is that, given only the distribution over x , it is impossible to distinguish between the two models ( f , z ) and ( ˆ f , ˆ z ) . This is problematic when one wants to discover interpretable factors of variations since z and ˆ z could be drastically different. There are essentially two strategies to go around this problem: (i) restricting the hypothesis class of decoders ˆ f [65, 26, 44, 54, 9, 73], and/or (ii) restricting/adding structure to the distribution of ˆ z [33, 50, 42, 47]. By doing so, the hope is that the only bijective mappings v keeping ˆ f and ˆ z into their respective hypothesis classes will be trivial indeterminacies such as permutations and element-wise rescalings. Our contribution, which is to restrict the decoder function ˆ f to be additive (Definition 1), falls into the first category. Other restricted function classes for f proposed in the literature include post-nonlinear mixtures [65], local isometries [16, 15, 29], conformal and orthogonal maps [26, 60, 9] as well as various restrictions on the sparsity of f [54, 73, 7, 71]. Methods that do not restrict the decoder must instead restrict/structure the distribution of the latent factors by assuming, e.g., sparse temporal dependencies [31, 38, 42, 40], conditionally independent latent variables given an observed auxiliary variable [33, 36], that interventions targeting the latent factors are observed [42, 47, 46, 8, 2, 3, 64, 10, 67, 72, 34], or that the support of the latents is a Cartesianproduct [68, 62]. In contrast, our result makes very mild assumptions about the distribution of the latent factors, which can present statistical dependencies, have an almost arbitrarily shaped support and does not require any interventions. Additionally, none of these works provide extrapolation guarantees as we do in Section 3.2.

Relation to nonlinear ICA. Hyv¨ arinen and Pajunen [32] showed that the standard nonlinear ICA problem where the decoder f is nonlinear and the latent factors z i are statistically independent is unidentifiable. This motivated various extensions of nonlinear ICA where more structure on the factors is assumed [30, 31, 33, 36, 37, 27]. Our approach departs from the standard nonlinear ICA problem along three axes: (i) we restrict the mixing function to be additive, (ii) the factors do not have to be necessarily independent, and (iii) we can identify only the blocks z B as opposed to each z i individually up to element-wise transformations, unless B = {{ 1 } , ..., { d z }} (see Section 3.1).

Object-centric representation learning (OCRL). Lin et al. [45] classified OCRL methods in two categories: scene mixture models [22, 23, 24, 51] & spatial-attention models [19, 12, 11, 18]. Additive decoders can be seen as an approximation to the decoding architectures used in the former category, which typically consist of an object-specific decoder f ( obj ) acting on object-specific latent blocks z B and 'mixed' together via a masking mechanism m ( B ) ( z ) which selects which pixel belongs to which object. More precisely,

and where B is a partition of [ d z ] made of equal-size blocks B and a : R | B | → R d x outputs a score that is normalized via a softmax operation to obtain the masks m ( B ) ( z ) . Many of these works also present some mechanism to select dynamically how many objects are present in the scene and thus have a variable-size representation z , an important technical aspect we omit in our analysis. Empirically, training these decoders based on some form of reconstruction objective, probabilistic or not, yields latent blocks z B that represent the information of individual objects separately. We believe our work constitutes a step towards providing a mathematically grounded explanation for why these approaches can perform this form of disentanglement without supervision (Theorems 1 & 2). Many architectural innovations in scene mixture models concern the encoder, but our analysis focuses solely on the structure of the decoder f ( z ) , which is a shared aspect across multiple methods. Generalization capabilities of object-centric representations were studied empirically by Dittadi et al. [14] but did not cover Cartesian-product extrapolation (Corollary 3) on which we focus here.

Diagonal Hessian penalty [58]. Additive decoders are also closely related to the penalty introduced by Peebles et al. [58] which consists in regularizing the Hessian of the decoder to be diagonal. In Appendix A.2, we show that 'additivity' and 'diagonal Hessian' are equivalent properties. They showed empirically that this penalty can induce disentanglement on datasets such as CLEVR [35], which is a standard benchmark for OCRL, but did not provide any formal justification. Our work provides a rigorous explanation for these successes and highlights the link between the diagonal Hessian penalty and OCRL.

̸

Compositional decoders [7]. Compositional decoders were recently introduced by Brady et al. [7] as a model for OCRL methods with identifiability guarantees. A decoder f is said to be compositional when its Jacobian D f satisfies the following property everywhere: For all i ∈ [ d z ] and B ∈ B , D B f i ( z ) = 0 = ⇒ D B c f i ( z ) = 0 , where B c := [ d z ] \ B . In other words, each x i can locally depend solely on one block z B (this block can change for different z ). In Appendix A.3, we show that compositional C 2 decoders are additive. Furthermore, Example 3 shows a decoder that is additive but not compositional, which means that additive C 2 decoders are strictly more expressive than compositional C 2 decoders. Another important distinction with our work is that we consider more general supports for z and provide a novel extrapolation analysis. That being said, our identifiability result does not supersede theirs since they assume only C 1 decoders while our theory assumes C 2 .

Extrapolation. Du and Mordatch [17] studied empirically how one can combine energy-based models for what they call compositional generalization , which is similar to our notion of Cartesianproduct extrapolation, but suppose access to datasets in which only one latent factor varies and do not provide any theory. Webb et al. [70] studied extrapolation empirically and proposed a novel benchmark which does not have an additive structure. Besserve et al. [5] proposed a theoretical framework in which out-of-distribution samples are obtained by applying a transformation to a single hidden layer inside the decoder network. Krueger et al. [39] introduced a domain generalization method which is trained to be robust to tasks falling outside the convex hull of training distributions. Extrapolation in text-conditioned image generation was recently discussed by Wang et al. [69].

3 Additive decoders for disentanglement & extrapolation

Our theoretical results assume the existence of some data-generating process describing how the observations x are generated and, importantly, what are the 'natural' factors of variations.

Assumption 1 (Data-generating process) . The set of possible observations is given by a lower dimensional manifold f ( Z test ) embedded in R d x where Z test is an open set of R d z and f : Z test → R d x is a C 2 -diffeomorphism onto its image. We will refer to f as the ground-truth decoder . At training time, the observations are i.i.d. samples given by x = f ( z ) where z is distributed according to the probability measure P train z with support Z train ⊆ Z test . Throughout, we assume that Z train is regularly closed (Definition 6).

Intuitively, the ground-truth decoder f is effectively relating the 'natural factors of variations' z to the observations x in a one-to-one fashion. The map f is a C 2 -diffeomorphism onto its image, which means that it is C 2 (has continuous second derivative) and that its inverse (restricted to the image of f ) is also C 2 . Analogous assumptions are very common in the literature on nonlinear ICA and disentanglement [33, 36, 42, 1]. Mansouri et al. [52] pointed out that the injectivity of f is violated when images show two objects that are indistinguishable, an important practical case that is not covered by our theory.

̸

We emphasize the distinction between Z train , which corresponds to the observations seen during training, and Z test , which corresponds to the set of all possible images. The case where Z train = Z test will be of particular interest when discussing extrapolation in Section 3.2. The 'regularly closed' condition on Z train is mild, as it is satisfied as soon as the distribution of z has a density w.r.t. the Lebesgue measure on R d z . It is violated, for example, when z is a discrete random vector. Figure 2 illustrates this assumption with simple examples.

Objective. Our analysis is based on the simple objective of reconstructing the observations x by learning an encoder ˆ g : R d x → R d z and a decoder ˆ f : R d z → R d x . Note that we assumed implicitly that the dimensionality of the learned representation matches the dimensionality of the ground-truth. We define the set of latent codes the encoder can output when evaluated on the training distribution:

When the images of the ground-truth and learned decoders match, i.e. f ( Z train ) = ˆ f ( ˆ Z train ) , which happens when the reconstruction task is solved exactly, one can define the map v : ˆ Z train →Z train as

This function is going to be crucial throughout the work, especially to define B -disentanglement (Definition 3), as it relates the learned representation to the ground-truth representation.

Before introducing our formal definition of additive decoders, we introduce the following notation: Given a set Z ⊆ R d z and a subset of indices B ⊆ [ d z ] , let us define Z B to be the projection of Z onto dimensions labelled by the index set B . More formally,

Intuitively, we will say that a decoder is additive when its output is the summation of the outputs of 'object-specific' decoders that depend only on each latent block z B . This captures the idea that an image can be seen as the juxatoposition of multiple images which individually correspond to objects in the scene or natural factors of variations (left of Figure 1).

Definition 1 (Additive functions) . Let B be a partition of [ d z ] 1 . A function f : Z → R d x is said to be additive if there exist functions f ( B ) : Z B → R d x for all B ∈ B such that

This additivity property will be central to our analysis as it will be the driving force of identifiability (Theorem 1 & 2) and Cartesian-product extrapolation (Corollary 3).

Remark 1. Suppose we have x = σ ( ∑ B ∈B f ( B ) ( z B )) where σ is a known bijective function. For example, if σ ( y ) := exp( y ) (component-wise), the decoder can be thought of as being multiplicative. Our results still apply since we can simply transform the data doing ˜ x := σ -1 ( x ) to recover the additive form ˜ x = ∑ B ∈B f ( B ) ( z B ) .

Differences with OCRL in practice. We point out that, although the additive decoders make intuitive sense for OCRL, they are not expressive enough to represent the 'masked decoders' typically used in practice (Equation (1)). The lack of additivity stems from the normalization in the masks m ( B ) ( z ) . We hypothesize that studying the simpler additive decoders might still reveal interesting phenomena present in modern OCRL approaches due to their resemblance. Another difference is that, in practice, the same object-specific decoder f ( obj ) is applied to every latent block z B . Our theory allows for these functions to be different, but also applies when functions are the same. Additionally, this parameter sharing across f ( B ) enables modern methods to have a variable number of objects across samples, an important practical point our theory does not cover.

3.1 Identifiability analysis

We now study the identifiability of additive decoders and show how they can yield disentanglement. Our definition of disentanglement will rely on partition-respecting permutations :

Definition 2 (Partition-respecting permutations) . Let B be a partition of { 1 , ..., d z } . A permutation π over { 1 , ..., d z } respects B if, for all B ∈ B , π ( B ) ∈ B .

Essentially, a permutation that respects B is one which can permute blocks of B and permute elements within a block, but cannot 'mix' blocks together. We now introduce B -disentanglement.

Definition 3 ( B -disentanglement) . A learned decoder ˆ f : R d z → R d x is said to be B -disentangled w.r.t. the ground-truth decoder f when f ( Z train ) = ˆ f ( ˆ Z train ) and the mapping v := f -1 ◦ ˆ f is a diffeomorphism from ˆ Z train to Z train satisfying the following property: there exists a permutation π respecting B such that, for all B ∈ B , there exists a function ¯ v π ( B ) : ˆ Z train B →Z train π ( B ) such that, for all z ∈ ˆ Z train , v π ( B ) ( z ) = ¯ v π ( B ) ( z B ) . In other words, v π ( B ) ( z ) depends only on z B .

Thus, B -disentanglement means that the blocks of latent dimensions z B are disentangled from one another, but that variables within a given block might remain entangled. Note that, unless the partition is B = {{ 1 } , . . . , { d z }} , this corresponds to a weaker form of disentanglement than what is typically seeked in nonlinear ICA, i.e. recovering each variable individually.

Example 1. To illustrate B -disentanglement, imagine a scene consisting of two balls moving around in 2D where the 'ground-truth' representation is given by z = ( x 1 , y 1 , x 2 , y 2 ) where z B 1 = ( x 1 , y 1 ) and z B 2 = ( x 2 , y 2 ) are the coordinates of each ball (here, B := {{ 1 , 2 } , { 3 , 4 }} ). In that case, a learned representation is B -disentangled when the balls are disentangled from one another. However, the basis in which the position of each ball is represented might differ in both representations.

Our first result (Theorem 1) shows a weaker form of disentanglement we call local B -disentanglement. This means the Jacobian matrix of v , D v , has a 'block-permutation' structure everywhere.

Definition 4 (Local B -disentanglement) . A learned decoder ˆ f : R d z → R d x is said to be locally B -disentangled w.r.t. the ground-truth decoder f when f ( Z train ) = ˆ f ( ˆ Z train ) and the mapping v := f -1 ◦ ˆ f is a diffeomorphism from ˆ Z train to Z train with a mapping v : ˆ Z train →Z train satisfying the following property: for all z ∈ ˆ Z train , there exists a permutation π respecting B such that, for all B ∈ B , the columns of D v π ( B ) ( z ) ∈ R | B |× d z outside block B are zero.

In Appendix A.4, we provide three examples where local disentanglement holds but not global disentanglement. The first one illustrates how having a disconnected support can allow for a permutation π (from Definition 4) that changes between disconnected regions of the support. The last two examples show how, even if the permutation stays the same throughout the support, we can still violate global disentanglement, even with a connected support.

We now state the main identifiability result of this work which provides conditions to guarantee local disentanglement. We will then see how to go from local to global disentanglement in the subsequent Theorem 2. For pedagogical reasons, we delay the formalization of the sufficient nonlinearity Assumption 2 on which the result crucially relies.

Theorem 1 (Local disentanglement via additive decoders) . Suppose that the data-generating process satisfies Assumption 1, that the learned decoder ˆ f : R d z → R d x is a C 2 -diffeomorphism, that the encoder ˆ g : R d x → R d z is continuous, that both f and ˆ f are additive (Definition 1) and that f is sufficiently nonlinear as formalized by Assumption 2. Then, if ˆ f and ˆ g solve the reconstruction problem on the training distribution, i.e. E train || x -ˆ f (ˆ g ( x )) || 2 = 0 , we have that ˆ f is locally B -disentangled w.r.t. f (Definition 4) .

The proof of Theorem 1, which can be found in Appendix A.5, is inspired from Hyv¨ arinen et al. [33]. The essential differences are that (i) they leverage the additivity of the conditional log-density of z given an auxiliary variable u (i.e. conditional independence) instead of the additivity of the decoder function f , (ii) we extend their proof techniques to allow for 'block' disentanglement, i.e. when B is not the trivial partition {{ 1 } , . . . , { d z }} , (iii) the asssumption 'sufficient variability' of the prior p ( z | u ) of Hyv¨ arinen et al. [33] is replaced by an analogous assumption of 'sufficient nonlinearity' of the decoder f (Assumption 2), and (iv) we consider much more general supports Z train which makes the jump from local to global disentanglement less direct in our case.

The identifiability-expressivity trade-off. The level of granularity of the partition B controls the trade-off between identifiability and expressivity: the finer the partition, the tighter the identifiability guarantee but the less expressive is the function class. The optimal level of granularity is going to dependent on the application at hand. Whether B could be learned from data is left for future work.

Sufficient nonlinearity. The following assumption is key in proving Theorem 2, as it requires that the ground-truth decoder is 'sufficiently nonlinear'. This is reminiscent of the 'sufficient variability' assumptions found in the nonlinear ICA litterature, which usually concerns the distribution of the latent variable z as opposed to the decoder f [30, 31, 33, 36, 37, 42, 73]. We clarify this link in Appendix A.6 and provide intuitions why sufficient nonlinearity can be satisfied when d x ≫ d z .

Assumption 2 (Sufficient nonlinearity of f ) . Let q := d z + ∑ B ∈B | B | ( | B | +1) 2 . For all z ∈ Z train , f is such that the following matrix has linearly independent columns (i.e. full column-rank):

where B 2 ≤ := B 2 ∩ { ( i, i ′ ) | i ′ ≤ i } . Note this implies d x ≥ q .

The following example shows that Theorem 1 does not apply if the ground-truth decoder f is linear. If that was the case, it would contradict the well known fact that linear ICA with independent Gaussian factors is unidentifiable.

Example 2 (Importance of Assumption 2) . Suppose x = f ( z ) = Az where A ∈ R d x × d z is full rank. Take ˆ f ( z ) := AV z and ˆ g ( x ) := V -1 A † x where V ∈ R d z × d z is invertible and A † is the left pseudo inverse of A . By construction, we have that E [ x -ˆ f (ˆ g ( x ))] = 0 and f and ˆ f are

B -additive because f ( z ) = ∑ B ∈B A · ,B z B and ˆ f ( z ) = ∑ B ∈B ( AV ) · ,B z B . However, we still have that v ( z ) := f -1 ◦ ˆ f ( z ) = V z where V does not necessarily have a block-permutation structure, i.e. no disentanglement. The reason we cannot apply Theorem 1 here is because Assumption 2 is not satisfied. Indeed, the second derivatives of f ( B ) ( z B ) := A · ,B z B are all zero and hence W ( z ) cannot have full column-rank.

Example 3 (A sufficiently nonlinear f ) . In Appendix A.7 we show numerically that the function

is a diffeomorphism from the square [ -1 , 0] × [0 , 1] to its image that satisfies Assumption 2.

Example 4 (Smooth balls dataset is sufficiently nonlinear) . In Appendix A.7 we present a simple synthetic dataset consisting of images of two colored balls moving up and down. We also verify numerically that its underlying ground-truth decoder f is sufficiently nonlinear.

3.1.1 From local to global disentanglement

The following result provides additional assumptions to guarantee global disentanglement (Definition 3) as opposed to only local disentanglement (Definition 4). See Appendix A.8 for its proof.

Theorem 2 (From local to global disentanglement) . Suppose that all the assumptions of Theorem 1 hold. Additionally, assume Z train is path-connected (Definition 8) and that the block-specific decoders f ( B ) and ˆ f ( B ) are injective for all blocks B ∈ B . Then, if ˆ f and ˆ g solve the reconstruction problem on the training distribution, i.e. E train || x -ˆ f (ˆ g ( x )) || 2 = 0 , we have that ˆ f is (globally) B -disentangled w.r.t. f (Definition 3) and, for all B ∈ B ,

where the functions ¯ v π ( B ) are from Defintion 3 and the vectors c ( B ) ∈ R d x are constants such that ∑ B ∈B c ( B ) = 0 . We also have that the functions ¯ v π ( B ) : ˆ Z train B →Z train π ( B ) are C 2 -diffeomorphisms and have the following form:

Equation (8) in the above result shows that each block-specific learned decoder ˆ f ( B ) is 'imitating' a block-specific ground-truth decoder f π ( B ) . Indeed, the 'object-specific' image outputted by the decoder ˆ f ( B ) evaluated at some z B ∈ ˆ Z train B is the same as the image outputted by f ( B ) evaluated at v ( z B ) ∈ Z train B , up to an additive constant vector c ( B ) . These constants cancel each other out when taking the sum of the block-specific decoders.

Equation (9) provides an explicit form for the function ¯ v π ( B ) , which is essentially the learned block-specific decoder composed with the inverse of the ground-truth blockspecific decoder.

Additional assumptions to go from local to global. Assuming that the support of P train z , Z train , is path-connected (see Definition 8 in appendix) is useful since it prevents the permutation π of Definition 4 from changing between two disconnected regions of ˆ Z train . See Figure 2 for an illustration. In Appendix A.9, we discuss the additional assumption that each f ( B ) must be injective and show that, in general, it is not equivalent to the assumption that ∑ B ∈B f ( B ) is injective.

3.2 Cartesian-product extrapolation

In this section, we show how a learned additive decoder can be used to generate images x that are 'out of support' in the sense that x ̸∈ f ( Z train ) , but that are still on the manifold of 'reasonable' images, i.e. x ∈ f ( Z test ) . To characterize the set of images the learned decoder can generate, we will rely on the notion of 'cartesian-product extension', which we define next.

Regularly

closed

Not regularly

closed

Path-connected

Not path-connected

Definition 5 (Cartesian-product extension) . Given a set Z ⊆ R d z and partition B of [ d z ] , we define the Cartesian-product extension of Z as

It is indeed an extension of Z since Z ⊆ ∏ B ∈B Z B .

[                            ]

[                           ]

[                           ]

[                            ]

Let us define ¯ v : CPE B ( ˆ Z train ) → CPE B ( Z train ) to be the natural extension of the function v : ˆ Z train →Z train . More explicitly, ¯ v is the 'concatenation' of the functions ¯ v B given in Definition 3:

where ℓ is the number of blocks in B . This map is a diffeomorphism because each ¯ v π ( B ) is a diffeomorphism from ˆ Z train B to Z train π ( B ) by Theorem 2.

We already know that ˆ f ( z ) = f ◦ ¯ v ( z ) for all z ∈ ˆ Z train . The following result shows that this equality holds in fact on the larger set CPE B ( ˆ Z train ) , the Cartesian-product extension of ˆ Z train . See right of Figure 1 for an illustration of the following corollary.

Corollary 3 (Cartesian-product extrapolation) . Suppose the assumptions of Theorem 2 holds. Then,

Furthermore, if CPE B ( Z train ) ⊆ Z test , then ˆ f ( CPE B ( ˆ Z train )) ⊆ f ( Z test ) .

Equation (11) tells us that the learned decoder ˆ f 'imitates' the ground-truth f not just over ˆ Z train , but also over its Cartesian-product extension. This is important since it guarantees that we can generate observations never seen during training as follows: Choose a latent vector z new that is in the Cartesian-product extension of ˆ Z train , but not in ˆ Z train itself, i.e. z new ∈ CPE B ( ˆ Z train ) \ ˆ Z train . Then, evaluate the learned decoder on z new to get x new := ˆ f ( z new ) . By Corollary 3, we know that x new = f ◦ ¯ v ( z new ) , i.e. it is the observation one would have obtain by evaluating the ground-truth decoder f on the point ¯ v ( z new ) ∈ CPE B ( Z train ) . In addition, this x new has never been seen during training since ¯ v ( z new ) ̸∈ ¯ v ( ˆ Z train ) = Z train . The experiment of Figure 4 illustrates this procedure.

About the extra assumption ' CPE B ( Z train ) ⊆ Z test '. Recall that, in Assumption 1, we interpreted f ( Z test ) to be the set of 'reasonable' observations x , of which we only observe a subset f ( Z train ) . Under this interpretation, Z test is the set of reasonable values for the vector z and the additional assumption that CPE B ( Z train ) ⊆ Z test in Corollary 3 requires that the Cartesian-product extension of Z train consists only of reasonable values of z . From this assumption, we can easily conclude that ˆ f ( CPE B ( ˆ Z train )) ⊆ f ( Z test ) , which can be interpreted as: 'The novel observations x new obtained via Cartesian-product extrapolation are reasonable '. Appendix A.11 describes an example where the assumption is violated, i.e. CPE B ( Z train ) ̸⊆ Z test . The practical implication of this is that the new observations x new obtained via Cartesian-product extrapolation might not always be reasonable.

Disentanglement is not enough for extrapolation. To the best of our knowledge, Corollary 3 is the first result that formalizes how disentanglement can induce extrapolation. We believe it illustrates the fact that disentanglement alone is not sufficient to enable extrapolation and that one needs to restrict the hypothesis class of decoders in some way. Indeed, given a learned decoder ˆ f that is disentangled w.r.t. f on the training support Z train , one cannot guarantee both decoders will 'agree' outside the training domain without further restricting ˆ f and f . This work has focused on 'additivity', but we believe other types of restriction could correspond to other types of extrapolation.

4 Experiments

We now present empirical validations of the theoretical results presented earlier. To achieve this, we compare the ability of additive and non-additive decoders to both identify ground-truth latent factors (Theorems 1 & 2) and extrapolate (Corollary 3) when trained to solve the reconstruction task on simple images ( 64 × 64 × 3 ) consisting of two balls moving in space [2]. See Appendix B.1

for training details. We consider two datasets: one where the two ball positions can only vary along the y -axis ( ScalarLatents ) and one where the positions can vary along both the x and y axes ( BlockLatents ).

ScalarLatents: The ground-truth latent vector z ∈ R 2 is such that z 1 and z 2 corresponds to the height (y-coordinate) of the first and second ball, respectively. Thus the partition is simply B = {{ 1 } , { 2 }} (each object has only one latent factor). This simple setting is interesting to study since the low dimensionality of the latent space ( d z = 2 ) allows for exhaustive visualizations like Figure 4. To study Cartesian-product extrapolation (Corollary 3), we sample z from a distribution with a L-shaped support given by Z train := [0 , 1] × [0 , 1] \ [0 . 5 , 1] × [0 . 5 , 1] , so that the training set does not contain images where both balls appear in the upper half of the image (see Appendix B.2).

BlockLatents: The ground-truth latent vector z ∈ R 4 is such that z { 1 , 2 } and z { 3 , 4 } correspond to the x, y position of the first and second ball, respectively (the partition is simply B = {{ 1 , 2 } , { 3 , 4 }} , i.e. each object has two latent factors). Thus, this more challenging setting illustrates 'blockdisentanglement'. The latent z is sampled uniformly from the hypercube [0 , 1] 4 but the images presenting occlusion (when a ball is behind another) are rejected from the dataset. We discuss how additive decoders cannot model images presenting occlusion in Appendix A.12. We also present an additional version of this dataset where we sample from the hypercube [0 , 1] 4 with dependencies. See Appendix B.2 for more details about data generation.

Evaluation metrics: To evaluate disentanglement, we compute a matrix of scores ( s B,B ′ ) ∈ R ℓ × ℓ where ℓ is the number of blocks in B and s B,B ′ is a score measuring how well we can predict the ground-truth block z B from the learned latent block ˆ z B ′ = ˆ g B ′ ( x ) outputted by the encoder. The final Latent Matching Score (LMS) is computed as LMS = arg max π ∈ S B 1 ℓ ∑ B ∈B s B,π ( B ) , where S B is the set of permutations respecting B (Definition 2). When B := {{ 1 } , . . . , { d z }} and the score used is the absolute value of the correlation, LMS is simply the mean correlation coefficient (MCC), which is widely used in the nonlinear ICA literature [30, 31, 33, 36, 42]. Because our theory guarantees recovery of the latents only up to invertible and potentially nonlinear transformations, we use the Spearman correlation, which can capture nonlinear relationships unlike the Pearson correlation. We denote this score by LMSSpear and will use it in the dataset ScalarLatents . For the BlockLatents dataset, we cannot use Spearman correlation (because z B are two dimensional). Instead, we take the score s B,B ′ to be the R 2 score of a regression tree. We denote this score by LMStree. There are subtleties to take care of when one wants to evaluate LMStree on a non-additive model due to the fact that the learned representation does not have a natural partition B . We must thus search over partitions. We discuss this and provide further details on the metrics in Appendix B.3.

4.1 Results

Additivity is important for disentanglement. Table 1 shows that the additive decoder obtains a much higher LMSSpear & LMSTree than its non-additive counterpart on all three datasets considered, even if both decoders have very small reconstruction errors. This is corroborated by the visualizations of Figures 4 & 5. Appendix B.5 additionally shows object-specific reconstructions for the BlockLatents dataset. We emphasize that disentanglement is possible even when the latent factors are dependent (or causally related), as shown on the ScalarLatents dataset (L-shaped support implies dependencies) and on the BlockLatents dataset with dependencies (Table 1). Note that prior works have relied on interventions [3, 2, 8] or Cartesian-product supports [68, 62] to deal with dependencies.

(a) Additive decoder

(b) Non-additive decoder

2

2

PredictedLatent

0

2

-1.5-1.0-0.50.00.51.01.52.0

PredictedLatent1

2

PredictedLatent

1

-1.5-1.0-0.50.00.51.01.52.02.5

PredictedLatent1

0.25

0.50

0.75

Ball 1 moving along x axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(a) Additive Decoder

0.25

0.50

0.75

Ball 1 moving along x axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(b) Non-Additive Decoder

Additivity is important for Cartesian-product extrapolation. Figure 4 illustrates that the additive decoder can generate images that are outside the training domain (both balls in upper half of the image) while its non-additive counterpart cannot. Furthermore, Table 1 also corroborates this showing that the 'out-of-support' (OOS) reconstruction MSE and LMSSpear (evaluated only on the samples never seen during training) are significantly better for the additive than for the non-additive decoder.

Importance of connected support. Theorem 2 required that the support of the latent factors, Z train , was path-connected. Appendix B.6 shows experiments where this assumption is violated, which yields lower LMSSpear for the additive decoder, thus highlighting the importance of this assumption.

5 Conclusion

We provided an in-depth identifiability analysis of additive decoders , which bears resemblance to standard decoders used in OCRL, and introduced a novel theoretical framework showing how this architecture can generate reasonable images never seen during training via 'Cartesian-product extrapolation'. We validated empirically both of these results and confirmed that additivity was indeed crucial. By studying rigorously how disentanglement can induce extrapolation, our work highlighted the necessity of restricting the decoder to extrapolate and set the stage for future works to explore disentanglement and extrapolation in other function classes such as masked decoders typically used in OCRL. We postulate that the type of identifiability analysis introduced in this work has the potential of expanding our understanding of creativity in generative models, ultimately resulting in representations that generalize better.

References

K. Ahuja, J. Hartford, and Y. Bengio. Properties from mechanisms: an equivariance perspective on identifiable representation learning. In International Conference on Learning Representations , 2022.

K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations, 2022.

K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 2013.

M. Besserve, R. Sun, D. Janzing, and B. Sch¨ olkopf. A theory of independent mechanisms for extrapolation in generative models. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI) , 2021.

J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

J. Brady, R. S. Zimmermann, Y. Sharma, B. Sch¨ olkopf, J. von K¨ ugelgen, and W. Brendel. Provably learning object-centric representations. In International Conference on Machine Learning , 2023.

J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems , 2022.

S. Buchholz, M. Besserve, and B. Sch¨ olkopf. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems , 2022.

S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Sch¨ olkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing, 2023.

C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. Monet: Unsupervised scene decomposition and representation, 2019.

E. Crawford and J. Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence , 2019.

A. S. d'Avila Garcez and L. Lamb. Neurosymbolic AI: The 3rd wave. ArXiv , abs/2012.05876, 2020.

A. Dittadi, S. S. Papa, M. De Vita, B. Sch¨ olkopf, O. Winther, and F. Locatello. Generalization and robustness implications in object-centric learning. In Proceedings of the 39th International Conference on Machine Learning , 2022.

D. Donoho and C. Grimes. Image manifolds which are isometric to euclidean space. Journal of Mathematical Imaging and Vision , 2003.

D. L. Donoho and C. Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences , 2003.

Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems , 2019.

M. Engelcke, A. R. Kosiorek, O. P. Jones, and I. Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations , 2020.

S. M. A. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, K. Kavukcuoglu, and G. E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems , 2016.

J. A. Fodor and Z. W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition , 1988.

A. Goyal and Y. Bengio. Inductive biases for deep learning of higher-level cognition. Proc. R. Soc. A 478: 20210068 , 2022.

K. Greff, A. Rasmus, M. Berglund, T. Hao, H. Valpola, and J. Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems , 2016.

K. Greff, S. van Steenkiste, and J. Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems , 2017.

K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. In Proceedings of the 36th International Conference on Machine Learning , 2019.

K. Greff, S. van Steenkiste, and J. Schmidhuber. On the binding problem in artificial neural networks. ArXiv , abs/2012.05208, 2020.

L. Gresele, J. V . K¨ ugelgen, V. Stimper, B. Sch¨ olkopf, and M. Besserve. Independent mechanism analysis, a new concept? In Advances in Neural Information Processing Systems , 2021.

H. H¨ alv¨ a, S. L. Corff, L. Leh´ ericy, J. So, Y . Zhu, E. Gassiat, and A. Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

S. Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena , 1990.

D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? In Advances in Neural Information Processing Systems , 2021.

A. Hyv¨ arinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems , 2016.

A. Hyv¨ arinen and H. Morioka. Nonlinear ICA of Temporally Dependent Stationary Sources. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , 2017.

A. Hyv¨ arinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks , 1999.

A. Hyv¨ arinen, H. Sasaki, and R. E. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In AISTATS . PMLR, 2019.

Y. Jiang and B. Aragam. Learning nonparametric latent causal graphs with unknown interventions, 2023.

J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

I. Khemakhem, D. Kingma, R. Monti, and A. Hyv¨ arinen. Variational autoencoders and nonlinear ica: A unifying framework. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , 2020.

I. Khemakhem, R. Monti, D. Kingma, and A. Hyv¨ arinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. In Advances in Neural Information Processing Systems , 2020.

D. A. Klindt, L. Schott, Y. Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethge, and D. M. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In 9th International Conference on Learning Representations , 2021.

D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville. Out-of-distribution generalization via risk extrapolation (rex). In Proceedings of the 38th International Conference on Machine Learning , 2021.

S. Lachapelle and S. Lacoste-Julien. Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Learning , 2022.

S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Y . Bengio, S. Lacoste-Julien, and Q. Bertrand. Synergies between disentanglement and sparsity: a multi-task learning perspective, 2022.

S. Lachapelle, P. Rodriguez Lopez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning , 2022.

B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences , 2017.

F. Leeb, G. Lanzillotta, Y. Annadani, M. Besserve, S. Bauer, and B. Sch¨ olkopf. Structure by architecture: Disentangled representations without regularization, 2021.

Z. Lin, Y. Wu, S. V. Peri, W. Sun, G. Singh, F. Deng, J. Jiang, and S. Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations , 2020.

P. Lippe, S. Magliacane, S. L¨ owe, Y. M. Asano, T. Cohen, and E. Gavves. iCITRIS: Causal representation learning for instantaneous temporal effects. In UAI 2022 Workshop on Causal Representation Learning , 2022.

P. Lippe, S. Magliacane, S. L¨ owe, Y. M. Asano, T. Cohen, and E. Gavves. CITRIS: Causal identifiability from temporal intervened sequences, 2022.

F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Sch¨ olkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning , 2019.

F. Locatello, B. Poole, G. Raetsch, B. Sch¨ olkopf, O. Bachem, and M. Tschannen. Weaklysupervised disentanglement without compromises. In Proceedings of the 37th International Conference on Machine Learning , 2020.

F. Locatello, M. Tschannen, S. Bauer, G. R¨ atsch, B. Sch¨ olkopf, and O. Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations , 2020.

F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems , 2020.

A. Mansouri, J. Hartford, K. Ahuja, and Y. Bengio. Object-centric causal representation learning. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations , 2022.

G. F. Marcus. The algebraic mind : integrating connectionism and cognitive science, 2001.

G. E. Moran, D. Sridhar, Y. Wang, and D. Blei. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research , 2022.

J. Munkres. Analysis On Manifolds . Basic Books, 1991.

J. R. Munkres. Topology . Prentice Hall, Inc., 2 edition, 2000.

J. Pearl. The seven tools of causal inference, with reflections on machine learning. Commun. ACM , 2019.

W. Peebles, J. Peebles, J.-Y. Zhu, A. A. Efros, and A. Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In Proceedings of European Conference on Computer Vision (ECCV) , 2020.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

P. Reizinger, L. Gresele, J. Brady, J. V. K¨ ugelgen, D. Zietlow, B. Sch¨ olkopf, G. Martius, W. Brendel, and M. Besserve. Embrace the gap: VAEs perform independent mechanism analysis. In Advances in Neural Information Processing Systems , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

K. Roth, M. Ibrahim, Z. Akata, P. Vincent, and D. Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In The Eleventh International Conference on Learning Representations , 2023.

B. Sch¨ olkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE - Advances in Machine Learning and Deep Neural Networks , 2021.

C. Squires, A. Seigal, S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In Proceedings of the 40th International Conference on Machine Learning , 2023.

A. Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on Signal Processing , 1999.

J. Von K¨ ugelgen, Y. Sharma, L. Gresele, W. Brendel, B. Sch¨ olkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

J. von K¨ ugelgen, M. Besserve, W. Liang, L. Gresele, A. Keki´ c, E. Bareinboim, D. M. Blei, and B. Sch¨ olkopf. Nonparametric identifiability of causal representations from unknown interventions, 2023.

Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective, 2022.

Z. Wang, L. Gui, J. Negrea, and V. Veitch. Concept algebra for text-controlled vision models, 2023.

T. W. Webb, Z. Dulberg, S. M. Frankland, A. A. Petrov, R. C. O'Reilly, and J. D. Cohen. Learning representations that support extrapolation. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Q. Xi and B. Bloem-Reddy. Indeterminacy in generative models: Characterization and strong identifiability. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , 2023.

J. Zhang, C. Squires, K. Greenewald, A. Srivastava, K. Shanmugam, and C. Uhler. Identifiability guarantees for causal disentanglement from soft interventions, 2023.

Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems , 2022.

Appendix

Table of Contents

Calligraphic & indexing conventions

S B General topology X Closure of the subset X ⊆ R n in the standard topology of R n X ◦ Interior of the subset X ⊆ R n in the standard topology of R n

A Identifiability and Extrapolation Analysis

A.1 Useful definitions and lemmas

We start by recalling some notions of general topology that are going to be used later on. For a proper introduction to these concepts, see for example Munkres [56].

Definition 6 (Regularly closed sets) . A set Z ⊆ R d z is regularly closed if Z = Z ◦ , i.e. if it is equal to the closure of its interior (in the standard topology of R n ).

Definition 7 (Connected sets) . A set Z ⊆ R d z is connected if it cannot be written as a union of non-empty and disjoint open sets (in the subspace topology).

Definition 8 (Path-connected sets) . A set Z ⊆ R d z is path-connected if for all pair of points z 0 , z 1 ∈ Z , there exists a continuous map ϕ : [0 , 1] → Z such that ϕ (0) = z 0 and ϕ (1) = z 1 . Such a map is called a path between z 0 and z 1 .

Definition 9 (Homeomorphism) . Let A and B be subsets of R n equipped with the subspace topology. A function f : A → B is an homeomorphism if it is bijective, continuous and its inverse is continuous.

The following technical lemma will be useful in the proof of Theorem 1. For it, we will need additional notation: Let S ⊆ A ⊆ R n . We already saw that S refers to the closure S in the R n topology. We will denote by cl A ( S ) the closure of S in the subspace topology of A induced by R n , which is not necessarily the same as S . In fact, both can be related via cl A = S ∩ A (see Munkres [56, Theorem 17.4, p.95]).

Lemma 4. Let A,B ⊆ R n and suppose there exists an homeomorphism f : A → B . If A is regularly closed in R n , we have that B ⊆ B ◦ .

Proof. Note that f ∣ ∣ A ◦ is a continuous injective function from the open set A ◦ to f ( A ◦ ) . By the 'invariance of domain' theorem [56, p.381], we have that f ( A ◦ ) must be open in R n . Of course, we have that f ( A ◦ ) ⊆ B , and thus f ( A ◦ ) ⊆ B ◦ (the interior of B is the largest open set contained in B ). Analogously, f -1 ∣ ∣ B ◦ is a continuous injective function from the open set B ◦ to f -1 ( B ◦ ) . Again, by 'invariance of domain', f -1 ( B ◦ ) must be open in R n and thus f -1 ( B ◦ ) ⊆ A ◦ . We can conclude that f ( A ◦ ) = B ◦ .

We can conclude as follow:

where the first inclusion holds by continuity of f

This lemma is taken from [42].

Lemma 5 (Sparsity pattern of an invertible matrix contains a permutation) . Let L ∈ R m × m be an invertible matrix. Then, there exists a permutation σ such that L i,σ ( i ) = 0 for all i .

[56, Thm.18.1 p.104].

̸

Proof. Since the matrix L is invertible, its determinant is non-zero, i.e.

̸

̸

where S m is the set of m -permutations. This equation implies that at least one term of the sum is non-zero, meaning there exists π ∈ S m such that for all i ∈ [ m ] , L i,π ( i ) = 0 .

Definition 10 (Aligned subspaces of R m × n ) . Given a subset S ⊆ { 1 , ..., m } × { 1 , ..., n } , we define

Definition 11 (Useful sets) . Given a partition B of [ d ] , we define

Definition 12 ( C k -diffeomorphism) . Let A ⊆ R n and B ⊆ R m . A map f : A → B is said to be a C k -diffeomorphism if it is bijective, C 2 and has a C 2 inverse.

Remark 2. Differentiability is typically defined for functions that have an open domain in R n . However, in the definition above, the set A might not be open in R n and B might not be open in R m . In the case of an arbitrary domain A , it is customary to say that a function f : A ⊆ R n → R m is C k if there exists a C k function g defined on an open set U ⊆ R n that contains A such that g ∣ ∣ A = f (i.e. g extends f ). With this definition, we have that a composition of C k functions is C k , as usual. See for example p.199 of Munkres [55].

The following lemma allows us to unambiguously define the k first derivatives of a C k function f : A → R m on the set A ◦ .

Lemma 6. Let A ⊆ R n and f : A → R m be a C k function. Then, its k first derivatives is uniquely defined on A ◦ in the sense that they do not depend on the specific choice of C k extension.

Proof. Let g : U → R n and h : V → R n be two C k extensions of f to U ⊆ R n and V ⊆ R n both open in R n . By definition,

The usual derivative is uniquely defined on the interior of the domain, so that

Consider a point x 0 ∈ A ◦ . By definition of closure, there exists a sequence { x k } ∞ k =1 ⊆ A ◦ s.t. lim k →∞ x k = x 0 . We thus have that

where we used the fact that the derivatives of g and h are continuous to go to the second line. Thus, all the C k extensions of f must have equal derivatives on A ◦ . This means we can unambiguously define the derivative of f everywhere on A ◦ to be equal to the derivative of one of its C k extensions.

Since f is C k , its derivative D f is C k -1 , we can thus apply the same argument to get that the second derivative of f is uniquely defined on A ◦ ◦ . It can be shown that A ◦ ◦ = A ◦ . One can thus apply the same argument recursively to show that the first k derivatives of f are uniquely defined on A ◦ .

Definition 13 ( C k -diffeomorphism onto its image) . Let A ⊆ R n . A map f : A → R m is said to be a C k -diffeomorphism onto its image if the restriction f to its image ˜ f : A → f ( A ) is a C k -diffeomorphism.

Remark 3. If S ⊆ A ⊆ R n and f : A → R m is a C k -diffeomorphism on its image, then the restriction of f to S , i.e. f ∣ ∣ S , is also a C k diffeomorphism on its image. That is because f ∣ ∣ S is clearly bijective, is C k (simply take the C k extension of f ) and so is its inverse (simply take the C k extension of f -1 ).

A.2 Relationship between additive decoders and the diagonal Hessian penalty

Proposition 7 (Equivalence between additivity and diagonal Hessian) . Let f : R d z → R d x be a C 2 function. Then,

Proof. We start by showing the ' = ⇒ ' direction. Let B and B ′ be two distinct blocks of B . Let i ∈ B and i ′ ∈ B ′ . We can compute the derivative of f k w.r.t. z i :

where the last equality holds because i ∈ B and not in any other block ¯ B . Furthermore,

where the last equality holds because i ′ ̸∈ B . This shows that D 2 f k ( z ) is block diagonal.

We now show the ' ⇐ = ' direction. Fix k ∈ [ d x ] , B ∈ B . We know that D 2 B,B c f k ( z ) = 0 for all z ∈ R d z . Fix z ∈ R d z . Consider a continuously differentiable path ϕ : [0 , 1] → R | B c | such that ϕ (0) = 0 and ϕ (1) = z B c . As D 2 B,B c f k ( z ) is a continuous function of z , we can use the fundamental theorem of calculus for line integrals to get that

(where D 2 B,B c f k ( z B , ϕ ( t )) ϕ ′ ( t ) denotes a matrix-vector product) which implies that

And the above equality holds for all B ∈ B and all z ∈ R d z .

Choose an arbitrary z ∈ R d z . Consider a continously differentiable path ψ : [0 , 1] → R d z such that ψ (0) = 0 and ψ (1) = z . By applying the fundamental theorem of calculus for line integrals once more, we have that

where the last equality holds by (23). We can further apply the fundamental theorem of calculus for line integrals to each term ∫ 1 0 D B f k ( ψ B ( t ) , 0) ψ ′ B ( t ) dt to get

and since z was arbitrary, the above holds for all z ∈ R d z . Note that the functions f ( B ) k ( z B ) must be C 2 because f k is C 2 . This concludes the proof.

A.3 Additive decoders form a superset of compositional decoders [7]

Compositional decoders were introduced by Brady et al. [7] as a suitable class of functions to perform object-centric representation learning with identifiability guarantees. They are also interested in block-disentanglement, but, contrarily to our work, they assume that the latent vector z is fully supported, i.e. Z = R d z . We now rewrite the definition of compositional decoders in the notation used in this work:

Definition 14 (Compositional decoders, adapted from [7]) . Given a partition B , a differentiable decoder f : R d z → R d x is said to be compositional w.r.t. B whenever the Jacobian D f ( z ) is such that for all i ∈ [ d z ] , B ∈ B , z ∈ R d z , we have

̸

where B c is the complement of B ∈ B .

In other words, each line of the Jacobian can have nonzero values only in one block B ∈ B . Note that this nonzero block can change with different values of z .

The next result shows that additive decoders form a superset of C 2 compositional decoders (Brady et al. [7] assumed only C 1 ). Note that additive decoders are strictly more expressive than C 2 compositional decoders because some additive functions are not compositional, like Example 3 for instance.

Proposition 8 (Compositional implies additive) . Given a partition B , if f : R d z → R d x is compositional (Definition 14) and C 2 , then it is also additive (Definition 1).

Proof. Choose any i ∈ [ d x ] . Our strategy will be to show that D 2 f i is block diagonal everywhere on R d z and use Proposition 7 to conclude that f i is additive.

Choose an arbitrary z 0 ∈ R d z . By compositionality, there exists a block B ∈ B such that D B c f i ( z 0 ) = 0 . We consider two cases separately:

̸

Case 1 Assume D B f i ( z 0 ) = 0 . By continuity of D B f i , there exists an open neighborhood of z 0 , U , s.t. for all z ∈ U, D B f i ( z ) = 0 . By compositionality, this means that, for all z ∈ U , D B c f i ( z ) = 0 . When a function is zero on an open set, its derivative must also be zero, hence DD B c f i ( z 0 ) = 0 . Because f is C 2 , the Hessian is symmetric so that we also have D B c D f i ( z 0 ) = 0 . We can thus conclude that the Hessian D 2 f i ( z 0 ) is such that all entries are zero except possibly for D 2 f i ( z 0 ) B,B . Hence, D 2 f i ( z 0 ) is block diagonal with blocks in B .

̸

Case 2: Assume D B f i ( z 0 ) = 0 . This means the whole row of the Jacobian is zero, i.e. D f i ( z 0 ) = 0 . By continuity of D f i , we have that the set V := ( D f i ) -1 ( { 0 } ) is closed. Thus this set decomposes as V = V ◦ ∪ ∂V where V ◦ and ∂V are the interior and boundary of V , respectively.

Case 2.1: Suppose z 0 ∈ V ◦ . Then we can take a derivative so that D 2 f i ( z 0 ) = 0 , which of course means that D 2 f i ( z 0 ) is diagonal.

Case 2.2: Suppose z 0 ∈ ∂V . By the definition of boundary, for all open set U containing z 0 , U intersects with the complement of V , i.e. ( D f i ) -1 ( R d z \ { 0 } ) . This means we can construct a sequence { z k } ∞ k =1 ⊆ V c which converges to z 0 . By Case 1 , we have that for all k ≥ 1 , D 2 f i ( z k ) is block diagonal. This means that lim k →∞ D 2 f i ( z k ) is block diagonal. Moreover, by continuity of D 2 f i , we have that lim k →∞ D 2 f i ( z k ) = D 2 f i ( z 0 ) . Hence D 2 f i ( z 0 ) is block diagonal.

We showed that for all z 0 ∈ R d z , D 2 f i ( z 0 ) is block diagonal. Hence, f is additive by Proposition 7.

A.4 Examples of local but non-global disentanglement

In this section, we provide examples of mapping v : ˆ Z train →Z train that satisfy the local disentanglement property of Definition 4, but not the global disentanglement property of Definition 3. Note that these notions are defined for pairs of decoders f and ˆ f , but here we construct directly the function v which is usually defined as f -1 ◦ ˆ f . However, given v we can always define f and ˆ f to be such that f -1 ◦ ˆ f = v : Simply take f ( z ) := [ z 1 , . . . , z d z , 0 , . . . , 0] ⊤ ∈ R d x and ˆ f := f ◦ v . This construction however yields a decoder f that is not sufficiently nonlinear (Assumption 2). Clearly the mappings v that we provide in the following examples cannot be written as compositions of decoders f -1 ◦ ˆ f where f and ˆ f satisfy all assumptions of Theorem 2, as this would contradict the theorem. In Examples 5 & 6, the path-connected assumption of Theorem 2 is violated. In Example 7, it is less obvious to see which assumptions would be violated.

Example 5 (Disconnected support with changing permutation) . Let v : ˆ Z → R 2 s.t. ˆ Z = ˆ Z (1) ∪ ˆ Z (2) ⊆ R 2 where ˆ Z (1) = { z ∈ R 2 | z 1 ≤ 0 and z 2 ≤ 0 } and ˆ Z (2) = { z ∈ R 2 | z 1 ≥ 1 and z 2 ≥ 1 } . Assume

Step 1: v is a diffeomorphism. Note that v is its own inverse. Indeed,

Thus, v is bijective on its image. Clearly, v is C 2 , thus v -1 = v is also C 2 . Hence, v is a C 2 -diffeomorphism.

Step 2: v is locally disentangled. The Jacobian of v is given by

̸

which is everywhere a permutation matrix, hence v is locally disentangled.

̸

Step 3: v is not globally disentangled. That is because v 1 ( z 1 , z 2 ) depends on both z 1 and z 2 . Indeed, if z 2 = 0 , we have that v 1 ( -1 , 0) = -1 = 0 = v 1 (0 , 0) . Also, if z 1 = 1 , we have that v 1 (1 , 1) = 1 = 2 = v 1 (1 , 2) .

Example 6 (Disconnected support with fixed permutation) . Let v : ˆ Z → R 2 s.t. ˆ Z = ˆ Z (1) ∪ ˆ Z (2) ⊆ R 2 where ˆ Z (1) = { z ∈ R 2 | z 2 ≤ 0 } and ˆ Z (2) = { z ∈ R 2 | z 2 ≥ 1 } . Assume v ( z ) := z + ( z ∈ ˆ Z (2) ) .

✶ Step 1: v is a diffeomorphism. The image of v is the union of the following two sets: Z (1) := v ( ˆ Z (1) ) = ˆ Z (1) and Z (2) := v ( ˆ Z (2) ) = { z ∈ R 2 | z 2 ≥ 2 } . Consider the map w : Z (1) ∪Z (2) → ˆ Z defined as w ( z ) := z -( z ∈ Z (2) ) . We now show that w is the inverse of v :

If z ∈ ˆ Z (2) , we have

If z ∈ ˆ Z (1) , we have w ( v ( z )) = z -✶ ( z ∈ Z (2) ) = z . (37) A similar argument can be made to show that v ( w ( z )) = z . Thus w is the inverse of v . Both v and its inverse w are C 2 , thus v is a C 2 -diffeomorphism on its image.

Step 2: v is locally disentangled. This is clear since D v ( z ) = I everywhere.

Step 3: v is not globally disentangled. Indeed, the function v 1 ( z 1 , z 2 ) = z 1 + ✶ ( z ∈ ˆ Z (2) ) is not constant in z 2 .

Example 7 (Connected support) . Let v : ˆ Z → R 2 s.t. ˆ Z = ˆ Z ( b ) ∪ ˆ Z ( o ) where ˆ Z ( b ) and ˆ Z ( o ) are respectively the blue and orange regions of Figure 6. Both regions contain their boundaries. The function v is defined as follows:

Step 1: v is a diffeomorphism. Clearly, v 1 is C 2 . To show that v 2 also is, we must verify that v 2 ( z ) is C 2 at the frontier between ˆ Z ( b ) and ˆ Z ( o ) , i.e. when z ∈ [1 / 4 , 1] ×{ 0 } .

v 2 ( z ) is continuous since

v ( z ) is C 1 since

v 2 ( z ) is C 2 since

We will now find an explicit expression for the inverse of v . Define

It is straightforward to see that w ( v ( z )) = z for all z ∈ ˆ Z . One can also show that w is C 2 at the boundary between both regions v ( ˆ Z ( b ) ) and v ( ˆ Z ( o ) ) , i.e. when z ∈ [1 / 4 , 1] ×{ 1 } .

Since both v and its inverse w are C 2 , v is a C 2 -diffeomorphism.

Step 2: v is locally disentangled. The Jacobian of v is

which is a permutation-scaling matrix everywhere on ˆ Z . Thus local disentanglement holds. Step 3: v is not globally disentangled. However, v 2 ( z 1 , z 2 ) is not constant in z 1 . Indeed,

̸

Thus global disentanglement does not hold.

A.5 Proof of Theorem 1

Proposition 9. Suppose that the data-generating process satisfies Assumption 1, that the learned decoder ˆ f : R d z → R d x is a C 2 -diffeomorphism onto its image and that the encoder ˆ g : R d x → R d z is continuous. Then, if ˆ f and ˆ g solve the reconstruction problem on the training distribution, i.e. E train || x -ˆ f (ˆ g ( x )) || 2 = 0 , we have that f ( Z train ) = ˆ f ( ˆ Z train ) and the map v := f -1 ◦ ˆ f is a C 2 -diffeomorphism from ˆ Z train to Z train .

Proof. First note that

which implies that, for P train z -almost every z ∈ Z train ,

But since the functions on both sides of the equations are continuous, the equality holds for all z ∈ Z train . This implies that f ( Z train ) = ˆ f ◦ ˆ g ◦ f ( Z train ) = ˆ f ( ˆ Z train ) .

By Remark 3, the restrictions f : Z train → f ( Z train ) and ˆ f : ˆ Z train → ˆ f ( ˆ Z train ) are C 2 -diffeomorphisms and, because f ( Z train ) = ˆ f ( ˆ Z train ) , their composition v := f -1 ◦ ˆ f : ˆ Z train → Z train is a well defined C 2 -diffeomorphism (since C 2 -diffeomorphisms are closed under composition).

Theorem 1 (Local disentanglement via additive decoders) . Suppose that the data-generating process satisfies Assumption 1, that the learned decoder ˆ f : R d z → R d x is a C 2 -diffeomorphism, that the encoder ˆ g : R d x → R d z is continuous, that both f and ˆ f are additive (Definition 1) and that f is sufficiently nonlinear as formalized by Assumption 2. Then, if ˆ f and ˆ g solve the reconstruction problem on the training distribution, i.e. E train || x -ˆ f (ˆ g ( x )) || 2 = 0 , we have that ˆ f is locally B -disentangled w.r.t. f (Definition 4) .

Proof. We can apply Proposition 9 and have that the map v := f -1 ◦ ˆ f is a C 2 -diffeomorphism from ˆ Z train to Z train . This allows one to write

Since Z train is regularly closed and is diffeomorphic to ˆ Z train , by Lemma 4, we must have that ˆ Z train ⊆ ( ˆ Z train ) ◦ . Moreover, the left and right hand side of (49) are C 2 , which means they have uniquely defined first and second derivatives on ( ˆ Z train ) ◦ by Lemma 6. This means the derivatives are uniquely defined on ˆ Z train .

Let z ∈ ˆ Z train . Choose some J ∈ B and some j ∈ J . Differentiate both sides of the above equation with respect to z j , which yields:

Choose J ′ ∈ B \ { J } and j ′ ∈ J ′ . Differentiating the above w.r.t. z j ′ yields

where B 2 < := B 2 ∩{ ( i, i ′ ) | i ′ < i } . For the sake of notational conciseness, we are going to refer to S B and S c B as S and S c (Definition 11). Also, define

Let us define the vectors

This allows us to rewrite, for all k ∈ { 1 , ..., d x }

We define

which allows us to write, for all k ∈ { 1 , ..., d z }

We can now recognize that the matrix W ( v ( z )) of Assumption 2 is given by

Since W ( v ( z )) has full column-rank (by Assumption 2 and the fact that v ( z ) ∈ Z train ), there exists q rows that are linearly independent. Let K be the index set of these rows. This means W ( v ( z )) K, · is an invertible matrix. We can thus write

which means, in particular, that, ∀ i ∈ { 1 , . . . , d z } , ⃗ b i ( z ) = 0 , i.e.,

̸

Since the v is a diffeomorphism, its Jacobian matrix D v ( z ) is invertible everywhere. By Lemma 5, this means there exists a permutation π such that, for all j , D j v π ( j ) ( z ) = 0 . This and (66) imply that

̸

To show that D v ( z ) is a B -block permutation matrix, the only thing left to show is that π respects B . For this, we use the fact that, ∀ B ∈ B , ∀ ( i, i ′ ) ∈ B 2 < , ⃗ c i,i ′ ( z ) = 0 (recall M ( z ) = 0 ). Because ⃗ c i,i ′ ( z ) = ⃗ c i ′ ,i ( z ) , we can write

̸

We now show that if ( j, j ′ ) ∈ S c (indices belong to different blocks), then ( π ( j ) , π ( j ′ )) ∈ S c (they also belong to different blocks). Assume this is false, i.e. there exists ( j 0 , j ′ 0 ) ∈ S c such that ( π ( j 0 ) , π ( j ′ 0 )) ∈ S . Then we can apply (69) (with i := π ( j 0 ) and i ′ := π ( j ′ 0 ) ) and get

̸

where the left term in the sum is different of 0 because of the definition of π . This implies that

̸

otherwise (70) cannot hold. But (71) contradicts (68). Thus, we have that,

From the above, it is clear that π -1 respects B which implies that π respects B (Lemma 10). Thus D v ( z ) is a B -block permutation matrix.

which allows us to write

The contraposed is

Lemma 10 ( B -respecting permutations form a group) . Let B be a partition of { 1 , . . . , d z } and let π and ¯ π be a permutation of { 1 , . . . , d z } that respect B . The following holds:

The identity permutation e respects B .

The composition π ◦ ¯ π respects B .

The inverse permutation π -1 respects B .

Proof. The first statement is trivial, since for all B ∈ B , e ( B ) = B ∈ B .

The second statement follows since for all B ∈ B , ¯ π ( B ) ∈ B and thus π (¯ π ( B )) ∈ B .

We now prove the third statement. Let B ∈ B . Since π is surjective and respects B , there exists a B ′ ∈ B such that π ( B ′ ) = B . Thus, π -1 ( B ) = π -1 ( π ( B ′ )) = B ′ ∈ B .

A.6 Sufficient nonlinearity v.s. sufficient variability in nonlinear ICA with auxiliary variables

In Section 3.1, we introduced the 'sufficient nonlinearity' condition (Assumption 2) and highlighted its resemblance to the 'sufficient variability' assumptions often found in the nonlinear ICA literature [30, 31, 33, 36, 37, 42, 73]. We now clarify this connection. To make the discussion more concrete, we consider the sufficient variability assumption found in Hyv¨ arinen et al. [33]. In this work, the latent variable z is assumed to be distributed according to

In other words, the latent factors z i are mutually conditionally independent given an observed auxiliary variable u . Define

We now recall the assumption of sufficient variability of Hyv¨ arinen et al. [33]:

Assumption 3 (Assumption of variability from Hyv¨ arinen et al. [33, Theorem 1]) . For any z ∈ R d z , there exists 2 d z +1 values of u , denoted by u (0) , u (1) , . . . , u (2 d z ) such that the 2 d z vectors

are linearly independent.

To emphasize the resemblance with our assumption of sufficient nonlinearity, we rewrite it in the special case where the partition B := {{ 1 } , . . . , { d z }} . Note that, in that case, q := d z + ∑ B ∈B | B | ( | B | +1) 2 = 2 d z .

Assumption 4 (Sufficient nonlinearity (trivial partition)) . For all z ∈ Z train , f is such that the following matrix has independent columns (i.e. full column-rank):

One can already see the resemblance between Assumptions 3 & 4, e.g. both have something to do with first and second derivatives. To make the connection even more explicit, define w ( z , k ) to be the k th row of W ( z ) (do not conflate with w ( z , u ) ). Also, recall the basic fact from linear algebra that the column-rank is always equal to the row-rank. This means that W ( z ) is full column-rank if and only if there exists k 1 , ..., k 2 d z ∈ [ d x ] such that the vectors w ( z , k 1 ) , . . . , w ( z , k 2 d z ) are linearly independent. It is then easy to see the correspondance between w ( z , k ) and w ( z , u ) -w ( z , u (0) ) (from Assumption 3) and between the pixel index k ∈ [ d x ] and the auxiliary variable u .

We now look at why Assumption 2 is likely to be satisfied when d x >> d z . Informally, one can see that when d x is much larger than 2 d z , the matrix W ( z ) has much more rows than columns and thus it becomes more likely that we will find 2 d z rows that are linearly independent, thus satisfying Assumption 2.

det(jac.T@jac)

1.2

Level sets of the first two components of f

7.5

1.2

0.40.

0.95

1.2

-0.010

det[W(z1,z2)]

0.400

0.60

0.8

1.0

6.0

1.0

0.8

0.70

0.50

0.80

1.0 -

8°0

0.025

0.300

0.200

0.6

4.5

0.6

0.90

0.8

0.60

0.6

0.100

0.4

3.0

3.0

0.4

0.40

0.4

0.050

0.050

0.025

0.2

0.0

1.5

1.5

0.2

0.0

0.20

0.2

0.0

0.200

0.010

0.000

0.2

1.2

1.0

0.80.6

0.4

0.2

0.0

0.2

0.0

0.2

1.2

-1.0

0.8

-0.6

0.4

0.2

0.0

0.2

0.00

0.2

1.2

1.0

-0.8

0.6

0.300

0.4

0.2

0.0

0.2

0.010

A.7 Examples of sufficiently nonlinear additive decoders

Example 8 (A sufficiently nonlinear f - Example 3 continued) . Consider the additive function

We will provide a numerical verification that this function is a diffeomorphism from the square [ -1 , 0] × [0 , 1] to its image that satisfies Assumption 2.

The Jacobian of f is given by

and the matrix W ( z ) from Assumption 2 is given by

Figure 7 presents a numerical verification that f is injective, has a full rank Jacobian and satisfies Assumption 2. Injective f with full rank Jacobian is enough to conclude that f is a diffeomorphism onto its image.

Example 9 (Smooth balls dataset is sufficiently nonlinear - Example 4 continued) . We implemented a ground-truth additive decoder f : [0 , 5] 2 → R 64 ∗ 64 ∗ 3 which maps to 64x64 RGB images consisting of two colored balls where z 1 and z 2 control their respective heights (Figure 8a). The analytical form of f can be found in our code base. The decoder f is implemented in JAX [6] which allows for its automatic differentiation to compute D f and D 2 f (Figures 8b & 8c). This allows us to verify numerically that f is sufficiently nonlinear (Assumption 2). Recall that this assumption requires that W ( z ) (defined in Assumption 2) has independent columns everywhere. To test this, we compute Vol ( z ) := √ | det( W ( z ) ⊤ W ( z )) | over a grid of values of z and verify that Vol ( z ) > 0 everywhere (Figure 8d). Note that Vol ( z ) corresponds to the 4 D volume of the parallelepiped embedded in R 64 ∗ 64 ∗ 3 spanned by the four columns of W ( z ) . This volume is > 0 if and only if the columns are linearly independent. Note that we normalize the columns of W ( z ) so that they have a norm

of one. It follows that Vol ( z ) is between 0 and 1 where 1 means the vectors are orthogonal, i.e. maximally independent. The minimal value of Vol ( z ) over the domain of f is ≈ 0 . 97 , indicating that Assumption 2 holds.

(a)

(b)

(c)

(d)

(e)

10

10

20

20

20

20

30

OE

30

40

40

50

50

60

10

20

50

60

10

20

sqrt(|det(W.T@W))

1.004

sqrt(|det(W.T@W)))

0.992

8860

0.996

1.000

0.978

0.979

0.675

0.977

0.976

0.978

7660

0.996

0.650

0.976

0

0.977

Height of first ball(z1)

8860

0.992

Height of first bal (z1)

0.976

0.625

0.988

0.975

0.600

0.996

。。

0.996

0.980

0.984

0.575

0.975

0.972

0.974

0.973

0.976

0.550

0.977

0.975

0.973

0.976

0.972

+0

0

0.998888

Height of second ball (z2)

2

3

0.988

4

0.968

0.972

0.500

0.525

0.500 0.525 0.550 0.575 0.600 0.625 0.650 0.675

Height of second ball (z2)

0.975

97

0.970

0.971

D

10

20

30

40

50

60

0

10

20

30

40

50

09

A.8 Proof of Theorem 2

We start with a simple definition:

Definition 15 ( B -block permutation matrices) . A matrix A ∈ R d × d is a B -block permutation matrix if it is invertible and can be written as A = CP π where P π is the matrix representing the B -respecting permutation π ( P π e i = e π ( i ) ) and C ∈ R d × d S B (See Definitions 10 & 11).

The following technical lemma leverages continuity and path-connectedness to show that the blockpermutation structure must remain the same across the whole domain. It can be skipped at first read.

Lemma 11. Let C be a connected topological space and let M : C → R d × d be a continuous function. Suppose that, for all c ∈ C , M ( c ) is an invertible B -block permutation matrix (Definition 15). Then, there exists a B -respecting permutation π such that for all c ∈ C and all distinct B,B ′ ∈ B , M ( c ) π ( B ′ ) ,B = 0 .

Proof. The reason this result is not trivial, is that, even if M ( c ) is a B -block permutation for all c , the permutation might change for different c . The goal of this lemma is to show that, if C is connected and the map M ( · ) is continuous, then one can find a single permutation that works for all c ∈ C .

First, since C is connected and M is continuous, its image, M ( C ) , must be connected (by [56, Theorem 23.5]).

Second, from the hypothesis of the lemma, we know that

where S ( B ) is the set of B -respecting permutations and R d × d S B P π = { MP π | M ∈ R d × d S B } . We can rewrite the set A above as

We now define an equivalence relation ∼ over B -respecting permutation: π ∼ π ′ iff for all B ∈ B , π ( B ) = π ′ ( B ) . In other words, two B -respecting permutations are equivalent if they send every block to the same block (note that they can permute elements of a given block differently). We notice that

Let S ( B ) / ∼ be the set of equivalence classes induce by ∼ and let Π stand for one such equivalence class. Thanks to (84), we can define, for all Π ∈ S ( B ) / ∼ , the following set:

where the specific choice of π ∈ Π is arbitrary (any π ′ ∈ Π would yield the same definition, by (84)). This construction allows us to write

We now show that { V Π } Π ∈ S ( B ) / ∼ forms a partition of A . Choose two distinct equivalence classes of permutations Π and Π ′ and let π ∈ Π and π ′ ∈ Π ′ be representatives. We note that

since any matrix that is both in R d × d S B P π and R d × d S B P π ′ must have at least one row filled with zeros. This implies that

which shows that { V Π } Π ∈ S ( B ) / ∼ is indeed a partition of A .

Each V Π is closed in A (wrt the relative topology) since

Moreover, V Π is open in A , since

̸

̸

Thus, for any Π ∈ S ( B ) / ∼ , the sets V Π and ⋃ Π ′ =Π V Π ′ forms a separation (see [56, Section 23]). Since M ( C ) is a connected subset of A , it must lie completely in V Π or ⋃ Π ′ =Π V Π ′ , by [56, Lemma 23.2]. Since this is true for all Π , it must follow that there exists a Π ∗ such that M ( C ) ⊆ V Π ∗ , which completes the proof.

̸

Theorem 2 (From local to global disentanglement) . Suppose that all the assumptions of Theorem 1 hold. Additionally, assume Z train is path-connected (Definition 8) and that the block-specific decoders f ( B ) and ˆ f ( B ) are injective for all blocks B ∈ B . Then, if ˆ f and ˆ g solve the reconstruction problem on the training distribution, i.e. E train || x -ˆ f (ˆ g ( x )) || 2 = 0 , we have that ˆ f is (globally) B -disentangled w.r.t. f (Definition 3) and, for all B ∈ B ,

where the functions ¯ v π ( B ) are from Defintion 3 and the vectors c ( B ) ∈ R d x are constants such that ∑ B ∈B c ( B ) = 0 . We also have that the functions ¯ v π ( B ) : ˆ Z train B →Z train π ( B ) are C 2 -diffeomorphisms and have the following form:

Proof. Step 1 - Showing the permutation π does not change for different z . Theorem 1 showed local B -disentanglement, i.e. for all z ∈ ˆ Z train , D v ( z ) has a B -block permutation structure. The first step towards showing global disentanglement is to show that this block structure is the same for all z ∈ ˆ Z train ( a priori , π could be different for different z ). Since v is C 2 , its Jacobian D v ( z ) is continuous. Since Z train is path-connected, ˆ Z train must also be since both sets are diffeomorphic. By Lemma 11, this means the B -block permutation structure of D v ( z ) is the same for all z ∈ ˆ Z train (implicitly using the fact that path-connected implies connected). In other words, there exists a permutation π respecting B such that, for all z ∈ ˆ Z train and all distinct B,B ′ ∈ B , D B v π ( B ′ ) ( z ) = 0 .

Step 2 - Linking object-specific decoders. We now show that, for all B ∈ B , ˆ f ( B ) ( z B ) = f ( π ( B )) ( v π ( B ) ( z )) + c ( B ) for all z ∈ ˆ Z train . To do this, we rewrite (50) as

but because B = π ( J ) = ⇒ D J v B ( z ) = 0 (block-permutation structure), we get

The above holds for all J ∈ B . We simply change J by B in the following equation.

Now notice that the r.h.s. of the above equation is equal to D ( f ( π ( B )) ◦ v π ( B ) ) . We can thus write

Now choose distinct z , z 0 ∈ ˆ Z train . Since Z train is path-connected, ˆ Z train also is since they are diffeomorphic. Hence, there exists a continuously differentiable function ϕ : [0 , 1] → ˆ Z train such that ϕ (0) = z 0 and ϕ (1) = z . We can now use (94) together with the gradient theorem, a.k.a. the fundamental theorem of calculus for line integrals, to show the following

which holds for all z ∈ ˆ Z train .

We now show that ∑ B ∈B c ( B ) = 0 . Take some z 0 ∈ ˆ Z train . Equations (49) & (98) tell us that

̸

̸

Step 3 - From local to global disentanglement. By assumption, the functions f ( B ) : Z train B → R d x are injective. This will allow us to show that v π ( B ) ( z ) depends only on z B . We proceed by contradiction. Suppose there exists ( z B , z B c ) ∈ ˆ Z train and z 0 B c such that ( z B , z 0 B c ) ∈ ˆ Z train and v π ( B ) ( z B , z B c ) = v π ( B ) ( z B , z 0 B c ) . This means

which is a contradiction with the fact that f ( π ( B )) is injective. Hence, v π ( B ) ( z ) depends only on z B . We also get an explicit form for v π ( B ) :

We define the map ¯ v π ( B ) ( z B ) := ( f π ( B ) ) -1 ( ˆ f ( B ) ( z B ) -c ( B ) ) which is from ˆ Z train B to Z train π ( B ) . This allows us to rewrite (98) as

Because ˆ f ( B ) is also injective, we must have that ¯ v π ( B ) : ˆ Z train B →Z train π ( B ) is injective as well.

We now show that ¯ v π ( B ) is surjective. Choose some z π ( B ) ∈ Z train π ( B ) . We can always find z π ( B ) c such that ( z π ( B ) , z π ( B ) c ) ∈ Z train . Because v : ˆ Z train →Z train is surjective (it is a diffeomorphism), there exists a z 0 ∈ ˆ Z train such that v ( z 0 ) = ( z π ( B ) , z π ( B ) c ) . By (103), we have that

which means ¯ v π ( B ) ( z 0 B ) = z π ( B ) .

We thus have that ¯ v π ( B ) is bijective. It is a diffeomorphism because

̸

where the first equality holds by (103) and the second holds because v is a diffeomorphism and has block-permutation structure, which means it has a nonzero determinant everywhere on ˆ Z train and is equal to the product of the determinants of its blocks, which implies each block D B v π ( B ) must have nonzero determinant everywhere.

Since ¯ v π ( B ) : ˆ Z train B → Z train π ( B ) bijective and has invertible Jacobian everywhere, it must be a diffeomorphism.

A.9 Injectivity of object-specific decoders v.s. injectivity of their sum

We want to explore the relationship between the injectivity of individual object-specific decoders f ( B ) and the injectivity of their sum, i.e. ∑ B ∈B f ( B ) .

We first show the simple fact that having each f ( B ) injective is not sufficient to have ∑ B ∈B f ( B ) injective. Take f ( B ) ( z B ) = W ( B ) z B where W ( B ) ∈ R d x ×| B | has full column-rank for all B ∈ B . We have that

where it is clear that the matrix [ W ( B 1 ) · · · W ( B ℓ ) ] ∈ R d x × d z is not necessarily injective even if each W ( B ) is. This is the case, for instance, if all W ( B ) have the same image.

We now provide conditions such that ∑ B ∈B f ( B ) injective implies each f ( B ) injective. We start with a simple lemma:

Lemma 12. If g ◦ h is injective, then h is injective.

Proof. By contradiction, assume that h is not injective. Then, there exists distinct x 1 , x 2 ∈ Dom ( h ) such that h ( x 1 ) = h ( x 2 ) . This implies g ◦ h ( x 1 ) = g ◦ h ( x 2 ) , which violates injectivity of g ◦ h .

The following Lemma provides a condition on the domain of the function ∑ B ∈B f ( B ) , Z train , so that its injectivity implies injectivity of the functions f ( B ) .

Lemma 13. Assume that, for all B ∈ B and for all distinct z B , z ′ B ∈ Z train B , there exists z B c such that ( z B , z B c ) , ( z ′ B , z B c ) ∈ Z train . Then, whenever ∑ B ∈B f ( B ) is injective, each f ( B ) must be injective.

Proof. Notice that f ( z ) := ∑ B ∈B f ( B ) ( z B ) can be written as f := SumBlocks ◦ ¯ f ( z ) where

Since f is injective, by Lemma 12 ¯ f must be injective.

̸

We now show that each f ( B ) must also be injective. Take z B , z ′ B ∈ Z train B such that f ( B ) ( z B ) = f ( B ) ( z ′ B ) . By assumption, we know there exists a z B c s.t. ( z B , z B c ) and ( z ′ B , z B c ) are in Z train . By construction, we have that ¯ f (( z B , z B c )) = ¯ f (( z ′ B , z B c )) . By injectivity of ¯ f , we have that ( z B , z B c ) = ( z ′ B , z B c ) , which implies z B = z ′ B , i.e. f ( B ) is injective.

A.10 Proof of Corollary 3

Corollary 3 (Cartesian-product extrapolation) . Suppose the assumptions of Theorem 2 holds. Then,

Furthermore, if CPE B ( Z train ) ⊆ Z test , then ˆ f ( CPE B ( ˆ Z train )) ⊆ f ( Z test ) .

Proof. Pick z ∈ CPE ( ˆ Z train ) . By definition, this means that, for all B ∈ B , z B ∈ ˆ Z train B . We thus have that, for all B ∈ B ,

We can thus sum over B to obtain

Since z ∈ CPE ( ˆ Z train ) was arbitrary, we have

where ¯ v : CPE B ( ˆ Z train ) → CPE B ( Z train ) is defined as

The map ¯ v is a diffeomorphism since each ¯ v π ( B ) is a diffeomorphism from ˆ Z train B to Z train π ( B ) . By (112) we get

and since the map ¯ v is surjective we have ¯ v ( CPE B ( ˆ Z train )) = CPE B ( Z train ) and thus

̸

A.11 Will all extrapolated images make sense?

Here is a minimal example where the assumption CPE B ( Z train ) ̸⊆ Z test is violated.

Example 10 (Violation of CPE B ( Z train ) ̸⊆ Z test ) . Imagine z = ( z 1 , z 2 ) where z 1 and z 2 are the x -positions of two distinct balls. It does not make sense to have two balls occupying the same location in space and thus whenever z 1 = z 2 we have ( z 1 , z 2 ) ̸∈ Z test . But if (1 , 2) and (2 , 1) are both in Z train , it implies that (1 , 1) and (2 , 2) are in CPE ( Z train ) , which is a violation of CPE B ( Z train ) ⊆ Z test .

A.12 Additive decoders cannot model occlusion

We now explain why additive decoders cannot model occlusion. Occlusion occurs when an object is partially hidden behind another one. Intuitively, the issue is the following: Consider two images consisting of two objects, A and B (each image shows both objects). In both images, the position of object A is the same and in exactly one of the images, object B partially occludes object A. Since the position of object A did not change, its corresponding latent block z A is also unchanged between both images. However, the pixels occupied by object A do change between both images because of occlusion. The issue is that, because of additivity, z A and z B cannot interact to make some pixels that belonged to object A 'disappear' to be replaced by pixels of object B. In practice, object-centric representation learning methods rely a masking mechanism which allows interactions between z A and z B (See Equation 1 in Section 2). This highlights the importance of studying this class of decoders in future work.

B Experiments

B.1 Training Details

Loss Function. We use the standard reconstruction objective of mean squared error loss between the ground truth data and the reconstructed/generated data.

Hyperparameters. For both the ScalarLatents and the BlockLatents dataset, we used the Adam optimizer with the hyperparameters defined below. Note that we maintain consistent hyperparameters across both the Additive decoder and the Non-Additive decoder method.

ScalarLatents Dataset.

Batch Size: 64

Learning Rate: 1 × 10 - 3

Weight Decay: 5 × 10 - 4

Total Epochs: 4000

BlockLatents Dataset.

Batch Size:

1024

Learning Rate: 1 × 10 - 3

Weight Decay: 5 × 10 - 4

Total Epochs: 6000

Model Architecture. We use the following architectures for Encoder and Decoder across both the datasets (ScalarLatents, BlockLatents). Note that for the ScalarLatents dataset we train with latent dimension d z = 2 , and for the BlockLatents dataset we train with latent dimension d z = 4 , which corresponds to the dimensionalities of the ground-truth data generating process for both datasets.

Encoder Architecture:

RestNet-18 Architecture till the penultimate layer ( 512 dimensional feature output)

Stack of 5 fully-connected layer blocks, with each block consisting of Linear Layer ( dimensions: 512 × 512 ), Batch Normalization layer, and Leaky ReLU activation (negative slope: 0 . 01 ).

Final Linear Layer (dimension: 512 × d z ) followed by Batch Normalization Layer to output the latent representation.

Decoder Architecture (Non-additive):

Fully connected layer block with input as latent representation, consisting of Linear Layer (dimension: d z × 512 ), Batch Normalization layer, and Leaky ReLU activation (negative slope: 0 . 01 ).

Stack of 5 fully-connected layer blocks, with each block consisting of Linear Layer ( dimensions: 512 × 512 ), Batch Normalization layer, and Leaky ReLU activation (negative slope: 0 . 01 ).

Series of DeConvolutional layers, where each DeConvolutional layer is follwed by Leaky ReLU (negative slope: 0 . 01 ) activation.

-DeConvolution Layer ( c : 64 , c : 64 , kernel: 4 ; stride: 2 ; padding: 1

-DeConvolution Layer ( c : 64 , c : 32 , kernel: 4 ; stride: 2 ; padding: 1 )

-DeConvolution Layer ( c : 32 , c : 32 , kernel: 4 ; stride: 2 ; padding: 1

-DeConvolution Layer ( c : 32 , c : 3 , kernel: 4 ; stride: 2 ; padding: 1 )

Decoder Architecture (Additive): Recall that an additive decoder has the form f ( z ) = ∑ B ∈B f ( B ) ( z B ) . Each f ( B ) has the same architecture as the one presented above for the nonadditive case, but the input has dimensionality | B | (which is 1 or 2, depending on the dataset). Note that we do not share parameters among the functions f ( B ) .

B.2 Datasets Details

We use the moving balls environment from Ahuja et al. [2] with images of dimension 64 × 64 × 3 , with latent vector ( z ) representing the position coordinates of each balls. We consider only two balls. The rendered images have pixels in the range [0, 255].

ScalarLatents Dataset. We fix the x-coordinate of each ball to 0 . 25 and 0 . 75 . The only factors varying are the y-coordinates of both balls. Thus, z ∈ R 2 and B = {{ 1 } , { 2 }} where z 1 and z 2 designate the y-coordinates of both balls. We sample the y-coordinate of the first ball from a continuous uniform distribution as follows: z 1 ∼ Uniform(0, 1). Then we sample the y-coordinate of the second ball as per the following scheme:

Hence, this leads to the L-shaped latent support, i.e., Z train := [0 , 1] × [0 , 1] \ [0 . 5 , 1] × [0 . 5 , 1] .

We use 50 k samples for the test dataset, while we use 20 k samples for the train dataset along with 5 k samples ( 25% of the train sample size) for the validation dataset.

BlockLatents Dataset. For this dataset, we allow the balls to move in both the x, y directions, so that z ∈ R 4 and B = {{ 1 , 2 } , { 3 , 4 }} . For the case of independent latents , we sample each latent component independently and identically distributed according to a uniform distribution over (0 , 1) , i.e. z i ∼ Uniform(0, 1). We rejected the images that present occlusion, i.e. when one ball hides another one. 2

For the case of dependent latents , we sample the latents corresponding to the first ball similarly from the same continuous uniform distribution, i.e, z 1 , z 2 ∼ Uniform (0, 1). However, the latents of the second ball are a function of the latents of the first ball, as described in what follows:

Intuitively, this means the second ball will be placed in either the top-left or the bottom-right quadrant based on the position of the first ball. We also exclude from the dataset the images presenting occlusion.

Note that our dependent BlockLatent setup is same as the non-linear SCM case from Ahuja et al. [3].

We use 50 k samples for both the train and the test dataset, along with 12 . 5 k samples ( 25% of the train sample size) for the validation dataset.

Disconnected Support Dataset. For this dataset, we have setup similar to the ScalarLatents dataset; we fix the x-coordinates of both balls to 0 . 25 and 0 . 75 and only vary the y-coordinates so that z ∈ R 2 . We sample the y-coordinate of the first ball ( z 1 ) from Uniform(0, 1). Then we sample the y-coordinate of the second ball ( z 2 ) from either of the following continuous uniform distribution with equal probability; Uniform(0, 0.25) and Uniform(0.75, 1). This leads to a disconnected support given by Z train := [0 , 1] × [0 , 1] \ [0 . 25 , 0 . 75] × [0 . 25 , 0 . 75] .

We use 50 k samples for the test dataset, while we use 20 k samples for the train dataset along with 5 k samples ( 25% of the train sample size) for the validation dataset.

B.3 Evaluation Metrics

Recall that, to evaluate disentanglement, we compute a matrix of scores ( s B,B ′ ) ∈ R ℓ × ℓ where ℓ is the number of blocks in B and s B,B ′ is a score measuring how well we can predict the ground-truth block z B from the learned latent block ˆ z B ′ = ˆ g B ′ ( x ) outputted by the encoder. The final Latent Matching Score (LMS) is computed as LMS = arg max π ∈ S B 1 ℓ ∑ B ∈B s B,π ( B ) , where S B is the set of permutations respecting B (Definition 2). These scores are always computed on the test set.

Metric LMSSpear: As mentioned in the main paper, this metric is used for the ScalarLatents dataset where each block is 1-dimensional. Hence, this metric is almost the same as the mean correlation coefficient (MCC), which is widely used in the nonlinear ICA literature [30, 31, 33, 36, 42], with the only difference that we use Spearman correlation instead of Pearson correlation as a score s B,B ′ . The Spearman correlation can capture nonlinear monotonous relations, unlike Pearson which can only capture linear dependencies. We favor Spearman over Pearson because our identifiability result (Theorem 2) guarantees we can recover the latents only up to permutation and element-wise invertible transformations, which can be nonlinear.

Metric LMStree: This metric is used for the BlockLatents dataset. For this metric, we take s B,B ′ to be the R 2 score of a Regression Tree with maximal depth of 10 . For this, we used the class sklearn.tree.DecisionTreeRegressor from the sklearn library. We learn the parameters of the Decision Tree using the train dataset and then use it to evaluate LMStree metric on the test dataset. For the additive decoder, it is easy to compute this metric since the additive structure already gives a natural partition B which matches the ground-truth. However, for the non-additive decoder, there is no natural partition and thus we cannot compute LMStree directly. To go around this problem, for the non-additive decoder, we compute LMStree for all possible partitions of d z latent variables into blocks of size | B | = 2 (assuming all blocks have the same dimension), and report the best LMStree. This procedure is tractable in our experiments due to the small dimensionality of the problem we consider.

B.4 Boxplots for main experiments (Table 1)

Since the standard error in the main results (Table 1) was high, we provide boxplots in Figures 9 & 10 to have a better visibility on what is causing this. We observe that the high standard error for the Additive approach was due to bad performance for a few bad random initializations for the ScalarLatents dataset; while we have nearly perfect latent identification for the others. Figure 14e shows the latent space learned by the worst case seed, which somehow learned a disconnected support even if the ground-truth support was connected. Similarly, for the case of Independent BlockLatents, there are only a couple of bad random initializations and the rest of the cases have perfect identification.

Add.

(In-Supp)

Non-Add.

(In-Supp)

Add.

(Out-Supp)

Non-Add.

(Out-Supp)

10

1

6×10 2

2×10 1

Reconstruction MSE (Log Scale)

Add.

(In-Supp)

Non-Add.

(In-Supp)

Add.

(Out-Supp)

Non-Add.

(Out-Supp)

50

75

100

LMS-Spearman (MCC)

Additive

Non-Additive

10

2

10

1

Reconstruction MSE (Log Scale)

Additive

Non-Additive

25

50

75

100

LMS-Tree

(a) Independent Latent Case

Additive

Non-Additive

10

2

2×10 2

Reconstruction MSE (Log Scale)

Additive

Non-Additive

60

80

100

LMS-Tree

(b) Dependent Latent Case

B.5 Additional Results: BlockLatents Dataset

To get a qualitative understanding of latent identification in the BlockLatents dataset, we plot the response of each predicted latent as we change a particular ground-truth latent factor. We describe the following cases of changing the ground-truth latents:

Ball 1 moving along x-axis: We sample 10 equally spaced points for z 1 from [0 , 1] ; while keeping other latents fixed as follows: z 2 = 0 . 25 , z 3 = 0 . 50 , z 4 = 0 . 75 . We will never have occlusion since the balls are separated along the y-axis z 4 -z 2 > 0 .

Ball 2 moving along x-axis: We sample 10 equally spaced points for z 3 from [0 , 1] ; while keeping other latents fixed as follows: z 1 = 0 . 50 , z 2 = 0 . 25 , z 4 = 0 . 75 . We will never have occlusion since the balls are separated along the y-axis z 4 -z 2 > 0 .

Ball 1 moving along y-axis: We sample 10 equally spaced points for z 2 from [0 , 1] ; while keeping other latents fixed as follows: z 1 = 0 . 25 , z 3 = 0 . 75 , z 4 = 0 . 50 . We will never have occlusion since the balls are separated along the x-axis z 3 -z 1 > 0 .

Ball 2 moving along y-axis: We sample 10 equally spaced points for z 4 from [0 , 1] ; while keeping other latents fixed as follows: z 1 = 0 . 25 , z 2 = 0 . 50 , z 3 = 0 . 75 . We will never have occlusion since the balls are separated along the x-axis z 3 -z 1 > 0 .

)

0.25

0.50

0.75

Ball 1 moving along x axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(a) Additive Decoder (Best) (LMSTree

: 99

.

9

)

0.25

0.50

0.75

Ball 1 moving along x axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(b) Non-Additive Decoder (Best) (LMSTree

: 83

.

9

)

0.25

0.50

0.75

Ball 1 moving along x axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(c) Additive Decoder (Median) (LMSTree

: 99

.

8

)

0.25

0.50

0.75

Ball 1 moving along x axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(d) Non-Additive Decoder (Median) (LMSTree

: 58

.

6

0.25

0.50

0.75

Ball 1 moving along x axis

4

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(e) Additive Decoder (Worst) (LMSTree

: 54

.

1

)

0.25

0.50

0.75

Ball 1 moving along x axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along x axis

2

0

2

0.25

0.50

0.75

Ball 1 moving along y axis

2

0

2

Predicted Latents

0.25

0.50

0.75

Ball 2 moving along y axis

2

0

2

Latent 1

Latent 2

Latent 3

Latent 4

(f) Non-Additive Decoder (Worst) (LMSTree

: 24

.

6

)

Figure 5 in the main paper presents the latent responses plot for the median LMStree case among random initializations. In Figure 11, we provide the results for the case of best and the worst LMStree among random seeds. We find that Additive Decoder fails for only for the worst case random seed, while Non-Additive Decoder fails for all the cases.

Additionally, we provide the object-specific reconstructions for the Additive Decoder in Figure 12. This helps us better understand the failure of Additive Decoder for the worst case random seed (Figure 12c), where the issue arises due to bad reconstruction error.

(a) Additive Decoder (Best)

(b) Additive Decoder (Median)

(c) Additive Decoder (Worst)

B.6 Disconnected Support Experiments

Since path-connected latent support is an important assumption for latent identification with additive decoders (Theorem 2), we provide results for the case where the assumption is not satisfied. We experiment with the Disconnected Support dataset (Section B.2) and find that we obtain much worse LMSSpear as compared to the case of training with L-shaped support in the ScalarLatents dataset. Over 10 different random initializations, we find mean LMSSpear performance of 69 . 5 with standard error of 6 . 69 .

For better qualitative understanding, we provide visualization of the latent support and the extrapolated images for the median LMSSpear among 10 random seeds in Figure 13. Somewhat surprisingly, the representation appears to be aligned in the sense that the first predicted latent corresponds to the blue ball while the second predicted latent correspond to the red ball. Also surprisingly, extrapolation

2

Predicted Latent

2

1

L

0

2

PredictedLatent1

occurs (we can see images of both balls high). That being said, we observe that the relationship between the predicted latent 2 ( ˆ z 2 ) and y-coordinate of second (red) ball is not monotonic, which explains why the Spearman correlation is so low (Spearman correlation scores are high when there is a monotonic relationship between both variables).

B.7 Additional Results: ScalarLatents Dataset

To get a qualitative understanding of extrapolation, we plot the latent support on the test dataset and sample a grid of equally spaced points from the support of each predicted latent on the test dataset. The grid represents the cartesian-product of the support of predicted latents and would contain novel combinations of latents that were unseen during training. We show the reconstructed images for each point from the cartesian-product grid to see whether the model is able to reconstruct well the novel latent combinations.

Figure 4 in the main paper presents visualizations of the latent support and the extrapolated images for the median LMSSpear case among random seeds. In Figure 14, we provide the results for the case of best and the worst LMSSpear among random seeds. We find that even for the best case (Figure 14b), Non-Additive Decoder does not generate good quality extrapolated images, while Additive Decoder generates extrapoalted images for the best and median case. The worst-case run for the Additive Decoder has disconnected support, which explains why it is not able to extrapolate.

)

(a) Additive Decoder (Best) (LMSSpear

: 99

.

9

)

(b) Non-Additive Decoder (Best) (LMSSpear

: 99

.

9

)

(c) Additive Decoder (Median) (LMSSpear

: 99

.

9

)

(d) Non-Additive Decoder (Median) (LMSSpear

: 76

.

1

(e) Additive Decoder (Worst) (LMSSpear

: 69

.

5

)

(f) Non-Additive Decoder (Worst) (LMSSpear

: 39

.

8

)

Predicted Latent

2

0.100.08-0.06-0.04-0.020.00 0.02

Predicted Latent1

2

Predicted Latent

-1

1.5-1.0-0.5 0.0 0.5 1.0 1.5

2.0

PredictedLatent1

2

Predicted Latent

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

-0.6-0.4-0.20.00.20.40.6

PredictedLatent1

Predicted Latent

2

1.0

0.5

0.0

0.5

1.0

2.0

2.5

2.01.51.00.5 0.0 0.5 1.0 1.5 2.0

PredictedLatent1

2

PredictedLatent

1

-1.5-1.0-0.5 0.0 0.5 1.0 1.5 2.0 2.5

PredictedLatent1

2

PredictedLatent

1.0

0.5

0.0

0.5

-1.0

1.5

2.0

-2

-1

0

2

Predicted Latent1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a theory to show how an additive decoder may be able to disentangle an image composed of several components. The proposed theory also shows how an additive decoder may produce novel images, possibly providing insights about the process performed by generative models.\n\nBesides the theory, paper also performs some experiments showing that additive decoders may have certain advantages in performing those tasks.\n\nSTRENGTHS:\nPaper is well written.\n\nThe topic and the experiments are interesting.\n\nIt has a broad and practical view. Literature review is relatively good.\n\nThe proposed theory might turn into a useful contribution for the research community.\n\nWEAKNESSES:\nExperiments are interesting but limited and disconnected from existing experiments in the literature, in my view.\n\n-----------------\n\nThere are a few publications that are relevant but not cited:\n\n–Li, N., Raza, M.A., Hu, W., Sun, Z. and Fisher, R, Object-centric representation learning with generative spatial-temporal factorization, NeurIPS 2021.\n\n–Yoon, J., Wu, Y.F., Bae, H. and Ahn, S., An investigation into pre-training object-centric representations for reinforcement learning, ICML 2023.\n\nBoth of the above papers have experiments on images that may be decomposed in the authors’ additive scenario and possibly be used as a baseline for comparison.\n\n-----------------\n\n“Reasonableness”, mentioned under section 3.2, seems to be an unclear definition underlying a significant portion of the theory. It appears that authors attempt to define the reasonableness, yet, it is not clear what is the difference between $Z^{test}$ and $Z^{train}$. Given the bijective assumption, the difference between $Z^{test}$ and $Z^{train}$ should refer to a specific region in the domain and range of the function. Yet, it is not clear what that difference is.\n\nI understand it is hard to define the limits of the underlying manifold of relevant images - that is exactly the heart of the difficulty in developing useful theory for deep learning. Could authors expand on their “reasonable” assumption? Perhaps identifying the boundaries of the “reasonable” manifold is hard, but it may be helpful to describe and contrast what is unreasonable. Perhaps providing a discussion and citing some previous studies on the underlying manifold of images would be helpful as well, e.g.:\n \nCohen, U., Chung, S., Lee, D.D. and Sompolinsky, H., 2020. Separability and geometry of object manifolds in deep neural networks. Nature Communications, 11(1), p.746.\n\n\n-----------------\n\nThere is an extrapolation study and a dataset called VAEC from the paper below.\n\nWebb, T., Dulberg, Z., Frankland, S., Petrov, A., O’Reilly, R. and Cohen, J., Learning representations that support extrapolation. ICML 2020.\n\nThe images in the VAEC dataset are designed as an extrapolation task. Do authors think this task can fit into their framework?\n\n\n-----------------\n\nIt seems that the notation D (for the Jacobian) is only defined in the appendix under Table 2. Since the notation is used in the main body of the paper, it would be useful to define it there. If instead of D, $\\nabla$ was used for the Jacobian, I would have inferred what authors mean by it. However, I was not sure about D until I found it in the appendix.\n\n-----------------\n\nFor assumption 2, it may be better to use “linearly independent” instead of “independent”.\n\nMoreover, it might be better to explain in words that: this assumption is requiring the … matrices, to have full column rank.\n\nOverall, the notion formalized in assumption 2 seems strange to me. Are authors familiar with the notion of curvature for functions and manifolds?\n\nIs there any precedence for the notion of nonlinearity defined under assumption 2 for any class of functions? I am not sure how authors’ notion of nonlinearity relates to known notions of nonlinearity/curvature, for example, the notions of curvature in differential geometry. Why should assumption 2 be satisfied over the entire manifold?\n\nIn its current form, assumption 2 stacks the first and second order derivatives together and then requires that the stacked matrix to have full column rank. It is not clear to me why stacking of these matrices is necessary. If the unrolled second derivatives have full column rank, would it be necessary for the first derivatives to have full column rank? If each of the first and second order derivatives, individually have full column rank, would that be sufficient?\n\n\n-----------------\n\nThis is not a weakness of the paper, but perhaps worth mentioning. It is common in the literature to use x as the input to a function/model, and use y or z as the output of a function/model. However, this paper uses z to denote the inputs and x to denote the output. This may sometimes be a bit confusing. But that is the authors’ choice, and this is just feedback.\n\nQUESTIONS:\nPlease see the questions under weaknesses, especially the questions about assumption 2. (Assumption 2 and what is built on it is my main concern about this work. I hope authors can be more clear and more convincing about their approach.)\n\nHave authors considered expanding their own experiments to something more sophisticated? For example, the shape of objects could be not just circles, but several other types: diamonds, triangles, rectangles, etc. The size of the objects could vary as well. \n\nDo authors think their extrapolation method can be applied to the VAEC dataset (or some modification of it)? The point is to connect this paper’s experiments to existing experiments in the literature. For example, when an object in the VAEC dataset is enlarged, the enlarged object can be considered an addition of two smaller objects which would fit the authors’ additive framework. Currently, this paper’s experiments seem to be isolated from the literature. If VAEC is not suitable, authors may want to consider some other datasets from the literature.\n\nLIMITATIONS:\nI did not see a discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyses the statistical identifiability of latent variables in an autoencoder with a so-called additive decoder. It is shown that under this class of decoders, the blocks of latent dimensions associated with the additive decoder can be identified. This result is further related to the ability of a model to extrapolate.\n\nSTRENGTHS:\n(These may be subject to change depending on the answers to my questions below)\n\n1. The paper is generally very well written in terms of giving intuitions behind the presented math (see the counter view in Weakness 1 below).\n2. The paper touches on an important topic (identifiability), and the focus on additive decoders is both interesting, relevant, and novel.\n3. The paper does a nice job of providing proof sketches, which is helpful since space constraints prevent the authors from including actual proofs in the main text.\n4. The paper does a very nice job of connecting assumptions and results to existing work in various branches of the literature. This is very helpful.\n5. Finally, I want to emphasize that the theoretical findings are both novel and interesting.\n\nWEAKNESSES:\n(These may be subject to change depending on the answers to my questions below)\n\n1. The mathematics is often phrased sufficiently convoluted that the phrased intuitions are required (see Strength 1 above). E.g. I found definition 3 to be nearly unreadable, and I could not verify if the clearly phrased intuitions (lines 177-178) actually describe the math (I trust that it does, but it's a problem that it is so difficult to verify).\n2. The 'additive decoder' construction seems quite similar to mixture models for which decades of work exist regarding identifiability. I was surprised to not see this link even briefly touched upon.\n3. I found the extrapolation part of the paper to be less convincing than the identifiability part. Bluntly put, I got lost in the many assumptions made (Corollary 3 holds under the assumptions of Theorem 2, which hold under the assumptions of Theorem 1 which holds under Assumption 1) that I was unable to tell which were the important assumptions for the particular corollary. Thus, I lost my intuition, and the following discussion (Lines 381-304) seems rather speculative. Fere I struggle to determine what's what.\n\nQUESTIONS:\n1. It seems to me that additive decoders are effectively a form of mixture model with non-trivial components. Here, we know quite a bit about conditions under which components can be identified up to permutation. This seems quite similar to the presented results. Can you elaborate on this connection? Am I misunderstanding something?\n2. In definition 2, you write \"let $\\mathcal{B}$ be a partition...\" Should this have been $\\mathcal{B}$ be the set of partitions...\"? Otherwise, I struggle to understand what $B \\in \\mathcal{B}$ actually means. But perhaps I misunderstood something.\n3. Can we agree that $v := f^{-1} \\circ \\hat{f}$ is a diffeomorphism simply because assumption 1 states that $f$ is a diffeomorphism or is there more to this? (I ask as the statement about $v$ appears in several places in the paper)\n4. How do you ensure that $f$ actually is a diffeomorphism? Here I mainly very about $f$ self-intersecting. I can see how it is easy to ensure that $f$ is an immersion, but in my reading of the paper you seem to require that $f$ is an embedding. Did I get this part right?\n5. I did not understand Assumption 2 at all. Can you explain it to me?\n6. In Theorem 2 it is assumed that $f$ is injective, which seems like a rather strong assumption. Can this be loosened?\n7. In line 271 it is stated $Z$ is *typically* a subset of $CPE(Z)$. What is meant by \"typically\"?\n8. I didn't quite understand the motivation behind the Cartesian-product extension (CPE). According to the intuition of Fig. 3, the CPE makes an axis-aligned extension, but isn't the entire issue regarding identifiability that we cannot assign much meaning to the axes (the axes concern the parametrization and not the underlying support)? Then, why is it natural to extend along the axes? (To be clear, I do see the point of making an extension when studying extrapolation, so my question is more why the said extension should be axis-aligned).\n\nLIMITATIONS:\nThere is no need for a discussion regarding societal impact, etc., in a purely theoretical paper such as the present.\n\nI wish the paper had had a greater discussion regarding the many assumptions made throughout the paper. Such a discussion would be akin to a \"limitations\" section often found in more empirical/methodological papers, and I don't see why a theoretical paper should not openly be discussing the limitations of the analysis (i.e. if the assumptions are appropriate).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the identification theory for the additive mixing function. Specifically, they transfer the existing nonlinear ICA conditions from the distribution (i.e., sufficient variability) to the nonlinear mixing function (i.e., sufficient nonlinearity). Under this model, they make the connection to extrapolation for generative models. Synthetic data experiments are designed to demonstrate their arguments.\n\nSTRENGTHS:\n1. This paper is well written — both the assumptions and the implications are adequately discussed and thus easy to understand.\n2. The block-wise identification result is novel as a result of the sufficient nonlinearity condition inspired by the sufficient variability condition in prior work.\n3. The connection to the extrapolation is interesting and yields a valuable understanding of current large models.\n\nWEAKNESSES:\n1. Some key assumptions, although discussed, are still evasive in their restrictiveness. The most notable is the sufficient nonlinearity assumption, which appears very restrictive. How to enforce this for the estimation model is challenging.\n2. The primary assumption, namely additivity, can be very stringent. It is hard to believe this would hold for any realistic data-generating process. This also somehow trivializes the significance of the extrapolation part. I would also like to learn about the relation to recent work [1].\n3. The experimental results lack detailed explanation. I struggle to make sense of the visualizations: what are the colored shades mean in Figure 4, and what does the color mean for the dots? Are the generating processes identical in the additive and the non-additive cases, i.e., does the only difference lie in the estimation model?\n\n[1]. https://arxiv.org/abs/2305.14229\n\nQUESTIONS:\nI would like to learn about the authors' response to the weaknesses listed above, which may give me a clearer perspective on the paper's contribution.\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the recently popular approach of constraining the nonlinear function to achieve identifiable disentanglement. Here the idea is that $\\mathbf{f}$ is additive i.e. made of constituent functions that operate independently on non-overlapping partitions of the latent function. Identifiable disentanglement is achieved in this situation with very mild conditions on the latent distribution. These additive decoders can be seen as rudimentary version of the decoders used in object-centric representation learning and thus help explain their generalization performance. In particular, the paper shows that by exploring the full cartesian product of the latent symbols the model can generate images that are out of the training images' support (called cartesian-product extrapolation).\n\nSTRENGTHS:\n- very well written paper with clear and intuitive examples despite the technical topic\n- a novel way in which we can understand disentanglement by restricting the nonlinearity from the point of view of OCRL is a nice new angle to this increasingly popular field\n- by making assumption about the structure of $f$ the authors are able make very mild assumptions about the distribution of the latent factors which is in contrast to the much more distribution\n- additive decoders and the relevant results here provide a nice simple baseline model upon which future works can build more realistic ocrl models\n- extrapolation guarantees is a nice addition, something previous works have been missing, and is something that hopefully will be adopted by the community (though it's unclear how that could be done; see below)\n- assumption of sufficient nonlinearity is an insightful result and its connection to previous literature is nicely illustrated (albeit in the appendix)\n\nWEAKNESSES:\nSince the latent variables have only very mild restrictions, the price is paid by having fairly strong restriction on the 'mixing' function i.e. block-wise additivity (likely problematic in many realistic situations such as images with occlusion) + requirements on nonlinearity. This is likely useful for OCRL (e.g. scene mixtures) but in general may be very restrictive in the more general nonlinear ICA/disentanglement and also abstracts away from the desired goal of traditional nonlinear ICA where the aim is to separate out sources from 'heavily mixed' signals. \n\nFurthermore, the additivity also leads to a slightly problematic level identifiability results in the sense that each partition-block $z_b$ is identified only up to arbitrary invertible nonlinear transformation (equation 8. & 9. and $v_B$ specifically) -- this can be interpreted as there being a nonlinear ICA / mixing problem completely *unsolved* and thus unidentified for each block. So even though we are no longer left with the generic unidentifiability problem of $x = f(z)$, we are still left with complete unidentifiability in each block $B$, $ x_B = f_B (z_B)$. It feels like the term \"partition-respecting permutation\" and \"B-disentanglement\" are thus very specific to this approach and not really corresponding to the commonly accepted definitions of disentanglement in literature. Indeed the authors write \"Thus, B-disentanglement means that the blocks of latent dimensions zB are disentangled from one 177 another, but that variables within a given block might remain entangled.\"\n\nRelated to this, for OCRL the model is quite simple, as admitted by the authors, and while they indeed may provide a good baseline (as mentioned above) it is hard to be sure whether the theorems fully explain their performance especially given these concerns about block-wise unidentifiability.\n \nIt would have been nice to see more extensive experiments and especially on more complex data.\n\nQUESTIONS:\nDo you believe the block-wise unidentifiability/entanglement is undesirable? Or do you believe my concerns above are not a problem? I'm talking practically speaking. What do you think is the impact of this on CPE? What if the blocks could also be fully disentangled -- what would this change (e.g. in CPE?). Again I'm interested in your thoughts on practical implications rather than theory.\n\nYou say that: \"is “imitating” a block-specific ground-truth decoder\". Could you please explain what you exactly mean by imitation? I find that quite vague. \n\nYou write that \"We believe it illustrates the  fact that disentanglement alone is not sufficient to enable extrapolation and that one needs to restrict the hypothesis class of decoders in some way.\" These sound like very generic statements -- do you have confidence that they hold much beyond this setting?\n\n\"our in this work to understand “out-of-support” generation is a step towards understanding theoretically why modern generative models such as DALLE-2 [42] and 53 Stable Diffusion [43] can be creative\" Could you please explain this more specifically? Do you believe additiveness to be important to this?\n\nLIMITATIONS:\nThere is some good discussion of limitations in the paper e.g. \"additive decoders make intuitive sense for OCRL, they are not expressive enough to represent the “masked decoders” typically used in\", \"Additionally, this  parameter sharing across f (B) enables modern methods to have a variable number of objects across  samples, an important practical point our theory does not cover.\" Appendix also has nice examples of what happens when some assumptions are violated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by the problem of disentanglement in generative models, this paper proposes a novel decoder architecture, so-called additive decoders, based on the addition of block-wise latent variables, where blocks of latents correspond to semantic factors.\n\nAn extrapolation property of their decoder is demonstrated, which essentially consists of forming new products of latent blocks unseen at train time.\n\nA theoretical analysis of their decoder is carried out. Several new definitions are made, as well as few necessary assumptions. Two theorems are presented on \"local disentanglement\" and \"global disentanglement\".\n\nAn empirical investigation based on synthetic image data, consisting of two balls randomly placed, is carried out, showing a case where their decoder improves upon a baseline case.\n\nSTRENGTHS:\nThe paper addresses the problem of disentanglement in an interesting way: through blocks of latent variables which contribute additively to the overall decoding. This is a natural and reasonable idea.\n\nThe motivation for this paper is good: restrict the decoder to be additive to address the problem of latent variable identifiability.\n\nThe proof technique appears to build on the well-established results (Hyvärinen, AISTATS 2019).\n\nWEAKNESSES:\nThe paper is quite technical. This makes it challenging to ensure all technical details are correct. I did not find any errors.\n\nThe authors write that this work may help explain the \"creativity\" of mainstream generative models like DALLE-2 and Stale Diffusion. It is not so clear to me that their analysis will be helpful.\n\nRegarding Assumption 2, which the authors write is \"key\" for Theorem 2, I am not clear on how realistic this assumption is in practice. I understand that it is motivated by similar assumptions made in the ICA literature. However I come away with the feeling that it may not have much practical relevance. The authors give a toy numerical example in Example 3, which is helpful. But for instance, has this assumption been verified for the additive decoders used in the Experiments (Section 4)?\n\nThe proposed additive decoder cannot handle occlusions, as noted by the authors in Section 4, and discussed in the appendix. This is a limitation, since real images may have occlusions. \n\nOnly small-scale synthetic data are included. It doesn't appear to me that training on larger scale data is feasible, but more discussion of this would be helpful. It's reasonable to develop theory on simple cases, perhaps even necessary in this cases, but we should be clear whether scaling the results can be expected.\n\nQUESTIONS:\nCan the authors further justify their statement that this work \"has the potential of expanding our creativity in generative model\" (line 368).\n\nCan assumption 2 be verified for the decoders in used in the Experiments (Section 4)?\n\nIs training on larger-scale data possible with this method? If not, why not?\n\nLIMITATIONS:\nUnfortunately I did not find any discussion of limitations in the main text. \n\nI don't think this paper requires a discussion on societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}