SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Example 1. To illustrate  $\mathcal{B}$ -disentanglement, imagine a scene consisting of two balls moving around in  $2D$  where the "ground-truth" representation is given by  $\pmb{z} = (x^{1},y^{1},x^{2},y^{2})$  where  $\pmb{z}_{B_1} = (x^1,y^1)$  and  $\pmb{z}_{B_2} = (x^2,y^2)$  are the coordinates of each ball (here,  $\mathcal{B} := \{\{1,2\} ,\{3,4\}\}$ ). In that case, a learned representation is  $\mathcal{B}$ -disentangled when the balls are disentangled from one another. However, the basis in which the position of each ball is represented might differ in both representations.
Without loss of generality, we assume that the partition  $\mathcal{B}$  is contiguous, i.e. each  $B\in \mathcal{B}$  can be written as  $B = \{i + 1,i + 2,\dots ,i + |B|\}$ .
Our first result (Theorem 1) shows a weaker form of disentanglement we call local  $\mathcal{B}$ -disentanglement. This means the Jacobian matrix of  $\pmb{v}$ ,  $D\pmb{v}$ , has a "block-permutation" structure everywhere.
Definition 4 (Local  $\mathcal{B}$ -disentanglement). A learned decoder  $\hat{\pmb{f}}:\mathbb{R}^{d_z}\to \mathbb{R}^{d_x}$  is said to be locally  $\mathcal{B}$ -disentangled w.r.t. the ground-truth decoder  $\pmb{f}$  when  $\pmb {f}(\mathcal{Z}^{\mathrm{train}}) = \hat{\pmb{f}} (\hat{\mathcal{Z}}^{\mathrm{train}})$  and the mapping  $\pmb {v}\coloneqq \pmb{f}^{-1}\circ \hat{\pmb{f}}$  is a diffeomorphism from  $\hat{\mathcal{Z}}^{\mathrm{train}}$  to  $\mathcal{Z}^{\mathrm{train}}$  with a mapping  $\pmb {v}:\hat{\mathcal{Z}}^{\mathrm{train}}\rightarrow \mathcal{Z}^{\mathrm{train}}$  satisfying the following property: for all  $z\in \hat{\mathcal{Z}}^{\mathrm{train}}$ , there exists a permutation  $\pi$  respecting  $\mathcal{B}$  such that, for all  $B\in \mathcal{B}$ , the columns of  $D\pmb{v}_{\pi (B)}(\pmb {z})\in \mathbb{R}^{|B|\times d_z}$  outside block  $B$  are zero.
In Appendix A.4, we provide three examples where local disentanglement holds but not global disentanglement. The first one illustrates how having a disconnected support can allow for a permutation  $\pi$  (from Definition 4) that changes between disconnected regions of the support. The last two examples show how, even if the permutation stays the same throughout the support, we can still violate global disentanglement, even with a connected support.
We now state the main identifiability result of this work which provides conditions to guarantee local disentanglement. We will then see how to go from local to global disentanglement in the subsequent Theorem 2. For pedagogical reasons, we delay the formalization of the sufficient nonlinearity Assumption 2 on which the result crucially relies.
Theorem 1 (Local disentanglement via additive decoders). Suppose that the data-generating process satisfies Assumption 1, that the learned decoder  $\hat{\pmb{f}}:\mathbb{R}^{d_z}\to \mathbb{R}^{d_x}$  is a  $C^2$ -diffeomorphism, that the encoder  $\hat{\pmb{g}}:\mathbb{R}^{d_x}\to \mathbb{R}^{d_z}$  is continuous, that both  $\pmb{f}$  and  $\hat{\pmb{f}}$  are additive (Definition 1) and that  $\pmb{f}$  is sufficiently nonlinear as formalized by Assumption 2. Then, if  $\hat{\pmb{f}}$  and  $\hat{\pmb{g}}$  solve the reconstruction problem on the training distribution, i.e.  $\mathbb{E}^{\mathrm{train}}||\pmb {x} - \hat{\pmb{f}} (\hat{\pmb{g}} (\pmb {x}))||^2 = 0$ , we have that  $\hat{\pmb{f}}$  is locally  $\mathcal{B}$ -disentangled w.r.t.  $\pmb{f}$  (Definition 4).
The proof of Theorem 1, which can be found in Appendix A.5, is inspired from Hyvärinen et al. [33]. The essential differences are that (i) they leverage the additivity of the conditional log-density of  $z$  given an auxiliary variable  $u$  (i.e. conditional independence) instead of the additivity of the decoder function  $f$ , (ii) we extend their proof techniques to allow for "block" disentanglement, i.e. when  $\mathcal{B}$  is not the trivial partition  $\{\{1\}, \ldots, \{d_z\}\}$ , (iii) the assumption "sufficient variability" of the prior  $p(z \mid u)$  of Hyvärinen et al. [33] is replaced by an analogous assumption of "sufficient nonlinearity" of the decoder  $f$  (Assumption 2), and (iv) we consider much more general supports  $\mathcal{Z}^{\mathrm{train}}$  which makes the jump from local to global disentanglement less direct in our case.
The identifiability-expressivity trade-off. The level of granularity of the partition  $\mathcal{B}$  controls the trade-off between identifiability and expressivity: the finer the partition, the tighter the identifiability guarantee but the less expressive is the function class. The optimal level of granularity is going to depend on the application at hand. Whether  $\mathcal{B}$  could be learned from data is left for future work.
Sufficient nonlinearity. The following assumption is key in proving Theorem 2, as it requires that the ground-truth decoder is "sufficiently nonlinear". This is reminiscent of the "sufficient variability" assumptions found in the nonlinear ICA literature, which usually concerns the distribution of the latent variable  $z$  as opposed to the decoder  $f$  [30, 31, 33, 36, 37, 42, 73]. We clarify this link in Appendix A.6 and provide intuitions why sufficient nonlinearity can be satisfied when  $d_x \gg d_z$ .
Assumption 2 (Sufficient nonlinearity of  $\pmb{f}$ ). Let  $q := d_z + \sum_{B \in \mathcal{B}} \frac{|B|(|B| + 1)}{2}$ . For all  $\pmb{z} \in \mathcal{Z}^{\mathrm{train}}$ ,  $\pmb{f}$  is such that the following matrix has linearly independent columns (i.e. full column-rank):
$$
\boldsymbol {W} (\boldsymbol {z}) := \left[ \left[ D _ {i} \boldsymbol {f} ^ {(B)} (\boldsymbol {z} _ {B}) \right] _ {i \in B} \left[ D _ {i, i ^ {\prime}} ^ {2} \boldsymbol {f} ^ {(B)} (\boldsymbol {z} _ {B}) \right] _ {(i, i ^ {\prime}) \in B _ {\leq} ^ {2}} \right] _ {B \in \mathcal {B}} \in \mathbb {R} ^ {d _ {x} \times q}, \tag {6}
$$
where  $B_{\leq}^{2} \coloneqq B^{2} \cap \{(i, i') \mid i' \leq i\}$ . Note this implies  $d_{x} \geq q$ .
The following example shows that Theorem 1 does not apply if the ground-truth decoder  $\pmb{f}$  is linear. If that was the case, it would contradict the well known fact that linear ICA with independent Gaussian factors is unidentifiable.
Example 2 (Importance of Assumption 2). Suppose  $\pmb{x} = \pmb{f}(\pmb{z}) = \pmb{A}\pmb{z}$  where  $\pmb{A} \in \mathbb{R}^{d_x \times d_z}$  is full rank. Take  $\hat{\pmb{f}}(z) \coloneqq \pmb{A}\pmb{V}\pmb{z}$  and  $\hat{\pmb{g}}(\pmb{x}) \coloneqq \pmb{V}^{-1}\pmb{A}^{\dagger}\pmb{x}$  where  $\pmb{V} \in \mathbb{R}^{d_z \times d_z}$  is invertible and  $\pmb{A}^{\dagger}$  is the left pseudo inverse of  $\pmb{A}$ . By construction, we have that  $\mathbb{E}[\pmb{x} - \hat{\pmb{f}}(\hat{\pmb{g}}(\pmb{x}))] = 0$  and  $\pmb{f}$  and  $\hat{\pmb{f}}$  are
$\mathcal{B}$ -additive because  $\pmb{f}(\pmb{z}) = \sum_{B \in \mathcal{B}} \pmb{A}_{\cdot,B} \pmb{z}_B$  and  $\hat{\pmb{f}}(\pmb{z}) = \sum_{B \in \mathcal{B}} (\pmb{A} \pmb{V})_{\cdot,B} \pmb{z}_B$ . However, we still have that  $\pmb{v}(\pmb{z}) := \pmb{f}^{-1} \circ \hat{\pmb{f}}(\pmb{z}) = \pmb{V} \pmb{z}$  where  $\pmb{V}$  does not necessarily have a block-permutation structure, i.e. no disentanglement. The reason we cannot apply Theorem 1 here is because Assumption 2 is not satisfied. Indeed, the second derivatives of  $\pmb{f}^{(B)}(\pmb{z}_B) := \pmb{A}_{\cdot,B} \pmb{z}_B$  are all zero and hence  $\pmb{W}(\pmb{z})$  cannot have full column-rank.
Example 3 (A sufficiently nonlinear  $\pmb{f}$ ). In Appendix A.7 we show numerically that the function
$$
\boldsymbol {f} (\boldsymbol {z}) := \left[ \boldsymbol {z} _ {1}, \boldsymbol {z} _ {1} ^ {2}, \boldsymbol {z} _ {1} ^ {3}, \boldsymbol {z} _ {1} ^ {4} \right] ^ {\top} + \left[ \left(\boldsymbol {z} _ {2} + 1\right), \left(\boldsymbol {z} _ {2} + 1\right) ^ {2}, \left(\boldsymbol {z} _ {2} + 1\right) ^ {3}, \left(\boldsymbol {z} _ {2} + 1\right) ^ {4} \right] ^ {\top} \tag {7}
$$
is a diffeomorphism from the square  $[-1,0] \times [0,1]$  to its image that satisfies Assumption 2.
Example 4 (Smooth balls dataset is sufficiently nonlinear). In Appendix A.7 we present a simple synthetic dataset consisting of images of two colored balls moving up and down. We also verify numerically that its underlying ground-truth decoder  $\pmb{f}$  is sufficiently nonlinear.
3.1.1 From local to global disentanglement
The following result provides additional assumptions to guarantee global disentanglement (Definition 3) as opposed to only local disentanglement (Definition 4). See Appendix A.8 for its proof.
Theorem 2 (From local to global disentanglement). Suppose that all the assumptions of Theorem 1 hold. Additionally, assume  $\mathcal{Z}^{\mathrm{train}}$  is path-connected (Definition 8) and that the block-specific decoders  $\pmb{f}^{(B)}$  and  $\hat{\pmb{f}}^{(B)}$  are injective for all blocks  $B\in \mathcal{B}$ . Then, if  $\hat{\pmb{f}}$  and  $\hat{\pmb{g}}$  solve the reconstruction problem on the training distribution, i.e.  $\mathbb{E}^{\mathrm{train}}||\pmb {x} - \hat{\pmb{f}} (\hat{\pmb{g}} (\pmb {x}))||^2 = 0$ , we have that  $\hat{\pmb{f}}$  is (globally)  $\mathcal{B}$ -disentangled w.r.t.  $\pmb{f}$  (Definition 3) and, for all  $B\in \mathcal{B}$ ,
$$
\hat {\boldsymbol {f}} ^ {(B)} (\boldsymbol {z} _ {B}) = \boldsymbol {f} ^ {(\pi (B))} \left(\bar {\boldsymbol {v}} _ {\pi (B)} (\boldsymbol {z} _ {B})\right) + \boldsymbol {c} ^ {(B)}, f o r a l l \boldsymbol {z} _ {B} \in \hat {\mathcal {Z}} _ {B} ^ {\text {t r a i n}}, \tag {8}
$$
where the functions  $\bar{\pmb{v}}_{\pi (B)}$  are from Definition 3 and the vectors  $c^{(B)}\in \mathbb{R}^{d_x}$  are constants such that  $\sum_{B\in \mathcal{B}}c^{(B)} = 0$ . We also have that the functions  $\bar{\pmb{v}}_{\pi (B)}:\hat{\mathcal{Z}}_{B}^{\mathrm{train}}\to \mathcal{Z}_{\pi (B)}^{\mathrm{train}}$  are  $C^2$ -diffeomorphisms and have the following form:
$$
\bar {\boldsymbol {v}} _ {\pi (B)} \left(\boldsymbol {z} _ {B}\right) = \left(\boldsymbol {f} ^ {\pi (B)}\right) ^ {- 1} \left(\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) - \boldsymbol {c} ^ {(B)}\right), f o r a l l \boldsymbol {z} _ {B} \in \hat {\mathcal {Z}} _ {B} ^ {\text {t r a i n}}. \tag {9}
$$
Equation (8) in the above result shows that each block-specific learned decoder  $\hat{f}^{(B)}$  is "imitating" a block-specific ground-truth decoder  $f^{\pi(B)}$ . Indeed, the "object-specific" image outputted by the decoder  $\hat{f}^{(B)}$  evaluated at some  $z_B \in \hat{\mathcal{Z}}_B^{\mathrm{train}}$  is the same as the image outputted by  $f^{(B)}$  evaluated at  $v(z_B) \in \mathcal{Z}_B^{\mathrm{train}}$ , up to an additive constant vector  $c^{(B)}$ . These constants cancel each other out when taking the sum of the block-specific decoders.
Equation (9) provides an explicit form for the function  $\bar{v}_{\pi(B)}$ , which is essentially the learned block-specific decoder composed with the inverse of the ground-truth block-specific decoder.
Additional assumptions to go from local to global. Assuming that the support of  $\mathbb{P}_{\mathbf{z}}^{\mathrm{train}}$ ,  $\mathcal{Z}^{\mathrm{train}}$ , is path-connected (see Definition 8 in appendix) is useful since it prevents the permutation  $\pi$  of Definition 4 from changing between two disconnected regions of  $\hat{\mathcal{Z}}^{\mathrm{train}}$ . See Figure 2 for an illustration. In Appendix A.9, we discuss the additional assumption that each  $\pmb{f}^{(B)}$  must be injective and show that, in general, it is not equivalent to the assumption that  $\sum_{B \in \mathcal{B}} \pmb{f}^{(B)}$  is injective.
Figure 2: Illustrating regularly closed sets (Definition 6) and path-connected sets (Definition 8). Theorem 2 requires  $\mathcal{Z}^{\mathrm{train}}$  to satisfy both properties.
3.2 Cartesian-product extrapolation
In this section, we show how a learned additive decoder can be used to generate images  $\pmb{x}$  that are "out of support" in the sense that  $\pmb{x} \notin f(\mathcal{Z}^{\mathrm{train}})$ , but that are still on the manifold of "reasonable" images, i.e.  $\pmb{x} \in f(\mathcal{Z}^{\mathrm{test}})$ . To characterize the set of images the learned decoder can generate, we will rely on the notion of "cartesian-product extension", which we define next.
Definition 5 (Cartesian-product extension). Given a set  $\mathcal{Z} \subseteq \mathbb{R}^{d_z}$  and partition  $\mathcal{B}$  of  $[d_z]$ , we define the Cartesian-product extension of  $\mathcal{Z}$  as
$$
\mathrm {C P E} _ {\mathcal {B}} (\mathcal {Z}) := \prod_ {B \in \mathcal {B}} \mathcal {Z} _ {B}, \text {w h e r e} \mathcal {Z} _ {B} := \left\{\boldsymbol {z} _ {B} \mid \boldsymbol {z} \in \mathcal {Z} \right\}.
$$
It is indeed an extension of  $\mathcal{Z}$  since  $\mathcal{Z} \subseteq \prod_{B \in \mathcal{B}} \mathcal{Z}_B$ .
Figure 3: Illustration of Definition 5.
Let us define  $\bar{\pmb{v}}:\mathrm{CPE}_{\mathcal{B}}(\hat{\mathcal{Z}}^{\mathrm{train}})\to \mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}})$  to be the natural extension of the function  $\pmb{v}$ :  $\hat{\mathcal{Z}}^{\mathrm{train}}\rightarrow \mathcal{Z}^{\mathrm{train}}$ . More explicitly,  $\bar{\pmb{v}}$  is the "concatenation" of the functions  $\bar{\pmb{v}}_B$  given in Definition 3:
$$
\bar {\boldsymbol {v}} (\boldsymbol {z}) ^ {\top} := \left[ \bar {\boldsymbol {v}} _ {B _ {1}} \left(\boldsymbol {z} _ {\pi^ {- 1} \left(B _ {1}\right)}\right) ^ {\top} \dots \bar {\boldsymbol {v}} _ {B _ {\ell}} \left(\boldsymbol {z} _ {\pi^ {- 1} \left(B _ {\ell}\right)}\right) ^ {\top} \right], \tag {10}
$$
where  $\ell$  is the number of blocks in  $\mathcal{B}$ . This map is a diffeomorphism because each  $\bar{\boldsymbol{v}}_{\pi (B)}$  is a diffeomorphism from  $\hat{\mathcal{Z}}_B^{\mathrm{train}}$  to  $\mathcal{Z}_{\pi (B)}^{\mathrm{train}}$  by Theorem 2.
We already know that  $\hat{f} (z) = f\circ \bar{v} (z)$  for all  $z\in \hat{\mathcal{Z}}^{\mathrm{train}}$ . The following result shows that this equality holds in fact on the larger set  $\mathrm{CPE}_B(\hat{\mathcal{Z}}^{\mathrm{train}})$ , the Cartesian-product extension of  $\hat{\mathcal{Z}}^{\mathrm{train}}$ . See right of Figure 1 for an illustration of the following corollary.
Corollary 3 (Cartesian-product extrapolation). Suppose the assumptions of Theorem 2 hold. Then,
$$
f o r \quad z \in \mathrm {C P E} _ {\mathcal {B}} (\hat {\mathcal {Z}} ^ {\text {t r a i n}}), \sum_ {B \in \mathcal {B}} \hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = \sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(\pi (B))} \left(\bar {\boldsymbol {v}} _ {\pi (B)} \left(\boldsymbol {z} _ {B}\right)\right). \tag {11}
$$
Furthermore, if  $\mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}}) \subseteq \mathcal{Z}^{\mathrm{test}}$ , then  $\hat{\pmb{f}}(\mathrm{CPE}_{\mathcal{B}}(\hat{\mathcal{Z}}^{\mathrm{train}})) \subseteq \pmb{f}(\mathcal{Z}^{\mathrm{test}})$ .
Equation (11) tells us that the learned decoder  $\hat{f}$  "imitates" the ground-truth  $f$  not just over  $\hat{\mathcal{Z}}^{\mathrm{train}}$ , but also over its Cartesian-product extension. This is important since it guarantees that we can generate observations never seen during training as follows: Choose a latent vector  $z^{\mathrm{new}}$  that is in the Cartesian-product extension of  $\hat{\mathcal{Z}}^{\mathrm{train}}$ , but not in  $\hat{\mathcal{Z}}^{\mathrm{train}}$  itself, i.e.  $z^{\mathrm{new}} \in \mathrm{CPE}_{\mathcal{B}}(\hat{\mathcal{Z}}^{\mathrm{train}}) \setminus \hat{\mathcal{Z}}^{\mathrm{train}}$ . Then, evaluate the learned decoder on  $z^{\mathrm{new}}$  to get  $x^{\mathrm{new}} := \hat{f}(z^{\mathrm{new}})$ . By Corollary 3, we know that  $x^{\mathrm{new}} = f \circ \bar{v}(z^{\mathrm{new}})$ , i.e. it is the observation one would have obtained by evaluating the ground-truth decoder  $f$  on the point  $\bar{v}(z^{\mathrm{new}}) \in \mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}})$ . In addition, this  $x^{\mathrm{new}}$  has never been seen during training since  $\bar{v}(z^{\mathrm{new}}) \notin \bar{v}(\hat{\mathcal{Z}}^{\mathrm{train}}) = \mathcal{Z}^{\mathrm{train}}$ . The experiment of Figure 4 illustrates this procedure.
About the extra assumption “ $\mathbf{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}}) \subseteq \mathcal{Z}^{\mathrm{test}}$ ”. Recall that, in Assumption 1, we interpreted  $f(\mathcal{Z}^{\mathrm{test}})$  to be the set of “reasonable” observations  $x$ , of which we only observe a subset  $f(\mathcal{Z}^{\mathrm{train}})$ . Under this interpretation,  $\mathcal{Z}^{\mathrm{test}}$  is the set of reasonable values for the vector  $z$  and the additional assumption that  $\mathbf{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}}) \subseteq \mathcal{Z}^{\mathrm{test}}$  in Corollary 3 requires that the Cartesian-product extension of  $\mathcal{Z}^{\mathrm{train}}$  consists only of reasonable values of  $z$ . From this assumption, we can easily conclude that  $\hat{f}(\mathbf{CPE}_{\mathcal{B}}(\hat{\mathcal{Z}}^{\mathrm{train}})) \subseteq f(\mathcal{Z}^{\mathrm{test}})$ , which can be interpreted as: “The novel observations  $\pmb{x}^{\mathrm{new}}$  obtained via Cartesian-product extrapolation are reasonable”. Appendix A.11 describes an example where the assumption is violated, i.e.  $\mathbf{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}}) \nsubseteq \mathcal{Z}^{\mathrm{test}}$ . The practical implication of this is that the new observations  $\pmb{x}^{\mathrm{new}}$  obtained via Cartesian-product extrapolation might not always be reasonable.
Disentanglement is not enough for extrapolation. To the best of our knowledge, Corollary 3 is the first result that formalizes how disentanglement can induce extrapolation. We believe it illustrates the fact that disentanglement alone is not sufficient to enable extrapolation and that one needs to restrict the hypothesis class of decoders in some way. Indeed, given a learned decoder  $\hat{\pmb{f}}$  that is disentangled w.r.t.  $\pmb{f}$  on the training support  $\mathcal{Z}^{\mathrm{train}}$ , one cannot guarantee both decoders will "agree" outside the training domain without further restricting  $\hat{\pmb{f}}$  and  $\pmb{f}$ . This work has focused on "additivity", but we believe other types of restriction could correspond to other types of extrapolation.
We now present empirical validations of the theoretical results presented earlier. To achieve this, we compare the ability of additive and non-additive decoders to both identify ground-truth latent factors (Theorems 1 & 2) and extrapolate (Corollary 3) when trained to solve the reconstruction task on simple images  $(64 \times 64 \times 3)$  consisting of two balls moving in space [2]. See Appendix B.1
ScalarLatents	BlockLatents (independent z)	BlockLatents (dependent z)
Decoders	RMSE	LMSpear	RMSEOOS	LMSOOSSpear	RMSE	LMSTree	RMSE	LMSTree
Non-add.	.06 ± .002	70.6±5.21	.18±.012	73.7±4.64	.02±.001	53.9±7.58	.02±.001	78.1±2.92
Additive	.06±.002	91.5±3.57	.11±.018	89.5±5.02	.03±.012	92.2±4.91	.01±.002	99.9±0.02
Table 1: Reporting reconstruction mean squared error (RMSE ↓) and the Latent Matching Score (LMS ↑) for the three datasets considered: ScalarLatents and BlockLatents with independent and dependent latents. Runs were repeated with 10 random initializations. RMSE $^{\text{OOS}}$  and LMS $^{\text{OOS}}$  are the same metric but evaluated out of support (see Appendix B.3 for details). While the standard error is high, the differences are still clear as can be seen in their box plot version in Appendix B.4.
for training details. We consider two datasets: one where the two ball positions can only vary along the  $y$ -axis (ScalarLatents) and one where the positions can vary along both the  $x$  and  $y$  axes (BlockLatents).
ScalarLatents: The ground-truth latent vector  $\mathbf{z} \in \mathbb{R}^2$  is such that  $\mathbf{z}_1$  and  $\mathbf{z}_2$  correspond to the height (y-coordinate) of the first and second ball, respectively. Thus the partition is simply  $\mathcal{B} = \{\{1\}, \{2\}\}$  (each object has only one latent factor). This simple setting is interesting to study since the low dimensionality of the latent space  $(d_z = 2)$  allows for exhaustive visualizations like Figure 4. To study Cartesian-product extrapolation (Corollary 3), we sample  $\mathbf{z}$  from a distribution with a L-shaped support given by  $\mathcal{Z}^{\mathrm{train}} := [0,1] \times [0,1] \setminus [0.5,1] \times [0.5,1]$ , so that the training set does not contain images where both balls appear in the upper half of the image (see Appendix B.2).
BlockLatents: The ground-truth latent vector  $\mathbf{z} \in \mathbb{R}^4$  is such that  $\mathbf{z}_{\{1,2\}}$  and  $\mathbf{z}_{\{3,4\}}$  correspond to the  $x, y$  position of the first and second ball, respectively (the partition is simply  $\mathcal{B} = \{\{1,2\}, \{3,4\}\}$ , i.e. each object has two latent factors). Thus, this more challenging setting illustrates "block-disentanglement". The latent  $\mathbf{z}$  is sampled uniformly from the hypercube  $[0,1]^4$  but the images presenting occlusion (when a ball is behind another) are rejected from the dataset. We discuss how additive decoders cannot model images presenting occlusion in Appendix A.12. We also present an additional version of this dataset where we sample from the hypercube  $[0,1]^4$  with dependencies. See Appendix B.2 for more details about data generation.
Evaluation metrics: To evaluate disentanglement, we compute a matrix of scores  $(s_{B,B^{\prime}})\in \mathbb{R}^{\ell \times \ell}$  where  $\ell$  is the number of blocks in  $\mathcal{B}$  and  $s_{B,B^{\prime}}$  is a score measuring how well we can predict the ground-truth block  $z_{B}$  from the learned latent block  $\hat{z}_{B^{\prime}} = \hat{g}_{B^{\prime}}(\pmb {x})$  outputted by the encoder. The final Latent Matching Score (LMS) is computed as LMS  $=$  arg max  $\pi \in \mathfrak{S}_B\frac{1}{\ell}\sum_{B\in \mathcal{B}}s_{B,\pi (B)}$  ,where  $\mathfrak{S}_{\mathcal{B}}$  is the set of permutations respecting  $\mathcal{B}$  (Definition 2). When  $\mathcal{B}\coloneqq \{\{1\} ,\dots ,\{d_z\} \}$  and the score used is the absolute value of the correlation, LMS is simply the mean correlation coefficient (MCC), which is widely used in the nonlinear ICA literature [30, 31, 33, 36, 42]. Because our theory guarantees recovery of the latents only up to invertible and potentially nonlinear transformations, we use the Spearman correlation, which can capture nonlinear relationships unlike the Pearson correlation. We denote this score by  $\mathrm{LMS}_{\mathrm{Spear}}$  and will use it in the dataset ScalarLatents. For the BlockLatents dataset, we cannot use Spearman correlation (because  $z_{B}$  are two dimensional). Instead, we take the score  $s_{B,B^{\prime}}$  to be the  $R^2$  score of a regression tree. We denote this score by  $\mathrm{LMS}_{\mathrm{tree}}$  . There are subtleties to take care of when one wants to evaluate  $\mathrm{LMS}_{\mathrm{tree}}$  on a non-additive model due to the fact that the learned representation does not have a natural partition  $\mathcal{B}$  .We must thus search over partitions. We discuss this and provide further details on the metrics in Appendix B.3.
4.1 Results
Additivity is important for disentanglement. Table 1 shows that the additive decoder obtains a much higher  $\mathrm{LMS}_{\mathrm{Spear}}$  &  $\mathrm{LMS}_{\mathrm{Tree}}$  than its non-additive counterpart on all three datasets considered, even if both decoders have very small reconstruction errors. This is corroborated by the visualizations of Figures 4 & 5. Appendix B.5 additionally shows object-specific reconstructions for the BlockLatents dataset. We emphasize that disentanglement is possible even when the latent factors are dependent (or causally related), as shown on the ScalarLatents dataset (L-shaped support implies dependencies) and on the BlockLatents dataset with dependencies (Table 1). Note that prior works have relied on interventions [3, 2, 8] or Cartesian-product supports [68, 62] to deal with dependencies.
(a) Additive decoder
(b) Non-additive decoder