SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\boldsymbol {f} ^ {\pi (B)}\right) ^ {- 1} \left(\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) - \boldsymbol {c} ^ {(B)}\right) = \boldsymbol {v} _ {\pi (B)} (\boldsymbol {z}) \text {f o r a l l} \boldsymbol {z} \in \mathcal {Z} ^ {\text {t r a i n}}. \tag {103}
$$
We define the map  $\bar{\pmb{v}}_{\pi (B)}(\pmb {z}_B)\coloneqq (\pmb {f}^{\pi (B)})^{-1}(\hat{\pmb{f}}^{(B)}(\pmb {z}_B) - \pmb {c}^{(B)})$  which is from  $\hat{\mathcal{Z}}_B^{\mathrm{train}}$  to  $\mathcal{Z}_{\pi (B)}^{\mathrm{train}}$ . This allows us to rewrite (98) as
$$
\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = \boldsymbol {f} ^ {(\pi (B))} \circ \bar {\boldsymbol {v}} _ {\pi (B)} \left(\boldsymbol {z} _ {B}\right) + \boldsymbol {c} ^ {(B)}, \text {f o r a l l} \boldsymbol {z} _ {B} \in \mathcal {Z} _ {B} ^ {\text {t r a i n}}. \tag {104}
$$
Because  $\hat{\pmb{f}}^{(B)}$  is also injective, we must have that  $\bar{\pmb{v}}_{\pi (B)}:\hat{\mathcal{Z}}_B^{\mathrm{train}}\to \mathcal{Z}_{\pi (B)}^{\mathrm{train}}$  is injective as well.
We now show that  $\bar{\pmb{v}}_{\pi (B)}$  is surjective. Choose some  $z_{\pi (B)}\in \mathcal{Z}_{\pi (B)}^{\mathrm{train}}$ . We can always find  $z_{\pi (B)^c}$  such that  $(z_{\pi (B)},z_{\pi (B)^c})\in \mathcal{Z}^{\mathrm{train}}$ . Because  $\pmb {v}:\hat{\mathcal{Z}}^{\mathrm{train}}\to \mathcal{Z}^{\mathrm{train}}$  is surjective (it is a diffeomorphism), there exists a  $z^0\in \hat{\mathcal{Z}}^{\mathrm{train}}$  such that  $\pmb {v}(\pmb {z}^{0}) = (\pmb {z}_{\pi (B)},\pmb {z}_{\pi (B)^c})$ . By (103), we have that
$$
\bar {\boldsymbol {v}} _ {\pi (B)} \left(\boldsymbol {z} _ {B} ^ {0}\right) = \boldsymbol {v} _ {\pi (B)} \left(\boldsymbol {z} ^ {0}\right). \tag {105}
$$
which means  $\bar{\pmb{v}}_{\pi (B)}(\pmb {z}_B^0) = \pmb {z}_{\pi (B)}$
We thus have that  $\bar{\pmb{v}}_{\pi (B)}$  is bijective. It is a diffeomorphism because
$$
\det  D \bar {\boldsymbol {v}} _ {\pi (B)} (\boldsymbol {z} _ {B}) = \det  D _ {B} \boldsymbol {v} _ {\pi (B)} (\boldsymbol {z}) \neq 0 \forall \boldsymbol {z} \in \hat {\mathcal {Z}} ^ {\text {t r a i n}} \tag {106}
$$
where the first equality holds by (103) and the second holds because  $\pmb{v}$  is a diffeomorphism and has block-permutation structure, which means it has a nonzero determinant everywhere on  $\hat{\mathcal{Z}}^{\mathrm{train}}$  and is equal to the product of the determinants of its blocks, which implies each block  $D_B\pmb{v}_{\pi(B)}$  must have nonzero determinant everywhere.
Since  $\bar{\pmb{v}}_{\pi (B)}:\hat{\mathcal{Z}}_{B}^{\mathrm{train}}\to \mathcal{Z}_{\pi (B)}^{\mathrm{train}}$  bijective and has invertible Jacobian everywhere, it must be a diffeomorphism.
A.9 Injectivity of object-specific decoders v.s. injectivity of their sum
We want to explore the relationship between the injectivity of individual object-specific decoders  $\pmb{f}^{(B)}$  and the injectivity of their sum, i.e.  $\sum_{B\in \mathcal{B}}\pmb{f}^{(B)}$ .
We first show the simple fact that having each  $\pmb{f}^{(B)}$  injective is not sufficient to have  $\sum_{B \in \mathcal{B}} \pmb{f}^{(B)}$  injective. Take  $\pmb{f}^{(B)}(\pmb{z}_B) = \pmb{W}^{(B)}\pmb{z}_B$  where  $\pmb{W}^{(B)} \in \mathbb{R}^{d_x \times |B|}$  has full column-rank for all  $B \in \mathcal{B}$ . We have that
$$
\sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = \sum_ {B \in \mathcal {B}} \boldsymbol {W} ^ {(B)} \boldsymbol {z} _ {B} = \left[ \boldsymbol {W} ^ {\left(B _ {1}\right)} \dots \boldsymbol {W} ^ {\left(B _ {\ell}\right)} \right] \boldsymbol {z}, \tag {107}
$$
where it is clear that the matrix  $\left[\boldsymbol{W}^{(B_1)}\dots \boldsymbol{W}^{(B_\ell)}\right]\in \mathbb{R}^{d_x\times d_z}$  is not necessarily injective even if each  $\boldsymbol{W}^{(B)}$  is. This is the case, for instance, if all  $\boldsymbol{W}^{(B)}$  have the same image.
We now provide conditions such that  $\sum_{B\in \mathcal{B}}\pmb{f}^{(B)}$  injective implies each  $\pmb{f}^{(B)}$  injective. We start with a simple lemma:
Lemma 12. If  $g \circ h$  is injective, then  $h$  is injective.
Proof. By contradiction, assume that  $h$  is not injective. Then, there exists distinct  $x_1, x_2 \in \operatorname{Dom}(h)$  such that  $h(x_1) = h(x_2)$ . This implies  $g \circ h(x_1) = g \circ h(x_2)$ , which violates injectivity of  $g \circ h$ .
The following Lemma provides a condition on the domain of the function  $\sum_{B\in \mathcal{B}}f^{(B)},\mathcal{Z}^{\mathrm{train}}$ , so that its injectivity implies injectivity of the functions  $f^{(B)}$ .
Lemma 13. Assume that, for all  $B \in \mathcal{B}$  and for all distinct  $\pmb{z}_B, \pmb{z}_B' \in \mathcal{Z}_B^{\mathrm{train}}$ , there exists  $\pmb{z}_{B^c}$  such that  $(\pmb{z}_B, \pmb{z}_{B^c}), (\pmb{z}_B', \pmb{z}_{B^c}) \in \mathcal{Z}^{\mathrm{train}}$ . Then, whenever  $\sum_{B \in \mathcal{B}} \pmb{f}^{(B)}$  is injective, each  $\pmb{f}^{(B)}$  must be injective.
Proof. Notice that  $\pmb{f}(\pmb{z}) \coloneqq \sum_{B \in \mathcal{B}} \pmb{f}^{(B)}(\pmb{z}_B)$  can be written as  $\pmb{f} \coloneqq \mathrm{SumBlocks} \circ \bar{\pmb{f}}(\pmb{z})$  where
$$
\bar {\boldsymbol {f}} (\boldsymbol {z}) := \left[ \begin{array}{c} \boldsymbol {f} ^ {(B _ {1})} \left(\boldsymbol {z} _ {B _ {1}}\right) \\ \vdots \\ \boldsymbol {f} ^ {(B _ {\ell})} \left(\boldsymbol {z} _ {B _ {\ell}}\right) \end{array} \right], \text {a n d S u m B l o c k s} \left(\boldsymbol {x} ^ {(B _ {1})}, \dots , \boldsymbol {x} ^ {(B _ {\ell})}\right) := \sum_ {B \in \mathcal {B}} \boldsymbol {x} ^ {(B)} \tag {108}
$$
Since  $f$  is injective, by Lemma 12  $\bar{f}$  must be injective.
We now show that each  $\pmb{f}^{(B)}$  must also be injective. Take  $\pmb{z}_B, \pmb{z}_B' \in \mathcal{Z}_B^{\mathrm{train}}$  such that  $\pmb{f}^{(B)}(\pmb{z}_B) = \pmb{f}^{(B)}(\pmb{z}_B')$ . By assumption, we know there exists a  $\pmb{z}_{B^c}$  s.t.  $(\pmb{z}_B, \pmb{z}_{B^c})$  and  $(\pmb{z}_B', \pmb{z}_{B^c})$  are in  $\mathcal{Z}^{\mathrm{train}}$ . By construction, we have that  $\bar{\pmb{f}}((\pmb{z}_B, \pmb{z}_{B^c})) = \bar{\pmb{f}}((\pmb{z}_B', \pmb{z}_{B^c}))$ . By injectivity of  $\bar{\pmb{f}}$ , we have that  $(\pmb{z}_B, \pmb{z}_{B^c}) \neq (\pmb{z}_B', \pmb{z}_{B^c})$ , which implies  $\pmb{z}_B \neq \pmb{z}_B'$ , i.e.  $\pmb{f}^{(B)}$  is injective.
A.10 Proof of Corollary 3
Proof. Pick  $z \in \mathrm{CPE}(\hat{\mathcal{Z}}^{\mathrm{train}})$ . By definition, this means that, for all  $B \in \mathcal{B}$ ,  $z_B \in \hat{\mathcal{Z}}_B^{\mathrm{train}}$ . We thus have that, for all  $B \in \mathcal{B}$ ,
$$
\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = \boldsymbol {f} ^ {(\pi (B))} \circ \bar {\boldsymbol {v}} _ {\pi (B)} \left(\boldsymbol {z} _ {B}\right) + \boldsymbol {c} ^ {(B)}. \tag {109}
$$
We can thus sum over  $B$  to obtain
$$
\sum_ {B \in \mathcal {B}} \hat {\boldsymbol {f}} ^ {(B)} (\boldsymbol {z} _ {B}) = \sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(\pi (B))} \circ \bar {\boldsymbol {v}} _ {\pi (B)} (\boldsymbol {z} _ {B}) + \underbrace {\sum_ {B \in \mathcal {B}} \boldsymbol {c} ^ {(B)}} _ {= 0}. \tag {110}
$$
Since  $z \in \mathrm{CPE}(\hat{\mathcal{Z}}^{\mathrm{train}})$  was arbitrary, we have
$$
\text {f o r a l l} \boldsymbol {z} \in \operatorname {C P E} \left(\hat {\mathcal {Z}} ^ {\text {t r a i n}}\right), \sum_ {B \in \mathcal {B}} \hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = \sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(\pi (B))} \circ \bar {\boldsymbol {v}} _ {\pi (B)} \left(\boldsymbol {z} _ {B}\right) \tag {111}
$$
$$
\hat {\boldsymbol {f}} (\boldsymbol {z}) = \boldsymbol {f} \circ \bar {\boldsymbol {v}} (\boldsymbol {z}), \tag {112}
$$
where  $\bar{v}:\mathrm{CPE}_{\mathcal{B}}(\hat{\mathcal{Z}}^{\mathrm{train}})\to \mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}})$  is defined as
$$
\bar {\boldsymbol {v}} (\boldsymbol {z}) := \left[ \begin{array}{c} \bar {\boldsymbol {v}} _ {B _ {1}} \left(\boldsymbol {z} _ {\pi^ {- 1} \left(B _ {1}\right)}\right) \\ \vdots \\ \bar {\boldsymbol {v}} _ {B _ {\ell}} \left(\boldsymbol {z} _ {\pi^ {- 1} \left(B _ {\ell}\right)}\right) \end{array} \right], \tag {113}
$$
The map  $\bar{v}$  is a diffeomorphism since each  $\bar{v}_{\pi (B)}$  is a diffeomorphism from  $\hat{\mathcal{Z}}_B^{\mathrm{train}}$  to  $\mathcal{Z}_{\pi (B)}^{\mathrm{train}}$ .
By (112) we get
$$
\hat {\boldsymbol {f}} \left(\mathrm {C P E} _ {\mathcal {B}} \left(\hat {\mathcal {Z}} ^ {\text {t r a i n}}\right)\right) = \boldsymbol {f} \circ \bar {\boldsymbol {v}} \left(\mathrm {C P E} _ {\mathcal {B}} \left(\hat {\mathcal {Z}} ^ {\text {t r a i n}}\right)\right), \tag {114}
$$
and since the map  $\bar{v}$  is surjective we have  $\bar{v} (\mathrm{CPE}_B(\hat{\mathcal{Z}}^{\mathrm{train}})) = \mathrm{CPE}_B(\mathcal{Z}^{\mathrm{train}})$  and thus
$$
\hat {\boldsymbol {f}} \left(\mathrm {C P E} _ {\mathcal {B}} \left(\hat {\mathcal {Z}} ^ {\text {t r a i n}}\right)\right) = \boldsymbol {f} \left(\mathrm {C P E} _ {\mathcal {B}} \left(\mathcal {Z} ^ {\text {t r a i n}}\right)\right). \tag {115}
$$
Hence if  $\mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}}) \subseteq \mathcal{Z}^{\mathrm{test}}$ , then  $\pmb{f}(\mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}})) \subseteq \pmb{f}(\mathcal{Z}^{\mathrm{test}})$ .
A.11 Will all extrapolated images make sense?
Here is a minimal example where the assumption  $\mathrm{CPE}_B(\mathcal{Z}^{\mathrm{train}}) \nsubseteq \mathcal{Z}^{\mathrm{test}}$  is violated.
Example 10 (Violation of  $\mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}}) \not\subseteq \mathcal{Z}^{\mathrm{test}}$ ). Imagine  $z = (z_{1}, z_{2})$  where  $z_{1}$  and  $z_{2}$  are the  $x$ -positions of two distinct balls. It does not make sense to have two balls occupying the same location in space and thus whenever  $z_{1} = z_{2}$  we have  $(z_{1}, z_{2}) \notin \mathcal{Z}^{\mathrm{test}}$ . But if  $(1, 2)$  and  $(2, 1)$  are both in  $\mathcal{Z}^{\mathrm{train}}$ , it implies that  $(1, 1)$  and  $(2, 2)$  are in  $\mathrm{CPE}(\mathcal{Z}^{\mathrm{train}})$ , which is a violation of  $\mathrm{CPE}_{\mathcal{B}}(\mathcal{Z}^{\mathrm{train}}) \subseteq \mathcal{Z}^{\mathrm{test}}$ .
A.12 Additive decoders cannot model occlusion
We now explain why additive decoders cannot model occlusion. Occlusion occurs when an object is partially hidden behind another one. Intuitively, the issue is the following: Consider two images consisting of two objects, A and B (each image shows both objects). In both images, the position of object A is the same and in exactly one of the images, object B partially occludes object A. Since the position of object  $A$  did not change, its corresponding latent block  $z_{A}$  is also unchanged between both images. However, the pixels occupied by object A do change between both images because of occlusion. The issue is that, because of additivity,  $z_{A}$  and  $z_{B}$  cannot interact to make some pixels that belonged to object A "disappear" to be replaced by pixels of object B. In practice, object-centric representation learning methods rely a masking mechanism which allows interactions between  $z_{A}$  and  $z_{B}$  (See Equation 1 in Section 2). This highlights the importance of studying this class of decoders in future work.
B Experiments
B.1 Training Details
Loss Function. We use the standard reconstruction objective of mean squared error loss between the ground truth data and the reconstructed/generated data.
Hyperparameters. For both the ScalarLatents and the BlockLatents dataset, we used the Adam optimizer with the hyperparameters defined below. Note that we maintain consistent hyperparameters across both the Additive decoder and the Non-Additive decoder method.
ScalarLatents Dataset.
- Batch Size: 64
- Learning Rate:  $1 \times 10^{-3}$
Weight Decay:  $5 \times 10^{-4}$
Total Epochs: 4000
BlockLatents Dataset.
- Batch Size: 1024
- Learning Rate:  $1 \times 10^{-3}$
Weight Decay:  $5 \times 10^{-4}$
Total Epochs: 6000
Model Architecture. We use the following architectures for Encoder and Decoder across both the datasets (ScalarLatents, BlockLatents). Note that for the ScalarLatents dataset we train with latent dimension  $d_{z} = 2$ , and for the BlockLatents dataset we train with latent dimension  $d_{z} = 4$ , which corresponds to the dimensionalities of the ground-truth data generating process for both datasets.
Encoder Architecture:
- RestNet-18 Architecture till the penultimate layer (512 dimensional feature output)
- Stack of 5 fully-connected layer blocks, with each block consisting of Linear Layer (dimensions:  $512 \times 512$ ), Batch Normalization layer, and Leaky ReLU activation (negative slope: 0.01).
- Final Linear Layer (dimension:  $512 \times d_z$ ) followed by Batch Normalization Layer to output the latent representation.
Decoder Architecture (Non-additive):
- Fully connected layer block with input as latent representation, consisting of Linear Layer (dimension:  $d_z \times 512$ ), Batch Normalization layer, and Leaky ReLU activation (negative slope: 0.01).
- Stack of 5 fully-connected layer blocks, with each block consisting of Linear Layer (dimensions:  $512 \times 512$ ), Batch Normalization layer, and Leaky ReLU activation (negative slope: 0.01).
- Series of DeConvolutional layers, where each DeConvolutional layer is followed by Leaky ReLU (negative slope: 0.01) activation.
- DeConvolution Layer ( $c_{in}$ : 64,  $c_{out}$ : 64, kernel: 4; stride: 2; padding: 1)
- DeConvolution Layer ( $c_{in}$ : 64,  $c_{out}$ : 32, kernel: 4; stride: 2; padding: 1)
- DeConvolution Layer ( $c_{in}$ : 32,  $c_{out}$ : 32, kernel: 4; stride: 2; padding: 1)
- DeConvolution Layer ( $c_{in}$ : 32,  $c_{out}$ : 3, kernel: 4; stride: 2; padding: 1)
Decoder Architecture (Additive): Recall that an additive decoder has the form  $\pmb{f}(\pmb{z}) = \sum_{B \in \mathcal{B}} \pmb{f}^{(B)}(\pmb{z}_B)$ . Each  $\pmb{f}^{(B)}$  has the same architecture as the one presented above for the non-additive case, but the input has dimensionality  $|B|$  (which is 1 or 2, depending on the dataset). Note that we do not share parameters among the functions  $\pmb{f}^{(B)}$ .
B.2 Datasets Details
We use the moving balls environment from Ahuja et al. [2] with images of dimension  $64 \times 64 \times 3$ , with latent vector  $(z)$  representing the position coordinates of each balls. We consider only two balls. The rendered images have pixels in the range [0, 255].
ScalarLatents Dataset. We fix the x-coordinate of each ball to 0.25 and 0.75. The only factors varying are the y-coordinates of both balls. Thus,  $z \in \mathbb{R}^2$  and  $\mathcal{B} = \{\{1\}, \{2\}\}$  where  $z_1$  and  $z_2$  designate the y-coordinates of both balls. We sample the y-coordinate of the first ball from a continuous uniform distribution as follows:  $z_1 \sim \mathrm{Uniform}(0, 1)$ . Then we sample the y-coordinate of the second ball as per the following scheme:
$$
z _ {2} \sim \left\{ \begin{array}{l l} \text {U n i f o r m} (0, 1) & \text {i f} z _ {1} \leq 0. 5 \\ \text {U n i f o r m} (0, 0. 5) & \text {e l s e} \end{array} \right.
$$
Hence, this leads to the L-shaped latent support, i.e.,  $\mathcal{Z}^{\mathrm{train}}\coloneqq [0,1]\times [0,1]\setminus [0.5,1]\times [0.5,1]$
We use  $50k$  samples for the test dataset, while we use  $20k$  samples for the train dataset along with  $5k$  samples (25% of the train sample size) for the validation dataset.
BlockLatents Dataset. For this dataset, we allow the balls to move in both the x, y directions, so that  $\mathbf{z} \in \mathbb{R}^4$  and  $\mathcal{B} = \{\{1,2\}, \{3,4\}\}$ . For the case of independent latents, we sample each latent component independently and identically distributed according to a uniform distribution over  $(0,1)$ , i.e.  $z_{i} \sim \mathrm{Uniform}(0,1)$ . We rejected the images that present occlusion, i.e. when one ball hides another one.
For the case of dependent latents, we sample the latents corresponding to the first ball similarly from the same continuous uniform distribution, i.e.,  $z_{1}, z_{2} \sim \mathrm{Uniform}(0, 1)$ . However, the latents of the second ball are a function of the latents of the first ball, as described in what follows:
$$
z _ {3} \sim \left\{ \begin{array}{l l} \text {U n i f o r m} (0, 0. 5) & \text {i f} 1. 2 5 \times (z _ {1} ^ {2} + z _ {2} ^ {2}) \geq 1. 0 \\ \text {U n i f o r m} (0. 5, 1) & \text {i f} 1. 2 5 \times (z _ {1} ^ {2} + z _ {2} ^ {2}) <   1. 0 \end{array} \right.
$$
<sup>2</sup>Note that, in the independent latents case, the latents are not actually independent because of the rejection step which prevents occlusion from happening.
$$
z _ {4} \sim \left\{ \begin{array}{l l} \text {U n i f o r m (0 . 5 , 1)} & \text {i f 1 . 2 5 \times (z _ {1} ^ {2} + z _ {2} ^ {2}) \geq 1 . 0} \\ \text {U n i f o r m (0 , 0 . 5)} & \text {i f 1 . 2 5 \times (z _ {1} ^ {2} + z _ {2} ^ {2}) <   1 . 0} \end{array} \right.
$$
Intuitively, this means the second ball will be placed in either the top-left or the bottom-right quadrant based on the position of the first ball. We also exclude from the dataset the images presenting occlusion.
Note that our dependent BlockLatent setup is same as the non-linear SCM case from Ahuja et al. [3].
We use  $50k$  samples for both the train and the test dataset, along with  $12.5k$  samples (25% of the train sample size) for the validation dataset.
Disconnected Support Dataset. For this dataset, we have setup similar to the ScalarLatents dataset; we fix the x-coordinates of both balls to 0.25 and 0.75 and only vary the y-coordinates so that  $\pmb{z} \in \mathbb{R}^2$ . We sample the y-coordinate of the first ball  $(z_1)$  from Uniform(0, 1). Then we sample the y-coordinate of the second ball  $(z_2)$  from either of the following continuous uniform distribution with equal probability; Uniform(0, 0.25) and Uniform(0.75, 1). This leads to a disconnected support given by  $\mathcal{Z}^{\mathrm{train}} := [0, 1] \times [0, 1] \setminus [0.25, 0.75] \times [0.25, 0.75]$ .
We use  $50k$  samples for the test dataset, while we use  $20k$  samples for the train dataset along with  $5k$  samples ( $25\%$  of the train sample size) for the validation dataset.
B.3 Evaluation Metrics
Recall that, to evaluate disentanglement, we compute a matrix of scores  $(s_{B,B^{\prime}})\in \mathbb{R}^{\ell \times \ell}$  where  $\ell$  is the number of blocks in  $\mathcal{B}$  and  $s_{B,B^{\prime}}$  is a score measuring how well we can predict the ground-truth block  $\pmb{z}_B$  from the learned latent block  $\hat{\pmb{z}}_{B^{\prime}} = \hat{\pmb{g}}_{B^{\prime}}(\pmb {x})$  outputted by the encoder. The final Latent Matching Score (LMS) is computed as LMS  $= \arg \max_{\pi \in \mathfrak{S}_{\mathcal{B}}}\frac{1}{\ell}\sum_{B\in \mathcal{B}}s_{B,\pi (B)}$  , where  $\mathfrak{S}_{\mathcal{B}}$  is the set of permutations respecting  $\mathcal{B}$  (Definition 2). These scores are always computed on the test set.
Metric LMSpear: As mentioned in the main paper, this metric is used for the ScalarLatents dataset where each block is 1-dimensional. Hence, this metric is almost the same as the mean correlation coefficient (MCC), which is widely used in the nonlinear ICA literature [30, 31, 33, 36, 42], with the only difference that we use Spearman correlation instead of Pearson correlation as a score  $s_{B,B'}$ . The Spearman correlation can capture nonlinear monotonous relations, unlike Pearson which can only capture linear dependencies. We favor Spearman over Pearson because our identifiability result (Theorem 2) guarantees we can recover the latents only up to permutation and element-wise invertible transformations, which can be nonlinear.
Metric  $\mathbf{LMS}_{\mathrm{tree}}$ : This metric is used for the BlockLatents dataset. For this metric, we take  $s_{B,B'}$  to be the  $R^2$  score of a Regression Tree with maximal depth of 10. For this, we used the class sklearn.tree.DecisionTreeRegressor from the sklearn library. We learn the parameters of the Decision Tree using the train dataset and then use it to evaluate  $\mathbf{LMS}_{\mathrm{tree}}$  metric on the test dataset. For the additive decoder, it is easy to compute this metric since the additive structure already gives a natural partition  $\mathcal{B}$  which matches the ground-truth. However, for the non-additive decoder, there is no natural partition and thus we cannot compute  $\mathbf{LMS}_{\mathrm{tree}}$  directly. To go around this problem, for the non-additive decoder, we compute  $\mathbf{LMS}_{\mathrm{tree}}$  for all possible partitions of  $d_z$  latent variables into blocks of size  $|B| = 2$  (assuming all blocks have the same dimension), and report the best  $\mathbf{LMS}_{\mathrm{tree}}$ . This procedure is tractable in our experiments due to the small dimensionality of the problem we consider.
B.4 Boxplots for main experiments (Table 1)
Since the standard error in the main results (Table 1) was high, we provide boxplots in Figures 9 & 10 to have a better visibility on what is causing this. We observe that the high standard error for the Additive approach was due to bad performance for a few bad random initializations for the ScalarLatents dataset; while we have nearly perfect latent identification for the others. Figure 14e shows the latent space learned by the worst case seed, which somehow learned a disconnected support even if the ground-truth support was connected. Similarly, for the case of Independent BlockLatents, there are only a couple of bad random initializations and the rest of the cases have perfect identification.
Figure 9: Reconstruction mean squared error (MSE)  $(\downarrow)$  and Latent Matching Score (LMS)  $(\uparrow)$  over 10 different random initializations for ScalarLatents dataset.
(a) Independent Latent Case
(b) Dependent Latent Case
Figure 10: Reconstruction mean squared error (MSE)  $(\downarrow)$  and Latent Matching Score (LMS)  $(\uparrow)$  for 10 different initializations for BlockLatents dataset.
B.5 Additional Results: BlockLatents Dataset
To get a qualitative understanding of latent identification in the BlockLatents dataset, we plot the response of each predicted latent as we change a particular ground-truth latent factor. We describe the following cases of changing the ground-truth latents:
- Ball 1 moving along x-axis: We sample 10 equally spaced points for  $z_{1}$  from [0, 1]; while keeping other latents fixed as follows:  $z_{2} = 0.25$ ,  $z_{3} = 0.50$ ,  $z_{4} = 0.75$ . We will never have occlusion since the balls are separated along the y-axis  $z_{4} - z_{2} > 0$ .
- Ball 2 moving along x-axis: We sample 10 equally spaced points for  $z_{3}$  from [0, 1]; while keeping other latents fixed as follows:  $z_{1} = 0.50$ ,  $z_{2} = 0.25$ ,  $z_{4} = 0.75$ . We will never have occlusion since the balls are separated along the y-axis  $z_{4} - z_{2} > 0$ .
- Ball 1 moving along y-axis: We sample 10 equally spaced points for  $z_{2}$  from [0, 1]; while keeping other latents fixed as follows:  $z_{1} = 0.25$ ,  $z_{3} = 0.75$ ,  $z_{4} = 0.50$ . We will never have occlusion since the balls are separated along the x-axis  $z_{3} - z_{1} > 0$ .
- Ball 2 moving along y-axis: We sample 10 equally spaced points for  $z_4$  from [0, 1]; while keeping other latents fixed as follows:  $z_1 = 0.25$ ,  $z_2 = 0.50$ ,  $z_3 = 0.75$ . We will never have occlusion since the balls are separated along the x-axis  $z_3 - z_1 > 0$ .
(a) Additive Decoder (Best)  $(\mathrm{LMS}_{\mathrm{Tree}}:99.9)$
(c) Additive Decoder (Median)  $(\mathrm{LMS}_{\mathrm{Tree}}:99.8)$
(e) Additive Decoder (Worst)  $(\mathrm{LMS}_{\mathrm{Tree}}:54.1)$
Figure 11: Latent responses for the cases with the best/median/worst  $\mathrm{LMS}_{\mathrm{Tree}}$  among runs performed on the BlockLatent dataset with independent latents. In each plot, we report the latent factors predicted from multiple images where one ball moves along only one axis at a time.
(b) Non-Additive Decoder (Best)  $(\mathrm{LMS}_{\mathrm{Tree}}:83.9)$
(d) Non-Additive Decoder (Median)  $(\mathrm{LMS}_{\mathrm{Tree}}:58.6)$
(f) Non-Additive Decoder (Worst)  $(\mathrm{LMS}_{\mathrm{Tree}}:24.6)$