SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Figure (a) shows latent representation outputted by the encoder  $\hat{\pmb{g}}(\pmb{x})$  over the training dataset, and the corresponding reconstructed images of the additive decoder with median LMSpear among runs performed on the ScalarLatents dataset. Figure (b) shows the same thing for the non-additive decoder. The color gradient corresponds to the value of one of the ground-truth factor, the red dots correspond to factors used to generate the images and the yellow dashed square highlights extrapolated images.
(a) Additive Decoder
Figure 5: Latent responses for the case of independent latents in the BlockLatent dataset. In each plot, we report the latent factors predicted from multiple images where one ball moves along only one axis at a time. For the additive case, at most two latents change, as it should, while more than two latents change for the non-additive case. See Appendix B.5 for details.
(b) Non-Additive Decoder
Additivity is important for Cartesian-product extrapolation. Figure 4 illustrates that the additive decoder can generate images that are outside the training domain (both balls in upper half of the image) while its non-additive counterpart cannot. Furthermore, Table 1 also corroborates this showing that the "out-of-support" (OOS) reconstruction MSE and  $\mathrm{LMS}_{\mathrm{Spear}}$  (evaluated only on the samples never seen during training) are significantly better for the additive than for the non-additive decoder.
Importance of connected support. Theorem 2 required that the support of the latent factors,  $\mathcal{Z}^{\mathrm{train}}$ , was path-connected. Appendix B.6 shows experiments where this assumption is violated, which yields lower  $\mathrm{LMS}_{\mathrm{Spear}}$  for the additive decoder, thus highlighting the importance of this assumption.
We provided an in-depth identifiability analysis of additive decoders, which bears resemblance to standard decoders used in OCRL, and introduced a novel theoretical framework showing how this architecture can generate reasonable images never seen during training via "Cartesian-product extrapolation". We validated empirically both of these results and confirmed that additivity was indeed crucial. By studying rigorously how disentanglement can induce extrapolation, our work highlighted the necessity of restricting the decoder to extrapolate and set the stage for future works to explore disentanglement and extrapolation in other function classes such as masked decoders typically used in OCRL. We postulate that the type of identifiability analysis introduced in this work has the potential of expanding our understanding of creativity in generative models, ultimately resulting in representations that generalize better.
This research was partially supported by the Canada CIFAR AI Chair Program, by an IVADO excellence PhD scholarship and by Samsung Electronics Co., Ldt. The experiments were in part enabled by computational resources provided by Calcul Québec (calculquebec.ca) and the Digital Research Alliance of Canada (alliancecan.ca). Simon Lacoste-Julien is a CIFAR Associate Fellow in the Learning in Machines & Brains program.
[1] K. Ahuja, J. Hartford, and Y. Bengio. Properties from mechanisms: an equivariance perspective on identifiable representation learning. In International Conference on Learning Representations, 2022.
[2] K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations, 2022.
[3] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[4] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 2013.
[5] M. Besserve, R. Sun, D. Janzing, and B. Scholkopf. A theory of independent mechanisms for extrapolation in generative models. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), 2021.
[6] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
[7] J. Brady, R. S. Zimmermann, Y. Sharma, B. Schölkopf, J. von Kugelgen, and W. Brendel. Provably learning object-centric representations. In International Conference on Machine Learning, 2023.
[8] J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, 2022.
[9] S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems, 2022.
[10] S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing, 2023.
[11] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. Monet: Unsupervised scene decomposition and representation, 2019.
[12] E. Crawford and J. Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[13] A. S. d'Avila Garcez and L. Lamb. Neurosymbolic AI: The 3rd wave. ArXiv, abs/2012.05876, 2020.
[14] A. Dittadi, S. S. Papa, M. De Vita, B. Schölkopf, O. Winther, and F. Locatello. Generalization and robustness implications in object-centric learning. In Proceedings of the 39th International Conference on Machine Learning, 2022.
[15] D. Donoho and C. Grimes. Image manifolds which are isometric to euclidean space. Journal of Mathematical Imaging and Vision, 2003.
[16] D. L. Donoho and C. Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 2003.
A Identifiability and Extrapolation Analysis 16
A.1 Useful definitions and lemmas 16
A.2 Relationship between additive decoders and the diagonal Hessian penalty 18
A.3 Additive decoders form a superset of compositional decoders [7] 19
A.4 Examples of local but non-global disentanglement 20
A.5 Proof of Theorem 1 22
A.6 Sufficient nonlinearity v.s. sufficient variability in nonlinear ICA with auxiliary variables 25
A.7 Examples of sufficiently nonlinear additive decoders 26
A.8 Proof of Theorem 2 27
A.9Injectivity of object-specific decoders v.s. injectivity of their sum 30
A.10 Proof of Corollary 3 31
A.11 Will all extrapolated images make sense? 32
A.12 Additive decoders cannot model occlusion 32
B Experiments 32
B.1 Training Details 32
B.2 Datasets Details 33
B.3 Evaluation Metrics 34
B.4 Boxplots for main experiments (Table 1) 34
B.5 Additional Results: BlockLatents Dataset 35
B.6 Disconnected Support Experiments 37
B.7 Additional Results: ScalarLatents Dataset 38
Calligraphic & indexing conventions
[n] := {1,2,...,n}
x	Scalar (random or not, depending on context)
x	Vector (random or not, depending on context)
X	Matrix
X	Set/Support
f	Scalar-valued function
f	Vector-valued function
|A	Restriction of f to the set A
Df, Df	Jacobian of f and f
D2f	Hessian of f
B ⊆ [n]	Subset of indices
|B|	Cardinality of the set B
xB	Vector formed with theith coordinates of x, for all i ∈ B
XB,B'	Matrix formed with the entries (i,j) ∈ B × B' of X.
Given X ⊆ R^n, XB := {xB | x ∈ X} (projection of X)
Recurrent notation
x ∈ Rdx	Observation
z ∈ Rdz	Vector of latent factors of variations
Z ⊆ Rdz	Support of z
f	Ground-truth decoder function
f	Learned decoder function
B	A partition of [dz] (assumed contiguous w.l.o.g.)
B ∈ B	A block of the partition B
B(i) ∈ B	The unique block of B that contains i
π: [dz] → [dz]	A permutation
SB := ∪B∈BB²
SC := [dz]2 \ SB
RdZ×dz := {M ∈ RdZ×dz | (i,j)∉SB ⇒Mi,j=0}
General topology
X̅	Closure of the subset X ⊆ R^n in the standard topology of R^n
X°	Interior of the subset X ⊆ R^n in the standard topology of R^n
A Identifiability and Extrapolation Analysis
A.1 Useful definitions and lemmas
We start by recalling some notions of general topology that are going to be used later on. For a proper introduction to these concepts, see for example Munkres [56].
Definition 6 (Regularly closed sets). A set  $\mathcal{Z} \subseteq \mathbb{R}^{d_z}$  is regularly closed if  $\mathcal{Z} = \overline{\mathcal{Z}^o}$ , i.e. if it is equal to the closure of its interior (in the standard topology of  $\mathbb{R}^n$ ).
Definition 7 (Connected sets). A set  $\mathcal{Z} \subseteq \mathbb{R}^{d_z}$  is connected if it cannot be written as a union of non-empty and disjoint open sets (in the subspace topology).
Definition 8 (Path-connected sets). A set  $\mathcal{Z} \subseteq \mathbb{R}^{d_z}$  is path-connected if for all pairs of points  $z^0, z^1 \in \mathcal{Z}$ , there exists a continuous map  $\phi : [0,1] \to \mathcal{Z}$  such that  $\phi(0) = z^0$  and  $\phi(1) = z^1$ . Such a map is called a path between  $z^0$  and  $z^1$ .
Table 2: Table of Notation.
Definition 9 (Homeomorphism). Let  $A$  and  $B$  be subsets of  $\mathbb{R}^n$  equipped with the subspace topology. A function  $f: A \to B$  is an homeomorphism if it is bijective, continuous and its inverse is continuous.
The following technical lemma will be useful in the proof of Theorem 1. For it, we will need additional notation: Let  $S \subseteq A \subseteq \mathbb{R}^n$ . We already saw that  $\overline{S}$  refers to the closure  $S$  in the  $\mathbb{R}^n$  topology. We will denote by  $\operatorname{cl}_A(S)$  the closure of  $S$  in the subspace topology of  $A$  induced by  $\mathbb{R}^n$ , which is not necessarily the same as  $\overline{S}$ . In fact, both can be related via  $\operatorname{cl}_A = \overline{S} \cap A$  (see Munkres [56, Theorem 17.4, p.95]).
Lemma 4. Let  $A, B \subseteq \mathbb{R}^n$  and suppose there exists a homeomorphism  $f: A \to B$ . If  $A$  is regularly closed in  $\mathbb{R}^n$ , we have that  $B \subseteq \overline{B^\circ}$ .
Proof. Note that  $\left.\pmb{f}\right|_{A^{\circ}}$  is a continuous injective function from the open set  $A^{\circ}$  to  $\pmb{f}(A^{\circ})$ . By the "invariance of domain" theorem [56, p.381], we have that  $\pmb{f}(A^{\circ})$  must be open in  $\mathbb{R}^n$ . Of course, we have that  $\pmb{f}(A^{\circ}) \subseteq B$ , and thus  $\pmb{f}(A^{\circ}) \subseteq B^{\circ}$  (the interior of  $B$  is the largest open set contained in  $B$ ). Analogously,  $\left.\pmb{f}^{-1}\right|_{B^{\circ}}$  is a continuous injective function from the open set  $B^{\circ}$  to  $\pmb{f}^{-1}(B^{\circ})$ . Again, by "invariance of domain",  $\pmb{f}^{-1}(B^{\circ})$  must be open in  $\mathbb{R}^n$  and thus  $\pmb{f}^{-1}(B^{\circ}) \subseteq A^{\circ}$ . We can conclude that  $\pmb{f}(A^{\circ}) = B^{\circ}$ .
We can conclude as follow:
$$
B = \boldsymbol {f} (A) = \boldsymbol {f} (\bar {A ^ {\circ}}) = \boldsymbol {f} (\bar {A ^ {\circ}} \cap A) = \boldsymbol {f} \left(\operatorname {c l} _ {A} \left(A ^ {\circ}\right)\right) \subseteq \operatorname {c l} _ {B} \left(\boldsymbol {f} \left(A ^ {\circ}\right)\right) = \operatorname {c l} _ {B} \left(B ^ {\circ}\right) = \bar {B ^ {\circ}} \cap B \subseteq \bar {B ^ {\circ}},
$$
where the first inclusion holds by continuity of  $f$  [56, Thm.18.1 p.104].
This lemma is taken from [42].
Lemma 5 (Sparsity pattern of an invertible matrix contains a permutation). Let  $\mathbf{L} \in \mathbb{R}^{m \times m}$  be an invertible matrix. Then, there exists a permutation  $\sigma$  such that  $\mathbf{L}_{i,\sigma(i)} \neq 0$  for all  $i$ .
Proof. Since the matrix  $L$  is invertible, its determinant is non-zero, i.e.
$$
\det  (\boldsymbol {L}) := \sum_ {\pi \in \mathfrak {S} _ {m}} \operatorname {s i g n} (\pi) \prod_ {i = 1} ^ {m} \boldsymbol {L} _ {i, \pi (i)} \neq 0, \tag {12}
$$
where  $\mathfrak{S}_m$  is the set of  $m$ -permutations. This equation implies that at least one term of the sum is non-zero, meaning there exists  $\pi \in \mathfrak{S}_m$  such that for all  $i \in [m]$ ,  $L_{i,\pi(i)} \neq 0$ .
Definition 10 (Aligned subspaces of  $\mathbb{R}^{m\times n}$ ). Given a subset  $S\subseteq \{1,\dots,m\} \times \{1,\dots,n\}$ , we define
$$
\mathbb {R} _ {S} ^ {m \times n} := \left\{\boldsymbol {M} \in \mathbb {R} ^ {m \times n} \mid (i, j) \notin S \Longrightarrow \boldsymbol {M} _ {i, j} = 0 \right\}. \tag {13}
$$
Definition 11 (Useful sets). Given a partition  $\mathcal{B}$  of  $[d]$ , we define
$$
S _ {\mathcal {B}} := \bigcup_ {B \in \mathcal {B}} B ^ {2} \quad S _ {\mathcal {B}} ^ {c} := \left\{1, \dots , d _ {z} \right\} ^ {2} \backslash S _ {\mathcal {B}} \tag {14}
$$
Definition 12 ( $C^k$ -diffeomorphism). Let  $A \subseteq \mathbb{R}^n$  and  $B \subseteq \mathbb{R}^m$ . A map  $\pmb{f}: A \to B$  is said to be a  $C^k$ -diffeomorphism if it is bijective,  $C^2$  and has a  $C^2$  inverse.
Remark 2. Differentiability is typically defined for functions that have an open domain in  $\mathbb{R}^n$ . However, in the definition above, the set  $A$  might not be open in  $\mathbb{R}^n$  and  $B$  might not be open in  $\mathbb{R}^m$ . In the case of an arbitrary domain  $A$ , it is customary to say that a function  $\pmb{f}: A \subseteq \mathbb{R}^n \to \mathbb{R}^m$  is  $C^k$  if there exists a  $C^k$  function  $\pmb{g}$  defined on an open set  $U \subseteq \mathbb{R}^n$  that contains  $A$  such that  $\left.\pmb{g}\right|_A = \pmb{f}$  (i.e.  $\pmb{g}$  extends  $\pmb{f}$ ). With this definition, we have that a composition of  $C^k$  functions is  $C^k$ , as usual. See for example p.199 of Munkres [55].
The following lemma allows us to unambiguously define the  $k$  first derivatives of a  $C^k$  function  $\pmb{f}:A\to \mathbb{R}^{m}$  on the set  $\overline{A}^{\circ}$ .
Lemma 6. Let  $A \subseteq \mathbb{R}^n$  and  $f: A \to \mathbb{R}^m$  be a  $C^k$  function. Then, its  $k$ th derivatives is uniquely defined on  $\overline{A^\circ}$  in the sense that they do not depend on the specific choice of  $C^k$  extension.
Proof. Let  $\pmb{g}: U \to \mathbb{R}^n$  and  $\pmb{h}: V \to \mathbb{R}^n$  be two  $C^k$  extensions of  $\pmb{f}$  to  $U \subseteq \mathbb{R}^n$  and  $V \subseteq \mathbb{R}^n$  both open in  $\mathbb{R}^n$ . By definition,
$$
\boldsymbol {g} (\boldsymbol {x}) = \boldsymbol {f} (\boldsymbol {x}) = \boldsymbol {h} (\boldsymbol {x}), \forall \boldsymbol {x} \in A. \tag {15}
$$
The usual derivative is uniquely defined on the interior of the domain, so that
$$
D \boldsymbol {g} (\boldsymbol {x}) = D \boldsymbol {f} (\boldsymbol {x}) = D \boldsymbol {h} (\boldsymbol {x}), \forall \boldsymbol {x} \in A ^ {\circ}. \tag {16}
$$
Consider a point  $\pmb{x}_0 \in \overline{A^\circ}$ . By definition of closure, there exists a sequence  $\{\pmb{x}_k\}_{k=1}^\infty \subseteq A^\circ$  s.t.  $\lim_{k \to \infty} \pmb{x}_k = \pmb{x}_0$ . We thus have that
$$
\lim  _ {k \rightarrow \infty} D g \left(\boldsymbol {x} _ {k}\right) = \lim  _ {k \rightarrow \infty} D h \left(\boldsymbol {x} _ {k}\right) \tag {17}
$$
$$
D \boldsymbol {g} \left(\boldsymbol {x} _ {0}\right) = D \boldsymbol {h} \left(\boldsymbol {x} _ {0}\right), \tag {18}
$$
where we used the fact that the derivatives of  $g$  and  $h$  are continuous to go to the second line. Thus, all the  $C^k$  extensions of  $f$  must have equal derivatives on  $\overline{A^\circ}$ . This means we can unambiguously define the derivative of  $f$  everywhere on  $\overline{A^\circ}$  to be equal to the derivative of one of its  $C^k$  extensions.
Since  $\pmb{f}$  is  $C^k$ , its derivative  $D\pmb{f}$  is  $C^{k - 1}$ , we can thus apply the same argument to get that the second derivative of  $\pmb{f}$  is uniquely defined on  $\overline{\overline{A}^{\circ}}$ . It can be shown that  $\overline{\overline{A}^{\circ}} = \overline{A^{\circ}}$ . One can thus apply the same argument recursively to show that the first  $k$  derivatives of  $\pmb{f}$  are uniquely defined on  $\overline{A}^{\circ}$ .
Definition 13 ( $C^k$ -diffeomorphism onto its image). Let  $A \subseteq \mathbb{R}^n$ . A map  $\pmb{f}: A \to \mathbb{R}^m$  is said to be a  $C^k$ -diffeomorphism onto its image if the restriction  $\pmb{f}$  to its image  $\tilde{\pmb{f}}: A \to \pmb{f}(A)$  is a  $C^k$ -diffeomorphism.
Remark 3. If  $S \subseteq A \subseteq \mathbb{R}^n$  and  $f: A \to \mathbb{R}^m$  is a  $C^k$ -diffeomorphism on its image, then the restriction of  $f$  to  $S$ , i.e.  $\left. f \right|_S$ , is also a  $C^k$  diffeomorphism on its image. That is because  $\left. f \right|_S$  is clearly bijective, is  $C^k$  (simply take the  $C^k$  extension of  $f$ ) and so is its inverse (simply take the  $C^k$  extension of  $f^{-1}$ ).
A.2 Relationship between additive decoders and the diagonal Hessian penalty
Proposition 7 (Equivalence between additivity and diagonal Hessian). Let  $\pmb{f} : \mathbb{R}^{d_z} \to \mathbb{R}^{d_x}$  be a  $C^2$  function. Then,
$$
\begin{array}{l}\forall \boldsymbol {z} \in \mathbb {R} ^ {d _ {z}}, \boldsymbol {f} (\boldsymbol {z}) = \sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(B)} (\boldsymbol {z} _ {B})\\\text {w h e r e} \boldsymbol {f} ^ {(B)}: \mathbb {R} ^ {| B |} \rightarrow \mathbb {R} ^ {d _ {x}} i s C ^ {2}.\end{array}\quad \Longleftrightarrow \quad\begin{array}{l}\forall k \in [ d _ {x} ], \boldsymbol {z} \in \mathbb {R} ^ {d _ {z}}, D ^ {2} \boldsymbol {f} _ {k} (\boldsymbol {z}) i s\\\text {b l o c k d i a g o n a l w i t h b l o c k s i n} \mathcal {B}.\end{array}\tag {19}
$$
Proof. We start by showing the “ $\Longrightarrow$ ” direction. Let  $B$  and  $B'$  be two distinct blocks of  $\mathcal{B}$ . Let  $i \in B$  and  $i' \in B'$ . We can compute the derivative of  $\pmb{f}_k$  w.r.t.  $\pmb{z}_i$ :
$$
D _ {i} \boldsymbol {f} _ {k} (\boldsymbol {z}) = \sum_ {\bar {B} \in \mathcal {B}} D _ {i} \boldsymbol {f} _ {k} ^ {(\bar {B})} \left(\boldsymbol {z} _ {\bar {B}}\right) = D _ {i} \boldsymbol {f} _ {k} ^ {(B)} \left(z _ {B}\right), \tag {20}
$$
where the last equality holds because  $i \in B$  and not in any other block  $\bar{B}$ . Furthermore,
$$
D _ {i, i ^ {\prime}} ^ {2} \boldsymbol {f} _ {k} (\boldsymbol {z}) = D _ {i, i ^ {\prime}} ^ {2} \boldsymbol {f} _ {k} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = 0, \tag {21}
$$
where the last equality holds because  $i' \notin B$ . This shows that  $D^2 f_k(z)$  is block diagonal.
We now show the “ $\Leftarrow$ ” direction. Fix  $k \in [d_x]$ ,  $B \in \mathcal{B}$ . We know that  $D_{B,B^c}^2 f_k(z) = 0$  for all  $z \in \mathbb{R}^{d_z}$ . Fix  $z \in \mathbb{R}^{d_z}$ . Consider a continuously differentiable path  $\phi : [0,1] \to \mathbb{R}^{|B^c|}$  such that  $\phi(0) = 0$  and  $\phi(1) = z_{B^c}$ . As  $D_{B,B^c}^2 f_k(z)$  is a continuous function of  $z$ , we can use the fundamental theorem of calculus for line integrals to get that
$$
D _ {B} \boldsymbol {f} _ {k} \left(\boldsymbol {z} _ {B}, \boldsymbol {z} _ {B ^ {c}}\right) - D _ {B} \boldsymbol {f} _ {k} \left(\boldsymbol {z} _ {B}, 0\right) = \int_ {0} ^ {1} \underbrace {D _ {B , B ^ {c}} ^ {2} \boldsymbol {f} _ {k} \left(\boldsymbol {z} _ {B} , \phi (t)\right)} _ {= 0} \phi^ {\prime} (t) d t = 0, \tag {22}
$$
(where  $D_{B,B^c}^2\pmb {f}_k(\pmb {z}_B,\phi (t))\phi '(t)$  denotes a matrix-vector product) which implies that
$$
D _ {B} \boldsymbol {f} _ {k} (\boldsymbol {z}) = D _ {B} \boldsymbol {f} _ {k} \left(\boldsymbol {z} _ {B}, 0\right). \tag {23}
$$
And the above equality holds for all  $B \in \mathcal{B}$  and all  $z \in \mathbb{R}^{d_z}$ .
Choose an arbitrary  $z \in \mathbb{R}^{d_z}$ . Consider a continuously differentiable path  $\psi : [0,1] \to \mathbb{R}^{d_z}$  such that  $\psi(0) = 0$  and  $\psi(1) = z$ . By applying the fundamental theorem of calculus for line integrals once more, we have that
$$
\begin{array}{l} \boldsymbol {f} _ {k} (\boldsymbol {z}) - \boldsymbol {f} _ {k} (0) = \int_ {0} ^ {1} D \boldsymbol {f} _ {k} (\boldsymbol {\psi} (t)) \boldsymbol {\psi} ^ {\prime} (t) d t (24) \\ = \int_ {0} ^ {1} \sum_ {B \in \mathcal {B}} D _ {B} \boldsymbol {f} _ {k} (\boldsymbol {\psi} (t)) \boldsymbol {\psi} _ {B} ^ {\prime} (t) d t (25) \\ = \sum_ {B \in \mathcal {B}} \int_ {0} ^ {1} D _ {B} \boldsymbol {f} _ {k} (\boldsymbol {\psi} (t)) \boldsymbol {\psi} _ {B} ^ {\prime} (t) d t (26) \\ = \sum_ {B \in \mathcal {B}} \int_ {0} ^ {1} D _ {B} \boldsymbol {f} _ {k} \left(\psi_ {B} (t), 0\right) \psi_ {B} ^ {\prime} (t) d t, (27) \\ \end{array}
$$
where the last equality holds by (23). We can further apply the fundamental theorem of calculus for line integrals to each term  $\int_0^1 D_B\pmb {f}_k(\psi_B(t),0)\pmb{\psi}_B'(t)dt$  to get
$$
\begin{array}{l} \boldsymbol {f} _ {k} (\boldsymbol {z}) - \boldsymbol {f} _ {k} (0) = \sum_ {B \in \mathcal {B}} \left(\boldsymbol {f} _ {k} \left(\boldsymbol {z} _ {B}, 0\right) - \boldsymbol {f} _ {k} (0, 0)\right) (28) \\ \Rightarrow \boldsymbol {f} _ {k} (\boldsymbol {z}) = \boldsymbol {f} _ {k} (0) + \sum_ {B \in \mathcal {B}} \left(\boldsymbol {f} _ {k} \left(\boldsymbol {z} _ {B}, 0\right) - \boldsymbol {f} _ {k} (0)\right) (29) \\ = \sum_ {B \in \mathcal {B}} \underbrace {\left(\boldsymbol {f} _ {k} \left(\boldsymbol {z} _ {B} , 0\right) - \frac {| \mathcal {B} | - 1}{| \mathcal {B} |} \boldsymbol {f} _ {k} (0)\right)} _ {\boldsymbol {f} _ {k} ^ {(B)} (\boldsymbol {z} _ {B}) :=}. (30) \\ \end{array}
$$
and since  $\mathbf{z}$  was arbitrary, the above holds for all  $\mathbf{z} \in \mathbb{R}^{d_z}$ . Note that the functions  $\pmb{f}_k^{(B)}(\pmb{z}_B)$  must be  $C^2$  because  $\pmb{f}_k$  is  $C^2$ . This concludes the proof.
A.3 Additive decoders form a superset of compositional decoders [7]
Compositional decoders were introduced by Brady et al. [7] as a suitable class of functions to perform object-centric representation learning with identifiability guarantees. They are also interested in block-disentanglement, but, contrarily to our work, they assume that the latent vector  $z$  is fully supported, i.e.  $\mathcal{Z} = \mathbb{R}^{d_z}$ . We now rewrite the definition of compositional decoders in the notation used in this work:
Definition 14 (Compositional decoders, adapted from [7]). Given a partition  $\mathcal{B}$ , a differentiable decoder  $\pmb{f}:\mathbb{R}^{d_z}\to \mathbb{R}^{d_x}$  is said to be compositional w.r.t.  $\mathcal{B}$  whenever the Jacobian  $D\pmb {f}(\pmb {z})$  is such that for all  $i\in [d_z],B\in \mathcal{B},z\in \mathbb{R}^{d_z}$ , we have
$$
D _ {B} \boldsymbol {f} _ {i} (\boldsymbol {z}) \neq \mathbf {0} \Longrightarrow D _ {B ^ {c}} \boldsymbol {f} _ {i} (\boldsymbol {z}) = \mathbf {0},
$$
where  $B^c$  is the complement of  $B\in \mathcal{B}$
In other words, each line of the Jacobian can have nonzero values only in one block  $B \in \mathcal{B}$ . Note that this nonzero block can change with different values of  $z$ .
The next result shows that additive decoders form a superset of  $C^2$  compositional decoders (Brady et al. [7] assumed only  $C^1$ ). Note that additive decoders are strictly more expressive than  $C^2$  compositional decoders because some additive functions are not compositional, like Example 3 for instance.
Proposition 8 (Compositional implies additive). Given a partition  $\mathcal{B}$ , if  $f: \mathbb{R}^{d_z} \to \mathbb{R}^{d_x}$  is compositional (Definition 14) and  $C^2$ , then it is also additive (Definition 1).
Proof. Choose any  $i \in [d_x]$ . Our strategy will be to show that  $D^2\pmb{f}_i$  is block diagonal everywhere on  $\mathbb{R}^{d_z}$  and use Proposition 7 to conclude that  $\pmb{f}_i$  is additive.
Choose an arbitrary  $\mathbf{z}_0 \in \mathbb{R}^{d_z}$ . By compositionality, there exists a block  $B \in \mathcal{B}$  such that  $D_{B^c} \mathbf{f}_i(\mathbf{z}_0) = \mathbf{0}$ . We consider two cases separately: