SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underbrace {D _ {j _ {0} ^ {\prime}} \boldsymbol {v} _ {\pi \left(j _ {0} ^ {\prime}\right)} (\boldsymbol {z}) D _ {j _ {0}} \boldsymbol {v} _ {\pi \left(j _ {0}\right)} (\boldsymbol {z})} _ {\neq 0} + D _ {j _ {0} ^ {\prime}} \boldsymbol {v} _ {\pi \left(j _ {0}\right)} (\boldsymbol {z}) D _ {j _ {0}} \boldsymbol {v} _ {\pi \left(j _ {0} ^ {\prime}\right)} (\boldsymbol {z}) = 0, \tag {70}
$$
where the left term in the sum is different of 0 because of the definition of  $\pi$ . This implies that
$$
D _ {j _ {0} ^ {\prime}} \boldsymbol {v} _ {\pi (j _ {0})} (\boldsymbol {z}) D _ {j _ {0}} \boldsymbol {v} _ {\pi \left(j _ {0} ^ {\prime}\right)} (\boldsymbol {z}) \neq 0, \tag {71}
$$
otherwise (70) cannot hold. But (71) contradicts (68). Thus, we have that,
$$
(j, j ^ {\prime}) \in S ^ {c} \Longrightarrow (\pi (j), \pi (j ^ {\prime})) \in S ^ {c}. \tag {72}
$$
The contraposed is
$$
(\pi (j), \pi \left(j ^ {\prime}\right)) \in S \Longrightarrow (j, j ^ {\prime}) \in S \tag {73}
$$
$$
(j, j ^ {\prime}) \in S \Longrightarrow (\pi^ {- 1} (j), \pi^ {- 1} \left(j ^ {\prime}\right)) \in S. \tag {74}
$$
From the above, it is clear that  $\pi^{-1}$  respects  $\mathcal{B}$  which implies that  $\pi$  respects  $\mathcal{B}$  (Lemma 10). Thus  $D\pmb {v}(\pmb {z})$  is a  $\mathcal{B}$ -block permutation matrix.
Lemma 10 ( $\mathcal{B}$ -respecting permutations form a group). Let  $\mathcal{B}$  be a partition of  $\{1, \ldots, d_z\}$  and let  $\pi$  and  $\bar{\pi}$  be a permutation of  $\{1, \ldots, d_z\}$  that respect  $\mathcal{B}$ . The following holds:
1. The identity permutation  $e$  respects  $\mathcal{B}$
2. The composition  $\pi \circ \bar{\pi}$  respects  $\mathcal{B}$ .
3. The inverse permutation  $\pi^{-1}$  respects  $\mathcal{B}$ .
Proof. The first statement is trivial, since for all  $B \in \mathcal{B}$ ,  $e(B) = B \in \mathcal{B}$ .
The second statement follows since for all  $B \in \mathcal{B}$ ,  $\bar{\pi}(B) \in \mathcal{B}$  and thus  $\pi(\bar{\pi}(B)) \in \mathcal{B}$ .
We now prove the third statement. Let  $B \in \mathcal{B}$ . Since  $\pi$  is surjective and respects  $\mathcal{B}$ , there exists a  $B' \in \mathcal{B}$  such that  $\pi(B') = B$ . Thus,  $\pi^{-1}(B) = \pi^{-1}(\pi(B')) = B' \in \mathcal{B}$ .
A.6 Sufficient nonlinearity v.s. sufficient variability in nonlinear ICA with auxiliary variables
In Section 3.1, we introduced the "sufficient nonlinearity" condition (Assumption 2) and highlighted its resemblance to the "sufficient variability" assumptions often found in the nonlinear ICA literature [30, 31, 33, 36, 37, 42, 73]. We now clarify this connection. To make the discussion more concrete, we consider the sufficient variability assumption found in Hyvarinen et al. [33]. In this work, the latent variable  $z$  is assumed to be distributed according to
$$
p (\boldsymbol {z} \mid \boldsymbol {u}) := \prod_ {i = 1} ^ {d _ {z}} p _ {i} \left(\boldsymbol {z} _ {i} \mid \boldsymbol {u}\right). \tag {75}
$$
In other words, the latent factors  $\mathbf{z}_i$  are mutually conditionally independent given an observed auxiliary variable  $\mathbf{u}$ . Define
$$
\boldsymbol {w} (\boldsymbol {z}, \boldsymbol {u}) := \left(\left(\frac {\partial}{\partial \boldsymbol {z} _ {i}} \log p _ {i} (\boldsymbol {z} _ {i} \mid \boldsymbol {u})\right) _ {i \in [ d _ {z} ]} \left(\frac {\partial^ {2}}{\partial \boldsymbol {z} _ {i} ^ {2}} \log p _ {i} (\boldsymbol {z} _ {i} \mid \boldsymbol {u})\right) _ {i \in [ d _ {z} ]}\right) \in \mathbb {R} ^ {2 d _ {z}}. \tag {76}
$$
We now recall the assumption of sufficient variability of Hyvarinen et al. [33]:
Assumption 3 (Assumption of variability from Hyvarinen et al. [33, Theorem 1]). For any  $\pmb{z} \in \mathbb{R}^{d_z}$ , there exists  $2d_{z} + 1$  values of  $\pmb{u}$ , denoted by  $\pmb{u}^{(0)}, \pmb{u}^{(1)}, \dots, \pmb{u}^{(2d_z)}$  such that the  $2d_{z}$  vectors
$$
\boldsymbol {w} \left(\boldsymbol {z}, \boldsymbol {u} ^ {(1)}\right) - \boldsymbol {w} \left(\boldsymbol {z}, \boldsymbol {u} ^ {(0)}\right), \dots , \boldsymbol {w} \left(\boldsymbol {z}, \boldsymbol {u} ^ {\left(2 d _ {z}\right)}\right) - \boldsymbol {w} \left(\boldsymbol {z}, \boldsymbol {u} ^ {(0)}\right) \tag {77}
$$
are linearly independent.
To emphasize the resemblance with our assumption of sufficient nonlinearity, we rewrite it in the special case where the partition  $\mathcal{B} := \{\{1\}, \ldots, \{d_z\}\}$ . Note that, in that case,  $q := d_z + \sum_{B \in \mathcal{B}} \frac{|B|(|B| + 1)}{2} = 2d_z$ .
Assumption 4 (Sufficient nonlinearity (trivial partition)). For all  $\mathbf{z} \in \mathcal{Z}^{\mathrm{train}}$ ,  $\mathbf{f}$  is such that the following matrix has independent columns (i.e. full column-rank):
$$
\boldsymbol {W} (\boldsymbol {z}) := \left[ \left[ D _ {i} \boldsymbol {f} ^ {(i)} \left(\boldsymbol {z} _ {i}\right) \right] _ {i \in [ d _ {z} ]} \left[ D _ {i, i} ^ {2} \boldsymbol {f} ^ {(i)} \left(\boldsymbol {z} _ {i}\right) \right] _ {i \in [ d _ {z} ]} \right] \in \mathbb {R} ^ {d _ {x} \times 2 d _ {z}}. \tag {78}
$$
One can already see the resemblance between Assumptions 3 & 4, e.g. both have something to do with first and second derivatives. To make the connection even more explicit, define  $\boldsymbol{w}(\boldsymbol{z}, k)$  to be the  $k$ th row of  $W(\boldsymbol{z})$  (do not conflate with  $\boldsymbol{w}(\boldsymbol{z}, \boldsymbol{u})$ ). Also, recall the basic fact from linear algebra that the column-rank is always equal to the row-rank. This means that  $W(\boldsymbol{z})$  is full column-rank if and only if there exists  $k_1, \dots, k_{2d_z} \in [d_x]$  such that the vectors  $\boldsymbol{w}(\boldsymbol{z}, k_1), \dots, \boldsymbol{w}(\boldsymbol{z}, k_{2d_z})$  are linearly independent. It is then easy to see the correspondence between  $\boldsymbol{w}(\boldsymbol{z}, k)$  and  $\boldsymbol{w}(\boldsymbol{z}, \boldsymbol{u}) - \boldsymbol{w}(\boldsymbol{z}, \boldsymbol{u}^{(0)})$  (from Assumption 3) and between the pixel index  $k \in [d_x]$  and the auxiliary variable  $\boldsymbol{u}$ .
We now look at why Assumption 2 is likely to be satisfied when  $d_x >> d_z$ . Informally, one can see that when  $d_x$  is much larger than  $2d_z$ , the matrix  $W(z)$  has much more rows than columns and thus it becomes more likely that we will find  $2d_z$  rows that are linearly independent, thus satisfying Assumption 2.
Figure 7: Numerical verification that  $\pmb{f} : [-1,0] \times [0,1] \to \mathbb{R}^4$  from Example 8 is injective (left), has a full rank Jacobian (middle) and satisfies Assumption 2 (right). The left figure shows that  $\pmb{f}$  is injective on the square  $[-1,0] \times [0,1]$  since one can recover  $\pmb{z}$  uniquely by knowing the values of  $f_1(z)$  and  $f_2(z)$ , i.e. knowing the level sets. The middle figure reports the  $\operatorname*{det}(D\pmb{f}(\pmb{z})^\top D\pmb{f}(\pmb{z}))$  (columns of the Jacobian are normalized to have norm 1) and shows that it is nonzero in the square  $[-1,0] \times [0,1]$ , which means the Jacobian is full rank. The right figure shows the determinant of the matrix  $W(z)$  (from Assumption 2, but with normalized columns), we can see that it is nonzero everywhere on the square  $[-1,0] \times [0,1]$ . We normalized the columns of  $D\pmb{f}$  and  $W$  so that the determinant is between 0 and 1.
A.7 Examples of sufficiently nonlinear additive decoders
Example 8 (A sufficiently nonlinear  $\pmb{f}$  - Example 3 continued). Consider the additive function
$$
\boldsymbol {f} (\boldsymbol {z}) := \left[ \begin{array}{l} \boldsymbol {z} _ {1} \\ \boldsymbol {z} _ {1} ^ {2} \\ \boldsymbol {z} _ {1} ^ {3} \\ \boldsymbol {z} _ {1} ^ {4} \end{array} \right] + \left[ \begin{array}{l} (\boldsymbol {z} _ {2} + 1) \\ (\boldsymbol {z} _ {2} + 1) ^ {2} \\ (\boldsymbol {z} _ {2} + 1) ^ {3} \\ (\boldsymbol {z} _ {2} + 1) ^ {4} \end{array} \right]. \tag {79}
$$
We will provide a numerical verification that this function is a diffeomorphism from the square  $[-1,0] \times [0,1]$  to its image that satisfies Assumption 2.
The Jacobian of  $\pmb{f}$  is given by
$$
D \boldsymbol {f} (\boldsymbol {z}) = \left[ \begin{array}{l l} 1 & 1 \\ 2 \boldsymbol {z} _ {1} & 2 \left(\boldsymbol {z} _ {2} + 1\right) \\ 3 \boldsymbol {z} _ {1} ^ {2} & 3 \left(\boldsymbol {z} _ {2} + 1\right) ^ {2} \\ 4 \boldsymbol {z} _ {1} ^ {3} & 4 \left(\boldsymbol {z} _ {2} + 1\right) ^ {3} \end{array} \right], \tag {80}
$$
and the matrix  $W(z)$  from Assumption 2 is given by
$$
\boldsymbol {W} (\boldsymbol {z}) = \left[ \begin{array}{c c c c} 1 & 0 & 1 & 0 \\ 2 \boldsymbol {z} _ {1} & 2 & 2 \left(\boldsymbol {z} _ {2} + 1\right) & 2 \\ 3 \boldsymbol {z} _ {1} ^ {2} & 6 \boldsymbol {z} _ {1} & 3 \left(\boldsymbol {z} _ {2} + 1\right) ^ {2} & 6 \left(\boldsymbol {z} _ {2} + 1\right) \\ 4 \boldsymbol {z} _ {1} ^ {3} & 1 2 \boldsymbol {z} _ {1} ^ {2} & 4 \left(\boldsymbol {z} _ {2} + 1\right) ^ {3} & 1 2 \left(\boldsymbol {z} _ {2} + 1\right) ^ {2} \end{array} \right]. \tag {81}
$$
Figure 7 presents a numerical verification that  $\mathbf{f}$  is injective, has a full rank Jacobian and satisfies Assumption 2. Injective  $\mathbf{f}$  with full rank Jacobian is enough to conclude that  $\mathbf{f}$  is a diffeomorphism onto its image.
Example 9 (Smooth balls dataset is sufficiently nonlinear - Example 4 continued). We implemented a ground-truth additive decoder  $\pmb{f}:[0,5]^2\to \mathbb{R}^{64*64*3}$  which maps to 64x64 RGB images consisting of two colored balls where  $z_{1}$  and  $z_{2}$  control their respective heights (Figure 8a). The analytical form of  $\pmb{f}$  can be found in our code base. The decoder  $\pmb{f}$  is implemented in JAX [6] which allows for its automatic differentiation to compute  $D\pmb{f}$  and  $D^2\pmb{f}$  (Figures 8b & 8c). This allows us to verify numerically that  $\pmb{f}$  is sufficiently nonlinear (Assumption 2). Recall that this assumption requires that  $W(z)$  (defined in Assumption 2) has independent columns everywhere. To test this, we compute  $Vol(z)\coloneqq \sqrt{|\operatorname*{det}(W(z))^{\top}W(z))|}$  over a grid of values of  $z$  and verify that  $Vol(z) > 0$  everywhere (Figure 8d). Note that  $Vol(z)$  corresponds to the 4D volume of the parallelepiped embedded in  $\mathbb{R}^{64*64*3}$  spanned by the four columns of  $W(z)$ . This volume is  $>0$  if and only if the columns are linearly independent. Note that we normalize the columns of  $W(z)$  so that they have a norm
of one. It follows that  $\operatorname{Vol}(z)$  is between 0 and 1 where 1 means the vectors are orthogonal, i.e. maximally independent. The minimal value of  $\operatorname{Vol}(z)$  over the domain of  $\pmb{f}$  is  $\approx 0.97$ , indicating that Assumption 2 holds.
(d)
Figure 8: Figure (a) shows an image the synthetic dataset of Example 9. Figure (b) shows the derivative of the image w.r.t.  $z_{1}$  (the height of the left ball) where the color intensity of each pixel corresponds to the Euclidean norm along the RGB axis. Figure (c) similarly shows the second derivative of the image w.r.t.  $z_{1}$ . Figure (d) is a contour plot of the function  $\sqrt{|\operatorname*{det}(\boldsymbol{W}(\boldsymbol{z})^{\top}\boldsymbol{W}(\boldsymbol{z}))|}$  where  $\boldsymbol{W}(\boldsymbol{z})$  is defined in Assumption 2 (here columns are normalized to have unit norm). The smallest value of  $\sqrt{|\operatorname*{det}(\boldsymbol{W}(\boldsymbol{z})^{\top}\boldsymbol{W}(\boldsymbol{z}))|}$  across domain is  $\approx 0.97$ , indicating that Assumption 2 is satisfied. See Example 9 and code for details. Figure 8e is a higher resolution rendering of the red region of Figure 8d (to make sure there is no singularity there).
A.8 Proof of Theorem 2
We start with a simple definition:
Definition 15 (B-block permutation matrices). A matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$  is a B-block permutation matrix if it is invertible and can be written as  $\mathbf{A} = \mathbf{C}\mathbf{P}_{\pi}$  where  $\mathbf{P}_{\pi}$  is the matrix representing the B-respecting permutation  $\pi$  ( $P_{\pi}e_{i} = e_{\pi(i)}$ ) and  $\mathbf{C} \in \mathbb{R}_{S_{\mathcal{B}}}^{d \times d}$  (See Definitions 10 & 11).
The following technical lemma leverages continuity and path-connectedness to show that the block-permutation structure must remain the same across the whole domain. It can be skipped at first read.
Lemma 11. Let  $\mathcal{C}$  be a connected topological space and let  $M: \mathcal{C} \to \mathbb{R}^{d \times d}$  be a continuous function. Suppose that, for all  $c \in \mathcal{C}$ ,  $M(c)$  is an invertible  $\mathcal{B}$ -block permutation matrix (Definition 15). Then, there exists a  $\mathcal{B}$ -respecting permutation  $\pi$  such that for all  $c \in \mathcal{C}$  and all distinct  $B, B' \in \mathcal{B}$ ,  $M(c)_{\pi(B'), B} = 0$ .
Proof. The reason this result is not trivial, is that, even if  $M(c)$  is a  $\mathcal{B}$ -block permutation for all  $c$ , the permutation might change for different  $c$ . The goal of this lemma is to show that, if  $\mathcal{C}$  is connected and the map  $M(\cdot)$  is continuous, then one can find a single permutation that works for all  $c \in \mathcal{C}$ .
First, since  $\mathcal{C}$  is connected and  $M$  is continuous, its image,  $M(\mathcal{C})$ , must be connected (by [56, Theorem 23.5]).
Second, from the hypothesis of the lemma, we know that
$$
M (\mathcal {C}) \subseteq \mathcal {A} := \left(\bigcup_ {\pi \in \mathfrak {S} (\mathcal {B})} \mathbb {R} _ {S _ {\mathcal {B}}} ^ {d \times d} P _ {\pi}\right) \setminus \{\text {s i n g u l a r m a t r i c e s} \}, \tag {82}
$$
where  $\mathfrak{S}(\mathcal{B})$  is the set of  $\mathcal{B}$ -respecting permutations and  $\mathbb{R}_{S_B}^{d\times d}\pmb {P}_\pi = \{\pmb {M}\pmb {P}_\pi \mid \pmb {M}\in \mathbb{R}_{S_B}^{d\times d}\}$ . We can rewrite the set  $\mathcal{A}$  above as
$$
\mathcal {A} = \bigcup_ {\pi \in \mathfrak {S} (\mathcal {B})} \left(\mathbb {R} _ {S _ {\mathcal {B}}} ^ {d \times d} P _ {\pi} \backslash \{\text {s i n g u l a r m a t r i c e s} \}\right), \tag {83}
$$
We now define an equivalence relation  $\sim$  over  $\mathcal{B}$ -respecting permutation:  $\pi \sim \pi'$  iff for all  $B \in \mathcal{B}$ ,  $\pi(B) = \pi'(B)$ . In other words, two  $\mathcal{B}$ -respecting permutations are equivalent if they send every block to the same block (note that they can permute elements of a given block differently). We notice that
$$
\pi \sim \pi^ {\prime} \Rightarrow \mathbb {R} _ {S _ {\mathcal {B}}} ^ {d \times d} \boldsymbol {P} _ {\pi} = \mathbb {R} _ {S _ {\mathcal {B}}} ^ {d \times d} \boldsymbol {P} _ {\pi^ {\prime}}. \tag {84}
$$
Let  $\mathfrak{S}(\mathcal{B}) / \sim$  be the set of equivalence classes induce by  $\sim$  and let  $\Pi$  stand for one such equivalence class. Thanks to (84), we can define, for all  $\Pi \in \mathfrak{S}(\mathcal{B}) / \sim$ , the following set:
$$
V _ {\Pi} := \mathbb {R} _ {S _ {\mathcal {B}}} ^ {d \times d} \boldsymbol {P} _ {\pi} \backslash \{\text {s i n g u l a r m a t r i c e s} \}, \text {f o r s o m e} \pi \in \Pi , \tag {85}
$$
where the specific choice of  $\pi \in \Pi$  is arbitrary (any  $\pi' \in \Pi$  would yield the same definition, by (84)). This construction allows us to write
$$
\mathcal {A} = \bigcup_ {\Pi \in \mathfrak {S} (\mathcal {B}) / \sim} V _ {\Pi}, \tag {86}
$$
We now show that  $\{V_{\Pi}\}_{\Pi \in \mathfrak{S}(\mathcal{B}) / \sim}$  forms a partition of  $\mathcal{A}$ . Choose two distinct equivalence classes of permutations  $\Pi$  and  $\Pi'$  and let  $\pi \in \Pi$  and  $\pi' \in \Pi'$  be representatives. We note that
$$
\mathbb {R} _ {S _ {\mathcal {B}}} ^ {d \times d} \boldsymbol {P} _ {\pi} \cap \mathbb {R} _ {S _ {\mathcal {B}}} ^ {d \times d} \boldsymbol {P} _ {\pi^ {\prime}} \subseteq \left\{\text {s i n g u l a r m a t r i c e s} \right\}, \tag {87}
$$
since any matrix that is both in  $\mathbb{R}_{S_B}^{d\times d}\pmb {P}_\pi$  and  $\mathbb{R}_{S_B}^{d\times d}\pmb{P}_{\pi '}$  must have at least one row filled with zeros. This implies that
$$
V _ {\Pi} \cap V _ {\Pi^ {\prime}} = \emptyset , \tag {88}
$$
which shows that  $\{V_{\Pi}\}_{\Pi \in \mathfrak{S}(\mathcal{B}) / \sim}$  is indeed a partition of  $\mathcal{A}$ .
Each  $V_{\Pi}$  is closed in  $\mathcal{A}$  (wrt the relative topology) since
$$
V _ {\Pi} = \mathbb {R} _ {S _ {B}} ^ {d \times d} P _ {\pi} \backslash \{\text {s i n g u l a r m a t r i c e s} \} = \mathcal {A} \cap \underbrace {\mathbb {R} _ {S _ {B}} ^ {d \times d} P _ {\pi}} _ {\text {c l o s e d i n} \mathbb {R} ^ {d \times d}}. \tag {89}
$$
Moreover,  $V_{\Pi}$  is open in  $\mathcal{A}$ , since
$$
V _ {\Pi} = \mathcal {A} \backslash \bigcup_ {\substack {\Pi^ {\prime} \neq \Pi \\ \text {closed in } \mathcal {A}}} V _ {\Pi^ {\prime}}. \tag{90}
$$
Thus, for any  $\Pi \in \mathfrak{S}(\mathcal{B}) / \sim$ , the sets  $V_{\Pi}$  and  $\bigcup_{\Pi' \neq \Pi} V_{\Pi'}$  forms a separation (see [56, Section 23]). Since  $M(\mathcal{C})$  is a connected subset of  $\mathcal{A}$ , it must lie completely in  $V_{\Pi}$  or  $\bigcup_{\Pi' \neq \Pi} V_{\Pi'}$ , by [56, Lemma 23.2]. Since this is true for all  $\Pi$ , it must follow that there exists a  $\Pi^*$  such that  $M(\mathcal{C}) \subseteq V_{\Pi^*}$ , which completes the proof.
$$
\bar {\boldsymbol {v}} _ {\pi (B)} (\boldsymbol {z} _ {B}) = \left(\boldsymbol {f} ^ {\pi (B)}\right) ^ {- 1} \left(\hat {\boldsymbol {f}} ^ {(B)} (\boldsymbol {z} _ {B}) - \boldsymbol {c} ^ {(B)}\right), f o r a l l \boldsymbol {z} _ {B} \in \hat {\mathcal {Z}} _ {B} ^ {\text {t r a i n}}. \tag {9}
$$
Proof. Step 1 - Showing the permutation  $\pi$  does not change for different  $z$ . Theorem 1 showed local  $\mathcal{B}$ -disentanglement, i.e. for all  $z \in \hat{\mathcal{Z}}^{\mathrm{train}}$ ,  $Dv(z)$  has a  $\mathcal{B}$ -block permutation structure. The first step towards showing global disentanglement is to show that this block structure is the same for all  $z \in \hat{\mathcal{Z}}^{\mathrm{train}}$  (a priori,  $\pi$  could be different for different  $z$ ). Since  $v$  is  $C^2$ , its Jacobian  $Dv(z)$  is continuous. Since  $\mathcal{Z}^{\mathrm{train}}$  is path-connected,  $\hat{\mathcal{Z}}^{\mathrm{train}}$  must also be since both sets are diffeomorphic. By Lemma 11, this means the  $\mathcal{B}$ -block permutation structure of  $Dv(z)$  is the same for all  $z \in \hat{\mathcal{Z}}^{\mathrm{train}}$  (implicitly using the fact that path-connected implies connected). In other words, there exists a permutation  $\pi$  respecting  $\mathcal{B}$  such that, for all  $z \in \hat{\mathcal{Z}}^{\mathrm{train}}$  and all distinct  $B, B' \in \mathcal{B}$ ,  $D_B v_{\pi(B')} (z) = 0$ .
Step 2 - Linking object-specific decoders. We now show that, for all  $B \in \mathcal{B}$ ,  $\hat{\pmb{f}}^{(B)}(\pmb{z}_B) = \pmb{f}^{(\pi(B))}(\pmb{v}_{\pi(B)}(\pmb{z})) + \pmb{c}^{(B)}$  for all  $\pmb{z} \in \hat{\mathcal{Z}}^{\mathrm{train}}$ . To do this, we rewrite (50) as
$$
D \hat {\boldsymbol {f}} ^ {(J)} (\boldsymbol {z} _ {J}) = \sum_ {B \in \mathcal {B}} D \boldsymbol {f} ^ {(B)} (\boldsymbol {v} _ {B} (\boldsymbol {z})) D _ {J} \boldsymbol {v} _ {B} (\boldsymbol {z}), \tag {91}
$$
but because  $B \neq \pi(J) \implies D_J v_B(z) = 0$  (block-permutation structure), we get
$$
D \hat {\boldsymbol {f}} ^ {(J)} (\boldsymbol {z} _ {J}) = D \boldsymbol {f} ^ {(\pi (J))} \left(\boldsymbol {v} _ {\pi (J)} (\boldsymbol {z})\right) D _ {J} \boldsymbol {v} _ {\pi (J)} (\boldsymbol {z}). \tag {92}
$$
The above holds for all  $J \in \mathcal{B}$ . We simply change  $J$  by  $B$  in the following equation.
$$
D \hat {\boldsymbol {f}} ^ {(B)} (\boldsymbol {z} _ {B}) = D \boldsymbol {f} ^ {(\pi (B))} \left(\boldsymbol {v} _ {\pi (B)} (\boldsymbol {z})\right) D _ {B} \boldsymbol {v} _ {\pi (B)} (\boldsymbol {z}). \tag {93}
$$
Now notice that the r.h.s. of the above equation is equal to  $D(\pmb{f}^{(\pi(B))} \circ \pmb{v}_{\pi(B)})$ . We can thus write
$$
D \hat {\boldsymbol {f}} ^ {(B)} (\boldsymbol {z} _ {B}) = D \left(\boldsymbol {f} ^ {(\pi (B))} \circ \boldsymbol {v} _ {\pi (B)}\right) (\boldsymbol {z}), \text {f o r a l l} \boldsymbol {z} \in \hat {\mathcal {Z}} ^ {\text {t r a i n}}. \tag {94}
$$
Now choose distinct  $z, z^0 \in \hat{\mathcal{Z}}^{\mathrm{train}}$ . Since  $\mathcal{Z}^{\mathrm{train}}$  is path-connected,  $\hat{\mathcal{Z}}^{\mathrm{train}}$  also is since they are diffeomorphic. Hence, there exists a continuously differentiable function  $\phi : [0,1] \to \hat{\mathcal{Z}}^{\mathrm{train}}$  such that  $\phi(0) = z^0$  and  $\phi(1) = z$ . We can now use (94) together with the gradient theorem, a.k.a. the fundamental theorem of calculus for line integrals, to show the following
$$
\int_ {0} ^ {1} D \hat {\boldsymbol {f}} ^ {(B)} \left(\phi_ {B} (\boldsymbol {z})\right) \cdot \phi_ {B} (t) d t = \int_ {0} ^ {1} D \left(\boldsymbol {f} ^ {(\pi (B))} \circ \boldsymbol {v} _ {\pi (B)}\right) \left(\phi (\boldsymbol {z})\right) \cdot \phi (t) d t \tag {95}
$$
$$
\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) - \hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B} ^ {0}\right) = \boldsymbol {f} ^ {(\pi (B))} \circ \boldsymbol {v} _ {\pi (B)} (\boldsymbol {z}) - \boldsymbol {f} ^ {(\pi (B))} \circ \boldsymbol {v} _ {\pi (B)} \left(\boldsymbol {z} ^ {0}\right) \tag {96}
$$
$$
\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = \boldsymbol {f} ^ {(\pi (B))} \circ \boldsymbol {v} _ {\pi (B)} (\boldsymbol {z}) + \underbrace {\left(\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B} ^ {0}\right) - \boldsymbol {f} ^ {(\pi (B))} \circ \boldsymbol {v} _ {\pi (B)} \left(\boldsymbol {z} ^ {0}\right)\right)} _ {\text {c o n s t a n t i n} \boldsymbol {z}} \tag {97}
$$
$$
\hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B}\right) = \boldsymbol {f} ^ {(\pi (B))} \circ \boldsymbol {v} _ {\pi (B)} (\boldsymbol {z}) + \boldsymbol {c} ^ {(B)}, \tag {98}
$$
which holds for all  $z\in \hat{\mathcal{Z}}^{\mathrm{train}}$
We now show that  $\sum_{B\in \mathcal{B}}c^{(B)} = 0$ . Take some  $z^0\in \hat{\mathcal{Z}}^{\mathrm{train}}$ . Equations (49) & (98) tell us that
$$
\begin{array}{l} \sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(B)} \left(\boldsymbol {v} _ {B} \left(\boldsymbol {z} ^ {0}\right)\right) = \sum_ {B \in \mathcal {B}} \hat {\boldsymbol {f}} ^ {(B)} \left(\boldsymbol {z} _ {B} ^ {0}\right) (99) \\ = \sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(\pi (B))} \left(\boldsymbol {v} _ {\pi (B)} \left(\boldsymbol {z} ^ {0}\right)\right) + \sum_ {B \in \mathcal {B}} \boldsymbol {c} ^ {(B)} (100) \\ = \sum_ {B \in \mathcal {B}} \boldsymbol {f} ^ {(B)} \left(\boldsymbol {v} _ {B} \left(\boldsymbol {z} ^ {0}\right)\right) + \sum_ {B \in \mathcal {B}} \boldsymbol {c} ^ {(B)} (101) \\ \Rightarrow 0 = \sum_ {B \in \mathcal {B}} \boldsymbol {c} ^ {(B)} (102) \\ \end{array}
$$
Step 3 - From local to global disentanglement. By assumption, the functions  $f^{(B)}: \mathcal{Z}_B^{\mathrm{train}} \to \mathbb{R}^{d_x}$  are injective. This will allow us to show that  $\pmb{v}_{\pi(B)}(\pmb{z})$  depends only on  $\pmb{z}_B$ . We proceed by contradiction. Suppose there exists  $(\pmb{z}_B, \pmb{z}_{B^c}) \in \hat{\mathcal{Z}}^{\mathrm{train}}$  and  $\pmb{z}_{B^c}^0$  such that  $(\pmb{z}_B, \pmb{z}_{B^c}^0) \in \hat{\mathcal{Z}}^{\mathrm{train}}$  and  $\pmb{v}_{\pi(B)}(\pmb{z}_B, \pmb{z}_{B^c}) \neq \pmb{v}_{\pi(B)}(\pmb{z}_B, \pmb{z}_{B^c}^0)$ . This means
$$
\pmb {f} ^ {(\pi (B))} \circ \pmb {v} _ {\pi (B)} (\pmb {z} _ {B}, \pmb {z} _ {B ^ {c}}) + \pmb {c} ^ {(B)} = \hat {\pmb {f}} ^ {(B)} (\pmb {z} _ {B}) = \pmb {f} ^ {(\pi (B))} \circ \pmb {v} _ {\pi (B)} (\pmb {z} _ {B}, \pmb {z} _ {B ^ {c}} ^ {0}) + \pmb {c} ^ {(B)}
$$
$$
\boldsymbol {f} ^ {(\pi (B))} (\boldsymbol {v} _ {\pi (B)} (\boldsymbol {z} _ {B}, \boldsymbol {z} _ {B})) = \boldsymbol {f} ^ {(\pi (B))} (\boldsymbol {v} _ {\pi (B)} (\boldsymbol {z} _ {B}, \boldsymbol {z} _ {B} ^ {0}))
$$
which is a contradiction with the fact that  $\pmb{f}^{(\pi(B))}$  is injective. Hence,  $\pmb{v}_{\pi(B)}(\pmb{z})$  depends only on  $\pmb{z}_B$ . We also get an explicit form for  $\pmb{v}_{\pi(B)}$ :