SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial \boldsymbol {\varphi} _ {k}}{\partial \boldsymbol {z} _ {k}} = \left(\frac {\partial \boldsymbol {C}}{\partial \boldsymbol {\varphi} _ {k}} ^ {\top} \frac {\partial \boldsymbol {C}}{\partial \boldsymbol {\varphi} _ {k}}\right) ^ {- 1} \frac {\partial \boldsymbol {C}}{\partial \boldsymbol {\varphi} _ {k}} ^ {\top} \frac {\partial \boldsymbol {f}}{\partial \boldsymbol {z} _ {k}} \quad \forall k \in [ K ], \tag {15}
$$
which gives all solutions  $\frac{\partial\varphi_k}{\partial z_k}$  if any exist. This system is overdetermined, and a (unique) solution exists if  $\frac{\partial C}{\partial\varphi_k}$  has full (column) rank. In other words, to execute this simplification step on  $P$ , we require that for all  $z\in P$  the  $M$  column vectors of the form
$$
\left(\frac {\partial C _ {1}}{\partial \varphi_ {k m}} (\varphi (z)), \dots , \frac {\partial C _ {N}}{\partial \varphi_ {k m}} (\varphi (z))\right) ^ {\top} \quad \forall m \in [ M ] \tag {16}
$$
are linearly independent. Each entry of a column vector describes how all entries  $C_n$  of the final output (e.g., the pixels of the output image) change with a single entry  $\varphi_{km}$  of the intermediate representation of component  $k$  (e.g., a single pixel of the component-wise image). It is easy to see that if even a part of the intermediate representation is not reflected in the final output (e.g., in the presence of occlusions, when a single pixel of one component is occluded), the entire corresponding column is zero, and the matrix does not have full rank.
To circumvent this issue, we realize that the LHS of Equation 15 only depends on the latents  $z_{k}$  of a single component. Hence, for a given latent  $z$  and a slot index  $k$ , the correct component function will have the same solution for all points in any (finite) set
$$
P ^ {\prime} (\boldsymbol {z}, k) \subseteq \left\{\boldsymbol {p} \in \operatorname {s u p p} P \mid \boldsymbol {p} _ {k} = \boldsymbol {z} _ {k} \right\}. \tag {17}
$$
We can interpret these points as the intersection of  $P$  with a plane in latent space at  $z_{k}$  (e.g., all latent combinations in the training set in which one component is fixed in a specific configuration). We can then define a modified composition function  $\tilde{C}$  that takes  $z$  and a slot index  $k$  as input and produces a "superposition" of images corresponding to the latents in the subset as
$$
\tilde {C} (\varphi , z, k) = \sum_ {\boldsymbol {p} \in P ^ {\prime} (\boldsymbol {z}, k)} C (\varphi (\boldsymbol {p})). \tag {18}
$$
Essentially, we are condensing the information from multiple points in the latent space into a single function. This enables us to write a modified version of Equation 14 as
$$
\sum_ {\boldsymbol {p} \in P ^ {\prime} (\boldsymbol {z}, k)} \frac {\partial \boldsymbol {f}}{\partial \boldsymbol {z} _ {k}} (\boldsymbol {p}) = \sum_ {\boldsymbol {p} \in P ^ {\prime} (\boldsymbol {z}, k)} \frac {\partial C}{\partial \varphi_ {k}} (\boldsymbol {\varphi} (\boldsymbol {p})) \frac {\partial \varphi_ {k}}{\partial \boldsymbol {z} _ {k}} (\boldsymbol {z} _ {k}) = \frac {\partial \tilde {C}}{\partial \varphi_ {k}} (\boldsymbol {\varphi}, \boldsymbol {z}, k) \frac {\partial \varphi_ {k}}{\partial \boldsymbol {z} _ {k}} (\boldsymbol {z} _ {k}) \quad \forall k \in [ K ] \tag {19}
$$
Now we can solve for  $\frac{\partial\varphi_k}{\partial\boldsymbol{z}_k}$  as in Equation 15, but this time requires only that  $\frac{\partial\tilde{C}}{\partial\varphi_k}$  has full (column) rank for a unique solution to exist, i.e.,
$$
\operatorname {r a n k} \frac {\partial \tilde {\boldsymbol {C}}}{\partial \varphi_ {k}} (\boldsymbol {\varphi}, \boldsymbol {z}, k) = \sum_ {\boldsymbol {p} \in P ^ {\prime} (\boldsymbol {z}, k)} \frac {\partial \boldsymbol {C}}{\partial \varphi_ {k}} (\boldsymbol {\varphi} (\boldsymbol {p})) = M \quad \forall \boldsymbol {z} \in P \quad \forall k \in [ K ]. \tag {20}
$$
In general, this condition is easier to fulfill since full rank is not required in any one point but over a set of points. For occlusions, for example, any pixel of a slot can be occluded in some points  $\pmb{p} \in P'$ , as long as it is not occluded in all of them. We can interpret this procedure as "collecting sufficient information" such that an inversion of the generally non-invertible  $C$  becomes feasible locally.
The requirement that  $\operatorname{supp} P$  has to be an open set, together with the full rank condition on the Jacobian of the composition function condensed over multiple points,  $\tilde{C}$ , is termed sufficient support in the main paper (Definition 3 and Assumption (A3)). As explained here, this means that the training distribution  $P$ , the composition function  $C$ , and derivatives of the function  $f$  on the training set specify a unique relationship between the component function  $\varphi_{k}$  and its derivatives:
$$
\frac {\partial \varphi_ {k}}{\partial \boldsymbol {z} _ {k}} = \left(\frac {\partial \tilde {\boldsymbol {C}}}{\partial \varphi_ {k}} ^ {\top} (\varphi , \boldsymbol {z}, k) \frac {\partial \boldsymbol {C}}{\partial \varphi_ {k}}\right) ^ {- 1} \frac {\partial \boldsymbol {C}}{\partial \varphi_ {k}} ^ {\top} \sum_ {\boldsymbol {p} \in P ^ {\prime} (\boldsymbol {z}, k)} \frac {\partial \boldsymbol {f}}{\partial \boldsymbol {z} _ {k}} (\boldsymbol {p}) \quad \forall k \in [ K ]. \tag {21}
$$
As explained above, this solution to the linear system of equations constitutes a system of partial differential equations (PDEs) in the set of component functions  $\varphi$  with independent variables  $z$ . We can see that this system has the form
$$
\partial_ {i} \varphi (z) = a _ {i} (z, \varphi (z)), \tag {22}
$$
where  $i \in [L]$  with  $L \coloneqq KD$  is an index over the flattened dimensions  $K$  and  $D$  such that the system of PDEs contains the functional relation between  $\varphi$  and its derivative from Equation 21 for all values of  $k$ .  $a_{i}$  is the combination of corresponding terms from the RHS of Equation 21. If this system of PDEs allows for more than one solution, Equation 21 does not uniquely determine the component functions.
However, if we have access to some initial point for which we know  $\varphi(\mathbf{0}) = \varphi^0$ , we can write
$$
\begin{array}{l} \varphi (z _ {1}, \dots , z _ {L}) - \varphi^ {0} = (\varphi (z _ {1}, \dots , z _ {L}) - \varphi (0, z _ {2}, \dots , z _ {L})) \\ + \left(\varphi \left(0, z _ {2}, \dots , z _ {L}\right) - \varphi \left(0, 0, z _ {3}, \dots , z _ {L}\right)\right) \tag {23} \\ + \ldots \\ + \left(\varphi (0, \dots , 0, z _ {L}) - \varphi (0, \dots , 0)\right). \\ \end{array}
$$
In each line of this equation, only a single  $z_{i} \eqqcolon t$  is changing; all other  $z_{1}, \ldots, z_{L}$  are fixed. Any solution of Equation 23, therefore, also has to solve the  $L$  ordinary differential equations (ODEs) of the form
$$
\partial_ {t} \varphi (z _ {1}, \dots , z _ {i - 1}, t, z _ {i + 1}, \dots , z _ {L}) = \boldsymbol {a} _ {i} \left(z _ {1}, \dots , z _ {i - 1}, t, z _ {i + 1}, \dots , z _ {L}, \varphi (z _ {1}, \dots , z _ {i - 1}, t, z _ {i + 1}, \dots , z _ {L})\right), \tag {24}
$$
which have a unique solution if  $\pmb{a}_i$  is Lipschitz in  $\varphi$  and continuous in  $z_i$ , as guaranteed by (A1). Therefore, 23 has at most one solution. This reference point does not have to be in  $z = 0$ , as a simple coordinate transform will yield the same result for any point in  $P$ . It is therefore sufficient that there exists some point  $\pmb{p}^0 \in P$  for which  $\varphi(\pmb{p}^0) = \hat{\varphi}(\pmb{p}^0)$  to obtain the same unique solution for  $\varphi$  and  $\hat{\varphi}$ , which is exactly what (A4) states. Overall, this means that Equation 21 does indeed specify unique component functions  $\varphi_k$ .
To recap, we have shown that the training distribution  $P$ , the composition function  $C$ , and knowledge of  $f(z)$  for  $z \in \operatorname{supp} P$  specifies a unique relationship between the component functions  $\varphi_k$  and their derivative (Equation 21). With additional knowledge of one initial point, this constrains the problem enough to get a unique solution for the component functions  $\varphi_k$ . Since  $P$  and  $C$  are fixed, this means that if the functions  $f$  and  $\hat{f}$  agree on  $P$ , they specify the same component functions, i.e.,
$$
\boldsymbol {f} \equiv_ {P} \hat {\boldsymbol {f}} \Rightarrow \varphi \equiv_ {P} \hat {\varphi} \tag {25}
$$
Step 4. Finally, we can conclude the model  $\hat{\pmb{f}}$  fitting the ground-truth generating process  $\pmb{f}$  on the training distribution  $P$ , through Equations 25, 13, and 10, implies that the model generalizes to  $Q$ . In other words, Equation 9 holds.
Figure 6: Schematic of the employed models. The monolithic model (right) uses the same architecture as each component model of the compositional model (left), except with a higher number of hidden units and channels to match the number of parameters.
Figure 6 shows a schematic of the employed compositional and monolithic model from Section 4.
Table 2 additionally reports the reconstruction quality measures as the mean squared error (MSE) for the experiments from Section 4.
B.1 Details about the compositional functions
As explained in Equation 8 in section 4, the composition function is implemented as a soft pixel-wise addition in most experiments. The use of the sigmoid function  $\sigma (\cdot)$  in the composition
$$
\boldsymbol {x} = \sigma \left(\tilde {\boldsymbol {x}} _ {1}\right) \cdot \tilde {\boldsymbol {x}} _ {1} + \sigma \left(- \tilde {\boldsymbol {x}} _ {1}\right) \cdot \tilde {\boldsymbol {x}} _ {2} \tag {26}
$$
was necessary for training stability. With this formulation, sprites can also overlap somewhat transparently, which is not desired and leads to small reconstruction artifacts for some specific samples. Implementing the composition with a step function as
$$
\boldsymbol {x} = \operatorname {s t e p} \left(\tilde {\boldsymbol {x}} _ {1}\right) \cdot \tilde {\boldsymbol {x}} _ {1} + \operatorname {s t e p} \left(- \tilde {\boldsymbol {x}} _ {1}\right) \cdot \tilde {\boldsymbol {x}} _ {2} \tag {27}
$$
instead would be more faithful to the ground-truth data-generating process, but is hard to train with gradient descent.
Note that both formulations could easily be extended to more than one sprite by simply repeating the composition operation with any additional sprite.
#	Train Set	Model	MSE ID ↓	MSE all ↓
1	Random	Monolithic	1.73e-3±1.5e-5	1.73e-3±1.5e-5
2	Random	Compositional	1.07e-3±2.6e-5	1.07e-3±2.6e-5
3	Orthogonal	Monolithic	8.49e-4±2.9e-5	4.06e-2±3.9e-3
4	Orthogonal	Compositional	6.94e-4±1.1e-5	1.24e-3±4.1e-5
5	Ortho.~N	Compositional	7.01e-4±8.7e-6	1.24e-3±2.6e-5
6	Diagonal	Compositional	8.87e-4±1.0e-4	1.39e-3±4.0e-4
7	Ortho. (broad)	Compositional	6.50e-4±1.5e-5	1.16e-3±3.3e-5
8	Diag. (broad)	Compositional	9.51e-4±2.8e-5	1.13e-3±3.1e-5
9	Ortho. (gap)	Compositional	7.36e-4±2.7e-5	2.64e-3±1.8e-4
10	Diag. (narrow)	Compositional	2.22e-3±1.2e-3	1.04e-2±7.1e-3
11	Orthogonal	Comp. (RGBa)	2.67e-4±2.8e-6	5.24e-4±3.7e-6
Table 2: We report the reconstruction quality measured as mean squared error (MSE, lower is better) for both the in-domain (ID) test set and the entire latent space, averaged over 5 random seeds.
In section 4, we also looked at a model that implements the composition through alpha compositing instead (see also Table 1, #11). Here, each component's intermediate representation is an RGBa image. The components are then overlaid on an opaque black background using the composition function
$$
x _ {\alpha} = x _ {1, \alpha} + (1 - x _ {1, \alpha}) \cdot x _ {2, \alpha} \tag {28}
$$
$$
x _ {\mathrm {R G B}} = x _ {1, \alpha} \cdot x _ {1, \mathrm {R G B}} + (1 - x _ {1, \alpha}) \cdot \frac {x _ {2 , \alpha}}{x _ {\alpha}} \cdot x _ {2, \mathrm {R G B}}. \tag {29}
$$
While this yields a compositional function, the sufficient support condition (Definition 3) is generally not fulfilled on the sprites data. The reason is that in fully transparent pixels  $(\alpha = 0)$ , changing the RGB value is not reflected in the output. Conversely, if a pixel is black, changing its alpha value will not affect how it is blended over a black background. As a result, most columns in the Jacobian  $\frac{\partial C}{\partial \varphi_k}$  (see also Equation 16) will be zero. Since the intermediate representations of each sprite will contain a lot of black or transparent pixels (the entire background), the rank of the Jacobian here will be low. In this case, the workaround from Equation 18 does not help since the low rank is not a result of another component in the foreground but of the specific parameterization of each component itself.
As stated in the main paper, the fact that this parameterization still produces good results and generalizes well is an indicator that there might be another proof strategy or workaround that avoids this specific issue.
C Details on the sufficient support assumption
The sufficient support assumption as stated in Definition 3 amounts to a rank condition on the Jacobian of the final output  $\pmb{x}$  by the intermediate component representation  $\tilde{\pmb{x}}$ : For each configuration of a given component,  $P$  must be sufficiently large so that tracking the dependence of each output dimension on each dimension of the component representation is possible. Step 3 of the proof of Theorem 1 outlines how this condition arises in general: For some interactions between components, it is possible that the final output  $\pmb{x}$  becomes invariant to changes in the representation  $\tilde{\pmb{x}}_k$  of component  $k$ . We can assume that the output is not generally invariant to component  $k$  (since the component could then just be dropped from the data-generating process) and instead only arises for certain configurations of components. We provide additional examples here that illustrate how such an invariance can arise for different choices of  $C$  and detail how a sufficient support can be chosen in these cases. Note that while it is possible to calculate the sufficient support condition directly, this is costly to do on a dataset scale since it involves calculating large Jacobians for many different data points. Instead, it is helpful to analyze which specific component configurations would violate the condition and choose the training set accordingly.
2d sprites with occlusion (orthogonal sampling) As explained in the main text and illustrated in Figure 4, occlusions make the output image invariant to changes of the background sprite in the
Figure 7: For each configuration of the background sprite, the training distribution must contain at least two samples in which the foreground sprites do not overlap the same pixels.
occluded pixels. The model does not receive a training signal for these pixels, and reconstruction becomes impossible. In the case of orthogonal sampling, the foreground sprite is fixed in its base configuration (orange triangle in the bottom left corner) whenever the background sprite is sampled randomly; see the first row of Figure 7. The foreground sprite, therefore, always occludes the same background pixels, for which the Jacobian  $\frac{\partial C}{\partial \varphi_{km}}$  is zero. Therefore, for each configuration of the background sprite, a second data point is required in which the foreground sprite occludes a different and distinct set of pixels (cyan square in the second row of Figure 7). We can clearly observe that this is the case here; therefore, the Jacobians  $\frac{\partial C}{\partial \varphi_{km}}$  from both samples have zero entries at different indices  $m$  and their sum has full rank. The resulting sufficient support has the shape illustrated on the left of Figure 5.
2d sprites with occlusion (diagonal sampling) As explained above, sufficient support for occluding sprites is guaranteed if the dataset contains at least two samples with the same background sprite and different foreground sprites that overlap distinct pixels (as also illustrated in Figure 4). In the diagonal sampling case, we, therefore, need to choose the width of the diagonal broad enough to guarantee that it contains such foreground sprites. The smallest possible width of the diagonal can be determined by finding the smallest possible x-offset (or y-offset) for which a pair of sprites of the smallest scale does not overlap. This is the case for a width of 0.2, which was used for the diagonal sampling case in Table 1, #6. The broad diagonal sampling case Table 1, #8 used a width of 0.4 (double the minimal width), while the narrow diagonal sampling case in Table 1, #10 used a width of 0.1 (half the minimal width).
Attributes and transformations of an object The sufficient support assumption is also necessary If the composition function  $C$  is chosen to model the interaction of multiple attributes or transformations on a single object. For example, if one component models the rotation of an object and another models its shape, and the composition happens in pixel space, then a circular shape will be invariant to rotations. The Jacobian of the output by the rotation component will not have full rank whenever the shape component is circular. A sufficient support must contain at least one non-circular sample for each rotation angle in the training set. See Appendix D for an empirical observation of this phenomenon.
Overlaying audio signals Sufficient support has to be guaranteed whenever the composition function contains  $\max (\cdot)$  or  $\min (\cdot)$  operations or similar operations with zero gradients. For example, if two or more audio signals are picked up by a microphone modeled with finite gain, a signal above a certain amplitude will saturate the microphone. The sufficient support assumption guarantees that for the configuration of one audio signal, there exists at least one sample in the dataset in which the other signals are below that threshold, such that changes in the signal are registered by the microphone.
Vector or matrix products If the intermediate component representations  $\tilde{\pmb{x}}$  are vectors or matrices and the composition function is their (outer) product, changes in the first vector (matrix) of the product do not change the output if the second vector (matrix) multiplies with a zero-element (or
Figure 8: Composition of object attributes. Each set of three figures shows a traversal of one component: rotation, color, and blur. The intermediate output of the first component function  $\varphi_{1}$  is an image of a white square at a specific angle; the intermediate output of the other two component functions are convolution filters. Note that for strong Gaussian blurs, the shape becomes circular, making the output invariant to rotations.
Figure 9: Generalizing to novel compositions of object attributes. For the data-generating process illustrated in Figure 8, a compositional model can learn to generalize to unseen compositions of rotation, color, and blur when all conditions are fulfilled (middle row). If sufficient support is violated, in this case due to the rotation invariance introduced by the Gaussian blur, the model is unable to learn the rotation transformation. he reported OOD MSE is calculated for samples with blur  $= 0$
zero-row/column). The sufficient support assumption guarantees that the training set contains some configuration of the second vector (matrix) with zero elements at different locations.
D Additional experiments
D.1 Composition of Object Attributes
To demonstrate that the considered function class does not only apply to the composition of objects but can also model the composition of object attributes, we consider a modification of our sprite setting, in which each image contains a single sprite with configured by the three attributes rotation, color, and blur. Figure 8 illustrates the resulting data. We chose these attributes because, as outlined in the previous section, they illustrate the connection between the sufficient support assumption and transformation invariances. Additionally, these attributes can be implemented with a simple composition operation: convolution. The resulting generating process can, thus, be formulated as
$$
\boldsymbol {f} (\boldsymbol {z}) = \boldsymbol {C} (\varphi (\boldsymbol {z})) = \varphi_ {1} \left(\boldsymbol {z} _ {1}\right) * \varphi_ {2} \left(\boldsymbol {z} _ {2}\right) * \varphi_ {3} \left(\boldsymbol {z} _ {3}\right), \tag {30}
$$
where  $\varphi_{1}$  renders the sprite with a specific rotation,  $\varphi_{2}$  generates a convolution filter that changes the color, and  $\varphi_{3}$  generates a Gaussian blur filter.
As shown in Figure 8, the sprite's appearance is invariant to rotations whenever a strong Gaussian blur is applied. Consequently, to fulfill the sufficient support condition, the support of the training distribution needs to be chosen such that rotations are observed on configurations for which the blur is not too strong. To demonstrate this, we train a compositional model on two different training distributions: The first one has a diagonal support as in Experiment 1 #6, which fulfills the sufficient support assumption. The second one has an orthogonal support as in Experiment 1 #3,4 chosen such that different rotations are only observed with a strong Gaussian blur. Figure 9 shows reconstructions of random OOD samples for both models. While the model learns to reconstruct the color correctly on both training sets, the rotation can only be learned in the first case when sufficient support is fulfilled.
Train Set	R2ID↑	R2OOD↑	ΔR2↓
Orthogonal	0.676±4.8e-3	0.632±2.9e-3	0.044±5.6e-3
Orthogonal with gap	0.560±1.5e-2	0.345±1.3e-2	0.215±2.0e-2
Table 3: Experiment results on CLEVR [35]. On an orthogonal training set (Equation 31) that fulfills the assumptions from Theorem 1, a compositional model is able to generalize compositionally, as indicated by the minuscule difference in performance on ID and OOD points. When the compositional support assumption is violated by introducing a gap in the support of one latent, generalization fails, and the difference between ID and OOD performance increases significantly. The overall lower ID performance on the training set with a gap (and the corresponding proportionally lower OOD performance) are due to the smaller number of samples in this training set, as explained below. Results are averaged over three seeds.
D.2 Compositional generalization on CLEVR
We additionally conduct experiments on the CLEVR dataset [35], a popular benchmark for compositional generalization and object-centric learning. The original CLEVR dataset consists of simple 3d scenes containing a varying number of objects configured by their x-position, y-position, shape, color, size, and orientation. While CLEVR comes with the code to generate new samples, generation cannot be controlled precisely: Objects are always positioned randomly, no specific combinations of object attributes other than the relation between shape and color can be controlled, and collisions are always avoided. As a result, we cannot easily generate large, controlled training sets with precisely defined supports and distributions as in Section 4 and instead opt to filter the existing dataset.
Specifically, we select all 13145 images containing exactly three objects (since Theorem 1 implicitly assumes that the number of slots is known). We filter an orthogonal training set similar to the one described in Equation 2, except that we have three slots and replace equality to the base configurations with a distance measure  $d$ , i.e.,
$$
\operatorname {s u p p} P = \left\{\left(\boldsymbol {z} _ {1} \in \mathcal {Z} _ {1}, \boldsymbol {z} _ {2} \in \mathcal {Z} _ {2}, \boldsymbol {z} _ {3} \in \mathcal {Z} _ {3}\right) | d \left(\boldsymbol {z} _ {1}, \boldsymbol {z} _ {1} ^ {0}\right) <   \delta \vee d \left(\boldsymbol {z} _ {2}, \boldsymbol {z} _ {2} ^ {0}\right) <   \delta \vee d \left(\boldsymbol {z} _ {3}, \boldsymbol {z} _ {3} ^ {0}\right) <   \delta \right\}. \tag {31}
$$