SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5 in the main paper presents the latent responses plot for the median  $\mathrm{LMS}_{\mathrm{tree}}$  case among random initializations. In Figure 11, we provide the results for the case of best and the worst  $\mathrm{LMS}_{\mathrm{tree}}$  among random seeds. We find that Additive Decoder fails for only for the worst case random seed, while Non-Additive Decoder fails for all the cases.
Additionally, we provide the object-specific reconstructions for the Additive Decoder in Figure 12. This helps us better understand the failure of Additive Decoder for the worst case random seed (Figure 12c), where the issue arises due to bad reconstruction error.
(a) Additive Decoder (Best)
(b) Additive Decoder (Median)
(c) Additive Decoder (Worst)
Figure 12: Object-specific renderings with the best/median/worst  $\mathrm{LMS}_{\mathrm{tree}}$  among runs performed on the BlockLatents dataset with independent latents. In each plot, the first row is the original image, the second row is the reconstruction and the third and fourth rows are the output of the object-specific decoders. In the best and median cases, each object-specific decoder corresponds to one and only one object, e.g. the third row of the best case always corresponds to the red ball. However, in the worst case, there are issues with reconstruction as only one of the balls is generated. Note that the visual artefacts are due to the additive constant indeterminacy we saw in Theorem 2, which cancel each other as is suggested by the absence of artefacts in the reconstruction.
B.6 Disconnected Support Experiments
Since path-connected latent support is an important assumption for latent identification with additive decoders (Theorem 2), we provide results for the case where the assumption is not satisfied. We experiment with the Disconnected Support dataset (Section B.2) and find that we obtain much worse  $\mathrm{LMS}_{\mathrm{Spear}}$  as compared to the case of training with L-shaped support in the ScalarLatents dataset. Over 10 different random initializations, we find mean  $\mathrm{LMS}_{\mathrm{Spear}}$  performance of 69.5 with standard error of 6.69.
For better qualitative understanding, we provide visualization of the latent support and the extrapolated images for the median  $\mathrm{LMS}_{\mathrm{Spear}}$  among 10 random seeds in Figure 13. Somewhat surprisingly, the representation appears to be aligned in the sense that the first predicted latent corresponds to the blue ball while the second predicted latent correspond to the red ball. Also surprisingly, extrapolation
Figure 13: Learned latent space,  $\hat{\mathcal{Z}}^{\mathrm{train}}$ , and the corresponding reconstructed images of the additive decoder with the median LMSpear among runs performed on the Disconnected Support dataset. The red dots correspond to latent factors used to generate the images.
occurs (we can see images of both balls high). That being said, we observe that the relationship between the predicted latent  $2(\hat{z}_2)$  and y-coordinate of second (red) ball is not monotonic, which explains why the Spearman correlation is so low (Spearman correlation scores are high when there is a monotonic relationship between both variables).
B.7 Additional Results: ScalarLatents Dataset
To get a qualitative understanding of extrapolation, we plot the latent support on the test dataset and sample a grid of equally spaced points from the support of each predicted latent on the test dataset. The grid represents the cartesian-product of the support of predicted latents and would contain novel combinations of latents that were unseen during training. We show the reconstructed images for each point from the cartesian-product grid to see whether the model is able to reconstruct well the novel latent combinations.
Figure 4 in the main paper presents visualizations of the latent support and the extrapolated images for the median  $\mathrm{LMS}_{\mathrm{Spear}}$  case among random seeds. In Figure 14, we provide the results for the case of best and the worst  $\mathrm{LMS}_{\mathrm{Spear}}$  among random seeds. We find that even for the best case (Figure 14b), Non-Additive Decoder does not generate good quality extrapolated images, while Additive Decoder generates extrapolated images for the best and median case. The worst-case run for the Additive Decoder has disconnected support, which explains why it is not able to extrapolate.
(a) Additive Decoder (Best)  $(\mathrm{LMS}_{\mathrm{Spear}}:99.9)$
(c) Additive Decoder (Median)  $(\mathrm{LMS}_{\mathrm{Spear}}:99.9)$
(e) Additive Decoder (Worst)  $(\mathrm{LMS}_{\mathrm{Spear}}:69.5)$
Figure 14: Figure (a, c, e) shows the learned latent space,  $\hat{\mathcal{Z}}^{\mathrm{train}}$ , and the corresponding reconstructed images of the additive decoder with the best/median/worst LMSpear among runs performed on the ScalarLatents dataset. Figure (b, d, f) shows the same thing for the non-additive decoder. The red dots correspond to latent factors used to generate the images and the yellow square highlights extrapolated images.
(b) Non-Additive Decoder (Best)  $(\mathrm{LMS}_{\mathrm{Spear}}:99.9)$
(d) Non-Additive Decoder (Median)  $(\mathrm{LMS}_{\mathrm{Spear}}:76.1)$
(f) Non-Additive Decoder (Worst)  $(\mathrm{LMS}_{\mathrm{Spear}}:39.8)$