SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we present additional results demonstrating the advantages of hierarchical VAEs versus non-hierarchical ones, which support the results in the main text. All models explored in this section were trained on fixate-1. For both cNVAE and VAE, we chose  $\beta$  values that maximized their informativeness (we discuss this choice in section 9.5). Specifically, we set  $\beta = 0.15$  for cNVAE and  $\beta = 1.5$  for VAE. These same models trained using the respective  $\beta$  values were also used for results presented in Fig. 3 and Fig. 4 in the main text.
10.1 Individual cNVAE latents exhibit greater linear correlation with ground truth factors
In the main text (Fig. 4) we demonstrated that the cNVAE latent representations  $z$  contained explicit information about ground truth factors  $g$ . We achieved this by training linear regression models that map the 420-dimensional latent space to each ground truth factor. Here, we investigate the degree that individual latents  $z_{i}$  are (Pearson) correlated with individual factors  $g_{j}$  (Fig. 11). For some ground truth factors, we find that cNVAE learns latents that are almost perfectly correlated with them. For instance,  $r = -0.91$  for  $F_{x}$  and  $r = 0.87$  for  $F_{y}$ , compared to modest correlation values of  $r = 0.34$  and  $r = 0.38$  for VAE. The higher linear correlation is consistently observed for cNVAE over VAE. Especially, we note that the VAE does not capture object-related variables at all, consistent with other results in the main paper (Figs. 3 and 4).
For the cNVAE, the indices for the selected highly correlated latents are as follows: [296, 248, 393, 284, 368, 92, 206, 105, 338, 60, 63], ordered left-to-right in the same manner as in Fig. 11. Here, an index of 0 corresponds to the very top latent operating at the spatial scale of  $2 \times 2$ , and an index of 419 corresponds to the very bottom one at scale  $8 \times 8$  (see Fig. 2). By considering the hierarchical grouping of these latents (see dashed lines in Fig. 3) we see that the majority of the highly correlated latents operate at the spatial scale of 8, with the exception of object-related ones corresponding to  $X_{obj}$ ,  $Z_{obj}$ ,  $V_{obj,y}$ , and  $V_{obj,z}$  which operate at  $4 \times 4$ .
In conclusion, Fig. 11 revealed that cNVAE learns individual latents that are highly correlated with individual ground truth factors, which provides complementary insights as to why cNVAE is so successful in untangling (Fig. 4). However, we note that a high correlation with individual latents is not the best way to capture the overall functional organization, which is better reflected for example in the mutual information heatmap of Fig. 3.
Figure 11: For each ground truth factor and model, we identified a single latent variable that displayed the highest (Pearson) correlation with that variable. The ground truth factors exhibited greater linearity within the latent space of cNVAE compared to VAE. Scatter points correspond to data samples from the test set. x-axis, the value of ground truth factors; y-axis, the value of latent variables.
10.2 Generated samples: non-hierarchical VAE completely neglects objects
If a generative model has truly learned the underlying structure of a dataset, then it should be able to generate fake samples that look like its training data. Here, we examine generated samples from both the cNVAE and VAE models, as illustrated in Fig. 12. For comparison, we also include five samples randomly drawn from the fixate-1 dataset that these models were trained on (Fig. 12, first row). The cNVAE samples appear indistinguishable from true samples; however, VAE fails to capture and generate objects, corroborating our previous results (see Fig. 3, Fig. 4, and Fig. 11). This striking divergence between cNVAE and VAE indicates that a hierarchically organized latent space is necessary for effectively modeling datasets that contain multiple interacting spatial scales.
10.3 Latent traversal reveals multi-scale understanding in cNVAE
One desired property of VAE latent representations is that continuous interpolation along a latent dimension results in interpretable effects on the corresponding reconstruction. Here, we provide visual examples of latent traversal in cNVAE, focusing on a latent dimension that effectively captures  $Y_{obj}$  (Figure 13). These examples demonstrate intriguing interactions between object velocity patterns and self-motion within the scene. Notably, in some examples such as rows 0 and 6, relocating the object from the bottom to the top of the scene induces a significant change in its linear velocity along the y-axis. This phenomenon arises due to the influence of self-motion, as seen in the background.
Figure 12: The first row shows random samples drawn from ROFL fixate-1 category. The second and third rows are samples generated from cNVAE and VAE models that were trained on fixate-1. Generated samples from cNVAE closely resemble real data; whereas, VAE ignores objects. Samples were generated at a temperature of  $t = 1$  without cherry-picking.
These illustrations collectively showcase that flow vectors in the visual scene are shaped both by local variables, such as the object's physical location, and their interactions with global-influencing variables like self-motion. By comprehending data across scales, cNVAE is able to capture such hierarchical and multi-scale properties inherent in naturalistic optic flow patterns. In contrast, non-hierarchical VAEs exhibit a complete failure in this regard.
10.4 All models achieve good reconstruction performance
All models considered in this paper are trained based (in part or fully) on reconstruction loss, corresponding to how well they could reproduce a presented stimulus. Here we report the models' reconstruction performance, to ensure that they are all able to achieve a reasonable performance. Figure 14 illustrates how the reconstruction loss (EPEPD, endpoint error per dim) and negative evidence lower bound (NELBO) depend on different values of  $\beta$  and model types. Both hierarchical and non-hierarchical VAE models perform well, with cNVAE exhibiting slightly superior performance for  $\beta < 0.8$ .
10.5 Untangling and reconstruction loss are strongly anti-correlated for cNVAE but not VAE
In this work, we demonstrated that a virtue of hierarchical VAEs is that they make information about ground truth factors easily (linearly) accessible in their latent representations. We were able to quantify this relationship because we used a simulation (ROFL) where ground truth factors were available. However, such a condition is not necessarily the case in real-world datasets that typically lack such direct knowledge [22].
To address this, we examined the relationship between informativeness and reconstruction loss, as well as NELBO, depicted in Fig. 15, to test whether there is a relationship between them. Strikingly, we observe a significant and strong negative linear relationship between reconstruction loss and informativeness in cNVAE ( $r = -0.91$ ,  $p = 8.9 \times 10^{-7}$ ,  $t$ -test). Conversely, no such correlation is observed for VAE ( $r = 0.10$ ,  $p = 0.72$ ,  $t$ -test). Notably, this relationship is absent for both models when comparing informativeness with NELBO (Fig. 15). These findings suggest that reconstruction
Figure 13: Latent traversal performed using a latent variable that effectively captures  $Y_{obj}$ . This variable belongs to a latent group operating at the spatial scale of  $2 \times 2$  (see Figs. 2 and 3; table 4).
loss—exclusively in the case of cNVAE—can serve as a viable proxy for informativeness, even in the absence of ground truth factors in real-world datasets.
10.6 Brain alignment requires more than just linear regression performance
In the field of neuroscience, linear regression  $R^2$  scores have commonly been employed to measure alignment between biological and artificial neural networks [30, 31, 34, 36]. However, we found
Figure 14: Both models demonstrate robust reconstruction performance, with cNVAE exhibiting a slight improvement. EPEPD, endpoint error per dim (lower is better). NELBO, negative evidence lower bound (lower is better).
Figure 15: Strong anti-correlation between reconstruction loss and informativeness (i.e., untangling) observed only for cNVAE. Each scatter point corresponds to a model trained with a different  $\beta$  value. EPEPD, endpoint error per dim (lower is better). NELBO, negative evidence lower bound (lower is better).
that relying solely on regression performance provided a weak signal for model selection, which is consistent with recent work [36-38, 112, 125]. In contrast, our alignment score defined in Fig. 8a provided a much stronger signal for differentiating between models (compare Fig. 7 and Fig. 8c).
To investigate this further, we conducted statistical tests, comparing pairs of models based on either their performance in explaining the variance of MT neurons or their brain-alignment scores. The results are presented in Table 5. We observed that ridge regression performance was not particularly effective in distinguishing between models. We measured the effect sizes between distributions of  $N = 141$  neurons, measuring how well ridge regression performance (Fig. 7) or alignment scores (Fig. 8c) discriminated between models. We found that the effect sizes were substantially larger for the alignment scores (Fig. 16). These findings suggest that while model performance is necessary, it is not sufficient for accurately measuring alignment between artificial and biological representations. In conclusion, refined and more comprehensive approaches are required to properly capture brain alignment [112-115].
Figure 16: Effect sizes are shown for the two different approaches in model comparisons: using ridge regression Performance (related to Fig. 7), versus, Alignment scores (related to Fig. 8c). Alignment demonstrates a substantially larger effect size, leading to a more pronounced differentiation among the models. Positive values indicate a larger value for hierarchical models (i.e., cNVAE > VAE or cNAE > AE). For different  $\beta$  values, model Performance difference fluctuates between positive and negative values. In contrast, the difference in model Alignment scores is consistently positive, indicating a greater brain alignment for hierarchical models.
10.7 Dissecting cross-validated model performance on neural data
One measure of neural alignment that we consider here used unsupervised latents to predict experimentally recorded neural responses (via linear regression). The neurons in this dataset had long presentations of optic flow stimuli, but only a fraction of the neurons (105 out of 141 neurons) had multiple repeats of a short stimulus sequence. Because these repeats facilitated a much clearer measure of model performance, we used these where available, following previous work [34]. However, due to the short stimulus sequence ( $\sim$ 5 sec; see Fig. 6b) used to estimate performance in the neurons with stimulus repeats, here we perform additional tests to verify that our conclusions hold using a more rigorous cross-validation procedure.
Specifically, here we used the long stimulus sequence to both train model parameters (80% of data) and determine hyperparameters (including regularization and optimal latencies) using a separate validation set (20%). We then predicted model performance on the repeats. This is in contrast to our approach in the main text, which used the repeat data itself to optimize hyperparameters, but as a result, risked overfitting on the short stimulus sequence. Indeed, we found that independent determination of hyperparameters (from the validation set) resulted in decreased model performance on the test set (Table 6) for both cNVAE and VAE models, although with comparable effect sizes. As a result, our conclusions of overlapping performance between cNVAE and VAE with a slightly better performance of the cNVAE were robust.
β	Performance (Figure 7)	Alignment scores (Figure 8c)
effect size	p-value	significant?	effect size	p-value	significant?
ae	-0.23	0.01	✓	1.2	8e-29	✓
0.01	-0.28	0.002	✓	0.32	0.0004	✓
0.1	-0.47	2e-07	✓	0.12	0.2	✗
0.15	-0.16	0.07	✗	0.46	4e-07	✓
0.2	-0.14	0.1	✗	0.33	0.0003	✓
0.3	-0.012	0.9	✗	0.69	4e-13	✓
0.4	-0.15	0.09	✗	0.65	7e-12	✓
0.5	-0.013	0.9	✗	0.65	7e-12	✓
0.6	0.29	0.001	✓	0.69	4e-13	✓
0.7	0.16	0.08	✗	0.71	1e-13	✓
0.8	0.44	1e-06	✓	0.73	4e-14	✓
0.9	0.24	0.008	✓	0.96	1e-20	✓
1.0	0.0016	1	✗	0.89	1e-18	✓
1.5	0.17	0.07	✗	1.2	3e-28	✓
2.0	0.037	0.7	✗	0.91	3e-19	✓
5.0	-0.24	0.008	✓	1.3	7e-31	✓
10.0	-0.056	0.6	✗	0.55	3e-09	✓
Table 5: Hierarchical architectures are significantly more aligned with MT neurons compared to non-hierarchical ones. Paired  $t$  -tests were performed, and  $p$  -values were adjusted for multiple comparisons using the Benjamini-Hochberg correction method [138]. Effect sizes were estimated using Cohen's  $d\left\lbrack  {139}\right\rbrack$  ,where positive values indicate cNVAE > VAE (or cNAE > AE, first row).
Model	Performance, R (μ ± se; N = 105)
β = 0.5	β = 0.8	β = 1	β = 5
cNVAE	.452 ± .025	.479 ± .025	.486 ± .024	.462 ± .025
VAE	.461 ± .027	.401 ± .029	.466 ± .028	.461 ± .027
cNAE	.460 ± .026
AE	.412 ± .031
Table 6: Model performance on the subset of neurons for which we had repeat data ( $N = 105$ ). Here, we used the repeat data as a held-out test set. We found that model performances remain statistically indistinguishable, providing additional support for our main results.
11 Additional Discussion
11.1 Visual-only cues and unsupervised learning are sufficient for untangling optic flow causes
It is well known that an observer's self-motion through an environment can be estimated using optic flow patterns alone [80], although it has been proposed that the brain also uses a combination of visual and non-visual cues [140], such as efference copies of motor commands [141], vestibular inputs
[142], and proprioception [143]. Indeed, using optic flow alone can be complicated by rotational eye movements and independently moving objects [48, 144, 145]. How can the brain estimate the motion of an object when self-motion is non-trivially affecting the entire flow field [146, 147], including that of the object itself? This is a critical motion processing problem [148], manifested in diverse ecological scenarios such as prey-predator dynamics, crossing the street, or playing soccer. Are retinal visual-only cues sufficient to extract information about different self-motion components, as well as object motion [149]?
Unlike the approach of Mineault et al. [34], our work is based on the premise that retinal-only signals contain sufficient information about their ground truth causes and that these factors can be extracted using unsupervised learning alone. In contrast, Mineault et al. [34] trained their DorsalNet by providing dense supervision on self-motion components, arguing that such supervised training is biologically plausible since these signals are approximately available to the observer from vestibular inputs and other such extra-retinal sources.
Our results (Fig. 4) provide proof-of-concept that it is possible to untangle ground truth causes of retinal optic flow in a completely unsupervised manner, using the raw optic flow patterns alone. Comparing the performance of alternative models revealed two essential ingredients for the success of our approach: (1) an appropriate loss function—inference [1]; and, (2) a properly designed architecture—hierarchical, like the sensory cortex [3].
11.2 Comparing cNVAE to NVAE, its non-compressed counterpart
In Fig. 4, we investigated the performance of the original non-compressed NVAE with an approximately matched latent dimensionality (440, slightly larger than 420 for cNVAE: more details are provided in section 9.3). We found that although NVAE did better than non-hierarchical VAE, it still underperformed cNVAE. This is because trying to match the latent dimensionality of NVAE with cNVAE necessitates reducing the number of hierarchical latent groups in the NVAE since most of its convolutional latents capture spatial dimensions rather than more abstract features as cNVAE does. From previous work [71], we know that the "stochastic depth" of hierarchical VAEs is the key reason behind their effectiveness; therefore, it was expected that reduced depth would hurt an NVAE with a matched number of latents.
It is worth noting that cNVAE and NVAE had a balanced performance across all ground truth factors (Fig. 4), in that they captured both global (self-motion-related) and local (object-related) aspects well. In contrast, non-hierarchical VAE completely ignored object-related variables and focused solely on the global scale determined by fixation point and self-motion. This suggests that hierarchical architecture is crucial for a more comprehensive understanding of multi-scale datasets such as ROFL.
11.3 Disentanglement is in the eyes of the beholder
The conventional definition of disentanglement in machine learning relies on relating the latent representation to predefined generative factors, a choice that is often not uniquely determined, and is often guided by heuristics [18]. For example, in the present work, we chose to represent velocities in Cartesian coordinates, rather than polar coordinates and judged the degree of disentanglement based on the relationship of the latents to these factors. Notably, these coordinate systems are related by a nonlinear transform, and latents linearly related to one coordinate system will then be less linearly related to the other.
Throughout our experiments, we observed one or more cNVAE latent variables that exhibited equally high mutual information with all components of self-motion velocity in Cartesian coordinates (e.g., latent dimensions 154 and 122 in Fig. 3). When we translated the generative factors into polar coordinates and looked at their relationship with these latents, we found that these latents were solely encoding the magnitude of self-motion, irrespective of its direction (Fig. 17). Thus, these seemingly highly entangled latent variables would have achieved almost perfect disentanglement scores had we employed polar coordinates instead of Cartesian coordinates to represent the ground truth velocities ( $d_{i} \approx 0.38 \rightarrow d_{i} \approx 0.81$ ).
Our findings shed light on the fact that disentanglement cannot be considered an inherent property of objective reality; rather, it is significantly influenced by our a priori determination of independent factors of variation within the data. Given the inherent complexity and richness of natural datasets, it
Figure 17: This latent variable cares about the magnitude of self-motion only. Its disentanglement score depends strongly on what we choose a priori as our independent factors of variation in data.
becomes challenging to unequivocally determine the true factors of variation a priori, pointing to the need for a more systematic approach.
When it comes to interpreting brain functions, perhaps considering the behavioral relevance of generative factors could guide our choices. For example, an animal directing its own movements does not pick grid coordinates but rather chooses direction and speed, possibly making a polar representation more "behaviorally relevant" to representations within its brain.
Another promising approach lies in focusing on the symmetry transformations inherent in the world [18, 150]. By seeking representations that respect the underlying geometrical and group-theoretical structures of the world [151, 152], and comparing them in principled ways [112-114], we may uncover new perspectives that provide valuable insights beyond the conventional and subjective notions of disentanglement.
12 Additional Background & Related Work
The deep connections between VAEs and neuroscience is explored in a review by Marino [17], but work looking at the synergy between neuroscience and VAEs has thus far been limited to the following recent papers.
Storrs et al. [15] trained PixelVAE [153] models on synthesized images of glossy surfaces and found that the model spontaneously learned to cluster images according to underlying physical factors like material reflectance and illumination, despite receiving no explicit supervision. Most notably, PixelVAE mimicked human perceptual judgment of gloss in that it made errors that were similar to those made by humans.
Higgins et al. [16] trained  $\beta$ -VAE [85] models on face images and evaluated the models on predicting responses of individual inferotemporal (IT) neurons from Macaque face patch. They found that  $\beta$ -VAE discovers individual latents that are aligned with IT neurons and that this alignment linearly increases with increased beta values.
Csikor et al. [68] investigated the properties of representations and inference in a biologically inspired hierarchical VAE called Topdown VAE that is capable of reproducing some key properties of representations emerging in V1 and V2 of the visual cortex. The authors showed that architectures that feature top-down influences in their recognition model give rise to a richer representation, such that specific information that is not present in mean activations becomes linearly decodable from posterior correlations.
Finally, Miliotou et al. [69] learned mapping from the latent space of a hierarchical VAE to fMRI voxel space. This allowed the use of the trained decoder to reconstruct images directly from brain responses. Importantly, they perform ablation experiments and find hierarchy is an essential component of their models.
13 Appendix: ROFL derivations and detailed description
ROFL aims to generate synthetic optical flow patterns that closely resemble those arising from ecologically relevant types of motion. This entails the consideration of both self-motion and object-motion components. Specifically, the self-motion component encompasses various subtypes such as translational and rotational motions induced by eye or head movements. Furthermore, it is crucial to have control over the relative prominence of these two types of motion. Depending on the specific configuration, the resulting flow pattern can exhibit an object-dominated characteristic when an abundance of objects is incorporated, or conversely, a self-motion-dominated nature if fewer or no objects are present. By carefully manipulating these factors, ROFL enables the generation of optical flow patterns that faithfully capture the dynamics of real-world scenarios involving an interplay between self-motion and object motion.
13.1 Setup
Self-motion is meaningful if the observer is moving relative to a static background (i.e., the environment). We consider a solid background wall at a distance  $Z = 1$  from the observer which extends to infinity in the  $X - Y$  plane. Define the following coordinate frames
1. Fixed coordinates, represented with capital letters:  $(X,Y,Z)$
2. Observer's coordinates, represented with lowercase letters:  $(x,y,z)$
Both coordinate systems are centered around the observer. The difference is that  $\hat{Z}$  is always perpendicular to the background plane, while  $\hat{z}$  is defined such that it is parallel to the fixation point.
All points in the space are initially expressed in the fixed coordinates. For example, any given point like  $\vec{P} = (X,Y,Z)$  has its coordinates defined in the fixed system. The same is true for fixation point,  $\vec{F} = (X_0,Y_0,Z)$ , and the observer's position  $\vec{O} = (0,0,0)$ , and so on. See Fig. 18.
13.1.1 Relating the two coordinate systems
How do we go from the fixed coordinates to the observer's coordinates? We need to find a rotation matrix that maps the  $\hat{Z}$  direction to the direction defined by the fixation point. By definition, this will determine  $\hat{z}$  and all the other directions in the observer's coordinates. First, let us express  $\vec{F}$  using its polar coordinates:
$$
\vec {F} = \left(X _ {0}, Y _ {0}, Z\right) \equiv \left(\| \vec {F} \|, \Theta_ {0}, \Phi_ {0}\right), \tag {19}
$$