SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to statistically verify that the learned projections to the latent space were meaningful, we next tested how well the projection matrices  $W_{i}$  fit to one half of the data generalized to the other half of the data. We identified events in the testing data for both the neural and stimulus data, and measured the degree of clustering for corresponding events between fMRI subjects and between the fMRI group-average and the stimulus. An example of the event representations in the latent space is shown in Figure 6, illustrating that event patterns are projected into event-specific clusters. The event clustering for the fMRI subjects was substantially above a random-projection baseline in both regions ( $p < 0.001$ ). The match between the neural representations and the stimulus representations was also significantly above baseline in both regions (Angular gyrus  $p < 0.001$ ; PMC  $p = 0.0027$ ). These results demonstrate that the learned mapping to a latent semantic space provides meaningful alignment between brains and provides a correspondence between brain data and stimulus features.
Figure 5: Learned projection of voxels and stimulus features, for the first semantic dimension. Fitting the H-HMM to the entire fMRI dataset produced projection matrices  $W_{i}$  for each subject and for the stimulus embedding. The voxel patterns (for the first four subjects) and sentences associated with the maximally-informative learned semantic dimension are visualized for both the angular gyrus (left) and posterior medial cortex (right). The voxel maps are broadly similar across subjects, but have distinct fine-scale structures specific to each individual. The scatter plots show the association between each sentence and the first semantic dimension across the 2,445 sentences in the dataset, highlighting example sentences that are strongly correlated or anti-correlated with this dimension.
A key parameter of our model is the dimensionality  $D$  of the latent semantic space. Intuitively, performance should suffer if  $D$  is too small (preventing proper discrimination between semantically-different events) or if  $D$  is too large (resulting in latent event representations that are largely orthogonal, which will fail to capture semantically-meaningful relationships and will fail to generalize to new events). We varied  $D$  and again performed our analysis of held-out performance, fitting the model on half the data and measuring the degree of clustering on testing data here, including both fMRI and stimulus representations in our variance-explained clustering measure. We observed (Figure 7) that performance peaked for  $D = 3$ , with slightly reduced performance for  $D = 2$  (the minimum dimensionality for which the HMM can compute pattern correlations) and substantially reduced performance as  $D$  increased to 8. We conclude that  $D = 3$  provides the optimal number of semantic dimensions events in this dataset, though datasets that are very large or explore a broader set of semantic topics could require a higher-dimensional latent space.
5 Limitations, Future Work, and Conclusions
The Hyper-HMM makes several assumptions about the cognitive structure of naturalistic datasets that could be relaxed in future work. First, it assumes that all subjects and the stimulus proceed through the same linear sequence of cognitive states. This is a reasonable assumption for data collected during tasks such as watching or recalling movies with linear narratives [5]; but may be less accurate for tasks such as recalling a series of unrelated short videos [28]. Second, we assume that neural dynamics are relatively constant within cognitive states, with sharp jumps between states; this model would fail to capture dynamics in brain regions with brief responses occurring only at event boundaries, such as the hippocampus [3, 9]. We also assume that brain responses in different people as well as stimulus embeddings are all linearly projectable to a common semantic space. Future work could extend this projection to be non-linear, as has been explored for characterizing relationships between brain regions within individuals [15, 37].
Figure 6: Learned projections  $W_{i}$  successfully generalize to held-out data. The H-HMM was trained on half of the experimental runs, and then the  $W_{i}$  matrices were fixed and temporal-only event alignment was performed on the held-out test data. The left panels show an example of these representations for three events in one held-out scanning run, showing that different events are projected to different parts of semantic space and that the stimulus events are well-aligned with the fMRI events. The right panels show the degree of clustering across all held-out events, measured separately for the clustering between fMRI subjects and for the match between fMRI data and the stimulus events. For both measures and both ROIs, clustering is significantly better than a random-projection baseline, indicating that our learned projection matrices generalize across experimental runs. **  $p < 0.01$ , ***  $p < 0.001$ , computed as the fraction of null values greater than each result.
Figure 7: Clustering performance when varying the dimensionality of the shared semantic space. Rather than using  $D = 3$  as in all previous analyses, we tested the sensitivity of our results to the dimensionality of the shared latent space. For each choice of  $D$ , we computed the degree of clustering (among both fMRI events and stimulus events) in the latent space, and compared it to a random-projection baseline. We found above-baseline performance across a range of  $D$  values, but clustering for held-out data is best for  $D = 3$ .
Although our results and model-fitting have focused primarily on brain-stimulus alignment, our model can be expanded to alignment of cross-modal, or even cross-species, information in the brain. Current neuroimaging techniques each exhibit tradeoffs between spatial resolution and temporal resolution, and there have been several proposed approaches for combining information across experiments in which brain responses to the same stimulus were measured in different neuroimaging modalities [7, 8, 24]. The H-HMM could provide a new avenue for integrating the strengths of modalities with different spatial and temporal properties, even if measured in different groups of experimental subjects. Analogously, brain alignment across species, in particular between humans and primates, have provided insights into common representational processes [12, 25, 48], and the H-HMM could be applied to map cross-species representations into a common latent space.
The Hyper-HMM provides a highly-flexible framework for identifying and aligning event representations across people and to features of a stimulus. Our experiments demonstrate that fitting this model to an fMRI dataset is computationally and statistically feasible, and can learn meaningful mappings of brain responses and stimulus features into a stable semantic space. This model has broad applications to the rapidly-expanding field of studies using continuous naturalistic stimuli to study dynamic cognition [29, 46], including the characterization of individual differences in brain responses and the identification of representational properties of high-level brain regions.
We thank Meir Meshulam for discussions and guidance in incorporating his dataset in this project. In addition, we are grateful to the labs of Mariam Aly, Christopher Baldassano, and Jim Haxby for their feedback at multiple stages of presenting our analyses and ideas. Lastly, we thank the reviewers of this paper for their insights leading to improvements in our final version, especially in the design of the simulation experiments.
[48] T. Xu, K.-H. Nenning, E. Schwartz, S.-J. Hong, J. T. Vogelstein, A. Goulas, D. A. Fair, C. E. Schroeder, D. S. Margulies, J. Smallwood, M. P. Milham, and G. Langs. Cross-species functional alignment reveals evolutionary hierarchy within the connectome. NeuroImage, 223:117346, 2020. ISSN 1053-8119.
[49] J. M. Zacks, T. S. Braver, M. A. Sheridan, D. I. Donaldson, A. Z. Snyder, J. M. Ollinger, R. L. Buckner, and M. E. Raichle. Human brain activity time-locked to perceptual event boundaries. Nature Neuroscience, 4(6):651–655, June 2001. doi: 10.1038/88486. URL https://doi.org/10.1038/88486.
[50] J. M. Zacks, B. Tversky, and G. Iyer. Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology: General, 130(1):29-58, 2001. doi: 10.1037/0096-3445.130.1.29. URL https://doi.org/10.1037/0096-3445.130.1.29.
Running the H-HMM code
Refer to README.md for a detailed explanation describing how to reproduce the main fMRI experiments, including the preprocessing steps for the fMRI data.
Algorithm for fitting H-HMM
Algorithm 1 describes the H-HMM's fits to a single fMRI run. Fitting to all scan sessions is performed similarly, except that the forward-backward function is applied to each run separately. For full details see the README.md file in the attached code directory.
Algorithm 1 HyperHMM algorithm
Parameter:  $L =$  event length   
Parameter:  $D =$  latent dimension   
 $n =$  number of subjects   
 $T =$  number of timepoints   
 $V_{i} =$  number of voxels/features in subject/model i   
 $X_{i} =$  data for subject/model i  $(V_{i}\times T)$ $K\gets$  round  $(T / L)$ $W_{i}\gets N_{D\times V_{i}}(0,1)\forall i\in 1..n$ $G_{e}\gets$  mean[mean  $[W_iX_i]]\forall e\in 1..K$ $\sigma \gets 4$    
while log-likelihood is improving do for all subjects i do  $\eta_{i}\gets$  forward_backward(log_probs  $(W_{i}X_{i},G,\sigma))$ $E_{i,e}\gets \sum_{t}\eta_{i,t,e}X_{i,t} / \sum_{t}\eta_{i,t,e}\forall e\in 1..K$  end for  $G\gets$  PCA([  $E_{i}]_{1..n},D)$ $W_{i}\gets$  [ridge  $(E_i,G)]\forall i\in 1..n$ $\sigma \gets \sigma *0.98$    
end while