Hyper-HMM: aligning human brains and semantic features in a common latent event space
Authors: Caroline Lee
Jane Han
Ma Feilong
Guo Jiahui
James Haxby
Christopher Baldassano
Venue: NeurIPS 2023 poster (2023)
Keywords: Brain Imaging
Other Cognitive Science
Other Neuroscience

Abstract: Naturalistic stimuli evoke complex neural responses with spatial and temporal properties that differ across individuals. Current alignment methods focus on either spatial hyperalignment (assuming exact temporal correspondence) or temporal alignment (assuming exact spatial correspondence). Here, we propose a hybrid model, the Hyper-HMM, that simultaneously aligns both temporal and spatial features across brains. The model learns to linearly project voxels to a reduced-dimension latent space, in which timecourses are segmented into corresponding temporal events. This approach allows tracking of each individual's mental trajectory through an event sequence, and also allows for alignment with other feature spaces such as stimulus content. Using an fMRI dataset in which students watch videos of class lectures, we demonstrate that the Hyper-HMM can be used to map all participants and the semantic content of the videos into a common low-dimensional space, and that these mappings generalize to held-out data. Our model provides a new window into individual cognitive dynamics evoked by complex naturalistic stimuli.

Hyper-HMM: aligning human brains and semantic features in a common latent event space
Caroline Lee Department of Psychology Columbia University New York, NY 214353@columbia. edu
Jane Han Department of Psychological and Brain Sciences Dartmouth College Hanover, NH
Jane. Han. wwwddrtmouth.cnommmmmmmmmmm edu
Ma Feilong
Guo Jiahui
Center for Cognitive Neuroscience
Center for Cognitive Neuroscience
Dartmouth College
Dartmouth College
Hanover, NH
Hanover, NH GaauuiGGuo@dartmouth edu
Feilong. Ma@dartmouth edu
James V. Haxby
Christopher Baldassano
Center for Cognitive Neuroscience Dartmouth College
Department of Psychology Columbia University
Hanover, NH
New York, NY
j ames. V. w..abbydaartoouthhh.. edu
c. bawlaassanooololseeeeeeeeeooooee www.:/awwoommmmaommmmmomma.....e edu
Abstract
Naturalistic stimuli evoke complex neural responses with spatial and temporal properties that differ across individuals. Current alignment methods focus on either spatial hyperalignment (assuming exact temporal correspondence) or temporal alignment (assuming exact spatial correspondence). Here, we propose hybrid model, the Hyper-MMM, that simultaneously aligns both temporal and spatial features across brains. The model learns to linearly project voxels to a reduced- dimension latent space, in which timecourses segmented into corresponding temporal events. This approach allows tracking of each individual's mental trajec- tory through event sequence and for alignment with other feature spaces such stimulus content. Using an fMRI dataset in which students watch videos of class lectures. demonstrate that the Hyper-HMM can be used map participants and the semantic content of the videos into a common oww-iimnsiiona space, and that these mappings generalize held-out data. Our model provides window into individual cognitive dynamics evoked by complex naturalistic stimuli.
Introduction
Dynamic, continuous stimuli such as movies, stories, music, or educational videos evoke complex response patterns throughout the brain that be captured using functional magnetic resonance imaging (fMRI). In addition to cognitive processes shared by all participants, these responses reflect individual differences in perception, learning, and memory. These differences result in the same semantic concept being represented at varying points in time or by different spatial patterns of brain activity. An ongoing challenge preserve these heterogeneous dynamics across individuals while capturing shared cognitive processes underlying human intelligence.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

One line of research has focused on the temporal differences between participants. Previous work using a Hidden Markov Model (HMM) approach T has found that there are differences the timing of event representations across development 9 or from repeated exposure to a stimulus [27] and has measured relationships between event timing and stimulus interpretation in individuals [4]] This approach requires either making the assumption that the event-specific spatial activity pattern across voxels is identical across participants, or fitting completely separate models for each participant (preventing the identification of corresponding events across people).
separate category of approaches has focused on the spatial differences across participants, learning a functional alignment across participants that accounts for differences in functional architecture 16 [9]. However, these approaches assume that all participants' neural responses are exactly syn- chronized in time. This assumption may be strongly violated, especially when studying idiosyncratic cognitive processes such as learning complex educational material.
Another challenge when using naturalistic stimuli is identifying the stimulus features that are driving neural activity. A common approach is construct vector representation of particular class of features in the stimulus, and then construct linear "encoding" or "decoding" model that maps these vectors onto patterns of brain activity or vice versa [2] 34 47l. However, these models again face the same challenges of temporal and spatial alignment: they assume that the neural response to a stimulus is immediate (or occurs at short, fixed delay learned by the encoding model) and that all participants represent stimulus features with the same pattern of brain activity (or are restricted to modeling a single participant at a time).
Contributions. Here we propose the Hyper-HMM (H-HMM), an extension of the HMM from Bal- dassano et al. [ that can simultaneously capture temporal and spatial differences across participants. The model can also align across entirely different kinds of feature spaces, creating mappings between neural data and a semantic embedding of the stimulus. We validate our model using simulated data and by using IMRI data from individuals watching course videos for a Computer Science class [30]. We can successfully map both the human participants and an embedding model of stimulus into a common latent space of semantic events, and this mapping generalizes dataset. Our novel approach is broadly applicable to any neuroimaging dataset with continuous naturalistic stimuli, and can provide a new way of characterizing and studying individual differences in cognitive processes.
held-out videos in this
2 Background
2.1 Temporal alignment between brains and with stimuli
Although neural responses are assumed to be tightly temporally-locked to the stimulus during traditional experimental designs with discrete stimuli, this is often not the case for more abstract cognitive trajectories evoked in naturalistic experiments. Especially in high-level regions, we could see responses that lag behind the stimulus (if viewer takes time to comprehend an event), run ahead of the stimulus (if' a viewer can predict upcoming events (27). or are generated based on a past stimulus rather than an ongoing stimulus (such as during narrative recall 5) As a further complication, these temporal shifts could differ in meaningful ways across people 41 One approach for identifying& a temporal alignment across datasets is to use Dynamic Time Warping, which stretches and compresses the temporal axis of each dataset to maximize their alignment [44]. This technique makes relatively few assumptions about the structure of the temporal alignment, but does require that the datasets can be matched together at the level of individual timepoints (i.e. that every 1-2 seconds of the stimulus can be directly mapped to 1-2 seconds of the neural response).
An alternative approach is to assume that datasets can be segmented into discrete, cognitively- meaningful events on the timescale of tens of seconds minutes. The theory that people naturally perceive and remember continuous stimuli as individual events has been extensively studied in cognitive psychology [9 (50), and signatures of this event segmentation can also be measured using neuroimaging [ 49 This work therefore suggests that events are a natural unit for representing cognitive dynamics and for performing temporal alignment, and we can employ Hidden Markov Models (HMMs) to track these discrete cognitive states [ 27 45 Current HMMs for neuroimaging. however, assume that the neural signature of an event a specific spatial pattern of activation across voxels in region of interest (ROI) which is identical across all experimental subjects.
[9]50]

---

2.2 Spatial hyperalignment between brains
Since the early days of fMRI, researchers have recognized that the specific spatial arrangement of functional units in the brain differs across individuals, motivating the use of "localizer" tasks to identify the functional layout of specific brain 42). One kind of method for performing multi- subject analyses is to use responses to a long naturalistic stimulus to learn a linear transformation that maps between individuals, method called "hyperalignmen"" [8]] There are now many variations and applications of this approach [19] 22], including models that map subjects into shared lower- dimensional space [4 I [6]. Though these methods can capture spatial variation in functional responses, they assume perfect temporal alignment across participants in their neural responses.
2.3 Mapping between stimulus features and neural responses
Extensive work has focused on bidirectionally mapping between features of the external world and the human brain [4] [3) These can take the form of encoding models that predict brain activity patterns
as a function of stimulus features [21 31 33 45 47 decoding models that produce estimates of
external stimuli (or internal mental states) given brain activity [20, 32 35
hybrid models that capture structure among both stimulus features and brain activity [23]. These models assume that the temporal correspondence between the stimulus and neural responses is known, allow for small
fixed lag in the neural responses by including time-shifted versions of the stimulus features.
3 Methods overview
3.1 Model description and fitting
The Hyper-HMM (Figure combines the strengths of these existing methods, removing the strong assumptions of spatial or temporal alignment across individuals and temporal synchronization the stimulus. Instead, we require only much more abstract level of correspondence across datasets, assuming that: a) all subjects and stimulus models proceed through the same sequence of semantic states (events); b) each event e can be represented as a low-dimensional vector Ge E IR" (where the latent dimensionality D is a parameter of the model); c) each subject and model can be mapped into this latent space using a linear transformation W, (unique to each subject and model); d) in the latent space, all timepoints during which this subject or model was in event e should be correlated with Ge-
To model both these temporal and spatial assumptions, we employ a chain-structured Hidden Markoy Model in which all participants transition sequentially through a sequence of event states, in Baldassano et al. [I. For the observation model, we define
p(Xi, =)
(I)
2 m
When a participant or stimulus model is in state (event) 81, = e, the latent-space mapping of the spatial voxel pattern embedding vector (W,X.,) should be similar to the group-level event representation Ge- All fMRI and stimulus datasets are therefore linked through their transform matrices W, into common latent semantic space, which reflects between-event similarity structure shared by all fMRI subjects and with the stimulus features. The 2() function represents z-scoring (standardizing representations have zero mean and unit variance), so that the Euclidean distance measure is proportional to the Pearson correlation between W.X., and G.-
We fit the model by iteratively estimating the temporal alignment 8i,t for each dataset and then updating the shared event representations and the W; projection matrices, On each itcration, we project each subject's 's data and the stimulus features into the low-dimensional shared space using the current W; matrices. We apply the forward- backbarkkkrrr algorithm from Baldassano et al. [ in this shared space, probabilistically segmenting each timeseries into events corresponding to the patterns Ge. The resulting probabilities 11.1.e = p( = e|X,, W,, G) can then be used to estimate the subject-specifie voxel representations of each event E,,- Σ, X,, Σ, ni,t.e-
To update the group-level representations G. we stack the event patterns E, from all subjects and the stimulus (concatenating along the voxel/embedding dimension for each event, in (16) and then project down onto principal components. In order to ensure that the fMRI data and stimulus embedding have equal vote in the shared event representations, the stimulus event patterns are
3

---

fMRI responses
X2
Timepoints
Stimulus features
X1
Sentences
Xs
Voxels
Embedding
Embedding
G
Voxels
Events
Es
Voxels
Embedding
Events
W2 2
Latent
E1
Ws
Events
W1
Latent
Latent
Figure 1: Hyper-HMM alignment across multiple brains and stimulus features. The H-HMM temporally divides each subject's brain data X, into discrete events with subject-specific patterns E,, and temporally divides the stimulus embedding Xş into event patterns Es. The event patterns from all subjects and the stimulus are constrained to linearly project (through matrices W;) to a shared, low-dimensional representation G.
(a)
10
(b)
Event 1
Event
Event 3
Event
fMRI subject
Stimulus embedding
Figure 2: Measuring learned event representations in the data, a) The model fit to voxel patterns (top) and stimulus (bottom) features, successfully learning a temporal alignment into corresponding events (middle) despite temporal and representational differences between the two datasets. b) Plotting the first two dimensions of the latent-space projection of event representations for IMRI and stimulus datasets shows clustering into corresponding events.

---

weighted n times as strongly as each fMRI dataset. Finally, we update the transform matrices, W using a ridge regression predicting G from subject patterns (with a = 10). We repeat this process (while annealing the HMM observation model variance O.. across iterations [ until the model's log-likelihood stops improving.
After fitting, the model provides temporal segmentation into events that corresponds across subjects and between the subjects and the stimulus (Figure 2h). For each event, all subjects' spatial patterns of IMRI activity and the stimulus feature representation can be projected to the same region of latent space, through the learned matrices W; (Figure[p).
3.2 Validation with simulated data
We tested the model's ability to recover ground-truth event structure under varying degrees of noise using simulated datasets. We used the firisim utility [33 the BrainIAK toolbox [26] generate a pure-noise dataset in the angular gyrus with the same number of subjects, runs, and timepoints per run as in the IMRI data used in our main experiments (see below). We simulated an event-structured signal by assigning blocks of timepoints to events, with events varying in length between 12 and 18 timepoints across subjects. Within each event there was a constant event pattern signal; these patterns came from the real fMRI data, averaged in 15-TR increments. By combining the signal and noise together with varying weights, we produced simulated datasets with realistic spatial and temporal correlation properties along with ground-truth timing and event patterns within each subject. We used these datasets to test the model's s ability to recover subject-specific event boundaries and estimate subject-specific spatial projections that generalized across runs.
For each noise level, we trained the model on only one half of the runs, and then froze the projection matrices W; and performed temporal-only aligmment on the held-out half of the runs. Ideally, these projection matrices should produce event representations on these held-out runs that are highly similar (across subjects and the stimulus embedding) for each event in the latent semantic space. To assess the degree of event clustering, we computed the sum of squared errors (SSE) between the latent-space event pattern for each fMRI subject and the mean pattern for this event across all subjects. We normalized this to yield a "variance explained" measure, dividing it by the SSE between all fMRI event patterns and the global mean pattern and then taking one minus this ratio. If the event patterns for all subjects map onto identical latent representations, this value will be 1, and patterns that show no clustering into events will yield variance explained of O. We repeated this process with the training and testing sets reversed, and averaged the results.
Next, we tested the model's ability to identify ground-trude event boundaries at each noise level. After fitting the model to the full dataset, we identified event boundaries for each subject as timepoints at which the most probable event switched, and computed the fraction of ground-truth boundaries that matched (within one timepoint) one of these model-derived boundaries.
3.3 Applying to an fMRI dataset
We used fMRI data collected by Meshulam et al. 30, in which undergraduate students (n=19) taking an introductory Computer Science course watched some of the course lecture videos in the scanner. There were five scan sessions throughout the course of the semester, divided into 21 separate scanning runs, for a total of 197 minutes of data per subject. We hypothesized that variations in expertise and concept understanding would lead to relatively large amount of variability in neural responses across participants, making it ideal for testing our alignment approach. The fMRI data (TR 2 seconds, voxel size 3mm 3mm 3mm) was preprocessed using IMRIPrep [1], including mapping data to the cortical fsaverage6 surface, and then cleaned via linear regression to remove variance associated with motion, cerebrospinal fluid signals, white matter signals, and low-fieqqeency drift. The timecourse of activity for each vertex was z-scored for each run.
We derived a semantic embedding of the lecture stimulus by transcribing the spoken dialogue using a speech recognition model (38) and then obtaining a vector representation of each sentence using a pre-trained transformer model 40 Note that we do not make use of any timing information about the correspondence between fMRI timepoints and stimulus sentences, giving the H-HMM full flexibility in assigning stimulus timepoints to neural responses in individual subjects.
5

---

Here we focus on data from the angular gyrus and posterior medial cortex (as defined in Baldassano al. (2), given their role in anticipating familiar narrative stimuli and learning schematic structures 27 We fit the H-HMM to all 21 fMRI runs, setting the number of events such that the average event duration was 15 TRs (30 seconds). Each run had a separate set of group-level events G, but the W; matrices were constrained to be the same across all runs (i.e. the W; matrices were computed via a single regression from all uubject/simmllusspeciifi events to all group-level events). For our primary analyses, chose latent dimensionality of D 3 for the group-level shared semantic space (see Figure 7 for an exploration of alternative dimensionalities).
We examined the learned projection matrices W; in both neural and stimulus space. For fMRI data, we visualized the first dimension of W; (which captured the largest share of the variance in semantic space) by left-multipliing it by the Gram matrix X,X? and then z-scoring. This produced weighted average the subject's spatial patterns, in which the weights were given by the similarity to the first dimension of W. We performed similar operation for the stimulus embeddings, measuring the correlation between individual sentence embeddings and the first column of X Ws. Sentences with particularly high or low correlations were those that were highly correlated or anticorrelated with this first semantic dimension.
To quantitatively evaluate these projection matrices, we used the variance-explained metric described in section 3.2 computing this value separately for 1) the clustering of the fMRI representations around their mean for each event and 2) the match between the stimulus representation and the mean fMRI representation for each event. As a baseline. we ran the same analysis. but used random projection matrices W; in which entries were sampled from a standard normal distribution N(0, 1). We also repeated this analysis while varying the dimensionality D of the latent semantic space.
Compute resources. Model fitting performed using maximum of 24 CPUs per fit. For fitting to the training data, compute time ranged from 12 - 16 minutes per split half, depending on the size of the ROL. The temporal-only alignment for the test data required minutes for each fit.
Results
4.1 Simulated data
In our experiments with simulated IMRI data, the H-HMM can effectively learn spatial alignments across subjects even under relatively high levels noise (Figure (1). Additionally, the model can successfully recover the majority of the ground truth event boundaries in each subject when the noise amplitude is up to twice as large as the signal amplitude (Figure Bp). These results demonstrate that the model and fitting procedure robust the noise properties of typical fMRI dataset.
(a)
(b)
Spatial alignment (variance explained)
0.7
0.9
Boundary recovery
0.6
0.8
0.7
0.4
0.6
0.3
0.5
2.0:1
1:1.0
1.1.5
1.20
1:25
1:3.0
1:3.5
1:4.0
2.0:1
1:10
1:1.5
1.2.0
1.2.5
1:3.0
1:3.5
1:4.0
Signal to noise
Signal to noise
Figure 3: Model validation in simulated data. The model exhibits high alignment in the latent projections of held-out fMRI events (i.e. high variance explained by the event clustering) until the noise amplitude several times larger than the simulated signal. b) The model can successfully recover ground-truth event boundaries for each subject up to high noise levels; the dotted line indicates the performance of null model that randomly selects timepoints event boundaries.

---

Angular Gyrus
Post Medial Cortex
fMRI Subjects
10
20
30
40
50
60
70
10
20
30
40
50
60
70
Timepoints (2s)
Timepoints (2s)
regular expressioos.
let's take reguler expressions. A protein string amino acids, s0 just a string consisting of those letters. Is in the language defined by these rules? from
And thet's well defined by these specific rules.
Stimulus
So, we might have the question, have given protein, is it, doas it follow these rules?
Here's another commercial computing.
And the answer in this case is yes.
this case, string the language.
So phrase, not an email addrress, it's not an email address.
10
20
30
Sentences
Sentences
Event
Event
Event
Event
Event
Figure 4: Example of learned temporal alignment of JMRI subjects and stimulus features. The H- HMM learns a segmentation of each neural timeseries and the stimulus embedding into corresponding events, with shared latent representations across brains and the stimulus. Here we visualize this temporal segmentation for the first half of one of the scan runs, for three of the subjects and the stimulus. The timing of transitions between cognitive states and the sharpness of these transitions vary across individuals. Sentences at the event transitions in the stimulus are shown, illustrating the semantic content present at the start of each event.
4.2 fMRI experiment
We fit the H-HMM to the full fMRI dataset for two brain regions (angular gyrus and posterior medial cortex) and examined the learned temporal and spatial alignment. An example of the temporal alignment for the first half of responses and the stimulus sentences are segmented into a shared set of events. This kind of semantic event segmentation is made possible by the learned projections W; that align all subjects and the stimulus embedding into common semantic space. A visualization of this alignment for the first amaximally-informatiee semantic dimension is shown in Figure 5 Note that the overall spatial topography corresponding to this dimension is similar across subjects (e.g. in angular gyrus, the posterior portion has generally positive weights while the anterior portion has generally negative weights) but the fine-scale details are different in each subject, reflecting individual variation in how this semantic dimension is spatially expressed. We can also examine the learned projection of the sentence embeddings onto this semantic dimension. The poles of this first dimension in angular gyrus appear to correspond approximately to simple concrete sentences versus abstract statements, and in posterior medial cortex to future-ories oriented statements versus present- tense descriptions of program structure. These axes of semantic variation have been observed in prior fMRI studies: Conca et al. 10 show differences in activity for concrete vs abstract and future- vs past- and present-tense sentences, and concrete vs abstract concepts engage different regions the semantic network (17).
is illustrated in Figure showing how each individual's 's neural
In order to statistically verify that the learned projections to the latent space were meaningful, we next tested how well the projection matrices W; fit to one half of the data generalized to the other half of the data. We identified events in the testing data for both the neural and stimulus data, and measured the degree of clustering for corresponding events between fMRI subjects and between the fMRI group-average and the stimulus. An example of the event representations in the latent space shown in Figure 6 illustrating that event patterns are projected into event- veen-specific clusters. The event clustering for the fMRI subjects was substantially above a random- projection baseline in both regions (p 0.001). The match between the neural representations and the stimulus representations was also significantly above baseline in both regions (Angular gyrus 0.001; PMC p 0.0027). These results demonstrate that the learned mapping to a latent semantic space provides meaningful alignment between brains and provides correspondence between brain data and stimulus features.
random--rojection
7

---

Angular Gyrus
Post Medial Cortex
Z Eanns
Subj
t fqns
Sabj
E tans
R fans
9 tqps
scmaartc
FIrSt somancic mmrorionnco (2)
hops that sometimes makes
can snite pregrams undirssand what it does.
funciliss, ot can more.
pretty sirplyy stractured or least has charactensties of semply derictured 68ts
program clearer and easer read.
Frrst suraaiic dimirsson (e)
Sontenee al algnmert to
Lecture
Lectuee
This one's not easy.
Hore's another indenending computetinn THAT we lafer on
That's cosk.
Figure 5: Learned projection of voxels and stimulus features, for the first semantic dimension. Fitting the H-HMM the entire fMRI dataset produced projection matrices W; for each subject and for the stimulus embedding. The voxel patterns (for the first four subjects) and sentences associated with the maximally maximally-informaatiy informative learned semantic dimension are visualized for both the angular gyrus (left) and posterior medial cortex (right). The voxel maps are broadly similar across subjects, but have distinct fine-scale structures specific to each individual. The scatter plots show the association between each sentence and the first semantic dimension across the 2,445 sentences in the dataset, highlighting example sentences that are strongly correlated anti-correlated with this dimension.
A key parameter of our model is the dimensionality D of the latent semantic space. Intuitively, performance should suffer if D is too small (preventing proper discrimination between semantically- different events) or if D is too large (resulting in latent event representations that are largely orthogonal, which will fail to capture semantically maaallly-m relationships and will fail to generalize to new events). We varied D and again performed our analysis of held-out performance, fitting the model on half the data and measuring the degree of clustering on testing data here, including both fMRI and stimulus representations in variance-explained clustering measure. We observed (Figure 7 that performance peaked for D = 3, with slightly reduced performance for = 2 (the minimum dimensionality for which the HMM performance as D increased to 8. We conclude that D = provides the optimal number of semantic dimensions events in this dataset, though datasets that very large explore broader set of semantic topics could require a higher-dimensional latent space.
compute pattern correlations) and substantially reduced
5 Limitations, Future Work, and Conclusions
The Hyper-HMM makes several assumptions about the cognitive structure of naturalistic datasets that could be relaxed in future work. First, it assumes that all subjects and the stimulus proceed through the same linear sequence of cognitive states. This is a reasonable assumption for data collected during tasks such as watching or recalling movies with linear narratives
(5); but may be less accurate for tasks such recalling a series of unrelated short videos (28) Second, we assume that neural dynamics are relatively constant within cognitive states, with sharp jumps between states; this model would fail to capture dynamics in brain regions with brief responses occurring only at event boundaries, such as the hippocampus [3 9. We also assume that brain responses in different people as well as stimulus embeddings are all linearly projectable to common semantic space. Future work could extend this projection to non-linear, as has been explored for characterizing relationships between brain regions within individuals [1 37
8

---

Latent space projections for three example events
Clustering performance on all test events
Variance explained
0.1
* *
Angular Gyrus
0.0
-0.1
-0.2
Event Event Event
fMRI subject
clustering
IMRI-stimulus match
Stimulus embedding
Variance explained
Post Medial Cortex
0.0
-0.1
-02
Figure Learned projections W; successfully generalize to held-out data. The H-HMM was trained on half of the experimental runs, and then the W; matrices were fixed and temporal-only event alignment was performed on the held-out test data. The left panels show an example these representations for three events in held-out scanning run, showing that different events projected to different parts of semantic space and that the stimulus events are well-aligned with the fMRI events. The right panels show the degree of clustering across all held-out events, measured separately for the clustering between fMRI subjects and for the match between f'RRI data and the stimulus events. For both measures and both ROIs, clustering is significantly better than random- projection baseline, indicating that our learned projection matrices generalize across experimental runs. ** p 0.01, *** 0.001, computed the fraction of null values greater than each result.

---

0.6
Angular Gyrus
Post Medial Cortex
Variance explained
0.4
0.2
Baseline
0.0
2
3
4
5
6 7 8
Shared dimensionality D
Figure 7: Clustering performance when varying the dimensionality of the shared semantic space. Rather than using D = 3 in all previous analyses, tested the sensitivity of our results to the dimensionality of the shared latent space. For each choice of D, we computed the degree of clustering (among both fMRI events and stimulus events) in the latent space, and compared it to candom-projection baseline. We found above-baseline performance across a range of values, but clustering for held-out data is best for D 3.
Although our results and model-fitiing have focused primarily on brain-stimulus alignment, our model can be expanded to alignment of cross-modal, even cross-species, information in the brain. Current neuroimaging techniques each exhibit tradeoffs between spatial resolution and temporal resolution, and there have been several proposed approaches for combining information across experiments in which brain responses to the same stimulus were measured in different neuroimaging modalities 77. 8 24 The H- H-HMM could provide new avenue for integrating the strengths of modalities with different spatial and temporal properties, even if measured in different groups of experimental subjects. Analogously, brain alignment across species, in particular between humans and primates, have provided insights into common representational processes
12 25 48), and the H-HMM could be applied to map cross-species representations into a common latent space.
The Hyper-HHee HMrM provides a highly-flcxible framework for identifying and aligning event represen- tations across people and to features of a stimulus. Our experiments demonstrate that fitting this model to an fMRI dataset computationally and statistically feasible, and can learn meaningful mappings of brain responses and stimulus features into stable semantic space. This model has broad applications to the rapidly-expanding field of studies using continuous naturalistic stimuli to study dynamic cognition (29 (46. including the characterization of individual differences brain responses and the identification of representational properties of high-level brain regions.
Acknowledgements
We thank Meir Meshulam for discussions and guidance in incorporating his dataset in this project. In addition, we are grateful to the labs of Mariam Aly, Christopher Baldassano, and Jim Haxby for their feedback at multiple stages of presenting our analyses and ideas. Lastly, we thank the reviewers of this paper for their insights leading to improvements in our final version, especially in the design of the simulation experiments.
10

---

References
[1] C. Baldassano, J. Chen, A. Zadbood, J. W. Pillow, U. Hasson, and K. A. Norman. Discovering event structure in continuous narrative perception and memory. Neuron, 95(3):709-721, 2017.
[2] C. Baldassano, U. Hasson, and K. A. Norman. Representation of real-world real-worrk event schemas during narrative perception. Journal of Neuroscience, 38(45):9689-9666) 2018. ISSN 0270- 6474.
[3] A. Ben-Yan Yakov and R. N. Henson. The hippocampal film editor: Sensitivity and specificity to event boundaries in continuous experience. J. Neurosci, 38047):10057-10066, Nov. 2018.
[4] N. Y. Bilenko and J. L. Gallant. Pyrcca: Regularized kernel canonical correlation analysis in python and its applications neuroimaging. Frontiers Neuroinformatiis, 10, Nov. 2016. doi: wttp:/fiinf.216.000000
URL httpt::tt///// org/ 10//10.33333 308.3389/280 3389/fninf fninf. 2016. 00049
[5] Chen, Y. C. Leong, C. J. Honey, C. H. Yong, K. A. Norman, and U. Hasson. Shared memories reveal shared structure in neural activity across individuals. Nature Neuroscience, 20 (1):115-125, Dec. 2016. doi: 10. 10.1838/nn.444 4450. URL BRLh:ttp:://00 org/ 10. 1038/nn. 4450
[6] P.-H. C. Chen, J. Chen, Y. Yeshurun, U. Hasson, Haxby, and P. J. Ramadge. A reduced- dimension fmri shared response model. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[7] R. Cichy, D. Pantazis, and A. Oliva. Resolving human object recognition in space and time. Nature neuroscience, 17, 01 2014.
[8] R. M. Cichy and A. Oliva. Am/eeg-fmn fusion primer: resolving human brain responses in space and time. Neuron, 107(5):772-781, 2020.
[9] S. Cohen, N. Tottenham, and C. Baldassano. Developmental changes in story-evoked responses in the neocortex and hippocampus. Elife, 11, July 2022.
[10] Conca, V. M. Borsa, S. F. Cappa, and E. Catricalà. The multidimennionalityy of abstract concepts: A systematic review. Neuroscience & Biobehavioral Reviews, 127:474-491, 2021.
[11] A. de Cheveigné, G. M. D. Liberto. D. Arzounian, D. D. Wong. J. Hjortkjær, S. Fuglsang, and L. C. Parra. Multiway canonical correlation analysis of brain data. Neurolmage. 186:728-744 740. Feb. 2019. doi: 10.1016/j nww.:j.nurrommmge.2.. 2018. 11.02......... URL https: org.oog///// 10 10. neuroimage 2018 11.8..1..0 11.026
[12] N. Eichert, F. C. Robinson, K. I.. Bryant, S. Jbabdi, M. Jenkinson, I., Li, K. Krug, K. E. Watkins, and R. B. Mars. Cross-species cortical alignment identifies different types of anatomical reorganization in the primate temporal lobe. eLife, 9:e53232, mar 2020. ISSN 2050-084X.
[13] C. T. Ellis, C. Baldassano, C. Schapiro, M. B. Cai,, and D. Cohen. Facilitating open-science with realistic fmri simulation: validation and application. PeerJ, 8:e8564, 2020.
[14] O. Esteban, C. Markiewicz, M. Goncalves, J. D. Kent, E. DuPre, R. Ciric, Salo, B. Pinsard, R. W. Blair, R. A. Poldrack, and K. J. Gorgolewski. IMRIPrep: robust preprocessing pipeline for functional MRI, Jan. 2022. httttttss///o...... wttp////....../o/oo..e.o 5898602
[15] M. Fang, C. Poskanzer, and S. Anzellotti. Multivariate connectivity: A brief introduction and open question. Frontiers in Neuroscience, 16, Jan. 2023. doi:
URI htR.https:: lattps:://olt org/ 10. 3w89/fnins...m..//.m.m fnwns.iiiiii..iiieeeiiiiinnoaoro 2022.1082120
[16] M. Feilong, S. A. Nastase, G. Jiahui, Y. O. Halchenko, M. I. Gobbini, and J. V. Haxby. The individualized neural tuning model: Precise and generalizable cartography of functional architecture in individual brains. bioRxiv, 2022. doi: 10.1 10000/2222.05.1.49922...
[17] M. Gilead, N. Liberman, and A. Maril. The language of future-thought: An fmri study of embodiment and tense processing, Neuroimage, 65:267-279, 2013.

---

[18] J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, M. Hanke, and P. J. Ramadge. A common. high- high-dimensional model of the representational space in human ventral temporal cortex. Newron, 72(2):404441, Oct. 2011. doi: 10.1016/j. neuron. 1. 1.08.026. URL http:////ooo..r///..o
j.wurnn..nnnnn.mmmmomemm 2011. 10.01.08.000. 026
[19] J. V. Haxby, J. S. Guntupalli, S. A. Nastase, and M. Feilong, Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. Elife, 9, June 2020.
[20] Horikawa and Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1), May 2017. doi: 10. 1e..0////ommss5555 1537.. URL https:// http://///o.../11..... 000m//ccmma/// 15037
[21] A. G. Huth, S. Nishimoto, A. T. Vu, and J. L. Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 7666):1210-1224 Dec. 2012. doi: 10. wttp:/i.eeuro.2212.10... 10.004, httpss//:://ddt org/ 10.1016/j neuron. 2012. 014
[22] G. Jiahui, M. Feilong, M. V. di Oleggio Castello, J. S. Guntupalli, V. Chauhan, J. V. Haxby, and M. I. Gobbini. Predicting individual face-selective topography using naturalistic stimuli. Neurolmage, 216: 16458, Aug. 2020. doi: 10.101 hew...oooimaee22011111. 201. 16658. URL https //doi.org// 1016/ neuroimage. 2019. 116458
[23] J.-R. King, Charton, D. Lopez-Paz, and M. Oquab. Back-to-back regression: Disentangling the influence of correlated factors from multivariate observations. Neurolmage, 220:1 17028. Oct. 2020. doi: 111.//....eoooiiiii neuroimage. 2020. 117028
170088. httls:ttss://dde htt///di.org//// 10. 10.o///....101111..
[24] D. Kozinska, F. Carducci, and K. Nowinski, Automatic alignment of ceg/meg and mri data sets. Clinical neurophysiology, 1 (2): 1553--1661, 2001.
[25] N. Kriegeskorte, M. Mur, D. A. Ruff, R. Kiani, J. Bodurka, H. Esteky, K. Tanaka, and P. A. Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey, Neuron, 60(6): 126-1 141, 2008. ISSN 0896-6273.
[26] M. Kumar, M. Anderson, W. Antony, C. Baldassano, P. Brooks, M. Cai, P.-H. C. Chen, C. T. Ellis, G. Saaeellman-Petrusete D. Huberdeau, J. B. Hutchinson, Y. P. Li, Q. Lu, J. R. Manning, A. C. Mennen, S. A. Nastase, H. Richard, A. C. Schapiro, N. W. Schuck, M. Shvartsman, N. Sundaram, D. Suo, J. S. Turek, D. Turner, V. A. Vo, G. Wallace, Y. Waang A. Williams, H. Zhang, X. Zhu, M. Capota J. D. Cohen, U. Hasson, K. Li, P. J. Ramadge, N. B. Turk-Browne, T. L. Willke, and K. A. Norman. BrainIAK: The brain imaging analysis kit. Aperture Neuro, (0) Feb. 2022. doi: 10.52294/31bb5b66 1188066--1144--- 2214-411411111 1e-11bbbao 8--00-00- -b-b:o--aaddaablllellllle decbólelda. 111.://.22224411112
URL htLphtts:://oo
2184- 8c00 210000aaddabbeeeddaa
[27] C. S. Lee, M. Aly, and C. Baldassano. Anticipation of temporally structured events in the brain. Elife, 10, Apr. 2021.
[28) H. Lee and J. Chen. Predicting memory from the network structure of naturalistic events. Nature Communications, 13(1), July 2022. doi: 10.1038//44166--022-333 2. URLhttpp //doi.org//il 1038/ 1038/ 941467-4477772 012-44477-222-11----2
[29] H. Lee, B. Bellana, and J. Chen. What can narratives tell us about the neural bases of human memory? Current Opinion in Behavioral Sciences, 32:111-119, Apr. 2020. doi: 10.1016/j. cobeha.2020.02.00 URL /tttps://doi..o org. 1016/ www.:////.................. wwweha.cbaacoooocc....o 2020. 02.0000..077
[30] M. Meshulam, L. Hasenfratz, H. Hillman, Y.-F. Liu, M. Nguyen, K. A. Norman, and U. Hasson. Neural aligmment predicts learning outcomes in students taking an introduction to computer science course. Nature communications, 12(1))1-14, 2021.
[31] J. Millet, C. Caucheteux, Y. Boubenec, A. Gramfort, E. Dunbar, C. Pallier, J.RR. King, et al. Toward a realistic model of speech processing in the brain with self-supervised learning. Advances in Neural Information Processing Systems, 55:33428-33445 2022.
12

---

[32] T. M. Mitchell, R. Hutchinson, R. S. Niculescu, F. Pereira, X. Wang. M. Just, and S. Newman. Learning decode cognitive states from brain images. Machine learning, 57(1-2):145-175 2004.
[33] T. M. Mitchell, S. V. Shinkareva, A. Carlson, K.-M. Chang, V. L. Malave, R. A. Mason, and M. A. Just. Predicting human brain activity associated with the meanings of nouns. science, 320(5880): 199-800)1111119988 2008.
[34] T. Naselaris, K. N. Kay, Nishimoto, and L. Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400 410,0000,,, 2011.
[35] K. A. Norman, S. M. Polyn, G. Detre, and J. V. Haxby. Beyond mind-reading: moxti-voxel pattern analysis of IMRI data. Trends in Cognitive Sciences, 1019))424-430, Sept. 2006. doi: t/016//..ics.2006.000
URL https:://oi..or//to..
tics. 2006. 07.005
[36] L. Paninski, J. Pillow, and Lewi. Statistical models for neural encoding, decoding, and optimal stimulus design. Progress in brain research, 165:493-507, 2007.
[37] C. Poskanzer and S. Anzellotti. Functional coordinates: modeling interactions between brain regions as points in a function space. Network Neuroscience, 6(4): 2022.
[38] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcl.eavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision, 2022.
[39] G. A. Radvansky and J. M. Zacks. Event boundaries in memory and cognition. Current Opinion in Behavioral Sciences, 17:133-140, Oct. 2017. doi: 10016//.coeea.2017...6 URL
https: org/10 10. cobeha. 2017 00.008.000 006
[40] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
org. www.a/wwwww...//////o/aaa attp//bb//1900
10084
[41] C. Sava-Segal, C. Richards, M. Leung, and E. S. Finn. Individual differences in neural event segmentation of contimuous experiences. Cerebral Cortex. Mar. 2023. doi: 10.1093/cercor/ bhad106. https:///// org/ 1ttgg/1..1900 1ww1100990rrreee http://erooobbable
[42] R. Saxe, M. Brett, and N. Kanwisher. Divide and conquer: A defense of functional localizers. Neurolmage, 3088) 1096. May 2006. doi: 10.1016/j new.oof.eeuommm..2000... 12.062. URL
https://doi http::///ooi.rg/11..11111 100 w0..///.....0//...... mwwroimageeeee..mm 2005. 12.022.22
[43] M. Schrimpf, I. A. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. B. Tenenbaum, and E. Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, :185461.221056661118 2021.
[44] L. J. Silbert, C. J. Honey, E. Simony, D. Poeppel, and U. Hasson. Coupled neural systems underlie the production and comprehension of naturalistic narrative speech. Proceedings of the National Academy of Sciences, 111(43), Sept. 2014. doi: 10. 00073//aas.132311222 URL 1073 /pas. http:/aa.....11111111113333 1323812111
https://doi.ox::
[45] M. Silva, C. Baldassano, and L. Fuentemilla. Rapid memory reactivation at movie event boundaries promotes episodic encoding. The Journal of Neurovcience. 39(43) 8548, Sept. 2019. doi: (0.1523/jneurosci.0k 03600 19.2019. 0360-19.2000 2019
org
100gg//1..11 10. 1523//neurosci. neurosci
[46] S. Sonkusare, M. Breakspear, and C. Guo. Naturalistic stimuli in neuroscience: Critically acclaimed. Trendy in Cognitive Sciences, 23(8):699-714, Aug. 2019. doi: 10.10166).tes..019. 05.004. https://t:://att
10100.0011....... tics. 2019. 05.00..000
[47] K. Vodrahalli, P.-H. Chen, Y. Liang, C. Baldassano, J. Chen, E. Yong, C. Honey, U. Hasson, Ramadge, K. A. Norman, and S. Arora. Mapping between fMRI responses to movies and their natural language annotations. Neuroimage, 180(Pt A):223-231, Oct. 2018.
13

---

[48] T. Xu, K.-H. Nenning, E. Schwartz, S.-J. Hong, J. T. Vogelstein, A. Goulas, D. A. Fair, C. E. Schroeder, D. Margulies, Smallwood, M. P. Milham, and G. Langs. Cross-species functional alignment reveals evolutionary hierarchy within the connectome. Newrolmage, 223: 17346, 2020. ISSN 1053-8119.
[49] J. M. Zacks, S. Braver, M. A. Sheridan, D. Donaldson, A. Z. Snyder, J. M. Ollinger, R. L. Buckner, and M. E. Raichle. Human brain activity time-locked to perceptual event boundaries. Nature Neuroscience. 4(6):651-655, June 2001. doi: 10.1038/88486. URL
https://0b..0rggg11011//////4
[50] M. Zacks, B. Tversky, and G. Iyer. Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology: General, 130(1))2--58, 2001. doi: 10. 10370 1096-3445.130.11 httL:/ttts://oii.or//1.00.000 wtt.///...rrg/111100///./// 3445. 130.1.29 13..
14

---

Appendix
Running the H-HMM code
Refer to README.mo for detailed explanation describing how to reproduce the main fMRI experiments, including the preprocessing steps for the IMRI data.
Algorithm for fitting H-HMM
Algorithm describes the H-HMM's fits to a single fMRI run. Fitting to all scan sessions is performed similarly, except that the forward-backwer function is applied to each separately. For full details see the README.nd file in the attached code directory.
Algorithm HMP algorithm
Parameter: L = event length
Parameter: D = latent dimension
number of subjects
T number of timepoints
V, number of voxels/features in subject/model i
X; data for subject/model i (V: x 7)
K + round (T/L)
W. NDx v, 10 vi I.n
G. - mean, [eean [W,X,]] Ve E 1..K
while log-likelihood is improving do for all subjects do
- forward backward(log Brrtoo__robbs,Wx,,,_ G,o))
Eic E, Mit. E. Mitte Ve 1.K end for
G PCAA(AF) I..,; D) W < [ridge( E,, G)] Vi 1..n
- a * 0.98
end while
15


Reviews:
[{"review_id": "WPWe4YxmhS", "reviewer": "Reviewer_1rZM", "summary": "In natural tasks that share input stimuli across participants, the cognitive states or neural responses of different participants might undergo approximately synchronous but slightly jittered dynamics. At the same time, the distribution of neural signals are not exactly consistent across participants at voxels with the same spatial coordinates. These two problems were addressed by two approaches separately by event segmentation and functional alignment. This paper proposes a new approach H-HMM that combines the advantage of both methods. It performed simulation to evaluate its performance and tested it on an fMRI datasets of college students watching the same series of lectures in computer science. The performance appears impressive and achieves what the model is designed for. Further, the paper also demonstrated alignment between fMRI data and semantic features of contents in the lecture.\n\n", "questions": "My major questions are asked in the weakness, which I think are addressable.", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Simultaneous achieving temporal and spatial alignment was not done before. \n* Comprehensive evaluation of the method is performed on both simulated and real data and showing good performance.\n* The illustration is generally clear and easy to understand.\n* The approach holds promise for a wide range of application and should be a significant contribution to the field", "weaknesses": "* I think there is a mismatch between the data size of simulated data and that of fMRI data or semantic features, making it a bit difficult to evaluate the expected performance of this method in general. The simulated data have only 5 voxels/features, which is rarely observed in the domain where this algorithm should be applied. The described numbers also do not match up: 4 events with 9-12 time points per event give rise to fewer than 48 time points, yet it is said that the simulated data have 60 time points. It is also strange that the event is represented by binary patterns. Maybe this is just for the purpose of easy visualization but I worry that the performance might depend on these unrealistic properties in the simulation. Doing it with similar property as would be expected for the fMRI data should be more convincing. If you encounter issue that the actual fMRI data have lower effective dimensionality due to smoothness, it may be simulated by spatial Gaussian process or other ways but I don't think it is justifiable to start with low-dimensional data, as you did not perform PCA on fMRI data to get similar dimensionality before application of the algorithm.\n* The description for the update of group-level G, starting from line 122, is a bit confusing. After the stacking, what is the dimensionality of the matrix? I assume the stacked data have a total number of elements as the number of events * (total number of voxels+ semantic features) * D (since E is in the D-dimensional space). If your stacked data have two dimensions (which I assume is the case since you can do PCA on it), which of the two dimensions (after explaining their size) do you treat as data features and which do you treat as samples in the definition of PCA? After projecting the stacked data for each event into D-dimensional PC space, do you need to do anything you do to get G (which is of the shape of number of events * D)? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) \nThe simulations were intended only to show proof-of-concept behavior for the model, demonstrating that the architecture and fitting procedure is able to capture varying temporal onset/offset of events and topographical differences across individuals while applying increasingly high spatial noise.Overall, simulating realistic fMRI data is an ongoing challenge, given the highly complex spatial and temporal correlations present in a real dataset. We hope to have access to simulated fMRI datasets that are more representative of real data sometime in the future.  \n\nThe 4 events were randomly selected in each individual to have anywhere from 12 - 18 time points per event. The range reported in the submission was from an earlier round of simulations and mistakenly used instead of the most recent parameters for the simulations. We aimed to ensure that each event would have high variance across individuals in terms of onset and offset times.\n\n2) \nThe voxel/feature vectors for each subject/model are concatenated together for each event, yielding a two-dimensional matrix with rows = events and columns = all concatenated voxels/features. After PCA, this long catenated dimension is reduced, yielding a matrix G where the dimensions are events x D (latent dimensionality). The values in the latent dimensions are the data features corresponding to each respective event. \n"}, {"review_id": "hUE3tye2bB", "reviewer": "Reviewer_jAF3", "summary": "UPDATE:\n\nI have raised my score and now support acceptance of this paper. I believe it will be a good contribution to the conference.\n\n-----------------------------\nThe authors propose an extension for an HMM model proposed by Baldassano and colleagues to align interindividually different brain responses in both space and time to analyze naturalistic stimulation settings. They test their approach through simulations and on an empirical dataset of 19 participants, who watched a series of 5 computer science lectures. They show that their model is robust to a number of noise levels in the simulation analysis. In the empirical analysis, the model appears to learn a meaningful latent space (concrete vs abstract and future-oriented vs present tense descriptions) and outperforms a null model.", "questions": "Overall, this work tackles an interesting problem and shows some first promising results, but I believe that it would require quite some work like more extensive simulations, comparisons to other methods and testing in a larger dataset or a second dataset (to demonstrate that this method is not only useful for applications to naturalistic stimulus analysis, but other areas of neuroscience research, for example aligning multiple imaging modalities) to meet the NeurIPS standard. \n\nNonetheless, I really enjoyed reading the paper and I strongly encourage the authors to continue this interesting work. Here are some questions/comments that would allow me to increase my score. I hope you find them helpful and constructive. \n\n**Major points**\n- The actual model could be explained in greater detail. It would be helpful to include pseudo-code for the algorithm and the cost function explicitly in the manuscript.\n- Please, explain (at least in the supplement) the choice of preprocessing, e.g. quality control (motion detection/scrubbing, motion artifact rejection thresholds), software versions, scanner/coil details, filtering, slice-time alignment if conducted etc. \n- The choice of the simulation parameters is not clearly motivated, can you explain why you chose to simulate datasets with only 6 participants or only 5 voxels, this does not correspond well to empirical situations, where the sample would be larger, and #voxels >> #stimuli, which would be important to test whether your reweighting (p.4 125-126) works. It would also be helpful to express the noise as SNR to assess whether your method assumes realistic noise.\n- For the empirical analysis, it is unclear how you construct your baseline. Please, be more specific about how you computed that, otherwise it is impossible to assess the model performance\n- Related to that Figure 5 is not very clear, I am assuming that the black dot and the black diamond in Fig 5 reflect the actual value, whereas the grey shaded area is the null-distribution? This should be clearly stated in the figure captions.\n- Figure 5: It is quite striking that the fMRI stimulus match is much worse on the empirical data than the clustering. Based on the simulations (Figure 2), this is unexpected. Could you elaborate on what the reasons for this could be? \n- Figure 5: The fMRI-stimulus match null-distribution(?) is centered on a negative R^2. Can you explain why this happens? Could this be suggesting that the baseline model is not appropriate?\n- Will code be provided to ensure reproducibility of the analyses?\n- p. 6 ll. 218-221: You state that the latent dimensions appear to map onto semantically meaningful dimensions (concrete vs abstract and future-oriented vs present tense descriptions). Is this in line with the literature? Further discussion of this would be helpful.\n- p. 9 l. 268-270: “Our experiments demonstrate that [...] this model […] can learn meaningful mappings” To show that these mappings are meaningful, it would be useful to show that the latent scores correlate with an external measure that was not shown to the model, but would be hypothesized to relate to the captured dimensions (e.g., IQ, exam scores, abstract thinking scores or something of that sort). Is any information like this available. That would strengthen your claims.\n\n\n**Minor points**\n- Could you elaborate on the relationship between your method and representational similarity analysis (https://doi.org/10.3389/neuro.06.004.2008)? You very briefly touch on related work (e.g. from Haxby), but I think this could be expanded a bit.\n", "limitations": "The authors have mentioned some limitations (linear assumptions), but the small sample size is not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The research problem addressed is an interesting and timely problem, namely how to compare spatially and temporally heterogeneous responses across participants\n- The paper is largely well-written\n", "weaknesses": "- The sample size is small\n- The approach is not compared to other state-of-the-art models, rather the authors compare it to a null model, which is poorly explained\n- The authors do not provide analysis code or sufficient detail to reproduce their results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) \nWe provided an algorithm (pseudocode) in the appendix submitted as part of our supplementary materials.\n\n2) \nWe used published data provided by Meshulam et al. here: https://openneuro.org/datasets/ds003233/versions/1.2.0 - please see the original dataset for details about the fMRI acquisition. We included a preprocessing script and directions for executing fMRIprep in the readme file in the supplementary material, and did not perform any additional preprocessing (e.g. slice-time alignment) beyond this.\n\n\n3) \nThe simulations were intended only to show proof-of-concept behavior for the model, demonstrating that the architecture and fitting procedure is able to capture varying temporal onset/offset of events and topographical differences across individuals while applying increasingly high spatial noise.Overall, simulating realistic fMRI data is an ongoing challenge, given the highly complex spatial and temporal correlations present in a real dataset. We hope to have access to simulated fMRI datasets that are more representative of real data sometime in the future. \n\n\n4) \nOur baseline was to use random projections for each subject/model into the latent space, with each element of the project matrix sampled from a standard normal distribution N(0, 1). This approach preserves the temporal structure for each subject (since this projection is held constant within subjects), while disrupting alignment between different subjects. \n\n\n5) \nYes, that is the correct interpretation of Figure 5. We only briefly mention the null distribution in the last line of the captions and should have indicated that they are represented by the gray shaded area. \n\n6) \nAligning fMRI data to the stimulus in the real dataset is a much more challenging task than fMRI-fMRI alignment. Rather than relating the same kind of data across people – which tends to have some similarities even before applying spatial or temporal alignment – this is an entirely different model with stimulus features that are only partially related to fMRI responses. This complexity is not straightforward to capture in our models, though note that at higher noise levels we do see performance that is reduced and more variable for the stimulus alignment (since there is only a single stimulus to align, as opposed to multiple subjects). \n\n7) \nWe would obtain R2 = 0 if the stimulus projections, on average, sit at the mean of the fMRI data. This means that random projections are likely to be below 0 (worse) since the stimulus projections will be randomly distributed with respect to the fMRI data. Because the fMRI data and stimulus model come from different representational spaces with differing dimensionalities, there is no simple baseline alignment between the two spaces.\n\n8) \nWe provided all code necessary to reproduce the entire pipeline from preprocessing the publicly available dataset (please see above response for preprocessing fMRI data) all the way to fitting the model in the supplementary material. \n\n9) \nThese axes of semantic variation have been observed in prior fMRI studies. Gilead et al., 2013 (DOI: 10.1016/j.neuroimage.2012.09.073) show differences in activity for concrete vs abstract and future- vs past- and present- tense sentences, and concrete vs abstract concepts engage different regions of the semantic network (Conca et al., 2021; DOI: 10.1038/s41598-021-02013-8). \n\n\n10) \nMapping the trajectories in the latent space to a behavioral measure would be a great validation of the method, but this would require a dynamic behavioral measure of second-by-second cognitive states; no such metric exists for this dataset, and it is also unclear how we could obtain this kind of measure without disrupting a student's comprehension of the material. Instead, we are arguing that our latent mappings are meaningful in the sense that they can generalize across runs (i.e. successfully align new fMRI-stimulus data). \n\n\nMinor 1) Similar to RSA, we make the assumption that event representations are linearly related across subjects and share the same similarity structure. However, RSA does not identify any temporal or spatial alignment, unlike work from Baldassano et al. or Haxby et al., respectively. \n"}, {"review_id": "W66Cdd25BN", "reviewer": "Reviewer_wrdu", "summary": "The authors develop a method to identify and align events in the brain and external stimulus. They iteratively fit a Hidden Markov Model to find the times of events, and spatial characteristics of those events.", "questions": "L235: \"resulting in latent event representations that are largely orthogonal, which will fail to capture semantically-meaningful relationships and will fail to generalize to new events.” I don’t understand. Orthogonal components in event relationships should be common and natural. Why is orthogonality considered as meritorious?\n\nThree dimensions is shockingly low dimensionality of semantic space. This seems to me like analyzing image data and find the dominant principal component, the sky.\n\nLinear projections onto semantic space is highly restrictive and unrealistic. But it could be a reasonable lower bound on alignment.", "limitations": "The authors do an excellent job of articulating their problem and the essence of their solution..", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This work is a minor update to past work, by accounting for time shifting as well.\nQuality: The authors use careful validation with simulated data, and careful cross-validation in real data.\nClarity: The paper is largely well written.\nSignificance: The main problem of comparing neural activity across subjects and referencing those patterns to interpretable stimulus-driven semantics is an important one, especially as the amount of data in the field continues to grow.", "weaknesses": "The events in this paper refer to extended time periods. Events are often described as instants in time, rather than extended periods. It would be helpful to carefully articulate the definition of events to prevent misunderstandings.\n\nThe authors make quite strong assumptions, despite protestations that their method is very general. In particular, I am skeptical about the \"constant events\" in the time course, and about the linear embedding of the semantic content.\n\nThe authors only test out a narrow range of timing differences (25%), which leads me toward greater skepticism about the generalizality of the author's results.\n\nSuggestion: It would be useful to test this method in a simulation with a deliberately time-warped movie, to check whether the method can recover simulated data can recover the true timercourse.\nMinor: Figure 2 caption has a typo: should read \"simulated data\", not \"stimulated data\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) \nHere we rely on a definition of events, and event segmentation, widely used in psychology and cognitive neuroscience. Continuous streams of information, as is the case in videos or text, can be divided into smaller and smaller chunks (e.g. a book consists of chapters, chapters are composed of paragraphs, paragraphs of sentences, sentences of words, so on and so forth). Psychologists use this definition to study the ways in which people organize meaningful chunks of information (Yates, Sherman, & Yousif, 2023; Zacks et al., 2007), and have found that this may be the foundation in remembering contextually- or semantically-linked information (Ezzyat & Davachi, 2011). With respect to temporally-constrained information, such as the class videos we use in our paper, an event consisting of semantically related adjacent timepoints is an extended period of time. \n\n2) \nOur simplifying assumptions are driven by the limitations of fMRI datasets, which have high levels of noise and generally consist of only ~1000 timepoint samples per subject. The “constant events'' we refer to in our paper depend on event stability, or a spatial (voxel) pattern in the brain that appears across adjacent time points. This pattern stability is viewed as the neural instantiation of a stable event representation (Antony, 2021; Baldassano et al., 2017; 2018). Brain representations are of course never fully static, but we can define events at the shortest timescale for which there are meaningful dynamics that generalize across subjects.The use of linear embeddings of the semantic content is a form of learning linear mappings from neuroimaging data to semantic spaces, a standard practice in psychology/cognitive neuroscience, such as when creating encoding models of the brain (Naselaris, 2011). \n\n\n3) \nThe simulations were intended only to show proof-of-concept behavior for the model, demonstrating that the architecture and fitting procedure is able to capture varying temporal onset/offset of events and topographical differences across individuals while applying increasingly high spatial noise. Overall, simulating realistic fMRI data is an ongoing challenge, given the highly complex spatial and temporal correlations present in a real dataset. We hope to have access to simulated fMRI datasets that are more representative of real data sometime in the future.\n\nQ1) Orthogonality across events would be indicative of events being equally distinct from each other, thus producing no meaningful event similarity structure shared across subjects. We believe that one strength of this model is the ability to learn a shared event space while preserving individual nuances. This aspect would be lost if the model were tuned to individuals without any regard to the shared event space.\n\nQ2) Given that the model is fit to data from single regions at a time, the low number of dimensions is consistent with pre-existing literature, such as seen with a commonly accepted fMRI dimensionality reduction algorithm, the Shared Response Model (SRM) <https://papers.nips.cc/paper_files/paper/2015/file/b3967a0e938dc2a6340e258630febd5a-Paper.pdf>.\n\nQ3) While linear projections onto a semantic space can be limiting at times, fMRI data itself is limited per subject. Our training and test datasets contained roughly 1000 timepoints on average, requiring the use of low-complexity models to accommodate the nature of the dataset. \n"}, {"review_id": "xkjJwwAwks", "reviewer": "Reviewer_jeCv", "summary": "This paper develops Hyper-HMM as a hybrid model that simultaneously aligns both temporal and spatial features across fMRI datasets. The proposed model learns a linearly project that maps voxels to a low dimensional latent space, in which timecourses are segmented into corresponding temporal events. The purpose of this model is to remove the effect of each individual’s mental trajectory through an event sequence, and to also align with other feature spaces like stimulus content. Overall, it is an interesting paper, however, there are several concerns about the machine learning novelty, validating the empirical studies, and the clear presentation of the proposed method.", "questions": "The followings are the major concerns and minor comments:\n\n\n1) It is still unclear to me as to what the novelty of this paper is in terms of machine learning. There may be some contributions to computational neuroscience in this paper; however, which part of the machine learning approach is new in this paper? The author(s) of the proposed paper should specify whether there is anything theoretically new for the proposed approach or if this is merely an application paper for HMM techniques. \n\n\n2) My second concern is the design of the empirical studies. Even though the study contains several beautiful figures, more numerical analyses would be helpful to convince the reader that the proposed method is effective. There is a lack of regular analysis and conventional machine learning metrics (such as accuracy, dice, etc.) which make it difficult to understand the results. \n\n3)  The proposed method should be benchmarked in comparison with related state-of-the-art techniques.\n\n\n4) In this paper, the notations are confusing. In regular papers, scalers are denoted by small letters, vectors are defined with small letters (highlighted in bold), and matrices are denoted by capital letters using bold. In this paper, there are a lot of conflicts. It is so hard to trace what is a set, a matrix, or even a distribution. \n\n\n5) There are some minor linguistic and typo problems in this paper.\n", "limitations": "Please refer to the question section\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Please refer to the question section\n", "weaknesses": "Please refer to the question section\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) \nOur model incorporates spatial alignment within the Baum-Welch update procedure for the temporal HMM, allowing for simultaneous spatial and temporal alignment. Although the components of this model are indeed taken from previous work, the combined model and its application to fMRI data are novel. We apply our approach to a dataset meant to maximally capture substantial variance across individuals learning and remembering complex pieces of information throughout an extended period of time (an entire semester), which has not been thoroughly examined in cognitive neuroscience/psychological sciences due to the methodological constraints we address in this paper.\n\n2) \nUnfortunately, our choices in evaluating the model deviated from conventional machine learning metrics. Because of the inherently difficult nature of obtaining “ground truth” labels of an individual’s true cognitive state on a second-by-second basis, metrics from supervised learning are not applicable here. We use conventional machine learning metrics such as R2 to test how well projections generalize across runs.\n\n3) \nAs of yet, there are not any suitable and related state-of-the art techniques against which we can benchmark our current model. Although there are existing spatial alignment, temporal alignment, and brain-stimulus encoding models available, none of these models have explored all three (spatial, temporal, stimulus feature) alignments simultaneously. This prevents us from applying them in cases when both temporal and feature dimensions are not aligned, such as for mapping between fMRI data (timepoints x voxels) and semantic models (sentences x features). \n\n4) \nOur notation in the main text uses capital letters for matrices (such as the weight matrix, data matrix, etc.) and lowercase letters for scalars, with the exception of the scalar dimensionality D. \n\n5) \nWe have identified a couple typos in the manuscript after the submission deadline; we hope that none of these caused confusion in understanding the paper. \n"}, {"review_id": "C8mkuIdM7D", "reviewer": "Reviewer_XnCP", "summary": "The authors propose an HMM-based model, Hyper-HMM, for characterizing variability in temporal and spatial dimensions in fMRI sequence datasets. The model is a chain-structured HMM where each discrete state (event) defines a relationship between neural activity and a stimulus embedding. Importantly, the discrete state defines the mean of the subject's neural activity projected to a lower-dimensional space. Each subject has a different lower-dimensional projection matrix, while the events and stimulus embeddings are shared across subjects. These features allow for modeling spatial variability across subjects (via different projection matrices) and different temporal alignments (discrete estimation separate for each subject). The stimulus embeddings allow for identifying shared structure. \n\nThe model is validated in a simulated experiment and on a dataset of fMRI recordings while subjects listened to computer science lectures. The authors examine recovery of latent states and clusterings of neural activity/stimuli in the simulated data. In the analysis of an fMRI dataset, the authors find variations in sequential activity & spatial coding across subjects. Importantly, they show the learned projections output statistically meaningful clusterings on heldout fMRI runs. ", "questions": "* Can the authors clarify how the stimulus embeddings are learned?\n\n* Why are the events $E$ defined in voxel space when the event segmentation is done in the lower dimensional projection space defined by $W_i$? How does this compare to computing the events $E$ based on projected activity $W_i X_i$? I encourage the authors to consider an alternative fitting method that learns $G$ via the projected neural activity $W_i X_i$. \n\n* How sensitive is the model to initialization / how do results vary across different initializations? \n\nMinor comments - typos\n* Figure 2 caption: stimulated -> simulated ", "limitations": "Yes. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The proposed methods are original and significant for the analysis of fMRI datasets during sequential tasks across subjects. The idea to also embed the stimulus makes the learned latent state clusterings more interpretable, and may also help with identifiability. The experimental results relating semantic content of course videos to fMRI recordings across subjects appear very significant and useful for scientific analysis. ", "weaknesses": "The clarity of the methods and experiment could be improved. One example is it is unclear how the stimulus embedding is learned. At some points, the text implies that the stimulus is treated akin to a subject where a projection matrix is learned for the stimulus in the same way it is for the subjects. However, the stimulus is not included in Algorithm 1. Including more details on how the stimulus is incorporated into the model and learned would help with clarity. \n\nNext, some aspects of the modeling approach appear inconsistent which may limit significance. For example, the events $E$ are defined in voxel space whereas the event segmentation is done in the lower dimensional projection space defined by $W_i$. Next, the model fitting approach appears somewhat ad hoc, and it is not clear it corresponds to a single objective such as maximum likelihood estimation. More justification for the proposed fitting approach and simulated data analysis could rectify these concerns. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) \nYes, the stimulus embeddings were treated like a subject when fitting the model (as you’ve already noted). We include one copy of the stimulus embeddings per each subject in order to prevent the model from over-favoring the human subjects during the forward-backward step. In Algorithm 1, we would adjust the number of subjects, n, to reflect this procedure. Note that because our approach does not require the number of voxels/features to be consistent across subjects/models, the fact that models are in a different feature space does not require any changes to the algorithm.\n\n2) \nEvents at the group level (G) are all defined in the low-dimensional space, and the HMM fitting takes place in this low-dimensional space. The event matrices E (in the original voxel/feature spaces) are computed and used only when updating the projection matrices W. The HMM update procedures, following the expectation-maximization algorithm (Baum-Welch), in combination with PCA updates to the projection matrices, maximizes the likelihood of data under the assumption that voxels have uniform error variance. https://www.sciencedirect.com/science/article/pii/S0169743998000902\n\n3) \nThere is no randomness in initialization - the HMM fitting starts by assuming very high measurement variance, which effectively starts the estimation procedure at the model's prior distribution.\n"}]

Author rebuttal:

Meta-review: This paper introduces an interesting method for aligning cognitive events across individuals using their brain recordings and stimulus features. I believe the authors have provided sufficient response to the reviewer's concerns, including running a realistic simulation. Some concerns were raised about the machine learning novelty in the paper, however, neurips has interest in machine learning methods applied in novel ways to answer neuroscientific problems, so this paper is appropriate for neurips. The authors are requested to address all the concerns raised in the review period and to add their new results and explanations.
Decision: Accept (poster)