Title: Hyper-HMM: aligning human brains and semantic features in a common latent event space
Venue: NeurIPS 2023
Accepted: yes
Keywords: Brain Imaging

Other Cognitive Science

Other Neuroscience
Abstract: Naturalistic stimuli evoke complex neural responses with spatial and temporal properties that differ across individuals. Current alignment methods focus on either spatial hyperalignment (assuming exact temporal correspondence) or temporal alignment (assuming exact spatial correspondence). Here, we propose a hybrid model, the Hyper-HMM, that simultaneously aligns both temporal and spatial features across brains. The model learns to linearly project voxels to a reduced-dimension latent space, in which timecourses are segmented into corresponding temporal events. This approach allows tracking of each individual's mental trajectory through an event sequence, and also allows for alignment with other feature spaces such as stimulus content. Using an fMRI dataset in which students watch videos of class lectures, we demonstrate that the Hyper-HMM can be used to map all participants and the semantic content of the videos into a common low-dimensional space, and that these mappings generalize to held-out data. Our model provides a new window into individual cognitive dynamics evoked by complex naturalistic stimuli.

Hyper-HMM: aligning human brains and semantic features in a common latent event space

Abstract

Naturalistic stimuli evoke complex neural responses with spatial and temporal properties that differ across individuals. Current alignment methods focus on either spatial hyperalignment (assuming exact temporal correspondence) or temporal alignment (assuming exact spatial correspondence). Here, we propose a hybrid model, the Hyper-HMM, that simultaneously aligns both temporal and spatial features across brains. The model learns to linearly project voxels to a reduceddimension latent space, in which timecourses are segmented into corresponding temporal events. This approach allows tracking of each individual's mental trajectory through an event sequence and for alignment with other feature spaces such as stimulus content. Using an fMRI dataset in which students watch videos of class lectures, we demonstrate that the Hyper-HMM can be used to map all participants and the semantic content of the videos into a common low-dimensional space, and that these mappings generalize to held-out data. Our model provides a new window into individual cognitive dynamics evoked by complex naturalistic stimuli.

1 Introduction

Dynamic, continuous stimuli such as movies, stories, music, or educational videos evoke complex response patterns throughout the brain that can be captured using functional magnetic resonance imaging (fMRI). In addition to cognitive processes shared by all participants, these responses reflect individual differences in perception, learning, and memory. These differences can result in the same semantic concept being represented at varying points in time or by different spatial patterns of brain activity. An ongoing challenge is to preserve these heterogeneous dynamics across individuals while capturing shared cognitive processes underlying human intelligence.

Jane Han

Hanover, NH

Guo Jiahui

Christopher Baldassano

One line of research has focused on the temporal differences between participants. Previous work using a Hidden Markov Model (HMM) approach [1] has found that there are differences in the timing of event representations across development [9] or from repeated exposure to a stimulus [27], and has measured relationships between event timing and stimulus interpretation in individuals [41]. This approach requires either making the assumption that the event-specific spatial activity pattern across voxels is identical across participants, or fitting completely separate models for each participant (preventing the identification of corresponding events across people).

A separate category of approaches has focused on the spatial differences across participants, learning a functional alignment across participants that accounts for differences in functional architecture [6, 16, 19]. However, these approaches assume that all participants' neural responses are exactly synchronized in time. This assumption may be strongly violated, especially when studying idiosyncratic cognitive processes such as learning complex educational material.

Another challenge when using naturalistic stimuli is identifying the stimulus features that are driving neural activity. A common approach is to construct a vector representation of a particular class of features in the stimulus, and then construct a linear 'encoding' or 'decoding' model that maps these vectors onto patterns of brain activity or vice versa [21, 34, 47]. However, these models again face the same challenges of temporal and spatial alignment; they assume that the neural response to a stimulus is immediate (or occurs at a short, fixed delay learned by the encoding model) and that all participants represent stimulus features with the same pattern of brain activity (or are restricted to modeling a single participant at a time).

Contributions. Here we propose the Hyper-HMM (H-HMM), an extension of the HMM from Baldassano et al. [1] that can simultaneously capture temporal and spatial differences across participants. The model can also align across entirely different kinds of feature spaces, creating mappings between neural data and a semantic embedding of the stimulus. We validate our model using simulated data and by using fMRI data from individuals watching course videos for a Computer Science class [30]. We can successfully map both the human participants and an embedding model of stimulus into a common latent space of semantic events, and this mapping generalizes to held-out videos in this dataset. Our novel approach is broadly applicable to any neuroimaging dataset with continuous naturalistic stimuli, and can provide a new way of characterizing and studying individual differences in cognitive processes.

2 Background

2.1 Temporal alignment between brains and with stimuli

Although neural responses are assumed to be tightly temporally-locked to the stimulus during traditional experimental designs with discrete stimuli, this is often not the case for more abstract cognitive trajectories evoked in naturalistic experiments. Especially in high-level regions, we could see responses that lag behind the stimulus (if a viewer takes time to comprehend an event), run ahead of the stimulus (if a viewer can predict upcoming events [27]), or are generated based on a past stimulus rather than an ongoing stimulus (such as during narrative recall [5]). As a further complication, these temporal shifts could differ in meaningful ways across people [41]. One approach for identifying a temporal alignment across datasets is to use Dynamic Time Warping, which stretches and compresses the temporal axis of each dataset to maximize their alignment [44]. This technique makes relatively few assumptions about the structure of the temporal alignment, but does require that the datasets can be matched together at the level of individual timepoints (i.e. that every 1-2 seconds of the stimulus can be directly mapped to 1-2 seconds of the neural response).

An alternative approach is to assume that datasets can be segmented into discrete, cognitivelymeaningful events on the timescale of tens of seconds to minutes. The theory that people naturally perceive and remember continuous stimuli as individual events has been extensively studied in cognitive psychology [39, 50], and signatures of this event segmentation can also be measured using neuroimaging [1, 49]. This work therefore suggests that events are a natural unit for representing cognitive dynamics and for performing temporal alignment, and we can employ Hidden Markov Models (HMMs) to track these discrete cognitive states [1, 27, 45]. Current HMMs for neuroimaging, however, assume that the neural signature of an event is a specific spatial pattern of activation across voxels in a region of interest (ROI) which is identical across all experimental subjects.

2.2 Spatial hyperalignment between brains

Since the early days of fMRI, researchers have recognized that the specific spatial arrangement of functional units in the brain differs across individuals, motivating the use of 'localizer' tasks to identify the functional layout of a specific brain [42]. One kind of method for performing multisubject analyses is to use responses to a long naturalistic stimulus to learn a linear transformation that maps between individuals, a method called 'hyperalignment' [18]. There are now many variations and applications of this approach [19, 22], including models that map subjects into a shared lowerdimensional space [4, 6, 11, 16]. Though these methods can capture spatial variation in functional responses, they assume perfect temporal alignment across participants in their neural responses.

2.3 Mapping between stimulus features and neural responses

Extensive work has focused on bidirectionally mapping between features of the external world and the human brain [34, 36]. These can take the form of encoding models that predict brain activity patterns as a function of stimulus features [21, 31, 33, 43, 47], decoding models that produce estimates of external stimuli (or internal mental states) given brain activity [20, 32, 35], or hybrid models that capture structure among both stimulus features and brain activity [23]. These models assume that the temporal correspondence between the stimulus and neural responses is known, or allow for a small fixed lag in the neural responses by including time-shifted versions of the stimulus features.

3 Methods overview

3.1 Model description and fitting

The Hyper-HMM (Figure 1) combines the strengths of these existing methods, removing the strong assumptions of spatial or temporal alignment across individuals and temporal synchronization to the stimulus. Instead, we require only a much more abstract level of correspondence across datasets, assuming that: a) all subjects and stimulus models proceed through the same sequence of semantic states (events); b) each event e can be represented as a low-dimensional vector G e ∈ R D (where the latent dimensionality D is a parameter of the model); c) each subject and model can be mapped into this latent space using a linear transformation W i (unique to each subject and model); d) in the latent space, all timepoints during which this subject or model was in event e should be correlated with G e .

To model both these temporal and spatial assumptions, we employ a chain-structured Hidden Markov Model in which all n participants transition sequentially through a sequence of event states, as in Baldassano et al. [1]. For the observation model, we define

When a participant or stimulus model i is in state (event) s i,t = e , the latent-space mapping of the spatial voxel pattern or embedding vector ( W i X i,t ) should be similar to the group-level event representation G e . All fMRI and stimulus datasets are therefore linked through their transform matrices W i into a common latent semantic space, which reflects between-event similarity structure shared by all fMRI subjects and with the stimulus features. The z () function represents z-scoring (standardizing representations to have zero mean and unit variance), so that the Euclidean distance measure is proportional to the Pearson correlation between W i X i,t and G e .

We fit the model by iteratively estimating the temporal alignment s i,t for each dataset and then updating the shared event representations G and the W i projection matrices. On each iteration, we project each subject's data and the stimulus features into the low-dimensional shared space using the current W i matrices. We apply the forward-backward algorithm from Baldassano et al. [1] in this shared space, probabilistically segmenting each timeseries into events corresponding to the patterns G e . The resulting probabilities η i,t,e = p ( s i,t = e | X i , W i , G ) can then be used to estimate the subject-specific voxel representations of each event as E i,e = ∑ t η i,t,e X i,t / ∑ t η i,t,e .

To update the group-level representations G , we stack the event patterns E i from all subjects and the stimulus (concatenating along the voxel/embedding dimension for each event, as in [16]) and then project down onto D principal components. In order to ensure that the fMRI data and stimulus embedding have an equal vote in the shared event representations, the stimulus event patterns are

fMRl responses

X2

Timepoints

Stimulus features

X1

Sentences

Xs

Embedding

Voxels

Embedding,

G

Events

Voxels

Es

Voxels

Embedding

E2

Events

Latent

E1

Ws

Events

W

Latent

Latent

(a)

10

20

30

40

50

(b)

Event 1

Event2

Event3

Event4

fMRI subject

0

10

20

30

40

50

Stimulus embedding

weighted n times as strongly as each fMRI dataset. Finally, we update the transform matrices, W i , using a ridge regression predicting G from subject patterns (with α = 10 ). We repeat this process (while annealing the HMM observation model variance σ m across iterations [1]) until the model's log-likelihood stops improving.

After fitting, the model provides a temporal segmentation into events that corresponds across subjects and between the subjects and the stimulus (Figure 2a). For each event, all subjects' spatial patterns of fMRI activity and the stimulus feature representation can be projected to the same region of latent space, through the learned matrices W i (Figure 2b).

3.2 Validation with simulated data

We tested the model's ability to recover ground-truth event structure under varying degrees of noise using simulated datasets. We used the fmrisim utility [13] in the BrainIAK toolbox [26] to generate a pure-noise dataset in the angular gyrus with the same number of subjects, runs, and timepoints per run as in the fMRI data used in our main experiments (see below). We simulated an event-structured signal by assigning blocks of timepoints to events, with events varying in length between 12 and 18 timepoints across subjects. Within each event there was a constant event pattern signal; these patterns came from the real fMRI data, averaged in 15-TR increments. By combining the signal and noise together with varying weights, we produced simulated datasets with realistic spatial and temporal correlation properties along with ground-truth timing and event patterns within each subject. We used these datasets to test the model's ability to recover subject-specific event boundaries and estimate subject-specific spatial projections that generalized across runs.

For each noise level, we trained the model on only one half of the runs, and then froze the projection matrices W i and performed temporal-only alignment on the held-out half of the runs. Ideally, these projection matrices should produce event representations on these held-out runs that are highly similar (across subjects and the stimulus embedding) for each event in the latent semantic space. To assess the degree of event clustering, we computed the sum of squared errors (SSE) between the latent-space event pattern for each fMRI subject and the mean pattern for this event across all subjects. We normalized this to yield a 'variance explained' measure, dividing it by the SSE between all fMRI event patterns and the global mean pattern and then taking one minus this ratio. If the event patterns for all subjects map onto identical latent representations, this value will be 1, and patterns that show no clustering into events will yield a variance explained of 0. We repeated this process with the training and testing sets reversed, and averaged the results.

Next, we tested the model's ability to identify ground-truth event boundaries at each noise level. After fitting the model to the full dataset, we identified event boundaries for each subject as timepoints at which the most probable event switched, and computed the fraction of ground-truth boundaries that matched (within one timepoint) one of these model-derived boundaries.

3.3 Applying to an fMRI dataset

We used fMRI data collected by Meshulam et al. [30], in which undergraduate students ( n =19) taking an introductory Computer Science course watched some of the course lecture videos in the scanner. There were five scan sessions throughout the course of the semester, divided into 21 separate scanning runs, for a total of 197 minutes of data per subject. We hypothesized that variations in expertise and concept understanding would lead to a relatively large amount of variability in neural responses across participants, making it ideal for testing our alignment approach. The fMRI data (TR = 2 seconds, voxel size = 3mm x 3mm x 3mm) was preprocessed using fMRIPrep [14], including mapping data to the cortical fsaverage6 surface, and then cleaned via linear regression to remove variance associated with motion, cerebrospinal fluid signals, white matter signals, and low-frequency drift. The timecourse of activity for each vertex was z-scored for each run.

We derived a semantic embedding of the lecture stimulus by transcribing the spoken dialogue using a speech recognition model [38] and then obtaining a vector representation of each sentence using a pre-trained transformer model [40]. Note that we do not make use of any timing information about the correspondence between fMRI timepoints and stimulus sentences, giving the H-HMM full flexibility in assigning stimulus timepoints to neural responses in individual subjects.

Here we focus on data from the angular gyrus and posterior medial cortex (as defined in Baldassano et al. [2]), given their role in anticipating familiar narrative stimuli and learning schematic structures [2, 27]. We fit the H-HMM to all 21 fMRI runs, setting the number of events such that the average event duration was 15 TRs (30 seconds). Each run had a separate set of group-level events G , but the W i matrices were constrained to be the same across all runs (i.e. the W i matrices were computed via a single regression from all subject/stimulus-specific events to all group-level events). For our primary analyses, we chose a latent dimensionality of D = 3 for the group-level shared semantic space (see Figure 7 for an exploration of alternative dimensionalities).

We examined the learned projection matrices W i in both neural and stimulus space. For fMRI data, we visualized the first dimension of W i (which captured the largest share of the variance in semantic space) by left-multiplying it by the Gram matrix X i X T i and then z-scoring. This produced a weighted average of the subject's spatial patterns, in which the weights were given by the similarity to the first dimension of W i . We performed a similar operation for the stimulus embeddings, measuring the correlation between individual sentence embeddings and the first column of X S X T S W S . Sentences with particularly high or low correlations were those that were highly correlated or anticorrelated with this first semantic dimension.

To quantitatively evaluate these projection matrices, we used the variance-explained metric described in section 3.2, computing this value separately for 1) the clustering of the fMRI representations around their mean for each event and 2) the match between the stimulus representation and the mean fMRI representation for each event. As a baseline, we ran the same analysis, but used random projection matrices W i in which entries were sampled from a standard normal distribution N (0 , 1) . We also repeated this analysis while varying the dimensionality D of the latent semantic space.

Compute resources. Model fitting was performed using a maximum of 24 CPUs per fit. For fitting to the training data, compute time ranged from 12 - 16 minutes per split half, depending on the size of the ROI. The temporal-only alignment for the test data required 3 - 6 minutes for each fit.

4 Results

4.1 Simulated data

In our experiments with simulated fMRI data, the H-HMM can effectively learn spatial alignments across subjects even under relatively high levels of noise (Figure 3a). Additionally, the model can successfully recover the majority of the ground-truth event boundaries in each subject when the noise amplitude is up to twice as large as the signal amplitude (Figure 3b). These results demonstrate that the model and fitting procedure are robust to the noise properties of a typical fMRI dataset.

(a)

(b)

explained)

0.7

T

recovery

Spatial alignment

0.9

工

0.6

0.8

0.5

(variance

0.7

Boundary

0.4

0.6

0.3

0.5

0.2

2.0:1

1:1.0

1:1.5

心

:2.5

1:3.5

.A.

2.0:1

1:1.0

1:1.5

1:2.0

1:2.5

3

1:3.5

1:4.0

Signal to noise

Signal to noise

Angular Gyrus

Post Medial Cortex

Subjects

fMRI

0

10

20

30

40

50

60

70

0

10

20

30

40

50

60

70

Timepoints (2s)

Timepoints (2s)

Sonowlet'stakealookatregularexpressions.

Sonowlet'stakealook atregularexpressions.

Andthat'swelldefinedbythesespecificrules.

Aproteinisastringofaminoacids,sojust

Stimulus

astringconsistingofthoseletters.

So,wemighthavethequestion,Ihavea

Isitinthelanguagedefinedbythoserules?

givenprotein，isit，doesitfollowtheserules?

Here'sanotherexamplefrom

Andtheanswerinthiscaseisyes.

commercialcomputing.

So in this case,this string

So this phrase,not an email

isin thelanguage.

address,it'snotanemailaddress.

0

10

20

30

40

0

10

20

30

40

Sentences

Sentences

Event1Event2Event3

Event4

Event 5

4.2 fMRI experiment

We fit the H-HMM to the full fMRI dataset for two brain regions (angular gyrus and posterior medial cortex) and examined the learned temporal and spatial alignment. An example of the temporal alignment for the first half of one run is illustrated in Figure 4, showing how each individual's neural responses and the stimulus sentences are segmented into a shared set of events. This kind of semantic event segmentation is made possible by the learned projections W i that align all subjects and the stimulus embedding into a common semantic space. A visualization of this alignment for the first (maximally-informative) semantic dimension is shown in Figure 5. Note that the overall spatial topography corresponding to this dimension is similar across subjects (e.g. in angular gyrus, the posterior portion has generally positive weights while the anterior portion has generally negative weights) but the fine-scale details are different in each subject, reflecting individual variation in how this semantic dimension is spatially expressed. We can also examine the learned projection of the sentence embeddings onto this semantic dimension. The poles of this first dimension in angular gyrus appear to correspond approximately to simple concrete sentences versus abstract statements, and in posterior medial cortex to future-oriented statements versus present-tense descriptions of program structure. These axes of semantic variation have been observed in prior fMRI studies: Conca et al. [10] show differences in activity for concrete vs abstract and future- vs past- and present-tense sentences, and concrete vs abstract concepts engage different regions of the semantic network [17].

In order to statistically verify that the learned projections to the latent space were meaningful, we next tested how well the projection matrices W i fit to one half of the data generalized to the other half of the data. We identified events in the testing data for both the neural and stimulus data, and measured the degree of clustering for corresponding events between fMRI subjects and between the fMRI group-average and the stimulus. An example of the event representations in the latent space is shown in Figure 6, illustrating that event patterns are projected into event-specific clusters. The event clustering for the fMRI subjects was substantially above a random-projection baseline in both regions ( p < 0 . 001 ). The match between the neural representations and the stimulus representations was also significantly above baseline in both regions (Angular gyrus p < 0 . 001 ; PMC p = 0 . 0027 ). These results demonstrate that the learned mapping to a latent semantic space provides meaningful alignment between brains and provides a correspondence between brain data and stimulus features.

Angular Gyrus

PostMedialCortex

Subj 2

Subj2

I[qns

I[qns

Subj3

Subj 3

Subj 4

Subj 4

First semantic dimension (z)

First semantic dimension (z)

-2

2

-2

2

program clearer and easierto read.

It'sjustanotherwayofformulating

loopsthatsometimesmakesa

to understand what it does.

canwrite aprogramtotry

It's an abstraction, but we

And a dot java file can be a set of

functions,butit canbemore.

characteristics ofsimplystructured data.

prettysimplystructuredoratleast has

Also,the data that we process oftenis

first semantic dimension (r)

first semantic dimension(r)

Sentence alignment to

Sentence alignment to

0.4

..Tia

3

Lecture

021

Lecture

1

0.0

3

2.

0.0-

2

3

4

—-021

-0.2

-04

0.4

Okay,what about this one?

This one's not so easy.

computationthatwe'lluselater on.

Here'sanotherinteresting

That's going to be our next task.

A key parameter of our model is the dimensionality D of the latent semantic space. Intuitively, performance should suffer if D is too small (preventing proper discrimination between semanticallydifferent events) or if D is too large (resulting in latent event representations that are largely orthogonal, which will fail to capture semantically-meaningful relationships and will fail to generalize to new events). We varied D and again performed our analysis of held-out performance, fitting the model on half the data and measuring the degree of clustering on testing data here, including both fMRI and stimulus representations in our variance-explained clustering measure. We observed (Figure 7) that performance peaked for D = 3 , with slightly reduced performance for D = 2 (the minimum dimensionality for which the HMM can compute pattern correlations) and substantially reduced performance as D increased to 8 . We conclude that D = 3 provides the optimal number of semantic dimensions events in this dataset, though datasets that are very large or explore a broader set of semantic topics could require a higher-dimensional latent space.

5 Limitations, Future Work, and Conclusions

The Hyper-HMM makes several assumptions about the cognitive structure of naturalistic datasets that could be relaxed in future work. First, it assumes that all subjects and the stimulus proceed through the same linear sequence of cognitive states. This is a reasonable assumption for data collected during tasks such as watching or recalling movies with linear narratives [5]; but may be less accurate for tasks such as recalling a series of unrelated short videos [28]. Second, we assume that neural dynamics are relatively constant within cognitive states, with sharp jumps between states; this model would fail to capture dynamics in brain regions with brief responses occurring only at event boundaries, such as the hippocampus [3, 9]. We also assume that brain responses in different people as well as stimulus embeddings are all linearly projectable to a common semantic space. Future work could extend this projection to be non-linear, as has been explored for characterizing relationships between brain regions within individuals [15, 37].

Latent space projections

Clustering performance

for three example events

onalltestevents

0.5

0.4

0.3

***

***

Angular Gyrus

0.2

0.1

0.0

0.1

-0.2

Event 2

● fMRI subject

Event6

fMRI clustering

fMRI-stimulus

Event9

Stimulusembedding

match

0.5

0.4

0.3

***

**

Post Medial Cortex

0.2

0.1

0.0

0.1

-0.2

0.6

Angular Gyrus

Post Medial Cortex

ained

0.4

expl

iance

0.2

以

Baseline

0.0

2

3

4

5

6

7

8

Shared dimensionality D

Although our results and model-fitting have focused primarily on brain-stimulus alignment, our model can be expanded to alignment of cross-modal, or even cross-species, information in the brain. Current neuroimaging techniques each exhibit tradeoffs between spatial resolution and temporal resolution, and there have been several proposed approaches for combining information across experiments in which brain responses to the same stimulus were measured in different neuroimaging modalities [7, 8, 24]. The H-HMM could provide a new avenue for integrating the strengths of modalities with different spatial and temporal properties, even if measured in different groups of experimental subjects. Analogously, brain alignment across species, in particular between humans and primates, have provided insights into common representational processes [12, 25, 48], and the H-HMM could be applied to map cross-species representations into a common latent space.

The Hyper-HMM provides a highly-flexible framework for identifying and aligning event representations across people and to features of a stimulus. Our experiments demonstrate that fitting this model to an fMRI dataset is computationally and statistically feasible, and can learn meaningful mappings of brain responses and stimulus features into a stable semantic space. This model has broad applications to the rapidly-expanding field of studies using continuous naturalistic stimuli to study dynamic cognition [29, 46], including the characterization of individual differences in brain responses and the identification of representational properties of high-level brain regions.

References

C. Baldassano, J. Chen, A. Zadbood, J. W. Pillow, U. Hasson, and K. A. Norman. Discovering event structure in continuous narrative perception and memory. Neuron , 95(3):709-721, 2017.

C. Baldassano, U. Hasson, and K. A. Norman. Representation of real-world event schemas during narrative perception. Journal of Neuroscience , 38(45):9689-9699, 2018. ISSN 02706474.

A. Ben-Yakov and R. N. Henson. The hippocampal film editor: Sensitivity and specificity to event boundaries in continuous experience. J. Neurosci. , 38(47):10057-10068, Nov. 2018.

N. Y. Bilenko and J. L. Gallant. Pyrcca: Regularized kernel canonical correlation analysis in python and its applications to neuroimaging. Frontiers in Neuroinformatics , 10, Nov. 2016. doi: 10.3389/fninf.2016.00049. URL https://doi.org/10.3389/fninf.2016.00049 .

J. Chen, Y. C. Leong, C. J. Honey, C. H. Yong, K. A. Norman, and U. Hasson. Shared memories reveal shared structure in neural activity across individuals. Nature Neuroscience , 20 (1):115-125, Dec. 2016. doi: 10.1038/nn.4450. URL https://doi.org/10.1038/nn.4450 .

P.-H. C. Chen, J. Chen, Y. Yeshurun, U. Hasson, J. Haxby, and P. J. Ramadge. A reduceddimension fmri shared response model. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

R. Cichy, D. Pantazis, and A. Oliva. Resolving human object recognition in space and time. Nature neuroscience , 17, 01 2014.

R. M. Cichy and A. Oliva. Am/eeg-fmri fusion primer: resolving human brain responses in space and time. Neuron , 107(5):772-781, 2020.

S. S. Cohen, N. Tottenham, and C. Baldassano. Developmental changes in story-evoked responses in the neocortex and hippocampus. Elife , 11, July 2022.

F. Conca, V. M. Borsa, S. F. Cappa, and E. Catricalà. The multidimensionality of abstract concepts: A systematic review. Neuroscience & Biobehavioral Reviews , 127:474-491, 2021.

A. de Cheveigné, G. M. D. Liberto, D. Arzounian, D. D. Wong, J. Hjortkjær, S. Fuglsang, and L. C. Parra. Multiway canonical correlation analysis of brain data. NeuroImage , 186:728-740, Feb. 2019. doi: 10.1016/j.neuroimage.2018.11.026. URL https://doi.org/10.1016/j. neuroimage.2018.11.026 .

N. Eichert, E. C. Robinson, K. L. Bryant, S. Jbabdi, M. Jenkinson, L. Li, K. Krug, K. E. Watkins, and R. B. Mars. Cross-species cortical alignment identifies different types of anatomical reorganization in the primate temporal lobe. eLife , 9:e53232, mar 2020. ISSN 2050-084X.

C. T. Ellis, C. Baldassano, A. C. Schapiro, M. B. Cai, and J. D. Cohen. Facilitating open-science with realistic fmri simulation: validation and application. PeerJ , 8:e8564, 2020.

O. Esteban, C. J. Markiewicz, M. Goncalves, J. D. Kent, E. DuPre, R. Ciric, T. Salo, B. Pinsard, R. W. Blair, R. A. Poldrack, and K. J. Gorgolewski. fMRIPrep: a robust preprocessing pipeline for functional MRI, Jan. 2022. URL https://doi.org/10.5281/zenodo.5898602 .

M. Fang, C. Poskanzer, and S. Anzellotti. Multivariate connectivity: A brief introduction and an open question. Frontiers in Neuroscience , 16, Jan. 2023. doi: 10.3389/fnins.2022.1082120. URL https://doi.org/10.3389/fnins.2022.1082120 .

M. Feilong, S. A. Nastase, G. Jiahui, Y. O. Halchenko, M. I. Gobbini, and J. V. Haxby. The individualized neural tuning model: Precise and generalizable cartography of functional architecture in individual brains. bioRxiv , 2022. doi: 10.1101/2022.05.15.492022.

M. Gilead, N. Liberman, and A. Maril. The language of future-thought: An fmri study of embodiment and tense processing. Neuroimage , 65:267-279, 2013.

J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, M. Hanke, and P. J. Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron , 72(2):404-416, Oct. 2011. doi: 10.1016/j. neuron.2011.08.026. URL https://doi.org/10.1016/j.neuron.2011.08.026 .

J. V. Haxby, J. S. Guntupalli, S. A. Nastase, and M. Feilong. Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. Elife , 9, June 2020.

T. Horikawa and Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications , 8(1), May 2017. doi: 10.1038/ncomms15037. URL https://doi.org/10.1038/ncomms15037 .

A. G. Huth, S. Nishimoto, A. T. Vu, and J. L. Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron , 76(6):1210-1224, Dec. 2012. doi: 10.1016/j.neuron.2012.10.014. URL https://doi.org/ 10.1016/j.neuron.2012.10.014 .

G. Jiahui, M. Feilong, M. V. di Oleggio Castello, J. S. Guntupalli, V. Chauhan, J. V. Haxby, and M. I. Gobbini. Predicting individual face-selective topography using naturalistic stimuli. NeuroImage , 216:116458, Aug. 2020. doi: 10.1016/j.neuroimage.2019.116458. URL https: //doi.org/10.1016/j.neuroimage.2019.116458 .

J.-R. King, F. Charton, D. Lopez-Paz, and M. Oquab. Back-to-back regression: Disentangling the influence of correlated factors from multivariate observations. NeuroImage , 220:117028, Oct. 2020. doi: 10.1016/j.neuroimage.2020.117028. URL https://doi.org/10.1016/j. neuroimage.2020.117028 .

D. Kozinska, F. Carducci, and K. Nowinski. Automatic alignment of eeg/meg and mri data sets. Clinical neurophysiology , 112(8):1553-1561, 2001.

N. Kriegeskorte, M. Mur, D. A. Ruff, R. Kiani, J. Bodurka, H. Esteky, K. Tanaka, and P. A. Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron , 60(6):1126-1141, 2008. ISSN 0896-6273.

M. Kumar, M. J. Anderson, J. W. Antony, C. Baldassano, P. P. Brooks, M. B. Cai, P.-H. C. Chen, C. T. Ellis, G. Henselman-Petrusek, D. Huberdeau, J. B. Hutchinson, Y. P. Li, Q. Lu, J. R. Manning, A. C. Mennen, S. A. Nastase, H. Richard, A. C. Schapiro, N. W. Schuck, M. Shvartsman, N. Sundaram, D. Suo, J. S. Turek, D. Turner, V. A. Vo, G. Wallace, Y. Wang, J. A. Williams, H. Zhang, X. Zhu, M. Capota ˘ , J. D. Cohen, U. Hasson, K. Li, P. J. Ramadge, N. B. Turk-Browne, T. L. Willke, and K. A. Norman. BrainIAK: The brain imaging analysis kit. Aperture Neuro , 2021(4), Feb. 2022. doi: 10.52294/31bb5b68-2184-411b-8c00-a1dacb61e1da. URL https://doi.org/10.52294/31bb5b68-2184-411b-8c00-a1dacb61e1da .

C. S. Lee, M. Aly, and C. Baldassano. Anticipation of temporally structured events in the brain. Elife , 10, Apr. 2021.

H. Lee and J. Chen. Predicting memory from the network structure of naturalistic events. Nature Communications , 13(1), July 2022. doi: 10.1038/s41467-022-31965-2. URL https: //doi.org/10.1038/s41467-022-31965-2 .

H. Lee, B. Bellana, and J. Chen. What can narratives tell us about the neural bases of human memory? Current Opinion in Behavioral Sciences , 32:111-119, Apr. 2020. doi: 10.1016/j. cobeha.2020.02.007. URL https://doi.org/10.1016/j.cobeha.2020.02.007 .

M. Meshulam, L. Hasenfratz, H. Hillman, Y.-F. Liu, M. Nguyen, K. A. Norman, and U. Hasson. Neural alignment predicts learning outcomes in students taking an introduction to computer science course. Nature communications , 12(1):1-14, 2021.

J. Millet, C. Caucheteux, Y. Boubenec, A. Gramfort, E. Dunbar, C. Pallier, J.-R. King, et al. Toward a realistic model of speech processing in the brain with self-supervised learning. Advances in Neural Information Processing Systems , 35:33428-33443, 2022.

T. M. Mitchell, R. Hutchinson, R. S. Niculescu, F. Pereira, X. Wang, M. Just, and S. Newman. Learning to decode cognitive states from brain images. Machine learning , 57(1-2):145-175, 2004.

T. M. Mitchell, S. V. Shinkareva, A. Carlson, K.-M. Chang, V. L. Malave, R. A. Mason, and M. A. Just. Predicting human brain activity associated with the meanings of nouns. science , 320(5880):1191-1195, 2008.

T. Naselaris, K. N. Kay, S. Nishimoto, and J. L. Gallant. Encoding and decoding in fmri. Neuroimage , 56(2):400-410, 2011.

K. A. Norman, S. M. Polyn, G. J. Detre, and J. V. Haxby. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences , 10(9):424-430, Sept. 2006. doi: 10.1016/j.tics.2006.07.005. URL https://doi.org/10.1016/j.tics.2006.07.005 .

L. Paninski, J. Pillow, and J. Lewi. Statistical models for neural encoding, decoding, and optimal stimulus design. Progress in brain research , 165:493-507, 2007.

C. Poskanzer and S. Anzellotti. Functional coordinates: modeling interactions between brain regions as points in a function space. Network Neuroscience , 6(4):1296-1315, 2022.

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision, 2022.

G. A. Radvansky and J. M. Zacks. Event boundaries in memory and cognition. Current Opinion in Behavioral Sciences , 17:133-140, Oct. 2017. doi: 10.1016/j.cobeha.2017.08.006. URL https://doi.org/10.1016/j.cobeha.2017.08.006 .

N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908. 10084 .

C. Sava-Segal, C. Richards, M. Leung, and E. S. Finn. Individual differences in neural event segmentation of continuous experiences. Cerebral Cortex , Mar. 2023. doi: 10.1093/cercor/ bhad106. URL https://doi.org/10.1093/cercor/bhad106 .

R. Saxe, M. Brett, and N. Kanwisher. Divide and conquer: A defense of functional localizers. NeuroImage , 30(4):1088-1096, May 2006. doi: 10.1016/j.neuroimage.2005.12.062. URL https://doi.org/10.1016/j.neuroimage.2005.12.062 .

M. Schrimpf, I. A. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. B. Tenenbaum, and E. Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences , 118(45):e2105646118, 2021.

L. J. Silbert, C. J. Honey, E. Simony, D. Poeppel, and U. Hasson. Coupled neural systems underlie the production and comprehension of naturalistic narrative speech. Proceedings of the National Academy of Sciences , 111(43), Sept. 2014. doi: 10.1073/pnas.1323812111. URL https://doi.org/10.1073/pnas.1323812111 .

M. Silva, C. Baldassano, and L. Fuentemilla. Rapid memory reactivation at movie event boundaries promotes episodic encoding. The Journal of Neuroscience , 39(43):8538-8548, Sept. 2019. doi: 10.1523/jneurosci.0360-19.2019. URL https://doi.org/10.1523/jneurosci. 0360-19.2019 .

S. Sonkusare, M. Breakspear, and C. Guo. Naturalistic stimuli in neuroscience: Critically acclaimed. Trends in Cognitive Sciences , 23(8):699-714, Aug. 2019. doi: 10.1016/j.tics.2019. 05.004. URL https://doi.org/10.1016/j.tics.2019.05.004 .

K. Vodrahalli, P.-H. Chen, Y. Liang, C. Baldassano, J. Chen, E. Yong, C. Honey, U. Hasson, P. Ramadge, K. A. Norman, and S. Arora. Mapping between fMRI responses to movies and their natural language annotations. Neuroimage , 180(Pt A):223-231, Oct. 2018.

T. Xu, K.-H. Nenning, E. Schwartz, S.-J. Hong, J. T. Vogelstein, A. Goulas, D. A. Fair, C. E. Schroeder, D. S. Margulies, J. Smallwood, M. P. Milham, and G. Langs. Cross-species functional alignment reveals evolutionary hierarchy within the connectome. NeuroImage , 223:117346, 2020. ISSN 1053-8119.

J. M. Zacks, T. S. Braver, M. A. Sheridan, D. I. Donaldson, A. Z. Snyder, J. M. Ollinger, R. L. Buckner, and M. E. Raichle. Human brain activity time-locked to perceptual event boundaries. Nature Neuroscience , 4(6):651-655, June 2001. doi: 10.1038/88486. URL https://doi.org/10.1038/88486 .

J. M. Zacks, B. Tversky, and G. Iyer. Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology: General , 130(1):29-58, 2001. doi: 10.1037/ 0096-3445.130.1.29. URL https://doi.org/10.1037/0096-3445.130.1.29 .

Appendix

Running the H-HMM code

Refer to README.md for a detailed explanation describing how to reproduce the main fMRI experiments, including the preprocessing steps for the fMRI data.

Algorithm for fitting H-HMM

Algorithm 1 describes the H-HMM's fits to a single fMRI run. Fitting to all scan sessions is performed similarly, except that the forward-backward function is applied to each run separately. For full details see the README.md file in the attached code directory.

Algorithm 1 HyperHMM algorithm

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an HMM-based model, Hyper-HMM, for characterizing variability in temporal and spatial dimensions in fMRI sequence datasets. The model is a chain-structured HMM where each discrete state (event) defines a relationship between neural activity and a stimulus embedding. Importantly, the discrete state defines the mean of the subject's neural activity projected to a lower-dimensional space. Each subject has a different lower-dimensional projection matrix, while the events and stimulus embeddings are shared across subjects. These features allow for modeling spatial variability across subjects (via different projection matrices) and different temporal alignments (discrete estimation separate for each subject). The stimulus embeddings allow for identifying shared structure. \n\nThe model is validated in a simulated experiment and on a dataset of fMRI recordings while subjects listened to computer science lectures. The authors examine recovery of latent states and clusterings of neural activity/stimuli in the simulated data. In the analysis of an fMRI dataset, the authors find variations in sequential activity & spatial coding across subjects. Importantly, they show the learned projections output statistically meaningful clusterings on heldout fMRI runs.\n\nSTRENGTHS:\nThe proposed methods are original and significant for the analysis of fMRI datasets during sequential tasks across subjects. The idea to also embed the stimulus makes the learned latent state clusterings more interpretable, and may also help with identifiability. The experimental results relating semantic content of course videos to fMRI recordings across subjects appear very significant and useful for scientific analysis.\n\nWEAKNESSES:\nThe clarity of the methods and experiment could be improved. One example is it is unclear how the stimulus embedding is learned. At some points, the text implies that the stimulus is treated akin to a subject where a projection matrix is learned for the stimulus in the same way it is for the subjects. However, the stimulus is not included in Algorithm 1. Including more details on how the stimulus is incorporated into the model and learned would help with clarity. \n\nNext, some aspects of the modeling approach appear inconsistent which may limit significance. For example, the events $E$ are defined in voxel space whereas the event segmentation is done in the lower dimensional projection space defined by $W_i$. Next, the model fitting approach appears somewhat ad hoc, and it is not clear it corresponds to a single objective such as maximum likelihood estimation. More justification for the proposed fitting approach and simulated data analysis could rectify these concerns.\n\nQUESTIONS:\n* Can the authors clarify how the stimulus embeddings are learned?\n\n* Why are the events $E$ defined in voxel space when the event segmentation is done in the lower dimensional projection space defined by $W_i$? How does this compare to computing the events $E$ based on projected activity $W_i X_i$? I encourage the authors to consider an alternative fitting method that learns $G$ via the projected neural activity $W_i X_i$. \n\n* How sensitive is the model to initialization / how do results vary across different initializations? \n\nMinor comments - typos\n* Figure 2 caption: stimulated -> simulated\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops Hyper-HMM as a hybrid model that simultaneously aligns both temporal and spatial features across fMRI datasets. The proposed model learns a linearly project that maps voxels to a low dimensional latent space, in which timecourses are segmented into corresponding temporal events. The purpose of this model is to remove the effect of each individual’s mental trajectory through an event sequence, and to also align with other feature spaces like stimulus content. Overall, it is an interesting paper, however, there are several concerns about the machine learning novelty, validating the empirical studies, and the clear presentation of the proposed method.\n\nSTRENGTHS:\nPlease refer to the question section\n\nWEAKNESSES:\nPlease refer to the question section\n\nQUESTIONS:\nThe followings are the major concerns and minor comments:\n\n\n1) It is still unclear to me as to what the novelty of this paper is in terms of machine learning. There may be some contributions to computational neuroscience in this paper; however, which part of the machine learning approach is new in this paper? The author(s) of the proposed paper should specify whether there is anything theoretically new for the proposed approach or if this is merely an application paper for HMM techniques. \n\n\n2) My second concern is the design of the empirical studies. Even though the study contains several beautiful figures, more numerical analyses would be helpful to convince the reader that the proposed method is effective. There is a lack of regular analysis and conventional machine learning metrics (such as accuracy, dice, etc.) which make it difficult to understand the results. \n\n3)  The proposed method should be benchmarked in comparison with related state-of-the-art techniques.\n\n\n4) In this paper, the notations are confusing. In regular papers, scalers are denoted by small letters, vectors are defined with small letters (highlighted in bold), and matrices are denoted by capital letters using bold. In this paper, there are a lot of conflicts. It is so hard to trace what is a set, a matrix, or even a distribution. \n\n\n5) There are some minor linguistic and typo problems in this paper.\n\nLIMITATIONS:\nPlease refer to the question section", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop a method to identify and align events in the brain and external stimulus. They iteratively fit a Hidden Markov Model to find the times of events, and spatial characteristics of those events.\n\nSTRENGTHS:\nOriginality: This work is a minor update to past work, by accounting for time shifting as well.\nQuality: The authors use careful validation with simulated data, and careful cross-validation in real data.\nClarity: The paper is largely well written.\nSignificance: The main problem of comparing neural activity across subjects and referencing those patterns to interpretable stimulus-driven semantics is an important one, especially as the amount of data in the field continues to grow.\n\nWEAKNESSES:\nThe events in this paper refer to extended time periods. Events are often described as instants in time, rather than extended periods. It would be helpful to carefully articulate the definition of events to prevent misunderstandings.\n\nThe authors make quite strong assumptions, despite protestations that their method is very general. In particular, I am skeptical about the \"constant events\" in the time course, and about the linear embedding of the semantic content.\n\nThe authors only test out a narrow range of timing differences (25%), which leads me toward greater skepticism about the generalizality of the author's results.\n\nSuggestion: It would be useful to test this method in a simulation with a deliberately time-warped movie, to check whether the method can recover simulated data can recover the true timercourse.\nMinor: Figure 2 caption has a typo: should read \"simulated data\", not \"stimulated data\".\n\nQUESTIONS:\nL235: \"resulting in latent event representations that are largely orthogonal, which will fail to capture semantically-meaningful relationships and will fail to generalize to new events.” I don’t understand. Orthogonal components in event relationships should be common and natural. Why is orthogonality considered as meritorious?\n\nThree dimensions is shockingly low dimensionality of semantic space. This seems to me like analyzing image data and find the dominant principal component, the sky.\n\nLinear projections onto semantic space is highly restrictive and unrealistic. But it could be a reasonable lower bound on alignment.\n\nLIMITATIONS:\nThe authors do an excellent job of articulating their problem and the essence of their solution..", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUPDATE:\n\nI have raised my score and now support acceptance of this paper. I believe it will be a good contribution to the conference.\n\n-----------------------------\nThe authors propose an extension for an HMM model proposed by Baldassano and colleagues to align interindividually different brain responses in both space and time to analyze naturalistic stimulation settings. They test their approach through simulations and on an empirical dataset of 19 participants, who watched a series of 5 computer science lectures. They show that their model is robust to a number of noise levels in the simulation analysis. In the empirical analysis, the model appears to learn a meaningful latent space (concrete vs abstract and future-oriented vs present tense descriptions) and outperforms a null model.\n\nSTRENGTHS:\n- The research problem addressed is an interesting and timely problem, namely how to compare spatially and temporally heterogeneous responses across participants\n- The paper is largely well-written\n\nWEAKNESSES:\n- The sample size is small\n- The approach is not compared to other state-of-the-art models, rather the authors compare it to a null model, which is poorly explained\n- The authors do not provide analysis code or sufficient detail to reproduce their results\n\nQUESTIONS:\nOverall, this work tackles an interesting problem and shows some first promising results, but I believe that it would require quite some work like more extensive simulations, comparisons to other methods and testing in a larger dataset or a second dataset (to demonstrate that this method is not only useful for applications to naturalistic stimulus analysis, but other areas of neuroscience research, for example aligning multiple imaging modalities) to meet the NeurIPS standard. \n\nNonetheless, I really enjoyed reading the paper and I strongly encourage the authors to continue this interesting work. Here are some questions/comments that would allow me to increase my score. I hope you find them helpful and constructive. \n\n**Major points**\n- The actual model could be explained in greater detail. It would be helpful to include pseudo-code for the algorithm and the cost function explicitly in the manuscript.\n- Please, explain (at least in the supplement) the choice of preprocessing, e.g. quality control (motion detection/scrubbing, motion artifact rejection thresholds), software versions, scanner/coil details, filtering, slice-time alignment if conducted etc. \n- The choice of the simulation parameters is not clearly motivated, can you explain why you chose to simulate datasets with only 6 participants or only 5 voxels, this does not correspond well to empirical situations, where the sample would be larger, and #voxels >> #stimuli, which would be important to test whether your reweighting (p.4 125-126) works. It would also be helpful to express the noise as SNR to assess whether your method assumes realistic noise.\n- For the empirical analysis, it is unclear how you construct your baseline. Please, be more specific about how you computed that, otherwise it is impossible to assess the model performance\n- Related to that Figure 5 is not very clear, I am assuming that the black dot and the black diamond in Fig 5 reflect the actual value, whereas the grey shaded area is the null-distribution? This should be clearly stated in the figure captions.\n- Figure 5: It is quite striking that the fMRI stimulus match is much worse on the empirical data than the clustering. Based on the simulations (Figure 2), this is unexpected. Could you elaborate on what the reasons for this could be? \n- Figure 5: The fMRI-stimulus match null-distribution(?) is centered on a negative R^2. Can you explain why this happens? Could this be suggesting that the baseline model is not appropriate?\n- Will code be provided to ensure reproducibility of the analyses?\n- p. 6 ll. 218-221: You state that the latent dimensions appear to map onto semantically meaningful dimensions (concrete vs abstract and future-oriented vs present tense descriptions). Is this in line with the literature? Further discussion of this would be helpful.\n- p. 9 l. 268-270: “Our experiments demonstrate that [...] this model […] can learn meaningful mappings” To show that these mappings are meaningful, it would be useful to show that the latent scores correlate with an external measure that was not shown to the model, but would be hypothesized to relate to the captured dimensions (e.g., IQ, exam scores, abstract thinking scores or something of that sort). Is any information like this available. That would strengthen your claims.\n\n\n**Minor points**\n- Could you elaborate on the relationship between your method and representational similarity analysis (https://doi.org/10.3389/neuro.06.004.2008)? You very briefly touch on related work (e.g. from Haxby), but I think this could be expanded a bit.\n\nLIMITATIONS:\nThe authors have mentioned some limitations (linear assumptions), but the small sample size is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn natural tasks that share input stimuli across participants, the cognitive states or neural responses of different participants might undergo approximately synchronous but slightly jittered dynamics. At the same time, the distribution of neural signals are not exactly consistent across participants at voxels with the same spatial coordinates. These two problems were addressed by two approaches separately by event segmentation and functional alignment. This paper proposes a new approach H-HMM that combines the advantage of both methods. It performed simulation to evaluate its performance and tested it on an fMRI datasets of college students watching the same series of lectures in computer science. The performance appears impressive and achieves what the model is designed for. Further, the paper also demonstrated alignment between fMRI data and semantic features of contents in the lecture.\n\nSTRENGTHS:\n* Simultaneous achieving temporal and spatial alignment was not done before. \n* Comprehensive evaluation of the method is performed on both simulated and real data and showing good performance.\n* The illustration is generally clear and easy to understand.\n* The approach holds promise for a wide range of application and should be a significant contribution to the field\n\nWEAKNESSES:\n* I think there is a mismatch between the data size of simulated data and that of fMRI data or semantic features, making it a bit difficult to evaluate the expected performance of this method in general. The simulated data have only 5 voxels/features, which is rarely observed in the domain where this algorithm should be applied. The described numbers also do not match up: 4 events with 9-12 time points per event give rise to fewer than 48 time points, yet it is said that the simulated data have 60 time points. It is also strange that the event is represented by binary patterns. Maybe this is just for the purpose of easy visualization but I worry that the performance might depend on these unrealistic properties in the simulation. Doing it with similar property as would be expected for the fMRI data should be more convincing. If you encounter issue that the actual fMRI data have lower effective dimensionality due to smoothness, it may be simulated by spatial Gaussian process or other ways but I don't think it is justifiable to start with low-dimensional data, as you did not perform PCA on fMRI data to get similar dimensionality before application of the algorithm.\n* The description for the update of group-level G, starting from line 122, is a bit confusing. After the stacking, what is the dimensionality of the matrix? I assume the stacked data have a total number of elements as the number of events * (total number of voxels+ semantic features) * D (since E is in the D-dimensional space). If your stacked data have two dimensions (which I assume is the case since you can do PCA on it), which of the two dimensions (after explaining their size) do you treat as data features and which do you treat as samples in the definition of PCA? After projecting the stacked data for each event into D-dimensional PC space, do you need to do anything you do to get G (which is of the shape of number of events * D)?\n\nQUESTIONS:\nMy major questions are asked in the weakness, which I think are addressable.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}