Learning shared neural manifolds from multi-subject FMRI data
Authors: Jessie Huang
Erica Lindsey Busch
Tom Wallenstein
Michal Gerasimiuk
Guillaume Lajoie
Guy Wolf
Nicholas Turk-Browne
Smita Krishnaswamy
Venue: ICLR 2022 Submitted (2022)

Abstract: Functional magnetic resonance imaging (fMRI) is a notoriously noisy measurement of brain activity because of the large variations between individuals, signals marred by environmental differences during collection, and spatiotemporal averaging required by the measurement resolution. In addition, the data is extremely high dimensional, with the space of the activity typically having much lower intrinsic dimension.  In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure. Specifically, we assume that while noise varies significantly between individuals, true responses to stimuli will share common, low-dimensional features between subjects which are jointly discoverable. Similar approaches have been exploited previously but they have mainly used linear methods such as PCA and shared response modeling (SRM). In contrast, we propose a neural network called MRMD-AE (manifold-regularized multiple-decoder, autoencoder), that learns a common embedding from multiple subjects in an experiment while retaining the ability to decode to individual raw fMRI signals. We show that our learned common space represents an extensible manifold (where new points not seen during training can be mapped), improves the classification accuracy of stimulus features of unseen timepoints, as well as improves cross-subject translation of fMRI signals. We believe this framework can be used for many downstream applications such as guided BCI training in the future. 

Under review as a conference paper at ICLR 2022
LEARNING SHARED NEURAL MULTI-SUBJECT FMRI DATA
MANIFOLDS
FROM
Anonymous authors Paper under double-blind review
ABSTRACT
Functional magnetic resonance imaging (fMRI) a notoriously noisy measurement of brain activity because of the large variations between individuals, signals marred by environmental differences during collection, and spatiotemporal averaging re- quired by the measurement resolution. In addition, the data is extremely high dimensional, with the space of the activity typically having much lower intrinsic dimension. In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn meaningful embedding of the data that denoises, and reveals its intrinsic structure. Specifically, we assume that while noise varies significantly between individuals, true responses stimuli will share common, dideensinionnl features between subjects which are jointly discoverable. Similar approaches have been exploited previously but they have mainly used linear meth- ods such as PCA and shared response modeling (SRM). In contrast, we propose a neural network called MRMD-AE (manifold-regil regularized multiple decoder, autoen- coder), that learns a common embedding from multiple subjects in an experiment while retaining the ability to decode to individual raw fMRI signals. We show that our learned common space represents an extensible manifold (where new points not seen during training can be mapped), improves the classification accuracy of stimulus features of unseen timepoints, as well as improves cross-subject transla- tion of fMRI signals. We believe this framework can used for many downstream applications such guided brain-computer interface (BCI) training in the future.
INTRODUCTION
Brain activation dynamics, as measured by fMRI, exist in extremely high-dimensional collection space and contain high levels of noise. Noise sources include measurement imprecisions and physio- logical factors such as a subject's 's blood pressure variability, head motion, and respiration (Brooks) ct al. 2013; Mumford Poldrack 2007). Morcover, even after accounting for the idiosyncrasies in noise, neural responses evoked by the same stimulus can vary greatly between different subjects. This makes using fMRI to extract meaningful signals or general trends that are task-relevant quite difficult. Experiments involving IMRI are expensive and usually involve pilot studies conducted in smaller scale with simpler stimuli and on a smaller group of subjects. Thus, it is desirable to have a method that can learn common trends from pilot data and can generalize a larger group of subjects or more complex tasks.
Like for other high dimensional biomedical data, it is broadly acknowledged that neural activity actually lies in lower- dimeersiiiiiionll latent space. Moreover we hypothesize that several features of these latent spaces are shared across subjects given that measurements are usually task-based and are taken when subjects experience the same stimuli. Dimensionality reduction methods have been used to discover such low-dimensional spaces and facilitate understanding the underlying brain activities. Methods such principal component analysis (PCA) and factor analysis have been the common choice (Smith et al. 2014). However, these linear methods are sensitive to noise and restrict interactions in the latent space. Nonlinear dimensionality reduction algorithms
shown to better capture the geometry of dige-dimnnsional biological data (Moon et al. 2019 Van der Maaten & Hinton 2008) and account for dynamics in neural activities (Gao al. 2020). PHATE (Moon et al.

---

Under review as a conference paper at ICLR 2022
2019), diffusion based manifold learning method, was shown to capture both global and local geometry of complex biological data, which appears to be good candidate to model fMRI data.
Using nonlinear manifold such PHATE reduce the dimensionality of data is appealing it can extract meaningful signals, denoise the data, and accelerate downstream analysis. However, unlike a method like PCA which learns a projection operator that can be applied to new data, manifold learning methods are fixed to the input data and do not naturally extend to new data from new tasks or from new subjects. To extend an existing manifold to out-of-sample data, landmark interpolation or Nyström extensions are commonly used, However it is usually unsatisfactory as shown by decremented performance(Appennn Fig. []. To tackle these shortcomings, we turn to neural networks such as autoencoders that provide nonlinear dimensionality reduction via a learned parametric function that readily applicable to new data. While our key goal is learning an informative owwddimensional data manifold shared across multiple subjects, this is not something automatically done by autoencoders. First, there nothing enforeing autoencoder latent embeddings to respect data manifold geometry, and indeed they often just spread out points for ease of decoding. Second, individual variation often dominates embeddings which prevents a common space to be automatically learned.
To address these issues, we propose regulaid-regulariizd multi-decoder autoencoder (MRMD-AB) that can process fMRI data from a group of subjects and extract common latent space representations that respect individual data geometry. Key features of the MRMD-AE include: (a) A common encoder that projects fMRI data from any subject on to a shared latent space, and subject-specific decoders that learn to reconstruct data for each subject faithfully. (b) A manifold-geomette regularization of the latent space based on a precomputed PHATE embedding, (c) Penalties for distances between individual patient encodings of common stimuli ensure that the latent space is not split into individual embeddings.
We show results on three types of tasks in two different datasets. First, we show that we are able to extend the manifold embedding to new timepoints of data not used in training. This shows the extensibility of our manifold in contrast to non-neural network based manifold learning methods. Second, we show that our latent space improves the ability to classify or infer stimulus features based on subject fMRI measurements. Surprisingly, the multiple decoder improves classification accuracy over common decoder, showing that this framework allows the network to separate common from individual variations. Finally, we show that even untrained, translation between subjects has increased accuracy withheld timepoints. Further, we show highly improved cross subject translation after training for this task.
2 RELATED WORK
Manifold Learning Manifold learning assumes that high- nigh-dimensional ambient data X e Rd lies on a low dimensional manifold. Given set of datapoints measured in the ambient space, assumed to be sampled from a manifold M, manifold learning optimizes a low dimensional Euclidean space that encodes the intrinsic geometry of M and the original data. Popular manifold learning methods include diffusion maps (Coifman & Lafon 2006). Laplacian Eigenmaps (Belkin & Niyogi 2003) and PHATE (Moon et al. (2019). We note that multiscale manifold learning methods (Wolf al. Kuchroo et al. (020) can be used to consider multiple manifolds that capture relations and structure in data at different resolutions. Other methods, such as tSNE (Van der Maaten & Hinton 2008) and UMAP (McInnes et al. 2018)) relax the manifold assumption to focus on neighborhood preservation in low dimensions. While these are often considered as variations of manifold learning, they are mostly suitable for visualization of clustered data that does not necessarily have global structure, and typically do not preserve overarching trends or relations between data regions (see, e.g., Moon et al.
2012 Brugnone et al.
2019 Gigante et al. 2019).
SAUCIE In (Amodio et al. 2.19), SAUCIE (Sparse Autoencoder for Clustering, Imputation, and Embedding) is an autoencoder based generative model that manipulated internal representations to force the network to produce specific desired transformations in the data. By imposing these structure on the latent representations of data, it can quickly extract information about datasets more massive than alternative approaches can easily handle. Further, this framework establishes that multiple regularizations, in conjunction, can be used to encourage interpretable latent representations that accentuate biologically reliable relations in data coming from multi-patient cohorts, as demonstrated, e.g., in immune response profiling (Zhao et al. 2020). Inspired by the success of this approach in
2

---

Under review as a conference paper at ICLR 2022
immunology, we apply a similar multi multi-objective neural network approach here to process multi- subject fMRI data and alleviate some of the main challenges associated with this data type.
Shared response modeling (SRM) is linear method to account for differences in functional topographies fMRI data in a low-dimensional space. SRM uses a singular value decomposition approach to learn set of low-dimensional signals (shared response) common multiple subject's fMRI data during the same experiment. SRM learns subject-specific transformation to align subject data with the shared model using an orthonommally-coonsrineee weight matrix, similar that of PCA (Chen et al. 2015).
SRM
3 PRELIMINARIES
Manifold learning and dimensionality reduction with PHATE tion in the latent space and balanced global and local structures of highly noisy fMRI data, use Potential of Heat-diffusion for Affinity-basee Transition Embedding (PHATE) (Moon et al. 2019), a manifold learning method based on diffusion geometry (Coifman & Lafon 2006). PHATE computes the diffusion operator which is the probability transition matrix P between data point pairs via an a-decay kernel. Then by raising to the power of t, the PHATE optimal diffusion time scale, simulates t-step random walk over the data graph. Diffusion potential distances D is extracted from pt, which accounts for both global and local data geometry. Finally, metric multi-dimensional scaling (metric MDS) (Abdi 2007), is applied to D to get the PHATE embedding of input data. PHATE has been shown to effectively capture geometry of high- sigh-dimensional biological data and was shown to preserve manifold structure in fMRI data in (Rieck et al. 2020). We utilize the extensible framework of PHATE and use PHATE embedding of the training data as regularization applied to the manifold embedding layer as shown in Fig. Note that we use PHATE over other popular dimensionality reduction methods such as tSNE or UMAP because of its established ability to retain
To capture the complex interac-
manifold and trajectory structure in biomedical data (Moon et al. 2019 Kuchroo et al.
2020 Chung
et al. 2020 Pappalardo et al.] 2020).
regglaty-reguulrrized Autoencoders (GRAE) The latent space of the vanilla autoencoders are usually difficult to interpret and not necessarily reflect data geometry (see, e.g., [Mishne et al.) 2019)] Moor et al. 2020: Duque et al. 2020). In order to encourage meaningful data geometry to emerge, or be retained, latent representations, the training process can be enhanced with appropriate regularizations, often derived from manifold learning principles. For example et al. (2015) and Yu ctt al. (2013) propose regularization terms that penalize inaccurate preservation of neighborhood relationships in the ambient space. Recent work by Duque et al. (2020) has shown that the geometry extracted by PHATE can be leveraged to formulate Geometry- Regularized Autoencoders (GRAE) that faithfully capture the instrinsic data geometry, while demonstrating advantages in extendibility and invertibility compared to previous approaches. In this work we adapt this approach to fMRI data in conjunction with adjusted architecture and additional regularizations that address further challenges encountered in these settings.
Cross-subject batch effects in fMRI Relative to other neuroimaging methods, fMRI is noninva- sive, minimal risk, and has broad coverage with high spatial resolution. fMRI experiments commonly use naturalistic stimuli such movies to probe arryy of cognitive mechanisms in conditions that closely mimic the human experience of the world. However, the data generated by such experiments notoriously high-dimensional and complex, as fMRI is rife with noise from sources like the scanner, subject movement, and the measured blood-oxygen-levee dee-eeee---eeeeedeen (BOLD) signal, which is a secondary effect of neural activity. This makes it difficult aggregate data across sessions, either within or between subjects in a study. Beyond the noise sources of fMRI data, another problem with aggregating data across subjects to uncover something common about brain functioning pertains to the idiosyneratic nature of functional topographies in the brain. In many brain regions, shared stimuli will reliably evoke similar patterns of neural activity across subjects (Nastase et al., 2019). However, that signal of interest could be located in slightly different anatomical locations across subjects or otherwise obscured by noise, causing a mismatch between functional and anatomical locations for a signal. A large body of neuroimaging research centers around rectifying this problem with a variety of linear functional alignment methods by learning either nigh-dimensional (Haxby et al., 2011 (Chen et al. 2015) common spaces. A solution this
Busch et al.) 2021) issue must consider both the complexity of the data at the individual subject level, where there is biological and environmental noise, and the broader cross-subject representational idiosynerasies.
3

---

Under review as a conference paper at ICLR 2022
4 MULTI-DECODER MANIFOLD-REGULARRIE AUTOENCODER
4.1 PROBLEM SETUP
Let x= (Xx: =,. ....., m. be set of common stimuli for timepoints, and let Y; (Y5,k be observed fMRI neuronal activations for subjects
Rº k = l..... m). 1, :..... 17 these stimuli. We seek to learn low dimensional latent space M C R (where d < q) with [- I...,,, ,, such that: (a) The mappings corresponding to each stimulus X₃ in subjects Y, and Yy are close to one another, i.e., - ll22 S c. (b) Stimulus features (for each X₂.) can be computed by function f(X,,) f(Xee of the low dimensional latent space. (c) Stimulus features can be inferred in an unseen/untrained subject Y from the same latent space via f(Y++1,&) + f((Xa), any timepoint k. (d) Subject-specific responses can be predicted on unseen stimulus X (e.g., at held-out timepoint m + 1) based on the shared latent space, i.e., gj g,(f(Y,,++)) subjects j. Note here that each Y,, is a high dimensional measurement consisting of raw activation values at hundreds to thousands of voxels depending on the respective region of interest (ROD) By contrast, expect f(Y)) to be vastly lower dimensional. We analyzed the spectral entropy of PHATE diffusion potential of the fMRI activation data used in the following experiments and found the intrinsic dimensions of the data ranging 10 to 40 (Appendix Table
associated map f IR + I2 and inverse maps 9j : R2
y,,m- for
4.2 MRDM-AE ARCHITECTURE
monifold
91
embeoding layer
09009000
shured embedding byyer
P1
Y Y/2
O2
00009000
000000
P2
Yn
9Tz
0 0
9,
9
8
input
common encoder f
Individual decoders Øy
reconstruction
Figure 1: Multi-decoder Manifold Regularized Autoencoder
Single Encoder Multiple Decoders To achieve the properties outlined in the previous section, we propose a manifold-regularizedd common-encoder,, multiple-decoder model (Figure [ The common encoder fY,,,) = Enc(Y, takes any set of neuronal activities from any subject and maps them in a many-to-one fashion to the same latent space L, with the same stimulus prediction. Our model features several decoders gy (f
YjA, to decode back to subject specific responses. The rationale for using common encoder is for the model to learn common (MRI language from which subject specific responses can be derived. While the fMRI data exhibits strong batch effect between subjects, the common encoder builds invariance in the latent space to individual noise the encoded space. The individual decoders enable incorporating subject- dependent interpretation of the latent information from the common encoder. Thus this framework pushes individual noise into the decoding space. In particular, the decoders can learn to reconstruct the subject-specific signals and recreate batch effect reconstruct data. The flexibility provided by individual decoders allow the single common encoder to only retain input-dependent variability between subject responses that is sufficient for the decoders to reconstruct the measurements. This base model is trained with a reconstruction error penalty for each subject seen at training time.
(fc,))(((Y..e))
f. f.gt.....a (Y...... Yn) = Σ |Y (f(,,))
j.k
'Duiing preprocessing. functional images were aligned to the common template hased on anatomical landmarks and time point of fMRT region of interest vector of length equal the numher of voxels.
4

---

Under review as a conference paper at ICLR 2022
Embedding Alignment Penalty Despite the common encoder (and individual decoders), the network may learn to separate subject-specific responses into their own latent sub-spaces for ease of decoding. To counteract this tendency explicitly push the latent encodings of individual responses to common stimuli together by penalizing Euclidean distances in the encoded space between timepoints corresponding to common stimuli. Therefore, we implement the embedding alignment penalty as L Zaliggment '........ (Yi,.... Y,,) Ei (Y, f(),,))l
Manifold Regularization Penalty In order to ensure smooth embedding space that enables faithful interpolation, inference, and extendibility to new data, a geometric regularization should be applied to ensure the latent space learned by the autoencoder follows the data geometry. To this end, within each decoder, we apply a regularization in the bottleneck layer using the manifold embedding P. that are learned with PHATE (Moon al. (2019)) using the train data for each subject j. We note that the strong subject batch effects in iMRI data make the task of directly learning a subject-independent manifold with PHATE challenging. Indeed, without careful consideration, often yields a group fMRI data geometry with variance dominated by batch effects, which would then submerge stimulus-related variance that is of actual interest. We therefore bypass this issue here by only considering individual-subject manifolds and letting the combined autoencoder deduce a latent representation that can easily be mapped to them. Thus, as shown in Figure we split each decoder into 9; = 9 ogg where intuitively we expect the first part 9 to "implement batch effects" going from the common latent representation to subject-dependent one that captures well the intrinsic structure of their stimuli responses. It is to this first layer of 9/ we apply the manifold regularization. The second part 9y will then map this intrinsic (manifold) representation to the observable measurements to finish reconstruction. In this way, we obtain an intermediary representation after the first layer of each decoder (where the single-layer limitation restricts the implemented "batch effects" to be relatively simple) that should match the PHATE embeddings. implemented in training via the manifold regularization as I. geometric (Yi,..., Y.) = Ils ((G,,))) - (1)
To combine three loss terms together, we introduce coefficients A, controlling the amount of alignment and geometric regularization (correspondingly) applied the hidden layers of the MRMD-AE, yielding the total loss
Combined Loss
L = Σ |& - (5(()))))) + j.k
Σ *jj.k
(I)
optimized over network functions f.g; = 1- o = 1,...,m, m. ww..mmmttreeeeeeeddddd via neural networks
and organized together as illustrated in Fig. 1
Additional Cross-Subject Training Penalty The multiple decoder framework allows us to add carefully designed loss penalties for specific tasks. For example, we can explicitly train for cross- subject translation. We achieve the purpose by adding the following regularization to encourage translation,
Imanslation 1. 9n (Y Yn) ΣΣ J,k 173
1Y - (f((()
(2)
In most of the following experiments this cross-subject penalty was not used, as we did not aim to train for translation. In the last section of Results we explore the use of this penalty to facilitate learning to translate between subjects' fMRI activities.
RESULTS
To empirically validate MRDM-AE, we test it on tasks of manifold extension, stimulus classification, and cross- cross-subject translation. The manifold-extension tasks test the ability of MRDM-AE to place points not seen during training appropriately within the learned data manifold in the embedded space. The classification tasks analyze the ability of MRDM-AE to capture stimulus-relevant signal from fMRI data, i.e., features of the movies from voxel activations. Cross-subject classification and translation tasks test the ability of MRDM-AE to capture shared latent space that can be used for common encoding and individual decoding. These tasks are tested on two different datasets comprising of subjects watching different movies. We compare MRDM-AE to several baselines including dimensionality reduction methods (PCA, PHATE), the shared response model (SRM) commonly used in fMRI data, well as ablations of our autoencoder method.
5

---

Under review as a conference paper at ICLR 2022
Datasets The Sherlock dataset is an open-access dataset of 16 participants watching 48- -immteete clip of the BBC television series "Sherlock." Data was collected in two runs of 946 and 1030 timepoints (repetition time; TRs) respectively. This data was downloaded from the DataSpace public repository| For full details on the stimulus, see the original publication of this dataset (Chen et al. (2017)) Annotations for the Sherlock movie stimulus were released with Vodrahalli et al. (2018) and available freely on Github These annotations were created by humans who viewed the film and labeled each timepoint on a variety of metrics. In our classification tasks, we used two sets of binary labels that correspond with visual and auditory aspects of the stimulus: whether given timepoint featured an indoor or outdoor scene (Indoor/Outdoor) and whether or not the timepoint featured music (Music Present).
As second testbed, we used the Study open-access dataset downloaded from Datal.ad full details for this can be found at the original publication of the movie dataset (Hanke et al. (2014) and the localizer extension dataset (Sengupta et al. (20T6)). Here we included data from 15 participants who completed both the nivie-viewing and funcciionl localizer tasks, where participants viewed 24 unique grayscale images of objects from categories (faces, bodies, houses, small objects, outdoor scenes, and scrambled images) in separate session to localize brain regions that reliably respond to a specific category. In the movie task, participants watched a 2-hour version of the movie Forrest Gump. In a series of classification tasks, we use labels for each timepoint of the movie indicating the time of day (2 levels), and the flow of time in the narrative (4 levels), annotated in (Husler & Hanke (2016). ). and image categories for the localizer data (6 levels).
Autoencoder Hyperparameters All autoencoder based models have three fully connected layers in the encoder and the decoder(s), with bottleneck latent space layer in between. The hidden neurons have the configurations of 256-128-66 128-64-20-64 124-128-256 Leaky ReLU activations are applied on all of the layers. We used the Adam optimization algorithm with learning rate of 0.001, and a batch size of 64. The neural networks were trained for 4000 epochs. When regularization is used, À and are either 0.1 or 0.01 and the same set of values are used across subjects for specific task.
Model Comparison We compare our proposed model with a standard autoencoder (AE) with the same hidden dimensions as the proposed MRMD-AF, but with single decoder for all subjects. We refer a standard autoencoder with manifold regularization applied to the bottleneck layer as MR-AE.
Dimensionality Reduction Methods Non-linear dimensionality reduction methods such as PHATE (and other methods such as I-SNE and UMAP) typically provides fixed latent representations for the input data, and do not provide a natural embedding function that can project new data to an existing manifold. PHATE preserves manifold structure while ISNE and UMAP usually shatter data trajectories to create clusters even when they do not exist (Moon et al. (2019). Kuchroo et al. (2020), Rieck et al. (0020). We therefore compare with an adaption of PHATE to interpolate new datapoints via landmark approach used by Moon et al. (2019), a subsampling method that was used and shown to be effective in improving scalability of PHATE. We also compare with linear PCA approach. For within-subject analyses, the dimensionality reduction was learned for subject one half of the dataset and applied to unseen timepoints. For cross- cross-subject analyses, the dimensionality reduction was learned all but one subject and applied to the held-out subject, and repeated holding out each subject once. For cross- cross-subject analyses, we also compared to SRM by learning a reduced- dimensional shared signal on data from all but one subject and evaluating the fit of the shared space on the held-out subject, repeating for all subjects.
Place Future fMRI Time Points on Learned Manifold We first show that MRMD-AE achieves efficient and accurate manifold extension by placing new fMRI time points at the right positions on learned manifold. Consider the fMRI time series contain two parts, including the first merain TRs that we receive as pilot data with which we learned individual manifolds Ptrasns a Meroin x d matrix, with each row being the PHATE coordinates of one TR. We would like to faithfully extend these learned manifolds to subsequently receive test data of future miest fMRI TRs, so that the extended manifold Ptest closely resemble the corresponding time points ground truth manifold P, that is learned from all Merain + Miest TRs.
Since the manifold coordinates are fixed the input data, we cannot directly compare the Prest with Rtest, the corresponding rows of P. Instead, we first compute the manifold distance matrices between each test and train time point pairs, where D(i, = l[Ptese(i.)
Ptrain (j. )ll. and the
ground truth Dii.j) j) = [[P(i.)- P(j ll. T E T Rerainn Then we evaluate the similarity using the MSE between D and D. Fig
2 shows that MRMD-AE achieves lower or matching MSE than the landmark approach, especially with lower percentages of train data. The extended manifold

---

Under review as a conference paper at ICLR 2022
Stimulus Classification Accuracy
0.9
Landmark
Landmark Acc
0.004
MRMD-AE MSE
MRMD-AE Acc
MR.AF MSE
0.8
MR-AE Acc
0.003
MSE
0.7
0.002
0.001
0.000
0.5
50
70
30
50
70
Train TR Percentage Figure 2: Comparison of extended manifold by MRMD-AE, MR-AE and PHATE Landmark on the Sherlock sarly-visute ROI. The MSE between extended manifold coordinates and ground truth were computed (lower is better). MRMD-AE extended manifold embeddings achieve highest classification accuracy in predicting movie stimulus labels. Error bars standard deviation across subjects.
Train TR Percentage
generated by MRMD-AE more closely resemble the ground truth. MRMD-AF also outperforms MR-AE, suggesting the advantage of per-subject decoders in reconstructing -ebjicc--ppccfic signals.
We also use the extended manifold embedding to predict movie stimulus labels. An support vector classifier (SVC) trained on each subject's extended manifold coordinates and the average accuracy and standard deviation via a 5-fold cross validation are reported. We found that MRMD-AE extended manifold achieves higher classification accuracies than the landmark approach, suggesting the embeddings retain more meaningful signals that can be used to recover signals of the movie stimulus (right panel Fig. 2 and T..
Effect of Latent Space Alignment Penalty We visualize the raw data and the embedding f(Y)), = 1, 2. 15 of the Forrest image localizer data using PHATE, and color the plots by subject ID in 3 The raw data visualization exhibits clear batch effect. By increasing À in Eq. ② and penalizing latent space misalignment, the batch effect in the latent space is eliminated. We computed the average Earth Mover's Distance (EMD) between subject embedding pairs and showed improved alignment between subjects in the latent space comparing to the raw data and the improved alignment by increasing alignment penalty.
Stimuli Classification Our goal to test how well our MRMD-AE generalizes to unseen times series of IMRI data. We evaluate the model by how well we can recover meaningful labels in the latent space. After training on the train half of data from all subjects, we applied the trained model on the test half of data and collected the latent space representation at the bottleneck layer. A SVC trained within each subject on the test embedding and the average classification accuracy and standard deviation via 5-fold cross validation are recorded. The average accuracy and standard deviation across subjects are reported.
Results are reported in I Sherlock movie annotation classification and table for Forrest movie and image localizer classification. MRMD-AE effectively recovers the latent factors corre- sponding to the stimuli and achieve the best or second best performance in all combinations of ROI and stimulus annotations. We showed previously that a learned PHATE manifold via landmark does not generalize to out-ll- data. Here confirm that the classification accuracy of the test set decreases significantly from the train set. The linear projection operator learned by PCA also fails to generalize well to new data (Appendix Fig. [ shows in-sample out-of-sample comparison for PCA and PHATE Landmark). MRMD-AE also outperforms per-subject individual MR-AE, suggesting the synergy between subjects offered by the shared encoder and multi-decoder framework benefits generalization and recovering meaningful signal from data. Appendix Figure shows that across subjects, MDMR-A consistently outperforms the other methods.
Untrained Cross-Subject Translation Our MRMD-AE framework allows integration of data from multiple subjects, thereby enabling translation between subjects. Similar as before, we trained MRMD-AE with the train half of all subjects' data. Then translate the test half timepoints of subject to each of the other subjects by Y,,, = 9, (Y:)). We evaluate the quality of translation by comparing the MSE and correlation between Y and Y 3 It should be noted that the model was not trained for translation, thus in training, the decoders did not learn to translate other subjects data. Yet. the common encoder maps the neural activities to the same latent space and builds invariance to individual noise in the encoded space, allowing the decoders to perform the translation task. SRM was
7

---

Under review as a conference paper at ICLR 2022
Table I: Sherlock within-subject manifold classification. Within each subject's data, latent space was learned on one half of the data and extended to the held-out timepoints. support vector classifier was trained/tested on the latent representation of the unseen timepoints with 5-fold cross validation to predict features of the movie stimulus. The mean and standard deviation test accuracy across subjects and folds were reported. MRMD-AE outperformed all comparison methods suggesting efficient manifold extension to new time points and better preservation of factors from the stimulus.
IndoorQuldons
MusicPresent
ROI
Method
Accuracy
Std. Dev.
Accuracy
Std. Dev.
PCA
0,5923
0.0629
0.5215
0.0247
PHATE-I PHATE-Landmard Lndmmark
0,5681
0.0333
0.5530
0.0302
carly-
AE
0.7379
0,0499
0.6840
0.0240
visual
Individual MR-AE
0.7289
0.0196
0.6746
0.0259
MRMD-AE
0.7760
0.0520
0.7170
0.0383
PCA
0,5948
0.0606
0.5544
0.0314
PHATT--andmark
0.5831
0.0290
0.5665
0.0296
early-
AE
0.7285
0.0445
0.6950
0.0268
auditory
Individual MR-AE
0.7213
0.0315
0.6875
0.0309
MRMD-AE
0.7523
0.0544
0.7128
0.0381
PCA
0.5622
0.0640
0.5216
0.0244
PHATE-Landmald
0.539S
0.0368
0.5342
0.0316
pmc
AE 0.7080
0.0446
0.6585
0.0243
Individual MR-AE
0.7096
0.0207
0.6600
0.0265
MRMD-AE
0.7318
0.0589
0.6845
0.0403
Table 2: Forrest within-subject manifold classification. For Flow of Time and Time Day classificationss latent spaces were learned one half of the movie timepoints and then extended to the held-out timepeints. For the Localizer classification, latent spaces were trained on the entirety of the movie watching dataset and then extended to the localizer dataset, which was collected in separate session. Results are reported as the average and standard deviation across subjects and cross validation folds.
Flow of Time
Time of Day
Localizer
ROI
Method
Acc.
Std. Dev.
Acc.
Std. Dev.
Acc.
Std. Dev.
PCA
0.4590
0.0186
0.6406
0.0322
0.3965
0.0464
PHATE-Lamdmark
0.2434
0.0135
0.5266
0.0233
0.3863
0,0182
early- visual
AE
0.5547
0.0210
0.6025
0.0298
0.4542
0.0561
Indiv. MR-AE
0.5771
0.0242
0.6695
0.0197
0.4321
0.0484
MRMD-AF
0.5845
0.0192
0.6912
0.0216
0.44668
0.0683
PCA
0.4690
0.0167
0.6450
0.0287
0.4442
0.0187
PanTa-Landmark
0.2540
0.0219
0.5244
0.0236
0.3817
0.0104
late-
AE
0.5613
0.0220
0.6278
0.0270
0.4153
0.0491
visual
Indiv. MR-AE
0.5773
0.0227
0.6659
0.0229
0.4023
0.0392
MRMD-AE
0.5837
0.0209
0.6816
0.0237
0.4255
0.0514
Table 3: Cross-subject Translation Comparison when MRMD-AE not trained for translation.
The translation accross subjects Y.- gj (Yr)), + j is compared with the target subject Y,- Lower MSE and higher correlation indicate better translation. MRMD-AE trained with latent space alignment penalties (A 0 in Eq. achieves better translation.
MSE
Correlation
ROI
Method
Reconstruction
Translation (untrained)
Reconstruction
Translation
(untrained)
PCA SRM
0.1548
1.3903
0.9193
0.2275
early- visual
0.1640
1.3504
0.9142
0.2327
MRMD-AE (A 0)
0.1797
1.4321
0,9058
0.2235
MRMD-AE (A 0)
0.1933
1.3626
0.2517
PCA
0.0752
1,5544
0,9616
0.1928
SRM
0.0766
1.5130
0.9609
0.1951
pmc
MRMD-AE (A 0)
0.0848
1.5540
0.9567
0.1900
MRMD-AE (A > 0)
0.1647
1.4795
0.9139
0.1977
8

---

Under review as a conference paper at ICLR 2022
developed to achieve this goal by mapping all subjects to shared response, then project to individual spaces. We show that our model achieves best translation performance on various brain ROIs in at least one metric. By incorporating latent space alignment penalty (A > 0 in Eq. D. MRMD-AE learns a more aligned shared latent space and allows for better translation across subjects. More experimental details can be found in the supplementary materials.
row data
lambda 0.1
lambda 10.0
lambda 100.0
10
EMD: 0.008517
0,008425
0.007875
0.004901
Figure 3: PHATE visualization of raw data and latent representation f(Y,), و 15, colored by subject. The raw data exhibits clear batch effect while the latent space is better aligned between subjects. The incorporation of higher levels of latent space alignment penalty by increasing À in Eq better removes batch effect in the shared space, demonstrated by decreasing EMD.
Trained Cross Subject Translation Our MRMD framework provides flexibility to specialize in different tasks by incorporating carefully designed loss penalties. By adding a cross-subject training penalty in Eq. to the combined loss, explicitly train our model for translating between subjects. We show in Figure the improved translation performance by using different combinations of loss penalties. Combining the alignment penalty with translation penalty achieves the lowest translation MSF.
Alignment Penalty
Translation Penalty
Transiatoon
MRMD-AE(L)
X
X
MRMD-AE(2)
X
MRMD-AE(3)
v
v
SAM
MRMD A1M-LNEE222E112EMMMMEEE
Figure 4: Cross subject translation MSE comparison. Combinations of latent space alignment penalty and translation penalty were used. MRMD-AE with better aligned latent space achieve better translation MSE. We also trained the model explicitly for translation by incorporating the translation loss, achieving the best translation performance with the lowest translation MSE.
DISCUSSION
Extracting meaningful brain mechanisms shared across subjects daunting task, chiefly due to Hiimulus-indeeente noise and the challenge of comparing subjects' activity pattern in common reference space. To address this, we proposed MRDM-AE, multi-decoder caanfold-regulariin autoencoder that takes fMRI data from group of subjects as input, first projects it to common latent space, then subject-specific decoders. We showed that the subject-specific decoders enabled the autoencoder to decouple shared information from individual variations such that the shared latent space offered higher-accurage classifications of underlying stimuli, and better manifold extensions to unseen data. Finally, we showed that the individual decoders can potentially be used for subject-to- subject translation, setting up fMRI response predictions subjects to unseen data based on training and calibration common stimuli. Overall, show the manifold learning and structural priors bring in improve the state of the art in shared latent space learning in fMRI from many subjects.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The results presented in the paper are reproducible. We will provide a link to an anonymous repository the reviewers and area chairs. The content of the link will contain complete instructions on how to run the code to reproduce the results presented for the paper. In addition, all details for the models and training procedures are described in the main paper under Sec. 4 and the Appendix.
REFERENCES
Hervé Abdi. Metric multidimensional scaling (mds): analyzing distance matrices. Encyclopedia of measurement and statistics, pp. 1-13, 2007.
Matthew Amodio, David Van Dijk, Krishnan Srinivasan, William S Chen, Hussein Mohsen, Kevin R Moon, Allison Campbell, Yujiao Zhao, Xiaomei Wang, Manjunatha Venkataswamy, et al. Explor- ing single-cell data with deep multitasking neural networks. Nature methods, (6(11):1139-11 2019.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373-133 1396, 2003.
Jonathan Charles William Brooks, Olivia Kate Faull, Kyle TS Pattinson, and Mark Jenkinson. Physiological noise in brainstem firi. Frontiers in human neuroscience, 7:623, 2013.
Nathan Brugnone, Alex Gonopolskiy, Mark W Moyle, Manik Kuchroo, David van Dijk, Kevin R Moon, Daniel Colon-Ramos, Guy Wolf, Matthew Hirn, and Smita Krishnaswamy. Coarse graining of data via inhomogeneous diffusion condensation. 2019 IEEE International Conference Big Data (Big Data), pp. 2624-2633. IEEE, 2019.
Erica I.. Busch, Lukas Slipski, Ma Feilong, J. Swaroop Guntupalli, Matteo Visconti di Oleggio Castello, Jeremy F. Huckins, Samuel A. Nastase, M. Ida Gobbini, Tor D. Wager, and James Haxby. Hybrid hyperalignment: A single high-dimensional model of shared information embedded in cortical patterns of response and functional connectivity. Neurolmage, 233, 2021.
Janice Chen, Yuan Chang Leong, Christopher Honey, Chung H. Yong, Kenneth A. Norman, and Uri Hasson. Shared memories reveal shared structure in neural activity across individuals. Nature neuroscience, 20(1):115-125, 2017. ISSN 1546-1726 1097-6256.
doi: 10. 10.8/mn.
4450. JtRpptttps:////oomeeee ncbi. nlm nih. nlm. nih. http:/hwwww/o.netttttttee///aaaa
/www. ncbi
Po-Hsuan (Cameron) Chen, Janice Chen, Yaara Yeshurun. Uri Hasson, James Haxby, and Pe- ter Ramadge. A reduced-dimensiog fmri shared response model. C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL ht http://es////oo.eeeinii..../ neurips. cc/ pdf
Sapper/2015/11e//200 htw.:///.......1.222222222225333
Katherine Minjee Chung, Jaffarguriqbal Singh, Lauren Lawres, Kimberly Judith Dorans, Cathy Garcia, Daniel B Burkhardt, Rebecca Robbins, Arjun Bhutkar, Rebecca Cardone, Xiaojian Zhao, et al. cedoorrine-exccrinn signaling drives associa-s panereatic ductal adenocarcinoma. Cell, (31::)8332-47 2020.
Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5-30, 2006.
Andrés F Duque, Sacha Morin, Guy Wolf, and Kevin Moon. Extendable and invertible manifold learning with geometry regularized autoencoders. In 2020 IEEE International Conference Big Data (Big Data), pp. 5027-5036. IEEE, 2020.
Siyuan Gao, Gal Mishne, and Dustin Scheinost. Non-linear manifold learning in fmri uncovers low-dimensional space of brain dynamics. bioRxiv, 2020.
Scott Gigante, Adam S Charles, Smita Krishnaswamy, and Gal Mishne. Visualizing the phate of neural networks. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2022
Michael Hanke, Florian J. Baumgartner, Pierre lbe, Falko R. Kaule, Stefan Pollmann, Oliver Speck, Wolf Zinke, and Jörg Stadler. A high-resolution 7-tesla fmni dataset from complex natural stimulation with an audio movie. Scientific Data, 1(1):140003, 2014.
James V Haxby, Swaroop Guntupalli, Andrew Connolly, Yaroslav Halchenko, Bryan R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):4044416, 2011. 2011.
CO Husler and M Hanke. An annotation of cuts, depicted locations, and temporal progression in the motion picture "forrest gump" [version I; peer review: 3 approved]. F1000Research, 5(2273), 2016.
Kui Jia, Lin Sun, Shenghua Gao, Zhan Song, and Bertram E. Shi. Laplacian auto-encoders: An explicit learning of nonlinear data manifold. Neurocomputingg 160:250-260) 2015.
Manik Kuchroo, Jessie Huang, Patrick Wong, Jean-Christophe Grenier, Dennis Shung, Alexander Tong, Carolina Lucas, Jon Klein, Daniel Burkhardt, Scott Gigante, et al. Multiscale phate exploration of sars-cov-2 data reveals multimodal signatures of disease. bioRxiv, 2020.
Leland McInnes. John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018.
Gal Mishne, Uri Shaham, Alexander Cloninger, and Israel Cohen. Diffusion nets. Applied and Computational Harmonic Analysis, 47(2):259-285, 2019.
Kevin R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B. Burkhardt, William S. Chen, Kristina Yim, Antonia van den Elzen, Matthew J. Hir, Ronald R. Coifman, Natalia B. Ivanova, Guy Wolf, and Smita Krishnaswamy, Visualizing structure and transitions in high-dimensional biological data. Nature Biotechnology,
2019.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt, Topological autoencoders. In International conference on machine learning, pp. 7045-7054, PMLR, 2020.
Jeanette Mumford and Russell A Poldrack. Modeling group fmrr data. Social cognitive and affective neuroscience, 2(3):251-255 2007.
Samuel A Nastase, Valeria Gazzola, Uri Hasson, and Christian Keysers. Measuring shared responses across subjects using intersubject correlation. Social Cognitive and Affective Neuroscience, 14(6). 2019.
Jenna I. Pappalardo, Le Zhang, Maggie K Pecsok, Kelly Perlman, Chrysoula Zografou, Khadir Rad- dassi, Ahmad Abulaban, Smita Krishnaswamy, Jack Antel, David van Dijk, et al. Transcriptomic and clonal characterization of cells in the human central nervous system. Science Immunology, (5 (5a)bbbb)8786 2020.
Bastian Rieck, Tristan Yates, Christian Bock, Karsten Borgwardt, Guy Wolf, Nicholas Turk-Browne, and Smita Krishnaswamy. Uncovering the topology of time-varying fmri data using cubical persistence. Advances in Neural Information Processing Systems, 33, 2020.
Ayan Sengupta, Falko R. Kaule, Swaroop Guntupalli, Michael B. Hoffmann, Christian Häusler, Jörg Stadler, and Michael Hanke. A studyforrest extension, retinotopic mapping and localization of higher visual areas. Scientific Data, 3(1), 2016.
Stephen M Smith, Aapo Hyvärinen, Gaël Varoquaux, Karla L Miller, and Christian F Beckmann. Group-pet for very large finri datasets. Neuroimage, 101:738-749, 2014.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Kiran Vodrahalli, Po- Hs-an Chen, Yingyu Liang, Christopher Baldassano, Janice Chen, Esther Yong, Christopher Honey, Uri Hasson, Peter Ramadge, Kenneth A. Norman, and Sanjeev Arora. Mapping between fmri responses movies and their natural language annotations. Neurolmage, 180, 2018.
11

---

Under review as a conference paper at ICLR 2022
Guy Wolf, Aviv Rotbart, Gil David, and Amir Averbuch. Coarse-grained localized diffusion. Applied and Computational Harmonic Analysis, 33(3):388 4000 2012.
Wenchao Yu, Guangxiang Zeng, Ping Luo, Fuzhen Zhuang, Qing He, and Zhongzhi Shi. Embed- ding with autoencoder regularization. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 208-223. Springer, 2013.
Yujiao Zhao, Matthew Amodio, Brent Vander Wyk, Bram Gerritsen, Mahesh M Kumar, David van Dijk, Kevin Moon, Xiaomei Wang, Anna Malawista, Monique M Richards, et al. Single cell immune profiling of dengue virus patients reveals intact immune responses to zika virus with enrichment of innate immune signatures. PLoS neglected tropical diseases, 14(3)=00001 12, 2020.
12

---

Under review as a conference paper at ICLR 2022
A SUPPLEMENTARY MATERIALS
IMRI pre-processing and region identification After all standard fMRI preprocessing steps, functional images from both the Sherlock and Forrest datasets were aligned to the common MNI template brain based on anatomical landmarks. In this anatomical template, we defined several regions of interest (ROIs) to target in analyses: the early visual cortex, late visual cortex, early auditory cortex, and the posterior medial cortex (PMC). We selected these regions as they robustly respond to audiovisual stimuli (early visual, late visual, and early auditory) and PMC in the original sherlock analyses for its memory involvement (Chen et al. (2017). We extracted timeseries data from the voxels in each of these regions (early visual: 307 voxels, late visual: 1008 voxels, early auditory: 1018 voxels, PMC: 481 voxels) and collapsed the spatial structure within this ROI into a [timepoints, voxels] matrix of activity for each voxel across time. Within this matrix, we z-scored each voxel across its timeseries to account for differences across voxels in mean activation.
targeted
Dimension reduction methods on out-of-sample data Dimensionality reduction methods do not generalize to out-of-sample data with decremented performance when extending trained models to new data points. In Figure [ we show that the stimuli classification accuracy decreases when PCA and PHATE (via landmark approach) are applied to out-of-sample data. Experiment was conducted on Sherlock early-visual ROI data.
Sherlock Indoor/Outdoor Accuracy, Early Visual ROI
Classification accuracy
0.8
0.6
0.4
0.2
0.0
PHATE
PHATE
Appendix Figure 1: Classification accuracy comparison. PCA and PHATE both fail to generalize to out-of-sample data, shown as decremented accuracy. Landmark approach was used to extend learned PHATE embedding new data.
Stimuli classification experimental details For 5-fold cross validation we proceed following the standard procedures. We split the dataset into groups at random, and for each group take as a hold out or test dataset and use the remaining groups as a training data set. Fit a classifier on the training set and evaluate folds. For each training set, we balance the samples to reach an even split between classes during training, the classification tasks are entirely balanced. We repeat the procedure for each subject. The mean accuracy from the 5-fold cross validation within each subject is reported as the skill score of the subject. Then we report the mean and standard deviation across subjects. For SVM classifiers, we used the C-support vector classification from Scikit-learn package, with an RBF kernel, regularization=10, and gamma set be 'scale" ,1 ///n//ffattt (nfeatures X X.var())
the test set, retain the test accuracy and summarize after all
Stimuli classification comparison across subjects Since we perform classification within each subject, we further look into how our model compare to the other methods for each subject. We present the results on the Sherlock early visual ROI and show that higher classification accuracy was achieved with MRMD-AE manifold embedding in 13 out of 16 subjects comparing to the next best method that is vanilla AE. Moreover, MRMD-AE outperformed individual MR-AE (third-best) in all subjects. Although there are not enough samples to quantify subject-level statistics, we show MRME-AE achieves better performance in almost all subjects(
Intrinsic dimensions of [MRI data We analyzed the intrinsic dimensions of fMRI data by exam- ining the spectral entropy of PHATE diffusion matrix of fMRT data. We refer the readers to Moon
13

---

Under review as a conference paper at ICLR 2022
0.25
Inbee/Deicen Clas
6.75
0.700
0.2-
0.015
06000
S1bject00
Appendix Figure 2: Classification accuracy comparison across subjects. Classification by indoorout- door label on Sherlock early visual ROI using the top performing models in Table 1 Left: The mean accuracy from the 5-fold cross-validation within each subject was reported. Error bar was standard deviation from the 5-fold cross-validation. MRMD-AE outperforms the next best method, a vanilla AE, in 13/16 subjects and outperforms the third best method, individual MR-AE, in all subjects. Right: Box plot of the same set of accuracy values show better performance of MRMD-AE compared to the other methods.
ct al. 2019) for how PHATE computes the diffusion operator and the relationship between diffusion operator the data geometry. For the purpose of current analysis, one consider the diffusion operator to represent the ambient data geometry and the spectral entropy analysis via Von Neumann Entropy similar the analysis of principle components that account for most variation in PCA. We report the intrinsic data dimensions of various ROIs of the Sherlock and Forrest IMRI datasets that we use in the experiments of this paper in Appendix Table I
Dataset
Sherlock
Forrest
ROI
early-visual
pmc
early-auditory
litu-visual
early-visual
Intrinsic Dimension
30
29
41
13
15
Appendix Table fMRI ROI data intrinsic dimensions
14


Reviews:
[{"review_id": "dErqMVS_h0f", "reviewer": "Reviewer_h5JS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\nThe problem tackled by the paper is interesting, and the proposed model is original. The model is sound, conceptually interesting, and reasonably described. It is based on recent techniques that are appropriately described and cited. The paper is clear and well structured.\n\n## Weaknesses\n1. It is not super clear why the proposed architecture uses a shared encoder and a separate decoder per subject. This choice seems central to the proposed method and should be better explained.\n2. It is also not clear which decoder was used to predict on held-out subjects. It seems like a decoder must be trained at some point on the held-out subjects, which might undermine the integrity of the cross-subject experiment.\n3. It is not clear how hyper-parameters lambda and mu were set, with a reasonable risk of overfitting the proposed benchmarks. The authors should explain how the hyper-parameters were chosen.\n4. \"MRMD-AE achieves lower or matching MSE than the landmark approach\" does not seem entirely supported by the data. The landmark approach is actually below MRMD-AE at 70% of train set, and the difference does not seem relatively smaller than the difference at other percentages.\n5. Tables with numbers are very non-intuitive, and might even be considered a ploy to hide small differences in performances between methods. The authors should use graphs, and move the Tables to the appendix if still considered necessary. If still using tables, the authors should remove non-significant digits (6 digits are not necessary when the standard deviation is at 5e-2.\n6. Figure 4: when using barplots, the y-axis should start at 0. Indeed, barplots use our intuitive perception of areas to convey the difference of values, but this perception is biased if the bar does not start at zero, and the difference of values is artificially increased.\n7. \"We will provide a link to an anonymous repository to the reviewers and area chairs\", yet the link was not provided.\n8. \"fMRI data is now being collected from thousands of subjects\" should be tempered by the fact the current study is limited to 16 subjects. \n\n### Unclear sentences\n- The \"Manifold Regularization Penalty\" section is quite hard to follow. The authors might improve it by using shorter sentences.\n- What is a \"soft\" regularization ?\n- What is a \"batch effect\" ? and a \"strong subject batch effect\" ? \n- What is a \"individual-subject manifolds subject\" ?\n\n### Typos\n- [Moon 2019a] and [Moon 2019b] are identical.\n- \"demonstraging\"\n- \"that that\"\n- \"Reduduction\"\n- \"We therefore compare\" (no dot)\n- \"TRs\" is not defined\n- Fontsize in Figure 2 is too small.\n- \"An. SVM\"", "summary_of_the_paper": "The paper proposes a model to learn a low-dimensional representation of fMRI data over multiple subjects of the same experiment. The model is built as an auto-encoder, with an encoder shared across subjects, and a separate decoder per subject. The model is regularized so that the first layer of each decoder gives a representation close to a pre-computed manifold embedding. An optional regularization also constrains the shared representation to be similar across subjects. \nThe paper then proposes a series of experiments to demonstrate the benefits of the learn representation. The experiments consider the tasks of (a) projecting new test samples to the learned manifold, (b) classify some stimulus features from the embedding (decoding task), and (c) predict brain recordings of a new subject.", "summary_of_the_review": "My recommendation (reject) is based on\n- a missing justification of the architecture design\n- a number of missing important details about the fitting procedure\n- a poor quality in the reporting of results", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Sf-ecvcepqK", "reviewer": "Reviewer_Rq5W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes an interesting and well-motivated contribution on a field typically overlooked in ICLR. Although some aspects of this paper exist in previous literature, to the best of my knowledge the ideas are novel and based on well-thought motivations stemming from specific challenges in the field of fMRI data. The empirical results - although in my opinion not sufficiently thorough - give a good indication that this new approach can be successfully applied on a multitude of tasks, improving previously used techniques. Given these reasons, I recommend acceptance of this work. However, my recommendation is only marginally above the acceptance threshold because the paper contains crucial flaws, mainly (1) a lack of sufficient ablation analysis and (2) several portions of the paper requiring further clarification. In this review I’ll first list what I think are the major weaknesses of this work, leaving some minor remarks and suggestions after.\n\n# Major weaknesses\n1. The paper mentions a “translation” task several times throughout the paper, but it never explains what it specifically consists of. Given this seems to be the task where the proposed architecture is the least strong (from Table 3), this is an important part to clarify.\n2. In the third paragraph of the Introduction, the paper mentions that typical autoencoders “often just spread out points for ease of decoding”. Many of the variational autoencoders developed from vanilla autoencoders actually try to solve this issue with AEs, by making the latent spaces more meaningful and smooth. Given how a direct development from AEs seems to directly tackle many of the issues which motivated the paper, I do not understand why the paper didn’t use VAEs as another baseline for comparison, and I find this to be an important weakness in the paper.\n3. The paper mentions in the “Related Work” section that it used a similar multi-objective NN approach like in SAUCIE; however, it is not clear what are the differences to this previous method which would grant novelty.\n4. Section 4.1 introduces a concept of common stimuli X for all subjects. Furthermore, this concept is used to show the properties that functions “f” and “g” should have, specifically in (b), (c ), and (d). However, this concept (ie., X) is not explored anywhere else in the paper, nor these properties are validated in the experiments. I find this to be a major point of confusion for the reader. \n5. Section 4.1 mentions “intrinsic dimensions of the data” but I think this concept is not explored nor explained (including in the appendix table referenced).\n6. A big point of the paper is how PHATE seems to correct for strong subject batch effects existent in fMRI, and how this is important in creating a subject-independent manifold. Although I think Figure 3 supported the claim of a subject-independent manifold, I don’t see how batch effects were accounted for at all; usually they relate to confounds like sex, ethnics, BMI, and others. In any case, why would batch effects be a problem in this case, as it is well-known how they can influence fMRI timeseries and therefore could be important for a rich low-dimensional manifold?\n7. A strong weakness of this paper is the lack of a more detailed ablation analysis on the several components introduced in the architecture. For example, the paper splits each decoder into two (Subsection “Manifold Regularization Penalty”) and tries to motivate this split into g’ and g’’ using very loose intuitions which I’m not even sure I agree with; an ablation analysis would further complement the paper to see how much influence this has in practice.\n8. Many details are missing in the paper to properly understand how the classification tasks are conducted. How does the paper decide where to “cut” the timeseries in order to conduct the SVM classifier and how is the 5-fold CV conducted/divided?  What hyperparameters were used for the SVM classifier? Given the datasets seem to have a label for each input, what is the reasoning for using a classification over many timepoints?\n9. The paper claims “We show that our model achieves best translation performance on various brain ROIs in at least one metric”. I do not understand where this is shown, given no figure or table is referenced in this paragraph, nor the concept of ROIs seem to be explored here.\n\n# Minor comments and suggestions\n1. I want to show my appreciation for the reproducibility statement, but as a suggestion, an anonymised code would have made this submission stronger and potentially clarify some of the questions I am asking in this review. \n2. Some acronyms appear in the text without being defined or specified with a reference beforehand, for instance BCI, BOLD, PCA and SVD. Although it’s reasonably fair to assume the majority of the ICLR community will know some of these terms, defining them it’s always better for preciseness, clarity, and avoidance of possible confusions for people in other fields reading this paper. I think for a computer science conference like this, the definition of BOLD signal should go beyond just spelling the acronym, as probably many people in the conference do not know what this means (e.g. in the Preliminaries section).\n3. In the fields of neuroscience, a standard dataset for any fMRI study is the Human Connectome Project (HCP) or, more recently, the big UK Biobank. Were these more widely-used datasets not used for some specific reasons? I’m aware that the UK Biobank requires a long application process, but all HCP data are publicly available. \n4. As a suggestion for future work, I’d say it would be interesting to use resting-state fMRI which is typically quite noisier than the task-specific fMRI used in this work. I believe it would be interesting to see whether the experiments on the manifold extension task would still work here.\n5. How imbalanced are the classification tasks? This is important to understand whether accuracy is a good-enough metric for this task.\n6. The paper claims that tSNE and UMAP “typically do not preserve overarching trends or relations between data regions”. For a complete related work section, the paper should probably mention that this is a contested statement and not so widely accepted as the paper claims. For example, some use UMAP for classification tasks (https://umap-learn.readthedocs.io/en/latest/supervised.html), and recent works explored preservation of global structure on real-world data, for instance: (1) https://www.nature.com/articles/nbt.4314, (2) https://www.nature.com/articles/s41540-021-00186-6, (3) https://www.nature.com/articles/s41467-020-15351-4.\n7. Footnote (1) is confusing. How can one time point in one region of interest be a vector of length equal to the number of voxels? When the functional images are aligned to a common template, it is common to extract a single averaged timeseries per region of interest, otherwise why is this alignment done, just as a preprocessing step? Indeed, from the definition it seems that each subject only has in fact one single timeseries, as Y only varies across j (subjects) and k (timepoints) as subscripts. This needs to be further clarified.\n\n# Minor comments to improve clarity and readability\n1. The paper uses a metric named MDS without explaining what it actually means. I don’t believe this is a metric well-known by the community.\n2. “ROI” is not defined in the paper even though it is used in a few places.\n3. In Section 4.1, point (c ) starts with “They”; please define what this actually corresponds to.\n4. In “Embedding Alignment Penalty” (in Section 4.2) there’s a typo in “common decoder (and individual encoders)” (should be the other way around)\n5. “TRs” is not defined in Section 5.\n6. How did the paper arrive at this set of hyperparameters? Manual choice? Hyperparameter sweep?\n7. Typo in “bottle neck” (needs to remove the space) (Section 5, “Autoencoder Hyperparameters”)\n8. Figure 2: yaxis should be only MSE (without the “vs” part)\n9. Figure 2: I find the MSE values extremely small, and the confidence intervals overlapping a lot. Therefore, I do not agree with the paper’s claims that MRMD-AE achieves significant results here. I believe the paper needs to tone down their conclusions in this subsection or explain how this small value (i.e., below 0.001) can be so significant.\n10. Typo: “An. SVM” (begin of page 7)\n11. Typo: “unseen times series. fMRI” (Section 5, “Stimuli Classification”).\n12. The paper uses a metric called “Earth Mover Distance” to support some of their claims (eg in Figure 3). However, this metric is not explained.\n13. In one paragraph in Section 5 it is written “SVM classifier” and 2 paragraphs later it is instead “A support vector classifier (SVC)”. Are these the same or did something change?\n14. Tables: I believe it would improve readability to reduce the number of decimal points. 2 decimal points in accuracy and 3 in standard deviation would probably suffice. \n15. Tables: What is the difference between “AE” and “Individual MR-AE”? If it is the usage of PHATE on MR-AE but not on an equivalent autoencoder with shared encoders and decoders, then please say in the text (and indeed this is a good ablation).\n16. Table 3 is not referenced in the text.\n17. Figure 3: Are the PHATE embeddings already in 2D space, or did the paper have to apply some technique like UMAP or tSNE for visualisation?\n", "summary_of_the_paper": "The paper proposes a new neural architecture named MRMD-AE that can be applied on noisy fMRI data in different tasks. Subject-specific decoders are used to more directly recover individual signals, while the encoder is shared across every subject under the assumption that every person will share common low-dimensional features for the same stimuli. A key component of this architecture is the usage of a regularisation term named PHATE (previously introduced in the literature) which allows the latent space to not be split into individual embeddings and be extendable for unseen data. The paper empirically shows how this architecture improves metrics on classification tasks when compared to previously used techniques. ", "summary_of_the_review": "Overall I think this is a very interesting and useful idea in the convergence of the fields of neuroimaging and learning representation. However, the lack of important experiments and explanations make me recommend only marginally above the acceptance threshold. I believe my questions can be mostly (if not all) addressed during a rebuttal process, after which I’d be happy to change my score as I think overall this should be shared among the ICLR community.\n\n------------------ EDITED AFTER REBUTTAL PERIOD:\nI followed the authors answers to my questions and other reviewer's questions. Although the authors didn't address all my questions (for example my comment titled \"Further discussion\") nor they uploaded a new pdf version to check corrections, I believe my main concerns were clarified. Assuming the authors introduce the written clarifications in the paper, I'm changing my review recommendation in two points:\n- Changing \"Correctness\" from 2 to 3.\n- Changing my \"Recommendation\" from 6 to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QxQdHlzdPHl", "reviewer": "Reviewer_nPEU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Clear and easy to understand introduction and motivation\nThe opening sections of the paper were clear and straightforward in motivating the functional alignment problem, manifold-based dimensionality reduction, and generating an approximate manifold embedding operator using a neural net. Likewise, the description of the problem setup, architecture choices and losses builds on itself in a way that was clear and easy to understand. I have only some minor suggestions for improvement here: \n\n[1a] As far as I can tell, the stimulus set $X$ and its elements $X_k$ is only used to say that the encoder maps the fMRI activation map to some representation of the stimulus, but $f'()$ is never defined explicitly. If so, it seems like unnecessary \"mathiness\" -- the notation is not buying additional precision or insight outside of saying this same thing in words. \n\n[1b] Expression 1 could color-code or otherwise annotate the distinct loss terms (reconstruction, latent closeness penalty, manifold regularization). \n\n# Weak evaluation\nHowever, the empirical evaluation is less well-motivated and weaker overall. A strong empirical evaluation would show [2a] unambiguously state-of-the-art performance relative to [2b] strong baselines, and [2c] evaluations or [2d] ablations that target the core contributions. I detail each in turn: \n\n[2a] Performance benefits seem modest at best. Some very rough statistics: with 15-16 subjects, the confidence interval should be roughly equal to half the width of the SD on each side (since the z- or t-multiplier for the conventional $\\alpha=0.05$ is roughly 2 and $\\sqrt{15}$ is roughly 4). This means that (very roughly) differences between the best and second-best performing models smaller than the SD may not be statistically reliable. MRMD-AE is rarely convincingly above the next-best model by even a single SD. These statistics are, of course, rough: variability may be much smaller across folds than across subjects, but proper analysis (likely some sort of repeated-measures model) may be needed to tease apart this variability and establish the strength of MRMD-AE more convincingly. \n\n[2b] Strong baselines are missing. First, I'm not sure I understand why SRM is missing from the classification baselines -- it should be applicable here, right? Second, all the baselines are either linear (PCA, SRM) or single-subject (PHATE, AE), and there is no nonlinear functional alignment baseline, even though they exist in the literature (for example: Deep Generalized CCA, Benton et al. ICLR 2017; Deep Hyperalignment, Yousefnezhad & Zhang NeurIPS 2017; Kernel Hyperalignment, NeurIPS 2012 -- and this is only a brief survey). More advanced linear methods likewise exist  (e.g. RSRM, Turek et al. ICASSP 2018; MN-SRM, and DP-SRM, Shvartsman et al. AISTATS 2018). Comparing to all of these methods is not critical but it's hard to get a sense of whether the new method truly exceeds the state of the art without at least a few stronger functional alignment baselines. \n\n[2c] As I see it, the key contributions here are the one-in-many-out architecture, and the additional manifold and latent space closeness regularization terms. A full set of evaluations would probe each in turn, showing how the present contribution performs best in out-of-sample performance of some reconstruction metric, some latent space closeness metric, and some manifold alignment metric. The paper reports untrained cross-subject translation (which is a sort of out-of-sample reconstruction metric), and the future-point-placement metric (which is sort of a latent space closeness metric). The benefit of the manifold regularization penalty is seen in the comparison to MR-AE. Importantly, the overarching narrative present in the front matter of the paper is much weaker here, and the link made above between the contributions and the evaluations is not made explicit. \n\n[2d] The ablation study is not particularly informative about the contributions of components outside of the translation penalty (since there is no model with translation but no alignment penalty, and no model with translation but no manifold regularization). To be clear: I don't think ablations studies are critical to this paper -- but the specific ablation study in the submission could be stronger. \n\n# Minor comments and typos\n- In the discussion, \"For many studies, fMRI data is now being collected from thousands of subjects\" is optimistic. I only know of the HCP and IMAGEN projects, both of which are massive multi-institution efforts. Most fMRI studies are still quite small (tens of subjects at most). \n- It would be nice (but not critical) to see some empirical demonstration of the benefit of splitting g() into g'() and g''(), and verification or citations of the claim that applying PHATE to the group data fails. \n- The motivation for the time-splitting experiment is a bit odd: in my experience pilot fMRI collections are still full length collections from different subjects, not short collections from the same subjects. \n- Figure 2 should identify the dataset used (I think it is Sherlock?). \n- I wonder if part of the reason that MRMD-AE fails to perform well on the localizer task is that it overfits to movie data, i.e. it is capturing movie-specific variability that is not present in the localizer data. \n- Section 3 first paragraph \"train data\" -> training data\n- Page 5 under \"manifold regularization penalty\" paragraph -- \"individual-subject manifolds subject\", second \"subject\" seems extraneous. \n- Last line of section 4: subjects fmri activities -> subjects' fmri activities. \n- First sentence on p7 Stimuli Classification section: \"unseen times series. fMRI data\" -> \"unseen fMRI data\" or \"unseen time series\" or \"unseen time series of fMRI data\" etc. \n\nIn sum, I think issues [2a] and [2b] are most critical and keep the paper from clearing the bar for ICLR. Addressing [2c] and to a lesser extent [2d] would also strengthen the paper, but would not be sufficient if [2a] and [2b] were not addressed. \n\nEDITED: I am adjusting my rating to a 6 based on additional information provided about the raw classification scores, but I'm still concerned about the baselines chosen. ", "summary_of_the_paper": "The submission proposes a new model for functional alignment of fMRI datasets from multiple subjects. It combines a one-in-many-out autoencoder with two regularization loss terms (one inspired by GRAE) to develop a model that can encode every subject's data to a shared latent space, from which each subject's data is then decoded by a separate decoder. Experiments are provided that demonstrate benefits to some downstream tasks. \n", "summary_of_the_review": "I think this is a well-motivated approach and the early parts of the paper are a pleasure to read: it was easy to understand what each piece of the contribution is meant to do, and its motivation and relation to past work. When it comes to the empirical evaluation, however, the paper runs into a common problem with noisy fMRI data, which is that it's hard to convincingly beat linear methods, and it then overstates (to my read) the strength of its empirical contributions. The performance gains are modest and nonlinear functional alignment methods (e.g. from the hyperalignment family such as KHA/DHA) and more advanced linear methods (such as RSRM and MN-SRM) are not included in the comparison. The structure of the latter portions of the paper suffers as well: while the specific modeling choices are well-motivated, the experiments do not fully probe the novel contributions. As a result, while I think this line overall is worth pursuing, I would like to see stronger empirical results for this work to be a successful ICLR submission. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2xYf144c4zD", "reviewer": "Reviewer_gwm6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1.\tApt controls and baselines are employed throughout. Ablations convey the utility of each component loss function employed to train the MRDM-AE model.\n2.\tAll components of MRD-AE modeling framework seem well motivated. \n3.\tThorough comparisons are employed with multiple datasets  \n4.\tThe paper is very clearly written and all analyses seem well-executed. \n\nComments/Critiques: \n1. Some results are not particularly exciting, for instance using the proposed MRDM-AE  framework does not result in a major improvement over a standard manifold regularized AE atleast in terms of stimulus decodability. \n2. It would have been useful to compare cross-subject alignment capabilities of the MRDM-AE framework against existing popular techniques for aligning cross-subject fMRI data like hyperalignment. \n3. Have the authors tried to assess extendibility/generalization to new subjects? For instance, once the MRDM-AE is trained, the authors could get the embeddings from a new subject and study stimulus decoding performance in this new scenario? In general, how could the proposed autoencoder framework be extended to a new subject? Do the authors envision this would involve simply training a new subject-specific decoder? \n4. How much does adding the translation penalty improve cross-subject translation in terms of correlation? \n5. It would be nice to also have some statistical significance analysis to assess whether the improvement of MRMD-AE over the dominant shared response modeling framework (SRM) is significant or not. \n6. It might also be useful to critically assess whether subject-specific decoders indeed capture individual variations or are they largely just capturing a common signal. For instance, are the reconstructed signals from each decoder truly subject-specific, in the sense of correlating best with the same subject’s signal than any other?  \n", "summary_of_the_paper": "This paper proposes a neural network-based modeling strategy to learn a common latent space from multi-subject fMRI data. In addition to capturing a useful common latent space, the proposed technique is further able to disentangle common representational patterns from subject-specific variations through the use of subject-specific decoders. The authors impose meaningful and desirable priors on the latent embedding, like geometric regularization and cross-subject embedding alignment. Unlike other manifold learning techniques, the proposed deep neural network modeling framework lends itself well to extendibility to new data (stimuli) since the PHATE embeddings are only required at training time in the geometric regularization loss. The proposed framework is tested on two large fMRI datasets and an improved stimulus decoding (from the shared space) and cross-subject translation accuracy is achieved over competitive baselines. ", "summary_of_the_review": "This paper presents a valuable methodological contribution to multi-subject fMRI data analyses. The merits of the proposed technique are well supported by the experiments and results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1zYH3BLhKfZ", "reviewer": "Reviewer_cq7U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n-   An interesting idea of constraining the embedding space with an existing non-generalizable manifold learning solution.\n-   A mostly-clear exposition and a well-written paper.\n\n\n## Weaknesses\n\n-   The paper mostly reuses existing ideas and thus has a limited technical novelty. Multi-loss optimization.\n-   The experiments are performed on a single ROI time courses, which diminishes the value of the work. The multivariate nature of fMRI is not taken into account and the experiments have mostly toy-example nature.\n-   Focus on sensory ROIs excludes chances of capturing semantic information encoded by the brain in he fMRI signal and thus devalues presented work for the neuroimaging community.\n\n\n## Questions to authors\n\n1.  Since the decoders are subject specific and training is happening on a few subjects simultaneously, it is unclear how a new subject is handled. Which $g'_i$ decoder is used in this case?\n2.  The input sample in the experiments is not specified making impossible to evaluate how close to fMRI signal is the input. Is this a window of the univariate ROI time-course per stimulus marker?\n\nInformation about what is the input data is only available in the supplement and thus difficult to obtain, which caused confusion in my case.\n", "summary_of_the_paper": "The papers presents a multi-decoder autoencoder model with an objective function encouraging representations of stimuli-related fMRI activity similarly across subjects and events. The main goal is to obtain an encoder that learns a consistent and reusable mapping from fMRI space to representation space.\n", "summary_of_the_review": "Two main concerns: limited technical novelty and limited value for neuroimaging. My recommendation is based on these limitations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the author present a method for learning a shared latent space between the fMRI activity of multiple individuals processing the same stimulus. The method consists of an auto-encoder with a single encoder and subject-specific decoders which is specifically regularized to decouple common and shared representations. This paper generated a lot of discussion between the reviewers and the authors, as well as between the reviewers. In light of these discussion, I cannot recommend acceptance at this point, as the paper is not ready. The main concerns were (1) about how the results and improvement are evaluated statistically, (2) that the baselines chosen were not strong enough and did not include existing approaches (neural or non-neural) and relatedly (3) that the paper was not framed correctly within the existing literature on finding shared spaces between participants, which would help with determining and understanding the novelty of the proposed approach. Some other smaller points were made by the reviewer can also strengthen the paper for a future submission in a neuroscience or machine learning venue.
Decision: Reject