Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The papers presents a multi-decoder autoencoder model with an objective function encouraging representations of stimuli-related fMRI activity similarly across subjects and events. The main goal is to obtain an encoder that learns a consistent and reusable mapping from fMRI space to representation space.

main_review: ## Strengths

-   An interesting idea of constraining the embedding space with an existing non-generalizable manifold learning solution.
-   A mostly-clear exposition and a well-written paper.


## Weaknesses

-   The paper mostly reuses existing ideas and thus has a limited technical novelty. Multi-loss optimization.
-   The experiments are performed on a single ROI time courses, which diminishes the value of the work. The multivariate nature of fMRI is not taken into account and the experiments have mostly toy-example nature.
-   Focus on sensory ROIs excludes chances of capturing semantic information encoded by the brain in he fMRI signal and thus devalues presented work for the neuroimaging community.


## Questions to authors

1.  Since the decoders are subject specific and training is happening on a few subjects simultaneously, it is unclear how a new subject is handled. Which $g'_i$ decoder is used in this case?
2.  The input sample in the experiments is not specified making impossible to evaluate how close to fMRI signal is the input. Is this a window of the univariate ROI time-course per stimulus marker?

Information about what is the input data is only available in the supplement and thus difficult to obtain, which caused confusion in my case.

summary_of_the_review: Two main concerns: limited technical novelty and limited value for neuroimaging. My recommendation is based on these limitations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a neural network-based modeling strategy to learn a common latent space from multi-subject fMRI data. In addition to capturing a useful common latent space, the proposed technique is further able to disentangle common representational patterns from subject-specific variations through the use of subject-specific decoders. The authors impose meaningful and desirable priors on the latent embedding, like geometric regularization and cross-subject embedding alignment. Unlike other manifold learning techniques, the proposed deep neural network modeling framework lends itself well to extendibility to new data (stimuli) since the PHATE embeddings are only required at training time in the geometric regularization loss. The proposed framework is tested on two large fMRI datasets and an improved stimulus decoding (from the shared space) and cross-subject translation accuracy is achieved over competitive baselines. 
main_review: Strengths: 
1.	Apt controls and baselines are employed throughout. Ablations convey the utility of each component loss function employed to train the MRDM-AE model.
2.	All components of MRD-AE modeling framework seem well motivated. 
3.	Thorough comparisons are employed with multiple datasets  
4.	The paper is very clearly written and all analyses seem well-executed. 

Comments/Critiques: 
1. Some results are not particularly exciting, for instance using the proposed MRDM-AE  framework does not result in a major improvement over a standard manifold regularized AE atleast in terms of stimulus decodability. 
2. It would have been useful to compare cross-subject alignment capabilities of the MRDM-AE framework against existing popular techniques for aligning cross-subject fMRI data like hyperalignment. 
3. Have the authors tried to assess extendibility/generalization to new subjects? For instance, once the MRDM-AE is trained, the authors could get the embeddings from a new subject and study stimulus decoding performance in this new scenario? In general, how could the proposed autoencoder framework be extended to a new subject? Do the authors envision this would involve simply training a new subject-specific decoder? 
4. How much does adding the translation penalty improve cross-subject translation in terms of correlation? 
5. It would be nice to also have some statistical significance analysis to assess whether the improvement of MRMD-AE over the dominant shared response modeling framework (SRM) is significant or not. 
6. It might also be useful to critically assess whether subject-specific decoders indeed capture individual variations or are they largely just capturing a common signal. For instance, are the reconstructed signals from each decoder truly subject-specific, in the sense of correlating best with the same subject’s signal than any other?  

summary_of_the_review: This paper presents a valuable methodological contribution to multi-subject fMRI data analyses. The merits of the proposed technique are well supported by the experiments and results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new neural architecture named MRMD-AE that can be applied on noisy fMRI data in different tasks. Subject-specific decoders are used to more directly recover individual signals, while the encoder is shared across every subject under the assumption that every person will share common low-dimensional features for the same stimuli. A key component of this architecture is the usage of a regularisation term named PHATE (previously introduced in the literature) which allows the latent space to not be split into individual embeddings and be extendable for unseen data. The paper empirically shows how this architecture improves metrics on classification tasks when compared to previously used techniques. 
main_review: The paper makes an interesting and well-motivated contribution on a field typically overlooked in ICLR. Although some aspects of this paper exist in previous literature, to the best of my knowledge the ideas are novel and based on well-thought motivations stemming from specific challenges in the field of fMRI data. The empirical results - although in my opinion not sufficiently thorough - give a good indication that this new approach can be successfully applied on a multitude of tasks, improving previously used techniques. Given these reasons, I recommend acceptance of this work. However, my recommendation is only marginally above the acceptance threshold because the paper contains crucial flaws, mainly (1) a lack of sufficient ablation analysis and (2) several portions of the paper requiring further clarification. In this review I’ll first list what I think are the major weaknesses of this work, leaving some minor remarks and suggestions after.

# Major weaknesses
1. The paper mentions a “translation” task several times throughout the paper, but it never explains what it specifically consists of. Given this seems to be the task where the proposed architecture is the least strong (from Table 3), this is an important part to clarify.
2. In the third paragraph of the Introduction, the paper mentions that typical autoencoders “often just spread out points for ease of decoding”. Many of the variational autoencoders developed from vanilla autoencoders actually try to solve this issue with AEs, by making the latent spaces more meaningful and smooth. Given how a direct development from AEs seems to directly tackle many of the issues which motivated the paper, I do not understand why the paper didn’t use VAEs as another baseline for comparison, and I find this to be an important weakness in the paper.
3. The paper mentions in the “Related Work” section that it used a similar multi-objective NN approach like in SAUCIE; however, it is not clear what are the differences to this previous method which would grant novelty.
4. Section 4.1 introduces a concept of common stimuli X for all subjects. Furthermore, this concept is used to show the properties that functions “f” and “g” should have, specifically in (b), (c ), and (d). However, this concept (ie., X) is not explored anywhere else in the paper, nor these properties are validated in the experiments. I find this to be a major point of confusion for the reader. 
5. Section 4.1 mentions “intrinsic dimensions of the data” but I think this concept is not explored nor explained (including in the appendix table referenced).
6. A big point of the paper is how PHATE seems to correct for strong subject batch effects existent in fMRI, and how this is important in creating a subject-independent manifold. Although I think Figure 3 supported the claim of a subject-independent manifold, I don’t see how batch effects were accounted for at all; usually they relate to confounds like sex, ethnics, BMI, and others. In any case, why would batch effects be a problem in this case, as it is well-known how they can influence fMRI timeseries and therefore could be important for a rich low-dimensional manifold?
7. A strong weakness of this paper is the lack of a more detailed ablation analysis on the several components introduced in the architecture. For example, the paper splits each decoder into two (Subsection “Manifold Regularization Penalty”) and tries to motivate this split into g’ and g’’ using very loose intuitions which I’m not even sure I agree with; an ablation analysis would further complement the paper to see how much influence this has in practice.
8. Many details are missing in the paper to properly understand how the classification tasks are conducted. How does the paper decide where to “cut” the timeseries in order to conduct the SVM classifier and how is the 5-fold CV conducted/divided?  What hyperparameters were used for the SVM classifier? Given the datasets seem to have a label for each input, what is the reasoning for using a classification over many timepoints?
9. The paper claims “We show that our model achieves best translation performance on various brain ROIs in at least one metric”. I do not understand where this is shown, given no figure or table is referenced in this paragraph, nor the concept of ROIs seem to be explored here.

# Minor comments and suggestions
1. I want to show my appreciation for the reproducibility statement, but as a suggestion, an anonymised code would have made this submission stronger and potentially clarify some of the questions I am asking in this review. 
2. Some acronyms appear in the text without being defined or specified with a reference beforehand, for instance BCI, BOLD, PCA and SVD. Although it’s reasonably fair to assume the majority of the ICLR community will know some of these terms, defining them it’s always better for preciseness, clarity, and avoidance of possible confusions for people in other fields reading this paper. I think for a computer science conference like this, the definition of BOLD signal should go beyond just spelling the acronym, as probably many people in the conference do not know what this means (e.g. in the Preliminaries section).
3. In the fields of neuroscience, a standard dataset for any fMRI study is the Human Connectome Project (HCP) or, more recently, the big UK Biobank. Were these more widely-used datasets not used for some specific reasons? I’m aware that the UK Biobank requires a long application process, but all HCP data are publicly available. 
4. As a suggestion for future work, I’d say it would be interesting to use resting-state fMRI which is typically quite noisier than the task-specific fMRI used in this work. I believe it would be interesting to see whether the experiments on the manifold extension task would still work here.
5. How imbalanced are the classification tasks? This is important to understand whether accuracy is a good-enough metric for this task.
6. The paper claims that tSNE and UMAP “typically do not preserve overarching trends or relations between data regions”. For a complete related work section, the paper should probably mention that this is a contested statement and not so widely accepted as the paper claims. For example, some use UMAP for classification tasks (https://umap-learn.readthedocs.io/en/latest/supervised.html), and recent works explored preservation of global structure on real-world data, for instance: (1) https://www.nature.com/articles/nbt.4314, (2) https://www.nature.com/articles/s41540-021-00186-6, (3) https://www.nature.com/articles/s41467-020-15351-4.
7. Footnote (1) is confusing. How can one time point in one region of interest be a vector of length equal to the number of voxels? When the functional images are aligned to a common template, it is common to extract a single averaged timeseries per region of interest, otherwise why is this alignment done, just as a preprocessing step? Indeed, from the definition it seems that each subject only has in fact one single timeseries, as Y only varies across j (subjects) and k (timepoints) as subscripts. This needs to be further clarified.

# Minor comments to improve clarity and readability
1. The paper uses a metric named MDS without explaining what it actually means. I don’t believe this is a metric well-known by the community.
2. “ROI” is not defined in the paper even though it is used in a few places.
3. In Section 4.1, point (c ) starts with “They”; please define what this actually corresponds to.
4. In “Embedding Alignment Penalty” (in Section 4.2) there’s a typo in “common decoder (and individual encoders)” (should be the other way around)
5. “TRs” is not defined in Section 5.
6. How did the paper arrive at this set of hyperparameters? Manual choice? Hyperparameter sweep?
7. Typo in “bottle neck” (needs to remove the space) (Section 5, “Autoencoder Hyperparameters”)
8. Figure 2: yaxis should be only MSE (without the “vs” part)
9. Figure 2: I find the MSE values extremely small, and the confidence intervals overlapping a lot. Therefore, I do not agree with the paper’s claims that MRMD-AE achieves significant results here. I believe the paper needs to tone down their conclusions in this subsection or explain how this small value (i.e., below 0.001) can be so significant.
10. Typo: “An. SVM” (begin of page 7)
11. Typo: “unseen times series. fMRI” (Section 5, “Stimuli Classification”).
12. The paper uses a metric called “Earth Mover Distance” to support some of their claims (eg in Figure 3). However, this metric is not explained.
13. In one paragraph in Section 5 it is written “SVM classifier” and 2 paragraphs later it is instead “A support vector classifier (SVC)”. Are these the same or did something change?
14. Tables: I believe it would improve readability to reduce the number of decimal points. 2 decimal points in accuracy and 3 in standard deviation would probably suffice. 
15. Tables: What is the difference between “AE” and “Individual MR-AE”? If it is the usage of PHATE on MR-AE but not on an equivalent autoencoder with shared encoders and decoders, then please say in the text (and indeed this is a good ablation).
16. Table 3 is not referenced in the text.
17. Figure 3: Are the PHATE embeddings already in 2D space, or did the paper have to apply some technique like UMAP or tSNE for visualisation?

summary_of_the_review: Overall I think this is a very interesting and useful idea in the convergence of the fields of neuroimaging and learning representation. However, the lack of important experiments and explanations make me recommend only marginally above the acceptance threshold. I believe my questions can be mostly (if not all) addressed during a rebuttal process, after which I’d be happy to change my score as I think overall this should be shared among the ICLR community.

------------------ EDITED AFTER REBUTTAL PERIOD:
I followed the authors answers to my questions and other reviewer's questions. Although the authors didn't address all my questions (for example my comment titled "Further discussion") nor they uploaded a new pdf version to check corrections, I believe my main concerns were clarified. Assuming the authors introduce the written clarifications in the paper, I'm changing my review recommendation in two points:
- Changing "Correctness" from 2 to 3.
- Changing my "Recommendation" from 6 to 8.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission proposes a new model for functional alignment of fMRI datasets from multiple subjects. It combines a one-in-many-out autoencoder with two regularization loss terms (one inspired by GRAE) to develop a model that can encode every subject's data to a shared latent space, from which each subject's data is then decoded by a separate decoder. Experiments are provided that demonstrate benefits to some downstream tasks. 

main_review: # Clear and easy to understand introduction and motivation
The opening sections of the paper were clear and straightforward in motivating the functional alignment problem, manifold-based dimensionality reduction, and generating an approximate manifold embedding operator using a neural net. Likewise, the description of the problem setup, architecture choices and losses builds on itself in a way that was clear and easy to understand. I have only some minor suggestions for improvement here: 

[1a] As far as I can tell, the stimulus set $X$ and its elements $X_k$ is only used to say that the encoder maps the fMRI activation map to some representation of the stimulus, but $f'()$ is never defined explicitly. If so, it seems like unnecessary "mathiness" -- the notation is not buying additional precision or insight outside of saying this same thing in words. 

[1b] Expression 1 could color-code or otherwise annotate the distinct loss terms (reconstruction, latent closeness penalty, manifold regularization). 

# Weak evaluation
However, the empirical evaluation is less well-motivated and weaker overall. A strong empirical evaluation would show [2a] unambiguously state-of-the-art performance relative to [2b] strong baselines, and [2c] evaluations or [2d] ablations that target the core contributions. I detail each in turn: 

[2a] Performance benefits seem modest at best. Some very rough statistics: with 15-16 subjects, the confidence interval should be roughly equal to half the width of the SD on each side (since the z- or t-multiplier for the conventional $\alpha=0.05$ is roughly 2 and $\sqrt{15}$ is roughly 4). This means that (very roughly) differences between the best and second-best performing models smaller than the SD may not be statistically reliable. MRMD-AE is rarely convincingly above the next-best model by even a single SD. These statistics are, of course, rough: variability may be much smaller across folds than across subjects, but proper analysis (likely some sort of repeated-measures model) may be needed to tease apart this variability and establish the strength of MRMD-AE more convincingly. 

[2b] Strong baselines are missing. First, I'm not sure I understand why SRM is missing from the classification baselines -- it should be applicable here, right? Second, all the baselines are either linear (PCA, SRM) or single-subject (PHATE, AE), and there is no nonlinear functional alignment baseline, even though they exist in the literature (for example: Deep Generalized CCA, Benton et al. ICLR 2017; Deep Hyperalignment, Yousefnezhad & Zhang NeurIPS 2017; Kernel Hyperalignment, NeurIPS 2012 -- and this is only a brief survey). More advanced linear methods likewise exist  (e.g. RSRM, Turek et al. ICASSP 2018; MN-SRM, and DP-SRM, Shvartsman et al. AISTATS 2018). Comparing to all of these methods is not critical but it's hard to get a sense of whether the new method truly exceeds the state of the art without at least a few stronger functional alignment baselines. 

[2c] As I see it, the key contributions here are the one-in-many-out architecture, and the additional manifold and latent space closeness regularization terms. A full set of evaluations would probe each in turn, showing how the present contribution performs best in out-of-sample performance of some reconstruction metric, some latent space closeness metric, and some manifold alignment metric. The paper reports untrained cross-subject translation (which is a sort of out-of-sample reconstruction metric), and the future-point-placement metric (which is sort of a latent space closeness metric). The benefit of the manifold regularization penalty is seen in the comparison to MR-AE. Importantly, the overarching narrative present in the front matter of the paper is much weaker here, and the link made above between the contributions and the evaluations is not made explicit. 

[2d] The ablation study is not particularly informative about the contributions of components outside of the translation penalty (since there is no model with translation but no alignment penalty, and no model with translation but no manifold regularization). To be clear: I don't think ablations studies are critical to this paper -- but the specific ablation study in the submission could be stronger. 

# Minor comments and typos
- In the discussion, "For many studies, fMRI data is now being collected from thousands of subjects" is optimistic. I only know of the HCP and IMAGEN projects, both of which are massive multi-institution efforts. Most fMRI studies are still quite small (tens of subjects at most). 
- It would be nice (but not critical) to see some empirical demonstration of the benefit of splitting g() into g'() and g''(), and verification or citations of the claim that applying PHATE to the group data fails. 
- The motivation for the time-splitting experiment is a bit odd: in my experience pilot fMRI collections are still full length collections from different subjects, not short collections from the same subjects. 
- Figure 2 should identify the dataset used (I think it is Sherlock?). 
- I wonder if part of the reason that MRMD-AE fails to perform well on the localizer task is that it overfits to movie data, i.e. it is capturing movie-specific variability that is not present in the localizer data. 
- Section 3 first paragraph "train data" -> training data
- Page 5 under "manifold regularization penalty" paragraph -- "individual-subject manifolds subject", second "subject" seems extraneous. 
- Last line of section 4: subjects fmri activities -> subjects' fmri activities. 
- First sentence on p7 Stimuli Classification section: "unseen times series. fMRI data" -> "unseen fMRI data" or "unseen time series" or "unseen time series of fMRI data" etc. 

In sum, I think issues [2a] and [2b] are most critical and keep the paper from clearing the bar for ICLR. Addressing [2c] and to a lesser extent [2d] would also strengthen the paper, but would not be sufficient if [2a] and [2b] were not addressed. 

EDITED: I am adjusting my rating to a 6 based on additional information provided about the raw classification scores, but I'm still concerned about the baselines chosen. 
summary_of_the_review: I think this is a well-motivated approach and the early parts of the paper are a pleasure to read: it was easy to understand what each piece of the contribution is meant to do, and its motivation and relation to past work. When it comes to the empirical evaluation, however, the paper runs into a common problem with noisy fMRI data, which is that it's hard to convincingly beat linear methods, and it then overstates (to my read) the strength of its empirical contributions. The performance gains are modest and nonlinear functional alignment methods (e.g. from the hyperalignment family such as KHA/DHA) and more advanced linear methods (such as RSRM and MN-SRM) are not included in the comparison. The structure of the latter portions of the paper suffers as well: while the specific modeling choices are well-motivated, the experiments do not fully probe the novel contributions. As a result, while I think this line overall is worth pursuing, I would like to see stronger empirical results for this work to be a successful ICLR submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a model to learn a low-dimensional representation of fMRI data over multiple subjects of the same experiment. The model is built as an auto-encoder, with an encoder shared across subjects, and a separate decoder per subject. The model is regularized so that the first layer of each decoder gives a representation close to a pre-computed manifold embedding. An optional regularization also constrains the shared representation to be similar across subjects. 
The paper then proposes a series of experiments to demonstrate the benefits of the learn representation. The experiments consider the tasks of (a) projecting new test samples to the learned manifold, (b) classify some stimulus features from the embedding (decoding task), and (c) predict brain recordings of a new subject.
main_review: ## Strengths
The problem tackled by the paper is interesting, and the proposed model is original. The model is sound, conceptually interesting, and reasonably described. It is based on recent techniques that are appropriately described and cited. The paper is clear and well structured.

## Weaknesses
1. It is not super clear why the proposed architecture uses a shared encoder and a separate decoder per subject. This choice seems central to the proposed method and should be better explained.
2. It is also not clear which decoder was used to predict on held-out subjects. It seems like a decoder must be trained at some point on the held-out subjects, which might undermine the integrity of the cross-subject experiment.
3. It is not clear how hyper-parameters lambda and mu were set, with a reasonable risk of overfitting the proposed benchmarks. The authors should explain how the hyper-parameters were chosen.
4. "MRMD-AE achieves lower or matching MSE than the landmark approach" does not seem entirely supported by the data. The landmark approach is actually below MRMD-AE at 70% of train set, and the difference does not seem relatively smaller than the difference at other percentages.
5. Tables with numbers are very non-intuitive, and might even be considered a ploy to hide small differences in performances between methods. The authors should use graphs, and move the Tables to the appendix if still considered necessary. If still using tables, the authors should remove non-significant digits (6 digits are not necessary when the standard deviation is at 5e-2.
6. Figure 4: when using barplots, the y-axis should start at 0. Indeed, barplots use our intuitive perception of areas to convey the difference of values, but this perception is biased if the bar does not start at zero, and the difference of values is artificially increased.
7. "We will provide a link to an anonymous repository to the reviewers and area chairs", yet the link was not provided.
8. "fMRI data is now being collected from thousands of subjects" should be tempered by the fact the current study is limited to 16 subjects. 

### Unclear sentences
- The "Manifold Regularization Penalty" section is quite hard to follow. The authors might improve it by using shorter sentences.
- What is a "soft" regularization ?
- What is a "batch effect" ? and a "strong subject batch effect" ? 
- What is a "individual-subject manifolds subject" ?

### Typos
- [Moon 2019a] and [Moon 2019b] are identical.
- "demonstraging"
- "that that"
- "Reduduction"
- "We therefore compare" (no dot)
- "TRs" is not defined
- Fontsize in Figure 2 is too small.
- "An. SVM"
summary_of_the_review: My recommendation (reject) is based on
- a missing justification of the architecture design
- a number of missing important details about the fitting procedure
- a poor quality in the reporting of results
review_title: null

Meta-review: In this paper, the author present a method for learning a shared latent space between the fMRI activity of multiple individuals processing the same stimulus. The method consists of an auto-encoder with a single encoder and subject-specific decoders which is specifically regularized to decouple common and shared representations. This paper generated a lot of discussion between the reviewers and the authors, as well as between the reviewers. In light of these discussion, I cannot recommend acceptance at this point, as the paper is not ready. The main concerns were (1) about how the results and improvement are evaluated statistically, (2) that the baselines chosen were not strong enough and did not include existing approaches (neural or non-neural) and relatedly (3) that the paper was not framed correctly within the existing literature on finding shared spaces between participants, which would help with determining and understanding the novelty of the proposed approach. Some other smaller points were made by the reviewer can also strengthen the paper for a future submission in a neuroscience or machine learning venue.
Decision: Reject