Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, different self-supervised learning architectures are adopted from NLP literature to predict brain activity. Specifically,

1. Autoencoders using LSTMS
2. GPT-style autoregressive models
3. BERT-style masked autoregressive model and a next sentence prediction task adapted to fMRI data (are the two sequences from the same dataset or not)
4. Same as 3 but a particular network (akin to an emebdding dimension) is dropped from all input embeddings

The paper explores training all 4 architectures using data from a large collection of fMRI experiments, and finds that all models train well. While there is no clear effect of model depth, models with higher capacity perform better.

Finally, the pretrained models are evaluated on two downstream datasets (for task classification) and are all found to beat traditional linear baselines.
strengths_and_weaknesses: Edit after rebuttal: I believe that the authors have sufficiently answered my questions and thought the response on using pertained GPT representation was especially interesting. I agree with other reviewers that in its current form, the usability of the networks for real-world downstream tasks is limited. However, I believe this paper is, to the best of my knowledge, highly novel and will spur interesting further research. I recommend it for acceptance.

#########################################################################################################

Strengths:
1. Clarity: This paper was very well written, easy to understand and well visualized. I enjoyed reading it!
2. Quality: I found the results presented convincing, although there is a lot to unpack and it is not fully apparent to me how the choice of hyper parameters, data construction schemes etc. affect the observed results
3. Originality & Significance: To the best of my knowledge, this is the first study to explore such pretraining schemes for fMRI data. It is a very exciting and significant direction for future research!

Weaknesses:
1. It took me a while to understand the need for both positional encodings and the TR encodings. Clarifying this earlier would improve readability in my opinion.
2. I would urge the authors to add standard error to the values in fig.5 over many random seeds.
3. Several (all?) of the analyses do not currently report standard error. Given the number of design choices made throughout, I believe this would be an important addition to improve the paper.
questions: Questions:
1. (Lines 237-238) Isn’t 1-5 TRs too small a gap considering HRF? Have the authors experimented with longer gaps?

1. Does the model use low-level cues like different repetition times for the task equivalent to next-sentence-prediction? One way to analyze this would be to correlate CE with some measure of difference in the TRs. Similarly, what about cues like participant ID or task type given that specific networks will be active depending on the task?
2. I am not sure I follow the experiment in section 3.4. If the experiment is to use GPT/BERT pretrained on text and then finetuned on the fMRI setup, why would that work?
3. How is the data quality/SNR related to training/validation/test loss? This was not clear to me but did the original pretraining also split train/val/test such that no participant contributed data to more than one set?

Misc. questions unrelated to my views of the current manuscript:
1. What is the relationship of ablating differentiating diff. networks on the downstream task accuracy? Is there a possibility for meaningful relationships between the same?
2. Relatedly, what is the impact of ablating a specific network on the MAD of different networks in the output? Is there some functional, anatomical organization to the same?
limitations: -NA-
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors benchmark a number of NLP architectures in the task of learning representations from fMRI data. The architectures include a seq2seq, transformer, and transformer-based BERT models. Unlike the original NLP models trained with cross entropy loss, the reconstruction loss is used in the presend work instead. The proposed objectives were pre-trained on 34 and evaluated on two publicly available datasets. The results show applicability of self-supervised pre-training for downstream tasks such as mental state decoding. Potentially interesting work and research direction but with a weak evaluation and inconclusive utility of the approach.

strengths_and_weaknesses: ## Strengths
- A novel application of causal and masked data modeling to fMRI data.
- A demonstration of viability of the masked self-supervised training for pre-training of models embedding fMRI, as shown in the transfer learning on a state-decoding task on fMRI. However, the experiments are limited to one downstream task on two datasets. Evaluation and the training is also problematic (see the Weaknesses below)
- A wide variety of considered NLP approaches.

## Weaknesses (approximately in the order of importance)
- Unfair evaluation. Unlike the cross entropy loss in the original NLP applications of the employed models, the reconstruction error introduces a problem in all models but the seq2seq autoencoder, where the bottleneck representation $h$ is used for downstream task. The downstream evaluating model $p(\cdot)$ needs to be a part of the evaluation on the input X, not just the "Linear Baseline" from the paper. Main concern is that the learned embeddings and the input data X are only a single linear transform apart from each other. If not that transform, the CSM and BERT models would be encouraged to learn identity functions, up to a linear autoencoder.
- Inconclusive evaluation. Only accuracy numbers are shown, but the numbers highly dependent on the balance in the dataset. To avoid gross misinterpretations the field uses now standard approaches such as ROC AUC or F1, which in the multiclass case will OVO MACRO or similar. Balanced accuracy is also a better choice. A confusion matrix would reveal some details, but still require a sumarization.
- Evaluation on the downstream task should not simply refer to Appendix but present confidence intervals and statistical significance of improvements. The standard way of comparing performance. Since this is an application paper the rigor of comparison is the most important metric of paper quality. For example, loss curves are unimportant as uninformative, since we're evaluating the utility of considered self-supervised approaches and a closer focus on downstream task, including the model variability.
- Lacking comparison with models pre-trained in supervised manner. Given the data, all of the models (architectures) considered in the manuscript could have been pre-trained with the downstream model attached, i.e. $p(f(X))$ trained to predict the states, but only the obtained embedding used further in evaluation. It may happen that supervised embedding would be as or more efficient than the proposed approach.
- The experiments do not support the claim that training on the homogenous dataset limits the generalizability of the pre-trained models. Moreover, the self-supervised or supervised learning objectives are agnostic to the homogeneity of the dataset. Thus this is a limitation of the experimental setup in the related work rather than the limitations of the pre-training models. Combining multiple inhomogeneous datasets in the medical imaging domain might lead to underspecification and shortcuts that the model can use. Hence, researchers need to control for additional factors (e.g., site effect, age, gender). To support the claim, further experiments are needed to see how the number of datasets and their homogeneity affects the generalizability of self-supervised learning objectives and whether it is a limiting factor.
- The authors claim that the data is minimally-preprocessed. However, this and related work follows = standard preprocessing such as spatial smoothing and detrending. Common choices for preprocessing pipelines are fMRIPrep, C-PAC, or SPM. In addition, the authors use DiFuMo, which is a matrix factorization method, which itself is a way to embed fMRI data. The authors need to clarify what is minimal. Moreover, self-supervised objectives would be agnostic again.
- Note, the statements about the way BERT is trained are not technically correct. BERT is pre-trained either on Masked LM or Next Sentence Prediction. Not Next Sentence Prediction and reconstruction error simultaneously as used in the manuscript.
questions: - Would the results and ranking still hold if the embeddings are properly evaluated using metrics not sensitive to a possible data imbalance (see details above)?
- How would the results compare with supervised embeddings (see details above)?
- How does $p(\cdot)$, the same model as used now for everything but the baseline, applied to the input $X$ rank against other models (see details above)?
limitations: - An inconclusive demonstration of utility of the proposed approach (major).
- Focus on the pre-training approaches based on masking and lack of mention or comparison with contrastive self-supervised learning, which shows strong results in a similar context.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors have extensively experimented with adapting some well-known NLP models to neuroimaging datasets. They have successfully trained autoencoding, causal, and BERT-styled models on either predicting the entire input sequence or predicting parts of the input sequence (mask)
They achieve excellent results in the downstream tasks, especially with the causal seq models, and have shown the generalizability of the trained model to achieve good results on the as-yet unseen datasets.
These types of research are important to further the field of machine learning in neuroimaging as labels are really hard to come by.
strengths_and_weaknesses: Strengths:
A large and extensive dataset was used.
Number of key models from NLP has been adapted.
The pretext task is straight-forward and simple.

Weaknesses:
The pretext tasks don't have anything particular to tie them to predicting brain states. The authors could have used many a datasets out there to predict something useful as in autism or depression as a downstream task.
Why did causal models have to perform better than the rest? No investigation on this has been attempted.

questions: 1. Can you explain why you needed a preliminary dimensionality reduction using DiFuMo? And instead just start with X in R^(txp) space where p is the number of voxels (at least in the grey matter)?

2. Why did you restrict yourself to brain decoding and not mental disorders?
3. Why isn't the training code being made available?
limitations: Brain dynamics often is defined by the interaction between networks. It would be have been great if there were some ablation studies done to check which interaction or single regions were most informative to the downstream task.
review_title: null

Meta-review: metareview: This submission solicited interesting discussions between the reviewers and the authors, and was seen as bringing interesting ideas to the table. The ideas of the masked self-supervised pre-training for brain imaging is an exciting one.

One strong concern is that the evaluation is not very conclusive, and it is not clear that the present work gives solid evidence that the pre-training actually is beneficial.
confidence: Less certain
Decision: Accept