Peaks2Image: Reconstructing fMRI Statistical Maps from Peaks
Authors: Raphaël Meudec
Jérôme Dockès
Demian Wassermann
Bertrand Thirion
Venue: Submitted to ICLR 2023 (2023)

Abstract: Neuroscience is striving to overcome the lack of power due to the small sample size of standard studies. An important step forward has been the creation of large-scale public image repositories, such as NeuroVault. Such repositories enable comparing images across studies and automatically associating them with cognitive terms. Yet, this type of meta-analysis faces a major roadblock: the scarcity and inconsistency of image annotations and metadata. Another resource containing rich annotations is the neuroscientific literature. However it only yields a handful of brain-space coordinates per publication, those of the main activity peaks reported in each study. In this work, we propose Peaks2Image, a neuralnetwork approach to reconstructing continuous spatial representations of brain activity from peak activation tables. Using reconstructions of studies published in the neuroscientific literature, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies. We validate the decoder on 43,000 NeuroVault images, successfully decoding 58 out of 81 concepts in a zero-shot setting.

Under review as a conference paper at ICLR 2023
PEAKS2IMAGE: RECONSTRUCTING FMRI STATISTI- CAL MAPS FROM PEAKS
Anonymous authors Paper under double-blind review
ABSTRACT
Neuroscience strives to overcome the lack of power due to the small sample size of imaging- mraging-bbssed studies. An important step forward has been the creation of large-scale public image repositories, such as Neuro Vault. Such repositories allow images to compared across studies and automatically associated with cognitive terms. Yet, this type of meta-analysis faces major roadblock: the scarcity and in- consistency of image annotations and metadata. Another resource containing rich annotations is the neuroscientific literature. However only yields handful of brain-space coordinates per publication, those of the main activity peaks reported in each study. This has led the community to mostly perform anallssaalyss based on these reported coordinates. In this work, we propose Peaks2lmage, a neural- network approach to reconstruct continuous spatial representations of brain ac- tivity from peak activation tables. Peaks2Image thus associates rich annotations from the neuroscientific literature with dense brain reconstructions. Using those reconstructions, we train decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image- based studies. We validate the decoder on 43,000 VeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting.
INTRODUCTION
Cognitive neuroscience aims to map cognitive processes onto brain regions. Functional Magnetic Resonance Imaging (fMRI) is one of the most powerful techniques available to identify such asso- ciations. This approach measures brain activity while subjects perform cognitive tasks in an MRT scanner, and then contrasts brain signals associated with different mental conditions. Statistical test- ing of these contrasts then identifies brain regions where the neural activity elicited by the conditions probed is significantly different. Unfortunately, the high cost of data acquisition limits the number of participants and tasks involved in each study (Poldrack et al. 2017). A small number of participants causes low statistical power and high proportion of false discoveries (Ioannidis] | BButton et al.) 2013). Analyzing a restricted set of cognitive tasks introduces the risk of over-interpreting statistical effects that not specific to the mental functions under study (Poldrack 2011).
Meta-analysis consists in aggregating the results of several studies to find effects that are reported consistently (Wager al. 2007). It helps overcome the challenge small sample sizes and uncovers more reliable associations between brain activity and mental function. When feasible, meta-analysis should use the full statistical brain maps produced by the original studies (Salimi-Khorsiid) 2009). Unfortunately, the vast majority of studies does not share the actual brain images. Instead, only the locations of the peaks of activation are reported. These are communicated in the form of tables, in scientific publications, containing stereolactic coordinates 3D coordinates in a standard spatial referential for the brain. This results in a poor representation of brain activity - most of the information contained in the original statistical maps is lost. Meta- ancay-an that rely peak activa- tion coordinates reported in publications are called Coordinate -aaed Meta-Analyses (CBMA).
al.
Recent efforts openly share full brain images and statistical maps such Neuro (Gor-) golewski et al. 2015))eeee or OpenNeuro (Gorgolewski et al. 2017), could facilitate Image-Bac Based Meta- Analysis (IBMA) relying on the brain images rather than coordinates. However, annotations avail- able on those large databases are scarce and inconsistent, leading to difficulties to capture the seman- ties of the cognitive processes associated with images (Menuet al. 2022). There is standard,

---

Under review as a conference paper at ICLR 2023
agreed-upon ontology vocabulary of mental functions. Moreover, all that is known is the task that the participant is performing. and ascribing mental functions to tasks is difficult (Poldrack et al.) 2011). Therefore, formally describing the mental states that underlie a brain image remains an open problem.
Scientific literature and image repositories therefore complementary: the literature provides rich descriptions of the studied cognitive tasks but poor spatial information, whereas image repositories contain complete representations of brain activity but lack useful annotations. In the present study, we introduce a meta-analysis method that combines the strengths of both.
Beyond the challenge of sample size, effects uncovered by individual neuroimaging studies suffer from a lack of specificity. When a cognitive task is studied in isolation, there is no way of knowing if the observed brain activations are specific to the mental functions of interest, or associated with a broader set of mental processes. Authors often over-interpret the observed associations, resulting in the fallacy of unwarranted reverse inference (Poldrack) 201T). One way of identifying more specific associations between brain activity and mental function is decoding: inferring the mental processes at play, given a brain image of neural activity. Indeed, to discriminate between a wide variety of cognitive states, decoding model must identify brain regions that characterize each state, rather than brain regions that are merely activated consistently. For the resulting associations to be specific, it is crucial that many and diverse cognitive states are decoded jointly. Due to the difficulty of formalizing the mental states associated with a brain image, high-quality labels are lacking for this supervised task. Performing such large-scale (sometimes called "open-ended") decoding is therefore challenging, and in practice most studies that claim to perform "open-ended" decoding only discriminate restricted set of cognitive concepts.
In this work, Peaks2Image learns to discriminate dozens of cognitive terms on the largest available image repository, Neuro To do so, leverages both the rich descriptions of mental processes found in the literature and the high-quality neural activity data found in full-brain statistical maps. Peaks2Image reduces the gap between IBMA and CBMA, by reconstructing brain maps from peaks coordinates contained in neuroscientific publications. We leverage brain images from an unlabeled dataset to extract peaks, and train Peaks2lmage to reconstruct images from the extracted peaks. We use Peaks2lmage to obtain for the first time images associated with neuroscientific studies that only provide stereotactic coordinates. We evaluate whether those reconstructions are relevant by using them for brain image decoding. We associate labels with the studies using some criteria the term- frequency inverse-docunte dfcuument-frequunnyy of the text. We use the decoding architecture from Neural Networks Dictionaries (NNoD) (Menuet al., 2022 trained on the neuroscientific corpus. We evaluate the decoding performance against 81 terms from NeuroVault. Peaks2lmage succesfully decodes 65 of them on thousands of brain images from the Neuro Vault database, without using any supervision from Neuro Vault samples during training. While the evaluation has been performed on a limited set of terms, Peaks2Image can decode zero-shot setting any term from its vocabulary.
2 RELATED WORK
Automated meta-analysis (Laird et al. 2005 Yarkoni et al. (001) has risen over the last few years to handle the growth of published neuroscientific studies. Handling neuroscientific concepts properly has emerged as a challenge for CBMA, leading to the use of more complex textual features and models. Dockes al. (200)) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. (2021) leveraged language models (Beltagy et al., 2019 to capture term relationships semantically. leading to the encoding of any query. However, CBMA suffers from the drastic information reduction inherent to peak reporting.
Using dense images yields more information for meta-analysis (Salimi-Khorshidd et al.] 2009). Thanks to the rise of large-scale databases of fMRI brain images such as Neuro Vault (Gorgolewski cet al. 2015) or OpenNeuro (Gorgolewski et al.) 2017), decoding can now be performed across mul- tiple studies (Mensch et al. 2017 Walters et al. 2022). Nonetheless, annotations associated with those images are often of low quality. Automatic strategies to improve this labeling are necessary to benefit properly from the scale of the data (Poldrack and Yarkoni 2016). Menuet al. (2022) leverages the Cognitive Atlas (Poldrack et al. (011) to improve the label quality of Neuro Vault im- ages, enabling the decoding of large of concepts. Overall, neuroscience meta-analyses have
2

---

Under review as a conference paper at ICLR 2023
increasingly relied data-drives approaches, learning more efficiently across large set of data [Beam et al. 2021).
Although there has been interest in enriching brain images to increase the amount of data (Zhang cct al. 2019), to generate or reconstruct fMRI images (Manning et al. 2018 Cai et al., 2016). there has been limited interest in generating reconstructions of statistical brain maps from peaks (Gor- goolewski et a 2019), probably due to the daunting data dimensionality. particular, we found no attempl merging rich textual information from neuroscientific studies with the dense spatial
information of brain images.
We focus on this particular aspect, proving that obtaining image representations for neuroscientific studies helps overcome the limitations of CBMA by generating corpus with both rich textual annotations and dense spatial representations.
METHOD
Neuroscientists most frequently extract peaks (local maxima) above a statistical threshold from brain images. An image containing hundreds of thousands voxels is thus reduced to a few datapoints less than 100 for most studies. We aim reconstruct brain images from those few datapoints. This reconstruction problem is an inverse problem between the set of peaks and the image from which they were extracted. From a given set of peaks Py R3xi extracted from an image y E I with m voxels, we learn the transformation 3 m I by minimization of the L2 error over the original image:
"" arg min Eyll[(py) -y11
(1)
To solve the minimization problem in eq [ we leverage rault,, an online repository of IMRI maps, from which we extract peak coordinates. This leads a training set of peaks and the corresponding brain images to be reconstructed. Once trained, Peaks2lmage can be applied to any set of peak coordinates from neuroscientific publications. This leads to combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode larger set of cognitive concepts.
of statistical maps taken from Neu-
We show in Figure summary of Peaks21mage. The figure divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.
3.1 RECONSTRUCTION
Peaks2lmage consists of a neural network that reconstructs brain images from activity peaks. We tame the effect of outliers by rescaling image values to a standard range, thus losing the statistical scale of the effect but keeping the overall statistical distribution. Images are also resampled at fixed resolution and masked keep the brain volume only.
Instead of leveraging the peak coordinates directly as sets problem (Zaheer et al. 2017 Peaks21mage builds an internal sparse Gaussian representation similar to the literature as by Laird et al. (2005); Dockès et al. 2020);; Ngo et al.] (2021). It performs transformation between the Gaussian representation space and the image space. Formally, we consider P (PI..., Pt) peaks extracted from image y where Vi E [1..]] p; E R Given fixed sampling grid G (g) ... gm) representing the voxels, where Vj E [1.mm].g g E RR3, and a Gaussian kernel K we define the vector Xy that samples P on the grid G: xy (æ1, .......0 Fm) with Vjj E [1..m] : E'= (()..g)).
High-resolution is not needed here, as population-level images, that represent the average of indi- vidual data with variable shapes, are intrinsically smooth. We can thus perform a drastic dimension reduction of the data, which is welcome given that the sample size is inherently limited. We use the Dictionary of Functional Modes (DiFuMo) probabilistic atlas (Dadi et al. 2020), that efficiently reduces the m = 50, 000 voxel values to Is = 1024 components. DiFuMo has been trained on thou- sands of resting-state and task fMRI with non-negativity and sparsity constraints, making this atlas
3

---

Under review as a conference paper at ICLR 2023
Languu Language Processing rra Search
Theary of Mind
Reconstructed DiFuMo
Raconstruction MIPP (Training)
Reconsfruction (Infaranca)
Sparse Gaussian Repree,
Decoding MLP
TF-IF
per
Training
Evalustion
TF-DF
Z
Extraction
DiFuMo Repres.
Neuroscientiic Study
Reconstructed
Neuroscientifc Study
NeuroVault Collection
NeuroVaut 4438 excluded
4438
Y 2
Reconstruction Task
A. Training
B. Reconstructian of Neuroscientific Studies
C. Zero-Shot Decoding
Figure I: Reconstructing Brain Activation Maps from Activation Peaks (A) We learn the re- construction process from NeuroVaut collection #4438. After selection and preprocessing of the images, we extract the peaks that are above a statistical threshold. From those peaks, we build an intermediate representation by placing Gaussian kernel each peak as in Dockés al.] (2020); Ngo et al. ) We compress the representation using the DiFuMo atlas. We regress the original DiFuMo components of each image by leveraging a 3-layer multi-layer perceptron (MIP). (B) Once the reconstruction model is trained, scraped studics, and reconstruct a brain map for each study using the above model, supporting fur- ther image-level computations such decoding. From the text, extract the tf-idf values for each threshold. (C) We perform decoding of Neuro Vault
build intermediate Gaussian- kernel representations for the
word, and infer labels on a percentile-bat images by training the newly acquired representations of neuroscientific studies. We decode broad set of cognitive processes benefiting from the large vocabulary.
relevant to represent population-les information. the rest of the paper, the term brain image refers to its 1024-dimeniional DiFuMo representation.
Mapping voxels R" coefficients a E R² along DiFuMo dictionary D mm of dimension k is mere linear regression problem following equation All the following steps are performed in the k-diiensional DiFuMo space.
(xx) arg min ||x
Da
aERt
The DiFuMo-encoded sparse Gaussian representation turned into a dense representation with 3 3-layer neural network where the input and output are DiFuMo coefficients. Using the DiFuMo atlas instead of the raw voxels helps us keep a relatively low number of parameters for this model. This transforms Eq into:
f" arg min (f - a where x= - G)
(3)
To train this reconstruction process, we use around 6k images from the #4438 NeuroVaule collection
wwtp.//ooooooooooooe.oooooooo.ee ottp://rrrrraaattooooooooooeoooo /4438)
The reconstructed images are obtained from the predicted DiFuMo components by linear combina- tions of the dictionary components: ý =D)"(a(xx))

---

Under review as a conference paper at ICLR 2023
3.2 DECODING FROM PEAKS
Peaks21mage outputs images from any set of positions. We apply it to neuroscientific publications to yield a of dense brain maps accompanied by rich textual annotations, reducing the technical gap between CBMA and IBMA. By doing this on the most comprehensive public literature corpus, we reconstruct 13,000 brain images from the peak activation tables.
We use these representations to decode cognitive processes in zero-shot multi-label setting. In- deed, the ability to decode cognitive concepts from the produced database ensures its validity and circumvents the biases inherent to image reconstruction quality metrics, such as the confounding effect of smoothness. We evaluate the decoding performance on the NeuroVaulh database (#4438 excluded).
We extract the term frequency-inverse document frequency (tf-idd) (Salton and Buckley 1988) from the studies' text. The tf-idf reflects the relevance of a specific term to study by giving higher power to terms that occurs repetitively in few studies. Based on it, we assign labels to studies, by setting a threshold the 95th percentile of the tf-idf. This ensures that the few studies that most relevant to each term get positive labels. The computed percentile can be null for some extremely rare terms. null tf-idf value, we only consider strictly positive tf-idf values.
When the percentile corresponds
We leverage a 3-layer dense neural network to predict the set of binary labels L = {0, 1) x {0, 1h,,1 denoting the total number of classes, from the input image X E IR compressed in DiFuMo space. The model is trained from the sole neuroscientific studies, without supervision from any NeuroVaull sample its labels. We were limited for the evaluation by the annotations available in Neuro Vault. In the following experiments, we predict only 1=81 terms from the 6308 in the tf-idf vocabulary. Peaks2Image could extend to set of cognitive terms as large as the vocabulary size without any additional cost.
EXPERIMENTS
4.1 DATASET DESCRIPTION
We leverage the Neuro Vault database. We exclude subset of images according to set of rules such as having incoherent range of values (indicating that an image does not contain Z or statistics), duplicated images
images with missing metadata to keep around 50k brain maps for our task. We split those maps into two different sets in a collection-wee wise manner. We take collection #4438, without labels (6k maps), to train the reconstruction task. We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.
We NeuroQuery Data Collection (Dockès et al. 2020) to fetch around 13k neuroscientific studies from PubMed Central. The tool helps us parse the activation tables of those studies, totalling 400k peaks. also transforms each study text into if-idf features across a vocabulary of 6308 terms.
For validation, we restrict the analysis to shared terms between the tags of YeuroVaule images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non- oonninnifi effects, keeping 81 terms to decode.
4.2 DATA PREPROCESSING
Image preprocessing Images were fetched from Neuro Vault using the Nilearn library (Abraham al. 2014). We resample all images to the standard MNI152 template and apply the MNI152 brain mask. We use the DiFuMo atlas dimension 1024 available in Nilearn. As the coefficients of the DiFuMo representation can range large values, scale them by constant to facilitate the convergence of the neural network.
Peak extraction We generate pairs of matching (peaks, image) pairs from the collection (#4438) from Neuro Vault images only. Similarly, we apply masking, resampling and DiFuMo compression without the normalization step. We use Nileamn's method for peak extraction, which first separates
5

---

Under review as a conference paper at ICLR 2023
the images into statistically active clusters and retrieves peak coordinates from each cluster. As these images represent statistical maps, equivalent to p-values voxel-level tests, we keep peaks with a p-value lower than 10 10-a and with minimal distance between peaks within the same cluster of 8mm. These are standard parameters from peak extraction brain imaging contexts.
Sparse Gaussian Representations From given set of peaks, build sparse Gaussian repre- sentation by placing Gaussian kernels at each peak position with 9mm full-width at half-maximum, following what is done in Dockès et al. (2020). Similarly the NeuroVault images, we extract the DiFuMo representations of the sparse Gaussian maps using Nilearn.
4.3 TRAINING DETAILS
The image reconstruction model is an MLP built from dense layers with 1024 units and sigmoid activation implemented using PyTorch (Paszke al. 2019). trained for 50 epochs using Adam optimizer and learning rate of 1e-3. The loss is the mean-squared error (MSE)
Reconstruction
over the DiFuMo components. We select the best set of weights over the validation set, which is composed of 20% of collection #4438.
We compare Peaks2lmage to linear models (linear regression, kernel Ridge). We use implementa- tions from scikit-learn (Pedregosa et al. 2011) with default parameters.
Decoding The decoder model is an MLP composed of one hidden layer of 300 units and ReLU activation. It is trained for 200 epochs the reconstructed neuroscientific studies using the Adam optimizer with a learning rate of 3e-3 on the binary cross-entropy loss. During evaluation, we apply additional sigmoid operation to the model outputs. We the Area Under the Receiver Oper- ating Characteristic Curve (ROC AUC) over each label of the testing set to evaluate the decoding performance. Chance level is 0.5.
We performed 20 different runs of the experiments with different splits to obtain confidence inter- vals.
RESULTS
5.1 RECONSTRUCTION
We evaluate the reconstruction produced by Peaks2lmage all NeuroVaull except collection #4438 which represents a testing set of around 43k images. As Peaks2lmage reconstructs a normalized image, we use metrics both the mean-squared error that has been used at training time, and the correlation between the original and its reconstruction in the voxel space. Even though the model has been trained with respect mean-squared error in DiFuMoo space, find correlation more appropriate to measure the effects in those maps.
Correlation in voxel space
linear regression
O00s
kernel ridge
sparse gaussians
peaks2image
0.2
0.0
0.2
0.4
0.6
0.8
Figure 2: Correlation between original images and reconstructions: We evaluate the reconstruc- tion for different models (Linear regression, kernel Ridge regression, sparse Gaussian representation and Peaks21mage) using correlation in voxel space. Evaluation is performed collection 4337 from Neuro (18k images).
We show the reconstruction performance of each model Fig. 2 Peaks2Image significantly out- performs linear models. As shown in the next section, although the correlations obtained with the

---

Under review as a conference paper at ICLR 2023
different methods comparable, the Peaks2lmage reconstruction yields much better performance on the downstream decoding task. This suggests that some aspects of the image reconstruction are not captured by simple metrics such the MSE or correlation. We plot examples of reconstructions for different labels in Fig. 7
5.2 DECODING PERFORMANCE
We validate the relevance of the reconstructions by decoding Neuro Vault images using reconstructed neuroscientific studies as training set. The problem is multi-label classification, which we evaluate by computing
ROC AUC over each label. We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from Neu- ro Vault. We apply label enrichment to the original image labels. This includes regular expressions map different terminologies for same concept, and fierarchy-basse inference of labels based on the Cognitive Atlas ontology. For a given term, we assign parent terms from the ontology used in (Menuet al. 2022)
This results in 43k images, totalling around 200 different cognitive processes. We evaluate only cognitive processes with at least 50 positive examples avoid results with small sample. This reduces the evaluation to 81 terms. The limitation of terms is induced by the evaluation against Neu- Vault, but Peaks2lmage could extend the prediction any word included in the tf-idf vocabulary.
In Fig. 3 we summarize the decoding performance by aggregating over Cognitive Atlas categories. Peaks21mage performs well over all the main concept families of Cognitive Atlas.
Tasks decoded
AUC per category
above 0.55 AUC
Action
2/5
Mean AUC
Attention
across category
3/4
AUC for individual
Emotion
task category
3/6
Executive
cognitive control
Language
9/9
Learning and memory
11/12
Misc
11/14
Perception
11/14
Reasoning
10/13
decision making
Social function
0/1
0.2
0.4
0.6
0.8
1.0
Figure 3: Zero-shot decoding performance on Neuro Vault: evaluation is performed on all Neu- roVallt except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2lmage decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC 0.55).
We show the importance of the reconstruction process in the decoding performance by running the same decoding task over different representations of the input. In particular, we compare Peaks2lmage reconstructions to the sparse Gaussian representations used in the literature. We show that, even though the correlation of sparse Gaussian representations remains close to Peaks2lmage reconstruction, Peaks2Image largely outperforms the sparse Gaussian model task. We observe a statistically significant gain 0.1 on the mean AUC across labels between Peaks2lmage and sparse Gaussian representations. We report the performance of each method in I
the decoding
We also compare Peaks2lmage to approaches from the literature. On terms that are shared with NeuroSynth (Yarooni et al.) (00IT) , we show that Peaks2lmage outperforms the former, while largely expanding the number of cognitive processes decoded.

---

Under review as a conference paper at ICLR 2023
Reconstruction Architecture
Decoding Model
Mean AUC across tasks
Peaks2Image
Logistic Regression
0.503 (# 0.001)
Sparse Gaussian
NNoD
0.613 (+: 0.004)
Kernel Ridge
NNoD
0.638 (#: 0.004) 0.698 (# 0.002)
Peaks2Image
NNoD
decoding from Peaks2lmage recon- structions outperforms the simple sparse Gaussian representation, well as kernel Ridge learner. Using the NNoD model with a hidden layer brings strong power over mere Logistic Regression.
Table 1: AUC performance of Peaks2lmage and baselines:
Peaks2lmage AUC Performance Relative Gain
vs Logistic Regression
4= 0.06 3e-131
vs Kernel Ridge
A 0.08 p 3e-124
vs sparse Gaussians
vs NeuroSynth 4= 0.06 0=0.2
no relative gain
-0.6
-0.4
-0.2
0.0
0.4 0.6
0.8
1.0
Figure 4: Distribution of relative AUC performance across tasks of Peaks21mage compared to other representations: We show the relative AUC gain of Peaks2lmage for each task, along with the difference of mean AUC across labels vs different methods. We perform a t-test to probe the statistical significance of the difference between Peaks2Image and other methods. We also display the average AUC difference (A). For NeuroSynth, we compute the difference on a subset of shared terms.
We plot in figure 5 the difference between NNoD performance and Peaks2Image performance. Par- ticularly, we add to this graph the performance Peaks2lmage on unseen terms, showing that Peaks2lmage enables the successful decoding of broader set of terms. Peaks2lmage cannot com- pete with NNoD on terms that are in the training set. NNoD has been trained specifically on Neu- Vaulu images, leading logically better decoding performances. However, Peaks2Image' value is that it makes it possible to decode unseen terms, leading to a larger set of decoded terms while keeping relatively good performances where NNoD outperforms
We noted an effect of NNoD being able to decode some terms unseen during training. We made sure that no data leakage occurred and compared results with other linear models which reached an expected 0.5 AUC. Our intuition is that the multi-label setting during training and the multi-layer structure lead a compression effect akin to PCA, leading to above chance decoding levels for certain tasks. This effect is negatively correlated with the number of positive samples in the testing set.
DISCUSSION
We introduced Peaks2lmage, model for reconstructing brain images from peak activity coordi- nates. By reconstructing dense spatial representations, Peaks2lmage bridges the gap between neu- roscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neu- roscientific studies, along with the reconstructions, Most importantly, the decoding performance reached without any labeling requirements on the images side, avoiding a time-consuming and error-prone task.
limitation of Peaks2lmage is our strategy regarding the studies text. hinders comparisons by limiting exact matches between vocabulary and tags from Vault. While Peaks2lmage could extend to any word contained in the neuroscientific literature, evaluation was reduced to limited number of concepts due to the lack of extensively annotated brain images. Exploration of more
8

---

Under review as a conference paper at ICLR 2023
AUC performance
risk seeking
response inhibition
mental rotation belief
0.9
reading
cueing
object perception
Peaks2lmage performance
0.8
0.7
0.6
social cognition
0.5
arithmetic processing detection cognitive control maintenance punishment processing pattern recognition
0.3
category seen in train
category not seen train
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
NNoD performance
Figure 5: AUC performance of our reconstructions Neuro samples: we compare the performance of the NNoD architecture on Neuro Vault (except collection #4438 used for the re- construction training) in two settings: training on NeuroVault data (collection 4438) or training reconstructions with percentile-inferred labels. NeuroVault training outperforms when it contains the term considered, but Peaks2lmage provides better guess when such training data avail- able.
powerful language models as Ngo al. (2021) is a future direction to better benefit from the continuous brain images generated for the neuroscientific studies. Though, the poor quality of Neu- VaValt annotations would still limit our ability to evaluate those models.
Further investigations are still necessary to study the impact of the dataset used for training. A promising avenue consists relying on abundant resting-state data, that do not carry cognitive la- bels, yet display rich repertoire of topographical maps, that can be extracted e.g. with Independent components analysis Smith et al. (2013).
Pcaks2lmage paves the way towards better generation of large sets of images for neuroscientific studies. shows that one can outrun the limitations of large-scale poor labeling quality by perform- ing zero-shot decoding. In particular, neuroscientific studies could become challenging application for recent methods in text-to-image representation learning such as |Radford et al.) (2021).
7 REPRODUCIBILITY STATEMENT
Data was collected from public sources using either the NQDC module from NeuroQuery or the Nilearn library. All scripts reproduce the experiments and figures will be released publicly after review, along with the downloading scripts and preprocessed data. In particular, we will share the DiFuMo components of all images from our current version of the NeuroVaull dataset. For neu- roscientific studies, we leveraged the data available at hhtpp:: /////ttiae giithu.. com/ http//euroquery// neuroquery_daet Saweerrydddatteeeeeeeeeee teatta/tteeeeee d/

---

Under review as a conference paper at ICLR 2023
REFERENCES
A. Abraham, F. Pedregosa, M. Eickenberg. P. Gervais, A. Muelerr, J. Kossaifi, A. Gramfort, B. Thirion, and G. Varoquaux. Machine learning for neuroimaging with scikit-learn. Frontiers in neuroinformatics, 8:14, 2014.
E. Beam, C. Potts, R. A. Poldrack, and A. Etkin. A data-driven framework for mapping domains of human neurobiology. Nature neuroscience, 24(12):1733-7744 2021.
1. Beltagy, K. Lo, and A. Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv: 1903. 10676, 2019.
K. S. Button, J. P. loannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, and M. R. Munafò. Power failure: why small sample size undermines the reliability neuroscience. Nature Reviews Neuroscience, 14(5):365, 2013.
M. Cai, N. W. Schuck, J. W. Pillow, and Y. Niv. A bayesian method for reducing bias in neural representational similarity analysis. Advances in Neural Information Processing Systems, 29, 2016.
K. Dadi, G. Varoquaux, A. Machlouzarides-Shali K. J. Gorgolewski, D. Wassermann, B. Thirion, and A. Mensch. Fine-grain atlases of functional modes for fmri analysis. Neurolmage, 221: 117126, 2020.
Dockès, R. Poldrack, R. Primet, H. Gözükan, Yarkoni, F. Suchanek, B. Thirion, and G. Varo- quaux. NeuroQuery, comprehensive meta-analysis of human brain mapping. Elife, 9, Mar. 2020.
K. Gorgolewski, O. Esteban, G. Schaefer, B. Wandell, and R. Poldrack. Openneuro-a free online platform for sharing and analysis neuroimaging data. Organization for human brain mapping. Vancouver, Canada, 1677(2). 2017.
K. J. Gorgolewski, G. Varoquaux, G. Rivera, Y. Schwarz, S. S. Ghosh, C. Maumet, V. V. Sochat, T. E. Nichols, R. A. Poldrack, J.-B. Poline, Neurovault. org: web-based repository for collecting and sharing unthresholded statistical maps of the human brain. Frontiers in neuroinformatics, 9:
8, 2015.
K. Gorgolewski, T. Yarkoni, and R. A. Poldrack. peaks2maps: reconstructing unthresholded statistical maps from peak coordinates using deep neural networks. F1000Research, 8, 2019.
J. P. loannidis. Why most published research findings are false. PLoS medicine, 2(8):e 24,, 2005.
A. R. Laird, J. Lancaster, and P. T. Fox. Brainmap. Neuroinformaties, 3(1):65-77, 2005.
J. R. Manning, X. Zhu, T. L. Willke, R. Ranganath, K. Stachenfeld, U. Hasson, D. M. Blei, and K. Norman. probabilistic approach discovering dynamic full-brain functional connec- tivity patterns. Neurolmage, 180:243-252. 2018. ISSN 199.-8119. doi: https://doi.or// 1016/j.eeuroimage2211....00 atriccee/p1//11033111111333 715
URL tettts://weett www.eiiiiinecrrect....ccmm...... wwwpiiirrtttccoooeeeooeemmmooe www.co///sce.... http/sciencce//// New advances in encoding and decoding of brain
signals.
A. Mensch, J. Mairal, D. Bzdok, B. Thirion, and G. Varoquaux. Learning neural representations of human cognition across many fmri studies. Advances in neural information processing systems, 30, 2017.
R. Menuet, R. Meudec, J. Dockès, G. Varoquaux, and B. Thirion. Comprehensive decoding mental processes from web repositories of functional brain images. Sei. Rep., 12(1):7050, Apr. 2022.
G. H. Ngo, M. Nguyen, N. F. Chen, and M. R. Sabuncu. Text2brain: Synthesis of brain activation maps from free-form text query. In International Conference on Medical Image Computing and Cosputer-Asiisted Intervention, pages 605-614, Springer, 2021.
10

---

Under review as a conference paper at ICLR 2023
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL
neurips cc/paper/ 9015 pytorch an imperative style high performance deep learning Library. pdf
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Couraaeeau M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
R. A. Poldrack. Inferring mental states from neuroimaging data: from reverse inference to large- scale decoding. Neuron, 72(5):622-697, 2011.
R. A. Poldrack and T. Yarkoni. From brain maps to cognitive ontologies: informatics and the search for mental structure. Annual review of psychology, 67:587, 2016.
R. A. Poldrack, A. Kittur, D. Kalar, E. Miller, C. Seppa, Y. Gil, D. S. Parker, F. W. Sabb, and R. M. Bilder. The cognitive atlas: toward a knowledge foundation for cognitive neuroscience. Frontiers in neuroinformatics, 5:17, 2011.
R. A. Poldrack, C. I. Baker, J. Durnez, K. J. Gorgolewski, P. M. Matthews, M. R. Munafò, T. E. Nichols, J.-B. Poline, E. Vul, and T. Yarkoni. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature reviews neuroscience, 18(2):1 1122:115-126 2017.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In Inter- national Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
G. Salimi-Khorshidi, S. M. Smith, J. R. Kelmer, T. D. Wager. and T. E. Nichols. Meta-analysis of neuroimaging data: comparison of image-based and coordinate- pooling of studies. Neuroimage, 45(3):810-833, 2009.
G. Salton and C. Buckley. Term-weighting approaches in automatic text retrieval. Information processing management, 24(5):513-522 1988.
S. M. Smith, C. F. Beckmann, J. Andersson, E. J. Auerbach, J. Bijsterbosch, G. Douaud, E. Duff, D. A. Feinberg, L. Griffanti, M. Harms, M. Kelly, T. Laumann, K. L. Miller, S. Moeller, A. Z. Snyder, A. T. Vu, M. w. Woolrich, J. Xu, E. Yacoub, K. Ugurbil, D. C. Van Essen, and M. E. Glasser. Resting-state fMRI in the Human Connectome Project Neuroimage, 80:144-163 168, Oct 2013.
S. Petersen, J. Power, G.
T. D. Wager, M. Lindquist, and L. Kaplan. Meta-analysis of functional neuroimaging data: current and future directions. Social cognitive and affective neuroscience, 2(2):150-158, 2007.
J. Walters, M. King, P. G. Bissett, R. B. Ivry, J. Diedrichsen, and R. A. Poldrack. Predicting brain activation maps for arbitrary tasks with cognitive encoding models. Neurolmage, page 119610. 2022.
T. Yarkoni, R. A. Poldrack, T. E. Nichols, D. C. Van Essen, and T. D. Wager. Large-scale automated synthesis of human functional neuroimaging data. Nature methods, 8(8):665- 670, 2011.
M. Zaheer, Kottur, S. Ravanbakhsh, B. Poczos, R. Salakhutdinov, and A. J. Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Zhuang, A. G. Schwing, and O. Koyejo. Fmri data augmentation via synthesis. 2019 JEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 1783-1787, 2019. doi: 10. 109//SBIIIB0 2019 87595855
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 RECONSTRUCTION OF NEUROVAULT IMAGES
We show example of the representation at different steps of the Peaks2Image model in figure
R
R
Sparse Gaussian Representation
Peaks2lmage Reconstruction
NeuroVault image
Extracted peaks
Figure 6: Successive steps Peaks2Image: we show the successive steps Peaks2Image rep- resentations. From left to right, collect brain images from Neuro Vault from which extract peaks above a statistical threshold, we transform the peaks into a sparse Gaussian representation from which we predict the Peaks2lmage reconstruction.
We also apply Peaks2lmage to Neuro images that are associated to set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig) 7p.
A.2 PER-LABEL DECODING PERFORMANCE
We ran 20 runs of the experiment. In figure we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.
12

---

Under review as a conference paper at ICLR 2023
Average of Gaussians sparse representations
Average of Reconstructions
audition
R
face
R
R
R
language production
R
R
pain
R
R
R
sadness
R
R
visual
R
L
R
Figure 7: Reconstructions of Neuro Vault samples: we show the average sparse Gaussian represen- tation used in standard meta-analysis procedures (left) along with the Peaks2lmage reconstruction (right) for Neuro samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the ini- tial map (sadness). In some rare cases (visual) focuses the activity patterns.
13

---

Under review as a conference paper at ICLR 2023
Performance
lanuuage production
reward ieami
interference
tallucination
empathy
strategy
prosady
thought
guilt
trait anxicty
utility
perceetual identifiiation
sadnees
incidental leamning
phonological processing
extinction
pavlovian cond onnng
expression
emoti onal
anwiety
divergent think
lyl
self monitaring
conceptual prim
verbal flucncy
mani pulation
salience
cagnitive load
behavioral inhibition
perception
negative
mecor planni ng
navigation
fargetting ng
strength
stress
reward valuation
uncertainly
pain
creative thinking
motor
source memory
processing
prospertive
cognitive effort
social inference
risk seek ng
adaptation
attention
surprise
audition
effort
response inhi bitinition
attitude
sleep
intelligence
monitoring
inference
inhibition
opisadic momory
risk processing ng
exp expectancy
language
mental imagery
arousal
decision making
Instrumental
efficiency
mental
agency
mathematical reasoning ng
visual memory
habit
visual search
judgment
anticipation
Instrumental conditioning ng
interference: contra
loss anticipation
face perception
auditory scene
search
semantic categorization
memory retrieval
concept
error detection
focus
speech processing
semantic memory
reasoning
priming
priming
visual recognition
sequence
autoblographica memory
emotion regulation
motion delection
word recognition
framing
reward anticipation
speech
face recoonition
attachment
reading
sustalined attention
deductive vo reasoni ng
response conflict bellef
domain specificity
color perception
binocular rivalry
teature detection
reinstatementt memory
analogical reasoning
logical reasoning
localization
visual working acquisition
comprehension visual
object perception
problem solving
discrimination
selective control
rule leamn ng
schema
attention
knowledge
perception
rei nonorrmment ieami ng
reward pracess ng
overt nam
Imagination
word generation
aculty
rhythm
object recognition
communication
fixation
goal
mental arithmetic
filtering
prototype
categorical perception
emotion recognition
emotion perception
response execution
response selection
spatial attention
visual attention
detection
decision
balance
lemma
action
social cognition
negative emation
visual recogn
warking memory
arithmetic processing
spatta) ability
memory
visual orientation
task difficulty
numerical comparison
introspcction
cogintoo
maintenance
attentional
contrel
pattern recognition
happiness
Chance level
parsing
eexical processing
shiment processing
0.2
Peaks2lmage to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes large part of those terms in zero-shot setting. Peaks21mage could extend to any word from the studies" vocabulary, but could be evaluated on the Neuro Vault annotations only.
Figure 8: Zero-shot decoding performance
Vault:
14


Reviews:
[{"review_id": "xalnM0a18P", "reviewer": "Reviewer_LK9E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a neural-network approach to reconstruct continuous spatial representations of brain activity from peak activation tables. Using reconstructions of studies published in the neuroscientific literature, the authors train a decoder leading to a much broader set of decoded terms than current image-based studies.", "summary_of_the_review": "This paper contains novel ideas such as utilizing the rich descriptions of the studied cognitive tasks in scientific reports to assist in brain decoding task. However, I have several concerns about the image reconstruction from peaks and experiments. For more details please see the ‘Strength And Weaknesses’.", "strength_and_weaknesses": "Strength:\n1.This paper is well-written and easy to read. The definition of tasks and problems are very clear.\n2.For a decoding task, the idea that fully utilize the rich descriptions of the studied cognitive tasks in scientific report is novel.\nWeaknesses\n1.My major concern is the necessity of image reconstruction from peaks. As illustrated in the paper, the authors train a reconstruction network to convert the peak coordinates into images. In my opinion, this ill-posed problem is too hard to be reasonable. Reconstructing the fMRI from only a few peak coordinates (less than 100 according to the paper) seems impossible to recover the details of images. Thus, the reconstruction is unnecessary. If the peak coordinates contain enough information for the following decoding tasks, reconstruction would introduce inevitable noise. If the peak coordinates are not enough, reconstruction still can’t help the decoding since extra information comes from the ‘guessing’ of the trained network. Can the authors give more explanation about this?\n2.Also, the experiments do not verify that image reconstruction can help in improving the decoding performance. The ablation study should compare with model using peak coordinates as input. So, I am wondering about the performance of model that uses peak coordinates as input.\n3.Experimental results are all shown in figures, which is good for analysis. However, giving some extra tables with exact values can be better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JJi8f6Av8O", "reviewer": "Reviewer_GJ9C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Peaks2Image as a neural-network approach to reconstruct continuous spatial representations of brain activity from peak activation tables. This study aims to learn a decoding model by using TF-IDF features as labels so that the number of decoded terms will be much broader than in current image-based studies. Based on 43,000 NeuroVault images, they successfully decoded 58 out of 81 concepts. In summary, this paper is interesting in terms of neuroscience, but it is unclear to what extent the proposed method is novel in terms of machine learning.", "summary_of_the_review": "This paper contains several controversial claims. \n\nThe paper uses Nilearn to estimate peaks and then uses these peaks to learn a model from data. However, there is no evaluation of the error associated with the selection of these peaks. Further, what evidence shows that you are more likely to see neural responses in these peaks than examining the whole brain data?\n\nAnother concern is the proposed formulation. As an example, what is the definition of the function f() in Equation 1? To learn f*, what constraints are considered? The same question can be asked for D in Equation 2. Due to the fact that D without any constraint may increase the chance of overfitting. \n\nEmpirical studies are another concern. The proposed method only compares with some classic machine learning approaches, such as Kernel Ridge or Space Gaussian. There are several groups of state-of-the-art techniques that aim at similar types of analysis and should be compared with the proposed approach, e.g., HTFA (Manning 2018), BRSA (Cai 2016), and DRD (Wu 2019). \n\nIn addition to the aforementioned concerns, the proposed approach should clarify some related concepts for multi-subject and multi-resource analysis. It seems that this study uses multi-subject data. However, I could not find anywhere that discusses how the author(s) functionally align brain responses from different subjects. Alternatively, Neuro Vault is a multi-resource repository where data are provided from different machines and environments. I also did not find any information on how the proposed method deals with bach effects to learn a model from multi-site data. You may provide additional information in a preprocessing/preparation section.\n", "strength_and_weaknesses": "Please refer to the review section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5VggSZAhHu4", "reviewer": "Reviewer_eWu3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Cognitive neuroscience aims to reveal the underlying brain regions corresponding to different cognitive processes, and such studies are usually performed by comparing and contrasting the fMRI scans of subjects performing designed cognitive tasks. A major drawback in this field is the high cost of data acquisition, which limits the sample size and hence undermines the statistical power. Meta-analysis over many individual studies, though may partially overcome the sample size issue, usually runs into data unavailability issues where only the locations (in form of standardized coordinates) of peaks of activations are reported in literature while the raw image data are not publicly shared.\n\nOn the other hand of the story are the recent efforts (e.g., NeuroVault, OpenNeuro) of aggregating and curating data from individual studies yielding large-scale public image repositories. In these repositories, however, the annotations and metadata are usually scarce and inconsistent.\n\nThe authors claimed that the rich text data from neuroscientific literature and the large-scale public image data are complementary. Leveraging the advantages from both sides, they proposed a novel meta-analysis method that can decode fMRI statistical maps corresponding to concepts in neuroscientific literature.", "summary_of_the_review": "I like the idea behind the paper, but the current presentation is selling it short, significantly. I do not believe the paper in its current form meets the publication standard at ICLR.", "strength_and_weaknesses": "Strengths\n1. The authors proposed an interesting workflow. If I understand it correctly, the authors are using the following priors: (1) fMRI brain maps is tightly correlated to the underlying cognitive states, and (2) distribution of peak locations in fMRI brain maps is highly correlated to the overall global pattern in fMRI brain maps. With these priors, the authors proposed to train a neural network model to predict the global fMRI pattern from sparse locations of peaks, and with that trained model they synthesized a paired dataset of (cognitive descriptions, global fMRI patterns) using the (cognitive descriptions, sparse locations of peaks) from the neuroscientific literatures. Finally, they trained a separate decoder with the (cognitive descriptions, global fMRI patterns) pairs to decode the former from the latter. The decoder is the ultimate piece of mastery that can describe the underlying cognitive states of an arbitrary fMRI image.\n\nWeaknesses\n1. In my humble opinion, the abstract may need to be better articulated. I was very disoriented and confused by the abstract until I finished reading the introduction section. Specifically, I was confused by (1) what the two complementary sources of data are, (2) why they are complementary, and (3) what is the proposed method trying to accomplish. For (1) and (2), even though the two sources of data are both explicitly mentioned, I did not have enough context to parse their relationship. In contrary, the introduction section has a very natural flow and is painless to follow.\n2. It is a bit misleading that when the authors mentioned the word “image” they did not necessarily mean “image”, but rather the “1x1024 vector representation in the DeFuMo space of the image”. I believe it might be helpful to provide a clarifying statement somewhere early in the paper to avoid potential misunderstanding.\n3. Figure 1 can be further polished. (1) Figure 1A did not clearly show what data format is provided and what are derived. It might be better if that information is illustrated more explicitly. (2) A lot of things are represented as “blue thin rectangle”, yet some of them are modules/operations while others are entities. It may be better to replace some of them. For example, “Reconstruction” and “decoder” can be replaced by sketches of the neural networks, and “Labels” can be replaced by some clipart object, etc. (3) The caption mentioned that in Figure 1B, the brain map is reconstructed using the trained model, so it would be better to show the sketch of model in an appropriate place and indicate that its weights are frozen.\n4. The reconstruction performance (Section 5.1, Figure 2) is quite poor. a correlation of 0.3 does not seem too promising.\n\nMinor Things\n1. Is there some typo in the following sentence? Page 5, 3rd paragraph. “When the percentile corresponds to a To avoid selecting studies with a null tf-idf value, we replace the percentile threshold with a strictly positive tf-idf value criteria.”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject