# Peaks2Image: reconstructing fMRI maps from stereotactic coordinates to enhance cognitive meta-analysis

Raphael Meudec, Jérôme Dockès, Fateme Ghayem, Demian Wassermann, Bertrand Thirion

► To cite this version:

Raphael Meudec, Jérôme Dockès, Fateme Ghayem, Demian Wassermann, Bertrand Thirion. Peaks2Image: reconstructing fMRI maps from stereotactic coordinates to enhance cognitive meta-analysis. 2025. hal-05243856

HAL Id: hal-05243856

https://inria.hal.science/hal-05243856v1

Preprint submitted on 7 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Peaks2Image: reconstructing fMRI maps from stereotactic coordinates to enhance cognitive meta-analysis

Meudec Raphaël¹, Dockès Jérôme¹, Ghayem Fateme¹, Wassermann Demian¹, Thirion Bertrand¹

1 Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France

* bertrand.thirion@inria.fr

## Abstract

To integrate novel findings and reach a consensus on the neural correlates of cognitive functions, human neuroscientists have relied heavily on previous literature to formulate, validate, or refine hypotheses. Since thousands of publications have reported the coordinates of regions identified in brain mapping, the community has largely performed meta-analyses based on these coordinates. A known problem with coordinate-based meta-analyses is that they rely on limited information from each publication, so terms appearing in few publications are not reliably mapped. Recently, the use of public images has stimulated meta-analysis, which has become possible due to the creation of large image repositories. NeuroVault is a prominent example of such a repository. These repositories allow researchers to compare images across publications, leading to image-based meta-analyses. However, image-based meta-analysis faces a major obstacle: the scarcity and inconsistency of image annotation and metadata.

In this paper, we propose *Peaks2Image*, a neural network approach that reconstructs continuous spatial representations of brain activity from peak activation tables. Peaks2Image provides dense brain reconstructions that can be combined with extensive annotations from the neuroscience literature. We use these annotations to train a decoder with textual features as labels. This results in a much broader set of decoded terms than current image-based studies. We validated the decoder using 47,000 annotated images and successfully identified 91 out of 108 terms. This is the most comprehensive decoding experiment to date with a model that was uniquely trained using the neuroscience literature.

## Author summary (should be 150-200 words)

We present the first successful attempt at using neuroscience literature to analyze large-scale functional magnetic resonance imaging (fMRI) databases. First, we trained a model that could reconstruct statistical brain images from activation coordinates. Next, we use the model to generate brain map reconstructions from coordinate tables in neuroscientific publications, creating a substantial collection of images with rich annotations. Using this enriched literature, we train a decoder to decode hundreds of cognitive processes. We applied the decoder to 47,000 NeuroVault images and decoded 91 out of 108 cognitive terms without human labeling supervision. Additionally, we demonstrate that image reconstruction is more effective when performed on task fMRI than on resting-state data.
# 1 Introduction

In recent decades, significant neuroscientific research has focused on mapping human cognition and dividing it into processes within the brain. Applications range from gaining insight into what makes us tick to improving post-surgical recovery. Importantly, human brain mapping remains a consensus-based science: evidence for the involvement of one or more brain regions in a cognitive process is largely determined by the neuroscience community's agreement [1,2].

Different meta-analysis tools have emerged to infer agreement across the neuroscience community [1,3-5]. These tools aggregate knowledge from the current literature and extract agreement on how cognition is mapped onto the brain in a reproducible and quantifiable manner [1]. Additionally, meta-analysis offers a partial solution to the pervasive issue of small sample sizes in neuroscience by providing priors that constrain data analysis and mitigate the risk of false detections. This is often implicitly implemented by masking strategies that build a binary mask of relevant regions from previous literature, thereby reducing the space of hypotheses under investigation.

Interpreting results from neuroimaging studies presents challenges, especially regarding the specificity of the neural correlates of cognitive processes. When a cognitive task is studied in isolation, it is impossible to determine whether the observed brain activations are specific to the mental functions of interest or associated with a broader set of mental processes [3]. One way to identify more specific associations between brain activity and mental function is *decoding*, or inferring the mental processes involved from a brain image of neural activity, especially when taking into account many alternative cognitive domains. To discriminate between a wide variety of cognitive states, a decoding model must identify brain regions that characterize each state rather than regions that are merely consistently activated [6]. For these associations to be specific, it is crucial that diverse cognitive states are decoded *jointly*. Due to the difficulty of formalizing the mental states associated with a brain image, high-quality labels are lacking for this supervised task. Performing large-scale decoding, sometimes called "open-ended", is therefore challenging. Thus, most decoding efforts only discriminate a limited set of cognitive concepts.

*Automated* meta-analysis methods [7] have been introduced to properly handle the growth of neuroscientific publications in recent years. NeuroSynth is a prominent example [8]. Handling neuroscientific concepts properly has emerged as a challenge for automated Coordinate-Based Meta-Analysis (CBMA), leading to the use of more complex textual features and models. *Neuroquery* [4] broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. *Text2Brain* [9] leveraged language models [10] to capture semantic relationships between terms, enabling the encoding of any query. However, CBMA suffers from the drastic information reduction inherent to peak reporting [11].

Using dense images provides more information for meta-analysis. Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault [12] or OpenNeuro [13], decoding can be achieved across multiple studies [14,15], making it robust with respect to study idiosyncrasies. However, the annotations associated with these images are often of low quality. Strategies that automatically improve labeling are necessary to properly benefit from the scale of the data [16]. The Neural Network on Dictionary (NNoD) [5] framework leverages the Cognitive Atlas [17] to improve the labels for NeuroVault images, enabling the decoding of a large set of concepts. Overall, neuroscience meta-analyses have increasingly relied on data-driven approaches to learn more efficiently across large sets of data [18].

There have been few attempts at generative approaches to increase the amount of data [19], to generate or reconstruct fMRI images [20,21] or apply self-supervised methods to neuroimaging signals [22]. However, there is still no systematic solution for
reconstructing statistical brain maps from peaks, likely due to the daunting dimensionality of the data. The only approach we know of is the *Peaks2Maps* method, presented in [23]. Notably, we have not identified any attempts to integrate the rich textual content of neuroscientific publications with the dense spatial data of brain images.

This study presents *Peaks2Image*, a method for reconstructing dense brain images from peak coordinates contained in Coordinate-Based Meta-Analysis (CBMA) databases <fnref n="1" />. *Peaks2Image* builds an enriched version of CBMA databases, enabling the decoding of dozens of cognitive terms from Image-Based Meta-Analysis (IBMA) databases without label supervision. Through this enriched database, we also created the first coordinate-based functional brain atlas, called the *Peaks2Image atlas*.

In this regard, we used peak coordinates, also known as stereotactic coordinates, which summarize the results of group studies in human neuroscience. We mapped these coordinates to dense activation images. Specifically, we used an unlabeled dataset of brain images to extract peaks and train *Peaks2Image* to infer images from which these peaks could be extracted. This process amounts to rebuilding full-brain activity landscapes from the maxima of these maps.

To evaluate Peaks2Image's performance, we decoded cognitive terms from the largest available IBMA-compatible repository, NeuroVault [12]. For each study, we extracted the reported cognitive processes, called labels or terms, by leveraging the term-frequency inverse document frequency of the study's text. Validation is not trivial. It should provide quantitative information and predictive validity, i.e., it should provide testable information on a set of fresh observations. For this, we turn to decoding [24, 25]. We used the decoding architecture from Neural Networks on Dictionaries (NNoD) [5] trained on a neuroscience literature corpus. We evaluated the decoding performance against 108 terms used as NeuroVault annotations.

This approach leverages the rich descriptions of mental processes found in the literature and the high-quality neural activity data found in full-brain statistical maps. Our experiments demonstrate that Peaks2Image successfully decoded 91 out of 108 cognitive terms from 47,000 brain images from the NeuroVault database. These results suggest that Peaks2Image can decode a large set of cognitive terms using the CBMA and IBMA databases without any labeling supervision.

## 2 Methods

This section introduces *Peaks2Image*, our contribution to the field. Peaks2Image is a model that reconstructs statistical images from activation peaks. Our goal is to transform the sparse information from neuroscientific literature into dense spatial images. Although this task is typically performed by a Kernel Density Estimation (KDE) in the framework of CBMA [4, 7, 9], we adopt a distinct approach by framing it as a learning problem. The learned model can build a dense spatial representation from any set of activation peaks, particularly those reported in the neuroscientific literature. We demonstrate the relevance of augmenting literature reports with such reconstructions by decoding cognitive processes without any labeled supervision.

First, we introduce the KDE representation in section 2.1. Then, we describe the reconstruction task (section 2.2) and the *Peaks2Image* model (section 2.3). In section 2.4, we describe the decoding experiments from the neuroscience literature used to study the decoding task. Finally, section 2.5 presents the variants of the Peaks2Image

<footnote n="1">¹It should be noted that positions reported in tables may also represent deactivations that reach a certain statistical significance. As is customary in CBMAs, we assume that these cases are in the minority and consistently refer to *peak* positions.</footnote>
model that we investigated. The Supplementary Material provides a detailed description of the data collection from the neuroscience literature.

In the following formulae, we represent scalars with lowercase characters, vectors as bold lowercase characters and matrices with bold uppercase characters. For $m \in \mathbb{N}$, $[m]$ will denote the $\{1, \dots, m\}$ set.

## 2.1 Kernel Density Estimation

The typical procedure for peak extraction from task fMRI involves thresholding brain maps at a given level, e.g. $p < 0.05$, FWER- or FDR-corrected, or $p < .001$ uncorrected, and retaining only the large connected components. Then, local maxima are reported that are, for example, 8 or 12mm apart among these supra-threshold regions. Thus, an image containing hundreds of thousands of voxels is reduced to a few coordinates — fewer than 100 for most studies. Recent meta-analysis models [4, 7, 9] often represent these few data points with a Kernel Density Estimation (KDE).

We compare the *Peaks2Image* model with recent KDE-based approaches. We use the *NeuroQuery* procedure to generate KDE representations from peaks. From a given set of peaks, we create a representation by placing a Gaussian kernel $\kappa_{\sigma}$ at each peak position with a full-width at half-maximum $\sigma$. Following previous work [4], we set $\sigma = 9mm$. Formally, we consider the set of peaks $\mathbf{A}(\mathbf{y}) = \{\mathbf{a}_1, \dots, \mathbf{a}_{l(\mathbf{y})}\}$ extracted from image $\mathbf{y}$ where $\forall i \in [l(\mathbf{y})], \mathbf{a}_i \in \mathbb{R}^3$. Given a fixed sampling grid $\mathbf{G} = (\mathbf{g}_1, \dots, \mathbf{g}_m)$ representing the voxels, where $\forall j \in [m], \mathbf{g}_j \in \mathbb{R}^3$, and the above-defined Gaussian kernel $\kappa_{\sigma}$, we define the KDE representation $\mathbf{x}_{\mathbf{y}}$ that samples $\mathbf{A}(\mathbf{y})$ on the grid $\mathbf{G}$:

$$ \mathbf{x}_{\mathbf{y}} = (x_1, \dots, x_m) \text{ where } \forall j \in [m] : x_j = \sum_{i=1}^{l(\mathbf{y})} \kappa_{\sigma}(\mathbf{a}_i, \mathbf{g}_j) \quad (1) $$

This heuristic may not fully capture the co-occurrence of specific active locations, which is believed to be a hallmark of cognitive functions. The motivation for learning more sophisticated image reconstructions is to capture the information carried by the co-occurrence of locations and to obtain more informative representations of each publication.

## 2.2 Reconstructing brain activation images

The few data points available in stereotactic tables are derived from intricate activation patterns. Our goal is to reconstruct the statistical images that produced those data points. This reconstruction problem is an inverse problem involving the set of peaks and the image from which they were extracted. From a given set of $l$ activation peaks $\mathbf{A}(\mathbf{y}) \in \mathbb{R}^{3 \times l}$ extracted from image $\mathbf{y} \in \mathbb{R}^m$ with $m$ voxels, we learn the transformation $f_{\theta}: \mathbb{R}^{3 \times l} \to \mathbb{R}^m$ by minimizing the $\mathbf{L}_2$ error over the original image:

$$ f_{\theta}^{*} = \arg \min_{\theta} \mathbb{E}_{\mathbf{y}} \| f_{\theta}(\mathbf{A}(\mathbf{y})) - \mathbf{y} \|_{2}^{2} \quad (2) $$

Since population-level images represent the average of individual data with variable shapes, high resolution is unnecessary. Therefore, we can perform a drastic reduction in the dimensions of the data, which is desirable given the limited sample size. We use the Dictionary of Functional Modes (DiFuMo) probabilistic atlas [26], which efficiently reduces the dimension of the data from $m = 50K$ voxels to $d = 1024$ components. DiFuMo is a linear decomposition similar to principal components analysis, with additional constraints of non-negativity and sparsity on the atoms of the dictionary. It has been trained on thousands of resting-state and task fMRI scans and can therefore
represent population-level information [22]. Using the DiFuMo atlas instead of the raw voxels helps us maintain a relatively low number of parameters.

Mapping voxels $\mathbf{y} \in \mathbb{R}^m$ to coefficients $\alpha \in \mathbb{R}^d$ along DiFuMo dictionary $\mathbf{D} \in \mathbb{R}^{m \times d}$ is simply done by linear regression, as shown in equation eq. (3). All subsequent steps are performed in the $d$-dimensional DiFuMo space.

$$ \alpha(\mathbf{y}) = \arg \min_{\alpha \in \mathbb{R}^d} \|\mathbf{y} - \mathbf{D}\alpha\|_2^2 \quad (3) $$

Therefore, eq. (2) turns into learning the transformation $f_{\theta}: \mathbb{R}^{3 \times l} \to \mathbb{R}^d$ by minimization of the $L_2$ error over the DiFuMo representation coefficients of the original images:

$$ f_{\theta}^* = \arg \min_{\theta} \mathbb{E}_{\mathbf{y}} \| f_{\theta}(\mathbf{A}(\mathbf{y})) - \alpha(\mathbf{y}) \|_2^2 \quad (4) $$

To solve the minimization problem in eq. (4), we use a training set of statistical maps from NeuroVault, an online repository of fMRI maps. We extract peak coordinates from these maps. We standardize these images to eliminate the influence of maps with extremely high statistical values. Standardization also facilitates model convergence. Details on peak extraction are provided in the appendix. This yields a training set of peaks and corresponding brain images that can be reconstructed.

The reconstructed images are obtained from the predicted DiFuMo coefficients via linear combinations of the dictionary components:

$$ \hat{\mathbf{y}} = \mathbf{D} f_{\theta}^*(\mathbf{A}(\mathbf{y})) . \quad (5) $$

In the rest of the paper, the term brain image refers to its $d = 1024$-dimensional DiFuMo representation coefficients $\alpha$.

## 2.3 Peaks2Image

Peaks2Image uses a Set Transformer architecture [27, 28] to solve eq. (4). The transformation from a given set of peaks $\mathbf{A}(\mathbf{y})$ into the DiFuMo representations $\alpha(\mathbf{y})$ of an image can be decomposed into an encoder $\phi: \mathbb{R}^3 \to \mathbb{R}^h$ applied to each element of the set followed by a decoder combining a pooling function pool and a row-wise feedforward layer $\rho$ such that $\rho \circ \text{pool}: \mathbb{R}^h \to \mathbb{R}^d$, where $h$ is the size of the hidden representations.

$$ \alpha(\mathbf{y}) = \rho(\text{pool}(\{\phi(\mathbf{a}_i), \mathbf{a}_i \in \mathbf{A}(\mathbf{y})\})) \quad (6) $$

There are several reasons to use a set transformer architecture for image reconstruction based on peak location occurrences. The pooling function's permutation-invariant property is highly relevant for peak locations because publications often do not report the statistical values associated with activation peaks. Therefore, no particular ordering of peaks can be inferred from the activation tables. Additionally, using sets of peaks as input eliminates the need for an intermediate representation. Thus, the Set Transformer model replaces the standard reconstruction obtained by applying a Gaussian kernel at peak locations, as described in [4, 7, 9]. It also captures co-activation patterns induced by functional networks across the brain. Therefore, we compare Peaks2Image to KDE representations and to Peaks2Maps [23], a Pix2Pix [29] architecture trained on NeuroVault statistical images.

In practice, the Set Transformer accepts a fixed-size input. Therefore, we set the maximum number of peaks to be selected at 150. Since only a few publications report more than 150 peaks, we randomly trim the table for those publications. For publications reporting fewer than 150 coordinates, we pad the set with zeros to obtain
<figure><img src="image_2.png" /><figcaption>Fig 1. Reconstructing brain activation maps from activation peak tables reported in the literature. (A) From a set of statistical images, we extract the peaks above a statistical threshold. We use a Set Transformer model to reconstruct the DiFuMo coefficients of the image after normalization. (B) Once the reconstruction model has been trained, it is applied to the coordinates from the peak activation tables in 19K scraped articles. This yields a brain map for each publication, which supports further image-based computations such as decoding. We extract the Term Frequency - Inverse Document Frequency (TF-IDF) values for each word from the article's text and infer labels using a percentile-based threshold. (C) We then train an image decoder based solely on the literature. For validation, we test the decoder on the NeuroVault repository. We decode a broad set of cognitive processes benefiting from the large vocabulary without any label supervision.</figcaption></figure>

the expected input size. We also pass a mask to the model to distinguish relevant peaks from padded ones. This mask ensures that the padded peaks are not considered in the pooling function.

## 2.4 Outputs: A functional brain atlas and a decoder trained using all available literature

Peaks2Image can reconstruct brain images from any set of peak coordinates. We use it on neuroscience publications to create dense brain maps with rich textual annotations, which reduces the technical gap between CBMA and IBMA. Using the NeuroQuery [4] dataset and Pubget [30], we gathered 19K neuroscience publications with stereotactic coordinates from PubMed Central. The tool parses the activation tables of these publications, totaling 400K peaks. We detail the data collection process further in the supplementary material. Note that the tables from each article were combined for reconstruction (meaning that the coordinates reported in the different tables were pulled together), as there is no simple way to leverage the association of the tables with the text. These maps can then be used to produce statistical summaries of the existing literature in the form of familiar brain maps. Therefore, we need to associate the reconstructions with descriptions of specific cognitive processes. Pubget extracts textual information from publications along the stereotactic coordinates. On top of this raw extraction, Pubget builds features such as the TF-IDF [31] across a large vocabulary of
6308 terms related to the field. The TF-IDF, which is also used in NeuroQuery and Text2Brain [4,8], reflects the relevance of a term to a study. It does so by giving more importance to terms that appear repeatedly in a few publications. We set a threshold at the 95th percentile of the TF-IDF to assign binary labels to publications, typically considering a few hundred positive samples per label. This ensures that the few publications most relevant to each term receive positive labels. The computed percentile may be 0 for some extremely rare terms. In this case, we only consider publications with strictly positive TF-IDF values. The result is a *functional brain atlas* in which each cognitive term receives a topography representing the meta-analytic reconstruction of that term.

To validate the reconstructions and their association with cognitive labels, we propose using a scheme that enforces external validity through out-of-sample generalization: decoding [32]. Thus, we decode statistical images from NeuroVault to identify the cognitive labels associated with each sample. The ability to decode cognitive concepts from the reconstructions ensures their validity and circumvents biases inherent to image reconstruction quality metrics. For example, the confounding effect of smoothness has been studied in [33]. Since we rely on information from the neuroscientific literature, we eliminate the constraint of having only a few labels associated with each brain image. We infer a much larger set of labels from the text of each publication. For the first time in cognitive neuroimaging, we perform decoding without label supervision by using only the peak activation tables in the most comprehensive public literature corpus. Using this dataset, we decode 108 cognitive processes from unseen NeuroVault images in a domain adaptation setting.

The supplementary material provides information on data collection and preparation, as well as training details.

Associating cognitive processes with brain images involves multi-label classification. The goal is to predict a set of labels $\mathbf{L} = \{0, 1\}^C$, where $C$ denotes the total number of classes, from the DiFuMo-space representation of an input image $\alpha(\mathbf{y}) \in \mathbb{R}^d$.

We use a three-layer dense neural network that is trained using only neuroscience publications, without any supervision from NeuroVault samples or their labels. For model evaluation, we are limited by the available NeuroVault annotations. In the following experiments, we predict only $C=108$ terms out of $C=6308$ considered in the TF-IDF vocabulary. However, note that Peaks2Image can be extended to a set of cognitive terms as large as the vocabulary used, with no additional labeling cost.

fig. 1 shows a summary of Peaks2Image. It consists of three parts: learning the reconstruction process from peaks (A), applying the reconstruction to neuroscientific publications (B), and decoding brain images with a model trained from those reconstructions in a domain adaptation setting (C).

## 2.5 Use of alternative latent representations provided by image autoencoders

Dimension reduction is a key component in tackling this reconstruction problem. We use the DiFuMo atlas to reduce the dimensionality of the problem, making it easier to capture co-activation patterns. We implemented other dimensionality reduction schemes for comparison. In particular, we consider the latent space of an autoencoder (AE) trained on NeuroVault data [34]. Using the latent representation of this convolutional AE, we continue to use latent variables that translate to local patterns, similar to DiFuMo loadings. This enables the AE-latent version of our model to compete with the DiFuMo version in terms of both Dice metrics [35] and AUC for the downstream multilabel classification task.

This alternative choice affects the Peaks2Image model in two different ways. First,
the model no longer outputs a DiFuMo representation of an image but rather the latent representation of the autoencoder. The encoder is built from a series of convolutional and max-pooling layers that transform the original brain volume into a compressed representation of 3x4x3 voxels across 512 channels. This increases the dimensionality of the Peaks2Image output from the 1024 DiFuMo components to a 18K latent space. Second, a necessary post-processing step is added to obtain images from the predicted values: applying the AE-based decoder. However, this does not drastically affect the model's computational efficiency, as reconstructing images from DiFuMo components also requires postprocessing to obtain the reconstructed brain image.

We process statistical images from NeuroVault using the procedure described in the original paper [34]. This yields a dataset that differs slightly from the task-version of Peaks2Image. We replace the DiFuMo representations of the images with their latent representations in the autoencoder space. We flatten these latent representations to fit the set transformer for this output. Postprocessing involves reshaping the predictions to the original latent dimensions before applying the decoder to produce the final image prediction.

In all decoding experiments, we use the AUC, as the chance level is normalized to .5, even for highly imbalanced data. We performed 20 runs to obtain confidence intervals for each cognitive task, rotating the training and validation sets from the neuroscientific literature. Evaluation is performed on all NeuroVault statistical images excluding IBC, which is used in the reconstruction training.

# 3 Results

The experimental results are presented as follows: In section 3.1 we quantitatively and qualitatively compare the reconstruction results obtained by Peaks2Image with those obtained by alternative techniques. Section 3.2 presents the results of the most comprehensive large-scale decoding experiment based on the neuroscience literature. In section 3.3 we contextualize these results by comparing them to alternative techniques. We then perform two ablation experiments to better assess the methodological contribution: In section 3.4, we examine the impact of the dimension reduction inherent in Peaks2Image. We also compare the method with an alternative reconstruction based on ICA decompositions of resting state data in section 3.5.

## 3.1 From peaks to dense spatial images

Before applying this reconstruction model to the neuroscientific literature, we measure the quality of the reconstructions produced on NeuroVault images. We use the Dice coefficient [35] (as in Text2Brain [9]) to evaluate the reconstructions produced by Peaks2Image on all NeuroVault statistical images, which represent a testing set of around 47K images. We excluded collection 6618, which was used to train the Peaks2Image model. Given a threshold t, the parametric Dice coefficient $D_t$ measures how well voxels in $\hat{y}$ reconstructed with a value greater than, t coincide with the voxels in the original image y. This eliminates correlation obtained in noise-only regions, and focuses on the most activated regions:

$$D_t = \frac{2 * \sum 1_{y>t} 1_{\hat{y}>t}}{\sum 1_{y>t} + \sum 1_{\hat{y}>t}} \quad (7)$$

Although the model was trained using mean-squared error in DiFuMo space, we find that the Dice coefficient is a more interpretable measure of reconstruction quality. We set the threshold at different percentile values of the original image to validate the
model's ability to recover the most active regions. We present the reconstruction performance of each model in fig. 2.

A. Reconstruction performance among most activated regions

<figure><img src="image_3.png" /></figure>

B. Comparison of most-activated voxels for different models

<figure><img src="image_4.png" /><figcaption>**Fig 2. Reconstruction performance of NeuroVault statistical images:** we reconstruct brain images taken from NeuroVault from their peak coordinates.</figcaption></figure>

**A.** We compute the Dice coefficient at different thresholds *t* (see eq. (7)). Dice coefficients measure how accurately the most activated regions are identified in the reconstruction process. The three variants of Peaks2Image outperform the other representation methods, meaning that Peaks2Image reconstructions better capture the activation patterns of the most active regions. **B.** We threshold the brain volume at different top *t%* thresholds and plot the projection of the thresholded maps on surface for different representations (from top to bottom, the NeuroVault original statistical image, Peaks2Image [task, autoencoder, ICA], Peaks2Maps, KDE). Peaks2Image provides a more truthful reconstruction than the other methods, capturing co-occurences patterns that are not visible with other representations.

As shown in fig. 2, Peaks2Image outperforms all other representations of statistical images. The KDE representation appears to be suboptimal because the resulting dice values are low at all thresholds. Simply positioning activation at the peak location is insufficient to capture the occurrence patterns among the most activated regions. The Gaussian prior in the KDE representation leads to either over-smoothed or over-activated regions.
We also consider Peaks2Maps [23], which uses a UNet architecture to reconstruct a brain image from the KDE representations. Peaks2Image produces regions that are more similar to the original image and misses fewer activations in regions with no reported peaks. This leads to larger Dice metrics. However, Peaks2Maps tends to produce reconstructions with less contrast (see fig. 2), as if the convolutions were blurring highly activated voxels. Using DiFuMo representations instead of raw voxels helps accurately produce highly activated regions and capture co-activation patterns.

We use the reconstructions to create a task-specific brain atlas using contrast images obtained for each term in the TF-IDF vocabulary. For a given term, we create a contrast map using a two-sample t-test comparing the reconstructed images of the publications with a positive TF-IDF scores to the rest of the corpus. Unlike NeuroSynth or NeuroQuery contrast maps, Peaks2Image can reconstruct negative activations. We show the atlas projected onto the surface in fig. 6. We also provide a synthetic representation of the set of components as a functional brain atlas in figure 3. To simplify, we retain the upper percentile of each activation map and cluster the components according to their spatial similarity using a hierarchical clustering algorithm with average linkage. Consequently, the clustering algorithm groups contrasts with a similar topography and plots them together. Interestingly, this clustering leads to the following chains of concepts: emotion, value, self representation, inhibition, language, reasoning, executive function, mathemtics, visuo-spatial functions, motor, vision, audition. This is compatible with the gradient view that emphasizes the difference between primary and perpetual systems [36], and more generally, the distinction between primary and associative, as well as higher-level, functions. Another observation is that not all brain regions are covered equally by the literature, either because they are difficult to map with functional neuroimaging (e.g., anterior temporal regions) or because their function is difficult to capture.

## 3.2 Brain decoding from the neuroscientific literature

We evaluate the relevance of reconstructions by using a model trained only on reconstructed neuroscientific publications to identify cognitive labels associated with NeuroVault images. This model has never encountered actual brain images; it has only seen texts from the literature. In this domain adaptation setting, we do not rely on NeuroVault annotations for training, which lifts the constraint of annotation availability for each brain image.

Identifying cognitive labels associated with brain images is a multi-label classification task that we evaluate by computing the ROC AUC [35] over each label. We evaluate the decoding of all NeuroVault images not used to train the reconstruction model. Similarly to NNoD [5], we apply label enrichment to the original image labels. This process uses regular expressions to map different terminologies for the same concept, as well as a hierarchy-based inference of labels based on the Cognitive Atlas ontology. For a given term, we also assign parent terms from the ontology. These procedures are described in detail in the supplementary material of the NNOD paper<fnref n="2" />.

We decoded 91 cognitive terms out of 108 among 47K NeuroVault images, without any NeuroVault label supervision. We summarize the decoding performance by aggregating over Cognitive Atlas categories in fig. 4. The affiliation of each cognitive term to Cognitive Atlas categories is listed in the Supplementary Information, and the per-term AUC values are given in table 2 in the Supplementary Information. Peaks2Image performs well across all main concept families of the Cognitive Atlas.

The reconstruction process plays a significant role in decoding performance. Thus,

<footnote n="2">²https://static-content.springer.com/esmart:3A10.1038/2Fs41598-022-10710-1/MediaObjects/41598_2022_10710_MOESM1_ESM.pdf</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig 3.** A summary functional brain atlas derived from the literature. (Left) The maps of the atlas are plotted after being thresholded to the upper 99th percentile. The color scheme is adjusted so that maps with similar content are the same color. (Right) Clustering of concepts according to the similarity of the reconstructed map. This representation is useful for summarizing the main consensus of the literature on brain topography. It also outlines cognitive concepts that activate similar brain areas. It also shows that some brain regions have not yet been assigned a clear functional role in the literature.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**Fig 4. Decoding performance on NeuroVault:** We evaluate the relevance of the reconstructions by identifying the cognitive labels associated with the NeuroVault images. Peaks2Image provides reconstructions for the neuroscientific literature, which we use to train a decoder by associating labels with TF-IDF representations of the text. The resulting decoder performs better than chance on 91 out of 108 cognitive processes over 47K images (AUC > 0.5 over at least 95% of the runs). Evaluation is performed on all NeuroVault except collection 6618. We excluded terms with fewer than 20 positive examples to ensure reliable results</figcaption></figure>

we compared the Peaks2Image reconstructions and the KDE representations used in the literature, as well as the Peaks2Maps, by assessing their performance in the decoding task. Peaks2Image outperforms all of these models in the decoding task (see fig. 5). We note that, although Peaks2Maps relies on a learning process for reconstruction, it performs worse than the KDE for decoding. We believe this is due to the lack of specificity in the reconstructed brain images. Peaks2Maps poorly reconstructs high-intensity voxels (see fig. 2), which probably inhibits the decoder from capturing the effect of specific voxels on cognitive processes.

## 3.3 Comparison to CBMA methods

We evaluate the strength of our reconstruction process by directly comparing it to coordinate-based methods, such as NeuroSynth and NeuroQuery.

Using Pubget, we fit the NeuroSynth and NeuroQuery models on the same corpus as Peaks2Image. We compare the Pubget version of NeuroSynth to the original NeuroSynth<fnref n="3" />. In the appendix, we demonstrate that both methods achieve comparable decoding results for terms from the original vocabulary. Using NeuroSynth on the same corpus as Peaks2Image allows us to compare the full vocabulary, not just the cognitive terms handled by the standard interface. As with Peaks2Image, we use the decoding capabilities of the NeuroSynth model to predict terms shared with the NeuroVault vocabulary. We use the chi-squared test statistics from NeuroSynth to associate labels with each statistical image, leveraging the correlation.

Since Pubget is developed by the NeuroQuery authors, we assume equivalence between the initial and fitted models. As with NeuroSynth, we generate a statistical

<footnote n="3">³implementation from https://www.github.com/neurosynth/neurosynth</footnote>
<figure><img src="image_7.png" /><figcaption>**Fig 5. Relative AUC performance across tasks compared to mean performance of all reconstruction methods:** We decode NeuroVault terms using each representation method as the training set. Then, we compute the AUC gain for each word relative to the mean performance. Then, we plot the aggregation of those gains across Cognitive Atlas categories. Peaks2Image largely outperforms other approaches in most categories.</figcaption></figure>

image for each term using the NeuroQuery model. Then, we compute the correlations of NeuroVault images with the reference maps.

As seen in fig. 5, Peaks2Image outperforms both NeuroSynth and NeuroQuery in most Cognitive Atlas categories. The NeuroSynth model competes with Peaks2Image with respect to well-established terms such as 'language' or 'memory', but it performs poorly with more specific terms.

## 3.4 Dimension reduction benefits to decoding accuracy

Dimensionality reduction is a key component in tackling this reconstruction problem. Switching from DiFuMo loadings to autoencoder representations yields slightly different samples. We noted that it tends to reconstruct sharper patterns, but capturing co-activation patterns poses a greater challenge. We attribute this difficulty to the decoder's depth; it comprises only four deconvolution layers. This probably limits its ability to reconstruct co-activation patterns distributed across the brain. Using the AE-latent version of our model has little impact on performance in the reconstruction (see fig. 2) and decoding tasks. We find that the DiFuMo loadings are a simpler, more interpretable way to reconstruct images.
## 3.5 Training Peaks2Image with resting-state instead of task-fMRI data

Task-fMRI data are expensive to collect and annotate, whereas a large amount of resting-state data is easily accessible. We investigate using the abundant resting-state data to learn the reconstruction process. We evaluated the ICA-trained model on the same NeuroVault dataset [12], which required an additional adaptation to account for the dataset shift. As shown in fig. 7 in the Supplementary Materials, NeuroVault maps yield a more accurate reconstruction than resting-state ICA maps.

## 4 Discussion

This paper tests the hypothesis that generative AI approaches can address the scarcity and inconsistency of annotations and metadata in brain image databases. For this, we introduce Peaks2Image, a model that reconstructs brain images from peak activity coordinates. To this end, we introduce Peaks2Image, a model that reconstructs brain images from peak activity coordinates. Peaks2Image bridges the gap between neuroscientific literature and brain image databases by using the available coordinates reported in publications to reconstruct dense images. Peaks2Image builds a database of dense spatial images based on the content of publications, improving the spatial information in coordinate-based analyses and the textual content in image-based databases. This database is the foundation for the first coordinate-based atlas of brain images derived from neuroscientific literature. Contrast maps can capture negative effects and probe the independence of certain brain regions in relation to specific cognitive tasks.

Peaks2Image builds dense spatial representations for neuroscientific publications. It improves upon the KDE that has been used in all CBMA so far, as well as in NeuroSynth and NeuroQuery. As shown in fig. 2, Peaks2Image reconstructions are more similar to brain images than KDE representations and better represent the most activated voxel patterns. Additionally, we measure the suboptimality of the KDE, which struggles to pinpoint the activation patterns commonly found in functional brain imaging. Our model, associated with TF-IDF features extracted from the publications, enables a more accurate and richer representation of each publication, further improving the description of each brain image. As shown in fig. 5 Peaks2Image outperforms NeuroSynth and NeuroQuery across most Cognitive Atlas categories, while expanding the number of cognitive processes that can be decoded. One might wonder whether poorly decoded terms would have well-decoded hypernyms. This appeared to be the case in some instances, such as when "mental arithmetic" received an AUC of .45 and "arithmetic processing" received an AUC of .79. However, this was not systematically the case. Compared with image-based models such as NNoD [5], it enables the decoding of a broader range of cognitive processes by leveraging the textual information of neuroscientific publications and reconstructions, all while maintaining a good decoding performance. Most importantly, the decoding performance is reached *without additional labeling effort* on activation images, avoiding a time-consuming and error-prone task. This represents the literature's first successful automatic decoding of 91 terms, among 108 that could be tested. This limited evaluation reflects the scope of our test set. Although Peaks2Image can be extended to any term in the vocabulary, our evaluation is limited by the availability of labels on NeuroVault images. As a side note, the vast majority of maps used from NeuroVault are single-subject maps (such maps represent 88% of NeuroVault content). Having a larger proportion of group-level maps may make our test set less idiosyncratic to individual configurations and increase accuracy.

We use Peaks2Image to train on task-fMRI brain images, which are costly to collect
and annotate. We investigate the impact of training on abundant resting-state data. Although resting-state data lacks cognitive labels, it displays a rich repertoire of topographical maps that we extract through Independent Component Analysis (ICA) [37]. Our model can be trained using any set of peaks and their corresponding ground images. We extract components from randomized ICA runs on HCP resting-state data. We obtained a model for statistical image reconstruction that nearly matched the performance of the task-supervised version in terms of Dice score. However, reconstructions from resting-state data result in lower decoding capabilities. Even though they reach good overall correlations. This suggests that brain maps obtained through the ICA components still lack the specific patterns necessary to properly distinguish cognitive processes, as discussed in [38, 39]. We attempted to address this issue by randomizing the ICA runs to obtain more varied and specific components. Unfortunately, this did not help capture useful, specific patterns for the decoding task.

Image compression is a key architectural element that allows Peaks2Image to capture patterns more effectively for reconstruction. We demonstrate that DiFuMo exhibits similar behavior to other compression strategies, particularly the latent representation of an autoencoder trained on NeuroVault images [34]. Our findings confirm that this domain-specific dimension reduction is well-suited to properly reconstructing brain images, as we observe similar reconstruction and decoding results. However, DiFuMo has several advantages over the autoencoder. For example, it captures co-activation patterns more effectively, as shown in fig. 2. The DiFuMo atlas representation is built around functional components best suited for characterizing co-occurring activation patterns. Additionally, the DiFuMo atlas is more easily interpretable than the autoencoder latent space, which uses several convolutional layers for image reconstruction.

An important limitation of the presented framework is its crude representation of texts using TF-IDF to represent the semantics of cognitive components. First, it hinders comparisons by limiting them to exact matches between vocabulary and tags from NeuroVault. Although Peaks2Image could be extended to include any word in the neuroscientific literature, the evaluation was limited to a small set of concepts due to the lack of extensively annotated brain images. Exploring more powerful language models as in Text2Brain [9] is a future direction that could better exploit the continuous brain images generated for the neuroscientific publications. This approach should facilitate understanding the semantics of queries rather than reducing the description of an image to just a few terms. However, poor-quality NeuroVault annotations would still limit our ability to evaluate these models. Second, it does not include the context in which each word is used. Applying binary labeling based on TF-IDF values is most likely a crude approximation of each word's semantic content. We believe that more advanced language models [40, 41] could better infer main topics of publications.

Peaks2Image paves the way for the improved generation of large sets of images for neuroscientific publications. It demonstrates how one can overcome the limitations entailed by poor image annotations by relying on a domain adaptation approach. In particular, neuroscientific publications could be a challenging application for generative approaches [42] and recent text-to-image representation learning methods, such as CLIP [43].
# 5 Supporting information

## 5.1 Code and data availability

The trained model is publicly available on Github<fnref n="4" />, among the code to reproduce the experiments. We also include pre-computed DiFuMo representations of NeuroVault images and fetched publications from pubget. The code is released under the MIT license.

## 5.2 Neuroscientific literature

### 5.2.1 Data collection

We leverage Pubget<fnref n="5" /> to fetch the neuroscientific literature, exploiting the Pubmed Central API. We fetch publications from an extensive list of journals<fnref n="6" />, resulting in 400K neuroscientific publications. While this seems like a large number, only a small fraction of those publications contain stereotactic coordinates. We identify those and extract a subset of 19K publications from which we obtain both the text and the activation tables. The tool helps us parse the activation tables of those publications, totalling 400K peaks. It also transforms each study text into TF-IDF features across a vocabulary of 6308 terms.

### 5.2.2 NeuroSynth and NeuroQuery models

Pubget also facilitates the preprocessing of the publications by providing directly the TF-IDF representation of the text. It facilitates the fit of NeuroSynth and NeuroQuery on the set of downloaded publications.

We use those versions to act as our baselines. We make sure that this newly fitted NeuroSynth version matches the original performance<fnref n="7" /> on shared terms.

NeuroSynth computes statistical maps from publications by performing a Chi-squared test on the co-occurrence of terms and locations reported across publications. They perform the decoding through correlation of a given image to those Chi-squared maps. We observe similar decoding results between our fitted version through Pubget and the original NeuroSynth one, with a slight improvement of the updated version (AUC of 0.663 vs 0.655 for NeuroSynth). This difference in performance can be explained through two main factors. First, the set of publications used in both versions are different. The original NeuroSynth version uses a set of 11K publications, while our updated version uses over 19K publications. Hence, the Chi-squared maps obtained for each term differ slightly. Second, Pubget uses the full body of the publication, while NeuroSynth only uses the abstract. This expands the vocabulary reach of this updated NeuroSynth version, and allows decoding for a broader set of terms.

## 5.3 NeuroVault

### 5.3.1 Data collection

We assess the quality of the informations captured from the literature through the NeuroVault database. We exclude a subset of images according to a set of rules such as having an incoherent range of values (indicating that an image does not contain Z or T statistics), duplicated images or images with missing metadata to keep 55K brain maps

<footnote n="4">⁴https://www.github.com/raphaelmeudec/peaks2image</footnote>

<footnote n="5">⁵https://www.github.com/neuroquery/pubget - v0.0.8</footnote>

<footnote n="6">⁶https://github.com/neuroquery/pubget/blob/main/docs/example_queries/journal_list_fmri_vbm.txt</footnote>

<footnote n="7">⁷Implementation from https://www.github.com/neurosynth/neurosynth</footnote>
for our task. We split those maps into two different sets in a collection-wise manner. We take collection 6618, without labels (6K maps), to train the reconstruction task. We consider the remaining 47K brain maps from 2376 different collections to evaluate the performance on the decoding task.

For validation, we restrict the analysis to terms shared between the tags of NeuroVault images described in NNoD [5] and the vocabulary from the TF-IDF. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 20 positive examples out of the 47K images in the test set to prevent any non-significant effects, keeping 108 terms to decode. We list those rare terms below.

**Filtered rare terms:** acuity agency, agreeableness, anxiety, appetite, attentional blink, attentional state, attitude, auditory masking, balance, binocular rivalry, concentration, cognitive effort, cognitive load, color perception, communication, competition, conceptual priming, consciousness, conversation, creative thinking, crossmodal, deductive reasoning, desire, distraction, divergent thinking, efficiency, emotional enhancement, expectancy, externalizing, extinction, facial expression, familiarity, feature detection, feature-based attention, filtering, focus, framing, generalization, goal, guilt, habit, hallucination, happiness, imagination, incidental learning, intelligence, interference, joint attention, knowledge, language comprehension, language production, lemma, lexical processing, lying, mathematical reasoning, meaning, metacognition, meter, mood, motor sequence learning, navigation, negative priming, overt naming, perceptual identification, phonological processing, positive priming, prosody, prospection, prospective memory, prototype, reinstatement, reward learning, rhythm, rule learning, sadness, semantic memory, shame, social norm processing, source memory, strategy, strength, surprise, sustained attention, task difficulty, task switching, thought, tongue response execution, trait anxiety, uncertainty, utility, verbal fluency, visual search, word generation, working memory retrieval

### 5.3.2 Image preprocessing

Images were fetched from NeuroVault early 2023 using the Nilearn library [44]. We resample all images to the standard MNI152 template and apply the MNI152 brain mask. We tame the effect of outliers by rescaling image values to a standard range, thus losing the statistical scale of the effects. We use the DiFuMo atlas in dimension *d* = 1024 available from Nilearn. As the coefficients of the DiFuMo representation can range to large values, we scale them to a constant to facilitate the convergence of the neural network.

### 5.3.3 Peak extraction

We generate matching (peaks, image) pairs from the collection 6618 from NeuroVault, without considering the associated labels. Similarly, we apply masking, resampling and DiFuMo compression without the normalization step. We use Nilearn for the peak extraction, which first separates the images into statistically active clusters and identifies local maxima as peak coordinates for each cluster. As these images represent statistical maps, equivalent to p-values of voxel-level tests, we keep peaks with a p-value lower than 10⁻³, and with a minimal distance between peaks within the same cluster of 8mm. These are standard parameters from peak extraction in brain imaging contexts [45].
### 5.3.4 Cognitive Atlas association

We present in this paper aggregation of metrics over Cognitive Atlas categories. We find this presentation more easily readable than representing each cognitive concept individually. We parse the mapping of terms upon Cognitive Atlas categories from the Cognitive Atlas website<fnref n="8" />. Additionally, for terms without a mapping, we look for categories assigned to its parents. We present in the table below the mapping between the Cognitive Atlas categories and the cognitive processes used in the paper.

### 5.3.5 NeuroSynth and NeuroQuery training

Pubget also facilitates the preprocessing of the publications by providing directly the TF-IDF representation of the text. It facilitates the fit of NeuroSynth and NeuroQuery on the set of downloaded publications.

We use those versions to act as our baselines. We make sure that this newly fitted NeuroSynth version matches the original performance on shared terms. We observe similar decoding results, with a slight improvement of the updated version (AUC of 0.663 vs 0.655 for NeuroSynth).

## 5.4 Peaks2Image training details

### 5.4.1 Reconstruction

The image reconstruction model is implemented using PyTorch [46]. It is a Set Transformer comprising an encoder-decoder architecture applied to each reported peak. The encoding function is composed of 2 induced set attention blocks (ISAB, as defined in the original paper [28]) with 2 heads and 32 inducing points. The decoder applies a pooling by multihead attention with 2 heads and 1 seed followed by row-wise feed forward layer applied on a set attention block as defined in the original paper [28]. The output dimension of the model is the chosen DiFuMo dimension of the images, aka 1024. It is trained for 50 epochs using Adam optimizer and a learning rate of 1e-3. The loss is the mean-squared error (MSE) over the DiFuMo components. We select the best set of weights over the validation set, which is composed of 20% of collection 6618.

### 5.4.2 Decoding

The decoder model is a multi-layer perceptron composed of one hidden layer of 300 units and ReLU activation. It is trained for 200 epochs on the reconstructed neuroscientific publications using the Adam optimizer with a learning rate of 3e-3 on the binary cross-entropy loss. During evaluation, we apply an additional sigmoid operation to the model outputs. We use the Area Under the Receiver Operating Characteristic Curve (ROC AUC) over each label of the testing set to evaluate the decoding performance. Chance level is 0.5. We performed 20 different runs of the experiments with different splits to obtain confidence intervals. AUCs were deemed significant when there 95% confidence intervals was situated above .5.

## 5.5 Learning from resting-state fMRI

Task-fMRI data are expensive to collect, and require neuroscientist expertise to design and evaluate tasks. They are still available in limited samples. For this reason, learning to reconstruct them from resting-state fMRI could be beneficial to increase statistical power.

<footnote n="8">⁸https://www.cognitiveatlas.org/concepts/categories/all</footnote>
<table><caption>**Table 1.** Mapping of the cognitive processes used in NeuroVault onto the Cognitive Atlas categories. The category of each cognitive term is either directly obtained from Cognitive Atlas categorization or inherited through a parent process category.</caption><thead><tr><td>Main category</td><td>Cognitive process</td></tr></thead><tbody><tr><td>Action</td><td>action, effort, left finger response execution, left hand response execution, left toe response execution, motor control, motor planning, response execution, response selection, right finger response execution, right hand response execution, right toe response execution, tongue response execution</td></tr><tr><td>Cognitive-affective processes</td><td>agency, agreeableness, appetite, association, attitude, balance, behavioral inhibition, belief, binocular rivalry, centration, cognitive effort, competition, conceptual priming, consciousness, crossmodal, cueing, domain specificity, efficiency, externalizing, extinction, feature-based attention, filtering, hallucination, integration, interference, interference control, introspection, loss, loss anticipation, memory trace, metacognition, meter, overt naming, phonological comparison, prospection, punishment processing, reinstatement, response bias, reward anticipation, reward learning, reward valuation, salience, sleep, social norm processing, strength, trait anxiety</td></tr><tr><td>Perception</td><td>acuity, action perception, audition, auditory coding, auditory masking, auditory scene, auditory sentence recognition, auditory tone detection, auditory tone discrimination, categorical perception, color perception, detection, discrimination, face perception, facial trustworthiness recognition, feature detection, localization, mental imagery, mental rotation, motion detection, navigation, negative priming, object perception, pattern recognition, perception, perceptual identification, positive priming, rhythm, spatial ability, visual form recognition, visual localization, visual object detection, visual object recognition, visual perception, visual recognition, visual sentence recognition</td></tr><tr><td>Learning and memory</td><td>adaptation, autobiographical memory, body maintenance, concept learning, episodic memory, face maintenance, face recognition, familiarity, forgetting, generalization, habit, incidental learning, instrumental conditioning, instrumental learning, knowledge, learning, memory, memory acquisition, memory retrieval, motor sequence learning, object maintenance, pavlovian conditioning, place maintenance, priming, prospective memory, prototype, reinforcement learning, reward processing, rule learning, source memory, spatial memory, string maintenance, tool maintenance, visual memory</td></tr><tr><td>Reasoning and decision making</td><td>analogical reasoning, animacy decision, arithmetic processing, categorization, concept, creative thinking, decision, decision certainty, decision making, decision under uncertainty, deductive reasoning, divergent thinking, feature comparison, framing, imagination, inference, intelligence, judgment, logical reasoning, loss aversion, mathematical reasoning, mental arithmetic, numerical comparison, problem solving, reasoning, risk, risk processing, risk seeking, schema, social inference, strategy, thought, uncertainty, utility</td></tr><tr><td>Attention</td><td>anticipation, attention, attentional blink, attentional state, distraction, fixation, focus, joint attention, search, spatial attention, spatial working memory, sustained attention, visual attention, visual search, visual working memory</td></tr><tr><td>Emotion</td><td>anxiety, arousal, emotion, emotion perception, emotion recognition, emotion regulation, emotional enhancement, empathy, facial expression, fear, guilt, happiness, mood, negative emotion, pain, pain habituation, sadness, shame, stress, surprise</td></tr><tr><td>Executive cognitive control</td><td>cognitive control, cognitive load, error detection, goal, inhibition, maintenance, manipulation, monitoring, planning, response conflict, response inhibition, selective control, self monitoring, task switching, updating, working memory, working memory retrieval</td></tr><tr><td>Social function</td><td>communication, lying, social cognition, theory of mind</td></tr><tr><td>Language</td><td>conversation, language, language comprehension, language production, lemma, lexical processing, meaning, naming, parsing, phonological processing, prosody, reading, semantic categorization, semantic memory, semantic processing, sentence processing, speech perception, speech processing, story comprehension, verbal fluency, visual word recognition, word generation, word recognition</td></tr><tr><td>Motivation</td><td>desire, expectancy, task difficulty</td></tr></tbody></table>
Components found in ICA usually represent a wide variety of activations. We aim to generate a large number of those components to learn the reconstruction process, avoiding the use of task-fMRI data. We build a dataset of corresponding peaks and images by applying Canonical ICA (CanICA) [47] on resting-state data from the Human Connectome Project (HCP) 900 release [37]. We randomly select sessions from between 10 and 30 subjects of HCP to perform over 2K ICA runs, producing each time between 30 and 60 components through CanICA. By repeating this operation multiple times on different subjects and sessions, we build a set of 21K ICA components. We standardize those components by subtracting the mean and reducing them to the unit variance. Those standardized components range to similar values as NeuroVault Z maps. We extracted peak coordinates from these components at different thresholds in {2., 3., 4.} in Z-scale, to vary the number of reported peaks per image. This improved the ability of the model to produce more nuanced statistical images, and better capture co-activation patterns in the peaks.

Among those components, we observe noisy patterns that result from artifacts during acquisition. We remove those components by applying a filter based on their correlation to statistical images. Each component is selected only if it reaches a correlation of 0.3 with at least one statistical image from collection 6618. This selection discards reconstruction patterns that would be related to imaging artefacts commonly obtained in ICA decompositions.

We treat each component as an individual image in the training set. Similarly to statistical images, we normalize those components and extract the peak positions of activated clusters. This change in the training dataset does not affect the architecture and settings of the Peaks2Image model. The predicted output is still a DiFuMo representation, which can be transformed into a brain image through eq. (5).

## Peaks2Image atlas

We plot in fig. 6 the contrast maps over reconstructions obtained from Peaks2Image for several terms. We compute the effect size for each term by first separating publications where the term occurs from publications where it does not.

## 5.6 Training Peaks2Image with resting-state instead of task-fMRI data

Task-fMRI is expensive to collect and annotate, whereas large amount of resting-state data are easily accessible. We investigate the use of the abundant resting-state data to learn the reconstruction process. We evaluate the ICA-trained model on the same dataset from NeuroVault [12], leading to an additional dataset-shift adaptation for this model.

ICA-based Peaks2Image remains on par in terms of Dice but yields low performance on the decoding task. We report the relative AUC performance of the ICA version in appendix. Although many brain patterns are well-captured, the ICA training base is not enough to capture specific patterns necessary for brain decoding. In particular, we observe performance reduction in some specific categories, such as attention or reasoning-related tasks.

We analyzed whether there were some spatial biases in the quality of ICs-based reconstruction. We used the Yeo 17 atlas to compute the per-network relative MSE. Specific networks suffer more during the reconstruction from ICA components: N1 & N2 (visual), and N6 (attentional). We show those differences in reconstruction in fig. 7. This can be put in perspective with the relative strength of ICA in the decoding of different tasks. We plot in fig. 7 the difference in the resulting AUC, as compared with
<figure><img src="image_8.png" /></figure>
<figure><img src="image_9.png" /><figcaption>**Fig 6.** Contrast maps for different terms from the vocabulary obtained from the reconstruction of the neuroscientific literature. We project the statistical images onto the surface, and apply a thresholding at the 90th percentile.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>**Fig 7. Comparison of ICA-based vs task-based reconstruction:** We investigate the effects of learning the reconstruction from resting-state data instead of task-fMRI images. (A) Relative AUC between task and ICA-based resting-state representations: for each task, we compare the AUC performance of the models trained on task-reconstructions or the ICs-based reconstructions. The resting-state ICs model performs worse, and tasks associated to brain areas with higher reconstruction errors pose a greater challenge. (B) Relative reconstruction error: we show the average AUC difference (bar) and task-specific AUC difference (dot) for ICA vs task-based models. Specific zones suffer more from training on ICA components instead of task data, in particular, the attentional network. This affects the downstream decoding capabilities, where almost all attentional tasks perform worse.</figcaption></figure>

the task model. Overall, the ICA-decoder performs worse, but the attentional tasks show an even larger loss in decoding capabilities.

## AUC values per task

The average AUC per task (across folds) is given in Table 2 for 189 tags that were observed in NeuroVault.
Acknowledgments

The authors gratefully acknowledge Benoit Malézieux, Julia Linhart, Cédric Rommel and Louis Rouillard for their help in the implementation of the Set Transformer blocks.
This work has received funding from ANR through the KARAIB AI chair (ANR-20-CHIA-0025-01).

664

665

666

667

668
<table><caption>Table 2. Average AUC value per concept</caption><thead><tr><th>Cognitive Label</th><th>AUC</th><th>Cognitive Label</th><th>AUC</th><th>Cognitive Label</th><th>AUC</th></tr></thead><tbody><tr><td>action</td><td>0.57</td><td>imagination</td><td>0.68</td><td>reasoning</td><td>0.60</td></tr><tr><td>acuity</td><td>0.88</td><td>incidental learning</td><td>0.95</td><td>reinforcement learning</td><td>0.30</td></tr><tr><td>adaptation</td><td>0.90</td><td>inference</td><td>0.81</td><td>reinstatement</td><td>0.77</td></tr><tr><td>agency</td><td>0.81</td><td>inhibition</td><td>0.79</td><td>response bias</td><td>0.72</td></tr><tr><td>agreeableness</td><td>0.79</td><td>instrumental conditioning</td><td>0.82</td><td>response conflict</td><td>0.67</td></tr><tr><td>analogical reasoning</td><td>0.78</td><td>instrumental learning</td><td>0.77</td><td>response execution</td><td>0.54</td></tr><tr><td>anticipation</td><td>0.69</td><td>intelligence</td><td>0.96</td><td>response inhibition</td><td>0.68</td></tr><tr><td>anxiety</td><td>0.93</td><td>interference</td><td>0.84</td><td>response selection</td><td>0.54</td></tr><tr><td>arithmetic processing</td><td>0.78</td><td>interference control</td><td>0.83</td><td>reward anticipation</td><td>0.60</td></tr><tr><td>arousal</td><td>0.74</td><td>interspection</td><td>0.57</td><td>reward learning</td><td>0.81</td></tr><tr><td>attention</td><td>0.80</td><td>joint attention</td><td>0.95</td><td>reward processing</td><td>0.81</td></tr><tr><td>attentional blink</td><td>0.73</td><td>judgment</td><td>0.60</td><td>reward valuation</td><td>0.85</td></tr><tr><td>attentional state</td><td>0.64</td><td>knowledge</td><td>0.70</td><td>rhythm</td><td>0.61</td></tr><tr><td>attitude</td><td>0.71</td><td>language</td><td>0.80</td><td>risk</td><td>0.74</td></tr><tr><td>audition</td><td>0.83</td><td>language production</td><td>0.64</td><td>risk processing</td><td>0.69</td></tr><tr><td>auditory scene</td><td>0.73</td><td>learning</td><td>0.73</td><td>risk seeking</td><td>0.67</td></tr><tr><td>autobiographical memory</td><td>0.58</td><td>lemma</td><td>0.83</td><td>rule learning</td><td>0.63</td></tr><tr><td>balance</td><td>0.62</td><td>lexical processing</td><td>0.86</td><td>sadness</td><td>0.97</td></tr><tr><td>behavioral inhibition</td><td>0.83</td><td>localization</td><td>1.00</td><td>salience</td><td>0.83</td></tr><tr><td>belief</td><td>0.85</td><td>logical reasoning</td><td>0.74</td><td>schema</td><td>0.76</td></tr><tr><td>binocular rivalry</td><td>0.90</td><td>loss</td><td>0.83</td><td>search</td><td>0.77</td></tr><tr><td>categorical perception</td><td>0.45</td><td>loss anticipation</td><td>0.67</td><td>selective control</td><td>0.70</td></tr><tr><td>cognitive control</td><td>0.79</td><td>loss aversion</td><td>0.62</td><td>self monitoring</td><td>0.87</td></tr><tr><td>cognitive effort</td><td>0.81</td><td>lying</td><td>0.82</td><td>semantic categorization</td><td>0.66</td></tr><tr><td>cognitive load</td><td>0.89</td><td>maintenance</td><td>0.84</td><td>semantic memory</td><td>0.64</td></tr><tr><td>color perception</td><td>0.62</td><td>manipulation</td><td>0.90</td><td>semantic processing</td><td>0.83</td></tr><tr><td>communication</td><td>0.78</td><td>mathematical reasoning</td><td>0.88</td><td>sentence processing</td><td>0.82</td></tr><tr><td>concept</td><td>0.78</td><td>meaning</td><td>0.83</td><td>sleep</td><td>0.87</td></tr><tr><td>conceptual priming</td><td>0.91</td><td>memory</td><td>0.75</td><td>social cognition</td><td>0.84</td></tr><tr><td>creative thinking</td><td>0.78</td><td>memory acquisition</td><td>0.62</td><td>social inference</td><td>0.81</td></tr><tr><td>cueing</td><td>0.36</td><td>memory retrieval</td><td>0.65</td><td>source memory</td><td>0.80</td></tr><tr><td>decision</td><td>0.83</td><td>memory trace</td><td>0.56</td><td>spatial ability</td><td>0.57</td></tr><tr><td>decision making</td><td>0.73</td><td>mental arithmetic</td><td>0.45</td><td>spatial attention</td><td>0.64</td></tr><tr><td>deductive reasoning</td><td>0.78</td><td>mental imagery</td><td>0.86</td><td>spatial memory</td><td>0.72</td></tr><tr><td>detection</td><td>0.94</td><td>mental rotation</td><td>0.47</td><td>spatial working memory</td><td>0.68</td></tr><tr><td>discrimination</td><td>0.62</td><td>monitoring</td><td>0.84</td><td>speech perception</td><td>0.86</td></tr><tr><td>divergent thinking</td><td>0.85</td><td>mood</td><td>0.91</td><td>speech processing</td><td>0.73</td></tr><tr><td>domain specificity</td><td>0.54</td><td>motion detection</td><td>0.27</td><td>story comprehension</td><td>0.91</td></tr><tr><td>efficiency</td><td>0.85</td><td>motor control</td><td>0.84</td><td>strategy</td><td>0.96</td></tr><tr><td>effort</td><td>0.81</td><td>motor planning</td><td>0.91</td><td>strength</td><td>0.85</td></tr><tr><td>emotion perception</td><td>0.32</td><td>motor sequence learning</td><td>0.83</td><td>stress</td><td>0.86</td></tr><tr><td>emotion recognition</td><td>0.34</td><td>naming</td><td>0.71</td><td>surprise</td><td>0.82</td></tr><tr><td>emotion regulation</td><td>0.58</td><td>navigation</td><td>0.88</td><td>sustained attention</td><td>0.65</td></tr><tr><td>emotional enhancement</td><td>0.87</td><td>negative emotion</td><td>0.32</td><td>task difficulty</td><td>0.73</td></tr><tr><td>empathy</td><td>0.86</td><td>negative priming</td><td>0.74</td><td>task switching</td><td>0.62</td></tr><tr><td>episodic memory</td><td>0.73</td><td>numerical comparison</td><td>0.44</td><td>thought</td><td>0.90</td></tr><tr><td>error detection</td><td>0.69</td><td>object perception</td><td>0.26</td><td>trait anxiety</td><td>0.87</td></tr><tr><td>expectancy</td><td>0.92</td><td>overt naming</td><td>0.55</td><td>uncertainty</td><td>0.63</td></tr><tr><td>extinction</td><td>0.94</td><td>pain</td><td>0.75</td><td>updating</td><td>0.60</td></tr><tr><td>face perception</td><td>0.75</td><td> parsing</td><td>1.00</td><td>utility</td><td>0.95</td></tr><tr><td>face recognition</td><td>0.35</td><td>pattern recognition</td><td>0.21</td><td>verbal fluency</td><td>0.86</td></tr><tr><td>facial expression</td><td>0.86</td><td>paylovian conditioning</td><td>0.89</td><td>visual attention</td><td>0.41</td></tr><tr><td>fear</td><td>0.86</td><td>perception</td><td>0.83</td><td>visual localization</td><td>0.63</td></tr><tr><td>feature detection</td><td>0.81</td><td>perceptual identification</td><td>0.92</td><td>visual memory</td><td>0.70</td></tr><tr><td>filtering</td><td>0.89</td><td>phonological processing</td><td>0.78</td><td>visual object recognition</td><td>0.37</td></tr><tr><td>fixation</td><td>0.74</td><td>planning</td><td>0.85</td><td>visual perception</td><td>0.84</td></tr><tr><td>focus</td><td>0.83</td><td>positive priming</td><td>0.74</td><td>visual recognition</td><td>0.55</td></tr><tr><td>forgetting</td><td>0.89</td><td>priming</td><td>0.87</td><td>visual search</td><td>0.66</td></tr><tr><td>framing</td><td>0.81</td><td>problem solving</td><td>0.62</td><td>visual word recognition</td><td>0.74</td></tr><tr><td>goal</td><td>0.81</td><td>prosody</td><td>0.96</td><td>visual working memory</td><td>0.51</td></tr><tr><td>guilt</td><td>0.94</td><td>prospective memory</td><td>0.73</td><td>word generation</td><td>0.62</td></tr><tr><td>habit</td><td>0.86</td><td>prototype</td><td>0.71</td><td>word recognition</td><td>0.76</td></tr><tr><td>hallucination</td><td>0.93</td><td>punishment processing</td><td>0.90</td><td>working memory</td><td>0.77</td></tr></tbody></table>
References

1. Wager TD, Lindquist M, Kaplan L. Meta-analysis of functional neuroimaging data: current and future directions. Social cognitive and affective neuroscience. 2007;2(2):150-158.

2. Poldrack RA, Baker CI, Durnez J, Gorgolewski KJ, Matthews PM, Munafò MR, et al. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature reviews neuroscience. 2017;18(2):115-126.

3. Poldrack RA. Inferring mental states from neuroimaging data: from reverse inference to large-scale decoding. Neuron. 2011;72(5):692-697.

4. Dockès J, Poldrack RA, Primet R, Gözükan H, Yarkoni T, Suchanek F, et al. NeuroQuery, comprehensive meta-analysis of human brain mapping. Elife. 2020;9.

5. Menuet R, Meudec R, Dockès J, Varoquaux G, Thirion B. Comprehensive decoding mental processes from Web repositories of functional brain images. Sci Rep. 2022;12(1):7050.

6. Varoquaux G, Schwartz Y, Poldrack RA, Gauthier B, Bzdok D, Poline JB, et al. Atlases of cognition with large-scale human brain mapping. PLoS computational biology. 2018;14(11):e1006565.

7. Laird AR, Lancaster JJ, Fox PT. Brainmap. Neuroinformatics. 2005;3(1):65-77.

8. Yarkoni T, Poldrack RA, Nichols TE, Van Essen DC, Wager TD. Large-scale automated synthesis of human functional neuroimaging data. Nature methods. 2011;8(8):665-670.

9. Ngo GH, Nguyen M, Chen NF, Sabuncu MR. Text2Brain: Synthesis of Brain Activation Maps from Free-Form Text Query. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2021. p. 605-614.

10. Beltagy I, Lo K, Cohan A. SciBERT: A pretrained language model for scientific text. arXiv preprint arXiv:190310676. 2019.

11. Salimi-Khorshidi G, Smith SM, Keltner JR, Wager TD, Nichols TE. Meta-analysis of neuroimaging data: a comparison of image-based and coordinate-based pooling of studies. Neuroimage. 2009;45(3):810-823.

12. Gorgolewski KJ, Varoquaux G, Rivera G, Schwarz Y, Ghosh SS, Maumet C, et al. NeuroVault. org: a web-based repository for collecting and sharing unthresholded statistical maps of the human brain. Frontiers in neuroinformatics. 2015;9:8.

13. Gorgolewski K, Esteban O, Schaefer G, Wandell B, Poldrack R. OpenNeuro—a free online platform for sharing and analysis of neuroimaging data. Organization for human brain mapping Vancouver, Canada. 2017;1677(2).

14. Mensch A, Mairal J, Bzdok D, Thirion B, Varoquaux G. Learning neural representations of human cognition across many fMRI studies. Advances in neural information processing systems. 2017;30.

15. Walters J, King M, Bissett PG, Ivry RB, Diedrichsen J, Poldrack RA. Predicting brain activation maps for arbitrary tasks with cognitive encoding models. NeuroImage. 2022; p. 119610.
16. Poldrack RA, Yarkoni T. From brain maps to cognitive ontologies: informatics and the search for mental structure. Annual review of psychology. 2016;67:587.

17. Poldrack RA, Kittur A, Kalar D, Miller E, Seppa C, Gil Y, et al. The cognitive atlas: toward a knowledge foundation for cognitive neuroscience. Frontiers in neuroinformatics. 2011;5:17.

18. Beam E, Potts C, Poldrack RA, Etkin A. A data-driven framework for mapping domains of human neurobiology. Nature neuroscience. 2021;24(12):1733-1744.

19. Zhuang P, Schwing AG, Koyejo O. FMRI Data Augmentation Via Synthesis. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019); 2019. p. 1783-1787.

20. Manning JR, Zhu X, Willke TL, Ranganath R, Stachenfeld K, Hasson U, et al. A probabilistic approach to discovering dynamic full-brain functional connectivity patterns. NeuroImage. 2018;180:243-252.
doi:https://doi.org/10.1016/j.neuroimage.2018.01.071.

21. Cai M, Schuck NW, Pillow JW, Niv Y. A Bayesian method for reducing bias in neural representational similarity analysis. Advances in Neural Information Processing Systems. 2016;29.

22. Thomas AW, Ré C, Poldrack RA. Self-supervised learning of brain dynamics from broad neuroimaging data. arXiv preprint arXiv:220611417. 2022;

23. Gorgolewski KJ, Yarkoni T, Poldrack RA. peaks2maps: reconstructing unthresholded statistical maps from peak coordinates using deep neural networks. F1000Research. 2019;8.

24. Cox DD, Savoy RL. Functional magnetic resonance imaging (fMRI) "brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage. 2003;19(2):261-270.

25. Varoquaux G, Poldrack RA. Predictive models avoid excessive reductionism in cognitive neuroimaging. Current Opinion in Neurobiology. 2019;55:1-6.
doi:10.1016/j.conb.2018.11.002.

26. Dadi K, Varoquaux G, Machlouzarides-Shalit A, Gorgolewski KJ, Wassermann D, Thirion B, et al. Fine-grain atlases of functional modes for fMRI analysis. NeuroImage. 2020;221:117126.

27. Zaheer M, Kottur S, Ravanbakhsh S, Poczos B, Salakhutdinov RR, Smola AJ. Deep sets. Advances in neural information processing systems. 2017;30.

28. Lee J, Lee Y, Kim J, Kosiorek AR, Choi S, Teh YW. Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks. arXiv:181000825 [cs, stat]. 2019;

29. Isola P, Zhu J, Zhou T, Efros AA. Image-to-Image Translation with Conditional Adversarial Networks. CoRR. 2016;abs/1611.07004.

30. Dockès J, Oudyk K, Torabi M, de la Vega AI, Poline JB. Mining the neuroimaging literature. bioRxiv. 2023;doi:10.1101/2023.10.30.564783.

31. Salton G, Buckley C. Term-weighting approaches in automatic text retrieval. Information processing & management. 1988;24(5):513-523.
32. Varoquaux G, Poldrack RA. Predictive models avoid excessive reductionism in cognitive neuroimaging. Curr Opin Neurobiol. 2019;55:1-6. 753 754

33. Triantafyllou C, Hoge RD, Wald LL. Effect of spatial smoothing on physiological noise in high-resolution fMRI. NeuroImage. 2006;32(2):551-557. doi:10.1016/j.neuroimage.2006.04.182. 755 756 757

34. Germani E, Fromont E, Maumet C. On the benefits of self-taught learning for brain decoding. GigaScience. 2023;12:giad029. doi:10.1093/gigascience/giad029. 758 759

35. Dice LR. Measures of the Amount of Ecologic Association Between Species. Ecology. 1945;26(3):297-302. doi:10.2307/1932409. 760 761

36. Margulies DS, Ghosh SS, Goulas A, Falkiewicz M, Huntenburg JM, Langs G, et al. Situating the default-mode network along a principal gradient of macroscale cortical organization. Proceedings of the National Academy of Sciences. 2016;113(44):12574-12579. doi:10.1073/pnas.1608282113. 762 763 764 765

37. Smith SM, Beckmann CF, Andersson J, Auerbach EJ, Bijsterbosch J, Douaud G, et al. Resting-state fMRI in the Human Connectome Project. Neuroimage. 2013;80:144-168. 766 767 768

38. Tuominen J, Specht K, Vaisvilaite L, Zeidman P. An information-theoretic analysis of resting-state versus task fMRI. Network Neuroscience. 2023;7(2):769-786. doi:10.1162/netn00302.

39. Demirtaş M, Ponce-Alvarez A, Gilson M, Hagmann P, Mantini D, Betti V, et al. Distinct modes of functional connectivity induced by movie-watching. NeuroImage. 2019;184:335-348. doi:10.1016/j.neuroimage.2018.09.042. 769 770 771

40. Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models; 2023. 772 773

41. Wu C, Lin W, Zhang X, Zhang Y, Wang Y, Xie W. PMC-LLaMA: Towards Building Open-source Language Models for Medicine; 2023. 774 775

42. Pinaya WHL, Tudosiu PD, Dafflon J, da Costa PF, Fernandez V, Nachev P, et al. Brain Imaging Generation with Latent Diffusion Models; 2022. 776 777

43. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. PMLR; 2021. p. 8748-8763. 778 779 780

44. Abraham A, Pedregosa F, Eickenberg M, Gervais P, Mueller A, Kossaifi J, et al. Machine learning for neuroimaging with scikit-learn. Frontiers in neuroinformatics. 2014;8:14. 781 782 783

45. Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013;14(5):365. 784 785 786

46. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, et al. Automatic differentiation in pytorch. 2017;.. 787 788

47. Varoquaux G, Sadaghiani S, Pinel P, Kleinschmidt A, Poline JB, Thirion B. A group model for stable multi-subject ICA on fMRI datasets. NeuroImage. 2010;51(1):288-299. doi:10.1016/j.neuroimage.2010.02.010. 789 790 791