Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a neural-network approach to reconstruct continuous spatial representations of brain activity from peak activation tables. Using reconstructions of studies published in the neuroscientific literature, the authors train a decoder leading to a much broader set of decoded terms than current image-based studies.
strength_and_weaknesses: Strength:
1.This paper is well-written and easy to read. The definition of tasks and problems are very clear.
2.For a decoding task, the idea that fully utilize the rich descriptions of the studied cognitive tasks in scientific report is novel.
Weaknesses
1.My major concern is the necessity of image reconstruction from peaks. As illustrated in the paper, the authors train a reconstruction network to convert the peak coordinates into images. In my opinion, this ill-posed problem is too hard to be reasonable. Reconstructing the fMRI from only a few peak coordinates (less than 100 according to the paper) seems impossible to recover the details of images. Thus, the reconstruction is unnecessary. If the peak coordinates contain enough information for the following decoding tasks, reconstruction would introduce inevitable noise. If the peak coordinates are not enough, reconstruction still can’t help the decoding since extra information comes from the ‘guessing’ of the trained network. Can the authors give more explanation about this?
2.Also, the experiments do not verify that image reconstruction can help in improving the decoding performance. The ablation study should compare with model using peak coordinates as input. So, I am wondering about the performance of model that uses peak coordinates as input.
3.Experimental results are all shown in figures, which is good for analysis. However, giving some extra tables with exact values can be better.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to read. 
The definition of tasks and problems are very clear. 
The idea that fully utilize the rich descriptions of the studied cognitive tasks in scientific report is novel, but reconstructing image from peaks is not quite reasonable. 
Reproducibility is good since details are all given in the experimental setup.
summary_of_the_review: This paper contains novel ideas such as utilizing the rich descriptions of the studied cognitive tasks in scientific reports to assist in brain decoding task. However, I have several concerns about the image reconstruction from peaks and experiments. For more details please see the ‘Strength And Weaknesses’.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Peaks2Image as a neural-network approach to reconstruct continuous spatial representations of brain activity from peak activation tables. This study aims to learn a decoding model by using TF-IDF features as labels so that the number of decoded terms will be much broader than in current image-based studies. Based on 43,000 NeuroVault images, they successfully decoded 58 out of 81 concepts. In summary, this paper is interesting in terms of neuroscience, but it is unclear to what extent the proposed method is novel in terms of machine learning.
strength_and_weaknesses: Please refer to the review section.

clarity,_quality,_novelty_and_reproducibility: Please refer to the review section.

summary_of_the_review: This paper contains several controversial claims. 

The paper uses Nilearn to estimate peaks and then uses these peaks to learn a model from data. However, there is no evaluation of the error associated with the selection of these peaks. Further, what evidence shows that you are more likely to see neural responses in these peaks than examining the whole brain data?

Another concern is the proposed formulation. As an example, what is the definition of the function f() in Equation 1? To learn f*, what constraints are considered? The same question can be asked for D in Equation 2. Due to the fact that D without any constraint may increase the chance of overfitting. 

Empirical studies are another concern. The proposed method only compares with some classic machine learning approaches, such as Kernel Ridge or Space Gaussian. There are several groups of state-of-the-art techniques that aim at similar types of analysis and should be compared with the proposed approach, e.g., HTFA (Manning 2018), BRSA (Cai 2016), and DRD (Wu 2019). 

In addition to the aforementioned concerns, the proposed approach should clarify some related concepts for multi-subject and multi-resource analysis. It seems that this study uses multi-subject data. However, I could not find anywhere that discusses how the author(s) functionally align brain responses from different subjects. Alternatively, Neuro Vault is a multi-resource repository where data are provided from different machines and environments. I also did not find any information on how the proposed method deals with bach effects to learn a model from multi-site data. You may provide additional information in a preprocessing/preparation section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Cognitive neuroscience aims to reveal the underlying brain regions corresponding to different cognitive processes, and such studies are usually performed by comparing and contrasting the fMRI scans of subjects performing designed cognitive tasks. A major drawback in this field is the high cost of data acquisition, which limits the sample size and hence undermines the statistical power. Meta-analysis over many individual studies, though may partially overcome the sample size issue, usually runs into data unavailability issues where only the locations (in form of standardized coordinates) of peaks of activations are reported in literature while the raw image data are not publicly shared.

On the other hand of the story are the recent efforts (e.g., NeuroVault, OpenNeuro) of aggregating and curating data from individual studies yielding large-scale public image repositories. In these repositories, however, the annotations and metadata are usually scarce and inconsistent.

The authors claimed that the rich text data from neuroscientific literature and the large-scale public image data are complementary. Leveraging the advantages from both sides, they proposed a novel meta-analysis method that can decode fMRI statistical maps corresponding to concepts in neuroscientific literature.
strength_and_weaknesses: Strengths
1. The authors proposed an interesting workflow. If I understand it correctly, the authors are using the following priors: (1) fMRI brain maps is tightly correlated to the underlying cognitive states, and (2) distribution of peak locations in fMRI brain maps is highly correlated to the overall global pattern in fMRI brain maps. With these priors, the authors proposed to train a neural network model to predict the global fMRI pattern from sparse locations of peaks, and with that trained model they synthesized a paired dataset of (cognitive descriptions, global fMRI patterns) using the (cognitive descriptions, sparse locations of peaks) from the neuroscientific literatures. Finally, they trained a separate decoder with the (cognitive descriptions, global fMRI patterns) pairs to decode the former from the latter. The decoder is the ultimate piece of mastery that can describe the underlying cognitive states of an arbitrary fMRI image.

Weaknesses
1. In my humble opinion, the abstract may need to be better articulated. I was very disoriented and confused by the abstract until I finished reading the introduction section. Specifically, I was confused by (1) what the two complementary sources of data are, (2) why they are complementary, and (3) what is the proposed method trying to accomplish. For (1) and (2), even though the two sources of data are both explicitly mentioned, I did not have enough context to parse their relationship. In contrary, the introduction section has a very natural flow and is painless to follow.
2. It is a bit misleading that when the authors mentioned the word “image” they did not necessarily mean “image”, but rather the “1x1024 vector representation in the DeFuMo space of the image”. I believe it might be helpful to provide a clarifying statement somewhere early in the paper to avoid potential misunderstanding.
3. Figure 1 can be further polished. (1) Figure 1A did not clearly show what data format is provided and what are derived. It might be better if that information is illustrated more explicitly. (2) A lot of things are represented as “blue thin rectangle”, yet some of them are modules/operations while others are entities. It may be better to replace some of them. For example, “Reconstruction” and “decoder” can be replaced by sketches of the neural networks, and “Labels” can be replaced by some clipart object, etc. (3) The caption mentioned that in Figure 1B, the brain map is reconstructed using the trained model, so it would be better to show the sketch of model in an appropriate place and indicate that its weights are frozen.
4. The reconstruction performance (Section 5.1, Figure 2) is quite poor. a correlation of 0.3 does not seem too promising.

Minor Things
1. Is there some typo in the following sentence? Page 5, 3rd paragraph. “When the percentile corresponds to a To avoid selecting studies with a null tf-idf value, we replace the percentile threshold with a strictly positive tf-idf value criteria.”
clarity,_quality,_novelty_and_reproducibility: Clarity and quality of the paper is mediocre. Novelty is good, but not remarkable.
summary_of_the_review: I like the idea behind the paper, but the current presentation is selling it short, significantly. I do not believe the paper in its current form meets the publication standard at ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The aim of this paper is to use sparse information from literature to solve the seemingly ill-posed problem of mapping from peak coordinates to fmri brain activity images. Prior work in deep learning in recent years has shown that going for such tasks can lead to highly non-trivial models. 

The reviewers although finding the paper well-written and clear are not entirely convinced about the merits of the paper. This indicates that although the paper is tackling an important and in this AC's opinion well-posed problem, there is still some work to do before it is ready for publication.

It is therefore recommended that the authors improve the paper further and submit to the next relevant conference. 
justification_for_why_not_higher_score: This is a potentially important and novel contribution that requires to be done completely right to get the impact it deserves.
Decision: Reject