Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a novel method incorporating decoded foreground attention and a new training scheme with encoder and decode in a loop to reconstruct image stimuli from fMRI data.
main_review: Strength:
The encoder decoder loop and use of foreground is relatively novel.

Weakness: 
It is not very convincing to me that the loop-enc-dec model and the use of foreground attention is able to reconstruct stimuli images from brain responses better than past attempts. The one (if not only) quantitive comparison the paper makes with past results (as in Figure 6) shows that this method barely has any improvement over other methods. In terms of qualitative comparison, it is not clear to me the reconstruction provided by this methods is better than those generated by other methods. 

Other than that, this paper would benefit a lot from a more thorough editing - some wordings are very confusing and there are a lot of typos. For example, "deep learing models" in the second paragraph of the introduction. And in the results section (3.2), 68.4% should be from Beliy et. al, not Fang et. al.


summary_of_the_review: This paper has some interesting and innovative ideas about recounting visual stimuli from brain responses. However the paper is not very well written, hard to understand and the results are not very convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a Loop-Enc-Dec framework to perform an image reconstruction in a neural decoding task. The solution is based on an end-to-end encoder-decoder model under the guidance of Foreground-attention to enhance the perceptual quality of reconstructed images. The experimental results show visible improvements in the quality of reconstructed images without requiring additional fMRI training data. Also, the proposed method shows improvements in the structural similarity of image reconstruction.
main_review: Strengths:
* Brain decoding is an interesting and still very much under-explored area
* Proposed a new framework by introducing the F-attention to enhance the image reconstruction further. 

Weaknesses:

* The experimental section is weak, and hard to interpret the reconstruction results.
** Previous works had shown reconstruction results on at least two datasets. Why do authors consider only one fMRI natural image dataset of 1250 images? (Why not the BOLD 5000 fMRI dataset? And Vim-1 dataset)
** Authors can also report the 2-way, 5-ways, and 10-way results for better comparison with previous methods. 
** From Figure 6, Without any statistical significance test on all the results, it is difficult to compare which method performs better than the remaining methods.
** Also, the proposed method has marginal improvement (68.9%) in terms of SSIM over the previous method (68.4%). No clear explanation for the comparison of the results.
** Figure 4 shows that ResNet is used as a pretrained encoder for extracting fMRI from stimuli, and what is the purpose of VGG19 in the Loop-Enc-Dec architecture? Throughout the paper, there is no discussion on VGG19.
** Any insights on self-attention?
** What is the dimension of Query, Key, and Value weight matrices?
** How is SSIM calculated?
** How do authors obtain λ value as 0.7 and μ as 0.3?
* Paper: M. Mozafari, L. Reddy and R. VanRullen, Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN, this paper is not cited at all, and no comparison with previous works in terms of evaluation metrics. 
* None of the image reconstruction results look comparable to Ground Truth or with previous methods.
* Cognitive insights are missing in this paper.
* Figure 8 illustrates that Lower visual areas have better reconstruction performance than Higher visual areas? Does this indicate the proposed model is reconstructing shapes and corners more? Is it because the number of voxels present in LVC is more than HVC?
* The total number of voxels present in VC?
* Among all the brain ROIs, which brain ROI is highly involved in image reconstruction?
* Using the Loop-Enc-Dec method to reconstruct natural images from fMRI is not new to this field.
* The mathematical notations used in Equations 5, 6, 7, 8, and 9 are difficult to follow.
** The naming conversion is bad.
summary_of_the_review: Although the paper tests all of their contributions, the benefits do not seem to be dramatic. I do not see proper evaluation metrics that are used in previous methods for the model evaluation and also not tested on other existing fMRI datasets. The experimental section is weak, and it is hard to reproduce the results, so the paper does not introduce a significantly better technique.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a model to decoding the fMRI signal from the human visual cortex by introducing the Foreground-attention. They also proposed a enc-dec training strategy called Loop-Enc-Dec, which is guided by the F-attention, to successfully reconstruct the visual images from the fMRI data. A higher score based on pairwise SSIM is achieved compared to previous works.
main_review: My main concern is that, (if I understand the training process correctly) the label information used for training the attention decoder is from another attention map generator (the F-attention labeling process), which produces artificial attention map. Then what is the meaning of fitting the BOLD signal to the artifical attention map (see Fig.3a&b)?  Obviously this kind of fitting can give an attention map based on the fMRI data, which is similar to the F-attention map given in Fig.3b after optimization. However, this may not reflect what the human attention looks like in this task. 
 
Other concerns:
1) The description of 'Visual attention labelling' should be made more clearer, e.g., how to extract the prominent objects in the images, is it a DNN-based objected detection algorithm? What does "perform a binary value" mean? What is a "tranform operation"?
2) The MSE loss in high dimension may not be easy to optimize (Eq.3, also in Eq.4). Could you please provide some demonstration of the training loss process, and give more illustration of how far the optimization goes?
3) The authors should make more clearer of the contribution of each part in their proposed model. Even though each part in explained in Sec.2, it is still very hard to know the underlying reason of such design. In addition, the ablation study does not make much sense. The increase of performance can be explained by including more training parameters. 
4) The choose of  $\alpha$ and $\beta$ in Eq.4 (also $\lambda$ and $\mu$ in Eq.5) seems to be empirical. 
5) The example of the decoded goldfish is not representative, because the decoded position of other images are also very accurate, as showed in Fig.6. 
6) The argument in Sec.3.4 is wrong. Firstly, the authors should define clearly about LVC and HVC. Otherwise it is hard to say which one is more responsible for the shape perception and contour perception. At least, it shouldn't be in primary visual cortex.
7) What is $d_k$ in Eq.2?

Suggestions:
1, The F-attention is the key in this work, but also a strong hypothesis assumed by the authors that human will adopt to do image classification in such fMRI task. However, the authors should be careful when use this term, as this kind of simple task may not require attention in the human brain. On the other hand, even the brain activation (at the temporal lobe) may reflect the attention info, it is unclear which kind of attention will be involved, only bottom-up attention, or a mixture of top-down feedback and bottom-up attention [1]? So it is better to use more accurate terms, such as 'visual saliency map [2]', etc.
2, To get the attention labels, it is better to use the eye tracking data which is a more direct reflection of the attention of participants, rather than relying on the algorithm-based attention labeling. Such kind of data (3T fMRI + eye tracking data) may be available on the website.
3, The terms in Eq.7&8&9 should be showed in the figure to help understanding. 

[1] Gilbert, C. D., & Li, W. (2013). Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5), 350-363.

[2] Li, Z. (2002). A saliency map in primary visual cortex. Trends in cognitive sciences, 6(1), 9-16.

summary_of_the_review: The idea using F-attention to guide the network training is fine from the view of network training, but the way of using it to do image reconstruction from the FMRI data seems incorrect.  Fitting the F-attention model to the artifical attention map does not make much sense.
review_title: null

Meta-review: This paper present a model for reconstructing images from fMRI recordings, based on an encoder and decoder used in a loop. The reviewers were unanimous in their opinion that this paper is not ready for publication at this stage. They raised concerns ranging from the quality of the result and how to compare them to previous methods, to the justification behind different modeling choices. The authors were gracious in their responses to the reviewers. I do not recommend acceptance at this stage,
Decision: Reject