Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel method for decoding images from FMRI, using a hierarchical VAE. First, a VAE is trained on images from imagenet. Then the encoder is removed, and a mapping is learned from the images to the latent variables of the VAE. The VAE is divided into lower and higher level areas (i.e., it's hierarchical), and lower and higher level brain recordings are used to feed into corresponding areas. The results are impressive, and to my knowledge, represent a new state of the art. 

After reading the other reviews, and the authors' responses to them, I am going to reduce my score from 10 to 8, as they used the test set to set hyperparameters. While they claim that the hyperparameters were not particularly impactful on the results, this is a clearly big mistake in terms of ML practice and should be avoided at all costs. Nevertheless, I believe their results still hold (as they used pretty standard hyperparameters, apparently), and the hierarchical nature of their model clearly makes a difference, and is worth publishing.

After reading the response to my reduction in score, I'm not clear that the authors totally understand that hyperparameters should not be tuned on the test set. They seem to think it is ok to tune training hyperparameters using the test set. This is still not the case. Again, I think there is still much to like here.
strength_and_weaknesses: This is an excellent paper, with excellent results. 

Strengths:

+ The approach is well-motivated by reference to lower and higher brain areas corresponding to lower and higher layers of the decoder.

+ Although the above point has been made before, the novelty here is that the decoder is used to map to the latent variables at different levels of the hierarchical VAE, rather than to the feature layers of a CNN.

+ The results are very strong, both quantitatively and qualitatively.

+ The direct comparison to a factored model with the ROIs randomly assigned to the hierarchy is a nice demonstration that it is the hierarchy that matters, not the division into regions. 

Weaknesses, with concrete, actionable feedback

- There are a few things that could be corrected easily, in terms of typos, etc. See below.

- why didn't you include LOC in the Ventral pathway model?

- The poorer results based on mapping all of the voxels to one latent might be a result of the LOC voxels being noisier. A fairer comparison would be to create three more baseline models, each one using the same areas as the three hierarchical models, but without the hierarchy. This would make a stronger argument for the approach. 


clarity,_quality,_novelty_and_reproducibility: In the abstract, you say: "The current architectures are bottlenecked because they fail to effectively capture the hierarchical processing of visual stimuli that takes place in the human brain. Motivated by that fact,..."
This is not a fact, it is an opinion. Please fix.

Figure 2b has a bug - you show the lower visual areas feeding into Z_2 and the higher ones into Z_1, which is the opposite of what the Figure caption says. Also, in the caption: "using by discarding" -> "by discarding." Also, there is no (a) or (b) in the figure. You could say "top" and "bottom" instead.

abstract, main text: You mention that you decode "natural" images and train the VAE on "natural" images. These images are not natural - that wording refers to images of natural scenes, like forests, woods, etc. Just remove the word "natural" everywhere.

Typos, wording:
1st line of page 3:  object -> objects
Fig 2 caption: using by -> by
sec 3.3, line 3: contain -> contains
page 6, line 3: thought -> though
line 7: fact -> facts
6th line from the bottom: "compares our method" - which method? I assume the ventral pathway. Again - why didn't you include LOC?
4th line from the bottom: you say you average the fMRI responses corresponding to the same category. I assume you mean corresponding to the same image, not category. If you are averaging responses over categories, that definitely needs more explanation.

Page 7, middle: "Qualitatively..." Qualitatively, the ventral and the dorsal both look about the same to me, and this is born out in the quantitative scores, which are very close. 
5th line from the bottom: there -> three. But I would just remove the whole "i.e., we aim to compare..." phrase - it isn't needed. Then replace the sentence that starts "The former ones..." with: "The three models, PS, DP, and VP, are hierarchical, whereas the naive baseline includes..." 
3rd line from the bottom: "by breaking down brain to two..." -> by breaking down the fMRI responses from two regions, V1 and V2, and discarding all other voxels, we are able to achieve better performance compared to mapping all the voxels into one latent vector. 

Page 8: "Furthermore, we observe that the SSIM gives a modest but consistent performance gain compared to PCC." No! That just means that SSIM is a slightly more lax measurement. The two measurements are not directly comparable. There is no reason (based on the numbers) to prefer one to the other. 
2nd line from the bottom: subject -> subjects.
summary_of_the_review: I can find few weaknesses and many strengths in this work. There is novelty (mapping to latents rather than features), a nice comparison between using just V1 and V2, using the dorsal pathway, and using the ventral pathway, as compared to a non-hierarchical model that uses all of the voxels. However, the final paper should include comparisons with each of the three models using the same areas without the hierarchy.

Again, in summary, I am reducing my score based on the hyperparameter tuning.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This study analyzes existing functional magnetic resonance imaging (fMRI) data from human participants that passively viewed a series of images. The analysis is based on using hierarchical variational autoencoders and then using fitting to map the activations of units in the model to the activation of voxels in order to reconstruct the images. Compared to a few baselines, the authors claim to obtain better qualitative results in reconstructions and better quantitative results in discriminating images across trials without cross-validation across images.
strength_and_weaknesses: Strengths

It is interesting to link neural representations to machine learning representations, if done correctly. 

Weaknesses
The title is somewhat misleading. While the word “imagery” can be used to denote the processing of images, a lot of people in the community have used the word to denote visual imagination, which is not what this study is about. The authors briefly mention decoding dreams and such in the conclusion as potential future work, but this is not what is shown here.

It is unclear what the goal of the algorithm should be. One goal could be to attempt to reconstruct images from fMRI voxel activity. The qualitative comparison of perhaps cherry-picked examples is not a convincing result and certainly not a rigorous way of presenting results. The authors present some quantitative results in Fig. 4, but they do not define what 2-way, 5-way, and 10-way represent. More critically, why is it relevant to reconstruct images? The purpose of the visual system is to extract useful information, not to reconstruct images. One should get a much better reconstruction from the activity of neurons in the retina. Whatever cortex does, it is not reconstruction. 

A central issue is to understand what goes into the training set and what goes into the test set. The first question is whether any of the results represent cross-validation across images. My understanding is that the authors are only using cross-validation across trials, which is not very interesting. Any powerful enough model can memorize the dataset. In this case, the only question is how consistent the data are across trials. If this is the main point, it would be useful to show the degree of consistency in the data across trials, showing the actual data across repetitions and measures of self-consistency. 

A different question, and a more interesting one is to perform cross-validation across very different images. In this case, there are a lot of questions that should be addressed. First, are the images in the training set and test set from the same categories (i.e., same imagenet labels)? How different are the images in the training set and the test set? How well can simple classifiers match the images in the test set to those in the training set? These are critical questions to assess what the models are doing or trying to do.


clarity,_quality,_novelty_and_reproducibility: As noted above, there are crtitical questions about the methodology that are not clearly described.

fMRI data have very low quality and signal-to-noise ratio. 

summary_of_the_review: This is an effort to try to reconstruct images from fMRI data. Although the proposed method performs better than the baselines suggested by the author, it is unclear whether this can really extrapolate and provide any real insights about the underlying representations
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors used a hierarchical VAE to learn representations from natural images and leverage their latent space hierarchy to learn voxel-to-image mappings. They showed that mapping V1/V2 responses to the early layer and higher visual areas to the deep layer of the latent hierarchy allow them to achieve superior image decoding performance than state-of-the-art approaches. 
strength_and_weaknesses: Strength: Their image decoding results are very impressive, far better than state-of-the-art.
Weakness: The insights are perhaps obvious, and the technical innovation is relatively minor. 

The comparison between V1/V2 versus ventral and dorsal is not completely fair and problematic.  These comparisons are confounded with different numbers of parameters, and thus the conclusion about the contribution of the different visual areas, ventral and dorsal streams are  potentially problematic.  It seemed that the more neural responses are used, the best would be the performance. For instance, if they include V3 in the ventral stream's model, would  the performance be even better?   To their credit,  they did show that the hierarchy is important, and their argument that the critical comparison is within the naive baseline which has the same number of parameters is not unreasonable. 
clarity,_quality,_novelty_and_reproducibility: Clear enough and well written. 
summary_of_the_review: This paper achieved the best performance in image decoding based on fMRI signals. The motivation is sound, the idea obvious. The technical innovation is minor.  it would become the new baseline for future algorithms to compare against. The comparison between V1/V2, ventral and dorsal streams are potentially problematic and incomplete.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors developed a natural images reconstruction model from fMRI with hierarchical variational autoencoders. Their model achieved better reconstructions compared to the state of the art and our ablation study indicates that the hierarchical structure of the latent space is responsible for that performance.
The main contributions are as follows: 1. The paper leveraged the hierarchical manner of human brain during visual stimuli and proposed a novel architecture with hierarchical variational autoencoders to reconstruct neural images. 2. Their architecture replicates the natural hierarchy of visual information processing in the latent space of a variational model. 3. Their experimental analysis suggests that hierarchical latent models provide better priors for decoding fMRI signals.
strength_and_weaknesses: Strength:
1. This is the first approach that uses HVAEs in the context of neural decoding.
2. The proposed method combines decoding models with neuroscience priors and the motivation makes sense.
3. The experiments are relatively sufficient.


Weaknesses:
(major)
1. Experiments have serious data leakage problem. The HVAE were pretrained in ImageNet and the dataset GOD you used is still collected based on ImageNet. Accordingly, I think the results of this paper is unbelievable.
2. Although authors has pointed out why they only used GOD dataset, there is still a lot of preprocessed datasets that can be used such as BOLD 5000 and NSD. NSD dataset is collected based on MS-COCO which can avoid aforementioned problem.
3. The compared methods are not comprehensive. I suggest that the authors add the method proposed in "Mind Reader:
Reconstructing complex images from brain activities"  in which the reconstruction quality is relatively better than the compared methods you have chosen.
4. The article introduced less content about HVAE, formula (1) is even traditional VAE, and used a lot of space introducing the background and evaluation indicators of neuroscience.  By the way, formula (1) and formula (2) needn't two labels.


(minor)
1. The writing of this paper is terrible.  A major refine of this manuscript is necessary.  Informal expressions such as "{fMRI,Image} pairs" and "Naturally, if our fMRI data are more fine grain, we can add additional latent layers" should be avoided.  Typos such as text in line 3 of page 5 should be corrected. In the experimental results, text wrapping charts is not acceptable for readers.  The structure of the article needs a careful adjustment such as the last two paragraphs of 3.1 should be placed into experimental details.
2. The main results part should contain a series of subtitle for easy understanding.
clarity,_quality,_novelty_and_reproducibility: The novelty this paper is modest but easy to understand. However, major flaw with respect to data using exists which leads to unbelievable experimental results.  The paper contains a large number of informal expressions and clerical errors, the structure is not clear, and the layout is confusing. According to this, the clarity and quality of the paper are  poor. The reproducibility is poor since the code is not available.
summary_of_the_review: The work applies an existing method to an existing task. The novelty this paper is modest but easy to understand. My major concern is the flaw with respect to data using exists which leads to unbelievable experimental results. Besides, the writing of this paper is terrible. I think the quality of the paper is not up to the average quality of the papers accepted by ICLR. I suggest the authors should change their experimental data and carefully revise the manuscript.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a novel method for decoding images from FMRI, using a hierarchical VAE. The paper uses the hierarchical bias of visual processing together with the hierarchy of the VAE to achieve good reconstruction results, both quantitatively and qualitatively. However, two reviewers have raised important problems with the method, including data leakage and hence concerns about the generalisability of the method to new images. Overall, the reviewers believe that the improvements shown in this paper come from using a better generative model rather than a fundamental new contribution of the authors, and it is not clear whether this work will be of value to either the ML or neuroscience communities in its current form.
justification_for_why_not_higher_score: The data leakage concern was not properly addressed and hence I do not believe this paper should be accepted in its current form.
justification_for_why_not_lower_score: NA
Decision: Reject