Neural Decoding of Visual Imagery via Hierarchical Variational Autoencoders
Authors: Eleni Miliotou
Panagiotis Kyriakis
Jason D Hinman
Andrei Irimia
Paul Bogdan
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural decoding
hierarchical variational autoencoders
neuroscience

Abstract: Reconstructing natural images from fMRI recordings is a challenging task of great importance in neuroscience. The current architectures are bottlenecked because they fail to effectively capture the hierarchical processing of visual stimuli that takes place in the human brain. Motivated by that fact, we introduce a novel neural network architecture for the problem of neural decoding. Our architecture uses Hierarchical Variational Autoencoders (HVAEs) to learn meaningful representations of natural images and leverages their latent space hierarchy to learn voxel-to-image mappings. By mapping the early stages of the visual pathway to the first set of latent variables and the higher visual cortex areas to the deeper layers in the latent hierarchy, we are able to construct a latent variable neural decoding model that replicates the hierarchical visual information processing. Our model achieves better reconstructions compared to the state of the art and our ablation study indicates that the hierarchical structure of the latent space is responsible for that performance. 

Under review as a conference paper at ICLR 2023
NEURAL DECODING OF VISUAL IMAGERY VIA HIER- ARCHICAL VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Reconstructing natural images from fMRI recordings is a challenging task great importance in neuroscience. However, current architectures fail to efficiently capture the hierarchical processing of visual information processing, which may bottleneck their representation capacity, Motivated by that fact, we introduce a novel neural network architecture for the problem of neural decoding. Our archi- tecture uses Hierarchical Variational Autoencoders (HVAEs) to learn meaningful representations of natural images and leverages their latent space hierarchy to learn voxel-to-image mappings. By mapping the early stages of the visual pathway to the first set of latent variables and the higher visual cortex areas to the deeper layers in the latent hierarchy, we are able construct latent variable neural decoding model that replicates the hierarchical visual information processing. Our model achieves better reconstructions compared the state of the art and our ablation study indicates that the hierarchical structure of the latent space is responsible for that performance.
INTRODUCTION
Decoding visual imagery from brain recordings is a key problem in neuroscience. This problem aims reconstruct the visual stimuli from fMRI recordings taken while the subject is viewing the stimuli. Even though some of the excitement is fulled by science fiction and the difficulty of the problem (. the scientific consensus is that neural decoding has real-world, important implications. is important for understanding how neural activity relates to external stimuli (2), for engineering application such as brain-computer interfaces 4 and for decoding imagery during sleep (4). Given its importance, neuroscience and machine learning researchers have jointly led the development of sophisticated deep learning architectures that allows us to design pipelines that map voxel-based recordings to the corresponding visual stimuli. Based on the target learning task, visual decoding can be categorized into stimuli classification, stimuli idenrification, and stimuli reconstruction. The former two tasks aim to predict the object category of the presented stimulus or identify the stimulus from ensemble of possible stimuli. The reconstruction task, which is the most challenging one and the main focus of this paper, aims construct a replica of the presented stimulus image from the fMRI recordings.
Related Work. The proposed methods for the problem of neural decoding can be broadly classified in three categories: non-deep learning methods, non- non-ggenraative deep learning methods and generative deep learning methods. The non-deep learning class consists of methods that are based on primitive linear models and aim in reconstructing low-level image features (5). Such approaches first extract handcrafted features from real-world images, such as multi-scale image bases (6) or Gabor filters 77. and then learn linear mapping from the fMRI voxel space to the extracted features. Due to their simplicity, linear models are not able to reconstruct complex real-world images and thus their applicability restricted to simple images containing only low-level features.
Methods that use convolutional neural networks as well as encoder-decoder architectures belong to the non-generative deep learning class. Horikawa et al. & demonstrated a homology between human and machine vision by designing an architecture with which the features extracted from convolutional neural networks can be predicted from fMRI signals. Based upon those findings, Shen et al. ( used a pretrained VGG-19 model to extract hierarchical features from stimuli images and learned mapping from the fMRI voxels in the low/high area to the corresponding low/high VGG-19 features. Beliy al. (9) designed a CNN-based Decoeer architecture, where the encoder learns

---

Under review as a conference paper at ICLR 2023
mapping from the stimulus images to the fMRI voxels and the decoder learns the reverse mapping. By stacking the components back-to-back, the authors train their network using self-supervision, thereby addressing the inherent scarcity of iMRI-image pairs. Following up that work, Gaziv et al. (10) improved the reconstruction quality by training on perceptual similarity loss function, which is calculated by first extracting multi-layer features from both the original and reconstructed images and comparing the extracted features layer-wise. Such a perceptual loss is known to be highly effective in assessing the image similarity and accounts for many nuances in the human vision
In the generative deep learning class, we have model architectures, such as generative adversarial networks (GANs) and variational autoencoders (VAEs). Shen et al. (I) extended their original method to make the reconstructions look more natural by conditioning the reconstructed images to be in the subspace of the images generated by a GAN, similar GAN-prior was used by Yves et al. in (12), where the authors also introduced unsupervised training on real-world images. Fang et al. (I3) leverage the hierarchical structure of the information processing in the visual cortex to propose two decoders, which extract information from the low and high visual cortex areas, respectively. The output of those decoders is used as a conditioning variable in a GAN-based architecture. Shen et al. (14) trained a GAN using a modified loss function that includes in addition to the standard adversarial loss. A line of work by Seeliger et al. (15), Mozafari et al. (I6) and Qiao et al. (17) assumes that there exists a linear relationship between the brain activity and the GAN latent space. These methods use the GAN as a real-world image prior to ensure that the reconstructed image has some "naturalness" properties. The work by VanRullen al. (18) and Ren et al. (19) utilize VAE-GANS (20). hybrid model in which the VAE decoder and GAN generator are combined. In the former work, the authors use the VAE to extract meaningful representations of the data and learn linear mapping between the latent vector and the fMRI patterns. In the later work, the authors propose a dual-VAE architecture where both the real-world images and fMRI voxels are converted into latent representations, which are then fed as conditioning variable in a GAN. Finally, the work by Lin et al. (2 leverages multi-modality and encodes the fMRI signals into a visual--anguagg latent space and a contrastive loss function to incorporate low-level visual features to the schematic pipeline, Then, the authors use a conditional generative model to reconstruct the images and obtain photo-realistie and accurate reconstructions.
image-space and perceptual loss
Contributions. In this paper, we purpose a novel architecture for the problem of decoding visual imagery from fMRI recordings. Motivated by the fact that the visual pathway in the human brain processes stimuli in a hierarchical manner, we postulate that such a hierarchy can be captured by the latent space of a deep generative model. More specifically, we use Hierarchical Variational Autoencoders (HVAE) (22) to learn meaningful representations of stimuli images and we train an ensemble of deep neural networks to learn mappings from the voxel space to the HVAE latent spaces. Voxels originating from the early stages of the visual pathway (V1, V2, V3) are mapped to the carlier layers of latent variables, whereas the higher visual cortex areas (LOC, PPA, FFA) are mapped to the later stages of the latent hierarchy. Our architecture replicates the natural hierarchy of visual information processing in the latent space of a variational model. Our experimental analysis suggests that hierarchical latent models provide better priors for decoding fMRI signals and, to the best of our knowledge, this is the first approach that uses HVAEs in the context of neural decoding.
2 VISUAL INFORMATION PROCESSING
In this section, we give a brief overview of the visual information processing in the human brain and describe the two streams hypothesis, which we use in our experimental architecture. Visual information received from the retina of the eye is interpreted and processed in the visual cortex. The visual cortex is located in the posterior part of the brain, the occipital lobe, and is divided into five distinct areas (V1 to V5) depending on the function and structure of the area. Visual stimuli received from the retina travel to the lateral geniculate nucleus (L.GN), located near the thalamus. LGN is multi-layered structure that receives input directly from both retinas and sends axons to the primary visual cortex (V1). VI is the first and main area of the visual cortex where visual information is received, segmented, and integrated into other regions of the visual cortex. Based on the two streams hypothesis (23). following V1, visual stimuli can take the dorsal pathway or ventral pathway. The dorsal pathway consists of the secondary visual cortex (V2), the third visual cortex (V3), and the fifth visual cortex (V5). The dorsal stream, informally known as the "where" stream, is responsible for -uuide behaviors and localizing objects in space. The ventral stream, also known as the
2

---

Under review as a conference paper at ICLR 2023
"what" stream, consists of V2 and fourth visual cortex (V4) areas and is responsible for processing information for visual recognition and perception. Visual processing occurs hierarchically at three distinct levels (24). The low-level includes the retina, lateral geniculate nuclei (LGN), and the primary visual cortex (V1). Low-level processing is the initial step when interpreting an image and it is the place where orientation, edges, and lines perceived. Sequentially, the mid-level processing consists of the secondary (V2), third (V3) and fourth (V4) which extract shapes, objects and colors. Finally, the high-level processing consists of category-selectiv areas such as the fusiform face area (FFA), lateral occipital (LOC), parahippocampal area (PPA) and medial temporal area (MT/V5). These areas show selective response to faces, objects/animals, places and motion, respectively.
Despite the evident hierarchical structure of vi- sual information processing, most current meth- ods for neural decoding fail to fully exploit that fact. Current methods take into account the hier- archy of visual information processing either by mapping the fMRI voxel to hierarchical CNN- extracted image features via regression models (5) or by training an end-to-end DNN model on feature loss function (12) 14). The major issue with such approaches
MT
V5
V3
LOC
V2
that the hierar- chy is taken into account in the feature space of CNN model, which is. in general, complex, high-dimensional space. In this work, pro- pose to take into account the aforementioned hierarchy in the latent space of a deep model. Latent spaces are known to produce compact, low-dimensional embeddings of the data and have recently shown impressive performance on image reconstruction and generation tasks (22). Given these facts, we postulate that a hicrarchical latent space provides better priors for decoding fMRI signals. The intuition is that each brain area, being "responsible" for certain set of features, better be mapped on compact, low-dimensional representation of those features. For example, given that V1 is broadly responsible for encoding low-level features (e.g., edges, orientations), it is sensible to map the fMRI voxels from the VI region onto representation of the underlying images features; and this mapping much easier to be learned the latent space, rather than the feature space.
Retina
PPA
FFA
V4
V2
V3
Figure 1:
Two stream hypothesis of visual information processing in the human brain.
METHOD
Leveraging the aforementioned intuition, we introduce neural decoding method that mimies the hierarchical visual information processing in the latent space. Our architecture has two main components: a Hierarchical Variational Autoencoder (HVAE) and a Neural Decoder. The HVAE is used for learning compact, hierarchical latent representations of real-world images and is trained using self-supervision. The Neural Decoder used for mapping the brain signals the HVAE hierarchical latent space and is trained via supervision on ( [mage pairs. In this section, we describe each of the components in more detail. Our architecture visualized in Fig for the special case of 2 latent hierarchical layers.
3.1 HIERARCHICAL VARIATIONAL AUTOENCODERS
To capture the inherent hierarchical structure of visual information processing, we propose model images via family of probabilistic models known as Hierarchical Variational Autoencoders (HVAEs). HVAES extend the basic Variational Autoencoder (VAEs) by introducing a hierarchy of latent variables. Formally, let be an image and z = {21, 22, ... 2r } be set of L latent variables. The generative po
distribution or decoder defined as (x(x (= II pe (2i+ Iz.) and is parametrized by 0. The prior distribution is defined as p(z) = p(z1)
(2,). The posterior p(z|x) is approximated by the variational distribution encoder (z go (z (x) II'_ 40 ( (z,). which is parametrized by o. Both the prior and the approximate posterior are represented by factorial Normal distributions. The variational principle provides tractable lower bound, known Evidence
p(2
3

---

Under review as a conference paper at ICLR 2023
EB1
NWV
D81
DB2
DB1
DB2
Figure 2: Outline of our method: a) We pretrain Hierarchical Variational Autoencoder large of images. Two layers of latent variahles 11, 7a inserted after each encoder (EB) and decoder (DB) block. b) We train the Neural Decoder by discarding the encoder from the previous step learning a map from fMRI voxels to the hierarchical latent space. The lower visual cortex (V1, V2, V3) is mapped to zn and the higher visual cortex (FFA, PPA, LOC) to Z2.
Lower Bound (ELBO), on the log- log-liielhhood. follows
Pa xx z)
log (x() 2 E (x(x)
log
= L(0, xx
9o (z/xx)
(1) where KL is the Kullback-Leible divergence. The encoder and decoder are implemented by deep neural networks and their parameters are jointly optimized using gradient descent on the ELBO criterion. Similarly to standard VAEs, the reparametrization trick (26) (27) is used to allow us to back-propagate the gradient thought the stochastic sampling involved in the computation of Eq
= = K (z) Ipo (z))) + Egg (x x [log pe (x/z)).
3.2 NEURAL DECODER
We now leverage the latent space of the HVAE learn a set of maps from the fMRI voxel space to the hierarchical latent variables. In more detail, each region of interest (ROI) mapped via dense neural network to specific subset of the latent space. Brain regions in the earlier states of the visual pathway are mapped to the earlier layers of the latent hierarchy, whereas voxels from the higher visual cortex areas mapped to the deep layers in the latent hierarchy. We assume that the HVAE has L. groups of latent variables 21, Z2, ......2 22, and that the fMRI voxels are partitioned into n non-overlapping brain regions of choice, i.e., y1, y2.-. Zz. Formally, the Neural Decoder is of maps from the i-th brain region to the i-th group of latent variables. Each of these maps is represented by deep neural network with parameters wi, i.e., 2; twe (y+), & is then obtained by passing the latent variables = { (y1), Van (y2). U.r (yL)) through the decoder model p(x z) defined in Sec. 3.1
2, L. The reconstruction
The loss function used for training the Neural Decoder is an important design choice. Classic per-pixel measures. such as Euclidean distance, commonly used for regression problems, or the related Peak Signal-to-Noise Ratio (PSNR), are insufficient for images, as they assume pixel-wise independence. Therefore, to encourage the Neural Decoder to learn reconstructions guided by human visual perception, use perceptual loss. Perceptual loss is class of loss functions that relies on the fact that CNNs extract hierarchical features. More specifically, deep features trained on supervised, self-supervised and unsupervised objectives are effective model of human visual perceptual similarity (II). For a given image x and its reconstruction &, their perceptual loss is:
1(x, x)
Σ H.W. Σ bb S A011.
4

---

Under review as a conference paper at ICLR 2023
where f²x f are the layer- activations of given, pretrained CNN model, b, I is channel- wise scaling vector. Intuitively, the perceptual loss in Eq. 2 exttrcts features for both the target and reconstructed image and then compares the features layer-wise using the Euclidean norm. To ensure that no bias is introduced during learning, it is important that the CNN used for evaluating Eq. i different than the one used for the encoder. In our implementation we use a pretrained AlexNet as well as the code provided by Zhang et al. (lI to compute the perceptual loss.
3.3 MODEL. TRAINING
For the encoder part of our HVAF, we use a pretrained VGG-19 model (28). This is a deep convolutional neural network of 19 convolutional layers and 3 fully connected layers. We use the weights from the model pretrained on ImageNet and discard the fully connected layers. We introduce latent variables by taking the output of given convolutional layer, flattening it, passing through a fully connected layer and, finally, through variational layer which outputs the latent variable. This latent variable is se-samlled avoid any dimension mismatch, and rerouted back to the main block, where it is aggregated with the output of the convolutional layer. Depending on how many latent layers we would like to insert, their exact position may vary. As an empirical design choice we choose to insert the latent layers equally spaced and after convolutional block. A latent layer is always inserted at the output of the penultimate convolutional block.
The decoder part of our HVAE transforms the hierarchical latent variables to output images and consists of 4 transposed convolutional layers. The number of decoder filters are [128, 64, 32, 16, 3) and all kernel sizes are set to 5. Each transposed convolutional layer is followed by a 2d batch normalization and a ReLU non-linearity. The output of each transposed convolutional layer is interleaved with the latent variables. More specifically, each latent variable is initially passed thought a fully connected layer, re-sampled to avoid dimension mismatch and then aggregated with the output of the corresponding transposed convolution. Similarly to the encoder, insert the latent variable such that we ensure symmetry and we always insert the penultimate latent variable before the first transposed convolution.
We start the training process by first deciding the number and position of the latent layers. The choice guided by the type of iMRI data that we have as well as the level of latent space coarse gaaasengaainng that we can achieve. For instance, if our fMRI data contains only the primary (V1) and the secondary (V2) visual cortex then we have two choices: a) we can either consolidate all voxels into a single vector and have single latent layer in our HVAE or b) we can have two vectors containing the voxels from each brain area and train the HVAE such that it has two latent layers 21, Z2 (example shown in Fig. 2) Naturally, if our fMRI data are more fine grain, we can add additional latent layers.
Following this design choice, the training proceeds in two phases: In the first phase, we pretrain the HVAE via self-supervision using the ELBO loss function [ on a large ensemble of 50,000 real-world images from the ImageNet database. These images come from the same categories as the images shown to the subjects but no test images representations and allows the HVAE decoder to adapt to the statistics of a large set of real-world images. In the second phase, the HVAE encoder is discarded, the HVAE decoder is kept fixed and the Neural Decoder is trained on supervised (fMRI, Image) pairs using the perceptual loss function Eq. 2 In this phase, we essentially learn a map from the voxels of each brain area the corresponding latent layer and then use that latent vector to reconstruct the image.
included. This phase gives meaningful latent
4 EXPERIMENTAL RESULTS
To evaluate the utility of our method in practice, we carry out a series of experimental simulations. To measure the performance of our method, we use both qualitative comparisons of the reconstructions as well as quantitative metrics. In what follows, we give the details of the dataset used, the metrics implemented and baseline comparisons.
We applied our pipeline commonly used, publicly available dataset known Generic Object Decoding (GOD). The dataset consists of high-resolution (500 x 500) stimuli images and their corresponding fMRI recordings. There exist 1250 (1200 train, 50 test) stimuli images selected from 200 object categories from the ImageNet database and the fMRI recording were obtained while 5 healthy subjects were viewing the stimuli (presentation experiment). The train- and test-fMRI data
Dataset:
5

---

Under review as a conference paper at ICLR 2023
consist of 1 and 35 (repeated recordings) per presented stimulus image, respectively. We use the post-processed fMRI data provided by Horikawa et al. (8, which contain voxels from brain areas W...V......FFA.PP...LOO
The temporal component of the fMRI signal is averaged-out and the input to the model is high-dimensional voxel vector. Even though there may be more comprehensive datasets, such as the BOLD 5000 (29) and the NSD (30) datasets (which in fact contain higher number of more diverse images), we choose to focus on GOD for two primary reasons: 1) the dataset provides post-processed IMRI data, and it has been used in numerous past studies (9)
Both of these facts facilitate the easy and fair comparison between different methods.
Ablation Study:
We perform an ablation study, with the number of hierarchical layers and, con- secutively, the number of brain regions, being the ablated parameter. Motivated by the two stream hypothesis ( for the neural processing of visual information, we consider the following variants:
1. Naive Baseline (NB): We consider only one latent layer 2NB and all fMRI voxels are mapped to B. There are approximately 5000 voxels in this variant.
2. Primary-Secondary (PS): We consider latent layers 2 2y2 and the voxels from V1, V2 are mapped to the corresponding latent layer. There are approximately 1500 voxels.
3. Dorsal Pathway (DP): We consider the 3 latent layers Zv2: Zy3 and voxels from V1, V2, V3 are mapped to the corresponding latent. There are approximately 2500 voxels.
4. Ventral Pathway (VP): We consider latent layers Zy1 Zv,, Zy4: ZPF and the voxels from V1, V2, V4, (FFA, PPA} are mapped to the corresponding latent layer. The voxels from FFA A and PPA are merged to a single area. There are approximately 3300 voxels.
We note that by using different ROIs and/or by combining them form different latent architectures, it is possible to obtain different ablated variants. We empirically noticed that by including the LOC, either concatenated part of the latest latent layer of the VP by creating new LOC-only latent layer, there no further performance improvements, only losses in terms of computational cost. Therefore, restrict our exposition to the aforementioned 4 variants.
The reconstruction quality is assessed both subjectively, i.e., by visual inspection of the output test images and comparison with the ground truth, as well objectively. Our quantitative evaluation relies on metrics that encode the spatial dependence such as the Pearson Correlation Coefficient (PCC) and the Structural Similarity Index Measure (SSIM).
Metrics:
Pearson Correlation Coefficient (PCC): This metric is extensively used in statistics to measure the linear dependence between variables. In the context of image similarity, PCC is computed on the flattened representations of the two images. The limitation of PCC is its sensitivity to edge intensity or misalignment, which makes the metric assign larger value blurry images (9)
Structural Similarity Index Measure (SSIM): Wang et al. proposed SSIM in 33 as a metric that quantifies the characteristies of human vision. Given a pair of images p, q. SSIM is computed weighted combination of luminance, contrast and structure. Assuming equal contribution of each measure, SSIM first computed locally in a common window of size x N, and then the global SSIM is computed by averaging the SSIM over all non-overlapping windows.
These image similarity metries defined are used for computing the correct identification rate in an n-way classification task. Let M E (PCC,SSIM) be a metric of choice, Pú be a reconstructed image and P. be a set containing the ground truth P. and set of 1 randomly selected target images. The Correct Identification Rate (CIR) is defined as follows:
N
CTR
Σ 1(i arg max M M(P. P)));
N
EyEP.
where N is the total number of images and the indicator function 1(-) has the value of 1 if the argument is true and 0 otherwise. The CLRY metric is essentially the frequency at which a reconstructed image can correctly identify the ground truth among randomly selected additional images. The chance level is 1/n.
Main Results:
We compare the performance of our method against several state of the art methods (SOTA) for the problem of neural decoding. The competitor methods are: the encoder-decoder based self-supervised method by Belyi al. (9, the end-to-end, GAN-based pipeline by Shen al. (14).

---

Under review as a conference paper at ICLR 2023
the GAN- cAn-iondit method by Shen et al. ① and the shape-schematic GAN by Fang et al. Figure shows qualitative results and compares our method against the aforementioned competitors. All dis- played images were reconstructed from the test-fMRI dataset. To improve the signal-to- genal-to-noise to-noise ratio, the test fMRI test samples are averaged across trials. The results shown were obtained using the Ventral Pathway variant, which gave the best performance. We directly respective papers by the authors. Our method tends to consistently produce more faithful reconstruc- tions. Note that, even though the GAN-based decoders tend produce more natural images, the recon- structions may deviate significantly from the stimulus image. This is because the GAN is introduced as an imaged prior, as noted by Belyi et al. (9). On the contrary, our method reconstructs the stimuli more faithfully, albeit the reconstructions appearing as noisier version of the ground truth. (I3).
the reconstructions reported in the
It is evident that the qualitative comparison highlights a trade- off between the naturalness of the reconstructed stimuli and the pixel-wise noise introduced in the reconstructions. To resolve the ambiguity, we perform additional quantitative compari- son using the CIR" metrie. For this part we compare against the method by Belyi al. (9) as well as the two variants of the method by Shen et ( We directly compare against the results as re- ported by the authors of (9). The results are shown in Fig. 4 For our method, we report the correct identification rate obtained using the Ventral Pathway variant and (CIRpec Rycc and CIR We observe that our method consistently
Ground
Fing al.
Bally
Shen al.
Shen al.
Figure 3: Qualitative comparison of reconstruction quality. average across the metrics (CT outperforms the competitors and, particularly in the 5-way and 10-way case, by a substantial margin. Additionally, observe our method shows small performance drop as increase n1 i.e, from 90% in the 2-way case to 79% in the 10-way case, whereas the performance loss for the competitor method substantially higher. This performance due to the following fact: Even though our method gives noisier reconstructions than the competitors, the high-level features such as color, texture and shapes are retained and, therefore, the task of identifying the correct ground truth from the reconstruction casier. In contrast, please observe in Fig. 3 that the competitor methods may substantially alter the color or texture the image, therefore leading to more frequent ground truth misidentification.
In the next experiment, we evaluate decoding performance of differ- visual pathways. The results are
Identification accuracy (%)
100
90
shown on Fig. [5] Qualitatively, the ventral stream seems be produc-
80
70
ing the best reconstructions, which
60
expected from a neuroscience per-
50
spective, given that this pathway's purpose is for visual perception and
40
30
contains high level areas (FFA-PPA) for object recognition. Interestingly enough, even though the Naive Base- line contains the available brain areas, the reconstruction quality is in- ferior, especially in the 2nd and 3rd images, which are far more complex.
20
10
2-way
5-way
10-way
Ours
Belyi et
Shen et al. Shen et
Figure 4: Conect identification ratio.
The V1-V2, Dorsal and Ventral variants essentially partition the brain areas into (progressively finer) segments and map the voxels from each area onto the hierarchical latent space of the HVAE decoder. Even though the increased performance among these variants may be partially explained by the
7

---

Under review as a conference paper at ICLR 2023
fact that the number of voxels increases, the main point of comparison should be against the Naive Baseline. The three models, PS, DP, and VP, are hierarchical, whereas the naive baseline includes all data but has hierarchy. Simply the fMRI responses from two regions, VI and V2 and discarding all other voxels we are able to achieve better performance than simply mapping all voxels in a big latent vector. This suggests that the hierarchy is far fed to the model. This is in line with previous studies which concluded that models trained on the whole visual cortex perform slightly worse than those trained on separate areas (3).
important than the amount of data that
Additionally, since the Naive Baseline essentially learns a map from all vox- els to single latent layer, it is natural assume that is fails due to massively overfitting. However, if overfitting is indeed the only reason for that failure, we would expect the reconstruction performance to decrease as we add more voxels to the model input. How- ever, the figure shows the exact oppo- site: the performance increases add more voxels. This suggests that overfitting is the only reason for the Naive model's 's failure and that the the hierarchical structure of the visual information processing needs to be ex- plicitly taken into account. However, one may hypothesize that the perfor- mance increase the Ventral Pathway model may come from the partitioning of the ROIs and that the hicrarchical structure has little impact. To test this, it is prudent to include variant in which the VP ROIs randomly shuffled to assess whether the hierarchical structure or the partitioning of the voxels drives the performance. We call this variant VP Permutations and it supplements the previous variants.
Ground truth
Naive
V1-v2
Dorsal
Ventral
Figure 5: Qualitative comparison for different pathways.
CIRPoc
CIR
CIR
CIR
CIR
SSMM
Naive Baseline
0.77
0.78
0.64
0.66
0.57
0.58
0.80
0.82
0.72
0.73
0.65
0.67
Dorsal Pathway
0.88
0.90
0.81
0.80
0.75
0.75
Ventral Pathway 0.91
0.92
0.84
0.85
0.79
0.79
VP Permutations 0.79
0.80
0.65
0.66
0.60
0.58
Table 1:
The n-way correct identification rate (n 2, 5, 10) for all ablated variants using the Pearson Correlation Coefficient (PCC) and the Structural Similarity Index Mcasure (SSIM) as selection criterion. We report the mean across subjects. The results for Permutations are averaged across permutations. The inter-subject deviation was in the range of 0.02 - 0.05. The chance levels are 0.5, 0.25, 0.10, respectively.
Following that, we present quantitative results on Table On this table we give the the n-way correct identification rate CL for n 2, 5, 10, for all ablated variants and the VP Permutations for both metrics (PCC and SSIM). The results on this table validate the aforementioned qualitative observations. The identification accuracy is progressively increasing as we partition the brain into more fine areas and as we add hierarchical layers in the HVAE onto which the brain areas are mapped. Additionally, we observe that the newly introduced VP Permutations variant leads to performance degradation, which suggests that the hierarchy and not the partitioning drives the performance result. However, we do note that we have a slight performance increase compared to the Naive Baseline, which indicates that merely partitioning the brain regions is beneficial, albeit not beneficial as accounting for the hierarchy.
To verify that method can be successfully applied all subjects and study potential inter-subject variation of the results, we show in Fig. the learning curves for all 5 subjects and for all metrics CT R 2, 10 and M (PCC, SSTM). The metrics were calculated using the test samples
8

---

Under review as a conference paper at ICLR 2023
C/R
CTR
C1R
Ci
C1R38MM
Figure 6: Learning curves for CIRI n = 2,5, 10 and M E SSIM) across all subjects. The horizontal axis is the number of epochs, Subject is marginally outperforming the other subjects and Subject I gives the worst performance (figure best viewed color).
and the ventral pathway variant. Even though the metrics appear similar across subject, after careful examination of the curves some subtle discrepancies and trends can be observed. Subject is consistently performing approximately 5% worse across all metries whereas Subject 3 is marginally outperforming the other subjects by 2%. The fact that the Subject gives the best reconstructions has been verified in previous studies (10) and is attributed to differences in the sigmal-to-noise ratio across subjects. Finally, Fig. 6 allows to study how training progresses and validate that no overfitting occurs. We observe that, in all cases, the metrics saturate at about 800 epochs, which gives us an empirical estimate of how many iterations model needs to achieve good performance.
Finally, we present on Fig. 7 qualitative comparison against the recent method by Lin at al. (21). Even though the images are different, we can extract some interesting conclusions. Our method seems be more faithful albeit having lower reconstruction quality. The method by Lin et al. has excellent reconstruction quality and only losing some elements of the faithfulness, which are hurting the schematic understanding. This may be because the later method is more sophisticated and handles schematic information in addition to the visual part of the modeling procedure. Since our method is not incorporating any schematic information, an inter- esting future direction is extend the model in order to see our pipeline can achieve similar quality.
Ours
Lin al.
Figure 7: Qualitative comparison with Lin
al. (1 on schematically-simee test images.
CONCLUSION
We addressed the problem of neural decoding from fMRI recordings and proposed novel architecture inspired by neuroscience. More specifically, motivated by the fact that the human brain processes visual stimuli in a hierarchical fashion, we postulated that this structure can be captured by latent space of hierarchical variational autoencoder (HVAE). Our HVAE serves as proxy learning meaningful latent representations of stimuli images and can be pretrained on a large dataset of high-resolution images. Following that, train our Neural Decoder learn map from the fMRI voxel space to the HVAE latent space. Our architecture replicates the visual information processing the human brain in the sense that earlier visual cortex areas (e.g.. primary- seconaay visual cortex) are mapped to the earlier latent layers, whereas voxels from the higher visual cortex (e.g.. PPA, FFA areas) are mapped the later latent layers. We validated our approach using fMRI recordings from visual presentation experiment involving 5 subjects and compared against other methods. Our work paves the way to constructing better models to replicate human perception and understanding the nuances of human visual reconstruction, both of which could utilized better understand the brain, assist people with visual disabilities and perhaps in decoding imagery during sleep.

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image recon- struction from human brain activity. PLOS Computational Biology, 15(1):1-23, 01 2019.
[2] Joshua I Glaser, Ari S Benjamin, Raeed H Chowdhury, Matthew G Perich, Lee E Miller, and Konrad P Kording. Machine learning for neural decoding. eNeuro, (44::ENEURO.0006 19.2020, August 2020.
[3] David M Brandman, Sydney S Cash, and Leigh R Hochberg. Review: Human intracortical recording and neural decoding for brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng.. 25(10))1687-1699
October 2017.
[4] Tomoyasu Horikawa, Masako Tamaki, Yoichi Miyawaki, and Yukiyasu Kamitani. Neural decoding of visual imagery during sleep. Science (New York, N.Y.), 340, 04 2013.
[5] Kendrick N. Kay, Thomas Naselaris, Ryan J. Prenger, and Jack L. Gallant. Identifying natural images from human brain activity. Nature, 152371855 Mar 2008.
[6] Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-Aki Sato, Yusuke Morito, Hiroki C Tanabe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using combination of multiscale local image decoders. Neuron, 60(5):115-929, December 2008
[7] Takashi Yoshida and Kenichi Ohki. Natural images are reliably represented by sparse and variable populations of neurons visual cortex. Nature Communications, 1 (1(1)8872 Feb 2020.
[8] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1):15037, May 2017.
[9] Roman Beliy. Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri 2019.
[10] Guy Gaziv, Roman Beliy, Niv Granot, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. Self-supervised natural image reconstruction and rich semantic classification from brain activity. bioRxiv, 2020.
[11] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. CoRR, abs/1801 1011 0111111139000 .03924, 2018
[12] Ghislain St- ves and Thomas Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1054-1061, 2018.
[13] Tao Fang, Yu Qi, and Gang Pan. Reconstructing perceptive images from brain activity by shape-semantic gan. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13038-13048 Curran Associates, Inc., 2020.
[14] Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, 13, 2019.
[15] K. Seeliger, U. Güçlü, L. Ambrogioni, Y. Güçlütürk, and M.A.J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolmage, 181:775- 785, 2018.
[16] Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstructing natural scenes from fmri patterns using bigbigan. CoRR, abs/2001.11761, 2020.
[17] Kai Qiao, Jian Chen, Linyuan Wang, Chi Zhang, Li Tong, and Bin Yan. Biggan-based bayesian reconstruction of natural images from human brain activity. CoRR, abs/2003.06105. 2020.
[18] Rufin VanRullen and Leila Reddy. Reconstructing faces from fmri patterns using deep generative neural networks. Communications Biology, 2(1):193, May 2019.
10

---

Under review as a conference paper at ICLR 2023
[19] Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang, Zhicheng Jiao, and Xinbo Gao. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. Neurolmage, 228: 228:117602, 2021.
[20] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. CoRR, abs/15 122.990000, 2015.
[21] Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. arXiv preprint arXiv:2210.01769, 2022.
[22] Arash Vahdat and Jan Kautz. Nvac: deep hicrarchical variational autoencoder, 2020.
[23] M A Goodale and D Milner. Separate visual pathways for perception and action. Trends Neurosci, 151):200- January 1992.
[24] Iris A Groen, Edward H Silson, and Chris Baker. Contributions of low- and high-level properties to neural processing of visual scenes in the human brain. Philos. Trans. R. Soc. Lond. Biol. Sci., (72(17 February 2017.
[25] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex, 28(12):4136-44) 4160, December 2018.
[26] Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[27] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric Xing and Tony Jebara, editors, Proceedings of the 31st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278-1286 1286, Bejing, China, 22-24 Jun 2014. PMLR.
[28) Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[29] Nadine Chang, John A. Pyles, Austin Marcus, Abhinav Gupta, Michael J. Tarr, and Elissa M. Aminoff. Bold5000, public fmri dataset while viewing 5000 visual images. Scientific Data, 6(1):49, May 2019.
[30] Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay, A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1): 16-126, Jan 2022.
[31] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing. (30))600 612, 2004.


Reviews:
[{"review_id": "jpBpVWpyAa", "reviewer": "Reviewer_H6z7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel method for decoding images from FMRI, using a hierarchical VAE. First, a VAE is trained on images from imagenet. Then the encoder is removed, and a mapping is learned from the images to the latent variables of the VAE. The VAE is divided into lower and higher level areas (i.e., it's hierarchical), and lower and higher level brain recordings are used to feed into corresponding areas. The results are impressive, and to my knowledge, represent a new state of the art. \n\nAfter reading the other reviews, and the authors' responses to them, I am going to reduce my score from 10 to 8, as they used the test set to set hyperparameters. While they claim that the hyperparameters were not particularly impactful on the results, this is a clearly big mistake in terms of ML practice and should be avoided at all costs. Nevertheless, I believe their results still hold (as they used pretty standard hyperparameters, apparently), and the hierarchical nature of their model clearly makes a difference, and is worth publishing.\n\nAfter reading the response to my reduction in score, I'm not clear that the authors totally understand that hyperparameters should not be tuned on the test set. They seem to think it is ok to tune training hyperparameters using the test set. This is still not the case. Again, I think there is still much to like here.", "summary_of_the_review": "I can find few weaknesses and many strengths in this work. There is novelty (mapping to latents rather than features), a nice comparison between using just V1 and V2, using the dorsal pathway, and using the ventral pathway, as compared to a non-hierarchical model that uses all of the voxels. However, the final paper should include comparisons with each of the three models using the same areas without the hierarchy.\n\nAgain, in summary, I am reducing my score based on the hyperparameter tuning.", "strength_and_weaknesses": "This is an excellent paper, with excellent results. \n\nStrengths:\n\n+ The approach is well-motivated by reference to lower and higher brain areas corresponding to lower and higher layers of the decoder.\n\n+ Although the above point has been made before, the novelty here is that the decoder is used to map to the latent variables at different levels of the hierarchical VAE, rather than to the feature layers of a CNN.\n\n+ The results are very strong, both quantitatively and qualitatively.\n\n+ The direct comparison to a factored model with the ROIs randomly assigned to the hierarchy is a nice demonstration that it is the hierarchy that matters, not the division into regions. \n\nWeaknesses, with concrete, actionable feedback\n\n- There are a few things that could be corrected easily, in terms of typos, etc. See below.\n\n- why didn't you include LOC in the Ventral pathway model?\n\n- The poorer results based on mapping all of the voxels to one latent might be a result of the LOC voxels being noisier. A fairer comparison would be to create three more baseline models, each one using the same areas as the three hierarchical models, but without the hierarchy. This would make a stronger argument for the approach. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_ymuf09B7l", "reviewer": "Reviewer_sMBW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors developed a natural images reconstruction model from fMRI with hierarchical variational autoencoders. Their model achieved better reconstructions compared to the state of the art and our ablation study indicates that the hierarchical structure of the latent space is responsible for that performance.\nThe main contributions are as follows: 1. The paper leveraged the hierarchical manner of human brain during visual stimuli and proposed a novel architecture with hierarchical variational autoencoders to reconstruct neural images. 2. Their architecture replicates the natural hierarchy of visual information processing in the latent space of a variational model. 3. Their experimental analysis suggests that hierarchical latent models provide better priors for decoding fMRI signals.", "summary_of_the_review": "The work applies an existing method to an existing task. The novelty this paper is modest but easy to understand. My major concern is the flaw with respect to data using exists which leads to unbelievable experimental results. Besides, the writing of this paper is terrible. I think the quality of the paper is not up to the average quality of the papers accepted by ICLR. I suggest the authors should change their experimental data and carefully revise the manuscript.", "strength_and_weaknesses": "Strength:\n1. This is the first approach that uses HVAEs in the context of neural decoding.\n2. The proposed method combines decoding models with neuroscience priors and the motivation makes sense.\n3. The experiments are relatively sufficient.\n\n\nWeaknesses:\n(major)\n1. Experiments have serious data leakage problem. The HVAE were pretrained in ImageNet and the dataset GOD you used is still collected based on ImageNet. Accordingly, I think the results of this paper is unbelievable.\n2. Although authors has pointed out why they only used GOD dataset, there is still a lot of preprocessed datasets that can be used such as BOLD 5000 and NSD. NSD dataset is collected based on MS-COCO which can avoid aforementioned problem.\n3. The compared methods are not comprehensive. I suggest that the authors add the method proposed in \"Mind Reader:\nReconstructing complex images from brain activities\"  in which the reconstruction quality is relatively better than the compared methods you have chosen.\n4. The article introduced less content about HVAE, formula (1) is even traditional VAE, and used a lot of space introducing the background and evaluation indicators of neuroscience.  By the way, formula (1) and formula (2) needn't two labels.\n\n\n(minor)\n1. The writing of this paper is terrible.  A major refine of this manuscript is necessary.  Informal expressions such as \"{fMRI,Image} pairs\" and \"Naturally, if our fMRI data are more fine grain, we can add additional latent layers\" should be avoided.  Typos such as text in line 3 of page 5 should be corrected. In the experimental results, text wrapping charts is not acceptable for readers.  The structure of the article needs a careful adjustment such as the last two paragraphs of 3.1 should be placed into experimental details.\n2. The main results part should contain a series of subtitle for easy understanding.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9ki3-KFKl9", "reviewer": "Reviewer_J68Z", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study analyzes existing functional magnetic resonance imaging (fMRI) data from human participants that passively viewed a series of images. The analysis is based on using hierarchical variational autoencoders and then using fitting to map the activations of units in the model to the activation of voxels in order to reconstruct the images. Compared to a few baselines, the authors claim to obtain better qualitative results in reconstructions and better quantitative results in discriminating images across trials without cross-validation across images.", "summary_of_the_review": "This is an effort to try to reconstruct images from fMRI data. Although the proposed method performs better than the baselines suggested by the author, it is unclear whether this can really extrapolate and provide any real insights about the underlying representations", "strength_and_weaknesses": "Strengths\n\nIt is interesting to link neural representations to machine learning representations, if done correctly. \n\nWeaknesses\nThe title is somewhat misleading. While the word “imagery” can be used to denote the processing of images, a lot of people in the community have used the word to denote visual imagination, which is not what this study is about. The authors briefly mention decoding dreams and such in the conclusion as potential future work, but this is not what is shown here.\n\nIt is unclear what the goal of the algorithm should be. One goal could be to attempt to reconstruct images from fMRI voxel activity. The qualitative comparison of perhaps cherry-picked examples is not a convincing result and certainly not a rigorous way of presenting results. The authors present some quantitative results in Fig. 4, but they do not define what 2-way, 5-way, and 10-way represent. More critically, why is it relevant to reconstruct images? The purpose of the visual system is to extract useful information, not to reconstruct images. One should get a much better reconstruction from the activity of neurons in the retina. Whatever cortex does, it is not reconstruction. \n\nA central issue is to understand what goes into the training set and what goes into the test set. The first question is whether any of the results represent cross-validation across images. My understanding is that the authors are only using cross-validation across trials, which is not very interesting. Any powerful enough model can memorize the dataset. In this case, the only question is how consistent the data are across trials. If this is the main point, it would be useful to show the degree of consistency in the data across trials, showing the actual data across repetitions and measures of self-consistency. \n\nA different question, and a more interesting one is to perform cross-validation across very different images. In this case, there are a lot of questions that should be addressed. First, are the images in the training set and test set from the same categories (i.e., same imagenet labels)? How different are the images in the training set and the test set? How well can simple classifiers match the images in the test set to those in the training set? These are critical questions to assess what the models are doing or trying to do.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-J2IXddOTu", "reviewer": "Reviewer_eDwx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors used a hierarchical VAE to learn representations from natural images and leverage their latent space hierarchy to learn voxel-to-image mappings. They showed that mapping V1/V2 responses to the early layer and higher visual areas to the deep layer of the latent hierarchy allow them to achieve superior image decoding performance than state-of-the-art approaches. ", "summary_of_the_review": "This paper achieved the best performance in image decoding based on fMRI signals. The motivation is sound, the idea obvious. The technical innovation is minor.  it would become the new baseline for future algorithms to compare against. The comparison between V1/V2, ventral and dorsal streams are potentially problematic and incomplete. ", "strength_and_weaknesses": "Strength: Their image decoding results are very impressive, far better than state-of-the-art.\nWeakness: The insights are perhaps obvious, and the technical innovation is relatively minor. \n\nThe comparison between V1/V2 versus ventral and dorsal is not completely fair and problematic.  These comparisons are confounded with different numbers of parameters, and thus the conclusion about the contribution of the different visual areas, ventral and dorsal streams are  potentially problematic.  It seemed that the more neural responses are used, the best would be the performance. For instance, if they include V3 in the ventral stream's model, would  the performance be even better?   To their credit,  they did show that the hierarchy is important, and their argument that the critical comparison is within the naive baseline which has the same number of parameters is not unreasonable. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject