MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity
Authors: Lynn Le
Paolo Papale
K. Seeliger
Antonio Lozano
Thirza Dado
Feng Wang
Pieter R. Roelfsema
Marcel van Gerven
Yağmur Güçlütürk
Umut Güçlü
Venue: NeurIPS 2024 poster (2024)
Keywords: space-time dependent reconstructions
biological constraints
neural decoding
deep neural network
receptive fields
visual cortex
multi unit activity
convolutional model
brain image reconstruction
naturalistic stimuli
Primary area: neuroscience_and_cognitive_science
TL;DR: This study uses a CNN to reconstruct images from macaque brain signals and explores how different brain areas process these signals at different timepoints.

Abstract: In this paper, we reconstruct naturalistic images directly from macaque brain signals using a convolutional neural network (CNN) based decoder. We investigate the ability of this CNN-based decoding technique to differentiate among neuronal populations from areas V1, V4, and IT, revealing distinct readout characteristics for each. This research marks a progression from low-level to high-level brain signals, thereby enriching the existing framework for utilizing CNN-based decoders to decode brain activity. Our results demonstrate high-precision reconstructions of naturalistic images, highlighting the efficiency of CNN-based decoders in advancing our knowledge of how the brain's representations translate into pixels. Additionally, we present a novel space-time-resolved decoding technique, demonstrating how temporal resolution in decoding can advance our understanding of neural representations. Moreover, we introduce a learned receptive field layer that sheds light on the CNN-based model's data processing during training, enhancing understanding of its structure and interpretive capacity.

MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity
Lynn Le', Paolo Papale 2, Katja Seeliger', Antonio Lozano², Thirza Dado',
Feng Wang?, Pieter Roelfsema 2,4.666 Marcel van Gerven', Yağmur Güglütürkk' Umut Güçlüt!
Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands Netherlands Institute Neuroscience, Amsterdam, Netherlands
Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
Centre for Neurogenomics and Cognitive Research, Vrije Universiteit, Amsterdam, Netherlands Institut de la Vision, Paris, France
Amsterdam University Medical Center, Amsterdam, Netherlands
$Correspondence Email: L.guclu@donddrs.nuu donders.uunn
Abstract
In this paper, we reconstruct naturalistic images directly from macaque brain sig- nals using convolutional neural network (CNN) based decoder. We investigate the ability of this CNN-based decoding technique to differentiate among neuronal populations from areas V1, V4, and ITT, revealing distinet readout characteristies for each. This research marks a progression from low-level to high-level brain signals, thereby enriching the existing framework for utilizing CNN-based decoders to decode brain activity. Our results demonstrate high-precision reconstructions of naturalistic images, highlighting the efficiency of CNN-based decoders in ad- vancing our knowledge of how the brain's 's representations translate into pixels. Additionally, we present a novel space-time tinee-elveerre decoding technique, demon- strating how temporal resolution in decoding can advance our understanding of neural representations. Moreover, we introduce a learned receptive field layer that sheds light on the CNN-based model's data processing during training, enhancing understanding of its structure and interpretive capacity.
Introduction
Artificial neural network models designed for decoding naturalistic images from neural activity signals significantly advance our understanding of how visual information is processed in the brain. Decoding models aim to disentangle patterns of neural responses to different stimuli, offering insights into how visual stimuli (e.g., brown horse or a white t-shirt) are represented by neural populations. Leveraging a large amount of naturalistic data facilitates the reconstruction of natural vision and allows for comprehensive analyses of visual features. Due to the immense variety of naturalistic visual space, reconstructing such stimuli from neural activity is considered the most challenging but also the most intriguing problem in neural decoding.
Convolutional neural networks (CNNs) have recently become cornerstone in neural decoding and encoding models. Their ability to study fundamental features carried by populations of neuronal signals has led significant advancements in understanding the computational mechanisms of natural scene perception. Encoding models provide valuable insights into how brain activity changes in response to stimuli, showcasing the features of CNNs that predict neural responses within specific
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

brain regions. Conversely, decoding models reveal the information content within brain areas without making assumptions about the representation of that information.
Despite the prowess of CNNs in distinguishing between different brain signal patterns, their ef- fectiveness in decoding shifts in neuronal activity remains limited. This limitation is crucial lack of variance in reconstructed images with changing brain signals suggests disconnect between the models and the brain's 's interpretive functions. This performance gap could stem from intrinsic model limitations or data distribution issues, Refining these methods is essential to enhance our understanding of how closely decoding models can approximate the complex processes underlying visual perception.
While CNNs engineered to handle complex, multi multi-dimennional data, including spatial (height and width) and depth (color) dimensions, there remain significant distinctions between CNN processing and human brain information processing. To address these challenges, we have developed fully convolutional decoding model trained from scratch using the THINGS dataset a highly diverse collection of naturalistic stimuli. This dataset enriches the model's exposure varied features, which is crucial for interpreting brain representations accurately.
In this paper, we present the following contributions:
We propose a CNN-based decoder trained to investigate the importance of spatial and temporal information carried in neuronal signals for high-fidelity visual reconstructions.
Homeomorphic decoder:
End-to-end inverse retinotopic mapping: We integrate an interpretable layer in the model, known as the end-to-end inverse retinotopic mapping. This layer dynamically learns to map brain signals to 2D image during training. The adaptive mechanism of this layer, influenced by the entire learning process, allows the decoder to organize its own input spatially.
Model inference and analysis: By performing model inferences with truncated brain data, our approach dissects how the network reorganizes its weights based on spatially separated brain regions for reconstructions. This method aligns with known neuroscientifie principles, enhancing the interpretability of decoded features.
Temporal dynamics: Our model incorporates specific time intervals for neuronal signal input, aligned with latency periods observed in the ventral visual pathway (VI to IT). This temporal aspect allows for deeper analysis of how visual processing evolves over time.
The remainder of this paper structured as follows. Section reviews related work in neural decoding using CNNs, highlighting advances and existing challenges. Section X details the materials and methods, including data acquisition, preprocessing, and the model architecture. Section 4 presents the results and discussion, evaluating the performance and interpretability of our model. Finally, 3 concludes the paper and suggests avenues for future research.
2 Related work
Deep neural networks (DNNs) and generative adversarial networks (GANs) have recently achieved notable success in decoding visual information from brain activity, particularly using fMRI data 12 6 g. Seeliger et al. (2018) (5] utilized GANs to reconstruct grayscale images and handwritten characters from fMRT data, demonstrating the versatility of adversarial methods. Nishimoto et al. (2011) 1 used voxel-wise modeling to reconstruct complex video stimuli from brain activity, focusing on capturing detailed neural responses in the visual cortex. Shen et al. (2019a) [8] further advanced this by training end-to-end models for image reconstruction, incorporating high-level feature losses into the GAN framework.
Reconstructing dynamic stimuli like videos presents additional challenges. Han et al. (2019) 6 used variational auto-encoders for video reconstruction, achieving low-level property reconstructions but struggling with finer details. This highlights the complexity of decoding dynamic visual information.
CNNs are effective for neural decoding due to their capability to process complex, multi-dimensional data. Sarraf and Tofighi (2016) [9] treated fMRI slices as separate images, but this method was limited by noise and did not respect neural topography. Approaches using 3D convolutions to preserve spatial

---

structure [TO], and geometric deep learning on cortical meshes promise in capturing brain features more effectively.
I
12
13
14
15
have shown
Decoding naturalistic stimuli using large datasets pushes conventional model boundaries 16
17
Le
al. (2022) 177 reconstructed images and videos from fMRI data by converting voxel responses into 2D representations aligned with the visual field, then applying fully convolutional networks trained with VGG feature loss and adversarial regularizers. This method showed significant improvements over previous techniques.
Our work uses multi-unit activity (MUA) data from Utah electrode arrays in the macaque visual cortex (V1, V4, IT), which offers higher temporal resolution and captures fine-graid seee-graind neuronal activity compared to fMRL. Bashivan al. (2019) [8] integrated receptive field concepts into neural network models for better interpretability. Building on this, we incorporate an end-to-end inverse retinotopic mapping layer within convolutional decoder. This layer dynamically maps brain signals to 2D images, improving spatial feature organization and providing deeper insights into neural processing.
3 Material & methods
3.1 Data
We used images from the THINGS database [19], containing high-resolution images across various object categories. Each image had three color channels (RGB) and was resized to 96 x 96 pixels to meet model input requirements and reduce computational complexity. Images were presented the lower right quadrant, shifted 150 pixels right and down from the central fixation point.
passive fixation task was conducted with a 7-year-old male macaque (Macaca Mulatta) across 22,348 trials (22,248 training and 100 test trials). The macaque maintained fixation on a central dot while images, presented in randomized sequence of four per trial, were displayed for 200 ms with 200 ms gray screen interval. The macaque was rewarded with juice for maintaining fixation. Ethical approval was obtained from the Royal Netherlands Academy of Arts and Sciences, adhering the NIH Guide for the Care and Use of Laboratory Animals, ensuring the macaque's well-being and minimizing stress.
MUA was recorded from 15 Utah electrode arrays implanted in V1 (7 arrays). V4 (4 arrays), and IT (4 arrays), capturing neural activity ms resolution over 200 ms per trial. Electrodes were selected using a self-correlation reliability score with threshold of 0.4, reducing the original 1024 electrodes 576 (including losses from broken electrode). Neural responses were z-scored (mean subtracted, divided by standard deviation). Time windows were defined as 0-125 ms for V1, 150 ms for V4, and 50-175 175 ms for IT. 25 ms smoothing window was applied, and data were temporally downsampled to either 8 Hz (averaged over 125 ms) or 40 Hz, ensuring consistent and normalized data for modeling.
3.2 Models
In this section, we describe the models used for decoding the neural responses into visual stimuli. The main model is homcomorphic decoder, and compare its performance against a baseline decoder. Additionally, we employ a discriminator to facilitate adversarial training.
3.2.1 Homeomorphie decoder
The homeomorphic decoder transforms neural responses into retinal embeddings and subsequently reconstructs the visual stimuli from these embeddings. The architecture leverages several neural network components, including pre-trained and end-to-end trained models.
Pre-trained inverse retinotopic mapping The first variant of our homeomorphic decoder uses pre-trained CNN to perform inverse retinotopic mapping. The model projects neural responses onto retinal embeddings using learned weights and subsequently reconstructs the visual image from these embeddings. The process involves two types of embeddings: spatial embeddings and spatiotemporal embeddings.
3

---

For spatial embeddings, the retinal embedding E is computed directly from the neural responses at single timepoint:
Easy
Σ
Te W
cay
cEMEA
where represents the neural responses over 576 electrodes at single timepoint, and W eay are the learned spatial weights for mapping the neural responses to retinal embeddings for each microelec- trode array (MEAg).
For spatiotemporal embeddings, the retinal embedding computed from the neural responses over multiple timepoints:
Eigyy
Rot W eay
EEROL
where represents the neural responses over 576 electrodes across 5 timepoints, and W Wezy are the learned weights for mapping the neural responses to retinal embeddings for each region of interest (ROI)).
The weights W are optimized by minimizing the following objective function:
W a
min Irre - fell2 + WillW - >2 + ||a]ß) + λada
W.a
where P. = -. Otec Eae Eay W Wery (S) is the estimated neural response with / representing acti- vations from a pre-trained Inception v1 network and Aw is the Laplace operator. Here, W can be considered the spatial weights interest and a the feature weights. The specific layers of the network used for different cortical areas are: conv2d2 for VI, mixed4a for V4, and mixed4d for IT. These embeddings leverage the pre-trained network to efficiently map neural activity patterns to their corresponding visual stimuli representations, forming a crucial component our decoder.
End-to-end trained inverse retinotopic mapping This variant of our homeomorphic decoder involves end-to-end training of the inverse retinotopic mapping from neural responses R to retinal embeddings E., allowing the model to learn optimal parameters directly from data without pre-trained CNN features.
The inverse receptive field (see Figure 6) computes E as:
Σ Te exp (-((s Te )
20 (-((=
(y
yc)
Eazy
20
cEMEA.
where Te represents the neural responses, te and Ye denote the spatial coordinates of electrode e, and ae represents the standard deviation parameter that determines the spatial spread of the receptive field associated with each electrode. The parameters Te, Ye, and d. are learned during training.
Once Rs is mapped onto E., pixel-to-pixel U-Net reconstructs the stimulus Stake U U_Net( E.). where Sjeal is the ground-truth stimulus. Loss components are functions Seeal and Stae. incorporat- ing E, into the objective function.
This end -ndd approach adapts to the specific characteristics neural responses and visual stimuli, improving reconstruction performance.
Pixel-to-pixel mapping The final component of our homcomorphic decoder employs U-Net architecture designed for pixel-to-pixel mapping of retinal embeddings to visual stimuli. The U-Net model a powerful neural network architecture commonly used for image segmentation tasks due to its ability to capture both local and global image features through its contracting and expansive paths connected via skip connections. We are using standard U-Net architecture; for details refer to Appendix A.I
3.2.2 Baseline decoder
We employ baseline decoder as reference for performance evaluation. This simpler model transforms neural responses directly into visual stimuli, without the use of intermediate retinal embeddings. adopts modified U-Net architecture, retaining only the expansive path from the homeomorphic decoder while omitting the contracting path and skip connections. The input is

---

576 X tensor representing neural responses from 576 electrodes at a single timepoint, and the output a RGB x 96 96 visual stimulus. Despite its simplicity, the baseline serves as crucial benchmark for assessing more complex models like the homeomorphic decoder. This approach is inspired by [8] which is regarded as a state-of-the-art reconstruction model 20
3.2.3 Discriminator
The discriminator plays an integral role in the training process by differentiating between real and fake visual stimuli, thus facilitating adversarial training. We employ a modified U-Net architecture, using only the contracting path, to serve as our discriminator. The input to the discriminator the visual stimulus, S, of dimensions RGB x 96 x 96. The output is scalar probability p, indicating the likelihood that the given image is real. Adversarial training, where the discriminator aims to distinguish real images from reconstructed images, drives the decoder to generate more realistic and accurate visual stimuli. The loss calculated from the discriminator's assessments is crucial for improving the fidelity of the decoded images.
3.3 Training
This section outlines the optimization procedures, loss functions, and strategies used to train the decoders and the discriminator for high-fidelity reconstruction of visual stimuli from neural responses. Source code is available on our GitHub repository!
3.3.1 Training parameters
The dataset comprised 22,348 training samples and 100 test samples, which were exclusively used for testing and never during training. We used the Adam optimizer with learning rate of 0.002 and beta coefficients of 0.5 and 0.999 to ensure convergence. The loss function included discriminator loss (Adiser) at 0.01, VGG feature loss (Bvgg) 0.9, and LI pixel-wise loss (3pix) 0.09 balance sensitivity. Training spanned 50 epochs on a Quadro RTX 6000 GPU, utilizing approximately 10,000 MiB of GPU memory.
3.3.2 Decoder training
The training of the decoders (both homcomorphic and baseline) involves a combination of losses designed to ensure realism and accuracy in the reconstructed images. The adversarial loss encourages the decoder generate realistic images that the discriminator cannot distinguish from real images. This loss is defined as the binary cross- cross-entropy loss between the discriminator's 's output and the true labels (1 for real images and for generated images):
Lady
Es... [lgg (Siea)) Es... [ls( D(S
To further enhance the quality of the reconstructed images, we use a feature matching loss based on the activations of a pre-trained VGG-19 network. The feature loss is the mean squared error (MSE) between the feature representations of the real and generated images various layers (conv1_2, conv2_2, conv3_4 conv4_4 4, and conv5_4) of the VGG-19 network:
Lieat Σ (Srea)) - ci (Stake)
where &i denotes the feature map at layer of the VGG-19 network. The pixel-wise loss ensures that the reconstructions are close to the original images in pixel space. We use the mean absolute error (MAE) to quantify this loss:
Lpixel
||Sreat Stakelli
The total loss for the decoder is weighted sum of the adversarial loss, feature loss, and pixel loss:
Ldecoder Cadv AfeatCreat + Cpixel
where Aadv, Areats and Apixel the weights for each respective loss component.

---

3.3.3
Discriminator training
The discriminator is trained to distinguish between real and generated images, using the adversarial loss defined as the binary cross-entropy loss:
discriminatoe -is Dog ((Sra))) Se [lg(( (SStake))
3.3.4 Optimization strategy
The decoder and discriminator are optimized using the Adam optimizer with default parameters (learning rate 0.0002, 81 = 0.5, B2 0.999). Early stopping is applied based validation set performance to prevent overfitting. An image buffer maintains a history of generated images to stabilize adversarial training by varying the discriminator's inputs, During training, the decoder and discriminator are updated alternately: 1) the discriminator using adversarial loss, and 2) the decoder using the total loss (adversarial, feature, and pixel losses). Training continues until convergence, monitored by evaluation metrics.
3.4 Evaluation metrics
To robustly evaluate the performance of decoding models, employ variety of metrics that assess the accuracy and quality of the reconstructed visual stimuli from both feature-wise and perceptual quality perspective.
Feature correlation One of the primary metrics used to evaluate stimulus reconstruction is the feature correlation between the reconstructed images and the original images. We use pre-trained AlexNet, extracting feature representations at various layers (conv1, conv2, conv3, conv4, conv5, fc6, fc7, fc8). The Pearson correlation coefficient is calculated between corresponding feature maps of the original and reconstructed images: Pa: = Pearson Φi (Sret), , di (S(tae))) where de denotes the feature map at layer I of AlexNet. High correlation values indicate that the reconstructed images capture similar feature representations as the original images. This standard metric for evaluating reconstruction quality was also used by Le al. (2022) [17.
Image colorfulness To evaluate the perceptual quality of the reconstructed images. we use the Hasler and Süsstrunk colorfulness metric [21). This metric quantifies the colorfulness of image, which is an important aspect of human visual perception. The metric is computed as: C' = Org 0.3prg where Org and Hrg are the standard deviation and mean of the color difference vector rg = (R G) across the image.
Occlusion analysis We performed spatial and spatiotemporal occlusion analyses to assess the contributions of different spatial and temporal regions to reconstruction accuracy. In spatial occlusion, parts of the input are systematically removed identify which regions are most critical for decoding. In spatiotemporal occlusion, specific segments of neural responses are occluded to evaluate the importance of different timepoints and spatial regions the reconstruction process.
Results and discussion
4.1 Stimulus reconstruction
The performance of different decoding models reconstructing visual stimuli from neural responses is evaluated across multiple dimensions, including model comparison, spatial occlusion analysis, and spatiotemporal occlusion analysis.
4.1.1 Model comparison
We compared the reconstruction performance of three variations of our homeomorphic decoder spatial, spatiotemporal, and end-to-end inverse retinotopic mapping against the baseline model, both qualitatively and quantitatively.
presentsss qualitative results. The spatial, spatiotemporal, and end-to-end decoders consistently outperformed the baseline, better preserving textures, shapes, and colors.

---

Stimuli
Spatial
Spatiotemporal
End-to-end
Baseline
Figure 1: Sample stimuli and corresponding reconstructions from models. The "Spatial" and "Spatiotemporal" column show results from the pre-trained inverse retinotopic mapping model, explained in Section 3.2.1 The "End-to-end" column shows reconstructions from the space-resolved model with a component that learns the neuron's receptive field explained in Section 3.2.1 3.2.1 shows the reconstructions of model we implemented explained in Section 3.2.2
"Baseline"
Table I provides quantitative comparison using Pearson correlations between reconstructed and original images across different AlexNet layers (Section 3.4 The spatiotemporal decoding model achieved the highest feature correlation across most layers, highlighting its superior ability to capture and reconstruct the features present in the original stimuli. Specifically, the spatiotemporal model excelled in the deeper layers cf and f fc8), which capture high-level feature representations, demonstrating the model's capability to capture both fine-graincd and abstract features encoded in the neural responses.
Table Feature correlations of reconstructions with original images across AlexNet layers.
Spatial
Spatiotempral
Baseline End-to-end
conv
0.358
0.372
0.348
0.267
conv2
0.320
0.334
0.303
0.221
conv3
0.429
0.443
0.407
0.326
conv4
0.385
0.401
0.369
0.316
conv5
0.292
0.318
0,282
0.203
FC6
0.344
0.377
0.325
0.235
FC7
0.534
0.579
0.541
0.434
FC8
0.579
0.610
0.543
0.446
The learned receptive field layer adapts receptive field sizes by spatial location, with larger fields in the periphery and smaller fields centrally (Figure 7 Despite some receptive fields being smaller than one pixel, all electrodes still contribute information, with single pixels used when needed. This effect likely due to the model's s limited 96 96 field view, constraining pixel allocation for very small fields.
We also ran model ablations assess the effect of different loss functions (discriminator, pixel, and VGG loss) on performance (Figure ITp. Additionally, we trained models on region-specific data (VI, V4, IT) to explore how brain region training affects reconstruction (Figure [2] These experiments highlight the importance of individual brain regions and the adaptability of models trained on occluded data.
4.1.2 Spatial occlusion analysis
Spatial occlusion analysis was conducted to identify the importance of different brain regions (V1, V4, IT) in the reconstruction process. This analysis involved occluding specific spatial regions the neural response inputs and examining the effect on the quality of the reconstructed images.
7

---

Stimuli
VI+V4+IT
VI
V4
IT
Spatial occlusion analysis of spatial model explained in Section column means included brain region.
Figure 2:
Title above
Figure 2 provides visualizations of example stimuli and their corresponding reconstructions when inputs from different brain regions (V1, V4, IT) were selectively occluded. The columns represent the regions of interest, while the rows present different example stimuli. For each region, the neural responses were set their baseline values (pre-stimmg ppre-stimulus onset) for the occlusion procedure.
To quantify the impact of occlusion on reconstruction quality, feature correlations with the original images were calculated using the pre-trained AlexNet, shown in Figure The bar plot depicts feature correlations across different AlexNet layers for reconstructions derived from V1, V4, ITT, and all regions combined.
4.2 Spatiotemporal occlusion analysis
Spatiotemporal occlusion analysis evaluates how neural responses from different time windows contribute to reconstruction quality, offering insights into the brain's temporal processing of visual stimuli.
Figure 3 shows example stimuli and their reconstructions when multiple time windows are occluded, with only one time window (highlighted in yellow) being included in the model during inference. Each column represents a different set of time windows being occluded, with the first column showing reconstruction using all time windows.
The colorfulness of reconstructions from V1, V4, and was evaluated using the Hasler and Süsstrunk metric. Figure shows the colorfulness scores for each brain region and combined data. V1- constrained reconstructions had the highest colorfulness scores and correlated strongly with early AlexNet layers (conv1, conv2), reflecting VI's role in processing basic features like edges and color. IT reconstructions, however, showed higher correlations with deeper AlexNet layers (fc7, fc8), which capture more abstract visual features.
These results highlight the hierarchical nature of visual processing, with V1 specializing in basic visual attributes and IT handling more complex features.
Conclusion
We presented a comprehensive approach to decoding naturalistic visual stimuli from neural responses using fully convolutional neural network trained from scratch. The use of the THINGS dataset enriched our model's feature set, crucial for accurately interpreting brain representations. Our homeomorphic decoder, enhanced with an end-to-end inverse retinotopic mapping layer, effectively integrates spatial and temporal information, leading to high-fidelity and interpretable reconstructions. Our evaluations highlighted the spatiotemporal decoding model's superior performance, evidenced by high feature correlations with deep layers of pre-trained AlexNet. Spatial and spatiotemporal

---

TW-15
TW-3
TW-3
TW-3
TW-3
TW-3
V44
Fast Sass
Torpet
Out
Out
Figure 3: Spatiotemporal occlusion analysis. Yellow indicates the active time window, with others occluded.
Colorfuless ROI constrained features
30
0.095
Distribution
Mean
25
0.062
IT
15
10-
0.05
0.10
0.15
0.20
1.25
0.30
Figure 4: Distribution of colorfulness metrics across V1, V4, and IT-constrained reconstructions, calculated using the Composite Colorfulness Score (CCS) based on RGB channel differences.
occlusion analyses provided insights into the contributions of different brain regions and time windows, affirming the hierarchical nature of visual processing. The end-to-end inverse retinotopic mapping facilitated accurate estimation of receptive fields, aligning with neurophysiological findings and enhancing model transparency. This work advances neural decoding by offering a scalable and interpretable framework for reconstructing high-quality images from brain signals. Future research will explore more sophisticated architectures, further integrating temporal information and applying this framework to other sensory modalities.
Broader impact
Neural decoding models reconstructing naturalistic images deepen our understanding of the link between neural activity and perception, with promising applications in visual neuroprosthetics. This study specifically reuses data originally collected as part of initiative aimed at restoring sight, adhering to ethical best practices by maximizing insights from single dataset and minimizing

---

the need for additional animal studies. This approach aligns with responsible research standards, balancing innovation with animal welfare.
However, caution is needed when applying these models to human neuroprosthetics due to the complexity of human brain activity and behavior. Human behavior encompasses interactions with the environment, complex motor actions, and neuroplasticity, elements that models trained exclusively on data from static image viewing may not fully capture. For instance, visual feedback from motor activities adds layers of complexity beyond current model capabilities. Additionally, using these models to identify stimulation sites for neuroprosthetics may not replicate natural neural responses precisely, highlighting the need for experiments/clinical trials with human subjects, nuanced model development and careful interpretation of outputs.
Limitations
This study applies invasive MUA recordings in macaques, expanding on prior work with fMRI by offering higher signal quality and detail. However, the applicability of these results non- invasive techniques like fMRI remains limited due to their lower signal-to-noise ratio and less detailed recording capabilities compared to electrode arrays. This distinction is important, as invasive neuroimaging remains rare in human research.
Additionally, transitioning this framework human intracranial applications poses challenges, including potential scar tissue formation, immune responses, and device rejection over long-term recordings, Anatomical differences, such vascular structures, may further impact device placement and stability. Future work could explore these adaptations for broader applications, including brain- computer interfaces (BCI) and neuroprostheties for individuals with acquired blindness, for which, careful regulatory guidance and additional research will be essential.
Acknowledgments
This work was supported by three grants of the Dutch Organization for Scientific Research (NWO): STW grant number P15-42 'NESTOR', ALW grant number 823-02-010 and Cross-over grant number 17619 'INTENSE' and grant number 124.005.022 'DBI2', a Gravitation program of the Dutch Ministry of Science, Education and Culture; the European Union's 's Horizon 2020 research and innovation programme: grant number 899287, 'NeuraViper'; the Human Brain Project, grant number 650003.
References
[1] Shinji Nishimoto, An T Thomas Naselaris, Yuval Benjamini, Bin and Jack Gallant. Reconstruct- ing visual experiences from brain activity evoked by natural movies. Current Biology, 21(11):1641-166 101
[2] Yagmur Güęllüüürk, Umut Güşlüü Katja Seeliger, Sander Basch, Rob van Lier, and Marcel van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. Advances in Neural Information Processing Systems, 30, 2017.
[3] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1):15037, 2017.
[4] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral Coriex, 28(12)4113-4166, 2018.
[5] Katja Seeliger, Umut Güçlü, Luca Ambrogioni. Yagmur Güglütürk, and Marcel Van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neuroimage, 181:775-785, 2018.
[6] Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Vari- ational autocneoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex. Newroimage, 198: 136, 2019.
[7] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLoš Computational Biology. 150))e 2019.
10

---

[8] Guohua Shen, Kshitij Dwivedi, Kei Majina, Tomoyasu Horikawa, and Yukiyasu Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computtatioaa Neuroscience, 13:432276, 2019.
[9] Saman Sarraf and Chassem Tofighi. Classification of Alzheimer's s disease using fMRI data and deep learning convolutional neural networks. arXiv preprint 1r33v:1003.06633 2016.
[10] Karl Bäckström, Mahmood Nazari, Irene Yu-Hua Gu, and Asgeir Store Jakola. An efficient 3D deep convolutional network alzheimer's discase diagnosis using MR images. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 149-153. 153. IEEE, 2018.
[11] Bruce Fischl and Martin Sereno. Microstructural parcellation the human brain. Neuroinage, 182:219- 231, 2018.
[12] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5115-5122 2017.
[13] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. arXiv:1801, 100.0 2018.
arXiv preprint
[14] Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. SplineCNN: Fast geometric deep learning with continuous b-spline kernels. In Proceedings of the IEEE Conference Computer Vision and Pottern recognition, pages 869-877, 877, 2018.
[15] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch- Gordan nets: fully fourier space spherical convolutional neural network. Advances Neural Information Processing Systems, 31, 2018.
[16] Yunpeng Bai, Xintao Wang. Yan-pei Cao, Yixiao Ge, Chun Yuan, and Ying Shan. Dreandiffusion: Generating high-quality images from brain EEG signals, arXiv preprins arXiv:2306. 16934, 2023.
[17] Lynn Le, Luca Ambrogioni, Katja Seeliger, Yağmur Güęlüüürk, Marcel van Gerven, and Umut Güçlü. Brain2pix: Fully convolutional naturalistic video frame reconstruction from brain activity. Frontiers in Neuroscience, 16, 2022.
[18] Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364464399 2019.
[19] Martin N Hebart, Oliver Contier. Lina Teichmann, Adam II Rockter, Charles Zheng, Alexis Kidder, Anna Corriveau, Maryam Paziri-Pashkam, and Chris Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife, 12:c82580, 2023.
[20] Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C Aoki, Kei Majima, Yusuke Muraki, and Yukiyasu Kamitami. Spurious reconstruction from brain activity. arXiv preprint arXiv:2405. 200002405.00078, 2024.
[21] David Hasler and Sabine Suesstrunk. Measuring colorfulness in natural images. In Human Vision and Electronic Imaging VIII, volume 5007, pages 87-95. SPIE, 2003.

---

A Appendix Supplemental material
A.1 U-net architecture
The contracting path follows the typical architecture of a convolutional network. It consists of repeated application of two 3 convolutions (unpadded convolutions) followed by a ReLU activation and a 2 x 2 max pooling operation with stride 2 for downsampling. At each downsampling step, the number of feature channels doubled.
Contracting path:
Expansive path: Every step in the expansive path consists of an upsampling of the feature map followed by 2 x 2 convolution ("up-convolution") that halves the number of feature channels, concatenation with the correspondingly cropped feature map from the contracting path, and two 3 x convolutions, each followed by a ReLU activation. The cropping is necessary due the loss of border pixels during convolutions in the contracting path.
Skip connections: Skip connections introduced between the contracting and expansive paths to combine low-level features with high-level features, facilitating finer reconstruction of the output image.
The imput to the U-net the retinal embedding E of dimensions 15 96 96 (15 channels and 96 96 spatial resolution) and the output is the reconstructed visual stimulus S of dimensions RGB x 96 x 96. The U-net architecture enables detailed reconstruction by preserving spatial information through its symmetrical structure.
Table 2: U-NET Layers
Layer
Shape
Configurations
Input
96
96
15
Conv2d
48 48 64
kernel sizc=4, stride=2, padding=1
LeakyReLU
48 x 48 x 64
negative_slon 2.1 implace=True
Identity
48 x 48 x 512
Identity parameters: count=512, depth=512
Skip I (Conv2d Cony lranspose2d)
48 x 48 x 512
See detailed breakdown
Skip 2 (Conv2d
Cony 2ranspose2d)
48 x 48 x 256
See detailed breakdown
Skip 3 (Conv2d
+ Conv Transposc2d)
48 x 48 x 128
See detailed breakdown
Skip 4 (Conv2d -> Conv Transpose2d)
48 x 48 x 64
See detailed breakdown
Conv' Trnn/ranppos2d
96 x 96 x 3
kernel size=4, stride=2, padding=1
Sigmoid
96 x 96 x 3
Output
96 x 96 x 3
DISCR
DISCR
Model (Thiining)
U-NET
Scalar
Model (Fixed)
Canzai
Loss Function
Figure 5: Overview of how the main reconstruction model is trained. A. The U-NET component trained with a stack of 2D tensors (illustrated grey) as input. These tensors are processed to produce reconstructions (depicted in yellow). The difference between the reconstructions and the target stimuli (represented in blue) are computed using the adversarial loss, feature loss, and pixel loss. B. Concurrently. the discriminator component undergoes its training phase. It evaluates the reconstructed outputs from the U-NET (labeled as 'fake images') alongside the original target images (labeled as 'real images'). This evaluation plays critical role in calculating the Adversarial Loss, which instrumental in guiding the parameter updates for the U-NET. This synergistic training approach ensures the progressive enhancement of the U-NET's ability to generate increasingly accurate and realistic reconstructed images.
12

---

A
Recan- struciinn
Torgot
Brain resporses
RF activation mags
(R)
(M)
REP
VI
V4
IT
(r))
(Rae))
Raqut
Receptive Field
(rw)
M
(Asa).Mal Meal
(Hage Kapl
Figure 6: A. The inverse receptive field layer produces for each brain response r E RR an RF activation map (M) (also known the embedding layer E) by using the learnable parameters (Pus-yy: σ)) in conjuction with the width (W) and height (H) of the desired model inputs (X) with a 2D Gaussian function. B. Let R be a matrix in R W such that each entry is r. [[] is multiplied element-wite with its corresponding M, and then stacked based on its electrode number, resulting in 15 in total (7 for VI, 4 for V4, and 4 for IT).
Initialized parameters
Learned parameters
[t dize (ploca))
6
Rt vine (plont))
6
Visual filld
0-
Visual field
201
(placls)
(phoths)
VI
V4
IT
Figure 7: The learned 2D Gaussian parameters spatial receptive field maps for mapping the neuronal signals in visual space as input for the reconstruction model. The "Visual field" shows the learned mappings 2D space. The plot adjacent shows the variations in size of these RFs function of distance from the foveal center, highlighting how the learned RFs expands with increased eccentricity.
13

---

AlexNee correlations of ROL-constrained features
0.188
C278
Bosii region
01245
0.391
01112
0.214
10375
11:77
TT-
0.191
0.198
0,264
0.331
0.152
0.18)
0331
0.352
(Tosined)
60
80
100
Relotive corvedaticn (%)
Conv1
Conv2
Conv3
Cocvv
Conv's
FC7 FC8
Figure 8: Relative correlation analysis of roi-constrained reconstructions with AlexNet features. This figure shows the relative correlation coefficients between features from roi-constrained recon- structions (V1, V4, IT) and corresponding AlexNet layers, normalized comparison. Higher relative correlations are indicated by deeper colors and larger bars, marking the roi reconstruction with the closest match to each AlexNet layer's processing characteristics.
brain region for fair
AlexNet correlations of brain features (temporal)
0,162
TW1
0229
0.07
0291
0.248
0.145
0.162
0.24
0L202
Time wimioo
TW3
0154
0.57
0.210
0.185
0.096
0.109
0.266
0.175
Bran
0:372
01354
0.433
04011
0:318
0.377
0.579
0610
20
40
60
80
100
Relative correlation (%)
PCS
Figure 9: Temporal relative correlation analysis across AlexNet layers. This figure illustrates relative correlation coefficients across multiple time windows and AlexNet layers, with color and bar size representing the highest relative (not absolute) correlations per brain region. The x-axis is normalized, allowing direct comparison of relative contributions across time points.
14

---

AlexNet correlations of ROI and time-constrained reconstructions
IT
V1+
0122
0:14
1.30
015
8.30
0.15
K0M
0.23
0:19
Relative cortelation (16)
021
0:23
1.51
1.27
1220
10.11
035
431
60
0.07
8.84
6100
11
0.85
EAIT
6:10
031
0.15
015
3.09
1.15
8.35
9:16
0.33
0.17
0.00
0.16
0.03
0.0
0.18
0.26
52E
OL1T
1.34
021
012
036
831
0:11
8:36
418
1018
0.20
1.2
0.6
112
1131
605
111
811
611
8:2
018
112
611
004
021
8.9
0.5
27
182
60
80
112
1.68
164
99
110
171
197
1:420
73
20
104
9
50
83
111
137
66
02
118
141
170
windows (ms))
Time windows (mnx)
Time windows (us)
Time windows (labele))
Convl
Conv2
Coar3
Conve
Coav5
FC6
FC7
PCS
Figure 10: Correlation of features across time and brain, where is varying timepoints of the initial timewindow for each ROI (V1: 0-27ms, V4: 33-60ms, IT: 66-93ms) after stimulus onset and +26 is the 26 shift of all of the three windows.
Model ablations
Stimuli
no ablation
no discr.
no L1
no VGG
Figure 11: Training model with ablated components.
15

---

Model trained on brain region of interest
Stimuli
VI+V4+IT
V1
V4
IT
Figure 12: Training model on various brain regions.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Figures
2 Table and their corresponding descriptions in the main text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This work does not include or introduce new results of purely theoretical
nature.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Section 3.1
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: A data manuseript for the THINGS macaque visual cortex dataset is currently in preparation (see httpp://things--imtiittts org). The code has been made available
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
guuides/CodeSubiistee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section 3.3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars on the quantitative results in Table are reported because would be too computationally expensive to create them, and this statistic does not impact the demonstrated functionality of the introduced method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 3.3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: All the reported data collection experiments strictly adhered to the local ethical guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The data was collected by authors of this paper. The stimulus dataset publicly available and properly referenced and credited. The models are original work, properly referenced and credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The newly created models and data in this paper documented in all necessary detail.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "AQed7IS8vY", "reviewer": "Reviewer_skQc", "summary": "This paper proposes a CNN-based decoder to reconstruct naturalistic images from macaque brain signals. To this end, the paper presents the Learned Receptive Field (LRF) layer to enhance the reconstruction and understanding of the model's structure and interpretive capacity. Here, the work aims to interpret brain activity by transforming low-level to high-level brain signals. It focuses on the readout characteristics of neuronal populations in areas V1, V4, and IT. The evaluations of the model are performed using the THINGS dataset, and the results demonstrate the effectiveness of an end-to-end model and provide interpretations, such as the crucial role of V1 in color processing.", "questions": "- Could you elaborate more on the motivation behind using Inception v1 for different cortical areas? How does this layer contribute to the generalization of the image reconstruction?\n\n- Is the model generalizable to other types of neural data from different species or different types of recording equipment?\n\n- In equation line 151, how does the standard deviation influence the spatial spread of the receptive field and overall reconstruction?\n\n- In Pre-trained Inverse Retinotopic Mapping, why do we need two embeddings? Can’t the model handle it with a unified spatiotemporal module instead of having separate embeddings?\n\n- What about employing an existing method to build a baseline, such as using a recurrent CNN or Vision Transformer? For a fair comparison, the evaluation should include some of the other state-of-the-art architectures.\n\n- Is the quantitative comparison presented in Table 1 a standard manner of measuring performance? Is it used in the literature? If not, what would be other metrics, such as the R² score?\n\n- What about the ablation analysis of the loss functions? Also, could you present the dynamic criterion of each loss during the training and validation phases?\n\n- How did you set the hyperparameters of the model? Also, the full implementation details of the model are missing.", "limitations": "- I think the main limitation of the paper is the limited area of applicability of the proposed framework.\n\n- The evaluation, both in terms of architecture and criterion, is missing to determine the necessity and effectiveness of all the modules of the model.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The integration of a U-Net architecture for the pixel-to-pixel mapping of retinal embeddings to visual stimuli is intriguing. \n\n- The proposed method is end-to-end with different parameters, making it adaptable to the target problem.\n\n- Though the method of the paper is data-specific, the proposed model’s results are somehow interpretable", "weaknesses": "- The proposed model is data-specific, which could potentially limit its application.\n\n- The overall loss consists of multiple criteria, each with its weight, which might make the model harder to train. Specifically, mixing - adversarial loss with pixel and feature loss can make it challenging to control the training dynamics.\n\n- The paper does not include a complete ablation analysis of the different modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. We appreciate your positive comments and are pleased to hear that you recognized our model's adaptability. Below, we address your points in further detail:\n\n\n**Weaknesses:**\n\n1. **Data-specific model**: While our model was trained on the THINGS dataset, which is extensive and diverse, it has demonstrated robust performance on limited data, which is crucial given the challenges of collecting invasive monkey data. Our model is also generalizable to various types of neural data and recording equipment, similar to earlier models successfully applied to human fMRI data (Le et al., 2022).\n\n\n2. **Loss functions and training dynamics**: Although our model employs multiple loss functions, their combination enhances training stability and performance. We have conducted additional ablation studies, which we will include in the revised manuscript, demonstrating the necessity of each loss component for optimal model performance.\n\n\n3. **Complete ablation analysis**: We acknowledge the importance of comprehensive ablation analysis. We have now performed additional ablation studies, examining the contributions of the discriminator, L1, and VGG losses, and will include these results in the revised manuscript (also see figure 1 provided in rebuttal).\n\n\n**Questions:**\n\n\n1. **Motivation for using Inception v1**: Inception v1 was chosen for its capability to capture features with varying receptive field sizes and complexities, aligning with the hierarchical processing in the ventral visual stream. This alignment allows effective generalization across visual stimuli and provides accurate predictions and meaningful MEIs for neurons in V1, V4, and IT. Previous studies (e.g., Yamins et al., 2014; Khaligh-Razavi & Kriegeskorte, 2014; Güçlü & van Gerven, 2015) have demonstrated the success of task-optimized DNN models in modeling visual areas.\n\n2. **Model generalization**: Our model is designed to be generalizable to different types of neural data and recording equipment, as demonstrated by its successful application to human fMRI data (Le et al., 2022).\n\n3. **Standard deviation in receptive fields**: The standard deviation models the size of the receptive field, directly influencing the accuracy of stimulus reconstructions. Precise modeling of receptive field sizes enhances the quality of the reconstructed images.\n\n4. **Pre-trained Inverse Retinotopic Mapping**: The use of two embeddings corresponds to our model’s variants for spatial and spatiotemporal reconstructions. Each variant requires specific embeddings to optimize performance in their respective domains.\n\n5. **Baseline comparisons**: We currently compare our model against two baselines: the Brain2Pix model (Le et al., 2022) and an improved version of Shen et al. (2019). These models represent the state-of-the-art in the field. We will emphasize this in the manuscript.\n\n6. **Performance metrics**: The metrics presented in Table 1 are standard in the context of image reconstruction and have been used in previous studies. We will clarify this in the revised manuscript.\n\n7. **Ablation analysis of loss functions**: We have now conducted ablation analyses on the discriminator, L1, and VGG losses. The results, showing decreased performance when these components are removed, will be included in the revised manuscript. Additionally, we trained separate models for V1, V4, and IT, in addition to the combined model, and will present these findings.\n\n8. **Hyperparameter settings and implementation details**: Our hyperparameters were set based on existing literature and detailed in section 3.6.4. We used the ADAM optimizer with specified parameters and provided full architecture details in the supplementary material. The training process has proven robust to hyperparameter variations, as confirmed by our pilot experiments. We have corrected the link to the full source code in our official comment to the AC, adhering to NeurIPS guidelines.\n\n\n**Limitations:**\n\n\n1. **Limited area of applicability**: In the revised manuscript, we will discuss potential extensions and adaptations of our framework, including applications in BCI communication and neuroprosthetics for individuals with acquired blindness.\n\n2. **Evaluation of architecture and criterion**: We have addressed this in our responses above, providing additional evaluation and justification for our model's architecture and criteria.\n\nThank you again for your insightful comments. We believe these revisions will significantly strengthen our manuscript and address the concerns raised.\n\n**References:**\n\n\n- Yamins, D. L., et al. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. *PNAS*, 111(23), 8619-8624.\n- Khaligh-Razavi, S. M., & Kriegeskorte, N. (2014). Deep supervised models may explain IT cortical representation. *PLoS Computational Biology*, 10(11), e1003915.\n- Güçlü, U., & van Gerven, M. A. J. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. *Journal of Neuroscience*, 35(27), 10005-10014.\n- Le, L., et al. (2022). Brain2pix: Video frame reconstruction from brain activity. *Frontiers in Neuroscience*, 16, 940972.\n- Shen, G., et al. (2019). End-to-end deep image reconstruction from human brain activity. *Frontiers in Computational Neuroscience*, 13, 432276.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0M3gXvPmQQ", "reviewer": "Reviewer_d65o", "summary": "The authors present a CNN-based decoder that illuminates the distinct information encoded in V1, V4, and IT neuronal populations. The decoding results are remarkably good, and their accomplishment in decoding natural images from neuronal-level signals is unprecedented and the best so far.", "questions": "No specific questions.", "limitations": "No concerns.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Their novel space-time resolved decoding technique and the learned receptive field layers are new and interesting. Perhaps most impressive of all is their unprecedented data, recorded simultaneously from 15 Utah arrays from a single monkey. It can be considered a milestone in neuroscience research.", "weaknesses": "Analytical innovation is relatively minor. I am not sure what new insights about the visual cortex we learned. Nevertheless, proving that it can be done (with 4-7 Utah arrays per area), and doing it beautifully is still a remarkable feat that should be admired and applauded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and positive review. We appreciate your recognition of our novel space-time resolved decoding technique and the data collected from 15 Utah arrays. Regarding the mentioned weakness, our primary aim was indeed to demonstrate the feasibility and beauty of using these advanced models for studying the visual cortex. While the new neuroscientific insights reported in this work may seem limited, we believe this research is an essential step towards deeper understanding in the future. Your acknowledgment of our effort is greatly appreciated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HYx08fdKqB", "reviewer": "Reviewer_R5Jx", "summary": "SETTING: Decoding (reconstructing) observed images via Utah-array recordings made from the visual cortex of a macaque.\n\nAPPROACH: A GAN, but taking (transformed--see below) neural data as input, and with some additional losses.  In particular, in addition to the standard adversarial loss, the generator/decoder is aligned with VGG-19 at various layers, and its output is penalized with a reconstruction penalty (mean absolute error).\n\nRaw neural data are not passed into the generator/decoder (except in the baseline model); instead, multi-unit activity is first mapped to a retinotopic/pixel space, either with (learned) isotropic Gaussian receptive fields, or by mapping to features of a pre-trained Inception network.\n\nRESULTS: The data and model evidently support high-quality reconstructions (Fig. 1).  The authors believe the decoder with a spatiotemporal inverse retinotopic map works best.  They also perform qualitative experiments to determine which cortical areas (V1, V4, IT) and time windows contribute what to image decoding.", "questions": "> The number of arrays is certainly much larger than what is used by almost any research group.  It would be very helpful to see how results scale with number of neurons (really, electrodes).  Such a figure would also perhaps help make sense of Fig. 2: how much of the degradation in image reconstruction there is due to using fewer neurons, and how much due to using neurons only from this or that area?  (If there isn't time to run this experiment, perhaps the authors have some other way of answering this question.)\n\n> Along these lines: The authors somehow only get ~600 neurons from these fifteen arrays, apparently because they reject neurons with \"intraclass correlations\" less than 0.4.  What are these ICCs?  (What are the classes?)  Does this really improve performance?  How did the authors come up with the threshold of 0.4?\n\n> Generically, I understand why one would want to map from electrode space into retinotopic/pixel space (basically so that the model can be a UNet from there on out.)  But I don't understand the explanation of the \"inverse retinotopic mapping\" the authors given is Section 3.  The authors give formulae for computing the retinal embedding E and say that the parameters of these functions are learned.  But where is E subsequently used?  For the \"pre-trained mapping,\" a cost function is given, but E doesn't appear in it.  Instead, it contains yet more weights, in this case mapping from a pretrained Inception network to neural responses.  (In section 3.3.1, the letter W or w is used with seven different subscripts, some of them seemingly referring to the same weights and others not.)\n\nIn the appendix, there is a figure (not referenced in the main text!) that seemingly corresponds to the end-to-end inverse retinotopic mapping.  But E doesn't appear in this figure either.  The caption has a broken reference to an equation.\n\nCan the authors clarify this part of the model?\n\n> The authors state in the text that the spatiotemporal model achieves the highest correlations with AlexNet features, but Table 1 shows it to achieve the lowest (excluding the baseline; the end-to-end model achieves the highest correlation).  Is the text or the table in error?\n\nThey also claim that the spatiotemporal model qualitatively outperforms the other models (Fig. 1), but that claim is not obvious to this reader's eye.  (The horse and the butterfly, e.g., look worse; may others are a push.)\n\n> Can the authors confirm that the 100 images used for testing were never used during training?\n\n> What conclusions do the authors draw from Fig. 3?\n\nIn any case, the results would seem to depend strongly on the values assumed for synaptic delays:  This determines what time window in IT corresponds to what time window in V1, etc. (otherwise the windowing scheme is essentially random).  Can the authors justify high confidence in this synaptic-delay time?\n\nMINOR:\n> Section 4.1.2 examines the effect on decoding of setting certain areas of visual cortex to their baseline values.  Why is this called \"spatial occlusion\"?\n\nAlso, the column headings are described as identifying which brain region has been excluded, rather than included, but that obviously can't apply to the second column (\"V1 + V4 + IT\").  I think the text is wrong here.  (But if not, perhaps the authors could reverse the labels, letting all column headings state which areas are *in*cluded.)\n\n> Figs. 8 and 9: I don't understand the lightness/darkness color scheme, which doesn't seem to correspond to the correlation values listed in the figure.  Can the authors clarify this?\n\n> Why is the decoder called \"homeomorphic\"?\n\n> \"...spatially downsampling data to 8 Hz...\"\nWhat does this mean (and how could spatial sampling be denominated in Hertz?)?\n\n> \"...x_e and y_e denote the spatial coordinates of electrode e....\"  I think it would be less confusing to say the \"retinotopic coordinates corresponding to electrode e,\" or something similar.\n\n> The reference to Fig. 9 (l. 288) should be to Fig. 3.\n\n> The supplementary figures should be referenced in the main text.  (Figure 6, e.g., would be very useful to see while reading Section 3.4.)\n\n> Fig. 3: \"...with the final column representing the full data reconstruction\" -> \"...with the *first* column.....\"", "limitations": "N/A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The image reconstruction is, to my knowledge, state of the art.  The authors also convincingly demonstrate the importance of the retinotopic mapping.  Finally, the experimental setup (15 Utah arrays in one animal!) is heroic.", "weaknesses": "I put much of the details of these weaknesses into the questions (below).\n\nIt's not clear what we learn from the results following Figure 1.  Part (but not all--see the questions) of the problem may be due to the presentation.  The model itself is also not presented clearly: The model figures (Figs. 5 and 6 in the Appendix--not referenced in the main text) do not cover the first type of inverse retinotopic map, and introduce yet more symbols not appearing in the text.  The equations in the text appear incomplete (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and thoughtful review. We appreciate your positive feedback on our contributions and the detailed suggestions for improvement. Your insights on our decoding approach are invaluable. We are glad to hear that our retinotopic mapping approach is recognized for its importance. We also appreciate your acknowledgment of our experimental setup. Below, we address your concerns and clarify the points raised.\n\n### Model Presentation:\nWe have fixed the figures to match the text and will clarify the presentation of our model and ensure that all relevant figures (Figs. 5 and 6) and equations are clearly referenced and explained in the main text. The inverse retinotopic mapping and its role in the model will be more clearly described.\n\n### Answer to Questions:\n\n**Scaling Results with Number of Neurons:**  \nWe agree that analyzing how results scale with the number of neurons would provide further valuable insights. Therefore, we provide new comparative results of reconstructing the images only using data from electrodes that are located in specific visual ROIs. The results of this new experiment can be found in the Rebuttal Figures Document. Specifically, we show the reconstruction results from the regions V1, V4, and IT, separately.\n\n**Threshold Explanation:**  \nWe apologize for the incorrect use of \"Intraclass Correlation Coefficient\" in our manuscript. The measure we used was a reliability score based on self-correlation, which evaluates the consistency of neural responses across repetitions. The threshold of 0.4 was selected to balance retaining sufficient electrodes while removing unstable signals. Approximately 600 electrodes remained from the original 1024, partly due to this threshold and partly because one electrode was broken.\n\n**What is E:**  \nThe inverse retinotopic mapping is learned by the model in an end-to-end manner. The retinal embeddings (E) are not pre-calculated in this end-to-end model, whereas in the spatial and spatiotemporal model they are. We apologize for the confusion as we have used E interchangeably with RFSimages in the figure and agree that it is confusing. We will fix this and clarify it in the manuscript.\n\n**Appendix Figure:**  \nThis figure is a depiction of how the inverse receptive field layer in the end-to-end model computes the retinal embeddings (E) based on input brain responses. Apologies for the broken reference, we will fix this in the manuscript.\n\nWe have removed this claim of qualitative outperformance.\n\n**Test Images:**  \nYes, definitely. We confirm that the 100 test images were never used during training.\n\n**Figure 3:**  \nFigure 3 illustrates that the models are capable of detecting temporal variations in the information carried by neurons. This finding is promising and warrants further investigation in future research. By occluding the time windows that are not highlighted in Figure 3, we infer that the neurons at the highlighted time points carry the specific visual information indicated by our model. As time progresses, we observe changes in this information. We will clarify this in the text in the manuscript.\n\n**Synaptic Delay:**  \nWe base this assumption of synaptic delay on literature (Kravitz et al., 2013).\n\n**Text and Table Consistency:**  \nWe will correct the inconsistency between the text and Table 1 regarding the spatiotemporal model’s performance and ensure the qualitative claims are supported by clearer visual evidence.\n\n**Training and Testing Data:**  \nWe confirm that the 100 images used for testing were never used during training and will explicitly state this in the revised manuscript.\n\n### Minor Issues:\nWe have addressed and fixed the minor issues mentioned in the paper:\n- The term \"spatial occlusion\" was used to indicate the model's inference when specific locations of the input are set to baseline values (i.e., no signal or activity). The occlusion occurs at the regions that we are not interested in. Baseline values are used instead of zero to account for existing noise.\n- The column headings in Section 4.1.2 are currently explaining which areas are included indeed. Will add it to the figure description.\n- In Figures 8 and 9, the x-axis represents relative correlation, showing the percentage of correlation when all correlations add up to 1 per brain region. This ensures a fair comparison by normalizing values per brain region. The color coding represents the highest relative correlation, not the highest absolute correlation, also depicted by larger bars. We will clarify the x-axis meaning and provide a detailed explanation of the color coding and correlations. The caption for Figure 9 will be adjusted as well. We added this explanation in the main text as well.\n- We referred to our technique as \"homeomorphic\" due to its topography-preserving nature, formulating the neuron-to-pixel mapping problem as an image-to-image translation problem after a (learnable) retinotopic projection. The model preserves the inherent topography of data throughout all the network layers, focusing on processing local features.\n\n  *Two shapes are said to be homeomorphic if there exists a continuous, one-to-one mapping between them that preserves the spatial relationships of points. If you can stretch or bend one shape into another without tearing or gluing parts together, the shapes are homeomorphic.*\n\n- Changed the reference from spatial downsampling in Hz to temporally downsampling data to 8 Hz.\n- Revised the description of electrode coordinates.\n- Fixed the reference to Fig. 3 and the labeling issues in Fig. 3.\n- Ensured all supplementary figures are properly referenced in the main text.\n\nThank you again for your constructive feedback. \n\n- Kravitz, D. J., Saleem, K. S., Baker, C. I., Ungerleider, L. G., & Mishkin, M. (2013). The ventral visual pathway: An expanded neural framework for the processing of object quality. *Trends in Cognitive Sciences, 17*(1), 26-49.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2SHXlYhlLE", "reviewer": "Reviewer_gW2o", "summary": "In this paper, authors record multi unit activity (MUA) from macaque ventral stream (V1, V4 and IT) and train a CNN decoder to reconstruct the visual stimuli. The authors present three variations decoding attempts: a baseline CNN decoder that maps MUA directly to image space, and two U-Net based decoders that differ in the fact that the first only uses spatial information (averaging over the time dimension), while the latter is time-resolved (spatiotemporal). Author find that the spatiotemporal decoder yield the highest performances as evaluated via feature correlation and present occlusion analysis (either masking brain regions or time windows).", "questions": "- Figure 7 (not discussed in the main text) reports the learned 2D receptive field. Most learned RF seem to have a smaller-than-one pixel size, does this imply that no information from that electrode is used at all?\n- Why do authors refer to their decoding technique as \"homeomorphic\"?\n- Would a model trained on occluded data still performed as poorly as the results presented in the paper?\n- Could the difference in measured performances between the baseline and homeomorphic model be simply attributed to the fact that the full model has higher complexity (more parameter), as the baseline decoder only leverages *half* of the U-Net architecture?", "limitations": "The authors have adequately addressed the limitations of their study.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper tackles an established field such as stimuli reconstruction using electrophysiological neural signals (MUA) as opposed to the more common fMRI techniques. This grants superior time-resolution which offers ways for novel experiments. Indeed author showed interesting effects of decoding from different time windows in different brain regions.\nFurthermore the paper is in general well structured and presented.", "weaknesses": "- The paper present some inconsistencies between Text and Figures.\n    * In line 267 it is said that the spatiotemporal model achieve the highest performance, however Table 1 reports the End-to-End models as the top scorer.\n    * In line 288 author link to Figure 9 while the correct one should be Figure 3. However, description of Figure 3 is still inconsistent as in line 290 authors say that last column report full-data reconstruction while Figure 3 has it in the first column.\n    * In Figure 8 meaning of x-axis is unclear. Moreover, Figure caption states that \"Deeply colored points indicate higher correlations [...]\", however it is unclear how this is assessed since for example the V4 row has higher correlation with fc7 and fc8 than IT (which is instead marked). Similar remarks hold for Figure 9, which has an incomplete caption (a trailing \"The\").\n- Hyperparameters for decoder loss (sec. 3.6.2) are missing.\n- Authors introduce the \"Learned Receptive Field\" (LRF) layer claiming it \"enhances understanding of (model) structure and interpretive capacity\", however no discussion about is presented. Only related result is Figure 7, which is never mentioned in the main text and presents somewhat puzzling results (see Questions section of this review).\n- In the occlusion analyses, author perform model *inferences* on truncated data, however a model *trained* on the same occluded data might compensate the measured decrease in performances.\n- Paper lacks comparisons with existing decoding techniques, albeit originally introduced for other - but structurally similar - modalities such as EEG signals (see Y. Bai et al., DreamDiffusion (2023)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback. We appreciate the opportunity to clarify and address your concerns.\n\n**Addressing Mentioned Weaknesses:**\n\n**Text and Figures:**\n\n- **Line 267 and Table 1**: We apologize for the oversight. There was a mistake in the header of Table 1. The correct table should have the \"End-to-End\" and \"Spatiotemporal\" headers switched. The rest of the text is correct in stating that the spatiotemporal model achieved the highest performance overall, consistent with Figures 8 and 9. We will fix the table header.\n\n- **Figure references**: Line 288 should refer to Figure 3. We will correct the description to accurately reflect the columns. Thank you for pointing this out.\n\n- **Figures 8 and 9**: In Figure 8, the x-axis represents relative correlation, showing the percentage of correlation when all correlations add up to 1 per brain region. This ensures a fair comparison by normalizing values per brain region. The color coding represents the highest relative correlation, not the highest absolute correlation, also depicted by larger partial bars. We will clarify the x-axis meaning and provide a detailed explanation of the color coding and correlations. The caption for Figure 9 will be adjusted as well.\n\n\n- **Hyperparameters for decoder loss**: The hyperparameters for the training are mentioned in section 3.6.1: “We used the ADAM optimizer with a learning rate of 0.002 and beta coefficients of 0.5 and 0.999 to enhance convergence. The loss function included a discriminator loss (α_discr) at 0.01, VGG feature loss (β_vgg) at 0.9, and L1 pixel-wise loss (β_pix) at 0.09 to balance error sensitivity.”\n\n\n- **Learned Receptive Field (LRF) layer**: We thank the reviewer for this point. Figure 7 demonstrates that the LRF layer adapts during end-to-end reconstruction, showing smaller receptive fields in the center and larger ones in the periphery, indicating that information is used differently based on spatial location. This suggests that the LRF learns a mechanism similar to the brain while reconstructing end-to-end. We will discuss these results in the main text with more emphasis.\n\n\n- **Occlusion analyses**: Our goal with the occlusion analyses was to highlight the spatial and temporal features carried by neurons, not necessarily to compare reconstruction quality. We agree that training a model on occluded data could yield different results and will address this in the discussion. We did train the model on occluded data (see the second figure in the rebuttal).\n\n\n- **Comparisons with existing decoding techniques**: We currently compare our model against two baselines: the Brain2Pix model from Le et al. (2021) and an improved version of Shen et al. (2019). These models represent the state-of-the-art in the field. We will emphasize this in the manuscript and cite the mentioned paper (Y. Bai et al., DreamDiffusion, 2023) as well as any other relevant ones in the revised manuscript.\n\n\n**Answer to Questions:**\n\n\n- **Figure 7**: Many learned receptive fields (RFs) are indeed smaller than one pixel. However, this does not mean no information from that electrode is used. If an RF estimate is less than a pixel, the model uses a single pixel for that RF, still containing necessary information. This results from the limited field of view of the model (96 by 96 pixels) being unable to accurately assign sufficient (integer number of) pixels to electrodes with relatively smaller receptive fields (e.g., those with visual angles less than one degree of arc).\n\n\n- **Homeomorphic decoding**: We referred to our technique as \"homeomorphic\" due to its topography-preserving nature, formulating the neuron-to-pixel mapping problem as an image-to-image translation problem after a (learnable) retinotopic projection. The model preserves the inherent topography of data throughout all the network layers, focusing on processing local features.\n\n\n- **Occluded data**: While training the model on occluded data might yield different results, our objective was to demonstrate feature dependencies of a model trained on full brain data, not to evaluate absolute model performance on occluded data. We also trained separate reconstruction models on V1, V4, and IT regions, in addition to the combined model, which will be included in the manuscript.\n\n\n- **Model complexity**: The difference in performances between the baseline and homeomorphic model can be attributed to the higher complexity of the full model. Our baseline model, based on Shen et al. (2019), uses only half of the U-NET architecture, while our enhanced generator includes a full U-NET with more parameters. While increasing the baseline's complexity might deviate from strict comparisons with existing models, it illustrates how increased model complexity affects results. We will include an additional run with a full U-NET in the revised manuscript, which showed improved performance compared to Shen et al. but still fell short of the full homeomorphic model’s performance.\n\n\nThank you again for your valuable feedback. We will revise the manuscript to address these points, improve clarity, and include the new analyses in the manuscript and the rebuttal images.\n\n\n**References:**\n\n\n- Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End-to-end deep image reconstruction from human brain activity. *Frontiers in Computational Neuroscience*, 13:432276, 2019.\n- Le, L., et al. (2022). Brain2pix: Video frame reconstruction from brain activity. *Frontiers in Neuroscience*, 16, 940972.\n- Y. Bai et al., DreamDiffusion (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have included additional results in the form of figures as requested. The ablation study on various losses is presented in the figure titled "Model Ablations," along with the corresponding model losses for each ablated model in the figure titled "Ablation Losses." Although these were run with fewer epochs than usual, we were already able to observe the effects of the ablations.

Additionally, we have trained our model on different brain regions in response to reviewer gW2o's request, depicted in the figure titled "Model Trained on Brain Region of Interest." The figures and detailed results provide insight into the model's performance under these conditions.

We hope these additions address your concerns and provide a clearer understanding of our model's robustness and versatility. The rest of the rebuttals are addressed individually to each reviewer.

Thank you for your valuable feedback and guidance.

Meta-review: In this paper, the authors present a method for image decoding from spatiotemporal information from neural recordings. The authors provide good reconstructions of the image, employing the temporal dimension for further accuracy as well as more insight into the brain. The authors also provide insight into the role of different regions in the decoding process. Some concerns where raised by the reviewers, and the author responses and clarifications should be part of the final document.
Decision: Accept (poster)