MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity

Authors: lynn  le
paolo  papale
katja  seeliger
antonio  lozano
thirza  dado
feng  wang
pieter  roelfsema
marcel  van gerven
yagmur  güçlütürk
umut  güçlü
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:20:25

Abstract: In this paper, we reconstruct naturalistic images directly from macaque brain signals using a convolutional neural network (CNN) based decoder. We investigate the ability of this CNN-based decoding technique to differentiate among neuronal populations from areas V1, V4, and IT, revealing distinct readout characteristics for each. This research marks a progression from low-level to high-level brain signals, thereby enriching the existing framework for utilizing CNN-based decoders to decode brain activity. Our results demonstrate high-precision reconstructions of naturalistic images, highlighting the efficiency of CNN-based decoders in advancing our knowledge of how the brain's representations translate into pixels. Additionally, we present a novel space-time-resolved decoding technique, demonstrating how temporal resolution in decoding can advance our understanding of neural representations. Moreover, we introduce a learned receptive field layer that sheds light on the CNN-based model's data processing during training, enhancing understanding of its structure and interpretive capacity.

Main paper: 
 introduction: 
 Artificial neural network models designed for decoding naturalistic images from neural activity signals significantly advance our understanding of how visual information is processed in the brain. Decoding models aim to disentangle patterns of neural responses to different stimuli, offering insights into how visual stimuli (e.g., a brown horse or a white t-shirt) are represented by neural populations. Leveraging a large amount of naturalistic data facilitates the reconstruction of natural vision and allows for comprehensive analyses of visual features. Due to the immense variety of naturalistic visual space, reconstructing such stimuli from neural activity is considered the most challenging but also the most intriguing problem in neural decoding.Convolutional neural networks (CNNs) have recently become a cornerstone in neural decoding and encoding models. Their ability to study fundamental features carried by populations of neuronal signals has led to significant advancements in understanding the computational mechanisms of natural scene perception. Encoding models provide valuable insights into how brain activity changes in response to stimuli, showcasing the features of CNNs that predict neural responses within specific brain regions. Conversely, decoding models reveal the information content within brain areas without making assumptions about the representation of that information.Despite the prowess of CNNs in distinguishing between different brain signal patterns, their effectiveness in decoding shifts in neuronal activity remains limited. This limitation is crucial -a lack of variance in reconstructed images with changing brain signals suggests a disconnect between the models and the brain's interpretive functions. This performance gap could stem from intrinsic model limitations or data distribution issues. Refining these methods is essential to enhance our understanding of how closely decoding models can approximate the complex processes underlying visual perception.While CNNs are engineered to handle complex, multi-dimensional data, including spatial (height and width) and depth (color) dimensions, there remain significant distinctions between CNN processing and human brain information processing. To address these challenges, we have developed a fully convolutional decoding model trained from scratch using the THINGS dataset -a highly diverse collection of naturalistic stimuli. This dataset enriches the model's exposure to varied features, which is crucial for interpreting brain representations accurately.In this paper, we present the following contributions:• Homeomorphic decoder: We propose a CNN-based decoder trained to investigate the importance of spatial and temporal information carried in neuronal signals for high-fidelity visual reconstructions.• End-to-end inverse retinotopic mapping: We integrate an interpretable layer in the model, known as the end-to-end inverse retinotopic mapping. This layer dynamically learns to map brain signals to a 2D image during training. The adaptive mechanism of this layer, influenced by the entire learning process, allows the decoder to organize its own input spatially.• Model inference and analysis: By performing model inferences with truncated brain data, our approach dissects how the network reorganizes its weights based on spatially separated brain regions for reconstructions. This method aligns with known neuroscientific principles, enhancing the interpretability of decoded features.• Temporal dynamics: Our model incorporates specific time intervals for neuronal signal input, aligned with latency periods observed in the ventral visual pathway (V1 to IT). This temporal aspect allows for a deeper analysis of how visual processing evolves over time.The remainder of this paper is structured as follows. Section 2 reviews related work in neural decoding using CNNs, highlighting advances and existing challenges. Section 3 details the materials and methods, including data acquisition, preprocessing, and the model architecture. Section 4 presents the results and discussion, evaluating the performance and interpretability of our model. Finally, Section 5 concludes the paper and suggests avenues for future research. 
related work: 
 Deep neural networks (DNNs) and generative adversarial networks (GANs) have recently achieved notable success in decoding visual information from brain activity, particularly using fMRI data [1, 2, 3, 4, 5, 6, 7, 8] . CNNs are effective for neural decoding due to their capability to process complex, multi-dimensional data. Sarraf and Tofighi (2016) [9] treated fMRI slices as separate images, but this method was limited by noise and did not respect neural topography. Approaches using 3D convolutions to preserve spatial structure [10] , and geometric deep learning on cortical meshes [11, 12, 13, 14, 15] , have shown promise in capturing brain features more effectively.Decoding naturalistic stimuli using large datasets pushes conventional model boundaries [16, 17] . Le et al. (2022) [17] reconstructed images and videos from fMRI data by converting voxel responses into 2D representations aligned with the visual field, then applying fully convolutional networks trained with VGG feature loss and adversarial regularizers. This method showed significant improvements over previous techniques.Our work uses multi-unit activity (MUA) data from Utah electrode arrays in the macaque visual cortex (V1, V4, IT), which offers higher temporal resolution and captures fine-grained neuronal activity compared to fMRI. Bashivan et al. (2019) [18] integrated receptive field concepts into neural network models for better interpretability. Building on this, we incorporate an end-to-end inverse retinotopic mapping layer within our convolutional decoder. This layer dynamically maps brain signals to 2D images, improving spatial feature organization and providing deeper insights into neural processing.3 Material & methodsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
data: 
 We used images from the THINGS database [19] , containing high-resolution images across various object categories. Each image had three color channels (RGB) and was resized to 96 × 96 pixels to meet model input requirements and reduce computational complexity. Images were presented in the lower right quadrant, shifted 150 pixels right and down from the central fixation point.A passive fixation task was conducted with a 7-year-old male macaque (Macaca Mulatta) across 22,348 trials (22,248 training and 100 test trials). The macaque maintained fixation on a central dot while images, presented in a randomized sequence of four per trial, were displayed for 200 ms with a 200 ms gray screen interval. The macaque was rewarded with juice for maintaining fixation. Ethical approval was obtained from the Royal Netherlands Academy of Arts and Sciences, adhering to the NIH Guide for the Care and Use of Laboratory Animals, ensuring the macaque's well-being and minimizing stress.MUA was recorded from 15 Utah electrode arrays implanted in V1 (7 arrays), V4 (4 arrays), and IT (4 arrays), capturing neural activity at 1 ms resolution over 200 ms per trial. Electrodes were selected using a self-correlation reliability score with a threshold of 0.4, reducing the original 1024 electrodes to 576 (including losses from a broken electrode). Neural responses were z-scored (mean subtracted, divided by standard deviation). Time windows were defined as 0-125 ms for V1, 25-150 ms for V4, and 50-175 ms for IT. A 25 ms smoothing window was applied, and data were temporally downsampled to either 8 Hz (averaged over 125 ms) or 40 Hz, ensuring consistent and normalized data for modeling.In this section, we describe the models used for decoding the neural responses into visual stimuli.The main model is a homeomorphic decoder, and we compare its performance against a baseline decoder. Additionally, we employ a discriminator to facilitate adversarial training.The homeomorphic decoder transforms neural responses into retinal embeddings and subsequently reconstructs the visual stimuli from these embeddings. The architecture leverages several neural network components, including pre-trained and end-to-end trained models.Pre-trained inverse retinotopic mapping The first variant of our homeomorphic decoder uses a pre-trained CNN to perform inverse retinotopic mapping. The model projects neural responses onto retinal embeddings using learned weights and subsequently reconstructs the visual image from these embeddings. The process involves two types of embeddings: spatial embeddings and spatiotemporal embeddings.For spatial embeddings, the retinal embedding E is computed directly from the neural responses at a single timepoint:E axy = e∈MEAa r e W exywhere r represents the neural responses over 576 electrodes at a single timepoint, and W exy are the learned spatial weights for mapping the neural responses to retinal embeddings for each microelectrode array (MEA a ).For spatiotemporal embeddings, the retinal embedding E is computed from the neural responses over multiple timepoints:E itxy = e∈ROIi R et W exywhere R represents the neural responses over 576 electrodes across 5 timepoints, and W exy are the learned weights for mapping the neural responses to retinal embeddings for each region of interest (ROI i ).The weights W are optimized by minimizing the following objective function:W * e , α * e = min W,α ∥r e -re ∥ 2 + λ 1 ∥W ∥ 1 + λ 2 (∥W ∥ 2 2 + ∥α∥ 2 2 ) + λ 3 ∆αwhere re = c α ec xy W exy f (S) cxy is the estimated neural response with f representing activations from a pre-trained Inception v1 network and ∆w is the Laplace operator. Here, W can be considered the spatial weights of interest and α the feature weights. The specific layers of the network used for different cortical areas are: conv2d2 for V1, mixed4a for V4, and mixed4d for IT. These embeddings leverage the pre-trained network to efficiently map neural activity patterns to their corresponding visual stimuli representations, forming a crucial component of our decoder.End-to-end trained inverse retinotopic mapping This variant of our homeomorphic decoder involves end-to-end training of the inverse retinotopic mapping from neural responses R to retinal embeddings E a , allowing the model to learn optimal parameters directly from data without pre-trained CNN features.The inverse receptive field (see Figure 6 ) computes E as:E axy = e∈MEAa r e exp - (x -x e ) 2 2σ 2 e + (y -y e ) 2 2σ 2 ewhere r e represents the neural responses, x e and y e denote the spatial coordinates of electrode e, and σ e represents the standard deviation parameter that determines the spatial spread of the receptive field associated with each electrode. The parameters x e , y e , and σ e are learned during training.Once R a is mapped onto E a , a pixel-to-pixel U-Net reconstructs the stimulus S fake = U_Net(E a ), where S real is the ground-truth stimulus. Loss components are functions of S real and S fake , incorporating E a into the objective function.This end-to-end approach adapts to the specific characteristics of neural responses and visual stimuli, improving reconstruction performance.Pixel-to-pixel mapping The final component of our homeomorphic decoder employs a U-Net architecture designed for pixel-to-pixel mapping of retinal embeddings to visual stimuli. The U-Net model is a powerful neural network architecture commonly used for image segmentation tasks due to its ability to capture both local and global image features through its contracting and expansive paths connected via skip connections. We are using a standard U-Net architecture; for details refer to Appendix A.1.We employ a baseline decoder as a reference for performance evaluation. This simpler model transforms neural responses directly into visual stimuli, without the use of intermediate retinal embeddings. It adopts a modified U-Net architecture, retaining only the expansive path from the homeomorphic decoder while omitting the contracting path and skip connections. The input is a 576 × 1 tensor representing neural responses from 576 electrodes at a single timepoint, and the output is a RGB × 96 × 96 visual stimulus. Despite its simplicity, the baseline serves as a crucial benchmark for assessing more complex models like the homeomorphic decoder. This approach is inspired by [8] , which is regarded as a state-of-the-art reconstruction model [20] .The discriminator plays an integral role in the training process by differentiating between real and fake visual stimuli, thus facilitating adversarial training. We employ a modified U-Net architecture, using only the contracting path, to serve as our discriminator. The input to the discriminator is the visual stimulus, S, of dimensions RGB × 96 × 96. The output is a scalar probability p, indicating the likelihood that the given image is real. Adversarial training, where the discriminator aims to distinguish real images from reconstructed images, drives the decoder to generate more realistic and accurate visual stimuli. The loss calculated from the discriminator's assessments is crucial for improving the fidelity of the decoded images.This section outlines the optimization procedures, loss functions, and strategies used to train the decoders and the discriminator for high-fidelity reconstruction of visual stimuli from neural responses. Source code is available on our GitHub repository1 .The dataset comprised 22,348 training samples and 100 test samples, which were exclusively used for testing and never during training. We used the Adam optimizer with a learning rate of 0.002 and beta coefficients of 0.5 and 0.999 to ensure convergence. The loss function included discriminator loss (α discr ) at 0.01, VGG feature loss (β vgg ) at 0.9, and L1 pixel-wise loss (β pix ) at 0.09 to balance sensitivity. Training spanned 50 epochs on a Quadro RTX 6000 GPU, utilizing approximately 10,000 MiB of GPU memory.The training of the decoders (both homeomorphic and baseline) involves a combination of losses designed to ensure realism and accuracy in the reconstructed images. The adversarial loss encourages the decoder to generate realistic images that the discriminator cannot distinguish from real images. This loss is defined as the binary cross-entropy loss between the discriminator's output and the true labels (1 for real images and 0 for generated images):L adv = -E Sreal [log D(S real )] -E Sfake [log(1 -D(S fake ))]To further enhance the quality of the reconstructed images, we use a feature matching loss based on the activations of a pre-trained VGG-19 network. The feature loss is the mean squared error (MSE) between the feature representations of the real and generated images at various layers (conv1_2, conv2_2, conv3_4, conv4_4, and conv5_4) of the VGG-19 network:L feat = l ∥ϕ l (S real ) -ϕ l (S fake )∥ 2 2where ϕ l denotes the feature map at layer l of the VGG-19 network. The pixel-wise loss ensures that the reconstructions are close to the original images in pixel space. We use the mean absolute error (MAE) to quantify this loss:L pixel = ∥S real -S fake ∥ 1The total loss for the decoder is a weighted sum of the adversarial loss, feature loss, and pixel loss:L decoder = λ adv L adv + λ feat L feat + λ pixel L pixelwhere λ adv , λ feat , and λ pixel are the weights for each respective loss component.The discriminator is trained to distinguish between real and generated images, using the adversarial loss defined as the binary cross-entropy loss:L discriminator = -E Sreal [log D(S real )] -E Sfake [log(1 -D(S fake ))]The decoder and discriminator are optimized using the Adam optimizer with default parameters (learning rate = 0.0002, β 1 = 0.5, β 2 = 0.999). Early stopping is applied based on validation set performance to prevent overfitting. An image buffer maintains a history of generated images to stabilize adversarial training by varying the discriminator's inputs. During training, the decoder and discriminator are updated alternately: 1) the discriminator using adversarial loss, and 2) the decoder using the total loss (adversarial, feature, and pixel losses). Training continues until convergence, monitored by evaluation metrics.To robustly evaluate the performance of our decoding models, we employ a variety of metrics that assess the accuracy and quality of the reconstructed visual stimuli from both a feature-wise and perceptual quality perspective.Feature correlation One of the primary metrics used to evaluate stimulus reconstruction is the feature correlation between the reconstructed images and the original images. We use a pre-trained AlexNet, extracting feature representations at various layers (conv1, conv2, conv3, conv4, conv5, fc6, fc7, fc8). The Pearson correlation coefficient is calculated between corresponding feature maps of the original and reconstructed images: ρ ϕ l = Pearson(ϕ l (S real ), ϕ l (S fake )) where ϕ l denotes the feature map at layer l of AlexNet. High correlation values indicate that the reconstructed images capture similar feature representations as the original images. This standard metric for evaluating reconstruction quality was also used by Le et al. ( 2022) [17] .Image colorfulness To evaluate the perceptual quality of the reconstructed images, we use the Hasler and Süsstrunk colorfulness metric [21] . This metric quantifies the colorfulness of an image, which is an important aspect of human visual perception. The metric is computed as: C = σ rg + 0.3µ rg where σ rg and µ rg are the standard deviation and mean of the color difference vector rg = (R -G) across the image.Occlusion analysis We performed spatial and spatiotemporal occlusion analyses to assess the contributions of different spatial and temporal regions to reconstruction accuracy. In spatial occlusion, parts of the input are systematically removed to identify which regions are most critical for decoding.In spatiotemporal occlusion, specific segments of neural responses are occluded to evaluate the importance of different timepoints and spatial regions in the reconstruction process.4 Results and discussionQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This work does not include or introduce new results of purely theoretical nature.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
stimulus reconstruction: 
 The performance of different decoding models in reconstructing visual stimuli from neural responses is evaluated across multiple dimensions, including model comparison, spatial occlusion analysis, and spatiotemporal occlusion analysis.We compared the reconstruction performance of three variations of our homeomorphic decoderspatial, spatiotemporal, and end-to-end inverse retinotopic mapping -against the baseline model, both qualitatively and quantitatively.Figure 1 presents qualitative results. The spatial, spatiotemporal, and end-to-end decoders consistently outperformed the baseline, better preserving textures, shapes, and colors.End-to-end Spatiotemporal Baseline Table 1 provides a quantitative comparison using Pearson correlations between reconstructed and original images across different AlexNet layers (Section 3.4). The spatiotemporal decoding model achieved the highest feature correlation across most layers, highlighting its superior ability to capture and reconstruct the features present in the original stimuli. Specifically, the spatiotemporal model excelled in the deeper layers (fc7 and fc8), which capture high-level feature representations, demonstrating the model's capability to capture both fine-grained and abstract features encoded in the neural responses. The learned receptive field layer adapts receptive field sizes by spatial location, with larger fields in the periphery and smaller fields centrally (Figure 7 ). Despite some receptive fields being smaller than one pixel, all electrodes still contribute information, with single pixels used when needed. This effect is likely due to the model's limited 96 × 96 field of view, constraining pixel allocation for very small fields.We also ran model ablations to assess the effect of different loss functions (discriminator, pixel, and VGG loss) on performance (Figure 11 ). Additionally, we trained models on region-specific data (V1, V4, IT) to explore how brain region training affects reconstruction (Figure 12 ). These experiments highlight the importance of individual brain regions and the adaptability of models trained on occluded data.Spatial occlusion analysis was conducted to identify the importance of different brain regions (V1, V4, IT) in the reconstruction process. This analysis involved occluding specific spatial regions of the neural response inputs and examining the effect on the quality of the reconstructed images. To quantify the impact of occlusion on reconstruction quality, feature correlations with the original images were calculated using the pre-trained AlexNet, as shown in Figure 8 . The bar plot depicts feature correlations across different AlexNet layers for reconstructions derived from V1, V4, IT, and all regions combined.Spatiotemporal occlusion analysis evaluates how neural responses from different time windows contribute to reconstruction quality, offering insights into the brain's temporal processing of visual stimuli.Figure 3 shows example stimuli and their reconstructions when multiple time windows are occluded, with only one time window (highlighted in yellow) being included in the model during inference. Each column represents a different set of time windows being occluded, with the first column showing the reconstruction using all time windows.The colorfulness of reconstructions from V1, V4, and IT was evaluated using the Hasler and Süsstrunk metric. Figure 4 shows the colorfulness scores for each brain region and combined data. V1constrained reconstructions had the highest colorfulness scores and correlated strongly with early AlexNet layers (conv1, conv2), reflecting V1's role in processing basic features like edges and color. IT reconstructions, however, showed higher correlations with deeper AlexNet layers (fc7, fc8), which capture more abstract visual features.These results highlight the hierarchical nature of visual processing, with V1 specializing in basic visual attributes and IT handling more complex features.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: See Section 3.1. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: All the reported data collection experiments strictly adhered to the local ethical guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion: 
 We presented a comprehensive approach to decoding naturalistic visual stimuli from neural responses using a fully convolutional neural network trained from scratch. The use of the THINGS dataset enriched our model's feature set, crucial for accurately interpreting brain representations. Our homeomorphic decoder, enhanced with an end-to-end inverse retinotopic mapping layer, effectively integrates spatial and temporal information, leading to high-fidelity and interpretable reconstructions.Our evaluations highlighted the spatiotemporal decoding model's superior performance, evidenced by high feature correlations with deep layers of pre-trained AlexNet. Spatial and spatiotemporal  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The data was collected by authors of this paper. The stimulus dataset is publicly available and properly referenced and credited. The models are original work, or properly referenced and credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Neural decoding models for reconstructing naturalistic images deepen our understanding of the link between neural activity and perception, with promising applications in visual neuroprosthetics. This study specifically reuses data originally collected as part of an initiative aimed at restoring sight, adhering to ethical best practices by maximizing insights from a single dataset and minimizing the need for additional animal studies. This approach aligns with responsible research standards, balancing innovation with animal welfare.However, caution is needed when applying these models to human neuroprosthetics due to the complexity of human brain activity and behavior. Human behavior encompasses interactions with the environment, complex motor actions, and neuroplasticity, elements that models trained exclusively on data from static image viewing may not fully capture. For instance, visual feedback from motor activities adds layers of complexity beyond current model capabilities. Additionally, using these models to identify stimulation sites for neuroprosthetics may not replicate natural neural responses precisely, highlighting the need for experiments/clinical trials with human subjects, nuanced model development and careful interpretation of outputs.This study applies invasive MUA recordings in macaques, expanding on prior work with fMRI by offering higher signal quality and detail. However, the applicability of these results to noninvasive techniques like fMRI remains limited due to their lower signal-to-noise ratio and less detailed recording capabilities compared to electrode arrays. This distinction is important, as invasive neuroimaging remains rare in human research.Additionally, transitioning this framework to human intracranial applications poses challenges, including potential scar tissue formation, immune responses, and device rejection over long-term recordings. Anatomical differences, such as vascular structures, may further impact device placement and stability. Future work could explore these adaptations for broader applications, including braincomputer interfaces (BCI) and neuroprosthetics for individuals with acquired blindness, for which, careful regulatory guidance and additional research will be essential.Contracting path: The contracting path follows the typical architecture of a convolutional network. It consists of repeated application of two 3 × 3 convolutions (unpadded convolutions) followed by a ReLU activation and a 2 × 2 max pooling operation with stride 2 for downsampling. At each downsampling step, the number of feature channels is doubled.Expansive path: Every step in the expansive path consists of an upsampling of the feature map followed by a 2 × 2 convolution ("up-convolution") that halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two 3 × 3 convolutions, each followed by a ReLU activation. The cropping is necessary due to the loss of border pixels during convolutions in the contracting path.Skip connections: Skip connections are introduced between the contracting and expansive paths to combine low-level features with high-level features, facilitating finer reconstruction of the output image.The input to the U-net is the retinal embedding E of dimensions 15×96×96 (15 channels and 96×96 spatial resolution) and the output is the reconstructed visual stimulus S of dimensions RGB × 96 × 96.The U-net architecture enables detailed reconstruction by preserving spatial information through its symmetrical structure. H×W ] be a matrix in R H×W such that each entry is r. R [H×W ] is multiplied element-wise with its corresponding M , and then stacked based on its electrode number, resulting in 15 X in total (7 for V1, 4 for V4, and 4 for IT). no VGG no L1 no discr. Justification: Figures 1, 2, 9 , Table 1 and their corresponding descriptions in the main text.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/neuralcodinglab/MonkeySeehttps://github.com/neuralcodinglab/MonkeySee 


Decision: In this paper, the authors present a method for image decoding from spatiotemporal information from neural recordings. The authors provide good reconstructions of the image, employing the temporal dimension for further accuracy as well as more insight into the brain. The authors also provide insight into the role of different regions in the decoding process. Some concerns where raised by the reviewers, and the author responses and clarifications should be part of the final document.

Reviews: review: paper_summary: paper_summary: This paper proposes a CNN-based decoder to reconstruct naturalistic images from macaque brain signals. To this end, the paper presents the Learned Receptive Field (LRF) layer to enhance the reconstruction and understanding of the model's structure and interpretive capacity. Here, the work aims to interpret brain activity by transforming low-level to high-level brain signals. It focuses on the readout characteristics of neuronal populations in areas V1, V4, and IT. The evaluations of the model are performed using the THINGS dataset, and the results demonstrate the effectiveness of an end-to-end model and provide interpretations, such as the crucial role of V1 in color processing.
strength_weakness: strength_weakness: - The proposed model is data-specific, which could potentially limit its application.

- The overall loss consists of multiple criteria, each with its weight, which might make the model harder to train. Specifically, mixing - adversarial loss with pixel and feature loss can make it challenging to control the training dynamics.

- The paper does not include a complete ablation analysis of the different modules.
questions: questions: - Could you elaborate more on the motivation behind using Inception v1 for different cortical areas? How does this layer contribute to the generalization of the image reconstruction?

- Is the model generalizable to other types of neural data from different species or different types of recording equipment?

- In equation line 151, how does the standard deviation influence the spatial spread of the receptive field and overall reconstruction?

- In Pre-trained Inverse Retinotopic Mapping, why do we need two embeddings? Can’t the model handle it with a unified spatiotemporal module instead of having separate embeddings?

- What about employing an existing method to build a baseline, such as using a recurrent CNN or Vision Transformer? For a fair comparison, the evaluation should include some of the other state-of-the-art architectures.

- Is the quantitative comparison presented in Table 1 a standard manner of measuring performance? Is it used in the literature? If not, what would be other metrics, such as the R² score?

- What about the ablation analysis of the loss functions? Also, could you present the dynamic criterion of each loss during the training and validation phases?

- How did you set the hyperparameters of the model? Also, the full implementation details of the model are missing.
limitations: limitations: - I think the main limitation of the paper is the limited area of applicability of the proposed framework.

- The evaluation, both in terms of architecture and criterion, is missing to determine the necessity and effectiveness of all the modules of the model.
paper_summary: paper_summary: The authors present a CNN-based decoder that illuminates the distinct information encoded in V1, V4, and IT neuronal populations. The decoding results are remarkably good, and their accomplishment in decoding natural images from neuronal-level signals is unprecedented and the best so far.
strength_weakness: strength_weakness: Analytical innovation is relatively minor. I am not sure what new insights about the visual cortex we learned. Nevertheless, proving that it can be done (with 4-7 Utah arrays per area), and doing it beautifully is still a remarkable feat that should be admired and applauded.
questions: questions: No specific questions.
limitations: limitations: No concerns.
paper_summary: paper_summary: SETTING: Decoding (reconstructing) observed images via Utah-array recordings made from the visual cortex of a macaque.

APPROACH: A GAN, but taking (transformed--see below) neural data as input, and with some additional losses.  In particular, in addition to the standard adversarial loss, the generator/decoder is aligned with VGG-19 at various layers, and its output is penalized with a reconstruction penalty (mean absolute error).

Raw neural data are not passed into the generator/decoder (except in the baseline model); instead, multi-unit activity is first mapped to a retinotopic/pixel space, either with (learned) isotropic Gaussian receptive fields, or by mapping to features of a pre-trained Inception network.

RESULTS: The data and model evidently support high-quality reconstructions (Fig. 1).  The authors believe the decoder with a spatiotemporal inverse retinotopic map works best.  They also perform qualitative experiments to determine which cortical areas (V1, V4, IT) and time windows contribute what to image decoding.
strength_weakness: strength_weakness: I put much of the details of these weaknesses into the questions (below).

It's not clear what we learn from the results following Figure 1.  Part (but not all--see the questions) of the problem may be due to the presentation.  The model itself is also not presented clearly: The model figures (Figs. 5 and 6 in the Appendix--not referenced in the main text) do not cover the first type of inverse retinotopic map, and introduce yet more symbols not appearing in the text.  The equations in the text appear incomplete (see questions).
questions: questions: > The number of arrays is certainly much larger than what is used by almost any research group.  It would be very helpful to see how results scale with number of neurons (really, electrodes).  Such a figure would also perhaps help make sense of Fig. 2: how much of the degradation in image reconstruction there is due to using fewer neurons, and how much due to using neurons only from this or that area?  (If there isn't time to run this experiment, perhaps the authors have some other way of answering this question.)

> Along these lines: The authors somehow only get ~600 neurons from these fifteen arrays, apparently because they reject neurons with "intraclass correlations" less than 0.4.  What are these ICCs?  (What are the classes?)  Does this really improve performance?  How did the authors come up with the threshold of 0.4?

> Generically, I understand why one would want to map from electrode space into retinotopic/pixel space (basically so that the model can be a UNet from there on out.)  But I don't understand the explanation of the "inverse retinotopic mapping" the authors given is Section 3.  The authors give formulae for computing the retinal embedding E and say that the parameters of these functions are learned.  But where is E subsequently used?  For the "pre-trained mapping," a cost function is given, but E doesn't appear in it.  Instead, it contains yet more weights, in this case mapping from a pretrained Inception network to neural responses.  (In section 3.3.1, the letter W or w is used with seven different subscripts, some of them seemingly referring to the same weights and others not.)

In the appendix, there is a figure (not referenced in the main text!) that seemingly corresponds to the end-to-end inverse retinotopic mapping.  But E doesn't appear in this figure either.  The caption has a broken reference to an equation.

Can the authors clarify this part of the model?

> The authors state in the text that the spatiotemporal model achieves the highest correlations with AlexNet features, but Table 1 shows it to achieve the lowest (excluding the baseline; the end-to-end model achieves the highest correlation).  Is the text or the table in error?

They also claim that the spatiotemporal model qualitatively outperforms the other models (Fig. 1), but that claim is not obvious to this reader's eye.  (The horse and the butterfly, e.g., look worse; may others are a push.)

> Can the authors confirm that the 100 images used for testing were never used during training?

> What conclusions do the authors draw from Fig. 3?

In any case, the results would seem to depend strongly on the values assumed for synaptic delays:  This determines what time window in IT corresponds to what time window in V1, etc. (otherwise the windowing scheme is essentially random).  Can the authors justify high confidence in this synaptic-delay time?

MINOR:
> Section 4.1.2 examines the effect on decoding of setting certain areas of visual cortex to their baseline values.  Why is this called "spatial occlusion"?

Also, the column headings are described as identifying which brain region has been excluded, rather than included, but that obviously can't apply to the second column ("V1 + V4 + IT").  I think the text is wrong here.  (But if not, perhaps the authors could reverse the labels, letting all column headings state which areas are *in*cluded.)

> Figs. 8 and 9: I don't understand the lightness/darkness color scheme, which doesn't seem to correspond to the correlation values listed in the figure.  Can the authors clarify this?

> Why is the decoder called "homeomorphic"?

> "...spatially downsampling data to 8 Hz..."
What does this mean (and how could spatial sampling be denominated in Hertz?)?

> "...x_e and y_e denote the spatial coordinates of electrode e...."  I think it would be less confusing to say the "retinotopic coordinates corresponding to electrode e," or something similar.

> The reference to Fig. 9 (l. 288) should be to Fig. 3.

> The supplementary figures should be referenced in the main text.  (Figure 6, e.g., would be very useful to see while reading Section 3.4.)

> Fig. 3: "...with the final column representing the full data reconstruction" -> "...with the *first* column....."
limitations: limitations: N/A.
paper_summary: paper_summary: In this paper, authors record multi unit activity (MUA) from macaque ventral stream (V1, V4 and IT) and train a CNN decoder to reconstruct the visual stimuli. The authors present three variations decoding attempts: a baseline CNN decoder that maps MUA directly to image space, and two U-Net based decoders that differ in the fact that the first only uses spatial information (averaging over the time dimension), while the latter is time-resolved (spatiotemporal). Author find that the spatiotemporal decoder yield the highest performances as evaluated via feature correlation and present occlusion analysis (either masking brain regions or time windows).
strength_weakness: strength_weakness: - The paper present some inconsistencies between Text and Figures.
    * In line 267 it is said that the spatiotemporal model achieve the highest performance, however Table 1 reports the End-to-End models as the top scorer.
    * In line 288 author link to Figure 9 while the correct one should be Figure 3. However, description of Figure 3 is still inconsistent as in line 290 authors say that last column report full-data reconstruction while Figure 3 has it in the first column.
    * In Figure 8 meaning of x-axis is unclear. Moreover, Figure caption states that "Deeply colored points indicate higher correlations [...]", however it is unclear how this is assessed since for example the V4 row has higher correlation with fc7 and fc8 than IT (which is instead marked). Similar remarks hold for Figure 9, which has an incomplete caption (a trailing "The").
- Hyperparameters for decoder loss (sec. 3.6.2) are missing.
- Authors introduce the "Learned Receptive Field" (LRF) layer claiming it "enhances understanding of (model) structure and interpretive capacity", however no discussion about is presented. Only related result is Figure 7, which is never mentioned in the main text and presents somewhat puzzling results (see Questions section of this review).
- In the occlusion analyses, author perform model *inferences* on truncated data, however a model *trained* on the same occluded data might compensate the measured decrease in performances.
- Paper lacks comparisons with existing decoding techniques, albeit originally introduced for other - but structurally similar - modalities such as EEG signals (see Y. Bai et al., DreamDiffusion (2023)).
questions: questions: - Figure 7 (not discussed in the main text) reports the learned 2D receptive field. Most learned RF seem to have a smaller-than-one pixel size, does this imply that no information from that electrode is used at all?
- Why do authors refer to their decoding technique as "homeomorphic"?
- Would a model trained on occluded data still performed as poorly as the results presented in the paper?
- Could the difference in measured performances between the baseline and homeomorphic model be simply attributed to the fact that the full model has higher complexity (more parameter), as the baseline decoder only leverages *half* of the U-Net architecture?
limitations: limitations: The authors have adequately addressed the limitations of their study.

Comments: title: We thank the reviewer for these clarification questions!
Responses to Rebuttal
comment: **Q1**. Let's first define the following:

- S := visual stimulus
  - S_real := ground-truth stimulus
  - S_fake := reconstructed stimulus
- R := neural response
- E := retinal embedding

After (pre-/end-to-end trained) inverse retinotopic mapping projects R onto E as in the (pre-/end-to-end trained) inverse retinotopic mapping subsection, pixel-to-pixel mapping reconstructs S_fake from E as S_fake = U_Net(E). Note that the U-Net architecture is provided in the appendix. Every loss component is a function of S_real and/or S_fake, which is how, e.g., E enters the objective function.

We will make sure to thoroughly review and clarify the relevant subsections in the revised manuscript.

**Q2**. Figure 3 demonstrates the temporal occlusion analysis results of five example stimulus-response pairs. While we can observe some qualitative trends in these results, we cannot easily draw general/strong conclusions from the figure alone.

For example, it appears that the form/shape information is present mostly in the first time window. Likewise, color information seems to be mostly present in the second and third time windows.

Also, quantitative results of the temporal occlusion analysis is provided in Figure 9 and 10, which complement Figure 3 and can be interpreted in a similar way to the spatial occlusion analysis results. That is, in terms of the internal representations of the AlexNet layer that show the highest correlation at a time window  (instead of only ROI as in the spatial occlusion analysis).

Likewise, we will make sure to thoroughly review and clarify the relevant  subsections, figure captions, as well as correcting the figure references  in the revised manuscript.
Thanks for your responses. Congratulation on your accomplishment.
The authors have answered some of my questions but not all.  The inverse retinotopic mapping still isn't adequately explained (the authors still haven't said how, e.g., E_axy enters the objective function).

It's still not clear what conclusions the reader is supposed to draw from Fig. 3.  The authors reply that "we infer that the neurons at the highlighted time points carry the specific visual information indicated by our model," but what information is this?  Can it be described even qualitatively?  E.g., "excluding TW-3 tends to make the images..."--what?  Blurrier?  If this analysis were redone with a different choice for synaptic delay (say, a factor of 2 larger), would we draw different conclusions?  

Still, the paper's strengths remain, and I stand by my original ratings.
Thank you very much. We greatly appreciate it.
I thank the authors for providing additional material in their rebuttal in such short time and for carefully addressing my specific concerns with a detailed reply. I believe their work was strengthened by the new material and most importantly the new result that even a complexity-matched model "still fell short of the full homeomorphic model's performance". I will increase my review score to reflect this change.