Title: MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity
Venue: NeurIPS 2024
Accepted: yes
Keywords: space-time dependent reconstructions

biological constraints

neural decoding

deep neural network

receptive fields

visual cortex

multi unit activity

convolutional model

brain image reconstruction

naturalistic stimuli
Abstract: In this paper, we reconstruct naturalistic images directly from macaque brain signals using a convolutional neural network (CNN) based decoder. We investigate the ability of this CNN-based decoding technique to differentiate among neuronal populations from areas V1, V4, and IT, revealing distinct readout characteristics for each. This research marks a progression from low-level to high-level brain signals, thereby enriching the existing framework for utilizing CNN-based decoders to decode brain activity. Our results demonstrate high-precision reconstructions of naturalistic images, highlighting the efficiency of CNN-based decoders in advancing our knowledge of how the brain's representations translate into pixels. Additionally, we present a novel space-time-resolved decoding technique, demonstrating how temporal resolution in decoding can advance our understanding of neural representations. Moreover, we introduce a learned receptive field layer that sheds light on the CNN-based model's data processing during training, enhancing understanding of its structure and interpretive capacity.

MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity

Abstract

In this paper, we reconstruct naturalistic images directly from macaque brain signals using a convolutional neural network (CNN) based decoder. We investigate the ability of this CNN-based decoding technique to differentiate among neuronal populations from areas V1, V4, and IT, revealing distinct readout characteristics for each. This research marks a progression from low-level to high-level brain signals, thereby enriching the existing framework for utilizing CNN-based decoders to decode brain activity. Our results demonstrate high-precision reconstructions of naturalistic images, highlighting the efficiency of CNN-based decoders in advancing our knowledge of how the brain's representations translate into pixels. Additionally, we present a novel space-time-resolved decoding technique, demonstrating how temporal resolution in decoding can advance our understanding of neural representations. Moreover, we introduce a learned receptive field layer that sheds light on the CNN-based model's data processing during training, enhancing understanding of its structure and interpretive capacity.

1 Introduction

Artificial neural network models designed for decoding naturalistic images from neural activity signals significantly advance our understanding of how visual information is processed in the brain. Decoding models aim to disentangle patterns of neural responses to different stimuli, offering insights into how visual stimuli (e.g., a brown horse or a white t-shirt) are represented by neural populations. Leveraging a large amount of naturalistic data facilitates the reconstruction of natural vision and allows for comprehensive analyses of visual features. Due to the immense variety of naturalistic visual space, reconstructing such stimuli from neural activity is considered the most challenging but also the most intriguing problem in neural decoding.

Convolutional neural networks (CNNs) have recently become a cornerstone in neural decoding and encoding models. Their ability to study fundamental features carried by populations of neuronal signals has led to significant advancements in understanding the computational mechanisms of natural scene perception. Encoding models provide valuable insights into how brain activity changes in response to stimuli, showcasing the features of CNNs that predict neural responses within specific

brain regions. Conversely, decoding models reveal the information content within brain areas without making assumptions about the representation of that information.

Despite the prowess of CNNs in distinguishing between different brain signal patterns, their effectiveness in decoding shifts in neuronal activity remains limited. This limitation is crucial - a lack of variance in reconstructed images with changing brain signals suggests a disconnect between the models and the brain's interpretive functions. This performance gap could stem from intrinsic model limitations or data distribution issues. Refining these methods is essential to enhance our understanding of how closely decoding models can approximate the complex processes underlying visual perception.

While CNNs are engineered to handle complex, multi-dimensional data, including spatial (height and width) and depth (color) dimensions, there remain significant distinctions between CNN processing and human brain information processing. To address these challenges, we have developed a fully convolutional decoding model trained from scratch using the THINGS dataset - a highly diverse collection of naturalistic stimuli. This dataset enriches the model's exposure to varied features, which is crucial for interpreting brain representations accurately.

In this paper, we present the following contributions:

Homeomorphic decoder : We propose a CNN-based decoder trained to investigate the importance of spatial and temporal information carried in neuronal signals for high-fidelity visual reconstructions.

End-to-end inverse retinotopic mapping : We integrate an interpretable layer in the model, known as the end-to-end inverse retinotopic mapping. This layer dynamically learns to map brain signals to a 2D image during training. The adaptive mechanism of this layer, influenced by the entire learning process, allows the decoder to organize its own input spatially.

Model inference and analysis : By performing model inferences with truncated brain data, our approach dissects how the network reorganizes its weights based on spatially separated brain regions for reconstructions. This method aligns with known neuroscientific principles, enhancing the interpretability of decoded features.

Temporal dynamics : Our model incorporates specific time intervals for neuronal signal input, aligned with latency periods observed in the ventral visual pathway (V1 to IT). This temporal aspect allows for a deeper analysis of how visual processing evolves over time.

The remainder of this paper is structured as follows. Section 2 reviews related work in neural decoding using CNNs, highlighting advances and existing challenges. Section 3 details the materials and methods, including data acquisition, preprocessing, and the model architecture. Section 4 presents the results and discussion, evaluating the performance and interpretability of our model. Finally, Section 5 concludes the paper and suggests avenues for future research.

2 Related work

Deep neural networks (DNNs) and generative adversarial networks (GANs) have recently achieved notable success in decoding visual information from brain activity, particularly using fMRI data [1, 2, 3, 4, 5, 6, 7, 8]. Seeliger et al. (2018) [5] utilized GANs to reconstruct grayscale images and handwritten characters from fMRI data, demonstrating the versatility of adversarial methods. Nishimoto et al. (2011) [1] used voxel-wise modeling to reconstruct complex video stimuli from brain activity, focusing on capturing detailed neural responses in the visual cortex. Shen et al. (2019a) [8] further advanced this by training end-to-end models for image reconstruction, incorporating high-level feature losses into the GAN framework.

Reconstructing dynamic stimuli like videos presents additional challenges. Han et al. (2019) [6] used variational auto-encoders for video reconstruction, achieving low-level property reconstructions but struggling with finer details. This highlights the complexity of decoding dynamic visual information.

CNNs are effective for neural decoding due to their capability to process complex, multi-dimensional data. Sarraf and Tofighi (2016) [9] treated fMRI slices as separate images, but this method was limited by noise and did not respect neural topography. Approaches using 3D convolutions to preserve spatial

structure [10], and geometric deep learning on cortical meshes [11, 12, 13, 14, 15], have shown promise in capturing brain features more effectively.

Decoding naturalistic stimuli using large datasets pushes conventional model boundaries [16, 17]. Le et al. (2022) [17] reconstructed images and videos from fMRI data by converting voxel responses into 2D representations aligned with the visual field, then applying fully convolutional networks trained with VGG feature loss and adversarial regularizers. This method showed significant improvements over previous techniques.

Our work uses multi-unit activity (MUA) data from Utah electrode arrays in the macaque visual cortex (V1, V4, IT), which offers higher temporal resolution and captures fine-grained neuronal activity compared to fMRI. Bashivan et al. (2019) [18] integrated receptive field concepts into neural network models for better interpretability. Building on this, we incorporate an end-to-end inverse retinotopic mapping layer within our convolutional decoder. This layer dynamically maps brain signals to 2D images, improving spatial feature organization and providing deeper insights into neural processing.

3 Material & methods

3.1 Data

We used images from the THINGS database [19], containing high-resolution images across various object categories. Each image had three color channels (RGB) and was resized to 96 × 96 pixels to meet model input requirements and reduce computational complexity. Images were presented in the lower right quadrant, shifted 150 pixels right and down from the central fixation point.

A passive fixation task was conducted with a 7-year-old male macaque ( Macaca Mulatta ) across 22,348 trials (22,248 training and 100 test trials). The macaque maintained fixation on a central dot while images, presented in a randomized sequence of four per trial, were displayed for 200 ms with a 200 ms gray screen interval. The macaque was rewarded with juice for maintaining fixation. Ethical approval was obtained from the Royal Netherlands Academy of Arts and Sciences, adhering to the NIH Guide for the Care and Use of Laboratory Animals, ensuring the macaque's well-being and minimizing stress.

MUA was recorded from 15 Utah electrode arrays implanted in V1 (7 arrays), V4 (4 arrays), and IT (4 arrays), capturing neural activity at 1 ms resolution over 200 ms per trial. Electrodes were selected using a self-correlation reliability score with a threshold of 0.4, reducing the original 1024 electrodes to 576 (including losses from a broken electrode). Neural responses were z-scored (mean subtracted, divided by standard deviation). Time windows were defined as 0-125 ms for V1, 25-150 ms for V4, and 50-175 ms for IT. A 25 ms smoothing window was applied, and data were temporally downsampled to either 8 Hz (averaged over 125 ms) or 40 Hz, ensuring consistent and normalized data for modeling.

3.2 Models

In this section, we describe the models used for decoding the neural responses into visual stimuli. The main model is a homeomorphic decoder, and we compare its performance against a baseline decoder. Additionally, we employ a discriminator to facilitate adversarial training.

3.2.1 Homeomorphic decoder

The homeomorphic decoder transforms neural responses into retinal embeddings and subsequently reconstructs the visual stimuli from these embeddings. The architecture leverages several neural network components, including pre-trained and end-to-end trained models.

Pre-trained inverse retinotopic mapping The first variant of our homeomorphic decoder uses a pre-trained CNN to perform inverse retinotopic mapping. The model projects neural responses onto retinal embeddings using learned weights and subsequently reconstructs the visual image from these embeddings. The process involves two types of embeddings: spatial embeddings and spatiotemporal embeddings .

For spatial embeddings , the retinal embedding E is computed directly from the neural responses at a single timepoint:

where r represents the neural responses over 576 electrodes at a single timepoint, and W exy are the learned spatial weights for mapping the neural responses to retinal embeddings for each microelectrode array (MEA a ).

For spatiotemporal embeddings , the retinal embedding E is computed from the neural responses over multiple timepoints:

where R represents the neural responses over 576 electrodes across 5 timepoints, and W exy are the learned weights for mapping the neural responses to retinal embeddings for each region of interest (ROI i ).

The weights W are optimized by minimizing the following objective function:

where ˆ r e = ∑ c α ec ∑ xy W exy f ( S ) cxy is the estimated neural response with f representing activations from a pre-trained Inception v1 network and ∆ w is the Laplace operator. Here, W can be considered the spatial weights of interest and α the feature weights. The specific layers of the network used for different cortical areas are: conv2d2 for V1, mixed4a for V4, and mixed4d for IT. These embeddings leverage the pre-trained network to efficiently map neural activity patterns to their corresponding visual stimuli representations, forming a crucial component of our decoder.

End-to-end trained inverse retinotopic mapping This variant of our homeomorphic decoder involves end-to-end training of the inverse retinotopic mapping from neural responses R to retinal embeddings E a , allowing the model to learn optimal parameters directly from data without pre-trained CNN features.

The inverse receptive field (see Figure 6) computes E as:

where r e represents the neural responses, x e and y e denote the spatial coordinates of electrode e , and σ e represents the standard deviation parameter that determines the spatial spread of the receptive field associated with each electrode. The parameters x e , y e , and σ e are learned during training.

Once R a is mapped onto E a , a pixel-to-pixel U-Net reconstructs the stimulus S fake = U_Net ( E a ) , where S real is the ground-truth stimulus. Loss components are functions of S real and S fake, incorporating E a into the objective function.

This end-to-end approach adapts to the specific characteristics of neural responses and visual stimuli, improving reconstruction performance.

Pixel-to-pixel mapping The final component of our homeomorphic decoder employs a U-Net architecture designed for pixel-to-pixel mapping of retinal embeddings to visual stimuli. The U-Net model is a powerful neural network architecture commonly used for image segmentation tasks due to its ability to capture both local and global image features through its contracting and expansive paths connected via skip connections. We are using a standard U-Net architecture; for details refer to Appendix A.1.

3.2.2 Baseline decoder

We employ a baseline decoder as a reference for performance evaluation. This simpler model transforms neural responses directly into visual stimuli, without the use of intermediate retinal embeddings. It adopts a modified U-Net architecture, retaining only the expansive path from the homeomorphic decoder while omitting the contracting path and skip connections. The input is a

576 × 1 tensor representing neural responses from 576 electrodes at a single timepoint, and the output is a RGB × 96 × 96 visual stimulus. Despite its simplicity, the baseline serves as a crucial benchmark for assessing more complex models like the homeomorphic decoder. This approach is inspired by [8], which is regarded as a state-of-the-art reconstruction model [20].

3.2.3 Discriminator

The discriminator plays an integral role in the training process by differentiating between real and fake visual stimuli, thus facilitating adversarial training. We employ a modified U-Net architecture, using only the contracting path, to serve as our discriminator. The input to the discriminator is the visual stimulus, S , of dimensions RGB × 96 × 96 . The output is a scalar probability p , indicating the likelihood that the given image is real. Adversarial training, where the discriminator aims to distinguish real images from reconstructed images, drives the decoder to generate more realistic and accurate visual stimuli. The loss calculated from the discriminator's assessments is crucial for improving the fidelity of the decoded images.

3.3 Training

This section outlines the optimization procedures, loss functions, and strategies used to train the decoders and the discriminator for high-fidelity reconstruction of visual stimuli from neural responses. Source code is available on our GitHub repository 1 .

3.3.1 Training parameters

The dataset comprised 22,348 training samples and 100 test samples, which were exclusively used for testing and never during training. We used the Adam optimizer with a learning rate of 0.002 and beta coefficients of 0.5 and 0.999 to ensure convergence. The loss function included discriminator loss ( α discr) at 0.01, VGG feature loss ( β vgg) at 0.9, and L1 pixel-wise loss ( β pix) at 0.09 to balance sensitivity. Training spanned 50 epochs on a Quadro RTX 6000 GPU, utilizing approximately 10,000 MiB of GPU memory.

3.3.2 Decoder training

The training of the decoders (both homeomorphic and baseline) involves a combination of losses designed to ensure realism and accuracy in the reconstructed images. The adversarial loss encourages the decoder to generate realistic images that the discriminator cannot distinguish from real images. This loss is defined as the binary cross-entropy loss between the discriminator's output and the true labels (1 for real images and 0 for generated images):

To further enhance the quality of the reconstructed images, we use a feature matching loss based on the activations of a pre-trained VGG-19 network. The feature loss is the mean squared error (MSE) between the feature representations of the real and generated images at various layers ( conv1_2 , conv2_2 , conv3_4 , conv4_4 , and conv5_4 ) of the VGG-19 network:

where ϕ l denotes the feature map at layer l of the VGG-19 network. The pixel-wise loss ensures that the reconstructions are close to the original images in pixel space. We use the mean absolute error (MAE) to quantify this loss:

The total loss for the decoder is a weighted sum of the adversarial loss, feature loss, and pixel loss:

where λ adv , λ feat, and λ pixel are the weights for each respective loss component.

3.3.3 Discriminator training

The discriminator is trained to distinguish between real and generated images, using the adversarial loss defined as the binary cross-entropy loss:

3.3.4 Optimization strategy

The decoder and discriminator are optimized using the Adam optimizer with default parameters (learning rate = 0.0002, β 1 = 0 . 5 , β 2 = 0 . 999 ). Early stopping is applied based on validation set performance to prevent overfitting. An image buffer maintains a history of generated images to stabilize adversarial training by varying the discriminator's inputs. During training, the decoder and discriminator are updated alternately: 1) the discriminator using adversarial loss, and 2) the decoder using the total loss (adversarial, feature, and pixel losses). Training continues until convergence, monitored by evaluation metrics.

3.4 Evaluation metrics

To robustly evaluate the performance of our decoding models, we employ a variety of metrics that assess the accuracy and quality of the reconstructed visual stimuli from both a feature-wise and perceptual quality perspective.

Feature correlation One of the primary metrics used to evaluate stimulus reconstruction is the feature correlation between the reconstructed images and the original images. We use a pre-trained AlexNet, extracting feature representations at various layers ( conv1 , conv2 , conv3 , conv4 , conv5 , fc6 , fc7 , fc8 ). The Pearson correlation coefficient is calculated between corresponding feature maps of the original and reconstructed images: ρ ϕ l = Pearson ( ϕ l ( S real ) , ϕ l ( S fake )) where ϕ l denotes the feature map at layer l of AlexNet. High correlation values indicate that the reconstructed images capture similar feature representations as the original images. This standard metric for evaluating reconstruction quality was also used by Le et al. (2022) [17].

Image colorfulness To evaluate the perceptual quality of the reconstructed images, we use the Hasler and Süsstrunk colorfulness metric [21]. This metric quantifies the colorfulness of an image, which is an important aspect of human visual perception. The metric is computed as: C = σ rg + 0 . 3 µ rg where σ rg and µ rg are the standard deviation and mean of the color difference vector rg = ( R -G ) across the image.

Occlusion analysis We performed spatial and spatiotemporal occlusion analyses to assess the contributions of different spatial and temporal regions to reconstruction accuracy. In spatial occlusion, parts of the input are systematically removed to identify which regions are most critical for decoding. In spatiotemporal occlusion, specific segments of neural responses are occluded to evaluate the importance of different timepoints and spatial regions in the reconstruction process.

4 Results and discussion

4.1 Stimulus reconstruction

The performance of different decoding models in reconstructing visual stimuli from neural responses is evaluated across multiple dimensions, including model comparison, spatial occlusion analysis, and spatiotemporal occlusion analysis.

4.1.1 Model comparison

We compared the reconstruction performance of three variations of our homeomorphic decoder spatial, spatiotemporal, and end-to-end inverse retinotopic mapping - against the baseline model, both qualitatively and quantitatively.

Figure 1 presents qualitative results. The spatial, spatiotemporal, and end-to-end decoders consistently outperformed the baseline, better preserving textures, shapes, and colors.

Spatial

End-to-end

Spatiotemporal

Baseline

Table 1 provides a quantitative comparison using Pearson correlations between reconstructed and original images across different AlexNet layers (Section 3.4). The spatiotemporal decoding model achieved the highest feature correlation across most layers, highlighting its superior ability to capture and reconstruct the features present in the original stimuli. Specifically, the spatiotemporal model excelled in the deeper layers ( fc7 and fc8 ), which capture high-level feature representations, demonstrating the model's capability to capture both fine-grained and abstract features encoded in the neural responses.

The learned receptive field layer adapts receptive field sizes by spatial location, with larger fields in the periphery and smaller fields centrally (Figure 7). Despite some receptive fields being smaller than one pixel, all electrodes still contribute information, with single pixels used when needed. This effect is likely due to the model's limited 96 × 96 field of view, constraining pixel allocation for very small fields.

We also ran model ablations to assess the effect of different loss functions (discriminator, pixel, and VGG loss) on performance (Figure 11). Additionally, we trained models on region-specific data (V1, V4, IT) to explore how brain region training affects reconstruction (Figure 12). These experiments highlight the importance of individual brain regions and the adaptability of models trained on occluded data.

4.1.2 Spatial occlusion analysis

Spatial occlusion analysis was conducted to identify the importance of different brain regions (V1, V4, IT) in the reconstruction process. This analysis involved occluding specific spatial regions of the neural response inputs and examining the effect on the quality of the reconstructed images.

V1+V4+IT

V1

V4

IT

1

Figure 2 provides visualizations of example stimuli and their corresponding reconstructions when inputs from different brain regions (V1, V4, IT) were selectively occluded. The columns represent the regions of interest, while the rows present different example stimuli. For each region, the neural responses were set to their baseline values (pre-stimulus onset) for the occlusion procedure.

To quantify the impact of occlusion on reconstruction quality, feature correlations with the original images were calculated using the pre-trained AlexNet, as shown in Figure 8. The bar plot depicts feature correlations across different AlexNet layers for reconstructions derived from V1, V4, IT, and all regions combined.

4.2 Spatiotemporal occlusion analysis

Spatiotemporal occlusion analysis evaluates how neural responses from different time windows contribute to reconstruction quality, offering insights into the brain's temporal processing of visual stimuli.

Figure 3 shows example stimuli and their reconstructions when multiple time windows are occluded, with only one time window (highlighted in yellow) being included in the model during inference. Each column represents a different set of time windows being occluded, with the first column showing the reconstruction using all time windows.

The colorfulness of reconstructions from V1, V4, and IT was evaluated using the Hasler and Süsstrunk metric. Figure 4 shows the colorfulness scores for each brain region and combined data. V1constrained reconstructions had the highest colorfulness scores and correlated strongly with early AlexNet layers ( conv1 , conv2 ), reflecting V1's role in processing basic features like edges and color. IT reconstructions, however, showed higher correlations with deeper AlexNet layers ( fc7 , fc8 ), which capture more abstract visual features.

These results highlight the hierarchical nature of visual processing, with V1 specializing in basic visual attributes and IT handling more complex features.

5 Conclusion

We presented a comprehensive approach to decoding naturalistic visual stimuli from neural responses using a fully convolutional neural network trained from scratch. The use of the THINGS dataset enriched our model's feature set, crucial for accurately interpreting brain representations. Our homeomorphic decoder, enhanced with an end-to-end inverse retinotopic mapping layer, effectively integrates spatial and temporal information, leading to high-fidelity and interpretable reconstructions. Our evaluations highlighted the spatiotemporal decoding model's superior performance, evidenced by high feature correlations with deep layers of pre-trained AlexNet. Spatial and spatiotemporal

/gid00021 /gid00028/gid00045/gid00034/gid00032/gid00047

/gid00016/gid00048/gid00047

/gid00016/gid00048/gid00047

/gid00016/gid00048/gid00047

/gid00016/gid00048/gid00047

/gid00016/gid00048/gid00047

/gid00023/gid00422

/gid00021/gid00024 /gid00496/gid00422/gid00426

/gid00021/gid00024 /gid00496/gid00424

/gid00036

/gid00021/gid00024 /gid00496/gid00424

/gid00036/gid00710/gid00423/gid00427

/gid00021/gid00024 /gid00496/gid00424

/gid00036/gid00710/gid00426/gid00423

/gid00021/gid00024 /gid00496/gid00424

/gid00036/gid00710/gid00428/gid00429

/gid00021/gid00024 /gid00496/gid00424

/gid00036/gid00710/gid00422/gid00421/gid00425

/gid00023/gid00425

/gid00047

/gid00422/gid00421/gid00421

/gid00496/gid00001/gid00047

/gid00423/gid00424/gid00424

/gid00047

/gid00422/gid00421/gid00421

/gid00496/gid00001/gid00047

/gid00422/gid00423/gid00428

/gid00047

/gid00422/gid00423/gid00427

/gid00496/gid00001/gid00047

/gid00422/gid00426/gid00424

/gid00047

/gid00422/gid00426/gid00423

/gid00496/gid00001/gid00047

/gid00422/gid00428/gid00430

/gid00047

/gid00422/gid00428/gid00429

/gid00496/gid00001/gid00047

/gid00423/gid00421/gid00426

/gid00047

/gid00423/gid00421/gid00425

/gid00496/gid00001/gid00047

/gid00423/gid00424/gid00422

/gid00047

/gid00422/gid00424/gid00424

/gid00496/gid00001/gid00047

/gid00423/gid00427/gid00427

/gid00047

/gid00422/gid00424/gid00424

/gid00496/gid00001/gid00047

/gid00422/gid00427/gid00421

/gid00047

/gid00422/gid00426/gid00430

/gid00496/gid00001/gid00047

/gid00422/gid00429/gid00427

/gid00047

/gid00422/gid00429/gid00426

/gid00496/gid00001/gid00047

/gid00423/gid00422/gid00423

/gid00047

/gid00423/gid00422/gid00422

/gid00496/gid00001/gid00047

/gid00423/gid00424/gid00429

/gid00047

/gid00423/gid00424/gid00428

/gid00496/gid00001/gid00047

/gid00423/gid00427/gid00425

/gid00047

/gid00422/gid00427/gid00427

/gid00496/gid00001/gid00047

/gid00423/gid00430/gid00430

/gid00047

/gid00422/gid00427/gid00427

/gid00496/gid00001/gid00047

/gid00422/gid00430/gid00424

/gid00047

/gid00422/gid00430/gid00423

/gid00496/gid00001/gid00047

/gid00423/gid00422/gid00430

/gid00047

/gid00423/gid00422/gid00429

/gid00496/gid00001/gid00047

/gid00423/gid00425/gid00426

/gid00047

/gid00423/gid00425/gid00425

/gid00496/gid00001/gid00047

/gid00423/gid00428/gid00422

/gid00047

/gid00423/gid00428/gid00421

/gid00496/gid00001/gid00047

/gid00423/gid00430/gid00428

/gid00010/gid00021

/gid00010/gid00021

/gid00715/gid00001/gid00424/gid00421/gid00421/gid00040/gid00046

/gid00715/gid00001/gid00042/gid00041/gid00046/gid00032/gid00047

/gid00715/gid00001/gid00001/gid00001/gid00423/gid00428/gid00040/gid00046

/gid00715/gid00001/gid00047/gid00045/gid00028/gid00036/gid00041/gid00032/gid00031/gid00001

/gid00001/gid00001/gid00001/gid00047/gid00036/gid00040/gid00032/gid00001

/gid00001/gid00001/gid00001/gid00050/gid00036/gid00041/gid00031/gid00042/gid00050/gid00046

L

Colorfulness ofROI-constrainedfeatures

35

V1

30

0.095

Distribution

Mean

V4

25

0.062

V1

IT

V4

20

0.058

IT

15

10

5

0

0.05

0.10

0.15

0.20

0.25

0.30

occlusion analyses provided insights into the contributions of different brain regions and time windows, affirming the hierarchical nature of visual processing. The end-to-end inverse retinotopic mapping facilitated accurate estimation of receptive fields, aligning with neurophysiological findings and enhancing model transparency. This work advances neural decoding by offering a scalable and interpretable framework for reconstructing high-quality images from brain signals. Future research will explore more sophisticated architectures, further integrating temporal information and applying this framework to other sensory modalities.

Broader impact

Neural decoding models for reconstructing naturalistic images deepen our understanding of the link between neural activity and perception, with promising applications in visual neuroprosthetics. This study specifically reuses data originally collected as part of an initiative aimed at restoring sight, adhering to ethical best practices by maximizing insights from a single dataset and minimizing

the need for additional animal studies. This approach aligns with responsible research standards, balancing innovation with animal welfare.

However, caution is needed when applying these models to human neuroprosthetics due to the complexity of human brain activity and behavior. Human behavior encompasses interactions with the environment, complex motor actions, and neuroplasticity, elements that models trained exclusively on data from static image viewing may not fully capture. For instance, visual feedback from motor activities adds layers of complexity beyond current model capabilities. Additionally, using these models to identify stimulation sites for neuroprosthetics may not replicate natural neural responses precisely, highlighting the need for experiments/clinical trials with human subjects, nuanced model development and careful interpretation of outputs.

Limitations

This study applies invasive MUA recordings in macaques, expanding on prior work with fMRI by offering higher signal quality and detail. However, the applicability of these results to noninvasive techniques like fMRI remains limited due to their lower signal-to-noise ratio and less detailed recording capabilities compared to electrode arrays. This distinction is important, as invasive neuroimaging remains rare in human research.

Additionally, transitioning this framework to human intracranial applications poses challenges, including potential scar tissue formation, immune responses, and device rejection over long-term recordings. Anatomical differences, such as vascular structures, may further impact device placement and stability. Future work could explore these adaptations for broader applications, including braincomputer interfaces (BCI) and neuroprosthetics for individuals with acquired blindness, for which, careful regulatory guidance and additional research will be essential.

Acknowledgments

This work was supported by three grants of the Dutch Organization for Scientific Research (NWO): STW grant number P15-42 'NESTOR', ALW grant number 823-02-010 and Cross-over grant number 17619 'INTENSE' and grant number 024.005.022 'DBI2', a Gravitation program of the Dutch Ministry of Science, Education and Culture; the European Union's Horizon 2020 research and innovation programme: grant number 899287, 'NeuraViper'; the Human Brain Project, grant number 650003.

References

Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology , 21(19):1641-1646, 2011.

Ya˘ gmur Güçlütürk, Umut Güçlü, Katja Seeliger, Sander Bosch, Rob van Lier, and Marcel van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. Advances in Neural Information Processing Systems , 30, 2017.

Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications , 8(1):15037, 2017.

Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral Cortex , 28(12):4136-4160, 2018.

Katja Seeliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güçlütürk, and Marcel Van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neuroimage , 181:775-785, 2018.

Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex. Neuroimage , 198:125-136, 2019.

Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLoS Computational Biology , 15(1):e1006633, 2019.

Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience , 13:432276, 2019.

Saman Sarraf and Ghassem Tofighi. Classification of Alzheimer's disease using fMRI data and deep learning convolutional neural networks. arXiv preprint arXiv:1603.08631 , 2016.

Karl Bäckström, Mahmood Nazari, Irene Yu-Hua Gu, and Asgeir Store Jakola. An efficient 3D deep convolutional network for alzheimer's disease diagnosis using MR images. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) , pages 149-153. IEEE, 2018.

Bruce Fischl and Martin I Sereno. Microstructural parcellation of the human brain. Neuroimage , 182:219231, 2018.

Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5115-5124, 2017.

Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. arXiv preprint arXiv:1801.10130 , 2018.

Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. SplineCNN: Fast geometric deep learning with continuous b-spline kernels. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition , pages 869-877, 2018.

Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-Gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems , 31, 2018.

Yunpeng Bai, Xintao Wang, Yan-pei Cao, Yixiao Ge, Chun Yuan, and Ying Shan. Dreamdiffusion: Generating high-quality images from brain EEG signals. arXiv preprint arXiv:2306.16934 , 2023.

Lynn Le, Luca Ambrogioni, Katja Seeliger, Ya˘ gmur Güçlütürk, Marcel van Gerven, and Umut Güçlü. Brain2pix: Fully convolutional naturalistic video frame reconstruction from brain activity. Frontiers in Neuroscience , 16, 2022.

Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science , 364(6439):eaav9436, 2019.

Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife , 12:e82580, 2023.

Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C Aoki, Kei Majima, Yusuke Muraki, and Yukiyasu Kamitani. Spurious reconstruction from brain activity. arXiv preprint arXiv:2405.10078 , 2024.

David Hasler and Sabine E Suesstrunk. Measuring colorfulness in natural images. In Human Vision and Electronic Imaging VIII , volume 5007, pages 87-95. SPIE, 2003.

A Appendix / Supplemental material

A.1 U-net architecture

Contracting path : The contracting path follows the typical architecture of a convolutional network. It consists of repeated application of two 3 × 3 convolutions (unpadded convolutions) followed by a ReLU activation and a 2 × 2 max pooling operation with stride 2 for downsampling. At each downsampling step, the number of feature channels is doubled.

Expansive path : Every step in the expansive path consists of an upsampling of the feature map followed by a 2 × 2 convolution ("up-convolution") that halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two 3 × 3 convolutions, each followed by a ReLU activation. The cropping is necessary due to the loss of border pixels during convolutions in the contracting path.

Skip connections : Skip connections are introduced between the contracting and expansive paths to combine low-level features with high-level features, facilitating finer reconstruction of the output image.

The input to the U-net is the retinal embedding E of dimensions 15 × 96 × 96 (15 channels and 96 × 96 spatial resolution) and the output is the reconstructed visual stimulus S of dimensions RGB × 96 × 96 . The U-net architecture enables detailed reconstruction by preserving spatial information through its symmetrical structure.

A

B

DISCR

DISCR

Tensor

Model (Training)

U-NET

Scalar

Model (Fixed)

Concat

Loss Function

Figure 5: Overview of how the main reconstruction model is trained. A. The U-NET component is trained with a stack of 2D tensors (illustrated in grey) as input. These tensors are processed to produce reconstructions (depicted in yellow). The difference between the reconstructions and the target stimuli (represented in blue) are computed using the adversarial loss, feature loss, and pixel loss. B. Concurrently, the discriminator component undergoes its training phase. It evaluates the reconstructed outputs from the U-NET (labeled as 'fake images') alongside the original target images (labeled as 'real images'). This evaluation plays a critical role in calculating the Adversarial Loss, which is instrumental in guiding the parameter updates for the U-NET. This synergistic training approach ensures the progressive enhancement of the U-NET's ability to generate increasingly accurate and realistic reconstructed images.

B

A

Recon-

struction

Target

Brain responses

RF activation maps

(R)

(M)

B2P

V1

V4

IT

(r1)

(rn+1)

(rm+1)

Inverse

+

+

Receptive

H

Field

(rn)

(rm)

(rp)

W

M

(μx,1,μy,1,z,1)

[MXH]

(μz,pHyp zp)

Initialized parameters

Learned parameters

8

8

size (pixels)

6

(pixels)

6

size

4

RF

2

RF

乙

Visual field

0

0

20

40

60

Visual field

0

0

20

40

60

Distance from center

Distance from center

(pixels)

(pixels)

V1

V4

IT

AlexNet correlations of ROI-constrained features

V1

0.228

0.188

0.278

0.208

0.137

0.155

0.323

0.330

Brainregion

V41

0.245

0.223

0.331

0.283

0.182

0.214

0.378

0.374

IT

0.194

0.186

0.264

0.234

0.152

0.180

0.348

0.352

All

0.358

0.320

0.429

0.385

0.292

0.344

0.534

0.579

(Trained)

0

20

40

60

80

100

Relative correlation (%)

Conv1

Conv2

Conv3

Conv4

Conv5

FC6

FC7

FC8

AlexNet correlations of brain features (temporal)

TWO

0.239

0.218

0.315

0.264

0.163

0.186

0.350

0.330

TW1 -

0.239

0.209

0.291

0.248

0.145

0.162

0.324

0.292

window

TW2

0.188

0.189

0.258

0.224

0.130

0.149

0.307

0.249

Time

TW3

0.154

0.157

0.210

0.185

0.096

0.109

0.246

0.175

TW4

0.147

0.152

0.212

0.189

0.096

0.113

0.244

0.204

Brain

0.372

0.334

0.443

0.401

0.318

0.377

0.579

0.610

0

20

40

60

80

100

Relative correlation (%)

Conv1

Conv2

Conv3

Conv4

Conv5

FC6

FC7

FC8

AlexNet correlations of ROI and time-constrained reconstructions

V1

V4

IT

V1+V4+IT

100-

0.01

0.13

0.14

0.07

0.10

0.26

0.13

0.12

0.08

0.07

0.18

0.11

0.06

0.04

0.07

0.33

0.29

0.25

0.18

0.20

0.17

Relative correlation

80-

60-

0.05

0.06

0.21

0.09

0.07

0.23

0.10

0.20

0.07

0.08

0.20

0.07

0.08

0.27

0.13

0.22

0.09

0.08

0.22

0.09

0.08

0.20

0.07

0.07

0.19

0.07

0.07

0.11

0.25

0.21

0.08

0.08

0.19

0.07

0.06

0.07

0.19

0.07

0.19

0.07

0.07

0.19

0.35

0.32

0.16

0.31

0.15

0.25

0.11

0.24

0.11

0.15

0.17

0.08

0.16

0.16

0.12

0.19

0.18

0.16

0.16

0.10

0.17

0.16

0.16

0.16

0.16

0.15

0.13

0.10

0.10

0.18

0.23

0.19

0.26

0.25

0.22

0.19

0.19

40-

0.16

0.20

0.19

0.18

0.18

0.28

0.21

0.20

0.18

0.18

0.21

0.20

0.17

0.17

0.17

0.32

0.29

0.26

0.21

0.21

20-

0.13

0.15

0.14

0.14

0.13

0.19

0.15

0.15

0.14

0.14

0.16

0.15

0.14

0.13

0.13

0.22

0.21

0.19

0.16

0.15

0.11

0.18

0.13

0.13

0.12

0.20

0.15

0.14

0.11

0.12

0.15

0.13

0.12

0.11

0.11

0.24

0.24

0.19

0.15

0.15

0

103

32

60

6

112

138

64

119

145

9

26

5

104

×

6

Time windows (ms)

Time windows (ms)

Time windows (ms)

Time windows (labeled)

Conv1

Conv2

Conv3

Conv4

Conv5

FC6

FC7

FC8

no ablation

no VGG

no L1

no discr.

Model ablations

V1+V4+IT

IT

V4

V1

Model trained on brain region of interest

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Figures 1, 2, 9, Table 1 and their corresponding descriptions in the main text.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This work does not include or introduce new results of purely theoretical nature.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: A data manuscript for the THINGS macaque visual cortex dataset is currently in preparation (see https://things-initiative.org ). The code has been made available 2 .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 3.3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars on the quantitative results in Table 1 are not reported because it would be too computationally expensive to create them, and this statistic does not impact the demonstrated functionality of the introduced method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 3.3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All the reported data collection experiments strictly adhered to the local ethical guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The data was collected by authors of this paper. The stimulus dataset is publicly available and properly referenced and credited. The models are original work, or properly referenced and credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The newly created models and data in this paper are documented in all necessary detail.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors record multi unit activity (MUA) from macaque ventral stream (V1, V4 and IT) and train a CNN decoder to reconstruct the visual stimuli. The authors present three variations decoding attempts: a baseline CNN decoder that maps MUA directly to image space, and two U-Net based decoders that differ in the fact that the first only uses spatial information (averaging over the time dimension), while the latter is time-resolved (spatiotemporal). Author find that the spatiotemporal decoder yield the highest performances as evaluated via feature correlation and present occlusion analysis (either masking brain regions or time windows).\n\nSTRENGTHS:\nThe paper tackles an established field such as stimuli reconstruction using electrophysiological neural signals (MUA) as opposed to the more common fMRI techniques. This grants superior time-resolution which offers ways for novel experiments. Indeed author showed interesting effects of decoding from different time windows in different brain regions.\nFurthermore the paper is in general well structured and presented.\n\nWEAKNESSES:\n- The paper present some inconsistencies between Text and Figures.\n    * In line 267 it is said that the spatiotemporal model achieve the highest performance, however Table 1 reports the End-to-End models as the top scorer.\n    * In line 288 author link to Figure 9 while the correct one should be Figure 3. However, description of Figure 3 is still inconsistent as in line 290 authors say that last column report full-data reconstruction while Figure 3 has it in the first column.\n    * In Figure 8 meaning of x-axis is unclear. Moreover, Figure caption states that \"Deeply colored points indicate higher correlations [...]\", however it is unclear how this is assessed since for example the V4 row has higher correlation with fc7 and fc8 than IT (which is instead marked). Similar remarks hold for Figure 9, which has an incomplete caption (a trailing \"The\").\n- Hyperparameters for decoder loss (sec. 3.6.2) are missing.\n- Authors introduce the \"Learned Receptive Field\" (LRF) layer claiming it \"enhances understanding of (model) structure and interpretive capacity\", however no discussion about is presented. Only related result is Figure 7, which is never mentioned in the main text and presents somewhat puzzling results (see Questions section of this review).\n- In the occlusion analyses, author perform model *inferences* on truncated data, however a model *trained* on the same occluded data might compensate the measured decrease in performances.\n- Paper lacks comparisons with existing decoding techniques, albeit originally introduced for other - but structurally similar - modalities such as EEG signals (see Y. Bai et al., DreamDiffusion (2023)).\n\nQUESTIONS:\n- Figure 7 (not discussed in the main text) reports the learned 2D receptive field. Most learned RF seem to have a smaller-than-one pixel size, does this imply that no information from that electrode is used at all?\n- Why do authors refer to their decoding technique as \"homeomorphic\"?\n- Would a model trained on occluded data still performed as poorly as the results presented in the paper?\n- Could the difference in measured performances between the baseline and homeomorphic model be simply attributed to the fact that the full model has higher complexity (more parameter), as the baseline decoder only leverages *half* of the U-Net architecture?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSETTING: Decoding (reconstructing) observed images via Utah-array recordings made from the visual cortex of a macaque.\n\nAPPROACH: A GAN, but taking (transformed--see below) neural data as input, and with some additional losses.  In particular, in addition to the standard adversarial loss, the generator/decoder is aligned with VGG-19 at various layers, and its output is penalized with a reconstruction penalty (mean absolute error).\n\nRaw neural data are not passed into the generator/decoder (except in the baseline model); instead, multi-unit activity is first mapped to a retinotopic/pixel space, either with (learned) isotropic Gaussian receptive fields, or by mapping to features of a pre-trained Inception network.\n\nRESULTS: The data and model evidently support high-quality reconstructions (Fig. 1).  The authors believe the decoder with a spatiotemporal inverse retinotopic map works best.  They also perform qualitative experiments to determine which cortical areas (V1, V4, IT) and time windows contribute what to image decoding.\n\nSTRENGTHS:\nThe image reconstruction is, to my knowledge, state of the art.  The authors also convincingly demonstrate the importance of the retinotopic mapping.  Finally, the experimental setup (15 Utah arrays in one animal!) is heroic.\n\nWEAKNESSES:\nI put much of the details of these weaknesses into the questions (below).\n\nIt's not clear what we learn from the results following Figure 1.  Part (but not all--see the questions) of the problem may be due to the presentation.  The model itself is also not presented clearly: The model figures (Figs. 5 and 6 in the Appendix--not referenced in the main text) do not cover the first type of inverse retinotopic map, and introduce yet more symbols not appearing in the text.  The equations in the text appear incomplete (see questions).\n\nQUESTIONS:\n> The number of arrays is certainly much larger than what is used by almost any research group.  It would be very helpful to see how results scale with number of neurons (really, electrodes).  Such a figure would also perhaps help make sense of Fig. 2: how much of the degradation in image reconstruction there is due to using fewer neurons, and how much due to using neurons only from this or that area?  (If there isn't time to run this experiment, perhaps the authors have some other way of answering this question.)\n\n> Along these lines: The authors somehow only get ~600 neurons from these fifteen arrays, apparently because they reject neurons with \"intraclass correlations\" less than 0.4.  What are these ICCs?  (What are the classes?)  Does this really improve performance?  How did the authors come up with the threshold of 0.4?\n\n> Generically, I understand why one would want to map from electrode space into retinotopic/pixel space (basically so that the model can be a UNet from there on out.)  But I don't understand the explanation of the \"inverse retinotopic mapping\" the authors given is Section 3.  The authors give formulae for computing the retinal embedding E and say that the parameters of these functions are learned.  But where is E subsequently used?  For the \"pre-trained mapping,\" a cost function is given, but E doesn't appear in it.  Instead, it contains yet more weights, in this case mapping from a pretrained Inception network to neural responses.  (In section 3.3.1, the letter W or w is used with seven different subscripts, some of them seemingly referring to the same weights and others not.)\n\nIn the appendix, there is a figure (not referenced in the main text!) that seemingly corresponds to the end-to-end inverse retinotopic mapping.  But E doesn't appear in this figure either.  The caption has a broken reference to an equation.\n\nCan the authors clarify this part of the model?\n\n> The authors state in the text that the spatiotemporal model achieves the highest correlations with AlexNet features, but Table 1 shows it to achieve the lowest (excluding the baseline; the end-to-end model achieves the highest correlation).  Is the text or the table in error?\n\nThey also claim that the spatiotemporal model qualitatively outperforms the other models (Fig. 1), but that claim is not obvious to this reader's eye.  (The horse and the butterfly, e.g., look worse; may others are a push.)\n\n> Can the authors confirm that the 100 images used for testing were never used during training?\n\n> What conclusions do the authors draw from Fig. 3?\n\nIn any case, the results would seem to depend strongly on the values assumed for synaptic delays:  This determines what time window in IT corresponds to what time window in V1, etc. (otherwise the windowing scheme is essentially random).  Can the authors justify high confidence in this synaptic-delay time?\n\nMINOR:\n> Section 4.1.2 examines the effect on decoding of setting certain areas of visual cortex to their baseline values.  Why is this called \"spatial occlusion\"?\n\nAlso, the column headings are described as identifying which brain region has been excluded, rather than included, but that obviously can't apply to the second column (\"V1 + V4 + IT\").  I think the text is wrong here.  (But if not, perhaps the authors could reverse the labels, letting all column headings state which areas are *in*cluded.)\n\n> Figs. 8 and 9: I don't understand the lightness/darkness color scheme, which doesn't seem to correspond to the correlation values listed in the figure.  Can the authors clarify this?\n\n> Why is the decoder called \"homeomorphic\"?\n\n> \"...spatially downsampling data to 8 Hz...\"\nWhat does this mean (and how could spatial sampling be denominated in Hertz?)?\n\n> \"...x_e and y_e denote the spatial coordinates of electrode e....\"  I think it would be less confusing to say the \"retinotopic coordinates corresponding to electrode e,\" or something similar.\n\n> The reference to Fig. 9 (l. 288) should be to Fig. 3.\n\n> The supplementary figures should be referenced in the main text.  (Figure 6, e.g., would be very useful to see while reading Section 3.4.)\n\n> Fig. 3: \"...with the final column representing the full data reconstruction\" -> \"...with the *first* column.....\"\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The authors present a CNN-based decoder that illuminates the distinct information encoded in V1, V4, and IT neuronal populations. The decoding results are remarkably good, and their accomplishment in decoding natural images from neuronal-level signals is unprecedented and the best so far.

STRENGTHS:
Their novel space-time resolved decoding technique and the learned receptive field layers are new and interesting. Perhaps most impressive of all is their unprecedented data, recorded simultaneously from 15 Utah arrays from a single monkey. It can be considered a milestone in neuroscience research.

WEAKNESSES:
Analytical innovation is relatively minor. I am not sure what new insights about the visual cortex we learned. Nevertheless, proving that it can be done (with 4-7 Utah arrays per area), and doing it beautifully is still a remarkable feat that should be admired and applauded.

QUESTIONS:
No specific questions.

LIMITATIONS:
No concerns.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a CNN-based decoder to reconstruct naturalistic images from macaque brain signals. To this end, the paper presents the Learned Receptive Field (LRF) layer to enhance the reconstruction and understanding of the model's structure and interpretive capacity. Here, the work aims to interpret brain activity by transforming low-level to high-level brain signals. It focuses on the readout characteristics of neuronal populations in areas V1, V4, and IT. The evaluations of the model are performed using the THINGS dataset, and the results demonstrate the effectiveness of an end-to-end model and provide interpretations, such as the crucial role of V1 in color processing.\n\nSTRENGTHS:\n- The integration of a U-Net architecture for the pixel-to-pixel mapping of retinal embeddings to visual stimuli is intriguing. \n\n- The proposed method is end-to-end with different parameters, making it adaptable to the target problem.\n\n- Though the method of the paper is data-specific, the proposed model’s results are somehow interpretable\n\nWEAKNESSES:\n- The proposed model is data-specific, which could potentially limit its application.\n\n- The overall loss consists of multiple criteria, each with its weight, which might make the model harder to train. Specifically, mixing - adversarial loss with pixel and feature loss can make it challenging to control the training dynamics.\n\n- The paper does not include a complete ablation analysis of the different modules.\n\nQUESTIONS:\n- Could you elaborate more on the motivation behind using Inception v1 for different cortical areas? How does this layer contribute to the generalization of the image reconstruction?\n\n- Is the model generalizable to other types of neural data from different species or different types of recording equipment?\n\n- In equation line 151, how does the standard deviation influence the spatial spread of the receptive field and overall reconstruction?\n\n- In Pre-trained Inverse Retinotopic Mapping, why do we need two embeddings? Can’t the model handle it with a unified spatiotemporal module instead of having separate embeddings?\n\n- What about employing an existing method to build a baseline, such as using a recurrent CNN or Vision Transformer? For a fair comparison, the evaluation should include some of the other state-of-the-art architectures.\n\n- Is the quantitative comparison presented in Table 1 a standard manner of measuring performance? Is it used in the literature? If not, what would be other metrics, such as the R² score?\n\n- What about the ablation analysis of the loss functions? Also, could you present the dynamic criterion of each loss during the training and validation phases?\n\n- How did you set the hyperparameters of the model? Also, the full implementation details of the model are missing.\n\nLIMITATIONS:\n- I think the main limitation of the paper is the limited area of applicability of the proposed framework.\n\n- The evaluation, both in terms of architecture and criterion, is missing to determine the necessity and effectiveness of all the modules of the model.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}