MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs
Authors: Jan Sobotka
Luca Baroni
Ján Antolík
Venue: NeurIPS 2025 poster ()
Keywords: Neural decoding
Neural population activity
Most exciting inputs
Image reconstruction
Visual cortex
Primary area: neuroscience_and_cognitive_science
TL;DR: MEIcoder uses most exciting inputs, SSIM loss, and adversarial training to reconstruct high-fidelity visual stimuli from neural population activity, excelling in data- and neuron-scarce settings.

Abstract: Decoding visual stimuli from neural population activity is crucial for understanding the brain and for applications in brain-machine interfaces. However, such biological data is often scarce, particularly in primates or humans, where high-throughput recording techniques, such as two-photon imaging, remain challenging or impossible to apply. This, in turn, poses a challenge for deep learning decoding techniques. To overcome this, we introduce MEIcoder, a biologically informed decoding method that leverages neuron-specific most exciting inputs (MEIs), a structural similarity index measure loss, and adversarial training. MEIcoder achieves state-of-the-art performance in reconstructing visual stimuli from single-cell activity in primary visual cortex (V1), especially excelling on small datasets with fewer recorded neurons. Using ablation studies, we demonstrate that MEIs are the main drivers of the performance, and in scaling experiments, we show that MEIcoder can reconstruct high-fidelity natural-looking images from as few as 1,000-2,500 neurons and less than 1,000 training data points. We also propose a unified benchmark with over 160,000 samples to foster future research. Our results demonstrate the feasibility of reliable decoding in early visual system and provide practical insights for neuroscience and neuroengineering applications.

MEIcoder:
Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs
Jan Sobotka 1.+
Luca Baroni²
Ján Antolik
EPFI. Switzerland
Charles University, Czechia
Abstract
Decoding visual stimuli from neural population activity is crucial for understand- ing the brain and for applications in brain-maching interfaces. However, such biological data is often scarce, particularly in primates or humans, where high- throughput recording techniques, such as two-photon imaging, remain challenging or impossible to apply. This, in turn, poses challenge for deep learning de- coding techniques. To overcome this, we introduce MElcoder, a biologically informed decoding method that leverages neuron-specii most exciting inputs (MEIs), structural similarity index measure loss, and adversarial training. ME- Icoder achieves state-of-the-art performance in reconstructing visual stimuli from single-cell activity in primary visual cortex (V1), especially excelling on small datasets with fewer recorded neurons. Using ablation studies, we demonstrate that MEIs are the main drivers of the performance, and scaling experiments, we show that MElcoder reconstruct high-fidelity natural-looking images from few 1,000-2,500 neurons and less than 1,000 training data points. We also propose a unified benchmark with over 160,000 samples foster future research. Our results demonstrate the feasibility of reliable decoding in early visual system and provide practical insights for neuroscience and neuroengineering applications.
1 Introduction
Recent progress in machine learning (ML), together with advances in collecting single-cell brain activity data, has enabled powerful data-driven approaches to model the brain. The dominant approach is to use ML models to characterize the stimulus-response function, i.e., to predict brain activity in
response to external variables (encoding), such as visual stimuli [2 9 24 27 28 46). The inverse
problem of decoding high-fide fidelityy stimuli from brain activity started to garner attention only relatively
recently [ 18] 33 37]. One of the main reasons for this is the inherent difficulty decoding
high-information content, such images, from small number of neurons that provide highly compressed and noisy version of the original stimulus (21] This inverse problem exaggerated by the scarcity of single-subject data, which is necessary to accurately capture the unique response- stimulus mapping of the subject's 's visual system. Therefore, there is a need for decoding techniques that can learn from limited training examples and from small number of recorded neurons.
However, training machine learning models from scratch on such scarce single-subject data tends to yield low-fidelity image reconstructions that lack sufficient detail [8] 33 [1]. Conversely, leveraging pre- ire-trained models, particularly those from the field of Generative AI (GenAl), provides high- resolution but unreliable reconstructions, often containing hallucinated content B1 35 37 Indeed, as 39 and our results in infsubsection show, prior decoding methods employing (text-guided) GenAI heavily biased toward their pre-training distribution of semantically rich images. This leads to deceptively realistic, yet often pixel-level inaccurate, image reconstructions that do not capture
"Correspondence to: jan. sobotka@epf1..h
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the true fine spatial characteristics of the visual stimulus (and occasionally outright hallucinations), hindering the reliability of these methods and limiting their scope of application [2T]
To remedy these problems, we develop an end-to-end trained decoding method called MElcoder, which utilizess prior knowledge from computational neuroscience and adversarial objectives to outper- form previous methods on three difficult datasets. MElcoder achieves high-fideling reconstructions by (1) utilizing a strong computational prior in the form of -spurfi most exciting inputs (MEIs), (2) a novel training loss based on the structural similarity index measure (SSIM), (3) an auxiliary adversarial training objective that pushes reconstructions toward a manifold of natural- -lookig im- ages, and (4) serfmietr-efficient architecture that allows training on multiple distinct datasets. In summary, the main contributions of this paper are as follows:
1. We develop method that achieves state-of-the-art performance in decoding visual stimuli from neural population activity in VI. This result demonstrates that decoding high-fideling reconstructions is feasible with the currently available single-subject data.
2. To understand the scaling behavior of our model, we analyze the relationship between performance, the number of available recorded neurons, and the amount of training data.
3. To stimulate further developments this area, we aggregate datasets from multiple sources into decoding benchmark with over 160,000 samples.
2 Related work
Most previous work on decoding brain signals has been done with magnetoenceehallgraphyy (MEG) and functional magnetic resonance imaging (fMRI) data. For example, (6] leveraged pre-trained image embeddings and pre-trained image generator to perform real-time decoding of MEG signals 41 used GenAl techniques, such as pre-trained diffusion models [9] 40.
into images. 30 35
36
to decode images from fMRL. Their techniques were able to reconstruct the semantic information in the visual stimuli, such as object categories, but were unable to capture low-level features of the images. Furthermore, study by (39 provided formal analysis to demonstrate that prior decoding approaches based on diffusion models suffer from so-called "output dimension collapse". which restricts their decodable features. Their study, as well as our results in subbecction 4.4, also show that prior GenAl- -based decoding techniques tend to hallucinate, leading to untrustworthy and spatially inaccurate reconstruction of novel images. These findings highlight the importance of choosing an appropriate prior and carefully balancing it with the neural data to achieve reliable reconstructions.
One of the first studies investigating decoding from neuron- data leveraged known retinotopy to reconstruct simple visual stimuli and mental imagery 443. Later, using Generative Adversarial Networks (GANs) [17 and other deep learning approaches, a series of works [88 25
31
33 51
showed promising initial results in decoding more complex stimuli from higher-order areas of the visual system, such as V4 and the inferior temporal cortex. For example, (25) incorporated
known biological properties of neurons in the visual system into their brain-inspired architccture to reconstruct images from sequences of spikes. More recently, (23 introduced a homeomorphic decoder with learned inverse retinotopic mapping to reconstruct naturalistic images from macaque brain signals. As evaluate their method in Isbbsecction 4.4| find that its fully end-to-end trained retinal embeddings are incapable of reconstructing high-fidelity images from our limited mouse data. Moreover, their architecture is not designed to work with multiple distinct datasets, which prevents it from integrating learning signals from data across different subjects.
Instead of training to decode images directly, the novel approach from pre-trains CNN encoder and, at inference time, performs an itcrative encoder inversion procedure. It begins by (randomly) initializing the pixels of the reconstructed image and then takes gradient steps the image pixels to minimize the difference between the ground-truth responses and the responses predicted by the encoder from the reconstructed image. In addition to the computational burden at inference time, this approach does not directly optimize for reconstruction quality at the pixel level, but rather for reconstruction in the space of neuronal responses predicted by the encoding model. This can lead to image artifacts and potentially limit its ability to accurately reconstruct target images.
Lastly, similarly as for the fMRI data, 31 leveraged pre-trained diffusion model and a CNN encoder to decode neural population activity. More specifically, their method, Energy Guided Diffusion, guided the inversion of a CNN encoder using a frozen diffusion model. This approach enabled sharper
2

---

Neuron Embeddings
Most Exciting Inputs
Pointwise Convolution
CNN Core
Reconstructed image
Fi 6,
Feed Forward
C;
Neural resporsies
Figure I: Architecture of MElcoder. First, individual neural responses r and their corresponding neuron embeddings get projected by feed-forward network into context representations C, (left). The context representations are then pointwise-multiplas with MEIs and passed through pointwise convolution layer (center). Finally, the output of the convolution layer is used by the CNN core to reconstruct the final image (right).
reconstructions by introducing strong bias toward the image statistics of the pre-trained diffusion model, but still optimized for reconstruction quality in the neural activity space as captured by the encoding model, rather than in the image pixel space. In turn, the resulting reconstructions from neuronal responses often contained spurious features that were not present the original images, which we also confirm in our experiments (subsection 4.4). Overall, these challenges highlight the need for new decoding approach that mitigates hallucination, preserves fine-grained visual details, and more effectively balances neural evidence with generative priors.
3 MElcoder
To overcome the limitations of previous methods, introduce MElcoder. It consists of two components: a single core module and one or more readin modules. Each readin trained separately for its respective single-subject dataset and acts an embedding function of the neural activity into the core's 's latent space. The core, on the other hand, is shared across all possibly heterogeneous multi- subject datasets and maps from its latent space into images. The purpose reuse learning signals across recordings from different subjects, even when their number of neurons and response-stimue -timulss mappings differ.
Core. To keep MElcoder parameter-efficien and suitable for low-data regimes, we build the core as six-layer convolutional neural network (CNN) with batch normalization, ReLU activation function, and dropout. Further details and hyperparameters can found /subsection A.3]
Readin. One of our main contributions comes in the readin module. While prior work in this area used simple beed-forward neural networks to translate the brain signals into the latent space 35 36,,
we found it to be highly suboptimal as it quickly overfits and does not generalize well. This motivated introduce novel readin module, where inject additional prior knowledge and regularization into the decoder in the form of most exciting inputs (MEIs). In the context of visual processing, MEIs are images that maximize the response of given neuron the visual system, thus being highly informative about the coding properties of individual neurons 4 32 45). They are easily obtainable from the training data, and their generation needs be done only once (subsection A.4).
The intuition behind MElcoder stems from the fact that MEIs contain information about the receptive fields of neuron the spatial patterns that most strongly stimulate them. This suggests straightfor- ward linear decoding approach: overlay the MEIs all neurons on top of each other, weighted by their respective neural responses, to reconstruct the image. This intuition is the primary building block of our method. However, in our method, since coding in V1 is not linear, we leave the combination of MEIs onto the nonlinear blocks in the MElcoder's core. Furthermore, to provide greater flexibility to the decoding process, we add learnable neuron embeddings that can encode additional properties of the neural code. We show the MElcoder pipeline Figure and describe it more formally below.
In of linear neurons, MEIs equivalent to the receptive fields.

---

Let L E R" be the vector of responses of n neurons, and h. w be the height and width of the images decode. The first step of our readin is to independently embed individual neural responses r; together with their corresponding learnable neuron embeddings
{ej E B Jj=1 using one-layer neural network Go : RI-d R- into context representations C -R.... The second step is pointwise-muliply the precomputed MEIs M E TR.,,,,w with the reshaped context representations C E RR..,,. to obtain neural maps H = M O C. Lastly, to obtain a constant numher of output channels for readins operating with possibly different numbers of neurons. we apply pointwise convolution to transform the neural maps of shape X h x w into compressed neural maps H. Rde.h. which form the input the core module of the decoder.
3.1 Training
Unless stated otherwise, we train the decoder end-to-end from random initialization. The full training objective consists of two terms: (1) SSIM-based reconstruction loss, and (2) adversarial loss. We combine the two using weighting coefficients ASSIM = 0.9 and 'ADV = 0.1.
SSIM-based reconstruction loss. Given that the images for reconstruction are encoded by natural vision, we employ a modification of the Structural Similarity Index Measure (SSIM) [47 steer the decoder toward perceptually important image features. Specifically, we use the negative log-SSIM loss defined as:
CSSiM y) = -lo (SSIMM,, 9) +1 +
(I)
2
where y. E R. are the ground-truth and reconstructed image, respectively, and 10-6 6 is introduced for numerical stability. As later shown in inssubsection 4.5 with ablation studies, we found this training objective more effective at producing perceptually accurate reconstructions compared to standard objectives such as the mean squared error (MSE). Unlike perceptual loss functions based on embeddings from pre-trained models, which we found to be unstable in training and produced fegu-frey artifacts, the SSIM objective required no further tuning and led to consistent results.
Ad Adversarial training. The limited amount of data may not give the decoder enough training signal to learn reconstruct high-fidelity natural-looking images, and may potentially lead overfitting. To counteract this, we use an auxiliary adversarial objective similar to that used in GAN More specifically, we train a secondary CNN classify whether given input image a reconstruction from our decoder or a reference (ground-truth) image from the dataset (ubsecttnn A.3) for details). Given this discriminator Dø. Re..,w -> [0. 1). we add the following loss for training the decoder:
LADV(y) = (Dx - 1)²
Note that unlike standard GANs, our decoder is not trained generatively and is conditioned only on neuronal responses. Moreover, its objective directly optimizes for spatially accurate reconstruction, and its priors are more aligned with the biological vision through the MEIs. These factors make MElcoder more reliable and faithful the true response-stinge stimulus function that is trying to capture.
We train MElcoder for 300 epochs using the Adam W optimizer [26] with a learning rate and weight decay found using hyperparameter search and the validation dataset. Similar early stopping (2).. we pick the best model from training based on the Alex(5) score measured on the validation dataset.
Experiments
We compare MElcoder to sfate-of-thh-ar baselines on three datasets, two of which represent data- and neuron-constrained settings. Given that there are currently no unified benchmarks for visual decoding from neural population activity with sufficient heterogeneity, we propose our own as an aggregation of previously published data sources.
4.1 Data
Brainreader dataset. The BRAINREADER data comes from mouse VI and was originally introduced by [... For our experiments, we use data from a single mouse, where recorded spike traces
3 Prevvios work has also found adversarial objectives effective for reconstructing visual stimuli "This name, not used by the original authors, is chosen differentiate this dataset.

---

were aggregated and averaged over a 500 ms time window following the presentation of grayscale images. We divide the dataset into training, validation, and test sets of 4,500, 500, and 100 samples, respectively. Each data point consists of a 36 x 64 px grayscale image sampled from ImageNet [12], along with evoked neuronal responses of 8,587 neurons. For one of our experiments, we use data from 8 mice, where the number of recorded neurons varies between the individual mouse datasets. We refer the reader tofsubsection A.2 for additional details.
SENSORIUM 2022 dataset. We repurpose the mouse dataset published by the SENSORIUM 2022 competition 48 for our decoding task. Specifically, the dataset used in our experiments contains time-binned recordings of responses from 8,372 neurons for 4,984 images, which we split into training and validation sets. For final evaluation, we use testing set of 100 images with corresponding 10-trial averaged neural responses. For one of our experiments, we pre- pre-train on data from 5 mice available in the SENSORIUM 2022 data corpus and fine-tune on single-mous, data.
Synthetic cat V1 dataset. Lastly, since most of the currently publicly available datasets suitable for decoding in V1 are greatly data-constrained and limited mouse monkey data, we leverage highly biologically realistic spiking model of cat VI from 3 to generate large synthetic dataset. This spiking model has been extensively validated in series of studies, demonstrating a wide range
of accurately replicated properties of V1 coding 3 42 4... For data generation, we sample 50,250 grayscale images from ImageNet and encode them into responses of 46,875 neurons using the spiking model. We split this additional synthetically generated dataset into training (45,000), validation (5,000), and test (250) sets, Additional details can be found in |uubsection A.2
4.2 Evaluation metrics
For robust evaluation, we follow previous studies 122
23 36
and use (1) SSIM, (2) Pearson
correlation between the pixel values of the reference and the reconstructed image, and (3) feature correlation with two-way identification using a pre-trained Alex For the feature correlation, we follow (36) and extract feature representations at the second and fifth layers of
-rretrranpe AlexNet to evaluate the two-way identification ability. This identification score refers to the percentage of correct comparisons assessing if the reference image embedding is more similar to the reconstructed image embedding or to randomly selected image embedding. We use the implementation of the two-way identification from [36]_ For additional experimental details, please refer to (subsection A.
4.3 Baselines
We compare MElcoder with five baseline methods, four of which were developed on population recordings of single-cell activity, and one of which was originally designed on fMRI. The first two baselines are the Inverted Encoder (InvEnc) [11] and Energy Guided Diffusion (EGG) [31] as described in section 2] InvEnc showed state-of-the-art performance on the BRAINREADER dataset, and both methods are representative of class of decoding methods that invert pre-trained encoder models for image reconstruction, minimizing eтrors in response space.
The third baseline is the homeomorphic decoder MonkeySee [23]. It employs the U-Net architecture, feature representations from a pre-trained CNN, and is trained end-to-end with VGG feature loss, L1 reconstruction loss, and an adversarial objective. Another baseline similar to MonkeySee is the CAE decoder [0], which employs a series of fully-connected layers followed by downsampling and upsampling convolutional blocks. Both methods showed improvements over the previous best-performing approaches, and we consider them strong representatives of direct pixel decoding.
The last baseline to which compare is MindEye2 (36.. which combines multiple pre-trained models from the field of GenAl with end-to-end trained networks. It demonstrated high-fidelity reconstruction on the fMRI Natural Scenes Dataset (1, outperforming all previous methods.
For all five baselines, we use the code provided by the original authors. To find the optimal hyperpa- rameters, we use the same procedure as for our method: we perform a hyperparameter search using only the training and validation datasets, and pick the best model checkpoint from training based on the performance on the validation data.
5

---

Table 1: Results on the test sets from the BRAINREADER and SENSORIUM 2022 datasets. Best results are highlighted red and second-best in bold. All values are means over three random seeds; standard deviations and results for higher-level metries are available in Tables 3 and respectively.
BRAINREADER
SENSORIUM 2022
Method
SSIM
PixCorr
Alex(2)
Alex(S)
SSIM
PixCorr
Alex(2)
Alex(S)
InvEne
.321
.611
.999
.896
.288
.433
.915
.720
EGG
.256
.495
.758
.659
.256
.365
.777
.755
MonkeySee
.232
.565
.967
.226
.185
.338
.564
.523
CAE
.256
.638
.930
.730
.287
.539
.656
.549
MindEye2
.277
.560
.46
.878
.210
.471
.877
.762
MindEye2 (FT)
.234
.516
.920
.836
.243
.499
.988
.799
MElcoder
.400
.679
.998
.990
.331
.503
.988
.896
MElcoder (FT)
.424
.706
.999
.977
.318
.486
.955
.908
Target
InvEnc
EGG
MonkeySee
CAE
MindEye2
MindEyc2 (FT)
MElcoder
MElcoder (FT)
Brainreader
SENSORIUM 2022
Figure 2: Reconstructions on the BRAINREADER (top) and SENSORIUM 2022 (bottom) datasets.
4.4 Results and discussion
Tables I and 2 show that MElcoder outperforms all baselines in decoding visual stimuli on all three datasets (considered as an aggregate over the metrics). Furthermore, we can see that the improvements most significant in the biological data-scarce and neuron-scarce settings (BRAINREADER and SENSORIUM 2022 datasets), highlighting the suitability of MElcoder for such cases. Closer qualitative evaluation in Figure 2] further confirms that MElcoder handles the low-data regime well, producing more detailed and faithful reconstructions compared to other methods. For example, while the MindEye2 reconstructions on the BRAINREADER dataset are relatively sharp but spatially inaccurate, and MonkeySee reconstructions are the opposite, MElcoder achieves both at the same time. This result can be explained by the difference in prior knowledge injected into these three decoders. Namely, MindEye2 reconstructions are steered toward natural- natural-looking, semantically rich images due to its GenAl components, and Monkey employs a weak prior the form of frozen feature embeddings from pre-trained CNN. MElcoder, by contrast, leverages MEIs prior that is more closely aligned with the neurons from which it is actually decoding.
We also test how well MElcoder handles multi- (pre-training and subsequent single-subject fine-tuning (FT). Comparing the FT version against others in Table [and Figure we can see that the split of the architecture into a core and subject-specific readins allows MElcoder to work well in this more heterogeneous training regime, outperforming the single-subject training in some cases.
"Combined data from 8 and 5 mice from BRAINREADER and SENSORIUM 2022 datasets, respectively.

---

Table 2: Results the test set of the SYN- THETIC CAT VI dataset. Best results are high- lighted in red and second- best in bold. All val- ues are means over three random seeds; standard deviations are available in Table 4
Target
EGG
MonkeySee
MindEyo2
SYNTHETIC CAT VI
Methed
SSIM
PixCor
Alex(2)
Alex(5)
InvEnc
.771
.833
.966
.978
FGG
.640
.667
936
.906
MonkeySee
.607
.723
.982
.958
Figure SYNTHETIC CAT VI reconstructions. See subsection A.12 for more examples.
CAE
.637
.792
.927
.776
MindEye2
.599
.757
.977
.999
MElcoder
.774
.987
MEls
X
x
X
Neuron embeddingg
x
X
X
X
Percentage change (%)
-20
-30
SSIM
PixCorr
Alex(2)
Alex(S)
Figure 4: Ablation study on sub-components of MElcoder: The percentage (y-axis) is calculated with
respect setting with no ablations. Reported values are the average across the three datasets.
4.5 Further analysis
A Ablation study. We conduct ablation studies to quantify the importance of individual components
of MElcoder. Specifically, we evaluate performance after removing MEIs from the readin module by using only the context representations C as the neural maps H (3). Similarly, we evaluate the metries after removing the neuron embeddings e; from the input of the neural network Iv. and after
substituting the standard MSE training objective in place of the SSIM- reconstruction loss. We report the average over the three datasets.
The results in Figure 4 show that MEIs have the most significant positive influence on the state-of-the-an thatttt--rrtppp performance of MElcoder. In- deed, compared to the individual ablations of the SSIM loss or the neuron embeddings, the removal MEIs leads the largest drop in performance in terms of all metrics. This result demonstrates the usefulness of MEIs as prior knowledge in this low-data regime.
MElcoder
MindEye2
SENSORIUM
Synthetic
1.0
1.0
0.8
0.9
0.9
0.6
Variance of errors. Here, we provide fur- ther evidence for the shortcomings of meth- ods that rely pre-trained GenAl models. Specifically, we quantify the reliability of MElcoder and MindEye2 by examining the
0.6
0.6
0.0
0.5
Alex!2]
Alax(5)
Alex/2)
Alex(S)
Alex!2]
Alex(5)
Metric
Metric
Metric
variance in their reconstruction quality across individual data points the test set. As shown in Figure 5] both the Alex(2) and A5) scores fluctuate much more for Mind- Eye2. indicating that MElcoder provides more consistent results. Figure 6 shows examples of hallucinated content in Mind- Fye2 images, supporting previous claims about spurious reconstructions [39]
Figure 5: Variance in reconstruction quality.
Target
MindCye2
Figure 6: Example of hallucinated content in recon- structed images by MindEye2 (left: woman face; right: giraffe).
7

---

Number points
0.8
100
500
1000
2000
3000
4500
Target
Brainreadder
PixCoor
0.3
3
Brainreader
SENSORIUM 2022
-0.2
Synthetic Cat V1
CENSORIN
1.0
Alex
0.8
Synthetic Cacv
5000
Number of data points
Figure 7: Relationship between MElcoder's performance and the number of training data points.
Number
Brainreader
SENSORIUM 2022
100
500
1000
2500
5000
8000
Target
Synthetic Cat
Biral meaderr
PrCCar
0.5
11
0.2
SENSORIUM
1.0
Alex(2)
Syntheri
10
10
10
Number of neurons
Figure 8: Relationship between MElcoder's performance and the number of neurons.
Amount of training data. To quantify the scaling behavior of MElcoder, we train it on varying sizes of the training set and evaluate using the full test set. As we can see in Figure 7 MElcoder's reconstructions start to capture ground-truth images after training with 1,000 or fewer training data points. Moreover, if we look at the Alex(2) performance of the second-best method from Table which was trained on all data, we see that MElcoder outperforms it already at 1,000 training data points, demonstrating its data efficiency. Lastly, it is worth noting that while the quantitative measures start to plateau, the qualitative (visual) results keep steadily improving, indicating limitations of the currently widely used metrics, such as PixCorr and two-way identification scores.
Number of neurons. Another challenge in reconstructing visual stimuli from neural population activity comes in the form of information scarcity, which results from the difficulty of (invasively) recording many cells in parallel and from neuronal noise [13). This problem manifests when decoding methods need to successfully invert the brain's unique stimulus-ene cennuiinnnnoding process using only a small subset of neurons, which might not carry sufficient information to fully decode the original stimulus. To balance the reconstruction quality with the costs of brain recordings. it therefore paramount to understand how the decoder's 's performance scales with the number of neurons. Figure 8 shows this relationship, and we can see that MElcoder reaches more than 95% two-way identification ability (Alex(2)) on all three datasets with just around 1,000 neurons and can accurately distinguish digits with around 2,500 neurons. Interestingly, the pixel correlation keeps increasing for both BRAINREADER and S SYNTHETIC CAT V1 datasets, indicating that the performance does not saturate

---

even with the 46,875 neurons available in the synthetic cat dataset, which is several-fold fold more than what most current biological datasets can provide in a single animal.
The scaling experiments above reveal several insights. Firstly, the similar sealing behavior of biological and SYNTHETIC CAT VI data highlights the potential of high-fidelity spik- ing models, which capture detailed visual cortex dynamics and provide abundant data for developing and benchmarking decoding methods. Second, the number of available neu- rons seems to be more limiting than the size of the training dataset, as can be seen by the steady qualitative improvement of reconstructions and unsaturated metrics with an increasing number of available neurons. Finally, our analysis indicates that recording between 1,000 and 2,500 neurons from mouse V1 is enough for fine-grained reconstructions with the power to discriminate between handwritten digits.
Most excited
Combined
Torget
MElcoder
+
Reconstructing artificial patterns. In Figure 9 provide an additional demonstration of how MElcoder successfully generalizes to out-of-distribution data using its strong MEI prior. Specifically, we create images with artificial patterns for which we predict neuronal responses from CNN encoder pre- trained on the BRAINREADER dataset, and then decode the we
Figure 9: Reconstructing artificial patterns using MEIs (center) and ME- Icoder (right). Shown are only the middle image regions for better visi- bility of MEls.
original images back using these encoded responses. can see, MElcoder captures exact shapes well, even though it has never encountered such artificial patterns in its training data. This demonstrates its generalization and reliability, which might be crucial for certain applications. In addition, we illustrate the underlying intuition behind our method: approximating the original images by summing the MEIs of all neurons, weighted by their corresponding neuronal responses ("Combined MEIs"). Additionally, we also show the MEI corresponding to the neuron with the highest predicted response (" ("Most excited MET"). The fact that even these simple linear MEI-based reconstructions already capture some of the basic characteristics of the original images further motivates the main building block of MElcoder.
Concept-based analysis. Finally, we analyze the learned decoding process of MElcoder, aiming to provide scientific insights into computations VI. Specifically, we implement a concept- based analysis that combines (1) non-negative matrix factorization (NMF) to learn dictionary of 32 feature bases ("visual concepts") from the feature maps at different layers of the MElcoder's core module, with (2) sensitivity analysis to observe how manipulating the response of single neuron changes the activation of these concepts. Intuitively, if the increased neuron activation increases the activation of certain concepts in our decoder, this would mean that in the brain, these visual concepts likely to be driving that particular neuron. We make the following observations from this analysis:
1. For the majority of neurons, the high- intennsttstty (bright) areas of the most active concepts become smaller and more focused as we traverse through the decoder's layers. This suggests the model learns a hierarchy, starting with coarse features and progressively refining them into more detailed structures, with the final location remaining in most cases consistent with the neuron's 's original MEL. An example of this for two neurons can be seen in in/iigure I0)
2.
Many of the neurons' top three feature bases include a concept that encodes the brightness of the image border (Figure 10). Together with finding (1), this suggests that many neurons encode the global lighting condition, and at the same time specialize in encoding smaller local structures at different places in the visual field (as supported by MEls). This might hint at how, through lateral cortical processing, the cortex fills in information where it is missing.
3. We identify a few neurons for which incrementally increasing the response results in an incremental shift of dark object in the reconstructed image (two examples in (Figure 12). This highlights neurons whose responses drastically affect the decoder's learned reconstruc- tion process. Interestingly, this emergent MElcoder's property mirrors key findings about functional asymmetries in the visual cortex. Namely, a study by 49 found a significant over-representation of "black-dominant" (OFF) neurons in the corticocortical output layers 2/3 of macaque VI. As the authors of [49] pointed out, their results suggested that the human perceptual preference for black over white is generated or greatly amplified in VI.

---

Image
MEI
Image
MEI
Layer 0
Layer
Layer
Layer 6
Figure 10: Input images, MEIs, and visual concepts with the highest activation gain for two example neurons from the BRAINREADER dataset. For each MElcoder core layer and neuron, the top three concepts (NMF features) are shown, ordered by the increase in the corresponding feature coefficient between forward pass with the neuron's 's response set to zero and one with it set to 10 p.99, where P.99 denotes the neuron's 's 99th activation percentile in the dataset.
This analysis showcases MElcoder's potential as a tool for scientific discovery, revealing how combines individual neuron contributions into coherent reconstruction. This demonstrates its utility for studying visual neural codes, and we believe further interpretability analysis represents an exciting avenue for future work. A more detailed discussion of the visual features reconstructed by MElcoder and how our analysis compares with existing work can be found in subsecction A.
5 Concluding remarks
Linitations. Motivated by applications in wrain-machine interfaces and neuroprosthetics for individu- als with acquired blindness, we focused primarily on data from V1, which encodes low-level features of visual stimuli. Although different species, which we would argue is still remarkable feat, we did not test our method on higher-order areas such as V4. However, despite the increased complexity of receptive fields of neurons in these areas, MElcoder can combine MEIs in a highly nonlinear fashion thanks to its core, and is able learn additional coding properties neurons in its Icarnable neuron embeddings. Another limitation of this study and room for additional investigation lies in the transfer learning capability of MElcoder (i.e.. pre-training on multi-subject data). Empirically, we found performance gains from transfer on the BRAINREADER dataset, but not on the SENSORIUM 2022 data.
showed state-of-the-art performance three VI datasets from two
Conclusion. We introduced MElcoder, novel decoding method that achieves state-of-the-art performance in reconstructing visual stimuli from neural population activity in V1. Leveraging MEIs, SSIM-based training objective, and adversarial training, MElcoder outperforms all baselines on three difficult datasets, especially excelling in data- and neuron-scarce settings. Compared to GenAI techniques that gained popularity in decoding. MElcoder better captures low-level features of images and does not suffer from high variance of reconstruction accuracy, making it more suitable for applications ranging from brain machine interfaces to uncovering brain information content. Our additional investigations also offer practical insights for neuroscience: We showed for the first time that MEIS are a powerful tool not only for understanding tuning properties of individual neurons, but also for reconstructing stimuli. Additionally, our scaling experiments demonstrate that it is feasible to achieve high-fidelity image reconstructions with as few as 1,000 to 2,500 neurons and limited single-subject training data. Finally, we showed the promise of accurate spiking models for developing decoding methods and presented an integrated benchmark pipeline that provides access to more than 160,000 samples to support future research in this area.
10

---

Acknowledgments
IS was supported by the Bakala Foundation during his studies at EPFL. LB and JA were supported by the EU Horizon 2020 Maria SklodowskaCurie grant agreement No 861423. JA was supported by the ERDF- Project Brain dynamics, No. 111.0001110022200800000000000000 and by the grant no. 25-180315 of the Czech Science Foundation (GACR).
References
[1] E. Allen, St-Y Y. Wu, J. L. Breedlove, J. Prince, I. T. Dowdle, M. Nau, B. Caron, Pestilli, I. Charest, Hutchinson, T. Naselaris, and Kay. massive fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116-12) 2022.
[2] Antolik, S. B. Hofer, J. A. Bednar, and D. Mrsic-Flogel. Model constrained by visual hierarchy improves prediction of neural responses to natural scenes. PLOS Computational Biology, 12(6):1-22, 2016.
[3] J. Antolik, R. Cagnol, T. Rózsa, C. Monier, Y. Frégnac, and A. P. Davison. A comprehensive data-driven model of cat primary visual cortex. PLOS Computational Biology, 20(8):1-42. 2024.
[4] P. Bashivan, K. Kar, and J. J. DiCarlo. Neural population control via deep image synthesis. Science, 364(6439) 2019.
[5] W. BBttaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski,
A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1:X.:::806.01261 2018.
[6] Y. Benchetrit, H. Banville, and J.-R. King. Brain decoding: toward real-time reconstruction of visual perception. In International Conference on Learning Representations, 2024.
[7] C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics).
Springer-Verlay Berlin, Heidelberg, 2006. ISBN 0387310738.
[8] N. Brackbill, C. Rhoades. A. Kling, N. P. Shah, A. Sher, A. M. Litke, and E. J. Chichilnisky. Reconstruction of natural images from responses of primate retinal ganglion cells. Elife, 9: e58516, 2020.
[9] S. Cadena, G. H. Denfield, E. Y. Walker, Gatys, A. S. Tolias, M. Bethge, and A. Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLOS Computational Biology, 15(4):1-27, 2019.
[10] Y. Chen, P. Beech, Z. Yin, S. Jia, Zhang, Z. Yu, and K. Liu. Decoding dynamic visual scenes across the brain hierarchy. PLOS Computational Biology, 20(8):1-25, 2024.
[11] E. Cobos, T. Muhammad, P. G. Fahey, Z. Ding, Z. Ding, J. Reimer, F. H. Sinz, and A. S. Tolias. It takes neurons to understand neurons: Digital twins of visual cortex synthesize neural metamers. bioRxiv, 2022.
[12] Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009.
[13] A. A. Faisal, L. P. J. Selen, and D). M. Wolpert. Noise in the nervous system. Nature Reviews Neuroscience, 9()):992 2008
[14] T. Fel, A. Picard, L. Béthune, T. Boissin, D. Vigouroux, J. Colin, R. Cadène, and T. Serre. Craft: Concept recursive activation factorization for explainability. In Conference Computer Vision and Pattern Recognition, 2023.
[15] T. Fel, E. S. Lubana, J. S. Prince, M. Kowal, V. Boutin, I. Papadimitriou, B. Wang, M. Watten- berg, D. E. Ba, and T. Konkle. Archetypal SAE: Adaptive and stable dictionary learning for concept extraction in large vision models. In International Conference on Machine Learning. 2025.
[16] A. Ghorbani, J. Wexler, J. Zou, and B. Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, 2019.

---

[17] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
[18] R. Hayashi and H. Kawata. Image reconstruction from neural activity recorded from monkey inferior temporal cortex using generative adversarial networks. In International Conference on Systems, Man, and Cybernetics, 2018.
[19] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances Neural Information Processing Systems, 2020.
[20] S. Joseph, P. Suresh, E. Goldfarb, L. Hufe, Y. Gandelsman, R. Graham, D. Bzdok, W. Samek, and
B. A. Richards. Steering CLIP's vision transformer with sparse autoencoders. In Mechanistic Interpretability for Vision CVPR 2025 (Non-proceedings Track), 2025.
[21] N. Kriegeskorte and Douglas. Interpreting encoding and decoding models. Current Opinion in Neurobiology, 55:167-179, 2019.
[22] L. Le. L. Ambrogioni, K. Seeliger, Y. Güşlüttürk, M. van Gerven, and U. Güşlü. Brain2Pix: Fully convolutional naturalistic video frame reconstruction from brain activity. Frontiers in Neuroscience, 16, 2022.
[23] L. Le, P. Papale, K. Seeliger, A. Lozano, T. Dado, Wang, P. R. Roelfsema, M. van Gerven, Y. Güglüüürk, and U. Güçlü. Monkeysee: Space-time-resol reconstructions of natural images from macaque multi-unit activity. In Advances in Neural Information Processing Systems, 2024.
[24] B. M. Li, I. M. Cornacchia, N. Rochefort, and A. Onken. VIt: large-scale mouse v1 response prediction using a vision transformer. Trunsactions on Machine Learning Research, 2023.
[25] W. Li, S. Zheng, Y. Liao, R. Hong, C. He, W. Chen, C. Deng, and X. Li. The brain-inspired decoder for natural visual image reconstruction. Frontiers in Newroscience, 17, 2023.
[26] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[27] K.-K. Lurz, M. Bashiri, K. Willeke, A. K. Jagadish, E. Wang, E. Y. Walker, S. A. Cadena, T. Muhammad, E. Cobos, A. S. Tolias, A. S. Ecker, and F. H. Sinz. Generalization in data-driven models of primary visual cortex. bioRxiv, 2021.
[28] L. T. McIntosh, N. Maheswaranathan, A. Nayebi, S. Ganguli, and S. A. Baccus. Deep learning models of the retinal response natural scenes. In Advances in Neural Information Processing Systems, 2016.
[29] N. Morgan and H. Bourlard. Generalization and parameter estimation in feedforward nets: Some experiments. In Advances in Neural Information Processing Systems, 1989.
[30] S. Nishimoto, A. T. Vu, T. Naselaris, Y. Benjamini, B. Yu, and J. L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 21(19): 1641-1646, 2011.
[31] P. A. Pierzchlewicz, K. F. Willeke, A. Nix, P. Elumalai, K. Restivo, T. Shinn, C. Nealley, G. Rodriguez, S. Patel, K. Franke, A. S. Tolias, and F. H. Sinz. Energy guided diffusion for generating neurally exciting images. In Advances in Neural Information Processing Systems, 2023.
[32] C. R. Ponce, W. Xiao, P. F. Schade, T. S. Hartmann, G. Kreiman, and M. S. Livingstone. Evolving images for visual neurons using deep generative network reveals coding principles and neuronal preferences. Cell, (774))999-1100.00
2019.
[33] X. Ran, J. Zhang, Z. Ye, H. Wu, Q. Xu, H. Zhou, and Q. Liu. Deep auto-encoder with neural response. arXiv preprint arX1v:2111.15309, 2021.
[34] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, 2016.
[35] S. Scotti, A. Banerjee, J. Goode, S. Shabalin, A. Nguyen, C. Ethan, A. J. Dempster, N. Ver- linde, E. Yundler, D. Weisberg, K. Norman, and T. M. Abraham. Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors. In Advances in Neural Information Processing Systems, 2023,
12

---

[36] P. S. Scotti. M. Tripathy, C. K. T. Villanueva, R. Kneeland, T. Chen, A. Narang, C. Santhi- rasegaran, Xu, T. Naselaris, K. A. Norman, al. Mindeye2: shared-subject models enable fmri-to-image with hour of data. In International Conference on Machine Learning, 2024.
[37] K. Seeliger, U. Güęlü, L. Ambrogioni, Y. Güiçlütürk, and M. A. J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neuroimage, 181:
775-785, 2018.
[38] G. Shen, K. Dwivedi, K. Majima, T. Horikawa, and Y. Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, 13, 2019.
[39] K. Shirakawa, Y. Nagano, M. Tanaka, S. C. Aoki, Y. Muraki, K. Majima, and Y. Kamitani. Spurious reconstruction from brain activity. Neural Networks, 190:107515, 2025.
[40] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
[41] Y. Takagi and S. Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Conference on Computer Vision and Pattern Recognition, 2023.
[42] M. M. Taylor, D. Contreras, A. Destexhe, Y. Frégnac, and Antolik. An anatomically constrained model of VI simple cells predicts the coexistence of Push-Pull and broad inhibition. Journal of Neuroscience, 41(37):7777-7122 2021.
[43] B. Thirion, E. Duchesnay, E. Hubbard, Dubois, J.-B. Poline, D. Lebihan, and Dehaene. Inverse retinotopy: inferring the visual content of images from brain activation patterns, Neu- roimage, (3): 1116, 2006.
[44] J. A. Tibor Rózsa, Rémy Cagnol. Iso-orientation bias of layer 2/3 connections: the unifying mechanism of spontaneous, visually and optogenetically driven v1 dynamics. bioRxiv, 2024.
[45] E. Y. Walker, F. H. Sinz, E. Cobos, T. Muhammad, E. Froudarakis, P. G. Fahey, A. S. Ecker, J. Reimer, X. Pitkow, and A. S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience, 22(12):2060-2065 2019.
[46] E. Y. Wang,
Ponder, M. A. Weis, Chang. T. Muhammad, S. Patel, Z. Ding, D. Tran, J. Fu, C. M. Sechnedder-Mizelll N. M. da Costa, R. C. Reid, Collman, K. Franke, A. Ecker, J. Reimer, X. Pitkow, F. H. Sinz, A. S. Tolias, and M. Consortium. Foundation model of neural activity predicts response to new stimulus types. Nature, 64008058):440-477,, 2025.
Fahey, Z. Ding, Papadopoulos,
[47] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-6) 612, 2004.
[48] K. F. Willeke, P. G. Fahey, M. Bashiri, L. Pede, M. F. Burg. C. Blessing, S. A. Cadena, Z. Ding, K.-K. Lurz, K. Ponder, et al. The sensorium competition predicting large-scale mouse 2022.
primary visual cortex activity. arXiv preprint arXiv:2206.08666
[49] C.-I. Yeh, D. Xing, and R. M. Shapley, "black" responses dominate macaque primary visual cortex vl. Journal of Neurescience, 29(38): 110::-11733316111 2009.
[50] R. Zhang, P. Madumal, T. Miller, K. A. Ehinger, and B. I. P. Rubinstein. Invertible concept-based explanations for enn models with nen-tegative concept activation vectors. In Conference on Artificial Intelligence, 2021.
[51] Y. Zhang, T. Bu, J. Zhang, S. Tang, Z. Yu, J. K. Liu, and T. Huang. Decoding Pixel-Level image features from Two-Photon calcium signals of macaque visual cortex. Neural Computing, 34(6): 1369-1397, 2022.
13

---

A Technical Appendices and Supplementary Material
A.1 Experimental details
We conducted the main experiments reported in Table I and Table three times with different random seeds; the resulting standard deviations are reported Table 3 and Table 4 Each of our experiments used one NVIDIA Tesla V100 GPU and required less than 32 GB of VRAM. The longest training run of our method (training with data from multiple subjects) took approximately four days, whereas the longest training run of the baselines took six days.
The code for our experiments and instructions obtaining the data are available in our public repository at haapp:://ggthutt www.g..t...oo.oeeeeeeeeo Joom/JJhnyyll 1etpp/111mmmmmmmmmcmmmmmme
Table 3: Quantitative results the test sets from the BRAINREADER and SENSORIUM 2022
datasets. Best results are highlighted red, and second-best bold.
BRAINREADER
SENSORIUM 2022
Method
SSIM
PixCom
Alex(2)
Alex(5)
SSIM
PixCorr
Alex(2)
Alex(5)
.321
.611
.989
.896
.288
.453
.915
.720
InvEnc
+.006
+.002
t.011
+.005
+.005
+.010
+.012
.56
.455
.758
.659
.256
.365
.777
.755
EGG
-006
+.001
+.015
+.021
+.009
+.020
+.007
+ +.024
232
.565
.967
.826
.185
.338
.564
MonkeySee
.008
1.024
1.009
1.014
1.004
.56
.638
.930
.730
.877
.539
.656
.549
CAE
.010
1.004
1.007
1.008
1.005
MindEye2
.77
.500
.946
.878
.210
.471
.877
.762
.039
1.027
1.020
1.027
1.028
1.036
1.037
1.048
.244
.516
.920
.836
.243
.499
.918
.799
MindEye2 (FT)
.4445
1.038
2.043
2.067
1.006
.008
2.013
+ 2.013
400
.679
.988
.990
.331
.003
.988
.896
MElcoder
.222
1.010
0.02
1.006
1.004
1 2013
2.004
1.06
.24
.706
.999
.977
.388
.86
.95
.908
MElcoder (FT)
-.006
+.012
+.002
+.010
+.003
+.011
+.004
+ +.003
A.2
Data
Each data point consists of z-scored image and associated neuronal responses (neural ac- tivity accumulated during +500 ms time win- dow after image stimulus onset). All images, except the hand-selected ones in the test sets of ARAIREEADER and SENSORIUM 2022, randomly sampled from ImageNet. The final sizes of the srayscale-mapper stimuli
Table 4: Quantitative results on the test set of the SYN- THETIC CAT VI dataset. Best results are highlighted in red, and second-best in bold.
SYNTHETIC CAT VI
Method
SSIM
PixCor
Alex(2)
Alex(5)
.771
.833
.986
.978
InvEne
+ 000
+.002
4.003
+.001
were: 36 64 px (BRAINREADER), 22 36
px (SENSORIUM 2022), and 20 x 20 px (SYNTHETIC CAT V1). Additionally, ad- dress the substantial variability of firing rates between different neurons, which could be detrimental to training, we rescale individ- ual neuronal responses by the inverse of the standard deviation estimated for each neuron. All z scoring statistics are obtained from the training sets. For further details on data col- lection of the biological datasets, please re- fer to the original works (11) and 48 for BRAINREADER and SENSORIUM 2022, respectively).
EGG
.640
.936
+.022
+.027
+.007
+.013
.607
.723
.982
.958
MonkeySee
+.031
+.019
+.00
+.021
.637
.793
.929
.775
CAF
+.006
+.005
4.004
+.006
.559
.757
.977
MindEye2
1.011
1.016
4.006
t.014
.774
.777
.944
.977
MElcoder
+-006
+-009
+.000
+.002
14

---

Synthetic Cat V1 dataset. We generated the SYNTHETIC CAT VI dataset using the biologically realistic spiking model from [3, which represents cortical layers 4 and 2/3, corresponding to a 5 mm patch of cat V1. When generating the samples, we first presented this encoding model with an image stimulus and then measured the evoked mean firing rates of 46,875 neurons in layer 2/3 over the following 560 ms time window. As for the other datasets, the image stimuli were sampled from ImageNet, converted to grayscale, downsampled, and then cropped to a size of 50 x 50 px. When we subsequently used the data to train and evaluate the decoding models, we only considered the central 20 20 px patch of the images. The reason is that the neurons in the encoding model have overlapping receptive fields that do not cover the whole visual field; therefore, their induced responses contain information only about the central patch of the presented images. To reduce the impact of noise and then averaging the corresponding neural responses to obtain the final neural activity.
evaluation, we created the test set by presenting the image stimuli 100 times
Combining the three datasets into single data corpus results in benchmark pipeline consisting of:
1. BRAINREADER dataset: Data from V1 of 22 mice, where each single-subject dataset contains around 5,000 data points, and the average/min/max number of neurons is 8,1 1/66,,2/// Data for each mouse was split into training, validation, and test sets by the original au- thors [Л]] The sets contain 40 repeated trials for each stimulus.
2. SENSORIUM 2022 dataset: Data from V1 of mice (using only the training recordings from the SENSORIUM 2022 competition), where each single-subject dataset contains around 5,000 data points, and the average/min/max number of neurons is 7,851/7,334/8.55 We split the dataset from each mouse into a training (4,500) and validation (500) set. The test sets are provided separately by the original authors and contain 10 repeated trials for each stimulus.
3. SYNTHETIC CAT VI dataset: Synthetic data from the spiking model of cat V1 (single- subject) containing 50,250 data points of neuronal responses 46,875 neurons. We split this data corpus into training (45,000), validation (5,000), and test (250) sets. The test set contains 100 repeated trials for each stimulus.
A.3 MElcoder details
Hyperparameters We provide hyperparameters of MElcoder used for the final experiments in Additional settings that we kept the same across all datasets include:
Table 5
Number of compressed neural map channels de (readin): 480
Number of CNN channels (core): 480, 256, 256, 128, 64,
Kernel sizes (core): 7, 5, 3,
Padding (core): 3, 2, 2, 1, 1,
Stride (core): 1, 1, 1,
Dropout probability (core): 0.35
Table 5: Hyperparameter search space for MElcoder, with final selected values underlined.
Dataset
Learning rate
Weight decay
Dimension of neuron embeddings
BRAINREADER SENSORIUM 2022
[le-3, Ie-4, 3e-5)
[3e-1, 8e-2, 3e-3)
{16, 32, 64)
[1c-3, Ic-4, 3c-5)
[3e-1, 8c-2, 3e-3)
{16, 32, 64)
SYNTHETIC CAT V1
[le-3, 1e-4, 3e-5)
(3e-1, 8e-2, 3e-3)
{16. 32, 64)
Discriminator: The discriminator used for the auxiliary adversarial objective implemented as a CNN with five layers of convolution, batch normalization, ReLU activation function, and dropout (p = 0.3). The output the last layer is flattened into a one-dimensional vector and transformed by a linear layer followed by the sigmoid activation function. The result is a predicted probability that the given discriminator's input reference image from the dataset (i.e., not reconstruction from the decoder).
15

---

We train the discriminator simultaneously with the decoding model using the Adam W optimizer [26] with the same learning rate and weight decay as the decoding model. The training objective as follows: Let ÄGT E R2 be the loss weighting factor, fGT E [0, far E [R+,, <R [0, CR RR]]] be the
target noising components, and y and y denote the reference (ground-truth) and reconstructed image, respectively. Then, our implementation of the discriminator loss LD is the following:
(y 9) = AGT (D&(y) - (GT) + - (1-)ar) (D&(9) - cx)²
(3)
The target noising component EGT for the reference image part is sampled from uniform distribution between 0 and ŞGT, while the noising component éR comes from a uniform distribution between 0 and fR- We found that by introducing noise into the discriminator training, we were able to better balance the decoder and the discriminator, and thereby stabilize the training. We note that this is reminiscent of one-sided label smoothing as introduced in [34], where the discriminator's positive targets smoothed from to 0.9, making its task harder. The specific hyperparameters used for the final experiments are given below:
Number of channels: 256, 256, 128, 64, 64
Kernel sizes: 7, 5, 3,
Padding: 2, 1, 1, 1,
Stride: 2, 2, 1, 1,
for {R 0.05
AGT 0.5
Overall, the selection of architecture and hyperparameters makes MElcoder more parameter-efficient. Namely, the CNN architecture achieves efficiency by parameter-sharing and by allowing only local connections. Furthermore, MElcoder reuses the core module (backbone) across datasets from different subjects (e.g.. different mice) and only trains new readin modules. By sharing the core module, MElcoder reduces the number of parameters threefold.
A.4 Most exciting inputs
The two main reasons why we decided use a computational prior in the form of MEIs are the following. First, strong priors help guide learning and prevent overfitting in data-limited regimes like our neural recording dataset [5] 7 Second, many neurons exhibit sparse firing, which can make it difficult for the decoder to learn the ttimulus-responss mapping for a neuron that was active for only handful of images the training set. To combat this, the MEI provides powerful "head-start" by giving the decoder dense, explicit template of each neuron's 's preferred stimulus, even for rarely firing neurons. This is visually demonstrated in Figure 9 where simply weighting MEIs by neural responses already forms coarse but recognizable reconstruction, highlighting the power of this prior.
Motivation.
to generate MEIs of all neurons in the given single-subject dataset. More specifically, train CNN-based encoding model the given dataset, randomly initialize an MEI image with zero mean and standard deviation of 0.15, and then iteratively optimize its pixel values using gradient ascent to maximize the encoder's prediction for the target neuron. After each optimization step, we normalize the image back to zero mean and standard deviation of 0.15 and clip pixel values that have an absolute value larger than one. Since there are more than 7,000 neurons in each dataset, we accelerate this MEI generation procedure by initializing and then optimizing MEIs of multiple neurons in parallel (batching inputs to the encoder). Note that this whole optimization procedure needs to be done only once for each dataset.
Generation. We follow previous work 4
32 45
While the entire MEI generation, including encoder training, took approximately 70 minutes (15 minutes training, 55 minutes generation) in our experiments, the decoder training required 1 hours on NVIDIA Tesla V100 GPU. This shows that the computational overhead of MEI generation is small compared to the decoder training itself.
For all cases of CNN-based encoding model (MEIs, InvEnc, and EGG), we the Gaus- sian readout architecture introduced by [27]. More specifically, we use the implementation and hyperparameters provided by [8
16

---

A.5 Sensitivity to quality of most exciting inputs
To understand the sensitivity of ME- Icoder's s performance to the quality of generated MEIs, we re-ran train- ing and evaluation on the BRAIN- READER dataset with MEIs with varying levels of Gaussian noise. As shown in Table 6. the performance degrades relatively slowly. In fact, even with highly noisy MEIs (std=1 and std=3 for MEIs with initial pixel values between -1 and 1), the perfor- mance remains par better than that the baselines (Table I).
Table 6: Quantitative results the test set of the BRAIN-
READER dataset. Standard deviation (std) corresponds to the Gaussian noise added to MElcoder's MEIs.
BRAINREADER
Method
SSIM
PixCom
Alex(2)
Alex(5)
MElcoder (no noise)
.400
.679
.988
.990
MElcoder (std=0.2)
.402
.675
.997
.982
MElcoder (std=0.5)
.390
.670
.990
.938
MElcoder (std=1)
.332
.625
.990
.937
MElcoder (std=3)
.287
.568
.984
.943
A.6 Comparison gradient-based linear receptive fields of neurons
MEIs can be seen nonlinear counterpart to traditional linear receptive fields of neurons. To compare these neural characterizations for decoding, we replaced MEIs in MElcoder with gradient- based linear receptive fields (LRFs) obtained from regularized regression trained on neural responses from the BRAINREADER dataset. shown in Table 7 this leads to degraded performance, but not as severe as with a complete removal of neural characterization done in the initial ablation study subsection 4.5) This demonstrates the importance of biologically informed computational prior, MEIs in particular, for the decoder's performance.
Table 7: Quantitative results the test set of the BRAINREADER dataset with different neural characterizations in MElcoder's readin.
BRAINREADER
Method
SSIM
PixCor
Alex(2)
Alex(5)
MElcoder (MEIs)
.400
.679
.998
.990
MElender (LRFs)
.364 (-9%)
(-2.4%)
.998 (-1%)
.949
(-4.1%)
A.7 MElcoder for highly non-linear neurons
To demonstrate that the state-of-the-ann performance of MElcoder does not severely degrade when trying to decode from highly non-linear neurons, such as those found in higher visual areas of the brain, conducted the following comparative experiment.
First, for each neuron in the BRAINREADER dataset, we calculated the so-called non-linearity index (NLI), which estimates how non non-linear individual neurons are [2] It is calculated the ratio between the prediction power of linear encoding model fitted to the data and the prediction power state-of-the-as thttte-rttttttt non-linear encoding model fitted to the data. Second, MElcoder only on subsets of the most linear and most non-linear neurons. As can be seen Table 8 MElcoder's ability decode images from the more non-linear neurons is very similar to its ability to decode from more linear neurons, suggesting that MElcoder can handle non-linearity of the code very well, at least within the context of VI.
trained and evaluated
Table 8: Quantitative results on the test of the BRAINREADER dataset when MElcoder is trained and evaluated only with subsets neurons.
BRAINREADER
Selection of neurons
SSIM
PixCorr
Alex(2)
Alex(5)
3,000 most non- non-linear neurons
.321
.630
.999
.958
3,000 least non-linear neurons
.324
.653
.996
.939
3,000 least non- con-linear neurons (nonzero NL.)
.345
.663
.999
.979
17

---

Table 9: Quantitative results on the test sets from the BRAINREADER and SENSORIUM 2022 datasets (higher-level metrics). Best results are highlighted in red, and second- in bold.
BRAINREADER
SENSORIUM 2022
Method
Incep
CLIP 1
Eff
SwAV +
Incep
CLIP
Eff 4
SwAV
InvEnc
.627
.636
.606
.448
.600
.564
.492
.243
EGG
.749
.610
.652
.497
.673
.579
.449
.232
MonkeySec
.655
.547
.653
486
.614
.513
.551
.301
CAE
.592
.541
.628
.612
.542
.535
.684
.737
MindEyc2
.772
.636
.571
.415
.643
.590
.440
.235
MindEyc2 (FT)
.725
.652
.584
465
.650
.595
.443
.219
MElcoder
.799
.679
.586
.489
.727
.590
.440
.276
MElcoder (FT)
.817
.702
.520
.408
.746
.620
.419
.216
A.8
Evaluation on higher-level metrics
Using the implementation from Mind- Eye2 [6.. we evaluated all methods on higher-level (semantic) metrics. Specifically, we measured the two-way identification ac- curacy with Inception ("Incep") and CLIP (ViT-L/14) embeddings, as well as the av- erage correlation distance in the embedding space of EfficientNet-Bl ("Eff"" and SwAV. ResNet50 ("SwAV"))
Table 10: Quantitative results on the test set of the SYNTHETIC CAT VI dataset measured on higher- level metrics. We highlight the best score in and the second-best score in bold.
SYNTHETIC CAT VI
Method
Incep
CLIP t
Eff 1
SwAV
InvEnc EGG
.884
.827
.326
.196
As can be seen in Table and Table 10 ME- Icoder remains highly competitive even these semantic metrics (Incep and CLIP: the higher the better, Eff and SwAV: the lower the better). On both the BRAINREADER and SENSORIUM 2022 datasets, MElcoder achieves the best performance, while on the SYNTHETIC CAT V1 data, it outperforms the other baselines on two out of four metrics, remaining on par on the other two performance measures. We can also see that the MElcoder fine-tuned ("FT") from multi-subject to single-subject data performs the best on the biological datasets. This further demonstrates MElcoder's ability to generalize across data from different subjects.
.766
.675
.283
.214
MonkeySce
.796
.760
.272
.193
CAE
.657
.622
.442
.573
MindEye2
.798
.780
.274
.208
MElcoder
.898
.761
.277
.181
A.9 Reconstructing higher-resolution images
Here, we demonstrate the scalability MElcoder higher-resolution visual stimuli. Specifically, us- ing the MEIs generated for the original resolution, train and test MElcoder on the BRAINREADER dataset with two-times higher image resolution 72 128 pixels.
As shown in Table 11, MElcoder's per- formance is maintained this image res- olution. In fact, although the task of re- constructing larger images is inherently more difficult, the higher-resolution ME- Icoder still outperforms the best alter- native baseline from the low-resolution setting Table Tp. This demonstrates that MElcoder is not limited to low- resolution stimuli and can effectively scale to more detailed visual inputs.
Table II: Quantitative results of MÉlcoder on the test set the BRAINREADER dataset.
BRAINREADER
Image resolution
SSIM
PixCor
Alex(2)
Alex(S)
px
.400
.679
.998
.990
128 px
.340
.646
.997
.980
18

---

Target
MElcoder
[72 128
Figure 11: Reconstructions of images from the BRAINREADER dataset with MElcoder trained and evaluated on 72 x 128 px images.
Target
115
0.00
1150
44.88
115
89.77
115
134.65
Target
11450
0.00
11450
31.31
11450
62.63
11450
93.94
Figure 12: Reconstructions of images from the BRAINREADER dataset as we vary the response level of neurons 150 (top) and 1450 (bottom).
A.10 MElcoder interpretability
Our analysis in subsection 4.5 shares methodological tools with methods like CRAFT [14 partieu- larly the use of NMF and sensitivity analysis. The main difference between CRAFT and our analysis that employ sensitivity analysis derive the importance of inputs (neuronal responses) on intermediate features (coefficients for combining NMF concepts). CRAFT, on the other hand, uses sensitivity analysis to assess the importance intermediate features (coefficients for combining NMF concepts) on the final output of the network. To measure the importance of inputs on the intermediate features, CRAFT employs implicit differentiation and gradient-bard based attribution maps.
Other techniques, such as ACE [6], ICE [50]. and sparse autoencoders (SAEs) [55 20 have explored similar ideas but different tasks and with different methodologies. ACE, for example, considered discovering visual features by segmenting images from the same semantically meaningful class and clustering these segments in embedding space of CNN. Then, akin to our sensitivity analysis, ACE perturbed the hidden states to evaluate the importance of individual segment clusters on the final prediction of the CNN classifier.
Reconstructed visual features. Analyzing which visual features are reconstructed with high fidelity can tell which of them well-encoded in the neural population of V1. Here, find two key patterns.
First, MElcoder's reconstruction performance directly reflects the known tuning properties of V1 neurons: consistently reconstructs distinct, high-contrast features like corners and edges with high fidelity, while struggling with low-frequency information such as gradual changes in shading (see, for example, figures 2 and This is a reflection the VI code, which is dominated by edge-detecting neurons that provide sparse signal for uniform surfaces.
Second, we see more global fallure mode where images containing dense, figh-freuuency textures across relatively small area of the image lead to globally degraded reconstruction, an effect especially pronounced in the SENSORIÜM 2022 dataset (Figures Saarressannn and [4]. This suggests that while V1 robustly encodes local details, the decoder can be "'verwhelmed"" when trying to synthesize a coherent global percept from a highly complex population signal, hinting at why the brain requires hierarchical processing.
A.11 Broader impacts
Our work advances neural decoding by enabling high-fidelity image reconstruction from limited neural activity, offering insights into visual processing and potential applications in brain-maching
19

---

interfaces. With its data- and neuron cuuron-fffiieeney, MElcoder lowers the requirements on invasive brain recordings, which we believe is of great importance for practical neuroengineering applications.
However, translating our findings to human applications requires caution, as models trained on constrained and multi-subject datasets may inherit biases, potentially limiting generalization across diverse populations. Therefore, future work should incorporate broader. more inclusive datasets and rigorous clinical validation to high performance, balancing scientific progress with ethical responsibility.
20

---

A.12 Additional reconstructed images
MonkeySee
MindEye2
(FT)
4
6
8
5
5
5
010
1
0
Figure 13: Additional reconstructed images from the BRAINREADER dataset.
21

---

Target
InvEnc
EGG
MonkeySeee
CAE
MindEye2
MindEye2 (FT)
MElcoder
MElcader (FT)
Figure 14: Additional reconstructed images from the SENSORIUM 2022 dataset.
22

---

Target
InvEnc
EGG
MonkeySee
CAE
MindEye2
MElcoder
Figure 15: Additional reconstructed images from the SYNTHETIC CAT V1 dataset.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide evidence for our contributions in the experiments section (4), and describe the scope and context of the work in the abstract, introduction ( and related work (2) sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer:
[NA]
Justification:
This work is supported by empirical findings.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe method in section 3 and subsection A.3 and provide experi-
mental details in /sectiion 4 subsection A. and subsection A.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code for experiments and instructions on obtaining the data are available at lhttps://gittee
Johnny 118mmooy1111//iiccooo/ 1118mmeicmmmmmm///m
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
Sublic/guidess www.e/Subiisiiiooool111111
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
cttpss.ccppubblci gwwdeu/ideeeee www.oo/wwwwswooooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all details in section 3 and section and provide additional information in issbsecction A.3
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For the main experiments (Table and Table 2). we report metrics averaged over three runs with different random seeds, with standard deviations shown in Table 3 and Table to capture variability from initialization and data sampling. Additional runs would be too computationally expensive and would not impact the demonstrated functionality of the method, which can also assessed by qualitative inspection.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

ieg. The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss compute resources subsection A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have carefully reviewed and adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential broader impacts in subsection A.I
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The biological datasets are publicly available and properly referenced and credited, while the synthetic dataset was generated by the authors using properly referenced spiking model (see subsscctoo 4.1p. The models are original work or properly credited.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code and related assets are available at htttss://ittub... com/
Johnny1 1at1111m/mmmeeeeeeeeeee
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
The core method development does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
30


Reviews:
[{"review_id": "gtMHkchnro", "reviewer": "Reviewer_2fvu", "summary": "In order to reconstruct high-quality visual stimuli at the pixel level from limited neural data, this work introduces MEIcoder, which incorporates neuron-specific most exciting inputs into a visual-stimuli decoding framework. In the experiments on three neural datasets, MEIcoder achieves the best reconstruction performance compared to alternative models. Furthermore, in scaling experiments, the results demonstrate the superiority of MEIcoder in dealing with small numbers of neurons and data points.", "questions": "1. When the number of neurons and the resolution of visual stimuli increase, is a more complex encoding model required to calculate MEIs, and can the computational cost still be ignored?\n2. Are there repeated trials for each visual stimulus in each dataset? How are the training set, validation set, and test set divided, and do they contain repeated images? Can the authors provide a more detailed description of the visual stimuli used in each dataset?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This work is well-motivated, and introducing MEIs into decoding models is novel and meaningful.\n2. MEIcoder outperforms several outstanding and well-known models in multiple metrics and visualization, demonstrating its effectiveness.\n3. Comprehensive ablation studies prove the role of each module of MEIcoder, especially MEIs.\n\nWeaknesses:\n\n1. My primary concern is the fairness of the experiments. First, the authors do not provide results of the datasets used in baseline models and instead conduct comparisons on new datasets (where only the Brainreader dataset has been used in InvEnc). Second, MindEye2 is more focused on semantically reconstructing visual stimuli, which inherently puts it at a disadvantage in low-level or even pixel-level metrics. Could the authors provide results using higher-level metrics employed in MindEye2? Finally, [1] has reconstructed high-quality movie stimuli from the neural activity of hundreds of mouse visual cortex neurons. I think this study should be included in the comparison, as it also targets small datasets and pixel-level reconstruction.\n2. Since the visual stimuli in this work have low resolution, it is unclear whether the model can be scaled up to handle higher-resolution visual stimulus reconstruction.\n\n[1] YeChen, et al. Decoding dynamic visual scenes across the brain hierarchy. PLOS Computational Biology 2024.", "was_revised": true, "final_justification": "Based on the additional results provided by the author, I raise my score.", "rebuttal": "We thank the reviewer for their thoughtful review. We sincerely appreciate the reviewer's recognition of the originality of the method by introducing MEIs into the image reconstruction process. We are also grateful for the reviewer's acknowledgment that MEIcoder outperforms several well-known baselines on multiple different datasets on both quantitative and qualitative levels, and that our extensive ablations strongly support our design decisions.\n\n---\n\n> **Weakness 1**: *Fairness of experiments.*\n\nThe reviewer raises important points about the fairness and comprehensiveness of our experimental evaluation. We address each of the concerns below.\n\n**On the choice of datasets:** We agree that comparing on the original datasets of baseline methods is ideal, which is also why we chose one of our two biological datasets to come from the baseline method InvEnc. However, for MindEye2, the original data is from functional magnetic resonance imaging (fMRI), which does not allow identifying detailed receptive fields or MEIs, essential for our method, and in turn decoding of fine spatio-temporal details of target images. This makes this dataset less suitable for testing our method, which focuses on decoding the exact fine geometric details, which are important for studying neural codes in early visual areas and for applications such as the development of prostheses for those with vision impairment. For MonkeySee, the dataset has not been released together with the published paper, and our repeated email inquiries to the authors were not answered. Similarly, data used by EGG was not released with the paper, and we were unable to find it elsewhere on the internet. This state of affairs also motivated us to introduce an easily accessible benchmark data corpus of over 160,000 data points, which we hope will make the area of brain activity decoding more accessible.\n\n**On the evaluation metrics:** While we wish the emphasize that the primary motivation of our work is to develop a method focused on high-fidelity reconstruction of low-level geometric structures in images (i.e. decoding the fine geometric detail of target images rather than just their overall semantics), which is crucial for studying visual representations in early visual areas (such as V1 or V2), as well as applications such as visual prosthetics targeting these low-level visual areas, we agree that a broader comparison is valuable. To this end, using the implementation from MindEye2 [2], we have evaluated all methods using the two-way identification accuracy with InceptionV3 (“Incep”) and CLIP (ViT-L/14) embeddings, as well as the average correlation distance in the embedding space of EfficientNet-B1 (“Eff”) and SwAV-ResNet50 (“SwAV”). As can be seen in the tables below, our method remains highly competitive even on these semantic metrics (Incep and CLIP: the higher the better, Eff and SwAV: the lower the better). On both the Brainreader and SENSORIUM 2022 datasets, MEIcoder achieves the best performance, while on the Synthetic Cat V1 data, it outperforms the other baselines on two out of four metrics, remaining on par on the other two performance measures. We can also see that the MEIcoder fine-tuned (“FT”) from multi-subject to single-subject data performs the best on the biological datasets. This further demonstrates MEIcoder’s ability to generalize across data from different subjects. \n\n- **Brainreader:**\n||Incep|CLIP|Eff|SwAV|\n|-|-|-|-|-|\n|**InvEnc**|.627|.636|.606|.448|\n|**EGG**|.749|.610|.652|.497|\n|**MonkeySee**|.655|.547|.653|.486|\n|**MindEye2**|.772|.636|.571|.415|\n|**MindEye2 (FT)**|.725|.652|.584|.465|\n|**MEIcoder**|.799|.679|.586|.489|\n|**MEIcoder (FT)**|**.817**|**.702**|**.520**|**.408**|\n\n- **SENSORIUM 2022:**\n||Incep|CLIP|Eff|SwAV|\n|-|-|-|-|-|\n|**InvEnc**|.600|.564|.492|.243|\n|**EGG**|.673|.579|.449|.232|\n|**MonkeySee**|.614|.513|.551|.301|\n|**MindEye2**|.643|.590|.440|.235|\n|**MindEye2 (FT)**|.650|.595|.443|.219|\n|**MEIcoder**|.727|.590|.440|.276|\n|**MEIcoder (FT)**|**.746**|**.620**|**.419**|**.216**|\n\n- **Synthetic Cat V1:**\n||Incep|CLIP|Eff|SwAV|\n|-|-|-|-|-|\n|**InvEnc**|.884|**.827**|.326|.196|\n|**EGG**|.766|.675|.283|.214|\n|**MonkeySee**|.796|.760|**.272**|.193|\n|**MindEye2**|.798|.780|.274|.208|\n|**MEIcoder**|**.898**|.761|.277|**.181**|\n\n*[2] Paul S. Scotti, et al. (2024). MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. ICLR 2024.*\n\n**On including [1] as a baseline:** We appreciate the reviewer for pointing us to this highly relevant work. Using the original codebase of [1], we evaluated it on our benchmark datasets. We find that while it performs strongly, MEIcoder still demonstrates superior performance in the majority of cases:\n- **Brainreader:**\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**MEIcoder**|**.400**|**.679**|**.998**|**.990**|\n|**Chen 2024 [1]**|.256|.638|.930|.730|\n\n- **SENSORIUM 2022:**\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**MEIcoder**|**.331**|.503|**.988**|**.896**|\n|**Chen 2024 [1]**|.287|**.539**|.656|.649|\n\n- **Synthetic Cat V1:**\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**MEIcoder**|**.774**|.777|**.994**|**.987**|\n|**Chen 2024 [1]**|.637|**.792**|.927|.776|\n\nWe will include these comparisons and discussions in the revised manuscript. Again, thank you for this excellent suggestion.\n\n*[1] Ye Chen, et al. Decoding dynamic visual scenes across the brain hierarchy. PLOS Computational Biology 2024.*\n\n---\n\n> **Weakness 2:** *Use of low-resolution images.*\n\nWe agree about the importance of demonstrating scalability to higher resolutions. To address this concern, we have conducted a new experiment on the Brainreader dataset using a two-times higher image resolution of 72x128 pixels. We are pleased to report in the table below that MEIcoder’s performance is maintained at this higher resolution, even with very limited hyperparameter search due to time constraints. In fact, although the task of reconstructing larger images is inherently more difficult, the higher-resolution MEIcoder still outperforms the best alternative baseline from the low-resolution setting in Table 1 of our paper. \n\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**MEIcoder (36x64px)**|.400|.679|.998|.990|\n|**MEIcoder (72x128px)**|.340|.646|.997|.980|\n\nThis demonstrates that our approach is not limited to low-resolution stimuli and can effectively scale to more detailed visual inputs. While this format does not allow posting images, and NeurIPS’ rules prohibit sharing (anonymous) links, we can confidently say that the higher-resolution image reconstructions look sharp and capture fine-grained structural details very well. It is also important to note that due to the time constraints, we did not train a more complex encoding model and generate new MEIs; instead, we reused the MEIs generated with the encoder trained on the lower resolution setting. Retraining of the encoder and the generation of MEIs tailored for this higher resolution could potentially bring further improvements. We will add these new results, including a comparison with baselines and a discussion on scalability, to the final manuscript.\n\n---\n\n> **Question 1:** *A more complex encoding model and its computational cost.*\n\nThis is an excellent question regarding the computational trade-offs of our framework. For the number of neurons, we did not need to increase the complexity of the encoding model when we scaled up from 8k neurons (Brainreader and SENSORIUM 2022 datasets) to 46k neurons (Synthetic Cat V1 dataset). The architecture and most of the hyperparameters were kept the same. With regard to the resolution of visual stimuli, we also did not need to increase the complexity of the encoding model to arrive at the results from training and evaluating MEIcoder on higher-resolution images. In fact, we reused the MEIs generated for the lower-resolution images. This shows that we can keep the computational cost of encoder training and MEI generation the same when we scale to higher-resolution images.\n\nIn summary, we can state that the computational cost of encoder training and MEI generation does not constrain the application of our method to larger-scale problems.\n\n---\n\n> **Question 2:** *Details about data.*\n\nWe thank the reviewer for requesting further clarification on our experimental setup and apologize for any ambiguity in the original submission.\n\n**Repeated trials:** To clarify, repeated trials were only available for the test sets, which is a common practice in neuroscience to obtain a more stable, averaged response for evaluation. Specifically, there are 40 trials for Brainreader, 10 for SENSORIUM 2022, and 100 for Synthetic Cat V1. All data points in the training and validation sets were from single trials.\n\n**Data splits and duplicates:** For Brainreader, we used the original train/validation/test splits provided by the dataset’s authors. For SENSORIUM 2022, we randomly subdivided the original training data into our own training (4,500 points) and validation (484 points) sets; a separate test set was provided by the organizers of the SENSORIUM 2022 competition. For Synthetic Cat V1, we partitioned a large set of generated data into training (45,000), validation (5,000), and test (250) sets. We have verified that there is no overlap of images across the splits within each dataset, nor across the training and validation splits of different datasets. The only duplicate images are in the test sets of Brainreader and SENSORIUM 2022, where the original authors used the same ImageNet images.\n\n**Stimuli description:** All images, except hand-selected ones in the test sets of Brainreader and SENSORIUM 2022, were randomly sampled from ImageNet. The final sizes of the grayscale-mapped stimuli were: 36 x 64 px (Brainreader), 22 x 36 px (SENSORIUM 2022), and 20 x 20 px (Synthetic Cat V1).\n\nWe will add all the above information to the final manuscript."}, {"review_id": "yh8FAfB1lf", "reviewer": "Reviewer_1gGm", "summary": "In this work, the authors introduce MEIcoder, a method for decoding the image presented to a sensory neural population based on their responses. The MEIcoder is unique in that it combines the use of MEIs as the characteristic features of the neuron, a structural similarity index measure to assess loss in training the decoder, and an adversarial training approach to refine the decoded image. MEIcoder outperforms other recent decoding approaches based on numerous metrics of decoded image quality.", "questions": "* As the authors noted, the MEI can be seen as a nonlinear counterpart to the traditional linear receptive field (with the caveat that MEI often fails to serve as an effective filter for a linear-nonlinear model of the neuron). I’m curious to learn how much of the result would hold if the MEIs were replaced by the simple gradient-based linear receptive field of each neuron? \n* The authors present little to no analyses on the trained decoder’s neuron embeddings and the context representations $C$. What information can be obtained about the neurons based on these fitted quantities?\n* Generally, I would have liked to see more discussion on how this decoding technique may be used to understand the properties of the neurons being decoded. In particular, what the fitted parameters in the decoder may be able to tell about the nature of the neuron and how this may change as this technique is applied to more nonlinear, higher-order sensory areas?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\n* The authors offer very well-written, to-the-point, and helpful introduction to the subject matter\n* The authors provide a comprehensive review of the related work and successfully compare their work to the previous work, often directly through replication of the published method. This significantly strengthens the authors’ claims.\n* All technical and mathematical expressions are coherent and accurate\n\nClarity:\n* The paper is very well structured and clearly written, with an excellent and to-the-point summary of the main contributions of the work.\n* The term FT is used without definition. While I infer this stands for “feature transfer”, a concrete definition in the text would be useful.\n\nSignificance:\nAs the author notes, there has been a significant advancement in employing recent advances in ML to develop state-of-the-art encoding / system-identification models, and it is precisely the advent of these models that has led to the development of neuron characterizing techniques such as MEI. On the other hand, achieving effective decoding of sensory stimuli presented based on a sensory neural population has remained challenging, especially in the regime of limited neuron counts. While most recent efforts in utilizing generative AI, especially those based on diffusion models, have seen success, as the authors have noted, these methods are prone to “hallucination”. \n* The MEIcoder presented in this work offers a simple yet effective method to “decode” sensory inputs from a neural population, employing MEI as the effective template that each neuron encodes about the image. I find this use of MEI quite innovative and exciting. However, I would have liked to see analyses showing that MEI is necessary, especially in comparison to more traditional neural characterizations such as linear receptive field. \n* Also, from the analyses presented in the work, it remains unclear how much the success of this method depends on the linearity (or for that matter, simple linear-nonlinear nature) of the neurons. In other words, would this kind of MEI-based decoding prove effective for highly nonlinear neurons? \n\nOriginality:\n* Use of MEIs as a strong characterization basis in decoding neural responses back to the input image is a very novel application of MEIs, which have been developed and used primarily as a method of verifying the fidelity of encoding models that were used to generate the MEIs and to qualitatively describe the nature of the neuron.\n* The use of context representations to modulate the contributions of the individual MEI is highly innovative, and understanding the context representation $C$ in itself could lead to potential insights into exactly how each neuron contributes to the scene understanding.", "was_revised": true, "final_justification": "I maintain my score.", "rebuttal": "We thank the reviewer for their positive and insightful review. We are delighted that the reviewer recognized the novelty of our work, particularly the “innovative and exciting” use of MEIs as a decoding prior and the originality of our context modulation mechanism. We also sincerely appreciate the reviewer’s acknowledgement of the paper’s clarity of writing and of the successful comparison to previous work, which significantly strengthens our claims. In our response, we address each reviewer’s concern individually.\n\n---\n\n> **Clarity:** *The term FT is used without definition.*\n\nWe thank the reviewer for pointing out this lack of clarity. We will explicitly define “FT” as “fine-tuning” upon its first use in the revised manuscript (Section 4.4, line 234) to avoid any confusion.\n\n---\n\n> **Significance, Question 1:** *Necessity of MEIs and comparison to gradient-based linear receptive fields of neurons.*\n\nWe appreciate this insightful question. To directly test the necessity of the MEI prior, we included in the initial submission an ablation study where we removed MEIs from the readin module and used only the context representations as the initial reconstruction template. As can be seen in Figure 4, this led to a significant degradation of performance. With regards to the comparison of MEIs to more traditional neural characterizations, we conducted additional experiments where we replaced MEIs with gradient-based linear receptive fields (LRFs) obtained from regularized regression trained on neural responses from the Brainreader dataset. As shown in the table below, this also leads to degraded performance, but not as severe as with a complete removal of neural characterization as done in the initial ablation study. This demonstrates the importance of biologically informed computational prior, MEIs in particular, for the decoder’s performance.\n\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**MEIcoder (original)**|.400|.679|.998|.990|\n|**MEIcoder (LRFs)**|.364 (-9%)|.663 (-2.4%)|.998 (-0%)|.949 (-4.1%)|\n\n---\n\n> **Significance:** *Would MEI-based decoding prove effective for highly nonlinear neurons?*\n\nThis is indeed a very interesting question. To answer it most thoroughly, one would repeat our experiments in higher visual areas. Unfortunately, in the very limited time available for this rebuttal, we were not able to run such complex experiments on new data. Nevertheless, we have attempted to address the question at least within the context of the V1 data that we have available. We have calculated a so-called non-linearity index (NLI), which estimates how non-linear individual neurons are [1]. It is calculated as the ratio between the prediction power of a linear encoding model fitted to the data and the prediction power of a state-of-the-art non-linear encoding model fitted to the data. As can be seen in the table below, our ability to decode images from the more non-linear neurons is very similar to our ability to decode from more linear neurons, suggesting that our method can handle non-linearity of the code very well, at least within the context of V1.\n\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**3000 most nonlinear neurons**|.321|.630|.999|.958|\n|**3000 least nonlinear neurons**|.324|.653|.996|.939|\n|**3000 least nonlinear neurons (nonzero NLI)**|.345|.663|.999|.979|\n\n*[1] Ján Antolík, et al. (2016). Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses to Natural Scenes. PLoS Computational Biology, 12, 1–22.*\n\n---\n\n> **Originality, Question 2, Question 3:** *Potential insights into how each neuron contributes to the scene understanding. What information can we obtain about the neurons based on fitted quantities such as neuron embeddings, context representations, and other decoders’ parameters? How may this change as this technique is applied to more nonlinear, higher-order sensory areas?*\n\nWe thank the reviewer for highlighting the innovative nature of our context modulation and for these suggestions on exploring the model’s interpretability. We have performed a new set of analyses to directly probe the function of the learned parameters and provide insights into the decoding process.\n\n**Insights into neuronal contributions:** We implemented a concept-based analysis that combines two techniques: First, we use Non-negative Matrix Factorization (NMF) to learn a dictionary of 32 “visual concepts” (or basis features) from the feature maps at different layers of our decoder. Second, we perform a sensitivity analysis to observe how systematically manipulating the response of a single neuron activates these specific concepts. This provides a direct, interpretable link from a single biological neuron’s activity to the activation of learned hierarchical visual features within the decoder. Our key findings from this analysis include:\n1. *Hierarchical feature learning:* We found that for the majority of neurons, the bright regions of the visual concepts they most strongly activate become smaller and more focused as we traverse through the decoder's layers. This suggests the model learns a hierarchy, starting with coarse, broad features and progressively refining them into more detailed structures, with the final location remaining consistent with the neuron’s original MEI.\n2. *Shared and specialized neuron functions:* Many neurons’ top-activated concepts include a feature that encodes the brightness of the border around the whole image. This suggests that many neurons encode global lighting conditions, while concurrently being individually specialized in encoding fine-grained local structures at different places in the visual field (as supported by MEIs).  This might hint at how, through lateral cortical processing, our cortex fills in information where it is missing (like in the blind spot, where no photoreceptors are present).\n3. *Dynamic object representation:* We identified a few neurons for which incrementally increasing their response results in an incremental shift of a dark object in the reconstructed image. This highlights that the model has learned not just to represent static features, but also to dynamically alter the scene’s composition based on the activity of specific, influential neurons. Interestingly, this emergent property of our model mirrors key findings about functional asymmetries in the visual cortex. More specifically, a study in macaque V1 by [2] found a significant over-representation of “black-dominant” (OFF) neurons in the corticocortical output layers 2/3. As the authors noted, their results strongly suggested that black-over-white human perceptual preference is generated or greatly amplified in V1.\n\nThese new analyses begin to leverage MEIcoder as a tool for scientific discovery, revealing how it has learned to interpret and combine the contributions of individual neurons to form a coherent reconstruction, demonstrating its utility for studying visual neural codes. We have prepared visualizations of these findings, and while NeurIPS policy prohibits ways of sharing them during the rebuttal period, we will add a dedicated section with these figures and a detailed discussion in the appendix of the final manuscript.\n\n**Neuron embeddings & context representations:** We investigated whether neuron embeddings implicitly learn the retinotopic organization of V1. Using Multidimensional Scaling (MDS) to project the 32-dimensional neuron embeddings into a 2D space, we did not find that they would recover the spatial coordinates of the neurons as learned in the Gaussian readout module of our encoder architecture [3]. Regarding the context representations, our sensitivity analysis mentioned above provides a direct visualization of their function. We found that their response-dependent spatial gain map, which multiplies the base MEIs, is often spread over the whole MEI and sometimes highlights a small part of the MEI while inhibiting other parts. We acknowledge that applying other interpretability techniques and relating the findings from MDS and related dimensionality-reduction techniques to known neuronal properties is an interesting avenue for future work.\n\n**Nonlinear, higher-order sensory areas:** Our aforementioned analysis with the non-linearity index (NLI) showed that MEIcoder is robust to the nonlinearities present within V1, but we agree that areas like V4 or IT might present different and interesting challenges. We hypothesize that the principles of our analysis would remain, but the findings would shift. The NMF concepts would likely evolve from focused local structures and Gabor-like edges to more holistic object parts or textures. This represents an exciting direction for future research, where interpretability frameworks could be used to map the functional organization of these higher-order areas.\n\n*[2] Chun-I Yeh, et al. (2009). “Black” responses dominate macaque primary visual cortex v1. The Journal of neuroscience: the official journal of the Society for Neuroscience, 29(38), 11753–11760.*\n\n*[3] Konstantin-Klemens Lurz, et al. (2021). Generalization in data-driven models of primary visual cortex. bioRxiv, doi: 10.1101/2020.10.05.326256.*"}, {"review_id": "CrtBhtsVtM", "reviewer": "Reviewer_KLSf", "summary": "This paper introduces MEIcoder, a method designed to reconstruct visual stimuli from neural population activity by leveraging Most Exciting Inputs. MEIcoder combines neuron-specific MEIs, structural similarity (SSIM)-based loss, and adversarial training to achieve state-of-the-art decoding performance across datasets from primary visual cortex (V1) recordings. Experiments on three datasets  demonstrate MEIcoder's superiority, particularly in data-scarce and neuron-constrained scenarios.", "questions": "Q1. Have you examined which specific visual features are consistently reconstructed with high fidelity versus those that are reconstructed poorly? Such analysis could provide valuable insights into V1 encoding mechanisms.\n\nQ2. Have you considered exploring reconstructions through a concept-based approach, such as applying sparse dictionary learning on early layers of the model, to interpret and explicitly identify the visual features driving neuronal responses?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "I appreciate this paper's attempt to bridge comp. neuro insights with practical decoding applications through MEI as biological priors. The work demonstrates clear technical merit and addresses a timely challenge in neural decoding.\n\nHere are, to me, the main pros:\n\n- Novel biological prior: The use of MEIs as structured priors is new and well-motivated, providing a principled way to inject neuroscience knowledge into the decoder\n- Data efficiency: The method works remarkably well with limited data (<1,000 training samples) and few neurons (1,000-2,500), which has important practical implications\n- Comprehensive evaluation: The paper includes thorough ablations, scaling experiments, and a valuable benchmark aggregating 160,000+ samples across datasets\n- Strong empirical results: Clear improvements over baselines across multiple metrics and datasets, with particularly notable gains in data-scarce settings\n\nHowever, I found several areas that could strengthen the contribution, i'll group them into Major (M) and minor (m):\nMajor Issues (M):\n\n* M1. Limited scientific insights: While the technical achievement is solid, the paper would benefit from exploring what MEIcoder reveals about visual processing. Currently, it demonstrates capability without leveraging this to advance neuroscience understanding. A section analyzing reconstruction or to understand specific pattern or even better to motivate a new hypothesis (and test it) could provide valuable insights into V1 encoding principles.\n\n* M2. Limited impact and generalization concerns: The evaluation is limited to V1 and relatively simple stimuli (small grayscale images). The work would be strengthened by discussing pathways to higher visual areas or more complex stimuli, even if not immediately feasible.\n\nNow for the minor Issues (m):\n\n* m1. MEI generation details: The circular dependency of needing a pre-trained encoder for MEI generation could be better addressed. How sensitive is performance to MEI quality?\n* m2. Baseline comparisons: Some baselines perform surprisingly well (e.g., InvEnc on BRAINREADER). The paper would benefit from deeper analysis of why MEIcoder's advantages are more pronounced in certain settings.\n* m3. Computational costs: No discussion of the computational overhead for MEI generation and the parameter-efficient architecture claims need more support.\n* m4. Loss function justification: While SSIM performs well empirically, the paper could strengthen the theoretical motivation for this choice over alternatives (e.g., Feature visualization in XAI usually preconditioning in Fourier domain...)\n\n\n\n**Overall**\n\nThis work makes a valuable contribution to neural decoding by incorporating biological priors through MEIs. The technical execution is solid, and the creation of a unified benchmark is commendable. The performance in data-scarce regimes addresses a real challenge in neuroscience applications.\n\nHowever, the impact is currently limited by its focus on engineering improvements rather than scientific discovery. The paper has excellent potential to bridge this gap : the authors have built a tool that could reveal new insights about visual processing. By adding analysis of what reconstruction patterns tell us about V1 computation, or how biological and synthetic data compare, this could become a significant contribution to both ML and neuroscience.\n\nI encourage the authors to view MEIcoder not just as an improved decoder, but as a lens for understanding visual processing. With modest additions exploring the neuroscience implications of their results, this could be an excellent paper that advances both fields.", "was_revised": true, "final_justification": "convincing rebuttal and adressing M1 and M2.", "rebuttal": "We thank the reviewer for their constructive and thoughtful feedback. We sincerely appreciate the reviewer for recognizing the novelty of our work in using MEIs as a principled biological prior, the practical importance of our method’s data efficiency in challenging data-scarce scenarios, and the rigor of our comprehensive evaluation. We also value the reviewer’s perspective that the creation of a unified benchmark with over 160,000 samples is commendable in itself. We address the reviewer’s comments below.\n\n---\n\n> **M1, Question 2:** *Limited insights for neuroscience and concept-based analysis of reconstructions.*\n\nWe agree that one of the key potential contributions of any neural decoder is the scientific insight it can offer. Our primary focus in this study was to develop a new method that would push the state-of-the-art performance forward and to assemble a new benchmark dataset so that future work in this field can proceed more rigorously. Nevertheless, following the reviewer's excellent suggestion in Question 2, we have implemented a concept-based analysis that combines **(1) Non-negative Matrix Factorization** (NMF) to learn a dictionary of 32 “visual concepts” from the feature maps at different layers of our decoder, with **(2) a sensitivity analysis** to observe how manipulating the response of a single neuron activates these specific concepts. Our key findings include:\n1. For the majority of neurons, the high-intensity (bright) areas of the most active feature bases (concepts) become smaller and more focused as we traverse through the decoder’s layers. This suggests the model learns a hierarchy, starting with coarse features and progressively refining them into more detailed structures, with the final location remaining consistent with the neuron’s original MEI. \n2. Many of the neurons’ top three feature bases include a concept that encodes the brightness of the image border. Together with finding (1), this suggests that many neurons encode the global lighting condition, and at the same time specialize in encoding smaller local structures and edges at different places in the visual field (as supported by MEIs). This might hint at how, through lateral cortical processing, the cortex fills in information where it is missing.\n3. We identified a few neurons for which incrementally increasing the response results in an incremental shift of a dark object in the reconstructed image. This highlights neurons whose responses drastically affect the decoder’s learned reconstruction process. Interestingly, this emergent MEIcoder’s property mirrors key findings about functional asymmetries in the visual cortex. Namely, a study by [1] found a significant over-representation of “black-dominant” (OFF) neurons in the corticocortical output layers 2/3 of macaque V1. As the authors of [1] pointed out, their results suggested that the human perceptual preference for black over white is generated or greatly amplified in V1.\n\nThis new analysis directly addresses the major concern (M1) about scientific insight and leverages MEIcoder as a “lens for understanding visual processing”, framing it as a promising avenue for future investigation.\n\n*[1] Chun-I Yeh, et al. (2009). “Black” responses dominate macaque primary visual cortex v1. The Journal of neuroscience: the official journal of the Society for Neuroscience, 29(38), 11753–11760.*\n\n---\n\n> **M2:** *Limited impact and generalization concerns.*\n\nWith regard to **higher-resolution stimuli**, we re-ran MEIcoder on the Brainreader dataset at the resolution of 72x128 pixels. As can be seen in the table below, our method’s strong performance holds in comparison to the lower-resolution version, demonstrating its applicability to more complex visual inputs. While we could not re-run all the baselines due to the limited time of the rebuttal period, we would like to point out that although the task of reconstructing larger images is inherently more difficult, the higher-resolution MEIcoder still achieves better performance than the best alternative baseline from the low-resolution setting (Table 1, original submission), strongly indicating that it will keep its competitive edge over the baselines also for the high-resolution condition.\n\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**MEIcoder (36x64px)**|.400|.679|.998|.990|\n|**MEIcoder (72x128px)**|.340|.646|.997|.980|\n\nWe would also like to note that we did not retrain a new encoder and generate new MEIs for the higher-resolution stimuli. Instead, we reused the MEIs generated with the encoding model trained originally on the setting with 36 x 64 px images. Encoder retraining and MEI generation tailored for this higher resolution could potentially bring additional improvements. We will add these results, including visual comparison, which we cannot share now due to NeurIPS policies, along with the interpretability results from M1, to the final manuscript.\n\nRegarding the extension to **higher visual areas**, we want to clarify that our focus on V1 is a deliberate and principled choice central to the paper’s primary motivations. As stated in our manuscript, our goals are to understand the nature of early visual representations through decoding and to advance neuroengineering applications like visual prosthetics. Both domains are primarily concerned with the precise geometric structure of visual stimuli as encoded in V1, rather than the abstract semantic content represented in higher cortical areas. Rigorous experimentation on data from higher visual areas and possible modifications of MEIcoder are unfortunately infeasible due to the short time available for this rebuttal and computational constraints, but represent an exciting avenue for future work.\n\n---\n\n**m1:** To address the question of MEIcoder’s sensitivity to MEI quality, we re-ran training and evaluation on the Brainreader dataset with MEIs with varying levels (standard deviation, std) of Gaussian noise. As shown in the table below, the performance degrades relatively slowly. In fact, even with highly noisy MEIs (std=1 and std=3 for MEIs with initial pixel values between -1 and 1), the performance remains on par or better than that of the baselines (Table 1).\n\n||SSIM|PixCorr|Alex(2)|Alex(5)|\n|-|-|-|-|-|\n|**MEIcoder (no noise)**|.400|.679|.998|.990|\n|**MEIcoder (std=0.2)**|.402|.675|.997|.982|\n|**MEIcoder (std=0.5)**|.390|.670|.990|.938|\n|**MEIcoder (std=1)**|.332|.625|.990|.937|\n|**MEIcoder (std=3)**|.287|.568|.984|.943|\n\n---\n\n**m2:** Our hypothesis is that MEIcoder’s advantage is most pronounced on datasets with higher levels of intrinsic neural noise. This is supported by our finding that the trained encoder's MSE is the highest on the SENSORIUM dataset, on which MEIcoder also achieves the largest improvements over baselines (Tables 1 and 2). Our hypothesis is that the MEI prior and the adversarial objective make our decoder more robust to this noise. The MEI provides a stable template of what the neuron prefers to see, while the adversarial loss encourages the generation of sharp, plausible image structures even when the neural signal is ambiguous.\n\n---\n\n**m3:** While the entire MEI generation, including encoder training, took around 70 minutes (15 min training + 55 min generation), the decoder training required 11 hours on one NVIDIA Tesla V100 GPU. This shows that the computational overhead of MEI generation is small relative to the decoder training itself. With regard to parameter efficiency, our claim that MEIcoder is parameter-efficient lies in the fact that we:\n1. Use a CNN architecture, which achieves efficiency by parameter-sharing and by allowing only local connections.\n2. Reuse the core module (backbone) across datasets from different subjects (e.g., different mice) and only train new readin modules. By sharing the core module, MEIcoder reduces the number of parameters threefold.\n\n---\n\n**m4:** Our choice of SSIM was primarily motivated by its established ability to better capture perceptually relevant structural information compared to pixel-wise losses like MSE. As argued and demonstrated in the original SSIM paper and subsequent works such as [2], it aligns more closely with the human visual system’s perception of image quality and is often a suitable choice in image restoration applications [3].\n\n*[2] Hamid Sheikh, et al. (2006). A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Transactions on Image Processing (Volume: 15, Issue: 11).*\n\n*[3] Hang Zhao, et al. (2017). Loss Functions for Image Restoration With Neural Networks. IEEE Transactions on Computational Imaging (Volume: 3, Issue: 1).*\n\n---\n\n> **Question 1:** *Which visual features are reconstructed with high fidelity?*\n\nWe thank the reviewer for this insightful question. We performed this analysis and found two key patterns. First, the decoder’s performance directly reflects the known tuning properties of V1 neurons: it consistently reconstructs distinct, high-contrast features like corners and edges with high fidelity, while struggling with low-frequency information such as gradual changes in shading. This is a reflection of the V1 code, which is dominated by edge-detecting neurons that provide a sparse signal for uniform surfaces. Second, we identified a more global failure mode where images containing dense, high-frequency textures across a relatively small area of the image lead to a globally degraded reconstruction, an effect especially pronounced in the SENSORIUM 2022 dataset. This suggests that while V1 robustly encodes local details, the decoder can be “overwhelmed” when trying to synthesize a coherent global percept from a highly complex population signal, hinting at why the brain requires hierarchical processing. Together, these and further analyses provide valuable insights into both the encoding capacity and integrative challenges of the V1 population code, and we will add a dedicated section with illustrative examples to the revised manuscript."}, {"review_id": "aGMphZXYNt", "reviewer": "Reviewer_iULc", "summary": "This paper uses a mixture of expert knowledge, and different techniques from machine learning to study the inverse problem of taking neuronal activity and converting it to observed images. This problem is a important and complex problem for the development of prosthesis for those with vision loss or vision impairment. This work takes a non-intuitive leap in the design and approach in this problem area by focusing on most exciting inputs, which appear to play an important role in how images are encoded by the eye to neuronal signals. The ability to replicate this behavior through deep learning processes efficiently is an important computational leap. The combination of hybrid techniques used to do this replication indicate that the vision encoding problem is not simple or straightforward and is actually quite bottlenecked when it comes to energy efficiency. The presentation of the work is acceptable. The techniques are rather novel for neuroscience, and do give some evidence for related problems in computer science.", "questions": "Due to limited time in my reviewing schedule, I don't have any further questions.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nVery good empirical study\n\nThe inverse problem is less looked at, but actually more important for making BCIs as the eventual purpose of BCIs is to *stimulate* neurons as if they were being stimulated by human appendages.\n\nThe usage of a computational prior is good, but needs to be better justified. Why is a computational prior needed for this exact problem?\n\nThe development of a dataset for further work is a strong contribution.\n\nSection 4.5 forms a good section for discussion that is more than typical for AI works.\n\n\nWeaknesses:\n\nAfter having carefully read the entire appendix, there are no ethical concerns.\n\nThe necessity or importance of using SSIMs need to be better explained.\n\nThe design choice of manifold and parameter efficiency needs to be better explained.\n\nThe related work section is acceptable, but does not unify the disparate works into a consistent story or narrative.\n\nSection 3 is a bit dry to read.", "was_revised": true, "final_justification": "I maintain my score.", "rebuttal": "We thank the reviewer for their supportive review and comments. We are particularly grateful that the reviewer recognized the importance of our work for the development of brain-machine interfaces, the strength of our empirical study, and the value of our new benchmark dataset as a “strong contribution” to the community. The reviewer’s feedback primarily points to valuable opportunities to improve the clarity and justification of our methodological choices. We agree that these explanations will strengthen the paper, and in our response below, we provide these detailed justifications for the use of computational priors, our SSIM-based loss, and the parameter efficiency of our architecture, which we are confident will fully address the reviewer’s points.\n\n---\n\n> **Strength 3:** *Usage of MEIs as a computational prior is good, but needs to be better justified. Why is a computational prior needed for this exact problem?*\n\nWe thank the reviewer for pointing out the soundness of MEIs as a computational prior and their follow-up question, which gets to the heart of our motivation. A computational prior is particularly crucial in this setting for two main reasons. First, neural recording datasets are often data-limited, and deep learning models benefit from strong priors to prevent overfitting and guide learning [1,2]. Second, and more specific to this problem, many neurons exhibit sparse firing. A decoder may struggle to learn the stimulus-response mapping for a neuron that was active for only a handful of images in the training set. The MEI provides a powerful “head-start” by giving the decoder a dense, explicit template of each neuron’s preferred stimulus, even for rarely firing neurons. This is visually demonstrated in Figure 9, where simply weighting MEIs by neural responses already forms a coarse but recognizable reconstruction, highlighting the power of this prior. We will include this justification in the next revision of the paper.\n\n*[1] Christopher Bishop (2006). Pattern Recognition and Machine Learning. Springer.*\n\n*[2] Peter W. Battaglia, et al. (2018). Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261.*\n\n---\n\n> **Weakness 1:** *Ethical concerns.*\n\nWe thank the reviewer for their careful assessment. We have included a “Broader Impacts” section (A.6) in the original submission that discusses potential ethical considerations and societal impacts of this research line. If the reviewer considers this to be insufficient, please let us know, and we would be happy to expand on ethical concerns further.\n\n---\n\n> **Weakness 2:** *Importance of SSIM.*\n\nWe thank the reviewer for this comment. Our choice is motivated by our comparison with MSE (Section 4.5, ablation study) and SSIM’s proven ability to capture structural similarity in a way that aligns better with human perception than pixel-wise losses [3,4]. Since our primary goal is geometric fidelity for applications like visual prosthetics, a loss function that prioritizes perceptually coherent structures over exact pixel intensities is more appropriate. We will ensure this reasoning is stated more clearly in Section 3.1.\n\n*[3] Zhoung Wang, et al. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600-612.*\n\n*[4] Hamid Sheikh, et al. (2006). A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Transactions on Image Processing (Volume: 15, Issue: 11).*\n\n---\n\n> **Weakness 3:** *The design choice of manifold and parameter efficiency needs to be better explained.*\n\nWe thank the reviewer and apologize for the lack of clarity. Regarding parameter efficiency, we will clarify that it refers to the following two properties of MEIcoder:\n1. MEIcoder employs a CNN architecture, which achieves efficiency by sharing parameters across different spatial locations within a feature map, and by using only local connections within its layers, as opposed to fully-connected networks.\n2. MEIcoder reuses the core module across datasets from different subjects (e.g., 8 different mice from Brainreader) and trains only new subject-specific readin modules. For example, on the Brainreader dataset, the core module has around 18.5M parameters, and a single readin module contains around 4.5M parameters. If we trained both the core and the readin for each of the 8 mice in the Brainreader dataset separately, we would need around 184M parameters. By reusing the same core module across subjects, MEIcoder is able to lower this from 184M to just 54.5M.\n\nWe are uncertain what the reviewer meant by “manifold”; if this refers to the latent space of our model, we will add more details on its structure in our new interpretability section. We would be grateful for any clarification the reviewer could provide on this point.\n\n---\n\n> **Weakness 4:** *Related work section.*\n\nWe appreciate this feedback. We will revise the related work section to not only list prior studies but also to synthesize them into a more cohesive narrative, highlighting the key challenges in the field (e.g., data limitation, geometric vs. semantic fidelity) and clearly positioning our work within this context.\n\n---\n\n> **Weakness 5:** *Section 3 is a bit dry to read.*\n\nWe thank the reviewer for this candid feedback. We will revise Section 3 to improve its readability, for instance, by adding more high-level motivation for our design choices and ensuring a clearer flow between the technical descriptions."}]

Author rebuttal:

Meta-review: This paper introduces MEIcoder, a method for reconstructing visual stimuli from neural population activity by leveraging neuron-specific Most Exciting Inputs (MEIs), structural similarity (SSIM)-based loss, and adversarial training. The model demonstrates strong performance in low-data and low-neuron regimes, offering both technical advances and potential utility for neuroscience and brain–machine interface applications.

The reviewers highlighted several strengths: the novelty of using MEIs as biologically grounded priors, strong empirical performance across datasets, data efficiency in scarce regimes, and a well-executed evaluation including ablations and scaling experiments. Weaknesses raised included fairness of baselines and evaluation metrics, limited discussion of scalability to higher-resolution or higher-order brain areas, and the need for deeper analysis of the neuroscience insights gained from the reconstructions.

The authors’ rebuttal and follow-up experiments convincingly addressed these concerns. They added comparisons with previously omitted baselines, demonstrated MEIcoder’s robustness on semantic metrics and at higher resolutions, and provided new analyses (e.g., concept-based decomposition, sensitivity analyses, and linear RF ablations) that underscored MEIs’ necessity and interpretability. These additions strengthened the claims and reassured reviewers about both methodological rigor and broader relevance.

After discussion, consensus among reviewers was broadly positive. Initial concerns about fairness, scalability, and interpretability were largely alleviated, with reviewers explicitly raising their scores. Some lingering limitations remain regarding generalization to higher-order visual areas and broader neuroscience insights, but reviewers agreed that the work is solid, innovative, and timely. The overall sentiment converged on acceptance.

Given all this, the AC recommends the paper to be accepted, as it makes a clear and well-supported contribution to neural decoding and will likely stimulate further work at the interface of AI and neuroscience.
Decision: Accept (poster)