Title: MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neural decoding

Neural population activity

Most exciting inputs

Image reconstruction

Visual cortex
Abstract: Decoding visual stimuli from neural population activity is crucial for understanding the brain and for applications in brain-machine interfaces. However, such biological data is often scarce, particularly in primates or humans, where high-throughput recording techniques, such as two-photon imaging, remain challenging or impossible to apply. This, in turn, poses a challenge for deep learning decoding techniques. To overcome this, we introduce MEIcoder, a biologically informed decoding method that leverages neuron-specific most exciting inputs (MEIs), a structural similarity index measure loss, and adversarial training. MEIcoder achieves state-of-the-art performance in reconstructing visual stimuli from single-cell activity in primary visual cortex (V1), especially excelling on small datasets with fewer recorded neurons. Using ablation studies, we demonstrate that MEIs are the main drivers of the performance, and in scaling experiments, we show that MEIcoder can reconstruct high-fidelity natural-looking images from as few as 1,000-2,500 neurons and less than 1,000 training data points. We also propose a unified benchmark with over 160,000 samples to foster future research. Our results demonstrate the feasibility of reliable decoding in early visual system and provide practical insights for neuroscience and neuroengineering applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses a mixture of expert knowledge, and different techniques from machine learning to study the inverse problem of taking neuronal activity and converting it to observed images. This problem is a important and complex problem for the development of prosthesis for those with vision loss or vision impairment. This work takes a non-intuitive leap in the design and approach in this problem area by focusing on most exciting inputs, which appear to play an important role in how images are encoded by the eye to neuronal signals. The ability to replicate this behavior through deep learning processes efficiently is an important computational leap. The combination of hybrid techniques used to do this replication indicate that the vision encoding problem is not simple or straightforward and is actually quite bottlenecked when it comes to energy efficiency. The presentation of the work is acceptable. The techniques are rather novel for neuroscience, and do give some evidence for related problems in computer science.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nVery good empirical study\n\nThe inverse problem is less looked at, but actually more important for making BCIs as the eventual purpose of BCIs is to *stimulate* neurons as if they were being stimulated by human appendages.\n\nThe usage of a computational prior is good, but needs to be better justified. Why is a computational prior needed for this exact problem?\n\nThe development of a dataset for further work is a strong contribution.\n\nSection 4.5 forms a good section for discussion that is more than typical for AI works.\n\n\nWeaknesses:\n\nAfter having carefully read the entire appendix, there are no ethical concerns.\n\nThe necessity or importance of using SSIMs need to be better explained.\n\nThe design choice of manifold and parameter efficiency needs to be better explained.\n\nThe related work section is acceptable, but does not unify the disparate works into a consistent story or narrative.\n\nSection 3 is a bit dry to read.\n\nQUESTIONS:\nDue to limited time in my reviewing schedule, I don't have any further questions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MEIcoder, a method designed to reconstruct visual stimuli from neural population activity by leveraging Most Exciting Inputs. MEIcoder combines neuron-specific MEIs, structural similarity (SSIM)-based loss, and adversarial training to achieve state-of-the-art decoding performance across datasets from primary visual cortex (V1) recordings. Experiments on three datasets  demonstrate MEIcoder's superiority, particularly in data-scarce and neuron-constrained scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nI appreciate this paper's attempt to bridge comp. neuro insights with practical decoding applications through MEI as biological priors. The work demonstrates clear technical merit and addresses a timely challenge in neural decoding.\n\nHere are, to me, the main pros:\n\n- Novel biological prior: The use of MEIs as structured priors is new and well-motivated, providing a principled way to inject neuroscience knowledge into the decoder\n- Data efficiency: The method works remarkably well with limited data (<1,000 training samples) and few neurons (1,000-2,500), which has important practical implications\n- Comprehensive evaluation: The paper includes thorough ablations, scaling experiments, and a valuable benchmark aggregating 160,000+ samples across datasets\n- Strong empirical results: Clear improvements over baselines across multiple metrics and datasets, with particularly notable gains in data-scarce settings\n\nHowever, I found several areas that could strengthen the contribution, i'll group them into Major (M) and minor (m):\nMajor Issues (M):\n\n* M1. Limited scientific insights: While the technical achievement is solid, the paper would benefit from exploring what MEIcoder reveals about visual processing. Currently, it demonstrates capability without leveraging this to advance neuroscience understanding. A section analyzing reconstruction or to understand specific pattern or even better to motivate a new hypothesis (and test it) could provide valuable insights into V1 encoding principles.\n\n* M2. Limited impact and generalization concerns: The evaluation is limited to V1 and relatively simple stimuli (small grayscale images). The work would be strengthened by discussing pathways to higher visual areas or more complex stimuli, even if not immediately feasible.\n\nNow for the minor Issues (m):\n\n* m1. MEI generation details: The circular dependency of needing a pre-trained encoder for MEI generation could be better addressed. How sensitive is performance to MEI quality?\n* m2. Baseline comparisons: Some baselines perform surprisingly well (e.g., InvEnc on BRAINREADER). The paper would benefit from deeper analysis of why MEIcoder's advantages are more pronounced in certain settings.\n* m3. Computational costs: No discussion of the computational overhead for MEI generation and the parameter-efficient architecture claims need more support.\n* m4. Loss function justification: While SSIM performs well empirically, the paper could strengthen the theoretical motivation for this choice over alternatives (e.g., Feature visualization in XAI usually preconditioning in Fourier domain...)\n\n\n\n**Overall**\n\nThis work makes a valuable contribution to neural decoding by incorporating biological priors through MEIs. The technical execution is solid, and the creation of a unified benchmark is commendable. The performance in data-scarce regimes addresses a real challenge in neuroscience applications.\n\nHowever, the impact is currently limited by its focus on engineering improvements rather than scientific discovery. The paper has excellent potential to bridge this gap : the authors have built a tool that could reveal new insights about visual processing. By adding analysis of what reconstruction patterns tell us about V1 computation, or how biological and synthetic data compare, this could become a significant contribution to both ML and neuroscience.\n\nI encourage the authors to view MEIcoder not just as an improved decoder, but as a lens for understanding visual processing. With modest additions exploring the neuroscience implications of their results, this could be an excellent paper that advances both fields.\n\nQUESTIONS:\nQ1. Have you examined which specific visual features are consistently reconstructed with high fidelity versus those that are reconstructed poorly? Such analysis could provide valuable insights into V1 encoding mechanisms.\n\nQ2. Have you considered exploring reconstructions through a concept-based approach, such as applying sparse dictionary learning on early layers of the model, to interpret and explicitly identify the visual features driving neuronal responses?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors introduce MEIcoder, a method for decoding the image presented to a sensory neural population based on their responses. The MEIcoder is unique in that it combines the use of MEIs as the characteristic features of the neuron, a structural similarity index measure to assess loss in training the decoder, and an adversarial training approach to refine the decoded image. MEIcoder outperforms other recent decoding approaches based on numerous metrics of decoded image quality.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n* The authors offer very well-written, to-the-point, and helpful introduction to the subject matter\n* The authors provide a comprehensive review of the related work and successfully compare their work to the previous work, often directly through replication of the published method. This significantly strengthens the authors’ claims.\n* All technical and mathematical expressions are coherent and accurate\n\nClarity:\n* The paper is very well structured and clearly written, with an excellent and to-the-point summary of the main contributions of the work.\n* The term FT is used without definition. While I infer this stands for “feature transfer”, a concrete definition in the text would be useful.\n\nSignificance:\nAs the author notes, there has been a significant advancement in employing recent advances in ML to develop state-of-the-art encoding / system-identification models, and it is precisely the advent of these models that has led to the development of neuron characterizing techniques such as MEI. On the other hand, achieving effective decoding of sensory stimuli presented based on a sensory neural population has remained challenging, especially in the regime of limited neuron counts. While most recent efforts in utilizing generative AI, especially those based on diffusion models, have seen success, as the authors have noted, these methods are prone to “hallucination”. \n* The MEIcoder presented in this work offers a simple yet effective method to “decode” sensory inputs from a neural population, employing MEI as the effective template that each neuron encodes about the image. I find this use of MEI quite innovative and exciting. However, I would have liked to see analyses showing that MEI is necessary, especially in comparison to more traditional neural characterizations such as linear receptive field. \n* Also, from the analyses presented in the work, it remains unclear how much the success of this method depends on the linearity (or for that matter, simple linear-nonlinear nature) of the neurons. In other words, would this kind of MEI-based decoding prove effective for highly nonlinear neurons? \n\nOriginality:\n* Use of MEIs as a strong characterization basis in decoding neural responses back to the input image is a very novel application of MEIs, which have been developed and used primarily as a method of verifying the fidelity of encoding models that were used to generate the MEIs and to qualitatively describe the nature of the neuron.\n* The use of context representations to modulate the contributions of the individual MEI is highly innovative, and understanding the context representation $C$ in itself could lead to potential insights into exactly how each neuron contributes to the scene understanding.\n\nQUESTIONS:\n* As the authors noted, the MEI can be seen as a nonlinear counterpart to the traditional linear receptive field (with the caveat that MEI often fails to serve as an effective filter for a linear-nonlinear model of the neuron). I’m curious to learn how much of the result would hold if the MEIs were replaced by the simple gradient-based linear receptive field of each neuron? \n* The authors present little to no analyses on the trained decoder’s neuron embeddings and the context representations $C$. What information can be obtained about the neurons based on these fitted quantities?\n* Generally, I would have liked to see more discussion on how this decoding technique may be used to understand the properties of the neurons being decoded. In particular, what the fitted parameters in the decoder may be able to tell about the nature of the neuron and how this may change as this technique is applied to more nonlinear, higher-order sensory areas?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn order to reconstruct high-quality visual stimuli at the pixel level from limited neural data, this work introduces MEIcoder, which incorporates neuron-specific most exciting inputs into a visual-stimuli decoding framework. In the experiments on three neural datasets, MEIcoder achieves the best reconstruction performance compared to alternative models. Furthermore, in scaling experiments, the results demonstrate the superiority of MEIcoder in dealing with small numbers of neurons and data points.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This work is well-motivated, and introducing MEIs into decoding models is novel and meaningful.\n2. MEIcoder outperforms several outstanding and well-known models in multiple metrics and visualization, demonstrating its effectiveness.\n3. Comprehensive ablation studies prove the role of each module of MEIcoder, especially MEIs.\n\nWeaknesses:\n\n1. My primary concern is the fairness of the experiments. First, the authors do not provide results of the datasets used in baseline models and instead conduct comparisons on new datasets (where only the Brainreader dataset has been used in InvEnc). Second, MindEye2 is more focused on semantically reconstructing visual stimuli, which inherently puts it at a disadvantage in low-level or even pixel-level metrics. Could the authors provide results using higher-level metrics employed in MindEye2? Finally, [1] has reconstructed high-quality movie stimuli from the neural activity of hundreds of mouse visual cortex neurons. I think this study should be included in the comparison, as it also targets small datasets and pixel-level reconstruction.\n2. Since the visual stimuli in this work have low resolution, it is unclear whether the model can be scaled up to handle higher-resolution visual stimulus reconstruction.\n\n[1] YeChen, et al. Decoding dynamic visual scenes across the brain hierarchy. PLOS Computational Biology 2024.\n\nQUESTIONS:\n1. When the number of neurons and the resolution of visual stimuli increase, is a more complex encoding model required to calculate MEIs, and can the computational cost still be ignored?\n2. Are there repeated trials for each visual stimulus in each dataset? How are the training set, validation set, and test set divided, and do they contain repeated images? Can the authors provide a more detailed description of the visual stimuli used in each dataset?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}