Title: NEED: Cross-Subject and Cross-Task Generalization for Video and Image Reconstruction from EEG Signals
Venue: NeurIPS 2025
Accepted: yes
Keywords: EEG

Cross-Subject

Cross-Task

Video Reconstruction

Image Reconstruction
Abstract: Translating brain activity into meaningful visual content has long been recognized as a fundamental challenge in neuroscience and brain-computer interface research. Recent advances in EEG-based neural decoding have shown promise, yet two critical limitations remain in this area: poor generalization across subjects and constraints to specific visual tasks. We introduce NEED, the first unified framework achieving zero-shot cross-subject and cross-task generalization for EEG-based visual reconstruction. Our approach addresses three fundamental challenges: (1) cross-subject variability through an Individual Adaptation Module pretrained on multiple EEG datasets to normalize subject-specific patterns, (2) limited spatial resolution and complex temporal dynamics via a dual-pathway architecture capturing both low-level visual dynamics and high-level semantics, and (3) task specificity constraints through a unified inference mechanism adaptable to different visual domains. For video reconstruction, NEED achieves better performance than existing methods. Importantly, Our model maintains 93.7% of within-subject classification performance and 92.4% of visual reconstruction quality when generalizing to unseen subjects, while achieving an SSIM of 0.352 when transferring directly to static image reconstruction without fine-tuning, demonstrating how neural decoding can move beyond subject and task boundaries toward truly generalizable brain-computer interfaces.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework named NEED to address the challenging problem of zero-shot cross-subject and cross-task generalization in EEG-based visual reconstruction. The framework comprises three core components: an Individual Adaptation Module (IAM) for normalizing inter-subject variability, a Dual-Stream EEG Encoder (DSGNet) for capturing both spatial and temporal features, and a Unified Inference Mechanism to enable task-adaptive reasoning. Extensive experiments on visual perception classification and visual reconstruction tasks demonstrate that NEED consistently outperforms existing baselines, highlighting its strong generalization ability and robustness.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n1. This work introduces the first unified framework, NEED, that achieves zero-shot generalization across both subjects and tasks in EEG decoding, addressing two fundamental challenges in the field.\n\n2. The proposed approach is comprehensively designed, encompassing critical aspects such as signal preprocessing, spatiotemporal feature extraction, and multimodal fusion.\n\n3. Extensive evaluations on benchmark datasets demonstrate significant performance improvements, particularly under cross-subject and cross-task scenarios. Thorough ablation studies (including IAM, PU, and SU modules) validate the contribution of each component and confirm the robustness of the overall framework.\n\n\n### Weakness\n\n1. The overall framework contains numerous modules, but the motivations behind some components are not clearly articulated. For example, in the Temporal Stream Processing part of DSGNet, the rationale for introducing the generalized Riemann-Liouville fractional transformation is not sufficiently explained.\n\n2. Some mathematical formulations lack complete variable definitions, which affects clarity and reproducibility. For instance, in Section 2.2 (Line 99), the multi-objective loss function is presented without adequate explanation of its variables. Similarly, in Section 2.3 (Line 140), the source of the enhanced visual features V1 is unclear.\n\n3. The section titles and their content are misaligned: Section 4.1 is titled \"Visual Perception Benchmark\", yet it only includes video perception results and omits image perception. Conversely, Section 4.2 is titled \"Video Reconstruction Results\", but includes both video and image reconstruction tasks. Renaming it to \"Visual Reconstruction Results\" would better reflect the content.\n\n4. The manuscript contains a number of typographical and formatting issues, such as:\n   - Most subscripts in equations are not enclosed using `\\textbf{}` for consistent formatting;\n   - Punctuation errors, e.g., missing periods in Lines 72 and 107, an extra period in Line 214, and a missing space in the formula in Line 98;\n   - Inconsistent figure references—some include a period after \"Fig\", while others do not. A thorough proofreading is recommended to ensure consistency.\n\nQUESTIONS:\n1. It is quite surprising that DSGNet achieves over a 100% improvement in 40-class classification accuracy on the SEED-DV dataset compared to baseline models. Could you provide more details on the experimental setup and share the corresponding code for DSGNet? Additionally, which specific design choices contribute to such a substantial performance gain?\n2. EEG2Video employs GLMNet as its EEG encoder, whereas NEED uses DSGNet. Although DSGNet clearly outperforms GLMNet in video perception classification, NEED does not show a correspondingly large improvement in video reconstruction performance compared to EEG2Video. Have you considered controlling for the encoder variable by using GLMNet within the NEED framework to allow for a more direct and fair comparison of the reconstruction capabilities?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified framework, NEED, designed to address cross-subject variability and taskspecificity in EEG-based visual reconstruciton. This framework supports zero-shot transfer between video and image tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The authors propose a pretrained Individual Adaptation Module to address subject variability and a unified inference mechanism to generate reconstructions across different tasks.\n2. The authors provide a detailed analysis of the impact of using different electrode combinations from brain regions on the reconstruction performance in video and image reconstruction tasks.\nWeaknesses\n1. The presentation of the paper needs further improvement, particularly in the Method section. There are many undefined symbols, which make it difficult to understand.\n2. There is a lack of a quantitative comparison in the video and image reconstruction tasks. The paper includes fMRI-based methods in the comparison, despite not using an fMRI dataset. Additionally, the experimental setup is not clearly stated, making it difficult to reasonably justify the model's performance on the SEED-DV dataset.\n\nQUESTIONS:\n1. What is the purpose behind the design of the Cross-Stream Integration mechanism as presented in the paper? For instance, why have you chosen to treat as a KV? What specific performance improvements does the addition of this MASK bring to the model?\n2. In the decoding module, the paper proposes modifications to common loss functions, such as incorporating the L2 norm based on temporal gradients or the Gram matrix. What is the purpose of these modifications? How would the model's performance be affected without them?\n3. In Table 2, was the performance of the proposed framework tested on the complete SEED-DV dataset, or was it only tested on the 10-class dataset?\n\nLIMITATIONS:\nNo, the limitations should be included for discussions as well.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript presents a cross-subject and cross-task EEG decoding framework composed of three main components: (1) multi-dataset pretraining, (2) a \"perception\" module that aligns EEG signals with a pretrained variational autoencoder (VAE), and (3) a \"semantic\" module that aligns EEG signals with both the vision and language encoders of CLIP. Additionally, the manuscript introduces a novel dual-stream EEG encoder designed to capture both the spatial and temporal characteristics of EEG recordings. The full model is trained on three large datasets in which participants viewed video clips and is subsequently evaluated on two additional datasets to assess cross-task and cross-subject generalisation. The main finding suggests that the proposed framework is capable of decoding EEG signals from a new participant performing a new task with promising accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n1. **Strong potential**: The manuscript addresses the challenging and important problem of decoding EEG signals across both participants and tasks. The proposed framework has significant potential and, with suitable modifications, could be extended to other neuroimaging modalities.\n\n2. **Text organisation**: The manuscript frequently refers back and forth between the main text and appendices, which disrupts the flow and makes it difficult to follow the authors' line of reasoning. In order to understand key details, I had to keep multiple tabs open simultaneously. This significantly affects the readability and accessibility of the work.\n\n3. **Overloaded scope**: Although the manuscript includes many valuable contributions, their presentation within a single compressed manuscript makes it difficult to clearly understand or scientifically assess the individual impact of each component.\n\n4. **Terminology**: The use of \"perception\" and \"semantic understanding\" throughout the manuscript is confusing. I suggest adopting more precise terms such as *visual* and *language features*, which better reflect the training objectives of the respective modules. The term \"perception\" typically refers to sensory processing (e.g., visual, auditory), while \"semantic\" can refer to either visual scene understanding (e.g., semantic segmentation) or linguistic meaning—both of which are overloaded in this context.\n\n5. **Missing details on VAE**: There is no information provided about the frozen VAE model. Was it trained by the authors on the same image dataset, or is it an existing state-of-the-art pretrained VAE? This needs clarification.\n\n6. **Reproducibility concerns**: In its current form, the manuscript lacks sufficient implementation details to allow reproducibility. Although the authors state they will release code upon acceptance, this is not a substitute for a reproducible submission.\n\n7. **Frozen adapter and prompting**: The role of the frozen adapter is unclear, and the manuscript does not explain what it refers to or how it functions. Additionally, the impact of prompt engineering on the large language model is not discussed, though this could significantly affect performance.\n\n8. **Dataset inconsistencies**: There are inconsistencies in the description of the THINGS EEG dataset. The stimuli in this dataset are presented for 100 ms with 100 ms inter-stimulus intervals, and include 16,540 images shown 4 times for training, and 200 images shown 80 times for testing. This does not align with the manuscript’s claim of “22,000 static images” and 500 ms stimulus duration, which should be corrected or clarified.\n\n9. **Appendix figure numbering**: Figures in the appendices should be labelled using a consistent scheme (e.g., A1, A2, etc.) rather than continuing the numbering from the main manuscript.\n\n10. **Figure clarity**: Figure 7 (in the appendices) contains many unexplained abbreviations, which makes it difficult to interpret. A more comprehensive caption and clearer legend would greatly improve readability.\n\n11. **Algorithm 1 clarity**: Algorithm 1 is not self-explanatory and would benefit from brief comments to aid understanding. Ideally, it should align more closely with the schematic in Figure 2, using consistent terminology and structure.\n\n12. **Need for a schematic of DSGNet**: A schematic representation of DSGNet would be highly informative and would help readers understand the model architecture at a glance.\n\nQUESTIONS:\n1. **BLIP redundancy and clarification**: The use of BLIP in both stages of the Semantic Understanding (SU) module appears redundant and potentially confusing. As illustrated in the schematic flowchart, BLIP seems to generate the same caption for two different images, which raises questions about its purpose in both stages. This redundancy needs to be clarified—are the captions generated independently, or reused across branches? Further explanation would help clarify the model's flow and the role of each BLIP instance.\n\n2. **Comparative evaluation with prior work**: To demonstrate the cross-subject generalisation capabilities of IAM, it would be valuable to directly compare its performance with existing EEG decoding frameworks such as NICE, ATM, or IDES. These previous methods often report strong within-subject decoding but struggle with generalisation across participants. A quantitative comparison would clarify how much IAM improves over these methods and could arguably represent the most impactful contribution of the paper. This deserves deeper experimental validation.\n\n3. **IAM loss components**: The IAM framework includes two main loss components: reconstruction and discrimination. However, the manuscript does not clarify the relative contribution of each to learning dataset-independent representations. An ablation or sensitivity analysis examining the effect of removing or varying the weight of each term would be informative. Similarly, it would be helpful to understand how each component of the IAM module, as illustrated in Figure 7, contributes to the overall performance.\n\n4. **Unclear evaluation setup in Table 1**: It is not clear which dataset is used for the evaluation results reported in Table 1. Are the competing methods trained and tested on the same dataset, and under the same experimental conditions? If this table is intended to demonstrate architectural effectiveness, it would also be appropriate to report the number of parameters and computational cost associated with each method.\n\n5. **Meaningfulness of comparisons in Table 2**: The results in Table 2 compare performance across different neuroimaging techniques, but it is unclear whether these results are based on the same set of images or tasks. If the underlying image sets differ, then the comparisons may not be meaningful. Please clarify whether the input stimuli are standardised across modalities, and if not, discuss the implications for interpreting these results.\n\n6. **Ablation insights and engineering influence**: The performance gap between the full model and individual ablation studies is substantial. This raises the question of whether any single component is inherently effective, or whether the observed gains primarily result from the specific combination and integration of all components. This is a critical question—does the strength of the model lie in its individual modules, or in the way they are engineered together? The manuscript would benefit from a deeper discussion of this point.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces NEED, a comprehensive framework for reconstructing visual stimuli (both videos and images) from EEG signals. The primary goal is to address two major challenges in the field: poor generalization across different subjects and the inability of models to transfer between different tasks (e.g., video vs. image reconstruction) without retraining. \n\nThe proposed NEED framework is a multi-stage, complex system featuring three main keypoints: \n1) an Individual Adaptation Module (IAM) to normalize EEG signals across subjects, \n2) a dual-pathway architecture (Perception and Semantic) to extract different levels of visual information, \n3) a unified inference mechanism for handling different tasks. \n\nThe extensive experiments, including several ablation studies, demonstrating the proposed model outperforms existing methods and achieves strong cross-subject and cross-task generalization, notably retaining over 90% of within-subject performance when generalizing to new subjects.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n1. Ambitious and Important Problem: The paper tackles two of the most significant and difficult challenges in BCI and neural decoding: cross-subject and cross-task generalization. Developing a single, unified model that can work \"out-of-the-box\" for new users and new visual paradigms is a crucial step toward practical, real-world applications. The ambition of this work is commendable.\n\n2. Comprehensive Experimental Evaluation: The authors have performed an impressive number of experiments on multiple public datasets (SEED-DV, THINGS-EEG, DEAP, etc.). The evaluation is multifaceted, using metrics for perceptual quality (SSIM, PSNR), semantic accuracy (CLIP-based classification), and temporal dynamics (FVD). The cross-subject analysis and ablation studies are thorough and provide strong evidence for the effectiveness of certain components, particularly the IAM.\n\n3. Strong Empirical Results: The reported results are strong. The DSGNet encoder shows significant classification improvements over numerous baselines (Table 1). The full NEED model achieves SOTA or competitive performance in both video and image reconstruction tasks (Table 2). The high performance retention rate for cross-subject generalization (over 90% with 19 training subjects) directly supports one of the main claims of the paper.\n\n\n# Weakness\n\nMy main concerns with this paper are its overwhelming complexity and the corresponding lack of clarity in the presentation, which makes the method difficult to understand and critically evaluate.\n\n1. Overwhelming System Complexity: The proposed NEED framework is exceedingly complex. It combines multiple pre-training stages, a dual-stream encoder (DSGNet), a dual-pathway understanding architecture (PU and SU), each of which contains multiple sub-modules (IAM, DFE, SMARN, CLIP, BLIP-2, LLM), and a sophisticated, multi-guidance inference mechanism (ControlNet, TaskAdapter, etc.). While the ablation studies attempt to justify each piece, the sheer number of components makes the system feel like an elaborate ensemble of many existing powerful models rather than a system with a few core, well-motivated innovations. It is difficult for a reader to discern whether the strong performance comes from a few key insights or simply from the brute-force combination of many SOTA components. The authors should consider if a simpler architecture could achieve comparable results, as complexity can obscure scientific understanding.\n\n2. Excessive Reliance on Supplementary Material: The paper is not self-contained, which severely hinders fluent reading and comprehension. Critical details about the methodology are deferred to the appendix. Many mathematical symbols and components are used in the main text without a clear definition. And many of method details forces a constant back-and-forth between the main text and the appendix, disrupting the flow of reading. Key architectural details for the DSGNet encoder and the crucial IAM are almost entirely in the appendix. The main text provides only a high-level overview, making it hard to understand the method from the main paper alone.\n\n3. High Information Density and Lack of Clarity: The \"Method\" section (Sec 2) is a whirlwind tour of dozens of advanced concepts packed into a very small space. Each component, from the Riemann-Liouville fractional transformation to the Spatiotemporal Memory-Augmented Recurrent Network (SMARN), is introduced and dismissed in a sentence or two.\n\n4. Motivation: The motivation for many design choices is not sufficiently explained. For instance, why was a Riemann-Liouville fractional transformation chosen for the temporal stream over more standard techniques? The justification is brief. Why is the SMARN architecture necessary, and how does it specifically improve upon standard RNNs or Transformers in this context?\n\n5. Clarity: The connections between the numerous components are not always clear. Figure 2 is incredibly dense and difficult to parse, with a multitude of pathways, losses, and modules. A simplified diagram focusing on the core data flow, followed by detailed diagrams for each major component, would be more effective.\n\nQUESTIONS:\nPlease refer to Weakness section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}