NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models

Authors: Chi-Sheng Chen
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: EEG, diffusion model, EEG to image, brain-computer interface, image reconstruction

Abstract:
NECOMIMI (NEural-COgnitive MultImodal EEG-Informed Image Generation with Diffusion Models) introduces a novel framework for generating images directly from EEG signals using advanced diffusion models. Unlike previous works that focused solely on EEG-image classification through contrastive learning, NECOMIMI extends this task to image generation. The proposed NERV EEG encoder demonstrates state-of-the-art (SoTA) performance across multiple zero-shot classification tasks, including 2-way, 4-way, and 200-way, and achieves top results in our newly proposed Category-based Assessment Table (CAT) Score, which evaluates the quality of EEG-generated images based on semantic concepts. A key discovery of this work is that the model tends to generate abstract or generalized images, such as landscapes, rather than specific objects, highlighting the inherent challenges of translating noisy and low-resolution EEG data into detailed visual outputs. Additionally, we introduce the CAT Score as a new metric tailored for EEG-to-image evaluation and establish a benchmark on the ThingsEEG dataset. This study underscores the potential of EEG-to-image generation while revealing the complexities and challenges that remain in bridging neural activity with visual representation.

Decision: reject
Reviewer scores: 3, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730133713134

thread_type:
official_review

<thread_object>
creator: reviewer_8ffj

<content>
summary:
The paper proposes a model named NECOMIMI, designed to generate images using EEG data alone. It introduces a novel NERV encoder and highlights two key innovations: first, using only EEG signals as embeddings in a diffusion model for image generation, and second, introducing a novel evaluation metric, CAT score, to assess the quality of generated images due to the lack of existing standards for EEG-to-image translation.

strengths:
1. The denoising method at line 318 in the two-stage process allows the model to produce finer-grained images.

2. Results in Tables 2 and 3 show that the experiments perform reasonably well on the ThingsEEG dataset.

3. The CAT score introduced in Section 4.4 reflects a stringent evaluation standard, with no EEG encoder exceeding 500 points, suggesting the metric’s reliability.

weaknesses:
1. The innovative aspect of the NERV encoder, starting from line 347, lacks sufficient mathematical justification for feasibility. The data presented in Tables 1, 2 and 3 may be coincidental without solid statistical analysis.

2. While Figure 1 demonstrates the model's generative capability, it lacks a thorough analysis of generated image quality. Similarly, Figures 4 onward depict images that visually lack resemblance to any original source.

3. The paper lacks a thorough analysis or proof regarding the selection of regularization parameters in the diffusion model and the settings of the fully connected layers and their impact on the generated images.

questions:
1. The theoretical analysis lacks robust supports, including the hyperparameter settings for the NERV encoder, which raises concerns about rigor. Experimental validation in this paper is not sufficiently convincing.

2. While the experimental section showcases generated images, these are primarily abstract landscapes, with little detail on improving output quality. Adding a discussion or analysis on image quality enhancement would provide a more complete perspective.

3. The denoising step in the two-stage process and the choice of regularization parameters lack sufficient analysis or validation, which limits the verification of the model’s feasibility. The techniques in this paper seem to differ little from the work of [1] Song et al. and Li et al. [2], what are the innovations in this paper?




[1] Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.

[2] Li D, Wei C, Li S, et al. Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion[J]. arXiv preprint arXiv:2403.07721, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730597224838

thread_type:
official_review

<thread_object>
creator: reviewer_j8js

<content>
summary:
A method of semantic image reconstructions from EEG recordings in presented.
The EEG singal is transformed using a transformer based network to be aligned with the corresponding image embedding.
The learned EEG embedding is used with pretrained image diffusion models to generate images in a two stage process.

strengths:
Competitive N-WAY classification results.

weaknesses:
The think overall the approach is similar to ATM-S("Visual decoding and reconstruction via eeg embeddings with guided diffusion"), specifically very similar encoder, and overall training procedure, there might be differences in the diffusion models, and image generation.
Given the similarities, I don't see how this work work is better.
The metrics presented are not better in a significant way, i.e. the difference is less than variability between subjects, an standard error or a statistical test will indicate that.
Further more for the classification tasks for high N-way (above 10) the results are worse (than ATM-S ), as indicated by the results in the paper.
In the ATM-S work there are also retrieval evaluation and pixel based evaluation, that lack here.
I don't think the papers convinces that this work is better or different in a significant way from ATM-S.

- No qualitative comparisons to other methods 
- Not sure why the new proposed metric "CAT" makes sense, classification is already semantic and not pixel based, additionally the difference all the model is insignificant(in a statistical test sense)
- Unclear specification for the encoder, which prevents reproduction of results (will be elaborated in questions)

questions:
The diagram and accompanying text for the encoder is unclear:
  - What is Spatial-temoral conv and Temporal-spatial conv and how would they operate on tokens.
  - What does the cross attention block do? Usually cross attention is applied on 2 different sets of representation, in the diagram there is only one.
  - EEG data has a time component how is this handled?

-Given the small amount of test data it would make sense to add error indication/ statistical tests.
 
-I think that displaying the reconstructions from the clip embedding is misleading and should be removed.

line 480-482: "we are currently unable to assess whether the brainwave data recorded from the subjects accurately
captures the complete information of the original images, as the subjects might have been distracted
and thinking about other things during the recording"
line 374: "we cannot guarantee that the subject’s thoughts during EEG
recording perfectly align with the ground truth image"
I think this undermines the whole experiment.
I would expected a significant part of the information in the EEG recordings to come from the visual cortex, that shouldn't be effected by subject's thoughts.

- It would make sense to provide qualitative comparisons to the other methods side by side.
 - Regarding CAT score a much simpler and more straight forward approach would be to do coarse level classification. (for example: animals, plants, inanimate, ...)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666546674

thread_type:
official_review

<thread_object>
creator: reviewer_jtpm

<content>
summary:
The paper introduces an approach for generating images from EEG signals. 
The proposed approach, NECOMIMI, leverages EEG for image generation by using a pre-trained Diffusion model. 
The EEG encoder is trained via contrastive loss by aligning EEG with image embeddings, enabling zero-shot classification.
To evaluate the quality of the generated images, the paper introduces a new metric, the Category-based Assessment Table (CAT) Score, specifically designed to assess EEG-to-image generation based on semantic information. 
Experimental results are provided for both classification and generative tasks.

strengths:
**Significance** The paper deals with a very interesting topic of image generation from EEG recordings.

**Clarity** The content of the paper is written coherently. The flow of the text is easy to comprehend and follow.

weaknesses:
**Originality and Contribution** The originality of the work is limited. Previous work has explored a similar approach for image generation from EEG [1] and fMRI [2]. The contribution of this work is not defined well enough. The experimental results need to include other datasets. For example, the datasets used in [1] could enable a better comparison with [1]. 

It is expected that the generated images will differ substantially from the ground truth, as the EEG recordings cannot capture the details present in the images Gifford et al. (2022). Additional experiments are needed to evaluate the image generation process. For example, are there any differences across the generated images across the image object categories the EEG stimuli belong? One would need to quantify the effect of the EEG signals for the image generation.

**Quality** It is not explained how the proposed CAT score is defined. Furthermore, it is not clear how the five tags are created.

The References section contains a substantial part of references that point to *arxiv.org*. Though some of *arxiv* papers are published in highly respected venues, it makes it cumbersome to navigate the related work. Some references are repeated; for example, Spampinato et al., Radford et al.

Table 1 mentions top-1 and top-5. However, in the table only one score is reported. It is unclear which one.

The sentence (LINE 047) containing 'EEG is one of the most ancient techniques' makes it sound unusual.

[1] Bai, Y., Wang, X., Cao, YP., Ge, Y., Yuan, C., Shan, Y. (2025). DreamDiffusion: High-Quality EEG-to-Image Generation with Temporal Masked Signal Modeling and CLIP Alignment. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15089. Springer, Cham.

[2] Paul S. Scotti et al., "Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors". In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, 2023, 24705–24728.

questions:
1. Why is the batch size set to 1024? Was it determined via hyper-parameter search?
2. How is the CAT score defined?
3. How are exactly the image tags extracted?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704353812

thread_type:
official_review

<thread_object>
creator: reviewer_kwkh

<content>
summary:
This paper develops another method for EEG to image reconstruction.
It develops a new EEG encoder and then uses an InfoNCE loss to learn
MLP projectors from a pre-trained image encoder and the EEG encoder.
The EEG	encoding + MLP projector is used for zero-shot classification
through	comparing encodings of EEG to the encodings of the images.
The EEG encoding is also used to generate reconstructed images through
a 2-stage diffusion process.

strengths:
The top-1 and top-5 classification accuracies for 2-way, 4-way and
200-way classification appear competitive with competitor algorithms.

weaknesses:
- The clarity of the paper could be improved.   It is hard to see what the new innovations are over the related work.

- The paper claims to introduce a new measure (Category-Based Assessment Table (CAT)Score) but it is
not fully described.  "each image is manually labeled with two tags for broad categories - one for a specific category and one for background content,resulting in a total of five tags per image". I don't see how you get five tags per image?    What does manually annotate mean?
The appendix says that all the category-based labels are generated by ChatGPT-4o with the prompt "5 one-word descriptions of the image, ranging from high level to low level".
Finally how are these matched (What kind of consideration is given to synonyms)?

- The reconstructions in the Appendix using the NERV (authors') encoder do not look anywhere near as good as the reconstructions shown in Figure 1. (see Questions)

- The reconstruction results as shown in the Appendix are visually inferior to those from the ATM-S paper as well as those from [Fei & de Sa 2024] Image Reconstruction from Electroencephalography Using Latent Diffusion ( https://arxiv.org/html/2404.01250v1 ).

questions:
From the Diagrams in Figure 2, it appears that the learned MLP projector after the EEG encoder that is used to better align images and EEG is not used in the image generation stage.  Could you please explain if this is just an error in the diagram or why it is not used?

Please explain how the CAT score is computed?   (see weaknesses above)

The reconstructions in the Appendix using the NERV (authors') encoder do not look anywhere near as good as the reconstructions shown in Figure 1.   Please explain.

It is stated that "The final results are averaged from the best outcomes of 5 random seed training sessions".  What does that mean?  Was best determined on the test data?

It is also stated that "the NERV model obtains the best results with 5 multi-heads...".   Was this determined using  test data or the training data?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper has significant weaknesses, including unclear descriptions of the proposed CAT score and insufficient methodological detail, which hinder reproducibility. Reviewers noted limited originality, with the approach closely resembling prior work like ATM-S, while failing to show significant improvements in results. The experiments lack breadth and statistical rigor, and the reconstructions are visually inferior to competing methods. Insufficient author responses and lack of qualitative comparisons further suggest the work is not ready for acceptance.