NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models

## Abstract

Abstract NECOMIMI ( NE ural- CO gnitive M ult I modal EEG-Infor M ed I mage Generation with Diffusion Models) introduces a novel framework for generating images directly from EEG signals using advanced diffusion models. Unlike previous works that focused solely on EEG-image classification through contrastive learning, NECOMIMI extends this task to image generation. The proposed NERV EEG encoder demonstrates state-of-the-art (SoTA) performance across multiple zero-shot classification tasks, including 2-way, 4-way, and 200-way, and achieves top results in our newly proposed Category-based Assessment Table (CAT) Score, which evaluates the quality of EEG-generated images based on semantic concepts. A key discovery of this work is that the model tends to generate abstract or generalized images, such as landscapes, rather than specific objects, highlighting the inherent challenges of translating noisy and low-resolution EEG data into detailed visual outputs. Additionally, we introduce the CAT Score as a new metric tailored for EEG-to-image evaluation and establish a benchmark on the ThingsEEG dataset. This study underscores the potential of EEG-to-image generation while revealing the complexities and challenges that remain in bridging neural activity with visual representation.

## 1 Introduction

Electroencephalography (EEG) is one of the most ancient techniques used to measure neuronal activity in the human brain [, ]. Its application has significant value in clinical practice, particularly in diagnosing epilepsy [], depression [] and sleep disorders [], as well as in assessing dysfunctions in sensory transmission pathways [] and more []. Historically, the analysis of EEG signals was limited to visual inspection of amplitude and frequency changes over time. However, with advancements in digital technology, the methodology has evolved significantly, shifting towards a more comprehensive analysis of the temporal and spatial characteristics of these signals []. As a result of this evolution, EEG has gained recognition as a potent tool for capturing brain functions in real-time, particularly in the sub-second range. Despite its advantages, EEG has traditionally suffered from poor spatial resolution, making it challenging to pinpoint the precise brain areas responsible for the measured neuronal activity at the scalp []. In recent years, there has been a surge of interest in utilizing EEG for more sophisticated applications, such as image recognition and reconstruction []. These advancements have led to significant improvements in the accuracy of image recognition tasks, underscoring the potential of EEG as a bridge between neural activity and visual representation [, ]. The growing interest in using EEG for image recognition is rooted in its ability to capture the temporal dynamics of neuronal activity, though its spatial resolution remains a challenge. Innovative methodologies, including deep learning techniques and generative models like Generative Adversarial Networks (GANs) [] and diffusion models [], have enhanced the accuracy and effectiveness of EEG-based systems, allowing for the generation of photorealistic images based on neural signals [, , ]. Notably, studies have demonstrated the feasibility of decoding natural images from EEG signals, employing innovative frameworks that align EEG responses with paired image stimuli []. However, most of the current works claiming to be EEG-to-image are essentially still image-to-image in nature, with EEG information primarily used to slightly guide the transformation of the input image by adding noise [, , , ]. In order to achieve a truly meaningful EEG-to-image generation, this work, named NECOMIMI (NEural-COgnitive MultImodal eeg-inforMed Image generation with diffusion models), introduces an innovative framework focused on EEG-based image generation, combining advanced diffusion model techniques.

This paper presents several key innovations as follows:

- •
We propose a novel EEG encoder, NERV, which achieves state-of-the-art performance in multimodal contrastive learning tasks.
- •
Unlike previous work that primarily focused on image-to-image generation with EEG features as guidance, we introduce a comprehensive two-stage EEG-to-image multimodal generative framework. This not only extends prior contrastive learning between EEG and images but also applies it to image generation.
- •
To address the conceptual differences between EEG-to-image and traditional text-to-image tasks, we propose a new quantification method, the Category-based Assessment Table (CAT) Score, which evaluates image generation performance based on semantic concepts rather than image distribution.
- •
We establish a CAT score benchmark standard using Vision Language Model (VLM) on the ThingsEEG dataset.
- •
Additionally, we uncover some notable findings and phenomena regarding the EEG-to-image generation process.

## 2 Related work

### 2.1 The potential of EEG data

In a typical experiment studying brain responses related to visual processes, a person looks at a series of images while a brain scanner or recording device captures their brain signals for analysis. There are various non-invasive methods to capture these brain responses, like fMRI, EEG, and MEG, each with different sensitivity levels. However, we still don’t fully understand what this data really means, and even more importantly, how to interpret it. In a pioneering study [], the researchers tried to generate impressions of what the subjects saw using fMRI images, based on a large image dataset taken from YouTube. However, this method has challenges, like the complexity and high cost of using an fMRI scanner. To overcome these drawbacks, a lot of research has shifted to using electrophysiological responses, particularly EEG, which has lower spatial resolution than most other methods but much higher temporal resolution. EEG recordings are also cheaper and easier to conduct, but the data is often noisy and affected by external factors, making it harder to reconstruct the original stimulus. Most image recognition and/or generation from brain signals nowadays is done using fMRI data [], while EEG, being noisier, is used much less often.

### 2.2 Using EEG information on image generation and reconstruction

Building on this shift towards EEG, prior to efforts in generating images directly from brain data, the concept of using EEG signals for image classification was introduced by the study []. This work first demonstrated the feasibility of decoding visual categories from EEG recordings using deep learning models, setting a foundation for leveraging neural signals in image-related tasks. However, the dataset they used was relatively small, which limited the generalization of their findings. Further advancements in generative models, specifically with the introduction of Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN), opened new possibilities for image generation. The VAE model proposed by [, ] achieved data generation and reconstruction by learning the latent distribution of data. The GAN model introduced by [] utilized adversarial training between a generator and a discriminator to produce highly realistic images. Building on these methods, Brain2Image [] was the first to use VAE to guide image generation from EEG features. Following that, EEG-GAN [] presented the first EEG-based image generation model, using LSTM [] to extract EEG information and guide the GAN for image generation. After this, there were still many EEG-to-image works based on GAN that emerged, with most of them focusing on improving the GAN architecture and the way it interacts with the EEG encoder, like in ThoughtViz [], VG-GAN-VC [], BrainMedia [], and EEG2IMAGE [], etc. However, in all these works, a common and challenging problem is figuring out how to effectively use EEG data to guide image generation and reconstruction.
This challenge of training neural networks to align multimodal information wasn’t effectively addressed until the emergence of CLIP [], which provided a much better solution. Since then, some works have also applied this approach to EEG-based image generation.

### 2.3 Contrastive learning-based works on EEG-image tasks

To the best of our knowledge, EEGCLIP [] was the first to use contrastive learning to align EEG and image data. However, in this work, this aspect was only an exploratory attempt and did not further utilize the framework for downstream tasks like zero-shot image recognition. The next challenge lies in designing a better EEG encoder for contrastive learning, based on the rich image embeddings extracted from a CLIP-based image pre-trained encoder. Some recent works have explored this direction, such as NICE [], MUSE [], ATM [], and []. Some researchers have even attempted quantum-classical hybrid computing and quantum EEG encoder [] to perform quantum contrastive learning []. Most current works focus on tackling zero-shot classification, where the model is tested on unseen both EEG data and images that it hasn’t encountered during training. The goal is to compute similarity scores for image recognition, aiming to enhance the model’s generalization performance on out-of-sample data. As contrastive learning architectures for EEG-based image recognition mature, and inspired by test-to-image frameworks in other generative fields, the invention of diffusion models has addressed the instability issues associated with previous GAN-based generation methods to some extent. While there are already EEG-based image reconstruction efforts using diffusion models, such as NeuroVision [], DreamDiffusion [], DM-RE2I [], BrainViz [], NeuroImagen [], and EEGVision [], most of these works still largely rely on image-based features, with EEG data serving as supplementary information for the diffusion process. While these methods have made significant strides in computer vision, they primarily rely on images as input and are not designed to process non-visual signals like EEG directly. Currently, models designed specifically for direct generation tasks using pure EEG features or embeddings, where EEG functions similarly to a prompt command, are still quite rare. This work seeks to introduce a flexible, plug-and-play architecture: NECOMIMI, which not only expands upon previous recognition-focused approaches but also extends them into EEG-to-image generation tasks based on modern diffusion models.

## 3 Methodology

### 3.1 Overview

Figure: Figure 2: The figure illustrates the entire workflow of the EEG-based image generation model.
Refer to caption: extracted/5899447/img/necomimi_abs1.png

This chapter provides a detailed overview of an advanced EEG-to-image generation model utilizing deep learning techniques and diffusion models. While the framework includes a one-stage image generation phase, we found that its performance was suboptimal. Consequently, the model is primarily designed as a two-stage process, which will be discussed in detail in later sections. The overall structure consists of four phases: the training phase, zero-shot testing, one-stage image generation, and two-stage image generation, each contributing to the transformation of raw EEG data into meaningful visual outputs.

### 3.2 Training Phase

In the initial training phase, both visual image $\in\mathbb{R}^{h\times w\times ch}$ and EEG data $\in\mathbb{R}^{e\times d}$ are processed in parallel to establish a shared embedding space, where $h$ is the height of the image, $w$ is the width of the image, $ch$ is the number of channels (e.g., RGB channels), $e$ is the number of electrodes (channels), and $d$ is the number of data points (time samples). Training set images are first passed through a pre-trained image encoder, which transforms the images into latent representations called image embeddings $I$. In this work, we use a pretrained Vision Transformer (ViT) [] from CLIP model [] as the image encoder, which outputs embeddings of size $\mathbb{R}^{1\times 1024}$ for each image. Simultaneously, the EEG signals from the corresponding sessions are processed by a custom EEG encoder to produce EEG embeddings $E$. As for the EEG encoder, in this work, we extended several existing works like NICE [], MUSE [], Nervformer [] and ATM [] to enable EEG-to-image capabilities. Additionally, we proposed a new EEG encoder, NERV, which is specifically designed for noisy, multi-channel time series data like EEG, based on a multi-attention mechanism.

These embeddings are projected into a unified space via an MLP Projector, where they are trained using the InfoNCE loss. This contrastive learning loss function ensures that corresponding image and EEG embeddings are aligned in the latent space, enhancing the model’s ability to understand and link neural patterns to visual stimuli. Standard contrastive learning employs the InfoNCE loss as defined by [, , ]:

$$ $\mathcal{L}_{InfoNCE}=-\mathbb{E}\left[\log\frac{\exp(S_{E,I}/\tau)}{\sum_{k=1 }^{N}\exp(S_{E,I_{k}}/\tau)}\right]$ (1) $$

where the $S_{E,I}$ represents the similarity score between the EEG embeddings $E$, and the paired image embeddings $I$, and the $\tau$ is learned temperature parameter.

### 3.3 Zero-shot Testing Phase

Once trained, the model enters the zero-shot testing phase. This phase focuses on evaluating the model’s ability to generalize to unseen data. Here, the EEG signals and images from the test set are encoded using the pre-trained encoders, and their respective embeddings are projected through the MLP Projector. The testing groups are separated into multiple divisions—2-way, 4-way, 10-way, 50-way, 100-way and beyond—allowing for a structured comparison between the EEG and image embeddings. The final similarity scores between embeddings determine the model’s classification accuracy, enabling the assessment of how well the model understands new EEG data without additional training.

### 3.4 One-stage Image Generation

In the one-stage image generation process, the EEG embeddings from the testing set are directly used as inputs to reconstruct images. By incorporating the IP-Adapter [], which was originally designed to use images as prompts, due to its compact design, enhances image prompt flexibility within pre-trained text-to-image models. We adapt it in this work as a means to transform EEG embeddings into "feature prompts" for the image generation process.
The conditioned embeddings are then processed by the Stable Diffusion XL-Turbo model [, ], a faster version of Stable Diffusion XL designed for rapid image synthesis, which reconstructs the final images based on the input EEG data. This method offers a streamlined approach to EEG-based image generation, relying on a single transformation stage to produce meaningful visual outputs from neural signals. The start of the EEG-conditioned diffusion phase is critical for generating images based on EEG data. This phase uses a classifier-free guidance method, which pairs CLIP embeddings and EEG embeddings $(I,E)$. By applying advanced generative techniques, the diffusion process is adapted to use the EEG embedding $E$ to model the distribution of the CLIP embeddings $p(I|E)$. The CLIP embedding $I$, generated during this stage, lays the foundation for the next phase of image generation. The architecture integrates a simplified U-Net model, represented as $\epsilon_{\text{prior}}(I^{t},t,E)$, where $I^{t}$ is the noisy CLIP embedding at a specific diffusion step $t$.

The classifier-free guidance method helps refine the diffusion model (DM) using a specific EEG condition $E$. This approach synchronizes the outputs of both a conditional and an unconditional model. The final model equation is expressed as:

$$ $\epsilon_{\text{prior}}^{w}(I^{t},t,E)=(1+w)\epsilon_{\text{prior}}(I^{t},t,E) -w\epsilon_{\text{prior}}(I^{t},t),$ (2) $$

where $w\geq 0$ controls the guidance scale. This technique allows for training both the conditional and unconditional models within the same network, periodically replacing the EEG embedding $E$ with a null value to enhance training variation (about 10% of the data points). The main goal is to improve the quality of generated images while maintaining diversity.

However, we were surprised to find that when using EEG embeddings directly as prompts for the diffusion model, the generated images mostly turned out to be landscapes, regardless of the category. We will discuss the detailed results in later sections. As a result, we attempted a 2-stage approach for image generation.

### 3.5 Two-stage Image Generation

The prior diffusion stage plays a crucial role in generating an intermediate representation [], such as a CLIP image embedding, from a text caption []. This representation is then used by the diffusion decoder to produce the final image. This two-stage process enhances image diversity, maintains photorealism, and allows for efficient and controlled image generation []. The two-stage image generation process introduces a more complex and refined method of synthesizing images from EEG data. In this approach, the EEG embeddings are first processed by a Diffusion U-Net, which applies additional transformations to enhance the representation of the neural data. After passing through the U-Net, the modified EEG embeddings are fed into the Stable Diffusion XL-Turbo model, with the assistance of the IP-Adaptor. This two-step transformation ensures a more nuanced generation process, potentially leading to higher-quality images by incorporating deeper layers of refinement.
The first step of stage-1 is training the prior diffusion model. The main purpose of training is to let the model learn how to recover the original embedding from noisy embeddings. The specific steps are as follows: (a) Randomly replace conditional EEG embeddings $c_{\text{emb}}$ with None with a 10% probability:

$$ $c_{\text{emb}}=\text{None},\quad\text{if }\text{random}()<0.1$ (3) $$

(b) Add random noise to the target embedding $h_{\text{emb}}$, perturb it using the scheduler at a timestep $t$, use the symbol $\mathcal{S}_{add_noise}$ to represent the scheduler add noise function:

$$ $\hat{h}_{\text{emb}}(t)=\mathcal{S}_{add_noise}(h_{\text{emb}},\epsilon,t)$ (4) $$

where $\epsilon\sim\mathcal{N}(0,I)$ is the random noise, and $t$ is a randomly sampled timestep.
(c) The model receives the perturbed embedding $\hat{h}_{\text{emb}}(t)$ and conditional embedding $c_{\text{emb}}$, and predicts the noise. Use the symbol $\mathcal{D}_{\text{prior}}$ to represent the diffusion prior function:

$$ $\epsilon_{\text{pred}}=\mathcal{D}_{\text{prior}}(\hat{h}_{\text{emb}}(t),t,c_ {\text{emb}})$ (5) $$

(d) Compute the loss using Mean Squared Error (MSE) between the predicted noise and the actual noise:

$$ $L=\frac{1}{N}\sum_{i=1}^{N}\left(\epsilon_{\text{pred}}^{(i)}-\epsilon^{(i)} \right)^{2}$ (6) $$

(e) Perform backpropagation on the loss $L$, and update the model parameters using the optimizer:

$$ $\theta\leftarrow\theta-\eta\nabla_{\theta}L$ (7) $$

where $\eta$ is the learning rate and $\theta$ represents the model’s parameters.

The last step of stage-1 is generation process. The main purpose of the generation process is to gradually denoise and generate the final embedding based on the conditional EEG embedding $c_{\text{emb}}$, starting from random noise. The specific steps are as follows: (a) Generate a sequence of timesteps $t$, which will be used for the denoising process, define $\mathcal{T}=\{t_{1},t_{2},\dots,t_{T}\}$ to represent the set of time steps sampled from the total steps $T$:

$$ $\{t_{1},t_{2},\dots,t_{T}\}\sim\mathcal{T}(T)$ (8) $$

where $T$ is the total number of denoising steps.
(b) Initialize random noise embedding $h_{T}$, which serves as the starting point for the generation process:

$$ $h_{T}\sim\mathcal{N}(0,I)$ (9) $$

(c) Starting from timestep $T$, iteratively apply the model to predict noise and denoise the embedding until $t=0$. Each step depends on the conditional embedding $c_{\text{emb}}$:

If using conditional embedding, perform both unconditional and conditional noise prediction at each step:

$$ $\epsilon_{\text{pred_cond}}=\mathcal{D}_{\text{prior}}(h_{t},t,c_{\text{emb}})$ (10) $$

$$ $\epsilon_{\text{pred_uncond}}=\mathcal{D}_{\text{prior}}(h_{t},t)$ (11) $$

Then combine the results using classifier-free guidance, define $\alpha_{\text{guide}}$ as the guidance scale:

$$ $\epsilon_{\text{pred}}=\epsilon_{\text{pred_uncond}}+\alpha_{\text{guide}} \times(\epsilon_{\text{pred_cond}}-\epsilon_{\text{pred_uncond}})$ (12) $$

Finally, update the noisy embedding based on the predicted noise, use the symbol $\mathcal{S}_{step}$ to represent the scheduler step function:

$$ $h_{t-1}=\mathcal{S}_{step}(\epsilon_{\text{pred}},t,h_{t})$ (13) $$

(d) After the denoising process is complete, $h_{output}$ represents the final generated embedding of a EEG, which is the model’s output:

$$ $h_{output}=h_{\text{generated}}\in\mathbb{R}^{1\times 1024}$ (14) $$

The stage-2 is input the $h_{output}$ into the IP-adaptor as a prompt to generate the image by Stable Diffusion XL-Turbo model.

### 3.6 NERV EEG Encoder

Figure: Figure 3: This diagram shows the overall structure and workflow of the NERV EEG encoder model.
Refer to caption: extracted/5899447/img/necomimi-nervformerv2.png

This diagram 3 illustrates the structure of NERV, a neural network encoder designed for EEG signal processing. The workflow starts with a linear projection of the flattened EEG nodes, followed by position encoding to retain temporal information. EEG signals pass through a Transformer layer and undergo instance normalization. The model then applies both spatial-temporal convolution (blue) to extract spatial features followed by temporal features and temporal-spatial convolution (yellow) to extract temporal features first, then spatial features. Multi-head self-attention mechanisms are applied to both feature sets, followed by layer normalization and residual connections. The cross-attention block (red) fuses the temporal and spatial features, which are further processed by a feed-forward layer before final output. The class token, position embeddings, and patch tokens are all part of the input sequence processed through these steps, ultimately yielding the output features for EEG-based tasks.

### 3.7 Category-based Assessment Table (CAT) Score

Unlike traditional image-to-image or text-to-image models driven by image representations, EEG-to-image models face unique challenges. In the current NECOMIMI architecture, the model can only capture broad semantic information from EEG signals rather than fine-grained details. For example, suppose the ground truth EEG data was recorded while a subject was observing an aircraft carrier. When using Model A as the EEG encoder in NECOMIMI, the generated image is a jet, while using Model B results in an image of a sheep. To objectively assess performance, we need a standard that scores Model A higher than Model B in such cases.

Why not use existing evaluation metrics? Traditional metrics like Structural Similarity Index (SSIM) [] measure structural similarity between the ground truth and generated image, while the Inception Score (IS) [] and Fréchet Inception Distance (FID) [] focus on the accuracy of image categories and its distribution. However, EEG captures more abstract semantic information, and we cannot guarantee that the subject’s thoughts during EEG recording perfectly align with the ground truth image. This makes traditional evaluation methods unfair for EEG-to-image tasks.

To address this, we propose the Category-Based Assessment Table (CAT) Score, a new metric specifically designed for EEG-to-image evaluation. In the ThingsEEG test dataset (which contains 200 categories with one image per category), each image is manually labeled with two tags for broad categories, one for a specific category, and one for background content, resulting in a total of five tags per image. We extracted the tags by ChatGPT-4o []. The entire test dataset thus comprises 200 images × 5 tags = 1,000 points. Using manual annotation, we can determine whether the categories of generated images match these labels, providing a fair assessment for EEG-to-image models. For more details on the ThingsEEG categories, please refer to the appendix.

## 4 Experiments

### 4.1 Datasets and Preprocessing

The ThingsEEG dataset [] consists of a large set of EEG recordings obtained through a rapid serial visual presentation (RSVP) paradigm. The responses were collected from 10 participants who viewed a total of 16,740 natural images from the THINGS database []. The dataset contains 1654 training categories, each with 10 images, and 200 test categories, each with a single image. The EEG data were recorded using 64-channel EASYCAP equipment, and preprocessing involved segmenting the data into trials from 0 to 1000 ms after the stimulus was shown, with baseline correction based on the pre-stimulus period. EEG responses for each image were averaged over multiple repetitions.

### 4.2 Experiment Details

Due to the significant impact that different versions of the CLIP package can have on the results of contrastive learning, this work ensures a fair comparison of various EEG encoders by rerunning all experiments using a unified CLIP-ViT environment, where available open-source code (e.g., [](^1^11[https://github.com/eeyhsong/NICE-EEG](https://github.com/eeyhsong/NICE-EEG)), [](^2^22[https://github.com/ChiShengChen/MUSE_EEG](https://github.com/ChiShengChen/MUSE_EEG)), [](^3^33[https://github.com/dongyangli-del/EEG_Image_decode](https://github.com/dongyangli-del/EEG_Image_decode))) was utilized. Another factor that can influence contrastive learning is batch size. Therefore, all experiments in this work were conducted with a batch size of 1024. The final results are averaged from the best outcomes of 5 random seed training sessions, each running for 200 epochs. We employ the AdamW optimizer, setting the learning rate to 0.0002 and parameters $\beta_{1}$=0.5 and $\beta_{2}$=0.999. The $\tau$ in contrastive learning initialized with $log(1/0.07)$. The NERV model achieves the best results with 5 multi-heads, while the Transformer layer has 1 multi-head and the cross-attention layer has 8 multi-heads. The time step is 50 in diffusion model. All experiments, including both EEG encoder training and prior diffusion model processing, were performed on a machine equipped with an A100 GPU.

### 4.3 Classification Results

In Table 1, the classification accuracy for both 2-way and 4-way zero-shot tasks is evaluated across ten subjects. Our new model NERV consistently achieves the best performance, particularly excelling in the 2-way classification task, where it maintains top accuracy across most subjects. It achieves an average accuracy of 94.8% in the 2-way classification and 86.8% in the 4-way classification, outperforming other methods like NICE [], MUSE [], and ATM-S []. While NICE and MUSE perform strongly in some subjects, they often fall short of NERV’s performance. NICE has an average of 91.3% in the 2-way task and 81.3% in the 4-way task, with MUSE trailing behind with averages of 92.2% (2-way) and 82.8% (4-way). ATM-S performs comparably to NICE and MUSE in some subjects but falls short on average with 86.5% in the 4-way classification. In Table 2, the results for the more challenging 200-way zero-shot classification task show that NERV also performs the best, especially in the top-1 accuracy. ATM-S and NERV perform similarly, but NERV shows stronger performance in most subjects. NERV achieves an average top-1 accuracy of 27.9% and top-5 accuracy of 54.7%, leading over all other methods. In contrast, Nervformer [] and BraVL [] show weaker performance, especially in the top-1 accuracy, where they average 19.8% and 5.8%, respectively. For the results of other 10-way, 50-way, and 100-way zero-shot classifications, please refer to the appendix. In summary, NERV consistently outperforms its competitors in both tasks, demonstrating the strongest zero-shot classification capability, particularly when distinguishing between a large number of categories, making it the most effective model in these experiments.

**Table 1: Overall accuracy (%) of 2-way and 4-way zero-shot classification using CLIP-ViT as image encoder: top-1 and top-5. The parts in bold represent the best results, while the underlined parts are the second best.**
|  | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Subject 5 | Subject 6 | Subject 7 | Subject 8 | Subject 9 | Subject 10 | Ave |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way | 2-way | 4-way |
| Subject dependent - train and test on one subject |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Nervformer | 89.9 | 76.9 | 91.3 | 80.7 | 91.6 | 80.8 | 94.3 | 85.9 | 86.3 | 70.4 | 91.1 | 82.5 | 92.5 | 81.6 | 96.2 | 88.3 | 92.0 | 83.7 | 92.4 | 83.1 | 91.8 | 81.4 |
| NICE | 91.7 | 80.4 | 89.8 | 77.4 | 93.5 | 83.7 | 94.0 | 84.9 | 85.9 | 70.3 | 89.1 | 81.7 | 91.2 | 81.7 | 95.8 | 89.2 | 87.9 | 76.5 | 93.8 | 87.1 | 91.3 | 81.3 |
| MUSE | 90.1 | 78.4 | 90.3 | 76.8 | 93.4 | 85.6 | 93.6 | 87.5 | 88.3 | 74.2 | 93.1 | 85.3 | 93.1 | 82.8 | 95.4 | 87.7 | 90.5 | 81.8 | 94.4 | 88.1 | 92.2 | 82.8 |
| ATM-S | 94.8 | 84.9 | 93.5 | 86.3 | 95.3 | 89.0 | 95.9 | 87.3 | 90.8 | 78.5 | 94.1 | 85.2 | 94.2 | 87.1 | 96.6 | 92.9 | 94.1 | 86.8 | 94.7 | 87.0 | 94.4 | 86.5 |
| NERV (ours) | 95.3 | 85.7 | 96.0 | 88.8 | 95.9 | 91.2 | 95.8 | 87.4 | 90.8 | 80.4 | 93.6 | 84.0 | 94.7 | 86.2 | 96.8 | 92.3 | 94.4 | 84.2 | 94.8 | 87.6 | 94.8 | 86.8 |

**Table 2: Overall accuracy (%) of 200-way zero-shot classification using CLIP-ViT as image encoder: top-1 and top-5. The parts in bold represent the best results, while the underlined parts are the second best.**
|  | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Subject 5 | Subject 6 | Subject 7 | Subject 8 | Subject 9 | Subject 10 | Ave |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 |
| Subject dependent - train and test on one subject |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| BraVL | 6.1 | 17.9 | 4.9 | 14.9 | 5.6 | 17.4 | 5.0 | 15.1 | 4.0 | 13.4 | 6.0 | 18.2 | 6.5 | 20.4 | 8.8 | 23.7 | 4.3 | 14.0 | 7.0 | 19.7 | 5.8 | 17.5 |
| Nervformer | 15.0 | 36.7 | 15.6 | 40.0 | 19.7 | 44.9 | 23.3 | 54.4 | 13.0 | 29.1 | 18.9 | 42.2 | 19.5 | 42.0 | 30.3 | 60.0 | 20.1 | 46.3 | 22.9 | 47.1 | 19.8 | 44.3 |
| NICE | 19.3 | 44.8 | 15.2 | 38.2 | 23.9 | 51.4 | 24.1 | 51.6 | 11.0 | 30.7 | 18.5 | 43.8 | 21.0 | 47.9 | 32.5 | 63.5 | 18.2 | 42.4 | 27.4 | 57.1 | 21.1 | 47.1 |
| MUSE | 19.8 | 41.1 | 15.3 | 34.2 | 24.7 | 52.6 | 24.7 | 52.6 | 12.1 | 33.7 | 22.1 | 51.9 | 21.0 | 48.6 | 33.2 | 59.9 | 19.1 | 43.0 | 25.0 | 55.2 | 21.7 | 47.3 |
| ATM-S | 25.8 | 54.1 | 24.6 | 52.6 | 28.4 | 62.9 | 25.9 | 57.8 | 16.2 | 41.9 | 21.2 | 53.0 | 25.9 | 57.2 | 37.9 | 71.1 | 26.0 | 53.9 | 30.0 | 60.9 | 26.2 | 56.5 |
| NERV (ours) | 25.4 | 51.2 | 24.1 | 51.1 | 28.6 | 53.9 | 30.0 | 58.4 | 19.3 | 43.9 | 24.9 | 52.3 | 26.1 | 51.6 | 40.8 | 67.4 | 27.0 | 55.2 | 32.3 | 61.6 | 27.9 | 54.7 |

### 4.4 Performance Comparison of different generative models

Here, we introduce our newly proposed CAT Score method, which quantifies and evaluates the quality of EEG-generated images based on semantic concepts rather than pixel structure. Detailed CAT Score labels can be found in the appendix. To our surprise, while our proposed NERV method achieved SoTA on the CAT Score, no EEG encoder has surpassed a score of 500 in this evaluation out of a possible 1000 points. This highlights both the rigor of the CAT Score and the challenging nature of the pure EEG-to-Image task.

**Table 3: Overall CAT score $\times 1000$ of NECOMIMI EEG-to-Image generation with several EEG encoders.**
|  | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Subject 5 | Subject 6 | Subject 7 | Subject 8 | Subject 9 | Subject 10 | Ave |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EEG Encoder |  |  |  |  |  | CAT Score |  |  |  |  |  |
| Nervformer | 432 | 457 | 429 | 454 | 475 | 463 | 404 | 438 | 427 | 410 | 438.9 |
| NICE | 426 | 456 | 445 | 447 | 411 | 454 | 438 | 443 | 426 | 429 | 437.5 |
| MUSE | 438 | 456 | 434 | 416 | 426 | 463 | 443 | 437 | 410 | 468 | 439.1 |
| ATM-S | 413 | 419 | 411 | 464 | 427 | 469 | 442 | 472 | 431 | 445 | 439.3 |
| NERV (ours) | 445 | 436 | 432 | 456 | 438 | 466 | 410 | 437 | 433 | 444 | 439.7 |

### 4.5 Findings in EEG-to-Image

Figure: Figure 4: The image illustrates the progression of visual representations generated using different embedding techniques in a diffusion model: (a) Top row: The original images shown to subjects (ground truth). (b) Second row: Images generated by the CLIP-ViT embeddings of the original images. (c) Third row: Images generated by one-stage method using pure EEG embeddings with NERV EEG encoder. (d) Fourth row: Images generated by two-stage NECOMIMI method using pure EEG embeddings with NERV EEG encoder.
Refer to caption: extracted/5899447/img/gen_img.jpg

We have observed some interesting findings from the pure EEG-to-Image process. As shown in the third row of Figure 4, the images generated by the diffusion model from embeddings compressed from EEG signals mainly consist of landscapes, which differ significantly from the original images (ground truth). Several factors may contribute to this phenomenon. For example, EEG signals are a high-noise, low-resolution form of data, capturing only certain aspects of brain activity. Moreover, we are currently unable to assess whether the brainwave data recorded from the subjects accurately captures the complete information of the original images, as the subjects might have been distracted and thinking about other things during the recording. This makes it difficult for the embeddings extracted from EEG signals to capture sufficient details, particularly when it comes to high-resolution object recognition (such as cats or specific items). As a result, the model tends to generate relatively vague or abstract images, like landscapes. Alternatively, the EEG signals may reflect higher-level abstract concepts or emotions associated with viewing the images rather than concrete objects or details. Since these abstract concepts are often related to the scene, background, or the brain’s broad perception of the environment, the model is more likely to generate abstract or general images, such as landscapes, instead of specific objects.

Additionally, the training of the model on EEG signals may still be insufficient. The diffusion model may not yet fully understand and generate images from EEG signals, especially when it lacks enough data or optimization to map EEG signals to specific visual information. As a result, the model might more easily generate the types of images it is "accustomed" to producing, such as landscapes, which may constitute a significant portion of the training data. The gap between the vision modality and the neural modality (EEG) is also substantial. EEG signals may not directly correspond to detailed objects in images, so the model tends to generate "safe options," like landscapes, which may have been more prevalent in the image generation samples during training. This leads to what can be described as "hallucinations." These factors collectively contribute to the significant differences between the images generated from EEG signals and the ground truth, particularly the failure in specific object recognition. This work can be considered a forward-looking exploration, as this field is just beginning to develop.

## 5 Discussion and Conclusion

The NECOMIMI framework expands previous works on EEG-Image contrastive learning classification by enabling image generation, filling a gap in prior research and opening new possibilities for EEG applications. We introduced the SoTA EEG encoder, NERV, which achieved top performance in 2-way, 4-way, and 200-way zero-shot classification tasks, as well as in the CAT Score evaluation, demonstrating its effectiveness in EEG-based generative tasks. A key finding is that the model often generates abstract images, like landscapes, rather than specific objects. This suggests that EEG data, being noisy and low-resolution, captures broad semantic concepts rather than detailed visuals. The gap between neural signals and visual stimuli remains a challenge for precise image generation. We also proposed the CAT Score, a new metric tailored for EEG-to-image generation, and established its benchmark on the ThingsEEG dataset. Surprisingly, we found that EEG encoder performance may not strongly correlate with the quality of generated images, providing new insights into the limitations and challenges of this task. In conclusion, NECOMIMI demonstrates the potential of EEG-to-image generation while highlighting the complexities of translating neural signals into accurate visual representations. Future research should focus on refining models to better capture detailed information from EEG signals.

## Acknowledgment

I would like to thank Aidan Hung-Wen Tsai for brainstorming this fascinating title with me, which originated from a conversation in Silicon Valley about NeuroSky’s cat-ear EEG product, Necomimi. I also appreciate Neuro Industry, Inc. for providing A100 computational resources on GCP. Special thanks to Dr. Guan-Ying Chen for proofreading and providing valuable feedback on revisions.

## Appendix A Appendix

### A.1 More EEG encoder classification performance comparison

**Table 4: Overall accuracy (%) of 10-way zero-shot classification using CLIP-ViT as image encoder: top-1 and top-5.**
|  | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Subject 5 | Subject 6 | Subject 7 | Subject 8 | Subject 9 | Subject 10 | Ave |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | 10-way | 10-way | 10-way | 10-way | 10-way | 10-way | 10-way | 10-way | 10-way | 10-way | 10-way |
| Subject dependent - train and test on one subject |  |  |  |  |  |  |  |  |  |  |  |
| Nervformer | 59.4 | 62.0 | 65.4 | 72.0 | 50.7 | 63.4 | 63.7 | 78.3 | 67.0 | 68.8 | 65.1 |
| NICE | 64.1 | 57.6 | 70.2 | 72.6 | 51.8 | 63.0 | 63.8 | 79.1 | 59.6 | 73.9 | 65.6 |
| MUSE | 61.0 | 56.1 | 70.8 | 71.3 | 55.1 | 70.1 | 66.2 | 76.9 | 62.8 | 73.2 | 66.4 |
| ATM-S | 72.5 | 70.4 | 76.3 | 74.1 | 64.6 | 72.2 | 73.6 | 83.2 | 70.6 | 75.8 | 73.3 |
| NERV (ours) | 72.2 | 74.3 | 75.9 | 76.7 | 62.5 | 71.8 | 70.4 | 81.8 | 70.9 | 73.8 | 73.0 |

**Table 5: Overall accuracy (%) of 50-way zero-shot classification using CLIP-ViT as image encoder: top-1 and top-5.**
|  | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Subject 5 | Subject 6 | Subject 7 | Subject 8 | Subject 9 | Subject 10 | Ave |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 |
| Subject dependent - train and test on one subject |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Nervformer | 28.4 | 66.0 | 32.0 | 71.8 | 37.4 | 73.9 | 44.8 | 81.6 | 24.6 | 57.1 | 33.8 | 74.4 | 33.6 | 69.2 | 49.9 | 87.2 | 36.8 | 75.6 | 38.8 | 76.6 | 36.0 | 73.3 |
| NICE | 36.0 | 72.2 | 30.2 | 66.8 | 43.0 | 77.8 | 44.0 | 80.3 | 24.8 | 58.2 | 35.6 | 70.4 | 36.9 | 72.5 | 53.3 | 86.0 | 34.4 | 65.4 | 45.8 | 82.8 | 38.4 | 73.2 |
| MUSE | 33.9 | 70.9 | 29.9 | 65.7 | 43.6 | 79.4 | 42.8 | 79.8 | 26.1 | 63.4 | 39.8 | 79.4 | 39.8 | 73.3 | 49.8 | 84.2 | 34.4 | 72.7 | 44.5 | 81.1 | 38.5 | 74.9 |
| ATM-S | 45.3 | 78.7 | 44.5 | 80.5 | 49.8 | 85.0 | 46.2 | 83.2 | 33.3 | 69.2 | 42.8 | 81.1 | 47.5 | 80.8 | 59.7 | 91.0 | 45.8 | 79.3 | 50.6 | 82.4 | 46.6 | 81.1 |
| NERV (ours) | 41.1 | 74.8 | 43.2 | 80.5 | 47.9 | 82.8 | 48.1 | 83.5 | 36.4 | 70.7 | 43.0 | 77.6 | 43.5 | 77.3 | 59.2 | 88.4 | 46.1 | 79.4 | 51.0 | 81.7 | 46.0 | 79.7 |

**Table 6: Overall accuracy (%) of 100-way zero-shot classification using CLIP-ViT as image encoder: top-1 and top-5.**
|  | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Subject 5 | Subject 6 | Subject 7 | Subject 8 | Subject 9 | Subject 10 | Ave |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 |
| Subject dependent - train and test on one subject |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Nervformer | 21.0 | 50.8 | 21.6 | 55.1 | 27.6 | 58.5 | 33.0 | 67.8 | 17.0 | 43.4 | 24.7 | 56.2 | 24.5 | 54.8 | 39.8 | 75.6 | 26.8 | 62.3 | 30.2 | 63.6 | 26.6 | 58.8 |
| NICE | 28.0 | 60.5 | 21.8 | 53.2 | 33.1 | 64.2 | 32.2 | 65.9 | 16.8 | 43.9 | 26.0 | 57.6 | 28.0 | 59.0 | 40.7 | 76.0 | 24.5 | 54.5 | 37.2 | 71.0 | 28.8 | 60.6 |
| MUSE | 25.4 | 56.7 | 21.2 | 49.8 | 33.9 | 67.6 | 32.2 | 65.7 | 18.0 | 49.6 | 30.4 | 67.2 | 29.5 | 60.8 | 39.0 | 73.3 | 26.1 | 58.7 | 33.6 | 67.0 | 28.9 | 61.6 |
| ATM-S | 34.9 | 67.7 | 33.1 | 66.9 | 38.1 | 74.3 | 36.0 | 70.2 | 24.6 | 55.6 | 28.4 | 67.4 | 35.1 | 67.9 | 48.3 | 82.1 | 33.2 | 68.6 | 39.1 | 73.0 | 35.1 | 69.4 |
| NERV (ours) | 31.1 | 64.4 | 33.1 | 66.9 | 36.6 | 74.1 | 39.0 | 70.2 | 26.1 | 57.1 | 32.9 | 65.2 | 34.2 | 66.0 | 50.4 | 78.0 | 35.5 | 67.7 | 41.1 | 72.5 | 36.0 | 68.2 |

### A.2 Details of Category-based Assessment Table (CAT) Score

All the category-based labels are generated by ChatGPT-4o (^4^44[https://chatgpt.com](https://chatgpt.com)), the prompt we used is "Please provide me with 5 one-word descriptions of the image, ranging from high level to low level.".

| Image Label | Test Image in ThingsEEG | Category-based label |
| --- | --- | --- |
| 00001_aircraft_carrier |  | Ship<br>Carrier<br>Deck<br>Island<br>Antenna |
| 00002_antelope |  | Animal<br>Antelope<br>Fur<br>Grassland<br>Horns |
| 00003_backscratcher |  | Object<br>Tool<br>Backscratcher<br>Wood<br>Handle |
| 00004_balance_beam |  | Structure<br>Beam<br>Wood<br>Grass<br>Support |
| 00005_banana |  | Fruit<br>Banana<br>Yellow<br>Spotted<br>Plate |
| 00006_baseball_bat |  | Sports<br>Bats<br>Baseball<br>Black<br>Grass |
| 00007_basil |  | Plant<br>Herb<br>Basil<br>Green<br>Leaves |
| 00008_basketball |  | Sport<br>Basketball<br>Ball<br>Orange<br>Court |
| 00009_bassoon |  | Instrument<br>Bassoon<br>Woodwind<br>Stage<br>Chair |
| 00010_baton4 |  | Race<br>Relay<br>Baton<br>Yellow<br>Hand |
| 00011_batter |  | Cooking<br>Batter<br>Mixing<br>Whisk<br>Bowl |
| 00012_beaver |  | Animal<br>Beaver<br>Fur<br>Tail<br>Paws |
| 00013_bench |  | Outdoor<br>Bench<br>Wooden<br>Garden<br>Trees |
| 00014_bike |  | Bicycle<br>Road<br>Wheels<br>Frame<br>Path |
| 00015_birthday_cake |  | Cake<br>Candles<br>Flames<br>Pink<br>Frosting |
| 00016_blowtorch |  | Tool<br>Blowtorch<br>Flame<br>Canister<br>Gas |
| 00017_boat |  | Boat<br>Water<br>Blue<br>Old<br>Rowing |
| 00018_bok_choy |  | Vegetable<br>BokChoy<br>Green<br>Leafy<br>Stems |
| 00019_bonnet |  | Hat<br>Bonnet<br>Ribbon<br>Fabric<br>Vintage |
| 00020_bottle_opener |  | Tool<br>Opener<br>Wooden<br>Bottlecap<br>Engraving |
| 00021_brace |  | Support<br>Brace<br>Joint<br>Black<br>Strap |
| 00022_bread |  | Food<br>Bread<br>Loaf<br>Slice<br>Crust |
| 00023_breadbox |  | Storage<br>Breadbox<br>Wooden<br>Bread<br>Countertop |
| 00024_bug |  | Insect<br>Bug<br>Leaf<br>Brown<br>Antennae |
| 00025_buggy |  | Vehicle<br>Buggy<br>Off-road<br>Wheels<br>Helmet |
| 00026_bullet |  | Ammunition<br>Bullet<br>Brass<br>Cartridge<br>Metal |
| 00027_bun |  | Food<br>Bun<br>Sesame<br>Bread<br>Round |
| 00028_bush |  | Plants<br>Bushes<br>Green<br>Mulch<br>Shrub |
| 00029_calamari |  | Food<br>Calamari<br>Fried<br>Plate<br>Lemon |
| 00030_candlestick |  | Candlesticks<br>Brass<br>Holders<br>Antique<br>Table |
| 00031_cart |  | Cart<br>Wheels<br>Wooden<br>Farm<br>Grass |
| 00032_cashew |  | Nuts<br>Cashews<br>Bowl<br>Snack<br>Glass |
| 00033_cat |  | Animal<br>Cat<br>Tabby<br>Fur<br>Whiskers |
| 00034_caterpillar |  | Insect<br>Caterpillar<br>Striped<br>Green<br>Leaf |
| 00035_cd_player |  | Device<br>CDPlayer<br>Portable<br>Gray<br>Buttons |
| 00036_chain |  | Metal<br>Chain<br>Links<br>Rusty<br>Wood |
| 00037_chaps |  | Clothing<br>Chaps<br>Leather<br>Fringe<br>Brown |
| 00038_cheese |  | Food<br>Cheese<br>Wedge<br>Yellow<br>Cracker |
| 00039_cheetah |  | Animal<br>Cheetah<br>Spotted<br>Hunt<br>Grassland |
| 00040_chest2 |  | Furniture<br>Chest<br>Wooden<br>Vintage<br>Lock |
| 00041_chime |  | Instrument<br>Chime<br>Percussion<br>Metal<br>Stand |
| 00042_chopsticks |  | Utensils<br>Chopsticks<br>Wooden<br>Metal<br>Case |
| 00043_cleat |  | Footwear<br>Cleats<br>Shoe<br>Green<br>Studs |
| 00044_cleaver |  | Tool<br>Cleaver<br>Blade<br>Handle<br>Steel |
| 00045_coat |  | Clothing<br>Coat<br>Black<br>Double-breasted<br>Hanger |
| 00046_cobra |  | Animal<br>Cobra<br>Snake<br>Hood<br>Sand |
| 00047_coconut |  | Fruit<br>Coconut<br>Shell<br>White<br>Husk |
| 00048_coffee_bean |  | Coffee<br>Beans<br>Roasted<br>Brown<br>Grinder |
| 00049_coffeemaker |  | Appliance<br>Coffeemaker<br>Machine<br>Carafe<br>Buttons |
| 00050_cookie |  | Cookies<br>Snack<br>Chocolate<br>Stack<br>Crumb |
| 00051_cordon_bleu |  | Food<br>Chicken<br>CordonBleu<br>Breaded<br>Stuffed |
| 00052_coverall |  | Clothing<br>Coverall<br>Workwear<br>Pockets<br>Green |
| 00053_crab |  | Animal<br>Crab<br>Beach<br>Claws<br>Sand |
| 00054_creme_brulee |  | Dessert<br>CrèmeBrûlée<br>Caramelized<br>Custard<br>Spoon |
| 00055_crepe |  | Dessert<br>Crepe<br>Chocolate<br>Banana<br>Plate |
| 00056_crib |  | Furniture<br>Crib<br>Wooden<br>Baby<br>Bedding |
| 00057_croissant |  | Pastry<br>Croissant<br>Flaky<br>Golden<br>Plate |
| 00058_crow |  | Bird<br>Crow<br>Black<br>Feathers<br>Beak |
| 00059_cruise_ship |  | Vessel<br>Cruise<br>Ship<br>Ocean<br>Deck |
| 00060_crumb |  | Crumbs<br>Plate<br>Food<br>Leftovers<br>White |
| 00061_cupcake |  | Cupcake<br>Dessert<br>Chocolate<br>Icing<br>Wrapper |
| 00062_dagger |  | Weapon<br>Dagger<br>Blade<br>Handle<br>Rock |
| 00063_dalmatian |  | Dog<br>Dalmatian<br>Spotted<br>White<br>Grass |
| 00064_dessert |  | Dessert<br>Berries<br>Cream<br>Trifle<br>Glass |
| 00065_dragonfly |  | Insect<br>Dragonfly<br>Wings<br>Striped<br>Branch |
| 00066_dreidel |  | Toy<br>Dreidel<br>Wooden<br>Spinning<br>Letters |
| 00067_drum |  | Instrument<br>Drum<br>Sticks<br>Blue<br>Percussion |
| 00068_duffel_bag |  | Bag<br>Container<br>Green<br>Straps<br>Eagles |
| 00069_eagle |  | Bird<br>Eagle<br>Flight<br>Wings<br>Sky |
| 00070_eel |  | Fish<br>Eel<br>Aquatic<br>Tank<br>Gravel |
| 00071_egg |  | Eggs<br>Bowl<br>Brown<br>Food<br>Shell |
| 00072_elephant |  | Animal<br>Elephant<br>Trunk<br>Zoo<br>Mammal |
| 00073_espresso |  | Drink<br>Espresso<br>Cup<br>Coffee<br>Saucer |
| 00074_face_mask |  | Gear<br>Mask<br>Helmet<br>Cage<br>Protection |
| 00075_ferry |  | Ferry<br>Boat<br>Transport<br>Water<br>Orange |
| 00076_flamingo |  | Bird<br>Flamingo<br>Pink<br>Water<br>Beach |
| 00077_folder |  | Folder<br>Office<br>Orange<br>Papers<br>Desk |
| 00078_fork |  | Utensil<br>Fork<br>Silver<br>Plate<br>Tablecloth |
| 00079_freezer |  | Appliance<br>Freezer<br>Storage<br>Cold<br>White |
| 00080_french_horn |  | Instrument<br>Horn<br>Brass<br>Coiled<br>Shiny |
| 00081_fruit |  | Fruits<br>Assortment<br>Tropical<br>Colorful<br>Fresh |
| 00082_garlic |  | Garlic<br>Bulb<br>Cloves<br>White<br>Peeled |
| 00083_glove |  | Gloves<br>Knitted<br>Patterned<br>Wool<br>Gray |
| 00084_golf_cart |  | Vehicle<br>GolfCart<br>White<br>Seats<br>Wheels |
| 00085_gondola |  | Boats<br>Gondolas<br>Venice<br>Water<br>Blue |
| 00086_goose |  | Bird<br>Goose<br>Flight<br>Wings<br>Lake |
| 00087_gopher |  | Animal<br>Gopher<br>Furry<br>Rodent<br>Field |
| 00088_gorilla |  | Animal<br>Gorilla<br>Primates<br>Silverback<br>Grass |
| 00089_grasshopper |  | Insect<br>Grasshopper<br>Antennae<br>Legs<br>Green |
| 00090_grenade |  | Weapon<br>Grenade<br>Metal<br>Pin<br>Explosive |
| 00091_hamburger |  | Food<br>Hamburger<br>Bun<br>Lettuce<br>Grilled |
| 00092_hammer |  | Tool<br>Hammer<br>Handle<br>Metal<br>Claw |
| 00093_handbrake |  | Automobile<br>Interior<br>Handbrake<br>Lever<br>Grip |
| 00094_headscarf |  | Headwear<br>Scarf<br>Fabric<br>Pink<br>Wrap |
| 00095_highchair |  | Red<br>Wooden<br>Chair<br>Highchair<br>Furniture |
| 00096_hoodie |  | White<br>Hoodie<br>Ground<br>Casual<br>Clothing |
| 00097_hummingbird |  | Hummingbird<br>Green<br>Feeder<br>Small<br>Bird |
| 00098_ice_cube |  | Ice<br>Cold<br>Frozen<br>Clear<br>Cubes |
| 00099_ice_pack |  | Gel<br>Blue<br>Reusable<br>Cold<br>Cooling |
| 00100_jeep |  | Off-road<br>Rugged<br>SUV<br>Adventure<br>Durable |
| 00101_jelly_bean |  | Colorful<br>Sweet<br>Candy<br>Vibrant<br>Chewy |
| 00102_jukebox |  | Retro<br>Vibrant<br>Music<br>Neon<br>Classic |
| 00103_kettle |  | Shiny<br>Stovetop<br>Whistling<br>Metallic<br>Classic |
| 00104_kneepad |  | Protective<br>Sporty<br>Durable<br>Cushioned<br>Ergonomic |
| 00105_ladle |  | Stainless<br>Sleek<br>Functional<br>Polished<br>Culinary |
| 00106_lamb |  | Adorable<br>Fluffy<br>Playful<br>Animal<br>Lamb |
| 00107_lampshade |  | Vintage<br>Floral<br>Fabric<br>Fringed<br>Ornate |
| 00108_laundry_basket |  | Laundry<br>Plastic<br>Basket<br>Towels<br>Grid |
| 00109_lettuce |  | Vegetable<br>Lettuce<br>Leafy<br>Fresh<br>Green |
| 00110_lightning_bug |  | Insect<br>Firefly<br>Antennae<br>Glowing<br>Segmented |
| 00111_manatee |  | Aquatic<br>Manatee<br>Underwater<br>Mammal<br>Floating |
| 00112_marijuana |  | Cannabis<br>Plant<br>Buds<br>Leaves<br>Green |
| 00113_meatloaf |  | Food<br>Meatloaf<br>Slice<br>Sauce<br>Hearty |
| 00114_metal_detector |  | Equipment<br>Detectors<br>Metal<br>Beach<br>Lineup |
| 00115_minivan |  | Vehicle<br>Minivan<br>Car<br>Blue<br>Electric |
| 00116_modem |  | Device<br>Modem<br>Router<br>Black<br>Connectivity |
| 00117_mosquito |  | Insect<br>Mosquito<br>Biting<br>Legs<br>Proboscis |
| 00118_muff |  | Accessory<br>Muff<br>Fur<br>Warm<br>Pink |
| 00119_music_box |  | Device<br>Music<br>Box<br>Crank<br>Punched |
| 00120_mussel |  | Seafood<br>Mussels<br>Shells<br>Steamed<br>Parsley |
| 00121_nightstand |  | Furniture<br>Nightstand<br>Wooden<br>Drawer<br>Lamp |
| 00122_okra |  | Vegetable<br>Okra<br>Green<br>Basket<br>Fresh |
| 00123_omelet |  | Breakfast<br>Omelet<br>Vegetables<br>Tomatoes<br>Herbs |
| 00124_onion |  | Vegetable<br>Onion<br>Red<br>Sliced<br>Raw |
| 00125_orange |  | Fruit<br>Orange<br>Citrus<br>Sliced<br>Juicy |
| 00126_orchid |  | Flower<br>Orchid<br>Yellow<br>Bloom<br>Petals |
| 00127_ostrich |  | Bird<br>Ostrich<br>Large<br>Plumage<br>Road |
| 00128_pajamas |  | Clothing<br>Pajamas<br>Striped<br>Blue<br>Fabric |
| 00129_panther |  | Animal<br>Panther<br>Black<br>Predator<br>Stealthy |
| 00130_paperweight |  | Office<br>Paperwork<br>Paperweight<br>Eyeball<br>Documents |
| 00131_pear |  | Fruit<br>Pear<br>Tree<br>Green<br>Ripe |
| 00132_pepper1 |  | Spice<br>Pepper<br>Ground<br>Black<br>Spoon |
| 00133_pheasant |  | Bird<br>Pheasant<br>Feathers<br>Colorful<br>Wild |
| 00134_pickax |  | Tool<br>Pickaxe<br>Wooden<br>Metal<br>Digging |
| 00135_pie |  | Dessert<br>Pie<br>Baked<br>Crust<br>Golden |
| 00136_pigeon |  | Bird<br>Pigeon<br>Grey<br>Perched<br>Feathers |
| 00137_piglet |  | Animal<br>Piglet<br>Spotted<br>Grass<br>Cute |
| 00138_pocket |  | Clothing<br>Jeans<br>Pocket<br>Denim<br>Stitched |
| 00139_pocketknife |  | Tool<br>Pocketknife<br>Blade<br>Compact<br>Multi-functional |
| 00140_popcorn |  | Snack<br>Popcorn<br>Bowl<br>Buttery<br>Crispy |
| 00141_popsicle |  | Dessert<br>Popsicle<br>Colorful<br>Frozen<br>Fruit |
| 00142_possum |  | Animal<br>Possum<br>Furry<br>Marsupial<br>Wild |
| 00143_pretzel |  | Snack<br>Pretzel<br>Salted<br>Baked<br>Dough |
| 00144_pug |  | Animal<br>Pug<br>Dog<br>Leash<br>Panting |
| 00145_punch2 |  | Tool<br>Punch<br>Metal<br>Office<br>Desk |
| 00146_purse |  | Accessory<br>Purse<br>Leather<br>Green<br>Handles |
| 00147_radish |  | Vegetable<br>Radish<br>Root<br>Fresh<br>Bunch |
| 00148_raspberry |  | Fruit<br>Raspberry<br>Red<br>Berry<br>Branch |
| 00149_recorder |  | Instrument<br>Recorder<br>Music<br>Notes<br>Sheet |
| 00150_rhinoceros |  | Animal<br>Rhinoceros<br>Horned<br>Savanna<br>Wild |
| 00151_robot |  | Robot<br>Toy<br>Humanoid<br>Black<br>White |
| 00152_rooster |  | Bird<br>Rooster<br>Feathers<br>Colorful<br>Comb |
| 00153_rug |  | Furniture<br>Rug<br>Patterned<br>Red<br>Ornate |
| 00154_sailboat |  | Boat<br>Sailboat<br>Ocean<br>White<br>Wind |
| 00155_sandal |  | Footwear<br>Sandals<br>Leather<br>Straps<br>Brown |
| 00156_sandpaper |  | Tool<br>Sandpaper<br>Abrasive<br>Roll<br>Rough |
| 00157_sausage |  | Food<br>Sausage<br>Sliced<br>Smoked<br>Meat |
| 00158_scallion |  | Vegetable<br>Scallion<br>Green<br>Fresh<br>Bundle |
| 00159_scallop |  | Seafood<br>Scallops<br>Seared<br>Plate<br>Garnish |
| 00160_scooter |  | Vehicle<br>Scooter<br>Electric<br>Green<br>Urban |
| 00161_seagull |  | Bird<br>Seagull<br>Beach<br>White<br>Walking |
| 00162_seaweed |  | Marine<br>Seaweed<br>Underwater<br>Aquatic<br>Sunlight |
| 00163_seed |  | Food<br>Seeds<br>Flax<br>Brown<br>Spoon |
| 00164_skateboard |  | Sport<br>Skateboard<br>Wheels<br>Outdoor<br>Deck |
| 00165_sled |  | Winter<br>Sled<br>Wooden<br>Snow<br>Sleigh |
| 00166_sleeping_bag |  | Camping<br>Sleeping<br>Bag<br>Outdoor<br>Frost |
| 00167_slide |  | Playground<br>Slide<br>Blue<br>Ladder<br>Outdoor |
| 00168_slingshot |  | Tool<br>Slingshot<br>Wooden<br>Rubber<br>Y-shaped |
| 00169_snowshoe |  | Footwear<br>Snowshoes<br>Yellow<br>Running<br>Winter |
| 00170_spatula |  | Utensil<br>Spatula<br>Metal<br>Slotted<br>Handle |
| 00171_spoon |  | Utensil<br>Spoon<br>Metal<br>Reflection<br>Curved |
| 00172_station_wagon |  | Vehicle<br>Station<br>Wagon<br>Red<br>Classic |
| 00173_stethoscope |  | Medical<br>Stethoscope<br>Instrument<br>Black<br>Diagnosis |
| 00174_strawberry |  | Fruit<br>Strawberry<br>Red<br>Ripe<br>Plant |
| 00175_submarine |  | Vessel<br>Submarine<br>Navy<br>Water<br>Stealth |
| 00176_suit |  | Clothing<br>Suit<br>Formal<br>Business<br>Tailored |
| 00177_t-shirt |  | Clothing<br>T-shirt<br>White<br>Event<br>Hanger |
| 00178_table |  | Furniture<br>Table<br>Wooden<br>Square<br>Drawer |
| 00179_taillight |  | Vehicle<br>Taillight<br>Pink<br>Classic<br>Chrome |
| 00180_tape_recorder |  | Device<br>Recorder<br>Cassette<br>Vintage<br>Audio |
| 00181_television |  | Electronics<br>Television<br>CRT<br>Screen<br>Retro |
| 00182_tiara |  | Crown<br>Tiara<br>Gold<br>Jewels<br>Red |
| 00183_tick |  | Insect<br>Tick<br>Parasite<br>Skin<br>Tiny |
| 00184_tomato_sauce |  | Food<br>Sauce<br>Tomato<br>Pot<br>Red |
| 00185_tongs |  | Utensil<br>Tongs<br>Metal<br>Grip<br>Kitchen |
| 00186_tool |  | Tools<br>Hammer<br>Pliers<br>Screwdriver<br>Utility |
| 00187_top_hat |  | Accessory<br>Top-hat<br>Cane<br>Gloves<br>Velvet |
| 00188_treadmill |  | Exercise<br>Treadmill<br>Machine<br>Indoor<br>Fitness |
| 00189_tube_top |  | Clothing<br>Top<br>Striped<br>Yellow<br>Knitted |
| 00190_turkey |  | Bird<br>Turkey<br>Feathers<br>Fanned<br>Brown |
| 00191_unicycle |  | Vehicle<br>Unicycle<br>Wheel<br>Tire<br>Seat |
| 00192_vise |  | Tool<br>Vise<br>Metal<br>Clamp<br>Adjustable |
| 00193_volleyball |  | Sport<br>Volleyball<br>Beach<br>Ball<br>Sand |
| 00194_wallpaper |  | Interior<br>Wallpaper<br>Pattern<br>Vintage<br>Wood |
| 00195_walnut |  | Food<br>Walnut<br>Nut<br>Shell<br>Brown |
| 00196_wheat |  | Crop<br>Wheat<br>Grain<br>Field<br>Stalk |
| 00197_wheelchair |  | Mobility<br>Wheelchair<br>Manual<br>Wheels<br>Seat |
| 00198_windshield |  | Vehicle<br>Windshield<br>Glass<br>Car<br>Street |
| 00199_wine |  | Beverage<br>Wine<br>Glass<br>Grapes<br>Red |
| 00200_wok |  | Cookware<br>Wok<br>Pan<br>Handles<br>Black |

## Appendix B The image generation results of NECOMIMI

In this section, we will present all the images generated by various EEG encoders within the NECOMIMI framework using a fixed random seed. These images are generated using the testing set of the ThingsEEG dataset in a zero-shot setting, meaning that the model has not seen these categories during the EEG-Image contrastive learning training process. All the images illustrate the progression of visual representations generated using different embedding techniques in a diffusion model: (a) Top row: The original images shown to subjects (ground truth). (b) Second row: Images generated by the CLIP-ViT embeddings of the original images. It is only related to the seed and has nothing to do with the subject and EEG encoder. (c) Third row: Images generated by one-stage method using pure EEG embeddings with the EEG encoder. (d) Fourth row: Images generated by two-stage NECOMIMI method using pure EEG embeddings with EEG encoder.

Figure: Figure 5: Random selected generated images in Subject 6 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub06_generated_image_0_composite_image_1.png

Figure: Figure 6: Random selected generated images in Subject 6 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub06_generated_image_10_composite_image_2.png

Figure: Figure 7: Random selected generated images in Subject 6 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub06_generated_image_90_composite_image_10.png

Figure: Figure 8: Random selected generated images in Subject 7 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub07_generated_image_0_composite_image_1.png

Figure: Figure 9: Random selected generated images in Subject 7 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub07_generated_image_10_composite_image_2.png

Figure: Figure 10: Random selected generated images in Subject 7 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub07_generated_image_90_composite_image_10.png

Figure: Figure 11: Random selected generated images in Subject 8 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub08_generated_image_0_composite_image_1.png

Figure: Figure 12: Random selected generated images in Subject 8 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub08_generated_image_10_composite_image_2.png

Figure: Figure 13: Random selected generated images in Subject 8 with NICE EEG encoder.
Refer to caption: extracted/5899447/NICE/NICE_sub08_generated_image_90_composite_image_10.png

Figure: Figure 14: Random selected generated images in Subject 6 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub06_generated_image_0_composite_image_1.png

Figure: Figure 15: Random selected generated images in Subject 6 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub06_generated_image_10_composite_image_2.png

Figure: Figure 16: Random selected generated images in Subject 6 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub06_generated_image_90_composite_image_10.png

Figure: Figure 17: Random selected generated images in Subject 7 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub07_generated_image_0_composite_image_1.png

Figure: Figure 18: Random selected generated images in Subject 7 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub07_generated_image_10_composite_image_2.png

Figure: Figure 19: Random selected generated images in Subject 7 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub07_generated_image_90_composite_image_10.png

Figure: Figure 20: Random selected generated images in Subject 8 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub08_generated_image_0_composite_image_1.png

Figure: Figure 21: Random selected generated images in Subject 8 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub08_generated_image_10_composite_image_2.png

Figure: Figure 22: Random selected generated images in Subject 8 with Nervformer EEG encoder.
Refer to caption: extracted/5899447/Nervformer/Nerv1_sub08_generated_image_90_composite_image_10.png

Figure: Figure 23: Random selected generated images in Subject 6 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub06_generated_image_0_composite_image_1.png

Figure: Figure 24: Random selected generated images in Subject 6 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub06_generated_image_10_composite_image_2.png

Figure: Figure 25: Random selected generated images in Subject 6 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub06_generated_image_90_composite_image_10.png

Figure: Figure 26: Random selected generated images in Subject 7 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub07_generated_image_0_composite_image_1.png

Figure: Figure 27: Random selected generated images in Subject 7 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub07_generated_image_10_composite_image_2.png

Figure: Figure 28: Random selected generated images in Subject 7 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub07_generated_image_90_composite_image_10.png

Figure: Figure 29: Random selected generated images in Subject 8 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub08_generated_image_0_composite_image_1.png

Figure: Figure 30: Random selected generated images in Subject 8 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub08_generated_image_10_composite_image_2.png

Figure: Figure 31: Random selected generated images in Subject 8 with MUSE EEG encoder.
Refer to caption: extracted/5899447/MUSE/MUSE_sub08_generated_image_90_composite_image_10.png

Figure: Figure 32: Random selected generated images in Subject 6 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub06_generated_image_0_composite_image_1.png

Figure: Figure 33: Random selected generated images in Subject 6 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub06_generated_image_10_composite_image_2.png

Figure: Figure 34: Random selected generated images in Subject 6 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub06_generated_image_90_composite_image_10.png

Figure: Figure 35: Random selected generated images in Subject 7 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub07_generated_image_0_composite_image_1.png

Figure: Figure 36: Random selected generated images in Subject 7 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub07_generated_image_10_composite_image_2.png

Figure: Figure 37: Random selected generated images in Subject 7 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub07_generated_image_90_composite_image_10.png

Figure: Figure 38: Random selected generated images in Subject 8 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub08_generated_image_0_composite_image_1.png

Figure: Figure 39: Random selected generated images in Subject 8 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub08_generated_image_10_composite_image_2.png

Figure: Figure 40: Random selected generated images in Subject 8 with ATM-S EEG encoder.
Refer to caption: extracted/5899447/ATM-S/ATM_S_sub08_generated_image_90_composite_image_10.png

Figure: Figure 41: Random selected generated images in Subject 6 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub06_generated_image_0_composite_image_1_2024.png

Figure: Figure 42: Random selected generated images in Subject 6 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub06_generated_image_10_composite_image_2_2024.png

Figure: Figure 43: Random selected generated images in Subject 6 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub06_generated_image_90_composite_image_10_2024.png

Figure: Figure 44: Random selected generated images in Subject 7 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub07_generated_image_0_composite_image_1_2024.png

Figure: Figure 45: Random selected generated images in Subject 7 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub07_generated_image_10_composite_image_2_2024.png

Figure: Figure 46: Random selected generated images in Subject 7 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub07_generated_image_90_composite_image_10_2024.png

Figure: Figure 47: Random selected generated images in Subject 8 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub08_generated_image_0_composite_image_1_2024.png

Figure: Figure 48: Random selected generated images in Subject 8 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub08_generated_image_10_composite_image_2_2024.png

Figure: Figure 49: Random selected generated images in Subject 8 with NERV EEG encoder.
Refer to caption: extracted/5899447/NERV/Nerv2_sub08_generated_image_90_composite_image_10_2024.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: From the Diagrams in Figure 2, it appears that the learned MLP projector after the EEG encoder that is used to better align images and EEG is not used in the image generation stage.  Could you please explain if this is just an error in the diagram or why it is not used?

Please explain how the CAT score is computed?   (see weaknesses above)

The reconstructions in the Appendix using the NERV (authors') encoder do not look anywhere near as good as the reconstructions shown in Figure 1.   Please explain.

It is stated that "The final results are averaged from the best outcomes of 5 random seed training sessions".  What does that mean?  Was best determined on the test data?

It is also stated that "the NERV model obtains the best results with 5 multi-heads...".   Was this determined using  test data or the training data?
soundness: 2
strengths: The top-1 and top-5 classification accuracies for 2-way, 4-way and
200-way classification appear competitive with competitor algorithms.
summary: This paper develops another method for EEG to image reconstruction.
It develops a new EEG encoder and then uses an InfoNCE loss to learn
MLP projectors from a pre-trained image encoder and the EEG encoder.
The EEG	encoding + MLP projector is used for zero-shot classification
through	comparing encodings of EEG to the encodings of the images.
The EEG encoding is also used to generate reconstructed images through
a 2-stage diffusion process.
weaknesses: - The clarity of the paper could be improved.   It is hard to see what the new innovations are over the related work.

- The paper claims to introduce a new measure (Category-Based Assessment Table (CAT)Score) but it is
not fully described.  "each image is manually labeled with two tags for broad categories - one for a specific category and one for background content,resulting in a total of five tags per image". I don't see how you get five tags per image?    What does manually annotate mean?
The appendix says that all the category-based labels are generated by ChatGPT-4o with the prompt "5 one-word descriptions of the image, ranging from high level to low level".
Finally how are these matched (What kind of consideration is given to synonyms)?

- The reconstructions in the Appendix using the NERV (authors') encoder do not look anywhere near as good as the reconstructions shown in Figure 1. (see Questions)

- The reconstruction results as shown in the Appendix are visually inferior to those from the ATM-S paper as well as those from [Fei & de Sa 2024] Image Reconstruction from Electroencephalography Using Latent Diffusion ( https://arxiv.org/html/2404.01250v1 ).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Why is the batch size set to 1024? Was it determined via hyper-parameter search?
2. How is the CAT score defined?
3. How are exactly the image tags extracted?
soundness: 2
strengths: **Significance** The paper deals with a very interesting topic of image generation from EEG recordings.

**Clarity** The content of the paper is written coherently. The flow of the text is easy to comprehend and follow.
summary: The paper introduces an approach for generating images from EEG signals. 
The proposed approach, NECOMIMI, leverages EEG for image generation by using a pre-trained Diffusion model. 
The EEG encoder is trained via contrastive loss by aligning EEG with image embeddings, enabling zero-shot classification.
To evaluate the quality of the generated images, the paper introduces a new metric, the Category-based Assessment Table (CAT) Score, specifically designed to assess EEG-to-image generation based on semantic information. 
Experimental results are provided for both classification and generative tasks.
weaknesses: **Originality and Contribution** The originality of the work is limited. Previous work has explored a similar approach for image generation from EEG [1] and fMRI [2]. The contribution of this work is not defined well enough. The experimental results need to include other datasets. For example, the datasets used in [1] could enable a better comparison with [1]. 

It is expected that the generated images will differ substantially from the ground truth, as the EEG recordings cannot capture the details present in the images Gifford et al. (2022). Additional experiments are needed to evaluate the image generation process. For example, are there any differences across the generated images across the image object categories the EEG stimuli belong? One would need to quantify the effect of the EEG signals for the image generation.

**Quality** It is not explained how the proposed CAT score is defined. Furthermore, it is not clear how the five tags are created.

The References section contains a substantial part of references that point to *arxiv.org*. Though some of *arxiv* papers are published in highly respected venues, it makes it cumbersome to navigate the related work. Some references are repeated; for example, Spampinato et al., Radford et al.

Table 1 mentions top-1 and top-5. However, in the table only one score is reported. It is unclear which one.

The sentence (LINE 047) containing 'EEG is one of the most ancient techniques' makes it sound unusual.

[1] Bai, Y., Wang, X., Cao, YP., Ge, Y., Yuan, C., Shan, Y. (2025). DreamDiffusion: High-Quality EEG-to-Image Generation with Temporal Masked Signal Modeling and CLIP Alignment. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15089. Springer, Cham.

[2] Paul S. Scotti et al., "Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors". In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, 2023, 24705–24728.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: The diagram and accompanying text for the encoder is unclear:
  - What is Spatial-temoral conv and Temporal-spatial conv and how would they operate on tokens.
  - What does the cross attention block do? Usually cross attention is applied on 2 different sets of representation, in the diagram there is only one.
  - EEG data has a time component how is this handled?

-Given the small amount of test data it would make sense to add error indication/ statistical tests.
 
-I think that displaying the reconstructions from the clip embedding is misleading and should be removed.

line 480-482: "we are currently unable to assess whether the brainwave data recorded from the subjects accurately
captures the complete information of the original images, as the subjects might have been distracted
and thinking about other things during the recording"
line 374: "we cannot guarantee that the subject’s thoughts during EEG
recording perfectly align with the ground truth image"
I think this undermines the whole experiment.
I would expected a significant part of the information in the EEG recordings to come from the visual cortex, that shouldn't be effected by subject's thoughts.

- It would make sense to provide qualitative comparisons to the other methods side by side.
 - Regarding CAT score a much simpler and more straight forward approach would be to do coarse level classification. (for example: animals, plants, inanimate, ...)
soundness: 3
strengths: Competitive N-WAY classification results.
summary: A method of semantic image reconstructions from EEG recordings in presented.
The EEG singal is transformed using a transformer based network to be aligned with the corresponding image embedding.
The learned EEG embedding is used with pretrained image diffusion models to generate images in a two stage process.
weaknesses: The think overall the approach is similar to ATM-S("Visual decoding and reconstruction via eeg embeddings with guided diffusion"), specifically very similar encoder, and overall training procedure, there might be differences in the diffusion models, and image generation.
Given the similarities, I don't see how this work work is better.
The metrics presented are not better in a significant way, i.e. the difference is less than variability between subjects, an standard error or a statistical test will indicate that.
Further more for the classification tasks for high N-way (above 10) the results are worse (than ATM-S ), as indicated by the results in the paper.
In the ATM-S work there are also retrieval evaluation and pixel based evaluation, that lack here.
I don't think the papers convinces that this work is better or different in a significant way from ATM-S.

- No qualitative comparisons to other methods 
- Not sure why the new proposed metric "CAT" makes sense, classification is already semantic and not pixel based, additionally the difference all the model is insignificant(in a statistical test sense)
- Unclear specification for the encoder, which prevents reproduction of results (will be elaborated in questions)

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The theoretical analysis lacks robust supports, including the hyperparameter settings for the NERV encoder, which raises concerns about rigor. Experimental validation in this paper is not sufficiently convincing.

2. While the experimental section showcases generated images, these are primarily abstract landscapes, with little detail on improving output quality. Adding a discussion or analysis on image quality enhancement would provide a more complete perspective.

3. The denoising step in the two-stage process and the choice of regularization parameters lack sufficient analysis or validation, which limits the verification of the model’s feasibility. The techniques in this paper seem to differ little from the work of [1] Song et al. and Li et al. [2], what are the innovations in this paper?




[1] Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.

[2] Li D, Wei C, Li S, et al. Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion[J]. arXiv preprint arXiv:2403.07721, 2024.
soundness: 2
strengths: 1. The denoising method at line 318 in the two-stage process allows the model to produce finer-grained images.

2. Results in Tables 2 and 3 show that the experiments perform reasonably well on the ThingsEEG dataset.

3. The CAT score introduced in Section 4.4 reflects a stringent evaluation standard, with no EEG encoder exceeding 500 points, suggesting the metric’s reliability.
summary: The paper proposes a model named NECOMIMI, designed to generate images using EEG data alone. It introduces a novel NERV encoder and highlights two key innovations: first, using only EEG signals as embeddings in a diffusion model for image generation, and second, introducing a novel evaluation metric, CAT score, to assess the quality of generated images due to the lack of existing standards for EEG-to-image translation.
weaknesses: 1. The innovative aspect of the NERV encoder, starting from line 347, lacks sufficient mathematical justification for feasibility. The data presented in Tables 1, 2 and 3 may be coincidental without solid statistical analysis.

2. While Figure 1 demonstrates the model's generative capability, it lacks a thorough analysis of generated image quality. Similarly, Figures 4 onward depict images that visually lack resemblance to any original source.

3. The paper lacks a thorough analysis or proof regarding the selection of regularization parameters in the diffusion model and the settings of the fully connected layers and their impact on the generated images.

Meta-review: The paper has significant weaknesses, including unclear descriptions of the proposed CAT score and insufficient methodological detail, which hinder reproducibility. Reviewers noted limited originality, with the approach closely resembling prior work like ATM-S, while failing to show significant improvements in results. The experiments lack breadth and statistical rigor, and the reconstructions are visually inferior to competing methods. Insufficient author responses and lack of qualitative comparisons further suggest the work is not ready for acceptance.

Decision: Reject