MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural Representations

Authors: yayun  wei
yilin  dong
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 12:23:21

Abstract: Decoding human visual representations from brain activity data is a challenging but arguably essential task with an understanding of the real world and the human visual system. However, decoding semantically similar visual representations from brain recordings is difficult, especially for electroencephalography (EEG), which has excellent temporal resolution but suffers from spatial precision. Prevailing methods mainly focus on matching brain activity data with corresponding stimuli-responses using contrastive learning. They rely on massive and high-quality paired data and omit semantically aligned modalities distributed in distinct regions of the latent space. This paper proposes a novel Multimodal Bidirectional Cycle Consistency (MB2C) framework for learning robust visual neural representations. Specifically, we utilize dual-GAN to generate modality-related features and inversely translate back to the corresponding semantic latent space to close the modality gap and guarantee that embeddings from different modalities with similar semantics are in the same region of representation space. We perform zero-shot tasks on the ThingsEEG dataset. Additionally, we conduct EEG classification and image reconstruction on both the ThingsEEG and EEGCVPR40 datasets, achieving state-of-the-art performance compared to other baselines.

Main paper: 
 introduction: 
 Understanding the complex mechanism of how the human brain perceives the world is fundamental to cognitive science and the development of deep learning. We argue that the behavior of humans recognizing objects and learning concepts are fundamentally linked to brain activity. This connection is inherently multimodal [1] , indeed, there exists lots of evidence from neuroscience suggesting that cognitive representations are cross-modal. For instance, the same evoke appears in the same brain neurons when humans watch a picture and hear someone's name or voice [2] . The correlation between brain activity signals (such as electroencephalography (EEG) signals) and external stimuli (such as visual and aural stimuli) elucidates how the brain perceives, processes, organizes, and understands the information from the external environment.Decoding brain activity signals aims to deepen our understanding of the brain and create user-friendly brain-computer interfaces (BCIs). EEG has garnered significant attention from researchers due to its non-invasive nature, high temporal resolution, and portability. Notably, learning robust visual neural representations from EEG signals has been a focal point of research [3, 4] . Recently, to explore the relationships between different modalities for decoding brain activity, Du et al. [5] constructed three trimodal datasets and pioneered to present a generic neural decoding method named BraVL, which employs intra-and inter-modality mutual information regularization principles to achieve multimodal learning of brain-visuallinguistic features. Subsequently, Song et al. [6] proposed NICE, a self-supervised framework for learning visual representations from EEG signals, particularly for object recognition. Specifically, they aligned these two modalities by leveraging contrastive learning to compute their similarity. Singh et al. [7] utilized a two-stage method that extracts robust EEG features and subsequently used learned representations for image generation and classification.However, these approaches above neglect the heterogeneity of modalities, as embeddings of different modalities are distributed in completely independent subspaces, increasing the difficulty of modal alignment [8] . To address this issue, we require a robust mechanism to close the modality gap and enable embeddings from different modalities with similar semantics to distribute in the same region of the representation space. Inspired by this motivation, we introduce Multimodal Bidirectional Cycle Consistency (MB2C), which enforces the generated features to approximate the distribution of realistic samples by learning and utilizing a cycle consistency loss between synthetic EEG/image representations and ground truth data, narrowing the gap between modalities. Furthermore, we combine MB2C with contrastive learning to jointly constrain the model's training, achieving cross-modal alignment between EEG and images.Beyond the challenges mentioned, due to privacy concerns and the expensive nature of collection equipment, obtaining large datasets of recorded brain activity signals poses challenges, resulting in a scarcity of paired (stimulus-response) training data. When there is a lack of sufficient paired data, models often suffer from poor generalization, especially in zero-shot tasks [9] . To address this, we introduce mixup-based data augmentation methods [10, 11] on zero-shot learning (ZSL) to compensate for the scarcity of paired brain-visual training data.Contributions. In summary, our main contributions are listed as follows:• We propose a novel multimodal framework for learning robust visual neural representations from EEG-based brain activity. • The Multimodal Bidirectional Cycle Consistency (MB2C) proposed is also utilized to narrow the gap between modalities and ensure that embeddings from different modalities can be distributed in the same region of the representation space. • We combine MB2C with contrastive learning to jointly constrain the model's training, effectively achieving cross-modal alignment between EEG and images. • We perform zero-shot recognition on the ThingsEEG dataset and EEG classification and image reconstruction tasks on the EEGCVPR40 dataset, achieving state-of-the-art performance compared to other baselines. 
related works 2.1 multimodal learning: 
 Our research focuses on multimodal learning. Human perception is inherently multimodal, constituting a complex process involving close collaboration between the brain and multiple sensory systems. Some studies try to decode brain activity and translate it into understandable outputs such as natural language or speech. Dewave et al. [12] translated brain dynamics into natural language using a quantized variational encoder to derive codex encoding and aligned it with pre-trained language models. Guo et al. [13] introduced Dual-Dual GAN that translates neural activity to speech. Other studies focused on sensory stimuli comprising visual stimuli (such as image and video). Kupershmidt et al. [14] proposed a self-supervised method by utilizing cycle consistency over Encoding-Decoding natural videos to achieve natural-movie reconstruction from brain activity. Xia et al. [15] and Takagi et al. [16] , both based on diffusion model to reconstruct images via functional magnetic resonance imaging (fMRI).Zero-shot Learning is a task that recognizes unseen classes using a model trained on seen classes. Generative Adversarial Network (GAN) [22] was proposed to train a generative model from atributive data distribution. The conventional methods used to generate unimodal data (such as text, image, audio, EEG), EEG-GAN [23] , a modification of Wasserstein GANs, was introduced to generate naturalistic EEG data. Recently, Singh et al. [24] proposed a contrastive learning-based method to extract features for conditional GAN (cGAN) to generate images from brain activity signals on EEGCVPR40 dataset that was used in the initial work [25] . Following this, Singh et al. [7] continued introducing a two-stage framework, EEGStyleGAN-ADA, using the CLIP [26] for joint representation learning and transforming the unseen images into EEG feature space with a pre-trained image encoder to reconstruct the images. Unlike GAN-based models, Bai et al. [27] introduced the Diffusion model DreamDiffusion based on the concept of Mind-vis [28] , achieving excellent performance. Additionally, Benchetrit et al. [29] proposed a MEG-based model called MEG-BD to ensure the temporal and spatial resolution of the generated images. 
methodology 3.1 overview: 
 The dataset is defined as 𝐷 = {(𝑥 In ZSL [30] , the dataset 𝐷 is partitioned into distinct seen and unseen subsets, serving as the training set 𝐷 𝑠 and testing set 𝐷 𝑢 , respectively. Similarly, the label set 𝑌 is divided into mutually exclusive seen 𝑌 𝑠 and unseen 𝑌 𝑢 subsets. It is crucial to emphasize that the unseen class labels 𝑌 𝑢 are unavailable in the training phase; they are used exclusively during testing. ZSL aims to predict images corresponding to previously unseen EEG data by leveraging information from EEG introduced during the testing phase.As illustrated in Figure 1 , we present multimodal bidirectional cycle consistency (MB2C). Firstly, we introduce mixup-based data augmentation methods and multimodal learning of Brain-Visual features. Subsequently, we introduce bidirectional cycle consistency loss based on bidirectional classified WGAN (BCWGAN).In ZSL, due to the training set only containing EEG-image pairs for seen classes, using a mapping learned from seen data without accommodating the unseen data can lead to a model bias towards these seen classes. To address this issue, we employ a mixup-based data augmentation method [10] , which generates diverse samples by performing convex combinations of two instances and their respective labels, preventing deep models from overfitting. Since image features are already preserved using a pre-trained image encoder, and recent studies suggest the superior performance of mixing features over original images [11] , we opt for a featurelevel mixup approach, where we randomly combine different class image features from the current batch to synthesize new image data. Additionally, for EEG data with a low signal-to-noise ratio, we apply data-level mixup to blend EEG raw data of different classes. Given randomly two different class EEG-image sample pairs (𝑥 𝑖 𝑒 , 𝑥 𝑖 𝑣 , 𝑦 𝑖 ) and (𝑥 𝑗 𝑒 , 𝑥 𝑗 𝑣 , 𝑦 𝑗 ) from our training data, the augmentation process using the following formulation:EQUATIONEQUATIONEQUATIONThe mixing ratio 𝛾 ∈ [0, 1] is randomly sampled from a Beta distribution. Values of 𝛾 close to 0 or 1 result in generated samples more similar to one of the input data, while values close to 0.5 lead to a more balanced mixture of the data. The generated image features and EEG data are fed into the visual projector and brain encoder, respectively, participating in the ensuing training process.Generating high-quality and discriminative images from EEG features is challenging due to the high noise in EEG embedding compared to the attribute vector (semantic embedding) for class and the inherent problems of mode collapse and convergence failure in GANs. Inspired by [31] , we employ a cross-modal contrastive learning approach to align images and EEG, maximizing the similarity between EEG signals and corresponding visual stimuli. Specifically, we input image and EEG signals stimuli-response pairs into the model, employing an image encoder and a brain encoder to learn image and EEG embeddings, respectively. Subsequently, we utilize the InfoNCE loss [32] as the objective function to maximize the cosine similarity between matching pairs while minimizing the cosine similarity between non-matching pairs. This cross-modal alignment approach at the instance level, rather than the pixel level, ensures distinguishable model outputs, enabling accurate retrieval of visual stimuli corresponding to EEG encodings. In a batch containing 𝑁 pairs of EEG-image, denote the embedding of the 𝑖th image as 𝑧 𝐼 𝑖 and that of the 𝑖th EEG as 𝑧 𝐸 𝑖 in the same bath. The InfoNCE loss for images is formulated as:EQUATION)where similarity is measured by dot product and the temperature hyper-parameter 𝜏 is used for adjusting distribution probability. Symmetrically, we define the loss for the EEG as follows:EQUATIONThe total loss of the InfoNCE thus expresses: Visual feature generation network. The visual generator 𝐺 𝑣 uses EEG embedding 𝑓 𝑒 (𝑥 𝑒 ) obtained from the brain encoder and a random noise distribution 𝑧 1 ∈ R 𝑑 𝑧 sampled from a Gaussian distribution 𝑁 (0, 1) to generate fake visual features intended to deceive the visual feature discriminator 𝐷 𝑣 . To distinguish between the synthetic features and the real features, the visual discriminator 𝐷 𝑣 employs a fully connected (FC) layer for binary classification. Additionally, introducing classification loss in the discriminator has demonstrated significant potential in auxiliary classifier GANs [34, 35] . Therefore, we incorporate an auxiliary classifier into 𝐷 𝑣 for classifying input samples into their respective image categories. Due to challenges associated with traditional GANs [22] such as training difficulties and the inability of generator and discriminator losses to guide training progress, we opt for a more stable training method known as WGAN [36] . Its loss function is defined as follows:EQUATIONEQUATIONEQUATIONIn the formula, E[.] represents the mathematical expectation, 𝜃 𝐺 𝑣 and 𝜃 𝐷 𝑣 respectively denote the parameters of the visual generator and visual discriminator. In the last term of equation ( 7), a gradient penalty term is employed to impose the Lipschitz constraint [37] . Specifically, 𝑓 𝑣 ( x𝑣 ) = 𝛾 𝑓 𝑣 (𝑥 𝑣 ) + (1 -𝛾)𝐺 𝑣 (𝑓 𝑒 (𝑥 𝑒 ), 𝑧 1 ; 𝜃 𝐺 𝑣 ), where 𝛾 ∼ 𝑈 (0, 1). The variables 𝛽 and 𝛼 are two hyperparameters, with 𝛽 representing the gradient penalty factor, and 𝛼 is utilized to weigh the contribution to the visual feature classification loss 𝐿 𝐶𝐿𝑆1 . Here, 𝐿 𝐶𝐿𝑆1 denotes the cross entropy loss between the visual features and their corresponding real labels 𝑦.EEG feature generation network. Considering the introduced multimodal cycle consistency loss, which involves generating EEG features from visual features, we propose the EEG Feature Generation Network 𝐸 = {𝐺 𝑒 , 𝐷 𝑒 }, analogous to 𝑉 = {𝐺 𝑣 , 𝐷 𝑣 }, 𝐸 consists of an EEG feature generator 𝐺 𝑒 and an EEG feature discriminator 𝐷 𝑒 . First, we concatenate the image embedding with a randomly sampled noise vector 𝑧 2 from a standard Gaussian distribution. The generator 𝐺 𝑒 takes the concatenated vector of EEG features and noise, represented as [𝑥 𝑣 , 𝑧 2 ], as the input to generate synthesized EEG features in the latent space. Meanwhile, the discriminator 𝐷 𝑒 discriminates between the real EEG features and synthetic features from 𝐺 𝑒 (𝑓 𝑣 (𝑥 𝑣 ), 𝑧 2 ; 𝜃 𝐺 𝑒 ). To improve the accuracy of the EEG feature generation, we introduce the EEG feature classification loss 𝐿 𝐶𝐿𝑆2 , which guides the EEG generation toward the specified direction. Therefore, the objective function can be formulated as follows:EQUATIONEQUATION)where equation ( 9) denotes Wasserstein adversarial loss, the first two terms calculate the Wasserstein distance between the synthesized EEG feature distribution and the real EEG feature distribution. The last term computes the gradient penalty by linearly interpolating between the real and generated EEG features. Equation ( 10) introduces the EEG feature supervised classification loss to ensure that EEG features are as discriminative as visual features.3.4.2 Multimodal Bidirectional Cycle Consistency Loss. In recent years, numerous studies have successfully employed GAN to transform extracted EEG features into corresponding image features [7, 24, 38] . However, existing methods solely rely on unidirectional generative networks, specifically transforming EEG into images. This limitation of the network architecture does not guarantee the accurate representation of the generated visual features corresponding to the underlying brain activities. To address this issue, we propose a bidirectional multimodal cycle consistency loss designed to encourage the visual features generated from EEG features to transform back to their respective EEG embeddings. Similarly, the EEG features generated from image features should also be capable of transforming back to their corresponding image embeddings.After 𝐺 𝑣 and 𝐺 𝑒 have generated the corresponding reconstructed features, we compute the bidirectional cycle consistency loss to update the parameters of 𝐺 𝑣 and 𝐺 𝑒 . In this way, 𝐺 𝑣 and 𝐺 𝑒 can perform feature reconstruction while retaining information from the original data. Formally, the loss is defined as follows:EQUATIONEQUATION)EQUATION)where 𝑁 denotes the number of samples and 𝜆 is the coefficient used in the computation. The bidirectional cycle consistency loss is computed using the mean squared error between the reconstructed features and real features, ensuring consistency between these features and closing the modality gap. 
experiments 4.1 datasets: 
 ThingsEEG dataset [39] . It comprises EEG data from 10 participants collected using the rapid serial visual presentation (RSVP) paradigm. Except for a single reference electrode, the data of 63 electrode channels can be obtained simultaneously, with a sampling frequency set at 1000Hz. Each participant took part in a total of 82,160 trials. The training set, which consists of 1,654 categories, each with 10 images, is repeated 4 times; The test set, on the other hand, comprises 200 categories, each with only 1 image, and is repeated 80 times. In the preprocessing stage, we average all EEG repetitions for each image to reduce noise impact and enhance the quality of the EEG signals. We select EEG time series from 0 ms to 1000 ms post-image onset. The EEG signals are down-sampled to 250 Hz, a frequency that captures the brain electrical activity induced by visual stimuli while significantly reducing the data volume for subsequent analysis. EEGCVPR40 dataset [25] . It is a subset of the ImageNet dataset [40] , comprising EEG-image pairs from 40 classes. During the experiment, 6 subjects viewed 2000 different images, where each image appeared once for 0.5 seconds, resulting in 12,000 visual-evoked EEG sequences. The EEG data are 128 channels with a sampling rate of 1000 Hz. For EEG data preprocessing, we select EEG data ranging from 20 to 460 ms and discard some faulty samples, resulting in 11,940 EEG-image pairs. To ensure a fair comparison, we have concatenated the data of all participants together, and the splitting of training and testing sets followed the same approach as [7] .This section describes the implementation details of the visual feature generation network 𝑉 = {𝐺 𝑣 , 𝐷 𝑣 } and the EEG feature generation network 𝐸 = {𝐺 𝑒 , 𝐷 𝑒 }. 𝐺 𝑣 (•) and 𝐺 𝑒 (•) are implemented with a hidden layer containing 900 hidden units, activated by LeakyReLU, and the output layer has dimensions identical to those of the image encoder. 𝐷 𝑣 (•) and 𝐷 𝑒 (•) have two hidden layers with 400 hidden units, applying a dropout rate of 0.25 and LeakyReLU activation. Furthermore, discriminators contain two fully connected layers, where one fully connected layer identifies whether the input features are real or not, and the other fully connected layer classifies the input features into the corresponding categories. In all datasets, the noise vector 𝑧 is set to a dimension of 100. We select Adam as the optimizer for the brain encoder, brain projector, and image projector with the learning rate set to 2 × 10 -4 . RMSProp is used as the optimizer for the feature generation network with a learning rate of 5 × 10 -5 . Default hyperparameters are set to 𝛽 = 1.0, 𝛼 = 1.0, and 𝜆 = 500. In the ThingsEEG dataset, the batch size is 256, and training epochs are set to 200. To prevent model overfitting or underfitting, we adopt an early stopping strategy. In the EEGCVPR40 dataset, we set the batch size to 64, and the training process runs for 2048 epochs. To ensure more accurate experimental results, we conduct each experiment three times with different random seeds and report the average results. All experiments are conducted using PyTorch on a GeForce 3090 GPU. The code has been released at: https://github.com/leeh2213/MB2C 𝑁 -way Zero-shot classification. We perform 𝑁 -way zero-shot classification tasks on the ThingsEEG dataset following the procedure outlined in Figure 1 for intra-subject and inter-subject experiments. As shown in Table 1 , the results indicate that our proposed method possesses excellent neural decoding capabilities. Specifically, in intra-subject experiments, MB2C achieves a top-1 accuracy of 50.47% and an average top-5 accuracy of 84.20% in a 50-way classification task, significantly surpassing chance levels of 2% and 10%, respectively. Even when tested with 200 unseen classes, the model maintains a top-1 accuracy of 28.45% and a top-5 accuracy of 60.37%. In the inter-subject experiments, due to individual differences among subjects, the neural decoding performance falls short of the intra-subject predictions but still significantly outperforms chance levels. We also compare MB2C with state-of-the-art methods, namely BraVL [5] and NICE [6] . Considering NICE with self-attention (NICE-SA) or graph attention (NICE-GA) as two variants that outperform the original NICE model, we select the bestperforming model for comparison. The results demonstrate that, regardless of whether the experiments were intra-subject or intersubject, MB2C outperforms the current state-of-the-art models in both 50-way and 200-way zero-shot classifications.Joint representation learning on EEGCVPR40 dataset. To verify the robustness of MB2C, we conduct experiments on the EEGCVPR40 raw dataset and the EEGCVPR40 filter dataset . During the model training process, we observe that directly using EEG-visual correlations for classification performed poorly. In the EEGCVPR40 dataset, each EEG-image pair corresponds to having real class labels. To improve the classification of EEG data, we train the EEG Encoder and two projection layers, similar to [7] , and then fine-tune the network using EEG data. This refinement results in excellent EEG classification results. As shown in Table 2 , our method outperforms the current state-of-the-art models in top-𝐾 accuracy, where 𝐾 ∈ {1, 5, 10}, on both the EEGCVPR40 raw dataset and the EEGCVPR40 filter dataset .Image reconstruction. Here we evaluate the effectiveness of MB2C in the image reconstruction task. Quantitative and qualitative experiments are conducted following the training flow outlined in Figure 2 . Due to the small size of the EEGCVPR40 dataset, we train StyleGAN from scratch to achieve stability and high fidelity. Additionally, we employ the more powerful generative model SDXL [45] , using EEG information as a conditional input for image reconstruction on the EEGCVPR40 filtered (5-95 Hz) and ThingsEEG datasets. Notably, to our knowledge, this is the first image reconstruction on the ThingsEEG dataset.The quantitative comparison results of image reconstruction are listed in Table 3 . We utilize five evaluation metrics that shed light on the quality and diversity of the synthesized images (resized to 128×128) to compare our model with some state-of-the-art methods. Inception score (IS), Frechet Inception Distance (FID), Kernel Inception Distance (KID), Per-pixel Correlation Coefficient (PCC), and Structural Similarity Index Measure (SSIM) are calculated based on reconstructed images. As summarized in Table 3 , our method outperforms other SOTA baselines. Meanwhile, the results of EEG-conditioned image reconstruction are shown in Figure 3 . As can be seen, the quality of the sampling of reconstructed images is semantically similar to the ground truth images, and our method maintains generation consistency in terms of shape, color, and semantics. Additionally, it can be observed that compared to recent work, our approach produces reconstructed results that are visually more similar to the ground truth both in terms of semantics and structure.To demonstrate the effectiveness of each component proposed in our method, we introduce two variants of MB2C for comparison: For a fair comparison, ablation experiments for InfoNCE-only and InfoNCE-CYC-only are conducted with a Mixup ratio coefficient of 0.75 (which yielded the best performance). We will discuss in detail the impact of the Mixup mixing coefficient on model performance in Section 4.6. Additionally, as ZSL is our primary classification task, all ablation experiments are performed on the ThingsEEG dataset. Table 4 displays the performance of each variant. Firstly, it is evident that adding the CYC loss individually on top of contrastive learning can improve overall performance, while jointly training the model with both CYC loss and CLA loss can significantly enhance performance, indicating that the constraint of cycle consistency effectively assists in multimodal alignment. Furthermore, the performance of each variant surpasses that of the methods presented in Table 1 , BraVL and NICE-GA, demonstrating the effectiveness of the MB2C approach. To visually demonstrate the effectiveness of MB2C, we perform similarity measures and visualization analysis on the ThingsEEG dataset. Firstly, to validate whether the EEG features we extract contain semantic information capable of distinguishing between different image classes, we categorize the image stimuli in the test set into six superclasses: animals, clothes, food, household, tools, and transportation. For instance, "tie" belongs to clothes, and "fork" belongs to tools. Then, we compute the cosine similarity of feature pairs for all subjects in the test set, as shown in Figure 4 (a). The results in Figure 4 (a) indicate that the similarity within the superclass is high, indicating that EEG features effectively reflect the semantic categories of corresponding images.In addition, we depict the t-SNE [46] plots of visual features and EEG features learned from the training and test sets in Moreover, except for a few categories, the spatial distributions of the EEG features in the test and training datasets also maintain high consistency.In Figure 5 , we randomly present the top-5 predicted results from subject 8. We can observe that they are semantically similar to the ground truth; for example, "gopher", "piglet", "beaver", "lamb", and "antelope" all belong to the category of animals.We conduct all experiments on the ThingsEEG dataset under the setting of 𝑁 -way Top-𝐾 classification task and present the optimal ratio of Mixup samples in Figure 6 . Given a fixed batch size 𝐵 of 256, within a range of ratio in {0, 0.25, 0.5, 0.75, 1.00}, indicating the choice of 0.75 sampling ratio achieves the best while considering both model performance and training computation (The higher the ratio, the more computations). Thus, we set the ratio to 0.75 under all experiment settings of the ThingsEEG dataset.  
conclusion: 
 In this study, our primary objective is to learn robust visual neural representations from EEG-based brain activity. We propose a novel multimodal framework that combines MB2C with contrastive learning to achieve cross-modal alignment. Notably, MB2C utilizes dual-GAN to generate modality-relevant features and inversely transform them back into corresponding semantic latent spaces, thereby narrowing the modality gap and ensuring embeddings of different modalities with similar semantics reside in the same region of the representation space. Our results demonstrate that decoding seen or even unseen visual categories from EEG signals is potential, and image reconstruction with EEG is also feasible. Lastly, although this paper focuses on EEG and images, we show that MB2C can generalize to other paired modalities. We believe our research holds significant value for practical BCIs and research in multimodal learning. 
 
 Appendix:  


Decision: The paper proposes a method for learning visual neural representations from EEG data.

The reviewers mention (+) the idea of applying adversarial learning to EEG visual decoding being interesting, and (+) the experiments showing the effectiveness of the method.

There are remaining concerns on (-) using GAN for image generation is not state of the art anymore, and (-) the overall quality of the results. They also mention (-) the authors needing to clarify how the data was filtered across different Hz ranges.

After rebuttal, the reviews are somewhat mixed, but positively tending in their ratings (1x weak accept, 2x borderline accept, and 1x borderline reject). Several reviewers increased their ratings such as one from borderline accept to weak accept and one from weak accept to borderline reject. While some reviewers mention open concerns ("not enough responses"), overall this paper could be worthy to be discussed as a poster for the conference.

Reviews: review: paper_summary: paper_summary: This paper proposes a new framework called Multimodal Bidirectional Circular Consistency (MB2C) for learning robust visual neural representations from electroencephalography (EEG) data. This method uses dual GAN to generate modality-related features and reversely transforms them back into the corresponding semantic latent space to narrow the gap between modalities and ensure that different modalities with similar semantics are embedded in the representation space of the same area. Through experiments on ThingsEEG and EEGCVPR40 data sets, the method performs well in zero-shot classification, EEG classification, and image reconstruction tasks, outperforming existing state-of-the-art models.
strength_weakness: strength_weakness: Innovation: A novel multi-modal bidirectional cycle consistency (MB2C) framework is proposed to innovatively solve the problem of cross-modal alignment.

Experimental results: Extensive experiments on multiple benchmark datasets show that the method outperforms existing methods in zero-shot classification, EEG classification, and image reconstruction tasks.

Comprehensiveness: The paper not only shows the overall effect of the method, but also conducts ablation experiments and hyperparameter sensitivity analysis, and explores the contribution of each component and the influence of parameters in detail.
limitations: limitations: Data dependence: Although the data augmentation method is used in the paper, the model still relies on a large amount of high-quality EEG and image paired data. When data is scarce or of low quality, model performance may be significantly affected, limiting its application in environments with limited data resources.
  
Incomplete ablation experiments: Although ablation experiments were performed, ablation experiments were not performed on the mixup method, resulting in the inability to fully evaluate the specific contribution of this data enhancement to the overall model performance. This incomplete experimental design limits in-depth understanding and validation of the role of various parts of the model.

Code openness: This method contains multiple complex components (such as double GAN and cycle consistency mechanism), and there are many implementation details.  The paper makes no mention of whether the code is public. If the code is not made public, it will affect the ability of other researchers to reproduce the experimental results and conduct further research on this basis, limiting the academic dissemination and practical application of this method.
paper_summary: paper_summary: The paper proposes an EEG visual decoding method based on adversarial learning and cycle consistency, called MB2C, and discusses its application in downstream tasks such as classification, image reconstruction, and retrieval.
strength_weakness: strength_weakness: 1. Applying adversarial learning techniques to EEG visual decoding is a very interesting idea.

2. Experimental results in downstream tasks indicate the effectiveness of the methods proposed in the paper for EEG visual decoding.

3. The paper is well-written up until the experiments.
limitations: limitations: 1. The quality of the generated images is not very high. Why not try more powerful and popular generative models such as SD or SDXL?

2. Since the dataset only contains 40 categories of images, with only 50 images each, I believe that training a new StyleGAN might lead to issues of model overfitting. For example, if the generative model can only produce these 40 categories of images, then the quality of the images entirely depends on how well the model fits these limited data. Consequently, the diversity of the images is fixed, making it inappropriate to assess the outcomes using the IS focusing solely on image quality and diversity. Moreover, since the image data used to train the StyleGAN includes a lot of solid color backgrounds (white, black), overfitting might make generated items like watches or cameras appear very similar to the GT images, reducing the disparity between the generative model and real data distributions. This might lead to better FID and KID results.  Perhaps it would be more appropriate to adopt a method similar to that used in the field of fMRI-to-image generation (use a pre-trained model).

3. On the EEGCVPR dataset, the authors chose to reconstruct images, while on ThingsEEG, they opted for similarity measures, which makes the experiments in the paper appear inconsistent and the motivation vague. Why not perform reconstruction experiments on ThingsEEG or use similarity measures on EEGCVPR? Furthermore, the image reconstruction seems to lack qualitative comparisons with baselines. Considering that the metrics are affected by the GAN trained with limited data, I am unable to determine if the image quality produced by this method is sufficiently better than baselines.

4. Minor error:
It seems that the third and fourth images in the second row of Figure 3 are swapped.

In conclusion, in terms of EEG feature decoding, this is a good work. I understand that the main intention of the authors is to demonstrate the decoding capabilities of their method in downstream tasks, but the paper's section on image generation has methodological and experimental issues, which greatly reduces the papers' quality.
paper_summary: paper_summary: This paper proposed a novel method to reconstruct images from EEG. The main idea is to enforce a cycle consistency loss between EEG embedding and image embeddings. The authors also utilized contrastive learning with mixup for better generalizability. Evaluations performed on two datasets show superior performance compared with previous work.
strength_weakness: strength_weakness: This task is of great interest for the practical use of portable BCI. The paper is well-written, with rich literature and a detailed description of the method. The cycle consistency loss is novel and useful in this task. Solid experiments were performed to support the proposed method.
limitations: limitations: 1. Figure 1 is hard to understand. The author may consider splitting this figure into multiple modules or redesigning this figure.

2. Can the author also provide pixel-level metrics such as PCC and SSIM in addition to the classification-based metrics?

3. Can the author provide more visual results other than Figure 3? It's also better to provide a visual comparison with previous works.

4. I am a bit confused about the multimodal claim. It seems only two modalities are involved, e.g., EEG signals and images. I suggest removing the multimodal claim to avoid confusion if these two are the only modalities involved. Otherwise, all EEG-to-image models will be multimodal. 

5. About the mixup-based data augmentation. From Eqn(1), the EEG data are linear combined to generate augmented data. Is it a valid operation? I can understand the mixup in the latent space for image features. But will the linear combination of EEG signals change the characteristics of the EEG signals?
paper_summary: paper_summary: The author proposes a novel Multimodal Bidirectional Cycle Consistency framework for visual neural representations. In their method, they use InfoNCE loss to train encoders to extract EEG and image embeddings first and feed these features to the generator of BCWGAN for cross-modal feature generation. The experiment result demonstrate the effectiveness of their feature.
strength_weakness: strength_weakness: The introduction of the method is very detailed and the experiment is thorough.
limitations: limitations: 1. Figures and tables:

(1) Figure 1 is a little bit complicated,  especially drawing the training objectives of the classification and discriminator into a graph feels a bit redundant. I suggest highlighting Multimodal Bidirectional Cycle Consistency Loss and InfoNCE loss. Do not draw other processes in too much detail.

(2) Some tables have a font that is too large, such as table 4.

2. Experiment result:

(1) Ablation study: it is recommended to correspond to the method, that is, with or without mixed-up data augmentation, infoNCE, and Multimodal Bidirectional Cycle Consistency Loss.

(2) Although the zero-shot classification result is good, the reconstruction results lack comparison with SOTA work in the past three years, such as "Seeing through the Brain: Image Reconstruction of Visual Perception
from Human Brain Signals" and "DreamDiffusion: Generating High-Quality Images from Brain EEG Signals". Also, the reconstruction image in Figure 3 also looks unsatisfactory.