A closed-loop EEG-based visual stimulation framework from controllable generation

Authors: Yiwei Kong, Dongyang Li, Jiahua Tang, Chen Wei, Quanying Liu
Primary area: applications to neuroscience & cognitive science
Keywords: Neural modulation; EEG; Close-loop;

Abstract:
Recent advancements in artificial neural networks (ANNs) have significantly refined methodologies for predicting the neural coding activities of the ventral visual stream in human and animal brains based on visual stimuli. Nevertheless, the endeavor to control visual stimuli to elicit specific neural activities continues to confront substantial challenges, including prohibitive experimental costs, the high-dimensional nature of stimuli, pronounced inter-individual variability, and an incomplete understanding of neuronal selectivity. To address these impediments, we propose a novel electroencephalography (EEG)-based closed-loop framework for visual stimulus. Leveraging this framework, we can identify the optimal natural image stimulus within a theoretically infinite search space to maximize the elicitation of neural activities that most closely align with desired brain states. Our framework employs advanced ANN ensemble models to ensure the reliability of neural activity predictions. Furthermore, we conceptualize the brain coding predicted by the ANN model as a non-differentiable black-box process, allowing us to directly analyze the relationship between the administered visual stimuli and the targeted brain activity. Our research demonstrates that, independent of the exactness of the ANN-predicted brain coding, the proposed framework can procure the theoretically optimal natural image stimulus at given cycle steps. Moreover, our method exhibits generalizability across different modalities of brain-specific activity regulation. Our code is available at https://anonymous.4open.science/status/closed-loop-F2E9.

Decision: reject
Reviewer scores: 6, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1730445672978

thread_type:
official_review

<thread_object>
creator: reviewer_4dny

<content>
summary:
This paper devised a closed-loop framework to find the visual stimuli that can elicit specific neural activities. The authors models the whole process as an MDP, and proposed to use the interactive search (mind matching) and heuristic search (genetic algorithm) to solve the problem. While claimed general, the authors specify the framework to  train the EEG encoding model to generate the synthesized EEG response and test it offline on the THINGS-EEG2 dataset. Visualized results demonstrate the possibility of the whole framework to find the appropriate visual stimuli in the search space. The authors also mentioned its possible impact and insights.

strengths:
1. The whole framework is novel and interesting. It addresses the challenge to find the corresponding stimuli that can evoke a specific brain signal pattern. The framework may have the potential to be applied to a more realistic scenario. 
2. The paper proposed two different settings for finding the visual stimuli: retrieval and generation, and provided corresponding solutions for them. 
3. The overall findings may provide interesting neuroscience intuitions and may ignite further contributions.

weaknesses:
1. One of the main claims by the authors is the adaptation of the whole close-loop framework. While the authors claim it can be simply replaced by recording EEG data from human participants, there are actually no more concrete demonstrations on how. For example, what is the "specific neural activity in the brain" in this paper and in a possible real scenario? What's the difference? And how difficult is it and how much effort will it take to apply the framework to the real world? It's always easy to just claim a methodology "generalizable", but without more justification that doesn't actually help strengthen the contribution of the paper.
2. Based on 1, I feel it is not sufficiently demonstrated in the paper what role the EEG plays in the whole framework. As far as I can understand from the current paper, it seems to be related to the reward $R$ in the MDP design, because it should provide signal based on the desired neural activities. However, we know neither how the reward is exactly calculated nor what kinds of the neural signal the authors are caring about (e.g., a specific frequency bank? a specific shape of waveforms? a specific activation from some brain area?). 
3. Besides the methodology, it's also not clear how the different part of this framework performs and contribute to the final result from the experimental aspect. While in the result section, we can see that the framework can yield promising visual stimuli result, it lacks either quantitative experiments and comparison between selection of algorithms, or a more detailed explanations on the presented ones. (See questions.) Therefore, it's unclear for me what the exact performance of the whole framework and individual parts compared to other solutions.
4. Overall, the presentation of this paper is unsatisfying (and that's probably why I have the concerns in 2 and 3). On the one hand, the author is presenting more well-known details in the main content but didn't make their own claims clear. For example, the algorithm 1 and algorithm 2 is a direct adaptation from previous work. Instead of using space to present them, I wish to see more on how the MDP is constructed. On the other hand, mixing citations with sentences (please use \citep instead \cite) and a few typos (in line 222, algorithm 1, the bracket is not matched) give me the feeling that the paper is not yet ready to be published.

questions:
1. What kind of the neural activity are you concerning in your experiment? How will you verify whether the activity is properly stimulated by your visual stimuli?
2. If the answer to the previous question is via the EEG encoder, then how can the encoder capture your concerned neural activity? How does encoder perform? How will the selection of the encoder influence the result?
3. What is the reward in the MDP?
4. For Figure 3.B, why do you choose subject 8 for demonstration? It seems the confidence interval is large. I wonder whether the similarity increase can pass the significance test.
5. How to interpret the spectrograms in Figure 4.C? I can't see the difference or some trends from the figure. 
6. How is Figure 4.D obtained? Why does the "random" also look so good?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562910834

thread_type:
official_review

<thread_object>
creator: reviewer_egtu

<content>
summary:
The authors proposed a closed-loop stimulation framework for EEG-based visual encoding, aiming to generate visual stimuli to elicit specific neural activities through controllable image generation strategy. In this framework, the authors control the stimulus image generation by approximating the brain activity evoked by the visual stimulation towards the desired neural response that corresponds to the candidate images rated by human users iteratively. Controlling visual stimuli in visual encoding studies is very important. Meanwhile, the stimulus images in most prior studies are relatively arbitrary as there is no standard criteria. The proposed framework provides a possible solution to this problem.

strengths:
The proposed closed-loop framework for synthetic visual stimuli generation in novel in several ways, in terms of the retrieval strategy for identifying candidate images, the feature selection approach, and the method to addressing the problem of unknown target query image. The framework and related methodologies are well designed and presented in general.

weaknesses:
The weaknesses of the manuscript lie in the lack of details and validations. For example, the details of encoding model are not sufficient. The authors described the architecture of the encoding model, however, the details for training such an encoding model is missing. he authors should provide details about training procedure, including data sources. Was the encoding model trained using data from multiple subjects or was it subject-specific? What is the method to validate the encoding model? More importantly, the encoding model was not adequately validated (at least I didn’t see any results related to the encoding model) given its critical role in the framework. In addition, what are the criteria and how to validate that the synthetic images are the “optimal” subset that can evoke specific neural activities? Similar issues exist in other modules, e.g., feature selection and interactive search. The authors are encouraged to validate each module separately rather than integratively.

questions:
1. The limitations of past studies on closed-loop neural encoding/decoding were not adequately justified, weakening the contribution of the study.
2. The subtitles are not well match the items in the framework, making the manuscript is not easy to follow.
3. The encoding model has not been adequately validated. This module is very critical for the proposed framework. In addition, the implement details of the encoding model are not clear, e.g., was the model trained using individual data or data from multiple subject? How many training samples are used to train the encoding model? How to validate the model?
4. Is the EEG encoder which has been aligned with CLIP image features a good choice? This alignment may introduce bias in feature representation of the target and generated EEG signals. Why not a naive EEG encoder? 
5. All the figures and tables are not referenced in the main text, making it quite difficult to read the figures. For example, what is encoded by the dot size in Figure 3c? What is the image with red boundary in Fig. 3d step 10?
4. Are there any failure cases? What I can imagine includes: 1) the random samples in the first round roulette wheel fail to cover the target; 2) The generated images at a certain iteration fail to cover the target. The authors are encouraged to discuss this issue. 
6. “Since different stimulus images in our framework can produce the same or similar EEG features”—this could attribute to the existence of Metamers. However, other factors can not be overlooked: 1) the limitation of EEG (low spatial resolution) in quantifying brain activity. It might be possible that different stimulus image evoke similar EEG responses due to the limitations of EEG. 2) The limitation of the model for EEG feature prediction (the encoding model 3.1). The authors are encouraged to make justifications more carefully.

Other issues:
“quotation marks” are not in right format
The font size of some text in the figures are too small to read.
Typos: in Figure 4 captions: (F) is for O2 channel?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664774451

thread_type:
official_review

<thread_object>
creator: reviewer_cryh

<content>
summary:
This is a highly innovative study demonstrating the capability to identify visual stimuli that closely match the original stimuli eliciting specific EEG activity patterns. The algorithm is well-explained and, to my knowledge, represents one of the first successful applications of this approach with EEG data.

strengths:
Very interesting study, timely, solves an important question, is generalizable.

weaknesses:
I can hardly identify any significant limitations in the current study. However, I have two questions:

The authors state that the identified stimulus is "optimal." Based on the MDP formulation of the algorithm, I understand that it finds a local minimum. Could you clarify how this approach ensures finding a global optimum, rather than a local one?

Why did you limit the comparison to the first 250 ms (Figure 4D)? While the initial 250 ms may indeed capture critical visual information, it is common in EEG analysis to display the full 1000 ms post-stimulus data. Could you elaborate on this choice?

questions:
The authors state that the identified stimulus is "optimal." Based on the MDP formulation of the algorithm, I understand that it finds a local minimum. Could you clarify how this approach ensures finding a global optimum, rather than a local one?

Why did you limit the comparison to the first 250 ms (Figure 4D)? While the initial 250 ms may indeed capture critical visual information, it is common in EEG analysis to display the full 1000 ms post-stimulus data. Could you elaborate on this choice?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703823740

thread_type:
official_review

<thread_object>
creator: reviewer_fvdh

<content>
summary:
This paper develops a method for choosing the optimal image stimulus to present to a human subject to elicit a specific desired pattern of neural activity (as measured using EEG).

strengths:
The problem that this paper takes on is very interesting. I am aware of previous research that has attempted to find preferred visual stimuli for single neurons, so as to figure out what that neuron "prefers", but this paper seems to be taking on a related but quite different issue, which is: given a whole pattern of population activity, what stimulus would elicit that overall pattern? This seems like a project that may have useful clinical applications in the future, as well as being scientifically interesting in its own right.

weaknesses:
I found the paper hard to follow. I admit that a contributing factor here may be my own lack of experience with respect to some of the techniques the paper uses, such as EEG data, diffusion models, and genetic algorithms. However, I do think that the presentation of the paper could be much clearer, and I will list some examples below of specific issues that came up with respect to clarity. 

- Most of the figures I did not understand, and as far as I could tell, the figures aren't referred to in the main text, so it was difficult to situate what the purpose of each figure was in the overall narrative of the paper. 
- It is unclear what the purpose of the MDP is in Section 3.2 (see Questions below).

It would probably have been useful to include a Supplemental section to explain some of the methods in more detail.

questions:
In Sec 3.2, what do the actions and states of the MDP refer to in this context? Are the actions features, because the algorithm is selecting features of the neural activity to represent? Or are the actions the selected images to be used as visual stimuli? 

What is the motivation for not updating the gradients in the model? The abstract says this allows "us to directly analyze the relationship between the administered visual stimuli and the targeted brain activity", but I wasn't sure why this is the case or where in the paper this motivation is fully explained or justified.

In Figure 1, what is the difference between "selection" and "action"?
In Fig 2, the distance metric seems to be applied to images, but I thought the point was to compare induced and target neural activities.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732220906736

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We are sincerely grateful to the reviewers for dedicating their time and effort to review our work. We are excited about the consensus among the reviewers regarding the novelty (FvdH, CRyH, EgtU, 4DNY) and significance (FvdH, 4DNY) of our work. Unfortunately, our past submission has been plagued by clerical errors and lack of clarity. So based on suggestions from reviewers, we have conducted a comprehensive revision of our manuscript to clarify the methodology, add missing technical details and polish our paper. We hope to do our best to clarify what we did not explain clearly in the manuscript to allay the concerns of the reviewers. In the following, we provide a Global Response to the reviewers' questions and concerns.

**Q1: What is the motivation of this work, particularly with regard to the establishment of the closed-loop system and black-box modeling?**

A1: In a closed-loop feedback system, brain activity is influenced by numerous factors, making it highly susceptible to interference. As a result, EEG signals recorded from a single stimulus may not fully capture the desired brain response. This necessitates multiple rounds of stimulation to reinforce and stabilize the response. As mentioned in the Introduction, it has been seen as an significant technique to focus on closed-loop regulation, both at the level of individual neurons and in larger-scale EEG systems. However, these studies have some limitations, such as lack of natural priors and generalization which human beings can understand. This underscores the necessity of the framework we propose. 

The black-box modeling we show in **Figure 1** is to verify the effectiveness of our training-free closed-loop framework. We assume that our framework still work even the specific structure of the encoding model is unknown. This can help us ignore the gains brought by the encoding model and focus on the advancement of the framework itself. So we use a simple encoding model with frozen weights, without the calculation of gradients. Given that the human brain does not provide direct access to neuronal activation patterns or the sequence of bioelectrical responses, these processes are not easily derivable. Therefore, in order to more generally simulate the specific process of our method in regulating human brain responses, it becomes necessary and meaningful to introduce a black-box model.

**Q2: How is the EEG encoding model trained to ensure the reliable synthesis of EEG?**

A2: In our framework, the validity and diversity of EEG signals are prioritized over strict reliability. The framework aims to regulate target-specific neural activity of individuals. The primary requirement for our pretrained encoding model is to encode natural image to EEG, to approximate the brain activity in response to visual perception. Previous studies have shown that EEG signals evoked by visual stimuli can be aligned with stimuli, which can be effectively modeled for downstream tasks such as decoding and reconstruction.

**Q3: How is the MDP algorithm implemented in this work, including a detailed explanation of the actions, rewards, and other relevant components?**

A3: Our closed-loop stimulus generation framework only depends on the current state and the current action (the current image search space), which has uncertainty and randomness in the search space. At the same time, the feedback of the action (selecting the stimulus image) and the reward (similarity score) is clear, so we model the entire iterative process through MDP. When performing retrieval as a setting for finding visual stimuli, the space of states (current stimulus sequence) and actions (selecting stimulus) is explicitly limited, that is, the complexity of the problem is controllable. When using a generative model and combining iterative generation of visual stimuli with evolutionary computation, we set the population fitness (similarity score threshold) as the condition for terminating evolution, so that the space of actions is also limited in this framework.

**Q4: Figures and tables are not referenced, making it quite difficult to read the figures. The presentation lacks clarity in certain areas, making the manuscript is not easy to follow.**

A4: Below, we outline the key changes made in response to your comments. A modified version of the manuscript for further clarification has been uploaded in the **PDF**. 

(1) We have revised the full manuscript, unified the mathematical notation, and refered each figure and table in a logical manner. Moreover, we have supplemented with more implementation details (**Section 4**). 

(2) We have updated the **Figure 1**, **Figure 2** and their captions to make our entire framework more clear. 

(3) In **Section 3.3** and **Section 3.4**, we have updated the description of the algorithm (or pseudocode) in two settings (retrieval and generation) to for better understanding to our implementation details and framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732588435417

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper revision

comment:
We would like to express our sincere gratitude to the reviewers for their insightful and valuable comments on our manuscript. Based on the feedback, we have made several updates to improve the clarity and quality of the writing, as well as to better support the experimental results. 

To facilitate the comparison between the two versions, we have listed the changes below:

**1. General Revisions**: In response to the written errors and unclear expressions raised, we have revised the manuscript to improve its clarity and readability. Specifically, we have corrected the mathematical notations, added the correct references for each figure and table, and provided additional detailed implementation information in **Section 4**.

**2. Figures and Descriptions**:

- **Figure 1**: For the extracted features section in our framework, we have updated the caption to provide a more comprehensive illustration. This revision offers a clearer understanding of our motivation.

- **Figure 2**: We have updated **Figure 2**, which offers a clearer and more concise presentation. On the left, the figure provides a step-by-step breakdown of the process, with more explicit descriptions of each step in the framework. On the right, it includes a detailed illustration of the pipeline, followed by descriptions of the two case studies.

**3. Section 3 Reorganization**:

We have reorganized the **Section 3 "Method"** and created a new **subsection 3.1: "Closed-loop Framework"** to further clarify the mathematical formula, making the section more intuitive to readers. In **Sections 3.3 and 3.4**, we have updated the descriptions of the algorithms (or pseudocode) for both the retrieval and generation settings, which now include more detailed steps and improved clarity to better convey our implementation process.

**4. Appendix Updates**:

- **Appendix A.1**: More details has been added regarding the dataset, the interactive search process, and the heuristic generation algorithm, providing a complete insight into our methodology.  (**Page 13, FvdH**)

- **Appendix A.2**: We have included extensive quantitive results of all subjects in both cases. We have also provided performance comparisons of iterative improvement in different target images across all 10 subjects in EEG semantic representation and spectrum intensity (**Table A.1**). Additionally, we have reported the improvement through in-subject t-tests across all subjects for both cases and correlation analysis between EEG features and CLIP representations (**Figure A.3**). (**Page 15, 4DNY**)

- **Appendix A.3**: In this section, we elaborate on the validity verification of synthetic EEG signals. Specifically, we conducted MSE and Pearson's correlation coefficient on synthetic EEG signals from AlexNet and CORnet-S across all 10 subjects. We also present results from training with ATM-S on two DNN models with pre-trained end-to-end models and randomly initialized end-to-end models (**Table A.2, Figure A.4- Figure A.8**). All these assesments are able to comprehensively vertify the performance of EEG encoding model and support our experiment settings on pretrained end-to-end Alexnet. (**Page 17, 4DNY and EgtU**)

- **Appendix A.4**: We have added supplementary examples, including EEG semantic representation and intensity cases, where we show the iterative complete process of different subjects and targets in each case. Additionally, we have provided examples where regulation failure gradually leads to Goal Drift (**Figure A.4.2**). (**Page 23, EgtU**)

**Changs 1&2&3** have led the improvements of the manuscript without affecting the experimental results presented in this paper. **Changes 4** brings some validations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents a closed loop methodology to identify optimal visual stimuli for maximally eliciting neural activation in the brain. The method relies on a combination of diffusion image generation and Markov-Decision-Process-like updates, iteratively generating new images and updating the latent representation of image-induced activity. The authors then test their data on pre-recorded EEG datasets, and use a neural network model trained to generate EEG data given a stimulus to test their framework. The reviewers were split, appreciating the general mathematical formalization of the method and goals of the paper. Several weaknesses were mentioned, with a large majority falling into the category of either clarity or insufficiency of the experimental validation. While the authors improved the clarity in new versions of the manuscript, there remains some question of how well this method will work in real applications, where a number of practical challenges of EEG stability/drift, changes in noise etc. can potentially reveal unforeseen instabilities in the method. Thus I believe a more thorough validation and, especially, either an experimentally validated test or proof that the simulated EEG suffices, would be key to a future submission.