MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Authors: Muzhou Yu
Shuyun Lin
Lei Ma
Bo Lei
Kaisheng Ma
Venue: ICML 2025 poster (2025)
Keywords: Human-Computer Interaction
Brain-Computer Interface
Generative AI
Primary area: applications->neuroscience_cognitive_science

Abstract: Advancements in generative models have promoted text- and image-based multi-context image generation. Brain signals, offering a direct representation of user intent, present new opportunities for image customization. However, it faces challenges in brain interpretation, cross-modal context fusion and retention. In this paper, we present MindCustomer to explore the blending of visual brain signals in multi-context image generation. We first design shared neural data augmentation for stable cross-subject brain embedding by introducing the Image-Brain Translator (IBT) to generate brain responses from visual images. Then, we propose an effective cross-modal information fusion pipeline that mask-freely adapts distinct semantics from image and brain contexts within a diffusion model. It resolves semantic conflicts for context preservation and enables harmonious context integration. During the fusion pipeline, we further utilize the IBT to transfer image context to the brain representation to mitigate the cross-modal disparity. MindCustomer enables cross-subject generation, delivering unified, high-quality, and natural image outputs. Moreover, it exhibits strong generalization for new subjects via few-shot learning, indicating the potential for practical application. As the first work for multi-context blending with brain signal, MindCustomer lays a foundational exploration and inspiration for future brain-controlled generative technologies.

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Muzhou Yu 1
Shuyun Lin
Lei Ma
Bo Lei
Kaisheng Ma
Overview of MindCustomer
Dual Contexts Generation
Visual Stimuli
Image Context
MindCustomer
Image Context
MindCustomer
Manting Any mattha Jun! Hara
Brain Context
Image Context
Blend
MindCustomer
"Text Context"
Brain Context
Brain Context
Triplet Contexts Generation
New Human Subject Few-Shot Generation
Image Context
MindCustomer
Image Context
MindCustomer
Image Context
MindCustomer
Brain Context
"Cyberpunk Style"
Brain Context
"Change Material"
Brain Context "Evening Sky with Grand Nebula"
Figure 1. MindCustomer seamlessly integrates visual brain signals into multi-context image generation. Given the scarcity of real-world brain data, it efficiently performs real-time single-image generation without additional inputs (e.g. masks or muti-views of the same image context). Text context optional, which provides more semanties control over the image and brain contexts.
Abstract
present MindCustomer to explore the blending of visual brain signals in multi-context image gener- ation. We first design shared neural data augmen- tation for stable cross-subject brain embedding by introducing the Image-Brain Translator (IBT) generate brain responses from visual images. Then, we propose effecttve cross-modal infor- mation fusion pipeline that mask-freet adapts distinct semantics from image and brain contexts within diffusion model. resolves semantic conflicts for context preservation and enables har- monious context integration. During the fusion pipeline, we further utilize the IBT transfer image context the brain representation to miti- gate the cross cross-modal disparity. MindCustomer en- ables cross-subject generation, delivering unified, high-quality, and natural image outputs. More-
Advancements in generative models have pro- moted text- and image-based multi-context im- age generation. Brain signals, offering direct representation of user intent, present new opportu- nities for image customization. However, faces challenges in brain interpretation, cross-modal context fusion and retention. In this paper, we
Eqqul contribution "Work done during internship at Beijing Academy of Artificial Intelligence. 'i'aan an Jiaotong Univer- sity "Tsinghua University Beejjig Academy of Artificial In- telligence "Peking University. Correspondence to: Lei Ma clei.ma@pku.edu.coo
Bo Lei <blei.2022@hotmaan how.mall.cmmm.>....... Kaisheng Ma <kaisheng @ mmail.ssinguu..cdu.rr
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
over, it exhibits strong generalization for new sub- jects via few-shot learning, indicating the poten- tial for practical application. As the first work for multi- multi-context blending with brain signal, Mind- Customer lays a foundational exploration and in- spiration for future brain-controlled generative technologies.
one generative model, it is important to ensure that the gen- erated images retain the semanties originating from each context, without being overshadowed or altered.
In this paper, we present MindCustomer to address this chal- lenging task. We incorporate the visually evoked brain sig- nals across different human subjects, rich in diverse seman- tic information from viewing natural images, as the brain contexts. MindCustomer leverages the diffusion model (Xu et al., 2023) the base structure for prompt-guided im- age generation to synthesize high-quality images. More importantly. we identify three critical points: (1) We in- troduce the Image-Brain Translator (IBT) for cross-subject brain data augmentation, ensuring subsequent stable embed- dings. Trained to generate pseudo-brain data from image inputs, IBT works alongside a semantic mapper to precisely capture semantic contexts in brain embeddings. (2) We pro- pose an effective cross modsl information fusion pipeline for multi-context preservation during diffusion generation. In order to blend different contexts, we design diffusion fine-tuning based on image context to learn the image se- manties, and then we freeze the fine-tuned diffusion model and lightly optimize the brain context with the target of the image context. This helps mitigate semantic conflicts between different contexts. (3) Besides, during the above process, we further utilize the IBT and brain embedding model to transfer image context to brain-embedding space to mitigate the representation disparity across modalities. Thus, the above-designed brain-blended cross-modal fusion enables final natural integration and semantic preservation across modalities, producing natural and high-quality blend- ing results. MindCustomer performs real-time generation a single image in light of the scarcity of neural data. Mcanwhile, it efficiently achieves multi-context blending without additional masks or samples.
1. Introduction
With the advance of generative models (Chang al., 2023; Ramesh et al., 2022a), text-to-image generation has gar- nered increasing attention. Users can achieve a desired visual effect or convey specific information by provid- ing diverse text prompts. More recently, advancements in large-scale models, such as diffusion (Rombach et al., 2022; Xu et al., 2023), CLIP (Radford et al., 2021b), and large language models (Naveed et al., 2023; OpenAI, 2023) have further enhanced image generation(Kawar (Kaaar al., 2023; Nichol et al., 2022). Based on this, numerous studies (Ding et al., 2024; Kumari et al., 2023; Ruiz et al., 2023; Gal et al., 2023c) have made significant strides in multi-context blended image generation from image and text prompts. It enables high-quality and customized creations with greater control, which has broad applications, including art creation, user-specific design, and virtual interactions.
Simultaneously, we notice that research on brain signal in- terpretation leveraging decoding technology enhances the understanding of biological information (Gong et al., 2024; Gao et al., 2024). However, current works are mainly limited to the brain signal reconstruction (Scotti et al.. 2024; Wang et al., 2024; Xia et al., 2024). As a signifieant semantic modality, brain signals straightforwardly reflect users' per- sonalized thoughts and preferences. We believe that it has a broader extension to customized image creation. Compared to traditional prompts from text and image modalities, brain signals offer a direct representation of user intent, paving the way to produce more efficient and personalized genera- tion. Based on the above observation, this paper pioneers the exploration of multi-context image generation blended with brain signals. It integrates cross-modal contexts from the image, text, and brain signal into a unified generative model to facilitate image customization.
As the first work, MindCustomer successfully implements multi-context blended with brain signal in image genera- tion. It achieves high-quality image customization guided by cross-subject brain contexts. Furthermore, in real-world scenarios where data from new subjects is often limited, we apply MindCustomer with few-shot learning to showcase its generalization capability. Figure 1 illustrates the overview and performance of MindCustomer.
In summary, our contributions are:
However, it lays several key challenges. (1) Cross-subject brain encoding: to achieve unified brain context extraction across different subjects, is crucial precisely capture se-
We explore the task of blending brain semantics in multi-context image generation for the first time, and present the effective MindCustomer to achieve it.
mantic brain embeddings under the shared data scarcity. (2) Multi-modal integration: the generative model should inte- grate implicit representations from multiple modalities in a cohesive manner; otherwise, the embeddings from different modalities may confuse the model's latent space. (3) Multi- context preservation: when multiple contexts are fused in
We design the shared neural data augmentation via IBT to ensure stable cross-subject brain encoding. Further- more, we present an effective pipeline to preserve and fuse contexts naturally during generation, alongside IBT to mitigate the cross-modal disparity.
2

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Visual Slimuli Subj-a
Subj-n
Brain
Diffusion Model
Embedder
Augment Subj-n IBT
Transfered (MRI
Image Emb
Image Context
Reconstruction
Subj-a
(2) Diffusion Model Fine-tuning
Opt
Real
Transfered 1MRI
Fine-tuned Diffusion
Embedder
Subj-n Shallow Layer
Subj-n fMRI
Shallow Layer
Visual Stimuli
Brain Context
Brain Emb
Image Context
Embedder
(3) Brain Embedding Optimization
Shared Deep Layer
Combined Emb
Image Emb
Fine-tuned Diffusion
Subject-wise Embeddings
Semantic Extractor
Brain Emb
Image Flow
SoftCLIP MSE Loss
Text Emb (optional)
Flow
Generation
(1) Brain Representation Pre-training
(4) Multi-modal Integration for Generation
Figure Pipeline of MindCustomer. (1) IBT simulates shared brain data across human subjects augment brain embedding and the semantic extractor is designed refine the representation. (2) We transfer the image context to the brain embedding space and utilize it for the diffusion fine-tuning. (3) The brain context optimized by using the fine-tuned diffusion, further ensuring cross-context adaptation. one, and finally generate the coherent blending results.
(4) We combine these multi-modal embeddings
MindCustomer achieves high-quality and seamless im- age generation guided by brain semanties. It can fur- ther generalize new subjects, performing strong real- world application capability.
FreeCustom (Ding al., 2024) designs a weighted mask strategy fuse input concepts with more focus. Overall, these advancements in image creation facilitate user-driven image generation guided by text and image prompts.
The latest developments hardware have allowed for the recording and analysis various brain signals in neuro- science labs (Sun et al., 2024; Taheri et al., 1994; Hari Lounasmaa, 1989). Brain signals, rich in semantic infor- mation, crucial for studying neural mechanisms (Daly & Wolpaw, 2008; Chaudhary et al., 2016; Weiskopf, 2012). Decoding visual brain signals has become key research area, helping researchers understand human visual percep- tion (Cox & Savoy, 2003; Horikawa Kamitani, 2015; Schoenmakers al., 2013: VanRullen & Reddy, 2018; Gu et al., 2023b; Quan et al., 2024). The development of gener- ative models like diffusion has significantly advanced high- quality brain signal decoding (Wang al., 2024: Xia et al., 2024; Scotti et al., 2023). A predominant approach (Scotti et al., 2024; Shen et al., 2024) involves aligning brain sig- nal encodings with the CLIP space (Radford et al., 2021b), which serves as a condition input for generative models to perform the reconstruction. Additionally, Takagi al. (Takagi & Nishimoto, 2023) reconstruct high-resolution images from brain activity while preserving rich semantic details, without the need for training or fine-tuning complex deep generative models. Mind-Vis Vis (Chen et al., 2023) uti- lizes Masked Autoencoders (MAE) to enhance brain signal encoding, enabling more accurate reconstructions. Given
2. Related Work
Advancements diffusion models (Ho al., 2020; Dhari- wal & Nichol, 2021; Song et al., 2021) have sparked interest in personalized image creation, allowing individuals to gen- erate images based on their creative intent. Many works adopt fine-tuning diffusion as core principle (Ruiz et al., 2023; Gal et al., 2023a). DreamBooth (Ruiz et al., 2023) fine-tunes the diffusion model to associate a unique iden- tifier with target subject, utilizing prior preservation loss to maintain model generalization in few-shot tuning. Textual Inversion (Gal al., 2023a) leverages text em- bedding that is learnable for encoding subject concept. Dii--Difusion (Li et al., 2023a), building on BLIP-2's ap- proach (Li et al., 2023b), pre-trains a multimodal encoder to generate visual representations aligned with text prompts, enabling a diffusion model to leverage these representations for learning sppject-specific features and creating novel ren- ditions. Recent research also explores merging multiple concepts for generation (Avrahami al., 2023; Tewel et al., 2023a; Gu et al., 2023a; Liu et al., 2023b;c). Custom Dif- fusion (Kumari et al., 2023) implements this by employing a constrained optimization method with a closure solution.
3

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
the variability of brain signals across individuals, cross- subject decoding methods have been proposed (Wang al., 2024; Xia et al., 2024) to integrate the encodings and re- constructions of multiple subjects within a unified model. Moreover, some works extend brain signals to more tasks (Xia et al., 2024; Shen et al., 2024; Chen et al., 2024a), such as visual grounding, risual-questionnansweess and styl- ized reconstruction. As a concurrent work, MindPainter (Yu et al., 2025) uses the brain signal as a condition for masked image painting via self-supervised learning. Dif- ferent from these works, this paper presents a mask-free, cross-subject, and multi-context (modalities in image, text, and brain) blending task for image generation. We design the shared neural data augmentation and cross-modal infor- mation fusing pipeline to effectively achieve high-quality image blending results. MindCustomer further broadens the way for contrllondollee image creation.
naturally blend with the provided contexts, mean pre- serving the relevant semantic characteristics. We propose MindCustomer to achieve this task. Considering the lim- ited availability of brain data from different subjects, it is difficult to construct large, labeled datasets for training gen- crative models. In this work, we perform efficient real-time generation on a single image, which holds greater practi- cal significance for real- applications. MindCustomer consists of four parts, which is illustrated in Figure 2.
Brain Representation Pre-training To tackle the incon- sistency for cross-subject brain embedding, inspired from (Scotti et al., 2024; Wang et al., 2024), we apply individual- to-shared encoding to encode fMRI voxels into the universal latent space of the pre-trained CLIP model. However, previ- ous methods only allow the subject-wise data for training, where the limited data affects the encoding capability. We in- troduce Image- Baage-Brain Translator Gr (IBT) to generate pseudo voxels to augment the training data without manual collect- ing. refers to the parameters of the model G. IBT is trained to simulate subject-specific brain signals from a given input image. During the brain representation pre-training stage, we concurrently feed IMRI data from multiple subjects (in- cluding true and generated fMRI) in response to the same stimulus into shared encoding model. This approach effee- tively augments the fMRI training dataset, as the stimulus presented to each subject differs from those in the original dataset. As a result, we can generate shared brain represen- tations across subjects, rather than treating each subject's s data as isolated samples, thereby enhancing the stability of the embedding training data. Specifically, we utilize the pairs of IMRI voxels B and corresponding visual stimuli I E I N X5266xX x in batch size of N to train the subject- wise IBT. We adopt the adaptive max pooling (Wang et al., 2024) to achieve the subject-invariant voxel sizes. Thus, we construct the fixed size of ground truth voxel B' from (13000~16000) to 8192. The training function of IBT G, can be formulated as:
3. Method
3.1. Preliminaries
NSD Data To better understand the brain-driven context, we first introduce the data related to visual brain signals. In this paper, we utilize the widely-used data, Natural Scenes Dataset (NSD) (Allen, 2022), as our brain contexts. This dataset includes high-resolution 7-Tesla fMRT scans from eight healthy adult participants, who were tasked with view- ing various natural images sourced from the MS-COCO dataset (Lin et al., 2014). In accordance with standard prac- tices (Scotti et al., 2024; Wang et al., 2024; Takagi & Nishi- moto, 2023; Shen et al., 2024; Chen et al., 2024a), we utilize all subject-wise data from four subjects (Subj1, 2, 5, 7) as the training data. Each subject viewed 8859 individual im- ages. And the remaining data of 982 images were viewed by all four subjects as the test data.
Diffusion Model Diffusion models (Rombach et al., 2022; Xu et al., 2023) have shown remarkable efficacy in synthe- sizing high-quality images. Leveraging extensive text-image pair datasets during training, they are capable of generating highly accurate visual representations from textual input. In this study, we leverage versatile diffusion (VD) (Xu al., 2023), which is multi-modal diffusion with independent flows for image and text prompts. This inherently enables us to condition the model with both image and text modalities, and we mainly focus on integrating visual brain signals with image modality in VD to generate high-quality images. We initialize our model with the publicly pre-trained VD, and the subsequent fine-tuning process is built upon this prior.
ES = CLIPItagge (7):
(1)
L = LMSE(S Er, B'),
We feed the I into the pre-trained CLIP ViT/L-14 14 (Radford et al., 202 to obtain the latent feature F, of size N x 1024.. LASSE is the MSE loss function. Note that the CLIP model remains frozen.
Next, we elaborate on the pre-training of cross- brain embedder. We denote different subjects as a, b, c, and d. For pair of (Ta, B,) from subject a, we first obtain the text caption T, from image La. and feed both Ia and T. into CLIP model to get the corresponding im- age and text CLIP embedding Fra and Fra- Then we fix the size of Bn to B', E 18192, and apply the IBT to generate B₁ E IR8192, B' E R and B'2 E R from input Ia- We employ the shallow subject-wise em-
3.2. MindCustomer
Given image contexts, brain signals from different human subjects and text prompts, our target is to perform mask-freee blending to generate images. The generation results should
4

---

MindCustomer: Multi-Contcee Image Generation Blended with Brain Signal
Brain Imago
VB
Subj1
Subj2
Subj5
Subj7
Subj1
"in the snow"
Subj2 Brain image
"bus in the background"
Subj5 Brain Image
"made of"
Subj7 Brain Image
"play"
Figure Multi-context generation results across different subjects.
Figure 4. We utilize the same image context (I) and visual stimuli (B) for different subjects blend. It demonstrates the robustness of MindCustomer and individual variability in generation.
bedders E, and deep shared embedders & to output em- beddings of voxels, (eza E S891 ,eTa IR.112), (c16,eT6), (ere,eTe), and (era,era), which will be supervised by CLIP- encoded text and image. Further, loss and MSE loss to optimize the error between the pre- dicted embeddings and corresponding CLIP embeddings feeb LSONCLIP + LISS, where
utilize the SoftCIIIP
LMSE N Σ 8; -S))²
(5)
S is the feature of the ground truth mapper, and is the predicted feature of the extractor M,, Note that the IBTs, shallow embedders, deep embedders, and semantic extractor are all MLP-based models. Especially, the IBT has only 3 layers of MLPs, which are light for efficient training.
exp( Rg/T) + log S 2n=1 exp(es Fa/T)
exple Fy/r)
log >> (.p Fn/T)
1- 1-1
(2)
Diffusion Fine-tuning with Embedding Optimization We observed that when directly fusing brain modality with im- age modality, the semantic gap between the modalities often results in inferior generation outcomes. Although alignment within the CLIP latent space has been attempted, the implicit and complex representations of brain signals still exhibit semantic discrepancies with the image space. Therefore, we propose to apply the IBT Gr to transform the image context / into brain voxel, aiming to eliminate inter-modal inconsistencies and reduce the semantic bias. We feed the transferred image context В, into brain embedder tain the embeddings ep and fine-tune the versatile diffusion to reconstruct the image. It not only allows the diffusion model to maintain the image context represented as the transferred brain signal but also benefits the later natural fusion brain context. The fine-tuning of the diffusion model with the parameters can be formulated as:
N
1
LMSE
M
ae E
(3)
N
With the pseudo voxel augmentation and pre-training pro- cess, we can embed the cross-subject voxels into latent CLIP embedding, which benefits accurate semantic extraction for diffusion-based generation. Furthermore, extract more semantically enhanced features, we introduce the mapper from ClipCap (Mokady et al., 2021) to guide our embed- ding model to explore the text semantics from brain signals. Since the mapper is formerly trained for image caption, it effectively transfers the vision features to language features, we utilize as the supervisor for enhanced brain embed- dings. We design the training of semantic extractor M, for the embeddings of voxels. The output features of the extractor Ma are also aligned with the loss of SoftCLIPP and MSE.
ob-
cime(() -E, Ex.o~N(0.1), [|le tle-co(ze,t,, (ite,,tee))] (6)
exp(8, : 8,/7)
expls; 8,/r)
CSunCLIP
log
where t = 1,...,2 T and x, is Gaussian corrupted image of input co( t) UNets-based denoising function.
exp(S; S., W exp(a; S1.
(4)
5

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Context Input
Image context gradually interpolated with brain context
Image
Brain
a
0.2
0.4
0.5
0.6
0.8
"in the snow"
Figure 5. Visualization of the generation process through gradual interpolation between image and brain contexts.
lmg/ Brain
Img/ / Brain
"in the snow"
"with beautiful sunset"
"a children's drawing"
"cyberpunk style
"snowy mountain with grand nebula"
"traveling with beautiful coastline"
Img/ Brain
Img Brain
"play"
"earth in the screen"
"christmas atmosphere*
"bus in the background"
"Victory Gate behind"
"in HongKong
Figure 6. Multi-context blending results with only different text contexts. MindCustomer robustly creates multi-context images that are content-consistent and naturally integrated.
Next, we empirically observe that when different contexts are fused often results in semantic conflict. Partial context may lost or disturbed. To better preserve the multi-semantics and enhance seamless fusion, we lightly optimize the brain context embedding e for approximating the semantics of the image context. The optimization only modifies the brain context embedding while we freeze the parameters of the diffusion model. We run this optimization for a few steps and this process can be formulated as:
diffusion for blended image outcomes. As for the generation only guided by dual contexts, we directly concatenate the embeddings of ep and e for the VD to obtain the results. As for triplet contexts, the concatenated dimension of embed- dings will exceed the model capacity. Thus, utilize linear interpolation of ép and optimized with a hyperparameter a E [0, 1] to build the combined embedding e.: e = a:) 6, + a 6b-
(8)
We employ the combined embedding e, to the image flow of the fine-tuned versatile diffusion, and meanwhile feed the text context embedding to the text flow, to jointly generate the final multi-context blended image. Note that the a can smoothly adjusted for asymptotical generation results balancing the image and brain contexts.
(braio(es) = E..e-Nio.... [[lc [ik- co
(7) adapt the semantics between the im- age and brain contexts in latent space that enables the diffu- sion model to better understand the fusion of both embed- dings to generate natural images.
Consequently,
New-subject Few-shot Generation Collecting brain sig- nals in real-world scenarios is time-consuming and costly, often resulting in limited data for new subjects. To address
Embeddings Integration Finally, we apply the fine-tuned
6

---

MindCoetomer: Multi-Context Image Generation Blended with Brain Signal
Img/ Brain
Baseline-2
Img/ Brain
Baseline-2
MindCustomer
Baseline-1
MindCustomer
Baseline-1
MMRI Recon
Blending
fMRI Recon
Blending
Figure 7. Baseline comparison. Baseline-1: We replace the brain context with its visual stimuli to construct a multi-image blending. Baseline-2: We reconstruct the brain context to the image and merge all the image contexts for blending. Benefiting from the designed cross-modal fusion pipeline, MindCustomer naturally preserves and fuses different contexts to generate more visually pleasing results.
(Krizhevsky al., 2012) are low-level property evaluation, and metrics of Inception (Szegedy et al., 2016) and CLIP (Radford et al., 2021a) l)) are high-level property evaluation. The implementation is described in Appendix-B.
Table Quantitative comparisons. We compare our method with the basclines on context similarity and generation quality.
Baseline-1
Baseline-2
Ours
CLIP-I
0.559
0.526
0.563
DINOv21
0.630
0.612
0.654
4.1. Qualitative Results
CLIP-IQAT
0.734
0,830
0.892
We use MindCustomer for multi-context image generation by integrating semantic information from brain signals. Im- age contexts from the NSD COCO dataset are paired with brain contexts and optionally enhanced with concise textual prompts to control spatial, environmental, and stylistic at- tributes. We present the customization results across all the subjects of NSD in Figure 3, showcasing the generation of high-quality and contextually faithful images. More results are presented in Appendix-D.I.
this, MindCustomer constructs a cuoss-subject model that integrates brain signals from different subjects into a unified framework, enabling better generalization to new subjects. Specifically, we first use limited new-subject data to train new IBT and fine-tune the brain embedding network. Lever- aging the Reset- train only the shallow embedder for the new subject while freezing the deep shared embedder, without IBT augmen- tation. The remaining generation steps are the same as the above processes. Empirical results show that using less than 20% of the full dataset is sufficient for few-shot learning to generate natural outputs.
operation (Wang et al., 2024), we
Since the visual stimuli in test data are shared across all four subjects, we conduct experiments under identical conditions using the same contexts for each subject. This setup allows us to evaluate the model's robustness and showcase the individual variability among different subjects. The results are shown in Figure 4.
4. Experiments
In the embedding interpolation of MindCustomer, we apply a hyperparameter a to control the ratio of transferred image embedding and brain instruction embedding. In Figure 5, we visualize the generation process via smoothly changing a. It can be observed that when the alpha value is around 0.5, our method achieves natural fusion of two embed- dings while preserving their respective feature information. Empirical experiments indicate that a range between 0.4 and 0.7 generally yields optimal blending results.
Evaluation Metrics To assess the semantic preservation of the generated image, we use the common metrics on image similarity, DINOv2 (Oquab al., 2024) and CLIP-I (Rad- ford al., 2021c). We further apply CLIP-IQA (Wang et al., 2023) to test the generation quality, a measure evaluating both the visual appeal and the conceptual perception of an image. Moreover, we adopt metrics of visual IMRI recon- struction to assess brain decoding performance. Following previous works (Wang et al., 2024; Xia et al., 2024), Pix- Corr, SSIM (Wang et al., 2004), AlexNet(2), and AlexNet(5)
Moreover, we present more visual results in Figure 6 with only different text contexts. As we can see, MindCus-
7

---

Muntommmmme:: Multi-Context Image Generation Blended with Brain Signal
Image
10%
20%
40%
Table 2. User study on multi-context image generation. Users are asked to score the generated images from 1 to 3 (3 is the best).
Brain
Method
Quality
Consisteneyl
Baseline
1.868
1.675
Ours
2.618
1.868
Table 3. Quantitative results on Subjl few-shot generation.
10%
20%
40%
CLIP-I
0.523
0.541
0.561
DINOV21
0.642
0.650
0.652
HIASS
CLIP-IQAT
0.944
0.947
0.952
Average 1
0,703
0.713
0,722
"in the snow"
in MindCustomer, we designed VD fine-tuning based on image context to learn the image semantics, and then we freeze the fine-tuned VD model, and lightly optimize brain contexts with the target of the image context. This helps mitigate semantic conflicts between different contexts. Dur- ing the above process, we further utilize the IBT to transfer image context to brain-embedding space to reduce the gap between modalities. Thus, the designed brain-blended cross- modal fusion pipeline allows the blending to better preserve each modality's 's semantics and produce more accurate, high- quality results. The quantitative metrics also demonstrate that MindCustomer significantly outperforms the baselines in terms of context similarity and image generation quality.
"eating"
Figure& 8. Few-shot generation the new human subject. We sepa- rately utilize 10%, 20%, and 40% of the subject's complete dataset perform generation fine-tuning, MindCustomer generalizes well to the new subject, indicating its real-world applicability.
tomer robustly creates multi-context images that are content- consistent and naturally integrated.
User Study We also conduct the human evaluation study, surveying our method and baseline. As the overall perfor- mance of Baseline-1 surpasses that of Baseline-2, we choose Baseline-1 1 as the baseline for this analysis. In the study, each generated image is scored according to the generation quality and the aligmment the contexts. In total, we col- lect 220 answers, whose results are summarized in Table 2. Participants exhibit notable preference for method in both perspectives. More details of the study protocols are described in Appendix-C.
4.2. Comparisons and Analysis
Baseline Unlike traditional methods (Ruiz al., 2023; Ding et al., 2024; Kumari et al., 2023), relying on masks mul- tiple images of the same object, MindCustomer enables mask-free, single-image generation, which makes direct comparisons challenging. To address this, we design two baselines using versatile diffusion (VD) (Xu et al., 2023), capable of multi-context blending. Baseline-1: We replace the brain context with its image-based visual stimuli and input all the contexts for VD generate results. Baseline-2: We use an SOTA IMRI-to-image model (MindEye (Scotti et al., 2023)) to reconstruct the brain context as another image context, and merge all the contexts for VD blending. Note that as baselines, VD requires no extra fine-tuning or optimization and relies on the original large text-image pre-training capability for direct inference.
4.3. Few-shot Generation on New Subject
In real-world scenarios, collecting brain signals from human subjects is highly costly, leading to data scarcity. There- fore, the ability of a designed algorithm to generalize to new individuals is crucial. To demonstrate MindCustomer's practical generalization capability, we pre-trained the model from Subj2. 5, and 7 and performed few-shot learning on Subj1. We fine-tuned our model using 10%, 20%, and 40% of the total data from Subj1, respectively.
Here present both qualitative and quantitative compar- isons of generated images in Figure 7 and Table 1. Due to the brain signal interpretation error caused by IMRI recon- struction and some semantic overlap between distinct image contexts resulting from the blending diffusion model, these baselines show some semantic deviation and loss. While
As shown in Figure 8, MindCustomerr achieved natural and competitive generation results even with limited data for fine-tuning. As the data point increased, the model consis- tently generated higher-quality images. The quantitative
8

---

MindCostomer: Multi-Context Image Generation Blended with Brain Signal
Image
Brain
(1)
(2)
(4)
(5)
MindCustomer
"Cyberpunk Style"
Figure 9. Qualitative ablation. (1) wo [IBT & fine-tuning & optimization]: Direct generation by concatenating CLIP-encoded image and Brain embedder-encoded contexts leads to semantic inconsistency and context overlap. (2) IBT: Adding fine-tuning and optimization fails to resolve these issues. (3) wo [fine-tuning & optimization]: Using IBT for alignment before concatenation allows partial explicit representation of semantic information. (4) wo optimization: Adding fine-tuning sigmificantly improves results, enabling better representation of different semantic contexts. (5) wo ClipCap: Without the introduced ClipCap for semantic enhancement in Brain Representation Pretraining may result low details context representation in blending results, MindCustomer: Incorporating all techniques, we can produce higher-quality image generation with detailed context fidelity.
Visual Stimull
UMBRAE MindCustomer Visual Stimuli
UMBRAE MindCustomer
Table Quantitative ablation. We compare MindCustomer with the ablations on context similarity and generation quality. (1) wo [IBT& & fine-tuning & optimization]: (2) wo IBT; (3) [fine-
tuning & optimization): (4) wo optimization; (5) wo ClipCap.
(1)
(2)
(3)
(4)
(5)
MindCustomer
CLIP-I
0,518
0.515
0.529
0,548
0,560
0.563
Visual Stimuli
Subj1
Subj2
Subj$
Sunj7
DINOv2
0.594
0.595
0.614
0.635
0.637
0.654
CLIP-IQAT
0.842
0.838
0.893
0.893
0.875
0.892
results of few-shot learning under different data volumes are presented in Table 3, indicating the promising general- ization ability of MindCustomer. More few-shot generation results are illustrated in Appendix-D.1.
Figure 10. Above: Comparison of brain decoding Subj1. Below: Brain decoding across subjects by MindCustomer.
4.4. Ablation
We conduct experiments to validate the components of MindCustomer. First, we design IBT to achieve modal alignment by mapping semantic representations from dif- ferent modalities into a shared implicit space, forming the basis for fusion. Second, we fine-tune the diffusion model on image contexts to preserve semantics and optimize brain contexts for high-level semantic adaptation, generating co- herent images. Figure 9 illustrates the key components of our design, and Table 4 provides quantitative ablation re- sults. Besides, visualize the brain reconstruction ability of MindCustomer in Figure 10. Compared with the SOTA method (Xia et al., 2024), MindCustomer also demonstrates superiority and robustness across diverse subjects. The quan- titative results on brain reconstruction and more detailed ablation experiments are conducted in Appendix-D.
5. Conclusion
In this paper, we propose novel brain-blended image customization method, MindCustomer. We introduce the IBT to enhance the data augmentation for cross-subject brain embedding. Further, the proposed fusion information pipeline, along with IBT, effectively adapts distinct seman- ties across multiple contexts. Thus, MindCustomer can generate context-preserved and naturally coherent blend- ing results. Besides, it can also generalize to new human subjects via few-shot learning, demonstrating the potential for real-world applications. In all, we hope MindCustomee serves as an exploration of brain context in the image cus- tomization task and gives inspiration for follow-up research, promoting the development of brain-computer interaction.

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Acknowledgements
Learning to follow image editing instructions. In CVPR, pp. 18392-18402. IEEE, 2023.
This work was supported by the Beijing Natural Sci- ence Foundation (Grant No.JQ24023 and No.F251020), Beijing Municipal Science & Technology Commission Project (No.Z231 100006623010), National Science and Technology Major Project (2022ZD01 163013 to B.L.), and Beijing Municipal Science and Technology Project
Cao, M., Wang, X., Qi, Z., Shan, Y., Qie X., and Zheng, Y. Masactrl: Tuning-free free mutual self-attention control for consistent image synthesis and editing. In ICCV, pp. 22503-22513. IEEE, 2023.
(No.22411000042240000
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M., Murphy, K. P., Freeman, W. T., Rubinstein, M., Li, Y., and Krishnan, D. Muse: Text-to- image generation via masked generative transformers. In ICML, volume 202 of Proceedings of Machine Learning PMLR, 2023.
Impact Statement
With the development of generative models and large lan- guage models, Al-driven personalized generation is gar- nering increasing attention and application. At the same time, advancements in neural data decoding are empower- ing researchers to delve deeper into brain-machine interfaces (BMIs) and human-computt interaction (HCI). The inte- gration of BMIs and generative models is poised to make personalized brain-controlled generation significant trend in the future of generative artificial intelligence. Mind- Customer represents the first exploration of multi-context image generation guided by brain semantics. We aim for to serve as foundational research, offering potential in- sights and inspiration for future research in related fields as well as practical applications. We believe personalized brain-controlled generation will lead groundbreaking de- velopments in areas such artistic creation, game design, and advertising industries.
Research, pp.
Chaudhary, U., Birbaumer, N., and Ramos-Murguiale A. Brain-computer interfaces for communication and rehabilitation. Nature Reviews Neurology, 12(9))513- 525, 2016.
Chee, J. Pearson's product-moment correlation: Sample analysis, 05 2015.
Chen, J., Qi, Y., Wang, Y., and Pan, G. Mind artist: Creating artistic snapshots with human thought. In CVPR, pp. 27197-27207. IEEE, 2024a.
Chen, X., Huang. I.., Liu, Y., Shen, Y., Zhao, D., and Zhao, H. Anydoor: Zero-shot object-level image customization. In CVPR, pp. 6593-6602. IEEE, 2024b.
Chen, Z., Qing, J., Xiang, T., Yue, W. L., and Zhou, J. H. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In CVPR, pp. 227 22710-22720. IEEE, 2023.
References
Abdal, R., Zhu, P.. Mitra, N. J., and Wonka, P. Styleflow: Attribute-condition exploration of stylegan-generaan generated images using conditional continuous normalizing flows. ACM Trans. Graph., 40(3):21:1-213: 2021.
Cox, D. D. and Savoy, R. L. Functional magnetic resonance imaging (fmri) "brain reading": detecting and classify- ing distributed patterns of fmri activity in human visual cortex. Neurolmage. 19(2):26--270, 2003.
Alaluf, Y., Richardson, E., Metzer, G., and Cohen-Or, D. neural space-time representation for text-to-image per- sonalization. ACM Trans. Graph., 2023.
Daly, J. J. and Wolpaw, J. R. Brain-computer interfaces in neurological rehabilitation. The Lancet Neurology, 7(11) 1032-1043, 2008
Allen, EJJ, S.-Y. G. W. Y. a. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. pp. 116.126, 2022.
Dhariwal, and Nichol, A. Q. Diffusion models beat gans on image synthesis. In NeurIPS, pp. 8780-8794, 2021.
Avrahami, O., Lischinski, D., and Fried, O. Blended diffu- sion for text-driven editing of natural images. In CVPR, pp. 18187-18197. IEEE, 2022.
Ding, G., Zhao, C., Wang, W., Yang. Z., Liu, Z., Chen, H.. and Shen, C. Freecustom: Tuning-free customized image generation for multi-concept composition. In CVPR, pp. 9089-9098. IEFF, 2024.
Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., and Lischinski, D. Break-a-scene: Extracting multiple con- cepts from single image. In SIGGRAPH Asia, pp. 96:1- 96: 12. ACM, 2023.
Evans, J. D. Straightforward statistics for the URL https:
behavioral sciences. 1995. / //api. www.steesmccoooooeee..... 121 121954573. www.sccho.a...oooooouossslalllla
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix:
10

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR. OpenReview.net, 2023a.
Horikawa, T. and Kamitani, Y. Generic decoding of seen and imagined objects using hierarchical visual features. CoRR, abs/15 10..06479, 2015.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang. H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models. In CVPR, pp. 6007-6017. IEEE, 2023.
Gal, R., Arar, M., Atzmon, Y., Bermano, A. H., Chechik, G., and Cohen-Or, D. Encoder- based domain tuning for fast personalization of text-to-image models. ACM Trans. 2023b.
Graph.,
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In NIPS, pp. 1 1106-1114, 2012.
Gal, R., Arar, M., Atzmon, Y., Bermano, A. H., Chechik, G., and Cohen-Or, D. Encoder- based domain tuning for fast personalization text-to-image models. ACM Trans. Graph., (144::55:11150:11 2023c.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J. Multi-concept customization of text-to-image diffusion. In CVPR, pp. 1931-1941 IEEE, 2023.
Gao, J., Fu, Y., Wang. Y. Qian, X., Feng. J., and Fu, Y. fmri- 3d: A comprehensive dataset for enhancing masi-based 3d reconstruction. CoRR, abs/2409.1 1315, 2024.
Lang, O., Gandelsman, Y., Yarom, M., Wald, Y., Elidan, G., Hassidim, A., Freeman, W. T., Isola, P., Globerson, A., Irani, M., and Mosseri, I. Explaining in style: Training a GAN to explain classifier stylespace. In ICCV, pp. 673-682 IEEE, 2021.
Gong, Z., Bao, G., Zhang, Q., Wan, Z., Miao, D., Wang, S., Zhu, L., Wang, C., Xu, R., Hu, L., Liu, K., and Zhang, Y. Neuroclips: Towards high-fidellts and smooth firi-to- video reconstruction. CoRR, abs/2410. 19452, 2024.
Li, D., Li, J., and Hoi, S. C. H. Blip-diffusion: Pre-trained subject representation for controllable text-to-image gen- eration and editing. NeurIPS, 2023a.
Gu, Y., Wang, X., Wu, J. Z., Shi, Y., Chen, Y., Fan, Z., Xiao, W., Zhao, R., Chang, S., Wu, W., Ge, Y., Shan, Y., and Shou, M. Z. Mix-of-show: Decentralized low-rank adaptation for multi contii- customization of diffusion models. In NeurlPS, 2023a.
D., Savarese, S., and Hoi, S. C. H. BLIP-2: boot- strapping language-image pre- pre-trainn traininiin with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023b.
Gu, V., Jamison, K., Kuceyeski, A., and Sabuncu, M. R. Decoding natural image stimuli from fmri data with a surface-based convolutional network. MIDL, volume 227 of Proceedings of Machine Learning Research, pp- 107-118. PMLR, 2023b.
Lin, T., Maire. M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: common objects in context. In ECCV (5), volume 8693 of Lecture Notes in Computer Science, pp. 750-755, Springer, 2014.
Hari, R. and Lounasmaa, O. V. Recording and interpretation of cerebral magnetic fields. Science, 244(4903):432-4777 1989.
Liu, X., Park, D. H., Azadi, S., Zhang, G., Chopikyan, A., Hu, Y.. Shi, H., Rohrbach, A., and Darrell, T. More control for free! image synthesis with semantic diffusion guidance. In WACV, pp. 289-299, IEEE, 2023a.
Härkönen, E., Hertzmann, A., Lehtinen, J., and Paris, S. Ganspace: Discovering interpretable GAN controls. In NeurlPS, 2020.
Liu, Z., Feng R., Zhu, K., Zhang, Y., Zheng, K., Liu, Y.. Zhao, D.,, Zhou, J., and Cao, Y. Cones: Concept neu- rons in diffusion models for customized generation. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 21548 21548-21561 21566. PMLR, 2023b.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-prompt image editing with cross cooss-attention control. In ICLR. OpenReview.net, 2023a.
Liu, Z., Zhang, Y., Shen, Y., Zheng, K., Zhu, K., Feng, R., Liu, Y., Zhao, D., Zhou, J., and Cao, Y. Cones Cus- tomizable image synthesis with multiple subjects. CoRR, 02s/2305.19322 2023c.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-prome image editing with cross-attention control. In ICLR. OpenReview.net, 2023b.
Mokady, R., Hertz, A., and Bermano, A. H. Clipcap: CLIP prefix for image captioning. CoRR, abs/2111.09734, 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, 2020.

---

MindCustomer; Multi-Context Image Generation Blended with Brain Signal
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Barnes, N., and Mian, A. A com- prehensive overview of large language models. CoRR, abs/2307.06433. 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.06125, 2022b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10674-10685. IEEE, 2022.
Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. GLIDE: towards photorealistic image generation and edit- ing with text-guided diffusion models. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804, PMLR, 2022.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pp. 22500-22510. IEEE, 2023.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Schoenmakers, S., Barth, M., Heskes, T., and van Gerven, M. Linear reconstruction of perceived images from hu- man brain activity. Neurolmage, 83:951-961, 20133
Oquab, M., Darcet, T., Moutakanni, T. Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M., Sharma, V. Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learn ing robust visual features without supervision. Trans. Mach. Learn. Res., 2024, 2024.
Scotti, P., Banerjee, A., Goode, J., Shahalin, S., Nguyen, A., cohen, c., Dempster, A., Verlinde, N., Yundler, E., Weis- berg, D., Norman, K., and Abraham, T. Reconstructing the mind's ' eye: fmri-to-image with contrastive learning and diffusion priors. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2023.
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, pp. 2065-2074. IEEE, 2021.
Scotti, P. S., Tripathy, M., Torrico, C., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K. A., and Abraham, T. M. Mindeye2: Shared- subject models enable imri-to-image with hour of data. In ICML. OpenRevie,.net, 2024.
Quan, R., Wang, W., Tian, Z., Ma, F., and Yang. Y. Psy- chometry: An omnifit model for image reconstruction from human brain activity. CoRR, abs/2403.20022,, 2024.
Shen, G., Zhao, D., He, X., Feng, L., Dong, Y., Wang, J., Zhang, Q., and Zeng, Y. Neuro-vision to language: Image reconstruction and language enabled interaction via brain recordings. CoRR, abs/2404. 194324 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021a.
Shen, Y. and Zhou, B. Closed-form factorization of latent semantics in gans. In CVPR, pp. 1532-1540. Computer Vision Foundation / IEEE, 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021b.
Shen, Y., Gu, J., Tang, X., and Zhou, B. Interpreting the latent space gans for semantic face editing. In CVPR, pp. 9240-9249. Computer Vision Foundation IEEE, 2020.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. In ICLR. OpenReview.net, 2021.
Radford, A., Kim, J. W., Hallacy. C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp.
Sun, Y., Chen, X., Liu, B., Liang, L., Wang, Y., Gao, S., and Gao, X. Signal acquisition of brain-compute interfaces: A medical-engineering crossover perspec- tive review. Fundamental Research, 2024. ISSN 2667-3258. doi:
PMLR, 2021c.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with CLIP latents. CoRR, abwp/204.00000 05,,6,11622002 2022a.
011. URL.ttpss///ww.eccenaddcccce com/ octeene/errtdde///////2222222222
12

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In CVPR, pp. 2818-2826. IEEE Computer Society, 2016.
Yu, M., Lin, S., Yan, H., and Ma, K. Mindpainter: Efficient brain-conditioned painting of natural images via cross- modal self-supervised learning. In AAAI, pp. 14468- 14476. AAAI Press, 2025.
Taheri, B. A., Knight, R. T., and Smith, R. L. A dry electrode for eeg recording. Electreencehhalggrppy and clinical neurophysiology. 90(5):76-3383, 1994.
Takagi, Y. and Nishimoto, S. High-resolution image recon- struction with latent diffusion models from human brain activity. In CVPR, pp. 14453-14463. IEEE, 2023.
Tewel, Y. Gal, R., Chechik, G., and Atzmon, Y. Key-locked rank one editing for text-to-image personalization. In SIGGRAPH (Conference Paper Track), pp. 12:1-12:11. ACM, 2023a.
Tewel, Y., Gal, R., Chechik, G., and Atzmon, Y. Key-locked rank one editing for text-to-image personalization. In SIGGRAPH (Conference Paper Track), pp. 12:1-12:11. ACM, 2023b.
VanRullen, R. and Reddy, L.. Reconstructing faces from fmnr patterns using deep generative neural networks. CoRR, abs/1810.03856, 2018.
Wang, J., Chan, K. K., and Loy, C. C. Exploring CLIP for assessing the look and feel of images. In AAAI, pp. 2555-2563. AAAI Press, 2023.
Wang, S., Liu, S., Tan, Z., and Wang, X. Mindbridge: A cross-subject brain decoding framework. In CVPR, pp. 11333-11342. IEEE, 2024.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4): 600-612, 2004.
Wasserman, N., Rotstein, N., Ganz, R., and Kimmel, R. Paint by inpaint: Learning to add image objects by re- moving them first. CoRR, abs/2404. 18212, 2024.
Weiskopf, N. Real-time fmri and its application to neuro- feedback. Neuroimage, 62(2):682 692, 2012.
Xia, W., de Charette, R., Öztireli, A. C., and Xue, J. UM- BRAE: unified multimodal brain decoding. In ECCV (7), volume 15065 of Lecture Notes in Computer Science, pp. 242-259. Springer, 2024.
Xu, X., Wang. Z., Zhang, E. J., Wang. K., and Shi, H. Versatile diffusion: Text, images and variations all in one diffusion model. In ICCV, pp. 7720-7731. IEEE, 2023.
Yang. B., Gu, S., Zhang, B., Zhang. T., Chen, X., Sun, X., Chen, D., and Wen, F. Paint by example: Exemplar- based image editing with diffusion models. In CVPR, pp. 18381-18391. IEEF, 2023.
13

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Appendix
*Note that all the appendix figures are at the end of the text.
A. More Related Works
Apart from the image customization, blending and composition introduced in the main paper, here describe other related works editing task. Image editing applies more precise control to the source image, allowing users slightly modify image contents according to external instructions. Many works perform image manipulation based on GAN models (Abdal et al., 2021; Härkönen et al., 2020; Lang et al., 2021; Patashnik et al.. 2021; Shen et al., 2020; Shen & Zhou, 2021) to edit images with contents and styles. Recently, the success of the diffusion model (Rombach et al., 2022; Xu et al., 2023) and large-scale pre-trained CLIP model (Radford et al., 2021b) also improves the development of editing (Avrahami et al., 2022; Hertz et al., 2023b; Ramesh et al., 2022b). Several methods integrate text conditions into the denoising process of latent diffusion to alter the resulting images (Brooks al., 2023; Kawar et al., 2023). Liu et al. (Liu al., 2023a) trained diffusion models to generate images that approximate the text condition, enabling edits the source images. Hertz et al. (Hertz et al., 2023a) enhanced cross- tross-attention mechanisms to more accurately fuse the source image with the condition, allowing for more precise modifications. Concurrently, research on image inpainting (Yang et al., 2023; Wasserman et al., 2024) has focused on utilizing manually added masks to incorporate text conditions, facilitating targeted image editing. In all, image editing focuses on more specific modifications to the reference image while customization and blending mainly deal with context integration.
B. Implementation
We utilize the training set of NSD data for the Brain Representation Pre-training, and use the test set as the brain context for generation. We train the subject-wise IBT with a learning rate of 56 for 200 epochs and then adopt Adam W schedule for our Brain Embedder with a learning rate of 3e for 600 epochs. MindCustomer is implemented with the publicly available versatile diffusion. The model supports multi-modal generation with multiple flows in the latent space (of size x 64 x 64), accompanied by VAE encoders and decoders for image and text. We fine-tune the VD for 200 epochs with learning rate of 5e-8 utilizing Adam schedule. Then optimize the brain embedding for 100 epochs with learning rate of 1e-5, 5, We present the intuitively detailed setting in Table 5. The time of one image generation is about 6 minutes on single Tesla A100 GPU. The efficiency enables us to perform single-image real-time creation for MindCustomer.
Table 5. Training parameters of MindCustomer
Optimizer
Learning rate
Weight decay
Training epochs
Batch size
LR scheduler
IBT
AdamW W
5c
1e-2
200
50
OneCycleLR
Brain Representation
AdamW
-3
1e-2
600
50
OneCycleLR
VD Finetune
Adam
5c
0
200
1
None
Embedding Opt
Adam
le 5
100
None
C. Details of User Study
We randomly selected 1 110 pairs of comparison images, including ours and the baseline, and divided them into 11 groups A total of 22 participants were involved, and to minimize bias caused by individual preferences, we randomly assigned every two participants to the same group, ensuring that each image received ratings from two different individuals. By comparing ours with the baseline, participants are required to score each generated image based on the quality and consistency from 1 to 3, with 3 being better. Ultimately, we collected 220 answers for the 110 randomly selected image pairs from the 22 participants. The results broadly reflect the participants' consistent preference for our approach.
14

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Table Pearson correlation coefficient and significance analysis on brain simulation of IBT with the ground truth. The range of Pearson and significance coefficient a is [-1, 1J and [0. 1J.
Subj1
Subj2
Subj5
Subj7
Avg
Pearson
0.389
0.423
0.502
0.389
0.426
1 1 5.73 x 10
1 1.170 x 10 12
7.370 x 10 19
199.250 10
0.999
D. More Experiments
D.I. Additional Results Multi-Conters Generation
In Figure 11, we provide more results of multi-context generation across different subjects. And in Figure 12, illustrate more few-shot generation results on Subj1.
D.2. Different Seeds for Generation
To demonstrate the robustness of MindCustomer, we present generation results across different random seeds in Figure 13, Our method is capable of producing images of consistently high quality, with minor variations. We also recommend that users experiment with multiple random seeds to obtain results that best suit their preferences.
D.3. Discussion Embedding Integration
In this paper, employ two different methods of embedding fusion: interpolation and concatenation. In the triplet contexts generation, we apply linear interpolation to the image and brain contexts to prevent exceeding the dimensional capacity of the VD model. In the case of dual contexts, we directly concatenate the two. To demonstrate the impact of these two fusion methods on generation quality, we input identical image and brain contexts with both fusion strategies in the absence of text context. In Figure 14, both methods successfully merge the information in a natural manner. However, we observe that, compared to concatenation, linear interpolation slightly alters the original semantic representations of the two modalities in the latent space, resulting in generated images with somewhat reduced fidelity and details. We speculate that this issue may be addressed through more sophisticated mechanisms, such cross- atteettttnn Additionally, optimizing this process with the currently limited brain data presents significant challenge. Nevertheless, fusion techniques, satisfactory results can be achieved. As research on neural data continues to evolve, we anticipate the development of more effective mechanisms to generate even higher-quality images.
method shows that even with simple
D.4. Text Context for Enhanced Generation
In Figure 15, we compare the impact of including versus excluding the text context generation results using the same image and brain contexts. It is evident that the text context provides additional semantic information or more precise interaction control between the image and brain contexts. This empowers MindCustomer to better meet user needs, enabling more accurate, richer, and higher-quality image generation.
D.5. Effectiveness of Image-Brain Tiiiint-rnnltoor
We conduct experiments to demonstrate the effectiveness of IBT in simulating pseudo-brain signals. First, we simulate brain signals for visual stimulus images from the test set and quantitatively assess them against ground truths (GT). Following standard practices in neuroscience for brain signal simulation, we measure the similarity between the simulated and GT brain signals using the Pearson correlation coefficient. In Table the average Pearson correlation coefficient across the four subjects is 0.426 (ranging from -1 to 1). According to relevant literature (Chee, 2015; Evans, 1995), it indicates a moderate positive correlation, meanwhile with a high confidence level of 99%. The results show that IBT can approximate the brain signal waves corresponding to the input images, providing a foundational prior for natural cross-modal fusion. Additionally, we visualize the brain signal simulations for four human subjects in Figure 16, further validating the method's effectiveness. Moreover, we transfer the brain signals derived from the images and directly fed them in the diffusion model to explicitly showease the semantic accuracy of the simulated brain signals in Figure 17.
15

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Table Ablations brain decoding. We compare our methad with the open-sourced SOTA methods. For fair comparison, all methods are trained on four subjects from NSD with only one generative model.
Method
PixCorrt
SSIM
AlexNet(2)
AlexNet(5y)
Inception
CLIP
MindBridge (Wang et al., 2024)
.151
.263
87.7%
95.5%
92.4%
94.7%
UMBRAE (Xia et al., 2024)
.283
.328
93.9%
96.7%
91.7%
93.5%
Ours
.157
.289
88.3%
95.5%
92.1% 1%
94.5%
wo IBT
.152
.272
87.9%
95.2%
91.5%
94.1%
D.6. Additional Results on Brain Decoding
Table shows the reconstruction performance, which highlights the impact of our Brain Representation Pre-training, where reconstructing brain signals with IBT improves cross-subject embeddings and achieves competitive results with SOTA methods (Wang et al., 2024; Xia et al., 2024). In Figure 18 & 19 & 20, we conduct more brain reconstruction comparisons with another SOTA MindBridge (Wang et al., 2024) and illustrations to demonstrate the effectiveness of our Brain Representation Pre-training. Morcover, we further illustrate the few-shot reconstruction capability of MindCustomerr as shown in Figure 20, where our method can also precisely reconstruct the visual stimuli under limited data.
E. Discussion and Limitation
MindCustomer utilizes visual brain signals as one of the key information sources in multi-context image generation. Here we discuss our approach with traditional methods (Alaluf et al., 2023; Cao et al., 2023; Chen et al., 2024b; Gal et al., 2023b; Tewel et al., 2023b) that rely solely on image and text modalities three aspects. (1) Data: Brain signal acquisition directly and implicitly reflects the user's intentions, without the need for additional modality conversion. In contrast, traditional methods often require users to explicitly convey their ideas through images or text, which may introduce deviations and complexities. (2) Technology: Traditional methods can directly embed prompts with the assistance of large-scale pre-trained models, such as CLIP and diffusion models. In contrast, brain-controlled image customization presents more challenges, requiring precise embedding of brain signals and bridging the implicit gaps between modalities to achieve naturally blended results. (3) Result: Since text and images can provide fine-grate mne-grained semantic information, image generation (e.g. customization, editing, manipulation) based on these modalities can achieve more detailed alignment. However, current brain signals primarily record coarse information due to noise or disturbance, making them more suitable for global context blending but potentially insufficient for handling detailed features or specific instructions.
Besides, prior works that focus on better decoding brain signals themselves (Scotti al., 2023; 2024; Wang al., 2024; Xia et al., 2024) and utilize neural data for image stylization (Chen et al., 2024a) or painting (Yu et al., 2025) have made tremendous progress in brain analysis and applications. Different from these works, MindCustomer further researches how to fuse brain signals with other contexts in image and text modalities to mask-freely create naturally blended images. Our experimental results have proven that MindCustomer is capable of effectively extracting brain signal semantics, and more importantly, it has demonstrated excellent performance in the personalized blending task. Therefore, we believe that previous works and MindCustomer can mutually inspire and complement each other, sparking more interesting and meaningful research and applications in the field of brain-computer interaction.
The limitation of this work primarily lies in current brain signal acquisition and analysis, which requires further development. This includes improving the quality (e.g. Signal to Noise Ratio) and diversity (e.g. type, content) of neural data, and expanding data samples, as these factors currently constrain our precision and richness of personalized generation. Further, we believe that the collection of brain imagery data specifically for brain-controlled generation tasks may inject new momentum into the development of this field. In the future, we will also explore and acquire these data. As brain- interface technology continues to evolve, contolcod creative research will become increasingly sophisticated. We hope MindCustomer provides critical foundation for advancing this emerging area of research.
16

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Image
Brain
MindCustomer
Image
Brain
MindCustomer
"with sunglasses"
"riding"
"outside"
"made of"
"turn nose appearance"
"Eiffel Tower in the behind"
Figure 11. More brain blin-blended multi-context generation results.
17

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Image
Brain
10%
20%
40%
10
: 10
"evening sky with grand nebula"
"Eiffel Tower in the behind"
Figure 12. More few-shot generation results on Subj1. We only use 10%, 20%, and 40% of the subject's 's complete training data for few-shot learning.
18

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Image
Brain
MindCustomer
"play"
Figure 13. Different seeds for generation.
Image
Brain
Interpolation
concatenate
Figure 14, Comparison on the methods brain and image embedding integration: interpolation and concatenation.
19

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Image
Brain
Dual Context Gen
Triplet Context Gen
"play"
Loduln
H
"made of"
"in the snow"
2,7
"bus in the background"
Figure 15. Generation results / contexts. It demonstrates that the text context provides additional semantic information precise control over the image and brain contexts.
20

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Value
Voxel Num
Subject1
Subject2
Subject5
Subject7
Prediction
wiee
Ground Truth
de M (1): Avg Diff: 0.25
2000
(1): Avg Diff: 0.2
(1): Avg Diff: 0.1
(1): Avg DifE: 0.1
Prediction
Mmn
Ground Truth
Maly
(2): Avg Diff: 0.2
(2): Avg Diff: 0.2
(2): Avg Diff: 0.15
(2): Avg Diff: 0.15
Prediction
1009 200 201
2000 1000
201
Ground
Truth
(3): Avg Diff: 0.15
(3): Avg Diff: 0.15
(3): Avg Diff: 0.15
(3): Avg Diff: 0.15
Figure Visualization of the predicted brain voxel by IBT with its ground truth (GT). It illustrates voxels brain signals with their corresponding fcature values. Here we randomly choose threc examples for each subject, and report the difference error between the prediction and the ground truth. We simply compute absolute difference value for each voxel, and then take the average of the absolute differences across all voxels, symbolized as Avg Diff. Note that the range of Avg Diff is [0, 2]. lower is better. It is observed that the subject- wise can effectively simulate brain signals with low differences to GT. facilitates the transferring accuracy image features to brain modality in our method.
21

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Visual Stimuli
IBT Subj1
IBT Subj2
IBT Subj5
IBT Subj7
Avg Diff: 0.2
Avg Diff: 0.3
Avg Diff: 0.15
Avg Diff: 0.1
Avg Diff: 0.15
Avg Diff: 0.2
Avg Diff: 0.2
Avg Diff: 0.25
Avg Diff: 0.15
Avg Diff: 0.15
Avg Diff: 0.3
Avg Diff: 0.2
Avg Diff: 0.3
Avg Diff: 0.2
Avg Diff: 0.1
Avg Diff: 0.2
Avg Diff: 0.3
Avg Diff: 0.2
Avg Diff: 0.2
Avg Diff: 0.3
Figure 17. We utilize the predicted brain voxels of IBT for direct reconstruction to explicitly demonstrate the effectiveness of accurate semantics. Same with Figure 16, here also report the Avg Diff of brain voxels between the prediction and the ground truth.
22

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
MindBridge
MindCustomer
Visual Stimuli
Subj1
Subj2
Subj5
Subj7
Subj1
Subj2
Subj5
Subj7
Figure Comparison on the cross-subject brain reconstruction. We compare our method with the SOTA MindBridge (Wang et al., 2024)
to show its competitive performance of MindCustomer on the brain decoding task.
MindCustomer
Visual Stimuli
Subj1
Subj2
Subj5
Subj7
301
Figure 19, More cross-subject brain reconstruction results of MindCustomer.
23

---

MindCustomer: Multi-Context Image Generation Blended with Brain Signal
Visual Stimuli
10%
20%
40%
100%
Figure 20. Few-shot brain reconstruction results on Subjl. Compared to full training (100%), ), our method can also roughly decode the hrain signals with limited data. No then 40%, MindCustomer is already capable of precise reconstruction.
24


Reviews:
[{"review_id": "jzspqOv6ZJ", "reviewer": "Reviewer_4Rs7", "summary": "This paper proposes MindCustomer, an image generation method with input of image, text and brain signals. The brain signals is converted by Image-Brain Translator(IBT) into image embeding space for subsequent text-image joint generation. The diffusion model is finetuned and the embeddings is optimized to achieve more plausible visual results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your reviews!\n\n***Rebuttal Link*** (https://anonymous.4open.science/r/ICML2025-Rebuttal-MindCustomer) is for supplemented results in rebuttal, please refer to it.\n\n**W1: Comments on contributions**\n- Diffusion model fine-tuning and linear interpolation are well-established techniques widely used in many studies. While we also employ these techniques in our work, they are **not** the core contributions we claim in this paper.\n\n- Our main contributions first lie in **as the first work, MindCustomer proposed and successfully achieved the personalized image creation task by integrating brain signals with other traditional modalities**. \n\n- Second, we need to clarify that the designed **IBT, a model that simulates pseudo-brain signals from input images (please note that it is not a brain signal encoding model described in the review**: \"The brain signals is converted by Image-Brain Translator(IBT) into image embeding space for subsequent text-image joint generation\", \"The brain representation pre-training (IBT)\", \"The introduction of ClipCap in IBT\"). **IBT simulates brain signals from images to enhance subsequent brain signal semantic encoding and reduce the modal gap in the following cross-modal fusion pipeline, which is independent of MindBridge**. Specifically, based on the proposed IBT, we simulate subject-wise fMRI for each visual stimulus. In this way, we can augment the fMRI training dataset due to the stimulus for each subject is different from the original dataset. Then during brain representation pre-training, we simultaneously feed different subjects' fMRI of one stimulus for shared learning in one encoding model. **The above process is different from MindBridge**. \n\n- Third, more importantly, we propose **an effective cross-modal information fusion pipeline**. In order to blend image context with brain context, we designed VD fine-tuning based on image context to learn the image semantics, and then we froze the fine-tuned VD model and lightly optimized brain contexts with the target of image context. **This helps mitigate semantic conflicts between different contexts**. Besides, during the above process, we further utilize the proposed IBT and brain embedding model to transfer image context to brain-embedding space to **reduce the gap between different modals**. Thus, the above-designed brain-blended cross-modal fusion enables final natural integration and semantic preservation across modalities, producing natural and high-quality blending results.\n\nThe above clarification can also be referred to the submitted paper (line 96, left - line 68, right). Therefore, **it is vital to highlight that these are our core contributions**, which fundamentally exceed simple VD fine-tuning or linear interpolation.\n\n**W2: Ablation study of ClipCap**\n\nIn the ***Rebuttal Link*** (**Rebuttal Figure 1 & Table 1**), we include an ablation study of ClipCap. The blending results without ClipCap show lower semantic details with complete MindCustomer. Both the qualitative and quantitative results demonstrate the role of the introduced ClipCap in enhanced brain semantic extraction. Adding this experiment makes our study more comprehensive. Additionally, please note that **we have also conducted detailed numerical and visual ablation studies on the proposed *core* techniques (e.g. IBT, cross-modal information fusion pipeline) in the submitted paper**.\n\n**W3: Clarity**\n\n- $\\eta$ in $G_{\\eta}$ refers to the parameters of the IBT model $G$. Thank you for pointing out. We will add its definition.\n- **We have defined $e_{p}$ in the paper (line 256) before using it in equation (8)**,  \"We feed the transferred image context $Bp$ into brain embedder $\\epsilon$ to obtain the embeddings $e_{p}$.\" We will also appropriately add a further description of it in the subsequent text.\n\n**Others: More results with only different text descriptions**\n\nIn the ***Rebuttal Link*** (**Rebuttal Figure 2**), we present more visual results with only different text contexts. As we can see, MindCustomer robustly creates multi-context images that are content-consistent and naturally integrated.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "The paper builds upon previous work in brain decoding and image generation, particularly leveraging advancements in models like MindBridge.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "## Strengths\n\n1. The integration of brain signals for image editing shows successful results, surpassing baseline methods.\n\n\n## Weaknesses\n\n\n1. The contribution is limited to trivial fine-tuning of Versatile Diffusion and linear interpolation of two embeddings. The brain representation pre-training (IBT) and new-subject few-shot generation are largely derived from MindBridge, making the paper limited in technical contribution.\n2. The introduction of ClipCap in IBT is not validated in ablation studies, making its actual contribution unclear.\n3. Clarity issues in the description:\n* The term \"$\\eta$\" in equation (1) is not defined.\n* The term \"$e_p$\" is not explained in equation (8).", "other_comments_or_suggestions": "Do you have more visual results with only different text descriptions?", "questions_for_authors": "Please refer to weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W92A6FMXKF", "reviewer": "Reviewer_FgVa", "summary": "This paper proposes a novel framework, MindCustomer, to explore the blending of visual brain signals in multi-context image generation. This approach enables cross-subject generation, delivering unified, high-quality,  and natural image generation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your reviews!\n\n***Rebuttal Link*** (https://anonymous.4open.science/r/ICML2025-Rebuttal-MindCustomer) is for supplemented results in rebuttal, please refer to it.\n\n**Q1: Claim of \"potential for practical application\"**\n\n- **As the first** to propose and achieve the fusion of fMRI, image, and text for customized creation, MindCustomer has demonstrated success on **the large-scale brain dataset NSD**. \n\n- We have discussed and explained this claim of potential application in detail in our **Discussion and Limitation section (Appendix D)**, please refer to it. **Constrained by the types of existing brain signal datasets**, we think that MindCustomer presents the potential application **in future real-world scenarios**.\n\n- Additionally, although constrained by the current types of brain signal data, we have also explored **few-shot generation for new subject in this paper**, simulating the effect of personalized generation when there is a limited amount of brain data in real-world scenarios. \n\nIn all, we believe that MindCustomer is an important foundational exploration, and in the future, more real-world data collection will be involved to promote the application of MindCustomer. \n\n**Q2: Reference to MindEye**\n\n- Regarding the performance of the fMRI reconstruction task, it is slightly lower compared to MindEye. We believe the reasons lie in the fact that the MindEye series used four more subjects' data for training (MindEye2) and employed additional information, such as retrieval, brain captions, and more diffusion models, to better achieve brain reconstruction tasks. In contrast, **for MindCustomer's multi-context blending task, this additional information was not needed**. As such, a direct comparison between the two **may not be fair**. Meanwhile, we **have cited and analyzed a series of brain reconstruction works, including MindEye, in our paper (line 71 in intro and line 164 in related work)**. We not only acknowledge and appreciate their contributions to brain signal decoding but also analyze the differences between our work and these previous studies in terms of tasks and challenges. Unlike prior works that focus on better decoding brain signals themselves, the core of our work lies in how to fuse brain signals with various cross-modal information to create naturally composed images. **Besides, we will also include more performance discussion of the MindEye series of work in the final version to highlight respective advantages.**\n\n- Nevertheless, we would like to emphasize that, firstly, **we have fairly compared MindCustomer with several SOTA methods in the brain reconstruction field**. MindCustomer demonstrated strong competitiveness. Secondly, as for the task and challenges in this paper, it is not only to have a model with a certain level of brain semantic encoding ability but **more importantly**, how to integrate information from different modalities to generate natural images that preserve semantics. Our experimental results have proven that MindCustomer **is capable of effectively extracting brain signal semantics**, and **more importantly, it has demonstrated excellent performance in the personalized blending task**. Therefore, we believe that MindEye indeed makes an important contribution to the field of fMRI reconstruction, while MindCustomer also has its own merits in personalized multimodal fusion creation. We will further clarify the relation in final version.\n\n**Q3：Figure 6 question**\n- Although images may contain richer semantics, we observe that inputting two images into versatile diffusion (VD), a mature image fusion method for blended generation, often causes **semantic overlap**, making it hard to preserve distinct semantics. To address this, we designed VD fine-tuning based on image context to learn the image semantics, and more importantly, we freeze the fine-tuned VD model, and lightly optimize brain contexts with the target of image context. This helps mitigate semantic conflicts between different image and brain contexts. During the above process, we further utilize the proposed IBT to transfer image context to brain-embedding space to reduce the gap between modals. Thus, **the above-designed brain-blended cross-modal fusion** allows the final multimodal blending to better preserve each modality's semantics and produce more accurate, high-quality results. We also have explained this in the paper **(line 353, right)**: “This is likely ... ”, please refer to it.\n\n- Therefore, as for the method design of MindCustomer, we **innovatively propose IBT to simulate the process of visual stimuli being transformed into brain signals**. This not only expands brain signal data and enhances encoding capability but also **provides the above solution of brain-blended cross-modal fusion**. Overall, MindCustomer introduces well-designed modules and a structured pipeline tailored to the task, making it a simple yet effective approach rather than a naive combination.", "claims_and_evidence": "1. This paper claims some potential for practical application. But there are no related applications introduced. It will be better that authors are able to further illustrate some specific real-world applications.\n\n2. Part of the contribution in this paper is summarized as the introduction of Image-Brain Translator (IBT). And authors provide some qualitative and quantitative results. But there are missing comparisons with other strategies handling the multi-identities generality such as MindEye2.\n\nReference: \nMindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data", "methods_and_evaluation_criteria": "The proposed approach is intuitive but the overall methodology is kind of combinational and has limited novelty. The evaluation criteria follows previous work and is reasonable.", "theoretical_claims": "There are no theoretical claims involved in this paper.", "experimental_designs_or_analyses": "1. About the figure 6, why is the proposed approach able to achieve superior performance over the baseline? Theoretically speaking, the image signal is supposed to capture more visualization information than the brain signal. \n\n2. For the brain decoding, how is performance compared with alternative designs such as MindEye1 ?\n\n\nMindEye1: Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Prior", "relation_to_broader_scientific_literature": "It is a cross-domain work that combines both brain-to-image study and image generation field research. Both fields attract significant attention in the research field.", "essential_references_not_discussed": "It will be better if authors can also include the MindEye series of work. \n\nMindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data\nMindEye1: Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Prior", "other_strengths_and_weaknesses": "N.A", "other_comments_or_suggestions": "N.A", "questions_for_authors": "Please kindly resolve the issues in Claims and Evidence sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0bPI7SqvK9", "reviewer": "Reviewer_2Q5v", "summary": "This paper proposes a new task: image customization with brain signals, which is novel and interesting. The authors introduce the image-brain translator and brain embedded to align various modalities, including images, fMRI, and texts, together for generating new images with a versatile diffusion model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your reviews!\n\n***Rebuttal Link*** (https://anonymous.4open.science/r/ICML2025-Rebuttal-MindCustomer) is for supplemented results in rebuttal, please refer to it.\n\n**Claims And Evidence: Reference error**\n\nThank you for pointing out the citation error. It should be: Takagi et al.[1] reconstruct high-resolution images from brain activity while preserving rich semantic details, without the need for training or fine-tuning complex deep generative models. Mind-Vis[2] utilizes Masked Autoencoders (MAE) to enhance brain signal encoding, enabling more accurate reconstructions. We will correct this citation error.\n\n[1] High-resolution image reconstruction with latent diffusion models from human brain activity\n\n[2] Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding\n\n**Other Strengths And Weaknesses1: Choice of adaptive max pooling**\n\n- Beyond unifying brain signals across subjects in scales, we believe adaptive max pooling helps mitigate the issue of sparse brain signal sampling, preserving critical semantic information. Additionally, reference MindBridge[3] shows that max-pooling provides better brain signal aggregation compared to average pooling or interpolation. We will further clarify this in our paper.\n\n- Besides, in the discussion and future work sections, we will acknowledge that this is not necessarily the optimal approach. In neuroscience, effective brain signal sampling requires a comprehensive analysis of cross-subject variations in numerical values, regions, and data quantity. We will explore the strengths and limitations of different sampling methods and provide further insights. We see this as a valuable direction for future work to design more effective sampling strategies.\n\n[3] MindBridge: A Cross-Subject Brain Decoding Framework\n\n**Other Strengths And Weaknesses2: The reason for original baseline design and results of recommended baseline**\n- The reason for us to choose the visual stimuli as the input rather than the fMRI-reconstructed image is that the images reconstructed via fMRI-to-image methods generally have lower semantic quality than the original visual stimuli. Then, using reconstructed images as inputs for blending would lead to weaker performance. Therefore, we chose to use the **higher-quality original visual stimuli** as the fusion input and set this as **a stronger baseline in our paper** to better demonstrate the effectiveness of our method.\n\n- Additionally, we also illustrate comparison results for using the output of a SOTA fMRI-to-image model (MindEye[4]) as input for another mature image-blending model (Versatile Diffusion[5]) as a baseline (advised in review). As shown in the ***Rebuttal Link*** (**Rebuttal Figure 3 & Table 2**), we conduct both qualitative and quantitative comparisons:  Due to the brain signal interpretation error caused by fMRI reconstruction and some semantic overlap between dual image contexts resulting from the blending diffusion model, this baseline shows some semantic deviation and loss. In contrast, the proposed **effective cross-modal fusing pipeline with the designed IBT model** addresses these issues. MindCustomer not only successfully blends different contexts in nature but also preserves semantics, thus generating high-quality results. **We also explained this in the submitted paper (Experimental analysis: line 350-357 (right), Intro: line 091-095 & 058-065 (right)), please refer to them.** We will also further supplement the above results and emphasize the analysis in the final version. \n\n[4] Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors\n\n[5] Versatile Diffusion: Text, Images and Variations All in One Diffusion Model\n\n**Others: More details of the user study**\n\nIn addition to the description of the User Study in the paper, we will provide more details about the study design in the Appendix. We randomly selected 110 pairs of comparison images, including ours and the baseline, and divided them into 11 groups. A total of 22 participants were involved, and to minimize bias caused by individual preferences, we randomly assigned every two participants to the same group, ensuring that each image received ratings from two different individuals. By comparing ours with the baseline, participants are required to score on each generated image based on the quality and consistency from 1 to 3, higher is better. Ultimately, we collected 220 answers for the 110 randomly selected image pairs from the 22 participants. The results broadly reflect the participants' consistent preference for our approach.", "claims_and_evidence": "Claims are reasonable and supported by references and evidence. \n\nThough, one reference mistake on ln139: \"Additionally, Takagi. etc (Takagi & Nishimoto, 2023) employs Masked Autoencoders (MAE) to improve the encoding of brain signals, enabling more accurate reconstructions.\" Takagi & Nishimoto did not use MAE to encode brain signals in the mentioned reference. Please double-check the reference.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental designs and analysis make sense.", "relation_to_broader_scientific_literature": "The proposed task is new in this area, which is interesting and could have an impact on the future application of the brain-computer interface.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I think the quality of this paper is good in general. It presents an interesting new task with careful design for multimodal alignment. However, I have the following concerns.\n\n- On ln185, the authors mentioned using adaptive max pooling to achieve the subject-invariant voxel sizes. However, various subjects have quite different activation regions even when viewing the same stimuli. Using adaptive max pooling as mentioned would lose or change the original position information of voxels. Therefore, this choice should be further discussed and justified. \n\n- The baseline in Section 4.2 is not convincing enough. Ideally, the comparison should be between another fMRI blending model, rather than image bleeding models (no matter whether it is mask-free or signle-image generation or not). I would suggest using the output of an fMRI-to-image model as the input of another image-blending model as a baseline.", "other_comments_or_suggestions": "Details of the user study should be introduced, such as study protocols, demographic information, etc.", "questions_for_authors": "See the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z2898tgxkp", "reviewer": "Reviewer_oSfB", "summary": "This paper introduces MindCustomer, a novel multi-context image generation framework that integrates visual brain signals with traditional image and text modalities to enable brain-controlled image customization. The authors propose a method to blend brain semantics into the image generation process, addressing key challenges such as multi-modal integration, and context preservation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your reviews!\n\n***Rebuttal Link*** (https://anonymous.4open.science/r/ICML2025-Rebuttal-MindCustomer) is for supplemented results in rebuttal, please refer to it.\n\n**Q1: Quantitative measures of ablation study and indicators of reconstructed image**\n\n**In the submitted paper, we have presented the quantitative measures of our ablation study in the right half of Table 1**, please refer to it. Due to page limitations, we reasonably merged these results with the baseline comparison within one table and provided a table reference in the relevant text (line 437). Furthermore, **we have listed the indicators of the reconstructed images in the submitted paper, please refer to Table 4**.\n\n**Q2: Table 4 illustration**\n\nAs the first work on multi-context (brain, image, text) blended generation, MindCustomer needs accurate context extraction. Table 4 demonstrates our method’s effectiveness in brain semantics extraction and its competitiveness with SOTA brain decoding methods. The main experiments in this paper show that our brain encoding, combined with the proposed effective cross-modal fusion pipeline, is sufficient to achieve high-quality blending. While existing fMRI reconstruction methods may offer better encoding, they introduce more conditions and complexity, making them inefficient in multi-context blended generation.\n\n**Q3: Further theoretical analysis**\n\nSince our proposed IBT is an MLP-based model with a relatively simple structure, we reduced some theoretical analysis in other aspects (e.g. robustness). Thank you for providing additional discussion ideas, which further enhance our theoretical analysis. We will explore the robustness of the IBT model across different data and individuals, as well as its generalization to data volume in the few-shot generation.\n\n**Q4: Metrics or qualitative analysis of semantic fidelity of the simulated brain signals**\n\nWe have provided additional qualitative and quantitative analyses of IBT-simulated brain signals in **Figures 15 and 16 of the original Appendix**. Please refer to them.\n\n**Q5: Comparison with other methods**\n\n- We have described a detailed analysis and differences with references (e.g. MindEye2, MindArtist), which can be found in the paper’s introduction (Left: Line 66-109, Right: Line 55-69) and related work section (Left: Line 150-164, Right: Line 135-155). Please refer to them. \n\n- To illustrate MindCustomer's performance, we add another baseline per the review's suggestion: using MindEye for fMRI-to-image reconstruction, followed by Versatile Diffusion (a mature image-blending model). In the baseline, VD requires no extra finetuning or optimization since it's pre-trained on large text-image datasets. In contrast, MindCustomer's finetuning adapts brain representation to diffusion's latent embedding, while optimization addresses semantic overlap between image and brain contexts—**unique innovations of our method**. As shown in the ***Rebuttal Link*** (**Rebuttal Figure 3 & Table 2**), due to the brain interpretation deviation caused by fMRI reconstruction and some semantic conflict from dual contexts resulting from the blending diffusion, the baseline shows some semantic deviation and loss. Benefiting from our unique designs, MindCustomer generates high-quality results. **We also explained this in the submitted paper (Experimental analysis: line 350-357 (right), Intro: line 091-095 & 058-065 (right)), please refer to them.** We will add the comparison results to better demonstrate MindCustomer's performance.\n\n**Q6: Clarity**\n\nThank you for the suggestion. We will revise the output of the Semantic Extractor in the figure clearly. It should add the output embedding of the Semantic Extractor with the ground truth to construct the loss for model training. Besides, in the section of Brain Representation Pre-training, we will reduce some unnecessary symbols to make it simpler and easier to understand.\n\n**Q7: Is the encoding model for predicting fMRI from images trained separately for each subject?**\n\nIBT is subject-wise, please refer to line 181 (right). Due to the significant differences in original brain signals from different subjects, we designed the IBT for each subject. Experiments **(e.g. Figure 15 & 16, and Table 6 in Appendix)** show that even when IBT is lightweight (several layers of MLP), it can still effectively fit each subject's brain signals. Therefore, the subject-wise IBT does not increase the complexity within an acceptable range.\n\n**Q8: Can each embedder obtain a different type of embedding?**\n\nIn line 203 (right): \"We employ the shallow subjectwise embedders $E_{∫}$ and deep shared embedders $E_{⌈}$ to obtain the two types of embeddings of voxels\", the two types refer to brain signal encodings supervised by CLIP-encoded **text** and **image**, respectively, as described in line 199 (right), rather than different types of encodings from different embedders. We will revise the description here for clarity.", "claims_and_evidence": "1.  As the first work,MindCustomer successfully implements multi context blended with brainsignal in image generation. （On the right side of line 70）\n\nEvidence:The paper provides a thorough literature review, highlighting that previous works have focused on text-to-image generation or brain signal reconstruction but not on using brain signals to guide multi-context image generation. \n\n2. It achieves high-quality image customization guided by cross-subject brain contexts.（On the right side of line 72）\nEvidence: The paper presents qualitative results showing high-quality image outputs that faithfully represent the provided contexts. \n\nQuantitative metrics (e.g., CLIP-I, DINOv2, CLIP-IQA) demonstrate superior performance compared to baseline methods. User studies also indicate a preference for MindCustomer's results over baselines.\n\n3. We propose the Image-BrainTranslator(IBT) for cross-subject brain data augmentation, ensuring stable embeddings.（On the right side of line 104）\n\nEvidence: The authors demonstrate the effectiveness of IBT through experiments showing that it can simulate brain signals from visual stimuli with moderate to high correlation to ground-truth fMRI data. Additionally, they show that IBT-augmented data improves the stability of brain embeddings.", "methods_and_evaluation_criteria": "Quantitative Metrics: The paper uses metrics such as CLIP-I, DINOv2, and CLIP-IQA to evaluate context similarity and image quality. These metrics are widely accepted in the field of image generation and are relevant for assessing the model's performance.\n\nI have some concerns regarding the ablation experiments presented in Figure 8. The figure only provides two pairs of Image/Brain context examples and does not quantitatively measure the true quality of the proposed IBT, fine-tuning, and optimization methods in this paper.\n\nThe results in Table 4 appear to indicate that the fMRI image reconstruction framework proposed in this paper underperforms compared to MindBridge and UMBRAE, suggesting that there is no actual performance improvement in this study.", "theoretical_claims": "The theoretical claims in the paper are primarily supported by empirical evidence rather than formal proofs. The authors provide convincing demonstrations of the effectiveness of their methods through experiments, comparisons, and user studies. However, there are potential areas for further theoretical analysis, such as:\n\nRobustness of the IBT: Analyzing the theoretical guarantees and limitations of the IBT in simulating brain signals. The correlation metric alone may not fully capture the semantic alignment between simulated and real brain signals. Additionally, the effectiveness of the IBT may depend on the quality and diversity of the training data.\n\nGeneralization Bounds: Exploring theoretical bounds on the model's generalization ability, especially in the context of few-shot learning. The generalization ability is based on empirical results from a limited number of subjects. Theoretical analysis could further explore the conditions under which the model generalizes well and the potential limitations in more diverse populations.", "experimental_designs_or_analyses": "The authors compare MindCustomer to several ablation variants, including models without IBT, without fine-tuning and optimization, and without optimization. The results are evaluated using the same metrics and qualitative analysis as in the main experiments.\n\nThe effectiveness of the IBT may vary across subjects due to inherent differences in brain signal patterns. The paper could benefit from a more detailed analysis of how well the IBT generalizes across different subjects. And while the Pearson correlation measures similarity, it does not directly assess semantic alignment. Additional metrics or qualitative analysis could provide deeper insights into the semantic fidelity of the simulated brain signals.\n\nWhile the ablation studies show improvements, they may not fully capture the interactions between components. More detailed analysis or additional ablation variants could provide deeper insights. For example, various indicators calculated from the reconstructed images of each part after ablation can be listed. The images displayed now are carefully selected and lack strong persuasiveness.", "relation_to_broader_scientific_literature": "The Image/Brain context-guided image generation proposed in this paper does not represent a particularly novel technology and can be viewed as a combination of existing techniques (e.g., MindEye2 (ICML 2024) and MindArtist (CVPR 2024)). It is recommended that the authors objectively evaluate the performance of the proposed methodological framework and emphasize its distinctions from previous work. This would help to highlight the contributions of this paper.", "essential_references_not_discussed": "See \"Relation To Broader Scientific Literature\".", "other_strengths_and_weaknesses": "The experiments in the article mainly focus on qualitative analysis, with too little emphasis on quantitative analysis. There is too little comparison with other models to fully demonstrate its effectiveness. It is strongly recommended to add quantitative indicators to key experiments such as ablation experiments, comparative experiments with image fusion, and generalization experiments. At the same time, some calculations of reconstructed image indicators can be added.\n\nThis paper proposes an Image/Brain context-guided image generation model that requires both Image and fMRI inputs during the inference process. Additionally, the textual inversion technology is not new and has been widely used. I believe that using a method similar to MindEye's [1] prior diffusion and SDXL for image generation, followed by optimization of image features, would not be inferior to the approach presented in this paper. Therefore, I am not convinced of the superiority of the proposed method over other existing methods.\n\nReferences\n\n[1] Scotti P, Banerjee A, Goode J, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors[J]. Advances in Neural Information Processing Systems, 2023, 36: 24705-24728.", "other_comments_or_suggestions": "After passing through Semantic Extractor, the figure points to the loss function. Suggest modifying the drawing to make it clearer.\n \nThe Method section of the article, especially the definition of variables in Brain Representation Pre training, is quite confusing. Although the author's meaning can be understood, it is difficult to read. I think it's because some variables were not used or further explained after being defined, such as EIa and eIa.", "questions_for_authors": "1. Is the encoding model for predicting fMRI from images trained separately for each subject?\n\n2. The article mentions the use of shallow subject-wise embedders  and deep shared embedders  to obtain two different types of embeddings. Can each embedder obtain a different type of embedding? Are they the same thing as the Shallow Layer and Shared Deep Layer in Figure 2 (1)? If so, there seems to be a sequential relationship, so how can two different types of embeddings be obtained from a shared deep layer? This part confuses me.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received mixed ratings from four reviewers. Three of them had positive comments on the technical novelty and the experimental verifications. One reviewer provided a rating of 'weak reject', and had questions only regarding some elaboration on the experimental results and discussion. In the rebuttal, the authors clearly echoed those questions with detailed information. Based on all these comments and the rebuttal, AC decided to recommend a weak acceptance for this submission.
Decision: Accept (poster)