XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation
Authors: Bowen Chen
Brynn zhao
Haomiao Sun
Li Chen
Xu Wang
Daniel Kang Du
Xinglong Wu
Venue: NeurIPS 2025 poster ()
Keywords: text-to-image generation
modulation
multi-subject controlled generation
Primary area: deep_learning

Abstract: Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.

Verse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation
Bowen Cher
Mengyi Zhad
Haomiao Sur
Li Cher Xu Wang,
Kang Du Xinglong Wu
Intelligent Creation Team, ByteDance
Schenbowen.cbr. zhaomengyi.o pl, sunhaomian ao, chenli. phd, wangxu, aangxu.ailab, ailab, dukang. daniel, wuxinglong) tbw.oonceeeetteecee.....o.o
XVerse
XVerse
Verse
X
Xverse
EESE
Figure I: XVerse enables single/multi-subbee personalization and the additional control of semantic attributes such as pose, style, and lighting. Input conditions are highlighted with red dots.
Abstract
Achieving fine- ane-ggrined control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, of- ten undermines the editability and coherence of Diffusion Transformers (DiTs).
"quual contribution.
Corresponding author, project lead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose novel multi-subjecc controlled gener- ation model XVerse. By transforming reference images into offsets for token- specific text-stream modulation, Verse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This ad- vancement significandly improves personalized and complex scene generation capabilities. Project Page: htttps://bytedaacee wig.aaaeeee.....bu....c. github. /it.io////o XVerse Github Link: httpp:://gitahe wtt.:/ab..coo/oyttdddr///////eee
1 Introduction
6 enabling the The field of text-to-image generation has advanced remarkably [1 2 3
4 5
creation of highly realistic and diverse images from textual descriptions. Initial breakthroughs in personalization focused on single subjects 77 9 [0] These methods demonstrated strong control over individual subject appearances and showed good editability. However, the growing demand for more complex visual narratives and personalized content has spurred interest in extending these capabilities to scenarios involving multiple subjects within a single, coherent scene. This transition to multi-subject personalization presents substantial challenges, particularly in preserving individual identity fidelity and alleviating attribute entanglement.
tried to leverage the attention mechanism in Diffusion Transformers (DiTs) [5] for injecting information from reference images. But this direct injection strong reliance image features can substantially impact the generation quality of the base model. This often leads to artifacts, distortions, attribute entanglement, and can compromise the overall structural integrity and coherence of the generated image. These limitations highlight a critical gap, underscoring the need for novel techniques that offer fine-grained, independent control over multiple subjects while preserving image quality, editability.
Many current ofte-of-theear multi-subjectt methods
To address these limitations, this paper introduces XVerse, a novel method for consistent multi-subject control of identity and semantic attributes. We identify that the inherent modulation vectors within DiT blocks [5). which are typically employed for general conditioning, represent an underexplored yet highly promising pathway to achieve nuanced, subject-specific control. Building upon this insight, Xeerse pioneers an approach centered on learning offsets within the text-stream modulation mechanism of DiTs. With the reference images provided, XVerse utilizes adapter transform them into share offsets and per-block offsets for token bokkenspecifit text-stream modulation. This technique allows for condition injection from diverse reference images while preserving the image's 's underlying structural integrity. To enhance fine-grained details, we incorporate VAF-encoded image features into the single-stream block of FL.UX (6. Instead of being the main conditioning factor, the VAE- derived features play a supporting role in enhancing details for the backbone network. This strategy successfully minimizes the occurrence of artifacts and distortions, enabling XVerse to achieve exceptional multi-subject controlled generation results (as shown in Figure Extensive testing conducted on our benchmark validates XVerse's 's exceptional performance in terms of both flexibility of editing and maintaining the appearance of the subject.
Our contributions are summarized follows:
We propose XVerse, novel framework for fine-grained multi-subject controllable genera- tion. Our approach integrates the reference images into the text-stream modulation offsets. Additionally, we leverages VAE-encoded image features to enhance fine-grained details which are difficult to expresss in the semantic space. This allows XVerse to achieve a high degree of consistency with reference images, while preserving the original diffusion model's editability by maintaining the image composition.
Through training high-quality datasets constructed by our data pipeline, Xerse achieves outstanding performance in generation tasks under multi-subject control. Moreover, due to the flexibility of text-stream modulation, XVerse also demonstrates strong generalization performance in tasks such as maintaining semantic information such as posture, lighting, and background.

---

We present comprehensive benchmark XeereeBench, which evaluates both single-subject and multi-subject controlled image generation. This benchmark provides rigorous method- ology for assessing a model's i ability to support flexible editing, maintain subject character- istics, and preserve distinct identities.
2 Related Work
2.1 Subject-driven Generation
Subject-driven generation tasks aimed at synthesizing user-specific content, and have made significant progress in recent years. Early endeavors predominantly concentrated on single-subject personaliza- tion. These approaches can be broadly classified into two main categories: (I) Fine-tuning tetee-tuiing--ase based methods, such as Textual Inversion 77] and DreamBooth [8]. which adapt pre-trained models to embed novel concepts from a few exemplar images of a single subject. (2) Tuning-free methods, including IP-Adapter [9] and Photoverse 10), leverage powerful vision encoders to inject subject identity directly into the generation process without test-time fine-tuning, thereby offering enhanced flexibility. However, extending personalization to accommodate multiple subjects within single, cohesive image presents substantial challenges, particularly in preserving individual identity fidelity and mitigating attribute entanglement. Recent progress leveraging DiTs [I5] has begun to address these complexities. For instance, OmniControl [TT] demonstrated versatile control by conditioning DiT imputs. Concurrently, line of research has focused unified frameworks for complex multi- subject image customization, exemplified by UniReal [[2), UNO [13], and DreamO [4]. While these methods significantly advance multi-subject generation, their primary strategies involve conditioning the input token sequence, imposing explicit constraints, or modifying attention mechanisms to guide the generative process. XVersee introduces distinct perspective, operating tuning- frrinng-reeeeee method for multi-subject personalization. With the utilization of the text-stream modulation mechanism DiTs, X can enable precise, subject-specific conditioning while preserving the structural integrity of the generated image
2.2 Modulation in Generative Models
Modulation mechanisms have been instrumental in enhancing the controllability and expressiveness of generative models. This concept involves dynamically adjusting model's internal activations or parameters based on conditioning information. StyleGAN [16) pioneered the use of modulation by introducing adaptive instance normalization (AdaIN), which modulates features using style vector. A key discovery associated with this approach was that even small modifications to these modulation parameters could induce smooth and semantically meaningful perturbations in the generated image. This insight spurred wave of research focused on leveraging modulation layers, leading to significant advancements and successful applications in tasks such as image editing and manipulation (17 18 19 20 21].
Transformer- based models, such as DiT's commonly employ modulation mechanisms like adap- tive layer normalization (AdaLN) [22]. These mechanisms allow conditioning information (e.g., text embeddings, timesteps) to guide image generation by modulating normalization layers within transformer blocks. Tokenverse (23 explored modulation for subject preservation, they typically rely on fine-tuning strategies. Such methods necessitate extensive training on predefined datasets to learn subject-specific modulations, thus limiting their adaptability to novel, unseen subjects without retraining or inference-time optimization. In contrast, XVerse is designed to inject rich identity and semantic information for arbitrary subjects without such subject-specife fine-tuning.
3 Method
3.1 Preliminaries
DiTs have introduced significant advancements in the quality, efficiency, and scalability of image synthesis, establishing DiT's as the foundational architecture for most state-of-the-art models. The Attention Block in DiT's process text and image tokens concurrently within their transformer layers. This attention mechanism provides a pathway for injecting control signals, such as features from control images, into the token representations. However, while injecting control signals through
3

---

Inject only singlobbleck
man wszninng o
V+-
T-Moe: n, +
I-lld: By +
Matic-lled Atteation
Tekeropeclic
Seake
Seaked Reidden Cunnectie
Reidiadl Consecion
Med.
Modulate Imags
Feed Ferward
DiT
Foed Farward
Fied Farward
Seak:
Figure 2: Overview of the XVerse framework. The reference images are processed by a T-Mod Resampler and subsequently injected into the per-token modulation adapter. Additionally, to supple- ment image details, the VAE-encoded features the reference image are also utilized as input to the single-stream block of DiTs.
attention can achieve good similarity improvements, this approach can also cause the model's sampling trajectory deviate, leading reduction in image generation quality. In this work, We try to further explore the potential of the modulation mechanism in DiTs to achieve more precise subject control editing.
Modulation Mechanism in DīTs. The modulation mechanism in DiTs, adapted from techniques popularized by StyleGAN[T6), refines neural network activations by applying learned scale factors and bias terms. For instance, models like Stable Diffusion [15] and FLUX [6] utilize Multi-Layer
Perceptron (MLP) process both the diffusion timestep t and CLIP [24] embedding of the text
prompt (..g.. fp = CL)P(p)). thereby deriving a conditioning vector y/:
MLP(t, fp).
(I)
This vector y is then further processed. typically through a linear layer, to generate a set of modulation parameters. In the DiT architecture, this yields twelve distinct parameters for each block. Six of these parameters modulate text features, and the remaining six modulate image features. The integration of these modulations is commonly achieved using techniques such Adaptive Layer Normalization (AdaLN) and residual connections. AdaLN is implemented prior to the Attention Layer and Berward Layer, and can be expressed as:
AdaLN((z, 0) a. T u(x)
(x)) B
(2)
where x is the input feature, p(x) and (x) are its mean and standard deviation respectively, and (scale) and { (bias) are modulation parameters derived from the conditioning vector Residual connections are added after the Attention layer and the Feed-Forward layer. They help stabilize training and enable deeper networks, typically taking the form:
Fout = FF(ma)) Jin-
where win is the input to layer or block, F(rin) is the learned residual mapping, Fout is the output, and y a scaling parameter applied to the residual term, also derived from the conditioning vector y. These mechanisms facilitate fine-grained control over feature representations at various stages within the network. Such adaptive modulation significantly enhances the model's capacity to align image generation with the conditioning inputs.
An essential aspect this modulation mechanism its ability operate separately from the main data flow of the attention mechanism. This separation allows for the precise integration of visual feature representation into specific words without interfering with the denoising process. This has the potential to minimize errors and enhance the overall quality of the generated image. Furthermore, text-based features often offer clearer semantic directionality compared to image-based features. This indicates that manipulating text-stream modulation signals can offer a more straightforward and comprehensible method to control the generative process. Consequently, Verse focuses on leveraging text-side feature modulation to attain precise control over image synthesis.

---

3.2 Multi-subject Controled Generation
The framework of XVerse is shown in Figure 2 We first leverage the modulation mechanism for its robust preservation of key attributes from the reference images. Subsequently, we inject fine- grained details with the VAE-encoded image features in single-stream blocks of FLUX model. The subsequent sections will detail these enhancements.
Enhanced Text-Stream Modulation with Image Feature Control. To enhance fine-graid aie-ggrined control over the generation process, we augment the existing text-stream modulation framework by introducing image features as an additional control signal. Specifically, given a conditioning image Le. we first extract its deep features using the CLIP model, represented as fc CL)P Subsequently, we combine these image features with the CLIP- encoded features fo of the text prompt, using perceiver resampler [25] as the text-stream modulation adapter (T-Mod Adapter). Here, the text features fp serve as the query vector, and the resampler outputs an offset Aemos:
A cross Resampler(fp:.
This output, Across encapsulates synergistic information from both the textual prompt and the conditioning image, acting as a corrective or refining signal. For example, if the text prompt is about "handbag" and the accompanying image Ie shows "brown leather handbag", Arnos helps the model adjust its representation from a generic handbag to the specific "brown leather handbag" with series of visual characteristies (such material, color, and style).
Crucially, Across is formulated as an offset vector targeting specific tokens. This offset is added to the corresoonding token embeddings injected into the model, enabling precise control over the semantics of each token while preserving the structure of the text-to-image result. The adjusted conditioning signal y is thereby derived:
MLP fp) + Across.
(5)
This adjusted conditioning signal y is then utilized to adjust the original modulation parameters (e.g.. scale and shift parameters) applied the network's activations, thereby refining the conditioning influence. Since our method primarily performs injection in the textual space, it can naturally generalize to the control of high-level semantic attributes such as pose, lighting, and style, without requiring additional differentiation for these conditions.
To further enhance the level of control over the final image output, we draw inspiration from StyleGAN's expansion of the W latent space to W+,, enabling each Transformer block i in our model to receive customized conditioning. This achieved by decomposing the Across signal. Instead of a single offset, we compute a shared component, Ashared (applied across all DiT blocks), and individual components for each block i, denoted as A per-block (which can be computed on a per-block or per-stage basis):
V MLP(L, fp) + Ashred + A Aper-block
This structured decomposition of the mmage-derived text offset facilitates more precise and adaptive control over the influence of text conditioning various levels of the generation process.
Refined Attention Module with Controlled VAE Integration. While using only text-stream modulation can achieve good editability and generation results, its ability to preserve detailed information is still limited. Inspired by OmniControl [1], we introduce VAE-encoded image features auxiliary module to further enhance the capability of our approach for maintaining consistency of fine-grained features. To avoid potential negative impacts of directly injecting image features (such as artifacts or degradation of image quality), we constrain the role of VAE features, making them primarily an auxiliary module for supplementing image details, rather than the dominant mechanism for feature injection. Specifically, we restrict the injection of VAE features only single-stream blocks within the FLUX model. To effectively distinguish different image patch regions within the latent space of the conditioning image. the position index for the latent is changed from (i, into individual index as UNO 13
3.3 Regularizations
In modulation space M+ feature vectors derived from highly similar subjects often become entan- gled, leading to subject confusion and unintended fusion in generated results. To address this,
5

---

Teage ENEEE
Anaring parple ekie and
Phrase Filtering
Face Extraction
Flerenced SAM2
wilking tien silowill Kearing a parpt shim and jeurs
purple ahirt
sidewokk
Figure 3: Training Data Construction Pipeline.
introduce two critical regularization techniques: region preservation loss and text-image attention loss, which facilitate multi- multi-subject disentanglement.
Region Preservation Loss. We construct new training sample by randomly selecting two existing samples and concatenating them in side-by-side left-right configuration, with their captions com- bined into unified overall caption. We randomly retain modulation injection from only one side (left or right), rather than both. For regions without modulation injection, we enforce consistency between the model's output and the T21 branch's 's output in those regions using 1.2 loss.
Lregion En. [[[(1 Me) O (Ver (zr,t, v,) Ve t, e)
(7)
where 2; denotes the noisy latent at timestep t, (2 t, y) denotes T2I branch's 's output, V (2,, v²) denotes the XVerse's output, and MM. denotes the modumotddaedd mask region. This approach only regularizes speject-specific features but also serves as data augmentation strategy for multi- subject datasets, enhancing the model's ability to distinguish and preserve subject characteristics.
Text-Image Attention Loss. To maintain the compositional and editability properties the T21 branch after modulation injection, we align the cross-attention dynamics between two branches. Specifically, we compute an L2 loss with normalization over the text-image cross-attention maps of the modulated model and the reference T2I branch. This encourages the modulated model to retain attention patterns that closely match those of the T2I branch, ensuring that semantic interactions between text and image regions remain consistent and editable.
3.4 Collection of Training data
High-quality multi-entity data remains scarce, and modulation injection necessitates knowledge of the corresponding text tokens in the target prompt for each conditional image. To address this, introduce a high-quality general-purpar purposse multi-entity data annotation pipeline.
Single-Image Data Construction. We curated a IM-scale dataset of images with resolutions exceeding 512 pixels from LAION [26), constructing a universal multi-entity single-image dataset via the data workflow depicted in 3 Specifically, we first employ Florence2 [27] for joint image caption generation and phrase grounding, followed by large language model (I.i.)-drrveen phrase filtering and classification to exclude non-entity terms (e.g., "sky", "water surface") that do not align with definitional criteria. Subject segmentation is then performed using SAM2 (28), with additional face detection and extraction applied specifically for human entities. While open-source data includes diverse scenarios, challenges such as suboptimal aesthetic quality and scene complexity impede model learning. To mitigate this, we supplement the dataset with an additional 1M-scale corpus of high-aesthetic qualiiyy images synthesized using FLUX [6]
Cross-Imape Data Construction. For human-centric data, we harvested 100K-scale single-person multi-view dataset from proprietary in-house collections, forming up to three image pairs per subject ID. For general objects, we leveraged the Subject200K [11] dataset. The construction workflow mirrors that of single-image data, with -ppaii--peccfic refinements: human data incorporates ID similarity thresholds, while general object data employs DINOv2 [29] similarity filtering-bor both applicd independently to enforce cross-view consistency.
Experiments
4.1 Experiments Setting
Implementation Details. In XVerse, we design three-stage training pipeline to achieve precise multi-subject controlled image generation. We first train the text-stream modulation adapter to

---

Single-Subjett
TripleSuujeet
1
H
Dual-Subject
Figure 4: Data distribution and samples for Bencs XVerseBench includes evaluations of single-subject, dual-subject, and triple-subject controlled image generation. The figure also illustrates the number of test samples allocated to each category.
establish foundational semantic alignment. It serves as the basis for conditional feature injection which ensures high-level consistency between injected images and generated results. Building upon the first stage, we introduce VAE-encoded features to enhance detail preservation. While maintaining the global structure and key characteristies obtained from stage one, this phase focuses on injecting fine- iine-grained visual details through hierarchical feature fusion. The first two stages employ hybrid training data containing both single-subject and multi-subject samples from our single-image dataset. However, we found that such reconstruction paradigm tends produce copy-paste effects during inference, resulting in reduced diversity and compromised structural fidelity. To address this issue, we propose third training phase incorporating cross- cross-image data where the subject appearances differ between injected and target images. This extension enhances the capability of our approach to establish robust appearance mapping under heterogeneous visual conditions. Throughout all stages, we maintain mixed training on both single-subject and multi-subject datasets.
Based on FLUX..1-dev 6 text-to- imae generation model, we employ LoRA 30 with rank of 128 to efficiently fine-tune the model while maintaining its generalization capabilities. We utilize two three-layer resamplers with intermediate dimension of 3072 to generate the shared offisets and per-block offsets for text-stream modulation. The model was trained for 70K, 150K, and 10K iterations in the respective stages. Both the text-stream modulation adapter and LoRA layers were optimized with a learning rate of 5e-6. The region preservation loss was assigned a weight of 10, while the text-image attention loss was weighted at 0.01. Training was conducted on 64 NVIDIA A800 GPUs (40GB each), taking approximately 7 days in total distributed as 2 days for Stage 1, days for Stage 2, and day for Stage 3.
XVerseBench Details. Existing controlled image generation benchmarks often focus on either maintaining identity or object appearance consistency, rarely encompassing datasets that rigorously test both aspects. To comprehensively assess the models' single-subject and multi-subject conditional generation and editing capabilities, constructed new benchmark by merging and curating data from DreamBench++
3 and Unsplash50 32)
Our resulting benchmark X VerseBench comprises 20 distinct human identities, 65 unique objects, and 45 different animal species/individuals. To thoroughly evaluate model effectiveness in subject-driven generation tasks, we developed test sets specifically for single-subject, dual-subject, and triple-subject control scenarios. This benchmark includes 300 unique test prompts covering diverse combinations of humans, objects, and animals. Figure 4 shows more detail information and samples for each categories. For evaluation, we employ suite of metrics to quantify different aspects of generation
quality and control fidelity: including DPG score [33 to assess the model's editing capability, Face ID similarity (34) and DINOv2 [29] similarity to assess the model's 's preservation of human
identity and objects, and Aesthetic Score (35] to measure to evaluate the aesthetics of the generated image. XVerseBench aims to provide a more challenging and holistic evaluation framework for state-of thetth---atttt art multi-subject controllable text-to-image generation models.
4.2 Comparisons with State-of-the-art Methods
We compare our proposed XVerse method with several leading multi-subject driven generation techniques, including MS-Difffusion (36), MIP-Adapter [37]. OmniGen (38), UNO [3] and DreamO
7

---

Table Quantitative results of single-subject and multi-subject driven generation on VerseBench.
Single-Subject
Multi-Subject
Overall
Method
DPG
ID-Sim
IP-Sim
AES
AVG
DPG
ID-Sim
IP-Sim
AES
AVG
ASS-Diffusion [36
96.94
6.58
51.06
59.69
53.57
87.27
4.81
40.90
55.87
47.21
50.39
MIP-Adapler
77.48
66.32
52.09
56.07
84.52
19.49
49.89
51.78
51.42
53.75
OmniGen 38
85.19
60.17
70.73
51.89
67.00
81.71
42.18
52.11
51.35
56.84
61.92
UNO 13
91.82
37.22
74.35
55.21
64.65
87.57
26.00
60.62
53.04
56.81
60.73
DreamO 14
97.51
58.74
67.69
53.80
69.44
89.75
44.21
60.87
51.16
61.50
65.47
X Verse (Ours)
93.50
63.02
71.35
56.63
71.13
91.77
51.03
61.04
53.68
64.38
67.76
Inputs
XVerse
Dream()
UNO
OmniGen
MIP
MS-Diffusion
person
A mun and woman standing side by side in park.
A litle girl standing beside a man.
walliing
the
branch above while white rige slowly approxches a eurioos kittin in forzs.
Figure 5: Qualitative comparison with different methods XVerseBench.
The generation results evaluated for both single-subject and multi-subject tasks, with quantitative findings presented in [able Our XVerse method achieves the highest Overall score of 67.76. significantly outperforming all other compared methods. This clearly indicates strong comprehensive advantage of our approach.
14
In the single-subject generation category, XVerse demonstrates exceptional performance, securing the top AVG score of 71.13. This underscores its robust capability in generating high-quality images focused on individual subjects. Notably, Xeess achieves the best identity similarity score (ID-Sim) of 63.02, suggesting superior preservation of subject identity. While DreamO leads in DPG with 97.51, Verse's s strong average performance, bolstered by competitive scores in IP-Sim (Object Similarity) at 71.35 and AES (Aesthetic Score) at 56.63, highlights its well-rounded excellence.
Mcanwhile, XVerree truly excels in the more challenging multi-subject generation tasks, achieving a leading AVG score of 64.38. This remarkable performance can be attributed to XVerse's 's novel approach of learning offsets within the text-stream modulation mechanism of DIT. This allows for precise conditioning from diverse image types while crucially preserving the image's 's structural integrity. Furthermore, the careful integration of VAE-derived features for detail refinement, rather than dominant conditioning, effectively mitigates artifacts and distortions, which is particularly vital for maintaining clarity and attribute disentanglement in multi-subject scenarios.
Figure 5 presents a qualitative comparison of our method with other state-of-the-art approaches. As illustrated, our model demonstrates superior capability in maintaining the consistency and relevance

---

Table 2: Ablation Study of Joint Training of Text Modulation and VAE-Encoded Features
EXP ID
EXP Name
Init
Steps
Single-AVG
Multi-AVG
AVG
30K
57.55
52.73
55.14
Text Modulation Only
Scratch
40K
58.30
53.54
55.92
30K
60.71
53.32
57.02
Single-Stream VAE Only
Scratch
40K
59.60
53.37
56.49
Text Modulation + VAE
Exp 30K
10K
62.62
55.16
58.89
Table 3: Ablation Study of Different Regulation Losses
EXP
Steps
Single-AVG
Multi-AVG
AVG
Bascline
30K
57.55
52.73
55.14
Baseline w/o RPL
30K
55.47
50.72
53,10
Baseline w/o TIAL
30K
53.42
44.51
48.97
between identities and associated objects within the generated images. This is a direct result of our refined modulation strategy. By carefully adjusting the modulation offsets, XVerse achieves enhanced text-image alignment, which is particularly evident in its accurate depiction of object quantities and the relationships between multiple subjects. Furthermore, when compared existing methods, our model consistently produces images with a higher degree of naturalness and visual plausibility. This improved realism underscores the advantages of our approach in editing the text-stream modulation pathway, allowing for more faithful and aesthetically pleasing image synthesis.
4.3 Ablation Studies
To further evaluate the effectiveness of different modules in X Xeese and analyze the impact of key components, we conduct a series of ablation studies. All quantitative comparisons are performed under the same configuration (e.g.. hyperparameters, training schedule), unless otherwise specified. Due to computational resource limitations, all experiments are executed on 16 NVIDIA A800 (40GB) GPUs.
Joint Training Text Modulation and VAE-Encoded Features We test three setups to examine text-VAE collaboration: (1) Text Modulation Only, (2) Single-Stream VAE Only, (3) Joint Training. The results are shown in Table 2 After 30K steps, Text Modulation Only scores 57.55 (single- subject) and 52.73 (multi-subject) AVG; extending to 40K steps (additional 10K) brings marginal gains (58.30/53.54) Single-Stream VAE Only outperforms text-only at 30K (60.71/53.32) but degrades 40K, reflecting standalone VAE instability (attribute blending disrupts consistency). Notably, Experiment (Joint Training, initialized from Exp. I's 30K checkpoint, 10K extra steps) boosts scores 62.62 (single) and 55.16 (multi), mitigating VAE injection instability and enabling more disentangled generation. Figure shows the qualitative comparison in both dual-subject and triple-subject controlled generation tasks. The experimental results show that by converting the reference image to text-stream modulation offset vectors, XVerse achieves the personalization of the condition subjects while maintaining the composition identical the original T2I outcomes. Additionally, the VAE features play supportive role by helping the model add specific details without compromising the overall structure and semantic coherence established by the text-stream modulation. This collaborative approach allows X Verse to maintain subject consistency while achieving a high degree of editability.
Impacts of Different Regularization Losses. To verify the necessity of Region Preservation Loss (RPL.) and Text-Image Attention Loss (TTAL), compare the baseline (both losses) with RPL. ablated and T'AL-ablated models. Other configurations (e.g., training steps) are identical to isolate loss impacts. As shown in Table 3 the baseline achieves the highest AVG (single: 57.55, multi: 52.73). RPL ablation drops performance by 2.04% (as it helps to maintain category consistency), while TIAL ablation causes 7.17% decline (as ensures precise attribute localization, reducing irrelevant feature leakage). This confirms RPL and TIAL are complementary and indispensable.

---

Table Ablation Study of Different VAE-encoded Feature Injection Strategies
EXP
Steps
Single-AVG
Multi-AVG
AVG
Single-stream, VAE Only All Blocks, VAE Only
30K
60.71 58.58
53.32
57.02
30K
51.35
54.97
Lxpo
T2
cacadad lagge forace.
XNex
Figure 6: Effect of text-stream modulation resam- pler and VAE-encoded image features.
Figure 7: The control of sementic attributes, such as cloth, pose, lighting, and style.
VAE-encoded feature injection blocks. We evaluate two VAE injection strategies to identify the optimal configuration: "Single-stream, VAE Only" (injecting exclusively into single-stream blocks) and "All Blocks, VAE Only" (injecting into both double-stream and single-stream blocks). As shown in Table 4 the single-stream injection consistently outperform the all-blocks approach, achieving average scores of 60.71 versus 58.58 for single-subject evaluations, and 53.32 versus 51.35 for multi-subject tests. The inferior performance of the all-blocks injection is attributed to feature interference, where redundant signals disrupt the core generation process. Focusing injection on the core single-stream pathways results in better alignment and overall effectiveness compared more diffuse injection strategy.
4.4 Applications
Figure 7 demonstrates XVerse's ' capability to control semantic attributes of the generated image. XVerse exhibits precise control over attributes such as lighting, subject posture, clothing, and artistic style. By injecting reference images toward targeted words, X'eesee can manipulate these semantic attributes without the need for extensive training data on specific attribute categories. These results further underscore the model's 's exceptional ability to generalize and edit effectively.
Conclusion
In this paper, we introduce XVerse, an innovative framework designed to excel in the complex task of precise multi-subject control within DiTs. XVerse injects reference image features through modulation offsets and control the token-specific representation within the DiT blocks. This approach facilitates adaptive, per-subject conditioning by precisely governing how textual embeddings are transformed and integrated throughout the diffusion process. Consequently. XVerse effectively mitigates common generation issues such attribute entanglement and artifacts, demonstrating particular excellence in both single-subject and multi-subject controlled generation tasks.
Limitations. While XVerse showcases significant progress in fine- fine-graine grained multi-subject control, there are certain limitations that need to be addressed and opportunities for future research. One major challenge is the lack of high-qualing large-scale cross- masss-imaee multi-subject datasets, which are essential for training and evaluating models that can understand and generate complex interactions between subjects. Additionally, our current research has mainly focused on the text-stream modulation pathway, leaving room for exploration in utilizing imdge-modulation techniques for precise pixel-level region-specific control.
10

---

References
[1] Ian Goodfellow, Jean Pouget-Abadee Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in neural information processing systems 27, 2014.
[2] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 1rXiv:1312.6114, 2013.
[3] Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pages
PMLR, 2015.
[4] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, 2020.
[5] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
[6] Black Forest Labs. Flux: Official inference repository for flux.1 models, 2024. Accessed: 2024-11-12
[7] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
[8] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500- 22510,2500000 2023.
[9] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint 1.tiv::2300.0672... 2023.
[10] Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song, Shizun Wang, Xu Wang, Hao Yang, Jing Liu, Kang Du, et al. Photoverse: Tuning-free image customization with text-to-image diffusion models. arXiv preprint w:Xiv:230900599 2023.
[11] Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer. arXiv preprint arXiv:2411.15098, 2024.
[12] Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al. Unireal: Universal image generation and editing via learning real-world dynamics. arXiv preprint arXiv:2412.07774, 2024.
[13] Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, and Qian He. Less-to- more generalization: Unlocking more controllability by in-context generation. arXiv preprint arXiv:2504.02216) 2025.
[14] Chong Mou, Yanze Wu, Wenxu Wu, Zinan Guo, Pengze Zhang, Yufeng Cheng, Yiming Luo, Fei Ding, Shiwen Zhang, Xinghui Li, et al. Dreamo: A unified framework for image customization. arXiv preprint arXiv:2504.16915, 2025.
[15] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 4195 2023.
[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.

---

[17] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 4014):1-14 2021.
[18] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In Advances in neural information processing systems, volume 33, pages 9841-9850, 2020.
[19] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to- to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287-2296, 2021.
[20] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images, ACM Transactions on graphics (TOG), 42(1):1-13, 2022.
[21] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on computer vision, pages 592-608, 2020.
[22] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
[23] Daniel Garibi, Shahar Yadin, Roni Paiss, Omer Tov, Shiran Zada, Ariel Ephrat, Tomer Michaeli, Inbar Mosseri, and Tali Dekel. Tokenverse: Versatile multi-concept personalization in token modulation space. arXiv preprint 22ti::2000.112222 2025.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, 2021.
[25] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716-23733 2022.
[26] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 400m:-400m: Open dataset clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 2021.
[27] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4818 4829, 2024.
[28] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007. 2024.
[29] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[30] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[31] Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned benchmark for personalized image generation. In The Thirteenth International Conference on Learning Representations, 2025.
12

---

[32] Zinan Guo, Yanze Wu, Chen Zhuowei, Peng Zhang, Qian He, et al. Pulid: Pure and lightning id customization via contrastive alignment. In Advances in neural information processing systems, volume 37, pages 36777-36804, 2024.
[33] Xiwei Hu, Rui Wang. Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with 1lm for enhanced semantic alignment. arXiv preprint wrXiv:2403:05135, 2024.
[34] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2019.
[35] discus0434. Aesthetic predictor v2.5: Siglip-based aesthetic score predictor, https://gittub pttp:///aaaeeeeeeeeepeeeeeeeeeee -2 2024. Accessed: 2024-12-08.
[36] Xierui Wang. Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi- subject zero-shot image personalization with layout guidance. In The Thirteenth International Conference on Learning Representations, 2025.
[37] Qihan Huang, Siming Fu, Jinlong Liu, Hao Jiang, Yipeng Yu, and Jie Song. Resolving multi- condition confusion for finetuning- free personalized image generation. In Proceedings the AAAI Conference on Artificial Intelligence, volume 39, pages 3707-3714, 2025.
[38] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprini arXiv:2409. / 1340, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We have presented the contributions of the paper in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We create separate "Limitations" part in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: [Ys]]
Guidelines: The experimental details have been outlined in the main text, and plan to make our code available at a later time.
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We plan to make our code available at a later time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Suubmisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In the beginning part of the experiments sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We are following the testing protocol of the previous method, where the results are represented by the mean performance of images generated by four random seeds to represent the model performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the beginning part of the experiments sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We will have the detailed usage guidelines with our realised code.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The benchmark used in the paper are all from open-source datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will have the detailed usage guidelines with our realised code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
The data we use is come from those open-access dataset.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLM for data labeling and model evaluation, and describe it in the paper.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Samples for Training Dataset
linle the sitr
Aman ontage, jumposult and land het sittng the sidweek looking his phone.
wermsn kneeling down large dag,
Amaz and woran bolling glass champagnee
Figure 8: Examples of training data for multi-subject controlled generation.
Figure presents examples of our training data for multi-subject controlled generation. As illustrated, the dataset covers a diverse range of scenarios, including human-object interactions, human-anima compositions, and complex multi-person scenes. For numan-centric data, we intentionally randomly select facial images or full-body images as reference inputs, This strategy can further enhance the model's generalization performance. By utilizing this diverse and extensive dataset, which encompasses wide range of scene variations and control types, our model is able to achieve impressive editing capabilities while maintaining high consistency with the reference images.
presentss
B Impact of Prompt Variation on the Generated Image
plotng
vaas plates
voara witinggg on the (Raying gaar
UIEEES GEEELE WITH den CBCA 3a0, KRDER KD THE BEE0
ehtiggo the cuep plying gaia:
Aedig Ontay oF dhang (Tayingg
paiar
Figure 9: Impact of prompt variation subject-controlled Image generation. The reference image and the initial output of our text-to-image generation model are shown on the left side. The right side illustrates the influence of different prompts on the generated output, with the prompt variances highlighted in red.
To evaluate the impact of prompt variation subject-controlled image generation, modified the injected words in the per-token text-modulation module while keeping the reference image constant. The results, shown in Figure 9 offer valuable insights. This visualization effectively illustrates
21

---

that a more detailed prompt description improves the preservation of the subject's identity in the generated image. Additionally. when the prompt closely matches the reference image, our model only incorporates intricate image details but also maintains high level of control over the subject's attributes, even allowing for successful changes such as gender. On the other hand, if there is significant semantic mismatch between the injected prompt and the reference image (e.g., trying to generate a person's image from prompts like "a dog" or "a tree stump"), the injection process consistently fails. This highlights model's s ability to accurately target and incorporate reference image features into specific words, enabling precise control over the generated output.
C Comparison of the CLIP-T and DPG scores
When evaluating text-to-image generation models, the CLIP-T 24 Score has been prevalent metric in prior studies, assessing semantic consistency by leveraging CLIP's image-text embeddings. However, our research highlights the superior efficacy of the DPG (Dense Prompt Graph) score, particularly for intricate prompts. While CLIP-T offers a broad measure of semantic aligmment, the DPG score specifically designed to evaluate model's capacity to interpret and execute detailed and complex textual instructions. It rigorously assesses editing abilities across multiple objects, diverse attributes, and intricate relationships, thereby capturing the nuanced and fine- fine-grainn mne-grained semantic alignment crucial for advanced compositional generation. This provides more comprehensive and robust evaluation for challenging scenarios.
Illustration of Region Preservation Loss
A picture divided into two parts. The left part is (prompt J). The right part is (prompt__2)
welking sidevolk wearing
steps laptop.
T21 Branch
noise
Zweoman
Apurple
Ajeans
Regeo
L-M_
Figure 10: Illustration of the region preservation loss.
Figure 10 shows the illustration of our region preservation loss. We form training samples by concatenating two existing samples, merging their captions, and randomly applying modulation to only one side. For the unmodulated regions, defined by Mc. we enforee consistency between our model's output (2or(z,,4,y)) and the text-to-image branch's output (Vo(z,, I, y)) an L2 loss (Eq. 1). By using this regularization, Verse can better inject the reference image into specific areas without affecting the generation of irrelevant areas, thereby achieving more precise generation control.
22

---

E Ablation Study for Text-Image Attention Loss
Grersted boag?
Kiees CAL
ai sa sbeut
The qualitative comparsion of Text- Image Attention Loss. This image shows the generated results and attention maps for "woman", "coffee cup", and "pink suit" for each method.
Figure 11:
To validate the effectiveness of our text-image attention loss, conducted experiment where excluded this regularization and examined the generated outputs along with their respective attention maps. The qualitative analysis presented in Figure clearly demonstrates the significance of this method. It demonstrates our method's s ability to maintain the structural and editable characteristics of the T2I branch following modulation injection. Through ensuring L2 consistency between the cross-attes aooss-tttnntion maps of the modulated model and the reference T2I branch, our approach ensures the reliable preservation of text-image semantic interactions. This ultimately enables precise control over semantics, as visually evidenced by the generated results and attention maps for specific prompts like "woman," "coffee cup," " and "pink suit."
F
Broader Impacts
Our model, XVerse, marks significant leap in multi-subject controllable text-to-image generation, leading to enhanced fidelity and editability. This breakthrough holds substantial positive societal impacts, particularly within the creative industries, where it can revolutionize the creation of person- alized and complex visual content. Furthermore, XVerse can transform education and training by providing more engaging and tailored visual aids, and contribute to content inclusivity by enabling the representation of wider range individuals and scenarios.
However, this powerful technology also presents potential negative societal impacts. The improved generation capability could lead to misinformation and deepfakes, raise privacy concerns used improperly, and potentially amplify biases present in training data. As foundational research, XVerse isn't directly tied to deployment. Yet, we believe it's crucial acknowledge these risks. Future work will explore mitigation strategies like content detection and ethical guidelines, contributing to the responsible advancement of generative AL.
23


Reviews:
[{"review_id": "m0BFbRtRXb", "reviewer": "Reviewer_S48T", "summary": "The paper proposes a solution to multiple subject control in DiT based image synthesizer and a benchmark dataset for evaluating single & multiple subject control tasks. The method achieve high-fidelity control of multiple (6+) subjects and their semantic attributes (poses, highlights and styles) during image generation.", "questions": "What about the stableness and performance under multiple subject injection? Till how many of subjects will the method still work in a stable way? It is better to provide evaluations under different subject amount.\n\nIf the authors can clear some of my doubts (about novelty and ablation studies), I would be happy to raise my scores.", "limitations": "The method is architecture-specific to the DiT Models.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe method achieves SOTA on both single and multiple subject control tasks (numerically and visually).\n\nThe method is designed in a dedicated way for preventing different subjects interfering with each others, and is stable across various scenes.\n\nThe method proposes to convert each reference images into subject-specific modulation offsets, which is a novel idea for token-level modulation.\n\nWeaknesses:\n\nThe writing of this paper is a bit confusing, the authors spends a lot of space to introduce the well-known AdaLayerNorm module in DiT, instead of introducing the module Perceiver Resampler used by the method's key mechanism. Also, some details is not well-explained in the paper, e.g. \"only injecting vae injection into a single block\". So which block? Why this block? Why not two or more blocks? Any further ablation studies?\n\nThe ablation study is not enough (maybe due to the complex training pipeline), no numerical results indicate the performance loss on each mechanism (like the effectiveness of global-local text-offsets).\n\nAs for the novelty, the main contribution is the injection of token-level modulation offsets, while the other of key mechanism have precedents, which lowers the overall novelty of the paper.", "was_revised": true, "final_justification": "After the rebuttal, the authors completed the most of missing experiments on ablations I required, which proves the integrity of the paper. Also, some of my doubts and misunderstandings are cleared. I have raised my rating for the paper.", "rebuttal": "Thank you for your insightful comments and valuable suggestions, which have helped us better refine our work. We address each point as follows:\n\n1. **Clarification on the Focus of AdaLayerNorm and Perceiver Resampler**\n\n\tThank you for your valuable comment. We devoted significant space to AdaLayerNorm because our work extends its application beyond its original purpose in DiT. Since AdaLN is known for accelerating model convergence [15], many works have not considered that it can be used for subject control. In contrast, we leverage it in XVerse to achieve fine-grained control over subject identity without extra test-time tuning, which is a key innovation of our method.\n\n\tIn contrast, the Perceiver Resampler is a more established framework and not our core contribution. However, we acknowledge the need for more detail. In the revised version, we will elaborate on how the Perceiver Resampler is utilized within our text-stream modulation adapter to enhance clarity.\n\n2. **Details on VAE Feature Injection into a Single Block**\n\n\tThank you for your feedback. The FLUX architecture includes two structures:\n\t- The double-stream module, consisting of 19 Transformer blocks, processes text and image streams independently, as shown on the right side of Figure 2. Each stream is equipped with its own projection layers, modulation parameters, and normalization layers.\n\t- The single-stream module, deployed after the dual-stream module, contains 38 Transformer blocks. It unifies text and image features without strict modality distinction.\n\n\tIn XVerse, the VAE encoded features do not interact with the latent features and text features in the double-stream module and are injected exclusively into the single-stream module of FLUX. This choice mitigates VAE's potential disruption to attention mechanisms and text-to-image aesthetics/structure. We conducted ablation studies on block selection (as shown below), confirming that limiting VAE injection to single-stream blocks best preserves generation quality, editability, and generalization. We will add more details to the revision.\n\n3. **Expanded Ablation Studies with Numerical Results**\n\n\tWe conducted a series of ablation studies to dissect critical components and their impacts on the framework, ensuring all experiments shared identical configurations (e.g., hyperparameters, training schedules) unless specified otherwise, and were executed on 16 NVIDIA A800 (40GB) GPUs due to computational constraints.\n\n\t - VAE-encoded feature injection blocks\n\n\t| | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|\n\t| EXP | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| Single-stream VAE Only | 30K | | 60.71 | 95.50 | 26.77 | 62.66 | 57.92 | | 53.32 | 91.02 | 14.47 | 52.55 | 55.26 |\n\t| All Blocks VAE Only | 30K | | 58.58 | 96.62 | 19.71 | 60.34 | 57.65 | | 51.35 | 88.63 | 11.15 | 50.33 | 55.29 |\n\n\tWe conducted an ablation study to dissect the role of VAE-encoded feature injection blocks in our framework. All experiments adopted identical experimental configurations except for the injection block designs (i.e., “Single-stream VAE Only” vs. “All Blocks VAE Only” as defined in the table). As shown in Table 1, injecting VAE-encoded features into the single-stream module of FLUX outperformed \"All Blocks VAE Only\" on both single-subject and multi-subject benchmarks. For instance, in single-subject AVG, it achieved 60.71 vs. 58.58, and in multi-subject AVG, 53.32 vs. 51.35. This highlights the superiority of targeted single-stream VAE feature injection.\n\n\t - Joint Training of Text Modulation and VAE-Encoded Features\n\n\t| | | | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n\t| EXP ID | EXP Name | Init | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| 1 | Text Modulation Only | Scratch | 30K | | 57.55 | 96.31 | 22.19 | 54.74 | 56.96 | | 52.73 | 91.25 | 18.45 | 47.12 | 54.14 |\n\t| 1 | Text Modulation Only | Scratch | 40K | | 58.30 | 96.61 | 24.01 | 56.31 | 56.29 | | 53.54 | 91.03 | 21.81 | 48.20 | 53.14 |\n\t| 2 | Single-Stream VAE Only | Scratch | 30K | | 60.71 | 95.50 | 26.77 | 62.66 | 57.92 | | 53.32 | 91.02 | 14.47 | 52.55 | 55.26 |\n\t| 2 | Single-Stream VAE Only | Scratch | 40K | | 59.60 | 94.97 | 22.93 | 62.94 | 57.59 | | 53.37 | 91.34 | 13.99 | 52.77 | 55.38 |\n\t| 3 | Text Modulation + VAE | Exp 1, 30K | 10K | | 62.62 | 96.28 | 39.53 | 57.54 | 57.14 | | 55.16 | 90.52 | 26.97 | 49.15 | 54.04 |\n\n\tWe designed three experiments to investigate joint training of text modulation and VAE-encoded feature injection. As shown in the table, Text Modulation Only model achieves average (AVG) scores of 57.55 and 52.73 on single-subject and multi-subject benchmarks, respectively, after 30K training steps. With an additional 10K training steps, its performance marginally improves to 58.30 and 53.54, respectively. In contrast, VAE-only attains slightly higher AVG scores of 60.71 and 53.32 after 30K training steps, yet its performance deteriorates with a further 10K training steps. This observation underscores the instability and challenges of VAE feature injection, which stem from conceptual confusion and attribute blending. Notably, Exp. 3—which employs 10K joint training steps for text modulation and VAE feature injection, using Exp. 1 (30K steps) as the initial checkpoint—exhibits performance improvements from 57.55 to 62.62 (single-subject) and from 52.73 to 55.16 (multi-subject). Collectively, our approach substantially alleviates prevalent challenges in VAE feature injection, thereby enabling a cleaner and more disentangled generation process.\n\n\tThe instability of pure VAE feature injection is evident from its lower ID-Similarity scores. Lacking semantic guidance, the VAE-only model often generates images that fail to retain the identity information of the reference image, resulting in inconsistent and lower-quality outputs. In contrast, text modulation, demonstrated in Exp. 3, serves as a crucial “stabilizer” and “guide”. It enables the VAE feature injection in a more controlled and semantically coherent manner, reducing conceptual errors and significantly enhancing the model’s stability and editability.\n\n\t - Impacts of CCL and CAL Regularization Losses\n\n\t| | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n\t| EXP | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| Baseline | 30K | | 57.55 | 96.31 | 22.19 | 54.74 | 56.96 | | 52.73 | 91.25 | 18.45 | 47.12 | 54.14 |\n\t| Baseline - CCL | 30K | | 55.47 | 95.89 | 15.19 | 54.11 | 56.72 | | 50.72 | 92.01 | 10.83 | 46.31 | 53.76 |\n\t| Baseline - CAL | 30K | | 53.42 | 96.31 | 7.47 | 52.43 | 57.47 | | 44.51 | 77.43 | 6.34 | 40.22 | 54.05 |\n\n\tWe evaluated CCL and CAL regularization losses by comparing against a baseline (retaining both losses). As shown in the table, the baseline model achieves the highest average scores on both single-subject and multi-subject benchmarks. This result underscores the effectiveness of the CCL and CAL regularization losses—since ablating either loss leads to performance degradation.\n\n4. **Novelty: Distinction from Prior VAE-Based Methods**\n\n\tThank you for your insightful feedback regarding the novelty of our work. We agree that VAE-encoded features are widely adopted to enhance identity fidelity, but our core innovation lies not in \"using VAE\" but in a novel integration mechanism that addresses limitations of existing methods. When extended to multi-subject control scenarios, traditional VAE-only methods often have difficulty preserving consistent identity fidelity across multiple subjects, frequently resulting in blurred or merged identities.\n\n\tOur XVerse solves these issues through two key innovations:\n\t- **Hybrid Injection Strategy**: Instead of global injection: the Double-stream module uses a text-modulation adapter for precise control over specific subject or high-level semantics (posture, shape). The Single-stream module receives direct VAE injection for the refinement of local details/textures. This enables implicit learning of injection regions without explicit attention masks, thereby preserving identity while protecting image structure.\n\t- **Novel Regularization Losses**: These guide the model to distinguish identity-controlled vs. text-controlled regions, achieving clearer disentangled representations and mitigating attribute blending.\n\n\tOur ablation experiments (as shown in Response 3) confirm this: a VAE-only model performs far worse than our complete XVerse, proving that our hybrid injection and regularization losses together enable such a high-fidelity, controllable generation. In summary, we build on VAE but solve its existing limitations via innovative architecture and specialized losses.\n\n5. **Stability Under Multi-subject Injection**\n\n\tThanks for your suggestion. We evaluated stability across 1Instance, 2-Instance and 3-Instance scenarios in XVerseBench. The results are presented as the average scores for editability (DPG-score), ID/IP similarity, and aesthetic quality:\n\n\t| Method | 1-Instance | 2-Instance | 3-Instance |\n\t|:---:|:---:|:---:|:---:|\n\t| OmniGen | 67.00 | 60.94 | 51.91 |\n\t| UNO | 64.65 | 58.63 | 52.11 |\n\t| DreamO | 69.44 | 64.45 | 50.97 |\n\t| XVerse | 71.13 | 64.85 | 62.96 |\n\n\tThe table illustrates that the model's performance varies with the number of subjects. Nevertheless, XVerse consistently outperforms state-of-the-art methods across all three protocols and demonstrates significantly greater stability, exhibiting an average performance decline of less than 10% even when applied to up to three subjects. You can also check out the examples in Figure 1. These show that our model can still produce high-quality images even when it is used with multiple subjects."}, {"review_id": "yyUoDLEaqG", "reviewer": "Reviewer_ATo1", "summary": "This paper introduces XVerse, which enables the composition and control of multiple subjects. The method uses a T-Mod Adapter to extract an offset vector ∆cross from reference images and applies it in the modulation space of DiTs. This control approach in the text space not only avoids interfering with the denoising process but also extends object manipulation to higher-level semantic attributes such as pose, lighting, and style.", "questions": "1. In Figure 2, what do the symbols c and x specifically represent? Additionally, could you clarify how the dual-branch architecture collaborates during training?\n2. If the dataset scale were reduced by half, how would this impact the model’s performance? Have you conducted any experiments or observations to assess this sensitivity?(You don't need re-training your model for this question.)", "limitations": "Please refer to Weakness.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Supports fine-grained control and composition of multiple objects.\n2. Demonstrates strong subject ID consistency.\n3. Provides valuable exploration of the modulation mechanism in DiTs, contributing technically to the community.\n4. The paper clearly explains the proposed method and is easy to read.\n\nWeaknesses\n1. According to Sec.3.3, the training appears to involve a dual-branch structure, but the Method section does not clearly describe how these two branches collaborate.\n2. The paper does not disclose the computational resources required for training. Given that at least 2M-scale datasets were collected, the training cost is likely substantial.\n3. The ablation study only analyzes the effect of the VAE-encoded features, and the two proposed losses, which are presented as key contributions, were not sufficiently studied or validated.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful questions. Your feedback is invaluable for improving the clarity and detail of our work. We address each of your points below:\n\n1. **Collaboration of the Dual-Branch Structure During Training**  \n\n\tWe appreciate the opportunity to clarify the synergy between our dual-branch structure. The T2I branch $V_θ$ provides a stable generation foundation, maintaining coherence and editability. The modulation branch $V_{\\theta'}$ injects specific subject features to achieve precise control. During each iteration, the input information is separately fed into the T2I branch and the modulation branch for forward propagation, and the model is optimized using the regularization loss mentioned in Section 3.3. This design achieves a balance between generation fidelity and specific subject control, enabling XVerse to realize both high-fidelity generation and accurate subject customization.\n\n2. **Computational Resources for Training**  \n\n\tWe acknowledge the importance of disclosing training resources. XVerse is trained efficiently using the following setup, leveraging parameter-efficient fine-tuning to reduce costs:\n\t- **Hardware**: 64×NVIDIA A800 (40GB) GPUs.\n\t- **Training Duration**: ~7 days total across three stages (Stage 1: 70K iterations-2days, Stage 2: 150K iterations-4 days, Stage 3: 10K iterations-1 day).\n\n3. **Ablation Studies for Regularization Losses**  \n\n\tWe conducted ablation experiments to evaluate the impacts of the CCL and CAL regularization losses, respectively, by comparing against a baseline model (which retains both losses). All experiments share identical configurations except for the regularization loss component. Owing to computational resource constraints, each experiment was executed on 16 NVIDIA A800 GPUs (40GB memory per GPU). As shown in the table, the baseline model achieves the highest average scores on both single-subject and multi-subject benchmarks. This result underscores the effectiveness of the CCL and CAL regularization losses—since ablating either loss leads to performance degradation.\n\n\t| | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n\t| EXP | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| Baseline | 30K | | 57.55 | 96.31 | 22.19 | 54.74 | 56.96 | | 52.73 | 91.25 | 18.45 | 47.12 | 54.14 |\n\t| Baseline - CCL | 30K | | 55.47 | 95.89 | 15.19 | 54.11 | 56.72 | | 50.72 | 92.01 | 10.83 | 46.31 | 53.76 |\n\t| Baseline - CAL | 30K | | 53.42 | 96.31 | 7.47 | 52.43 | 57.47 | | 44.51 | 77.43 | 6.34 | 40.22 | 54.05 |\n\n4. **Symbols in Figure 2**  \n\n\tIn Figure 2, $x$ is the latent image feature, and $c$ is the text conditioning feature. The two-branch structure illustrated in Figure 2 depicts the double-stream blocks of the Flux model, which is subsequently followed by the single-stream blocks that integrate both the text feature and the latent feature. Note that Figure 2 does not include the dual-branch architecture described in Section 3.3; we will provide a more detailed explanation in the revised version.\n\n5. **Influence of Training Dataset Size and Quality**  \n\n\tThank you for your question. We recognize that dataset scale can impact model performance. A reduced dataset scale, such as halving the current size, would likely increase the risk of overfitting to the training data, potentially undermining the model’s generalization ability to unseen categories.  \n\n\tHowever, compared to the sheer quantity of data, the diversity and quality of the dataset are more critical. Our experiments indicate that even a smaller set of high-quality cross-image data, which covers diverse scenarios and maintains cross-view consistency, can significantly enhance the model’s performance in test scenarios. This diversity enables the model to generate more natural and coherent images across different contexts.\n\n6. **Additional: More Expanded Ablation Studies with Numerical Results**\n\tWe also conducted a series of ablation studies to dissect critical components and their impacts on the framework, ensuring all experiments shared identical configurations (e.g., hyperparameters, training schedules) unless specified otherwise, and were executed on 16 NVIDIA A800 (40GB) GPUs due to computational constraints.\n\n\t- Joint Training of Text Modulation and VAE-Encoded Features  \n\n\t| | | | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n\t| EXP ID | EXP Name | Init | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| 1 | Text Modulation Only | Scratch | 30K | | 57.55 | 96.31 | 22.19 | 54.74 | 56.96 | | 52.73 | 91.25 | 18.45 | 47.12 | 54.14 |\n\t| 1 | Text Modulation Only | Scratch | 40K | | 58.30 | 96.61 | 24.01 | 56.31 | 56.29 | | 53.54 | 91.03 | 21.81 | 48.20 | 53.14 |\n\t| 2 | Single-Stream VAE Only | Scratch | 30K | | 60.71 | 95.50 | 26.77 | 62.66 | 57.92 | | 53.32 | 91.02 | 14.47 | 52.55 | 55.26 |\n\t| 2 | Single-Stream VAE Only | Scratch | 40K | | 59.60 | 94.97 | 22.93 | 62.94 | 57.59 | | 53.37 | 91.34 | 13.99 | 52.77 | 55.38 |\n\t| 3 | Text Modulation + VAE | Exp 1, 30K | 10K | | 62.62 | 96.28 | 39.53 | 57.54 | 57.14 | | 55.16 | 90.52 | 26.97 | 49.15 | 54.04 |\n\n\tWe designed three experiments to investigate joint training of text modulation and VAE-encoded feature injection. As shown in the table, Exp. 1 (Text Modulation Only) achieves average (AVG) scores of 57.55 and 52.73 on single-subject and multi-subject benchmarks, respectively, after 20K training steps. With an additional 10K training steps, its performance marginally improves to 58.30 and 53.54, respectively. In contrast, Exp. 2 (VAE-only) attains slightly higher AVG scores of 60.71 and 53.32 after 30K training steps, yet its performance deteriorates with a further 10K training steps. This observation underscores the instability and challenges of VAE feature injection, which stem from conceptual confusion and attribute blending. Notably, Exp. 3—which employs 10K joint training steps for text modulation and VAE feature injection, using Exp. 1 (30K steps) as the initial checkpoint—exhibits performance improvements from 57.55 to 62.62 (single-subject AVG) and from 52.73 to 55.16 (multi-subject AVG). Collectively, our approach substantially alleviates prevalent challenges in VAE feature injection, thereby enabling a cleaner and more disentangled generation process.  \n\n\tThe instability of pure VAE feature injection (Exp. 2) is evident from its lower ID-Similarity (ID-Sim) scores. Lacking semantic guidance, the VAE-only model often generates images that fail to retain the identity information of the reference image, resulting in inconsistent and lower-quality outputs. In contrast, text modulation, demonstrated in Exp. 3, serves as a crucial “stabilizer” and “guide”. It enables the VAE feature injection in a more controlled and semantically coherent manner, reducing conceptual errors and significantly enhancing the model’s stability and editability.\n\n\t -  VAE-encoded feature injection blocks  \n\n\t| | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|\n\t| EXP | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| Single-stream VAE Only | 30K | | 60.71 | 95.50 | 26.77 | 62.66 | 57.92 | | 53.32 | 91.02 | 14.47 | 52.55 | 55.26 |\n\t| All Blocks VAE Only | 30K | | 58.58 | 96.62 | 19.71 | 60.34 | 57.65 | | 51.35 | 88.63 | 11.15 | 50.33 | 55.29 |\n\n\tWe conducted an ablation study to dissect the role of VAE-encoded feature injection blocks in our framework. All experiments adopted identical experimental configurations except for the injection block designs (i.e., “Single-stream VAE Only” vs. “All Blocks VAE Only” as defined in the table). As shown in Table, injecting VAE-encoded features into the single-stream module of FLUX (Single-stream VAE Only) outperformed \"All Blocks VAE Only\" on both single-subject and multi-subject benchmarks. For instance, in single-subject AVG, it achieved 60.71 vs. 58.58, and in multi-subject AVG, 53.32 vs. 51.35. This highlights the superiority of targeted single-stream VAE feature injection."}, {"review_id": "YIW5jsvmBb", "reviewer": "Reviewer_CRkS", "summary": "This paper describes a method, XVerse, to perform multi-subject diffusion-based image customization. Motivated by the limitation in feature injection in attention layers, as commonly done in existing works, the authors proposed two key designs as their contributions: first, the use of learned offsets applied to text-stream modulation of DiTs to control identity and semantic attributes, and second, the use of VAE-encoded features for attention injection only in single blocks of FLUX model for fine-grained details. The authors also construct a multi-entity dataset for the purpose of training the proposed model.", "questions": "Ln 234-235: are there visual examples that show how the “cross-image data” used in the third training phase looks like. How were they curated?\n\nLn 196-201: how exactly are text-image attention loss formulated? \n\nSupp D. Is there an ablation study for the region preservation loss?\n\nFigure 3: is it hard to tell from the figure how training data is constructed. Adding step indicators, specifying the filtered output (e.g. I assume the segmented \"sidewalk\" is discarded), and adding more captions may help making the figure more clear. \n\nIt appears that in Figure 1, bottom-right corner, \"Light\" is a controllable attribute in Xverse, whereas at ln210, it is mentioned that \"non-entity terms\" are excluded. How does XVerse learn to control semantic concepts like lighting and style with this exclusion?", "limitations": "limitations are discussed. No discussion on societal impact.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is overall well-written and well grounded to improve upon recent progress in image customization.  but a few concerns remain:\n\nConcern 1: Comparison between XVerse and TokenVerse[23]\n- Much of the proposed method section 3.2 appears highly similar to TokenVerse, in terms of adding offset to token embeddings to adjust the modulation parameters, and the design to perform this injection in the W+ space. It'd help if the authors discuss difference to this work, and, despite slight difference in customization setting, compare experiment results between the two in concept-driven image generation (where concepts come from reference images, which should be supported by the proposed pipeline).\n\nConcern 2: Copy-and-paste effect\n- Figure 5: Visual results of XVerse look relatively realistic, but display more “copy-and-paste” effects compared to baselines like DreamO, which may partially explain its relatively high performance in similarity scores. It’d be useful to show diverse samples given the same prompts to shed light on the learned target distribution.\n\nConcern 3: clarity in the method section\n- the design to inject features for fine-grained details are vaguely explained. Is this done as a feature injection into the attention layers? If so, how does it mitigate the argued weakness of leveraging attention mechanism in image customization (ln21-27)?\n\nConcern 4: quantitative results are missing in ablation study. \n\nBased on the above concerns, the novelty of the paper appears incremental (primarily combining the designs of TokenVerse[23] and Omnicontrol[11]) but creditable. The effectiveness of the method can be validated by the evaluation, though some additional comparisons could strengthen the claim (e.g. comparison to TokenVerse, quantitative ablation study). Some details of the method are also missing, which may be a problem for reproducibility. Overall, I consider this an incremental work that shows creditable improvement on multi-subject image editing.", "was_revised": true, "final_justification": "Based on the other reviewers' feedback and the authors' responses, I believe this work provides a solid improvement over multi-subject image editing, and thus I recommend an accept rating, on the ground that the authors will incorporate clarifications and additional details they provided in the rebuttal process.", "rebuttal": "Thank you for your thorough review and insightful questions. Your feedback has been invaluable in improving the clarity and depth of our work. We have responded to each of your comments below:\n\n1. **Distinction from TokenVerse and Comparative Analysis**  \n\n\tThe core difference between XVerse and TokenVerse [23] lies in tuning requirements and generalization capability:  \n\t- **TokenVerse**: Relies on fine-tuning on small-scale predefined datasets to learn subject-specific modulations, requiring reference one image per concept for overfitting-based customization. This limits its applicability to unseen subjects without retraining.  \n\t- **XVerse**: Uses an adapter to convert reference images and their descriptions into token-specific offsets. By converting reference images into token-specific offsets, XVerse achieves control without test-time tuning, eliminating the need for training on specific subjects. As a result, XVerse can generalize any concept from reference images after just a single training.  \n\n2. **Addressing \"Copy-and-Paste\" Effects with Diverse Samples**  \n\n\tThe evaluation in XVerseBench all uses the average results of 4 seeds, which, to some extent, can evaluate the stability of the model under the same prompt. From the visualization results, it can also be seen that compared to methods like OmniGen, Uno, DreamO that only use VAE, our model maintains better editability and stability. Subsequently, we will add some generation results corresponding to different seeds under the same prompt in the supplementary documentation to further demonstrate the model's stability.  \n\n\tDuring training, we did observe the copy-paste phenomenon and took targeted measures to mitigate it: the third training phase specifically incorporated cross-image data to foster more robust appearance mapping across diverse visual conditions. As our work prioritized methodological innovations, we acknowledge that the cross-image dataset, while effective to a degree, lacks the scale and diversity needed to fully eliminate such effects (as mentioned in the Limitations part). In future work, we aim to further increase data diversity, which will help minimize the copying effect and enhance the originality of the generated images.  \n\n3. **Clarification on Fine-Grained Feature Injection**  \n\n\tThank you for your valuable comment. The features are directly injected into the attention of single-stream blocks. Notably, within the self-attention of double-stream modules, a deliberate design choice is made: while latents and text features cannot interact with VAE features, VAE features retain the ability to perceive latents and text features for effective feature aggregation. This mechanism prevents the degradation that would occur if VAE features were directly inserted into the single stream. We will strengthen the more detailed description of VAE injection in the revised version.  \n\n\tTo mitigate the argued weakness of leveraging the attention mechanism in image customization, we have adopted the following strategies:  \n\t- We only perform feature injection in the single-stream blocks, while using text-modulation adapters in the double blocks for semantic control. As shown in the supplementary ablation experiments, global injection would lead to a significant decline in the naturalness of the generated images and cause obvious feature fusion in multi-condition scenarios.  \n\t- Two regularization losses are introduced to constrain the application scope of the VAE. This helps the model better identify the regions where different conditions should be injected, thereby reducing confusion and fusion.  \n\n4. **More Quantitative Ablation Study**  \n\n\tWe conducted a series of ablation studies to dissect critical components and their impacts on the framework, ensuring all experiments shared identical configurations (e.g., hyperparameters, training schedules) unless specified otherwise, and were executed on 16 NVIDIA A800 (40GB) GPUs due to computational constraints.  Due to the limitation on the length, we are unable to provide the complete ablation implementation details in this response. You can refer to our reply to Reviewer 1 for more details on the quantitative metrics and analysis.\n\n5. **More details of method for reproducibility**  \n\n\tThank you for your valuable feedback. In the revised version, we will enhance the supplementary document by providing more comprehensive details, including the hyperparameter settings and detailed training configurations. Additionally, we plan to opensource the model and related code in the future, which we believe will further facilitate reproducibility and foster further research in this area.  \n\n6. **Formulation for text-image attention loss**  \n\n\tThank you for your question. To more effectively constrain the region of feature injection, we performed two forward propagations in each iteration: one for text-to-image generation, and the other for controlled subject generation. The text-image attention loss in XVerse is formulated as an L2 loss with normalization over the text-image cross-attention maps of the modulated model and the reference T2I branch. Specifically, it encourages the modulated model to retain attention patterns that closely match those of the T2I branch, ensuring that semantic interactions between text and image regions remain consistent and editable.  \n\n\tMathematically, it can be expressed as:  \n\t$$L_{attention}=E\\left[\\left\\| Norm\\left(A_{\\theta \\prime}\\right)-Norm\\left(A_{\\theta}\\right)\\right\\| ^{2}\\right],$$ \n\twhere $A_{\\theta'}$ denotes the text-image cross-attention maps of XVerse's modulated branch, $A_{\\theta}$ denotes those of the T2I branch, and $Norm(\\cdot)$ represents the normalization operation applied to the attention maps. You can also refer to Sections D and E in the supplementary document to help understand these two regularization loss functions.  \n\n7. **More details for the training data**\n\t  \n\tFor human-centric cross-image data, we collected a 100K-scale single-person multi-view dataset from proprietary in-house collections, forming up to three image pairs per subject ID. These were curated by clustering facial images and selecting diverse views (e.g., facial and full-body shots) to enhance generalization. For general objects, we leveraged the Subject200K dataset, applying DINOv2 similarity filtering to ensure cross-view consistency.  \n\n\tThe data curation involved generating image descriptions, detecting/segmenting subject terms using Florence2 and SAM2, and aligning conditioning images with subject terms via DINOv2 scores to form cross-image pairs. This process ensures robust appearance mapping under heterogeneous visual conditions.  \n\n8. **How does XVerse learn to control semantic concepts like lighting and style with this exclusion?**  \n\n\tThank you for your question. We excluded non-entity terms (such as \"light\" and \"style\") during data processing primarily to enhance the accuracy of detection and segmentation for entities, which is crucial for effectively learning subject-specific features. XVerse's text flow modulation mechanism enables precise control over semantic concepts such as lighting and style. This is thanks to text flow modulation operating at the semantic level of text prompts, allowing the model to naturally extend control from entity features to related semantic attributes without relying on enough explicit non-entity annotations in the training data."}, {"review_id": "zb8vHoeo1M", "reviewer": "Reviewer_yqTe", "summary": "This paper investigates the subtask of text-to-image generation—target customization—by projecting target identities into the text modulation space of Flux to achieve this goal.", "questions": "See Weaknesses", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Advantages:\n1. The paper is fluently written with clear expressions, and the figures are appealing.\n\nDisadvantages:\n1. My primary concern is that projecting target identities into Flux's text modulation space, in my view, is fundamentally identical in principle to the numerous previous methods that projected target identities into the original text representation space. The only difference is that the modulation space here is a secondary projection of the text space. This explains why, similar to past approaches, projecting target identities solely into the text space results in insufficient fidelity, necessitating the concatenation of image VAE features.\n2. Regarding the ablation study, I am more interested in the differences between projecting target identities into Flux's original text representation space and its text modulation space.\n3. Using VAE-encoded image features to enhance target identity fidelity has become a widely adopted technique.\n4. The generated images in this paper exhibit a pronounced \"copying\" effect. As shown in Figure 1, the characters' postures and expressions are highly similar to the reference images.\n5. Here's a question: In the current task of target-customized characters, do you think network architecture or data is more important? I speculate that a primary reason for the obvious copying effect in this paper is the insufficient diversity of target samples during data construction. Although the third-stage training in this paper aims to alleviate this issue, the problem persists.", "was_revised": true, "final_justification": "Having carefully studied the author's rebuttal, their response has largely dispelled my doubts. The author frankly explained the \"Copy-and-Paste\" effect and appropriately discussed the importance of network architecture and data. I am well aware of the difficulties involved here, and on the whole, this paper has achieved a good level of completion. Therefore, I have finally decided to change the result to \"borderline acceptance\".", "rebuttal": "Thank you for your comprehensive review and thoughtful questions. Your feedback is instrumental in enhancing the clarity and depth of our work. We have addressed each of your comments below:\n\n1. **Differences from text representation space feature injection methods**  \n\n\tThank you for your insightful observation. While there may seem to be similarities in projecting identities into text-related spaces, our approach differs fundamentally from prior methods like PhotoMaker.  \n\n\tPrevious strategies often directly modify token values through addition or subtraction, which can disrupt the model's inherent text-to-image structure and compromise editability. In contrast, XVerse introduces perturbations to the distribution within the text modulation space, minimizing interference with the base model's structural integrity.  \n\n\tAs visualized in Figure 6, our text-stream modulation effectively injects semantic features from reference images while preserving the original structure of text-to-image outputs. This distinction allows us to maintain strong editability while achieving high fidelity, reducing the reliance on VAE features compared to approaches that alter token values directly.\n\n2. **Using VAE-encoded image features to enhance target identity fidelity has become a widely adopted technique**  \n\n\tWe appreciate the reviewer's attention to our use of VAE. We agree that VAE-encoded features are widely adopted to enhance identity fidelity, but our core innovation lies not in \"using VAE\" but in a novel integration mechanism that addresses limitations of existing methods. Categorizing all VAE-based schemes as identical overlooks critical methodological differences.  \n\n\tRecently, VAE-only methods demonstrate strong performance in specific settings, particularly excelling at controlled generation for individual subjects. However, when extended to multi-subject control scenarios, they often have difficulty preserving consistent identity fidelity across multiple subjects, frequently resulting in blurred or merged identities.  \n\n\tOur XVerse solves these issues through two key innovations:  \n\t- **Hybrid Injection Strategy**: Instead of global injection: the Double-stream module uses a text-modulation adapter for precise control over specific subject or high-level semantics (posture, shape). The Single-stream module receives direct VAE injection for the refinement of local details/textures. This enables implicit learning of injection regions without explicit attention masks, thereby preserving identity while protecting image structure.  \n\t- **Novel Regularization Losses**: These guide the model to distinguish identity-controlled vs. text-controlled regions, achieving clearer disentangled representations and mitigating attribute blending.  \n\n\tOur ablation experiments (Due to the limitation on the length, we are unable to provide the complete ablation implementation details in this response. You can refer to our reply to Reviewer 1 for more analysis) confirm this: a VAE-only model performs far worse than our complete XVerse, proving that our hybrid injection and regularization losses together enable high-fidelity, controllable generation—our core innovation.  \n\n\t| | | | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n\t| EXP ID | EXP Name | Init | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| 1 | Text Modulation Only | Scratch | 30K | | 57.55 | 96.31 | 22.19 | 54.74 | 56.96 | | 52.73 | 91.25 | 18.45 | 47.12 | 54.14 |\n\t| 1 | Text Modulation Only | Scratch | 40K | | 58.30 | 96.61 | 24.01 | 56.31 | 56.29 | | 53.54 | 91.03 | 21.81 | 48.20 | 53.14 |\n\t| 2 | Single-Stream VAE Only | Scratch | 30K | | 60.71 | 95.50 | 26.77 | 62.66 | 57.92 | | 53.32 | 91.02 | 14.47 | 52.55 | 55.26 |\n\t| 2 | Single-Stream VAE Only | Scratch | 40K | | 59.60 | 94.97 | 22.93 | 62.94 | 57.59 | | 53.37 | 91.34 | 13.99 | 52.77 | 55.38 |\n\t| 3 | Text Modulation + VAE | Exp 1, 30K | 10K | | 62.62 | 96.28 | 39.53 | 57.54 | 57.14 | | 55.16 | 90.52 | 26.97 | 49.15 | 54.04 |\n\n\t| | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n\t| EXP | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| Baseline | 30K | | 57.55 | 96.31 | 22.19 | 54.74 | 56.96 | | 52.73 | 91.25 | 18.45 | 47.12 | 54.14 |\n\t| Baseline - CCL | 30K | | 55.47 | 95.89 | 15.19 | 54.11 | 56.72 | | 50.72 | 92.01 | 10.83 | 46.31 | 53.76 |\n\t| Baseline - CAL | 30K | | 53.42 | 96.31 | 7.47 | 52.43 | 57.47 | | 44.51 | 77.43 | 6.34 | 40.22 | 54.05 |\n\n\t| | | | | | Single | | | | | | Multi | | |\n\t|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|\n\t| EXP | Steps | | AVG | DPG | ID-Sim | IP-Sim | AES | | AVG | DPG | ID-Sim | IP-Sim | AES |\n\t| Single-stream VAE Only | 30K | | 60.71 | 95.50 | 26.77 | 62.66 | 57.92 | | 53.32 | 91.02 | 14.47 | 52.55 | 55.26 |\n\t| All Blocks VAE Only | 30K | | 58.58 | 96.62 | 19.71 | 60.34 | 57.65 | | 51.35 | 88.63 | 11.15 | 50.33 | 55.29 |\n\n\tIn summary, we build on VAE but solve its existing limitations via innovative architecture and specialized losses.\n\n3. **\"Copy-and-Paste\" Effects**  \n\n\tWhen there is no text controlling semantics such as expressions and poses, the model mimics the characteristics of the given condition image. However, as shown in Figure 1, our model can still retain a certain ability to edit semantics such as pose and lighting.  \n\n\tDuring training, we did observe the copy-paste phenomenon and took targeted measures to mitigate it: the third training phase specifically incorporated cross-image data to foster more robust appearance mapping across diverse visual conditions. As our work prioritized methodological innovations, we acknowledge that the cross-image dataset, while effective to a degree, lacks the scale and diversity needed to fully eliminate such effects (as mentioned in the Limitations part). In future work, we aim to further increase data diversity, which will help minimize the copying effect and enhance the originality of the generated images.  \n\n4. **The importance of Network Architecture or Data**  \n\n\tWe recognize that both model structural optimization and data updates are critical to performance improvement. Our experiments confirm that even a smaller set of high-quality cross-image data can serve as a strong foundation for enhancing model performance in test scenarios, as such diversity helps the model generate more natural and coherent images across contexts. However, the transformative leap in our model’s capabilities stems from its structural innovations: by leveraging text modulation, we preserve the aesthetic quality and structural integrity inherent to text-to-image generation, while our strategic use of regularization losses and constrained VAE feature injection positions effectively mitigates the \"copy-paste\" artifacts often induced by VAE. These design choices enable the model to fully capitalize on high-quality data while overcoming key limitations that would otherwise hinder generation quality.  \n\n\tTo further validate the distinctiveness of our model design, we conducted an ablation study comparing various methods on the same dataset. The results, presented in response 2, clearly illustrate that the VAE module and the Text-Modulation adapter serve complementary functions. This controlled experiment underscores that the architectural innovations in our model are crucial for fully leveraging the data’s potential, rather than depending solely on data quality."}]

Author rebuttal:

Meta-review: This paper receives positive ratings of (4, 4, 5, 5). The reviewers generally acknowledge the performance of the proposed method, but raised on questions on ablations, designs, and writing. After reading the paper, review and rebuttal, the AC finds no reason to overturn the decision of reviewers. An acceptance is recommended. The authors are advised to revise the paper according to the reviewers' comment.
Decision: Accept (poster)