Title: Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion models

text-to-image diffusion models

text-to-image generation
Abstract: Text-to-image diffusion models rely on text embeddings from a pre-trained text encoder, but these embeddings remain fixed across all diffusion timesteps, limiting their adaptability to the generative process. We propose Diffusion Adaptive Text Embedding (DATE), which dynamically updates text embeddings at each diffusion timestep based on intermediate perturbed data. We formulate an optimization problem and derive an update rule that refines the text embeddings at each sampling step to improve alignment and preference between the mean predicted image and the text. This allows DATE to dynamically adapts the text conditions to the reverse-diffused images throughout diffusion sampling without requiring additional model training. Through theoretical analysis and empirical results, we show that DATE maintains the generative capability of the model while providing superior text-image alignment over fixed text embeddings across various tasks, including multi-concept generation and text-guided image editing. Our code is available at https://github.com/aailab-kaist/DATE.


Peer reviews:
comments: SUMMARY:
The paper introduces Diffusion Adaptive Text Embedding (DATE), a method that improves text-to-image diffusion models by dynamically updating text embeddings at each diffusion timestep. Unlike traditional approaches with fixed embeddings, DATE refines embeddings based on intermediate image data during sampling, enhancing text-image alignment without additional model training. Theoretical and empirical results show that DATE maintains generative quality while improving performance in tasks like multi-concept generation and text-guided editing.

STRENGTHS AND WEAKNESSES:
Pros:
1. The paper presents an interesting formulation of the problem and offers a unique perspective.
2. The overall writing is clear and easy to follow.
3. The results seem promising in some cases.

Cons:
1. The results presented in this paper are limited to some easy cases. As the authors mainly focus on optimizing the text embedding, they should put more attention on complex, long texts to show the improvement they claim to achieve.
2. Additionally, there are actually many powerful image generation models like GPT-4o, Gemini, FLUX that can achieve good image generation quality while accurately following the text prompts. The authors should compare with them (although some are not open-sourced, they can be inferenced for comparison).
3. They authors use CLIP Score or ImageReward as the optimization function but also use them as the evalution may lead the optimization into sub-optimal and overfitiing. The authors should explain its validity.

QUESTIONS:
Can the method be transferred to other stronger base models like SD-XL, FLUX or other DiT-based image generation models? If so, I would like to see how actually it improves.

LIMITATIONS:
Please see the weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Diffusion Adaptive Text Embedding (DATE), a novel method for improving text-to-image diffusion models by dynamically updating text embeddings during the sampling process.  Experimental results across several tasks, including t2i generation, multi-concept generation, and text-guided image editing.\n\nSTRENGTHS AND WEAKNESSES:\nThanks to the authors for their efforts in this work. \n\n**Strengths**    \nThis manuscript is well-presented, with comprehensive experimental results and solid theoretical analysis. The idea that updates the textual embedding with an optimization strategy is novel. The experiments are conducted across different network architectures (e.g., Stable Diffusion v1.5 and PixArt-α), and include detailed ablation studies. In addition, the authors provide a clear theoretical formulation of their method, including optimization objectives and update strategies.\n\n**Weaknesses**    \n1. Despite the effective optimization formulation, the proposed framework relies on the evaluation function $h$, such as CLIP score,  which is known to exhibit biases [1]. Such reliance may limit the robustness or generalizability of the method under varying evaluation metrics. while it also has a trade-off of the Clip score and ImageReward.\n2. Some arguments are overclaims; e.g., the related work section (lines 70–72) lacks discussion of relevant approaches such as SD3 [2], which utilizes multiple text embeddings.\n3. There is also some related work on text embedding tuning [3], text-to-image generation based on the evaluation function [4], which needs to be discussed.\n4. A key concern is that, although the experimental results demonstrate that the proposed embedding tuning improves performance somehow, it alters the embedding space of the original input text. In other words, it may deviate from the user's original intent in generation.\n\n$\\small{\\text{ [1] Ruiz, Nataniel, et al. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.}}$\n$\\small{\\text{ [2] Esser, Patrick, et al. \"Scaling rectified flow transformers for high-resolution image synthesis.\" Forty-first international conference on machine learning. 2024.}}$   \n$\\small{\\text{ [3] Gal, Rinon, et al. \"An image is worth one word: Personalizing text-to-image generation using textual inversion.\" arXiv preprint arXiv:2208.01618 (2022).}}$     \n$\\small{\\text{ [4] Fan, Ying, et al. \"Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.\" Advances in Neural Information Processing Systems 36 (2023): 79858-79885.}}$\n\nQUESTIONS:\nSee the Weakness part.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Diffusion Adaptive Text Embedding (DATE), a method for dynamically updating text embeddings at each diffusion sampling timestep in text-to-image diffusion models. Existing methods typically use a fixed text embedding throughout the reverse diffusion process, which may lead to suboptimal semantic alignment. DATE refines the text embedding per timestep based on intermediate perturbed images by optimizing a differentiable image-text alignment score (e.g., CLIP score or ImageReward). The paper formulates this as a constrained optimization problem and proposes a lightweight gradient-based update rule that does not require additional model training. DATE is model-agnostic, efficient (can be applied at test time), and improves performance on tasks such as multi-concept generation and text-guided image editing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n* The idea of introducing time-varying text embedding updates during the diffusion sampling process is both innovative and intuitive Furthermore, optimizing the alignment score at intermediate steps of the diffusion process, rather than only at the final output, is a novel training strategy. The method is accompanied by theoretical derivations. \n* The paper presents well-designed experiments and extensive ablation studies.\n\nWeaknesses:\n\n* Although the authors employ strategies such as sparse updates to mitigate runtime, DATE still incurs a noticeable overhead compared to standard diffusion sampling (over 30% inference time). This may limit its applicability in resource-constrained settings. \n\n* The effectiveness of DATE is dependent on the choice of reward function (CLIP, ImageReward). There exists a risk of overfitting to proxy metrics, which may not fully reflect human judgment or semantic correctness.\n\nQUESTIONS:\n1. How well does DATE generalize to long and complex prompts? Are there cases where it introduces instability or performance degradation? \n2. Does the stronger text-image alignment induced by DATE reduce image diversity or cause overfitting to the reward function? \n3. Can DATE be effectively combined with prompt tuning or soft prompt methods? Would such combinations be complementary or interfere with each other?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Diffusion Adaptive Text Embedding (DATE), a framework for improving text-to-image diffusion models by dynamically updating the text embedding at each diffusion timestep. Unlike traditional models that use a fixed text embedding across all timesteps, DATE adapts the embedding based on the current denoised image, enhancing semantic alignment. The update process is guided by maximizing a text-image alignment metric using a gradient-based optimization. Theoretical analysis supports that DATE improves alignment without degrading generative quality. Experiments show consistent performance gains across tasks such as multi-concept generation and text-guided image editing.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nS1. This paper presents a clearly motivated framework for text-to-image generation using diffusion models. Guiding the diffusion model with adaptive text embeddings is both intuitive and effective, since it does not require the training of the diffusion model parameters.\n\nS2. The proposed method is based on strong mathematical formulation (Sec. 3.1. ~ 3.2.), supported by theoretical analysis (Sec. 3.3., Appendix A), which makes the proposed method solid.\n\nS3. The experimental results (including supplementary materials) are impressive. DATE outperforms baselines in FID [1], CLIP Score [2] and ImageReward [3]. Notably, optimizing the method with a specific metric (e.g., CLIP Score [2]) still leads to improvements across other evaluation metrics, including FID [1] and ImageReward [3].\n\nS4. The proposed method demonstrates good applicability and generalizability. DATE can be applied for both text-guided image editing and multi-concept generation, and can be incorporated into orthogonal prior work (e.g. DDPM Inversion [4]), further extending its versatility.\n\n**Major Weakness**\n\nW1. The comparison with prior work is limited. Baselines including prompt optimization-based methods [5, 6, 7], reward-based methods [8], and other guidance strategies [9, 10] should be involved for a more comprehensive and strong evaluation. Is optimizing the text embedding truly the most effective approach for improving text-to-image diffusion models, compared to other techniques?\n\nW2. Along with W1, the paper should better clarify its novelty compared to prior prompt optimization-based works, especially with regard to what DATE offers beyond existing strategies.\n\n\n**Minor Weakness**\n\nW3. The paper would benefit from visualizing diverse qualitative results from both the baseline algorithms and the proposed method for a more fair comparison.\n\nW4. DATE requires x2.5 GPU memory for a batch size of 4 (Table 8) compared to the method using fixed text embedding. Is there a lightweight method (revised version) that can reduce the GPU memory usage?\n\nReferences\n\n[1] Heusel, Martin, et al. \"Gans trained by a two time-scale update rule converge to a local nash equilibrium.\" in NeurIPS (2017).\n\n[2] Hessel, Jack, et al. \"Clipscore: A reference-free evaluation metric for image captioning.” in EMNLP (2021).\n\n[3] Xu, Jiazheng, et al. \"Imagereward: Learning and evaluating human preferences for text-to-image generation.\" in NeurIPS (2023).\n\n[4] Huberman-Spiegelglas, Inbar, Vladimir Kulikov, and Tomer Michaeli. \"An edit friendly ddpm noise space: Inversion and manipulations.\" in CVPR (2024).\n\n[5] Mo, Wenyi, et al. \"Dynamic prompt optimizing for text-to-image generation.\" in CVPR (2024).\n\n[6] Um, Soobin, and Jong Chul Ye. \"Minority-Focused Text-to-Image Generation via Prompt Optimization.\" in CVPR (2025).\n\n[7] Hao, Yaru, et al. \"Optimizing prompts for text-to-image generation.\" in NeurIPS (2023).\n\n[8] Kim, Semin, et al. \"Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models.\" arXiv preprint (2025).\n\n[9] Sadat, Seyedmorteza, et al. \"CADS: Unleashing the diversity of diffusion models through condition-annealed sampling.\" in ICLR (2024).\n\n[10] Um, Soobin, and Jong Chul Ye. \"Self-guided generation of minority samples using diffusion models.\" in ECCV (2024).\n\nQUESTIONS:\nQ1. The authors used two metrics (CLIPScore and ImageReward) for optimization. Can another metric can be applied for optimization?\n\nQ2. For further questions and concerns, please refer to the Major and Minor Weaknesses sections above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}