Title: Rare Text Semantics Were Always There in Your Diffusion Transformer
Venue: NeurIPS 2025
Accepted: yes
Keywords: generative models

diffusion transformer

text-to-vision
Abstract: Starting from flow- and diffusion-based transformers, Multi-modal Diffusion Transformers (MM-DiTs) have reshaped text-to-vision generation, gaining acclaim for exceptional visual fidelity. As these models advance, users continually push the boundary with imaginative or rare prompts, which advanced models still falter in generating, since their concepts are often too scarce to leave a strong imprint during pre-training. In this paper, we propose a simple yet effective intervention that surfaces rare semantics inside MM-DiTs without additional training steps, data, denoising-time optimization, or reliance on external modules (e.g., large language models). In particular, the joint-attention mechanism intrinsic to MM-DiT sequentially updates text embeddings alongside image embeddings throughout transformer blocks. We find that by mathematically expanding representational basins around text token embeddings via variance scale-up before the joint-attention blocks, rare semantics clearly emerge in MM-DiT’s outputs. Furthermore, our results generalize effectively across text-to-vision tasks, including text-to-image, text-to-video, and text-driven image editing. Our work invites generative models to reveal the semantics that users intend, once hidden yet ready to surface.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ToRA (Token Spacing and Residual Alignment), a training-free method for improving the generation of rare semantic concepts in Multi-modal Diffusion Transformers (MM-DiTs). The core insight is that rare semantics already exist within the text embeddings but are not effectively accessible during generation. The method operates by expanding representational basins around text token embeddings via variance scale-up before joint-attention blocks. TORA consists of two components: (1) Token Spacing, which scales up variance in the principal semantic space using PCA and singular value decomposition, and (2) Residual Alignment, which rotates the residual space toward semantic directions to mitigate potential side effects. The method is evaluated on text-to-image generation (using RareBench, T2I-CompBench, GenEval), text-to-video generation (CogVideoX), and text-driven image editing (Stable Flow).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**Strong empirical performance across multiple tasks**: The method shows consistent improvements across text-to-image generation (Table 1), text-to-video generation (Figure 8), and text-driven image editing (Table 2). On RareBench, TORA achieves substantial gains over baselines across all categories when evaluated with both GPT-4o and human evaluation.\n\n**Training-free**: Unlike existing methods that require finetuning (Figure 2a), optimization during inference (Figure 2b), or external LLM modules (Figure 2c), TORA requires no additional training.\n\n\n**Comprehensive evaluation framework**: The evaluation spans multiple benchmarks (RareBench for rare concepts, T2I-CompBench and GenEval for common prompts) and includes both automated metrics and human evaluation, providing robust evidence for the method's effectiveness.\n\n**Good compatibility with existing methods**: The paper demonstrates that TORA can be combined with existing approaches like R2F, showing improved performance in \"R2F + Ours\" configurations (Table 1), indicating practical compatibility.\n\n## Weaknesses\n**Clarity**: the manuscript is sometimes vague and misses details. For example:\n* In Section 3.1, the authors are not specific enough regarding when PCA is computed, whether it is precomputed using many samples and fixed afterwards, or whether it is computed dynamically for each new batch. Additionally, it is not clear whether the PCA decomposition is shared between transformers blocks, although I assume a different PCA is computed for each layer.\n* The writing style is non-standard and difficult to follow on a first read. The paper would benefit from a clearer separation between background, methodology, and experiments. For example, Section 2 mixes background information with experimental results, making it hard to distinguish prior work from novel contributions. This lack of structure is especially challenging for readers unfamiliar with interventions on the neural network activations, such as those proposed in this manuscript.\n* Another area lacking clarity is the discussion of anisotropy and isotropy in the introduction (third paragraph). The explanation of these concepts is vague, particularly regarding why, after clustering, the cosine similarity within clusters is low while the global average similarity is high. This is counterintuitive, as clustering based on cosine similarity should group similar tokens together. The authors should provide a clearer explanation for this phenomenon in the background.\n\n**Baseline comparisons**: The paper primarily compares against base models and R2F. It would be relevant to compare ToRA with methods that fine-tune the diffusion transformer or perform inference optimizations. Even if the method does not beat those baselines, it would be important for practitionners to understand how large the gap would be.\n\n**Figure quality issues**: Figure 2 contains elements that are not clearly explained. Specifically, it is not clear what the \"rotating arrow\" in Figure 2b represents.\n\n\n\nOverall, the empirical results seem strong, however the background section deserves a major rewrite in my opinion.\n\nQUESTIONS:\n**PCA**: When is the PCA computed in practice? Is it pre-computed using a fixed dataset, computed per-batch, or computed dynamically for each input?\n\n**Computational overhead**: What is the actual computational cost of applying TORA during inference? The paper claims \"low-overhead\" but provides no quantitative analysis.\n\n**Failure mode analysis**: Under what conditions does the method fail or produce worse results? The authors do not expand on the failure modes.\n\nLIMITATIONS:\nThe discussion of limitations is deferred to the appendix. Naturally, the clarity of the writing is not discussed as a limitation, although it is the strongest limitation in my opinion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

comments: SUMMARY:
This paper investigates the generation of rare prompts in multi-modal diffusion transformers. Without requiring additional training or external modules, this paper finds that rare semantics can merge from MM-DiT by proposing a variance scale-up method. Experimental results show that the proposed method achieves strong performance in various tasks, from text-to-image generation to image editing.

STRENGTHS AND WEAKNESSES:
**Strength:**
1. The paper presents an insightful analysis on the isotropic properties of rare semantics.
2. Without requiring additional training, the proposed ToRA method significantly improves text-image alignment for rare prompts.
3. This work conducts extensive experiments to support its claims and performance improvements. 
3. The paper is clearly written and well presented. 

**Weaknesses:**
1. The proposed ToRA method performs well on rare prompts. How does it perform on common prompts?
2. Do common and rare prompts require different parameters or settings? If so, in real applications, how does the system determine whether a given prompt is rare or common?
3. Equation 2 shows that ToRA scales the top-k singular values. Is k determined by the entire Rarebench dataset, or set differently for different prompts? How does k influence the performance using different groups of prompts, similar to Figure 10(a)?

QUESTIONS:
Please refer to the strength and weaknesses.

LIMITATIONS:
This paper discusses the limitations but does not address the broader impact.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
This paper proposes a simple method to help Multi-modal Diffusion Transformers better handle rare or unusual prompts in text-to-vision tasks. By adjusting the text embeddings before attention using the proposed principle of variance scale up, the model can bring out rare concepts more clearly, without needing extra data, training, or tools.

STRENGTHS AND WEAKNESSES:
# Strengths

1. The paper is overall well written and clearly structured.
2. The proposed method operates without requiring external knowledge, enhancing its general applicability across tasks.
3. The method demonstrates superior performance compared to baseline approaches in the reported experiments.

# Weaknesses

These are not necessarily weaknesses, but addressing them may help strengthen the paper:

1. It appears that the parameter $k$ in Equation (2) plays a significant role in the proposed method. An ablation study on $k$ would help clarify its impact on performance.
2. In line 175, the authors mention "decomposing the space of original text embeddings" using PCA. PCA typically requires a set of vectors—could the authors clarify what this set consists of in this context?
3. The exposition in Section 3.2 could benefit from further revision. In particular, the definition of $G_{\theta}$ in Equation (3) is unclear. Do the authors intend for $G_{\theta}$ to denote any matrix satisfying Equation (3), or is it defined in a more specific way?
4. What would be the performance if $\sigma < 1$, for example $\sigma = 0.5$ or $\sigma = -1.3$? Some clarification or empirical discussion would help readers understand the the impact of the variance scaling up under such settings.

QUESTIONS:
see Strengths And Weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This work introduces TORA, a training-free method to enhance rare semantic emergence by expanding text token embeddings’ representational basins via variance scale-up before joint-attention blocks. By amplifying local isotropy and refining residual alignment, TORA improves semantic distinguishability without external modules or retraining. It achieves robust generalization across text-to-image, text-to-video, and image editing tasks, demonstrating superior performance on rare prompts while preserving common concept accuracy. The method’s low computational overhead and compatibility with existing pipelines make it a practical solution for boosting MM-DiT’s semantic fidelity. All code will be publicly released.

STRENGTHS AND WEAKNESSES:
Strengths:
1. TORA achieves rare semantic alignment without requiring additional training, data, or external modules.
2. TORA enhances local isotropy in text embeddings by expanding representational basins through variance scale-up and refining residual alignment.
3. TORA generalizes effectively across diverse text-to-vision tasks (text-to-image, text-to-video, and image editing) and achieves state-of-the-art performance on rare prompts.

Weaknesses:
Hard to find many weaknesses, it's a strong paper that is well written, with a clear argument, clear analysis, clear solution that seems to work after both quantitative and qualitative results.

QUESTIONS:
1. Does ToRA work for other problems like missing objects or wrong numbers of objects?

LIMITATIONS:
yes, the limitations have been addressed
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03