A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers
Authors: Trung X. Pham
Kang Zhang
Ji Woo Hong
Chang D. Yoo
Venue: ICLR 2026 Poster (2026)
Keywords: conditional embeddings
diffusion models
generative AI
transformer-based diffusion
sparse representation learning
efficient learning
Primary area: generative models
TL;DR: Conditional embeddings in diffusion Transformers are highly redundant, with semantics concentrated in a few dimensions, enabling large-scale pruning without harming generation quality.

Abstract: Diffusion Transformers have achieved state-of-the-art performance in class-conditional and multimodal generation, yet the structure of their learned conditional embeddings remains poorly understood. In this work, we present the first systematic study of these embeddings and uncover a notable redundancy: class-conditioned embeddings exhibit extreme angular similarity, exceeding 99% on ImageNet-1K, while continuous-condition tasks such as pose-guided image generation and video-to-audio generation reach over 99.9%. We further find that semantic information is concentrated in a small subset of dimensions, with head dimensions carrying the dominant signal and tail dimensions contributing minimally. By pruning low-magnitude dimensions--removing up to two-thirds of the embedding space--we show that generation quality and fidelity remain largely unaffected, and in some cases improve. These results reveal a semantic bottleneck in Transformer-based diffusion models, providing new insights into how semantics are encoded and suggesting opportunities for more efficient conditioning mechanisms.

Published as a conference paper at ICLR 2026
A HIDDEN SEMANTIC BOTTLENECK IN CONDITIONAL EMBEDDINGS OF DIFFUSION TRANSFORMERS
Trung X. Pham, Kang Zhang, Ji Woo Hong, Chang D. Yoo* Korea Advanced Institute of Science and Technology (KAIST) (trungpx, zhangkang, jiwoohong93, cd_yoo)kkiist.ac akaist.ac.krr
ABSTRACT
Diffusion Transformers have achieved state-of-the-arl performance class- conditional and multimodal generation, yet the structure of their learned conditional embeddings remains poorly understood. In this work, we present the first systematic study of these embeddings and uncover notable dundancy: class-conditioned embeddings exhibit extreme angular similarity, exceeding 99% ImageNet-1K,, while condiion tasks such pose-guided image generation and video-to-audio generation reach over 99.9%. We further find that semantic information concentrated small subset of dimensions, with head dimensions carrying the dominant signal and tail dimensions contributing minimally. By pruning low-magnitude dimensions-remoovngg removing up to two-thirds of the embedding space-we we show that generation quality and fidelity remain largely unaffected, and some cases improve. These results reveal semantic bottleneck in Transformer- based diffusion models, providing new insights into how semantics are encoded and suggesting opportunities for more efficient conditioning mechanisms.
INTRODUCTION
$6.2%%
Transformer-basee diffusion models have re- cently emerged tatt-off-the-art architec- tures for generative modeling tasks across diverse domains, including class-conditional image synthesis DiT (Peebles & Xie, 2023), MDT (Gao et al., 2023), SIT (Ma et al., 2024), DiT (Yao et al., 2025), Model-Guidance (MG) (Tang et al., 2025), REPA (Yu et al., 2025), poso-guided per- son image generation (Pham al., 2024), and video-to-audio generation (Pham et al., 2025b). These models combine the expres- sive capacity of Trannformer backbones with diffusion processes to generate figh-fiielity, semantically consistent outputs. A key com- ponent of such models is the conditional em- bedding vector, often formed by summing class label and timestep embeddings and injected via adaptive layer normalization (AdaLN). Yet despite their state-of-the-art performance and broad adoption, the role and embeddings remain poorly understood.
Simaarity IPR (%)
DIT
LightningDiT
SIT
MDT
RERA
Figure Hidden Semantic Bottleneck: Extreme Alignment and Dimensional Sparsity. Conditional vectors č'in sfate-of-the- art Transformer diffusion models on ImageNet- 1K exhibit very high pairwise cosine similarity (mostly 90 90-99%) while concentrating semantic information in only a few of 1,152 dimensions.
internal structure of these learned conditional
In this work, we present a systematic analysis conditional embeddings in diffusion trans- formers and uncover two key findings. (1) Class-condition vectors exhibit extreme alignment, with cosine similarity exccoding 99% on ImageNet-1K across multiple statc-of trat-off-tho-att tho-art meth-
"Corresponding Author

---

Published as a conference paper at ICLR 2026
Conditions Video, person, pose, class,
Encode
Transformers-based Diffusion
Generated
AdaLN
DIT, MDT, SIT, REPA, MG, LightningDiT, X-MDPT, MDSGen...
"bird"
X3
D1 82 D3 E
Figure 2: Transformer-bassel diffusion models inject conditions as globally compact vectors Ts via AdaLN for outputs such as images or mel-spectrograass.
ods (Fig. 1, white bar). (2) The learned conditional vector č is markedly sparse: only about 10 20 of its 1,152 dimensions carry substantial magnitude, yielding normalized participation ion rate (uPR) of just 1 1-2% (Fig. 1, gray bar). When we prune up to 66% of dimensions and perform inference with the resulting sparsified ?, generation quality remains essentially unchanged, exposing significant phee-parmeteriiiaionnn These findings challenge common assumptions about semantic conditioning and indicate that diffusion transformers encode conditioning signals far more compactly than previously believed, offering new design perspective for generative models. Our contributions are as follows:
Extreme similarity. We present the first systematic analysis showing that, in discrete class-conditional tasks (c.g., ImageNet), transformer- based diffusion models learn class- conditioned embeddings with up to 99% pairwise cosine similarity, and in continuous- condition lasks (e.g., pose-guided image or video-conditioned audio generation), the similarity exceeds 99.9%.
Sparse representations. We find that semantic information is concentrated in small set of embedding dimensions, while most remain near zero, revealing highly sparse conditional representations.
Redundancy and pruning. We demonstrate that aggressively pruning low low-maguitude dimensions preserves or even improves generation quality, highlighting substantial redundancy and enabling more efficient conditioning.
Mechanistic insight. We provide hypotheses, supported by analyses and theoretical reasoning, to explain the emergence of high similarity, sparsity, and pruning effectiveness.
2 RELATED WORK
Diffusion transformers and conditioning via AdaLN. Diffusion models have progressed from U-Net backbones (Rombach et al., 2022) to transformer- designs such as DiT (Peebles & Xie, 2023), SiT (Ma et al., 2024), LightningDiT (Yao et al., 2025), MG (Tang et al., 2025), X-MDPT (Pham et al., 2024), MDSGen (Pham et al., 2025b), and UCGM (Sun et al., 2025), achieving strong results across image, audio, and multimodal generation.
These models embed conditional signals-class labels, poses, or video features inttrres-into timestep embeddings and inject them via adaptive layer normalization (AdaLN) (Fig. 2), where condition vectors modulate all layers through learned scale shift parameters. Unlike the distributed conditioning of U-Nets, this global AdaLN mechanism molivates our study of how semantic information is encoded in transformer conditional vectors.
Prior work on conditional embedding analysis. Li et al. (2023) examined activation sparsity in Transformers for NLP and ImageNet with classification, but systematic studies of conditional embeddings in gencrative diffusion remain scarce. Early efforts targeted U-Net conditioning (Rombach et al., 2022; Saharia et al., 2022), while transformer- focused on architectural or training advances (Peebles & Xie, 2023; Ma et al., 2024; Yu et al., 2025; Tang et al., 2025; Gao el al., 2023; Pham et al., 2024; 2025b). We fill this gap with analysis of transformer conditional embeddings and their link to representation collapse.
models
Collapse in contrastive learning. Representation collapse mapping diverse inputs to nearly identical embeddings well known in contrastive learning (Grill et al., 2020: Zbontar ct al., 2021). We observe related effect in diffusion transformers: conditional embeddings across classes reach extreme angular similarity (>99% cosino) without harming generation quality, indicating distinct embedding usage compared to contrastive methods.
Hyperspherical embeddings and compressed codes. Our findings align with hyper- spherical embedding (Liu et al., 2017) and information bottleneck theory (Tishby et al., 2000),
2

---

Published as a conference paper at ICLR 2026
Cosine Similatty 11000 classes)
Gosine Smilariyy iZoomed 10 dasses]
1,0000
6.0060
0.3033
00002
0.5032
0.0052
0.0022
2.0055
2.0255
6.954
74
0.5238
6.5240
1,0000
0.775
0.003
0.0022
0.0022
50326
stc 144 19
0.963
0.9676
0000
0.945
0 0.0
200
Class
0.9652
0.645
0.0622
0605
0.000
1.0003
0.9012
3.0000
20001
000
0.09.5
0.9027
2992
0.5950
0.0000
0.9923
0.5043
0.00
0.9003
0.9911
1,0000
29024
0.062
800
0.166
6.4967
0.9555
0,992
0.9944
0.996
0.9927
09954
3,0000
6440
1.3000
9.95
500 300
760
244
308
252
722
Class
Class
Figure 3: Cosine similarity of conditional vectors č= + t across 1000 ImageNet classes using REPA-XL (Yu et al., 2025). Despite distinct semanties, embeddings show over 99% similarity for nearly all class pairs. Left: full x000 1000 matrix showing global aligument. Right: zoomed 10x 10 subset. for randomly chosen classes. Additional results for other SOTA methods appear in the Appendix.
which describe semantic compression into low-dimensional subspaces. Similar trade-ofis appear in VAEs and multimodal systems (Kingma & Welling, 2013; Tsai et al., 2019). Diffu- sion transformers further compress conditioning into small set of active head dimensions, leaving others largely redundant.
Conditioning injection: U-Net vs. Transformers. U-Net diffusion models inject conditions at multiple spatial scales via concatenation or cross-attention (Rombach et al., 2022; Dhariwal & Nichol, 2021), allowing localized fcature extraction. Transformers, in contrast, apply global AdaLN modulation, which likely drives the observed sparsity and high similarity in conditional embeddings as semantics collapse into a few dominant dimensions.
3 EMERGENT PROPERTY I: NEAR- CAA-UNNFFORM COSINE SIMILARITY
3.1 SETUP
We systematically analyze six state-of-the-art diffusion transformer models-Di - (Peebles & Xie, 2023), MDT (Gao et al., 2023), SiT (Ma et al., 2024), REPA (Yu et al., 2025), LightningDFTT (Yao et al., 2025), and Model-Guided (Tang et al., 2025) using their official pretrained checkpoints released on GitHub (XL models). The primary analysis is conducted on ImageNet-1K, where we compute pairwise cosine similarity matrices across all class- conditioned vectors غ' 1522 Each E is formed by summing the learned class embedding and timestep embedding, resulting in the final conditional vector injected into the denoising transformer backbone. To assess generality across domains, we extend the analysis to pose- guided image synthesis using X-MDPT (Pham et al., 2024) (c e 12224) and video-to-audio generation with MDSGen (Pham et al., 2025b) (c E I8 1276)), ), again utilizing publicly available pretrained weights. This consistent evaluation sctup ensures reproducibility and enables direct comparison across models and tasks.
3.2 COSINE SIMILARITY HEATMAPS
Fig. 3 (left) shows the full 1,000-class cosine-similarity matrix, where trained models reach up to ~ 99% similarity across class pairs. For clarity, Fig. 3 (right) provides zoomed-in view of 10 randomly sampled classes, revealing the same strong alignment. Additional results for other methods are provided in the Appendix.
3.3 CROSS- TASK EXAMINATION
The strong alignment of conditional embeddings extends beyond class-condiiional image generation to pose-guided image synthesis and vidco-to-audio generation (Fig. 4). X-MDPT (Pham et al., 2024) and MDSGen (Pham et al., 2025b) exhibit extreme cosine similarity-un up to 99.999.9 99.98% on DeepFashion and 99. 99.99% on VGGSound even with randomly varying test samples and conditions (e.g., persons, poses, videos). Because MDSGen shows patterns nearly identical to X-MDPT, its cosine heat map is omitted. This striking consistency indicates that diverse inputs yield almost identical embeddings before denoising; we examine possible explanations in the following sections.
3

---

Published as a conference paper at ICLR 2026
.0000
XHD77
Percentage of [E]yam
Sample Index
Percentage of
0.99990
30-
30-
23
20
0.90405
10-
10-
0.0
3.0
Sample Index (c) cosine heatmap
(a) original scale
(b) norno lized scale
Figure 4: Sparsity and alignment of conditional embeddings in X-MDPT (Pham et al., 2024). (a) and (b): With == 0.1, over 51% of components in the conditional vectors have magnitudes below the threshold, highlighting significant sparsity. Remarkably, pruning these dimensions has minimal effect on generation quality. (c) Cosine similarity between random test samples in DecpFashion exceeds 99.9%, confirming extreme alignment across conditional embeddings.
EMERGENT PROPERTY II: SPARSE MAGNITUDE DISTRIBUTION
4.1 MAGNITUDE HISTOGRAMS
Fig. 5 shows the histogram of absolute component values of only about 10-20 of the 1,152 dimensions exceed 0.1 in magnitude, and roughly 10 exceed 1. Fig. visualizes the learned conditional vector for each method, underscoring its pronounced spansity. For completeness, we include continuous tasks such as X-MDPT and MDSGen in the Appendix: their embeddings appear less sparse, consistent with the higher participation ratio (PR) reported Tab. 1.
LightringhT
MDT
2211110
imagetuale
magritute
map fade
HG
SIT
Figure 5: Magnitude histogram distribution of learned conditional vector embedding R gx1152_ Most dimensions have near-zero values (< 0.01), with only ~ - 20 dimensions showing dominant magnitudes. This sparsity holds across multiple models, including DiT, MDT, LightningDiT, MG, SiT, and REPA. It is best viewed with 300% zoomed in.
Vieu
000
300
400
68
1000
1200
280
1000
1209
200
10.0
120
WS
RERA
SIT
Divendan
Figure 6: Original distribution of learned conditional vector embedding غe 12x11 Most dimensions have near-zero values (< 0.01), with only ~ 20 dimensions showing dominant magnitudes. This sparsity holds across multiple models, including DiT, MDT, LightningDiT, MG, SiT, and REPA. It is best viewed with 200% zoomed in.
4

---

Published as a conference paper at ICLR 2026
Table 1: Participation Ratio (PR) in learned conditional embeddings of stato-of-the-art
models on Imagenet-1K 1K generation (discrete) and DecpFashion/VVGGS.. (continuous).
Embedding Metries
DIT
SIT
MDT
LightaingDIT
MG
REPA
X-MDPT
MDSGen
Condition Dim (d)
1152
1152
1152
1024
PR (a)
120.69
26.25
18.45
23.70
19.98
17.67
495.75
104.22
nPR (Oaeem)
10.17%
2.28%
1.60%
2.05%
1.73%
1.53%
48.42%
13.57%
Cosine Sim. (cx)
0.9001
0.9852
0.3805
0.9779
0.9934
0.9946
0.9998
4.2 DIMENSION CONTRIBUTION: PARTICIPATION RATIO
To quantify how many dimensions effectively contribute, we compute the participation ratio (PR) on absolute magnitudes U, = lc|:
a = PR(v) = (E i= v
a Onorm d
with d is dimension.
(1)
Ei=1 uz
PR estimates the number of coordinates carrying most of the total magnitude. Tab. shows that state-of-the-art models (MDT, LightningDiT, MG, REPA) rely on less than 2% of dimensions, whereas continuous tasks such as X-MDPT and MDSGen (13 48%) and exhibit even higher cosine similarity (up to 99.99% vs. 90 99.4%)
larger fraction
This suggests that continuous-condition embeddings both engage more dimensions and distribute information more uniformly, naturally leading to stronger alignment across samples compared discrete class-conditional ImageNet generation.
FROM OBSERVATION TO ACTION: PRUNING REDUNDANT DIMENSIONS
Role of tail dimensions. To quantify sparsity and effective di- mensionality, we define the spar- sity ratio threshold 7
Orginal
Generaned removed benf-dimension (D
RCPA
c- (0.200)
C-n OU SOS
1- (3.0%)
y- (2.2%)
(848%)
Stail(r) = = |a| <T).
(2)
With r= 0.01, we observe a spar- sity ratio of = 0.38 0.40. We define the high-magnitude "head" as (T) # |c]| > 7} and the low-magnitude "tail" as the remaining coordinates. Using REPA representative model, we progressively prune é at thresh- olds E (0.01, 0.02.... and find removing up to 66% of dimensions (Tab. 2), showing substantial redundancy.
2V1 SITEST
Figure 7: Class-conditionedd image generation with head removal. ImageNct samples after pruming top- magnitude dimensions of ē (threshold r); removing only few head dimensions markedly degrades quality.
that FID and CLIP scores remain stable-even after
Pruning at ~88 of dimensions) preserves or improves image quality (Fig. 8, Tab. 2). Pruning during late de- noising steps yields larger FID gains and modest CLIP improve- ments, aligning with the gradual rise in cosine similarity loward the final steps. For consistency, we re- port statistics using the conditional vector at the initial step to and dis- cuss possible explanations in the following sections. Next, we examine in dctail how head dimensions influence generation quality and clarify how their role differs from that of the tail dimensions.
0.01 (removing
Table 2: Performance and semantic metrics under spar- sification. 4; prune every step, to: prune only at start, In-knn prune during last k steps.
Prine
Threshold 7
# Removed Dims
FID
IS 1
CLIPT
Buseline (REPA)
0/1152 (0%)
7.1004
176.02
29.746
0.01 (t))
448/1152 (38.94%)
7.2143
171.99
29.737
+= 0.01 (ta)
418/1152 (38. 91%)
7.1690
175.97
29.807
0.01 (t., k..)
448/1152 (38.94%)
7.1598
175.49
29,805
7 0.02 (t.)
762/1152 (66.21%)
9.2201
125.15
29.221
T- 0.05 (6.)
1110/1 (96.41%)
56.2308
20.47
22.177
(t.)
1149/1152 (99.800 (90.80%)
356.135
1.77
5.0 (t))
2/1152 (0.20%
7.8478
164.15
29.355
1.0 (5)
8/1152 (0.60%)
523.7637
1.95
22.090
Role of head dimensions. As shown in Fig. 7, removing only a few migh-nag dimensions (e.g., 4-6/1152) dramatically degrades generation quality. In contrast, pruning up
5

---

Published as a conference paper at ICLR 2026
Original
Generated outpets with removed tall-dimensions 1152)
0.01
0.02
0.03
0.04
0.05
0.1
0.5
1.0
x-
19%6
0/115
448/1152
962/1152
961/11552
1064/1152
3138/1152
1142/1152
1146/1152
114901152
Figure 8: Class-conditioned image generation (tail removal). ImageNet samples with progressive removal low-magnitude dimensions in (threshold T on absolute value). Image quality remains high or better baseline REPA even when 38->80% of dimensions pruned (generated images in the second column), as long as key head dimensions are retained.
to 66% of low-magnitude tail dimensions (762/1152) leaves quality largely intact. Variance analysis (Fig. 9) further reveals that only ~55-20 head dimensions carry most of the variance, particularly across classes, highlighting their critical role compared to the tails.
0.30
Head
0.00200-
2.0019
Zoom: Top 20
LightningD
0.00175
0.25
MDT MG RERA
Mcsn Vonaance
0.00125
0.00150
0.0034
VerianCe
0.20
0.2
0.001.0-
0.0030
0.15
0.10
0.00050
0.0
0.00025
0.85
Tail
DrT
Lightrighht
140
5iT
0.00
200
400
600
800
1000
1200
Method
Sorted Dimension
(a) Average variance across dimensiooss.
(b) Variance per-dimensioo (sorted).
Figure 9: Variance concentration in conditional vectors (a) Mean variance across models stays non-zero, showing no collapse. (b) Variance is concentrated in only 15 20 head dimensions (<2%), while the remaining 98% of tail dimensions show minimal variation, indicating that semantic information is confined to small subspace.
Continuous task. We apply the same pruning procedure to poso-guided person image generation with X-MDPT (Pham et al., 2024) and observe consistent behavior (Fig. 10). Unlike class-conditional ImageNet, this task requires slightly higher threshold to induce similar sparsity: 0.1 yields 0.38 0.40 (Fig. 4 a,b) while preserving generation quality (Fig. 10). More qualitative results are available in the Appendix.
UNDERLYING MECHANISMS BEHIND SIMILARITY, SPARSITY, AND PRUNING: HYPOTHESES
6.1 HOw CAN MODEL GENERATE CORRECT OUTPUTS DESPITE HIGH SIMILARITY?
Although conditional vectors exhibit high pairwise cosine similarity, our variance analysis (Fig. 9a) shows no embedding collapse: the mean variance across models remains small but clearly non-zero (0.0002-00 -.00-0. This contrasts with the feature collapse often seen in castive learning, where embeddings converge single point and variance approaches

---

Published as a conference paper at ICLR 2026
Conditions
Original
Generated outputs with removed tail- diim-diissss (D 1024)
Style
Pose
X-MDPT
Spars (1- 0.1)
1- 02
0.3
0-0.4
T-0.5
--100
5.0
Figure 10: Pose- cosditindd person image generation. DeepFashion samples under different spursification of is the magnitude threshold for pruning. Person-consistent images remain high quality even when 50 75% of dimensions are zeroed, as long as key head dimensions are preserved. Best viewed 200% zoom; more samples appear the Appendix.
zero (Wang & Isola, 2020; Zhang et al., 2022). Further, the cimensionnthreetee analysis in Fig. reveals that with т 0.01, a large fraction (50 90%) of dimensions retain low magnitude, saturating near = 0.02, indicating that informative components are broadly distributed rather than concentrated a few directions.
We hypothesize that diffusion transformers preserve this subtle but global structure because each denoising step predicts fine-grained Gaussian noise, providing a rich and stable training signal. Consoquently, even when conditional embeddings lic within narrow cone in feature space, their nuanced directional differences lifferences-amplliie through adaptive layer normalization, the expressive Transformer backbone, and iterative refinement remain sufficient to guide accurate class-conditionall pose-guided, and video-conditionee generation. deeper theoreli- cal explanation of this robustness remains an open problem, calling for rigorous analysis in future work.
Percentage of sT
Percentage of [C ([mom
LightningDiT MDT
LightningDiT
MDT
MG
MG
REPA
REPA
SIT
0.83
0.22
5.04
0259
0283
8.80
0.20
0123
6542
T
(a) Dimensions below threshold 7 (original scale). (b) Dimensions below threshold 7 (normalized),
Figure 11: Sparsity of conditional embeddings. With r 0.01, over 80% of components in the mean conditional vector fall below the threshold, revealing a highly sparse representa- tion that starts to saturate near 0.02.
6.2 HYPOTHESIZING THE STRUCTURE OF CONDITIONAL EMBEDDINGS
High Cosine Similarity. We hypothesize that the extreme cosine similarity among class embeddings arises from the dynamics of diffusion Transformer training (Fig. 12 top). Since the model conditions embeddings across all timesteps f, it favors embeddings that provide a stable, robust signal for denoising, resulting in globally aligned embeddings:
cosine(Gy, Cyr) RJ 0.99 Vy + y'
(3)
Despite this high similarity, semantic differences are encoded in a small subset of high- magnitude head dimensions,
Cy hy.d + Cy,taily
|| Cyy.haad >
|gg,tiil|
(4)

---

Published as a conference paper at ICLR 2026
are sufficient to modulate Adaptive LayerNorm parameters ((c)),((cy))
(c,) = WyC B(cy) = Wyyy-
Those subtle differences are progressively amplified by the itcrative diffusion process, enabling correct and high-quality class-conditional generation despite the high overall cosine similarity.
Observed sparsity in learned embeddings. Conditional em- beddings are highly sparse: for d 1152, only about 1 2% of di- mensions reach large magnitudes (~ 5-8), while most remain ncar zero (10 30-1) This head lail pattern indicates that semantie in- formation resides in a small sub- space, aligning with our pruning results. We quantify sparsity using the normalized participation ratio Onorm, which confirms that the ef- fective dimensionality is far below d. As shown in Fig. 12 (bottom), monitoring this metric (nPR) while training the REPA B-2 model on ImageNet- 1K for 200k steps shows drop from about 90% early in training to under 6%, with the de- cline continuing revealing natural sparsification dynamic in diffusion transformers. Ex- tended analyses appears in the Appendix.
Cosine similartty of
0.8
0.6
Training Step
0.8
nPR (anom)
0.6
0.4
0.2
25000
50000
75000 100000 125000 150000 175000 200000 Training Step
Figure 12: Training dynamics of conditional em- beddings. Top: batchwise cosine similarity of č during training. Bottom:
participation ratio (nPR) over train- ing steps, showing progressive sparsification.
Pruning Improves Generation. We conjecture that pruning low-magnitude embedding dimensions acts as a form of noise suppression in diffusion Transformers. Let the conditional embedding be e Rd and decompose it as
Chead Chead Ctail,
|[Creaa > ||cta]]|
(6)
In practice e is mapped to Adaptive LayerNorm parameters 7(c), B(c) that modulate hidden states h:
h - n(h) n-((h) AdaLN(h c) 7(c) e
B(c)-
(hh)
Because semantic information concentrates in Chead: the tail Call contributes only weak, low-variance signals (as shown before in Fig. 9b). Supporting this view, Fig. 13 visualizes class embeddings: when only head dimensions are retained, class clusters remain well separated, whereas tail-only embeddings collapse into an entangled cloud. Retaining these noisy tail dimensions perturb 7(c), B(c) and inject interference into the denoising trajectory, particularly in later inference steps where precision is critical.
We empirically observe that pruning (zeroing out) Ctail at the initial step to at the final steps preserves generation quality, with late-step pruning yielding the strongest FID improvements. This supports the view that pruning suppresses interference and sharpens the semantic subspace. A more detailed analysis is provided in the Appendix.
7 DISCUSSION
Our results reveal a semantic bottleneck in transformer- based diffusion models: conditional embeddings place most semantic content in small set of high-magnitude dimensions, leaving the majority near zero and largely redundant. For class-conditional ImageNet generation, this effect is strongest, with only few dominant dimensions and very low normalized participation ratio (nPR). Contimuous-condinn
tasks (e.g., pose-guided image or video- to-audio generation) show milder form of this sparsity, exhibiting more high-magnitude

---

Published as a conference paper at ICLR 2026
SME
USKE 1
[(a]
Figure 13: t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from SiT-XL on ImageNet-1K; similar trends appear in other models.
dimensions and higher nPR values. The consistency of this pattern across architectures and tasks points a general property of how diffusion transformers encode conditioning signals.
Relation to Contrastive Learning Collapse The observed alignment of conditional embeddings bears resemblance to representation collapse contrastive learning methods like SimCLR Chen et al. (2020), SimSiam (Chen & He, 2021), BYOL (Grill et al., 2020), VICReg (Bardes et al., 2022), and Barlow Twins (Zbontar et al., 2021). In those settings, collapse leads to trivial embeddings and degraded downstream performance unless variance- promoting regularizers or repulse components (negative samples) are used (Zhang et al., 2022). Interestingly, diffusion transformers avoid such pitfalls: despite extreme angular similarity, they maintain strong generation quality. We hypothesize that AdaLN amplifies nigh-magnitude dimensions sufficiently to preserve semantic distinctiveness during denoising, and that diffusion modcls' itcrative refinement mitigates the impact of collapsed embeddings.
Why high cosine similarity occurs only in transformers. Based additional experiments with U-Net models, we clarify that high cosine similarity arises primarily in transformers, and not in U-Nets when timestep embeddings are removed. However, similar redundancy emerges in U-Net diffusion models once timestep embeddings are included. This distinction appears tied to conditioning mechanisms: AdaLN in transformers promotes compression into dominant dimensions, whereas U-Nets use concatenation or cross-attention, preserving richer represcntations.
Relation to information bottleneck and AdaLN. The sparsity mirrors information bottleneck behavior (Tishby et al., 2000). where networks distill essential features. AdaLN's linear scaling amplifies few dominant dimensions, rendering others redundant. Implica- tions and risks. Compact embeddings may conflate unrelated semantics (cosine similarity * semantic similarity). This could limit controllability in multi-conditional tasks or fine-graines editing.
Broader impact. Our observations of extreme similarity, sparse embeddings, and effective pruning in transformer based diffusion models suggest that similar redundancy patterns could also appear in other generative frameworks, such U-Net diffusion models (when timestep embeddings are included), GANs, or autoregressive models. This points to a potential principle of compact and efficient conditioning, which may inspire future work lighter models and interpretable embeddings across tasks and modalities.
CONCLUSIONS
We have uncovered an interesting phenomenon in transsormee-bbsed diffusion models: rxme angular similarity and semantic sparsity in conditional embeddings. Our extensive analyses reveal that only a small subset of high-magnitude dimensions carry semantic information, while the majority of dimensions are redundant. Despite this, diffusion transformers maintain robust generation quality even when up to 66% of the conditional vector is pruned or masked, These findings suggest a fundamental overparameterization of conditional encoding and motivate rethinking conditioning mechanisms for efficiency and interpretability. Future architectures could benefit from compressed or hybrid conditioning strategies that maintain semantic fidelity while reducing computational overhead. Exploring these directions may lead to more controllable, efficient, and versatile generative models across vision, andio, and multimodal domains.
9

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported by the Institute for Information & Communications Technology Planning & Evaluation (LTTP) grant funded by the Korea government (MSIT) (No. RS- 2021-11211381, Development of Causal Al through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and (No. RS-2022-II220184, Develop- ment and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics) and partially supported by the KAIST Jang Young Sil Fellow Program.
REFERENCES
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varranceiivarranne-ovvrinne regularization for self-supervised learning. In International Conference on Learning Representations, 2022. URL https://openrevies net/fforumm:dxxxx6662220
9
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020. 9
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In IEEE/CVF Conference on Computer Vision and Paltern Recognition. (CVPR), 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.
Advances in neural information processing systems, 34:8780 8794, 2021. 3
Shanghua Gao, Pan Zhou, Ming- Cheng, and Shuicheng Yan. Masked diffusion trans- former a strong image synthesizer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 23164-23173, 2023. 1, 2, 3, 15
Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Newral Information Processing Systems, 33:21271 -12244, 2020. 2, 9
Ji Woo Hong, Tri Ton, Trung X Pham, Gwanhyeong Koo, Sunjae Yoon, and Chang D Yoo. Ita-mdt: Image- teagee-iiistee-dppptiieee virtual try-on. In Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 28284 28294, 2025. 30
masked diffusion transformer framework for image-based
Young Hoon Jung, Seong Kwang Hong, Hee Scung Wang, Jae Hyun Han, Trung Xuan Pham, Hyunsin Park, Junyeong Kim, Sunghun Kang, Chang D Yoo, and Keon Jae Lee. Flexible piezoelectric acoustic sensors and machine learning for speech processing. Advanced Materials, 32(35):100002,, 2020. 30
Young Hoon Jung, Trung Xuan Pham, Dias Issa, Hee Scung Wang, Jac Hee Lee, Mingi Chung, Be-Yeon Lee, Gwangsu Kim, Chang D Yoo, and Keon Jae Lee. Deep learning-based noise robust. flexible piezoelectric acoustic sensors for speech processing. Nano Energy, 101:107610, 2022. 30
Junyeong Kim, Minuk Ma, Trung Pham, Kyungsu Kim, and Chang D Yoo. Modality shifting attention network for multi-modal video question answering. In Proceedings of the IEEE/CV conference on compuler vision and pallern recogniitin,, pp. 10106-10111 2020. 30
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXie preprint ar Xrv: 1312:1311..6 6114, 2013.7 3
Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, and Chang D Yoo. Flexiedit: Frequency-awant latent refinement for enhanced non-rigid editing, In European Conference on Computer Vision, pp. Springer, 2024. 30
10

---

Published as a conference paper at ICLR 2026
Gwanhyeong Koo, Sunjae Yoon, Younghwan Lee, Ji Woo Hong, and Chang D Yoo. Flowdrag: 3d-aware drag-bast based image editing with mesh-guided deformation vector flow fields. In International Conference Machine Learning, pp. 31467-31448 PMLR, 2025. 30
Donghoon Lee, Hyunsin Park, Trung Pham, and Chang D Yoo. Learning augmentation network via influence functions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10961-10970, 2020. 30
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ / /opprr/v/ep/.r...... net/foorunPdd-TJ2ncccll.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 212 220, 2017. 2
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Bolli, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23 40. Springer, 2024. 2, 3
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVE International Conference Compuler Vision, pp. 4195-4205, 2023. 1, 2, 3, 15, 16
Trung Pham, Chaoning Zhang, Axi Niu, Kang Zhang, and Chang D Yoo. On the pros and cons of momentum encoder in self-supervised visual representation learning. arXiv preprint arXiv:2208.0577 2022a. 30
Trung X Pham, Rusty John Lloyd Mina, Dias Issa, and Chang D Yoo. Self-supervised learning with local attention-aware feature. arXiv preprint arXio:2108. 00475, 2021. 30
Trung X. Pham, Kang Zhang, and Chang D. Yoo. Cross-view masked diff'usion transformers for person image synthesis. In Forty-firs firs-f International Conference on Machine Learning, 2024. URL https: http:////peeeeeeeeeee met/forummid=joon IkNNkqy.. 1, 2, 3, 4, 6, 28, 30
Trung X Pham, Zhang Kang, Ji Woo Hong, Xuran Zheng, and Chang D Yoo. E-md3e: Taming masked diffusion transformers for efficient zero-shol object customization. arXiv preprint or 2rvv 09164, 2025a. 30
Trung X. Pham, Tri Ton, and Chang D. Yoo. MDSGen: Fast and efficient masked diffusion temporal-aware transformers for open-domain sound generation. In The Thirleenth International Conference on Learning Nepresentationss 2025b. URL net:/Eoruu/idyyFEa/gtt.. 1, 2, 30
Trung Xuan Pham, Rusty John Lloyd Mina, Thanh Nguyen, Sultan Rizky Madjid, Jinwoong Choi, and Chang Dong Yoo. Lad: A hybrid deep learning system for benign paroxysmal positional vertigo disorders diagnostie. IEEE Access, 2022b. 30
Trung Xuan Pham, Axi Niu, Kang Zhang, Tee Joshua Tian Jin, Ji Woo Hong, and Chang D Yoo. Self-supervised visual representation learning via residual momentum. IEEE Access, 2023. 30
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Hish-resolution image synthesis with latent diffusion models. In Proceedings of the IEFE/CV conference on computer vision and pattern recognition, pp. 10684-10695, 2022. 2.3
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to to-gg diffusion models with deep language understanding. Advances in newral information processing systems, 35:36479 364944 2022. 2
11

---

Published as a conference paper at ICLR 2026
Peng Sun, Yi Jiang, and Tao Lin. Unified continuous generative models. arXiv preprint arXiv:2505.074 07447, 2025.
Zhicong Tang, Jianmin Bao, Dong Chen, and Baining Guo. Diffusion models without classifier-free guidance. arXiv preprint arx:::250 1215,, 2025. 1, 2, 3, 16, 19, 28
Naftali Tishby, Fernando C Percira, and William Bialck. The information bottlenock method. arx preprint
2000. 2,
Pham Xuan Trung and Chang D Yoo. Short convolutional neural network and mffees for accu- rate speaker recognition systems. International Technical Conference on Srrcits/Sysstem Computers and Communications (ITC-CSCC), 2019. 30
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaignnd multimodal language sequences. In Proceedings of the conference. Association for computational linguisties. Meeting, volume 2019, pp. 6558, 2019. 3
Thang Vu, Hyunjun Jang, Trung X Pham, and Chang Yoo. Cascade rpn: Delving into high-quality region proposal network with adaptive convolution. Advances neural information processing systems, 32, 2019. 30
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through aligmment and uniformity on the hypersphere. In ICML, 2020. 7
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognation Conference, pp. 15703-15712, 2025. 1, 2, 3, 13, 15, 19, 28
Eunseop Yoon, Hee Suk Yoon, Mark A Hasegawa-JJhnsson and Chang D Yoo. Can video I1ms refuse to answer? aligument for answerability in video large language models. The Thirteenth International Conference on Learning Representations, 2025. 30
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusson transformers is easier than you think. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttps://openreviez: net f (its/foommmiddDDszGGZY
1, 2, 3, 16, 28
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. International Conference on Machine Learning, 2021. 2,
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? unified understanding with self-supervised contrastive learning. In International Conference on Learning Representations, 2022. URL https:// //oenreviei net/f http://rrmmmmi bigbbcq04 7, 9
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
In this appendix, we provide the details of experimental setups as well as more comprehensive analysis results for various approaches.
A.1 SETUP DETAILS
We generate 5,000 samples from the public checkpoints each method (5 samples per ImageNet class) and evaluate FID and IS using the LightningDiT (Yao et al., 2025) evaluation code. Inference follows the default hyperparameters and sampling steps specified by each model, using XL-size variants when available. For continuous tasks (X-MDPT and MDSGen with AdaLN), we adopt their largest released models (L-size and B-size, respectively). During inference, modify only the conditional vector keeping all other components unchanged.
A.2 EXPLANATION HYPOTHESIS (EXTENDED)
Pruming Improves Generation. We retain the decomposition used throughout the рaper:
c= Chead + Ctails l/Ceadd > ||eaal|-
Here Chead denotes the high-variance, semantically informative dimensions, while Ctail corre- sponds to low-magnitude, low-variance dimensions.
Conditioning implemented through Adaptive Layer Normalization (AdaLN). For a hidden activation h E R4:
AdaLN(h (c) ((c) h (()
o(h) + +c((),
with linear projections
7(c) Wyc. [(c) Wsc.
Lincarity implics that
7(c) = ((hhoad) (Ctail), ((c) = B (Cheadd) + 3(craa).
Empirically, Var[y(Cuau) and Var[8(ccaa] are negligible compared to their head counterparts. However, we hesize that these weak terms can propagate as noise through the denoising trajectory, with a potentially larger effect later inference steps (t 0), where precision critical.
Define a pruning operator P(-) that zeros out tail dimensions:
P(c) = Chead
Pruning can be applied either at the initial step during the final few steps of inference. While early-step pruning reduces redundancy early, we empirically observe that late-step pruning consistently yields stronger improvements in FID, supporting the hypothesis that late-step pruning suppresses residual noise and sharpens semantic guidance:
AdaLN(h e') (Cheadd) h- u(h) ((Chead)-
h-w(h) ((h)
removing weak tail dimensions reduces interference while focusing conditioning dominant semantic directions, explaining why pruning preserves or can even improve generative quality.
Thus, pruning acts as an effective noise filter:
High Cosine Similarity. We empirically observe that the cosine similarity between class embeddings remains extremely high (> 0.99) across nearly all timesteps of denoising, We hypothesize that this is consequence of dynamic training in diffusion Transformers: conditioning is applied across all timesteps f, and the network learns to maintain a stable, robust signal. This encourages embeddings to align along similar directions, while semantic distinctions are preserved in a small subspace of head dimensions:
Cy head + Cy,tail, Gyy.eadd >> |Cg,tail]]-
13

---

Published as a conference paper at ICLR 2026
Even with globally aligned embeddings, the high-magnitude head dimensions provide sufficient directional cues to modulate Adaptive LayerNorm parameters:
7(2y) = Wy B(cy) = Wycy.
These small differences are progressively amplified by the iterative denoising process.
Thus, while embeddings appear nearly parallel in the full space, the effective semantic subspace defined by the head dimensions ensures that generation remains accurate and high-quality. This also explains why pruning tail dimensions, which contain low-magnitude, redundant signals, does not harm gencration and can somctimes improve quality.
A.3 ANALYSIS OF EMBEDDING SPARSITY
Empirical observation. In the pretrained diffusion Transformer embeddings analyze 1152) only a small subset of dimensions about 1% to 2% e%xibiii large absolute values (typical magnitude 5-8), while the rest remain near-zero (typical magnitude ~ 10-3-10 We refer to the large-magnitude coordinates as the head and the rest as the tail.
Metrics. To quantify sparsity and effective dimensionality,
the following statistics:
Sparsity ratio at threshold т:
1 s(7) 7 7 [cl: r}.
With r set to a small constant (e.g., 0.5) this yiclds s = 0.01-0.02 empirically.
Participation ratio (PR) on absolute magnitudes 8, |c|l (a measure of effective dimensions):
Ei= v Ei=1 u
a= PR())
d PR(U).
Onormalized
PR gives estimate of how many coordinates carry most of the total magnitude; we find PR < (order tens).
This normalization maps the range of values to Onormalized (0, 1], nPR (Anormalized) 1 when all d coordinates contribute equally. nPR ss k/d when effectively only k coordinates carry the magnitude.
Interpretation hypotheses. We offer several plausible, non-exclusive explanations for this sparse phenomenon:
Projection and scale effects. The learned linear projections Wy, Wa (and any subsequent layers) can amplify few coordinates of c if their corresponding projection weights are large, producing a few dominant coordinates in the final modulation parameters.
2. Stable conditioning across timesteps. Because conditioning is applied across many timesteps, the optimizer favors a stable, low-dimensional conditioning signal to avoid disturbing denoising dynamics; encoding semantics in a few robust axes avoids noisy, volatile conditioning.
3. Implicit sparsity from
wwtrraatiinn/egglaiiaatoooo Weight decay, initializa- tion, and training dynamics may implicitly encourage small-magnitude coordinates; only the coordinates providing robust semantic signal are driven large magnitudes.
Consequences for AdaLN modulation. Since AdaLN uses 7(c) W.c and B(c) = Wgc, large entries in dominate the modulation:
7(c) Chead.
B(c) Ro WyChead-
Thus, the denoising network effectively receives conditioning from low-dimensional subspace, explaining why zeroing most coordinates (sparsification) has a small empirical impact and why pruning even improve performance by removing weak, noisy contributions.
14

---

Published as a conference paper at ICLR 2026
A.4 MORE VISCALIZATIONS OF OTHER METHODS
A.4.1 COSINE SIMILARITY WITH PAIRWISE ANALYSIS.
For completeness, we present full pairwise cosine similarity matrices for all six state-of-the- art diffusion transformer models evaluated on ImageNet. Each matrix reports the cosine similarity between conditional embeddings for every pair of ImageNet-IK classes, offering comprehensive view of how uniformly aligned these vectors are across the label space. The results reinforce the main paper's findings: near-uniform similarity is pervasive across models and classes, with the sole exception of DiT, whose lowest pairwise similarity is about 88%. Notably, DiT also delivers weaker generative performance (higher FID) compared to the other models, further distinguishing it from the rest of the evaluated methods.
Cosine Similarity 11000 classes)
Cosine Similarity IZoomed classes)
1.02
2.02
1.0003
0:0259
0.0042
20004
0,5005
2,00CC
2.2042
6932
0.0007
0.872
1.92
0.2042
1,0000
50006
0.5024
2.5546
5.5022
8.803
0.3003
-:44
A13
0.4011
0.0000
6,000
3
5.6157
0.41:4
0.4213
2.80
430
Class
6:92
Class
115 106
0.000
0.0604
0,50555
11050
5,5039
1 0.8933
0,5000
0.
620
65
0.504
0.3246
0.3020
2.0:47
0.5253
3.0020
2,0002
5154
0.2272
0.003
0.02
0.02
758 614
0.699
0:916
0.6445
6.9190
3.8455
0.9211
830
-C.82
02087
0.2287
0.0003
2.2133
0,5040
2.020
5,5272
05222
1,0000
0.8333
120
5:54
08:2
0.8181
0.2641
2.8308
0,800
3.8/25
5.8913
02877
0.2252
100002
0.88
200
400
500
802
955
420
37
4:3
153
753
617
516
7.5
320
Class
Class
Figure 14: Cosine similarity of conditional vectors č= y across 1000 ImageNet classes using DiT-XI, (Peebles & Xie, 2023).
Costne Simiartty 11,00 cesssss
Costne Similartt (Zoomed 10 cladpos)
1.0000
0:2704
0 0:8792
0.677C
3,STEC
0.605C
3.036
6.4020
0.50773
467
1,0000
035/28
2,3505
0,3007
0.500
2.5435
0.50
-0.0
100
076C
0-3003
C000
0.777
3.3645
0.9757
2.0344
08
97
105
627
0002
1.775
0.722
100
3.99
0.500
300
Class
Class
14 Rg
4664
0. 4607
0.577
0 4777
1.6003
3.445
6.4772
0.985
600
277
0.0232
0.2874
0.3822
2,3883
0,5644
1.0000
2.0099
0.0857
0.5042
032
0.9865
0:9652
0 6:601
3.773
0.6053
0.0059
2,000
6,6000
000
0.3222
0.385
0.3822
0 0.543
7,3842
0.577
09057
23922
3,0000
0.55.
847
26
273
156
185
Class
Class
Figure 15: Cosine similarity of conditional vectors غ = y + t across 1000 ImageNet classes using SightninDDiT-X (Yao et al., 2025).
Casine Simiirity 11000 dassest
Cosini Similarity (Zoomod 10 classos)
2.000
0
000000
0,9763
0.3673
(L3004
09729
2.9781
0,5400
0.999
-0.9
0.
94744
0 64452
1003
0 GGRE
3 1404
3.9447
5.400
540
0212
0,2000
0.303
100022
2,383
0.2874
0.0022
2.007
5,055
0.0055
Class
Class
55 F14
9700
0.9425
6907
6097
7,3600
0.6611
3,0050
0.085
0.9244
0.7522
0.392
0 0454
7.3541
1.0007
0.595
2.5949
0,5951
0.59
800
700
0.560
14
0.978
0.9962
0.992
0 5997
3,950
0.9933
8933
1,000
8,9907
0.99/
800
0.2200
0.2312
03802
05732
3.3526
0.599:
0.993
2.007
2,0000
0.5025
0.A4
0.94
6.9556
6955
1.440
0.6431
6637
3.4411
0,500
316
144
660
662
Class
Class
Figure 16: Cosine similarity of conditional vectors č= y + across 1000 ImageNet classes using MDT-XL (Gao al., 2023).
15

---

Published as a conference paper at ICLR 2026
Cosine Sindarit 11000 classes))
Cosine Similartt IZeemed 10 dassess
1.0000
6,0929
0.2097
0.0092
0.3042
2.044
0.985
3,0000
0.5613
0.0055
0.0036
5.030
0.9942
0.9922
0.95:25
00000
05955
0.9929
0.9913
0.9927
0.9906
11611
800
0.325
990
0.4642
0.497
0.9604
0.6426
0.6410
1.0003
3.9415
5,9915
0.4973
600
0.2046
0.0031
0.2244
0.004L
02000
0.0052
22022
20028
0.2241
9394
0.375
0.66-6
6.6639
6.9635
10000
000
0.990
0.59
0.352
0.994
009LE
0.9933
0.7922
25954
.0000
0.300
00000
7M
Class
Class
Figure 17: Cosine similarity of conditional vectors č= across 1000 ImageNet classes using MG-XL (Tang et al., 2025).
classes)
Similarity
dasses)
3,000
11 1.000
0,006:
0.3943
22995
0.3559
-6.455
0,560
3,6000
6-9552
0.0023
0.9023
2.9027
5.9002
200
0L993
0,9452
1.0009
999939
0.9920
0.9664
0.9924
0.9944
2.915
0.350
400
Class
0.94
0.4453
0.4403
1.0000
0.945
3.6461
2.9000
6.4447
0.315
52
0.300
0.1290
0.5049
0.352
0.2592
03092
0.2247
22054
2.049
320
0.560
0.6667
0.36-4
0.0023
0.9355
0.915
9399
0.3550
8,8069
0.3663
0.50:
0.5947
0.9025
0.204E
23015
20003
3.0000
000
100
150
253
3G2
159
376
22c
258
230
Class
Class
Figure 18: Cosine similarity of conditional voctors č= y + across 1000 ImageNet classes using REPA-XL (Yu et al., 2025).
Costne Similarity (1a00classas) classes)
Casine Similaiity IZoomed classes)
1000
2.99922
0.5/22
0.9569
0.0000
6.9857
5.9477
EU9772
3.9432
6409
230
2.00
8.665
4477
6477
1,0000
6613
3.977
2. 9888
0 6774
64617
4017
400
2.92
Class
5.99
0.3622
0.2772
0.3951
2.5
0.300
5.02
237
03742
0.0003
0.0879
2.0055
0.5700
3.002
2.0002
0 0.080
0.2872
0.0002
5:34
-245
0.362
0.2259
0.670
0.5900
5.5872
03713
1:000
0.0047
84651
6.9775
6874
0.507
403
500
329
820
252
400
744
459
571
425
Class
Class
Figure 19: Cosine similarity of conditional vectors == y across 1000 ImageNet classes using SIT-XL (Peebles & Xie, 2023).
16

---

Published as a conference paper at ICLR 2026
A.4.2 T-SNE DISTRIBUTION ANALYSIS.
To further examine the role of head and tail dimensions in conditional embeddings, we provide t-SNE visualizations for all evaluated methods under targeted perturbations. Specifically, we manipulate either the sigh-magnttude (head) or low-magnitude (tail) dimensions of the embeddings and observe how these changes affect the overall distribution of class representations for all 1,000 ImageNet. classes.
These visualizations illustrate that removing or altering head dimensions strongly disrupts the scparability of class clusters, while perturbing tail dimensions has minimal impact, highlighting the concentration of semantic information in a small subset dimensions.
Results for each method are shown in Fig. 20 - Fig. 25, providing comparative view of how different architectures encode and distribute semantic information their conditional embeddings.
15KE
6SNE Dimensions
1SRE
5
Dimensions scale]
(4)
scale]
Figure 20: (DiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from DiT-XT, on ImageNet-1K; similar trends appear other models.
TSKE
bSNe Dimensions
1:RE
Full
Dimensions scale]
(4) Dimensions scale]
Figure 21: (LightningDiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c.d) collapse into entangled points, revealing weak semantic structure. Results are from LightningDiT-XT. on ImageNet-1K: similar trends appear in other models.
13122
[a] Full Dimonsions
(B) Head Dimensions
Tall Dimensions [real scale]
(4) Tail Dievensions ((a) scalle
Figure 22: (MDT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embodding (a), while
tail-only emboddings (c,d) collapse entangled points, revealing weak semantic structure. Results are from MDT-XL, on ImageNet-1K:, similar trends appear in other models.
17

---

Published as a conference paper at ICLR 2026
SOKE
(2) Full Dimensions
(b) Head Dimensions
[c) Tall Dimensions Ercal scale]
(4) Tail Dimonstons ((a) scalle]
Figure 23: (MG) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from MG-XL on ImageNet-1K; similar trends appear in other models.
Organs
kl 3
bi 2.22 tp
5GRE
Figure 24: (REPA) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from REPA-XL on ImageNet- 1K; similar trends appear in other models.
SINC
KONE
1SRE
(a) Full Dimersions
(b) Head Dimersions
Tail Dimensions [real scale]
(4) Tuil Dimenssion ((a) scall)
Figure 25: (SIT) t-SNE of class embeddings by head vs. tail dimensions, Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from SIT-XT. on ImageNet-IK; similar trends appear in other models.
18

---

Published as a conference paper at ICLR 2026
A.4.3 ADDITIONAL BASELINES WITH SPARSE CONDITIONING
We extend the evaluation from the main paper two additional strong baselines: Light- ningDiT (Yao et al., 2025) and MG (Tang et al., 2025), following the same experimental protocol. As shown in Tab. 3, pruning low-magnitude dimensions in the conditional vec- tor consistently improves both FID and CLIP scores. These results reinforce our main finding that dense conditional embeddings contain noisy, low-utility dimensions and that sparsification can yield more efficient and effective generative models.
Table 3: More baselines. Performance and semantic metrics under sparsification. 4; prune every step, to: prune only at start, kn prune during last steps.
Prune
Threshold
# Removed Dims
FID 4
IS t
CLIP
Baseline MG (Tang et al., 2025)
0/1152 (0%)
7.2478
174.5151
30.199
== 0.01 (t:) 7= 0.01 (fo) 0.01 (ln-ka)
448/1152 (88)94%)
7.2791
170.55
30.140
Thil
448 448/1152 (38 94%)
7.2466
174.5537
30.199
448/1152
(38.949)
7.2455
174.3103
30.198
Baseline LightningDüT (Yao et al., 2025)
0/1152 (0%)
7.0802
169.8574
30.720
т = 0.01 (f.)
448/1152 (38.91%)
7.0130
166.0569
30.7045
Tail
7 0.01 (fu)
448 148/1152 (38 94%)
7.0712
169.9164
30.729
0.01 (ba-ka)
448 /155 (8%)
7.0745
169.9236
30.729
A.4.4 VARIANCE DISTRIBUTION ANALYSIS.
We analyze the per-dimension variance of the conditional embeddings by first computing the mean vector for each method and then measuring the variance across classes for each dimension. As expected, high-magnitude dimensions (head dimensions) exhibit substantially higher variance than the low-magnitude (tail) dimensions, reinforcing the observation that semantic information is concentrated in the head.
An exception is DiT, where the conditional vectors have smaller absolute values (maximum around 0.8, compared to 4-8 for other models), resulting in different variance pattern. These results, visualized in Fig. 26 to Fig. 31, provide further evidence of the head tail structure and its connection to semantic encoding in diffusion transformer embeddings.
For continuous-condition tasks such as pose-guided person image generation and video- geieed audio generation, the learned embeddings higher participation-ratio scores reported in Tab. of the main paper. Detailed variance and mean analyses for these tasks provided in Fig. 33 and Fig. 32.
noticeably less sparse, consistent with the
A.5 ADDITIONAL QUALITATIVE RESULTS
We present an extended set of qualitative results for both class-conditional image generation on ImageNet and pose-guided person image synthesis. These visualizations highlight the impact of pruning cow-magni dimensions in the conditional embedding vector.
Across a wide range of samples, we observe that removing these tail dimensions often preserves the generation quality and, in some cases, even enhances visual fidelity sharpness. This supports our main finding that semantic information concentrated in a small subset of head dimensions, while the majority of the embedding space is redundant.
Representative examples are provided in Fig. 34 through Fig. 41, demonstrating consistent trends across different models, classes, and poses.
A.6 USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used solely as a writing-assistance tool to polish grammar and improve sentence clarity. All research ideas, experimental design, analyses, and results were conceived and executed entirely by the authors. The LLM did not contribute to research ideation, data analysís, or the generation of any scientific content.
19

---

Published as a conference paper at ICLR 2026
Variance per Dimension
0.25
0.20
Variance
0.15
0.10
0.05
Mean Vector per Dimension
1
Mean Value
-1
2
-3
-5
-6
200
400
600
800
1000
1200
Dimension
Figure 26: Variance per dimension of the conditional vector learned by MDT-XL.
Variance per Dimension
0.25
0.20
Variance
0.10
0.05
0.00
Mean Vector per Dimension
Mean Value
-1
-2
-3
200
400
600
800
1000
1200
Dimension
Figure 27: Variance per dimension of the conditional vector learned by hightningDiT-XI
20

---

Published as a conference paper at ICLR 2026
Variance per Dimension
0.12
Variance
0.06
Mean Vector per Dimension
0-
-1
Mean Value
-3
-4
-6-
-7
200
400
600
800
1000
1200
Dimension
Figure 28: Variance per dimension of the conditional vector Iearned by MG-XL.
Variance per Dimension
0.12
0.10
0.08
Variance
0.06
0.04
0.02
Mean Vector per Dimension
0
Mean Value
-4
6-
200
400
600
800
1000
1200
Dimension
Figure 29: Variance dimension of the conditional vector learned by REPA-XL.
21

---

Published as a conference paper at ICLR 2026
Variance per Dimension
Variance
Mean Vector per Dimension
-1
Mean Value
-2
3-
-5-
200
400
600
800
1000
1200
Dimension
Figure 30: Variance dimension of the conditional vector learned by SIT-XL.
Variance per Dimension
0.0020
Variance
0.0010
0.0005
Hyppl
Mean Vector per Dimension
0.8
Mean Value
0.6
0.2
0.0
200
400
600
800
1000
1200
Dimension
Figure 31: Variance per dimension of the conditional vector learned by DIT-XL.
22

---

Published as a conference paper at ICLR 2026
Variance per Dimension
0.14
0.12
0.10
Variance
0.08
0.06
0.02
Mean Vector per Dimension
Mean Value
0
100
200
300
400
500
600
700
800
Dimension
Figure 32: Variance per dimension of the conditional vector learned by MDSGen.
Variance per Dimension
Variance
0.2
0.1
Mean Vector per Dimension
Mean Value
-4
-8
200
400
600
800
1000
Dimension
Figure 33: Variance per dimension of the conditional vector learned by X-MDPT.
23

---

Published as a conference paper at ICLR 2026
(REPA) 0.00
0.02
0.00
0.05
0-1.1
5,0
Figure 34: Class-conditional ImageNet generation with pruned embeddings (1). Removing low-magnitude dimensions from preserves or slightly improves image quality, confirming that semantic information is concentrated in few head dimensions.
T- (REEAA)
C ULO3
K- 0,02
T- 0.03
T 0.04
T- 0.05
TH 0.1
t-
t- 1.0
HOBBC
Figure 35: Class-conditional ImageNet generation with pruned embeddings (2). Removing low-maguitude dimensions from preserves slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.
24

---

Published as a conference paper at ICLR 2026
(REPA)
BO
Figure 36: Class-conditional ImageNet generation with pruned embeddings (3). Removing low- ma-ninage dimensions from preserves slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.
T- (REPA)
T-
Figure 37: Class-conditional ImageNet generation with pruned embeddings (4). Removing low-magnite www-manntuude dimensions from preserves or slightly improves image quality, confirming that semantic information is concentrated in few head dimensions.
25

---

Published as a conference paper at ICLR 2026
Conditions
Generated outputs
Style
Rose
X-MDPT (t- 0) Sparse (t- 0.1)
T-02
T-
0.3
t-0.4
0.5
T-1.0
5.0
Figure 38: Pose- Pose-guided person image synthesis with pruned embeddings (1). Pruning tail dimensions maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.
Conditions
Generated outputs
Style
Pose
X-MDPT (e- 0) Sparse (t- 0.1)
1- 0.2
1-03
t-04
T-5.5
1-1.0
5.5
Figure 39: Pose-guided person image synthesis with pruned embeddings (2). Pruming tail dimensions in maintains pose fidelity and visual quality, highlighting the redundancy of low- magnitude embedding dimensions.
26

---

Published as a conference paper at ICLR 2026
Conditions
Generated outputs
Style
Pose
K-MDFT (c- () Sparse (1 0.3)
0-02
1- 0.3
0.4
t-5
1-100
Figure 40: Pose-guided person image synthesis with pruned embeddings (3). Pruning tail dimensions in maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.
Conditions
Generated outputs
Style
Poae
MDDD 0) Sparse (r- 0.1)
T=02
r-
c- 0.4
r-
t- 1.0
5.0
Figure 41: Posc-guided person image synthesis with pruned embeddings (4). Pruming tail dimensions in maintains pose fidelity and visual quality, highlighting the redundancy of low-maguitude embedding dimensions.
27

---

Published as a conference paper at ICLR 2026
B MORE BASELINES AND ANALYSIS
B.1 Separating Timestep Embedding and Conditions Analysis.
Table 4: Separate timestep (t) and conditions (y). Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models on Imagenet-1K cans-conditioned generation. With denotes the methods used: AdaLN, and denotes the method used: concatenation.
Metrics
DiT
SIT
MDT
LightningDiT'
MG
REPA
UVIT
Embed.
Cosine Sim,
0.9001
0,9852
0.9905
0.9779
0.9934
0.9946
0.97917
y+t
nPR (Chorra) %
10.47
2.28
1.60
2.05
1.73
1.43
50.06
Cosine Sim.
0.7774
0.5436
0.8540
0.7166
0.6853
0.5194
0.00165
y
PR (Onorm) %
70.14
37.43
36.75
36.42
43.67
41.60
63.52
y
B.2 Text-conditioned Methods and Model Sizes.
Table 5: Separate timestep (t) and conditions (y). Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models text or vidco-conditioned generation. With denotes the methods used: AdaLN, and 3 denotes the method used: cross-attention.
Metries
X-MDPT-L'
X-MDPT-B'
X-MDPT-S'
SD3.0 (2B)
SD3.0
MDSGea'
AudioLDM Embed.
Cosine Sim.
0.9962
0.9999
y
8PR [Owon) ) %
48.42
53.41
54.79
13:57
y ++1
Cosine Sim.
0.9862
0,9009
0,9492
0.9949
0.9937 24.25
0.9918 9.56
0.1406
aPR (Owen.) ) %
20.88
29.75
37.51
52.39
63.09
B.3 More Quantitative Metrics.
Table 6: Precision and Recall with previous metrics: FID, IS, and CLIP.
Method
FIDI
ISt
CLIP
Precision"
Recallf
Remark
REPA (Yu et ul., 2025)
7.1694
176.02
29.746
0.8032
0.6236
Baseline
Pruned (7 = 0.01) to
7.1690
175.97
29.807
0.7878
0.6252
Ours
Pruned (r 0.01) fm
7.1598
175.49
29.805
0.8045
0.6381
Ours
Model-Guide (Tang al., 2025)
7.2478
174.5151
30.199
0.7842
0.6633
Bascline
Pruned (7 0.01) to
7.2466
174.5537
30.199
0.7854
0.6625
Ours
Pruned [r = 0.01) km
7.2455
174.3103
30.198
0.7898
0.6644
Ours
LightningDIT (Yao et al., 2025)
7.0802
169.8574
30.720
0.7928
0.6248
Baseline
Pruned (r = 0.01) to
7.0712
169.9164
30.729
0.7906
0.6256
Ours
Pruned (r = 0.01) 4.
7.0745
169.9236
30.729
0.7935
0.6265
Ours
Table 7: Quantitative metrics on the DeepFashion dataset of pose-guide person image generation task with masked diffusion transformers.
Method
FID
SSIM
LPIPS.
PSNR
Remark
X-MDPT (Pham et al., 2024)
18.6372
0.6798 0.6792
0.1672
17.336
Baseline
Pruned (7 0.1) 40%
18.6692
0.1673
17.328
Ours
B.4 Computational Reduction Analysis.
We provide the benefit of sparse embedding when prunning in Fig. 42.
28

---

Published as a conference paper at ICLR 2026
Computation Ops Sparsity
Runtime Sparsity (SPJ)
209
Bartiee
309
vccler
1344
130
Spartty
Figure 42: Dense vs. Sparse vectors. Compared the computation overhead. It shows that a sparse vector is more efficient in computation and has faster runtime than a dense vector (baseline).
B.5 Stability Analysis with Limited Timestep.
Durign the rebuttal, the reviewer asked for the generalization of limited timestep embedding. We perform a more comprehensive experiment in Fig. 43.
Cosine similarity of c
1.0
0.8
500 steps
200 steps
0.6
100 steps
50 steps
0.4
Training Step
500 steps
0.8
nPR (anorm)
200 steps
0.6
100 steps
50 steps
0.4
0.2
25000
50000 75000 100000 125000 150000 175000 200000 Training Step
Figure 43: Training dynamics of conditional embeddings. Top: batchwise cosine similarity of during training. Bottom: participation ratio (nPR) over training steps, showing progressive sparsification. With limited timesteps: 50, 100, 200, and 500.
29

---

Published as a conference paper at ICLR 2026
B.6 ADDITIONAL RELATED WORK
B.6.1 MULTIMODAL REPRESENTATION AND EMBEDDING SPACE
In this section, we provide a broader overview of foundational techniques in multimodal representation learning that inform our semantic bottlencck design. Early works explored diverse strategies for modality alignment and flexible fusion and data augmentation techniques, as well as in vision and speech processing (Loe al., 2020; Jung al., 2020; 2022; Vu el al., 2019; Trung & Yoo, 2019). Specifically, techniques focusing on modality-speciffe constraints (Kim et al., 2020) and self-supervised alignment. (Pham et al., 2021; 2023) laid the groundwork for establishing robust embedding spaces. Recent advances have further refined these representations through deep architectural optimizations, which are essential for maintaining semantic integrity within the conditioning signals of diffusion models.
B.6.2 GENERATIVE MODELING AND DIFFUSION REFINEMENT
The integration of Transformers into diffusion frameworks has shifted the focus toward scalable and semantically aware conditioning. Recent studics, such as MDSGen (Pham el al., 2025b) and MD3C (Pham et al., 2025a), demonstrate the efficacy of complex multi- data scheduling and contrastive constraints generative lasks. Additionally, exploring the capabilities of these models in specialized domains (Yoon et al., 2025; Hong et al., 2025; Pham et al., 2024; Koo et al., 2024; 2025) and structural adaptation (Pham et al., 2022b;a) provides comprehensive context for our proposed Diffusion Transformer with semantic bottlenecking.
30


Reviews:
[{"review_id": "zXNkzKRgvK", "reviewer": "Reviewer_LE2p", "summary": "This paper discovered a phenomenon in XL-size DiT and its variants: extremely high cosine similarity and semantic sparsity in conditional embeddings. By unveiling this discovery, the authors aim to gain a deeper insight into how diffusion transformers encode conditioning signals and call for further exploration of more efficient and compact conditioning mechanisms.", "questions": "Apart from what has been mentioned in the weaknesses section:\n1. It would be better to also report more evaluation metrics, such as precision and recall values, in class-conditional image generation, following similar evaluation settings in DiT, REPA, etc. \n2. In Section 5, the authors use REPA “as a representative model,” but didn’t comment later on whether the observations on REPA also universally exist in other models. \n3. In Section 5 “Continuous work”, the results are mostly qualitative not quantitative. Could more standard quantitative evaluation metrics, such as FID/SSIM/LPIPS, be added to compare the pruning and no-pruning cases?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Besides discovering the similarity and sparsity from condition embeddings in DiT models, this paper also tries to convert the observation to action by pruning the condition and provides multiple hypotheses for further exploration.", "weaknesses": "By discovering this phenomenon, the authors hope that “future architectures could benefit from compressed or hybrid conditioning strategies that maintain semantic fidelity while reducing computational overhead.” However, the conditional embedding carries not only the condition information but also the timestep information, which is important for image generation diffusion models. It’s not enough to only evaluate the semantic fidelity of the conditional embedding.\n\nMeanwhile, the paper does not present experiments that demonstrate the utility of this discovery, such as proving that pruning indeed \"reduces computational overhead\".\n\nFurthermore, the observation only builds upon the XL-size model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bigDRUO02y", "reviewer": "Reviewer_DXYW", "summary": "This paper presents a systematic empirical analysis of conditional embeddings in transformer-based diffusion models. Through extensive experiments across state-of-the-art models and multiple datasets, the authors show that conditional embeddings exhibit large redundancy and sparsity. The class-conditioned embeddings across different classes can achieve over 99% cosine similarity, which is kind of surprising, and only a small subset of dimensions carries meaningful semantic information. By pruning up to two-thirds of embedding dimensions, they demonstrate that generative quality is unaffected or even improved.", "questions": "1. Can the authors clarify whether they have empirically tested for similar redundancy/sparsity in conditional embeddings in U-Net based diffusion models? If so, does the semantic bottleneck extend beyond the transformer/AdaLN regime?\n\n2. Could the authors provide detailed ablations on how different conditioning injection techniques (addition vs. concatenation vs. cross-attention), projection, or other schemes affect the redundancy and sparsity? Would, for example, nonlinear projection heads mitigate the observed bottleneck?\n\n3. Although some of the dimensions can be pruned and make little difference in the generation quality, they seem to make little improvement. How would this be useful? \n\n4. Despite claims of negligible drops or marginal improvements in generation quality after pruning, would pruning induce minor artifacts like edge collapse or structure distortion? Could the authors present more failure case analyses?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.  The paper evaluates a broad set of transformer-based diffusion models, including DiT, MDT, SiT, REPA, LightningDiT, etc, on diverse benchmarks. And the paper shows evidence of near-uniform cosine similarity and embedding sparsity. The breadth of experiments convincingly substantiates the paper’s core claims.\n\n2. The figures and results tables in the paper vividly illustrate the redundancy and sparsity of class embeddings, especially the high cosine similarity and the dominance of a few embedding dimensions, and the visual/quantitative impact of pruning. The ablation and t-SNE plots reinforce key claims regarding semantic encoding and redundancy.", "weaknesses": "1. The theoretical analysis, while mathematically sound in defining PR and sparsity, stops short of providing a fundamental theoretical explanation of why such extreme redundancy and cosine similarity emerge in the conditional embeddings of transformer-based diffusion models. The paper’s stated hypotheses remain largely empirical and conceptual, lacking formal proofs or broader generalization to other conditioning modalities or even to transformer architectures outside the diffusion context. For instance, while the AdaLN/linear projection argument is plausible, its theoretical justification is not fully fleshed out.\n\n2. The investigated tasks are representative, but would be better if they could be extended further. For example, would the text-to-audio diffusion models also demonstrate such highly correlated class embeddings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AVB9k2LwBZ", "reviewer": "Reviewer_jVCn", "summary": "This is a very interesting work. This paper makes significant contributions to the understanding of conditional embeddings in diffusion Transformers, a critical component of state-of-the-art generative models.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. It is the first work to systematically investigate the internal structure of conditional embeddings in diffusion Transformers, uncovering two core emergent properties—extreme angular similarity (exceeding 99% on ImageNet-1K and 99.9% on continuous-condition tasks) and high-dimensional sparsity (only 1–2% of dimensions carrying substantial semantic information). This fills a critical gap in existing literature, which has primarily focused on architectural advances rather than the intrinsic characteristics of conditional encoding.\n2. The experiments are well-designed and extensive, covering six diffusion Transformer models (DiT, MDT, SiT, REPA, LightningDiT, MG) on discrete class-conditional image generation (ImageNet-1K) and two continuous-condition tasks (pose-guided image synthesis with X-MDPT and video-to-audio generation with MDSGen). \n3. The demonstration that pruning up to 66% of low-magnitude dimensions preserves or even improves generation quality provides actionable insights for model optimization. This offers a potential path toward more efficient conditioning mechanisms—particularly valuable for resource-constrained applications.", "weaknesses": "1. Although the author reveals the redundancy problem in condition embeddings of diffusion models via empirical analysis, they do not provide any further method to improve the efficiency or performance of existing models based on the observation.\n2. More conditions should be considered. For instance, user-generated descriptions or prompts are more general than class-conditional prompts. I suggest the author investigate more types of conditions.\n3. Maybe the time embedding and prompt embedding should be considered separately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PxpxHnoYbQ", "reviewer": "Reviewer_TVaD", "summary": "This paper investigates the properties of conditional embeddings in conditional diffusion models, revealing that they are sparse yet locally uniform across dimensions. The analysis is conducted on several representative class-conditional diffusion models and is extendable to continuous class embeddings and diverse conditioning signals, e.g., video-to-audio and pose-guided image generation. By removing redundant dimensions, slight improvements in FID and CLIP scores are observed.", "questions": "What is the setting of k in Table 2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The finding that conditional embeddings in DiT are sparse but locally uniform is meaningful, as it reveals redundancy in the conditional representations of current conditional diffusion models. This insight can motivate new directions for future research.\n- Both qualitative and quantitative experiments on diffusion models with various types of conditions validate that conditional embeddings in current DiTs exhibit significant redundancy.\n- The authors demonstrate that simply removing redundant dimensions in conditional embeddings at inference time can enhance generation quality to some extent, highlighting the potential of optimizing conditional embeddings to improve DiTs.", "weaknesses": "- The proposed hypotheses lack experimental support. For example, regarding the hypothesis about the cause of high cosine similarity in conditional embeddings, the authors argue that DiTs favor embeddings providing stable and robust signals for denoising because they condition on embeddings across all timesteps. Maybe a training log similar to Fig. 12, but for a model trained on fewer timesteps, could validate this hypothesis. Since inference performance is not the concern here, such a model could be trained on a subset of timesteps.\n- There is a contradiction between Lines 454–457, which state that high cosine similarity occurs only in transformers and not in U-Nets, and Lines 467–470, which state that similar redundancy can also appear in U-Net-based diffusion models. Moreover, the authors neither present nor discuss experimental results on U-Net diffusion models.\n- The authors claim that AdaLN amplifies high-magnitude dimensions of conditional embeddings, thereby preserving semantic differences, but this claim also lacks experimental validation. What happens if AdaLN is replaced with another conditioning mechanism, such as cross-attention?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)