Representation Alignment for Diffusion Transformers without External Components
Authors: Dengyang Jiang
Mengmeng Wang
Liuzhuozheng Li
Lei Zhang
Haoyu Wang
Wei Wei
Guang Dai
Yanning Zhang
Jingdong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Transformers
Self-Representation Alignment
Primary area: generative models
TL;DR: Self-Representation Alignment for Diffusion Transformers

Abstract: Recent studies have demonstrated that learning a meaningful internal represen-
tation can accelerate generative training. However, existing approaches necessi-
tate to either introduce an off-the-shelf external representation task or rely on a
large-scale, pre-trained external representation encoder to provide representation
guidance during the training process. In this study, we posit that the unique dis-
criminative process inherent to diffusion transformers enables them to offer such
guidance without requiring external representation components. We propose Self-
Representation Alignment (SRA), a simple yet effective method that obtains rep-
resentation guidance using the internal representations of learned diffusion trans-
former. SRA aligns the latent representation of the diffusion transformer in the
earlier layer conditioned on higher noise to that in the later layer conditioned on
lower noise to progressively enhance the overall representation learning during
only the training process. Experimental results indicate that applying SRA to
DiTs and SiTs yields consistent performance improvements, and largely outper-
forms approaches relying on auxiliary representation task. Our approach achieves
performance comparable to methods that are dependent on an external pre-trained
representation encoder, which demonstrates the feasibility of acceleration with
representation alignment in diffusion transformers themselves.

Published as a conference paper at ICLR 2026
REPRESENTATION ALIGNMENT FOR DIFFUSION TRANSFORMERS WITHOUT EXTERNAL COMPONENTS
Dengyang Jiang?.1;
2:13 Mengmeng Wang
Liuzhuozheng Li
Lei Zhang?
Haoyu Wang" Wei Wei Guang Dai
Yanning Zhang*
Jingdong Wang
SGIT AI Lab, State Grid Corporation of China "Northwestern Polytechnical University
Zhejiang University of Technology "Baidu Inc.
Quick overview at: https::/wwwwwjyy..tttt.t.. io/sra
Code is available al: https: com/ htpp:s//gtthu...cmm///wwww.yt
ABSTRACT
Recent studies have demonstrated that learning a meaningful internal represen- tation can accelerate generative training. However, existing approaches necessi- tate to either introduce an off-the-shelf external representation task or rely on a large-scale, pre-trained external representation encoder to provide representation guidance during the training process. In this study, we posit that the unique dis- criminative process inherent to diffusion transformers enables them to offer such guidance without requiring external representation components. We propose Self- Representation Alignment (SRA), simple yet effective method that obtains rep- resentation guidance using the internal representations of learned diffusion trans- former. SRA aligns the latent representation of the diffusion transformer in the earlier layer conditioned on higher noise to that in the later layer conditioned on lower noise progressively enhance the overall representation learning during only the training process. Experimental results indicate that applying SRA to DiTs and SiTs yields consistent performance improvements, and largely outper- forms approaches relying on auxiliary representation task. Our approach achieves performance comparable to methods that are dependent on an external pre-trained representation encoder, which demonstrates the feasibility of acceleration with representation alignment in diffusion transformers themselves.
Generative
Loss
Represontation Loss (a.g.. loss)
Loss
Alignmont
Generative
Loss
Self-Alignment
latent representiation
latent represantation
reprosentation
Diffusion
Regulation
Diffusion Model
Representation Model (e.g. DINOvZ)
Diffusion Model
Diffusion Model
Model
Process
(e.g., mask)
Representation Training Paradigm Representation
(c) Representation Component
external representation task to guide diffusion transformer. Middle: Methods like REPA leverage an external representation foundation model as guidance. Right (our approach): We do not use any external representation component but still obtain such guidance through proposed self-representation alignment technique.
Figure 1: Left: Methods like MaskDiT and SD-DiT
INTRODUCTION
Diffusion transformers (Peebles Xie 2023 Ma et al. and vision transformers (Dosovitskiy et al.
2024
Chen et al.
2024a
Team 2025
2021
Liu et al.
202
Wang et al. 2021 have held
This work was completed during the internship at SGIT Al Lab. State Grid Corporation of China. *Corrsponning author. PProject lead.

---

Published as conference paper at ICLR 2026
Increasing Block Layers
t=
layer1
layer3
layerS
layert3
layer18
layer23
layer28
0.0.3
t=5.5
Decreasing Noise Level
Validation Acc. (%)
t- 0.7
Layer Index
PCA Visualization Result
(b) Linear Probing Result
Figure 2:
We empirically investigate the trend of latent representations in the original SiT-XL/2. Left: Using PCA Abdi & Williams (2010) to visualize the latent features, we observe that the features lead process from bad to good when increasing block layers and decreasing noise level. Right: A similar trend also be seen in the linear probing results ImageNet. Investigation of DiT is provided in Appendix A which leads to the similar results as SiT.
the dominant positions in visual generation and representation because of their scalability during pre-training (Brooks et al. 2024 Chen al. 2022 Oquab et al. 2023 Sun et al. 2024) and gener-
alization capacity for downstream tasks (Kirillov et al. 2023 Liu al. 2024
Lin et al.
2025
Recently, many works (Zheng et al. 2023 Zhu al. 2024a eu al.
2025 Krause et al.
2025) have
explored leveraging representation components of vision transformers for diffusion transformer's training and have shown that learning high-quality internal representation can not only speed up the generative training progress but also improve the generation quality. These works either utilize the external discriminative loss in representation learning (e.g., MAE's (He et al. 2022). IBOT's (Zhou et al. )022)) shown in Figure T left) leverage large-scale pre-trained representation foundation model (e.g., DINOv2 (Oquab et al. 2023), CLIP (Radford et al. (02I)) shown in Figure Mi give representation guidance for the diffusion transformer during the original generative training.
However, the former methods require an external representation task, and the latter method relies on powerful external pretrained encoder, which limits usage scenarios when there are no good external encoders. Thereout, intriguing yet underexplored problem has come to light: Can we obtain such representation guidance without external representation components?
Different from the representation model that takes a clean image as input and then outputs semantically-rich feature, diffusion model often takes noise latent as input and ob- tains cleaner one step-by-step. In other words, the generative mechanism by which the diffusion model operates can generally considered as a bad to good process. Inspired by this behavior, hypothesise that the representations in it also follow such trend. To testify this, we perform an empirical analysis with recent diffusion transformers (Ma et al.) 20244 [Peelles & Xic (023). As shown in Figure leett), first find out that the latent features in the diffusion transformer progressively refined, moving from bad to good, as block layers increase and noise level decreases. Next, akin the results in previous studies (Yu al. 2025 Xiang et al. 2023), the diffusion transformer already learns meaningful discriminative representations as shown in Fig- ure rright). Meanwhile, although the accuracy drops off after reaching a peak about layer 20 because the model needs to shift away to focus on generating images with the quality of the representations basically transfers from bad good by increasing block layers and decreasing noise level. These indicate that the diffusion transformer gets roughly from bad-to-good discriminative process when only generative training performed.
Our observations:
observe that
details,
The observations above motivate us to align the weaker representations in the dif- fusion transformer to the better ones the original generative training, thereby not only giving supervision to intermediate layers for fast convergence (Lee et al. 2015), but also enhancing the overall representation learning of the model without involving any external representation compo- nent. Driven from this inspiration,
Our approach:
present Self- Representation Alignment (SRA). As shown in Figure ( right), SRA does not need any representation component; in essence, it aligns the output latent representation in earlier layer conditioned higher noise that later layer conditioned on
2

---

Published as a conference paper at ICLR 2026
lower noise to achieve self reffeeepreentttinnn alignment. Meanwhile, in order to boost the performance, we obtain the target features from another model that shares the same architecture with the trainable model but updates weight by weighted moving average (EMA)i Furthermore, the student's output latent feature is first passed through the projection layers to conduct slight nonlinear transformation for better representation extraction and then aligned with the target feature output by the teacher. In a nutshell, our SRA can offer a flexible way to integrate representation guidance without external component needs and architectural modification.
Finally, we conduct comprehensive experiments to evaluate the effect of SRA. After a series of component-we analyses. we show that SRA brings significant performance improvements both DiTs (Peebles & Xie, 2023) and SiT's (Ma et al. (024).. Moreover, our ablation study highlights the crucial role of internal representations in the success of SRA, which supports our central hypothesis: diffussoon transformers can achieve representation aligument without external components.
In summary, our main contributions are follows:
We analyze the representations in diffusion transformers and assume that the unique discriminative process makes it possible to achieve representation alignment without external components.
We introduce SRA, simple yet effective method that aligns the output latent representation of the diffusion transformers in the earlier layer conditioned on higher noise to that in the later layer conditioned on lower noise to achieve self relf-repreenntaionn alignment.
With SRA, both DiTs and SiTs achieve sustained training speed acceleration and nontrivial generation performance improvement.
METHOD
2.1
PRELIMINARY: TRAINING OBJECT OF DIT AND SIT
As our method is built upon the denoise-based model (DiT) and flow-based model (SiT), to facilitate more seamless introduction of SRA in the subsequent part, now present brief overview of the training object of these two types of models. We omit the class- conss-conntiion here for simplicity. We also leave more detailed mathematical descriptions of these types of models in Appendix
Denoise-based models learn to transform Gaussian noise into data samples through a step-by-step denoising process. Given pre-defined forward process that gradually adds noise, these models learn the reverse process to recover the original data.
For data point Xy from distribution Xy p(x), the forward process follows: g(xx(x+-1)
(xx;; /1-3,xx, 3,I). The model learns to reverse this process using a neural network Eç(X,, t) that predicts the noise added at each step. The network trained using a simple mean squared error objective that measures how well it can predict the noise:
Csiple Ex,, e, |lle Ec(x;, t)][
Different from the denoise-based model, the flow-based model aims to learn a velocity field (x,, () that governs a probability flow ordinary differential equation (PF ODE). This PF ODE allows the model to sample data by flowing toward the data distribution. This forward process is defined as:
X, 0xXo + OcC, where xo ~ p(x) is the data, ~ NN(,, I) is Gaussian noise, and at and ar are monotone decreasing and increasing functions of t E [0, T". respectively, The PF ODE is given by:
Og dT 1, or 0g = 0,
* = =V(xx,,t)
(3)
where the marginal distribution of this ODE at time (x()) matches that of the forward process. To learn the velocity field, the model trained minimize the following loss function:
Evelecity = Ex.,e. [ll (xl. f) - â₂ []
(4)
For the sake of simplicity, in the following part, we use generative loss (Egea) to uniformly represent the two generative training objectives of denoise-based and flow-based methods.
IIn the following part, use term 'student' to represent the trainable model and 'teacher' to represent the FMA model in SRA for simplicity.
3

---

Published as a conference paper at ICLR 2026
2.2 SILF-REPRESENTATION ALIGNMENT
Previous studies have demonstrated that learning good internal representations can both speed up the diffusion transformer's training convergence. In SRA, insight is aligning the student's latent feature in the earlier layer conditioned on higher noise with that in the later layer conditioned on the lower noise of the teacher conduct reteeeeprennntnonnn alignment without requiring any external representation components. As depicted in Figure E the goal of our simple training framework is to let the diffusion transformer not only predict noise or velocity but also align with better visual rep- resentations from itself. This operation thereby provides a simple way to enhance the representation learning in the diffusion transformer during generative training without the need to design complex representation regulation or introduce external representation foundation models.
Formally, let ʃ be the trainable stu- dent model and f. be the teacher model. Considering input noise la- tent, timestep, and condition to be Xt, f, and c. Then, we can ob- tain the student encoder latent out- put = 1 (x,, c) E BBxNXD where D, size. number of patches and the em- bedding dimension for and m de- note the output from the th layer in f. Similarly, the output of the teacher can be expressed as c) RBxNx1 x In our SRA, we set S n, k 2 0, and S (t k) < E Hence, we conduct self-representation align- ment using the teacher's output y. and the student's output transform (y( E xBxNxD, where (y) is a projection of the student encoder out- y that through a lightweight train- able MLPs head jæ- Notably, this projection head can be discarded optionally after training, which enables SRA to provide guidance without altering any architecture in diffusion transformers.
Lgen
DIT/SIT Block
Later Layer Lower Noiso
DiT/SiT Block
DIT/SIT Block
Csa
DIT/SIT Block
DIT/SIT Block
DIT/SIT Block
Earlier Layer
sg
Higher Norse
MLPs
DIT/SIT Block
DIT/SIT Block
are the batch-
DIT/SIT Block
DIT/SIT Block
DIT/SIT Block
DiT/SiT Block
f (x:-ktt
Figure 3: Overall framework. SRA aligns the student's la- tent representation in the earlier layer conditioned on higher noise (green branch) to that of the teacher in the later layer conditioned lower noise (blue branch) to achieve self- representation alignment. We use a noo-ggradeent (sg) op- erator the teacher to let gradients flow only through the student, and update the teacher's 's parameters with an expo- nential moving average (ema) of the student's parameters.
In particular, SRA attains self-alignment by minimizing the patch-wise distance between the teacher's output (y.) and the student's output variant (y)):
N
Csa(Ses v) = Ex,,ti
Σ NN
1
dist(y! [di
(5)
where [i] is a patch index, (: :) pre-defined distance calculation function, and Ses th the parameters of student diffusion transformer and the projection head.
Finally, we add the abovementioned two objectives for joint learning:
L = Lgen + AL
(6)
where À 0 hyperparameter that controls the trade-off between the generation object and the self-representation alignment object.
2.3 TEACHER NETWORK
In SRA, we do not need an off-the-shelf teacher to give the guidance. Meanwhile, using the output of the same model as the target to conduct supervision would cause poor results (both found in
previous work (Chen & He 202T Grill et al. 2020 and our experiment). Thus, we build the
For SiTs, fmaz and for DiTs, tmaz 1000. In practice, truncate (t k) to O if it less than
4

---

Published as a conference paper at ICLR 2026
teacher from past iterations of the student network using exponential moving average (EMA) the student weights. In specific, the updated role of EMA is S = aç + (1-a)6a, where a E [0. 1) is the momentum coefficient. Note that this is widely used in self-supervised learning (SSL) (Zhou et al. 2022 Caron et al. 2021 He et al. 2020). But we find the default updating role in SSL does 0.9999 unchanged works surprisingly well in framework.
perform well here, instead, Experiments and analyses of different a settings are presented in Appendix F Moreover, we do not use other operations like clustering constraints (Caron al. et al. 2020; Richemond et al., (2000), and centering (Caron et al. 2021 Zhou et al. (022) in SSL because we find that the training progress is already stable enough without applying these tricks.
a 2018 2020), batch normalizations (Grill
3 EXPERIMENT
In this section, we mainly focus on answering the following questions:
How each design choice and component in SRA influence the performance? (Table
Does SRA work different basclines across different model sizes? (Figure 4 and Table
Can SRA show comparable or superior performance against methods that leverage either the ex- ternal representation task or the external representation encoder? (Table Table Figure
Does SRA genuinely enhance the representation capacity of the baseline model, and is the gener- ation capability indeed strongly correlated with the representation guidance? (Figure
3.1 EXPERIMENTAL SETUP
Implementation details. Unless otherwise specified, the training details strictly follow the setup in DiT (Peebles & Xie 2023) and SiT (Ma al. 2024), including Adam (Loshchilov & Hutter 2017) with a constant learning rate of Ie-4, no weight decay, batchsize of 256, using the Stable Diffusion VAE (Rombach et al.) 2022b) to extract the latent, and etc. For model configurations, we use the B/2, L/2, and XI/2 architectures introduced in the DiT and SiT papers, which process inputs with a patch size of Additional experimental details, hyperparameter settings, linear probing details, and computing resources, are provided in Appendix
Evaluation. We report Fréchet inception distance (FID (Heusel et al. 2017 ). sFID (Nash et al. 2021) inception score (IS (Salimans et al.) 20166), precision (Pre.) and recall (Rec.) (Kynkäänniemi et al. 2019). To ensure a fair comparison with previous methods, we also use the ADM's TensorFlow evaluation suite (Dhariwal & Nichol, 2021 I with 50K samples and the same reference statistics. A
detailed breakdown of each evaluation metric is included in Appendix [
Methods for Comparison. We compare with recent advanced methods based on diffusion mod- els: (a) Pixel diffusion:: ADM, VDM++, Simple diffusion, CDM, (b) Latent diffusioo with U-Ner: LDM, and (c) Latent diffusion with transformers: DiT, SiT, SD-DiT, MaskDiT, TREAD, REPA, and MAETok. We give detailed descriptions of each method in Appendix LL Note that in the diffusion transformers family, we choose compare with DiT/SiT and their modifications with representation components involved but do not compare with works aiming at designing advanced architecture like lightningDiTT (Yao & Wang 2025) and DDT (Wang et al. 2025a).
3.2 COMPONENT-WIS. WISE ANALYSIS
Below, we provide detailed analysis of the impact of each component. We use SiT-B/2 and train with SRA for 400K iterations for evaluation. Results are shown Table We also provide some hyperparameter setting principles for more casily transferring SRA to new models in Appendix G
Block layers for alignment. We begin by analyzing the effect of using different blocks of student and teacher for alignment. We observe that using the teacher's 's last but not least few layers (e.g.. 8) to regulate the student's 's first layers (e.g., 3) leads to optimal performance. We assume that the first few layers need more guidance so they catch semantically meaningful representation for subsequent generation. Miln there is strong correlation between the quality of the representations the teacher's layers and the performance of the corresponding aligned student (we give the quantitative
5

---

Published as a conference paper at ICLR 2026
Table I: Component-wise analysis ImageNet 256x without classifier-free guidance. J and T indicate whether lower or higher values are better, respectively. m - n denotes aligning features from mt layer of the student with that from nh layer of the teacher. [0, kmax) denotes time interval is chosen randomly from to kmax. PH. denotes whether to use the projection head.
Block Layers
Time Interval
A
PH.
FID/
IS1
SiT-B/2 Baseline Ma et al
2024
33.02
43.71
00 0.2)
0.2
34.85
40.28
4 + 8
0, 0.2)
0.2
30.00
47.78
4 + 10
0, 0.2)
0.2
v
30.65
46,69
12
0.2.2)
0.2
33.19
43.30
0.,02)
0.2
32.14
46,36
+
0, 0.2)
0.2
29.31
50.13
8
0, 0.2)
0.2
29.10
50.20
(0,0.2)
0.2
37.08
41.54
3 8
0.0
0.2
31.07
47.32
0.1
0.2
29.55
49.01
0.2
0.2
30.70
47.72
1
0.10.)
0.2
29.38
49.32
+ 8
00 0.2)
0.2
29.10
50.20
00 0.3)
0.2
29.15
50.01
+ 8
[0, 0.2)
0.1
30.65
48.31
[0, 0.2)
0.2
29.10
50.20
00, 0.2)
0.3
29.28
49.72
0.2)
0.4
29.75
49.30
[0, 0.2) 00 0.2)
0.2
34.23
41.07
0.2
29.10
50.20
results and analysis in the latter Section 3.4). Based on these results, we set alignment layers as 3 + 8, 6 16, and 8 20 for B, L, and XL models respectively as defaul
Time interval for alignment. We then study the time interval (meaning the same as & in Sec- tion used for alignment. Here, study fixed and dynamic intervals. Notably, find that using the teacher's s features input with a lower noise level than the student's leads to performance enhancement, and interval value of 0.1 the mean of 0.1 is optimal. We hypothesize that this is because lower noise levels can offer better representation guidance, but an excessively large time focus only on optimizing alignment
interval hinder the model's learning process, causing loss at the expense of neglecting the generative aspects. As dynamic interval shows slightly better performance, we apply time interval as 0.2 in our feature experiments
Regularization coefficient for alignment. We also examine the effect of the coefficient À of self- alignment loss. Note that the results are all better than the baseline, and there only little variance for different X. We choose 0.2 as default since it achieves overall better performance.
Effect of projection head for alignment. We finally examine the effect of the projection head for alignment. Surprisingly, We observe that using this simple head to post-possess the student's output much better than directly using it to align. We hypothesize this slight operation enables the model to learn more effective hidden representations for subsequent projection head to conduct transforma- tion for final alignment, rather than explicitly aligning the entire latent feature that could potentially disrupt the original generation field that each layer and timestep is responsible for (Zhang et al. 2024 2023 Frenkel et al. 2024). Hence, we keep using the projection head in future experiments.
3.3
SYSTEM-LEVEL COMPARISON
In this section, a system-level comparative study is performed to assess recent diffusion- based ap- proaches against diffusion transformers with SRA.
by default because DIT's discriminative behavior across layers is slightly different from SiT's (see Figure 2 and Figure For DiTs, set time interval [[0, 200) by default since DiT's adopt the linear variance schedule with t where E 1, 2,.,... 999) Z.
For DiTs, we set alignment layers + 7.6 + and

---

Published as a conference paper at ICLR 2026
DIT-B
DITL
30-
DITAL
SRA
DITL -SRA
SRA
184
10-
FID 100-SSK
14-
Training Time [N]
Thaining Time (h)
Truining Time (h)
SIT-8
SIT-L
27.5
SITXL
35.0
22.3
20.0
Ro
37.5
13.0
12.0
Training Time [N]
Traniig Time (h)
Training Time (h)
Figure 4: FID comparisons with vanilla DìTs and SiTs on ImageNet 256x256 without classifier- free guidance (CFG). The training time (h) tested on single machine with A100 GPUs.
Figure 5: Selected samples ImageNet 256 from the SIT-XL + SRA. We use classifier-free guidance with w 4.0. More uncurated samples are provided in Appendix L
SRA accelerate the convergence of models in different types and sizes. As our method requires an additional forward pass of the teacher model. For a fair comparison, we report the FID compar- ison with the baseline under training time. We also give specific training speed and GPU memory usage in Appendix H as well as FID vs. training time against REPA in Figure 6 As shown in Figure. 4 diffusion transformers trained with SRA demonstrate substantial improvements in perfor- mance at the same training time. Moreover, similar to the observation in some SSL works (Oquab) et al., 2023 Fan et al. 2025), we notice that the effect of SRA in a larger size model is more sig- nificant, which is probably because the larger model tends to provide richer guidance. Mcanwhile, the benefits of SRA do not saturate even when the models have already achieved a low FID score. We assume that this is likely due to the teacher's constantly improving capacity, which allows it to provide better and better representation guidance for the student when the training goes on.
Meanwhile, we also conduct text-to-image experiment following REPA to use COC02014 (Ln jet al. 2014) and MMDiT (Esser et al., (2224) to further demonstrate the effect of SRA, the results in Table shows that SRA can naturally extend to text-to-image generation. Our sepferepresnntaioon
7

---

Published as conference paper at ICLR 2026
Table 3: System-level comparison on ImageNet 256x256 with Classifier-fre Guidance (CFG).
The best and second-best results on each metric are highlighted bold and underlined.
Model
Epochs
Tokenizer
FID
sFID4
1St
Pre.t
Rec.1
Pixel diffusion
ADM-U
400
3,94
6.14
186.7
0.82
0.52
VDM++
560
2.40
225.3
Simple diffusion
800
2.77
211.8
CDM
2160
4.88
158.7
Latent diffusion, U-Net
LDM-4
200
LDM-VAE
3.60
247.7
0.87
0.48
Latent diffusion, Transformer
DiT-XL/2
1400
SD-VAL SD-VAE
2.27
4.60
278.2
0.83
0.57
SiT-XL/2
1400
SD-VAE
2.06
4.50
270.3
0.82
0.59
SD-DiT
480
SD-VAE
3.23
MaskDiT
1600
SD-VAE
2.28
5.67
276.6
0.80
0.61
DIT TREAD
740
SD-VAE
1.69
4.73
292.7
0.81
0.63
SiT REPA
800
SD-VAE
1.42
4.70
305.7
0.80
0.65
SiT MAETok
800
MAE-Tok
311.2
SIT SRA (ours)
400
SD VDE
1.85
4.50
297.2
0.82
0.61
SIT SRA (ours)
800
SD- VD-V
1.58
4.65
311.4
0.80
0.63
Table 4: System-level comparison on ImageNet 512x512 with Classifier-free Guidance (CFG).
Model
Epochs
FID,
sFID,
ISt
Pre.1
Rec.1
Pixel diffusion
VDM++
2.65
278.1
Simple diffusion
800
4.28
171.0
Latent diffiusion, Transformer
DIT-XL/2
600
3.04
5,02
240.8
0.84
0.54
SIT-XL/2
600
2,62
4.18
252.2
0.84
0.57
MaskDiT
600
2.50
5.10
256.3
0.84
0.56
+ REPA
2.08
4.19
274.6
0.83
0.58
SiT + SRA (ours)
200
2.17
4.15
279.3
0.83
0.59
alignment strategy works, as evidenced by our experiments: applying SRA to MMDIT without any hyperparameter tuning improves FID from 5.66 to 4.75) and PickScore (from 20.65 to 21.14) over the baseline, and is comparable to REPA.
Table 2: FID and Pickscore (Kirstain et al. 2023) comparisons with vanilla MMDiT and REPA on COCO2014.
SRA demonstrates superior comparable perfor- mance compared to other methods. We also provide quantitative comparison between SiT-XI XL with SRA and other recent methods shown in Table 3 and Table 4 ImageNet 256x256, Our method already outperforms the original SiT-XL model with 1000 fewer epochs, and it is further improved with longer training. At 800 epochs, SRA achieves FID of 1.58 and IS of 314.4. It is worth noting that this result is far superior methods (e.g., MaskDiT) that rely external representation task and is comparable with methods (e.g.. REPA) that depend heavily on an external pre- pre-trained representation model. In higher resolution settings, the performance of SRA surpasses its baseline using 3x fewer training iterations. Training with the same iterations, SRA outperforms REPA in terms of three metries (sFID, IS, and Rec) and on par with REPA in other metrics.
Method
FID
PickScoref
ODE, NFE=50, Trained for 150K iter
MMDIT
5,86
20.05
MMDIT REPA
4.60
20.88
MMDiT SRA
4.85
21.14
SRA shows a more sustained boost than REPA. As shown in Figure 6 although REPA con- verges quickly the beginning of training due to its large-scale, pre-trained representation model for guidance, the performance saturates after about 200 epochs. On the contrary, the progressively higher-quality guidance teacher have throughout training in SRA makes method to provide more sustained boost without the need for any external models. This observation of saturation ef- fects and potential detrimental impacts of REPA in later stages has also been corroborated by several
8

---

Published as a conference paper at ICLR 2026
STILL
Teas her Loger for Aligrment
Sarker
Vaisaation CN)
FD--SI
Layer
Validation Acc. PS]
Training Herationa
Linear Probing Result (tm0.1)
(b) Linear
Result
Validation Acc.
Linear Probing Compartsion
Figure 7: We empirically investigate the effect of representations in SRA. (a) and (b): Linear prob- ing result vanilla SiT-XL trained for 1400 epochs and SiT-XL SRA trained for 800 epochs. (c): Linear probing (similar plot of DiT + SRA is provided in Appendix Jp. (d): Linear probing results of layers of the teacher and the student used for alignment during training.
FID plot of SiT-XL + SRA with different teacher's 's output layers for alignment
contemporary studies (Wang et al. 2025b Zhang et al. 2025 We believe this limitation of REPA be offset by the sustained enhancement facilitated by SRA.
3.4 ABLATION STUDY
SiT-XL
Since SRA introduces repre- sentation guidance in an im- plicit way, we thus aim at testing whether representation truly matters in SRA. We give the answer with the following experiments.
22.5
SiT-XL
REPA
20.0
SiT-XL
17.5
15.0
12.5
10.0
Enhanced representation ca- pacity with SRA. We first compare the representation ca- pacity of vanilla SiT and SiT trained with SRA. As shown Figure (a)) and Figure
7.5
5.0
120
240
400
580
Training Time (h)
Figure Training time (h) vs. FID plot without CFG.
SRA consistently improve the quality of latent representation in the diffusion transformer, as indi- cated by better linear probing results across different blocks and timesteps.
Tight coupling between generation quality and representation guidance in SRA. We then in- vestigate the correlation between generation performance and the representation guidance (detailed experimental setup can be found in Appendix in SRA. Figure (c)) reveals a strong correlation between linear probing accuracy and FID scores as the teacher network layers for alignment are varied. This finding underscores that the model's generative capabilities are indeed closely tied to the effectiveness of the self-representation guidance mechanism.
(())
Consistent representation improvement during training in SRA. We finally verify whether the representational capacity of the student and teacher models continuously improving during the training process and whether the teacher can always give better guidance for the student. As shown in Figure (d)). the teacher's representation quality consistently improves throughout training (from 38.1 at 200K iterations to 54.2 at 800K iterations) and consistently outperforms the student model each stage. This empirical evidence directly supports our claim that the teacher's 's improving capacity provides better representation guidance training proceeds.
RELATED WORK
Here, we highlight key related studies and defer discussion of other relevant studies to Appendix K
Representations guidance for diffusion transformer's training, Many recent works have at- tempted to introduce representation guidance in diffusion transformer training. MaskDiT (Zheng et al. 2023) and SD-DiT (Zhu et al. 2024a) add MAE's (He et al.) (022) and IBOT's (Zhou et al.)
2022 learning task into the original DiT's training progress. TREAD (Krause et al. 2025 de-

---

Published as a conference paper at ICLR 2026
sign token routing strategy with MAE loss to speed up the diffusion model's training. REPA Yu iet al. 2025) utilizes a large-scale data pre-trained representation model for regulating the diffusion model's 's latent feature. VA-VAE (Yao & Wang 2025) and MAETok (Chen et al. 2025) align the latent distribution of the tokenizer with the external representation foundation model and show this alignment is beneficial to resulting diffusion models. Different from this work, we aim to look for representation guidance in the diffusion model itself and its own training paradigm.
5 LIMITATION AND FUTURE WORK
Due to limitations in computation resources, we are unable to conduct large-scale text-to-video (12v) pretraining with SRA. However, we maintain that applying SRA video domain is conceptually sound for the following reasons. First, even within the image domain (already exist strong encoder like DINOv2), our method achieves performance comparable to that of REPA, this supports our confidence in the effectiveness of approach in text-to-video generation, where currently there is no well-pretrained strong encoder for open-domain videos (there is some video pretrained model, c.g., VideoMAE (Wang et al. 2023). which is still not strong enough for open-domain video encoder, compared to DINOv2 for image encode ). Second, in the video domain, numerous studies (Wiede- mer et al. 2025 Zhu et al. al.] (024b)) have shown that large-scale pretrained 12v models already capture rich and transferable representations suitable for various understanding tasks, further reinforcing the potential of SRA in video-related applications.
Moreover, as mentioned in FLUX.2 (Labs) 2025), when scaling to larger, more diverse data distri- butions beyond ImageNet and COCO, the approach that does not align with an external represen- tation encoders like DINOv2 shows superior generation quality compared to REPA. This suggests that SRA, which does not rely on external encoders, may have better scalability and generalization potential when applied to more complex, large-scale generation tasks, as the capacity of external en- coders may be fundamentally limited by their pre- vre-tranning data distribution and thus fail provide meaningful guidance when the target domain exceeds their representational coverage.
CONCLUSION
In this study, we show that diffusion transformers can provide representation guidance by themselves to boost generation performance with our proposed SRA, which aligns their latent representation in the earlier layer conditioned higher noise that in the later layer conditioned lower noise to progressively enhance the representation learning without external components. Considering the simplicity and effectiveness of SRA, we believe it will facilitate more future research to extend SRA in other scenarios.
7 ACKNOWLEDGMENT
This work was supported by Zhejiang Province Natural Science Foundation of China under Grant No. LQN25F030008 and the National Natural Science Foundation of China under Grant No. 62403429.
This manuscript referred to some color schemes and structural options from REPA (Yu et al. 2025) manuscript. We also want to thank Sihyun Yu, Sizhe Dang, and Zanyi Wang for the helpful discus- sions and suggestions during the progress of SRA project.
REFERENCES
Hervé Abdi and Lynne Williams. Principal component analysis. Wiley interdisciplinary reviews:
computational computationall statistics, 2(4):433 4599 2010.
3 mentioned by Waver Zhang al. 2025 video generation model uses REPA by employing Qwen2.5 VL (Bai et al. (2025) to extract video representations. is observed that the early training process benefits from the alignment and the alignment does not bring gain when the training converges. This might stem from the representations extracted from Qwen2.5-VT. VT. are not meaningful enough for video generation. Using internal representations of diffusion transformer potentially remedies this.
10

---

Published as a conference paper at ICLR 2026
Michael Samuel Albergo and Eric Vanden Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations, 2023.
Vanden-Eijnden.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2. 5-vl technical report. arXiv preprint ar arviv:2502. 139.2333....923. 2025.
Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 22669-22679 2023.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEÎT: BERT pre-training of image transform- ers. In International Conference on Learning Representations, 2022.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joc Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. OpenAl Blog, 2024.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-99244 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-super vision transformers. In Proceedings of the International Conference on Computer Vision, 2021.
Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang, Ze Wang. Zicheng Liu, Difan Zou, and Bhiksha Raj. Masked autoencoders are effective tokenizers for diffusion models. arXiv preprint 2502 2502.04500, 2025.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representa- fions, 2024a.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 19830-19843, 2023.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. International Journal of Computer Vision, 132:208 223, 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self supe-supereieed vision transformers. In Proceedings of the IEEF/CVF international conference computer vision, pp- 9640-9649, 2021.
Xinlei Chen, Zhuang Liu. Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401. 14404, 2024b.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR' '05), volume 1, pp. 886-893. leee, 2005.
11

---

Published as a conference paper at ICLR 2026
Javier Martín Daniel Verdú. Flux.1 I lite: Distilling flux1.dev for efficient text-to-image generation. https:/huugingfuuecoofrpppp 2024.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307. 08691,,8811111 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Ieee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image is worth 16x16 scale, 2021.
words: Transformers for image recognition at
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3 117:3-1,,20 2018
Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rab- bat, Nicolas Ballas, Yann LeCun, Amir Bar, and Saining Xie. Scaling language-fret free visual repre- sentation learning. arXiv preprint arXiv:2504.01017, 2025.
Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. arXiv preprint arXiv:2412.01 2024.
Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. Implicit style-content separation using b-lora. European Conference on Computer Vision, pp. 181-198. Springer, 2024.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 12996):179--1899 2021.
Baan-Bastien Grill, Florian Strub, Florent Altch'e. e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21283 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
12

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023.
J Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18(4): 203-210, 1986.
Dengyang Jiang. Haoyu Wang, Lei Zhang, Wei Wei, Guang Dai, Mengmeng Wang, Jingdong Wang, and Yanning Zhang. Low-biased general annotated dataset generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025.
Tero Karras, Miia Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 1166544--65166 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Man-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4015-4026, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for -ottte--mmae generation. Advances neural information processing systems, 36:36652-3666 -366663,,, 2023.
Felix Krause, Timy Phan, Vincent Tao Hu, and Bjöm Ommer. Tread: Token routing for efficient architecture-annosiic diffusion training. arXiv preprint arXiv:2501.04765, 2025.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. Advances in Neural Information Processing Systems, 37:122458- 12244833,1 2025.
Black Forest Labs. Flux. Jttt::://gihhub..com/bbddd
2024.
Black Forest Labs. FLUX.2: Frontier Visual Intelligence. https://bfl.a///n/ flux- 2 2025.
Chen-Ye Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Proceedings of the Eighteenth International Conference Artificial Intel- ligence and Statistics (AISTATS), pp. 562-570. PMIR, 2015.
Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Tor- ralba, and Sanja Fidler. Dreamteacher: Pretraining image backbones with deep generative mod- els. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 16698- 16708, 2023.
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised representation generation method. Advances in Neural Information Processing Systems, 37: 125441-125468, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
13

---

Published as a conference paper at ICLR 2026
Weifeng Lin, Xinyu Wei, Renrui Zhang, Le Zhuo, Shitian Zhao, Siyuan Huang, Junlin Xie, Yu Qiao, Peng Gao, and Hongsheng Li. Pixwizard: Versatile sagge-to-image visual assistant with open- language instructions. In International Conference Learning Representations, 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02744 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li. Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38- 55. Springer, 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao-Yusl Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11988 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04376 2023.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Fric Vanden-Fijnden and Sain- ing Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23-40. Springer, 2024.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 14297-14306, 2023.
Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion models beat gans on image classification. arXiv preprint 0tt.::23070087222 2023.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2/03.0384/ 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pp. 8162-8171. PMLR, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nico- las Ballas, Wojciech Galuba, Russ Howes, Po- Yao (Bernie) Huang, Shang- Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4205, 2023.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 10619-10629, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Interna- fional conference machine learning, pp. 8748 8763. PMLR, 2021.
14

---

Published as a conference paper at ICLR 2026
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint TrXiv:2404.13680 2024.
Pierre H. Richemond, Jean-Bastien Grill, Florent Altch'e Corentin Tallec, Florian Strub, Andrew Brock, Samuel L. Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. Byol works even without batch statistics. arXiv preprini arXiv:2010.10241 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the the IEEF/CVF confer-
ence on computer vision and pattern recognition, pp. 10684-10695, 2022b.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processigg systems, 29, 2016.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp. 32211 -32252. PMLR, 2023.
Quan Sun, Jinsheng Wang. Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-18b: Scaling clip 18 billion parameters. arXiv preprint arXiv:2402.04252. 2024.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Z-Image Team, Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Zhaohui Hou, Shijie Huang. Dengyang Jiang, Xin Jin, Liangehen Li, Zhen Li, Zhong- Yu Li, David Liu, Dongyang Liu, Junhan Shi, Qilong Wu, Fengyi Yu, Chi Zhang, Shifeng Zhang, and Shilin Zhou. Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint Xiviv:251 1.22699, 2025.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang. Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14549-14560, 2023.
Shuai Wang. Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. arXiv preprint arXiv:2504.0577 2025a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568-578, 2021.
Ziqiao Wang, Wangbo Zhao, Yuhao Zhou, Zekai Li, Zhiyuan Liang, Mingjia Shi, Xuanlei Zhao, Pengfei Zhou, Kaipeng Zhang, Zhangyang Wang, Kai Wang, and Yang You. Repa works un- til it doesn't stope-,, holistic alignment supercharges diffusion training. arXiv preprint 20X. 2025b.
WanTeam. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. 2025.
Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. Video models are zero-shot learners and reasoners. 2025.
arXiv preprint arXiv:2509.203:
Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders unified self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15802-15812, 2023.
15

---

Published as a conference paper at ICLR 2026
Xingyi Yang and Xinchao Wang. Diffusion model representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19938-18999, 2023.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang. Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with am expert transformer. arXiv preprint arXiv:2408.066 000.00006002...0 2024.
Jingfeng Yao and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501001423: 2025.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In International Conference Learning Representations, 2025.
Yifu Zhang, Hao Yang, Yugi Zhang, Yifei Hu, Fengda Zhu, Chuang Lin, Xiaofeng Mei, Yi Jiang, Bingyue Peng, and Zehuan Yuan. Waver: Wave your way to lifelike video generation. arXiv preprint arXiv:2508. 15761, 2025.
Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong- Yee Lee, Oliver Deussen, and Changsheng Xu. Prospect: Prompt spectrum for attribute-aware per- sonalization diffusion models. ACM Transactions on Graphics (TOG), 42(6):1-14, 2023.
Zhanjie Zhang, Quanwei Zhang, Huaizhong Lin, Wei Xing, Juncheng Mo, Shuaicheng Huang, Jin- heng Xie, Guangyuan Li, Junsheng Luan, Lei Zhao, Dalong Zhang, and Lixia Chen. Towards highly realistic artistic style transfer via stable diffusion with step-aware and layer-aware prompt. arXiv preprint 14X44:2404.11474 2024.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09335,, 2023.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In International Conference on Learning Repre- sentations, 2022.
Rui Zhu, Yingwei Pan, Yehao Li, Ting Yao, Zhenglong Sun, Tao Mei, and Chang Wen Chen. Sd- dit: Unleashing the power of sule-super discrimination in diffusion transformer. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8435-8445, 2024a.
Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, and Gang Hua. Exploring pre-traine pre-trained text-to-video diffusion models for referring video object segmentation. European Conference on Computer Vision, pp. 452-469. Springer, 2024b.
16

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This manuscript does not involve any topic and experiment related to ethics.
REPRODUCIBILITY STATEMENT
This manuscript provides comprehensive description of the implementation details and hyperpa- rameter settings. The associated code made publicly available for reproducing our results.
USE OF LLMS
In this paper, large language models (LL.Ms) collaborative editors polish the paper, elevating both the precision and readability of our manuscript.
APPENDIX
A INVESTIGATION OF REPRESENTATIONS IN DIT
We also perform a similar analysis with DiT like those have done in Figure leff) (PCA visual- ization) and Figure 2 right) (linear probing), the results showed in Figure In short, also observe that the representations in DiT basically lead a process from coarse to fine when increasing block layers and decreasing noise level SiT's.
Increasing Block Layers
layar3
layera
layert3
layerts
layer28
t=500
Decreasing Noise Level
Validation Acc. (%)
t=700
1-100
Layer Index
(a) PCA Visualization Result
(b) Linear Probing Result
Figure 8: We also empirically investigate the representations in diffusion transformers across differ- ent blocks and timesteps with the original DiT-XL/2 checkpoint trained for 7M iterations. Similar to SiT, the latent representations in DiT basically follow the bad to good process, as block layers increase and noise level decreased.
B DESCRIPTIONS FOR TWO TYPES OF BASELINE MODELS
In this paper, we use DiT SiT as our baseline models. We now provide an overview of two types of generative models that are variants of denoising autoencoders and used to learn the target distribution. Specifically, we discuss Denoising Diffusion Probabilistic Models (DDPMs) like DiT in Section B.I and Stochastic interpolant models like SiT in Section B.2
B. DENOISING DIFFUSION PROBABILISTIC MODELS (DIT)
Denoise-based models (Ho et al. 2020 Nichol & Dhariwal 2021) aim to model the target distribu- tion ( p(x) by learning gradual denoising process that transforms Gaussian distribution
17

---

Published as a conference paper at ICLR 2026
(O,,)) I) into p(x). Formally, diffusion models learn reverse process ( (p(x;-1 xx) corresponding to pre-defined forward process ( q(xx (x)) ), which incrementally adds Gaussian noise to the data starting from p(x) over a sequence of time steps ( E ...... T), with T' > 0 fixed.
For a given Xo ~ p(xo), the forward process q(1, xx-1) ) is defined as: q(x, x,,1 ((x;; VT Bixo I)T), where Br (0, 1) pre-defined small hyperparameters. DDPM (Ho et al. 2020 formalizes the reverse process p( p(x,-1(xx [xx) as:
p(x, xx, N/ (N(xx (1
- (x:
V1 0g (x(xx,t)): E (x,,,)).
of
Va:
where a₁ = = 3r. at = II l', and Ex( 1) parameterized by a neural network.
The model is trained using simple denoising autoencoder objective:
Csimple = ex |lee - Ec Ex(x:
(8)
For the covariance term Ex(x,, ). DDPM Ho al. 2020) demonstrated that setting it as ( (of1
) with B, = of is sufficient. Subsequently, Improved DDPM & Dhariwal, 2021 showed that performance can be enhanced by jointly learning (xx,t) along with (<((x,,)) in dimension-wise manner using the following objective:
Lv cxp(e log 3r + - v) log 3t),
where e is component per model output dimension, and 8t 1-at-13.
With sufficiently large T and an appropriate scheduling of Be, the distribution p(xr) approaches an isotropic Gaussian distribution. Thus, sampling is achieved by starting from random Gaussian noise and iteratively applying the reverse process (((xx-1(xx) ) to recover data sample Xo (Ho et al. 2020
B.2 STOCHASTIC INTERPOLANTS MODELS (SIT)
Unlike DDPMs, flow-based models (Liu et al. 2023 Lipman et al. 2022) describe continuous dime-dependent process involving data ( X,, ~ p(x) and Gaussian noise ( (0, ) over t [0,1]:
0, with and or being decreasing and increasing functions of t. respectively, The process is governed by a probability flow ordinary differential equation (PF ODE):
0x + d.C, Og = 1,
(10)
*; vxx, t),
(11)
where the distribution of the ODE time matches the marginal distribution (x)).
and can be approximated by a model (x(x,, t) trained to minimize the objective:
Qvllcity Ex..e.t [ t) å,x; - ó,c|13)
(12)
This also corresponds to reverse stochastic differential equation (SDE) given by:
dx, = v(xi, t)dt 2 sx t)dt Vwidw.
(13)
As shown in (Albergo & Vanden-Eijjdenn 2023 any functions a and a, that satisfy the following three conditions:
1. og + of 0, V1 [0,
2. 04 and ar are differentiable, VI E [0,
0i oo 0, as 01
lead to an unbiased interpolation process between Xo and e. Example choices include linear inter- polants (ar, = - f, a, t) pariance-preserving (VP) interpolants (a, cos 7 o2 sin( (t) (Ma et al. 2024).
An advantage of stochastic interpolants is that the diffusion coefficient w, can be independently selected during sampling with the reverse SDE, even after training. This capability simplifies the design space compared score-based diffusion models (Karras al. 2024..
18

---

Published as a conference paper at ICLR 2026
Table 5: Default hyperparameter setup. Unless other otherwise specified, these sets of hyperparameters for different models. our component-wise analysis experiment, settings are also kept the same except those point out in Table
SiT-B
SiT-L
STT-XL
DiT-B
DIT-L
DTT-II
Architecture
Input dim.
32x 32
32x 32xx
32x:2xx4
32xx3xx4
32xx32x4
32x4
Patch
2
2
Num. layers
12
24
28
12
24
28
Hiddon dim.
768
1024
1152
768
1024
1152
heads
SRA
Alignment blocks
16 [o, 0.2)
Alignment time interval
[0, 0.2) smooth-fs
[o, 0.2) smooth-F 0.999
[[0, 200) smooth-f f 0.999
[0, 2001 smooth-/ 0.999
[o, 200) smooth-Fi
Objective
smooth-f f 0.999
decay
0.999
0.999
Using projection head
Optimization
Batch size Optimizer
256 AdamW
256 Adam W 0.0001 (0.9. 0.999)
256 Adam
256 Adam
256 Adam 0.0001 (0.9. 0.999)
Adamn
Ir
0,0001 (0.9, 0.999)
0.0001 (0.9. 0.999)
0,0001 (0.9. 0.999)
0.0001 (0.9, 0.999)
[2)
Interpolants or Denolsing
1-
1 ppeddcction Fuller Mansyama 250
T
1000 moise prediction DDPM 250
1000 nosse nosse-prediction DDPM 250
1000 noise prede-prediction DDPM
Training objective
v prediction Murer-Marayama 250
predicction Fuler- manue-Manuyama
Sampler
Sampling steps
250 1.8 (if used)
250
Guidance Scale
C HYPERPARAMETER AND MORE IMPLEMENTATION DETAILS
More implementation details. We implement our models based on the original DiT and SiT im- plementation. To speed up training and save GPU memory, we use mixeddprecision (fp16) with gradient clipping and FusedAttention (Dao 2023) operation for attention computation. We also pre-compute compressed latent vectors from raw pixels via stable diffusion VAE (Rombach et al., 20226) and use these latent vectors. We do not apply any data augmentation, but we find this does not lead to big difference, as similarly observed in MaskDiT (Zheng et al. 2023) and REPA YYu al. 2025). We also use stabi lityyii/ ad-vae-ft- fe-ama for encoding images latent vectors and decoding latent vectors to images. For the projection head used for nonlinear transformation, we use two-layer MLP with SiLU activations (Elfwing al. 2018). We use smooth-l, objective for align- ment because we find it perform slightly better than 2 and its gradient transitions are smoother than f1 When using SiT-XL generate images with classifier-free guidance (Ho & Salimans 2022), the guidance interval introduced in the study (Kynkäänniemi et al. 2025) with the same setting used REPA (Yu et al. 2025) is applied, which has been demonstrated yield a slight performance improvement.
Sampler. For DIT, use the DDPM sampler and set the number of function evaluations (NFE) as 250 by default. For SiT, we use the SDE Euler- Maruyama sampler (for SDE with w, = o2) and set the NEF as 250 by default. The settings also align with those used in DiT and SiT papers.
Dataset. We use ImageNet (Deng et al. 2009), where each image is preprocessed to the resolu- tion of 256 512x512 (denoted as ImageNet x56 or ImageNet 512x512), and follow ADM (Dhariwal & Nichol 2021) for other data preprocessing protocols.
Linear probing. We follow the setup used REPA Yu et al.) 2025) and I-DAE (Chen et al. 2024b), and use the code-base of ConvNeXt (Liu et al. 2022) to conduct the experiment. Specifically, we use Adaptive.AggPooling and a batch normalization layer to process the output the latent feature by the model, then train a linear layer for 80 epochs. The batch size is set to 4096 with cosine decay learning rate scheduler, where the initial learning rate is set to 0.001.
Computing resources. We use 8 NVIDIA A100 80GB GPUs or 8 NVIDIA L40S 48GB GPUs for training largest model (XL); and use NVIDIA A 1100 80GB GPUs or 4 NVIDIA L40S 48GB GPUs for training smaller model (L, B), our training speed is about 2.12 step/s with global batch size of 256. When sampling, we use either NVIDIA A100 80GB GPUs or NVIDIA L40S 48GB GPUs or NVIDIA RTX 4090 24GB GPUs to obtain the samples for evaluation.
19

---

Published as a conference paper at ICLR 2026
D EVALUATION METRIC
In this section, we define the key metrics used to assess the performance of our model. Each metric summarized below:
Fréchet Inception Distance (FID) (Heusel et al., 2017)
Purpose: Measures the similarity between the feature distributions of real and generated im- ages.
Methodology: Uses the Inception-v3 network (Szegedy et al. 2016) to extract features. As- sumes both feature distributions are multivariate Gaussian and computes the Fréchet distance between them.
Interpretation: Lower FID scores indicate better similarity (and thus higher-quality generated images).
Spatial Fréchet Inception Distance (sFID) (aash et al. 2021
Purpose: Extends FID by incorporating spatial information to better capture the structural fidelity of generated images.
Methodology: Computes FID using intermediate spatial features (rather than global features) from the Inception-v3 network.
Inception Score (IS) (Salimans et al. 2016)
Purpose: Evaluates the quality and diversity of generated images. Methodology: Uses the Inception-v3 network to compute class probabilities (logits) for gener- ated images. Measures the KI--divergence between the marginal class distribution of generated images and the conditional class distribution of single image (after softmax normalization). Interpretation: Higher values indicate both high image quality (confident predictions) and diversity (uniform marginal distribution).
Precision and Recall for Distributions (Precision/Recall) (Kynkäänniemi et al. 2019)
Purpose: Evaluates the trade-off between sample quality (precision) and distribution coverage (recall).
Methodology: Precision: Fraction of generated images deemed realistic by classifier (relative real images). Recall: Fraction of the real image manifold covered by generated samples.
METHODS FOR COMPARISON
Next, we explain the key ideas behind the methods used for evaluation and comparison.
ADM (Dhariwal & Nichol 2021) enhances U-Net-based architectures for diffusion models and introduces classifier-guides sampling, a technique used to balance the quality-diversee tradeoff and improve overall performance.
VDM++ (Kingma & Gao 2023) improves diffusion model training efficiency through an adap-
tive noise scheduling mechanism that dynamically adjusts noise levels during optimization.
Simple Diffusion (Hoogeboom et al., (023) targets high-resolution synthesis by simplifying both network architectures and noise schedules through systematic exploration of lightweight design choices.
CDM (Ho al. 2022) achieves high-fidelity generation via a cascaded pipeline: training low- resolution diffusion models first, then progressively refining details through super-resolution diffusion stages.
LDM Rombaac et al. 2022a) accelerates training while maintaining generation quality op- erating in compressed latent spaces, modeling image distributions at reduced dimensionality compared to pixel space.
DiT (Peebles & Xie 2023) employs pure transformer backbone for diffusion models, incor- porating AdalN-zero modules as an important component modification against vanilla vision transformer (Dosovitskiy al. 202T) of its architecture.
SiT (Ma al. 2024) provides an extensive analysis of how the training efficiency of DiTs can
be improved by transitioning from discrete diffusion to continuous based modeling, which can be seen flow-hased version of DIT.
20

---

Published as a conference paper at ICLR 2026
SD-DiT (Zhu et. 2024a) leverages IBOT's (hoou et al.
2022
training paradigm that combines
DINO loss (Caron et al. 2021 and BEIT loss [Bao et al. 2022 for efficiently training diffusion transformers.
2023) proposes an asymmetric encoder-decodes scheme for efficient training of diffusion transformers, where they train the model with an auxiliary mask reconstruc- tion task similar to MAE (He et al. 2022).
MaskDiT (Zheng et al.
TREAD (Krause et al. 2025) introduces dynamic token routing strategy combined with the mask reconstruction task similar to MAE (He et al. 2022) and MaskDiT (Zheng et al. 2023 to accelerate the training of diffusion models.
REPA (Yu et al. 2025) achieves significant improvements in both training efficiency and gener- ation quality by aligning the latent feature of the diffusion model with that of large-scale data pre-trained representation model (e.g., DINOv2 (Oquab et al. 2023
MAETok changes the SD- DA- MAE-Tok, which is trained with auxiliary mask reconstruc- tion loss and aligns loss with three representation targets (HOG's (Dalal & Triggs) 20005), DI-
NOv2's (Oquab et al., 2023), and CLIP's (Radford et al. 2021h and obtains diffusion trans- former with better generation performance.
F TEACHER NETWORK UPDATING ROLE
Table 6: Different EMA update rule attempts, other settings are con- sistent with the default setting of Table in the main paper.
at of EMA
FID
1St
0.0
35.71
42.18
0.996
1.0
33.17
44.96
0.9999
29.10
50.20
In other generative learning studies, the EMA model is often used solely for evaluation. However. as we need it to provide guidance during training, we study different updating methods. Here, we investigate three different strategies to build the teacher. First, we find that using teacher copied from a student (ox 0) would impair the performance, which testifies to our argument in Section Next, we consider using the strategy widely used in self-supervise learning works (Caron et a 2021 Zhou et al. 2022) that updates the momentum coefficient a from 0.996 to during training. However, in framework, this does not work well. Finally, we use the of 0.9999 unchanged. which is commonly used in other generative learning works (Dhariwal & Nichol 2021 Peebles & Xie 2023) and find it is also suitable for framework. In analysis, first, student copy (a= 0.0) performs the worst : this is consistent with the observation in self-supervised learning: student copy leads to unstable training and cannot converge; there is diffusion loss in approach, so the training still converges. Second, using a= 0.9999 better than 0.996 > 1.0: In EMA, a larger a in- dicates a smoother update. We analyze that the reason for keeping such a very large value of a works well in SRA is that: in our method, the enhanced representation learning is ultimately for the service of generation. Thus, when the update of teacher model relatively intense, the alignment would be less stable, the model may focus more on alignment loss and overlook diffusion loss, thereby destroying some of the original generation behavior. On the contrary, maintaining a in a very large value (0. (9.9999) ensures the alignment target is extremely robust and slow to change, preventing the ali alignment loss from destabilizing the primary diffusion loss. Thus, we set the momentum coefficient 0.9999 our default.
G PRINCIPLE OF OUR HYPERPARAMETER
Although method requires some hyperparameter tuning, we believe that some settings model- independent, while some settings have selection principles. We now give the detailed explanations, and we hope these explanations can help to develop SRA to other new models more easily.
21

---

Published as a conference paper at ICLR 2026
Projection Head. In generative models, different layers often believed to have different generation responsibilities. We hypothesize that using the lightweight projection head instead of directly conducting the alignment can lead to relatively soft alignment, and could be the reason for avoiding disruption to the model's original generative behavior. We believe this principle and findings with the design of projection head can be easily transferred new models.
a in EMA decay. In EMA, larger indicates smoother update, We analyze that the reason for keeping such a very large value of a works well in SRA is that: in our method, the enhanced representation learning is ultimately for the service of generation. Thus, when the update of teacher model is relatively intense, the alignment would be less stable, the model may focus more alignment loss and overlook diffusion loss, thereby destroying some of the original generation behavior. On the contrary, maintaining a a very large value (0.9999) ensures the alignment target is extremely robust and slow to change, preventing the alignment loss from destabilizing the primary diffusion loss. This can also be indirectly verified by our following analysis of X. Given this analysis, we believe the choice of is will be well-matched most of the new model.
Block layers and time intervals. Principle of the selection of block layers: the block layer for the teacher is selected so that the corresponding representation has strong semantics (See Figure 2 (b)). The block layer for students is the first few layer as the first few layers are more about semantics learning, which is similar to REPA, and discussed in REPA. Principle of the time interval: The intuitive guidance could be that the target representation for the teacher timestep has better semantics, and the teacher timestep is not too far from the current timestep, i.e., balance between teacher representation semantics and the semantic distance between teacher (timestep sampled from the interval) and student (current timestep). These principles are also applicable to other diffusion transforms (e.g, MMDiT as shown Table 2
Coefficient X. The first principle we adopt À is maintain the diffusion loss (approximately 0.7) and the alignment loss (approximately 1.6) in the same scale. Our experimental results indicate that performance is slightly better when the alignment loss is relatively smaller, when these two points are satisfied, we consider this little performance variance is not suspicious. We also conduct additional experiments as follows, suggested that a smaller (e.g.. 0.02) diminishes the regularization effect of the alignment loss, while a larger À (e.g., 2) makes the alignment loss to dominate training instead of assist generation, which can harm sample quality (this can also indirectly verifies why EMA performs well when a large fixed ois set.). Hence, once we know the scale of diffusion loss and the above principles we give, the selection will be very easy to new model.
H TRAINING SPEED AND GPU MEMORY USAGE AGAINST BASELINES
Table 7: Memory useage, per-epoch training time comparison. "Param" " counts only the additional parameters required by the auxiliary forward pass. These results are tested with total batch size of 256 on a single machine with 8 A100 GPUs.
Method
Mem (GB)
Time (h)
Param
SiT-B
12.85
0.096
SiT-B + SRA
13.23
0.115
87 M
SiT-L
25.68
0.282
SIA
26.65
0,330
305 M
SiT-XL
29.08
0,394
S
30.24
0.476
481 M
DIT-B
13.96
0,173
SRA
14.78
0,209
87 M
DiT-L
27.85
0,503
SRT
29.26
0.558
305 M
DiT-XL
30.68
0.719
DIT-XL+ SRA
32.31
0.804
481 M
As our approach needs one more forward pass on the diffuusion backbone to compute the target representation. We now provide specific comparison GPU memory usage and training time with
22

---

Published as a conference paper at ICLR 2026
baselines. Noting that the extra pass is only for forward, and no gradient computation is needed, so we can use half preciisee for fast forward. In addition, it is not needed to pass the whole network, and only partial pass is needed: 8/12 blocks in SiT-B, 20/28 blocks in SiT-XL. So it can be that the additional cost are not very large, and the results in Figure F and Table also indicate that method can improve the performance of the model within the same training time.
I DETAILED SETUP OF ABLATION STUDY
We now give the detailed experimental setup of Figure 7 in our ablation study. The accuracy rate of the horizontal axis in the figure obtained by using the linear probing results of the original SiT-XL/2 checkpoint training for 7M iterations, while the vertical axis is the FID evaluation result of training 400K iterations with SRA without classifier- free guidance (CFG). Since we do introduce any representation component and use the representation supervision signal only in the generative training process, we consider this experiment to validate the effectiveness of our approach.
J ABLATION RESULTS OF DIT
We also perform a similar analysis with DiT like those have done in Figure 7e, the results are showed Figure g In short, we also observe that the generative capability DiT with SRA indeed strongly correlated with the representation guidance as observed in SiT with SRA.
Teacher Layer for Alignment
FID-50K
12
18
16
20
30
40
50
60
Validation Acc. (%)
Figure 9: We also investigate the correlation between generation performance and the representation guidance of DiT SRA. A similar tight coupling can also be seen.
K MORE DISCUSSION ON RELATED WORK
We now provide a detailed literature review of other related work.
EMA model as teacher. Unlike traditional knowledge distillation (Gou et al. (022)) that uses well pre-trained model as the teacher, using EMA model (Hunter 1986) the teacher can be seen self- distillation because the weight of the teacher model is obtained by the weighted moving average of the student. This method often needs a feasible pretext task succeed. For example. MoCo (He et al. 2020: Chen et al. 202T) sets the teacher's 's output as momentum queue and uses contrastive learning to guide the student model; DINO (Caron al. 2021 Oquab et al. 2023) feeds two views of images to the teacher, and the trainable student then forces the student's output distribution to be close to that of the teacher. Our work also shares some similarities, where we set aligning the student model's latent feature in the earlier layer conditioned higher noise with that in the later layer conditioned on lower noise of the teacher as our pretext task for training.
Diffusion transformers. Currently, the diffusion model is progressively transitioning from U-Net- based architecture to a Transformer-bast based one, owing to the superior scalability of the latter. At the beginning, U-ViT (Bao al. 2023) shows transformer-based backbones with skip connections can
23

---

Published as a conference paper at ICLR 2026
be effective backbone for training diffusion models. Then, DiT (Peebles & Xie 2023) shows skip connections are not even necessary components, and a pure transformer architecture can be a scal- able architecture for training denoise-based models. Based on DiT, SiT (Ma et al. 2024) shows the model can be further improved with continuous stochastic interpolants (Albergo & Vanden-Eijnden 2023). Moreover, Stable diffusion 3 (Esser al., 2024), Z--Image (Team al. 20.5) and FL.UX Labs), (024) show pure transformers can be scaled up for challenging text-to-image generation, and this characteristic is also verified by Sora (Brooks al. 2024), CogvideoX (Yang et al. (024)
and Wan (Wan' leam 2025) in text-to-vider field. Our work focuses on improving the training of DiT (and SiT) architecture based on our proposed self-represeentation alignment technique.
Exploring representation capacity of diffusion models. With the success of diffusion models to generate detailed images, many works have attempted to test whether discriminative semantic in- formation can be found in diffusion models. GID (Mukhopadhyay et al., 2023) and DDAE (Xiang) jt al. 2023) first observe that the intermediate representations of diffusion models have discrimina- live properties. Driving from this finding, I-DAE (Chen al.) 2024b) deconstructs diffusion models to be a self-supervised Learner. Moreover, Repfusion (Yang & Wang 2023). DiffusionDet (Chen) et al., 2023), and DreamTeacher (Li et al. 2023) diffusion models to perform various down- stream tasks (e.g., semantic segmentation and object detection). Our work also tries to explore the representation capacity of the diffusion model, but focus on leveraging the representations in diffusion transformers to enhance their generation capacity.
Applying representation guidance to other generation tasks. In addition pre-training of the class-conditional diffusion model. applying representation guidance can benefit other gener- ation tasks. For example, lbGen (Jiang et al., (225)) utilizes text features from CLIP (Radford) iet al. (02T) as a low-biased reference to regulate diffusion model for low-biased dataset synther- ies. RCG (Li et al. (024) focuses on unconditional generation, which first use the features from a self-supervised image encoder to serve as the 'label' for image generation by a second generator. Diff-AF (Preechakul et al., 2022) uses learnable encoder for discovering the high-level semantics and diffusion model for modeling stochastic; this dual-encoding improves the realism of the gen- crated images on attribute manipulation and image interpolation tasks. Different from these works, our study focuses the class-conditions diffusion transformer's 's pre-training and exploiting repre- sentation guidance in itself and its own training paradigm. But we also believe our plug-and-play method can be easily applied to other tasks with benefits.
Knowledge distillation for diffusion model. Knowledge distillation (Gou et al. 2021 is also widely used in diffusion models. Its purposes can roughly be divided into two types: improving gen- eration performance and accelerating inference sampling. To achieve the first goal, A more powerful pretrained model is often used to guide the diffusion model during training (Daniel Verdū 2024 Yu itt al. 2025, Fang et al.) 2024). For example, TinyFusion (Fang et al.) 2024) combines progressive optimizations for U-Net backbones, enabling deployment on edge devices. REPA ( Yu et al. 2025) distills the knowledge from large-scale, pretrained represen- tation foundation model (Oquab et al.) [2023] Radford et al.) (02T)) and finds that this distillation can improve both training efficiency and generation quality. Meanwhile, to achieve the second goal, the student and teacher models will be initialized by pre-trained models (here can be one model or two different models), then the training target is to make the prediction of the student model with fewer steps conform to that of the teacher model with multiple steps (Luo et al. 2023 Ren et al., 2024 Meng et al. 2023 Song et al. 2023). thereby achieving the result of accelerated sampling and generation speed. For example, CM (Song et al. 2023) constrains the student's 's outputs of adjacent points sampling path to be consistent with teacher's. s. LCM (Luo et al.) 2023) leverages this idea in the latent space. Our study also have some similarities, while we do not need a pretrained model and improve the generation performance of diffusion transformers by applying proposed SRA.
distillation with architecture-speciff
L MORE QUALITATIVE RESULTS (CFG 4.0)
24

---

Published as a conference paper at ICLR 2026
Figure 10: Uncurated samples of loggerhead turtle (class label: 33).
Figure 11: Uncurated samples of sulphur-crested cockatoo (class label: 89).
25

---

Published as a conference paper at ICLR 2026
Figure 12: Uncurated samples of golden retriever (class label: 207).
Figure 13: Uncurated samples of white fox (class label: 279).
26

---

Published as a conference paper at ICLR 2026
Figure 14: Uncurated samples of tiger (class label: 292).
Figure 15: Uncurated samples of red panda (class label: 387).
27

---

Published as a conference paper at ICLR 2026
Figure 16: Uncurated samples acoustic guitar (class label: 402).
MIeIONNE alj
Figure 17: Uncurated samples of balloon (class label: 417).
28

---

Published as a conference paper at ICLR 2026
GONNOY JUN VEHEONATI BRIEOVADI 709 VPC
Figure 18: Uncurated samples of baseball (class label: 429).
ORO GREEE SHOR
Figure 19: Uncurated samples of fire truck (class label: 555).
29

---

Published as a conference paper at ICLR 2026
Figure 20: Uncurated samples of laptop (class label: 620).
Figure 21: Uncurated samples of ice eream (class label: 928).
30

---

Published as a conference paper at ICLR 2026
Figure 22: Uncurated samples cheeseburger (class label: 933).
Figure 23: Uncurated samples of cliff drop-off (class label: 972).
31

---

Published as a conference paper at ICLR 2026
Figure 24: Uncurated samples of coral reef (class label: 973).
Figure 25: Uncurated samples of lakeside (class label: 975).
32


Reviews:
[{"review_id": "SfUdRo6Czn", "reviewer": "Reviewer_YcnN", "summary": "This paper proposes SRA, a model designed to enhance the generation performance of diffusion transformer models. The key idea of SRA originates from an observation that earlier timesteps and shallow layers of a diffusion transformer capture coarse latent features, while later timesteps and deeper layers perform progressive refinement. Instead of relying on a pre-trained representation encoder (a recent approach for improving diffusion transformer performance) the authors employ an EMA-updated teacher network derived from the training process itself. Through this self-updating training mechanism, the model refines its representations over time. The proposed method is evaluated across various models and achieves performance comparable to those that use large pre-trained encoders.", "questions": "- Given sufficient GPU capacity, it seems feasible to combine this method with REPA for joint training. Have the authors explored this possibility?\n\n- There seems to be a noticeable performance difference depending on whether the projection head is used or not. Could the authors elaborate on why the projection head contributes so significantly to performance, and what specific role it plays in the model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper presents an intuitive and elegant idea: achieving refinement within the model itself during training, without depending on large-scale pre-trained encoders. This demonstrates the potential for applying the method to settings where such encoders are unavailable.\n\n- The proposed method can be easily integrated into existing diffusion training pipelines, and the authors confirm its applicability to both flow-based and diffusion-based models.\n\n- The linear probing results of the learned latent representations provide yet convincing evidence that the latent quality has been improved.", "weaknesses": "- The method requires model-specific hyperparameter tuning. For each new model, appropriate block layers need to be identified, and while the time interval and lambda parameters are less sensitive, they still require some search.\n\n- Since the training process involves an EMA teacher model, the approach is likely to increase GPU memory consumption. It would be helpful if the authors could quantify the additional GPU usage and training time per epoch compared to the baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y2v5D8MB2F", "reviewer": "Reviewer_NzJf", "summary": "This paper introduces Self-Representation Alignment (SRA), leveraging internal representation guidance during diffusion transformer training. The core idea is to align latent representations from earlier layers (conditioned on higher noise) with those from later layers (conditioned on lower noise), without relying on any external models. To achieve this, the authors employ an EMA teacher of the same network and apply a lightweight projection head with a patch-wise distance loss. Experimental results show that SRA attains performance comparable to methods using external alignment.", "questions": "- Figure 6 shows that SRA keeps improving while external alignment methods saturate early. It gives a good signal, but it’s still unclear whether SRA eventually converges or continues improving over longer training. Could the authors provide longer training curves or convergence analysis?\n\n- I’m also curious whether SRA can naturally extend to text-to-image generation. Would the same self-alignment strategy work there, or are there additional challenges?\n\n- (Minor) Typo in line 373 - “Tabel” → “Table”.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation and flow of ideas in this paper are clear. It starts by analyzing the internal representations of diffusion transformers and then proposes a way to effectively use them to speed up training and improve performance. The paper also shows self-alignment can serve as a good alternative in the absence of external encoders.\n\n- The paper is well-written and easy to follow. The figures are clear and helpful, and the authors include an analysis of training cost, which is one of the considerations with having another network forward pass. \n\n- Experiments across different resolutions (e.g., 512×512) and models (DiTs and SiTs) show that the method scales well and generalizes effectively.", "weaknesses": "- The EMA teacher is one of the key components, but the paper gives little discussion about it. From Table 5, the results are quite sensitive to how $\\alpha$ is updated - some settings (e.g., 0.0 and 0.996 → 1.0) even show worse FID than the baseline. A deeper analysis beyond saying it is “commonly used” in other works would make this part more convincing.\n\n- The paper does not include experiments or discussions demonstrating SRA’s effectiveness in cases where strong external encoders are unavailable - even though this is the main motivation. It would be helpful to include or discuss results in another setting (e.g., video generation) to make the motivation more convincing.\n\n- The authors claim that “the teacher’s constantly improving capacity allows better representation guidance as training proceeds,” but there is no evidence supporting this. Showing how the teacher’s representation quality (e.g., via linear probing) improves over time would support the claim. Otherwise, the statement should be toned down.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LpMmejQVgW", "reviewer": "Reviewer_1H2k", "summary": "This paper aims to accelerate the training of diffusion transformers by learning meaningful internal representations while training the model. In particular, this paper proposes SRA, which align the diffusion model with a better representation without external guidance. The key idea is to introduce a teacher network (EMA), and then use different noise and layers of the teacher network for alignment to self-distill a better representation. Experiments demonstrate that SRA accelerates generative training without an external model.", "questions": "Please answer the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is overall easy to follow, and the method is simple yet effective.\n\n2. Using self-distillation from the teacher network to boost the training of diffusion models is an intriguing idea.\n\n2. SRA shows faster training with respect to time compared to vanilla Diffusion Transformers, showing the effectiveness of the method.", "weaknesses": "1. The authors claim that later blocks with small noise have fine-grained features compared to the first blocks with large noise. However, it does not indicate that later blocks with small noise have better representations to be aligned. For instance, an extremely larger layer even has more detailed features, but it is not beneficial (e.g., 4 -> 12 performs worse than naive SiT in Table 1). Therefore, I think it needs additional / more analysis to justify the effectiveness of the proposed method.\n\n2. The choice of block layers is quite sensitive. In particular, it sometimes performs worse than vanilla SiT. It (1) makes the practitioners to difficult to use this technique to improve the training speed of Diffusion Transformers, and (2) makes it difficult to extend the suggested framework to different domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VL8YCRFU7a", "reviewer": "Reviewer_MZU4", "summary": "This paper introduces Self-Representation Alignment (SRA), which improves representation learning quality in diffusion/flow transformers (DiTs/SiTs) without relying on pre-trained encoders or auxiliary representation learning tasks. SRA achieves accelerated training speed and improved generation performance in practice. The key insight is that diffusion/flow transformers learn a hierarchy of representations which progress from coarse features (in earlier layers and with higher noise levels) to fine features (in later layers and with lower noise levels). SRA leverages this internal structure in a student-teacher framework, where the teacher is an EMA version of the student network. The diffusion/flow training objective is then augmented with an alignment loss that encourages the projection of the student's coarse representations to match the teacher's more refined representations. Large-scale experiments are performed to evaluate SRA's empirical performance.", "questions": "1. Could the authors comment on the generalizability of the chosen hyperparameters to other settings/tasks and provide some scientific explanations and/or general rules for selecting these hyperparameters?\n2. Could the authors discuss the limitations of the proposed method in the paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The observation of the hierarchy of the internal representations in diffusion/flow transformers is interesting, which is a natural motivation for the proposed method.\n2. The key idea of using the EMA model's representation in later layer and with lower noise level for self-guidance is novel. This has the advantage of eliminating the need of pretrained encoder or auxiliary representation learning tasks for representation alignment.\n3. Thorough empirical evaluation and ablation studies are performed to demonstrate the proposed method's competitive empirical performance.\n4. The paper is easy to understand with a clear description of the preliminaries and the proposed method, as well as a well-organized section for experiment result.", "weaknesses": "1. The primary weakness is a lack of a principled foundation, as many core design choices feel arbitrary and are determined through empirical trial-and-error rather than guided by a clear theoretical or analytical framework. Below are some examples:\n\n    (a) The projection head is a bit counterintuitive but is shown to be crucial during training with no clear justification. Intuitively, it feels more natural to perform alignment without projection or with projection for both teacher and student outputs.\n\n    (b) It is unclear why a fixed EMA decay of a particular value works well and whether this is still the optimal value for new tasks/architectures.\n\n    (c) The choices of the layers and time intervals for self-guidance feel ad-hoc.\n\n    (d) It is suspicious that there is only a little performance variance for different values of the alignment loss's weight $\\lambda$, given that it controls the trade-off between the diffusion loss and alignment loss.\n\n2. The proposed method requires tuning many hyperparameters and its performance seems highly sensitive to these specific hyperparameter choices, for which the paper offers no clear scientific explanation or general rule. As a result, it is unclear whether these hyperparameter will generalize to new architectures/tasks/settings.\n\n3. Although the authors propose several hypotheses to explain why the chosen hyperparameters make sense, it still does not provide sufficient scientific intuition for why the proposed alignment is useful as the hypotheses are unverified.\n\n4. Another major weakness is that there is no discussion of the limitations of the proposed method in the paper.\n\n5. There are some minor typos in the paper:\n\n    (a) \"Dose\" -> \"Does\" in Line 227,\n\n    (b) \"Tabel 2\" and \"Tabel 3\" -> \"Table 2\" and \"Table 3\" in Line 373,\n\n    (c) \"Defualt\" -> \"Default\" in Line 918,\n\n    (d) \"useage\" -> \"usage\" in Line 1068.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)