LayerSync: Self-aligning Intermediate Layers
Authors: Yasaman Haghighi
Bastien Van Delft
Mariam Hassan
Alexandre Alahi
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion models
Self distillation
Primary area: generative models

Abstract: We propose LayerSync, a domain-agnostic approach for improving the generation quality and the training efficiency of diffusion models. Prior studies have highlighted the connection between the quality of generation and the representations learned by diffusion models, showing that external guidance on model intermediate representations accelerates training. We reconceptualize this paradigm by regularizing diffusion models with their own intermediate representations. Building on the observation that representation quality varies across diffusion model layers, we show that the most semantically rich representations can act as an intrinsic guidance for weaker ones, reducing the need for external supervision. Our approach, LayerSync, is a self-sufficient, plug-and-play regularizer term with no overhead on diffusion model training and generalizes beyond the visual domain to other modalities. LayerSync requires no pretrained models nor additional data. We extensively evaluate the method on image generation and demonstrate its applicability to other domains such as audio, video, and motion generation. We show that it consistently improves the generation quality and the training efficiency. For example, we speed up the training of flow-based transformer by over 8.75$\times$ on ImageNet dataset and improve the generation quality by 23.6\%.

Published as a conference paper at ICLR 2026
LAYERSYNC: SELF-ALIGNIN INTERMEDIATE LAYERS
Yasaman Haghighi" Bastien van Delft" Mariam Hassan
Alexandre Alahi
Ecole Polytechnique Fédérale de Lausanne (EPFL) {yasaman. waaamma.hgghihh.... haihighi, bashi,,astieeee bastien......
vandel mariam. hassan, a alexandree.aaaatie www.aaaaaaaaal...oooeeea
ABSTRACT
We propose LayerSync, a domain-agnosttic approach for improving the generation quality and the training efficiency of diffusion models. Prior studies have high- lighted the connection between the quality of generation and the representations learned by diffusion models, showing that external guidance model interme- diate representations accelerates training, We reconceptualize this paradigm by regularizing diffusion models with their own intermediate representations. Build- ing on the observation that representation quality varies across diffusion model layers, we show that the most semantically rich representations can act as an in- trinsic guidance for weaker ones, reducing the need for external supervision. Our approach, LayerSyne,
self-sufficient, plug-and-play regularization term with overhead on diffusion model training and generalizes beyond the visual domain to other modalities. LayerSyne requires no pretrained models or additional data. We extensively evaluate the method on image generation and demonstrate its ap- plicability to other domains such as audio, video, and motion generation. We show that consistently improves the generation quality and the training efficiency. For example, we speed up the training of flow-based transformers by over 8.75x x on ImageNet dataset and improve the generation quality by 23.6%. The code is avail- able at https: //tgthtb...////////// com http://aaaaaeeelaaaaeeeeeee git
Image
Audio
Motion
90
ImageNet
8.99
0.25
0.52
SIT-XL2
7.7%
SIT-XL/2 LayerSync
0.48
50
23.6%
20.4%
StopGradd
Transformer Block
FID 50K
0.20
Transformer Block
6.87
Al Alignment
FIB
FAD
Transformer Black
10
Transformer Black
822
8.75x
50k
100K
200K
20h
4.50
0.12
0.26
Training Iterations
Baseline
Baseline + LayerSyne
(a)
(b)
Figure 1: LayerSync improves training efficiency and generation quality via internal repre- sentation alignment. (a) LayerSyne improves representations by aligning shallow layers with se- mantically rich deep layers. (b) LayerSyne achieves over 8. 8.75x training acceleration the Ima- geNet 256x256. (c) LayerSyne consistently improves generation quality across multiple modalities: 23.6% on FID for images (ImageNet 256x256), 24% on FAD for audio (MTG-Jamendo), and 7.7% on human-motion FID (HumanMI.3)).
"Equal contribution

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Denoising generative models, such as diffusion (Ho et al. 2020 Song et al. 2020 Song & Ermon 2019) and flow matching models (Lipman et al., (022), have demonstrated remarkable success in modeling complex data distributions, achieving state-of-the-arl performance across a range of gen- erative tasks. However, this success comes at significant computation cost. Thus, new promising line of research has emerged to improve the training efficiency of these models by improving the
models' intermediate representations (Yu et al.] 2024 [Wang et al.) [2025] [Wang & He (2025). It has been shown that the quality of a diffusion model's intermediate representations is intrinsically linked its generative performance. As result, explicitly guiding these representations accelerates
training and improves generation quality (Yu al. 2024).
Building on this insight, the most dominant approach (Yu et al.. 2024 Wang al. 2025) has been to leverage powerful external guidance from large pre-trained models, by aligning the internal features of diffusion model with those of high-capacity vision models like DINOv2 (Oquab et al. 2023) or vision- vision-language language models (VLMs) like DW (Wang et al. 2024). These methods demonstrate that access to strong semantic features can accelerate training by an order of magnitude. While effective, this paradigm comes with several limitations. It introduces a dependency on massive ex- ternal models that are themselves costly to train, require large amounts of data, and may not be available for domains beyond natural images. Additionally, this reliance rameters introduces extra overhead into the training pipeline. For instance, in the case of Wang et al.) (225), training is indeed faster in terms of iterations but involves calling 9-billion-parametee parameter VLM at each step. These limitations motivate the development of more self-contained and generalizable alternatives.
external data and pa-
A recent step in this direction is the Dispersive Loss (Wang & He 2025), a self-contained regularizer that encourages internal representations to spread out in the feature space, analogous to the repulsive force in contrastive learning (Oord et al. (2018). Although this approach demonstrates the potential for internal regularization, substantial performance gap remains compared to methods that leverage external representations. In this paper, we propose a self-contained method with more directed learning signal reduce this gap.
Our work is motivated by two key observations: First, while diffusion models learn powerful rep- resentations, their quality is highly heterogeneous across the model's depth. As demonstrated by previous works (Mukhopadhyay et al. 2024, Kim et al.) 2025 Stracke et al.) 2025). certain inter- mediate layers capture more semantically rich and useful information than others. Second, when models incorporate knowledge through external guidance using DINOv2 for instance, regularizing early layers seemed more effective than regularizing the deeper ones (Yu et al. 2024 Wang et al., 2025). Based on these two observations, clear opportunity presents itsell:: can the model's 's own strongest layers act as an intrinsic guidance to improve its weaker ones through allf-aligmmmn?
To this end, we propose LayerSyne, simple yet powerful regularization framework that aligns model's own intermediate layers. LayerSync is a parameter-free, plug-and-play solution that op- erates without any external models or data, making it truly self-contained method. LayerSyne introduces negligible computational overhead, yet its effectiveness is substantial. Our experiments show that LayerSyne consistently outperforms prior self-contained methods across all tested con- figurations. For image generation, LayerSync accelerates training on ImageNet 256x256 (Deng al., 2009) by more than 8.75x. This leads to a new state-of-the-art in purely self-supervised image generation on ImageNet, demonstrating the strength of our self- self-alignment objective and substan- tially narrowing the gap between self-contained approaches and those relying on external guidance. Furthermore, due to its self-contained nature, LayerSync seamlessly generalizes to other modalitie lidaliiies. Our experiments show that for audio generation, LayerSync leads to 21% improvement in FAD- 10K on MTG-Jamend (Bogdanov et al.) 2019). to 7.7% improvement in FID for human motion genera- tion on HLmanML33D and 54.7% in FVD for video generation on CLEVRER (Yi et al. 2019) (see Appendix A). To the best of our knowledge, it is the first time that self-contained method proves to accelerate diffusion model training seamlessly across different domains.
Additionally, an analysis of the internal features confirms that LayerSync strengthens the model's s representations, leading to a 32.4% improvement in classification and a 63.3% improvement in semantic segmentation accuracy.
2

---

Published as a conference paper at ICLR 2026
Our main contributions are follows:
We introduce LayerSync, a minimalist, parameter-free, and self-contained regularization method that leverages a diffusion model's own layers as an intrinsic guidance via self- alignment.
We demonstrate the domain-agnostie versatility of LayerSync by successfully applying it to image, audio, human motion, and video generation.
We show that our self-supervised method not only accelcrates training but also improves the representations across the model's layers.
RELATED WORK
Representation learning with diffusion models. Denoising Generative Models, including both diffusion (Ho et al., 2020; Song et al. 2020 Song & Ermon] 2019) and flow matching models (Lipman al. 2022). trained as multi-level denoising autoencoders (Vincent et al. 2008), naturally give nise to discriminative representations. A line of work has specifically evaluated the quality of these representations, showing that diffusion features can be effectively used across a variety of tasks (Mukhopadhyay et al. [2024 Kim et al., 2025 Sttackk et al. 2025 and, in some cases, achieve performance comparable to self self-supervised representation learning methods (Stracke et al.] (025). However, the quality of the representations varies across model layers, with the final layers, just before the model begins decoding, consistently containing more semantically rich features (Kim al. 2025 Xiang al.) 2023), regardless of whether the architecture is U-Net (Ronneberger et al.
2015) or a Transformer (Vaswani et al. (017). Our work is directly built upon those insights. We demonstrate that the semantically rich representations in the intermediate layers can be leveraged as guiddance signal to enhance the quality of earlier-layer representations.
Representation regularization for improving diffusion models. It has been shown that represen- tation quality is closely linked to generative performance (Yu et al.) 2024). One line of work im- proves generation by regularizing model representations through alignment with strong pretrained networks. For example, Yu et al. (2024) aligns diffusion features with self-supervised features from DINOv2 (Oquab et al., 2023). y while Wang et al. (2025) demonstrates that leveraging vi- sion-language models (VLMs) (Wang et al. (2024) yield further improvements. Although such approaches accelerate training and enhance generation quality, they remain constrained by the need for high-quality external representations, which not readily available in non-visual domains. Additionally, they introduce computational overhead, as pretrained models must be inferred at each training step. Another group of work adopts self-supervised strategies that rely on EMA (Expo- nential Moving Average) (Tarvainen & Valpola 2017) models to guide the representations. Zheng al.] (2023) integrates a generative diffusiin process with an auxiliary mask reconstruction task. Zhu et al. (2024); ; Diang et al.) (2025) align representations between teacher and student encoders joint embedding space. While being self-contained, such methods increase computational cost by requiring an additional forward pass through the EMA model at each training step. A recent work (Wang& He| 2025) proposes dispersing representations in the feature space, analogous to the repulsive force in contrastive learning. This approach introduces Similarly, we present a self-contained, overhead-fit free solution; however, we leverage semantically richer internal representations to guide and improve the learning of weaker ones.
additional training overhead.
METHOD
3.1 PRELIMINARIES
We adopt the generalized perspective of stochastic interpolants (Ma et al.) 2024), which provides unifying framework for both flow-based and diffusion-based models. Here is a brief overview; we refer Appendix Section N for more details.
Stochastic interpolants are generative models that learn to reverse a process that gradually converts a data sample xo into simple noise This is achieved by defining path between them:
X 04430 + O1C,
3

---

Published as a conference paper at ICLR 2026
where de and de are functions of time controlling the mix of data and noise at time t. To generate new data, the model must learn to travel backward along this path, from noise to data. The direction and speed at any point x, and time is given by velocity field. The true velocity is the time derivative of the path: 6x + ozC.
Since this true velocity is unknown during generation, neural network 0, t) is trained predict it. The model learns by minimizing the velocity loss, which measures the squared difference between the predicted velocity and the ground-truth velocity:
velociiy (0) == Exo.e,t [Ilv((x,,)) () (á,x0 + o
(1)
Once trained, the model can generate new data by starting with a random noise sample and following the velocity field it has learned.
3.2 DIFFUSION MODELS INTERMEDIATE REPRESENTATIONS
Representation Hierarchy. We investigate the representations learned by pre-trained SiT model (Ma et al. 2024) on ImageNet (Deng et al. 2009). Through linear probing on downstream tasks (classification, segmentation) and Centered Kernel Alignment (CKA) with DINOv2 (Oquab al., 2023) features, we observe a clear hierarchy in representation quality across layers. As shown Figure 4 deeper layers exhibit superior discriminative capabilities, consistent with established principles of hierarchical feature learning in deep networks (LeCun et al., (201)). This pattern of increasing semantic richness culminating in peak before the final decoding blocks is known characteristic of diffusion model representations (Mukhopadhyay et al. 2024).
Corrolation Heatmap
Mack
Internal Block Structure of Diffusion Transformers. Beyond the hierarchy of feature quality, we observe that the internal structure of Diffusion Transformers at convergence follows specific correlation patterns as shown in Figure | An al. 2025). These blocks are highly correlated and naturally segregate int three functional groups: (1) an initial group focusing on local features, (2) middle group of highly correlated blocks capturing global features, and (3) final group acting as a decoder project back the latent space.
and discussed (Raghu et al. 2021
Bick
Figure 2: Correlation be- tween the blocks of SiT-XL/2 at Convergence.
3.3 LAYERSYNG.
We propose self-contained regularization approach, named LayerSyne, designed to improve the generation quality and training dynamics of diffusion models. The core principle behind LayerSync is intra-model self-alignment, where the model trained to guide itself. Building on the hierarchy of representation quality across layers, we use the context-rich deep layers as "intrinsic guidance" to provide a direct signal to the earlier "weak" " layers, thereby enhancing the model's entire feature from within. The extra alignment between the model's intermediate layers aligns with the block structure of the diffusion transformer at convergence.
LayerSync achieves this alignment by maximizing the similarity between the feature representations of designated strong and weak blocks. The similarity is computed for each patch in the represen- tation and then averaged over the whole patch sequence for each image. Let fe be the network transformer and let f designate the network up to the k-th layer. Let x, be the input marginal dis- tribution time t, with
Uniform( (0, 1) and Xt, we define the loss for LayerSyne between layer l and & with & < k as follows:
N
1
sim f (x) stopgrad (1ø (c)
LaayerSyn (0) -Ex,,
(2)
N
4

---

Published as a conference paper at ICLR 2026
Training Iteration
100k
200k
400k
100k
200k
400k
100k
200k
400k
SIT XL/2
LayerSyne Dispersive
Figure 3: LayerSync improves generation quality without relying on external representation. We compare the images generated by SiT-XL/2 when regularized with dispersive and LayerSync. All the models trained for 400K iterations, share the same noise, sampler, and number of sam- pling steps, and none of them use classifier-free guidance.
where n is the patch index and sim(:.) pre-defined similarity function. We experimented with different similarity functions and opted for cosine similarity in all following experiments. This regularization term is integrated into the primary training objective as weighted sum:
== Lvelocity AayyyeSyue
where the hyperparameter À > 0 balances the standard denoising task with our internal representa- tion alignment. Algorithm T in the Appendix summarizes proposed approach.
While LayerSyne internally regularizes the network by aligning representations from its early layers with those from its own deeper, semantically richer layers, beyond the immediate benefit of improv- ing shallow layers, we bypothesize that this method may induce a virtuous cycle in the recursive refinement process. The enhancement of early-layer features is expected facilitate the learning of more robust deep-layer representations, which subsequently offer a more refined target for internal alignment, potentially leading to a cascading improvement of the model's feature space.
3.4 LAYER SELECTION
key design consideration for LayerSync is the selection of the layers to align. As detailed in Section 3.2,, the layers of Diffusion Transformers naturally converge toward structure three groups of blocks focusing on local features, global features, and decoding. While our experiments indicate that LayerSync yields performance gains across wide range of configurations, we observe that respecting this structure leads to optimal acceleration (see Section D
Based on these observations. we propose a selection strategy guided by three principles. First, draw- ing on established findings (Xiang et al., 2023) regarding the functional specialization of layers in generative transformers, we exclude the hnnal 20% of blocks from being chosen as the reference layer, as those are primarily specialized for low-level decoding tasks, making them suboptimal as guidance targets. Second, based on the findings from (Raghu et al., 2021), we exclude the very first layers, prior work finds that early blocks focusing on local features improve performance and generalization (An et al. 2025). Third, to ensure a meaningful semantic gap between the represen- tations, enforce minimum distance between the aligned and reference layers (e.g.. 8 blocks for SiT-XL and SiT-L and for SiT-B). We validate this heuristic and the robustness of the method to different layer selection through experiments summarized in Tables 6 11 and 12
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
We conduct comprehensive set of experiments to validate the effectiveness of LayerSync. Our evaluation structured along three axes:
We first study extensively the performance and training efficiency of LayerSyne in large- scale class-conditional image generation (Section 4.1)
We then assess the domain-agnostic capabilities of our method by applying it to generative tasks audio (Section 4.2). human motion (Section 4.3). and video (Appendix Section
Finally, we perform an in-depth analysis to quantify the impact of LayerSync on the quality and structure of the learned internal representations (Section 4.4
4.1 IMAGE GENERATION
Implementation details. We strictly follow the setup in SiT (Ma et al. 2024). Specifically, we use ImageNet (Deng et al. 2209) and follow ADM (Dhariwal & Nichol 2021) for data preprocessing. The processed image will have the resolution of 256 256 and is then encoded into compressed vector 2 E 333x3224 using the Stable Diffusion VAE (Rombach et al. 2022). For model configu- rations, we use the B/2, L/2, and XL/2 architectures by Ma al. (2024), which process inputs with patch size of 2. More details about the architectures and the number of parameters are provided the Section Q Additional experimental details, including hyperparameter settings and computing resources, provided in Section D
Evaluation metrics. We report Fréchet inception distance (FID; Heusel et al. (017)). Inception Score (Salimans et al. 2016), Precision, and Recall (Kynkäänniemi et al. 2019) using 50,000
samples. We provide details of each metric in Section
Baselines. We compare our results with Dispersive (Wang & He 2025), the only self-contained, zero-cost method that accelerates training. For the sake of completeness, we also compare our method with several recent diffusion-based generation methods. For pixel-based approaches compare with ADM (Dhariwal & Nichol| (201)). VDM++ (Kingma & Gao 2023), Simple diffusion (Hoogeboom al., 2023). CDM (Ho et al.] 2022). For latent-based approaches we compare with LDM [Rombach et al. (2022), U-ViT-H/2 Bao et al. (2023). DiffiT (Hatamizadeh et al. 2024). HDDvv-XLL22 (Gao et 2023), MaskDiT (Zheng all (023), SD-DiT (Zhu et al.) 2024). DiT (Peebles & Xie 2023), and SIT eM al. 2024). We also compare our approach with REPA (Zhang atal|[2225 and REED (Wang et al. 2025), which rely on external representations. Additionally, we
compare two autoregressive methods, VAR (Tian et al. (024) and D-JEPA (Chen et al. 2024).. For both methods, we selected the models with a similar number of parameters to SiT-XL. Finally, we
compare results with the EMA-based method SRA (Jiang al. 2025). A detailed description of each bascline is provided in Section PP
Results. As shown in Table I our method consistently improves diffusion transformer training and more effective than Wang & He 2025). Our method results in 8.75 acceleration compared to SiT-XL baseline, reaching FID of 8.29 after only 160 epochs, and in 4.7x acceleration compared to the baseline trained with Dispersive Loss. In Table 2 we compare LayerSync with recent state- of-the-art diffusion model approaches. In particular, on SiT-XL//2, we reach FID 1.89 after 800 epochs, setting a new state-of-the-art in pure self-supervised generation, decreasing the gap with methods like Yu et al. (2024) that rely on external representations. We also qualitatively compare the progression of generation results in Figure where we use the same initial noise across different models. Additional comparison metrics with SRA are provided in Table 15
4.2 AUDIO GENERATION
Implementation details. We use the MTG-Jamendo dataset (Bogdanov et al. 2019), a large-scale collection containing over 55,000 full-length songs. For training, we process the audio by creating random 10-second samples, which are sampled at a standard rate of 44. 1 kHz. We condition using the metadata provided with the dataset by conditioning the generation labels associated with each sample. Our audio generation model is an adaptation of the Scalable Interpolant Transformer (SIT-XI.) (a et al. 2024), consistent with the 28-le architecture used in
the genre and instrument

---

Published as a conference paper at ICLR 2026
Table 1: FID comparisons of class- conditional generation on ImageNet 256x256. No classifier-free guidance (CFG; Ho & Salimans| (2022)) is used. The sampler used is the ODE-based Heun method, except for the last section, which uses the SDE-based Euler method following Ma et al. (2024)
Table 2: System-level comparison on ImageNet 256x256 with CFG. Models with additional
CFG scheduling Kynkäänniemi et al. (2024) are marked with an asterisk (*).
Model
Epochs
FID
1ST
Pre.
Rec.1
Pixe diffusion
ADM-U
400
3.94
186.7
0.82
0.52
VDM++
560
2.40
225.3
Simple diffusion
800
2.77
211.8
Model
#Params
Fjochs
FID
CDM
2160
4,88
158.7
130M
36.19
Dispersive LayerSyne
(10.35)
Autoregressive Iouge Generation
130M
30.00 (17.1%)
VAR-d20
350
2.57
302.6
0.83
0.56
SiT-L/2
458M
80
21.41
D-JEPA-L
480
1.58
303.1
0.80
0.61
Dispersive
16.68 (22.
Latent diffusion, U-Net
LayerSyne
(30.7%)
LDM-4
200
3.60
247.7
0.87
0.48
SIT-X1.2
675M
17.97
Dispersive
675M
80
15.95 (1 (1133)
Latent diffusion, Transformer + U-Net hybrid
LayerSyne
675M
11.24 (37.5%)
U-VIT-H/2
240
2.29
263.9
0.82
0.57
DifiT
1.73
276.5
0.80
0.62
4 Dispersive
675M
200
10.64 (12.666)
MDTv2-XL/2*
1080
1.58
314.7
0.79
0,65
LayerSyne
675M
200
8.28 (32.0%6
Latenu diffusion, Tronsformer
SiT-XL/2
Dispersive
675M
10.11
(12.998)
MaskDiT
1600
2.28
276.6
0.80
0.61
LayerSyne
675M
400
6.94 (31.45))
SD-DIT
480
3.23
SiT-XL/2
675M
800
8.99
DFT-XL/2
1400
2.27
278.2
0.83
0.57
Dispersive
675M
800
8.08 (10.19)
SRA*
800
1.58
311.4
0.80
0.63
LayerSyne
SiT-XL/2
1400
2.06
270.3
0.59
SiT-XL/2 (w/ SDE)
675M
1400
8.3
REPA
800
1.80
284.0
0.81
0.61
Dispersive
675M
800
7.71 (7.1%)
+ REPA
800
1.42
Dispersive
675M
1200
7.43 (10.5))
200
1.80
267.5
0.81
0.61
LayerSyne
8.29
+. Dīspersive
1120
1,97
+ LayerSyne
+ LayerSync
675M
675M
200
400
6.51 (21.6%)
7.78 (6.3%)
LayerSyne
800
1.89
265.34
0.81
0.60
LayerSyne
675M
6.32 (23.9%)
+ LayerSync*
800
1.49
285.2
0.80
0.63
Table 3: Quantitative results for audio generation on the MTG-Jamendo dataset. We report Fréchet Audio Distance (FAD) using CLAP embeddings. Our method significantly outperforms the baseline with no change in parameter count.
Method
#Params
Epoch
FAD (CLAP) +
SiT-XL (baseline)
756M
465
0.333
+ LayerSyne (Ours)
756M
465
0.263 (21.0%)
SiT-XI. (haseline)
756M
655
0.251
+ LayerSyne (Ours)
756M
650
0.199 (20.7%)
our vision experiments. The model is configured to operate on patchified latent representations with patch size of one. These latents are obtained from the pre-trained Variational Autoencoder (VAE) of the Stable Audio Open model (Evans et al.) 2025), which provides a compact representation of the raw audio waveforms. The model was trained on 64 GH200 GPUs with a global batch size of 1024. In our experiment, we align layer 8 with 21 using cosine similarity between the patch-wise representations of these two layers.
Evaluation metrics. To quantitatively assess the quality and realism of the generated audio, we re- port the Fréchet Audio Distance (Kil)) et al. 2018) with 10,000 samples using the widely- used CLAP embeddings (Zhao et al. 2023).
Results. LayerSync improves the final FAD-IOK by 20.7% at 650 epochs as seen in Table The model trained with LayerSync reaches the final performance of the baseline model around epoch 500 150 epochs earlier. The convergence speed is therefore improved by 23%.
4.3 TEXT-CONDITIONAL HUMAN MOTION GENERATION
To demonstrate that LayerSync can be applied in domains with limited datasets and compact archi- tectures, we consider the task of human motion generation. Given sentence that describes motion
7

---

Published as a conference paper at ICLR 2026
Table 4: Quantitative results for text-conditional human motion generation task on HL33 dataset. LayerSync improves both FID and R- R-Prrecision.
Method
FID 4
R-Precision
MDM
600K
0.5206
0,7202
LayerSyne (Ours)
600K
0.4801 (7.7%)
0.7454 (3.45))
Validation Acc. (%)
CKA (vs, DINOV2-91
miou (%%
SIT-XU/2 SIT-XL/2 LayerSync
SIT-XL/2
SIFXU/2 SIT-KL/2 LayerSync
SITXL/2 LayerSync
Layer Index
Layer Index
(a) Classification Tiny Ima- geNet dataset.
(b) Semantic Segmentation
(c) Alignment to DINOv2-g.
PASCAL VOC dataset.
Figure 4: Assessing the quality of intermediate features shows that LayerSync improves average validation accuracy across layers (shown with dashed lines in the figures) for both classification and segmentation, and enhances alignment with DINOv2. In this experiment, layer 8 is aligned with layer 16.
sequence of actions, the task is generate the corresponding human motion. Each motion quence consists of a series of human poses, where each pose is represented by 22 joints defined as 3D points in space.
Implementation details. We follow the exact setup as MDM Tevet al. 2022) using a transformer with 8 layers. We use HumanML3D dataset (Guo et al. (022). We train the model with and without LayerSync for 600K iterations. We align layer 3 with 6. More details are provided in Section M
Evaluation metrics. We report FID and R- R-Precision (top 3) (Kynkäänneemi et al. 2019) detailed in Section O
Results. The results summarized in Table 4 show that LayerSyne improves FID by 7.7% and R- Precision by 3.4%, confirming its effectiveness even with small architectures and limited datasets.
4.4 REPRESENTATION LEARNING
To evaluate the effect of LayerSync, we analyze the model's intermediate representations. We com- pare SiT-XL/2 model trained with LayerSyne for 160 epochs against a baseline SiT-XL/2 trained for 1400 epochs as they both have similar FID. This ensures that both models exhibit compara- ble generative performance, allowing us isolate the impact of our regularization on the learned representations, independently of the final generation quality.
We consider linear probing for classification on Tiny ImageNet dataset (Deng et al., 2009) linear probing for segmentation on the PASCAL VOC dataset (Fieeringhmm 2010)) and Centered Kernel Alignment (CKA) (Kornblith et al. 2019) with DINOv2 embeddings (Oquab et al. 2023 to measure the distance between the model representations. More implementation details provided in Section
Our empirical results, summarized in Figure lead to two interesting observations. First, Layer- Sync induces a more homogeneous distribution of high-quality features across the network's 's layers, leading to 32.4 % improvement in the average validation accuracy for classification, 63.3 % in aver- age mIOU, and 88.2 improvement in average alignment with DINOv2. Secondly, we observe not only shift in the block with the best performance in downstream tasks but also improvement in
8

---

Published as a conference paper at ICLR 2026
Table Quantitative evaluation of SiT-XL/2 trained on ImageNet 256 256 (Deng et al. 2009) comparing REPA and REPA + LayerSync. Both models are trained for 200k iterations and batch size of 1024. REPA applied on Layer 10 and LayerSync aligns layer 8 with 16. The results show that LayerSync can be combined with approaches that rely external representations, such as REPA, to further accelerate training.
Method
FID
sFID +
Inception Score
REPA
7.88
4.81
126.39
REPA + LayerSyne
7.01
4.78
129.85
the best performing block. While an increase in mean performance is an intuitive consequence of regularizing weaker layers toward nigh-peeforming one, the emergence of new peak that signifi- cantly surpasses the baseline's 's maximum is a non-trivial finding.
We conclude that the representational benefits of LayerSyne are not merely a byproduct of accel- erated convergence. Even when the baseline model is afforded more than 8x larger training bud- get match generative performance, its internal representations remain significantly inferior. We therefore hypothesize that LayerSync acts as powerful structural regularizer that fundamentally alters the model's optimization trajectory. By imposing an internal semantic constraint, it guides the network to discover a more efficient and globally coherent feature hierarchy, one that remains inaccessible to the unconstrained model.
Furthermore, it is noteworthy that the quality of representations learned with LayerSync approaches that models trained with powerful external guidance. Our peak classification accuracy, for exam- ple, is comparable the results reported by (Yu et al., 2024). We interpret this as evidence for initial hypothesis: LayerSync establishes a virtuous cycle that progressively refines the entire fea- ture hierarchy. Notably, the layer of peak performance often shifts to align with the chosen reference layer. A more detailed analysis is provided in Section G
4.5 LAVERSYNC COMBINED WITH EXTERNAL REPRESENTATIONS
In this experiment, we investigate whether LayerSyne can combined with external representa- tion guidance to further accelerate training. The results summarized in Table 5 show that the two approaches are synergistic; combining LayerSyne with REPA (Yu et al.) 2024) yields better per- formance than REPA alone. This suggests that the internal structural alignment of LayerSync (see Section and the external semantic injection of REPA operate as complementary axes of improve- be used as standalone method when external models unavailable or
ment. Thus, LayerSync too costly, or combined with external representations to maximize performance. Additional details provided in Section H
5 ABLATION STUDY
Layer Selection. To empirically validate the robustness of our layer selection strategy, we con- ducted an experiment with randomized layer pairings. For both SiT-XL and SiT-L architectures, detailed in Tables IT and T2 We observe that regardless of the selected layer, we always have gain. However, the gain is optimal following the layer selection strategy discussed in Section 3.4 The results in Table demonstrate remarkable consistency. The low standard deviation in the FID (0.8 for SiT-XL) confirms that the specific choice of layers is not very sensitive hyperparame- ter. This robustness validates claim that LayerSyne is practical, plug-and-play method that provides significant performance gains without necessitating an expensive search for optimal layer combinations.
Effect of X. We examine the effect of the regularization coefficient on SIT B/2 in Table 7 and observe that our method robust wide range of values for À and consistently improves FID.

---

Published as a conference paper at ICLR 2026
Table Performance of LayerSyne with randomized layer pairings. Results show the mean FID, IS and standard deviation (in parentheses) over independent runs, confirming the robustness of Layer- Sync to layer selection.
Method
Model
FID & (STD)
IS (STD)
Baseline
SiT-XL
17.98
50.41
LayerSyne- Ours
SiT-XL
12.24 (0.8)
71.47 (2.6)
Table Ablation study for X. We train SiT-B/2 for 400K iterations while aligning block with 8.
We observe that our approach is robust for wide range of X. The baseline SiT-B/2 has FID 19.19.
0.1
0.2
0.3
0.5
0.7
Average (Std)
FIDL
36.19
31.63
31.02
31.6
31.17
31.36
31.356 (0.27)
1St
44.9
46.12
44.56
45.65
45.2
45.286 (0.61)
DISCUSSION
LayerSync is a regularization framework that promotes feature consistency actoss model's depth. aligns intermediate layers by encouraging those with weaker representations to become more similar but not identical those with richer features. This self-alignment propagates strong semantic information, which we found accelerates training and improves generative performance.
This similarity between layers raises natural question: does LayerSyne make layers redundant, potentially allowing for model pruning?
Our experiments (Appendix B) show that while models trained with LayerSync more robust to layer removal than their baseline counterparts, performance still degrades significantly. This in- dicates that despite the improved alignment, each layer retains a unique function essential to the model's 's capacity. Consequently, naively pruning a trained model did not prove superior to simply training a smaller architecture from scratch. Similar findings have been reported for LLMs: interme- diate transformer blocks often exhibit high correlation. While removing these blocks has little effect on easier tasks such as question answering (QA), degrades performance on more challenging tasks (Gromov et al. (024).. Therefore, high inter-block correlation does not necessarily imply that the blocks are redundant.
We also wish to emphasize that the long-term effects of regularization might also demand further study, Although we did not observe the performance degradation seen in other methods with external guidance reported in (Wang et al. 2025), future work could explore scheduling the LayerSync loss to preemptively address any potential long-term downsides.
Finally, the alignment loss function itself presents key area for future research. We selected cosine similarity due to its strong empirical performance on images and its effective transfer to audio. However, developing novel alignment losses specifically engineered for different data domains, such as the hierarchical nature of text or the temporal patterns in time-series data, is an interesting and potentially impactful research direction.
7 CONCLUSION
In this paper, we introduced LayerSyne, a simple yet novel self-supervised regularization method for improving diffusion transformers. We demonstrated that model's later-layer representations can effectively guide its earlier layers, enhancing feature quality and accelerating training at no additional cost. As general framework, LayerSyne requires external guidance and is readily applicable to different data domains.
This work opens several avenues for future research in training efficiency, representation learning, and self-supervised learning. We believe the core principle of LayerSync is broadly applicable and encourage exploring its potential in other generative architectures beyond diffusion models.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We are committed to open-sourcing the full codebase and all experiment configurations used in this paper upon acceptance, support transparency and facilitate future research in this area.
ETHICS STATEMENT
This work investigates self-supervised method for accelerating the training of diffusion models across multiple modalities. All experiments were conducted using publicly available datasets. No personal sensitive data used, and we do not anticipate any direct ethical risks associated with this research.
10 ACKNOWLEDGMENT
This research was supported by the Swiss National Science Foundation (SNSF) under Grant No. 10003100, and by Innosuisse the Swiss Innovation Agency (Ref. 127.265 IP-ICT). Computational resources were provided as part of the Swiss Al Initiative by grant from the Swiss National Supercomputing Centre (CSCS) under project ID al Alps.
REFERENCES
Jie An, De Wang, Pengsheng Guo, Jiebo Luo, and Alex Schwing. On inductive biases that en- able generalization in diffusion transformers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 22669-22679, 2023.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mnte-ammedd dataset for automatic music tagging. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308. 2017.
Dengsheng Chen, Jie Hu, Xiaoming Wei, and Enhua Wu. Denoising with joint-embedding predic- tive architecture. arXiv preprint arXiv:2410.0375:. 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdty2: Masked diffusion trans- former is a strong image synthesizer. arXiv preprint arXiv:2303.1 14389, 2023.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. /7887, 2024.
11

---

Published as a conference paper at ICLR 2026
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng, Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM international conference multimedia, pp. 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 5152-5161, 2022.
Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. European Conference on Computer Vision, pp. 37-55. Springer, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023.
Dengyang Jiang, Mengmeng Wang, Liuzhuozheng Li, Lei Zhang, Haoyu Wang, Wei Wei, Guang Dai, Yanning Zhang, and Jingdong Wang. No other representation component is needed: Diffusion transformers can provide representation guidance by themselves. arXiv preprint arXiv:2505.02831, 2025.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fry 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv: 1..Xiv:8811... 882.08466, 2018.
Dahye Kim, Xavier Thomas, and Deepti Ghadiyaram. Revelio: Interpreting and leveraging semantic information in diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4659-4669, 2025.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances Neural Information Processing Systems, 36:65484-6551 2023.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
Solomon Kullback and Richard A Leibler. information and sufficiency. The annals of mathe- matical staristies, 22(1):79-88 86, 1951.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Tuomas Kynkäääniemi,, Mika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. Advances in Neural Information Processing Systems, 37:1 17711-248-11223 122483, 2024.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553):436 444, 2015.
12

---

Published as a conference paper at ICLR 2026
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open- large video generation model. arXiv preprint arX1::2412.00131, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: skinned multi-person linear model. ACM Transactions on Graphics (TOG), 34(6):1-16, 2015.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Eijnden, and Sain- ing Xie. Sit: Exploring flow and biffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23-40. Springer, 2024.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp.
2019.
Soumik Mukhopadhyay, Matthew Gwilliam, Yosuke Yamaguchi, Vatsal Agarwal, Namitha Pad- manabhan, Archana Swaminathan, Tianyi Zhou, Jun Ohya, and Abhinav Shrivastava. Do text- free diffusion models learn discriminative visual representations? In European Conference on Computer Vision, pp. 253 272. Springer, 2024.
Charlie Nash, Jacob Menick. Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint 1xXi:22003.08411 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini arXiv: 187.048887. 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX1vv2304.07/93, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 1211-22 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, and Björn Ommer. Cleandift: Diffusion features without noise. In Proceedings of the Computer Vision and Pattern Recognition
Conference, pp. 117-127, 2025.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2818-2826, 2016.
13

---

Published as a conference paper at ICLR 2026
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight- con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
Keyu Tian, Yi Jiang Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-84865, 2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoinn Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.2033 14, 2025.
Chenyu Wang, Cai Zhou, Sharut Gupta, Zongyu Lin, Stefanie Jegelka, Stephen Bates, and Tommi Jaakkola. Learning diffusion models with flexible representation guidance. arXiv preprint arXiv:2507.08990, 2025.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai. Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing Aision-languee model's perception of the world at any resolution. arXiv preprint 2tt.:2499.1219/ 2024
Runqian Wang and Kaiming He. Diffuse and disperse: Image generation with representation regu- larization. arXiv preprint arXiv:2506.0900 2025.
Weilai Xiang, Hongyu Yang. Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-super learners. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15802-15812, 2023.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint 19000:191.001422, 2019.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint 14XV::241000644000 2024.
Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang. Fanqing Meng, Xiangpeng Wan, Junchi Yan, and Yu Cheng. Videorepa: Learning physics for video generation through relational alignment with foundation models. arXiv preprint trXiv:2505.2366 2025.
Tianqi Zhao, Ming Kong, Tian Liang, Qiang Zhu, Kun Kuang, and Fei Wu. Clap: Contrastive language-audio pre-training model for multi-modal sentiment analysis. In Proceedings of the 2023 ACM international conference multimedia retrieval, pp. 622-626. 2023.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint wrXiv:2306.09305 2023.
Rui Zhu, Yingwei Pan, Yehao Li, Ting Yao, Zhenglong Sun, Tao Mei, and Chang Wen Chen. Sd-dit: Unleashing the power of self supervised discrimination in diffusion transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8435-8445, 2024.
14

---

Published as a conference paper at ICLR 2026
A VIDEO GENERATION
We study the effectiveness of LayerSyne for both video generation and fine-tuning existing video diffusion models on new datasets.
Training from Scratch on CLEVRER. We train SiT-XL model with 3D patchification of size (1,2,2) on the CLEVRER dataset Yi et al. 2019). Due to the high computational cost of video training from scratch, we limit the run to 24k steps on 16 GPUs, using this as a proof of concept to demonstrate the effectiveness of LayerSync.
Fine-tuning on MixKit dataset. We use MixKit dataset (Lin al. 2024) fine-tune Wan2. 1.3B foundation model (Wan et al. 2025). Each video has 81 frames. We align layer 8 with layer 16 with a regularizer weight of 0.02. The model fine-tuned via LoRA adapters for 1,100 steps using 4 GPUs.
Baselines. We refer to fine-tuning without any extra guidance as vanilla.
Evaluation metrics. We rely on Fréchet Video Distance (FVD; Unterthiner et al. (018)) for eval- uation. We generate 5,000 videos of 16 frames for SiT-XL and 1,000 videos 81 frames for the fine-tuned 1 model.
Results. As shown in Table LayerSyne consistently outperforms vanilla training across both fine-tuning and training from-scratch setups, achieving the lowest FVD scores in every scenario.
Table FVD scores (L) for video generation. We observe that LayerSync consistently improves FVD for both fine-tuning and training from scratch.
Wan2.1 (MixKit)
SiT-XL (CLEVRER)
Vanilla LayerSync (Ours)
321.84 304.68
265.50
120.13
B DROPPING BLOCKS
To investigate the effect of dropping blocks, train SiT XL/2 and SiT XL/2 with LayerSync (aligning layers 7 and 16) for 120k iterations. We then drop four blocks between the aligned layers. Quantitative results presented in Table and qualitative examples are shown in Figure indicating that the model trained with LayerSync is more robust to block removal.
We also experimented dropping blocks outside the aligned layers. As summarized in Table 9 and Figure 6) this leads to more significant degradation in sample quality, suggesting that the drop of blocks outside the synced range has more detrimental effect. Although LayerSyne improves robustness to dropped blocks, doing so still results in an increase in FID.
Table 9: Comparison of FID, sFID, Inception Score (IS), Precision, and Recall when dropping specific blocks from the model. The model trained with LayerSync is more robust to block removal.
Skipped blocks
FID +
sFID 4
IS f
Precision
Recall f
SiT XL/2 SiT XL/2 + LayerSync
37.03
5,49
35.41
0.53
0.61
25.72
5.05
48.49
0.61
0.59
SIT XI./2
[9,11,13,15]
211.66
93.92
4.02
0.01
0.10
SIT XI./2 + LayerSyne
[9,11,13.15]
55.07
7.85
23.04
0.39
0.63
SIT XL/2 + LayerSyne
19,11,13,15,21]
86.11
18.30
16.79
0.29
0.46
SIT XL/2 + LayerSyne
[1.9,11,13.15]]
92.84
22.28
15.38
0.26
0.44
15

---

Published as a conference paper at ICLR 2026
SIT XL/2
SIT SIT LayerSync
Figure 5: Qualitative comparison of generated samples from SiT XL/2 and SiT XL/2 with Lay- erSync when layers 7 and 16 are synced. After dropping blocks [9, 11, 13, 15], we observe that LayerSyne helps preserve visual quality despite block removal.
No Drop
Drop Blocks 9, 11, 13, 15
Drop Blocks 11, 13, 15,
Drop Blocks 1, 9, 11, 13, 15, 21
Figure 6: Qualitative results when dropping blocks from SiT XL/2 with LayerSync, where layers and 16 are synced. Dropping blocks outside the synced layers leads to more noticeable degradation in sample quality.
16

---

Published as a conference paper at ICLR 2026
C ALIGNMENT CONSIDERING THE TIMESTEP
We apply alignment on SiTXL/2 for the last 75 %, 50 and 25 of timesteps. The model is trained on ImageNet 256 x 256 for 80 epochs. The results summarized in TOllws that the best performance is achieved when alignment applied on all the timesteps, which confirms that improving the weak representations is beneficial regardless of the timestep.
Table 10: Applying LayerSync on specific timesteps shows that alignment is beneficial for all the timesteps.
Timestep
FID |
25%
18.28
50%
18.77
75%
17.68
100%
16.03
D ABLATION ON BLOCK SELECTION
Ablation study on block selection for SiT-XL/2 and SiT-L/2 summarized in Tables 11 and 12 shows that LayerSync consistently improves the generation quality, but the gain is suboptimal when distance between the blocks is low when aligning with the decoder blocks (very last layers). All the models are trained for 100K iterations with 16 GPUs and a batch size of 1024.
Table 11: Ablation study block selection for SIT-XL/2. The results show that LayerSync consistently improves the generation quality, but the gain is suboptimal when the distance between the blocks low or when aligning with the decoder blocks (very last layers).
Model
IS
FID 4
sFID
Vanilla SiT-XL/2
50.408
26.534
5.035
6 6
775.221
15.386
4.672
LayerSyne 7 +- 17
75.727
15,433
4.694
LayerSyne 7 +- 15
74.987
15.740
4.611
LayerSyne
17
73.791
16.058
4.695
LayerSyne 8 +- 18
73.549
16.078
4.589
LayerSyne +-
72.821
16.276
4.638
LayerSync 8 +- 20
71.477
16.568
4.662
LayerSyne t- 20
71.016
16.661
4.695
LayerSyne 8 +- 19
71.094
16.680
4.658
LayerSyne f- 14
72.750
16.697
4.695
LayerSyne 10 f- 18
70.209
16.999
4.638
LayerSyne 9 +- 19
69.831
17.091
4.701
LayerSyne 11
21
68.952
17.096
4.673
LayerSync 7 21
69.438
17.126
4.685
LayerSyne 11 19
68.464
17.615
4.628
LayerSync 9 +- 22
68.127
17.823
4.717
LayerSyne 8 21
67.633
18.032
4.705
LayerSync 6 +-
57.956
22.630
4.903
LayerSyne +-
56.552
23.307
4.892
LayerSyne 6 +-
55.717
23,924
4.954
LayerSyne < 23
61.851
19.966
4.769
LayerSyne 12 23
59,692
20.711
4.737
LayerSync 15 + 23
55.235
23.030
4.783
17

---

Published as a conference paper at ICLR 2026
Table 12: Ablation study on block selection for SiT-L/2. The results show that LayerSync con- sistently improves the generation quality, but the gain is suboptimal when the distance between the blocks low when aligning with the decoder blocks (very last layers).
Model
1S
FID
sFID
LayerSync 6 <- 15
64.638
19.165
4.827
LayerSyne 8 16
62.794
19.566
4.856
LayerSyne 7 15
62.854
19.663
4.841
LayerSyne 6 = 14
62.616
19.945
4.791
LayerSyne 5 13
62.158
20.048
4.778
LayerSyne 7 < 16
61.4
20.055
4.865
LayerSync 7 18
60.533
20.324
4.820
LayerSync 6 -- 16
61.304
20.444
4.794
LayerSync 5 12
60.807
20.492
4.775
LayerSync 5 t 11
61.104
20.696
4.821
LayerSync 15
60.362
20.734
4.824
LayerSync 5 <- 17
61.178
20.771
4.878
LayerSyne f= 8
50.473
26,511
5.084
LayerSync 5 7
49.948
27.004
5.171
LayerSync 5 t 6
47.19
28.741
5.21
LayerSync 11 +- 20
43.253
30.57
5.172
LayerSync 17 t- 20
43.563
31.056
5.308
LayerSyne 16 +- 20
43.242
31.24
5.316
LayerSync 13 +- 20
42.535
31.407
5.182
LayerSyne 14 -- 20
41.626
32.35
5,282
LayerSync 15 + 20
41.381
32.622
5,403
E LAYERSYNC VS. INCREASING LEARNING RATE
To show that LayerSync impact is not simply due to an increase in the gradient magnitude, designed two sets of experiments below. All the models are trained with GPUs, batch size 1024 for 100k iterations (80 epochs). We then report the gradient norms and the FIDs of different configurations. We consistently observed that the gradient norm with LayerSync is actually smaller than that of the baseline, indicating that our method is not simply equivalent to using larger learning rate.
(1) Global learning rate increase.
Starting from the default learning rate of 1 x 10-4, we trained models with higher learning rates 2 x 10-1 and 5 10-1, For learning rates above 5 x 10-4, 2, the model diverges. While increas- ing the learning rate can partially accelerate training, the resulting FID scores remain worse than those obtained with LayerSync with the default learning rate 1 x 10 as shown in Table 13 The visualization of gradient norm is provided Figure 7a
Table 13: Effect of Global Learning Rate. Simply increasing the global learning rate improves FID slightly but does match the performance gains of LayerSyne.
Method
Ir.
FID
SiT-XL/2
1 10
26.53
SiT-XL/2
2 x 10
24.95
LayerSync
1 10
16.03
(2) Higher learning rate on early blocks only.
We then increased the learning rate only for the first blocks and compared this to model trained
with LayerSync aligning layers 8-16 at a global learning rate of x 10 10-4, Again, for learning
rates above 1 10 training diverges. As summarized in Table 14 for 2 x 10 4, x 10 10m4, and x 10-3 3, the FID improvements do not match those obtained with LayerSyne. The visualization of the gradient norms provided in Figure 7b
18

---

Published as a conference paper at ICLR 2026
Table 14: Effect of Early-Layer Learning Rate. Increasing the learning rate specifically on early
layers (first 8 blocks) is beneficial but still underperforms compared to LayerSyne.
Method
Ir. Early Layers
General lr.
FID
SiT-XL/2
1 1 10
1 10
26.53
SiT-X1//2
2 10-1
1 10
19.24
SiT-X1/22
5 x 10-1
1 10
24.63
LayerSyne
1 x 10-4
1 10
16.03
Norm Training Iteration
Gradient Norm Training Beeatiin
Lageriyne 20.04 SAALD 2013--0
220-2-9 -le41
0.20
1.5
505.2 (1-b0
0.25
SIALD Joanly Monday
Gradeer Nomm
0.50
Gradeentorm
3.2
9.70
3.0
0.55
0.5
0.5
0.55
20200
42003
E0000
80002
102030
Timestop
Timestop
(b) Gradient norm when changing the learning rate of carly blocks.
(a) Gradient norm for different learning rates.
Figure 7: Gradient norm visualization revealed that LayerSync impact is not similar to simply increasing the learning rate as the model trained with LayerSyne has lower gradient norm than the baseline.
F LAYERSYNC VS SELF-REPRESENTATION ALIGNMENT USING EMA
We compare our LayerSyne approach with the concurrent work SRA (Jiang et al. 2025) terms of training time and computational overhead. We computed the metrics for SiT-XL/2 using 4 GH200 GPUs and a batch size of 32 per GPU. Results are reported in Table [5] We show that LayerSync requires 25.5% fewer Flops, is 40.5% faster in real-time, and reaches FID 5% higher.
Table 15:
Comparison between LayerSync and SRA. LayerSync results in lower FID while being less computationally expensive.
Method
FID L
Wall-clock time/step +
GFiops
STT-XL/2+SRA
1.58
0.617
30762
LayerSne
1.49
0.367
22910
G THE VIRTUOUS CYCLE AND THE EVOLUTION OF REPRESENTATIONS
To provide empirical support for the "virtuous cycle" and analyze the evolution the feature hi- erarchy, we conducted two additional studies evaluating the segmentation performance (mloU) of internal representations throughout the network on PASCAL VOC (Everingham et al. 2010).
First, tracked the evolution of representations throughout the training process (from 100k to 600k steps). As shown in Figure Sal we observe that while the relative structure feature quality across layers remains stable, the performance monotonically improves across the entire hierarchy.
Our second study evaluates models trained with different alignment targets at the same training stage (100k steps) (see Figure This comparison provides the most compelling evidence for our hypothesis. When we synchronize an early block (c.g., block 6) with a deeper target (block 18 vs. block 10), observe two critical effects:
19

---

Published as a conference paper at ICLR 2026
0.4
miu (%0
0.3
0.24
mlou
016
100K
200K soeps
6c-24 (FID:16.60)
sceps
(FID:15.39)
0.1
400K steps
0.05
Bx-28 (FID:16.08)
stops
10c--88 (FID:17.00)
600K steps
SIT-XL (FID:26.53)
24
Layer Index
Layer Index
(a) Evolution representations throughout the train- ing process assessed by semantic segmentation on
(h) Evaluation representations different lay- ers alignments assessed by semantic segmentation on
PASCAL VOC dataset.
PASCAL VOC dataset.
Figure Evaluation of the representations assessed by semantic segmentation on PASCAL VOC dataset. (a) Evolution throughout the training process. (b) Evaluation of representation for different layer alignment.
The model guided by the deeper layer achieves superior down- stream performance and lower FID (15.39 vs. 23.90), indicating that guidance from deeper layers correlates with better overall representations.
Global Improvement:
Accelerated Maturation: Notably, using deeper target shifts the peak performance of the network to earlier layers. By effectively "pulling" semantic richness from the deep target the earlier block, the early layers appear to acquire higher-level features sooner in the depth hierarchy.
These observations are consistent with the hypothesized virtuous cycle: guiding early block with a stronger target improves its representation, which in turn provides higher-quality input to subsequent layers. This likely facilitates the learning of stronger deep representations, which then serve as even better guides, progressively refining the entire hierarchy.
H OPTIMAL PLACEMENT OF EXTERNAL GUIDANCE WHEN COMBINED WITH LAYERSYNC
To maximize the synergy between internal and external alignment, we investigated the optimal depth for applying REPA. The results summarized Table 16 and show that applying REPA before the synchronization range leads to no significant synergies. The most effective strategy integrates the externa signal between the aligned layers. For instance, using LayerSyne to align layers 8 and 16 while applying REPA at layer 10 yields the best performance. All the models are trained for 50k iterations with 16 GPUs and batch size of 1024. show that the trend continues, we continue training the models to 200k iterations and report the results in the main paper.
I EVOLUTION OF BLOCK STRUCTURE
comparison between the block structure of the SiT-XL/2 and SiT-XL/2 + LayerSync is provided in Figure 9 showing that LayerSync imposes the structural equilibrium early in training.
20

---

Published as a conference paper at ICLR 2026
Table 16: Qualitative comparison between different combinations of REPA and LayerSync. The results show that combining LayerSyne with REPA can further accelerate the training, and the best place apply REPA is between the syncing layers.
Method
REPA Layer
LayerSync Layer
FID
SiTXL/2
59.45
SiTXL/2 LayerSync
8-16
46.26
SiTXL/2 REPA
46.06
SiTXL/2 REPA + LayerSyne
8-16
43.55
SiTXL/2 REPA + LayerSyne
10
8-16
29.68
Wer Sevieity (fear State)
Codas Siviarity leatmra
20,
590,
20.
lot
Mer.
bicce
bicc?
tic.
(a) Correlation between blocks of SiT-XL/2 after 120k iterations.
(b) Correlation between blocks SIT-XL/2 with LayerSync after 120k iterations.
Figure 9: Evolution of the Block Structure. (a) Early in training (120k iterations), the vanilla model lacks this structure. (b) LayerSyne imposes such structural equilibrium early in training (120k iterations).
J LAYERSYNC ALGORITHM
We present the algorithmic formulation of LayerSync Algorithm
Algorithm LayerSync
Require: Weak Representation Zμ E BxPDD Strong Representation Z R BxPDD
where B is the batch size, P the number of patches and D the feature dimension. D 1.2-normalize embeddings L2-normalize embeddings
1: Znorm <- normalize Zk. dim -1) norm
2: Z &- normalize Zz', dim = -1)
3: ClayerSync f- similarity ( Zearm [.]]] Z [.j])
> Negative similarity across patches
4: return Layerryyne
K FLOP COMPARISON
We compare the computational complexity of Dispersive Loss, which computes pairwise distances, with LayerSync in Table [7 LayerSync more efficient in terms of computational complexity. as the pairwise comparisons in Dispersive Loss result in quadratic cost with respect batch size.
21

---

Published as a conference paper at ICLR 2026
Table 17:
Comparison of computational complexity between the Dispersive Loss (pairwise dis- tances) and LayerSyne. B is the batch size and D is the feature dimension.
FLOPs
Scaling w.r.t Batch Size
Dispersive
(
Quadratic (B²) Linear (B)
LayerSyne
(
L IMAGE GENERATION EXPERIMENTAL DETAILS
We use a node of 4 GH200 GPUs and batch size of 256. The details of hyperparameters and sampler are provided in Tables 18 and T9
We use the Tiny ImageNet dataset (Deng et al., 2009). upsample the images to 256 x 256, and train linear classification heads for 50 epochs. Performance is evaluated on the validation set.
Classification.
Segmentation. For segmentation, we use the PASCAL VOC dataset (Everingham et al. 2010) and train linear heads for 25 epochs.
CKA. For CKA evaluations (Kornblith et al. 2019), use 4,000 samples from ImageNet 256 x 256.
Table 18: Hyperparameter setup for main experiments.
Table
(SiT-B)
Table 1 (SiT-L)
Table
(SiT-XL)
Table
Architecture
Input dim.
x x
32 x 32 x
32
32
x 32 4 28
Num. layers
12
24
28
Hidden dim.
768
1024
1152
1152
Num. heads
12
16
16
16
LayerSyne
0.3
0.2 (8,18)
0.2 (8,16)
0,2 (8,16)
Syncing layers (im-..))
(4,7)
cos. sim.
cos. sim.
cos. sim.
cos. sim.
Optimization
Batch size Optimizer
256 Adam W 0.0001
256 AdamW W 0.0001
256 AdamW 0.0001
256 Adam W 0.0001
Interpolants
O₁
t 1-t v-prediction SDE Euler-Maruyama 250 1.37
1-t v-prediction ODE Heun 250
1-t v-prediction ODE Heun 250
1-t v-prediction ODE Heun 250
Training objective Sampler
Sampling steps
Guidance
M
HUMAN MOTION GENERATION EXPERIMENTAL DETAILS
Task. Given sentence that describes motion sequence of actions, the task is to generate a corresponding human motion. Each motion sequence consists series of human poses, where each pose is represented by 22 joints defined as 3D points in space.
22

---

Published as a conference paper at ICLR 2026
Table 19: Hyperparameter setup for figures and ablation experiments.
Figure (SiT-XI.)
Table 6 (SiT-XI. and SiT-B)
Table 7 (SiT-B)
Architecture
Input dim.
32 32 4
32 x 32 x 4
32 x 32 x 4
Num. layers
28
12
Hidden dim.
1152
768
Num. heads
16
12
LayerSync
À
0.2
0.3
Alignment depth (im(..)
(8,16)
(2,8) cos. sim.
cos. sim.
cos. sim.
Optimization
Training iteration
400K
400K 256 Adam W 0.0001
400K 256 Adam W 0.0001
Batch size
256 Adam W 0.0001
Optimizer
Ir
Interpolants
1 1-t v-prediction ODE Heun 250
1-t v-prediction ODE Heun 250
1 1-t v-prediction ODE Heun 250
Training objective
Sampler
Sampling steps Guidance
Dataset. We rely on HumanMI dataset (Guo al. 2022 that contains 44,970 motion annota- tions across 14,646 motion sequences from the AMASS (Mahmood et al.] (2019) and HumanAct 2 (Guo et al. 2020) datasets, along with corresponding text descriptions, and is widely used for the task of text-conditional human motion generation. Motions in the HumanML3D dataset follow the skeleton structure of SMPL (Loper et al. (013) with 22 joints. Each pose p in the motion sequence represented vector of size 237,
(r
where r" E IR is the root (pelvis joint) angular velocity along the Y-axis; (F" E R are the root linear velocities in the XZ-plane; rd the root height; j" R " R and j" Roj the local joint positions, velocities, and rotations in the root space, with j indicating the number of joints; of I represents foot-ground contact features.
Implementation details. We use the exact setup MDM (Tevet et al. 2022), we train up to 600K iterations using a H100 GPU. We syne block with block 6.
N STOCHASTIC INTERPOLANTS
We adopt the generalized perspective of stochastic interpolants (Ma et al. (024) which provides unifying framework for both flow-based and diffusion-basee models.
At the core of these models a process that gradually transforms a real data sample xo p(x) into a simple noise sample ~ No00, T). This process is defined by:
0230 + O.C,
(4)
where 0g and d are functions of time, respectively decreasing and increasing, that control the mix of data and noise, satisfying the boundary conditions ag σT and ay do 0. The generative process aims to reverse this path. This can be model through deterministic trajectory commonly described as the probability flow ordinary differential equation (PF-ODE).
23

---

Published as a conference paper at ICLR 2026
x = ux,,t),
(5)
where v(x,, t) is the velocity field, specifying the direction and magnitude of movement at any point X; any time to go from noise back to data. The velocity fields defined as the time derivative of the interpolant:
xxx t) X1 [x,x==x Ex X x + 8,Ele x].
However, since those conditional expectations are intractable, model ux(x,, () is trained to approx- imate by minimizing the flow matching loss defined as:
Cveelciit (0) = Ex [1x00(xe,)) 6x 5
The data then generated by integrating equation 5 from t=1 t=0 using any standard ODE solver starting from random noise sample x ~ (0, There exists also an alternative way to model the reverse process using Stochastic Differential Equation (SDE). The SDE shares the same marginal probability densities (x) as the PF-ODE but follows stochastic, rather than deterministic, trajec- tory. The general form of this reverse SDE is:
dxc; (v(x, 1 ())((x),t) dt + Vwidw,
where Wi diffusion coefficient and dw; is a standard Wiener process, and s(x;, t) the score function, defined as the gradient of the log-density of the data. The velocity and the score are not independent, they are two sides of the same coin as the score can be derived from the velocity field and vice versa.
O EVALUATION METRICS DETAILS.
O.1 IMAGE
FID. FID. Heusel et al. (2017) measures the distance between the real and generated data distri- butions the feature space of pretrained Inception-v3 network (Szegedy et al.) (016). It computes the Fréchet distance (Heusel et al.) 2017) between two multivariate Gaus- sians fitted to the feature embeddings, capturing both the quality and diversity of generated samples. Lower values indicate better performance.
sFID. Nash et al. 2021) compares local image patches instead of global image statistics. By focusing patch-level embeddings, sFID provides more fine-grained evaluation of spatial consistency and local realism in the generated samples.
Inception Score. Salimans al. (2016) computes the Kullback-Leible (KL) divergence (Kullback & Leibler 1951) between conditional and marginal label distributions predicted by an Inception network.
Precision and Recall. Kynkäänniemi al. (2019) measures the fraction of generated samples that lie within the support of the real data distribution in feature space. High recall reflects the diversity of generated samples, indicating that the model captures the variability of the real data distribution.
O.2 AUDIO
FAD: Kilgour et al. (2018) like FID for images, is reference-based metric that measures the perceptual similarity between the distribution of generated samples and the distribution of real audio.
24

---

Published as a conference paper at ICLR 2026
0.3 VIDEO
FVD: Unterthiner et al. (2018) extends the idea of FID to videos by measuring the distance between real and generated video distributions in pretrained spatiotemporal feature space. Specifically, it uses embeddings from Carreira & Zisserman (2017), pretrained on large- scale video datasets, capture both spatial and temporal dynamics.
0.4 MOTION
FID: Computed in the same way as for images, but using T2M features instead of Inception features.
(Guo et al.) 2022) motion
R-Precision: Measures the relevancy of the generated motions to the input prompts.
P EXTENDED RELATED WORK
In what follows, we summarize the main baseline methods used our evaluation:
ADM (Dhariwal & Nichol, 202T): Builds upon U-Net-based diffusion models by intro- ducing classifier-guidee sampling, allowing fine-grained control over the trade-off between generation quality and diversity.
VDM++ (Kingma Gao 2023): Proposes an adaptive noise schedule that adjusts dynam- ically during training, improving convergence and sample quality.
Simple diffusion (Hoogeboom et al. 2023): Simplifies both the noise schedule and archi- tectural components, enabling high-resolution image generation with improved computa- tional efficiency.
CDM (Ho et al. 2022): Introduces cascaded diffusion models that progressively refine images from low high resolution using super-resolution stages, achieving better detail synthesis.
LDM (Rombach et al.
2022): Trains diffusion models in a compressed latent space learned
by a VAE, drastically reducing training cost while maintaining image fidelity.
U-VIT (Bao et ) Combines ViT-based backbones with U-Net-style skip connec- tions in the latent space, bridging the benefits of transformers and convolutional inductive biases.
DiffIT (Hatamizadeh et al. 2024): Enhances transformer-based diffusion models using time-aware multi-head self atienttention, boosting sample efficiency and reducing training time.
MDTv2 (Gao al. 2023): Employs asymmetric encoder-decoder transformer archi- tecture with U-Net-inspired shortcuts in the encoder and dense skip connections in the decoder, improving video generation quality and coherence.
MaskDiT (Zheng al. 2023): Introduces masked modeling into diffusion transformers by training with an auxiliary mask reconstruction objective, leading to better efficiency and generalization.
SD-DIT (Zhu al., 2024): Builds on MaskDiT by incorporating self-supervised discrim- ination objective using momentum encoding, enhancing the semantic richness of internal representations.
DiT (Peebles & Xie
2023): Proposes a pure transformer architecture for diffusion, using AdaL.-zero modules to stabilize training and seale to large model sizes efficiently.
SiT (Ma et al. 2024): Investigates the link between training efficiency and flow-based perspectives by transitioning from discrete-time diffusion to continuous flow matching, showing improved sample quality and convergence rates.
D-JEPA (Cen et al. 2024h: Integrates Doint-Embedding Predictive Architectures (JEPA) into generative modeling by reframing masked image modeling as generalized next-token prediction task, utilizing diffusion or flow matching loss to model per-token probability distributions in continuous space.
25

---

Published as a conference paper at ICLR 2026
VAR (Tian et al.
2024): Redefines autoregressive image generation coarse-to toarse-to-fine to-fine "next-scale prediction' process, diverging from standard raster-scan next-token prediction, allowing for faster inference and scaling laws similar to Large Language Models.
SRA Jiang et al. 2025): An EMA-based method that enables diffusion transformers to enhance their own representation learning and generation quality by aligning latent outputs from earlier, noisier layers with those from later, cleaner layers, eliminating the need for external guidance models.
DETAILS OF SIT MODEL
The architecture of the SiT block is provided in Figure 10 and more details the model parameters are summarized in Table 20
Scani,,
Pointewwe ove
Fooaforward
Layer Norm
Scatle
Scale, Stift
HUP
Tokens
Timestep
Figure 10: Visualization of a single SiT block.
Table 20: The number transformer layers, hidden dimensionality, and number of attention heads for SiT models used in experiments.
Config
#Layers
Hidden dim
#Heads
B/2
12
768
12
1024
XL/2
28
1152
16
R ATTENTION MAPS PCA OVER LAYERS
We visualize the learned representations by applying PCA to the features of SiT-XL/2 models trained on ImageNet 256. We add different levels noise to the input image and visualize resulting features. We compare two variants: the baseline SiT-XL/2 and SiT-XL/2 with LayerSync, where block 8 synced with block Both models are trained for 400K iterations on single node with GH100 GPUs. Our results show that LayerSync results in more discriminative features, particularly the carlier blocks.
Figure 11: Input image to the model
26

---

Published as a conference paper at ICLR 2026
Figure 12:
Visualization of SIT-XL/2 model features with 10% noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right comer.
Figure 13: Visualization of S/T-XL/2 model + LayerSync features with 10% noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right corner.
27

---

Published as a conference paper at ICLR 2026
Figure 14: Visualization of SIT-XL/2 model features with 30% % noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right comer.
Figure 15: Visualization of S/ model + LayerSync features with 30 % noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right corner.
28

---

Published as a conference paper at ICLR 2026
Figure 16: Visualization of SIT-XL/2 model features with 50% noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right comer.
Figure 17: Visualization of S/T-XL/2 model + LayerSync features with 50 % noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right corner.
29

---

Published as a conference paper at ICLR 2026
Figure 18: Visualization of SIT-XL/2 model features with 70% noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right comer.
Figure 19: Visualization of S/T-XI/2 model + LayerSync features with 70 % noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right corner.
30

---

Published as a conference paper at ICLR 2026
Figure 20: Visualization of SIT-XL/2 model features with 90% noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right comer.
Figure 21: Visualization of S/T-XL/2 model + LayerSync features with 90% noise added to the input image. The top-left plot shows the features from the first block, and subsequent blocks are visualized row by row, ending with the final block in the bottom-right corner.
31

---

Published as a conference paper at ICLR 2026
S QUALITATIVE EXAMPLES
We provide qualitative examples in Figure 22 The model is trained for 800 epochs on ImageNet dataset (Deng et al. 2009) and the samples are generated using classifier-free guidance with scale of 4 and the ODE Heun sampler.
Additionally, qualitative comparisons between the baseline SiT-XL/2, SiT-XL/2 regularized with Dispersive, and SiT-XL/2 regularized with LayerSyne trained on ImageNet dataset are shown in Figure 23 All models are trained for 400K iterations and share the same noise, sampler, and num- ber of sampling steps. The samples are generated using ODE Heun sampler and no classifier- free guidance is used. LayerSync improves generation quality without relying on external representation.
Figure 22: Selected samples from the SiT XL/2 with LayerSyne on ImageNet 256x We use classifier- free guidance with a CFG of 4.0.
32

---

Published as a conference paper at ICLR 2026
Training Iteration
100k
200k
400k
100k
200k
400k
SIT XL/2
LayerSync Dispersive
SIT XL/2
Dispersive
LayerSync
SiT XL/2
Dispersive
LayerSync
Figure 23: Qualitative comparison of SiT-XL/2 when regularized with Dispersive and LayerSyne.
33


Reviews:
[{"review_id": "rFePg00AVG", "reviewer": "Reviewer_Sohi", "summary": "This paper introduces LayerSync, a parameter-free, self-contained regularization strategy for diffusion models. Unlike previous work that aligns diffusion model representations with large external models, LayerSync aligns a model’s own weaker intermediate layers to its stronger, semantically richer layers, thus providing internal guidance. The method is simple (involving an intra-model alignment loss over patchwise cosine similarities), incurs negligible overhead, and is broadly applicable across data modalities—including images, audio, motion, and video. Empirical results on ImageNet, MTG-Jamendo (audio), HumanML3D (motion), and CLEVRER (video), as well as comprehensive ablations and internal feature analysis, demonstrate that LayerSync consistently outperforms prior self-contained baselines and narrows the performance gap with externally guided methods.", "questions": "Please refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- LayerSync is evaluated across four distinct modalities (image, audio, motion, and video), each with strong quantitative results and improvements in training efficiency, generation quality, and representation quality (Tables 1–4, 7; Figure 1). These cross-domain demonstrations suggest the approach’s generality.\n- Unlike methods requiring external pretrained models (e.g., DINOv2, VLMs), LayerSync requires no additional data or parameters, making it attractive for non-visual domains and resource-limited scenarios.\n- For image generation on ImageNet, LayerSync accelerates convergence by over 8.75x (Table 1/Figure 1b) compared to baseline diffusion transformers, and outpaces self-contained alternatives like Dispersive.\n- Qualitative and quantitative comparisons (e.g., Figure 2, Figure 3, Figure 15-16) support claims about quality and internal changes, and supplementary figures show robust improvements (e.g., noise robustness, block dropping effects).", "weaknesses": "- The strategy for picking “strong” and “weak” layers is based on architectural distance and exclusion of “final 20%” blocks. While Table 5 suggests robustness to randomization, the method remains ad-hoc, and a more systematic (possibly data-driven or information-theoretic) approach could yield better guarantees. There’s no analysis as to how the reference layer selection interacts with model depth or varying data/architecture.\n- The paper is primarily empirical; while this is fine, and the practical results are compelling, the absence of theoretical grounding for the expected improvements (e.g., analysis on generalization, capacity, information propagation) or limitations of the approach means its scope is less clear. For instance, are there pathological settings (e.g., excessive over-alignment leading to collapse) or fundamental limits with this approach where external representations would still be needed?\n- Although the paper notes that training with LayerSync increases resilience to block removal, it is not clear whether stronger self-alignment could lead to diminished feature diversity, especially in representations, and whether downstream performance is robust to such collapse.\n- For audio, motion, and video, the paper mostly reports aggregate scores (e.g., FAD, FVD, FID) with limited visualization or error analysis of failure modes. Given the modality-general claim, richer qualitative evidence would reinforce the universality claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jCYaybMGwB", "reviewer": "Reviewer_2qbB", "summary": "This paper aims to accelerate the training of diffusion transformers without external guidance to avoid dependency on an external model that introduces additional training cost and limited domains. To this end, the authors propose LayerSync, a simple yet effective model that uses features from a later layer of the diffusion model itself as an alignment target. They show that LayerSync accelerates training of diffusion models in a domain-agnostic manner.", "questions": "Please answer the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a simple yet effective framework that accelerates the training of the diffusion transformer.\n\n2. This paper shows that LayerSync is modality-agnostic (image, audio, human motion), demonstrating the effectiveness of the proposed method in various domains that may have challenges in using pre-trained representations.\n\n3. This paper introduces the heuristic guidance to find a layer to align, which is highly practical for using diffusion models in different domains.", "weaknesses": "1. A comparison to a more alignment method is needed. For instance, despite SRA [1] using the EMA model (which can make training of DiT slower), I think comparison with it in terms of training time (e.g., wall-time clock) is important to claim that we indeed need an alignment strategy that does not depend on an additional module.\n\n2. The authors claim that REPA has limitations in that a pre-trained representation model is not available for various domains. However, several studies have proposed methods to obtain a pre-trained representation model in a domain-agnostic manner [2-4]. If we use such models, we may accelerate the training of the diffusion model even faster than LayerSync (in terms of training time), and we can use pre-computed features to reduce the memory cost.\n\n[1] Jiang et al., No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves, Arxiv 2025 \\\n[2] Tamkin et al., DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning, NeurIPS 2021 \\\n[3] Baevski et al., Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language, Arxiv 2022 \\\n[4] Jang et al., Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gp5YNd3JAp", "reviewer": "Reviewer_eqEG", "summary": "The paper presents LayerSync as a self-sufficient, plug-and-play regularization for diffusion models across modalities, but the claims are undermined by a lack of solid theoretical grounding and insufficiently rigorous empirical validation. The experimental comparisons appear incomplete, and I identified errors in data analysis that could misrepresent the method’s effectiveness (e.g., the reported speedups and generalization results may be overstated). Without stronger theoretical justification, more comprehensive baselines, and corrected analyses, the work risks misleading reviewers and does not convincingly support its central contributions.", "questions": "- The paper lacks solid theoretical analysis of the LayerSync mechanism itself. In the original SiT architecture, skip connections between blocks already help maintain feature consistency across layers. The additional loss introduced by LayerSync may simply increase the overall gradient magnitude, which could accelerate convergence in a manner similar to using a larger learning rate. While this is a hypothesis, it raises questions about the underlying mechanism driving the observed improvements.\n- I strongly encourage the authors to provide more theoretical insights and empirical evidence regarding LayerSync in their rebuttal. For example, is the gradient norm significantly larger compared to the baseline? Would directly amplifying the feature gradients at corresponding locations in the baseline yield similar training benefits? Addressing these points would greatly clarify the contribution and effectiveness of LayerSync.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "- The core idea behind LayerSync is notably simple and clear, which contributes to the overall readability and accessibility of the paper.\n- The authors provide comprehensive empirical validation across multiple domains, including image, video, and audio generation, demonstrating the versatility of their approach.\n- Compared to the selected baselines, the proposed method exhibits a tangible improvement in convergence speed, as evidenced by the experimental results.", "weaknesses": "- The choice of SiT as the sole baseline is not sufficiently convincing. Although Table 1 demonstrates some improvements when LayerSync is added to SiT under various model sizes, the baseline itself performs relatively poorly. Achieving gains over a weak baseline does not provide strong evidence of the proposed method's effectiveness.\n- In the experiments presented in Table 2, LayerSync does not outperform SiT-XL/x+REPA or SiT-XL+REED in terms of FID or training epochs. However, the authors highlight their own results in bold, which may mislead reviewers. Furthermore, the experimental comparison remains limited to relatively weak baselines and does not include more recent and competitive approaches such as DJEPA or VAR. As a result, the evaluation is not comprehensive, and it is difficult to assess the true effectiveness of the proposed architecture.。", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tPUjaBw49H", "reviewer": "Reviewer_5YQE", "summary": "The paper presents LayerSync, a self-contained, plug-and-play regularization method for diffusion models leveraging their own intermediate representation. Based on the analysis on internal representations, LayerSync uses the model’s semantically rich layers as intrinsic guidance requiring no additional parameters or external components. Experimental results show consistent improvements in generation quality and training efficiency across multiple domains including, image, audio, video, and human motion.", "questions": "- How does LayerSync perform with CFG in Table 2?\n- The overall writing and presentation could be improved for better readability. Some sections, such as Section 4.3 (Results), would need detailed explanations. There are also a few minor typos and formatting issues (e.g., line 265 “table → Table”, 266 “SiTXL → SiT-XL”, 327 “10.000 → 10,000”, 355 missing space after “Results.”, 359 missing comma before “we”, Eq (6) notation issue).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The overall flow of the paper is clear. To overcome the limitation of relying on external alignment, the authors first analyze the internal representations of diffusion models using discriminative tasks. They then propose a simple self-contained alignment between earlier and later layers, where the latter hold richer semantic information. Finally, they validate the idea through cross-domain experiments and detailed analysis of internal representations.\n\n- The main strengths of the paper are broad generalization and thorough experiments. The proposed method is lightweight and domain-agnostic, showing consistent improvements across different modalities without the need for extra losses or external modules.", "weaknesses": "- The paper gives a simple heuristic for choosing which layers to align (Section 3.4), but more experiments on this would help. In Figure 3, layer 20 seems to work best for two tasks, yet the paper aligns layer 8 → 16 instead. Also, both SiT-L and SiT-XL use the same 8th layer (from Table 8). A layer-pair ablation could show which combinations are most effective and why.\n- The authors claim that “LayerSync establishes a virtuous cycle that progressively refines the entire feature hierarchy,” but this isn’t clearly supported. It would be more convincing to show how the representations evolve during training or how this behavior changes with different layer selections.\n- It seems the alignment is applied without considering timestep. Is there any analysis on how aligning at different timestep affects the results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)