Boosting Generative Image Modeling via Joint Image-Feature Synthesis
Authors: Theodoros Kouzelis
Efstathios Karypidis
Ioannis Kakogeorgiou
Spyros Gidaris
Nikos Komodakis
Venue: NeurIPS 2025 spotlight ()
Keywords: Diffusion models
Representation learning
Primary area: deep_learning
TL;DR: Jointly modeling image laments and high-level semantic features (e.g. DINO),  significantly enhances  generative quality and training efficiency.

Abstract: Latent diffusion models (LDMs) dominate high-quality image generation, yet integrating representation learning with generative modeling remains a challenge. We introduce a novel generative image modeling framework that seamlessly bridges this gap by leveraging a diffusion model to jointly model low-level image latents (from a variational autoencoder) and high-level semantic features (from a pretrained self-supervised encoder like DINO). Our latent-semantic diffusion approach learns to generate coherent image-feature pairs from pure noise, significantly enhancing both generative quality and training efficiency, all while requiring only minimal modifications to standard Diffusion Transformer architectures. By eliminating the need for complex distillation objectives, our unified design simplifies training and unlocks a powerful new inference strategy: Representation Guidance, which leverages learned semantics to steer and refine image generation. Evaluated in both conditional and unconditional settings, our method delivers substantial improvements in image quality and training convergence speed, establishing a new direction for representation-aware generative modeling.

Boosting Generative Image Modeling via Joint Image-Feature Synthesis
Theodoros Kouzelis Archimedes, Athena RC National Technical University of Athens
Efstathios Karypidis Archimedes, Athena RC National Technical University of Athens
loannis Kakogeorgiou Archimedes, Athena RC IIT, NCSR "Demokritos"
Spyros Gidaris
Nikos Komodakis Archimedes, Athena RC University of Crete IACM-Forth
valeo.ai
S
p(A)
sel( in (ll)))))
hangge
pla.s)
DINO
((fa)))( us i))
46-1
Figure 1: ReDi: Our generative image modeling framework bridges the gap between generative modeling and representation learning by leveraging a diffusion model that jointly captures low-level image details (via VAE latents) and high-level semantic features (via DINOv2). Trained to generate coherent image-feature pairs from pure noise, this unified latent-semantic dual-space diffusion approach significantly boosts both generative quality and training convergence speed.
Abstract
Latent diffusion models (LDMs) dominate high-quality image generation, yet inte- grating representation learning with generative modeling remains challenge. We introduce a novel generative image modeling framework that scamlessly bridges this gap by leveraging diffusion model jointly model low-level image latents (from variational autoencoder) and high-level semantic features (from a pretrained self-supervised encoder like DINO). Our latent-semantic diffusion approach leams to generate coherent Seaatrf pairs from pure noise, significantly enhancing both generative quality and training efficiency, all while requiring only minimal modifications to standard Diffusion Transformer architectures. By eliminating the need for complex distillation objectives, unified design simplifies training and unlocks a powerful new inference strategy: Representation Guidance, which leverages learned semantics to and refine image generation. Evaluated in both conditional and unconditional settings, our method delivers substantial im- provements in image quality and training convergence speed, establishing a new direction for representation-awaal generative modeling. Project page and code: http:://rppeeentatiimoffffff htw.fffuuiiooii.uu.biio.oem..
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

40
XL/-XL/2
SiT SE--L//S R
30
DIT-XL/2+BaD1
20
S1T-XL/2+
S1T-X1/2
SIT-XL/2+RED 2-REPA
20
31T-XL/2+BeD1
F1110
x 23 speed-up
5
x11
X6
3.3
3.3
100K
400K
IM
2M
4M
7M
100K
400K
1M
2M
4M
Training Iteration
Training Iteration
Figure 2: Accelerated Training. Generative performance curves on Imagenet 256 256 without Classifier-Free Guidance. Left: Our ReDi accelerates convergence of DiT-XL/2 and SiT-XL/ XL/21 by approximately x23. Right: ReDi converges x6 faster than REPA. When applied on top of REPA delivers a x1l speed-up.
Introduction
Latent diffusion models (LDMs) (Rombach et al.) 22022) have emerged as a leading approach for high-quality image synthesis, achieving state-of-the-ard results (Rombach et al., 2022, Peebles Xie
2023 Ma et al. 2024). These models operate in two stages: first, a variational autoencoder (VAE) compresses images into a compact latent representation (Rombach et al.) 2022); second, a diffusion model learns the distribution of these latents, capturing their underlying structure.
Leveraging their intermediate features, pretrained LDMs have shown promise for various scene understanding tasks, including classification (Mukhopadhyay et al. (023), pose estimation (Gong al., 2023), and segmentation (Li et al., | Liu al.] 2023] Delatolas et al. 2025). However, their discriminative capabilities typically underperform specialized (self-supervised) representation learning approaches like masking-based (He et al., 2022), contrastive (Chen et al.) 2020), self- distillation (Caron al. 222 l), vision-language contrastive (Radford et al. 202 methods. This limitation stems from the inherent tension LDM training the need to maintain precise low-level reconstruction while simultaneously developing semantically meaningful representations.
This observation raises fundamental question: How can we leverage representation learning to enhance generative modeling? Recent work by Yu et al. (2025) (REPA) demonstrates that improving the semantic quality of diffusion features through disti listinlation of pretrained self-supervised repre- sentations leads to better generation quality and faster convergence. Their results establish clear connection between representation learning and generative performance.
Motivated by these insights, we investigate whether a more effective approach to leveraging represen- tation learning can further enhance image generation performance. In this work, contend that the answer is yes: rather than aligning diffusion features with external representations via distillation, we propose to jointly model both images (specifically their VAE latents) and their high-level semantic features extracted from pretrained vision encoder (e.g.. DINOv2 (Oquab et al. (2024)) within the same diffusion process. Formally, as shown in Figure we define the forward diffusion process as (x,,, 2, x 2t-1) for í = 1, .... T, where Xg x and Zo = are the clean VAE latents and semantic features, respectively. The reverse process po (Xt-1; 1: -t-1 X4, 2;) learns to gradually denoise both modalities from Gaussian noise.
This joint modeling approach forces the diffusion model to explicitly learn the joint distribution of both precise low-level (VAE) and high-level semantic (DINOv2) features. We implement this approach, called ReDi (Representation Diffusion). within the DiT (Peebles & Xie, 2023) and SiT (Ma al. 2024) frameworks with minimal modifications to their transformer architecture: we apply standard diffusion noise to both representations, combine them into single set of tokens, and train the standard diffusion transformer architecture to denoise both components simultaneously.

---

Compared to REPA, our joint modeling approach offers three key advantages. First, the diffusion process explicitly models both low-level and semantic features, enabling direct integration of these complementary representations. Second, our method simplifies training by eliminating the need for additional distillation objectives. Finally, during inference, our unified approach enables Repre- sentation Guidance where the model uses its learned semantic understanding to iteratively refine generated images, improving quality in both conditional and unconditional generation.
Our contributions can be summarized as follows:
1. We propose ReDi, a novel and effective method that jointly models seange-compreseed latents and semantically rich representations within the diffusion process, significantly improving image synthesis performance.
2. We provide a conerete implementation of our approach for both diffusion (DiT) and flow- matching (SiT) frameworks, leveraging DINOv2 (Oquab et al. 2024) as the source of high- quality semantic representations.
3. We also introduce Representation Guidance, which leverages the model's semantic predictions during inference to refine outputs, further enhancing image generation quality.
4. We demonstrate that our approach boosts performance in both conditional and unconditional generation, while significantly accelerating convergence (see Figure 2).
2 Related work
Representation Learning. Various approaches aim to learn meaningful representations for down- stream tasks, with self-supervised learning emerging as one of the most promising directions. Early approaches employed pretext tasks such as predicting image patch permutations (Noroozi & Favaro 2016) or rotation angles (Gidaris et al., 2018). while more recent methods utilize contrastive learn-
ing (Chen al. 2020; Van den Oord et al.] 2018:: Misra Maaten 2020). clustering- based ohjec-
tives (Caron et al. 2008)]20110099 and self-distillation techniques (Grill et al. 2020 Chen & He
2021 Caron et al. | [021]] Gidaris et al. 202T). The introduction of transformers enabled Masked
Image Modeling (MIM), introduced by BEiT (Bao et al. (202)) and evolved through SimMIM (Xie) ct al. 20222, , MAE He et al. (2022), AttMask (Kakogeorgiou et al., 2022). iBOT (Zhou et al. 2022] )
and MOCA Gidaris et al. 2024). with DINOv2 (Oquab et al. 2024) achieving state-of-the-art
performance through scaled models and datasets. Separately, contrastive vision-language pretraining, initiated by CLIP (Radford et al.] (2021a). established powerful joint image-text representations. Subsequent models like SigLIPIZhin et al. (2023) and SigLIPv2 (Tschannen et al.] (025)) refined this framework through enhanced training techniques, excelling in zero-shot settings and image retrieval (Kordopatis-Zilos et al. 2025). Building on these advances, we leverage pretrained DINOv2 visual representations enhance image generative modeling performance.
Diffusion Models and Representation Learning Due to the success of diffusion models, many
recent works leverage representations learned from pre-trained diffusion models for downstream tasks (Fuest et al. 2024). In particular, intermediate U-Net (Ronneberger et al.) 2015) features have
been shown capture rich semantic information, enabling tasks such semantic segmentation
(Baranchuk et al. [202] ZZhao et al. (202)), semantic correspondence (Luo et al. [023] (Zhang et al.)
2023 Hedlin et al.] (023), depth estimation (Zhao et al. (023). and image editing (Tumanyan ct al. 2023). Furthermore, diffusion models have been used for knowledge transfer by distilling learned representations through teacher-student frameworks (Li al. 2023a) or refining them via reinforcement learning (Yang& & Wang 2023). Other works have shown that diffusion models learn strong discriminative features that can leveraged for classification (Mukhopadhyay et al.) [2023] Xiang et al. (2023). In a complementary direction, REPA ( YYu et al.) (223)) recently demonstrated that aligning the internal representations of DłT (Peebles & Xie| Xie| (0233) with a powerful pre-trained visual encoder during training significantly improves generative performance. Motivated by this observation, we propose to integrate images and semantic representations into a joint learning process.
Multi-modal Generative Modeling Unifying the generation across diverse modalities has recently attracted widespread interest. Notably, CoDi (Tang et al. 2023) leverages a diffusion model that enables generation across text, image, video, and audio in an aligned latent space. A joint represen- tation for different modalities has been shown to have great scalability properties (Mizrahi et al.) 2023). For video generation, WVD Zhang et al. 2024) incorporates explicit 3D supervision by
3

---

VAE Letent
Noiey VAR Tatent
DAT/BAT
Prodicted VAE Labent
Image
Toloon Porio
DLT/BLT Black
PLT/S1T Block
DLT/344 Block
Prodicted DIMDx2
DINDy2
Noiay D1M0v2 PCA
Figure 3: Given an input image, the VAE latent and the principal components of DINOv2 are extracted.
Both modalities noised and fused into joint token sequence, given as input to DiT or SiT.
learning the joint distribution of RGB and XYZ frames. To capture richer spatial semanties, GEM (Hassan et al., 2024) generates paired images and depth maps. MT-Diffusion (Chen et al. 2024 learns to incorporate various multi-modal data types with a multitask loss including CLIP (Rafford al., lb) image representations. However, they do not quantitatively assess how this impacts the generative performance. VideoJam (Chefer et al. 2025) models joint image-motion representation that boosts temporal coherence and introduces a theoretically motivated Classifier- Free Guidance (CFG) Ho & Salimans| (2022) variant to condition on both motion and text. Inspired by this approach and building on the standard CFG framework, propose Representation Guidance, incorporating the visual representations as an additional guidance signal during inference.
Method
3.1 Preliminaries
Denoising Diffusion Probabilistic Models (DDPM) Diffusion models (Ho et al., 2020) generate data by gradually denoising noisy input. The forward process corrupts input X (e.g., image or its VAE latent) over T steps by adding Gaussian noise:
X
oo + V=
where x, is the noisy input step de are constants that define the noise schedule, and NN(, is the Gaussian noise term. Following Ho et al. (2020), the reverse process learns to denoise x by predicting the added noise e using a network E (-)-) with parameters The training objective is:
simple Exx,,,,//Ee/(x,,
=ll
Although we also include the variational lower bound loss from Nichol & Dhariwal 2021 to learn the variance of the reverse process, we omit hereafter for brevity.
Unless otherwise specified, we focus on class-conditional image generation throughout this work. For notational simplicity, omit explicit class conditioning variables from all mathematical formu- lations.
Diffusion Transformers (DiT) The DiT Peebles Xie (2023) implements eg using a Vision Transformer Dosovitskiy et al. (202)). Given the "patchified" input x, E LxC, (I tokens of dimension Cz), the model first computes embeddings:
h, X4 W emb,
Wend RCxCCa
The transformer processes h, E LxCaa to produce o, E LxCa The final noise prediction is computed as:
co(x,, t) o, W dec:
Waee CaxC.
3.2
Joint Image-Representation Generation
Our goal is to train single model to jointly generate images and their semantic-aware visual represen- tations by modeling their shared probability distribution. This approach captures the interdependent

---

structures and features of both modalities. While we frame our approach using DDPM. it is also applicable to models trained with flow-matching objectives Ma al. (2024) (see (see Appendix A).
high-level overview of our method is depicted in Figure Let denote a clean image, Xn = (1) RL its VAE tokens (produced by the VAE encoder (.), and Zg (t) E RLXC. its patch-wise visual representation tokens (extracted by pretrained encoder (+), e.g., DINOv2 Oquab al. (2024) To match the spatial resolution of Xo, we assume & (-) includes a bilinear resizing operation.
During training, given X and Zu, we define a joint forward diffusion processes:
x X vT - artes 24 = v va + vI - 0162
(5)
where &q controls the noise schedule and E, ~ (o),,,), E, ~ -N(O,I) I) are Gaussian noise terms of dimensions RL and I2 fxxxx,, respectively.
The diffusion model (X,, ze, t) takes as input x, and zt. along with timestep t, and jointly predicts the noise for both inputs. Specifically, it produces two separate predictions: eg (x,, Z,, f) for the image latent noise Ex, and eğ (xe; Ze, t) for the visual representation noise E.. The training objective combines both predictions:
Cjoint
JE Xo.
(x zc,t) Exl1 + >x ze,t) - E 1
(6)
where X balances the denoising loss for z₁ By default, use A₂ in experiments.
3.3 Fusion of Image and Representation Tokens
YAB Latent
We explore two approaches to combine and jointly process x, and Ze in the diffusion trans- former architecture: (1) merging tokens along the embedding dimension, and (2) maintaining separate tokens for each modality (see Fig. Both methods require only minimal modifi- cations to the DiT architecture, specifically defining mopelity-specific embedding matrices W emb E RC2xCa and W emb RC,, along with prediction heads We dee E RCaxC, and W dec CCCxC= for X2 and z, respectively.
Merged Tokens
Separate Tokens
DINDV2 PCA
Figure 4: An illustration of our proposed token fusion approaches: (a) The tokens of the VAE la- tents and the DINOv2 are merged channel-wise, (b) The tokens are concatenated along the sequence dimension.
Merged Tokens The tokens are embedded separately and summed channel-wise:
h = x; W cmb + 2t Winb E R.CCK
(7)
The transformer processes h, to produce o,, with predictions:
c = o, W
cp o, W
(8)
doc
This approach enables early fusion while maintaining computational efficiency, as the token count remains unchanged.
Separate Tokens
Tokens are embedded separately and concatenated along the sequence dimension: |x, W
Z; W m
2L2
where [,, :] denotes sequence- wise concatenation. The transformer outputs separate representations
O: Jo of], with predictions:
o
W dee
of dec
(10)
This method provides greater expressive power by preserving modality-speciff information through- out processing, at the cost of increased computation due to increased token count.
Unless stated otherwise, we use the merged tokens approach for computational efficiency.
For notational clarity, we incorporate the patchification step (typically with 2 2 patches in DIT architectures) into the encoder definitions E, and E..
5

---

3,4 cetpriiionally-Reddeee Visual Representation
In practice, the channel dimension of visual representations (C₂) significantly exceeds that of image latents (Cz), i.e., O2 > Cz. We empirically observe that this imbalance degrades performance, as the model disproportionately allocates capacity to visual representations at the expense of image latents.
To address this, we apply Principal Component Analysis (PCA) to reduce the dimensionality of 2o from C2 C² (where C²2 < C22., preserving essential information while simplifying the prediction task. The PCA projection matrix is precomputed using visual representations sampled from the training set. All visual representations Sections 3.2 and refer these PCA-reduced versions.
3.5 Representation Guidance
To ensure the generated images remain strongly influenced by the visual representations during inference, introduce Representation Guidance. This technique during inference modifies the posterior distribution to: (x( z,) x (x(x)) (2))))))))))))) where w, controls how strongly samples are pushed toward higher likelihoods of the conditional distribution po( (x.). Taking the log derivative yields the guided score function:
Vx, log pu (Xe, z,) Vx, log po (Xe) + w. (Vx, log po pa(ze (x+))
(11)
=vx, log po (x+) + wr (vx, log po (xt: 21) Vx, log (x.)).
(12)
By recalling the equivalence of denoisers and scores (Vincent, 2011). we implement this representation-guidee prediction éo (x,, Ze, t) at each denoising step as follows:
(X,, 2e, () Ep (xr. () + wr (EO(X., Zz, () EØ Ex(x,, ())-
(13)
Following Ho & Salimans (022)) ) we train both eØ(x,, Z:, t) and eø ex jointly. Specifically,
during training, with probability Pdrop, zero out z, (setting eæ (x,, .) = EØ (xe, 0, t)) and disable the visual representation denoising loss by setting À = 0 in Eqqution
Experiments
4.1 Setup
Implementation details. We follow the standard training setup of DiT (Peebles & Xie 2023
and SiT (Ma al. 2024), training ImageNet at 256 x 256 resolution with a batch size of 256. Following ADM'S preprocessing pipeline (Dhariwal & Nichol 2021), we center-crop and resize all images 256 x 256. Our experiments utilize transformer architectures B/2, L/2, and XL/2 all using a 22 2 patch size. For unconditional generation, we simply set the number of classes to 1, maintaining the original architecture. Images are encoded into VAE latent representations using SD-VAE-FT-EWS (Rombach et al., 2022) that produces outputs with x8 spatial downsampling factor and 4 output channels. For 256 256 images, this results in 32 x 32 4 latent features. Through patchification with 2 x 2 patches, the VAE encoder (.) yields L = 256 tokens, each with C² = 16 channels (4 channels 2x2 patch size). For semantic representation extraction, we employ DINOv2-B with registers DDarett et al. 2023; Oquab et al. 2024). The 768-dimensional embeddings are reduced to 8 dimensions via PCA (trained on 76,800 randomly sampled ImageNet images). After bilinear interpolation to match the VAE's 32 x 32 x 4 spatial resolution and 2 x 2 patchification, the encoder Eg() produces 256 tokens with C, = 32 channels each (8 channels 2x2 patch size).
Sampling. For DiT models, we adopt DDPM sampling, while for SiT models, we employ the SDE Maruamaa sampler. The number of sampling steps is fixed 250 across all experiments. When using Classifier-Free Guidance (CFG) (Ho & Salimans 2022). we apply it only to the VAE channels, with a guidance scale of w 2.4 (see Figure 6). For Representation Guidance, we set Parop 0.2, the guidance scale to w, = 1.5 for models and w. 1.1 for XL models.
Evaluation. To benchmark generative performance, we report Frechet Inception Distance (FID) (Heusel et al. (017), sFID (Nash et al. 2021). Inception Score (IS) (Salimans et al. 2016), Precision (Pre.) and Recall (Rec.) (Kynkäänniemi et al. 2019) using 50k samples and the ADM's TensorFlow
evaluation suite (Dhariwal & Nichol 2021).

---

Table FID Comparisons. FID scores on ImageNet 256 x 256 without Classifier- Free Guidance for DiT and SiT models of various sizes with REPA and ReDi (ours).
Table 2: Comparison with State-of-the-art. Quantita- tive evaluation on ImageNet 256 x 256 with Classifier Free Guidance. Both REPA and ReDi (ours) employ SiT-XL/2 as the base model.
MODEL
PPARMM
ITER.
FIDL
MODEL
EPOCHS FIDI SFIDI 1st PRE.1 REC.
DiT-L/2
458M
400K
23.2
Autoregressive Models
w/ REPA
458M
400K
15.6
VAR
350
1.80
365.4
0.83
0.57
w/ ReDi (ours)
458M
400K
10.5
MagViTv2
1080
1.78
319,4
0.83
0,57
MAR
800
1.55
303.7
0.81
0.62
SiT-L/2
458M
400K
18.5
w/ REPA
458M
400K
9.7
Latent Diffusion Models
w/ ReDi (ours)
458M
400K
9.4
LDM
200
3.60
247.7
0.87
0.48
H/2
240
2.29
5.68
263.9
0.82
0.57
DiT-XL/2
675M
400K
19.5
XX/2
1400
2.27
4.60
278.2
0.83
0,57
w/ REPA
675M
100K
12.3
MaskDiT
1600
2.28
5.67
276.6
0.80
0.61
DIT-XL/2 REPA
675M
7M
9.6
SD-DiT
480
3.23
675M
850K
9.6
SiT-XL/2
1400
2.06
4.50
270.3
0.82
0.59
w/ ReDi (ours)
675M
400K
8.7
FasterDiT
2.03
4.63
264.0
0.81
0.60
MDT
1300
1.79
4.57
283.0
0.81
0.61
SiT 2XL/2
675M
400K
17.2
w/ REPA
675M
400K
7.9
Leveraging Visual Representations
w/ ReDi (ours)
675M
400K
7.5
REPA
800
1.80
4.50
284.0
0.81
0.61
ReDi (ours)
350
1.72
4.68
278.7
0.77
0.63
SiT-XL/2
675M
7M
8.3
ReDi (ours)
800
1.61
4.66
295.1
0.78
0.64
w/ REPA
675M
4M
5.9
w/ ReDi (ours)
675M
700K
5.6
ReDi (ours)
675M
4M
3.3
4.2
Enhancing the performance of generative models
DiT & SiT. To demonstrate the effectiveness of our approach, we present performance gains for various- sized DiT and SiT models in Table 1] Our method, ReDi, consistently delivers substantial improvements across models of different scales. Notably, BiT-XL/2 with ReDi achieves an FID of 8.7 after just 400k iterations, outperforming the baseline DiT-XL/2 trained for 7M steps. Similarly, SiT-XL/2 with ReDi reaches an FID of 7.5 at 400k iterations, surpassing the converged SiT-XL at 7M steps. Additionally, Tabll 2 reports results for SiT-XL/2 with Classifier-Fins Free Guidance (CFG) Ho & Salimans| (2022). Once again, ReDi yields significant improvements, achieving an FID of 1.72 just 350 epochs, outperforming the baseline trained to convergence over 1400 epochs.
Comparison with REPA. We further compare our results with REPA, which also leverages DINOv2 features to enhance generative performance. Our approach, ReDi, consistently achieves superior generative performance with both DiT and SiT as the base models. As shown in [able I DiT-L/2 with ReDi achives an FID of 10.5 significantly outperforming DiT-L/2 with REPA. Notably, even surpasses REPA trained for the same number of iterations with the larger DiT-XL/2, which achieves a higher FID of 12.3. Further for SiT-XL models, ReDi attains an FID of 5.6 in just 700k iterations. while REPA A requires 4M iterations to reach an FID of 5.9. These results highlight the effectiveness of our method in leveraging visual representations significantly boost generative performance.
ReDi is complementary to REPA. Interestingly, we observe that the joint modeling objective of our ReDi and the alignment objective of REPA are complementary. As presented in Table 5 REPP + ReDi matches the FID of the dully-converged REPA after only 350K iterations, and at 1M iterations reaches an FID of 3.6. For the implementation details, see Appendix B.3]
Accelerating convergence. The aforementioned results indicate that ReDi significantly accelerates the convergence of latent diffusion models. As illustrated in Figure 2] ReDi speeds up the convergence of DiT-XL/2 and XL/2 by approximately x23, respectively. Even when compared with REPA ReDi demonstrated a x6 faster convergence. When ReDi is applied on top REPA, A, the convergence is x 11 faster.
Comparison with state-of-the-art generative models. Ultimately, we provide a quantitative comparison between ReDi and other recent generative models using Classifier-Free Guidance (CFG)
7

---

Table 3: Unconditional Generation FID Per- formance. Results on ImageNett 256 x 256. For comparison, we include conditional generation results (shown in gray). Models at 400K steps. RG denotes using Representation Guidance.
Table FID with Representation Guidance.
FID scores on ImageNet 256 x 256. RG denotes
Representation Guidance. Models at 400K steps.
MODEL
#PARAMS
FID
DiT-B/2 w/ ReDi
130M
25.7
MODEL
#PARAMS
FID
DiT-B/2 w/ ReDi+ RG
130M
20.2
DiT-B/2 (conditional)
130M
43.5
DiT-XL/2 w/ ReDi
675M
8.7
DiT-B/2
130M
69.3
DiT-XL/2 w/ ReDi+ RG
675M
5.9
w/ ReDi (ours)
130M
51.7
w/ ReDi+RG (ours)
130M
47.3
Table 5: ReDi with REPA. FID scores on Ima-
geNet 256x256 w/o CFG.
DiT-XL/2 (conditional)
675M
19.5
DiT-XL/2
675M
44.6
MODEL
#ITER.
FID
w/ ReDi (ours)
675M
25.1
SiT-X SiT-XL/2 w/ REPA
4M
5.9
w/ ReDi+RG (ours)
675M
22.6
SiT XL/2 w/ REPA+ReDi
350K
5.9
SiT-XL/2 2 w/ REPA+ReDi
1M
3.5
Image
DINOV2
Figure 5: Selected samples from our SiT-XL/2 w/ ReDi model trained on ImageNet 256 x 256. Images and visual representations are jointly generated by our model. We use Classifier-Fres Guidance with 4.0.
Our method already outperforms both the vanilla SiT-XL XL and SiT-XL with REPA with only 350 epochs. At 800 epochs ReDi reaches an FID of 1.64. We provide qualitative results of both generated images and visual representations in Figure 5
(Ho & Salimans 2022 in Table 2
Improving Unconditional Generation. establish the effectiveness our method in improving generative models, we further present experiments for unconditional generation using DiT. As shown in Table 3, our ReDi significantly improves generative performance for various model sizes. Specifically, with our ReDi FID drops from 69.3 to 51.7 for B and from 44.6 to 25.1 for XL models.
4.3 Impact of Representation Guidance on generative performance.
Class Conditional Generation. In Table 4 we present the impact of Representation Guidance (RG) on generative performance. We observe that for both and XL models, Representation Guidance unlocks further performance enhancements by guiding the generated image closely follow the semantic features of DINOV2. Particularly for DiT-XL w/ ReDi the FID drops from 8.7 to 5.9. We also present qualitative results in Figure
Unconditional Generation. Representation Guidance is especially useful in unconditional gener- ation scenarios, where the absence of class or text conditioning prevents the use of Classifier- Guidance to enhance performance. As demonstrated in Table 3 Representation Guidance enhances the performance of ReDi with both and XL models, further further closing the performance gap between unconditional and conditional generation. Notably, ReDi with Representation Guidance achieves an FID of 22.6, approaching the performance of the class-conditioned DiT-XL. XL/2 (FID of 19.5).

---

VAE-only CFG
45
43
DINGY2
VAE & DINOv2 CFG
40
36.9
FID
FID
35
31.9
29.2
29,
27
27:5
25.7
690-90
25
2
w/o
2
8
12
16
32
Classifter-Free Guidance weight w
DINOv2
# Principal Components
Figure 6: VAE-only &AE& DINOv2 CFG.
FID scores for SiT-XL with ReDi (trained for 400K steps)
Figure 7: Effect of number of principal com- ponents. FID of DiT-B/2 w/ ReDi with dif- ferent number of DINOv2 Principal Components. The vanilla DiT-B/2 is illustrated with gray. No Classifier-Free Guidance is used.
function of Classifier- Free Guidance weight w, comparing two con- figurations: (1) applying CFG only VAE latents (VAE-only CFG) versus (2) applying CFG to both VAE and DINOV2 representations
(VAE& & DINOv2 CFG).
Table 6: Performance of Modality Combination Strategies. FID scores ImageNet 256 x 256 without CFG for DiT-B/2 with ReDi using Separate Tokens (SP) and Merged Tokens (MR). See Appendix B for details on throughput measurements.
MODEL
#ToKENS THROUGHPUT
FID4
DiT-B/2
256
4.52
43.5
w/ ReDi (MR)
256
4.51
25.7
w/ ReDi (SP)
512
2.26
24.7
4.4 Analysis
Dimensionality reduction ablation. We begin the analysis of our method by ablating the impact of dimensionality reduction on the visual representations, as shown in Figure 7 Initially, we observe that jointly learning little principal component yields significant improvements in generative performance. Increasing the component count continues to improve performance, up to r = 8, beyond which further components begin degrade the quality of generation. This suggests optimal intermediate subspace where compressed visual features retain sufficient expressivity to guide generation without dominating model capacity.
Merged Tokens vs. Separate Tokens. In Table 6) we evaluate the effectiveness of the two explored integration strategies, Merged Tokens (MR) and Separate Tokens (SP), for joint learning of image VAE latents and visual representations, using DiT-B/2 as our base model. While both approaches achieve comparable performance gains, SP demonstrates slightly better results. This advantage comes at significant computational cost: SP doubles the transformer's 's input sequence length by introducing 256 additional DINDv2 tokens. resulting approximately 2x greater compute demands during both training and inference (Kaplan et al. 2020). The MR strategy, by contrast, maintains the original sequence length while delivering similar performance improvements, thereby preserving computational efficiency as measured by throughput.
VAE-only Classifier-Free Guidance. As ReDi jointly models both VAE latents and visual repre- sentations, we investigate two Classifier-Free Guidance (CFG) strategies: applying CFG exclusively VAE latents (VAE-only CFG) versus applying to both modalities simultaneously (VAE & DINOv2 CFG). Our experiments in Figure demmonsrrrte that VAE-only CFG achieves superior results, yielding an FID of 2.39 compared 2.86 for the VAE & DINDv2 CFG approach. Notably, VAE-only CFG also shows greater robustness variations in the CFG weight parameter.

---

Conclusion
In this work, we explore the relationship between semantic representation learning and generative performance in latent diffusion models. Building recent insights, we introduced ReDi, novel framework that integrates bigh-level semantic features with low-level latent representations within the diffusion process. Unlike prior approaches that rely auxiliary objectives, ReDi jointly models two distributions. We demonstrate that this simple approach more effective at leveraging the semantic features and leads to drastic improvements in generative performance. We further proposed Representation Guidance, a novel guidance method that leverages the jointly learned semantic features to enhance image quality. Across both conditional and unconditional settings, ReDi consistently improves generation quality and accelerates convergence, highlighting the benefits of our approach.
This work has been partially supported by project MIS 5154714 14 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGeneration Program and by Institute of Informatics and Telecommunications, National Center for Scientific Research "Demokritos". Hardware resources were granted with the support of GRNET. Also, this work was performed using HPC resources from GENCI-IDRIS (Grants 0024-AD01101288445
Acknowledgements
References
Bao, F., Nie, S., Xue, K., Cao, Li, C., Su, H., and Zhu, J. are worth words: vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 72669-22673 2023.
Bao, H., Dong, L., Piao, S., and Wei, F. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
Baranchuk, D., Voynov, A., Rubachev, I., Khrulkov, V., and Babenko, A. Label-efficient semantic segmentation with diffussion models. In International Conference on Learning Representations, 2022.
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision, pp. 132-149, 2018.
Caron, M., Bojanowski, P., Mairal, J., and Joulin, A. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2959-2968, 2019.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural Information Processing Systems, 33:9912 9924, 2020.
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Chefer, H., Singer, U., Zohar, A., Kirstain, Y., Polyak, A., Taigman, Y., Wolf, L., and Sheynin, S. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arX1v:2502.02449, 2025.
Chen, C., Ding, H., Sisman, B., Xu, Y., Xie, O., Yao, B. Z., Tran, S. D., and Zeng, B. Diffusion models for multi-modal generative modeling. arXiv preprint arXiv:2407.17571, 2024.
Chen, T., Korbblith, S., Norouzi, M., and Hinton, G. simple framework for contrastive learning of visual representations. In JCML, 2020.
Chen, X. and He, K. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, Vision transformers need registers. arXiv preprint 1xXXi:2309.16888 2023.
10

---

Delatolas, T., Kalogeiton, V., and Papadopoulos, D. P. Studying image diffusion features for zero-shot video object segmentation. arXiv preprint arXiv:2504.05440 2025.
Dhariwal, P. and Nichol, A. Q Diffusion models beat GANs on image synthesis. In NeurIPS, 2021.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Elfwing, S., Uchibe, E., and Doya, K. Sigmoid- linear units for neural network function approximation reinforcement learning. Neural networks, 107:3-11, 2018.
Fuest, M.. Ma, P., Gui, M., Schusterbauer, J., Hu, V. T., and Ommer, B. Diffusion models and representation learning: A survey. arXiv preprint arXiv:2407.00788, 2024.
Gao, S., Zhou, P., Cheng, M.-M., and Yan, S. Mdtv2: Masked diffusion transformer is strong image synthesizer. arXiv preprint 2.33:::33331143888 2023.
Gidaris, S., Singh, P., and Komodakis, N. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Gidaris, S., Bursuc, A., Puy, G., Komodakis, N., Cord, M., and Pérez, P. Obow: Online bag-of- of-uuall----- words generation for supervupe learning. In CVPR, 2021.
Gidaris, S., Bursuc, A., Siméoni, O., Vobecký, A., Komodakis, N., Cord, M., and Perez, P. MOCA: Self-supervised representation learning by predicting masked online codebook assignments. Trans- actions on Machine Learning Research, 2024.
Gong, J., Foo, L. G., Fan, Z., Ke, Q., Rahmani, H., and Liu, J. Diffpoos: Toward more reliable 3d pose estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2023.
Grill, J.-B.. Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C.. Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach self- supervised learning. Advances in Neural Information Processing Systems, 33:21271-21284, 2020.
Hassan, M., Stapf, S., Rahimi, A., Rezende, P., Haghighi, Y., Brüggemann, D., Katircioglu, L., Zhang, L.. Chen, X., Saha, S., et al. Gem: A generalizable ego- vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control. arXiv preprint arXiv:24/2.11198, 2024.
He, K., Chen, X., Xie, S., Li, Dollár, P., and Girshick, Masked autoencoders are scalable vision learners. In CVPR, 2022.
Hedlin, E., Sharma, G., Mahajan, S., Isack, H., Kar, A., Tagliasacchi, A., and Yi, K. M. Unsupervised semantic correspondence using stable diffusion. In Thirty-seventh Conference on Neural Informa- fion Processing Systems, 2023. URL https::/ wtt.:e//oooommiii-ssooiiaaa
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12594 2022.
Ho, J., Jain, A., and Abbeel, Denoising diffusion probabilistic models. Advances neural 2020.
information processing systems,
Kakogeorgiou, Gidaris, S., Psomas, B., Avrithis, Y., Bursuc, A., Karantzalos, K., and Komodakis, N. What to hide from your students: Attention-guided masked image modeling. In ECCV, 2022.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, arXiv preprint arXiv:2001.0836) 2020.
11

---

Kordopatis- Zilos, G., Stojnić, V., Manko. A., Suma, P., Ypsilantis, N.-A., Efthymiadis, N., Laskar, Z., Matas, J., Chum, O., and Tolias, G. ILIAS: Instance-level image retrieval at scale, 2025.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assessing generative models. Advances neural information processing systems, 32, 2019.
Li, D., Ling, H., Kar, A., Acuna, D., Kim, S. W. Kreis, K., Torralba, A., and Fidler, S. Dreamteacher: Pretraining image backbones with deep generative models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 16698-16708, 2023a.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 144406.18366 11838, 2024.
Li, Z.,, Zhou, Q. Zhang, X., Zhang, Y., Wang, Y., and Xie, W. Open-vocabulary object segmentation with diffusion models. 2023b.
Lipman, Y., Chen, R. T. Q.. Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. The Eleventh International Conference on Learning Representations, 2023. URL http://ormmnddpplleea
btttss:/oppnnee
Liu, J., Hu, T., Sonke, J.-j., and Gavves, E. Beyond generation: Exploring generalization of diffusion models in few-shot segmentation. In Proceedings of the NewrIPS 2023 Workshop Diffusion cc/ Models, 2023. httpht//s//:eueeee wwterrips.cc.citttieic witp/viitttaaaaa 22////223/////// 74849 Poster.
Luo, G., Dunlap, L., Park, D. H., Holynski, A., and Darrell, T. Diffusson hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36:475000-475555
2023.
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden-Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, pp. 23 40, 2024.
Misra, I. and Maaten, L. d. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Mizrahi, D., Bachmann, R., Kar, O., Yeo, T., Gao, M., Dehghan, A., and Zamir, A. 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems, 36:58363- 58408, 2023.
Mukhopadhyay, S., Gwilliam, M., Agarwal, V., Padmanabhan, N., Swaminathan, A., Hegde, S., Zhou, T., and Shrivastava, A. Diffusion models beat gans on image classification, 2023.
Nash, C., Menick, J., Dieleman, S., and Battaglia, P. W. Generating images with sparse representations. arXiv preprint 113.iv21033.388/4 2021.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, volume 139, pp. 8162-8177 18-24 Jul 2021.
Noroozi, M. and Favaro. P. Unsupervised learning of visual representations by solving jigsaw puzzles. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), ECCV, pp. 69-844 84, 2016.
Oquab, M.. Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N.. Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J.. Labatut, P., Joulin, A., and Bojanowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. URL httptttss///peneee nett/forumP.--aGSSUBB..
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 4195-4205, 2023.
12

---

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G. Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICMI,, 2021a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. $748-8763, PmLR, 2021b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer- assisted wwwtvvettoooMMICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp- 234 241. Springer, 2015.
Salimans, T., Goodfellow, I., Zaremba, W. Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Tang, Z.,, Yang, 7.,, Zhu, C.,, Zeng, M., and Bansal, M. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36: 16083-16099, 2023.
Tian, K., Jiang, Y., Yuan, Z., Peng. B., and Wang. L. Visual autoregressivve modeling: Scalable image generation via next-scale prediction, arXiv preprint arXiv:2404.02905, 2024.
Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., Hénaff, O., Harmsen, J., Steiner, A., and Zhai, X. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprinl 2502. 14786, 2025.
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug-and-play diffusion features for text-driven image-to-imane translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
Van den Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Vincent, P. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674 2011. doi: 10.1162/NECO a 00142.
Xiang. W., Yang, H., Huang, D., and Wang, Y. Denoising diffusion autoencoders are unified self- supervised learners. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15802-15812, 2023.
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu, H. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9653-9663, 2022.
Yang, X. and Wang, X. Diffusion model as representation learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18938-189488 2023.
Yao, J., Wang, C., Liu, W., and Wang, X. Fasterdit: Towards faster diffusion transformers training without architecture modification. In NewrlPS, 2024.
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L.,, Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Hauptmann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion tokenizer is key to visual generation. In The Twelfth International Conference Learning Representations, 2024.
Yu, S., Kwak, S., Jang, H., Jeong, J.. Huang, J., Shin, J., and Xie, S. Representation alignment for generation: Training diffusion transformers easier than you think. ICLR, 2025.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, Sigmoid Loss for Language Image Pre- Pre-Tranning In ICCV, pp. 11941-11952, 2023.
13

---

Zhang, J., Herrmann, C., Hur, J., Polania Cabrera, L., Jampani, V., Sun, D., and Yang, M.-H. A tale of two features: Stable diffusion complements dino for shot semantic correspondence. Advances in Neural Information Processing Systems, 36:45533 45547, 2023.
Zhang, Q. Zhai, S., Bautista, M. A., Miao, K., Toshev, A., Susskind, J., and Gu, J. World-consistent video diffusion with explicit 3d modeling. arXiv preprint arx.::2412.01821, 2024.
Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., and Lu, J. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5729-5739, 2023.
Zheng, H., Nie, W., Vahdat, A., and Anandkumar, A. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.0933 2023.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICIR), 2022.
Zhu, R., Pan, Y., Li. Y., Yao, T., Sun, Z., Mei, T., and Chen, C. W. Sd-dit: Unleashing the power of supe-supe discrimination diffusion transformer. In CVPR, pp. 8435-8445, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We experimentally demonstrate the effectiveness of joint image-representttion generation in speeding up and improving conditional generative modeling in (Table and
Table 2. and unconditional generation in Table 3 The effect of Representation Guidance on generative performance is also validated in Table 4 and Table
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of our work Appendix E
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We present all the implementation details for experiments in section 4] Table 7 and Table 8
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Upon acceptance, we will release code publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. ct/ppi.cc guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: To the best of our knowledge, we have provided all the necessary hyperparam- eters, dataset splits, and experimental details fully reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Training large diffusion flow models with multiple random seeds exceeds our computational resources. This process is extremely essource-intensig and remains uncommon the diffusion research community.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We fully disclose the hardware used for our experiments in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: hitt::/nnesisppeee http://urippscccppllleeeeeeeeeee
Answer: [Yes]
Justification: We have read the code of ethics and made sure that the paper conforms to it in every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact of our work in Appendix F
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all models, datasets, and code implementations used to run our experiments and evaluate our models.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre /0tps//eec.ooooeeeeee////2////// for what should or should be described.
21

---

Appendix
Contents
A
ReDi with Stochastic Interpolant Models (SiT)
22
Stochastic Interpolant Models (SiT)
22
Joint Image-Representatiio Generation with Si1
22
Additional Implementation Details
23
Architecture details
23
B.2
Optimization details
23
B.3
Further implementation details
23
Detailed Benchmarks
23
D
Baseline Generative Models
24
Limitations Future Work
25
Broader Impact
25
Additional Qualitative Results
26
A ReDi with Stochastic Interpolant Models (SiT)
In the main paper, we introduced ReDi within the DDPM framework, as employed by DiT models. In this section, we begin with a brief overview of Stochastic Interpolant Models Ma et al. (2024) and then describe how ReDi can be applied in this setting.
A.1 Stochastic Interpolant Models (SIT)
Following flow-based models Lipman et al. (203), ), stochastic interpolants involve a continuous dime-dependent process transforming data distribution Xo p(x) into Gaussian noise NN I):
0x800 + 0,6, 0g =
og 0,
(14)
where and or are increasing and decreasing functions of t respectively.
Given this process, the marginal probability distribution px(x) of x, in 14 coincides with the distribution of the probability flow ordinary differential equation with velocity field:
*; v(xe; t).
(15)
The velocity field can be upproximated by neural network vo t) by minimizing the following training objective:
Lvelocity (0) = Exo,est vo (x(,) âz xo - δx E
(16)
A.2 Joint Image- Retresttprtnnnntiionn Generation with SiT
During training, given a VAE latent image and a visual representation we define joint interpolation process:
axXo + drEn, 2z 0:20 + orE2,
(17)
22

---

The model (g Zz, t) takes as input and 2;, along with timestep t. and jointly predicts the velocity for both inputs. Specifically, produces two separate predictions: vø (x,, Z,, for the image latent velocity and (x Re, () for the visual representation velocity Vz. The training objective combines both predictions:
Cjoint
[ (Xe, Zz, ai Xo - oi Ezl1 Aellvv Ze, t) Oe 20 - ox Ex e=li] (18)
Na zo.t
A 1, at and 1 - in our where 2₂ balances the velocity loss for 2,. By default, experiments.
B Additional Implementation Details
B.1 Architecture details
We present in Table 7 the configurations of the different-sized DiT and SiT models used in our experiments.
Table 7: Model configuration details. The configurations are the same for both DiT and SiT models.
MODEL SIZE
B/2
L/2
XL/2
Input Size
32
x 32
32 X 32 x 4
32 x 32
Patch Size
2
2
2
# Layers
12
24
28
# Heads
12
16
16
Hidden Dim.
768
1024
1152
B.2 Optimization details
We present in (Table the optimization hyperparameters used for all experiments presented in the paper.
Table 8: Optimization details. The optimization hyperparameters for both DiT and SiT models.
Batch Size
256
Optimizer
AdamW
LR
10-1
(31, B2)
(0.9, 0.999)
Computational Resources. For both training and sampling we use NVIDIA A100 40GB GPUs. Throughput, presented in Table 6lis measured single NVIDIA A100 40GB GPU with a batch size of 64 as the number of images generated per second using 250 sampling steps.
B.3 Further implementation details
ReDi with REPA experiment. To apply the Representation Alignment objective (REPA) top of ReDi we follow the implementation of (Yu et al.) 2025) and employ projection layer the Sth transformer layer. The projection is three-layer MLP with SiLU activations (Elfwing et al. 2018). The weight on alignment loss is 'REPA = 0.5.
C Detailed Benchmarks
We provide a detailed evaluation of the main experiments presented the main paper, including additional metrics and training iterations. Specifically, [Table 9 details the performance of the SiT-XL/2 ReDi models. Further Table 10 presents results for the ReDi with REPA (SiT-XL/2). For all models, use the evaluation metrics reported in the original publications.
23

---

MODEL
#ITERS.
FID
SFID
1St
PREC.
REC.
SiT-XL/2 Peebles & Xie 2023
7M
8.3
6.3
131.7
0.68
0.67
w/ ReDi
50K
56.1
18.9
23.8
0.44
0.47
w/ ReDi
100K
23.1
5.9
61.5
0.64
0.57
w/ ReDi
200K
12.6
5.7
97.3
0.69
0.61
w/ ReDi
300K
9.7
5.3
117.3
0.71
0.62
w/ ReDi
400K
7.5
5.1
129.5
0.72
0.62
w/ ReDi
4M
3.3
4.8
188.9
0.74
0.68
Table 9: Detailed evaluation for SiT-XL/2 w/ ReDi. All results are reported without classifier-free
guidance.
MODEL
#1TERS.
FID4
SFID
ISt
PREC.1
REC.
SiT-REPA-XL/2 Yu et al. 2025
400K
7.9
5.1
122.6
0.70
0.65
SiT-REPA-XL/
4M
5.9
5.7
157.8
0.70
0.69
w/ ReDi
50K
44.8
18.7
32.8
0.50
0.49
w/ ReDi
100K
15.2
5.6
85.3
0.68
0.59
w/ ReDi
200K
8.3
5.2
122.3
0.71
0.61
w/ ReDi
300K
6.3
5.1
140.6
0.73
0.62
w/ ReDi
400K
5.3
4.9
149.8
0.74
0.63
w/ ReDi
1M
3.5
4.64
177.9
0.75
0.69
Table 10: Detailed evaluation for ReDi with REPA. All results are reported without classifier-free guidance.
D Baseline Generative Models
We provide here a brief description of the baseline approaches presented in the main paper. Specifi- cally, consider (a) Autoregressive Models, (b) Latent Diffusson Models, and (c) REPA (Yu al. 2025 that also leverages visual representations to enhance generative performance.
(a) Autoregressive Models
(Tian al., 2024) proposes scalable generative framework that autoregressively predicts higher-resolutin image details from lesolurenol contexts across multiple scales.
MagViTv2 Yu et al. 2024) introduces a lookup-free quantization method enabling large vocabulary that able to improve the generation quality of autoregressive models.
MAR (Li al. 2024) proposes an autoregressive image generation framework that eliminates the need for vector quantization
(b) Latent Diffuusio Models
LDM (Rombach et al. 2022) proposes latent diffusion models, modeling the image distribu-
tion in compressed latent space produced by KL- or VO-regularized autoencoder.
U U-ViT-H/2 Bao al. (2023) proposes ViT-based (Dosovitskiy al. 2021 latent diffusion model that incorporates skip connections.
DiT Peebles & Xie (2023) proposes pure transformer backbone for training diffusion models and incorporates AdalN-zero modules.
MaskDiT (Zheng et al. 2023 trains diffusion transformers with an auxiliary mask recon- struction task
MDT GGo et al. (2023) introduce an effective mask latent modeling scheme, and design an asymmetric masking diffusion transformer.

---

SD-DiT (Zhu et al. 2024) extends the MaskDiT architecture by incorporating a discrimina- tion objective using momentum encoder.
2024 improves diffusion transformer training by moving from discrete diffusion to continuous flow-based modeling.
SiT (Ma et al.
FasterDiT Yao et al. 2024) incorporates supervision of the velocity direction into the denoising objective, significanlly accelerating the training process.
(c) Leveraging Visual Representations
REPA (Yu et al. 2025) aligns the representations of diffusion transformer models to the representations of self-supervised models.
E Limitations & Future Work
This section outlines some limitations of current work and highlights promising directions for future research.
Multiple visual representations. In this work. we demonstrate the effectiveness of jointly modeling the visual representations from DINOv2 during the diffusion process. A promising direction for future research is to investigate whether integrating multiple visual representations, each capturing different semantic or structural properties, can further boost generative performance.
Different dimensionality reduction approaches. We have shown that projecting visual representa- space with PCA effectively compresses visual features while retaining
tions into a sufficient information. An interesting direction for future work is explore more sophisticated compression techniques, such as training autoencoder, to better capture and retain the expressivity of these features.
Broader Impact
Generative models carry substantial risk of misuse. Their application can lead to various negative socictal impacts, most notably the spread of disinformation. Enhancements in generative performance, achieved by method, may further increase the realism of generated content, potentially making disinformation even more convincing.
25

---

G
Additional Qualitative Results
wo/ RG
W, 1.1
W, 1.2
Wr 1.3
1.4
1.5
Wt = 1.6
Figure 8: The effect of Representation Guidance. Samples from our DiT-XL/2 w/ ReDi model trained on ImageNet 256 256 for 400k steps with different Representation Guidance weights wy.
26

---

Figure 9: Uncurated generation results of SiT-XL/2 w/ ReDi. We use Classifier Free Guidance with w 4.0. Class label 88.
Figure 10: Uncurated generation results of SiT-XL/2 w/ ReDi. We use Classifier- Sriiffeer-Feeeeee Guidance
with w 4.0. Class label = 89.
27

---

Figure 11: Uncurated generation results of SiT-XL/2 w/ ReDi. We use Classifier- Free Guidance with w 4.0. Class label 207.
Figure 12: Uncurated generation results of SiT-XL/2 w/ ReDi. We use Classifier- Free Guidance with w 4.0. Class label 250.
28

---

Figure 13: Uncurated generation results of 2XL/2 w/ ReDi. We use Classifier- Free Guidance with w 4.0. Class label 417.
Figure 14: Uncurated generation results of X w/ ReDi. We use Classifier- Free Guidance
with w 4.0. Class label 555.
29

---

Figure 15: Uncurated generation results of 2LL2 w/ ReDi. We use Classifier- Guidance with w 4.0. Class label 928.
Figure 16: Uncurated generation results of SiT-XL/2 w/ ReDi. We use Classifier- Free Guidance
with w 4.0. Class label 933.
30


Reviews:
[{"review_id": "40x7OMyTf1", "reviewer": "Reviewer_UuCC", "summary": "This paper proposes a new technique, called ReDi, to improve the training and inference of generative diffusion models for images.\nThe paper places itself in the context of transformer latent diffusion.\nThe main idea of the paper is to learn to noise/denoise both the image latent and a secondary image representation derived from a pre-trained DINOv2 model.\nThe secondary image representation acts as regularizer like is commonly done in self-supervised learning.\nFurthermore this secondary objective yields an additional signal for guidance during inference which the authors coin adequately as representation guidance.\nExperiments on ImageNet256 demonstrate an improvement over REPA, in particular when NOT using Classifier-Free Guidance (CFG).\nWhen using CFG, there's little room for improvement in the first place for ImageNet256, but the proposed method still manages to show a little improvement for class-conditional generation and a significant one for unconditional generation.", "questions": "I do not have questions, my only suggestion would to dig deeper on trying various secondary representations to help uncover what a good representation should have.", "limitations": "I didn't see any discussion on limitations or potential negative social impact. \nI would assume that it shares those with generative modeling in general, e.g. that ultimately could allow to generate fake data that is indistinguishable from reality.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is easy to read and clear\n2. The method looks simple to implement\n3. The experiments show convincing gains over REPA\n4. Conceptually, I can see this applying to other domains than images but somehow this does require a (good) secondary representation to be available for the target domain.\n\nWeaknesses\n1. Experimental setting limited to ImageNet256\n2. Secondary representation limited to one choice (DINOv2), it would have been interesting to try several alternatives for secondary representations.\n3. Strength (4) raises the question of what is a good secondary representation, which in turn points back to weakness (2), this discussion is lacking in the paper \n4. Consequently, the paper feels more like an engineering a solution to natural image generation that probing the underlying workings of diffusion models.", "was_revised": true, "final_justification": "I am raising my score significantly as the points I raised were very satisfactorily answered, changing my initial perception of the paper.", "rebuttal": "We appreciate your insightful comments and efforts in reviewing our manuscript. Below, we provide our responses to each of your comments:\n\n**\"A little improvement for class-conditional generation\"**\n\nWe respectfully clarify that ReDi's improvements on class conditional generation *are significant both with and without CFG*, even though we agree with the reviewer's assessment that \"*there's little room for improvement in the first place for ImageNet256*\" in the CFG setting. We summarize the performance gains over the baseline (SiT) and REPA in what follows:\n\nAs presented in Table 2 ReDi reaches an FID of 1.72 at 350 epochs, surpassing both the baseline SiT and REPA trained for 1400 and 800 epochs, respectively. These results show that even in the CFG setting, ReDi accelerates convergence by **> x4** over SiT and by **> x2** over REPA.\n\n\n**W1. Experimental setting limited to ImageNet256**\n\nWe appreciate the reviewer’s concern regarding the applicability of our method to other datasets.  To address this, we conducted an additional experiment on Food101 dataset [1]. While this serves as a preliminary result in a smaller-scale setting, it provides valuable empirical evidence that ReDi’s significant improvements are not limited to ImageNet256.\n\n\n| **Model** | **Iter** | **FID** |\n|:---------------|:--: |:----------------:|\n|     SiT-B/2   | 400K|           14.5 |\n|         SiT-B/2 w/ ReDi   |400K |            9.2 |\n\n\nDue to our limited computational resources and the short timeframe of the rebuttal period, we were unable to conduct a large-scale experiment, e.g., on ImageNet512. However, we plan to include such experiments in the final version of our paper. \n\n\n**W2-W3. Alternative representations**\n\nWe thank the reviewer for their suggestion to explore alternative representations. \nAlthough we selected DINOv2 for its state‑of‑the‑art performance across a wide spectrum of vision tasks, we agree that evaluating other representations can provide additional insight. As such, we have already highlighted the potential for evaluating a combination of different representations in the \"Limitations and Future Work\" section (Appendix D).\n \nTo address this concern, we conducted some preliminary supplementary experiments using 2 alternative representations, namely MAE and JEPA, with different number of principal components (#PC). Given the time constraints of the rebuttal period, a more thorough exploration was not possible. However, we plan to include additional alternative representations, such as CLIP, SIGLIP, and MOCOv3, in the final version of our paper.\n\nFor all experiments, we use SiT-B/2 as the base model and train for 400K steps. We observe that using DINOv2 outperforms the other self-supervised encoders. This result is consistent with the ablation conducted by  REPA [2]. \n\n **Model** | **Visual Representation** | **#PC** | **FID** |\n|:---------------|:--: |:--: |:----------------:|\n|   SiT-B/2            | -|- |           33.0 |\n|   SiT-B/2 w/ ReDi    |  MAE    | 8|            28.5 |\n|   SiT-B/2 w/ ReDi    |  MAE    | 12|          25.4|\n|   SiT-B/2 w/ ReDi    |  JEPA    | 8|            29.3 |\n|   SiT-B/2 w/ ReDi    |  JEPA    | 12|          25.7|\n|   SiT-B/2 w/ ReDi    |  DINOv2    | 8|         21.4 |\n\n\n\n**W4. Engineering a solution** \n\nWe respectfully disagree with the reviewer's assessment of our method as \"an engineering solution for natural image generation\". \n \n \nModeling the joint distribution of precise low-level features (via the VAE latents) and semantic high-level features (via DINOv2) is well motivated by the recent literature [2] that established a connection between representation learning and generative performance. \n\nFurther, we emphasize that *we did not apply any engineering tricks* during training (e.g., adjusting LR, optimization parameters, etc.) nor any changes in the standard diffusion transformer architecture (e.g., RMSNorm [3], SwiGLU [4], and RoPE [5]). Thus, our work presents a fundamental finding and not an engineering solution: integrating image latents and semantic representations within one probabilistic model demonstrably simplifies and improves diffusion‑based generation.\n\n**References**\n\n[1]  Food-101–mining discriminative components with random forests. In\nECCV, 2014\n\n[2] Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025\n\n[3] Root mean square layer normalization. In NeuRIPS 2019\n\n[4] Glu variants improve transformer. In Arxiv 2020\n\n[5] Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 2024"}, {"review_id": "68RBy5P56I", "reviewer": "Reviewer_NELV", "summary": "- This paper presents a joint diffusion training approach using low-level VAE representations and high-level DINOv2 representations, called **ReDi**.\n- The joint training significantly improves the resulting diffusion model across the board—for both conditional and unconditional generation. The results are in the same vein as REPA (Yu 2024), but ReDi achieves even greater improvements. Additionally, joint training speeds up convergence substantially.\n- The paper also introduces a sampling-time self-guidance trick called **Representation Guidance**, where the guidance signal comes from the denoising high-level DINOv2 representation. This signal is amplified in a way similar to classifier-free guidance (CFG), and the method effectively narrows the performance gap between conditional and unconditional generation.", "questions": "- I’m personally curious about *why and how* jointly modeling “more stuff” (like DINO, which should make the task harder) actually leads to **faster convergence**. More broadly, I’d like to understand how joint modeling in general helps convergence.\n    - To answer these questions, it would be useful to test different joint representation choices. For example, one potential baseline is to jointly model with the **class label** itself.\n        - My hypothesis is that class labels might outperform DINO representations (though I could be wrong). If they do, it might suggest that the advantage comes from the sharpness of the conditioning signal.\n        - This would also shed light on why Representation Guidance helps: it could serve as a “sharpened” signal that accelerates denoising, akin to conditional generation. While it doesn’t provide a strong condition from the beginning, it offers a slightly stronger guide during sampling, enhancing overall generation quality.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The improved convergence speed and convergence quality have practical significance. Achieving **23× faster convergence** and **cutting FID in half** is a very strong result—this alone is worthy of publication.\n- In addition to the substantial FID improvement, the idea of Representation Guidance raises intriguing research questions (see Weaknesses).\n\n**Weaknesses:**\n\n- There is limited investigation into *why and how* joint modeling with DINO representations leads to improvements. Although REPA observed that aligning a diffusion model’s representation with DINO helps, a deeper mechanistic understanding is lacking. ReDi’s approach is also structurally different from REPA, which warrants a fresh explanation.\n    - Relatedly, Figure 7 shows that increasing the number of DINO channels can degrade performance. The paper suggests this happens because the model devotes too much capacity to modeling the DINO representation. While plausible, the experiments may not be well-controlled enough to support that conclusion. For instance, the loss weighting should be adjusted to maintain a constant ratio between the VAE and DINO losses. Without this control, it’s difficult to draw strong conclusions.\n    - Even with such control, an open question remains: different DINO channels may encode fundamentally different types of information, and perhaps only some of that information is beneficial for diffusion. The paper too quickly attributes the observed degradation to limited capacity, without adequately exploring whether the nature—not just the volume—of the DINO information plays a role.\n- There is also limited analysis of why **Representation Guidance** (a form of self-guidance) is effective. It’s curious that a signal generated by the model itself—when amplified—can lead to better diffusion outcomes. Understanding this phenomenon might uncover important new insights.", "was_revised": true, "final_justification": "I think this is a strong paper with solid results and good experimental design. The gains are meaningful (especially the faster convergence and improved FID), and the idea of Representation Guidance is interesting and opens up a lot of directions.\n\nI had some questions about why joint modeling helps and how RG works, and while those are still open to some extent, the authors’ rebuttal and extra experiments helped clarify things. I found the responses reasonable and convincing overall.\n\nThere’s definitely more to explore here, but the current version already stands on solid ground. I’m keeping my score at 5: Accept.", "rebuttal": "We sincerely appreciate the reviewer's thoughtful questions about the fundamental mechanisms behind our approach's success. While these open-ended inquiries deserve careful consideration, providing comprehensive answers within the rebuttal period is difficult. For now, we share below some additional experiments, our current perspectives and note that these questions inspire valuable future research directions - particularly a deeper theoretical or mechanistic analysis of the ReDi's joint diffusion mechanisms.\n\n**W1.1 Why and how joint modeling leads to improvements.**\n\nWe agree with the reviewer's assessment that the significant performance improvement demonstrated by ReDi raises interesting research questions regarding the underlying mechanisms of our method. \n\nBoth our work and REPA are motivated by the observation that diffusion models tend to learn features that lack the depth and versatility seen in self-supervised approaches like DINOv2. Our hypothesis is that this limitation arises because the denoising objective does not differentiate between meaningful semantic information and irrelevant image details.\n\nREPA showed that aligning the DiT representations with DINOv2 can mitigate the deficiencies of the denoising objective, enabling the model to learn stronger representations. This resulted in significant improvements, indicating that learning high-level semantic representations is essential for generative modeling.\n\nThis observation directly motivates our joint modeling approach. Rather than relying on distillation, ReDi simultaneously learns stronger semantic representations by learning to denoise the first principal components of  DINOv2, which encode the high-level semantic information of the image.\n\n\n**W1.2 Loss weight and #PC**\n\nWe thank the reviewer for their suggestion to further explore a more optimal weighting between ReDI's  VAE and DINOv2 denoising losses. We agree that such an exploration is an interesting direction that we plan to do in future research. As an initial step in this direction, we conduct an experiment with PC=16 and $\\lambda_z=0.5$. Additionally, we conduct an experiment where we use the 8 components *after the first 8 PCs* i.e. $z[8:16]$. We highlight these new experiments with **bold** in the table below: \n **Model** | **PC** | **$\\lambda_z$** | **FID** |\n|:---------------|:--: |:--: |:----------------:|\n|   SiT-B/2            | -|- |           33.0 |\n|   SiT-B/2 w/ ReDi    |  0:8    | 1|            25.7 |\n|   SiT-B/2 w/ ReDi    |  0:16    | 1|          29.1|\n|   **SiT-B/2 w/ ReDi**    |  **0:16**    | **0.5**|            **31.5** |\n|   **SiT-B/2 w/ ReDi**    |  **8:16**    | **1**|            **25.5** |\n\nThese results show that although the 8 components after the first 8 PCs ($z[8:16]$) contain useful information and match the performance of the first 8 PCs ($z[0:8]$), modeling both together negatively impacts performance. This holds *even when we adjust the loss weight to $\\lambda_z=0.5$*. While we agree that a deeper exploration is necessary, this preliminary result supports our explanation for the performance degradation observed as the number of DINO channels increases.\n\n\n**W1.3 Different DINO channels may encode fundamentally different types of information.**\n\nWe once again thank the reviewer for their suggestion to explore *which* principal components (PCs) are most beneficial for ReDi. Indeed, different PCA components encode different types of information. Specifically, the first PCs capture low-frequency global structures, while the last PCs capture high-frequency details of DINOv2. Thus, we agree that determining whether low-frequency, high-frequency, or a combination of both is most beneficial for joint modeling is an interesting question.\n\nIn this direction, we conducted two additional experiments (**in bold**) where we fixed the number of PCs to 8 and varied their position, with the 1st PC indexed at 0:\n **Model** | **#PC** | **Position** | **FID** |\n|:---------------|:--: |:-- |:----------------:|\n|   SiT-B/2            | -|- |           33.0 |\n|   SiT-B/2 w/ ReDi    |  8    | 0:8 (first)|            25.7 |\n|   **SiT-B/2 w/ ReDi**    | **8**   | **760:768 (last)**|           **29.4**|\n|   **SiT-B/2 w/ ReDi**    |  **8**   | **384:392 (middle)**|           **30.1** |\n\nThese results indicate that the low-frequency semantic information is the most beneficial for ReDi. \n\n\n\n**W2. Why Representation Guidance (a form of self-guidance) is effective?**\n\nThis is indeed an interesting observation, our intuition is that  Representation Guidance (RG) can be viewed through the same framework as Classifier‑Free Guidance (CFG):\n\n - In CFG $\\nabla \\log p(x_t|c) -\\nabla \\log p(x_t)$ acts as the gradient of an implicit classifier (amplifying the likelihood of $p(c | x_t)$), producing images that follow the condition $c$. \n -  In RG $\\nabla \\log p(x_t,z_t) -\\nabla \\log p(x_t)$ acts as the gradient of an implicit feature extractor  (amplifying the likelihood of $p(z_t | x_t)$), producing images that follow the (noisy) semantic structure $z_t$ at each step $t$.\n\nWe will highlight this connection more explicitly in the final version of the paper.\n\n\n\n**Q1. Why and how jointly modeling “more stuff” (like DINO, which should make the task harder) actually leads to faster convergence?** \n\nMore generally, jointly modeling multiple modalities has been systematically shown to benefit image and video generative models, offering strong scalability properties. We discuss such methods in the *Multi-modal Generative Modeling* paragraph of our Related Work section.\n\nSpecifically for ReDi, we have already analyzed our hypothesis for the observed performance gains in response to W1.1.\n\n**Q2. Would jointly modeling the class label outperform DINO?** \n\nWe believe that the reviewer raises an interesting question. As an initial experiment, we attempted to encode the class label as a one-hot vector and jointly generate it as a separate token. However, we observed that this straightforward implementation is not successful. Specifically, the model was unable to learn the discrete \"class label\" distribution (potentially due to the denoising objective), and the training loss increased over time rather than converging. Still, we consider this an interesting direction, and further experimentation beyond the rebuttal period is needed."}, {"review_id": "cVMBY3rbOU", "reviewer": "Reviewer_wr8D", "summary": "The paper proposes to jointly model the image latents and semantic features (e.g., DINO) within the same diffusion process. During inference, it introduces representation-guided sampling to leverage the semantic features. Experimental results show that the proposed ReDi improves generative quality for both DiT and SiT, and achieves faster convergence during training.", "questions": "- How about the speed? The sampling speed might be slower than the baselines due to the extra prediction of the DINO feature?", "limitations": "Yes, limitation of impact discussed in Appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## **Strengths**\n- The motivation of the proposed method is well-explained. In the introduction it discusses the gap in discriminative capabilities between LDMs and representation learning methods. \n- Compared to the existing work with similar motivation  e.g., REPA that achieves the semantic alignment via distillation, the paper proposes a new direction that encourages diffusion models to explicitly learn the joint distribution of the VAE latents and semantic features. It could offer new insights to the research community.\n- The experiment results show the superiority of the proposed method\n\n## **Weaknesses**\n- There is no analysis of the generated DINO features. For example, it would be good to demonstrate their discriminative capabilities via linear probing results. In Figure 5, there are qualitative examples of the generated DINO features; however, it would be better to compare them with the ground truth DINO features as well.\n- There is no experimental result with higher resolution (512 x 512) input. It would be good to show the proposed method can also effectively synthesize the semantic features and improve the generative quality of higher resolution inputs.", "was_revised": true, "final_justification": "After reading the rebuttal, I would like to raise my score to accept as the authors addressed my main concerns. Though some additional results were promised but not included in the rebuttal, the provided materials convinces me that the paper could make a solid contribution to the community.", "rebuttal": "We appreciate your insightful comments and efforts in reviewing our manuscript. Below, we provide our responses to each of your comments:\n\n**W1.1 Quantitative analysis of the generated DINO features**\n\nWe appreciate the reviewer’s suggestion to demonstrate the discriminative capabilities of ReDi's generated features. Although the primary aim of this work is to improve generative image modeling, we agree that this is an interesting experiment. \n\nSince we operate on only the first 8 Principal Components (PC) of DINOv2 features, both ground-truth and generated representations have reduced discriminative capabilities, making linear probing less effective. Instead, we use Attentive Probing [1], which is more suitable for this low-dimensional setting.\n\nOur evaluation setup is the following: We first train an Attentive Probing Head on 8-PC DINOv2 features from ImageNet. Then compare the classification accuracy on 50K (50 for each ImageNet class) generated samples between:\n- **Ground-Truth Features:** PCA-reduced features extracted from generated images with DINOv2.\n- **Generated Features:** Features directly generated by ReDi.\n\nWe employ SiT-XL w/ ReDi trained for 4M iterations and use Classifier-Free Guidance as in Table 2, to generate the samples.\n\n **Features** | **Accuracy** |\n|:---------------|:----------------:|\n|   **Ground-Truth Features**            |           90.5 |\n|   **Generated Features**            |           81.2 |\n\n\nWe note that the top-1 accuracy is high for both the ground truth and the generated features, since they are produced in a class-conditioned setting with CFG. This pushes the generated samples to align with the condition, thereby limiting diversity. As such, these results are not directly comparable to those typically reported in representation learning papers, such as DINOv2.\n\nThe results demonstrate that ReDi’s generated features largely preserve the discriminative capacity of the ground truth DINOv2 features extracted from generated images.\n\n**W1.2 Visual comparison between ground truth and generated DINO features**\n\nWe appreciate the reviewer’s suggestion to include a visual comparison between ground truth and generated DINO features. Because the NeurIPS 2025 rebuttal guidelines do not allow new figures to be uploaded at this stage, we cannot include the requested visualization in the rebuttal. We will include the suggested visual comparison in the final version of our paper.\n\n**W2. Higher Resolution (512 x 512)**\n\nWe agree with the reviewer that it would be interesting to validate the effectiveness of ReDi in improving the generative quality of higher-resolution images. However, due to our limited computational resources and the limited time frame of the rebuttal period, we were unable to conduct an experiment on ImageNet512. However, we plan to include such experiments in the final version of our paper. \n\n\n**Q. How about the speed?**\n\nWe respectfully emphasize that we have explicitly measured the sampling speed of ReDi in terms of throughput (generated images per second) and reported the results in Table 5. The Merged Tokens strategy (our standard approach), which preserves the original sequence length, achieves a throughput of 4.51, while Vanilla DiT-B/2 achieves 4.52. This demonstrates that ReDi (MR) maintains the same computational cost (99.78%) as Vanilla DiT-B/2.\n\n**References**\n\n[1] Scalable pre-training of large autoregressive image models. In ICML'24"}, {"review_id": "VJ2g2eoYb3", "reviewer": "Reviewer_8DbC", "summary": "This paper proposes a novel generative paradigm named joint image-feature synthesis ReDi. Inspired by REPA , this paradigm first performs understanding (feature synthesis) followed by synthesis (image synthesis) within the backbone's forward pass. The significance of this method lies in two aspects: (1) Its novelty may provide valuable insights for designing future unified understanding and synthesis algorithms , and (2) it has the potential to enhance the model's generalization capabilities and training speed.", "questions": "No.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**:\n\n1. The greatest strength of this paper lies in its highly interesting and novel algorithm. Additionally, its proposed representation guidance also offers valuable insights.\n\n2. This paper is written in a very logical manner and the experiments are adequate and sound.\n\n3. The experimental results are outstanding; ReDi outperforms REPA in both model performance and training speed.\n\n**Weakness**:\n\n1. The paper lacks some analytical experiments, such as: the FID change curve during training, a visualization of the \"synthesized\" representation's evolution during the sampling process, and a comparison of the \"synthesized\" representations between REPA and ReDi to better explain why ReDi performs better.\n\n2. In addition to DINOv2, the authors should explore other representations, such as MAE, LLM Embeddings (e.g., Gemma and T5 Embedding) corresponding to prompts, or even representations from a REPA pre-trained diffusion model.\n\n3. The statement \"This suggests an optimal intermediate subspace where compressed visual features retain sufficient expressivity to guide generation without dominating model capacity\" only provides an intuitive explanation of the experimental results. Is there a more in-depth explanation regarding Figure 7?", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and efforts in reviewing our manuscript. Below, we provide our responses to each of your comments:\n\n**W1. Lack of some analytical experiments**\n\n- **FID change curve during training.** \n\nWe respectfully emphasize that in Figure 2, we demonstrate the progression of FID during training for both DiT-XL w/ ReDi and SiT-XL w/ ReDi, and compare with the vanilla DiT/SiT and REPA.\n- **Visualization of the \"synthesized\" representations sampling process.**\n\n We appreciate the reviewer’s suggestion. Because the NeurIPS 2025 rebuttal guidelines do not allow new figures to be uploaded at this stage, we cannot include the requested visualization in the rebuttal. We will, however, add a figure illustrating how ReDi’s synthesized representations evolve over the course of inference in the camera‑ready version.\n- **Comparison of the \"synthesized\" representations between REPA and ReDi.**\n\nWe appreciate the reviewer’s suggestion to include a comparison between the features generated by ReDi and REPA. We will include a visual comparison of the generated features from both methods in the final version of our paper.\n\n\n\n**W2. Alternative Representations**\n\nWe thank the reviewer for their suggestion to explore alternative representations. \nAlthough we selected DINOv2 for its state‑of‑the‑art performance across a wide spectrum of vision tasks, we agree that evaluating other representations can provide additional insight. \n \nTo address this concern, we conducted some preliminary supplementary experiments using 2 alternative representations, namely MAE and JEPA, with different number of principal components (#PC). The additional experiments suggested by the reviewer, such as LLM embeddings corresponding to prompts and representations from a REPA, are not trivial to implement within the short rebuttal timeline. However, we agree that these would be interesting to explore, and we plan to include such experiments in the final version of our paper.\n\nFor all experiments, we use SiT-B/2 as the base model and train for 400K steps. We observe that using DINOv2 outperforms the other self-supervised encoders. This result is consistent with the ablation conducted by  REPA. \n\n **Model** | **Visual Representation** | **#PC** | **FID** |\n|:---------------|:--: |:--: |:----------------:|\n|   SiT-B/2            | -|- |           33.0 |\n|   SiT-B/2 w/ ReDi    |  MAE    | 8|            28.5 |\n|   SiT-B/2 w/ ReDi    |  MAE    | 12|          25.4|\n|   SiT-B/2 w/ ReDi    |  JEPA    | 8|            29.3 |\n|   SiT-B/2 w/ ReDi    |  JEPA    | 12|          25.7|\n|   SiT-B/2 w/ ReDi    |  DINOv2    | 8|         21.4 |\n\n\n\n\n**W3. A more in-depth explanation regarding Figure 7 (Principal Components Ablation)**\n\nWe agree with the reviewer that further exploration is needed to better explain the performance degradation observed when increasing the number of DINOv2 PC's. To support our claim that this degradation occurs due to the model disproportionately allocating capacity to predict the visual representations at the expense of image latents, we conduct two additional experiments.\n\nOne experiment with PC=16 and $\\lambda_z=0.5$ and additionally an experiment where we use the 8 components *after the first 8 PCs* i.e. $z[8:16]$. We highlight these new experiments with **bold** in the table below: \n\n **Model** | **PC** | **$\\lambda_z$** | **FID** |\n|:---------------|:--: |:--: |:----------------:|\n|   SiT-B/2            | -|- |           33.0 |\n|   SiT-B/2 w/ ReDi    |  0:8    | 1|            25.7 |\n|   SiT-B/2 w/ ReDi    |  0:16    | 1|          29.1|\n|   **SiT-B/2 w/ ReDi**    |  **0:16**    | **0.5**|            **31.5** |\n|   **SiT-B/2 w/ ReDi**    |  **8:16**    | **1**|            **25.5** |\n\nThese results show that although the 8 components after the first 8 PCs ($z[8:16]$) contain useful information and match the performance of the first 8 PCs ($z[0:8]$), modeling both together negatively impacts performance. This holds true *even when we adjust the loss weight to $\\lambda_z=0.5$*.  These experiments support our explanation for the performance degradation observed as the number of DINO channels increases."}]

Author rebuttal:

Meta-review: This paper enhances generative modeling by jointly modeling image latents and high-level DINO features, which significantly improves generative quality and training efficiency. The idea is well-motivated, the overall method is technically solid, and the results demonstrate clear improvements in faster convergence and reduced FID. Reviewers found the experimental design convincing, providing sufficient evidence of the method's impact and reproducibility. With unanimous support from the reviewers, the consensus is to accept the paper. AC agrees with this consensus and recommends accepting the paper.
Decision: Accept (spotlight)