There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-Training
Authors: Jiachen Lei
Keli Liu
Julius Berner
Y HoiM
Hongkai Zheng
Jiahong Wu
Xiangxiang Chu
Venue: ICLR 2026 Poster (2026)
Keywords: Image Generation
Diffusion models
Pixel-space generation
Primary area: generative models

Abstract: Pixel-space generative models are often more difficult to train and generally underperform compared to their latent-space counterparts, leaving a persistent performance and efficiency gap. In this paper, we introduce a novel two-stage training framework that closes this gap for pixel-space diffusion and consistency models. In the first stage, we pre-train encoders to capture meaningful semantics from clean images while aligning them with points along the same deterministic sampling trajectory, which evolves points from the prior to the data distribution. In the second stage, we integrate the encoder with a randomly initialized decoder and fine-tune the complete model end-to-end for both diffusion and consistency models. Our framework achieves state-of-the-art (SOTA) performance on ImageNet. Specifically, our diffusion model reaches an FID of 1.58 on ImageNet-256 and 2.35 on ImageNet-512 with 75 number of function evaluations (NFE) surpassing prior pixel-space methods and VAE-based counterparts by a large margin in both generation quality and training efficiency. In a direct comparison, our model significantly outperforms DiT while using only around 30\% of its training compute. Furthermore, our consistency model achieves an impressive FID of 8.82 on ImageNet-256, significantly outperforming its latent-space counterparts. This marks the first successful training of a consistency model directly on high-resolution images without relying on pre-trained VAEs or diffusion models. Our codes are available at: \href{https://github.com/AMAP-ML/EPG}{https://github.com/AMAP-ML/EPG}

Published as a conference paper at ICLR 2026
THERE IS No VAE: END-TO-END PIXEL-SPACE GEN- ERATIVE MODELING VIA SELF-SUPERVISED PRE- TRAINING
Jiachen Lei"
Keli Liu'
Julius Berner
Haiming Yu²
Hongkai Zheng"
Jiahong Wu
Xiangxiang Chu
AMAP, Alibaba Group, 2 Caltech
ABSTRACT
Pixel-space generative models are often more difficult to train and generally derperform compared to their latent-space counterparts, leaving a persistent per- formance and efficiency gap. In this paper, we introduce novel two-stage training framework that closes this gap for pixel-space diffusion and consistency models. In the first stage, we pre-train encoders to capture meaningful semantics from clean images while aligning them with points along the same deterministic sam- pling trajectory, which evolves points from the prior to the data distribution. the second stage, integrate the encoder with randomly initialized decoder and fine-tune the complete model end-to-end for both diffusion and consistency models. Our framework achieves state-of-the-art (SOTA) performance geNet. Specifically, our diffusion model reaches an FID of 1 1.58 on ImageNet-256 and 2.35 on ImageNet-512 with 75 number of function evaluations (NFE) sur- passing prior pixel-space methods and VAE-based counterparts by a large margin both generation quality and training efficiency. a direct comparison, our model significantly outperforms DiT while using only around 30% of its train- ing compute. Furthermore, our consistency model achieves an impressive FID of 8.82 on ImageNet-256, significantly outperforming its latent-space counterparts. This marks the first successful training of a consistency model directly on high- resolution images without relying pre-trained VAEs diffusion models. Our codes are available at: https://github.co wwwp:/ttuuboo//MMaaaaa Swa.m/MAPPMMLEEEB
Ima-
1.5
JT-GG16
2.0
SIT XL/2 MasKDNT
UMIT2222
FID-50K
3.5
4.0
2
2/7
22
Inference compute (GFLOPs, log scale)
Figure 1: (Left) Our diffusion model achieves SOTA performance while maintaining significant inference efficiency. The z-axis indicates the log-scaled GFLOPs for generating an image. The bubble size denotes number of model parameters. (Right) Images generated by our diffusion model.
"Corresponding Author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Diffusion- based generative models have become the cornerstone of modern image synthesis. This family includes both traditional, iterative diffusion models (Ho al., 2020; Song et. 2021; Karras et al., 2022) and more recent one/few-shot generators like consistency models (Song et al., 2023; Geng et al., 2025; Lu & Song, 2025) and flow maps (Boffi et al., 2025). Much of their success on high-resolution tasks relies training in compressed latent space (Rombach et al., 2022) of pre-trained VAES (Kingma & Welling, 2013). However, this reliance on VAEs introduces its own set of significant challenges. Firstly, training the VAE itself is difficult due to the need to balance compression with figh-fidelity reconstruction. Besides, even when trained properly, the VAE often produces imperfect reconstructions for latents far from the training set. While pre-training the VAE on massive datasets can mitigate this, it still induces a permanent performance bottleneck, as the generative model's 's ability to adapt to new data is always limited by the VAE's fixed capacity.
To simplify the overall training pipeline and bypass the performance bottleneck of VAEs, numerous works (Jabri al., 2023; Hoogeboom al., 2023; Ho al., 2021; Dhariwal & Nichol, 2021) have explored training diffusion models directly on raw pixels, developing specialized model architec- tures (Jabri al., 2023) or training techniques (Hoogeboom et al., 2023). Despite these efforts, none have achieved training performance and inference efficiency comparable to VAE-based meth- ods primarily due to the high computational cost originating from the model backbone or the slow convergence rate, representing the two major challenges of operating in pixel space.
In our work, we aim at addressing these challenges to bridge the performance and efficiency gap between pixel-space training and its latent-space counterparts. We take inspiration from self- supervised learning (SSL) approaches: the encoder served as general visual semantic learner and decoder spec-specific prediction head (He et al., 2021: Chen al., 2021; Chen & He, 2020). This decomposition significantly improves training efficiency and model performance in downstream tasks. Motivated by this, we hypothesize that the encoder-decoder role in diffusion- models can also be decomposed in a similar way. Specifically, we argue encoders in diffusion-basse generative models primarily learn high-level visual semantics from noisy inputs and the decoders act as low-level pixel generators conditioned on encoder representations.
generative
As a result, we decompose the training paradigm into two separate stages as in SSL. During pre- training. the encoder captures meaningful visual semantics from images at varying noise levels. Note that this is different from the Gaussian noise data augmentation utilized in SSL, as the noise level follows pre-defined diffusion schedules and is much higher than the one used in augmentation. Subsequently, after pre-training, the decoder is fine-tuned end-to-end with the encoder under task- specific configurations and predicts pixels given representations from the encoder. This framework endows the model with solid discriminative capability at the start of training, while offering a more flexible way for it to adapt learned representations to contain detailed visual semantics, which is preferable in generative tasks. However, designing such a pre-training method is non-trivial, as typical visual learning methods struggle in learning meaningful semantics from images with larger noise levels. In particular, directly utilizing SSL as a pre-training method is ineffective, due to its representation collapse on images of strong noise.
To address this, by extending the idea of rRCM (Lei et al., 2025), we pre-train encoders to capture visual semantics from clean images while aligning them with corresponding points on the same de- terministic sampling trajectory, which evolves points from pure Gaussian to the data distribution. In practice, this is realized by matching images with shared noise but different noise levels as in con- sistency tuning (Song et al.. 2023). This pre-training approach reformulates representation learning on noisy images as a generative alignment task, connecting features of noisy samples to their pro- gressively cleaner versions. Subsequently, given the pre-trained encoders, we fine-tune it alongside a randomly initialized decoder in an end-to-end manner under task-specific configurations.
To verify the effectiveness of our method, we conduct thorough experiments on ImageNet dataset without relying on any external models. We focus on two distinct downstream generative tasks: training diffusion and consistency models. Built upon pre-trained weights, our diffusion model achieves a SOTA FID of 1.58 on ImageNet-256 and 2.35 on ImageNet-512 with 75 NFEs, sur- passing prior and concurrent pixel-space methods by large margin in both generation quality and efficiency. Besides, we close the gap with the dominant latent-space paradigm and achieve perfor- mance that surpasses leading latent diffusion counterparts, such as DiT (Peebles & Xie, 2023) and
2

---

Published as a conference paper at ICLR 2026
SiT (Ma et al., 2024), while using approximately 30% of its training compute. Moreover, sistency model achieves a remarkable FID score of 8.82 in a single generation step, demonstrating superior performance and training efficiency compared to the latent-space counterparts. This result, to the best of our knowledge, marks the first time a consistency model has been successfully trained directly on ImageNet-256 without utilizing pre-trained VAEs or diffusion models. While rely on an SSL pre-training, this is significantly more efficient and shows a promising way of stabilizing consistency model training.
Our contributions can be summarized as follows:
i. We propose a novel training framework that enables efficient, performant, and scalable pixel- space generative modeling in high resolution. By identifying the semantic roles of the encoder and decoder, we establish that training a diffusion model can be framed as a self-supervised learning problem, similar training an image classifier. Moreover, our empirical results vali- date that the key to successful pixel-space generation is the high-quality semantic representation that is temporally consistent across noise levels.
ii. Our End-to-end Pixel-space Generative model (EPG) demonstrates state-of-the-ar results for pixel-space generation on ImageNet dataset. For the first time, it closes the performance and efficiency gap with leading VAE-based counterparts. Besides, to the best of our knowledge, are the first achieve strong generation performance ImageNet-256 with consistency models directly trained in pixel space and without relying on pre-trained VAEs or diffusion models.
iii. terms of both GFLOPs and training time, our model, built on Vision Transformer (Vaswani et al., 2017), maintains significant computational efficiency across different resolutions. This is achieved by fixing input token length through proportionally adjusting the patch size as image resolution increases e.g., 16x 16 on ImageNet-256 and x2x32 on ImageNet-512.
BACKGROUND
In this section, we introduce necessary backgrounds on diffusion models and consistency models. Let Pdata (x) denote the probability density of target data æ, (x) the probability density of att), (r) (a)) Diffusion models (DM) approximate Pdata by learning to reverse a pre-defined forward stochastic differential equation (SDE) that gradually transports points =00) from Pdtta (*) to a prior distribution. The corresponding reverse SDE generate samples by evolving points from the prior distribution back to the data distribution, guided by the score function Vx log Pr()). To estimate the score, diffusion models train a neural network so (=x(t), t) using the objective (Karras et al., 2022)
arg min LE A(t) (|se(x((t) (0))
Here, A(t) is a scalar weighting function. In our work, following EDM (Karras et al., 2022), we define the forward process as dæ = /tdw., E [0, [0, = 80. w is the standard Wiener process. The corresponding reverse SDE can be depicted by: dx = 2tV log (x()) + 22fdw with w being a standard Wiener process that flows backwards in time. A key property of the reverse SDE is the existence of the probability flow (PF) ordinary differential equation (ODE) that shares the same marginal distribution (x(
x log (z().. Let t)) denote the solution of equation from 0
da Vv
f(x(t),)) x(t) /. -
V log (x)
Diffusion models generate samples by solving equation 3 numerically over discrete time intervals. As a result, it requires multiple function evaluations, incurring heavy computational cost.
Consistency models (CM) (Song et al., 2023) address this sampling inefficiency through directly approximating the trajectory end point f(x(t), t) with a neural network fo(a(t), t), which is trained by enforcing the self-consistency condition: fo (x))) fo(#(s), s), 1, E 00, T). Note that a=())
3

---

Published as a conference paper at ICLR 2026
Stop grad
Predicted Xt,
Compute Loss
Projector lg-
Projector Agg
Decoder Dg
Align
Step
Festurcs
Encoder Rar
Ficooder Fi
Encoder Equa))
Encoder
(Pay Rigus? Temporally adjacent data pains
Add noise
Augmented data pairs
(a) Pre-train encoder Eg.
(b) Simple end-to-end fine-tuning.
Figure 2: Overview of our framework. During pre-training. we train encoder by learning visual semantics (blue branch) and aligning them along ODE sampling trajectories (yellow branch). A- is the exponential moving average of 0, and sg is the stop gradient operation. After pre-training, discard the projector and train Eg along side a randomly initialized decoder Do in end-to- end manner under specific generative training settings (e.g., denoising or consistency training). The encoder Ee maps noisy images into features, which are then reconstructed into clean pixels by the decoder Dg. In our work, we choose vision transformer as our backbone.
and (s)) lie on the same PF ODE trajectory and fo satisfies the boundary condition: fo(xx00)0) 2x (0)). To ensure this, prior works parameterize fe as
fo (t), t) (f)=(() + Cout (t)Fo(x(t).)) t),
(4)
where Cakip(t) (t) and (t) are scalar functions and satisfy (0) = 1, Coat (0) = 0. CMs discretize the time horizon into N N-1 non-overlapping time intervals { J in E [amin T), and optimize
the following consisiency training objective, which minimizes metric between adjacent points on the PF ODE sampling trajectory
arg min E ( bn), Seg(o) (8t..)1))
(5)
where d is a distance metric, sg means stop gradient operation, It,, It,-, are adjacent points on the same PF ODE trajectory. During training, Lt, is sampled from the forward SDE given clean image Ttg- Lt,, 1
is sampled by following equation 2, yet the score at æt., is necessary but unknown during training. To address this, one can either utilize a pre-trained diffusion model, or leverage the unbiased estimator -E| -B|"===aa [x]] (Song et al., 2023) to approximate the score value, meaning that Va log (z()) = - -xx xt/ As a result, according to equation 2, Zt,-, can be approximated by
Dt Dt. + In-
where the noise is the same perturbation used when creating It,, . In our work, we sample at,-- by following equation 6 without relying on any trained diffusion models. To further improve model performance, iCT (Song & Dhariwal, 2023) proposes annealing the discretization steps N following pre-defined discretization schedule, which increases N step-wisely according to training steps. With an initially small N, this approach reduces the gradient variance at higher noise levels and accelerate the model convergence speed at small diffusion time steps, providing solid supervising signals for aligning predictions on images of strong noise magnitude.
ECT (Geng al., 2025) extends the above framework by additionally taking advantage of trained diffusion model weights to initialize Se. and trains fa on continuous time intervals. To produce paired noisy samples (f),,, xrr)), r, t E [amin, T), they sample from a predefined distribution p(() and derive r with a deterministic mapping function p(r(t). As training proceeds, the ratio r/t gradually increases and approximates 1.0. This reduces early-stage variance of equation 5, while reducing the bias as r/t becomes larger, ultimately improving model generation quality.
"The min is chosen such that Pamsn (m) Rd Péato (ar) and Fto can he approximately seen samples from the data distribution Paato (x))
4

---

Published as a conference paper at ICLR 2026
3 METHOD
3.1
OVERVIEW
In this section, we first formalize the core principles our pre-training method and subsequently introduce the fine-tuning stage. Building upon the diffusion framework established in Section 2, we adhere to the SDE formulations and notational conventions defined therein.
3.2 REPRESENTATION CONSISTENCY LEARNING
Specifically, the pre-training objective is composed of two components: contrastive loss that facil- itates semantic learning, and a representation consistency loss that enforces alignment of semantics across points the same ODE trajectory. Both objectives are in the form of the equation 5, while adopting the NT-Xent (Chen et al., 2020) as the distance metric. Formally, the NT-Xent metric for both terms can be written
exp (q (1/lqq/g)))) Xem (","")== loog cxp (q + exp (xp(q.q-/T) /T)²
where q is encoded sample, with q and q representing its positive and negative counterparts respectively. For the contrastive loss, the positive pairs are constructed through data augmentation, while other samples in the batch treated negatives. The representation consistency loss temporally adjacent points along the ODE trajectory as positive pairs, e.g., (It.. Itt,-)), while adjacent points from different trajectories serve as negatives, e.g., (Itt,, vt,, ,). The temporal data pairs is crafted following equation 6 as discussed in Section 2. As result, the overall pre-training objective can be formally depicted as
dem. Lo (11(11, ; Lg- (Eg- (y2, (o)) + dNN.xeet (Eo( Ln); Eeg(o) (rt,, 1 1))
Contrastive Loss
Representation Consistency Loes
where f denotes model parameter and is updated to minimize the above objective, (y1 y2) are augmented views of æ, Eg- and ( are target models used when computing corresponding loss term. 0 denotes exponential moving average of 6, respectively. Lg projector layer, 3-layer MLP, U(0, NN. and X(() 1.
Figure 2a illustrates the pre-training framework, which features three branches: a semantic learning branch (Fa- and Lg-), an online branch (F& and Lg),, and semantic alignment branch (Fag(o)) The input of each encoder includes learnable token [CLS], a time condition token, and image tokens. Notably, the online model Es encodes both augmented samples and noisy samples æt,, that contain more noises during each step. When computing losses, both contrastive and representation consistency loss utilize model outputs at the class token position, while contrasting the projector outputs and encoder outputs respectively.
A design of the original pre-training framework involves annealing EMA coefficient of 0- following a manually designed schedule. This mechanism aims to regulate the learning rate of representations on clean images, mitigating the challenges of aligning them with noisy samples at large diffusion time steps, critical for representation quality at high noise levels. This philosophy parallels consistency model training, where consistency at small time steps governs behavior at larger steps (Song & Dhariwal, 2023). However, this approach introduces intricately coupled hyper- parameters and a brittle training process, where even minor deviations from the preseribed config- uration risk training collapse. As result, such fragility poses significant barrier to adaptation in image generation tasks, where achieving better generation quality demands greater hyperparameter flexibility.
In contrast, the temperature value T in representation consistency loss intuitively plays a pivotal role in determining the learned representation quality on noisy images. Specifically, small enforces strong separability between samples on distinct PF ODE trajectories while tightly aligning points within the same trajectory to their clean-image endpoints. To begin with, set the temperature value to 0.1, achieving downstream performance comparable to our final results However, its early training exhibits transient instability. We speculate this is because the gradient signals from noisy
5

---

Published as a conference paper at ICLR 2026
samples introduces bias the model lacks meaningful features to reconcile alignment. To mitigate this, we implement a linear interpolated temperature schedule: T(t) = T1 * (1 (1-14 -) + T2 * t, with
T2. E [0, 1]. leading a loose aligmment for points large time steps. As training progresses, 22 converges to Ti via a cosine schedule. Crucially, this schedule operates independently hyper-parameters. While this design resolves potential early-stage instability, we emphasize that fixed small temperature value (c.g., 0.1) remains a viable solution and acts as strong baseline. We ablate this design in Appendix C. We also study the learned semanties of pre-trained model on noisy images (See Appendix E).
other
3.3 FINE-TUNING
After pre-training, the projector discarded and combine Eg with randomly initialized decoder Da and then fine-tune the complete model fo end-to-end under either diffusion or consistency model training configurations (see Figure 2b).
Diffusion model. We fine-tune the complete model with the denoising objective in equation 1. We also incorporate time-dependent weighting function and sample noise levels following a LogNormal distribution. We choose diffusion model, instead of flow matching (Liu et al., 2022; Lipman et al., 2023), as it works seamlessly with our pre-training method, which builds on the theoretical founda- tions of consistency models (Song et al., 2023). We argue that our pre- treirriining model also supports flow matching in downstream task as the time condition and noise schedules are properly aligned, akin to the recent effort (Lu & Song, 2025).
Consistency model. In early experiments, we observe standard consistency training suffers from slow convergence and suboptimal generation quality due to the supervising signals only come from the clean data. To address this, we empirically introduce an auxiliary loss between the model outputs fo ) and their corresponding clean images Ty used to generate the noisy inputs Ft, It can be formally depicted by
arg min E (Wo (n), bn), W (Z [0))
0
Here, We is frozen copy of the pre-trained encoder Eg (without projector layer Lg) and won't 't be updated during fine-tuning. This approach provides complementary supervision when training con- sistency model while incurring negligible cost, and has similar spirit to the concurrent works (Sto- ica et al., 2025; Wang & He, 2025). Note that we do not use any external models. Instead, we take full advantage of our pre-trained weights. As a result, we train consistency models by combining equation 5 and equation 9.
We defer detailed diffusion and consistency model training settings (e.g. weighting, time proposal distribution, parameterization, etc.) to Appendix B.2.
EXPERIMENT
Dataset and network architecture. We conduct experiments on ImageNet- (Deng et al., 2009) dataset. We use vision transformer (ViT) (Vaswani al., 2017) as our backbone, and utilize 16x 16 patch size on ImageNet-256 and 32x32 on ImageNet-512. The model input tokens include a learn- able token [CLS], time token, and image tokens. During fine-tuning, we use same number of blocks in encoder and decoder. To further improve training efficiency in pixel space, we add residual connections between encoder and decoder as in (Hoogeboom et al., 2023; Bao et al., 2023), and additionally incorporate time condition into decoder using N--ero (Peebles & Xie, 2023). We defer our network configuration the Appendix B.3.
Training. In main results (Table 2 4), we use 1024 batch size in both pre-training and fine-tuning. We pre-train our model for 600K steps (480 epochs). Besides, during fine-tuning, we train diffu- sion models for IM steps (800 epochs), and train consistency models for 700K steps (560 epochs). settings across We train models in FP16 mixed precision mode, and use similar including param-ear experiments within each training stage. More implementation details, and computational cost, are deferred to Appendix B.2. We also display more qualitative results in Appendix F.

---

Published as a conference paper at ICLR 2026
Table System-level comparison ImageNet-256 with CFG. For latent-space models, display model parameters and sampling GFLOPs of both the VAE and the generative model. We report GFLOPs of our EPG following DÝT. t; both model parameters and GFLOPs are composed of two generative models and one classifier. Text in gray: method that requires external models in addition to VAE. Line in blue adopt class-balanced sampling.
Model
FID4 1st Precision" Recallt NFE, Epochs
#Params
GFLOPs
Models in Latent Space
LDM (Rombach et al., 2022)
3.60 247.7
0.87
0.48
250x2
167
+ 400M
+ 104
USP DiT-X1/2 (Chu et al., 2025)
2.33 267.0
250x2
240
84 + 675M
312 119
MaskDiT (Zheng et al., 2024)
2.28 276.6
0.80
79x2
1600
675M 312 + 119
DiT-XL/2 (Peebles & Xie, 2023)
2.27 278.2
0.83
0.57
250x2
1400
84 675M 312 119
SiT- XL/2 (Ma et al., 2024)
2,06 277.5
0.83
0.59
250x2
1400
84 + 675M 312+ 119
REPA (Yu et al., 2025)
1.42 305.7
0.80
0.65
434
800
84 675M 312 119
RAE (Zheng et al., 2025)
1.28 262.9
50x2
800
415 + 839M 107 + 146
Models in Pixel Space
ADM (Dhariwal & Nichol, 2021)
3.94 215.8 0.83
0.53
500
963
673M
761
RIN (Jabri et al., 2023)
3.42 182.0
1000
480
410M
334
SiD (Hoogeboom al., 2023)
2.44 256.3
250x2
800
2.46B
555
VDM++ (Kingma & Gao, 2023)
2.12 278.1
250x2
2.46B
555
PixNerd-XL/16 (Wang al., 2025)
1.93 297.0
0.79
0.59
100x2
320
700M
134
PixelFlow (Chen ct al., 2025c)
1.98 282.1
0.81
0.60
677M
2909
SiD2 (Hoogchoom et al., 2025)
1.72
137
EPG-XL/16
2.04 283.2
0.80
0.61
75
800
583M
128
EPG-XXI/16
1.87 287.6
0.80
0.63
75
800
789M
176
EPG-G/16
1.70 297.8
0.80
0.63
75
1600
1391M
321
JYT-H/16 (Li & He, 2025)
1.86 303.4
0.78
0.62
191
600
953M
182
JiT-G/16 (Li & He, 2025)
1.82 292.6
0.79
0.62
191
600
2B
383
EPG-XXL/16
1.81 294.6
0.80
0.61
75
600
789M
176
EPG-G/16
1.75
275.1
0.80
0.62
75
600
1391M
321
EPG-G/16
1.58
298.4
0.80
0.63
75
1600
1391M
321
Evaluation. For a fair comparison with existing works, we use evaluation suite provided by ADM (Dhariwal Nichol, 2021). Evaluation metrics include Frechét Image Distance (FID) (Heusel et al., 2017), sFID (Nash et al., 2021), Inception Score (IS) (Salimans et al., 2016), Precision and Recall (Kynkäänniemi et al., 2019).
report generation results with 50K images sampled by utilizing the Heun ODE sampler (Karras et al., 2022) with 32 sampling steps in Table 3 and addi- tionally with classifier-free guidance (CFG) (Ho Salimans, 2022) Table 1 and 2, where we use interval-efg (Kynkäänniemi et al., 2024) with the recommended guidance interval (0.19, 1.61]. In ablation studies, we sample diffusion models using Euler deterministic sampler with 50 sampling steps, and report FID of diffusion models with 10K images. For consistency models, we report FID) with 50K images produced via one-step sampling across all experiments.
Pre-training cost comparison with VAE. On ImageNet-226, utilizing 8xH200, compare pre-training cost with the sd-vae-mse". which widely adopted VAE-based methods. It is orig- inally trained for 840K steps in total with batch size of 192 on large-scale open-sourced datasets. To achieve our performance reported in Table 1, 2, and 4, the pre-training takes 57 hours, 111 hours, and 100 hours respectively. In contrast, it takes 160 hours train sd- sd-vae-mse. Since our pre-training cost is smaller than the VAE model, we only report our fine-tuning cost (in epochs) when comparing with latent-space methods. We benchmark overall training efficiency against DiT on ImageNet-256 in Table 5.

---

Published as a conference paper at ICLR 2026
Table 2: System-level comparison on ImageNet-512 with CFG. Settings are the same as Table
Model
FID 1St Precisiont Recallf NFE4 Epochs
#Params
GFI.OPs|
Models Latent Space
U-ViT-H/4 (Bao et al., 2023)
4.05 263.8
0.84
0.48
50x2
400
84M + 501M 1260 + 133.0
XV/ (Peebles & Xie, 2023)
3.04 240.8
0.84
0.54
250x2
600
84M + 675M 1260
524.6
SiT-XL/2 (Ma et al., 2024)
2.62
252.2
0.84
0.57
250x2
600
84M + 675M 1260 + 524.6
MaskDiT (Zheng al., 2024)
2.50 256.7
0.83
0.56
79x2
800
84M 675M 1260 524.6
Models Pixel Space
RIN (Jabri et al., 2023)
3.95
216.0
1000
800
320M
415
ADM (Dhariwal & Nichol, 2021)
3.85
221.7
0.84
0.53
500
1081
731M
2813
SiD (Hoogeboom et al., 2023)
3.02 248.7
250x2
800
2.46B
PixelNerd (Wang et al., 2025)
2.84 245.6
0.80
0.59
100x2
320
700M
583
VDM++ (Kingma & Gao, 2023)
2.65 278.1
250
800
2.46B
EPG-L/32
2.43 289.7
0.82
0.57
75
600
540M
113
EPG-L/32
2.35 295.4
0.82
0.57
75
800
540M
113
Table 3: Class-conditional generation per- formance comparing with DřT and SiT.
Benchmarking few-step class-conditional image generation ImageNet-256.
Table 4:
Model
FID
sFID
1S1
Epochs
MParams
Model
FID NFEJ Epochs
#Params
DIT-B/2
84M-130M
Models in Latent Space
SIT-B/2
33.0
6.46
43.71
80
84M-130M
EPG-B/16
52.0
8.25
24.34
80
229M
ICT-XL/2 (Song et al.,, 2023)
34.24
84M + 675M
EPG-B/16
31.9
6.80
42.70
160
229M
20.30
2
84M + 675M
EPG-B/16
25.1
6,27
54.25
240
229M
Shorteur-XI/2 XL/2 (Frans et al., 2025)
10.60
250
84M + 675M
7,80
250
84M + 675M
DIT-XL/2
19.5
80
84M-675M
SIT-X1/2
17.2
80
84M-675M
IMM (Zhou et al., 2025)
8.05
6395
84M 675M
EPG-XL/16
15,9
6.28
72.43
80
583M
Models in Pixel Space
DIT-XL/2
6.85
121.50
84M-675M
EPG-L/16
8.82
560
540M
SIT-XL/2
8.3
6.32
131.70
1400
84M-675M
EPG-XL/16
6.6
5.16
141.09
800
583M
4.1
MAIN RESULTS
Diffusion Model. In Table 1, when compared to diffusion models trained the latent space, EPG- XXL/16 surpasses one of the leading models SiT-XL/2 with similar overall model parameters, e.g., 789M v.s. 759M, and approximately 50% of its training cost. We display the FID convergence of EPG-XL/16 in Figure 7. Our EPG models are in parallel to methods like REPA and RAE and can also benefit from incorporating external supervision, which we leave to our future work. When comparing with pixel-space diffusion models, EPG achieves the best FID while bridging the perfor- mance and efficiency gap with VAE-based counterparts. Our EPG models also demonstrate strong scalability with respect model sizes. By further increasing model parameters 1391M. our EPG-G/16 model achieves 1.58 FID with class-balanced sampling, outperforming JiT-G/16 by large margin. We attribute this the strong semantic quality and consistency from the pre-trained weights. Our performance advantage over JiT also reveals that training diffusion model with the original pixel-wise prediction loss alone inefficient in capturing rich semantics thus leading to severe overfitting (Li & He, 2025) and poor scalability. We compare class-conditional image gener- ation performance with SiT and DiT different sizes in Table 3.
EPG's performance ImageNet-512 (Table 2) confirms its ability to scale efficiently to high- resolution generation. With large 32x32 patch size, delivers strong generation quality with sig- nificantly low computational overhead (GFLOPs), all without relying external VAEs. The results on ImageNet position EPG as a scalable and efficient solution for pixel-space generation.
Consistency Model. To the best of our knowledge, model without relying on pre-trained VAEs or diffusion models. As shown in Table 4, EPG-L/16 reaches 8.82 FID with one step generation, substantially outperforming iCT-XL/2, its latent-space
the first to successfully train consistency
8

---

Published as a conference paper at ICLR 2026
Table Training efficiency comparison with DiT with x H200 GPU. f: measured with official codes.
Table 6: Comparing FID scores with
baseline methods in downstream tasks.
Model
FID
Cost (hours)
#Params
Experiment
DM
CM
sd-vac-mse
160
84M
REPA SiT-B
72.71
Our Pre-train
57
106M
MoCov3 ViT-B
56.26
36.77
Scratch
59.69
NaN
DIT-XL/2
2.27
506
675M
rRCM
46.51
37.55
EPG-XL/16
2.04
139
583M
EPG-B/16
41.36
33.12
EPG-XXL/16
1.87
160
789M
Pre-craiiing for
Scale Pretzaining 66 for
Sca Fre-tral dParmm for DM
Scale Pro-tra #Paramo
bc255 bs1024
bs254 bs1024
56M
C4M
300
107M
307M
80-
223H
225M
0-
103K
200k
200k
10ck
200k
300k
103k
200k
200k
100k
200k
300k
Toiring Stops
Traiing Stopo
Training Steps
Trining Steps
Figure 3: Performance in downstream tasks scales with pre-training compute budget. DM, CM: fine-tune under the settings of diffusion consistency model.
counterpart. Besides, EPG-L/16 outperforms Shorteut-XL/2 in one-step generation, while leaving small performance gap its 4-step generation result. Noticeably, IMM marginally surpasses model by 0.77 FID but requires 11x more training compute, highlighting practical advantages of our method, We display the FID convergence of EPG-L/16 in Figure
4.2 ABLATION STUDY
Comparing with pixel-space baselines. On ImageNet-224, we compare with several pixel-space baselines training diffusion and consistency model. (1) REPA (Yu et al., 2025), (2) load pre- trained MoCo v3 (Chen et al., 2021) weights, (3) train from scratch, and (4) pre-train with original rRCM framework (Lei et al., 2025). We display FID results in Table 6. The model equipped with REPA yields the worst generation quality. Besides, The model loaded with pre-trained MoCo v3 weights demonstrate better results and training stability than the one trained from seratch. com- parison with the model pre-trained by rRCM, EPG demonstrates consistently better generation performance in both diffusion and consistency training tasks, indicating the remarkable effective- ness of our improvements. Notably, our framework is much concise and does introduce coupled rrper-parameters. We defer implementation details of the baseline methods to Appendix B.
Scalability. We study the scalability of our method on pre-training batch size and model parame- ters with results shown in Figure 3. Specifically, we compare downstream model performance by increasing the pre-training batch size from 256 to 1024 (first two columns). In addition, we study the impact of the model parameters by scaling the encoder from 64M to 107M and 225M (last two columns). The decoder scaled accordingly during fine-tuning, resulting in complete models of 1 166, 229M, and 540M parameters, respectively. As demonstrated, the generation performance on downstream tasks improves as the pre-training budget increases. This scalability positions our framework as promising solution for high-resolune regouresolttion and multi-mode image generation, where training and sampling efficiency are critical.
RELATED WORK
Training diffusion model in pixel-space Training diffusion models directly in pixel space is noto- riously challenging due to high computational costs and slow convergence on high-resolution data. Prior work addresses this through optimizing model architectures (Jabri et al., 2023), improving diffusion formulations (Hoogeboom et al., 2023; Kingma & Gao, 2023), and decomposed training stages with masked pre-training (Lei et al., 2023) or cascaded diffusion models (Ho et al., 2021). While these efforts enhance generation quality of pixel-space diffusion, they still lag behind latent-

---

Published as a conference paper at ICLR 2026
space methods in both performance and efficiency. Our work bridges this gap, establishing new state-of-thh-ar for pixel-space approaches utilizing similar training cost as latent-space baselines yet with fewer sampling steps.
Accelerate diffusion model training. On one hand, current works in diffusion models are focusing on accelerating training speed by incorporating additional representational supervision. However, as far as we know, all these efforts remain confined in the latent-space training formulation. Yu et al. (2025) aligns intermediate features of diffusion models with those from off-the-shelf SSL models. As follow up works, several variants (Leng et al., 2025; Yao et al., 2025) have been proposed. Leng al. (2025) train both VAE and diffusion models in an end-to-end manner, achieving superior training efficiency yet it still relies on trained VAEs. In contrast, another line of works (Chu et al., 2025; Stoica al., 2025; Wang & He, 2025) speed up model training by incorporating representation learning loss, without relying on external SSL models. The most related work to ours is USP (Chu al., 2025), which contains a strong representation learning stage to accelerate the training of diffusion models. In contrast, our study focuses on the image space instead of latent space. In addition, Stoica et al. (2025) and Wang & He (2025) accelerate model training by incorporating contrastive loss, imposed either predicted clean images intermediate features.
Few-step generative models. Consistency models (Song & Dhariwal, 2023; Song et al., 2023; Lu & Song. 2025) enable single-step sampling by enforeing consistency across deterministic diffusion trajectories. While theoretically appealing, CMs face amplified training challenges due to sparse supervision narrow noise intervals. Solutions like weight initialization from pretrained diffusion models (Geng et al., 2025) mitigate this but remain infeasible without access to trained diffusion models. Meanwhile, Shortcut model (Frans al., 2025) and IMM (Zhou et al., 2025) provide insightful solutions for achieving few-step generation with solid theoretical innovations. In contrast, from an empirical perspective, we offer an effective solution for training one-shot generators on high-resolution images directly in pixel space.
6 DISCUSSION
Conclusion. We present a two-stage training framework for training diffusion and consistency mod- els in pixel space. It successfully bridges the performance and efficiency gap between pixel-space trainings and the latent-space approaches, reaching SOTA performance ImageNet dataset. By identifying the semantic roles encoder and decoder in diffusion based generative models, we pro- vide a practical pathway for efficiently training the generative models. We hope it will motivate future efforts to improve generation quality with well-established insights from the traditional visual domain.
Advantages over VAE-based methods/LDDM framework. Our two-stage training framework fun- damentally challenges the dominant LDM paradigm. It avoids the training complexity and perfor- mance bottleneck rooted in the VAF-based LDM framework, lowering the barrier to adapt gener- ative models to new and unseen data. Besides, the pre-training is significantly efficient than VAE and the pre-trained model could immediately benefit the downstream generative model training (see Figure 7). As reported in Table 5, our diffusion model surpasses DiT in both performance and efficiency by a large margin. Moreover, by increasing model patch size proportionally to image resolutions (e.g. 16x16 on ImageNet- and 32x32 on ImageNet-511 our method benefits from significant training efficiency across resolutions in terms of both GFLOPs and training time.
Limitations. Our EPGs currently under-perform leading latent-space methods like REPA and RAE. However, the performance gap can be largely explained by a significant disparity in training com- pute. Our experiments in Table support this, showing that the performance limited by the overall training budget, not by a fundamental limitation of our approach. This finding positions our method as a highly scalable and promising approach for training pixel-space diffusion-based generative models. A natural next step, which we leave for future work, is to leverage powerful off- tee-the-shfff backbones to accelerate this scaling and further close the performance gap.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: vit backbone for diffusion models, 2023. URL URLhttps://arae httppp:s//axxiooooo oww.voorrggoooo arggabs//209 12152.
Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden-Fijnden. How build consistency model: Learning flow maps via self-distillation, 2025. URL
.or.v.oorrgooooo. abs/
2505. 18825.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL https:
/ //arxiv.org/abbbbb. /btp///a.//...... 14294.
Chubin Chen, Sujie Hu, Jiashu Zhu, Meiqi Wu, Jintao Chen, Yanxun Li, Nisha Huang, Chengyu Fang. Jiahong Wu, Xiangxiang Chu, al. Taming preference mode collapse via directional de- coupling alignment in diffusion reinforcement learning, arXiv preprint arXiv:2512.24140 2025a. 2025a.
Chubin Chen, Jiashu Zhu, Xiaokun Feng, et al. -r-guidance: Stochastic self guidance for training- free enhancement of diffusion models. arXiv preprint arXiv:2508, 1 12880, 2025b.
Shoufa Chen, Chongjian Ge, Shilong Zhang, Peize Sun, and Ping Luo. Pixelflow: Pixel-space generative models with flow, 2025c. URL https: aaaaaaivvvvoo.m ort.
07963.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. URL https: http:///xxvvv/./e// 2002. 05709.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020. URL
https://axxia: 1ttp://aaa....t//////1111111..3
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self supervveed vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9640-9649, 2021.
Xiangxiang Chu, Renda Li, and Yong Wang. Usp: Unified self-supervised pretraining for image generation and understanding. arXiv preprint arXiv:2503.065 06132, 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. URL https: /arxivvivvv////////i//aav///aiia org//b/g//s htrg/bbs/2105 /210..2211..0 05233.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shorteut models. In The Thirteenth International Conference on Learning Representations, 2025. URL
https:/ //pppppeevvieeeeeeeee/eeeeeeeeee net. htw.:r/reeeee.nt...oooo...doooe.
Zhengyang Geng, Ashwini Pokle, Weijian Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. In The Thirteenth International Conference Learning Representations, 2025. URL
1i==mmxxodxxxxxxs https: apenreview net hetee//ooondddooo/eeeeme
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. URL http:///axxwv.or///oaa/ 06377.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:/ /ttttppss//poooeedeee.eeoruoooon http:eeeeree.c.cccpeee.eeeeeeee faleeeiieeeeee//////// pater//2n7naaaaaad0000reeeeeeeee
In I. Guyon, U. Von Luxburg,

---

Published as a conference paper at ICLR 2026
Jonathan Ho and Tim Salimans. Classifier- diffusion guidance, 2022. URL https: //axiv.. org/ a 2207 12598.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL
https: /tt///rrae /arxivvvvvv// orggabss////oooooooooo.oo.ss...s 20006 11239.
Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Sal- imans. Cascaded diffusion models for high fidelity image generation, 2021. URL https:
//aaxiv.orggaaaabb. 2t..//11....... 15282.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images, 2023. URL https: arxiv. org/ abs./bb....1159
Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Sali- mans. Simpler diffusion (sid2): 1.5 fid imagenet512 with pixel-space diffusion, 2025. URL https: arxiv. org/ abs/ 2410 19324.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation, 2023. URL https: wwwpvv.wvvvv///////////e..///aaa wwgg/bas/
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577 2022.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances Neural Information Processing Systems, 36:65484-65511 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL https:
//taxxv..rrg////eee...........lt abs 111/11111166 6114.
Tuomas Kynkäänniemi, Miia Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. Advances in Neural Information Processing Systems,
2024.
Tuomas Kynkäääniemmi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models, 2019. URL https: //arxiv.. org/abs/ 904. 06991.
Jiachen Lei, Qinglong Wang, Peng Cheng, Zhongjie Ba, Zhan Qin. Zhibo Wang, Zhenguang Liu, and Kui Ren. Masked diffusion models are fast distribution learners. arXiv preprint arXiv:2306. 1363, 2023.
Jiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Chaowei Xiao, Zhongjie Ba, Kui Ren, Jun Zhu and Anima Anandkumar. Robust representation consistency model via contrastive denoising. In The Thirleenth International Conference on Learning Representations, 2025. URL
net forum inda of rmbJRJárH.
Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers, 2025. URL
/ettps///ariv...
abss 2250 10483.
Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise, 2025. URL
https: / //arxiivv... org/ abs /2/122511111 13720.
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised representation generation method, 2024. URL wtw.t:tpps///m http:::s//axxx...rrr.. wttpvv/ooggg///oo.oooo/ooo.. abs//31b//2222 03701.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL https: //arxaaa..vv////a//aa/eeaeaaaava org/ abs /22202222222 02747.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.08. 00300, 2022.
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models, 2025. URL https: //wxxvixvvv///./////a/////aaa//a org.orr///ss.. /241 11081.
12

---

Published as a conference paper at ICLR 2026
Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijndee and Saining Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers, 2024. URL https:://// http///axxvvv.o.///. abs / 2201. 08740.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprini arXiv:2103.03844) 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https:
//arxiv. org. abs/ 02222.097444
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL https: ptctps://arxiv.ot /arxiv.org//////............
abs/2112.10755
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. /4189, 2023.
Yang Song, Jascha Sohh-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021. URL 13456.
https: a tttpss///arxvv.orggge wwtp:/o////oo//////o/
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. URL https://arxiae wwg/ abs////////////////// 2303. 20....166666 01469.
George Stoica, Vivek Ramanujan, Xiang Fan, Ali Farhadi, Ranjay Krishna, and Judy Hoffman. Contrastive flow matching. arXiv preprint arXiv:2506. 05350, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Runqian Wang and Kaiming He. Diffuse and disperse: Image generation with representation regu- larization. arXiv preprint arXiv:2506.0902) 2025.
Shuai Wang, Ziteng Gao, Chenhui Zhu, Weilin Huang, and Limin Wang. Pixnerd: Pixel neural field diffusion, 2025. URL wwtphttppseec /ttp///avvvv///////////a////.avv http://aaaa....//aa///////////// httrrg/ab///502.. 23268..
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization orwooorgg//.....ooooo...o 2501.11111 01 01423.
dilemma in latent diffusion models, 2025. htRp://s:://aat
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think, 2025. URL https: / /ttp/axivvv//////////////////.vl org/ abs /2000 06940.
Boyang Zheng, Nanye Ma. Shengbang Tong, and Saining Xie. Diffusion transformers with repre- sentation autoencoders, 2025. URL https:/ /ttpa//a.......aa////////3 11690.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. In Transactions on Machine Learning Research ('MMLR), 2024.
Linqi Zhou, Stefano Enmon, and Jiaming Song. Inductive moment matching, In Forty-second International Conference on Machine Learning, 2025. URL https::/ net/ forum? perrumid-pwate Ld-pwNu00yUUUU
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
To enhance the clarity and readability of this work, we utilized DeepSeek for language refinement several sections (e.g., abstract and introduction) of the manuscript. The tool's suggestions were reviewed and edited to ensure technical accuracy and alignment with the paper's scientific content.
B IMPLEMENTATION DETAILS
B. BASELINE METHODS
We introduce the implementation details for baseline methods listed in Table 6.
MoCo v3. Utilizing pre-trained MoCo v3 ViT-B model (pre-trained for 300 epochs), we attach an additional decoder comprising 12 VIT layers and fine-tune the complete model end-to-end. Note that we also add residual connections between the encoder and decoder, and adaLN-Zero in the decoder. Similar to our Base/16 model, the complete model contains 229M parameters. To enable class-conditional denoising training, prepend positional embedding. Besides, we concatenate the class and timestep embeddings with the noisy image tokens and feed them input tokens the model.
embedding to the encoder's original
REPA in pixel space. We use the official implementation and train SiT-L/16 with the default hyper- parameter settings.
rRCM. We use the official checkpoint of pre-trained rRCM encoder. has the same structure as our Base model while being pre-trained with 4096 batch size for 600k steps.
Table 7:
Network configurations of EPG. Encoder and decoder settings are separated by comma.
t
ViT Block
AderN-
xl.
Small
6,
768, 768
12, 12
116M
Base
12. 12
768, 768
12, 12
229M
c
y
ViT Block
Large
16.
16.
768, 1584
583M
XXL
12. 12
768. 1920
12. 16
789M
12, 12
768,
2688
12,21 21
1391M
VTT Block
Table 8: Network configurations of encoder in pre- pre-training, named as Representation Consistency Model (RCM) for clarity.
VIT Block
c+yy
X
Model
Blacks
Dim
Hends
Parauns
RCM-S
6
768
64M
Figure 4: The network architecture of EPG utilized in downstream generative tasks.
RCM-B
RCM-L
16
1024
225M
B.2 IMPLEMENTATION DETAILS
Data pre-processing. We pre- mrepprocess ImageNet dataset by center-cropping images into 256 512x512 and save images in lossless PNG format.
Training settings. We adopt the definition of SDEs and the time discretization strategy from EDM (Karras al., 2022), we discussed in Section 2. The time horizon is divided into N non- overlapping intervals, where each interval endpoint n is mapped to discretized time step (or noise level) via the following equation
11 In 1/p + N
2/0 1_ min
1/D with n E (0, N - 1].
(10)
hhtps///www.depseck.com
14

---

Published as a conference paper at ICLR 2026
Table 9: Our pre-training and fine-tuning diffusion settings. In denoising training, first sample from a continuous lognormal distribution and then map the nearest discrete value b.. t: the denoiser predicts clean image æ at different noise levels.
Pro-training
Denoising Training
Consistency Training
Total discrctization steps N
from 20 1280
1280 Constant
(Continnous) follow ECT (Geng al. 2025)
N p(r/t) Weighting X(t)
follow iCT (Song& & Dhariwal, 2023)
Proconditioning
1/ 1/v C + datu
1/ V dutu
1/ 1/v data
Time condition
c(t) 1000 } lut 24(0, N)
c(t) 1000 Int
e(t) 1000 Int
Noise Sampling
tw lognormall (-1.2, 1.6)
Segnormal-OO... 1.6). p(r(t)
Training objective
Equation
Equation
Equation Equation Equation
Model parameterization
a-prediction"
Shifting
image size/64
smage size/64
image size/64
(Hoogehoom al., 2023)
Here, Omar 80, mmi = 0.002, = N is total discretization step. To regularize model input, also employ the preconditioning G.2 from EDM with data 0.5.
We consider our pre-training as combination of representation learning and consistency train- for representation learning are grounded in SSL. best practices: we use MoCo v3's augmentation strategies (Chen et al., 2021) for contrastive loss computation, set the EMA coefficient to 0.99 for the momentum encoder Eg-, and adopt a lin- ear learning rate schedule scaled to batch size (6e-4 for 1024 batch size). We use Adam W optimizer with default beta values. Built on top of these, we determine hyper-pane eyper-parameters of the representation consistency loss in equation 8. In specific, we adopt the time discretization annealing schedule from iCT (Song& & Dhariwal, 2023), and implement a temperature schedule linearly interpolated between 71 0.1 and 72 0.2. See Section C for ablation on this design.
ing (Song & Dhariwal, 2023). Therefore,
After pre-training, we fine-tune model under diffussinn/cosiistenyy model training configurations. We utilize z-prediction for diffusion model and the EDM formulation (Equation for consistency model. We also scale the noise level according to image_size of training data. Detailed pre-training and fine-tuning syper-parameters
displayed in Table 10 and 11. For clarity, following (Lei et al., 2025), we name the pre-trained model as Representation Consistency Model (RCM) in our discussions below. We display the RCM model configurations in Table
B.3 NETWORK CONFIGURATION.
We list our model configurations Table 7, and network structure in Figure We use the Small and Base model in our ablation studies.
B.4 COMPUTATIONAL COST
Utilizing 8 8xH200 GPUs, the pre-training takes 57 hours for RCM-B/16. 100 hours for RCM-L/16 on ImageNe and 111 hours for RCM-L/32 on ImageNet-512. For the denoising task, training on ImageNet-256 takes 139 hours for EPG-XL/16, 160 hours for EPG-XXL/16, and 267 hours for EPG-G/16 (on H200 GPUs). On ImageNet-512 training the EPG-L/32 model takes 100 hours. The EPG-L/16 consistency training ImtgeNet-256 takes 156 hours.
C ADDITIONAL ABLATION STUDY
In this section, we conduct ablation studies on the temperature value and training objectives of pre- training, and auxiliary loss in consistency fine-tuning. Besides, given pre-trained encoder, also study how the flexibility of updating its parameters impacts downstream performance by controlling its learning rate. By default, We experiment with EPG-B/16 and pre- mre-train the encoder for 300K steps. Subsequently, we fine-tune downstream diffusion models for 100K steps with 1024 batch size, and consistency models for 200K steps with 256 batch size.
15

---

Published as a conference paper at ICLR 2026
Table 10: Hyper-parameters used in Pre- Pre-training.
RCM-S, RCM-B
RCM-L
Bs
1024
Training steps
600k
Lr
6c-4, Cosine Decay AdamW((0.) 0.999)
Optimizer
Wd
0.03
0.05
linear interpolation with anncaling
T for Rep consistency loss
71
0.1, 72 0.2
for contrastive loss
0.2
EMA momentum encoder Eg-
0.99
Diffusion settings
see Table 9
Data augmentation
Follow MoCo v3 (Chen et al., 2021)
Table 11: Hyper- prppe--praamiers in Fine-tuning. We use step-wisely decayed learning rate when fine-tuning con- sistency model: in the starting 400k steps, Ir=le-4, from 400k to 500k, Ir=3e-S, and Ir=8e-6 thereafter. 1; the dropout is employed after each Attention block in decoder.
Denoising Training (Table 1) Denoising Training (Table Consistency Training (Table 4)
Dataset Bs Training steps
ImageNet-256
ImageNet-512
ImageNet-256
1024
1024
1024
1000/2000K
1000K
700K
Lr
lc-4
Ic-4
step-wisely
Grad clip Optimizer
0.5
0.5
AdamW((.9) 0.999)
Adam (00.9, 0.999)
Ad,9 0.99)
Wd
0.01
0.01
0.01
Dropout
0.5
Patch size
16x 16
32x32
16x 16
Diffusion settings
See Table
See Table
See Table
EMA value
0.9999
0.9999
0,9999
Data augmentation
Horizontal Flip
Horizontal Flip
Horizontal Flip
Impact of auxiliary loss in training consistency model. In Table 13a, we compare consistency models trained without auxiliary loss (equation 9) or without loading RCM weights. We use pre- trained weights from Table at 600K step.
Impact of temperature value. We compare different temperature settings for representation con- sistency loss (1) No representation consistency loss (2) 7 72 0.1: fixed temperature 0.1, (3) 72 0.5: linear interpolation with 72 0.5, and (4) Ours: linear interpolation with 72 0.2. We
display results in 13b.
Impact of pre-training loss terms. In Table 13c, conducted a detailed ablation study to isolate the contribution of each loss term to the final downstream generation performance. We observe that removing the contrastive loss causes performance to degrade severely, even more than training from scratch. This is because the encoder fails to learn effective visual semantics, ultimately leading
Table 12: Ablation studies on auxiliary loss, temperature value and pre-training loss terms.
Model
CM
Model
DM
CM
Model
DM
CM
Scratch
NaN
wlo consistency
58.18
56.41
EPG B/16
$2.67
46.99
w/o aux
129.16
0.1
52.79
46.44
who consistency
58.18
56.41
w/o RCM
65.05
7 = 0.1,
0.5
56.22
45.69
w/o construst
115.48
117.63
EPG-B/16
36.75
0.1,
0.2
52.67
46.99
Scratch
77.25
NaN
(a) Auxiliary loss.
Temperature value of RCL.
(c) Pre- Pre-training loss term.
16

---

Published as a conference paper at ICLR 2026
to collapsed representation across different noise levels. In contrast, model trained from scratch is not constrained by this poor initialization, making easier to optimize.
Impact of the flexibility of encoder. We pre-trained RCM weights from Table at 600K step and adopt the XL architecture during fine-tuning, Specifically, we compare three different settings: updated. (2) Lrd=0.5:
(1) Frozen: the pre-trained encoder is frozen and only decoder parameters
we apply the layer-wise learning rate decay to encoder, where shallow layers (e.g. patch embedding
layer) have smaller learning rate while deeper layers (e.g. last ViT block) have larger learning rate.
(3) Default: default setting in our experiments, where the encoder is trained alongside decoder with constant learning rate. For reference, we also present the FID of diffusion model trained from scratch. The results are displayed in Table 13.
The performance improves as encoder parameters are updated with greater flexibility. This result indicates that the pixel reconstruction inherent in denoising requires the encoder to supply low-level information, a capability it must adapt from its initial pre-training on high-level semantics.
D DETAILED QUALITATIVE RESULTS
Training dynamics of downstream models. We display the FID scores of downstream models with respect to training steps in Figure 7.
Visualization of diffusion model encoder features. To provide intuitive comparison between our EPG diffusion variant and the one trained from scratch. We visualize output of their encoder features respectively Figure and 6. The results showcase the encoder features from our diffusion model yields stronger semantic structure.
E SEMANTICS LEARNED BY RCM ON NOISY SAMPLES.
To demonstrate that the RCM encodes noisy samples into meaningful representations, compute the per-channel standard deviation of the fz-normalized encoder outputs averaged across 1000 sam- ples at various time steps. For RCM, the channel-wise std approximates Va dimension. In contrast, the encoder trained solely with contrastive loss exhibits near-zero std across channels at varying noise levels, indicating representation collapse, as noted in prior work (Chen & He, 2020).
1 where d is the feature
QUALITATIVE RESULTS
We display more qualitative results in figures 9 10 12 15 16 17 18. Images are downsampled and compressed to optimize storage efficiency.
17

---

Published as a conference paper at ICLR 2026
Noise decrease
Figure 5: Visualizing encoder features of EPG-XL diffusion variant.
Noise decrease
Figure 6: Visualizing encoder features of diffusion model trained from scratch.
53
20.50K
30
20
10-
160
Epochs
Figure 7: Training progress of downstream generative models: (Left) EPG-XL diffusion variant, (Right) EPG- L consistency variant.
18

---

Published as a conference paper at ICLR 2026
Table 13: Trade-off between the flexibility of and faith-
fulness to encoder representation.
vd
Output
No consistency
Encoder
FID
random
Default
12.46
100
200
Lrd=0.5
18.56
Time step
Frozen
22.44
Figure 8: Avg per-channel std of RCM's outputs varying time steps.
Figure 9: Images generated by EPG-L via one-step sampling.
19

---

Published as a conference paper at ICLR 2026
Figure 10: Uncurated samples generated by EPG-XL, 32-step Heun ODE sampler, classifier-free guidance 4.5, class 1.
K
Figure 11: Uncurated samples generated by EPG 32-step Heun ODE sampler, classifier-free guidance 4.5, class 22.
Figure 12:
Uncurated samples generated by EPG-XL. 32-step Heun ODE sampler, classifier free guidance 2.5,
class 89.
20

---

Published as a conference paper at ICLR 2026
Figure 13: Uncurated samples generated by EPG-XL. 32-step Heun ODE sampler, classifier-free guidance 4.5, class 207.
Figure 14: Uncurated samples generated by EPG-XL. 32-step Heun ODE sampler, classifier free guidance 4.5, class 250.
NAN
GOi
Figure 15: Uncurated samples generated by EPG-XL. 32-step Heun ODE sampler, classifier- free guidance 4.5, class 429.
21

---

Published as a conference paper at ICLR 2026
Figure 16: Uncurated samples generated by EPG-XL. 32-step Heun ODE sampler, classifier- free guidance 4.5, class 620.
Figure 17: Uncurated samples generated by EPG-XI. 32-step Heun ODE sampler, classifier-free guidance 4.5, class 812.
Figure Uncurated samples generated by EPG-XL. 32-step Heun ODE sampler, classifier-free guidance 3.5,
class 933.
22


Reviews:
[{"review_id": "4I1oRYB7oo", "reviewer": "Reviewer_bsaA", "summary": "The authors propose a two-stage framework for (1) representation learning with consistency regularization, and (2) fine-tuning for the generative model. To make the optimization of the representation learning in the first stage work, the authors leverage EMA and stop gradients to form the training objectives. The fine-tuning is conducted using a diffusion model and extra regularization to ensure that the encoder remains structured and meaningful.", "questions": "Have you evaluated the encoder's performance?\n\nHow's the performance if you don't use the consistency regularization? Or simply using it without the contrastive objective.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors carefully designed the training objectives for leveraging consistency loss in the training of the encoder.\n\nThe authors focus on important questions and give good justifications for their initiatives.", "weaknesses": "My main concern is that I am not sure if the paper can be claimed as a \"pixel-based\" generative model, since:\n\n(1) The consistency regularization is conducted in the latent space.\n\n(2) The generative model is also trained in the diffusion model. (Also, I do think the authors should specify how the generative model is trained in their diagram, as from the current one, you might think they are learning a generative model in the 1st stage of learning representations, which is not the case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0aJVTvL2ks", "reviewer": "Reviewer_FWeF", "summary": "This paper proposes a novel two-stage training framework (tentatively referred to as EPG) for advancing end-to-end pixel-space generative models, addressing the long-standing limitations of Variational Autoencoder (VAE)-dependent paradigms in the field of diffusion and consistency models. The framework decouples the complex generative task into self-supervised pre-training and end-to-end fine-tuning phases: in the first phase, the encoder is trained independently via contrastive loss and representation consistency loss to learn noise-robust visual features; in the second phase, the pre-trained encoder is concatenated with a randomly initialized decoder for end-to-end optimization targeting downstream generative tasks. Experiments on ImageNet-256 and ImageNet-512 datasets demonstrate competitive performance—achieving FID scores as low as 2.04 and 2.35 respectively with only 75 inference steps—and pioneering pixel-space consistency model training without VAE reliance, yielding 8.82 FID in single-step generation. This work bridges the efficiency gap between pixel-space and latent-space models while simplifying the training pipeline.", "questions": "What is the impact of ODE path sampling density on the representation consistency loss and final generation quality? Could you supplement ablations or visualizations to clarify this mechanism?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Paradigm Innovation: Breaking VAE Dependence\nThe paper makes a significant paradigm contribution by eliminating the need for VAEs in high-quality generative modeling—a critical limitation of mainstream approaches like Stable Diffusion and DiT . VAEs introduce inherent trade-offs between compression ratio and reconstruction quality, and require costly joint fine-tuning across domains . By decoupling representation learning from pixel reconstruction, the proposed two-stage framework resolves these issues: the self-supervised pre-training phase ensures robust feature extraction from noisy data, while the lightweight fine-tuning phase avoids the complexity of VAE optimization. This \"de-VAE\" design aligns with emerging trends in pixel-space generation and significantly lowers the barrier to adapting generative models to new domains.", "weaknesses": "Limited Evaluation on High-Resolution and Diverse Datasets\nWhile the paper demonstrates results on ImageNet-256/512, it lacks validation on higher-resolution tasks (e.g., 1024×1024 FFHQ) where pixel-space models historically struggle . The choice of ImageNet (natural images) also raises questions about generalizability to other domains (e.g., medical imaging, satellite imagery) where VAE artifacts are particularly problematic. Furthermore, key video generation metrics like temporal consistency or FVD are absent, despite the paper hinting at extensibility to video—leaving uncertainty about whether the framework can capture spatiotemporal dependencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bBRoK4b6QD", "reviewer": "Reviewer_nxB1", "summary": "The paper proposes a two-stage training framework — pre-training (representation consistency + contrastive learning) to help the encoder learn semantic features, followed by end-to-end fine-tuning with a randomly initialized decoder. The framework can be applied to pixel-space diffusion and consistency models, achieving strong results on ImageNet-256.\n\nThe paper is clearly written and easy to follow. The motivation, methodology, and experimental evidence are all presented in a logical and convincing manner.\n\nIn terms of originality, while the approach does not introduce fundamentally new techniques, it leverages existing methods in a novel way to improve the performance of high-dimensional pixel-space generative models, which is meaningful and valuable.\n\nThe experiments are relatively comprehensive and well-designed.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Please refer to Summary.", "weaknesses": "My main concern lies in the comparison with the latest latent-space methods, particularly RAE [1], where the proposed approach still lags behind in performance (ImageNet-256 FID 2.04 vs. 1.51). Moreover, most of the baselines used in the paper are relatively outdated (especially the pixel-level models), and the reported improvements over them are not very substantial.\n\nCompared to VAE-based approaches, the proposed two-stage training pipeline does not demonstrate a clear advantage. Notably, recent works such as REPA-E [2] have also shown that end-to-end joint training of VAEs and diffusion models is feasible and effective.\n\nOverall, while the method is conceptually simple, it does not appear to be sufficiently effective or provide strong new insights.\n\n[1] Diffusion Transformers with Representation Autoencoders\n[2] REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7TTAZPib0X", "reviewer": "Reviewer_jexG", "summary": "This paper proposes a novel two-stage training framework for pixel-space consistency models. In the first stage, a portion of the generative model, referred to as the encoder, is pre-trained using self-supervised learning (SSL) methods, providing beneficial initialization. In the second stage, the entire model is fine-tuned with an additional network component, termed the decoder. This approach eliminates the need for latent diffusion training and achieves superior generation performance compared to existing end-to-end pixel-space generative models.", "questions": "- Have you experimented with performing the second-stage training while keeping the encoder, pre-trained in the first stage, frozen? Readers may be interested in understanding the trade-off between the flexibility of $f_\\theta$ and the faithfulness to the representations learned in stage 1.\n- How would the intermediate features obtained by $E_\\theta$ differ if both $E_\\theta$ and $D_\\theta$ were trained in a single stage without the contrastive loss? If possible, visualizing this difference would be of interest to readers.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The motivation is clearly articulated. In particular, the introduction effectively positions the work, helping readers understand its aims and benefits.\n- The proposed method does not require external models, which may enhance its applicability to other domains.\n- The overall training pipeline is presented clearly.\n- The empirical results are impressive, and the evaluation is extensive.", "weaknesses": "[Major comments]\n- Although two types of contrastive loss are proposed for the pre-training stage (Equation (8)), a more in-depth analysis of how each contributes to performance improvement (rather than reporting only FID scores) would be beneficial.\n- It is unclear why $(\\cdot)^-$ is applied to the first term and $sg$ to the second term in Equation (8). Could you provide further insight into this design choice?\n- The fine-tuning stage lacks clarity, as $\\theta'$ is not defined.\n\n[Notation errors] There are several notation errors and unclear formulations:\n- $\\omega$ in the caption of Figure 2 should be $\\theta$?\n- The definitions of $(\\cdot)^-$ in Lines 177 and 193 are inconsistent.\n- $\\tau_2(t)$ in Line 274 should be $\\tau$?\n- What is $\\theta'$ in Equation (9)?\n- In Equation (9), does $E_{\\theta'}$ include the projection used in the pre-training stage?\n\n[Minor]\n- The reference for VAE cites the wrong publication year.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)