Adapting Self-Supervised Representations as a Latent Space for Efficient Generation
Authors: Ming Gui
Johannes Schusterbauer
Timy Phan
Felix Krause
Joshua M. Susskind
Miguel Ángel Bautista
Björn Ommer
Venue: ICLR 2026 Poster (2026)
Keywords: generative models
visual synthesis
diffusion
flow matching
Primary area: generative models
TL;DR: We show that fine-tuned self-supervised tokens can serve as compact latents, enabling faithful single-token reconstruction and efficient generation.

Abstract: We introduce Representation Tokenizer (RepTok), a generative modeling framework that represents an image using a single continuous latent token obtained from self-supervised vision transformers. Building on a pre-trained SSL encoder, we fine-tune only the semantic token embedding and pair it with a generative decoder trained end-to-end using a standard flow matching objective. This adaptation enriches the token with low-level, reconstruction-relevant details, enabling faithful image reconstruction. To preserve the favorable geometry of the original SSL space, we add a cosine-similarity loss that regularizes the adapted token, ensuring it remains smooth and suitable for generation.Our single-token formulation resolves the spatial redundancies of the 2D latent space, simplifies architectures, and significantly reduces training costs. Despite its simplicity and efficiency, RepTok achieves competitive results on class-conditional ImageNet generation and extends naturally to text-to-image synthesis, reaching competitive zero-shot performance on MS-COCO under extremely limited training budgets. Our findings highlight the potential of fine-tuned SSL representations as compact and effective latent spaces for efficient generative modeling. We will release our model to facilitate further research.

Published as a conference paper at ICLR 2026
ADAPTING SELF-SUPERVISED REPRESENTATIONS AS A LATENT SPACE FOR EFFICIENT GENERATION
Ming Gui
Johannes Schusterbauer
Timy Phan 1.2
Felix Krause
Josh Susskind
Miguel Angel Bautista
Björn Ommer
'CompVis @ LMU Munich
Munich Center for Machine Learning (MCML)
*Apple
ABSTRACT
We introduce Representation Tokenizer (Rep'Tok), generative modeling frame- work that represents an image using a single continuous latent token obtained from self-supervised vision transformers. Building on pre-trained SSL encoder, we fine-tune only the semantic token embedding and pair it with generative decoder trained jointly using standard flow matching objective. This adaptation enriches the token with low-level, reconstruction-relewe construction. To preserve the favorable geometry of the original SSL. space, we add a cosine- loss that regularizes the adapted token, ensuring the latent space remains smooth and suitable for generation. Our single-token formulation resolves the spatial redundancies of the 2D latent space and significantly reduces training costs. Despite its simplicity and efficiency, RepTok achieves competitive results on class-conditional ImageNet generation and extends naturally to text-to-imann to-image syn- thesis, reaching competitive zero-shot performance on MS-COCO under extremely limited training budgets. Our findings highlight the potential of fine-tuned SSL rep- resentations as compact and effective latent spaces for efficient generative modeling. We release our code at https: /ttp//gittiaa/e com/ Seann/ComVVis///pee
details, enabling faithful image re-
INTRODUCTION
In recent years, diffusion- (Ho et al., 2020 Kingma et al. 2021 Song & Ermon 2019) and flow-
based (Lipman et al.) 2023: Liu et al. 2003b; Ma et al. 2024) models have emerged powerful
generative modeling frameworks, capable of synthesizing high-quality images (Ramesh et al. 2022]
Rombach al.) 2022, Dhariwal & Nichol 2021) and videos (Ho al., 2022). However, these models typically come with substantial computational demands since they regress vector fields in the high-dimensional pixel space of images. Latent Diffusion Models (Rombach al. [0022) address this challenge by decomposing the generative modeling task into two stages. By first compressing images into lower-dimensional latent space via pre-trained Variational Autoencoder (Kingma et al. 2013). LDMs abstract away imperceptible details, enabling the generation process to solely focus on semantic content and drastically reducing computational costs during training and inference (Esser et al., 2021 2024 Fuest et al.) 2024 [Schusterbauer et al. 2024).. However, despite these computational advantages, the latent space is still organized two-dimensional grid structure, which fails to exploit the high spatial redundancies inherent to natural images.
Recent efforts have sought to improve latent generative paradigms along two directions. TiTok Yu et al. 2024a) tries to exploit spatial redundancies and replaces the default 2D spatial grid in latent diffusion with a transformer-ban based encoder-decoder that represents images as ID latent sequences, achieving compact encodings with as few as 32 discrete tokens. In parallel, REPA (Yu et al. 2024b) leverages the rich representations of pre-trained self self-super learning (SSL) models to accelerate the convergence of latent diffusion models, by distilling the semantic knowledge into the diffusion model via a cosine similarity loss between their respective feature representations.
"quual Contribution
Experimentation, including use of pre-trained models, were completed by university collaborators.

---

Published as a conference paper at ICLR 2026
DIT-XU/2
SIT-XU/2
DIT-XL/2
REPA
SIT-XU/ REPA
Training GFLOPs
GFID-50K
9FID-S0K
50
-91.7%
10a
107
1023
1024
50K
400K TM
2M
4M
7M
Encader
Model
Total compute (GFLOPS)
Training Iteration
Figure I: Comparison of our single-token MLP-Mixer generator against transformer-based baselines models like REPA. RepTok attains competitive generative
(DiT, SiT), as well as representation-aliee performance while reducing training cost by over 90% owing to its compact latent space and lightweight architecture. All results reported without CFG. For fair comparison, we employ an encoder and decoder trained on general-domain data.
In this work, we extend these two directions by exploring more powerful uses of SSL representations. While REPA accelerates training primarily through feature alignment on the 2D spatial grid, we demonstrate that self-supervised models can be leveraged more directly: with minimal but crucial fine-tuning, pooled ID SSL representations themselves constitute effective latent spaces for generative modeling. These representations exhibit smooth, semantically structured geometry that well-suited for generation, while simultaneously eliminating the spatial redundancies inherent in 2D grid-based latents. Specifically, we show that the pooled 1D output from the [cls] s] token alone provides a compact yet expressive representation that not only captures high-level semanties but also preserves sufficient spatial detail to enable high-fidelity reconstruction.
Our Representation Tokenization approach, termed RepTok, builds a pre-trained SSL encoder that is lightly fine-tuned and trained jointly with generative decoder. We train the decoder with standard flow matching objective, complemented by ppsine-similarity loss that regularizes the latent representation to remain close to its original smooth and semantically structured space, which is well- suited for generation. Without auxiliary perceptual (Zhang et al. (018)) or adversarial (Esser et al.. 2021) losses, the resulting model is able to faithfully decode the single-token latent representation into the pixel space. This design enables highly efficient image synthesis training, allowing us to use simple, attention-free architectures such as MLP-Mixen (Tolstikhin al. 2021) for accelerated ImageNet training (see Figure D]. Furthermore, we show that the framework naturally extends to text-to-image (T2I) synthesis: by incorporating cros-attenttion to integrate textual conditioning, model achieves competitive zero-shot performance on the COCO (Lin et al. 2014) benchmark under an extremely constrained training budget (see Figure 7
We state contributions follows:
We show that self-supervised vision transformers can be used more powerfully than just guiding generative training: with minimal adaptation the semantic token, their smooth and semantically structured latent spaces can directly act encoders for generative modeling. By injecting the necessary fine-grained information into this semantic token, we enable faithful reconstruction while simultaneously eliminating the spatial redundancies inherent in 2D grid-based latents. Coupled with a generative decoder, this setup allows accurate image reconstruction from a single continuous token.
Exploiting this autoencoder design, introduce lightweight and optionally attention-free pipeline for latent generative modeling. This drastically reduces training compute while preserving quality, achieving competitive ImageNet generation at fraction of the cost of transformer-base based diffusion baselines.
We show that RepTok scales effectively to rext-to-image synthesis, achieving competitive zero-shot results on MS-COCO with under 20 hours of training on four A100 GPUs.
2 RELATED WORK
Latent space generation Early approaches such PixeIVAE and
al. 2016
Razavi et al. 2019 Van Den Oord et al. (017)) demonstrated that generative modeling within compact
latent spaces significantly improves sampling quality and efficiency. VQGAN (Esser et al.) 2021
integrates vector- quantized variational autoencoders with adversarial losses to construct discrete latent codebooks. Subsequently, these discrete tokens leveraged by autoregressive transformers
2

---

Published as a conference paper at ICLR 2026
Encoder-Decader Decoder Training
(8)
Latent Space Training
(D)
Image
Rupresentatoo Genorator
Inference
(c)
Generstive
Deccaler
Representation
Goncrater
Decoder
Figure Overview of our pipeline. (a) Joint fine-tuning of the [cls] token of SSL encoder & and training of the generative decoder 'D for image reconstruction. (b) Training of the generation model to synthesize frozen encoder outputs, which constitute the latent space, 2 = {zx). (c) Inference pipeline, where the latent space is first generated and subsequently decoded into the pixel space.
for image generation tasks. Latent Diffusion Models (LDMs) (Rombach et al. 2022) brought this
concept into the diffusion models, operating in learned spatial latent spaces that preserve semantic
content and abstract away perceptual detail. This approach has since become foundational across
modalities including images (Peebles Xie 2023] Ma et al. 2024: Pernias
2024). audio (Liu Blattmann et al. 2023b Kong et al. 2024h.
et al. 2023a), and video (Ho et al. 2022
Pre-trained representations in diffusion models Leveraging pre-trained representations has been shown to improve image generation Kouzelis et al. (2025); Yao et al. (2025). REPA Yu et al. 2024b Wang et (025) accelerates diffusion training by aligning diffusion features with DINO embeddings. Closely related to our approach is RCG (Li et al. 2024), which employs a two-stage pipeline: first generating predefined semantic representation and then transporting it to the pixel space. However, RCG primarily targets unconditional synthesis and thus leaves the representation space unchanged. In contrast, our objective is faithful reconstruction and generation, similar to the role of the latent space in VAEs. This requires not only semantic but also low-level visual information. We address this by injecting fine-grained details into the representation space, enabling both faithful reconstruction and high generative performance. Concurrent works like RAE (Zheng al. 2025 and SVG (Shi et al. 2025) use the full spatial grid of SSL features, thus operating in high-dimension structured latent space. Our approach is fundamentally different: RepTook relies solely the pooled semantic token, discarding all spatial tokens and learning to represent an image with a single compact vector. This yields a much more aggressive compression. Additionally, key difference SVG whose objective is aligning to the SSL representation, we directly employ the pooled semantic output as the latent representation itself.
Global information latent spaces Recent work has explored 1D tokenization beyond spatial grid latents. TYTok Yu al. 2024a) encodes images into compact sequences of as few as 32 discrete tokens with a ViT encoder, enabling efficient autoregressive generation. ElasticTok (Yan et al. (024) extends this idea with adaptive token counts per frame, while FlexTok (Bachmann al. (025) introduces variable-length ordered tokens for coarse-to-fine generation. Our approach differs in the following key aspects: First, we operate in continuous latent space, avoiding quantization and enabling fully differentiable diffusion training. Second, we directly exploit the [cls] ] token of SSL vision transformers as compact latent, yielding smooth and semantically structured manifolds. Unlike discrete tokenizers, Diffusion Autoencoders (Preechakul et al., (022) extract semantic information into a continuous latent space and utilize a jointly trained diffusion model for reconstruction. As the latent space is mostly semantic, image reconstruction requires an additional subcode æ7, obtained by mapping the image back to the Gaussian noise space using conditional DDIM sampling (Song et al. 2020). By contrast, concurrent work, AToken (Lu et al.) 2025), proposes a unified visual tokenizer designed to operate consistently across multiple modalities.
method reconstructs images faithfully from single latent alone. A
3

---

Published as a conference paper at ICLR 2026
ImageNet-1K
MS-COCO
GT
TiTok
Fok
Ours
GT
TYTok
FlexTok
Ours
32
16
16
4
Figure 3: We introduce Replok, a compact visual tokenizer that builds upon pre-trained SSL represen- tations. Our approach augments these representations with additional necessary information to enable images to be faithfully encoded single continuous token, which allows for both high-fidelity image reconstruction and synthesis. The third row indicates the number of tokens for reconstruction.
3 METHOD
3.1 PRELIMINARIES
Flow Matching models learn vector fields that map between two terminal distributions: p(ae), typically simple prior distribution such standard Gaussian distribution, and p(æ1), the target data distribution. Let Rd be the space that 2o and Iı reside in, and let ug (t, represent the time- dependent vector field to be learned with & [0, 1]. The underlying dynamics of flow matching models are then governed by the ordinary differential equation (ODE) dr = ng (xr t). A common choice for the interpolant between zp and Tı is the linear interpolant (Liu al.) (023b), defined
as 1 + (1 t) The vector field uh can then be optimized using the following training
objective with randomly sampled f and the corresponding xy (Lipman al. 2023
Liu al. 2023b
Schusterbauer et al. 2025):
L = Re lIvo (x+, t) (#1 - -
To sample from a flow matching model, one simply integrates along the trajectory defined by the learned ODE, This can be accomplished using numerical integration techniques such the forward Fuler method, with the update rule given by Fitts = 2t + LA Vo (It, where V/ E [0, 10,11,44 = 1/N, and N being the number of function evaluations (NFE).
3.2 REPTOK: REPRESENTING IMAGES AS A SINGLE TOKEN
TYTok (Yu et al. 2024a) represents significant advancement over traditional VAEs by overcoming their inherent 2D tokenization grid constraints. Unlike conventional approaches, where each token is restricted to attending only to a fixed image grid, TiTok enables tokens to attend freely to the entire image. However, despite these improvements, TYTok typically still relies on multiple tokens to effectively encode an image. In this work, we show that continuous latent spaces can achieve even greater efficiency few-token regimes. Specifically, we demonstrate that a single continuous token, derived from a pre-trained encoder, can be used together with a generative decoder to synthesize high-fidelit, reconstructions.
Finetuned Self-supervised Models are Faithful Encoders It well established that models such as CLIP (Radford et al. 2021), MAE (He et al. 2022) and DINO (Caron et al., 2021 Oquab et al. 2024) models encode highly informative representations and demonstrate strong understanding of images, as evidenced by their effectiveness in various downstream tasks, including image classification
(Radford et al., [02T Caron et al.) 202T]] Oquab al. 2024) and semantic segmentation (Zhang et al. 2023). This capability is further demonstrated by the existence of unCLIP models (Ramesh et al.
2022 Rombach al. 2022), which can generate image variations from noise using only a single CLIP embedding. While this observation confirms that generative models can synthesize images from extremely compact bottlenecks (for unCLIP RRamesh et al. 2022 z E R 1Ix112) ), we hypothesize that the variations of the outputs arise from the fact that CLIP models are not explicitly trained to preserve exact pixel locations but instead optimize a contrastive loss with corresponding textual deseriptions, thereby capturing only high-level semantic features.
Motivated by these observations, we explore and unlock the potential of leveraging a pretrained encoder that already possesses comprehensive understanding of image content. To this end,
4

---

Published as a conference paper at ICLR 2026
Image A
Interpolation
Image B
Figure 4: Latent space interpolation. We observe smooth transitions not only in semantic content but also in spatial configuration. This indicates that our method successfully integrates low-level spatial information while preserving the properties of the pretrained encoder's latent space, and facilitates generation within the learned representation. We provide more samples the Appendix.
introduce a novel training strategy that leverages pretrained self- seff-supervised learning (SSL) model with a transformer- based architecture as the encoder. These models typically incorporate class token (typically referred to as the [cls] token) that is trained, either explicitly or implicitly, to aggregate information from image patches. However, such pretrained models are often optimized for downstream tasks and may consequently, as an example, underrepresent low-level visual details propose targeted adaptation strategy
critical for image reconstruction. To address this limitation, that only updates the class token embedding while keeping the remainder of the encoder frozen. Remarkably, find that this minimal intervention is sufficient inject the necessary visual detail into the representation. Empirical results reveal that with only the class token being fine-tuned, the system is capable producing reconstructions with high fidelity across range of SSL backbones including DINOv2 (Oquab et al. 2024), MAE (He et al. 2022) and CLIP (Radford et al. 2021) We demonstrate our reconstructions Figure
Training the Encoder together with Generative Decoder While the SSL-pretrained encoder remains largely frozen, a supervisory signal is still required to inject reconstruction-relevan information into the class token. Additionally, decoder is necessary to map the resulting single- token latent representation back into pixel space. To this end, we jointly train the encoder and generative decoder D in continuous manner, using simple but effective flow matching loss.
The generative decoder is trained end-to-end alongside the encoder to learn a mapping from randomly sampled Gaussian noise the target image We follow principles similar to the conditioning mechanism employed in MMDiT (Esser et al. (024) and concatenate the latent token Exx) with the noisy image tokens. The resulting training objective is formulated as a flow matching loss as in Equation (I. which optimizes both the encoder and the decoder:
C = Er (t, Tz, 2) - (z1 zo)ll
To improve computational efficiency and remain consistent with the SiT framework (Ma et al. 2024). we adopt pretrained 2D VAE (Rombach et al. 2022) so that the generative decoding process operates within a learned latent space rather than directly in pixel space.
Cosine-Similarity Loss We observe that the [cls] ] tokens of self- supervised vision encoders already provide a smooth, semantically structured space. Hence, our goal during training is to maintain this well- rell-reeuuar space while still allowing the token to integrate the fine-grained information the decoder needs for faithful reconstrue- tions. Freezing the [cls] token leads to poor reconstruction quality, as indicated Figure 5 Conversely, leaving the encoder completely unconstrained pulls the token far away from the well-regularized space, removing the prerequisite for later generative modeling. We find that only unfreezing the [cls] while fixing all other encoder weights strikes a good balance between integration of more information and maintaining the original regularization. To constrain the token from deviating its pre-trained representation, introduce
Figure 5: Fine-tuning the [cls] token. From left: GT, frozen, finetuned.
5

---

Published as a conference paper at ICLR 2026
Table 1: State-of-the-att comparison between tokenizers for reconstruction and class-conditional
ImageNet generation. { metrics sourced from [Bachmann et al. 2025)
Tokenizer
tokens
global
continuous
rFID
gFID
LDM (Rombach
2022
32x32
0.90
7.76
LlamaGen (Sun et
2024
16x16
X
2.19
3.06
TiTek-L'
Yu al
2024a
32
V
X
2.21
2.77
TīTek-B
Yu al
2024a
64
1.70
2,48
TiTok-S' Yu el al.
X124a
128
X
1.71
1.97
FlexTok!
a12-d12
Bachmann et al
2025
1-256
X
4,20
3.83
FlexTok' d1 8-118
Bachmann et al
2025
1-256
X
1.1
2,02
FlexTok" d13 d18-d28
Buchmann et al
2025
1-256
X
1.45
1.86
RepTok (owrs)
1.85
1.88
cosine-similarity alignment term
Leos (z) X(1 cos(2, 2trozen))
Zrrozen
Etrozen (I)) E(x).
(3)
where Zfrozen is the token output from the frozen SSL model, is the fine-tuned counterpart, and explicitly controls the allowed deviation. Reducing À relaxes the constraint; increasing it restricts the token more tightly to its source. With this alignment mechanism, we retain the well-behaved SSL latent space for later generative modeling, while additionally enriching the token with the additional information the generative decoder needs to faithfully reconstruct. We observe that incorporating the cosine similarity loss prevents the embedding from drifting away from the well-regularized latent space, also under extended training, as illustrated in Figure We directly condition the generative decoder on those representations and focus on preserving their structured properties while injecting additional information to enable both faithful reconstruction and generative abilities.
3.3 SINGLE TOKEN GENERATION FOR IMAGE SYNTHESIS
Since RepTokk projects images into a continuous latent space = (typically in R1x768), it becomes feasible model and sample from this space using separate generative model G. To this end, we again employ flow matching (Lipman et al., 2023) for latent space generation. We discover that utilizing a frozen SSL. model, with only the class token finetuned, provides an effective alternative regularization mechanism to the conventional approaches using Kullback-Leein Leibler (KI.) divergence (Rombach al. [022) or vector quantization (Austin al., 2021 1. Yu al.) [2024]] Tian et al. 2024). By preserving the structural properties of the learned feature space, the frozen encoder inherently constrains the latent representations and facilitates the generation process without requiring explicit
KL or vector quantization regularization.
Attention-free ImageNet Generation Typical diffusion models operate on high-dimensional image latent spaces consisting of multiple tokens, where capturing global structure and local detail relies on modeling interactions across tokens. This is commonly achieved through attention (Vaswani et al.) 2017). While effective, introduces significant computational overhead. contrast, when inputs are aggressively compressed into single token, token-to- to-kkee-- interactions become unnecessary. We show that in this highly compressed regime, generative modeling can be effectively performed using an attention-free, pure MI.P-based architecture such as MI.-MMxx (Tolstikhin et al. (202T). Despite its architectural simplicity and lack of self-attention, our MI.P-only approach performs remarkably well. This highlights novel and computationally efficient approach to diffusion modeling, where architectural complexity is shifted to the pre-trained compression stage without sacrificing flcxibility generality. For text-to-image synthesis, still use attention for text conditioning, but the compactness of our latent space keeps the associated cost minimal. In particular, because the number of tokens in latent space small, the quadratic scaling of attention remains inexpensive.
EXPERIMENTS
We evaluate RepTok classccon ImageNet-Ik (Deng al. 2009) and show the scalability of our approach on text-to-image -moge (T2I) generation. We evaluate reconstruction performance with reconstruction FID (rFID), PSNR, SSIM, and LPIPS, and generation performance with generation

---

Published as conference paper at ICLR 2026
Table FID comparison on the ImageNet 256 256 benchmark, including parameter counts and training FLOPs. Stage refers to the training of the generative decoder, while Stage 2 corresponds to the main generative model training. As all models rely on the SD- VAE and REPA uses DINOv2 as well, we exclude these shared pre-training costs from FLOP estimates.
Stage
Stage 2
Model
FIDD
PFlops
Train Steps
Params (M)
GFlops/Iter
PFlops
Total PFlops
DiT-X1/2
19.5
400K
675
118.6
12.1K
12.1K
+REPA
12.3
400K
675
140.5
14.4K
14.4K
SYT-L/2
18.8
400K
458
77.5
7.9K
7.9K
+REPA
9.7
400K
458
99.4
10.2K
10.2K
SiT-XL/2
17.2
400K
675
118.6
12.1K
12.1K
+REPA
7.9
400K
675
140.5
14.4K
14.4K
SiT-XL/2
8.3
7M
675
118.6
212.5K
212.5K
+CFG=1.5
2.06
675
118.6
212.5K
212.5K
+REPA
5.9
4M
675
140.5
143.9K
143.9K
+REPA, CFG=1.5
1.42
4M
675
140.5
143.9K
143.9K
RepTok
5.4
30.4K
100K
276
23.0
0.6K
31.0K
RepTok
3.4
30.4K
700K
276
23.0
4.1K
34.5K
+CFG=1.5
3.22
30.4K
700K
276
23.0
4.1K
34.5K
RepTok-L
2.06
30.4K
460K
516
25.0
11.7K
42.1K
+CFG=1.5
1.88
30.4K
460K
516
25.0
11.7K
42.1K
FID (gFID), consistent with prior work (Bachmann al.
2025
Yu al. 2024a). All models operate
at 256 resolution; implementation and training details are provided in the Appendix.
4.1 CLASS-CONIE CONDITIOALL GENERATION
We jointly train the SSL encoder (only the [cls] ] token parameters trainable) and generative flow matching- -ecoodr for reconstruction in a first stage. We use DINOv2 (Oquab et al. 2024) as SSL encoder, but show in Section 4.3 that our method also generalizes to other SSL methods. For latent space synthesis, we train lightweight, attention free generator (MLP-Mixer) over the continuous [cls] token, where encode images using the previously trained SSL encoder model. We inject class information by concatenating learned class embedding, which we randomly drop during training enable classifier-free guidance (Ho & Salimans 202T). We apply CFG in limited interval E [0.3, 0.9] following Kynkäänniemi et al. (2024).
Quantitative Comparison Table 2 compares our method against recent, state-of-the-art transformer- based generative models on Ima- geNet 256 x 256. For each model, report the FID score, number of training iterations, parameter count, per-iteration compute GFLOPs, and the resulting total training compute in Peta-FLOPs. FLOPs are estimated from a single forward pass (batch size 1), and scaled linearly with the effective batch size and the number of train- ing steps; we follow the convention of counting only the forward pass. Our model achieves highly competitive FID scores while re- quiring significantly less total compute than other baselines such as DiT and SiT. We note that classifier-free guidance (CFG) yields only limited improvements our setting, a phenomenon also re- ported by RCG.
Table Reconstruction per- formance ImageNet 2562.
FID@50K
PSNR 1
RCG
3.20
9.31
Ours
1.85
14.94
Rep'ook with spatial and 1D tokenizers for both reconstruction and class-conditional generation on ImageNet. Despite using just one continuous token, RepTok matches even outperforms several spatial and non-spatial baselines in rFID while remaining competitive on gFID relative to recent discrete tokenizers. Additional results Table S compare RepTok to RCG (Li al. 2024), method which relies purely semantic codes. RepTok achieves significantly higher PSNR and lower FID, indicating that our continuous token preserves more information than pure semantics and delivers stronger performance across both perceptual and pixel-level metrics.
7

---

Published as a conference paper at ICLR 2026
Figure 6: Uncurated MLP-Mixer ImageNet generations (CFG=3.5). More samples in the Appendix.
Text-to Image Training Efficiency
Text- to-t-t--ln toxt-to-image Performance
Zero-shet FID-30k
Würstchen
50
20
0.24
D
40
10
PisAiig
Ours
0.23
10?
10a
10a
FID.5k
CLIP Score
Training days on 8xH100
(123
M)
Gemma (2.5
Figure 7: Left: Training days vs gFID, zero-shot evaluation on MS-COCK (Lin et al. 2014). Data sourced from MicroDiT (Sehwag et al. 2024). Right: Scaling the frozen language backbones results in improved performance. Language models: CLIP, VL, and Gemma-2B.
Efficiency We measure training compute in floating point operations (FLOPs). In the single-token latent space, token-to-token interactions are unnecessary. We therefore adopt a pure MLP-Mixer the latent space generator model. The combination of representing an image with single token and the MLP-only architecture reduces training FLOPs by an order of magnitude compared to attention-bast based diffusion in latent space, shown in Figure [1] Despite comparable number of parameters across both models, our approach still achieves substantially lower computational footprint, requiring only 1.7% of the FLOPs consumed by SiT (Ma et al. 2024). Our overall FLOPs remain significantly lower, also when accounting for the inference cost of the corresponding first-stage encoder.
Qualitative Comparison Figure 3 shows high-fidelity reconstructions from a single token
ImageNet validation images and strong out-of-distributioo reconstructions on MS-COCO (Lin et al. 2014), despite training only on ImageNet. Figure 6 presents class-conditional samples; despite the simple architecture and low compute budget, quality remains competitive with attention-based image generation models. We provide more uncurated, qualitative samples in the Appendix.
Latent Space Interpolation A key advantage of self-supe encoders is the smoothness of their latent spaces, yielding a geometry well-suited for generation. F ssows that our training preserves this property, where we linearly interpolate between latent representations, which produces gradual transitions in both high-level semantics and low-level visual details. We observe continuous changes in object shape, size, emergence, and rotation (see more samples in the Appendix).
4.2 ENABLING REPTOK FOR T2I GENERATION
We scale RepTok to text-to-image generation using 120M image- text pairs from COYO (Byeon al. 2022), recaptioned using VL3- 1B (Zhu al. 2025). We first train the magguage-agnostie
Table 4: Our approach generalizes to other self-supervised encoders. class-conditional ImageNet (Deng et al. 2009).
We compare 10k FID on
SSL method
rFID J.
PSNR
SSIM
LPIPS
gFID 1
w/o prior
13.99
19.64
47.19
0.23
128.54
CLIP Radford al. 2021]
13.66
14.24
31.69
0.44
30.56
MAE al 2014
9.09
13.79
30.28
0.45
28.48
DINOVZ Oquab et al 2024
7.95
14.94
33.26
0.41
20.75
8

---

Published as a conference paper at ICLR 2026
cactus with cartoon hoppy An avocada chair the shape Sahara dessert
cat wearing round sunglasses lying on beach
close-up of horse's face with details of mone and eyes
halved avocado
Figure 8: RepTok text-to-image results with taasformer-basee latent space model (CFG scale 3.5).
encoder-decoder using DINOv2 our SSL encoder and a Flow Matching transformer as decoder. During generative model training, we concatenate four learnable tokens with the noisy [cls] token from the SSL encoder and apply cross-attention to the frozen outputs of the language model. Similar to prior work, we evaluate our method on the COCO validation set (Lin al. 2014). We report FID, CLIP Score (Hessel al. 202T), as well as validation loss, as (Esser et al. 2024 Polyak al. 2024 found that it correlates with human evaluations.
Quantitative Results Figure 7 (left) shows that our method achieves substantially lower training cost than prior text-to-image models while maintaining competitive zero-shot FID. Since the language backbone frozen and only provides conditioning, it can scaled independently without impacting the training of the generative model. Figure 7 (right) shows the performance for language backbones with increasing scale: CLIP (Radford et al. 2021), Intern (Zhu et al.] (025). and Gemma-2B (Team-Gemma al. 2024 Larger language models consistently improve performance across all metrics. All results are obtained after 200k training iterations with a batch size of 256.
Fin.eince
Qualitative Results Figure 8 shows qualitative text-to-image results. Our model is able to produce realistic images after only 200k training iterations. Despite the short training time (< 20 hours on 4x 110 GPUs), the generations capture fine details and adhere closely to the prompt. This highlights the efficiency and scalability of Rep'Tok for text-to-image synthesis. Interestingly, we observe that the SSL. encoder and generative decoder trained exclusively on ImageNet can already be repurposed for text-to-image generation. We show qualitative samples and discuss this further in the Appendix.
4.3 ABLATIONS
Generalization to other SSL methods Our method generalizes to a number of self-supervised vision encoders, shown in Table While the main results based DINOv2, observe similarly strong reconstruction quality and generative performance when using alternative SSL methods such MAE and CLIP. In contrast, when using a randomly initialized encoder with no prior information, the generative decoder loss enforces a strong pixel-wise reconstruction but leaves the resulting latent space completely unstructured and hard capture for the generative model, as reflected in the high generation FID. A semantic prior enforces a geometry in which semantically similar images are drawn together and dissimilar images are pushed apart. This naturally induces smooth, low-dimensional manifolds which promotes stable generations.
Cosine Similarity Loss We introduced a cosine similarity loss in Equation (3 that incentivizes the semantic token to remain close to the SSL encoder's original to preserve the beneficial properties of the pre-trained space. Here, similar to previous work Yao et al. 2025) Tschannen et al. 2024). Stronger
observe trade-off between generation and reconstruction, visualized in Figure regularization improves the generative performance (gFID), but at the cost of reduced pixel-wise reconstruction (PSNR). Mild regularization significantly improves the generative quality, indicating

---

Published as a conference paper at ICLR 2026
Cosine Similarity
1,00
0.75
Inverted PSAR
0.1
0.01
14
20
20k
Sok
1ock
150k
10 Cosine Loss Weight A constrined
10.2
10
Training Steps
rellsed
Figure 9: The parameter À of the cosine similarity loss in Equation allows to trade off between pixel-wise reconstruction and generation capabilities. Relaxed constraints (low A) improve pixel-wise reconstruction (PSNR in right plot), but result in poor generation capabilities (gFID in right plot).
MLP Mixer Scaling Properties on ImageNet 256x256
0.65
500
Valication Loss
Parameters (MS
RID 10k
0.60
300
200
B0k Taininn Steps
Training Steps
Model
Figure 10: Scaling analysis of the MLP Mixer Tolstikhin al. 2021 scales: larger variants consistently yield improved performance, and continued training further improves results. The largest XL model evaluated contains 516M parameters.
We observe that the model
better latent space for generation, while minimally degrading reconstruction quality. À allows to balance between preserving high-level semantic content and reconstructing low-level visual details.
Scaling Analysis of MLP-Mixer Since parameter scaling a key feature of Transformers, we further evaluate whether this also holds for latent space MLP-Mixer architecture (Tolstikhin et al. 2021 In Figure 10 we show that the MLP Mixer also scales with parameter size and FLOPs.
Inference Speed We report wall-clock inference time for each component in Ta- 5). Our MLP-Mixer typically saturates 40 NFEs, while the SiT-based decoder saturates at 25 NFEs for reconstruction quality (Figure S3}. In comparison, SiT reports saturation at 64 NFEs for the ODE solver (and 256 for SDE). Our overall inference time is dominated by the generative decoder, while the MLP-Mixer generator adds minor overhead.
Table 5: Inference speed comparison.
(ms)
Time
STEXI
64
118.6
38.0
2,43
Ours
MLP-Mixer
3.1
Decener (SIT-XL.)
1186
3800
5 CONCLUSION
In this work, we introduced RepTok, framework that adapts celf-supervised representations into a compact latent space for generative modeling. By fine-tuning only the class token of an SSL encoder and regularizing it with a cosine-similarity loss, we obtain a single continuous token that retains the smooth geometry of the original space while enriching it with reconstruction--ellvann information. Coupled with a generative decoder trained via flow matching, this setup enables faithful reconstructions and efficient image synthesis without reliance costly attention mechanisms or auxiliary losses. Our experiments demonstrate that this single-token formulation achieves competitive results in class-conditional generation fraction of the computational cost. We further show that RepTok scales to more complex text-to- to-iagge settings. Overall. these findings highlight the potential of leveraging SSL representations themselves build lightweight but effective generative models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like thank Nick Stracke for helpful discussions and feedback. This project has been supported by the bidt project KLIMA-MEMES, the German Federal Ministry for Economic Affairs and Climate Action within the project "NXT GEN AI METHODS Generative Methoden für Perzeption, Prädiktion und Planung". the project "GeniusRobot" (011S24083) funded by the Federal Ministry of Research, Technology and Space (BMFTR), and the Horizon Europe project ELLIOT (GA No. 101214398). The authors gratefully acknowledge the Gauss Center for Supercomputing for providing compute through the NIC on JUWELS/JUPITER at JSC and the HPC resources supplied by the NHR @ FAU Erlangen.
REFERENCES
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing 2021.
systems, 34:1
Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar, Elmira Amirloo, Alaacldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences of flexible length. arXiv preprint 2XXi:25502.1367. 2025.
Andreas Blattmann, Tim Dockhom, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311. 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22563-22575, 2023b.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset.
coonkaaaobbain/
coyo-dataset 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2021.
Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z. Kaplan, and Enrico Shippole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers, 2024. /ttpttss:: http:///wwwwwww..////oo/o
11000......... 11605
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Patrick Esser. Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-furst international conference machine learning, 2024.
Michael Fuest, Pingchuan Ma, Ming Gui, Johannes Schusterbauer, Vincent Tao Hu, and Bjorn Ommer. Diffusion models and representation learning: survey. arXiv preprint arXiv:2407.00783 2024.
11

---

Published as a conference paper at ICLR 2026
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: latent variable model for natural images. arXiv preprint arXiv:1611.05013 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv:2104.08711 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Diederik Kingma, Max Welling, et al. Auto-cncoding variational bayes, 2013.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong. Xin Li, Bo Wu, Jianwei Zhang, et Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:24/2.03600 2024.
Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Ko- modakis. Boosting generative image modeling via joint image-feature synthesis. arXiv preprint MrXiv:2504.16064 2025.
Tuomas Kynkääänieemi, Miik Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. Advances in Neural Information Processing Systems, 2.P:12225812244 2024.
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised representation generation method. Advances Neural Information Processing Systems, 37: 125441-125468, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, pp. (1500-21474, 2023a.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL http:s///pppeeeeeeeeeeeeeee/rrre
Bkg6RiCqY7
Jiasen Lu, Liangchen Song, Mingze Xu, Byeongjoo Ahn, Yanjun Wang, Chen Chen, Afshin Dehghan, and Yinfei Yang. Atoken: A unified tokenizer for vision. arXiv preprint arXiv:2509. /4476, 2025.
12

---

Published as a conference paper at ICLR 2026
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- and Saining Xie. Sit: Exploring flow and diffusion-baan based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23 40. Springer, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp. 1-31, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4201 2023.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. Würstchen: An efficient architecture for large-scall text-to-image diffusion models. In The Twelfth International Conference Learning Representations (ICLR 2024). OpenReview, 2024.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13721 2024.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 10619-10629, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PmLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URLphtts::/axxx..ooc////
061 25
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Johannes Schusterbauer, Ming Gui, Pingchuan Ma, Nick Stracke, Stefan Andreas Baumann, Vin- cent Tao Hu, and Björn Ommer. Fmboost: Boosting latent diffusion with flow matching. In European Conference on Computer Vision, pp. 338-355.. Springer, 2024.
Johannes Schusterbauer, Ming Gui, Frank Fundel, and Bjöm Ommer. Diff2flow: Training flow matching models via diffusion model alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28347-28357, 2025.
Vikash Sehwag, Xianghao Kong, Jingtao Li, Michael Spranger, and Lingjuan Lyu. Stretching each dollar: Diffusion training from scratch on a micro-budget. arXiv preprint arXiv:2407.15811, 2024.
Noam Shazeer. Glu variants improve transformer, 2020. URL https: http:::/axxiv..ooo http:://arxxv.oor///bb..
2002. 05202
Minglei Shi, Haolin Wang. Wenzhao Zheng, Ziyang Yuan, Xiaoshi Wu, Xintao Wang, Pengfei Wan, Jie Zhou, and Jiwen Lu. Latent diffusion model without variational autoencoder. arXiv preprint arXiv:2510. 15301, 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02550 02502, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.
Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2026
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. 2023. URL hullhttps///ann httppp:://arxi...o orq/ wttp//////////ooo//////////aaae/ /al//ba///2/ /204
09864
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation, 2024. URL https
//arxiv. org/ abs 2406. 06525
Gemma Team-Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-84865, 2024.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative anfinit-vvccabllig transformers. In European Conference on Computer Vision, pp. 292-309. Springer. 2024.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ziqiao Wang, Wangbo Zhao, Yuhao Zhou, Zekai Li, Zhiyuan Liang, Mingjia Shi, Xuanlei Zhao, Pengfei Zhou, Kaipeng Zhang, Zhangyang Wang, et al. Repa works until it doesn't t: Early-stopped, holistic alignment supercharges diffusion training. arXiv preprint arXiv:2505. 16792, 2025.
Wilson Yan, Volodymyr Mnih, Aleksandra Faust, Matei Zaharia, Pieter Abbeel, and Hao Liu. Elastictok: Adaptive tokenization for image and video. arXiv preprint arXiv:24 10.08836, 2024.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501.01123, 2025.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37:128940-12896, 2024a.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.069400 2024b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL https: //axxiv.crr/abs//1110
Junyi Zhang, Charles Hermaann Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming- Hsuan Yang. tale of two features: Stable diffusion complements dino for sero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36:45533-45547,, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018
Boyang Zheng, Nanye Ma. Shengbang Tong, and Saining Xie. Diffusion transformers with represen- tation autoencoders. arXiv preprint arXiv:2510.1169) 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.
14

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIAL:
ADAPTING SELF-SUPERVISED
REPRESENTATIONS AS A LATENT SPACE FOR EFFICIENT GENERATION
A IMPLEMENTATION DETAILS
Generative Decoder Our generative decoder is implemented DiT-XL/2 (Peebles & Xie 2023 and trained for one million steps with a learning rate of 10 using the AdamW (Loshchilov & Hutter 2019) optimizer, linear warm-up of 2000 steps and global batch size of 512 8 H100 GPUs. Our implementation uses RoPE (Su et al., [ Crowson et al. 2024). RMSNorm (Zhang & Sennrich 2019) and SwiGLU (Shazeer 2020) activation functions, we find that these modifications improve the stability and performance of our generative decoder. We concatenate the SSL embedding to the decoder patch tokens and apply full self-attention over all tokens.
MLP Mixer We adopt standard MLP-Mixed (Tolstikhin al. 2021 architecture, where all conditioning information: CLIP text embeddings for text-to-image (T21) generation and class tokens for class-condin conditonnii image generation is concatenated with the noisy image token and passed through the model. Our implementation follows the configuration provided by the lucidrains 1 GitHub repository, with hidden dimension of 1280, depth of 28 layers, an expansion factor of 4 for the channel MLP, and 2 for the token MLP.
B ADDITIONAL RESULTS
Qualitative examples per token type. As discussed in the main paper, DINOv2 (Qquab et al. 2024) offers two different types of tokens (besides patch tokens). First, the standard [cls] ] token and additionally a set of register tokens (Darcet al. 2024). In Figure (S4 we provide a qualitative comparison of the differences in outcome between these two token types. We keep the SSL back- bone frozen and only train our generative decoder. We can observe that the [reg] token contains more knowl- edge about appearance, location, and object orientation compared the [cls] token. However, none of the ap- proaches gives proper pixel-wise reconstructions, again highlighting the need to integrate further information from the SSL encoder.
GT
[cis]
[reg]
Figure S4: [cls] vs [reg] qualitative comparison.
Performance vs test-time compute. Figure S3 shows the number of function evaluations (NFE) vs reconstruction FID (rFID) on the ImageNet et al. (2009) validation dataset. Performance improves with increasing number of function evaluations (NFE), but saturates around 20. We hypothesize that the strong conditioning signal from the generative decoder reduces the need for additional refinement steps.
Token type DINOv2 (Oquab et al. 2024) pro-
Table 6: Ablation of token type. Conditioning on DINOv2's (Oquab et al.) 2024) register tokens improves pixel-wise metrics, indicating stronger local information.
vides access to both a [cls] J token and a set of register tokens. We compare their usefulness as latent representations for our generative decoder in Table Using frozen [cls] ] token results in strong reconstruction FID, indicating good semantic alignment, but yields low pixel-level scores such as PSNR and SSIM. In contrast, the register token captures more fine-gra grained visual details, improving pixel-wise reconstruc- tion quality. This suggests that while the [cls] token emphasizes semantic content, the register token retains more low-level and regional information.
Token
rFID
PSNR 1
SSIM 1
LPIPS
[reg] [cls]
14.90
12.85
29.07
0.52
14.13
12.59
28.41
0.54
https:://gii http://ggtttuuuoooo
15

---

Published as a conference paper at ICLR 2026
Image A
Interpolation
Image B
Figure S1: More single token latent space interpolation results. We observe smooth transitions only in semantic content but also in object spatial configuration, and especially in object rotation (see dog).
Image A
Interpolation
Image
unCLIP
Ours
Figure S2: Qualitative interpolation comparison unCLIP Ramesh al. (202)); Rombach al.
(2022) The results show that representations from unCLIP primarily capture semantic information and lack low-level detail, leading to less coherent transitions. contrast, our approach preserves both semantic and structural continuity, enabling visually consistent interpolations. We use the pretrained Stable Diffusion 2.1 unCLIP checkpoint.
Scaling the Decoder to 512px We further fine-tune our encoder-decoder model for 100K iterations at 512" resolution, starting from weights pre-trained at 2562. The model successfully adapts to the higher-resolution setting as visualized in Figure 56
More qualitative samples We provide additional qualitative results to further illustrate the capa- bilities of our model: text-conditional generations shown in Figure S10) and uncurated class- conditional ImageNet generations in Figure S12
Additional Tokens Some SSL encoders, such as DINOv2, provide additional global register tokens beyond the [cls] token. Incorporating these tokens increases latent capacity and improves reconstruction quality (see Figure SS]] left). However, these tokens are typically unregularized and therefore do not inherit the favorable semantic and well-structured properties that our approach relies This also shows in the worse generative performance (gFID in Figure S7 right). Moreover, using multiple global tokens requires SSL models that have additional non-spatial tokens, as there is no straightforward way to repurpose spatial tokens into meaningful global representations. Given focus on efficiency and structured single-token representations, we therefore restrict our method to the [cls] token, though multi-token extensions remain interesting direction for future work.
16

---

Published as a conference paper at ICLR 2026
CLI
InterVLL
Gemma28
63
55
FID 10k
50
53
45
40
30
Decoder NFE
CLIP Score
Figure S3: Effect of decoder inference steps (left) and effect of CFG scales (right). We evaluate both on the COCO validation set[Lin al. (2014). More decoder inference steps yield better decoding results. CLIP score rises with larger CFG scales, while FID improves only within moderate range.
CLIP
MAE
DINOV2
wio prior
20
FFD 10k
26
15
14
50k 100k
150k
200k
250c
3004
50k
100c
150k
2002
250k
303k
Training lberation
Taining Iteration
Figure S5: Comparing SSL priors over training steps. Our approach generalizes to different self-supervised methods. While the unregularized model without prior knowledge shows remarkable pixel-wise reconstruction, the latent space is not amenable for generation (see Table 4 and Figure
C LIMITATIONS
While our single-token representation enables highly efficient generation and significant compute savings, it may limit expressiveness in capturing fine-grained details, particularly for complex or high-resolution scenes. Extending our approach to support richer multi-token representations while preserving efficiency is an interesting direction for future work. While our experiments demonstrate that the single-token embedding preserves certain low-level spatial structures, achieving fine-grained control over object location and scene composition remains an open challenge.
Snnonstructtin-Geeeaattiie trade-off Another limitation of our method lies in the trade-off im- posed by cosine similarity regularization. While stronger regularization enhances the smoothness and structure of the latent space, which crucial for stable generative modeling. it can also suppress low- level detail, leading to degraded pixel-wise reconstructions. This trade-off may limit the applicability of our approach in scenarios where very high visual reconstruction fidelity is critical.
Unleashing T2I for ImageNet- Autoencoder We investigate the capabilities of our amage-trained encoder-decoder framework. Figure [STT] shows qualitative text-to-image samples. Despite being trained exclusively on ImageNet, the latent space does not overfit and shows strong
GT
Recon
Figure S6: Fine-tuned secoder-decoder model on the higher 5122 resolution.
17

---

Published as a conference paper at ICLR 2026
[cls]
[cls] reg
[els] reg tokens
[ola] tokens
10k
PSNN (B8)
10k
FFID
100k
200k
2008 400k
100k
2004
b00k
4004
100k
200k
2008
400k
10k
20k
40k
Training Steps
Training Steps
Training Steps
Training Steps
Reconstruction
Generation
Figure S7: Token number ablation. Increasing the number of tokens form of additional register tokens from DINOv2 Darcet al. al.] (2024) improves reconstruction quality. However, since register tokens do not have the favorable SSL properties, their space is not amenable for generation.
5K
10K
30K
80K
Figure S8: Uncurated coss-conditional ImageNet generation results over training iterations (5k, 10k. 30k, and 80k). Note that our model produces good results as early as 30k training steps.
generalization, generating diverse and high-quality images that extend well beyond the ImageNet manifold. Although the model generates plausible images, find it struggles with compositional prompts that require placing multiple objects within a scene (e.g., a cat and a dog side-by-side). This limitation is expected, since the object-centric bias of ImageNet offers little exposure multi-object scenes. However, finetuning our encoder on more diverse data alleviates this issue and enables the generation of multi-object content.
18

---

Published as a conference paper at ICLR 2026
Reconstruction
Generation
GT
Random
Ours
Random
Ours
Figure S9: Qualitative comparison between refers to class-conditional samples with the same class as the corresponding GT image. While random initialization achieves stronger pixel-level reconstruction, lacks the structured priors of pre-trained self-supervised encoders, resulting in poor generative performance. In contrast, our method balances reconstruction and generation.
encoder and ours. Generation
coT the snow with blue eyes
fierce lion, colorful, low- poly. poly-hd, polygon mesh
cute puppy swimming in the water
living room, bright modern Scondinavian style
Figure S10: Additional text-to-image generation results with CFG scale of 7.5 and RepTok encoder- decoder trained on the COYO dataset.
cup of coffee on wooden table
child's drawing of grassland with wild flowers
ascending into clouds
A landscape with rolling sond dures
Figure S11: T2I generation results (CFG scale 3.5), using RepTok solely trained on ImageNet data with latent space transformer. The autoencoder also transfers effectively to T2I tasks, producing visually compelling results.
19

---

Published as a conference paper at ICLR 2026
Figure S12: Uncurated class-conditional generation results of RepTok with CFG scale of 3.5.
20


Reviews:
[{"review_id": "gHx8tXzKbV", "reviewer": "Reviewer_wivx", "summary": "This paper presents a method for tokenizing an image into a single continuous token using an SSL-pretrained model. Based on this single token, the authors design a generative decoder and a representation generator to perform reconstruction and generation.", "questions": "As the generation process involves two flow-matching steps, I am curious about the inference speed of the proposed model. Could the authors provide a comparison of inference time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper provides strong evidence that SSL models can provide strong generative information using only a single image token.\n2. The corresponding generative decoder and representation generator are well designed.\n3. The proposed method shows very high training efficiency.", "weaknesses": "1. Lack of representation comparison: As the work is heavily based on the image representations, it would be helpful to show the correspondence between generation quality and representation performance under different [CLS] settings (e.g., frozen or fine-tuned with varying $\\lambda$).\n2. The authors experimented with different SSL models for the proposed method. How does the performance change when combining multiple models, such as training a lightweight linear projection on top of the concatenated [CLS] tokens (or through other fusion strategies)?\n3. The proposed method heavily relies on image SSL models, which may limit its generalizability to video generation or multimodal generation tasks.\n4. The paper only conducts experiments at a resolution of 256x256. Using a single token to encode an image may be ineffective for higher-resolution settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q3PvamoiF1", "reviewer": "Reviewer_TyLM", "summary": "This paper proposes RepTok, an approach for training image generative models using pre-trained self-supervised learning (SSL) models. During the encoder-decoder training phase, the SSL model is fine-tuned efficiently via a learnable [CLS] token parameter, and a flow matching head is trained. In the image generation training phase, images are generated in the SSL model's latent space using flow matching based on an MLP model. The paper validates the feasibility of using SSL as an encoder and demonstrates impressive performance in both image reconstruction and generation.", "questions": "Refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The paper offers several intriguing insights in its model architecture design. These include the indirect fine-tuning of the pre-trained SSL model via a learnable [CLS] token, the use of cosine similarity as a latent space regularizer, and the employment of an MLP-Mixer as the backbone for the generative stage. I believe these interesting design choices could provide inspiration for subsequent researchers.\n2.  The proposed method is notably lightweight. On one hand, the model architecture does not require training an excessive number of parameters; on the other hand, building upon a pre-trained SSL model keeps the training cost for the generative model relatively low, suggesting potential for practical application.\n3.  The paper is generally well-written and clear, and the experiments are reasonably comprehensive.", "weaknesses": "1.  Although the structure is lightweight, the scaling potential of the method is concerning. Firstly, while the reconstruction results in Figure 3 appear superior to FlexTok, they do not seem to surpass those of the VAE. Furthermore, as shown in Table 3, RepTok's final performance is lower than that of REPA-CFG 1.5. Therefore, I would like the authors to provide an analysis of how this method could be further scaled, including a discussion of performance bottlenecks, and to present additional experimental results demonstrating the method's potential at larger scales.\n2.  The authors' proposed method (two-level flow matching) bears formal resemblance to Transition Matching [1]. Furthermore, the idea of replacing VAEs with self-supervised models has also been explored, including in works such as RAE [2] and SVG [3]. While direct comparison with concurrent work is not strictly necessary, I encourage the authors to supplement the discussion with a clarification of the distinctions and a more in-depth comparative analysis.\n3.  Since the authors' core claim also includes that \"SSL provides a better semantic space than VAE\" (not merely lightweight), additional experimental results are needed to substantiate this point, particularly regarding performance aspects. This could include evaluations on avoiding distortions and handling challenging, complex, or compositional prompts.\n4.  The citation format is informal; it should be (Ho et al. 2020), not Ho et al. (2020).\n\nref:\n\n[1] Transition Matching: Scalable and Flexible Generative Modeling\n\n[2] Diffusion Transformers with Representation Autoencoders\n\n[3] Latent Diffusion Model without Variational Autoencoder", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h63uQUYGbs", "reviewer": "Reviewer_GhVu", "summary": "This paper introduces RepTok, a generative modeling framework that leverages the [cls] token from a pre-trained self-supervised vision transformer as a single-token, continuous latent space useful for image generation tasks. \n\nThe paper fines only the [cls] token from a pretrained SSL network and regularizing it with a cosine similarity loss. The authors enable faithful image reconstruction and efficient generative modeling. The approach achieves competitive results on class-conditional ImageNet generation and zero-shot text-to-image synthesis on MS-COCO.\n\nThis paper, to my knowledge, is the first single-token encoder paper that achieves reasonably high performance on image genertion tasks", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(+) novelty: using a single [cls] token to represent the image in the continuous space to my knowledge is novel (and neat), what is better is that it can be minimally fine-tuned from a pretrained SSL network. The paper can encourage more work for compact visual tokenizer that improves both token redundancy and improve tokenization/reconstruction quality\n\n(+) Competitive performance: the performance is good for the first paper with a single-token compression. The experiements are also relatively thorough\n\n(+) Generalization: can use different SSL models, such as DINOv2, MAE, CLIP", "weaknesses": "(-) The method requires careful tuning of the cosine similarity loss parameter and careful choices of freeze/unfreeze tokens to balance between reconstruction fidelity and generative quality. This trade-off may be non-trivial in practice and may need tuning for different backbones\n\n(-) A single-token representation is neat, but it will be nice to see the tradeoff between more tokens vs single token.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9dXvZsFMeJ", "reviewer": "Reviewer_kicm", "summary": "This paper adapts pre-trained self-supervised representations into a single continuous latent token space and trains a lightweight latent generator with a generative decoder. This idea is potentially impactful for compute-efficient generation. However, at the reported compute and model sizes, the performance falls short of the current SOTA, which lacks a matched-budget comparison. Also, the Flops reported in Table 3 appears to exclude the generative decoder, making the efficiency claims for end-to-end generation unclear. The paper also lacks a scaling study of compute resource and paramters which is an important property.", "questions": "1. What are the total compute resources and paramters combining the Generative Decoder with the MLP Mixer? Please report FID, GFlops/Iter, total PFlops in Table 3 under matched evaluation settings and include the stage-one generative decoder in compute resources and parameters.\n2. Can RepTok exceed SOTA FID when scaled up in Table 3? Please include a scaling analysis of the MLP-Mixer sizes and longer training, reporting FID vs. total PFLOPs/params under matched evaluation settings.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It is a noval approach that uses a single continuous latent token adapted from self-supervised representations with a lightweight latent generator.\n2. RepTok shows flexibility as it extents to text-to-image synthesis.\n3. The paper is generally clear and well organized.", "weaknesses": "1. Although the compute and model sizes of latent generator is smaller, the gFID in Table 2 and FID in Table 3 are not competitve with SOTA. It remains unclear how RepTok performs under matched evaluation settings against baselines.\n2. Table 3 appears to report their compute resources and parameters only for the MLP Mixer, excluding the stage-one Generative Decoder. This undermines the fairness of the compute comparison and weakens the efficiency claim.\n2. The paper does not provide a scaling study for the MLP-Mixer generator (e.g., FID vs. parameters/train steps). Table 3 reports only a single 276M model, whereas the only scaling evidence shown concerns the frozen language backbones in Figure 7. Since parameter scaling is a key feature of DiT/SiT, reporting two training-step settings is insufficient to establish scaling behavior.\n3. There are many instances of incorrect citation formatting: using author-in-text citations (\\citet{...}) instead of the required parenthetical author-year style citations (\\citep{...}).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)