Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Authors: Philippe Hansen-Estruch
David Yan
Ching-Yao Chuang
Orr Zohar
Jialiang Wang
Tingbo Hou
Tao Xu
Sriram Vishwanath
Peter Vajda
Xinlei Chen
Venue: ICML 2025 poster (2025)
Keywords: Auto-Encoder
Visual Compression
Latent Diffusion Models
Vision Transformer
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Visual tokenization via auto-encoding empowers state-of-the-art image and video generative models by compressing pixels into a latent space. However, questions remain about how auto-encoder design impacts reconstruction and downstream generative performance. This work explores scaling in auto-encoders for reconstruction and generation by replacing the convolutional backbone with an enhanced Vision Transformer for Tokenization (ViTok). We find scaling the auto-encoder bottleneck correlates with reconstruction but exhibits a nuanced relationship with generation. Separately, encoder scaling yields no gains, while decoder scaling improves reconstruction with minimal impact on generation. As a result, we determine that scaling the current paradigm of auto-encoders is not effective for improving generation performance. Coupled with Diffusion Transformers, ViTok achieves competitive image reconstruction and generation performance on 256p and 512p ImageNet-1K. In videos, ViTok achieves SOTA reconstruction and generation performance on 16-frame 128p UCF-101.

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Philippe Hansen-Estruch 12 David Yan? Ching- Yao Chuang? Orr Zohar Jialiang Wang? Tingbo Hou?
Tao Xu² Sriram Vishwanath
Peter Vajda? 2 Xinlei Chen
Abstract
the tokenizers themselves, predominantly based convo- lutional neural networks (LeCun et al. (9998) (CNN), have rarely been the main focus of scaling efforts.
Visual tokenization through auto-encoding en- hances state-of-the-art image and video genera- tive models by compressing pixels into a latent space. However, questions persist regarding how the design of the auto-encoder affects both recon- struction and downstream generative performance. This paper investigates the impact of scaling auto- encoders for reconstruction and generation by sub- stituting the convolutional backbone with an en- hanced Vision Transformer for Tokenization (Vi- Tok). This paper's results show that scaling the auto-encoder bottleneck correlates with improved reconstruction, though its relationship with gen- erative performance is trast, scaling the encoder does not lead to gains, while scaling the decoder enhances reconstruc- tion with minimal effect on generation. These findings indicate that scaling the cxisting auto- encoder paradigm does not significantly improve generative performance. When paired with Dif- fusion Transformers, ViTok achieves competitive image reconstruction & generation performance on 256p and 512p ImageNet-IK. K. For videos, Vi- Tok achieves state-of-the-art both reconstruc- tion& & generation performance on 128p UCF-101.
In this paper, we explore whether visual tokenizers require the same scaling efforts as generators. To do this, we first examine two potential bottlenecks: model architecture and data. We begin by replacing convolutional backbones with Transformer-based auto-encoder (Vaswani et al. 2017). specifically adopting the Vision Transformer (ViT) (Doso-
vitskiy et al. [021]) architecture enhanced with Llama (Tou- vron et al. 2023). which has demonstrated effectiveness at scale (Gu & Dao 2023 Sun et al. 2024). Our resulting auto- encoder design, which we refer as Vision Transformer To- kenizer or ViTok, performs well with the generative pipeline in Diffusion Transformers (DiT) (Peebles & Xie, 2023).
complex. In con-
Second, we train our models on large-scale image datasets that significantly exceed ImageNet-IK. (Deng et al. 2209) and extend our approach to videos, ensuring that our tok- enizer scaling is not constrained by data limitations. Within this framework, we investigate three aspects:
Scaling the auto-encoding bottleneck. Bottleneck size correlates with reconstruction metrics. However, when the bottleneck becomes large, the generative per- formance decreases due to increased channel sizes.
Scaling the encoder. Scaling the encoder does not im- prove the generation outcomes and can even be detri- mental. In particular, more complex latents can be more difficult to decode and model, reducing overall perfor- mance.
1. Introduction
Modern methods for high-fidelity image and video gen-
eration (Brooks et al. 2024; Polyak et al. 2024 Genmo 2024 Esser et al. 2024) rely on two components: a visual tokenizer that encodes pixels into a lower-dimensional la- tent space and subsequently decodes, and generator that models this latent representation. While many studies have improved generators through scaling of Transformer-baat based architectures /Vawwani et al. 2017 Dosovitskiy et al. 2021).
Scaling the decoder. Scaling the decoder improves reconstruction quality, but its influence on downstream generative tasks remains mixed. We hypothesize that the decoder acts in part as a generator, filling in local textures based on limited information.
'T Austin GGenAL, Meta *Stanford University 'Fundamental Al Rescarch, Meta. Correspondence to: Philippe <philippchansenee edu..
Collectively, these results indicate that scaling the auto- encoder tokenizer alone is not an effective strategy to en- hance generative metrics within the current auto-encoding paradigm (Esser et al. (02T). We also observe that similar bottleneck trends apply to video tokenizers. However, Vi- Tok leverages the inherent redundancy in video data more
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Scaling the encoder doesn't 't necessarily improve reconstruction or generation performance. Small encoders are optimal
Patch/Tubelet Embed
Increasing the bottleneck size enhances reconstruction quality but degrades generation performance when too large
Llama Encoder
Latent Bottleneck
Scaling the decoder improves reconstruction, but has more minimal benefits in generation
The perceptual and GAN losses trades off image fidelity (SSIM/PSNR) for image quality (FID/IS), effectively transforming the decoder into generative model
Llama Decoder
Patch/Tubelet Decode
Videos follow same reconstruction trends as images, but achieve better reconstruction metrics given the same compression rate of pixels per channel
Figure 1. Our learnings from scaling ViTok. We present ViTok, an asymmetric auto-encoder combining Vision Transformers (VITs) and an enhanced Llama architecture for reconstruction and generation. Visual inpuls are embedded as patches/tubelets, processed by compact Llama Encoder, and bottlenecked into latent codes. A larger Llama Decoder reconstructs the input. We detail findings scaling the encoder, adjusting the bottleneck size, and expanding the decoder on the right.
effectively, achieving superior reconstruction metrics than for images at fixed compression rate of pixels per channel. We summarize our findings and our method in Figure [
encoder outputs parameters for multivariate Gaussian dis- tribution- mean 2m and variance 2, with channel size.:
2 ~~ N(m 20) Z = fo(X) #x
We compare our best-performing tokenizers against prior state-of-the-ar methods. ViTok achieves image reconstruc- tion and generation performance at 256p and 51 resolu- tions that matches or surpasses current ofute-of-thh-art tok- enizers the ImageNet-1K K (Deng et al. 2009) and COCO (Lin et al. 2014) datasets, all while utilizing 2-5x fewer FL.OPs. In video applications, ViTok surpasses current state-of-the-art methods, achieving state-of-the results in 16-frame 128p video reconstruction and class-conditions video generation on the UCF-101 (Soomro) 2012) dataset.
The sampled latent vector is then fed into a decoder Dø. with parameters v. which reconstructs the input im- age X = (z). The primary objective of the auto-encodie is to minimize the mean squared error between the recon- structed and original images, LREC (X, X). To regularize the latent distribution to a unit Gaussian prior which is necessary to recover the variational lower bound, KL di- vergence regularization term is added, which we refer to LXL. Recent advancements in VAËs used for down- stream generation tasks (Esser et al.) 2021 Rombach et al., 2022 incorporate additional objectives improve the vi- sual fidelity of the reconstructions. These include a per- ceptual loss based on VGG features (Johnson et al. 2016) LIPPS and an adversarial GAN objective, LGAN (Goodfel low al. 20144). The comprehensive loss function for the auto-encoder, LAB(X, X, Z). is formulated as:
2. Background
We review background on continuous visual tokenizers and then describe ViTok to enable our exploration.
2.1. Continuous Visual Tokenization
The Variational Auto-Encoder ( (VAE) (Kingma & Welling 2013) is a framework that takes a visual input X e xx (where T = 1 for images and T > 1 for videos) is processed by an encoder fe. parameterized by This encoder performs spatial-temporal downsampling by factor of g x p p. producing latent code. The
LAR X, X, = CREE (x, X) + (Z)
(I)
+ 1 Xx X) + XLAN X)
where 3, 17, and À are parameters that control the trade-off between visual fidelity and quality (Section 3.4).
2

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Table Model Sizes and FLOPs for ViTok. We describe ViTok variants by specitying the encoder and decoder sizes separately. For
example, ViTok S-B/4x16 refers to model with an encoder Small (S) and decoder Base (B), using tubelet size 4 and = 16.
Model
Hidden Dimension
Blocks
Heads
Parameters (M)
GFLOPs
Small (S)
768
43.3
11.6
Base (B)
768
12
12
85.8
23.1
Large (L)
1152
24
16
383.7
101.8
2.2. Scalable Auto-Encoding Framework
2017). Inception Score (IS) Sali- mans et al.) 2016), Structural Similarity Index Measure (SSIM) (Wang et al. 2004), and Peak Signal-to-Noise Ratio (PSNR). For video, we report rFID (frame-wise FID) and Fréchet Video Distance (rFVD) (Unterthiner et al.) 2019).
(FID) (Heusel et al.
We develop a visual tokenizer based on a Vision Transformer (ViT) architecture, replacing CNNs for better scalability. Building on ViViT (Arab et al. 202I). our framework handles both images and videos. A 3D convolution with kernel and stride size X p X tokenizes the input X into Xembed E BxLxx1, where L = T x H W A ViT encoder processes Xembed, and a linear projection produces a compact representation = fa (Xembed) E Bx Following the VAE formulation, we recover 2 E BxLxe and define the latent space dimensionality as
p
Generation experiments and metrics. To assess our tok- enizers in a large-scale generative setting, we train a class- conditional DīT-L (Peebles & Xie 2023) with 400M param- eters for 500,000 steps and batch size of 256, applying classifier-free guidance (CFG) (Ho & Salimans 2022) on
(2) which controls the compression ratio. The decoder upsam- ples from è to Cg channels via a linear projection, pro- cesses it with a VIT decoder, and uses a 3D transposed convolution to recover X. This defines our Vision Trans- former Tokenizer (ViTok). Figure Tillustrates the process, and Table provides ViTok size details for reference.
E == L c,
a DDIM sampler (Song et al. 2020) over 250 steps and a CFG scale of 1.5, We apply the same Llama upgrades to our DiT as for our tokenizers. We measure generation qual- ity using gFID and gIS (gInception Score) calculated over 50,000 samples. Since ViTok can directly output continuous tokens, we can feed the noised latents + directly into DiT without patchifying and predict the noise.
3. Bottlenecks, Scaling, and Trade-offs in Visual Tokenization
2.3. Experiment Setup and Training
To address instability in VAE frameworks, we use a two- stage training approach. Stage 1 trains with MSE, LPIPS, and KL losses (B3 = 1 x 10 auto-encoding. Stage 2 incorporates the GAN, freezes the encoder, and fine-tunes the decoder with À = 1.0.
In Section - we introduced ViTok and its training pro- cess. Here, we explore the impact of scaling three key fac- tors-bottleneck size, encoder size, and decoder size-on reconstruction and generation performance.
= 1.0, x = 0) for stable
Architecture, datasets, and training details. We use Vision Transformer architecture (ViT) for the encoder and decoder, incorporating SwiGLU and 3D Axial RoPE for spatiotemporal modeling. We train on large-scale datasets: Shutterstock (450M images) and ImageNet-1K for images, and Shutterstock videos (30M videos) for video. Evaluation is performed on ImageNet-IK, COCO-2017, UCF-101, and Kinetics-700. Stage 1 runs for 100k steps with batch sizes of 1024 (images) and 256 (videos). Stage 2 fine-tunes for another 100k steps. We use AdamW (8) 0.9, B2 = 0.95), 1x
3.1. E as the Main Bottleneck in Image Reconstruction
In prior discrete cases performance depends on the tokens (L) and the codebook size (Oord et al.] 2017 Mentzer et al. 2023). For ViTok, the analogous factor is E (Equation which proves be the critical determinant of reconstruction performance. As described in Equation 2 E represents the total number of floating points (also dependent on precision) and relates to the compression ratio, defined as TxHx x3 We train ViTok S-B on stage 1 (Section 22.3 on 256p image reconstruction, sweeping patch sizes p (32, 16, 8) and channel widths c {4, 8, 16, 32, 64). yielding E' values from 2 to 2 Patch size affects L HxWW and therefore FLOPs due to the quadratic nature of attention, while c controls the bottleneck between encoder and decoder. The results are summarized in Figure 3 with more details of the experiment provided in Appendix
E
weight decay of x 10-4 and a cosine decay schedule. For Stage 2, we use Style- GAN (Karras al. 2019) discriminator with learning rate of 2 x 10 and a 25k-step warmup. Training uses bfloat1 6 autocasting, with EMA (0.9999) introduced in Stage 2.
a peak learning rate of 256
p²
Reconstruction evaluation metrics. To gauge recon- struction quality, we use the Fréchet Inception Distance
Figure 3 shows strong correlation between E. and recon-
3

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Reconstruction with Floating Points
Ground Truth
16384
8192
4096
2048
1024
Figure 2. 256p image reconstruction visualization over floating points E. Example reconstruction for varying values on ViTok S-B/16, achieved by adjusting the channel size 64, 32, 16, 8, 4 for each image across the row. As E decreases, high-frequency details diminish, with small colors and fine details gradually lost. When E < 4096, textures merge, and significant detail loss becomes apparent.
2²
200
26
Dataset
c Values
Patch Sizes
R
24
250
11k 0.91
25
32
x32
x16
150.
200
26
25
2 150
6-100
2-2
21
100
2-4
Dataset
2a
50-
0.92 R
2-1
11k
0.94
50
R 0.75
2'
2
2
2
2
2
2
2
Floating Points, E
Floating Points,
Floating Points,
Floating Points,
Figure 3. 256p image reconstruction sweep over floating points E. We evaluate ViTok S-B using combinations of patch sizes p E 8, 16, 32 and channel widths c E 4, 8, 16, 32, 64 to investigate 256² how = pa e influences FID and IS in reconstruction tasks. This indicates that E is the primary bottleneck for reconstruction, irrespective of the code shape or FLOPs expended.
Figure 4. 256p image generation over E. We evaluate each tok- enizer from Figure 3 on DiT following Section 2.3 Our results show no strong linear correlation between log and generation performance, Instead, second-order trend reveals an optimal E for each patch size p. This highlights the necessity of optimizing both E and to balance reconstruction with generation.
struction metrics (rFID/rIS), indicating that E. is key pre-
E 4096; p = 4, 4096; and p = 32, 32, E 2048. In particular, gFID and gIS remain poor for 32, indicating that excessive harms performance.
dictor of reconstruction quality. Performance trends are con- sistent across datasets, with minor rFID variations due to
validation set sizes (50k for ImageNet-IK vs 5k for COCO).
Low E bottlenecks the generative model due to poor re- construction, while high (driven by large c) complicates convergence and degrades gFID and gIS. This aligns with recent findings on the trade-off between rFID and gFID in latent diffusion models (Yao & Wang 2025 The balance of E and is crucial: they must be low enough to aid gen- eration, but high enough to ensure quality reconstructions. Generation visualizations are provided in the Appendix B
For a fixed E, varying patch sizes (c = similar performance, suggesting that FLOPs do not improve results for given F. This establishes F. the primary bottleneck. Figure 2 visualizes reconstructions for different E values. As E decreases, high-frequency details are lost; for E < 4096, textures degrade significantly.
HxW) yields a
Exp²
Regardless of code shape or flops ex- pended in auto-encoding, the total number of float- ing points in the latent code (E) is the most predic- tive bottleneck for visual reconstruction.
Finding 1:
Scaling F. does not consistently im- prove generative performance. Instead, optimal re- sults are achieved by tunigg both E and c. low E limits the quality of reconstruction, while a high E and channel size hinder the generation perfor- mance.
Finding 2:
3.2. The Impact of E in Image Generation
In this section, explore how E impacts generative perfor- mance using the tokenizers from Figure 3 and the training protocol in Section 2.3 Results are shown in Figure 4
3.3. Scaling Trends in Auto- muto-Encodiig
Unlike reconstruction, the generative metrics (log(gFID) and glS) do not show a linear correlation with log(E) E)). In- stead, each patch size has an optimal F:: p = 16, c = 16,
We explore how scaling impacts auto-encoding in recon- struction and generation tasks using ViTok, fixing parame- ters to p = 16,c 16. We sweep sizes S-S, B-S, S-B, B-B,

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
200
1.8
R 0.00
R 0.04
6.8
R
0.24
165
R 0.23
6.6
160
6.4
1.6
6.15
6.2
1.5
192
6.0
150
190
188
145
2
2
2 Encoder Size (M)
Encoder Size (M)
Encoder Size (M)
Encoder Size (M)
Figure Encoder scaling on 256p image reconstruction. We evaluate reconstruction metrics of ViTok over model sizes S-S, B-S, S-B, B-B. B-L, L-L with fixed p = 16, = 16. There no correlation between encoder size and reconstruction performance, indicating that scaling the encoder is unhelpful.
Figure 7. Encoder scaling on 256p image generation. evalu- ate each tokenizer from Figure on DiT following Section 2.3 We plot encoder size over generation metric results. There a negative correlation between encoder size and final performance indicating that scaling the encoder harmful for generation.
200
6.8
165
R²
198
1.7
196
6.6
160
1.6
2
194
G-155
6.2
6.0
1.4
5.8
145
20
2'
2
25
2
27
2
2
2
Decoder (M)
Decoder (M)
Decoder Size
Decoder Size (M)
Figure 6. Decoder scaling on 256p image reconstruction. Using the results from Fi we plot various decoder sizes (S, L) over reconstruction performance. There is strong correlation between decoder size reconstruction performance, which in- dicates scaling the decoder improves reconstruction. Although, increasing the decoder size from Base to Large does not provide the same boost of performance doubling E to 8192 from 4096.
Figure 8. Decoder scaling on 256p image generation. Using the results from Figure we plot various decoder sizes B. L) over generation performance. We plot decoder size over generation metric results for CFG scales 1.5. Unlike reconstruction, is no correlation between decoder size and generation performance. This indicates that scaling decoder minimal benefits overall and most scaling efforts should focus on the generator.
S-L, B-L,, L-L,, defined in Table [ following the training protocol in Section 2.3 Results are in Figures and and
but raises training and inference costs.
Scaling the encoder provides recon- struction benefits and may harm generation, while scaling the decoder improves reconstruction but of- fers limited generation gains.
Finding 3:
Figure 5 shows no correlation between encoder size and reconstruction performance. In contrast, shows correlation between decoder size and reconstruction quality. However, E remains the dominant factor: Going from Base to Large reduces rFID from 1.6 to 1.3 for E = 4096, while doubling E to 8192 with Base decoder drops rFID to 0.8. Scaling the decoder can be beneficial, but scaling the encoder provides no significant advantage.
With the findings so far, we believe simply scaling the current auto-encoding (Esser al.) (022] based tokenizers does not automatically lead improved downstream generation performance. Therefore, for generation tasks, it is more cost-effective to concentrate scaling efforts on the generator itself, rather than the visual tokenizer.
Figures] and examine the impact of scaling the encoder and decoder on generation performance. F shows slight negative correlation between encoder size and gen- eration quality, suggesting that larger encoders either have no benefit or may harm performance. Similarly, Figure g indicates that scaling the decoder offers minimal gains for generation. Unlike reconstruction, scaling the encoder or decoder does not significantly improve generation quality
3.4. A Trade-Off in Decoding
We compare the manner in which different losses balance SSIM/PSNR against FID. SSIM/PSNR measure visual delity or how much of the original information is preserved, while FID focus on visual quality and how closely outputs
5

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
27.0
Models & Fit
2
R
0.81
26.5
2
(SSI
0.80
Models &
27
12 1.0%PP
26.0
L2
2
0.5%LP
2
L1 + 1.0 DYLP
25.5
12-1P-GAN
2
If 0.92
2
2 2 Floating Points, E
2
rFID
IFID
Floating Points, E
Figure 9. Metric trade-offs in 256p image reconstruction. We train ViTok S-B/16 varying the LPIPS (LP in figure) weight À E [0.0, 0.5, 1.0) and using either LI or L.2 MSE reconstruction loss (Equation[] D. Additionally, we finetune ViTok S-B/16 with GAN and include the result as L.2+LP+GAN. The results indicate that enhancing rFID) scores through increased perceptual and visual losses requires a trade-off with rSSIM//SNRR, resulting in loss of information from the original image.
over E. We train Vi- Figure 10. 256p video reconstruction results over
Tok S-B on 16x256x 256 videos at 8 fps, varying p E {8, 16, 32) and q E {1, 2, 4, 8} with 16. Reconstruction performance is evaluated using rFID and rFVD on the Kinctics-700, UCF101, and Shuterstock datasets. The results exhibit a similar trend to im- age reconstruction in Figure demonstrating strong correlation between E and reconstruction performance. Expectantly, videos are more compressible than direct scaling from images would suggest; instead of requiring 16x E, achieving comparable rFID to 256p image reconstruction only necessitates 4 88
match the real dataset. This comparison shows how differ- ent losses choices can change the role of the decoder from strictly reconstructing to more actively generating content.
p 8) for 16x256x256 video yields 4096 tokens. Based on Section 3.3 we use small ViTok S-B variant to reduce computational costs as scaling the encoder does not aid
We conducted these experiments on ViTok by fixing p 16, 16, and E' 4096. We then trained with stage and varied the LPIPS loss weight À E (0.0, 0.5, 1.0) combined with the choice of L1 or L2 reconstruction loss (Equation I) We also include our Stage 2 results following Section 2.3 to see the final effect of the generative adversarial objective.
performance. We sweep patch sizes p E {8, 16, 32} and temporal strides E f1, 4,8). Figure 10 shows that log( F)) correlates with reconstruction metrics FFVD/FFID.
Videos require E 2 16384 to achieve an rFID of 2.0, whereas images need E 4096. The smaller-than taallee-tha--eepeetteee expected impact scaling image-derived E' across video frames high- lights ViTok's 's ability to exploit video compressibility.
Figure FigureDssows a clear trade-off among these losses. With- out perceptual loss, we get worse rFID scores but better ISSIM/PSNR, indicating that a strict MSE-based approach preserves the most original information. Increasing À grad- ually lowers SSIM/PSNR while improving FID/IS. Finally, fine-tuning the decoder with GAN pushes these generative metrics further, achieving an rFID of 0.50 at the cost of a lower SSIM/PSNR. These results demonstrate that at a fixed F, aiming for higher visual quality requires sacrificing some traditional compression fidelity.
Videos share the same reconstruction bottleneck characteristies as images with respect to E. However, auto-encoding exploits video com- pressibility, allowing E to scale more efficiently relative to total pixels than in images.
Finding 5:
4. Experimental Comparison
Finding 4: There is a trade-off between rFID (vi- sual quality) and iSSIM/PSSN (visual fidelity), influenced by the weights of Equation | Conse- quently, the decoder can be trained as a generation model, which extends the main generator.
In this section, we compare our auto-encoders to prior work on image reconstruction at 256p and 512p. as well as video reconstruction with 16 frames at 128p. We utilize the S-B/16 and S-L/16 variants for images and the S-B/4x8 variant for videos. Training these tokenizers follows the Stage 1 and Stage 2 protocol outlined in Section 2.3
3.5. Video Results
4.1. Image Reconstruction and Generation
We extend ViTok to video tasks to study the impact of E video reconstruction and explore redundancy in video data. Using 16-frame videos at 8 fps and 256p resolution, we maintain consistency with image experiments. Furthermore, tokenizing videos can result in long sequences; for example, a tubelet size of 4x8 (temporal stride q = 4, spatial stride
We evaluate our models on image reconstruction and class- conditional generation using ImageNet- IK and COCO- 2017 at 256p and 512p resolutions. For reconstruction, we compare against otate-of-the-art continuous tokenizers, including SD-VAE 2.x (Rombach et al. 2022), SDXL-
6

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Table 256p image reconstruction comparison. We assess the reconstruction performance of ViTok on ImageNet- IK and COCO-2017 validation sets, benchmarking them against CNN-based tokenizers with an equivalent compression ratio (48 pixels per channel), Our ViTok S S-B/16 tokenizer achieves state-of- the-ar art (SOTA) rFID scores on both ImageNet- IK and COCO datasets, outperforming other CNN-based continuous tokenizers while utilizing significantly fewer FLOPs. Furthermore, ViTok maintains competitive performance in SSIM and PSNR metrics compared to prior methods. When scaling decoder size Large, ViTok improves all reconstruction numbers.
ImageNet rPSNR1
COCO iPSNR
Name
Params (M)
GFL.OPs
rFID
ISSIM1
rFID
ISSIM
SD-VAE SDXL-VAR
59.3
162.2
0.78
25.08
0.705
4.63
24.82
0.720
0.68
26.04
0.834
4.07
25.76
0.845
OAI Cosmos- CI
0.81
24.43
0.786
4.59
24.19
0.800
2.02
31.74
0.700
5.6
31.74
0.703
ViTok S-B/16
129.0
34.8
0.50
24.36
0.747
3,94
24.45
0.759
ViTok S-L/16
426.8
113.4
0.46
24.74
0.758
3.87
24.82
0,771
Table 512p image reconstruction comparison. We assess the reconstruction performance of top-performing tokenizers on ImageNet-Ik IK and COCO-201 validation sets, benchmarking them against a CNN based tokenizer with an equivalent compression ratio (48 pixels per channel). Our ViTok S-B/16 tokenizer maintains state-of the art (SOTA) results across metries.
ImageNet rPSNR
COCO
Name
Params(M)
GFLOPs
rFID
1SSIM
FFID
rPSNR
SSIM
SD- VAE
59.3
653.8
0.19
ViTok S-B/16
129.0
160.8
0.18
26.72
0.803
2.00
14
0.790
VAE (Podell et al. 2023), and COSMOS (NVIDIA 2024). Discrete tokenizers are excluded due to incompatibility with direct comparisons.
2023),
Consistency Decoder (OpenAl
state-of-the-ar (SOTA) rFVD results compared to other CNN-based continuous tokenizers with the same total com- pression ratio. Additionally, our approach significantly re- duces FLOPs compared to Transformer- based prior method LARP, underscoring the efficiency and versatility of ViTok.
As shown in Table 2 our ViTok S-B/16 achieves SOTA rFID scores on I and COCO, with competitive iSSIM and rPSNR metries. Scaling the decoder to size L fur- ther improves performance. Notably, ViTok reduces FLOPs compared to prior CNN-based methods. For 512p recon- struction (T) ViTok achieves SOTA performance with significantly fewer FLOPs, outperforming existing methods in both metries and computational efficiency.
We further evaluate our models on class-conditional video generation using the UCF-101 dataset. We train DiT-L model for 500K steps on the UCF-101 training set, comput- ing gFID and gFVD metrics. The results are summarized in Table ViTok achieves SOTA gFVD scores at 1024 tokens. Example video generations using our 1024-token configuration are illustrated Figure 21
For class-conditiona, image generation, train DiT-XL (675M parameters) for 4M steps paired with ViTok S-B/16, using 256 tokens for 256p and 1024 tokens for 512p genera- tion. Results in Table T ViTok performs competitively against DDEE At 512p, ViTok matches other methods, demonstrating its effectiveness at higher resolutions. Gener- ated images are visualized in Figures 16 and 17
5. Related Work
Image tokenization. High-resolution images have been compressed using deep auto-encoders (Hinton et al. 2012 Vincent al. 2008). process that involves encoding an im- age into lower-dimensional latent representation, which then decoded to reconstruct the original image. Variational auto-encoders (VAEs) (Kingma & Welling 2013) extend this concept by incorporating probabilistic meaning to the encoding. VQVAES (Oord et al. (017) introduce a vector quantization (VQ) step in the bottleneck of the auto-encoder, which discretizes the latent space. Further enhancing the vi- sual fidelity of reconstructions, VQGAN (Esser et al. 202T) integrates adversarial training into the objective of VQVAE. Finally, FSQ (Mentzer al. 2023) simplifies the training
4.2. Video Reconstruction and Generation
For our video comparison, our reconstruction metrics are computed on the UCF-101 training set and compared against state-of-the-art methods including TATS (Ge et al. 2022). LARP (Wang et al. 2024a). and MAGViTv1/v2 (Yu al. 2023b The results are presented in Table 4 Our tokeniz- ers demonstrate very competitive performance relative to prior work. Specifically, S-B/4x8 (1024 tokens) achieves

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Table 128p Video Reconstruction. evaluate ViTok S-B/4x8 on video reconstruction for 16x 128x video on UCF-101 101 train set and compare to prior that utilizes similar compression ratios. ViTok S S-B/4x8 achieves SOTA performance in rFVD. rFVD, IPSNR, and rSSIM metrics compared prior work. ViTok also reduces the total FLOPs required from prior transformer based methods.
Method iSSIMt
(M))
GFLOPs
Compression
# Tokens
rFID
rFVD]
iPSNR*
TATS
32
Unk
2048
162
MAGVIT
158
Unk
1280
25
22.0
0.701
MAGVÎTv2
158
Unk
1280
16
LARP-L-Long
174
505.3
1024
20
ViTok S-B/4x8
129
160.8
1024
2.13
8
30.11
0.923
fine-tunes only the decoder for downstream generative tasks. Causally Regularized Tokenization (CRT) (Ramanujan et al.)
Table Class conditional image generation results. We evalu- atc our tokenizers class-conditional generation at resolutions of 256p and 512p on ImageNet-IK compared to prior methods. ViTok performance competitive with prior continuous diffusion generation methods for both 256p and 512p generation.
2024) and ElasticTokk (Yan al. 2024 both develop causal I-D tokenizers tailored for autoregressive image models. Finally, JetFormer (Tschannen et al. 2025) unifies tokeniza- tion and generation by training an end and-o--end autoregressive model that directly produces high-resolution images, remov- ing the need for separate stage-1 and stage-2 components.
256p Generation gFID|
512p Generation gFID
Method
gIS f
glS
LDM
3.60
247.70
DiT-XL/2
2.27
278.24
3.04
240.82
Video tokenization. VideoGPT (Yan et al., 2021) pro- poses using 3D Convolutions with VQVAE. TATS (Ge) et al. 2022) utilizes replicate padding to reduce temporal corruption issues with variable-length videos. Phenaki (Vil legas et al. 2022) utilizes the Video Vision Trans- former (Arnab et al. (VVII( architecture with factor- ized attention using full spatial and casual temporal attention. MAGViTvI (Yu et al. 2023a b) utilizes 3D convolution with VQGAN to learn a video tokenizer coupled with a masked generative portion. Finally, LARP (Wang al. 2024a) is concurrent work that tokenizes videos with ViT into discrete codes similar to TiTok's architecture (Yu et al., 2024,, our work differs as we use continuous codes and don't concatenaee latent tokens to the encoder.
VQGAN
15.78
TiTok-B
2.48
2.49
ViTok S-B/16
2.45
284.39
3.41
251.46
Table 6. 1 128p class conditional video generation. We evaluate ViTok S-B 4x8 on class-conditional generation 16x x28x 128 on the UCF- ICF-101 dataset compared to prior methods. ViTok S S-B/4x8 achieves SOTA video generation performance when used with a comparable compression ratio with prior methods.
Tokenizer
Params
gFID
gFVDJ
TATS
321M
332
MAGVIT
675M
76
MAGViTv2
177M
58
W.A.L.T
177M
46
LARP-L-Long 177M
57
High resolution generation. High resolution image generation has been done prior from sampling VAEs, GANs (Goodfellow et al. els (Sohl-Dickstein
ViTok S-B/4x8
400M
6.67
27
2014), and Diffusion Mod-
al.
2015
Song & Ermmn 2019
process of VQVAE by avoiding additional auxiliary losses.
Song et al. 2020 Ho et al.
2020 While some work per-
While ConvNets have traditionally been the backbone for auto-encoders, recent explorations have incorporated
form image synthesis in pixel space (Dhariwal & Nichol 2021), many works have found it more computationally
Vision Transformers (Vaswani et al., [017] Kolesnikov
effective to perform generation a latent space from an auto-encoder (Rombach et al.) 2022). Typically, the U-Net architecture (Ronneberger et al.) 2015) has been used for diffusion modeling, although recently transformers have been gaining favor in image generation. MaskGIT (Chang) et al. 2022) combines masking tokens with a schedule to generate images and Diffusion Transformers (Peebles & Xie 2023) proposes to replace the U-Net architecture with a ViT. With Some methods use regreseereesive modeling to
let al. 2020 (ViT) to auto-encoding. UViM (olesnikov
2022) adopts an asymmetric Vision Transformer (ViT) cncoder-decoder to unify wide range of vision tasks, including panoptic segmentation and depth prediction. ViT-VQGAN (Yu et al. 2022) replaces the convolutional auto-encoder of VQGAN with ViT blocks, yielding better re- construction quality and more favorable scaling. TYTok YYu ct al. 2024) introduces a compact 1-D ViT tokenizer that distills latents from a VQGAN, freezes the encoder, and
generate images (Ramesh et al. 2021 Yu et al.
2023a
8

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
2021) encodes im- ages with VQVAE and then uses next token prediction to generate the images. While most auto-regressive image gen- erators rely on discrete image spaces, MAR (Li et al., (024) proposed a synergized next token predictor that allows for visual modeling in continuous latent spaces.
al. 2024).
DALL-E (Ramesh et al.
ulators. 2024. URL https://openai. com/reseearhh/videe cenerntion--odell-s--wweeeo
2024.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In CVPR, 2022.
Deng, J., Dong, W., Socher, R., Li, L..J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
6. Conclusion
In this paper, explored scaling in auto-encoders, intro- ducing ViTok, a ViT-style auto-encoder. We investigated scaling bottleneck sizes, encoder sizes, and decoder sizes, finding a strong correlation between the total number of floating points (E) and visual quality metrics. Our results show that scaling the auto-encoder alone does not signifi- cantly improve generative performance. From our sweep, we develop SOTA visual tokenizers. ViTok achieves compet- itive performance with vate-of-the-an tokenizers, matching rFID and rFVD metrics while using fewer FLOPs.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs on image synthesis. In NIPS, 2021.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Duggal, S., Isola, P., Torralba, A., and Freeman, W. T. Adaptive length image tokenization via recurrent alloca- tion, 2024. URL
7. Limitations
02393
While ViTok demonstrates strong performance, our find- post-partlularrly the negative results- constrained by the experimental setup and assumptions. These limitations are shared to prevent redundant efforts and to provide valu- able insights for future work. Despite these constraints, the lessons learned offer contributions to understanding the core challenges in image and video generative methods.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Esser, P., Kulal, S., Blattmann, A., Entezari, R. Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
Ge, S., Hayes, T., Yang. H., Yin, X., Pang, G., Jacobs, D., Huang, J., and Parikh, D. Long video generation with time-agnostic vggan and time- transformer, 2022.
Impact Statement
This paper presents work whose goal is to advance the field of generative modeling and visual compression. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here as our benchmarks are purely academic in nature.
Genmo. Mochi 1. models 2024.
Goodfellow, I. Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
References
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M.,, and Schmid, C. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6836-6846, 2021.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arX15:2312.00752, 2023.
Ba, L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv:1607.064500 2016.
Hansen-Estruch P., Vishwanath, S., Zhang, A., and Tomar, M. Unified auto-encoding with masked diffusion, 2024. URL
wwwpa///.ogg/b////aaeee.oessee/e abs/2 2406. 17688.
Beyer, L., Zhai, x., and Kolesnikov, A. Big vision.
ht https://githun http://itthuu.oooomaoo gwwg//oooleeeeeoooooo research/ big visionson 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CVPR, 2016.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luh- man, E., et al. Video generation models as world sim-
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In CVPR, 2022.

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by two time-scale update rule converge to local Nash equilibrium. In NIPS, 2017.
NVIDIA. Cosmos-tokenizer, https: /tte:://ggttuuuuuu
NVID TA/ Cotia///oooooosooeeee izer 2024. Accessed: 2025-01-05. A tokenizer designed for efficient processing of large-scale datasets.
Hinton, G. E., Srívastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. Improving neural net- works by preventing co-adaptation of feature detectors. 2012.
Oord, A. d., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In NIPS, 2017.
OpenAl. Consistency decoder. /tttts://githhie com/ httpaaa//cosssssteeyecccoooe 2023. Accessed: 2023-10-17.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Py PyTorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
Ho. J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NIPS, 2020.
Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Com- puter Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part II 14. pp. 694-711. Springer, 2016.
Peebles, W. and Xie, S. Scalable diffusion models with Transformers. In 2023.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint urXiv:2307.01952, 2023.
Karras, T., Laine, S., and Aila, A style-based generator architecture for generative adversarial networks, 2019.
URL http:ttss//rxxi..or///////// 04444 04948
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., Yan, D., Choudhary, D., Wang, D., Sethi, G., Pang. G., Ma, H., Misra, I., Hou, J., Wang, J., Jagadeesh, K., Li, K., Zhang, L., Singh, M., Williamson, M., Le, M., Yu, M., Singh, M. K., Zhang, P., Vajda, P., Duval, Q. Girdhar, R., Sumbaly, R., Rambhatla, S. S., Tsai, S., Azadi, S., Datta, S., Chen, S., Bell, S., Ramaswamy, S., Sheynin, S., Bhattacharya, S., Motwani, S., Xu, T., Li, T., Hou, T., Hsu, W.-N., Yin, X., Dai, X., Taigman, Y., Luo, Y., Liu, Y.-C., Wu, Y.-C., Zhao, Y., Kirstain, Y., He, Z., He, Z., Pumarola, A., Thabet, A., Sanakoyeu, A., Mallya, A.. Guo, B., Araya, B., Kerr, B., Wood, C., Liu, C., Peng, C., Vengertsev, D., Schonfeld, F.,, Blanchard, E., Juefei-Xu, F., Nord, F,, Liang. J., Hoffman, J., Kohler, J., Fire, K., Sivakumar, K., Chen, L., Yu, L., Gao, L., Georgopoulos, M., Moritz, R.. Sampson, K., Li. S., Parmeggiani, S.. Fine, S., Fowler, T., Petrovic, V., and Du, Y. Movie gen: A cast of media foundation models, 2024. URL https: wtw.iv..rg//bb//2400000
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In ICLR, 2013.
Kolesnikov, A., Beyer, L.,, Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big Transfer (BiT): General visual representation learning. In ECCV, 2020.
Kolesnikov, A., Pinto, A. S., Beyer, L., Zhai, X., Harm- sen, J., and Houlsby, N. Uvim: A unified modeling approach for vision with learned guiding codes, 2022. URL http::ttt///. wwwaiiivvcommm org. htcp//bs//2200 10337
Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., and Farhadi, A. Matryoshka represen- tation learning, 2024. URL htttps://arxiv..oo abs//2055.31
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al. Gradient-basee based learning applied to document recognition. Proceedings of the IEEE, 86(1 11:2778-2282 1:228-23224, 1998.
Ramanujan, V., Tirumala, K., Aghajanyan, A., Zettlemoyer, L., and Farhadi, A. When worse is better: Navigating the -nnrrsssioongeeaattion tradeoff in visual tokenization. arXiv preprint arXiv:2212..16322 2024.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406.1 1838, 2024.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. ICML, 2021.
Lin, M., Chen, Q., and Yan, S. Network in network. In ICLR, 2014.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arv9922309 15505, 2023.
Rippel, O., Gelbart, M. A., and Adams, R. P. Learning ordered representations with nested dropout, 2014. URL ht.ps::/arxiv.or//aaa 40150915
10

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, N., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In NeurIPS, 2017.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In MICCAI, 2015.
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. NeurlPS, 2016.
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In ICML, 2008
Shazeer, Glu variants improve transformer. arXiv preprint wrXiv:2002.0520, 2020.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. ICML,, 2015.
Wang. H., Suri, S., Ren, Y., Chen, H., and Shrivastava, A. Larp: Tokenizing videos with a learned autoregres- sive generative prior. arXiv preprint arXiv:2410211264. 2024a.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. NIPS, 2019.
Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., and Qiao, Y. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14549-14560, 2023.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S.. and Poole, B. Score-based genera- tive modeling through stochastic differential equations. arXiv:2011. 13456, 2020.
Wang, X., Zhou, X., Fathi, A., Darrell, T., and Schmid, C. Visual lexicon: Rich image features in language space, 2024b. URL https://arxiv..r /bbs/ 2412.06774
Soomro, K. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 2024.
Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13 (4):600-612, 2004. doi: 10.1 1..1109/TIP.20038889999
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:4406.06525, 2024.
Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A. Videogpt: Video generation using vq-vac and transformers. arXiv preprint arXiv:2104.10157 2021.
Tomar, M., Mlansen-Estruee P., Bachman, P., Lamb, A., Langford, J., Taylor, M. E., and Levine, S. Video oc- cupancy models, 2024. URL RRLLhtts:// http::://xxvv..rrr/
Yan, W., Zaharia, M., Mnih, V., Abbeel, P., Faust, A., and Liu, H. Elastictok: Adaptive tokenization for image and video. arXiv preprint 24 2024.
abs/2407.0955
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv: 13971, 2023.
Yao, J. and Wang. X. Reconstruction generation: Taming optimization dilemma in latent diffusion models, 2025, URL /ttp::////xxvvvvv www.:iiw.orrr.oooooooooo aabs///b////a 20t1.. .14233111333...
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang. Z., Va- sudevan, V., Ku, A., Yang, Y., Ayan, K., et al. Scaling autoregressive models for content-rich text-to-image gen- eration. arXiv preprint 2206:::2266.770000 2(3):5, 2022.
Tschannen, M., Pinto, A. S., and Kolesnikov, A. Jetformer: An autoregressive generative model of raw images and text, 2025.
19722
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023a.
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges, 2019. URL https: /tpps:///rxvv

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng. Y., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023b.
Yu, Q., Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for reconstruc- tion and generation. arXiv preprint 40X.5.2406.07550, 2024.
Zohar, O., Wang, X., Dubois, Y., Mehta, N., Xiao, T., Hansen-Estruch, P., Yu, L., Wang, X., Juefei-Xu, F. Zhang, N., et al. Apollo: An exploration of video un- derstanding in large multimodal models, arXiv preprint arXiv:2412.10360, 2024.
12

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
In the appendix section we include more details on experiments, architecture details, and visualizations.
We provide additional details on the implementaiton of ViTok. Our implementation is based on the VideoMAAEv (Wang) et al. 2023) codebase and inspired by the Big Vision codebase (Beyer et 2022). Utilizing PyTorch (Paszke et al., 2019). we employ Distributed Data Parallel (DDP) for efficient multi-GPU training, along with activation checkpointing, b precision, and Torch Compile optimizations. For image models, we train using 8 NVIDIA H100 GPUs, where ViToo S-B/16 requires approximately 6-12 hours for stage and 3-6 hours for stage 2 on 256p and 5 512p resolutions. comparison, DiT image models take around 72-96 hours train for 4 million steps on the same resolutions. For video models. ViTok S-B/4x8 is trained on 16 NVIDIA H100 GPUs, taking about 24 hours for stage and 12 hours for stage on 256p, 16-frame videos, and 12 hours for 128p, 16-frame videos. DiT video models require roughly 48-96 96 hours train for 500k steps with batch size of 256. Our transformer architecture is based the Vision Transformer (ViT) (Dosovitskiy et al. 2021 and modified to incorporate elements from the Llama architecture, including SwiGLU (Shazeer 2020) activation functions and 3D axial Rotary Position Embeddings (RoPE) (Su et al. 2024). The architecture consists of Transformer blocks (Vaswani et al. 2017) with multi-head self-attention and MLP layers, enhanced by residual connections (He et al. 2016) and layer normalization (Ba et al., 2016), closely following the Masked Autoencoder (MAF) design (He et al. 2022). Additionally, we integrate video processing code from Apollo (Zohar et al. 2024). Unified Masked Diffusion Tlansen-Estruce et al.
).24), and Video Occupancy Models (Tomar al. 2024), enabling ViTok to effectively handle and exploit redundancy in video data, thereby improving both reconstruction metrics and compression efficiency. Overall, ViTok leverages advanced training techniques and architectural innovations to achieve state-of-the the---rrttt art performance in image and video reconstruction and generation tasks.
A. Extra Experiments
A.1. Detailed 256p Image Results
2
Patch Size
Channels
1.0
40
x32
64
200
0.9
x16
32
29
8
(55IM
0.8
IPSNR
4
30
2-2
0.7
100
2-4
0.6
2-6
50
0.5
20
2
2 213
2
2 Floating Points, E
2
2 2
Floating Points, E
Floating Points,
Floating Points, E
Figure 11. 256p Detailed Image Reconstruction Results with Fixed Architecture Size. We provide more details for the sweep in Fo the just the ImageNet-1K validation set. For 1024 < E S. 16381, where intersections of E. exist across patch sizes, we see very little variation in performance for fixed E. This indicates that is the main bottleneck for visual auto-encoding and is not influence by increasing FLOPs.
We provide further detail of the ImageNet- 1K validation reconstruction results from Figure 3 in Figure [I] Here we show different patch sizes and channels over F. This shows that regardless of patch size and FLOPs usage, E is highly correlated with the reconstruction perforance
A.2. E trends 512p reconstruction.
To examine how resolution size affects F,, we scale up the resolution from 256p to 512p. We test ViTok S-B/16 over p E 8, 16, 32. The results of the sweep are shown in Figure 12 The results follow trend similar to that in Figure 3 with E exhibiting consistent correlation relationships. While FID and IS are challenging to compare across resolution r achieving comparable rSSIM and rPSNR performance at 512p requires 4 x E' from 256p. This suggests that maintaining performance across resolutions requires preserving the same compression ratio, HxWxx
E
'The Inception V3 network used for FID and IS calculations resizes images 299p hefore feature computation, leading potential information loss during downsampling.
13

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
0.95
Dataset
Dataset
250
i1k R 0.99
240
0.85
30
FFDD
2 230
0.80
2-2
220
0.75
Dataset
x32
Dataset
2-4
caco R
0.99
x16
0.70
coco R 1.00
24
11k = 0.99
x8
i1k R 1.00
2
2
2 Floating Points,
2
2
2 Floating Points, E
2
2
Floating Points, E
Floating Points,
Figure 12. 512p Image reconstruction over E. We evaluate ViTok S-B trained with stage (Section 2.3 across all combinations of patch sizes E 8, 16, 32 and a fixed channel width = 16, analyzing how the total floating-poing operations, calculated as E = 512²
influence reconstruction metrics such FID, IS, SSIM, and PSNR. E shows trends similar 256p results (Figure 3 However, achieving comparable :PSNR/SSSMM to 256p requires x E' for 512p reconstruction, which indicates that compression ratio of pixels to channels should be fixed to maintain performance.
A.3. GAN Fine-tuning Ablation
230
Discrm LR 1
225
Discrim LR 2e-5
0.79
26.0
Weight
n2 2*
215
Pull Finetune
SSM
0.78
PSNR
210
0.77
200
0.76
24.3
195
80000
190
60000
0.75
Step
Step
Step
Step
Figure 13. Finetuning the Decoder with GAN. We study the effects of finetuning the decoder in ViTok S-B/16 on 256p images. We compare: (1) GAN finetuning, (2) different discriminator learning rates, (3) an increased GAN loss weight (0.1), 1), and (4) a finetuning of all model parameters (including the encoder). The best results occur with discriminator learning rate of 2 10 while higher rates cause instabilities. We also observe a clear shift toward more generative behavior: as the decoder gains better IS/FID, sacrifices some SSIM/PSRR, reflecting its transition into stronger generative component.
In Figure 13
study how various loss settings affect finetuning of the GAN decoder. Our goal is highlight the trade-off and the transition of the decoder to more generative behavior. We use ViTok S-B/16 on 256p images, following the protocol in Section 2.3 for stage 2 fine-tuning from a model trained on stage 1.
We compare:
Finetuning the decoder with the same Stage loss (no GAN).
Finetuning with discriminator learning rates (f1 10-5,2 10 10-5,4x 10 53) and GAN weight of 0.05,
Finetuning the full encoder/decoder with the GAN.
Using a higher GAN weight of 0.1 with discriminator learning rate of 1 x 10-5
the best setting is a GAN weight of 0,05 and a discriminator learning rate of 2 10-5, A higher discriminator learning rate causes training instabilities, while lower rate degrades performance. Full fine-tuning produces results, but slightly underperforns when compared to fine-tuning only the decoder. Fine-tuning without a GAN yields no improvement, confirming that GAN training the key factor behind the better results.
From Figure 13
14

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Finally, we inherent trade-off: improving FID tends worsen SSIM/PSNR, indicating that the decoder focuses on visual fidelity, it shifts more toward generative outputs. This demonstrates the evolving role of the decoder generative model to improve visual performance.
A.4. Latent ViTok and Masked ViTok
In this section, we describe two variants of ViTok that provide different potential directions for tokenization. First describe and evaluate our latent variation that does ID tokenization and can form more arbitrary code shapes, then discuss and evaluate our masking variant that allows for variable, adaptive tokenization.
Latent ViTok Variation. Another variant of ViTok involves utilizing latent codes following Titok (Yu et al. 2024 Initially, after applying tubelet embedding, concatenate set of ID sincos initialized latent tokens with dimensions Matent x Cj to the tubelet token sequence Xembed This combined sequence is then processed through the encoder and bottleneck using a linear layer. Subsequently, the tubelet tokens are discarded, and the latent tokens generated by the encoder form Z = Intemt X 2c, from which we sample 2 ~ Z. This gives us a 1D code with easy shape manipulation since L and c are arbitrarily decided and do not depend on p. In the decoder, 2 is upsampled to Cg. and we concatenate a flattened masked token sequence of length L x Cg with the upsampled latent code lamnn x Cg. The decoder then predicts X in the same manner as the simple ViTok variation using masked tokens. This approach allows for a more adaptive compression size and shape using self-attention. In addition, it accommodates arbitrary code shapes of different lengths than L, provided there redundancy in the code. A trade-off compared to the simple ViTok is the increased total sequence length and computational cost (FLOPs) during encoding and decoding. We refer this variant as Latent ViTok.
2
Dataset Group
225
Simple VITok
200
30
2a
Latent Vitok
0.85
175
28
22
0.80
#SWR
e
150
21.
125
26
100
0.70
24
2 75
0.65
2-1
2 Floating Paints, E
2
2
24
2 Floating Points, E
2
2
Floating Points,
Floating Paints,
Figure 14. 256p Simple vs Latent ViTok Results. We implement a latent variant of ViTok S-B/16, with 16 and T. (64, 128, 256, 512, 1024) latent tokens appended the original patch embedding, then processed using full self-attention, and subse- quently bottlenecked to = 16. Although this latent variant slightly underperforms the simpler version in rFID/1S, it remains comparable overall and follows the same rules as Consequently, provides an alternative Simple ViTok with greater control over the latent space.
We train latent ViTok on stage 1 (Section 2.3) where we fix = 16 and sweep the number of latent tokens E {64, 128, 256, 512, 1024) to adjust E. The results are shown in Figure 14 Our simple variant outperforms the latent version for most values of E, although the latent version achieves slightly better rSSIM/PPSNR for certain choices of E. This indicates that the latent approach a promising alternative to simple ViTok for more control over the latent space; however, it comes with an increased computational cost due to the longer sequence of concatenated tokens. We leave this implementation out of ViTok due to added complexity.
Token Compression via Random Masking. The simplest bottlenecking process ViTok involves manipulating e, which does not compress the number of tokens; the token count remains equivalent the number tokens post-patchin (L) or equivalent to the number of latent tokens (Imtem). However, manipulating p does not provide a fine grain control over the token count.
To form another bottleneck, we can instead manipulate the main sequence of patch tokens by masking a random power of two number of tokens, starting with tokens at the end of the sequence and masking towards the beginning. This is similar to the method developed in Elastic Tok (Yan et al., 2024). Matryoshka representation learning (Kusupati et al. 2024) encodes coarse-to fone information in a single vector, letting any prefix serve as a valid embedding and yielding up to 14x
15

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
compression with loss in ImageNet accuracy. Learned ordered representations with nested dropout (Rippel al. 2014 train networks rank latent dimensions by importance, so early codes cover low-budget tasks while later codes refine detail. Adaptive length tokenization via recurrent allocation (Duggal et al. 2024) iteratively refines image patches, producing variable 32 256 1-D tokens that scale with entropy and task difficulty. Finally, Visual Lexicon (Wang et al. (024b) maps images into the language token space, retaining fine visual detail just a few tokens that can be mixed seamlessly with ordinary text prompts. For example, if we randomly select 256 as the masking amount for a sequence of 1024 tokens, then the last 256 tokens will masked out and replaced with learned masked token of dimension c. This directional masking strategy enforces an ordered structure the tokens. We set the minimum length to The length of the code at inference, leval, forms another axis to change the code shape (Section 3 and E Leval 2c.
200
Dataset Group
26
Simple
Mask Latent
0.75
25
simple
150
0.70
24
2-
2 125
23
0.65
22
100
22
0.60
21
21
0.55
50
20
2 Floating Points, E
21z
2
2 2
2
2 Floating Points,
210
2
Floating Paints, E
Floating Points, E
Figure 15. 256p Adaptive Masking ViTok Results. We investigate variations ViTok S-B/16 that apply token masking after encoding. We consider two approaches: Mask Simple, which masks the patch tokens following encoding, and Mask Latent, which introduces latent tokens (like the architecture used for Figure 14 and masks them. At stage training time we randomly selected token lengths (32, 64, 128, 256) with = 16, then at inference evaluate each model every token length and compare the simple ViTok baseline similar E. While the masking variations underperform the simple variant, they still perform strongly. Mask Simple tends to perform better higher E, while Mask Latent achieves better results lower E.
We now train our mask ViTok on stage 1 (Section 2.3 and investigate potential adaptive tokenization schemes. We first apply this masking strategy to the simple version of ViTok, directly masking the patch tokens after they have been processed by the encoder. We then explore the same approach on the latent version of ViTok. Both methods are trained with token lengths {32, 64, 128, 256} and e = 16 on ViTok S-B/16 using 256p images.
Figure 13 compares these masking methods with simple ViTok across different E. While all masking variants slightly underperform the simple ViTok, their overall performance remains strong. In particular, masking patches directly is more effective for higher E > 4096, whereas masking latent tokens performs better when E < 4096. These findings highlight how ViTok can be adapted for flexible token lengths during inference, and illustrate how our method can be extended to learn an ordered structure of tokens. Though more work here is needed to improve performance further.
B. Visualizations
In this section, we provide extra visualizations of generation examples from our various models and sweeps.
B.1. Image Generation
Here provide generation examples from our final models and sweep conducted in Figure The p visuals are shown in Figure 18 the p 32 visuals are shown in Figure [9 and the p 8 visuals can be found in Figure 20
Final ViTok model generations. are shown Figure 16 and Figure 17
We provide example generations from the DiT models trained in Table Visualizations
B.2. Video Generations
We include more video generation results in this section from Table and show example generations at 1024 tokens in
Figure 21
16

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Figure 16. 256p image generation examples. We show randomly selected 256p image generation examples from our DiT-XL. trained using the ViTok S-B/16 variant for 4 million steps batch size 256. Images were sampled with 250 steps using the DDIM sampler and a CFG weight of 4.0.
Figure 17. 512p image generation examples. We show randomly selected 512p image generation examples from our DIT-XL trained using ViTok S-B/16 variant for 4 million steps at batch size 256. Images were sampled with 250 steps using DDIM sampler and a CFG weight of 4.0.
17

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Patch Size 8, Channel
SDDDS
bood
Manso Iont
Patch Size 8, Channel 16
T
Patch Size 8, Channel 64
Figure 18. Channel size generation visualization 256p for = S. We show example generations for various compression ratios on ViTok S-B/8 from Figure Here 4 has the best visuals that look close good images, while 16 generally looks good well but not as good, e = 64 looks very poor and the images do not look realistic. 18

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Patch Size 16, Channel 4
EXON
MEBI
DD
130
Patch Size 16, Channel 16
Patch Size 16, Channel 64
Figure 19. Channel size generation visualization 256p for = 16. We show example generations for various compression ratios on ViTok S-B/16 from Figure Here 16 has the best visuals that look close good images, while 64 suffers artifacts that worsen image quality. 4 suffers from poor reconstruction quality from the auto-encoder.
19

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Patch Size 32, Channel 4
H
Patch Size 32, Channel 16
q न
Patch Size 32, Channel 64
IIP
ma
Figure 20. Channel size generation visualization 256p for 32, We show example generations for various compression ratios on ViTok S-B/32 from Fig Here 64 has the best visuals overall but the high channel sizes make the image quality look poor and jumbled. Both e 16 and suffers from poor reconstruction quality from the auto-encoder. 20

---

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
Figure 21. 128p video generation examples. We show randomly selected 16x 128 128 video generation examples from our DiT-L trained with ViTok S-B/4x8 variant. Vidcos are sampled with 250 steps and CPG weight of 2.0.
21


Reviews:
[{"review_id": "iidpfOe72k", "reviewer": "Reviewer_mrYe", "summary": "This paper introduces ViTok, a Vision Transformer-based autoencoder, and systematically investigates the impact of scaling bottleneck size, encoder, and decoder architectures on image and video reconstruction and generation performance. Key findings reveal that the total latent floating points E is the dominant factor for reconstruction quality: scaling E enhances reconstruction capabilities but does not continuously improve generation performance. Meanwhile, excessive increases in channel size c degrade generation quality due to rising complexity in the latent space. Experimental results demonstrate architectural scaling asymmetry: encoder scaling yields no significant reconstruction gains and may even reduce generation performance, whereas decoder scaling improves reconstruction quality but offers limited marginal benefits for generation. By incorporating perceptual loss (LPIPS) and adversarial training (GAN), the model achieves flexible trade-offs between visual fidelity (SSIM/PSNR) and generation quality (FID/IS), enabling the decoder to serve dual roles in reconstruction and generation. Notably, video data exhibits higher compression efficiency due to temporal redundancy, achieving superior reconstruction quality at the same E compared to images. Experimental validation shows that ViTok achieves state-of-the-art performance on 256p/512p ImageNet-1K/COCO image tasks and 16-frame 128p UCF-101 video tasks, with computational costs (FLOPs) reduced by 2–5× compared to traditional CNN methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The paper systematically validates four core claims through experiments: the total latent floating points E is the decisive factor for reconstruction quality, with sweeps over patch size p and channel number c revealing a strong positive correlation between E and reconstruction metrics (rFID/rIS) (Figure 3), where scaling E significantly improves image/video structural similarity (SSIM/PSNR) but excessive channel size c degrades generation performance due to increased latent space complexity; encoder scaling shows weak negative correlations with generation metrics (gFID/gIS) (Figures 5, 7), indicating that expanding encoder depth/width yields no reconstruction gains and may harm generation quality; decoder scaling demonstrates strong positive correlations with reconstruction quality (rFID/rIS) (Figure 6) but limited marginal improvements for generation (Figure 8), highlighting its role in enhancing local texture details while contributing minimally to global generative capabilities; video data achieves superior reconstruction quality at the same E compared to images due to temporal redundancy, with experiments showing that rFID comparable to 256p images requires only 4–8× E instead of 16× direct pixel scaling (Figure 10), validating ViTok’s efficient exploitation of spatio-temporal redundancy. These findings provide theoretical foundations for visual tokenizer architecture design and reveal the inherent trade-offs between reconstruction and generation in current autoencoding paradigms.", "methods_and_evaluation_criteria": "ViTok uses 3D ViT with SwiGLU and RoPE for spatiotemporal modeling, trained on large-scale datasets (Shutterstock 450M images/30M videos). Standard metrics (FID, SSIM, PSNR, FVD) and ablation studies validate the bottleneck analysis. Generation is paired with DiT-L for downstream tasks. Rigorous ablation of architectural choices (encoder/decoder/bottleneck) and large-scale training.", "theoretical_claims": "No formal theoretical proofs are provided. The paper focuses on empirical analysis of scaling trends.", "experimental_designs_or_analyses": "Comprehensive ablation studies (e.g., varying p, c, and model size) and comparisons across datasets (ImageNet, COCO, UCF-101).", "relation_to_broader_scientific_literature": "This work discusses the effect of scaling autoencoders. It has little impact to other fields except image generation.", "essential_references_not_discussed": "Essential references have been discussed.", "other_strengths_and_weaknesses": "To my understanding, this work demonstrates that scaling autoencoders does not effectively improve image generation performance. The paper conducts a detailed analysis of model scaling in terms of bottleneck size, encoder, and decoder architectures, showing that ViTok improves reconstruction capabilities. However, generation performance is much more important than reconstruction. How might the findings help researchers design better generators?", "other_comments_or_suggestions": "None.", "questions_for_authors": "Why in some experiments, better reconstruction performance does not lead to better generation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUHzUOfA0g", "reviewer": "Reviewer_ZzLx", "summary": "This paper conducts an empirical study on scaling up the autoencoder for generative models, with a ViT-style autoencoder. The authors investigate bottleneck size, encoder size, and decoder size, and find the generation performance is not significantly better with scaling up the autoencoder. As a result, the authors also provide a setting of the autoencoder that achieves state-of-the-art performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Claims are supported by experiments. While some claims may sound too strong if not with the corresponding context. For example, L28 \"when the bottleneck becomes large, generative performance declines due to increased channel sizes.\", is not always true if without context (increasing the bottleneck channel from 4 to 8 sometimes can improve the generation performance).", "methods_and_evaluation_criteria": "Method and evaluation makes sense. It is better to include human evaluation but it is too expensive given the large amount of experiments.", "theoretical_claims": "The paper is not about theoretical proofs.", "experimental_designs_or_analyses": "Experimental designs and analyses are generally good. One potential issue is that, when changing the design choice in the autoencoder, the way to train the latent diffusion model is fixed and classifier-free guidance (CFG) is also fixed. It is a reasonable way for ablation, while changing the autoencoder may potentially change the property of the latent space and thus change the optimal setting for training latent diffusion, for example, optimal CFG value may change.", "relation_to_broader_scientific_literature": "This paper conduct comprehensive experiments on scaling up the existing autoencoder based on GAN, LPIPS, L1 losses for helping understand the properties of the autoencoders.", "essential_references_not_discussed": "I didn't find key missing related work.", "other_strengths_and_weaknesses": "The findings in this paper could be very helpful to the field for understanding and improving the current autoencoders.\n\nWhile it could make this work stronger if having more interesting and deeper insights behind these findings.", "other_comments_or_suggestions": "While I believe the findings are valuable, \"scaling the auto-encoder alone does not significantly improve generative performance\" as the key learning of this paper is not a very rigorous conclusion to me, as there are potentially other ways to scale up.", "questions_for_authors": "Is there any hypothesis about why scaling up the autoencoder does not significantly help generation? Do these findings give some hint on how to properly scale up the autoencoder, or do they just suggest that scaling up the autoencoder is not a promising direction?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qQZpGn9YBZ", "reviewer": "Reviewer_ocDZ", "summary": "This paper conducts empirical studies on training Variational Autoencoders (VAEs) for latent diffusion models using Vision Transformers (ViT) as backbones instead of commonly used CNN-based models. It reports two main findings: (1) The reconstruction quality of the VAE does not necessarily determine the generative performance of the subsequent diffusion model; (2) Scaling the encoder size contributes minimally to reconstruction quality, whereas scaling the decoder size substantially improves performance, suggesting the decoder functions primarily as a generator.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The paper’s claims are generally supported by comprehensive experimental evidence. Particularly, the extensive experiments effectively illustrate the authors’ main points regarding the relationship between VAE reconstruction quality, model scaling (encoder vs. decoder), and the generative performance of the diffusion model. However, the manuscript lacks depth in explaining why scaling the decoder significantly boosts reconstruction but does not substantially benefit the subsequent generative phase.", "methods_and_evaluation_criteria": "The methods and evaluation criteria chosen by the authors are appropriate for the research objectives. The choice of ViT-based VAEs and comparisons regarding encoder and decoder scaling are logical for investigating these questions. Nevertheless, the paper notably lacks sufficient visual demonstrations of reconstruction quality, which are crucial for applications dependent on accurate latent representations, such as conditional generation tasks.", "theoretical_claims": "The paper does not explicitly provide theoretical proofs or extensive theoretical justifications. Most claims are empirical and based solely on experimental results.", "experimental_designs_or_analyses": "Experimental designs are robust and extensive, effectively validating the primary empirical claims. However, certain important analyses are missing. Specifically, the authors fail to deeply analyze and explain the following critical points:\n- How to optimally choose patch size and latent dimensionality given a fixed total floating-point budget (E)? While the authors suggest floating-point numbers in the latent codes are a bottleneck, the implications of different patch sizes on the complexity and generative quality remain unclear. Larger patch sizes would reduce token count and improve inference speed, but it is not evident from the current experiments whether this would degrade generation quality, increase model fitting difficulty, and why this occurs.\n- Why does scaling the decoder improve reconstruction significantly but yield limited benefits for the second-stage generative performance? The paper lacks an in-depth theoretical or analytical exploration of this phenomenon.", "relation_to_broader_scientific_literature": "The paper’s contributions mainly involve verifying and empirically supporting insights that are already widely accepted among researchers working with generative models, though not formally documented in existing literature. Its results clarify the roles of encoder and decoder scaling in ViT-based VAEs within latent diffusion models, contributing valuable experimental evidence for future reference.", "essential_references_not_discussed": "I didn't find there were essential reference missed", "other_strengths_and_weaknesses": "Strengths:\n- Solid and extensive experimental validation clearly supporting the empirical claims.\n\nWeaknesses:\n- The results feel somewhat incremental, providing little novel insight beyond empirical verification.\n- Lack of deep theoretical analysis or explanation of the observed phenomena.\n- Insufficient visual evidence provided for reconstruction quality.\n- Limited applicability of the ViT-based VAEs to a fixed image resolution.", "other_comments_or_suggestions": "The paper could significantly improve by providing more visual results for reconstructions, deeper theoretical or analytical insights, and explicitly addressing the trade-offs regarding latent dimension, patch size, and inference complexity.", "questions_for_authors": "1. Given that floating-point numbers in the latent codes (E) represent a key bottleneck, how should one optimally select the patch size and latent dimensionality for the generative model? Specifically, would larger patch sizes, which significantly reduce inference complexity, negatively affect generative performance, and why? Have you conducted experiments or theoretical analyses addressing this issue?\n2. While scaling the decoder substantially enhances reconstruction quality, why does this improvement not notably translate into better generative performance in the second stage? Can you provide deeper theoretical or analytical insights into this observation?\n3. The paper provides limited visual demonstrations (only Figure 2) of reconstruction quality. Could the authors provide more comprehensive visual evidence? This is especially important for applications relying heavily on accurate latent representations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lj93UBEBsE", "reviewer": "Reviewer_QXLn", "summary": "Most current-day generative models operate on abstract representations rather than pixels, which are often produced by a VAE-like \"stage 1\" model. These models are commonly trained separately from the subsequent generative modeling step (\"stage 2\"), such as diffusion modeling. The central question the authors tackle in this submission revolve around scaling three aspects of VAEs, namely the encoder and decoder capacities, as well as the bottleneck size. The paper demonstrates that the most effective way to improve reconstruction performance is to increase the bottleneck size, instead of scaling encoder or decoder. When it comes to generation, there is a sweet-spot to that; after a certain point generation performance suffers with too expressive latents. The paper further demonstrates how scaling the encoder is not beneficial, while scaling the decoder can slightly improve reconstruction fidelity. Overall, the paper presents a \"negative\" result in that scaling VAE encoder and decoder size does not significantly improve downstream generative performance.\n\n## update after rebuttal\nThe authors did not submit a rebuttal, which leaves all my questions and concerns open. Due to the uncertainty about those parts I lower my rating, but I still think the submission is overall a positive contribution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Most claims are supported by enough evidence. There are, however, a few areas I would like to point out:\n1. In several places (e.g. L333 \"outperforming other CNN-based continuous tokenizers while utilizing significantly fewer FLOPs.\", or L373 \"achieves state-of-the-art (SOTA) rFVD results compared to other CNN-based continuous tokenizers with the same total compression ratio.\") the paper suggests that ViT-based autoencoders are better than CNN-based ones. These claims are made in comparison to other models, trained on other datasets, with different training schedules and objectives. To support these suggestions, I'd like to see more controlled experiments; especially since the paper highlights the benefits of Llama's architecture modifications over more vanilla Transformers.\n2. Section 3.2 and Finding 2: The authors claim that high E complicates convergence and degrades generation performance. This is demonstrated empirically in Fig 4 for a fixed stage 2 model size. That said, this result ignores the effect stage 2 model size / compute has on the optimal stage 1 compression ratio. Indeed, Esser et al. [1] demonstrate that larger stage 2 models benefit from less stage 1 compression. Stage 2 is usually the one that is scaled, which makes it important for explorations into stage 1 to be done with that in mind.\n\n[1] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, Esser et al. 2024", "methods_and_evaluation_criteria": "The evaluation criteria make sense to me. The authors evaluate on two common domains, image and video generation, and evaluate common metrics. I also am glad to see that the authors analyze per-image reconstruction metrics, and not only FID.", "theoretical_claims": "The paper does not make theoretical claims. All claims are empirical.", "experimental_designs_or_analyses": "The experimental design and analyses appear sound. The authors provide enough implementation details that I would feel comfortable reimplementing the experiments.", "relation_to_broader_scientific_literature": "The submission is timely, as there is a recent uptick in investigations into the role of tokenizers for downstream generative modeling tasks. Especially in the diffusion model literature, the effect of the first stage autoencoders has been understudied. Several tokenizer-focused papers, such as ViT-VQGAN or TiTok, investigate the stage 1 tokenizer scaling to some extent, but don't make it the main focus.", "essential_references_not_discussed": "1. Recently, there has been a focus on training tokenizers jointly with a regularization objective to make the latents more predictable by a downstream generative model, e.g. see [2,3]. These works show an alternative view on tokenization, that differs from the strictly separated stage 1 vs stage 2 view adopted in this paper. I would suggest the authors to add a discussion on these kinds of works and how it relates to their findings.\n2. The appendix shows preliminary experiments towards 1D token representations, as well as ordered token sequences. I would suggest the authors to discuss the following relevant works: [4-7]\n\n[2] When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization, Ramanujan et al. 2024\n\n[3] JetFormer: An Autoregressive Generative Model of Raw Images and Text, Tschannen et al. 2024\n\n[4] Matryoshka representation learning, Kusupati et al. 2022\n\n[5] Learning ordered representations with nested dropout, Rippel et al. 2014\n\n[6] Adaptive Length Image Tokenization via Recurrent Allocation, Duggal et al. 2024\n\n[7] Visual Lexicon: Rich Image Features in Language Space, Wang et al. 2024", "other_strengths_and_weaknesses": "The authors investigate an understudied problem and show a clear negative result about scaling tokenizer capacity. This is a useful finding to the community, and may help reduce wasteful experiments in this area. It is also commendable that the tokenizers are not trained in a data-constrained setting, such as ImageNet, which would make me question some of the results. Overall the paper is well written and presented.\n\nTo me the main limitations of the submission lie in that stage 2 scaling is not taken into account, which may distort some of the findings. For example, it has been shown that scaled-up stage 2 models can take advantage of more expressive latent spaces, yet the experiments in this paper would suggest that the stage 1 bottleneck size should not be scaled beyond a certain threshold. Unlike for the bottleneck size, I am not aware of other papers investigating the impact of stage 1 encoder and decoder capacity when scaling stage 2 models, which makes me slightly less confident about that part of the results.", "other_comments_or_suggestions": "1. L143, Sec 3.1. Missing word in third line of paragraph, \"For , ...\"\n2. Nitpick: The compression ratio in Sec 3.1 also depends on the precision of the latents and inputs.\n3. L438: \"workk\"\n4. Tables 2, 3, 4: I suggest showing the compression ratios (both token count and number of channels) of different tokenizers, along the Params and GFLOPs. As also noted by the paper, the compression ratio is one of the most significant hyperparameters that determines the final reconstruction performance; more so than encoder and decoder capacity (and presumably architecture). To illustrate the necessity for that, one could list a simple reshaping operation that performs 16x spatial compression at the cost of 16x the number of channels as a baseline with 0 Params, 0 GFLOPs, and perfect reconstruction metrics. These numbers can arbitrarily be gamed if the compression ratio is not shown.\n5. Table 2: Why are the Params and GFLOPs of SDXL-VAE, OAI, and COSMOS-CI not listed?\n6. Figures 16-20: The images are very compressed, and show a lot of banding. It's hard to evaluate some aspects of the tokenizer quality visually to to that strong compression.", "questions_for_authors": "For Section 3.2 / Figure 4, did the authors run any stage 2 scaling experiments? How does the U-shaped curve shift for larger stage 2 models / comptue?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper analyzes the effects of scaling different parts of a ViT-based autoencoder on the performance of image/video reconstruction and generation. The authors did not provide any rebuttal. The final recommendations are 4 "Weak accept". The reviewers generally agree that
- experimental designs and analyses are appropriate and comprehensive,
- claims are supported by experiments, and
- findings are helpful for understanding and improving autoencoders (for reconstruction).

Their major concerns include
- only empirical results with no theoretical proofs or extensive theoretical justifications, and
- lack of positive directions on how to improve generation performance.

Taking both the above pros and cons into account, I recommend "Weak accept".
Decision: Accept (poster)