Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization
Authors: Yifan Chang
Jie Qin
Limeng Qiao
Xiaofeng Wang
Zheng Zhu
Lin Ma
Xingang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Generative Model
Image Quantization
Autoregressive Modeling
Image Generation
Image Synthesis
Primary area: generative models
TL;DR: We present FVQ, a discrete tokenizer framework that resolves key VQ challenges, sets new state-of-the-art results, and enables vanilla AR models to outperform VAR and DiT.

Abstract: Vector quantization (VQ) is a key component in discrete tokenizers for image generation, but its training is often unstable due to straight-through estimation bias, one-step-behind updates, and sparse codebook gradients, which lead to suboptimal reconstruction performance and low codebook usage. 
In this work, we analyze these fundamental challenges and provide a simple yet effective solution. 
To maintain high codebook usage in VQ networks (VQN) during learning annealing and codebook size expansion, we propose VQBridge, a robust, scalable, and efficient projector based on the map function method. VQBridge optimizes code vectors through a compress–process–recover pipeline, enabling stable and effective codebook training. 
By combining VQBridge with learning annealing, our VQN achieves full (100\%) codebook usage across diverse codebook configurations, which we refer to as FVQ (FullVQ).
Through extensive experiments, we demonstrate that FVQ is effective, scalable, and generalizable: it attains 100\% codebook usage even with a 262k-codebook, achieves state-of-the-art reconstruction performance, consistently improves with larger codebooks, higher vector channels, or longer training, and remains effective across different VQ variants. Moreover, when integrated with LlamaGen, FVQ significantly enhances image generation performance, surpassing visual autoregressive models (VAR) by 0.5 and diffusion models (DiT) by 0.2 rFID, highlighting the importance of high-quality tokenizers for strong autoregressive image generation.

Published as a conference paper at ICLR 2026
SCALABLE TRAINING FOR VECTOR-QUANTIZE NET- WORKS WITH 100% CODEBOOK UTILIZATION
Yifan Chang
Jie Qin
Limeng Qiao Xingang Wang!.5 1,5
Xiaofeng Wang²
Zheng Zhu
Lin Ma
CCASIA
Meituan
GigaAl *UCAS
Luoyang Institute for Robot and Intelligent Equipment
Performance Comparison
Model Architecture
diccrete tokeens
codebook
40
VQ
just for tralving
VQBridge
VOVAE/VOGAN
FVQ (ours)
Figure 1: Performance comparison and model architecture. Left: Performance comparison showing generation (FID) and reconstruction (rFID) quality across different methods. Right: Architectural comparison between baseline VQVAE/VOGAN and the proposed FVQ model, with VQBridge component for training.
ABSTRACT
Vector quantization (VQ) is a key component in discrete tokenizers for image generation, but its training is often unstable due to straight-through estimation bias, tre-step-behind updates, and sparse codebook gradients, which lead to suboptimal reconstruction performance and low codebook usage. this work, we analyze these fundamental challenges and provide simple yet effective solution. To main- tain high codebook usage in VQ networks (VQN) during learning annealing and codebook size expansion, we propose VQBridge, robust, scalable, and efficient projector based on the map function method. VQBridge optimizes code vectors through compress-process-reonn
pipeline, enabling stable and effective code- book training. By combining VQBridge with learning annealing. our VQN achieves full (100%) codebook usage across diverse codebook configurations, which we refer to as FVQ (FullVQ). Through extensive experiments, we demonstrate that FVQ is effective, scalable, and generalizable: it attains 100% codebook usage even with a 262k-codebook, achieves state-of-the-an reconstruction performance, consis- tently improves with larger codebooks, higher vector channels, or longer training, and remains effective across different VQ variants. Moreover, when integrated with LlamaGen, FVQ significantly enhances image generation performance, surpassing visual autoregressive models (VAR) by 0.5 and diffusion models (DiT) by 0.2 rFID, highlighting the importance of high-quality tokenizers for strong autoregressive image generation. https: / / thub. cwm/c watmm//fhaang-cc///ecc
INTRODUCTION
The scalability of autoregressive models has been well demonstrated in large language models (Rad- ford et al., 2018; Brown al., 2020; Touvron al., 2023; Grattafiori et al., 2024; Bai et al., 2023). Inspired by this, pioneering works (Esser et al., 2021; Yu et al., 2022; Ramesh et al., 2021: Tian et al., 2024) have explored the application of autoregressive models in visual generation tasks. To enhance scalability and reduce computational costs, common approach (Van Den Oord et al., 2017:
Razavi et al., 2019) involves introducing discrete tokenizers that employ vector quantization (VQ) to convert continuous images into discrete tokens. Autoregressive models trained on these tokens in a self-supervised strategy to predict the next tokens. Generally, the reconstruction capability of the tokenizer directly determines the upper bound of the generative model's 's performance (Esser

---

Published as a conference paper at ICLR 2026
et al., 2021; Yu et al., 2024a). Therefore, the design of discrete tokenizers plays crucial role in
autoregressive visual generation.
However, due to the information loss introduced by vector quantization, the performance of discrete tokenizers has long been limited compared with continuous tokenizers (Kingma, 2013). The intu- itive insight is that enlarging the codebook could reduce the information loss during quantization. Nevertheless, prior work (Yu et al., 2022; Sun et al., 2024; Ma et al., 2025) has shown that simply increasing the codebook size or code dimension often leads to codebook collapse, which causes a severe reduction in codebook usage and consequently undermines the generative capability of downstream models (Huh et al., 2023; Luo al., 2024). Despite many efforts to mitigate this issue, when scaling to larger settings (e.g., codebook size of 262k and code dimension of 256) (Shi et al., 2025), full codebook usage remains unattainable and the performance is still unsatisfactory.
To address this issue, we revisit VQ from fundamental mechanism and three fundamental chal- lenges (Huh et al., 2023), together with empirical observations that shed light on resolving them. VQ relies on the straight-through estimator (STE) (Bengio et al., 2013) to resolve the non-differentiabiiiy problem, but this introduces three additional challenges: straight-through estimation bias, one-step- behind update, and sparse codebook gradients. The first two challenges cause misalignment in optimization across the network, leading to instability, while sparse codebook gradients prevent the codebook from being sufficiently trained. In practice, only small subset of code vectors updated, which is the primary cause of collapse.
Surprisingly, based our observations, these challenges can be effectively mitigated using only learning annealing and a sufficient powerful projector that jointly optimizes the codebook. Prior works (Zhu al., 2024a;b) employ linear layer as the projector for joint codebook optimization, showing some improvement. However, our further experiments reveal that a linear projector alone is fragile: it makes the network highly sensitive to the learning rate and insufficiently capable when scaling larger codebooks. To enable robust and scalable VQ training, we propose robust, scalable and efficient projector, called VQBridge, as shown in Figure 1. VQBridge adopts a compress-procestt proe-poo-ocsoooeeeeee pipeline: it first compresses the set of code vectors into smaller number, then models interactions via VIT blocks (Dosovitskiy et al., 2020), and finally recovers the vectors to the original codebook size and dimensionality. This design allows VQ to rapidly achieve and sustain 100% codebook usage for arbitrary codebook configurations, enabling effective optimization VQ.
Using VQBridge combined with learning annealing, we conduct extensive experiments to validate its effectiveness, scalability, and generalization. On the ImageNet benchmark (Deng et al., 2009), FVQ consistently achieves 100% codebook usage across all configurations, including a 262k-codebook. while substantially enhancing the reconstruction performance of discrete tokenizers (rFI)= 0.88), without introducing additional parameters inference cost. FVQ exhibits strong scalability, with performance consistently improving as the codebook size, vector channel, and number of training epochs increase. Furthermore, the approach remains effective when applied multi-code representa- tion VQ (Lee et al., 2022; Tian et al., 2024), demonstrating robust generalization. Besides, FVQ leads to substantial improvements in generation performance: for instance, LlamaGen-Xi improves FID from 3.39 to 2.07, outperforming comparable visual autoregressive models VAR (Tian et al., 2024) (FID=2.57) and diffusion models DiT (Peebles & Xie, 2023) (FID=2.27). These results highlight that even a vanilla autoregressive framework can achieve strong generation capabilities when the tokenizer exhibits high-quality reconstruction.
In summary, our contributions include:
We analyze the fundamental challenges arising from vector quantization and provide effective observations, leading to simple yet scalable training framework (FVQ).
Through extensive experiments, we show that FVQ effective, scalable, and generalizable. Specifically, it achieves 100% codebook usage across all configurations, attains state-of-the-art reconstruction performance, and consistently improves as the codebook size, vector channel, and number of training epochs increase, remains effective across different types of VQ.
3. We demonstrate that FVQ substantially enhances the generative capability of autoregressive models. Without any bells and whistles, vanilla autoregressive framework equipped with FVQ outperforms ad advanced visual autoregressive models (VAR) and diffusion models (DiT).
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
Experiment
I-NNE
Standard
Wf VGDrdge
Stundard (1aagan77))
Liwer Lager (PsgeeSTS)
WC W.VGDndgg Oaagen
% Lugel
4.00
Toy experiment and t-SNE visualization results. Left: A dynamic vector quantization toy experiment where the target is fixed optimization objective and the model is optimized with SGD. Using the standard method with only STE leads a spiral trajectory. while VQBridge produces a piecewise linear trajectory. With VQBridge, 2g closely follows ze. reducing their distance already in the early stages of training. Right: Due to sparse gradients from the codebook, the standard method achieves only 3% codebook usage. Adding linear layer for joint optimization improves the usage but remains insufficient, whereas VQBridge achieves 100% usage.
Figure 2:
Vector Quantized Network. A vector quantized network (VQN) contains Encoder E(-, Vector Quantization Layer -(:,-) and Decoder (-)). Given vector x, the encoder maps it into latent representation Ze- The codebook is a collection of K code vectors, denoted as C -2,1.2,..... Cx J. C2,....
where each vector c, has same dimension ze. In quantization layer Q(+, :), 2g quantized into by by assigning it into code vector from codebook C' using distance measure d(:;)) )):
2g = C
where
x = arg min d(ze; C,).
Therefore, VQNs can denoted:
g =
C)) D(zg).
(2)
VQNs predict the output ý and compute loss with target y : L)y. The objective of VQNs is to optimize the following loss function:
min E (PD))))))) E,O.D
(3)
where T is training dataset. The extension to image reconstruction straightforward by applying the formulation to 2D data; see Appendix F for details.
Optimization Challenge. Since the arg min operation in the quantization layer Q(.) non- differentiable, a common approach is to apply the straight-through estimator (STE) during backprop- agation (Bengio et al., 2013), which allows gradients to bypass the discrete operation:
Z Ze + sg((z, -ze)],
(4)
where sgl: doopggradient operation. In addition, a commitment loss (Van Den Oord al., 2017) is introduced to ensure that the gradient flow on both sides of the quantization layer remains accurate:
Lemi (Ze, Zg)= d(zzg sg(z.]) 3d(zz: sg(z,,).
Although this approach makes the objective differentiable and thus optimizable, it introduces several additional challenges.
Estimation Bias. The straight-through estimation error, defined as 8 2 Ze, introduces bias into the training process. It directly affects the decoder D(.)) and indirectly impacts the encoder E() through the commitment loss (Huh et al., 2023).
Challenge I: Straight-
Challenge 2: One-step-behind Update. Through derivation (see Appendix D). we show that the commitment loss causes the model to align with historical representations instead of the current ones (Lańcucki et al., 2020; Huh al., 2023). Specifically. the codebook is updated as
Zg ((+1) + (1 - n) Za (4) + 1 ze (1)
(6)
3

---

Published as a conference paper at ICLR 2026
and the decoder is trained on the previous codebook output. This mismatch leads misalignment between the encoder and decoder. Ideally, both should be updated using the current representation:
(++1) + (1 7) Zg (1) + Ze (++1))
(7)
Challenge 3: Sparse Codebook Gradients. The codebook C suffers from sparse gradients-oni the selected code vectors receive updates, while the unselected ones get no gradient signal and are likely to remain unused indefinitely (Zheng & Vedaldi, 2023; Shi et al., 2025). This results in codebook collapse and limited representational capacity.
We use a toy experiment to illustrate the impact of the STE. As shown in Figure 2, the optimization trajectories of ze and By form a spiral shape, revealing hindered optimization (Challenge 1 & 2). Moreover, the t-SNE visualization further shows that directly applying the STE leads to low usage, with most codebook entries left unused. This phenomenon arises from the sparse gradients received by the codebook (Challenge 3).
METHODOLOGGY
3.1
PRELIMINARY OBSERVATIONS
16K
16k Annealing
262k
262k Annealing
190
Usage (P))
Usage (1)
60-
50-
C3
MLP-6 VCBridge
MLP-5 VQBridge
Usage
Usage
Linear VOBridge
Lineer
VCSridge
28 Epoch
Epoch
Epoch
Epoch
Figure 3: Codebook usage during training. With codebook size of 16k, the lincar-layer baseline can achieve 100% usage, but usage drops significantly when learning annealing is applied. With larger codebook size of 262k, even S-layer MLP fails maintain sufficient usage. contrast, VQBridge rapidly achieves 100% usage under all configurations and with learning annealing, which greatly facilitates VQN optimization.
We denote the corresponding sets of Ze, c, and 2g as P., C., and Hz. respectively. The distance between the distributions of the two sets can be formally defined
D(P,, C.) = Pa
Σ 7.EP
min d(z,, C2). c,e,,
Our work is based on the simple and effective linear reparameterization (Huh et al., 2023; Zhu et al., 2024a;b), which can be generalized as a mapping function f(-) from C, to a new set Ć., with joint optimization over c: ć= [C2)). Consequently, the training objective shifts to minimizing the distance between P. and C.: D(P C.). ). A smaller D(P,, C.) indicates smaller estimation error However, estimation error does not necessarily reflect optimization effectiveness, since a small number of active codebook vectors may lead to lower loss but fail yield meaningful performance gains (Huh et al., 2023). Codebook usage can therefore serve as complementary metric, as higher usage indicates that more vectors are effectively contributing, which is more informative.
Observation 1 To mitigate the straight-through estimation bias, rapid alignment of the two distribu- tions is crucial, reducing D(Pz, 2. C.)) early in training. Furthermore, estimation error must remain low for stable optimization. In practice this requires using larger codebook while maintaining high usage from early to late training. We present the effect of codebook size on the loss in Appendix
Observation 2 Learning annealing is necessary, as a smaller learning rate helps reduce the one- step-behind effect and improves the overall alignment of the model. We establish an upper bound on the effect of the one-step stei-hehindd update and demonstrate that learning annealing serves as an effective remedy. Further derivations are presented in Appendix E.
4

---

Published as a conference paper at ICLR 2026
Observation 3 Linear layer is weak. As shown in Figure 3, observe that when linear layer is used as the mapping function f()), the codebook usage drops significantly during learning annealing. Additionally, under codebook scaling, the linear mapping struggles model capacity with a 5-layer MLP still result in poor usage. This suggests that the representational power of such mappings is inadequate for aligning the distributions of Ps and C..
more. Attempts increase
Based on the above observations, stable and scalable training of VQN requires strong and robust mapping function f(),, combined with learning annealing. In the following section, we introduce our proposed design, referred to as VQBridge.
16584
Decoder
Linear and Reshape
UnPatchify
Layer Norm
Quant
VQBridge
N
ViT Block
Layer Norm
Encoder
Patchify
Reshape and Linear
Figure 4: FVQ framwork overview. VQBridge first applies ID patchification to the code vectors, then performs global interaction using ViT blocks, and finally restores them their original size.
3.2 VQBRIDGE DESIGN
The design objective of VQBridge is achieve strong fitting capability, enabling rapid align- ment between two distributions, while maintaining scalability and training stability. Inspired by the architecture of Diffusion Transformers (DiT) (Peebles & Xie, 2023), the module follows pipeline, where the compression step enables efficient scaling to large codebooks (e.g.. 262k entries). To simplify the problem, we assume that codebook and the mapped codebook é = f(C) share the same size and vector channel. Details are provided in Figure 4.
compress-process-reed
First, we perform ID patchify operation. Given codebook C containing K code vectors, we divide it into groups, each containing K vectors. For each group, the vectors are compressed into a single vector via shared linear projection Wcomp
IR ta xd', followed by LayerNorm operation:
h (C, W Woop)) g 1.....P
(8)
Next, N VIT blocks are applied as scalable layers to process the intermediate representation H E Rx H [h,, h hp). where each h, is compressed intermediate representation. This design enables global information exchange and allowing the model to fit complex relationships between code vectors:
H' ViT (H).
Finally, ID umpatchify operation restores the representation to its original size. Specifically, H' is first normalized via LayerNorm, then expanded to times dimension through a shared linear
projection Woxp E dd Ka
H = LN H Weep:
(10)
followed by a reshape operation that converts each expanded vector back into p vectors within group:
Č = reshape(H)) R kxd
(11)
This yields a codebook Ć the same size as the input to VQBridge, where the code vectors have undergone full mutual interaction.
5

---

Published as a conference paper at ICLR 2026
Note that both VQBridge and the original codebook C can be discarded after training, maintaining only the mapped codebook Ć. This implies that VQBridge introduces additional computational overhead during inference, preserving the original VQN's workflow and computational cost.
EXPERIMENTS
4.1 MAIN EXPERIMENTS
Table 1: Reconstruction performance of different tokenizers on 256 ImageNet 50k validation set using the same Encoder and Decoder. trained on OpenImages (Kuznetsova et al., 2020). trained for 330 epochs. trained for 120 epochs. We highlight the best results in bold and underline the second-best results.
Codebook Size
Vector Channel
Codebook Usaget
Method
Tokens
rFID
LPIPS!
VOGAN (Esser al., 2021)
16 16
1,024
256
7.94
44%
VQGAN (Esser et al., 2021)
16 x 16
16,384
256
4.98
0.17
5.9%
SD (Rombach et al., 2022a)
16 16
16,384
8
5.15
MaskGIT (Chang et al., 2022)
x
1,024
256
2.28
LlamaGen (Sun et al., 2024)
16 X 16
16,384
8
2.19
0.14
97%
LlamaGen (Sun al., 2024)
16 16
16,384
256
9.21
0.29%
VQGAN-LC (Zhu et al., 2024a)
16 X 16
16,384
3.01
0.13
99%
VQGAN-LC (Zhu et al., 2024a)
16 X 16
100,000
8
2.62
0.12
99%
IBQ' (Shi et al., 2025)
X 16
16,384
256
1,55
0.23
97%
FVQ
x 16
16,384
256
1.30
0.13
100%
FVQ*
16 x 16
16,384
256
1.17
0.13
100%
Table 2: Generative model comparison class-conditional ImageNet 256 256. The metrics
include Fréchet Inception Distance (FID), Inception Score (IS), Precision (Pre), and Recall (Rec). We highlight the best results in bold and underline the second-bes results.
Type
Model
#Para
FID
ISt
Pret
Reet
Diff.
DiT-L/2 (Peebles & Xie, 2023)
458M
5.02
167.2
0.75
0.57
Diff.
DiT-XL/2 (Peebles Xie, 2023)
675M
2.27
278.2
0.83
0.57
Mask.
MaskGIT (Chang et al., 2022)
227M
6.18
182.1
0.80
0.51
VAR-d16 (Tian al., 2024)
310M
3.30
274.4
0.84
0.51
VAR
VAR-d20 (Tian et al., 2024)
600M
2.57
302.6
0.83
0.56
AR
VQGAN (Esser et al., 2021)
227M
18.65
80.4
0.78
0.26
AR
LlamaGen-L (Sun al., 2024)
343M
3.81
248.3
0.83
0.52
AR
LlamaGen-XL (Sun et al., 2024)
775M
3,39
227.1
0.81
0.54
AR
IBQ-B (Shi et al., 2025)
342M
2.88
254.4
0.84
0.51
IBQ-L (Shi et al., 2025)
649M
2.45
267.5
0.83
0.52
FVQ-L (LlamaGen-L)
343M
2.39
276.6
0.84
0.56
AR
FVQ-XL XXL
775M
2.07
287.0
0.83
0.58
Implementation Details. For visual reconstruction, we adopt a codebook with size of 16.384 and vector channel of 256. VQBridge is configured with latent dimension of 256, a depth of 2 and patch size of 16. The encoder and decoder configurations are identical to those in VQGAN (Esser et al., 2021; Sun et al., 2024), utilizing 2 layers of ResBlocks. The model is trained for 40 epochs with base learning rate of le-4 and batch size of 128. Learning annealing starts with 4-epoch warmup, followed by a learning rate decay to 1% of the original value. The Adam optimizer (Kingma, 2014) is employed with 31 0.9 and B2 0.95. Finally, we scale the training time to 120 epochs. For visual generation, LlamaGen (Sun et al., 2024) is utilized as the generator, with training configurations aligned with the open-source implementations. Additional details provided in the Appendix G.

---

Published as a conference paper at ICLR 2026
Original Image
FVQ (ours)
VQGAN
FVQ
Reconstruction
Generation
Figure 5: Qualitative results of FVQ. Left: reconstruction performance compared against baseline methods. Right: generation results.
Reconstruction Results. In Table I, we compare existing tokenizers using the same encoder and decoder architectures, most of which trained on ImageNet- 256x256 with a downsampling ratio of 16. FVQ achieves an rFID of 1.30 after 40 epochs, outperforming all other discrete tokenizers. Scaling the training time to 120 epochs further improves the rFID to 1.17. By addressing the opti- mization challenges of vector quantization, our method can be extended to other discrete tokenizers, unlocking their full potential (see Section 4.4).
Visual Generation. We report the results of LlamaGen (Sun et al., 2024) generators in Table 2, comparing them with prior generative models. FVQ improves the FID of LlamaGen-L from 3.81 1 to 2.39, surpassing other models with similar parameter sizes. Notably, FVQ-L (343M) even outperforms VAR-d20 (600M, 2.39 vs. 2.57), and FVQ-XL exceeds DīT-XL/2 (2.07 vs. 2.27), demonstrating the potential of autoregressive (AR) methods. These results further emphasize that the tokenizer the key to AR visual generation. Previous AR models are constrained by suboptimal tokenizers, whereas FVQ significantly enhances the generative capabilities of AR models, fully unlocking their potential. See Figure 5 for visualization results.
4.2 ABLATION STUDIES
To identify the most efficient and effective setup, we explore various settings VQBridge Figure 6, focusing the interplay between its parameters, codebook's 's size and vector channel.
2.00
2.00
2.00
regonatnaction (FF10)
1.75
veckor chanmel=120
channem23i
1.75
wedor channel-128
channet-286
1.75
codebookk size-lk
codebook sizen16x
1.50
1.50
1.25
1.25
1.25
1.00
1.00
1.00
0.75
0.75
0.75
Latent Dim. (0')
Dapth (V)
(f)
Figure 6: VQBridge desin ablation results. We vary VQBridge configurations including the latent dimension (d'), VIT block depth (N), and patch size (p), well codebook parameters such the vector channel and codebook size.
Latent Dimension (d'). left subfigure, increasing d' from 64 to 256 consistently improves the reconstruction quality. Interestingly, the best results appear when the latent dimension d' matches the vector channel size, indicating that a balanced latent embedding dimension and code vector width facilitates more effective representation learning. Further enlarging d' to 512 does not provide additional benefits, and may even degrade performance due to over over-paramenn seeeraaramtterzztiion and optimization difficulties. Therefore, set d' equal the vector channel size in final configuration.
7

---

Published as a conference paper at ICLR 2026
Depth of VIT Block (N). The middle subfigure shows that using single VIT block leads to suboptimal results. Increasing the depth to N = significantly improves performance, while further stacking blocks (N 4) actually degrades performance due to increased optimization difficulty and higher computation cost. Therefore, N = is adopted as our default.
Patch Size (p). The right subfigure presents the effect of patch size under different codebook sizes, with both parameters following a 4" scaling pattern. Our experiments reveal a clear scaling relationship: when K = 4k, the optimal patch size p 4, while when K 16k, the optimal patch size is 16, indicating that patch size and codebook size should be scaled together to achieve optimal performance. This scaling relationship reflects a balance between the computational pressure on the Patchify stage and the ViT blocks.
Our final model adopts latent dimension d' consistent with the vector channel dimension (default 256). 2 VIT blocks, and p 16 with a K = 16k codebook. When scaling up the codebook, we simultaneously increase the patch size by the same factor of 4x. This scaling strategy ensures optimal performance across different model configurations while preserving computational efficiency.
4.3 SCALING STUDIES
Table presents comprehensive evaluation of scalability across different model configurations, examining the effeets of codebook size, vector channels, and training epochs reconstruction quality and codebook usage. The results demonstrate performance differences between VQGAN and FVQ across multiple scaling dimensions. All models trained on ImageNet (Deng et al., 2009) and evaluated on both the ImageNet val set (50k) and COCO 2017 val set (5k) (LAION, 2022).
Table 3: Results of scalability from codebook size, vector channel and training epoch. rFID INIK denotes the results evaluated on the ImageNet- Ik validation set, while rFID denotes the results evaluated on the COCO validation set. (-Ann.) denotes results without learning annealing.
Training Epoch
Codebook
Vector Channel
Model
Ratio
iFIDINKK
iFIDcoco
Usage
Size
VQGAN VQGAN
40
16
16k
8
2.19
8.43
97%
16k
256
9.21
0.29%
FVQ
16k
256
2.61
100%
4
FVQ
40
16
16k
4
1.71
7.64
100%
FVQ
16k
1.47
7.15
100%
FVQ
16k
256
1.30
6.84
100%
FVQ
120
16
16k
256
1.17
6.62
100%
FVQ
40
16
65k
256
1.14
6.39
100%
FVQ
40
16
65k
512
1.09
6.38
100%
FVQ
65k
512
1.00
6.16
100%
11
FVQ
262k
256
0.95
6.04
100%
12
FVQ
120
16
262k
256
0.88
5.83
100%
13
VOGAN
16k
0.98
5.03
97%
14
FVQ
40
8
16k
256
0.39
3.84
100%
Model Performance and Stability. FVQ consistently outperforms VQGAN in both reconstruction quality and codebook usage. As shown in Figure FVQ achieves 100% codebook usage early in training and maintains it throughout. Learning annealing further improves alignment in VQNs (from 2.61 to 1.30), noted in Observation 2, which is also confirmed by our experiments. FVQ sustains full codebook usage across all tested configurations, whereas VQGAN experiences severe codebook collapse in high-capacity settings, with usage dropping to only 0.29% when employing 256 vector channels, validating Observation 3. This stability advantage is crucial for practical applications, ensures reliable performance scaling without catastrophic failure modes.
Codebook Size Scaling. Our experiments reveal clear scaling benefits with increased codebook capacity. For FVQ with 40 training epochs and 256 vector channels, expanding the codebook from 16k to 262k entries yields substantial improvements in reconstruction quality, with rFID decreasing from 1.30 to 0.95 and rFIDcoco from 6.84 to 6.04. This trend indicates that larger codebooks provide enhanced representational capacity without suffering from underutilization issues.
8

---

Published as a conference paper at ICLR 2026
Vector Channel Effects. Increasing vector channels from 4 to 256 leads to substantial improvements in reconstruction quality across both datasets. On ImageNet, rFIDINIK decreases from 1.71 to 1.30, while COCO, rFIDcoco improves from 7.64 to 6.84. This trend demonstrates that higher- dimensional vector representations enable more expressive feature encoding, allowing the model capture finer visual details and achieve better reconstruction fidelity. Importantly, these findings provide a crucial foundation for integrating semantic tokenizer (e.g., CLIP (Radford et al., 2021) with 512-dimensional embeddings) with vector quantization.
Training Duration Impact. Extending training from 40 to 120 epochs yields consistent but di- minishing returns on ImageNet, while on COCO results in steady improvements. with rFID increasing by approximately 0.2 at each evaluation interval.
Compression Trade-ofTs. Higher compression ratios (e.g.. 16x) make the reconstruction task more challenging, reducing reconstruction quality, while lower compression ratios (e.g.. 8x) facilitate learning and improve reconstruction fidelity. Both models benefit from reduced compression, but FVQ maintains its stability advantage across compression levels, achieving rFIDINIK of 0.39 and rFIDcoco of 3.84 at 8x compression with full codebook usage.
These findings establish FVQ as more scalable and reliable architecture for vector quantization tasks, demonstrating predictable performance improvements with increased computational resources while maintaining full codebook usage across diverse configurations.
4.4 GENERALIZATION STUDIES
Recently, several multi-code representation methods have been proposed to improve VQ. where multiple tokens are used to represent a single spatial location (e.g.. RQ-VAE (Lee al., 2022) and VAR (Tian et al., 2024)). To demonstrate the generalization ability of our approach, further conduct experiments on these two methods.
Table 4: Multi-Code Representation VQ. 68010step denotes code map constructed through 10 sequential quantization steps with step sizes 1, 2, 3, 4, 5, 6, 8, 10, 13, 16, where 680 is the sum of squares of the sequence.
Training Epoch
Token Map
Codebook Size
Vector Channel
Method
rFID]
RQ-VAE (l.ce et al., 2022)
10
8 8 x 4
16,384
256
4.73
RQ-VAE (Lee et al., 2022)
50
8
8 4
16,384
256
3,20
FVQ (RQ-VAF)
8
8
16,384
256
2.98
(Tian et al., 2024)
40
6801ostep
4,096
1.00
FVQ (VAR)
40
6801axtep
16,384
256
0.80
We follow the original training protocols as closely as possible. The results are reported in Table 4, where our method consistently improves performance. Notably, with only 10 epochs of training, our approach surpasses RQ-VA VAE trained for 50 epochs. This highlights the strong generalization ability and training efficiency our method.
MORE DISCUSSION
In this section, we clarify why shared MLP projector collapses under learning-rate annealing, whereas VQBridge remains stable. We focus on the structural limitations of the MLP under straight- through estimation and contrast them with the dense, cross-code interactions introduced by VQBridge.
MI.P + STE leads to low-rank, usage-sparse updates. the baseline, all code vectors are transformed by shared ML. fo:
é = fo(Co), I,...,K K. K.
(12)
9

---

Published as a conference paper at ICLR 2026
Although gradients from active codes update the shared parameters, only codes in the active set A contribute to the STE loss at iteration t. The parameter gradient is
aL 08
J(ck)
gh:
(13)
REAL
where J(cx) (fo(c=)/00 is the Jacobian. Since [A.] < K for large codebooks, this update lies in a low-dimensional subspace defined by (J(C=)REEA Inactive codes are only indirectly updated by
Δč₃ ~ J(c)) A0
(14)
which are typically weak (due to small Jaoss-Jacobian interactions) and poorly aligned with the data regions those codes should represent.
Why learning-rate annealing exacerbates collapse. The usage of a code is determined by nearest- neighbor assignments. Usage changes only when a code crosses a Voronoi boundary. Before annealing, relatively large and noisy updates sometimes make such boundary crossings possible. However, once the learning rate decays,
Ad Th
J(:) gi:
Me 0.
(15)
=EA,
the induced changes on inactive codes,
Ad ~ ((cx) A0..
(16)
become extremely small and resemble noise. These updates are insufficient to shift a code across any Voronoi boundary, so its nearest-neighhor assignments remain unchanged and its usage stays zero The optimization therefore converges to degenerate state where only subset of codes is active, leading to codebook collapse.
Why VQBridge remains stable. VQBridge avoids this failure mode by introducing explicit cross- code interaction. The codebook C' E R xxd first patchified, processed by several ViT blocks with global self-attention, and then unpatchified produce Ó = fo(C). Each output code aggregates information from multiple input codes, and gradients propagate through dense attention weights:
0L
Okk Jkygj,
(17)
00
where Ouk denotes attention weights. This mechanism directly transfers strong gradients from frequently used codes to many other codes within and across patches. The resulting updates are dense, globally coupled, and aligned with data geometry. Even under learning-rate annealing, inactive codes receive meaningful gradients, allowing them to re-enter active usage and preventing collapse.
Summary. The shared MLP projector collapses under learning-rate annealing because its pointwise architecture and STE produce inherently low-rank, usage-sparse updates that cannot revive inactive codes once the learning rate becomes small. VQBridge introduces structured cross-code interaction via self-attention, enabling dense and data-aligned gradient flow across the entire codebook. This design keeps all codes trainable and maintains near-complete codebook usage throughout training.
CONCLUSION
We have systematically analyzed the challenges inherent in discrete tokenizers and identified the root causes of unstable training and codebook collapse. Based on these insights, we proposed FVQ with the VQBridge projector, enabling robust, scalable, and high-quality VQ optimization. Our experiments show that high codebook usage and state-of-the-art reconstruction are achievable across diverse settings, and that strong tokenizers are essential for effective autoregressive image generation. Beyond improving current models, our work provides foundation for extending VQ training to larger-scale and potentially unlabeled datasets. We believe these findings will inform future research on autoregressive visual generation and contribute to the development of more capable multimodal learning systems.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
Section 3.2 describes the detailed design and implementation of FVQ. Appendix G provides compre- hensive information on the experimental parameter settings, which can help readers reproduce the results reported in this paper. In addition, all datasets, preprocessing steps, and training protocols clearly documented in the main text and supplementary materials to facilitate reproducibility. To further ensure reproducibility, we will release all source code, models, and configuration files immediately after the review process. These resources will allow other researchers to replicate our experiments and validate our findings.
REFERENCES
Alpha-VI. ALL..-VL..M. Large-dit-imagenet.
http://///////.t.//////////aa/aa tree/f7fel 921.9222333333333111111133330330
ImageNet, 2024.
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yoshua Bengio, Nicholas Lčonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rXiv:1308.3432. 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1.KIv:8809.11000 2018
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrlPS, volume 33, pp. 1877-1901, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman, Maskgit: Masked generative image transformer. In CVPR, pp. 1 11305-11311 11315 2022.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICML, pp. 1691 1703. PMLR, 2020.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, pp. 248-255, 2009.
Jacob Devlin, Ming- Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480,, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 34: 8780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Heigold, S Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2020.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp. 12873-12883, 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Ficet, Mohammad Norouzi, and Tim Salimans, Cascaded diffusion models for high fidelity image generation. JMLR, 23(1):2249-2281, 2022.
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks. In International Conference on Machine Learning, pp. 14096-14113. PML.R. 2023.
11

---

Published as a conference paper at ICLR 2026
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. CVPR, pp. 5967-5976, 2017.
Minguk Kang, Jun-Yan Zhu, Richard Zhang. Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. CVPR, pp. 10124-10134, 2023.
Diederik P Kingma. Auto-encoding variational bayes, arXiv preprint arXiv: 14, 2013.
Diederik Kingma. Adam: method stochastic optimization. arXiv preprint 1416:9141,.6980, 2014.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tusct, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 128(7):1956-19)) 2020.
LAION. Laion-coco 600m. https: //aps:://laion ai ai/bllg//////// laallogglaioncccoooo 2022.
Adrian Lańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alumäe, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2020.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook Shin Han. Autoregressive image generation using residual quantization. In CVPR, pp. 11513-11522, 2022.
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised representation generation method. NeurlPS, 37:1 37:125441- 2-441-1225550 1225666, 2024a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024b.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02743 2022.
Zhuoyan Luo. Fengyuan Shi, Yixiao Ge, Yujiu Yang. Limin Wang. and Ying Shan. meenmmag An open- source project toward democratizing rugo-segres visual generation. arXiv preprint arXiv:2409.0 04410. 2024.
Chuofan Ma, Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502. 2025.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23-40. Springer, 2024.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vg-vae made simple. arXiv preprint arXiv:2309. 15505, 2023.
Ziqi Pang. Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, Yu- Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. arXiv preprint arXiv:2412.01827, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, 4195 4205, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. article, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, pp. 8748-8763. PmLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and 1lya Sutskever. Zero-shot text-to-in text-to-image generation. In Marina Meila and Tong Zhang (eds.), ICML, volume 139, pp. 8821 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NewrIPS, volume 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022a.
12

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. CVPR, pp. 10674-10685, 2022b.
Axel Sauer. Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan large diverse datasets. In SIGGRAPH, pp. 1-10, 2022.
Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang. Ying Shan, and Limin Wang. Taming scalable visual tokenizer for autoregressive image generation. arXiv preprint arXiv2412.00299,, 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02503 2020.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan, Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525 2024.
Keyu Tian, Yi Jiang. Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.022 022255,,,,20, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. NeurIPS, 29, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NewrlPS, volume 30, 2017.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma latent diffusion models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15703-15712 2025.
Jiahui Yu, Xin Li, Jing Yu Koh, Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In ICLR, 2022.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang. Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In ICLR, 2024a.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chich worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406.07555 2024b.
Chich Chen. An image is
Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantization for tokenized image synthesis. In CVPR, pp. 18467-18476, 2023.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, pp. 586-595, 2018.
Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In ICCV, pp. 22798-22807, 2023.
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for
high-fidelity image generation. Advances in Neural Information Processing Systems, 15:23412-23425, 2022.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2.006. 00.30000000000...00.00 2023.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vggan to 100,000 with utilization rate of 99%. arXiv preprint arXiv:2406.1 1837, 2024a.
Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Addressing representation collapse in vector quantized models with one lincar layer. arXiv preprint arXiv:241 1.0208,, 2024b.
13

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Image Quantization. To convert image features into discrete tokens, VQ-VAE (Van Den Oord et al., 2017) maps encoded features to the nearest vectors in a learned codebook. VQ-V (Razavi et al., 2019) improves quantization with a hierarchical encoding strategy, and VQGAN (Esser et al., 2021) enhances perceptual quality via adversarial and perceptual losses. However, scaling the codebook size and vector channel often leads to codebook collapse- low usage and poor reconstruction. To mitigate this, VIT-VQGAN (Yu et al., 2022) reduces the vector channel and adds l2 normalization to improve codebook usage. FSQ (Mentzer et al., 2023) and LFQ (Yu et al., 2024a) adopt similar low-channel designs to maintain usage with larger codebook. Reg-VO, VQ (Zhang et al., 2023) introduces prior regularization, while CVQ (Zheng & Vedaldi, 2023) resets underused codebook entries. Yet, these methods still struggle when both size and channel are scaled, limiting reconstruction quality, Recently, (Huh et al., 2023) specifically analyzes the impact of the Buraigh-TThrouge Estimator (Bengio et al., 2013). IBQ (Shi et al., 2025) achieves high usage with large codebook and high channel via index backpropagation. VQGAN-LC (Zhu al., 2024a) initializes codebook with pretrained features and employs linear projector to jointly transform embedding. Sim VQ (Zhu et al., 2024b) further highlights the importance of joint transformation and provides theoretical support for linear mappings. We argue that scaling both the codebook size and vector channel requires strong projector to fully optimize the codebook and ensure effective usage.
Image Generation. PixeCCNN (Van den Oord et al., 2016) and iGPT (Chen al., 2020) generates RGB pixels in scanning order. VQGAN (Esser et al., 2021) improves (Van den Oord et al., 2016; Chen et al., 2020) by using an autoregressive approach in the latent space of VQVAE (Van Den Oord et al., 2017). VQVAE2 (Razavi et al., 2019) and RQ-Transformer (Lee al., 2022) adopt the same approach but with additional scales. MaskGIT (Chang et al., 2022) employs masked prediction similar to BERT (Devlin al., 2018). 8). LlamaGen (Sun al., 2024) demonstrates that autoregressive generation is possible even without visual bias, while VAR (Tian et al., 2024) explores visual autoregressive generation using multi-scale methods. RandAR (Pang et al., 2024) trains transformer in random orders to improve performance. Additionally, diffusion models (Ho et al., 2020; Song et al., 2020; Lipman et al., 2022: Rombach al., 2022a;b; Peebles Xie. 2023) generate images by denoising from noise. MAR (Li et al., 2024b) uses diffusion loss to climinate the vector quantization process.
B LIMITATIONS AND FUTURE WORK
We primarily focus on the VQVAE/VQGAN-baser learning paradigm to validate the effectiveness of our method. The solutions to codebook collapse and unstable training are general and can be further applied to models like UniTok (Ma et al., 2025) for additional improvements. Additionally, we only use LlamaGen as the generator; employing other strategies, such as random order learning and parallel decoding, can further enhance the generative capabilities of AR models.
Pretraining Tokenizer. We plan to train more advanced discrete tokenizers larger-scale datasets, such as Openlmages (Kuznetsova et al., 2020) and LAION-COCE (LAION, 2022), to further enhance the reconstruction capabilities of tokenizers and create better conditions for autoregressive generation.
Unified Tokenizer. Our results demonstrate that the vector channel can be scaled larger dimen- sions. This facilitates alignment with visual representation tokenizers like CLIP (Radford et al., 2021) (dim=512), enabling unified tokenizer accomplish both generative and understanding tasks.
Generator Capacity. Larger codebooks increase the parameter count of the generator's final layer, which can make optimization more challenging. A more balanced approach is to scale up the generator accordingly, Due to limited computational resources, we have not explored substantially larger codebooks and generators, which remains future work.
C ADDITIONAL EXPERIMENTS IN OBSERVATION
In Figure 7, VQ loss tends to stabilize at a lower value as the codebook size increases. Early in training, the loss drops quickly because only few code vectors active. This allows the encoder

---

Published as a conference paper at ICLR 2026
6.75
Method
Codebook Size
usage
rFLD
linear
16k
100%
2.10
liner+AAmaaling
16k
53%
1.52
VQBridge
10k
100%
2.61
VOBrdge+Anmnaaing
16k
100%
1.30
MIP-5
262k
20%
2.06
MLP StAnealnng
262k
22%
1.75
VQBridge
262k
100%
1.40
6850
VQBridge+Annealing
262k
100%
0.95
20244
200
Figure 7: Effect of codebook size training loss
Table 5: Additional observational results.
fit the codebook entries easily. Additional results are reported in Table 5, which demonstrate the necessity of using VQBridge together with annealing.
D DERIVATION OF THE ONE-STEP-BEHING FORMULA
Consider 1-th iteration VQ training with the following components: encoder output ze (4)
Rd.
quantized output 2," (t).
The Vector Quantization commitment loss is defined as:
Lvg
(z.") (2("),21) 2Z!
(18)
The gradient of the commitment loss with respect to the vector code Zg is:
(#)
Zo
Zi
(1)
2!!))
(19)
Ozg
(3)
(t+1)
The update rule for zq
when optimizing with SGD is:
x - dLyg
Ze Zi
(t)
(1+1)
(20)
Ozy (t)
Substituting the gradient:
4
(++1) = 21(1) 17.
(21)
Expanding and rearranging:
(1+1) q
(0)
172 (1) + 7 (1 17)
2g
(2)
+ 17 (()
(22)
From the above, we arrive at Equation The correct update is given by Equation 7, and we derive an executable formula following (Huh et al., 2023). First,
z (t+1) = 2. - -1 aL
(23)
Oz.
Substituting Equation 23 into Equation 7:
(++1)
(1 11) z (t)
+ z 1++1)
(24)
Trask
(1 1/) z
(t)
1
(25)
Oze
8Lask
(1 n! x (1)
+ 2 (t)
-1²
(26)
Oz.-
commitment loss
can be observed that the additional term is multiplied by n When annealing applied, the contribution of this term is negligible, which underscores the importance of learning annealing.
2

---

Published as a conference paper at ICLR 2026
E EFFECT OF
ON TRAINING STABILITY
In the (t + 1)-th forward pass, quantization uses the updated codebook:
x ((+1)) = arg min Iize (t-1) - k
(27)
2q
(t+1) c 1)+
(28)
Comparing with the quantization point used for backpropagation step t: Zzg (1) (t-11)
220) we observe
temporal misalignment: Step t encoder updates using signals around ((t-1); Step forward
quantization centers have changed to C(t), This inconsistency constitutes one-step-behind problem.
Mathematical Analysis. The quantization center shift can be expressed as:
(1+1)
2n
(t) (((-1)
(t-1). z(0)
(29)
Quantization center shift
This decomposes into two components:
(c (t) g(++))
(t-1)
(c
(t-1)
c c((-1))
(30)
Codehook update effect
Cluster switching
By the triangle inequality,
2g (++1)
Zg 10 S |/c 2 || + || |[ -(1r)) 11.
(31)
Bounded Change Analysis. Assume bounded changes:
(0) -c-cct-1 ((--1) S & Vi, (++1) 2. (t) 52.
(32)
Define the assignment margin at time as the difference between the distances to the nearest and second-nearest centers (evaluated w.r.t. {c(-11) ((-1))); J):
I (t) 2nd
(1- 1z. - c
(33)
Using the triangle inequality one obtains that if (()) > 28, then the nearest-center assignment is preserved (i.e. (++1) = (*)); conversely, when m < 25_ cluster switching may occur. In the
no-switching case
||2 (++1) 2g (+) S dc: while in the worst case (arbitrary switching) may only guarantec
(34)
12g (1+1) Zu (4) S de + D Dinax:
(35)
where Dmax maxi, (|((--1) c, ((-1)) || (time-local) upper bound on inter-center distances.
Finally, if the task loss L-Lipschitz w.r.t. Zg. then
(++1)) - Liz,, (9) s L.|2, (t-1) Z. [0)]
(t+11)
(36)
Combining the above yields the following interpretation: when encoder outputs lie away from cluster boundaries (m (+) > 26g) and codebook updates are small (8, small). loss fluctuations induced by one-step-behing behind are controlled by LỘ.; however, near boundaries cluster switching can produce center jumps of order inter-cluster distances, which the Lipschitz property amplifies into large loss fluctuations and thus destabilizes training.
Implication for the Decoder. Since the quantized vector 2, is directly fed into the decoder, fluc- tuations in Zy propagate to the reconstruction stage. In the stable case (no switching, small 6.), the decoder receives smoothly varying inputs and can learn consistent mapping. In contrast, when clus- ter switching occurs, 2g may undergo discontinuous jumps of order Dmax. forcing the decoder to fit inconsistent latent representations across consecutive steps. This misalignment not only destabilizes optimization but also degrades reconstruction quality, as the decoder struggles to reconcile abrupt changes in its input space.
3

---

Published as a conference paper at ICLR 2026
F VECTOR QUANTIZATION FOR IMAGE RECONSTRUCTION
discrete tokenizer contains Encoder, Codebook and Decoder. Given an image E #x the Encoder compresses it into image feature map Z R where H x W and h denote the spatial resolution of the image and feature map, D denotes the channel number feature map. Formally,
Z = Encoder(I).
(37)
The Codebook contains K vectors C fc1, C2,... .... Cy' }, where each vector is D-dimensitional vector. By searching for the most similar vector in the codebook, Ze is mapped a feature Zy. and the indices form the token map T. We denote the search process Quant(-.)) We have
C₁ = Quant(c* cC),
(38)
where x = arg (1..2 1 Cs c2|12. and
Zy xx (2²1}.
(39)
where E (l, E (1,-.,,))) w). The Decoder reconstructs Zn into i. Formally,
i= Decoder(2), (2).
(40)
Typically, compound loss is optimized to train the model. We have
C = free + Leommit Aperehpere AGLG;
(41)
where Lree = I
I | Lcommit ||sg] Z..] Z, + B||sg] [ Zell2. Lpare perceptual loss
like LPIPS (Zhang et al., 2018), LG a discriminative loss like PatchGAN (Isola et al., 2017), B, Xperer êG are loss weights, and sg[-] is stop-gradient operation.
G FURTHER DETAILS ON EXPERIMENTAL CONFIGURATIONS
Table 6: Experimental Configurations on reconstruction and generation. Warmup(0.1) indicates that 10% of the training time used for learning rate warmup.
Config
Reconstruction
Generation
Base Batch Size
128
256
Training Epochs
40 or 120 Adam
300-350
Optimizer
AdamW x 10-4
Base Learning Rate
x 10-4
Learning Rate Schedule
w0rmup(0.1)
warmup(O)
lincar(0..) (bascl((0.))lincar((..)) [baselr(0.3)
(basel((..5))Iimca((0555)
Warmup Learning Rate
0.005 to to 0.01
Linear Learning Rate Optimizer Parameters
1 to 0.01
61 0.9, 81 = 0.95
8 = 0.9, 32 = 0.95 weight decoy 5 x 10
Loss Weight
commit loss 0.25
perceptual loss Aperc = 1.0
discriminative loss Àg 0.5
The training recipe Table follows LlamaGin (Sun al., 2024), where the learning rate is calculated as (batch size base batch size) x base learning rate. The learning rate schedule specifies the proportions of different phases: warmup 10.1) indicates 10% of the training epochs, and [baseIr(0.3)- linear(0.7)) represents the remaining 90%, with 30% maintaining a constant learning rate and 70% applying linear decay. All training is conducted on A100 GPUs, and evaluation follows the LlamaGen protocol. We use Fréchet Inception Distance (FID) the primary metric, sampling 50,000 images and evaluating FID using code from ADM (Dhariwal & Nichol, 2021). Additionally, we report Inception Score (IS), Precision, and Recall.
4

---

Published as a conference paper at ICLR 2026
H FURTHER RESULTS COMPARISON
Additional reconstruction results. We present results for more metrics, including RSNR and SSIM, in Table 7. We report additional comparison results in Table highlighting the superiority of FVQ. As shown in the Table 8, FVQ achieves the best reconstruction performance (rFID=0.88) among all discrete tokenizers, outperforming Open-MAGVIT2 and IBQ, even though they utilize larger architectures.
Table 7: Additional Metries and Comparisons. trained for 120 epochs.
Method
Codebook Size
rFID
RSNR
SSIM
Usage
LlamaGen (Sun al., 2024)
16,384
2.19
20.79
0.675
97.0%
FVQ
16,384
1.30
22.36
0.648
100%
FVQ
16,384
1.17
22.45
0.653
100%
FVQ
262,144
0.95
22.90
0.665
100%
FVQ'
262,144
0.88
22.95
0.672
100%
Table 8: Reconstruction performance of different tokenizers on 256 256 ImageNet 50k validation set. trained on Openlmages (Kuznetsova et al., 2020). t 'trained for 330 epochs. trained for 120 epochs. trained with 4 ResBlocks.
Codehook Size
Vector Channel
Codehook
Method
Tokens
rFTD
LPIPS
Usage
VAE-SD" (Rombach al., 2022a) VAE-MAR (Li al., 2024b)
16
16
0.87
16
1.22
VAVAE (Yao 2025)
16
0.28
VQ-VA (Q-veE-based Methods
VQGAN (Esser et al., 2021)
16 16
1,024
256
7.94
44%
VQGAN (Esser ct al., 2021)
16 16
16,384
256
4.98
0.17
5.9%
VQGAN-SD* (Rombach et al. 2022a)
16 16
16,384
8
5.15
MaskGIT (Chang et al., 2022)
16 16
1,024
256
2.28
LlamaGen (Sun et al., 2024)
16 16
16,384
8
2.19
0.14
97%
LlamaGen (Sun al., 2024)
16 16
16,384
256
9.21
0.29%
VQGAN-LC (Zhu ct al., 2024a)
16 16
16,384
8
3.01
0.13
99%
VQGAN-LC (Zhu ct al., 2024a)
16 16
100,000
2.62
0.12
99%
Open-MAGVIT2 (Lao al., 2024)
16,384
1.58
0.23
100%
Open-MAGVV112t (Luo et al., 2024)
262,144
1.17
0.20
IBQ (Shi et ul., 2025)
16x 16
16,384
256
1.55
0.23
97%
IBQ (Shi al., 2025)
16 16
16,384
256
1.37
0.22
96%
IBQ (Shi al., 2025)
262,144
256
1.00
0.20
84%
FVQ
16,384
256
1.30
0.13
100%
FVQ' FVQ
16 16
16,384
256
1.17
0.13
100%
16 16
262,144
256
0.95
0.12
100%
FVQ'
262,144
256
0.88
0.12
100%
1-D Token Sequence
Titok-L (Yu et al., 2024b)
32
4.096
16
2.21
Titok-B (Yu et al., 2024b)
64
4.096
16
1.70
Titok-S (Yu et al., 2024b)
128
4,096
16
1.71
Multi-Code Representation
RQ-VAFroes
al., 2022)
16,384
256
4.73
RQ-VA RA- (Lee 2022)
16,384
256
3.20
MoVQ (Zheng 2022)
1,024
1.12
VAR (Tia et al,, 2024)
68010atep
4,096
1,00
FVQ RRQ-VEE
8 x x
16,384
256
2.98
FVQ (VAR)
180
16,384
256
0.80

---

Published as a conference paper at ICLR 2026
Additional ablation experiments. We perform addtional ablation experiments to study the ef- fects of warmup and learning annealing. Our results indicate that warmup considerably influences performance, in line with the observations reported in (Huh et al., 2023). In contrast, the choice of learning annealing schedule-linee or cosine-has hase a negligible impact on the outcome. The detailed results are summarized in Table 9. Table 10 reports the computational cost of different projectors, highlighting the efficiency of VQBridge.
Table 10: Estimated Model Computational
Table 9: Additional Ablation Experiments.
Cost (in FLOPs)
Method
Codebook Size
rFID
Codebook Size
Linear
MIP-5
VQBridge
haseline w/ linear
16,384
1.30
16,384
1.07G
5.37G
4,30G
w/o warmup w/ cosine
16,384
1.70
262,144
17.18G
85,90G
55.89G
16,384
1.32
Additional generative results. We report more additional generative results in Table 11. FVQ- XL (775M, 2.07) surpasses VAR-d24 (1.0B, 2.09), RandAR-XXL. (1.4B, 2.15) and L-DiT (3.0B, 2.10; 7.0B, 2.28), further demonstrating the importance of tokenizers and the superiority of AR methods. FVQ can also further enhance the capabilities of other AR models. Due to time and computational constraints, we plan to extend this work the future.
Table 1I: Generative model comparison on class-conditional ImageNet 256 x 256. The metrics include Fréchet Inception Distance (FID), Inception Score (IS), Precision (Pre), and Recall (Rec). trained with a learning rate schedule.
Type
Model
#Para
FID
ISt
Pref
Reef
GAN
BigGAN (Brock al., 2018)
112M
6.95
224.5
0.89
0.38
GAN
GigaGGAN (Kang al., 2023)
569M
3.45
225.5
0.84
0.61
GAN
StyleGan-XL (Sauer al., 2022)
166M
2.30
265.1
0.78
0.53
Diff
ADM (Dhariwal Nichol, 2021)
554M
10.94
101.0
0.69
0.63
Dif.
CDM (Ho al., 2022)
4.88
1.88.7
Diff.
LDM-4-G (Rombach et al., 2022a)
400M
3.60
247.7
Diff.
DIT-L/2 (Peehles & Xie, 2023)
458M
5.02
167.2
0.75
0.57
Diff.
DIT-XL/2 (Peebles & Xie, 2023)
675M
2.27
278.2
0.83
0.57
Diff.
L-DiT-38 (Alpha-V1.. VL.LMM 2024)
3.0R
2.10
304.4
0.82
0.60
Diff.
L-DiT-7B (Alpha-VLLL)) VLLM, 2024)
7.0B
2.28
316.2
0.83
0.58
Diff.
MaskDiT (Zheng al., 2023)
675M
2.28
276.6
0.80
Diff.
SIT (Ma 2024)
675M
2.06
270.3
0.82
0.59
MuskGIT (Chung et al., 2022)
227M
6.18
182.1
0.80
0.51
Mask.
RCG (L. et al., 2024a)
502M
3.49
215.5
MAR
MAR-L (1. et al., 2024b)
479M
1.98
290.3
MAR
MAR-H (Li et al., 2024b)
943M
1.55
303.7
0.81
0.62
VAR
VAR-d16 (Tian et 2024)
310M
3.30
274.4
0.84
0.51
VAR
VAR-d20 (Tian et al., 2024)
600M
2.57
302.6
0.83
0.56
VAR
VAR-d24 (Tian 2024)
1.0B
2.09
321.9
0.82
0.59
VAR
VAR-d30
al., 2024)
2.0B
1.92
323.1
0.82
0.59
AR
VQGAN (Esser et al., 2021)
227M
18.65
80.4
0.78
0.26
AR
VITVQ et al., 2022)
1.7B
4.17
175.1
RQTran. (Lee et 2022)
3.8B
7.55
AR
RandAR-L (Pang
2024)
343M
2.55
288.9
0.81
0.58
AR
RandAR-XL (Pang al., 2024)
775M
2.25
317.8
0.80
0.60
AR
RundAR-XXL (Pang et al., 2024)
1.4B
2.15
321.9
0.79
0.60
AR
LamaGGen-L et al., 2024)
343M
3.81
248.3
0.83
0.52
AR
LlamaGie-XXl (Sum et al,, 2024)
775M
3.39
227.1
0.81
0.54
AR
LlamaGen-XMe (Sun et al., 2024)
14B
3.09
253.6
0.83
0.53
AR
IBQ-B (Shi et al., 2025)
342M
2.88
254.4
0.84
0.51
IBQ-I. (Shi al., 2025)
649M
2.45
267.5
0.52
AR
IBQ-XL (Shi al., 2025)
1.1B
2.14
279.0
0.83
0.56
AR
FVQ-L (LlamuGen-L)
343M
2.39
275.8
0.84
0.56
AR
FVQ-XL (LlamaGen-XL)
775M
2.07
287.0
0.83
0.58

---

Published as a conference paper at ICLR 2026
I FURTHER VISUALIZATION ANALYSIS
Qualitative Reconstruction and Generation Samples. Figure and generation samples in Figure 9-Figure 11.
We present reconstruction samples in
Raw
BA-VQ
Raw
BA-VQ
Figure 8: FVQ-16,384 (rFID=1.17) reconstruction samples.
Brambling g (10)
Macww (85)
Cookatco
Golden Retrever (207)
Figure 9: 256 x 256 FVQ-XL (FID=2.07) samples. Classifier-free guidance scale=1.65.
7

---

Published as a conference paper at ICLR 2026
Eslimo (246)
Tabby (281)
Schoomer (780)
(012)
Figure 10: 256 256 FVQ-XL (FID=2.07) samples. Classifier free guidance scale=1.65.
TOWAO
Bellon (417)
Beacon
Fountain
Figure 11: 256 256 FVQ-XL (FID=2.07) samples. Classifier-fres free guidanee scale=1.65.
J TRAINING COST
We report additional details regarding training resources. The VQBridge module introduces only 3M extra parameters, accounting for approximately 4% of the total model size. This increase minor and does not lead additional computational burden. Furthermore, the overall training time remains effectively unchanged, aside from normal variations in server throughput. These results indicate that VQBridge provides performance gains with negligible training overhead.
8

---

Published as a conference paper at ICLR 2026
Table 12: Training Cost. This table summarizes the parameter overhead and training time of different configurations. VQBridge introduces only small number of additional parameters while keeping the overall training time effectively unchanged, indicating negligible computational overhead.
Model
Module
Total Parameters
Extra Parameters
Training Time
VOGAN (Sun al., 2024)
w/ nothing
76.07M
0M
~2hh
FVQ
w/ linear
14.
0.66M
~27h
FVQ
w/ VQBridge
79.23M
3.15M
~27h
K
LLM USAGE STATEMENT
In the preparation of this paper, large language model (LLM) was only used as general-purosse writing assistant to improve grammar, phrasing, and clarity. The LLM did not contribute to the research ideation, experimental design, analysis, the generation of scientific content. All ideas, results, and conclusions presented in this paper are solely the work of the authors, who take full responsibility for the content.


Reviews:
[{"review_id": "ifT3vBkPN9", "reviewer": "Reviewer_M7kB", "summary": "This paper studies presents a well justified and supported study into vector quantization for image tokenization. Generating discrete tokens from images is a non-trivial, yet crucial task for application in a myriad of modern vision methods. To address the training and scaling of the codebooks as compared to other methods, the author tackle challenges of Straight Through Estimation bias, lagging updates, and sparse updates. They present a method in VQBridge that uses mixing between entires at training, better initial alignment, and a different update method to handle the mentioned challenges. The authors accompany this introduced method and theory with good ablation studies and experiments to motivate the application of the introduced method in different studies, making this a very useful study in the explored topic.", "questions": "1. Expanding on the results in table 3, Is codebook size the only factor that influences the optimal patch size? Have the authors studied, as an example, how the patch size and vector channels interact ? \n2. The smaller codebooks produced by FVQ seem to be provide strong results in reconstruction. Does the same benefit of not needing a larger codebook extend to downstream tasks ? Have the authors been able to study downstream tasks from models using different-size codebooks generated by FVQ ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The presented challenges are each sufficiently well motivated with examples and theory \n2. Each presented challenge is suitably addressed in the building of the suggested VQBridge method \n3. The suggested method is studied in a large domain of suitable of applications, showing good results in the strong suite of experimentation\n4. Comprehensive ablation studies are included across different vector and codebook sizes, as well as the training efficiency is explained - which justify the claims of scalability \n5. Generalization studies are added to show applicability in multi code representation as well \n6. The effects of the use of the tokens produced are also studied in downstream tasks, which motivates the applicability of the method well", "weaknesses": "1. The argument for the proposed method could be made stronger with the inclusion of more details on computational cost. In particular, the added computational cost for training and how the training cost differs between the compared methods since only training epochs is used as an example \n2. The larger codebooks retain the 100% utilization and are still shown to have great performance. Still, given that the patchification process induces sharing between entires, some insight into the uniqueness of the tokens generated and/or the distribution of active tokens could further ground the presented results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CBFt5sREzw", "reviewer": "Reviewer_kziK", "summary": "This paper addresses three key errors in Vector-Quantized Networks (VQNs): the bias introduced by the straight-through estimator (STE), the misalignment caused by one-step-behind updates, and the inefficiency resulting from sparse codebook utilization. These issues become particularly severe when scaling up codebook size and vector dimensions. The authors propose FVQ, a training framework that combines a novel projector module called VQBridge with learning rate annealing. VQBridge is designed as a \"compress-process-recover\" module utilizing ViT blocks to enable global joint optimization of the entire codebook. The experimental results are compelling: the method achieves 100% codebook utilization across various scales, including a massive 262k codebook, and establishes new state-of-the-art reconstruction performance. Furthermore, when integrated into autoregressive image generation pipelines, it significantly enhances generation quality, surpassing strong baselines such as VAR and DiT, thereby unlocking greater potential. The proposed framework also maintains high scalability across other methods.", "questions": "1. Q1: Mathematical Formulation: Could the authors please verify the mathematical formulation in the methodology section? Specifically, what is the precise meaning behind the dimensionality of the matrices described in the framework?\n\n2. Q2: Training Efficiency: Could the authors provide an estimate of the relative increase in total training time (e.g., for the 16k codebook configuration) when incorporating VQBridge compared to the VQGAN baseline? This would help practitioners better assess the trade-off between final performance and computational budget.\n3. Loss Function: The authors mention in the paper that beyond estimation error, \"codebook usage can therefore serve as a complementary metric.\" However, the paper does not explicitly state whether the framework involves a specific loss function for the codebook. Could the authors clarify how the codebook loss function is incorporated?\n\n4. Q3: Design Choices: The scaling law between codebook size `K` and patch size `p` (`p = sqrt(K)/4`) is found empirically. Was there any intuition or theoretical justification for this relationship? Why does compressing the codebook by this specific factor lead to optimal performance?\n5. Q4: Failure Mode Analysis: In the rare case that codebook usage does not reach 100%, what are the observed failure modes? Does the training become unstable, or does performance just plateau at a sub-optimal level?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis is thorough and precise: This paper presents a clear and well-structured examination of the fundamental optimization challenges in VQ training. The explanations are well-supported by comprehensive comparative experiments and mathematical proofs. It is particularly noteworthy how the paper elaborates on the positive role of learning rate annealing in mitigating the one-step-behind update error, which significantly enhances the theoretical credibility.\n\n2. High-Impact Results: The achievement of 100% codebook utilization at scale (up to 262k codes) is a major result . The reported reconstruction (rFID) and generation (FID) metrics are state-of-the-art, convincingly demonstrating the practical benefit of solving the codebook collapse problem.\n3. The design is elegant and practical: The VQBridge module presents a clever and well-designed solution. The \"compress-process-recover\" pipeline not only makes scaling to large codebooks computationally feasible but also significantly enhances the representational capacity of the projection function, effectively overcoming the codebook collapse issue inherent in linear projectors. A key advantage is that the module is employed exclusively during training, introducing zero overhead during inference, which makes it highly appealing for practical deployment.\n4. Extensive Experimentation: The authors perform a wide array of experiments, including ablation studies on the VQBridge components, scalability studies across codebook size, vector channels, and training epochs, and generalization tests on multi-code VQ methods (RQ-VAE, VAR). This leaves little doubt about the robustness and effectiveness of the proposed method.", "weaknesses": "1. Issues with the Mathematical Formulation of the Framework: Mathematical definitions are provided for the framework in the methodology section; however, there appears to be an apparent contradiction between the statement \"For each group, the vectors are compressed into a single vector\" and its corresponding matrix representation. Additionally, there seem to be inconsistencies in the tensor shapes described in the subsequent reshaping operation. The authors need to verify the accuracy of this part of the mathematical representation.\n\n2. Computational Cost of Training: Although the inference cost remains unchanged, the introduction of the ViT-based VQBridge significantly increases the computational cost of training, particularly for large codebooks. The authors do not discuss these newly introduced training costs in the paper.\n3. Ablation on Annealing Schedule: The paper heavily relies on the combination of VQBridge + Learning Annealing. While an ablation in Table 9 shows that the schedule type (cosine/linear) has a negligible impact, it does not ablate the *necessity* of the specific annealing parameters (warmup proportion, decay schedule) used. The robustness of the method to different annealing strategies could be better explored.\n4. Comparison to Closest Contemporaries: While comparisons to VQGAN-LC are provided, a more direct and detailed comparison with other recent methods aiming for high usage with large codebooks (like IBQ) in terms of training efficiency, stability, and final performance on a level playing field would further strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13rZMxtigB", "reviewer": "Reviewer_wFm6", "summary": "This paper addresses the long-standing problems of training instability and codebook collapse in vector-quantized (VQ) networks, especially when scaling up the codebook size. The authors attribute these issues to three fundamental challenges: straight-through estimator (STE) bias, one-step-behind updates, and sparse codebook gradients. To solve this, the paper proposes a training framework called FVQ (FullVQ), featuring a novel, ViT-based projector named VQBridge. VQBridge uses a \"compress-process-recover\" pipeline to jointly optimize the codebook vectors. This method, combined with learning annealing, aims to enable stable and scalable training. Experimental results demonstrate that FVQ achieves 100% codebook usage across various configurations (up to 262k) and attains state-of-the-art image reconstruction performance (rFID 0.88). More importantly, FVQ significantly boosts the performance of downstream autoregressive (AR) image generation models, enabling LlamaGen-XL to outperform strong diffusion models like DiT-XL/2.", "questions": "1. (Re: Weakness 2) Could the authors elaborate on the practical training time (wall-clock) and VRAM overhead of VQBridge compared to baselines (e.g., VQGAN or VQGAN-LC w/ linear projector)? Table 10 gives FLOPS, but what is the practical impact for reproducibility and adoption? \n2. (Re: Weakness 3) The interaction with learning annealing is interesting. In Figure 3, annealing drastically hurts the linear projector's usage but has no effect on VQBridge's. Yet, Table 3 shows annealing is vital for VQBridge's performance (rFID). Could the authors clarify this? Does VQBridge's powerful joint optimization prevent the collapse that annealing would otherwise cause in a weaker projector?\n 3. The VQBridge design is inspired by DiT. Did the authors experiment with other powerful interaction mechanisms? It's surprising that a 5-layer MLP failed so badly (20% usage). Why do the authors believe the ViT block (Patchify -> Process -> Unpatchify) architecture is so much more effective than a deep MLP for this task?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Significance: Addresses a core bottleneck in VQ training—codebook collapse—which is critical for scalable discrete representation learning.\n 2. Effectiveness: The results are outstanding. Achieving 100% usage on a 262k codebook and an rFID of 0.88 is a major engineering and empirical achievement. \n3. Downstream Impact: One of the paper's strongest points is demonstrating how a good tokenizer empowers AR models. By only swapping the tokenizer, LlamaGen-L's FID improved from 3.81 to 2.39, and LlamaGen-XL's from 3.39 to 2.07, surpassing DiT-XL/2 (2.27). This is a very compelling result. \n4. Clarity: The paper is clearly written and well-organized. The problem, challenges, and solution are all well-articulated. 5. Thoroughness: The experimental evaluation is comprehensive, covering detailed architectural ablations, scaling analysis, and generalization tests.", "weaknesses": "1. Limited Novelty: The core idea of using a projector for joint codebook optimization is not entirely new. The paper itself cites prior work that used linear layers for this. The main novelty lies in the design of a much more powerful, DiT-inspired projector (VQBridge). While highly effective, this feels more like an excellent architectural improvement than a fundamental conceptual leap.\n 2. Training Overhead: The VQBridge module introduces significant computational cost during training. Table 10 shows that for a 16k codebook, VQBridge has ~4x the FLOPS of a linear layer, and for 262k, it's ~3.2x. While the paper correctly states it adds no inference cost, this training overhead is a practical limitation that warrants more discussion. \n3. Analysis of Annealing: The paper claims learning annealing is crucial and shows a large performance gain in Table 3. However, the analysis in Figure 3 is confusing: it shows a linear layer's usage drops significantly with annealing, while VQBridge's remains at 100%. This suggests VQBridge is simply robust to annealing, rather than it being a synergistic component. Why does annealing hurt the linear projector's usage so badly? This is not fully explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sqZAHqLEFh", "reviewer": "Reviewer_93aA", "summary": "This paper proposes a new framework for VQ-VAE called VQBridge, which enables robust and scalable codebook optimization. The proposed FVQ achieves 100% codebook usage for various codebook size. Extensive experiments demonstrate that FVQ outperforms prior methods and significantly boosts the generative performance of autoregressive models.", "questions": "- Why does learning rate annealing have such a significant impact on the MLP?\n\n- Does using a large codebook pose challenges for generation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written and easy to follow.\n\n- The proposed method achieves robust and scalable codebook optimization, enabling 100% codebook usage even with very large codebooks.\n\n- The authors conduct various experiments to show the effectiveness of the proposed results, both quantitatively and qualitatively.", "weaknesses": "- While the empirical results are strong, the paper does not include an in-depth theoretical justification for why the proposed method achieves 100% codebook usage while VQGAN-LC cannot, especially considering that the only difference between VQGAN-LC and FVQ is the network architecture (MLP vs. ViT).\n\n- The authors need to provide experiments using larger codebooks (e.g., 65k or 262k) for image generation to demonstrate the necessity of using a large codebook.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7CkBZPAosK", "reviewer": "Reviewer_pncL", "summary": "This paper investigates the training challenges of vector quantization (VQ) in image generation, identifying straight-through estimation bias, one-step-behind updates, and sparse gradients as the root causes of instability and codebook collapse. To address these issues, the authors propose VQBridge, a projector with a “compress–process–recover” pipeline, and combine it with learning rate annealing to form FVQ. Experiments show that FVQ consistently achieves 100% codebook utilization, significantly improves reconstruction and generation quality, and enables autoregressive image models to outperform both existing autoregressive and diffusion models, while maintaining strong scalability and generalization.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clearly identifies and analyzes the three core challenges of VQ training (STE bias, one-step-behind updates, sparse gradients) with both theoretical and empirical support.\n2. Proposes a simple yet effective solution (VQBridge) that ensures 100% codebook utilization, addressing long-standing codebook collapse issues.\n3. Strong experimental results: FVQ outperforms prior tokenizers in both reconstruction and generation quality, and even enables autoregressive models to surpass diffusion models.\n4. Demonstrates robustness and scalability across codebook sizes (up to 262k), vector channels, and training epochs, with consistent improvements.\n5. Shows generalization to multi-code VQ methods (RQ-VAE, VAR), suggesting broader applicability.", "weaknesses": "1. Novelty may be somewhat incremental, as the idea of using projectors (linear layers, joint optimization) has been explored; the main contribution is scaling with a stronger projector (ViT-based).\n2. Experimental comparisons focus mainly on ImageNet and a few baselines; evaluation on more diverse datasets or real-world tasks would strengthen claims.\n3. While FVQ achieves 100% codebook usage, the practical benefit of full utilization versus ~95–99% is not fully clarified.\n4. The method’s additional training cost and complexity (e.g., ViT blocks in VQBridge) are not extensively analyzed.\n5. Limited discussion of limitations or potential failure cases (e.g., extremely large-scale pretraining, cross-modal tokenizers).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)