VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers
Authors: Xianghong Fang
Yuan Yuan
Dehan Kong
Tim G. J. Rudner
Venue: ICLR 2026 Poster (2026)
Keywords: VQ-Transplant
Plug-and-play integration
Computational cost reduction
Pre-trained tokenizers
Primary area: generative models
TL;DR: VQ-Transplant enables efficient plug-and-play integration of new vector quantization techniques into frozen models with minimal computational overhead, achieving high fidelity while reducing training costs by 95%.

Abstract: Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-art VQ-based models requires significant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limitation, we propose VQ-Transplant, a simple framework that enables plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining when modifying the quantization method. To mitigate decoder-quantization mismatch, we introduce a lightweight decoder adaptation strategy (trained for only 5 epochs on ImageNet-1k) to align feature priors with the new quantization space. In our empirical evaluation, we find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95%. VQ-Transplant democratizes quantization research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance.

Published as a conference paper at ICLR 2026
EFFICIENT VQ-MODULE INTEGRATION FOR PRE-TRAINED VISUAL TOKENIZERS
Xianghong Fang
Yuan Yuan?
Dehan Kong" Vijil
Tim G. J. Rudner!
University of Toronto
Boston College
ABSTRACT
Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-ard VQ-based models requires signifi- cant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limi- tation, we propose VQ-Transplant, simple framework that enables plug-and play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retrain- ing when modifying the quantization method. To mitigate decooder-quaniizatoon mismatch, introduce a lightweight decoder adaptation strategy (trained for only epochs on ImageNet- 1ek to align feature priors with the new quantization space. In our empirical evaluation, near state-of-the- -ftte-af-te reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95%. VQ-Transplant democratizes quantiza- tion research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance. Code and models are available at VO-Transplant.
find that VQ-Transplant allows obtaining
INTRODUCTION
Vector Quantization (VQ) a cornerstone of modern discrete visual tokenization frameworks, enabling efficient learning of discrete representations critical for downstream tasks including visual generation (van den Oord et al., 2017; Esser et al., 2021; Lee et al., 2022; Sun et al., 2024; Tian et al., 2024) and vision-language modeling (Ramesh et al., 2021; Bao et al., 2022; Li et al., 2024; Wu et al., 2024). State-of-the-ee VAR (Sun et al., 2024) leverage adversarial training (Goodfellow et al., 2014) to achieve exceptional reconstruction fidelity by aligning synthesized outputs with real-data distributions.
visual tokenizers such as VQGAN (Esser et al., 2021; Sun et al., 2024) and
However, achieving oate-offttheaat fidelity through adversarial training requires substantial com- putational resources to enable training on large-scale datasets like ImageNet (Deng et al., 2009) and OpenImages (Kuznetsova et al., 2018), as detailed Table 1. Moreover, adversarial training is inherently unstable (Isola et al., 2017; Karras et al., 2019), further complicating the process. These challenges severely limit the exploration of novel quantization algorithms, particularly in soouure-connsrainne enviromments where end-to-end training of full mncoder-decoder architectures is computationally infeasible.
This practical constraint raises a critical yet underexplored question:
Can we decouple the development of VQ methods from the computational overhead associated with training tokenizers from scratch?
Current approaches (Esser et al., 2021; Sun al., 2024; Tian al., 2024; Li et al., 2025; Han et al., 2025) treat the VQ module and the encoder-decoder framework as monolithic system, necessitating joint optimization of all components. Such tight integration not only significantly increases computational cost but also impedes the integration of novel quantization techniques into pre-trained models. For instance, replacing a frozen tokenizer's native VQ module with an improved
Corresponding authors: xi xinghong. fang@ma i utoronto. ca, dehan. kongautoronto.cc and tim.

---

Published as a conference paper at ICLR 2026
quantization variant may lead to Recoder-quantizzn distributional mismatch since the decoder's conditioning depends on the original quantization space.
In this paper, we propose VQ- IQ-Transplant, computationally efficient framework enabling plug-and- play replacement of arbitrary VQ algorithms within pre-trained visual tokenizers (c.g., VAR (Tian et al., 2024)), without costly end-to-end retraining from scratch. Our key insight is preserve the frozen encoder-decoder parameters of state-of-the-art tokenizers while replacing their native VQ modules with superior alternatives. To address distributional discrepancies between new quantization spaces and frozen decoders, we propose a lightweight decoder adaptation strategy-requirin only five epochs of training on ImageNet- Ik-to to align the decoder's learned feature priors with the new quantizer's s latent space. This decoupling eliminates the need for prohibitive full-model retraining, allowing cheap and rapid iterations the development of novel VQ algorithms.
We demonstrate the effectiveness of the VQ- Transplant framework by evaluating multiple VQ approaches for visual tokenization tasks. Building upon distributional alignment princi- ples (Fang et al., 2025), we introduce MMD VQ. which leverages maximum mean discrep- ancy to align the distributions of feature and codebook vectors, to improve compatibility with VQ-Transplans For example, when integrating MMD VQ into the pre-trained VAR tokenizer (Tian al., 2024), VQ-Transplant achieves superior reconstruction fidelity (0.81 rFID) while being 21.8x faster than training vanilla VAR (0.92 rFID), as shown in Table 1.
Table I: Comparison of computational cost between different tokenizers.
Tokenizers
Dataect
CIPUS
Training Hours
Speedop
GEN csun al., 2024)
ImageNer
2
Imagellwdeer 2015 2015)
ImageNet
32 x
VAR al. UniTic
A100
2025)
Opealinaees
256
290.9
VO-Trampplant (Ours)
ImtggeNss-11
2 A100
22
Our key contributions are as follows:
1. Our primary contribution is Vr-aaassplant, computationally efficient framework for discrete visual tokenization designed to enable cheap and repid exploration of novel Vector Quantization (VQ) techniques. By enabling seamless plug-and-play integration of new quantization algo- rithms into pre-trained tokenizers, VQ-Transplant obviates resource-intensive retraining, thereby significantly lowering the barrier innovation in VQ research.
2. Our secondary contribution is MMD-VQ, a novel VQ method introduced within this framework. Specifically designed to enable improved compatibility with Toanspspl,,n, MMD VQ leverages maximum mean discrepancy to achieve distributional alignment. This method demonstrates superior reconstruction fidelity compared to the vanilla VAR approach, further validating the effectiveness and versatility of the VQ- (Q-Transplant framework.
RELATED WORK
Visual Tokenizer for Generative Models. The rapid evolution of generative models has catalyzed substantial interest in visual tokenization, a pivotal component that bridges raw visual signals and latent representations for synthesis. Contemporary visual generative models predominantly adhere to two paradigms (Wang et al., 2024): language model-based and diffusion- former harnesses sequence modeling to frame visual generation as next-token prediction (van den Oord et al., 2017; Esser et al., 2021; Yu et al., 2023; Sun et al., 2024; Ma et al., 2025), relying on discrete tokenizers such as VQVAE (van den Oord al., 2017). In contrast, diffusion models (Ho et al., 2020; Song et al., 2021a;b) utilize continuous tokenizers (e.g., VAEs (Kingma & Welling, 2014; Rombach al., 2022; Chen et al., 2025a;b; Xie al., 2025)) encode images into latent distributions.
approaches. The
Adversarial Training in Discrete Visual Tokenizers. Visual tokenizers define the expressivity boundary of generative systems (Esser et al., 2021; Yan et al., 2021; Hong et al., 2023). While early VQVAES (van den Oord et al., 2017) were plagued by over-smoothed reconstructions and suboptimal perceptual quality, modern discrete tokenizers (Esser et al., 2021; Li et al., 2025; Tian et al., 2024) leverage adversarial training techniques, such as PatchGAN (Isola et al., 2017) and StyleGAN (Karras et al., 2019), to enhance reconstruction fidelity by aligning reconstructed outputs with real data distributions. However, such adversarial training is computationally intensive, often requiring exten- sive training on large-scate scale datasets like ImageNet (Deng al., 2009) or Openlmages (Kuznetsova et al., 2018), as detailed in Table 1, and is prone to optimization instability. These challenges hinder researchers exploring novel VQ algorithms, particularly those with limited computational resources.
2

---

Published as a conference paper at ICLR 2026
To address this training burden, introduce VQ-Transplant, computationally efficient framework that enables plug-and-play integration of arbitrary VQ algorithms into pre-trained visual tokenizers, cireumventing the need for costly end-to-end retraining from scratch.
Vector Quantization in Discrete Visual Tokenizers. As the core component in discrete visual tokenizer, VQ acts compressor that discretizes continuous latent features into discrete visual tokens by mapping them to the nearest code vectors within a learnable codebook, utilizing the corresponding code indices as visual tokens. Despite its widespread adoption, vanilla VQ (van den Oord et al., 2017) suffers from critical limitations, including training instability and codebook collapse. Recent efforts(Huh et al., 2023; Razavi et al.. 2019; Zheng & Vedaldi, 2023) have sought to mitigate these issues through enhanced codebook update mechanisms. However, their effectiveness remains heavily dependent on meticulous codebook initialization, thereby limiting their robustness and generalizability (Fang et al., 2025). To address these challenges, Fang et al. (2025) introduced Wasserstein VQ, a theoretically principled approach that reformulates codebook learning as the problem of distribution matching between code and feature vectors. Nevertheless, Wasserstein VQ critically relies on Gaussian distribution assumptions. When feature vectors in real-world applications deviate from Gaussianity, Wasserstein VQ reduces to merely aligning first- and second-order statistics between features and code vectors, failing to achieve effective distribution alignment.
3 PRELIMINARIES: DISCRETE VISUAL TOKENIZERS
discrete visual tokenizer typically consists of three key components: an encoder & a VQ module 2g. and decoder Dep Given input image x E IR H XW x3 the encoder &n first produces set of d-dimensional feature embeddings x, = Ex(x) E RR x xd with a spatial downsampling factor of fx The VQ module then discretizes these continuous features through wearest-neighhor codebook lookup, where each spatial feature 2 E Rdd maps to its closest codebook entry ex:
1 arg ming || ex1
(1)
yielding spatial visual token ri E zt The quantized latent representation z = 2x(2!)) = e is then decoded to reconstruct the image क. (2,()))
While VQVAE (van den Oord et al., 2017) demonstrated the feasibility of discrete visual tokenization, generated images in early work exhibited oversmoothed reconstructions and suboptimal perceptual fidelity. Subsequent improvements (Esser et al., 2021; Li et al., 2025; Tian et al., 2024) have integrated adversarial training (Isola et al., 2017: Karras et al., 2019) and VGG-based perceptual regularization (Simonyan & Zisserman, 2015). The VQGAN framework (Esser et al., 2021) optimizes a composite objective:
[(0, o. (p) = | 21 + 3 (s((zz)) z. 112 + |sg(z.) - xgg Ap + Ag (2)
where sg(-) is the stop- groddiradeent operation. Leer computes feature spauce discrepancies using frozen VGG network (Zhang et al., 2018), and LGAN uses hinge-based adversarial losses (Lim & Ye, 2017). Hyperparameters 8, Xp and Ag balance the losses. Recent VQGAN- -ased methods (Sun al., 2024; Tian et al., 2024; Li et al., 2025) achieve state-of-the-art reconstruction fidelity. However, their reliance on adversarial training incurs substantial computational costs, as shown in the Table
THE VQ-TRANSPLANT FRAMEWORK
4.1 TWO-STAGE TRANAPSSPL
To mitigate the computational overhead of end-to-end retraining, we propose VQ-Transplant, computationally efficient framework enabling plug-and-play replacement of arbitrary VQ modules within pre-trained visual tokenizers. Our method has two parts: (I) VQ module substitution and (2) decoder adaptation, as detailed below.
Stage I: VQ Module Substitution, Given a pretrained discrete visual tokenizer with encoder Eg.,
decoder Dip" and native VQ module Qureitainn we substitute Queiraii with new VQ module Quew while freezing 0* and l²" Let x. Eg. (x) denote the encoder's 's latent embedding, and 2 (Ø) Q (z.) represent the quantized latent from the new VQ module. The training objective for Qnew is:
Lvo(ø) ||sg(z.)
Zy (+) 112 + Lunique (Q
3

---

Published as a conference paper at ICLR 2026
Pretrained Visual Tokenizer
e
2q
Encoder
VQ Module
Decoder
Eg-
Queitain
D
VQ Module Substitution
Decoder Adaptation
Eg--
Ze nEE zn 20(0)
D
x Eq-
Ze Qnow 2q
D
x (1))
Substitute Queataii with 2
Initialize Dep by Dep
Optimize 2ne
Q via Lvglo)
Optimize Do via Deccddrr(p)
Figure 1: Illustration of the VQ- VQ-Trassplant framework. Block represents pretrained visual tokenizer with three key components: an encoder, a decoder, and a native VQ module. Block 2 and denote the VQ module substitution and decoder adaptation stages in the VQ- TQaansssllant framework.
where Lunique is a uniqueness-enforcinng loss in new VQ modules (e.g.. Wasserstein loss for Wasser- stein VQ (Fang et al., 2025)), and y balances the terms. This stage ensures minimization of the quantization eπror while accounting for its inherent constraints.
Stage II: Decoder Adaptation. While Stage might effectively reduces quantization error, the frozen decoder D.' remains suboptimal for reconstructing inputs from the updated quantized representations zy (6) due to mismatch between the decoder and the quantized space. To address this incompatibility, we propose lightweight decoder adaptation scheme that preserves the frozen encoder Egr, and optimized VQ module Q while updating only the decoder parameters. Specifically, we compute quantized features as Zg = Q (Eg. (x)) and reconstruct the image via @(p)) = Dz where is initialized from s²" The decoder is optimized through composite loss:
(Decotee |cx(+p) - =113 + Xp + AGLCAN(())
(4)
where the hyperparameters Ap and Ag balance the terms. We follow Tian et al. (2024), Chen et al. (2025a), and Li et al. (2025) and employ an identical frozen DINO-S (Caron et al., 2021; Oquab al., 2024) diseriminator, which shares similar architecture to StyleGAN (Karras et al., 2020; 2019). To improve discriminator training. we incorporate DiffAug (Zhao et al., 2020), consistency regularization (Zhang et al., 2019), and LeCAM regularization (Tseng et al., 2021) as implemented in Tian et al. (2024). We also discuss another optimization approach in Appendix C, where the encoder, decoder, and VQ module are jointly optimized, instead of optimizing the decoder alone.
4.2 MMD-VQ: DISTRIBUTION- ALLIDDD
ALIGNED VECTOR QUANTIZATION
To enhance compatibility with the VQ-Transplant framework, we propose MMD- novel vec- tor quantization method that leverages Maximum Mean Discrepancy (MMD; Gretton et al., 2012: Sriperumbudur et al., 2009) align the distributions of underlying features and the codebook. The motivation for using MMD is grounded in a well-established theoretical principle: a vector quan- tizer approaches optimality when its codebook distribution closely matches the feature distribution, minimizing quantization eгror while maximizing codebook utilization, as rigorously supported by prior work (Fang et al., 2025; Graf Luschgy, 2000). MMD achieves this alignment through charac- teristic kernels that guarantee universal distribution matching. Unlike previous deeeseen approaches (Fang et al., 2025), MMD-VQ makes no parametric assumptions and robustly aligns feature and codebook distributions even for complex, non-Gaussian data. This flexibility makes MMD-VQ particularly suitable for the "Q-Transplan framework, especially in scenarios where feature distributions deviate from idealized parametric forms, such maulti-modal -eiled, or otherwise non-Grn Gaussian distributions. A more detailed discussion of the motivation and advantages of MMD-VQ is provided in Appendix B.
4

---

Published as a conference paper at ICLR 2026
Specifically, collect feature vectors X {21. 22, ...: 2N (spatial features x!/ and codebook vectors Y = {e1, e2, ..., Cx) The squared MMD distance is defined as:
N
K
K
1
2 k(z;; ej); (5)
AMD (XY))
k(z, 23)
k(s,,) NK
N
K
==
where -(.,-) denotes a characteristic kemnl. Crucially, DMMDD(X,
0 iff Px = Py,
making MMD useful nonparametric divergence metric. We employ a multi- Gaulsian kernel k(xr v) = Σ, exp(- || yI /2o and define Lunique DAMD (X, Y) in Equation (3) to achieve distribution alignment.
EMPIRICAL EVALUATION
We present comprehensive empirical evaluation of (Q-Transplant from three perspectives. First,
we investigate the integration of multi-scale VQ algorithms within VQ-Transplant in Section 5,1.
Second, we evaluate ixed-scale VQ algorithms under the VQ-Transplant framework in Section 5.2. Both studies are conducted using the ImageNet- 1k dataset (Deng et al., 2009). Finally in Section 5.3, we examine the generalization capability of -rQanaaspl across diverse datasets to further illustrate the strengths and potential of the proposed framework.
Experiment Setup. This work investigates the VQ-Transplant framework using a pre-trained VAR tokenizer (Tian et al., 2024) throughout all experiments. During VQ module substitution, we freeze the encoder-decoder parameters the pre-trained model and replace its native multi-scale VQ module with either new multi-scale VQ modules fixed-scale VQ modules. For the latter approach, implement parallel quantization system wherein 32-dimensional feature vectors are partitioned into two 16-dimensional sub-vectors. These sub- vectors undergo independent quantization via separate VQ modules before concatenation to form the final 32-dimensional vectors for decoder input. During decoder adaptation, we freeze parameters for both the encoder and transplanted VQ modules. We implemented the VO-transplant using five distinct quantization algorithms: Vanilla VQ (van den Oord et al., 2017), EMA VQ (Razavi al., 2019), Online VQ (Zheng & Vedaldi, 2023), Wasserstein VQ (Fang et al., 2025), and our proposed MMD VQ, adapted respectively for both multi-scale and fixed-scale configurations in the pretrained VAR tokenizer. Additional implementation details are provided in Appendix A.
To quantify VQ performance, we report the quantization error (E) and codebook utilization (24). For reconstruction quality, we measure: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), Fréchet Inception Distance (r-FID; Heusel et al., 2017), perceptual similarity (LPIPS; Zhang et al., 2018), and reconstruction inception score (r-IS; Salimans al., 2016).
Evaluation Metrics.
Baselines. We compare the proposed MMD VQ within TQ-Transplant framework with set of baselines: DQVAE (Huang et al., 2023a), DF-VQGAN (Ni et al., 2022), DiVAE (Shi et al., 2022), RQVAE (Lee et al., 2022), VQGAN (Esser et al., 2021), VQGAN-FC (Yu et al., 2022), VOGAN- EMA (Razavi et al., 2019), VQGAN-LC (Zhu et al., 2024), Llama GEN (Sun et al., 2024), and VAR (Tian al., 2024), all evaluated the ImageNet-1k dataset.
Main Results. As demonstrated in Table 2, our VQ-Transplant framework equipped with MMD- VQ and MMD-VAR outperform competing baselines in critical reconstruction fidelity metrics, including r-FID and r-IS. This validates the framework's 's capacity to deliver industry-level reconstruction fidelity when integrated with compatible quantization algorithms like MMD VQ and MMD-VAR. Beyond superior reconstruction fidelity, our VQ- TQ-Transplant implementations achieve significant efficiency gains: As quantified in Table 1, MMD VQ and MMD VAR demonstrate 21.8x faster training than standard VAR (Tian et al., 2024) while simultaneously exceeding the reconstruction performance of the original VAR tokenizer. Specifically, MMD VQ achieves 0.86 r-FID and MMD VAR reaches 0.81 F-FID, outperforming the bascline VAR tokenizer's 's 0.92 r-FID.
5.1 INTEGRATION OF MULTI-SCALE VECTOR QUANTIZATION ALGORITHMS
We investigate the integration of five multi-scale VQ algorithms within the VQ-Transplant framework: Vanilla VAR, EMA VAR, Online VAR, Wasserstein VAR, and MMD VAR. As shown in Table 3, when
5

---

Published as a conference paper at ICLR 2026
Table 2: Reconstruction performance on the ImageNet-IK dataset. "FS VQ" indicates fixed-scale quantization
methods, while "MS VQ" corresponds multi-scale quantization methods. T: results cited from VQGAN-
LC (Zhu et al., 2024); ": results cited from Llama GEN (Sun et al., 2024).
Method
VQ Types
Tokens
Codebook Size
21 (t)
r-FID r-st
LPIPS
PSNR
SSIM
DOVAE
VQ
1024
4,08
DIVART
ES VQ
16384
4.07
ROVAE!
FS VQ
256
16384
3,20
RQVAET
FS VQ
512
163884
2.69
ROVAE!
FS VQ
1024
16384
1.83
DF-VQGAN!
FS VQ
256
12288
5.16
DQ-ANTANT
ES VQ
1024
8192
1.38
Llama GEN
FS VQ
16384
97.0%
2.19
20.79
67.5
FS VQ
16384
3.4%
5,96
0.17
23.3
52.4
VQGANT
FS VQ
256
50000
1.1%
5,44
0.17
22.5
52.5
FS VQ
256
100000
0.5%
5.44
0.17
22.3
52.5
FS VQ
256
16384
11,2%
4.29
0.17
22.8
54.5
VQGAN-FCI
FS VQ
256
50000
3.6%
4.96
0.15
23.1
54.7
FS VQ
256
100000
1.9%
4.65
0.15
22.9
55.1
FS VQ
256
16384
83.2%
3,41
0.14
23.5
56.6
VQGAN-EMA
FS VQ
256
50000
40.2%
3,88
0.14
23.2
55.9
FS VQ
256
100000
24.2%
3.46
0.13
23.4
56.2
FS VQ
256
16384
99.9%
3.01
0.13
23.2
56.4
FS VQ
50000
99.9%
2.75
58.4
VQGAN-LC
FS VQ
256
100000
99.9%
2.62
0.12
23.8
58.9
FS VQ
512
16384
99.8%
1.05
191.2
0.115
24.31
63.7
MMD VQ (Ours)
FS VQ
512
32768
99.9%
0.97
194.1
0,110
24.53
64.7
VQ
99.9%
VAR
MS VQ
680
4096
100 %
0.92
198.6
0.100
24.37
63.9
MMD VAR (Ours)
MS VQ
680
4096
100 %
0.91
199.2
0,108
24. 16
63.2
VAR (Ours)
VQ
8192
0.81
0.104
24.37
Table Reconstruction performance multi-scale VQ algorithms on ImageNet- 1K dataset. For each codebook size and cach phase, the best- pest-performing result is highlighted in bold.
Methods
Phase
Tokens
Codebook Size K ((.)
PSNR(f)
SSIMeT>
LPIPS
rFFD(.))
r-S(())
Tokenizer wio
VAR Tokenizer (Tiaa et al., 2024)
680
4096
0.283
100%
24.37
63.9
0.100
0.92
196.6
Vasilla VAR
Substitution
680
4096
0.305
38.2%
23.53
61.2
0.137
1.54
180.3
EMA VAR
Substitution
680
1096
0.321
99.9%
23.34
60.0
0.151
2.28
171.8
1.56
136.9
Subvtitution
MMD VAR
Substitution
680
1096
0.255
100%
24.52
64.4
0.116
1.52
189.4
Vasalla VAR
Substittion
680
$192
0.309
22.9%
23.29
61.0
0.139
1.93
179.3
EMA VAR
Subcitttion
680
$192
0.312
99.8%
23.45
60.5
0.111
2.18
173.6
680
Wasserstein
Subaiittion
680
190.
MMD VAR
Substittion
680
8192
0.234
100%
24.73
65.1
0.111
1.49
190.4
Adaptation
EMA VAR
Adaptation
680
4096
0.321
99.9%
22.90
0.139
1.69
177.4
Online VAR
Adaptition
680
4776
0.276
99.
23.74
61.8
0.117
1.05
193.0
Wasserstein VAR
Adaptation
680
4096
0.255
100%
24. 10
62.9
0.109
0.93
196.9
MMD VAR
Adaptation
680
0.255
100%
24.16
63.2
0,108
199.2
Adaptation
EMA VAR
Adaptation
680
$192
0.312
99.3%
23.02
0.137
L.15
191.9
Online VAR
Adaptation
680
$192
0.269
73.9%
23.87
62.5
0.113
1.00
193.1
Wasserstein VAR
Adaptation
680
8192
0.240
1001%
24.40
64.1
0,104
0.83
198.8
MMD VAR
Adaptation
630
100%
24.37
63.8
0,104
201.0
Figure 2: Comparison of reconstructed ImageNet-Ik Ik images based on MMD VAR: (Top) VQ Module Substitution Stage; (Bottom) Post-Decoder Adaptation Phase.

---

Published as a conference paper at ICLR 2026
Table 4: r-FID progression during decoder adaptation ImageNet-1K K across epochs.
Methods
Tokens
Codebook Size K
Epoch I
Epoch 2
Epoch 3
Epoch
Epoch 5
Wasserstein VAR
680
4096
1.006
0.968
0.957
0.953
0.926
MMD VAR
680
4096
1.023
0.987
0.950
0.937
0.911
Wasserstein VAR
680
8192
0.910
0.903
0.860
0.832
0.833
MMD VAR
680
8192
0.909
0.880
0.890
0.847
0.806
Table 5: Reconstruction performance of multi-scale VQ algorithms on ImageNet-1K K dataset. The suffixes "-a", " "-b", " "-c", and " "-d" correspond to decoder adaptation for 5, 10, 15, and 20 epochs, respectively. For each codebook size, the best-performig result is highlighted in bold.
Methods
Phose
Tokens
Crdebook Size
((.)
24 (1)
PSNR(T)
SSIM())
LPIPS
--1((7)
VAR Tokenizer (Tian ot al., 2024)
680
4096
0.283
100%
24.37
63.9
0,100
0.92
198.6
MMD VAR-4
Adagcarion Adageation
4096
0.255
100%
24.16
0.108
0.91
199.2
VAR-2
8192
24.37
201.0
MMD VAR-h
Adageation
680
4096
0.255
100%
24.11
63.1
0.108
0.87
199.9
MMD VAR-b
Adageation
680
8192
0.234
100%
24.35
63.9
0.103
0:78
201.0
MMD VAR-c
Adapeation
4096
0.255
100%
24.13
63.0
0.108
0.82
198.3
VAR-c
Adagcation
100%
24.39
201.0
MMD VAR-d
Adugcation
680
4096
0.255
100%
24.24
63.3
0.108
0.79
199.1
MMD VAR-J
Adsgcation
680
8192
0.234
100%
21.36
63.7
0.103
0.74
201.0
replacing the native VQ modules in pretrained VAR tokenizer, MMD VAR demonstrates the highest level of compatibility. Wasserstein VAR follows closely, achieving nearly equivalent compatibility by achieving both lower quantization error and 100% codebook utilization, outperforming other methods like Vanilla VAR, EMA VAR, and Online VAR. These results suggest that VQ algorithms based distributional alignment minimize information loss during substitution, thereby enabling superior reconstruction performance compared to conventional approaches.
However, substituting the VQ module introduces misalignment between the features expected by the decoder and those produced by the new quantized latent space. Two key observations support this: First, while MMD VAR achieve lower quantization error than the original VAR tokenizer (0.255 and 0.234 vs. 0.283), their reconstruction metrics (r-FID and r-IS in Table 3) remain inferior to those of the original model (Tian al., 2024). Second, Figure reveals blurring and loss of fighfrreuency detail in reconstructions using the substituted VQ modules. These findings confirm a persistent mismatch between the decoder's learned priors and the altered quantized latent space.
To resolve this misalignment, we implemented lightweight decoder adaptation on ImageNet- 1k, fine-tuning the decoder parameters for just epochs. This adaptation aims to align the decoder's learned priors with the altered quantized space. After adaptation, both Wasserstein VAR and MMD VAR surpass the performance of the original VAR tokenizer on both r-FID and r-IS metrics (Table 3). Additionally, Figure 2 demonstrates, the resulting reconstructions exhibit dramatically improved visual fidelity. Further reconstruction samples comparing performance after decoder adaptation are provided in Figure 8 (Appendix).
Analyses On Adaptation Epochs. Table 4 tracks the progression of r-FII) metrics during decoder adaptation ImageNet- 1K across training epochs. We observe consis- tent r-FID improvements for both MMD VAR and Wasser- stein VAR as the number of adaptation epochs increases. To examine whether MMD VAR could achieve further gains beyond 5 epochs, we extended the adaptation to 20 epochs. As reported Table 5, MMD VAR with codebook sizes of 4096 and 8192 achieves improved r-FID scores of 0.79 and 0.74, respectively, representing a further im- provement over the epoch-5 basclines of 0.91 and 0.81. Figure shows the r-FID curves over 20 epochs. Despite some fluctuations, there is a clear overall downward trend, demonstrating that continued adversarial training effectively converts the reduction in quantization error achieved by MMD VAR in Stage into improved reconstruction performance.
1.6
K = 4096
1.4
K 8192
1.2
1.0
0.8
0.6
16
Adaptation Epochs
Figure 3: r-FID performance as a func- tion of adaptation epochs.
Comparison Between From-scratch Training and VQ-Transplant. We also compare from-scratch training with VQ-Transpla As shown in Table 6, by examining the MMD VAR on ImageNet- 1k, find that even when from-scratch training is run for longer than VQ-Transplant, it still yields
7

---

Published as a conference paper at ICLR 2026
Table 6: Reconstruction performance of multi-scale VQ algorithms ImageNet- K dataset. The suffxxes "-a," "b,," " and "-c" correspond to from-scratch training for 5, 6, and 7 epochs, respectively. For each codebook size, the best-performing result is highlighted in bold.
Methods
Strateyy
Training Hours
Tokens
Codebook Size A
(4(1))
PSNR(1)
SSIMI")
[-HIDiL)
(-1S(t)
MMD VAR
vo-Teunspluut VQ-Tramphan
22
680
419%
0.255
100%
24.16
0.108
0.91
199.2
MMD
0.234
100%
24.37
201.0
MMD
From-sentch training
25
6811
100%
23,63
60.7
0.120
1.40
MMD VAR-a
Fum-sentch tnining
25
680
8193
0.659
100%
23.58
60.6
0.120
1.26
191.0
MMD VAR
From seracch training
30
680
4895
0.768
100%
23.74
61.1
0.120
1.36
191.9
MMD VAR-b
Fom-semnch training
30
680)
8192
0.665
100%
23:70
60.9
0.119
1,29
196.2
MMD VAR-c
From-snnch training
35
680
4096
0.753
100%
23.74
61.1
0.119
1.34
190.4
MMD VAR
From scratch training
35
680
8192
0.662
100%
23.75
60.9
0.119
1,26
191.0
relatively poor reconstruction performance. This outcome is expected, as discrete tokenizers typically require hundreds of epochs to achieve high-quality visual reconstruction when trained from scratch. In contrast, VQ-Transplant offers more favorable trade-off between performance and training time, yielding accurate reconstruction while consuming significantly fewer computational resources.
Joint Optimization of Encoder, Decoder, and VQ in Stage II We also discuss an alternative optimization approach Appendix C, where the encoder, decoder, and VQ module are jointly optimized, instead of optimizing only the decoder. As shown in Table 14, although that joint optimization could cause mode collapse, in practice it is stable, just like the decoder-only approach, and for most VQ methods it further improves reconstruction performance. However, these performance gains come at the cost of increased training time.
might worry
Compatibility of VQ- Tr-Traasplant with Pretrained LDM Tokenizer. To evaluate the compatibility of Vr-Transplant with different tokenizers, we applied it to the pretrained LDM-16 continuous tokenizer (Rombach et al., 2022). As shown in Table 16, VQ-Transplant achieves reasonable performance on LDM- 16. Nevertheless, its adaptability is lower compared to VAR-based models, as shown in Table 3, particularly with respect to r-FID and r-IS metrics. We provide a detailed discussion of this performance gap in Appendix D.
5.2 INTEGRATION OF FIXED-SCALE VECTOR QUANTIZATION ALGORITTMS
We further investigate integrating five fixed-scale VQ algorithms within the VQ-Transplant frame- work: Vanilla VQ. EMA VQ, Online VQ. Wasserstein VQ, and MMD VQ. As shown in Table 7, nearly identical performance patterns emerge to those observed in Table 3, demonstrating consistent transferability of conclusions from multi-scale to fixed-scale VQ algorithms. Three key observations VQ algorithms (e.g.. Wasserstein VQ and MMD VQ) consistently achieve the lowest quantization error across all codebook sizes, demonstrating their superiority in minimizing information loss when replacing VQ modules. (2) Crucially, reduced quantization error does not translate to improved r-FID or r-IS metrics relative to the original VAR tokenizer (Tian et al., 2024), indicating misalignment between the decoder's learned priors and the modified quantized la- tent space. (3) After realigning decoder priors with the altered latent space via decoder adaptation, all models substantially improve reconstruction fidelity, confirming the importance of decode adaptation in the VQ- TQ-Transplant framework. Representative reconstruction samples post-adaptation are shown Figure 9 the Appendix.
follow: (1) Distribution-alignmae
5.3
PROSS-DATARE GENERALIZATION OF VQ-TRANSPLANT
Finally, we evaluate the cross-dataset generalization capacity of the VQ-Transplant framework. While prior experiments demonstrate strong reconstruction fidelity on ImageNet- 1k (Deng et al., 2009), its generalizability across diverse domains remains underexplored. This limitation arises because the original VAR tokenizer was trained on OpenImages (Kuznetsova et al., 201 -h-hre ImageNet- 1k is subset-raising a critical question: Can the framework generalize to datasets structurally distinct from both ImageNet-IK Ik and OpenImages? To answer this, empirically validate VQ- "Q-Transplant' 's cross- dataset performance on three divergent datasets: CelebA-HQ (Karras et al., 2017), FFHQ (Karras al., 2019), and LSUN-Churches (Yu et al., 2015).
As detailed in Tables 8, 9, and 10, Wasserstein VQ and MMD nQ-implemented via the VQ- Transplant framework with fixed-scale quantization crantization-ddmmonstaateee alization, achieving state-of ootee----hh----- the-art reconstruction performance across all three benchmarks. This generalization capacity is visually corroborated by high-fidelity samples in Figures 4, 5, and 6. Most
demonstrate exceptional cross-datase gener-
8

---

Published as a conference paper at ICLR 2026
Table 7: Reconstruction performance of fixed-scale VQ algorithms ImageNet-IK IK dataset. For each codebook size and each phase, the best-performing result is highlighted in bold.
VQ=
K
E(1)
2 (1)
PSNR(()
SSIM(1)
(L)
rBD((
r-S(()
Vanilla VQ
Substitution
512
16384
0,423
0.8%
22.03
53.0
0.244
10.61
104.7
Subsriturion
512
24.71
0.134
Online VQ
Subsütttion
512
16384
0,286
42.0%
24.32
62.7
0.149
2.20
174.1
Wasserstein
Subsitution
0.130
MMD VQ
Substitution
512
16384
0.234
99.8%
24,89
65.4
0.130
1.84
183.7
Vanilla VQ
Subsitution
32768
EMA VQ
Substitution
512
32768
0.223
100%
24.89
65.7
0.129
1881
183.0
Online
Substitution
512
32768
0.279
22.7%
24.43
63.1
0.146
2.13
175.6
Wasserstein VQ
Suhstitttion
512
32768
0,215
99.7%
25.06
66.3
0,126
1.79
184.8
MMD VQ
Substitution
512
32768
0,216
99.9%
25
66.2
0.124
1.73
187.0
Vanilla VQ
Subsitution
512
65536
0.422
0.2%:
22.04
53.1
0.243
10.89
103.8
Substitutonn
Online VQ
Substitution
512
65536
0,280
13,5%
24.42
63.2
0.147
2.28
174.2
Wasserstein VQ
Substittion
512
65536
0,201
99.6%
25.22
66.9
0.121
1,76
186.0
MMD VQ
Subsitution
512
65536
0,201
99.9%
25.24
66.8
0.121
1.69
187.3
Vanilla VQ
Adaptation
512
16384
0.423
00,8%
21.36
51.3
0,208
5.02
118.4
EMA
Adaptation
512
16384
100%
63.2
0.118
Online VQ
Adaptation
512
16384
0.286
42.0%
23.78
60.9
0.132
1.49
178.9
Wasserstein VQ
Adaptation
512
16384
0.231
99.8%
24.36
64.0
0.114
1.04
191.3
MMD VQ
Adaptation
512
16384
0.234
99.8%
24.31
63.7
0.115
1.05
191.2
Vanilla VQ
Adaptation
512
32768
0,422
0.5%
21.22
51.0
0.209
5.11
117.4
EMA VQ
Adaptation
512
32768
0.223
100%
24.24
63.6
0,113
0.99
192.2
Online VQ
Adaptation
512
32768
0.279
22.7%
23.86
61.6
0.129
1,40
181.3
Wasserstei VQ
Adaptation
512
32768
0.215
99.7%
24.37
64.3
0,111
0.98
193.9
MMD VQ
Adaptation
512
32768
0.216
99.9%
24.53
64.7
0.110
0,97
194,1
Vanilla VQ
Adaptation
512
65836
0,422
0.2%%
21.19
50.7
0.209
5.05
118.9
EMA VQ
Adaptation
512
65536
0.217
65.5%
24.36
64.1
0.111
0.99
194.3
Online VQ
Adaptation
512
65536
0.280
13.5%
23.84
61.6
0,130
1.38
182.9
Wasserstein VQ
Adaptation
512
65536
0,201
99.6%
24.68
65.4
0.106
0,92
195.5
MMD VQ
Adaptation
512
66536
0.201
65.0
0.86
Table 8: Reconstruction performance the FFHQ dataset. t: results cited from (Zhu et al., 2024).
VQs
Phase
Tokens
Codebook Size
8(L)
U (1)
PSNR(1) SSIM(1)
LPIPS (L)
r-FID(L)
ROVAE!
Full Training
256
2048
22.9
67.0
0.13
7.04
VQ-WAB
Full Training
256
1024
22.5
66,5
0.12
4,20
MOVAET
Full Training
256
1024
78.2%
4.55
VQGANT
Full Training
256
16384
2.3%
24.4
63.3
0.12
5.25
VQGAN-FCI
Training
256
16384
10.9%
64.6
VQGAN-EMA
Full Training
256
16384
68.2%
25.4
66.1
4.79
VQGAN-LC
Full Training
256
100000
99.5%
26.1
69.4
0.08
3.81
Wasserstein
Substitution
16384
0.153
99.7%
27.59
77.2
MMD VQ
Substitution
512
16384
0.153
99.9%
27.63
77.0
0.075
2.21
Wasserstein VQ
Substitution Substitution
512
32768
0.142
09.7%
27.83
77.7
0.072
2.27
MMD VQ
512
32768
0.142
99.9%
27.88
77.7
0.073
2.09
Wasserstein VQ
Adaptation Adaptation
512
16384
0.153
99.7%
27.25
75.4
0.075
1.81
MMD VQ
512
16384
0.153
99.9%
27.09
74.7
0.075
1.99
Wasserstein VQ
Adaptation Adaptation
512
32768
0.142
99.7%
27.33
75.7
MMD VQ
512
32768
0.142
99.9
27.43
75.5
0.073
1.37
Table Reconstruction performance on the CelebA-HQ dataset. For each codebook size and each phase, the best best-performing result is highlighted in bold.
VQs
Phase
Tokens
Codebook Size K
E(L)
24 (1)
PSNR(1)
SSIM(1)
LPIPS ())
r-FID(L)
Wasserstein VQ
Substitution Substitution
512
16384
0.129
99.5%
27.92
77.6
0.070
3.30
MMD
512
16384
99.6%
28.02
77.5
0.069
2.96
Wasserstein VQ
Aduptation
512
16384
0.129
99.5%
27.45
75.2
0.071
3.02
MMD VQ
Adaptation
512
16384
0.129
99.6%
27.71
75.7
0.071
2.60
Table 10: Reconstruction performance the Churches dataset. For each codebook size and each phase, the best rest-performing result highlighted in bold.
VQs
Phase
Tokens
Codebook Size K
2(3)
(((11)
PSNR(")
SSIM(1)
LPIPS (J)
r-FIX(])
Wasserstein VQ
Substiution
16384
23.35
2.76
MMD VQ
Substitution
512
16384
0.204
99.7%
23.34
64,2
0.141
2.72
Wasserstein VQ
Adaptation
S12
16384
0.204
99.6%
22.63
62.3
0.122
1.79
MMD VQ
Aduptation
99.7%
22.51
0.124
1.87

---

Published as a conference paper at ICLR 2026
Figure 4: Reconstructed FFHQ Samples. (Top) Original inputs; (Middle) Wasserstein VQ reconstruc-
tion; (Bottom) MMD VQ reconstruction. All images are 256 256 resolution.
Vrssckk a 50
25
Figure 5: Reconstructed CelebA-HQ Samples. (Top) Original inputs; (Middle) Wasserstein VQ
reconstruction; (Bottom) MMD VQ reconstruction. All images are 256 x 256 resolution.
Figure 6: Reconstructed I. WUU--UNNCchhie Samples. (Top) Original inputs; (Middle) Wasserstein VQ recon- struction; (Bottom) MMD VQ reconstruction. All images 256 256 resolution.
notably on FFHQ (Table 8), Wasserstein VQ achieves a record r-FID of 1.21, significantly outperform- ing all baselines including RQVAE (Lee et al., 2022), VQGAN (Esser et al., 2021). VQGAN-FC (Yu et al., 2022), VQGAN-EMM (Razavi et al., 2019), VQ-WAE (Vuong et al., 2023), MQVAE (Huang et al., 2023b), and VQGAN-LC (Zhu al., 2024).
CONCLUSION
In this paper, we proposed VQ-Transplant, computationally efficient framework that enables rapid plug-and-play integration of novel VQ algorithms into pre-trained visual aokenizers-elimiattiin costly retraining requirements- as well MMD-VQ. quantization method that uses maximum mean discrepancy to force distributional alignment and ensure compatibility with VQ-Transplant. We hope that this work will help democratize quantization research by enabling resource-efficiens integration of novel VQ techniques while matching industry-level reconstruction performance.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
To ensure full reproducibility of our results, the following resources are included in the supplemental materials: (1) complete training and evaluation souree code, (2) execution scripts for all experiments, (3) comprehensive training logs capturing model dynamics, and (4) final model outputs and evaluation artifacts. To further support the research community, all resources cecuurces-includin pre-trained model weights, detailed documentation, and configuration fiel-will be publicly released on GitHub. This release will enable independent verification our findings and facilitate future research.
REFERENCES
Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Hao Chen, Ze Wang, Xiang Li, Ximeng Sun, Fangyi Chen, Jiang Liu, Jindong Wang, Bhiksha Raj. Zicheng Liu, and Emad Barsoum. Softvq-vae: Efficient 1-dimensional continuous tokenizer. In CVPR, 2025a.
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang. Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. In ICLR, 2025b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Xianghong Fang, Litao Guo, Hengchao Chen, Yuxuan Zhang, XiaofanXia, Dingjie Song, Yexin Liu, Hao Wang, Harry Yang, Yuan Yuan, and Qiang Sun. Enhancing vector quantization with distributional matching: A theoretical and empirical study, ArXiv, 2025.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. In NeurlPS, 2014.
Siegfried Graf and Harald Luschgy. Foundations of quantization for probability distributions. Springer Science & Business Media, 2000.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Scholkopf, and Alex Smola. A kernel two-sample test. JMLR, 2012.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In CVPR. 2025.
Martin Heusel, Hubert Ramsauer. Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining text-to-video generation via transformers. In ICLR, 2023.
Mengqi Huang. Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. CVPR, 2023a.
Mengqi Huang, Zhendong Mao, Quang Wang, and Yongdong Zhang. Not all image regions matter: Masked vector quantization for autoregressive image generation. In CVPR, 2023b.
11

---

Published as a conference paper at ICLR 2026
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- through estimator: Overcoming optimization challenges in vector quantized networks. In ICML, 2023.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ArXiv, 2017.
Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tusst, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. IJCV, 2018.
Doyup Lee, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. ArXiv, 2024.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. In ICLR, 2025.
Jae Hyun Lim and J. C. Ye. Geometric gan. ArXiv, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding. ArXiv, 2025.
Minheng Ni, Chenfei Wu, Haoyang Huang, Daxin Jiang, Wangmeng Zuo. and Nan Duan. Nüwa-lip: Laaguage-guidee image inpainting with defect-free vggan. In CVPR, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Q. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. TMLR, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurIPS, 2019.
Robin Rombach, A. Blattmann. Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016.
Jie Shi, Chenfei Wu, Jian Liang, Xiang Liu, and Nan Duan. Divae: Photorealistic images synthesis with denoising diffusion decoder. ArXiv, 2022.
12

---

Published as a conference paper at ICLR 2026
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In JCLR, 2013.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021a.
Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Scholkopf, and Gert R. G. Lanckriet. Hilbert space embeddings and metrics on probability measures. JMLR, 2009.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. ArXiv, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling:
Scalable image generation via next-scale prediction. In NeurIPS, 2024.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In CVPR, 2021.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.
Tung-Long Vuong, Trung-Nghia Le, He Zhao, Chuanxia Zheng, Mehrtash Harandi, Jianfei Cai, and Dinh Q. Phung. Vector quantized wasserstein auto-encoder. In ICMI,, 2023.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng. Zuxuan Wu, and Yu- Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. In NeurlPS, 2024.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation. ArXiv, 2024.
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang. Muyang Li, Ligeng Zhu. Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. In ICLR, 2025.
Wilson Yan, Yunzhi Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. ArXiv, 2021.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans the loop. ArXiv, 2015.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. In ICLR, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, -iunn Yang. Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer. CVPR, 2023.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. ArXiv, 2019.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In NewrIPS, 2020.
Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In ICCV, 2023.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vggan to 100,000 with utilization rate of 99%. ArXiv, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXPERIMENTAL DETAILS
Data Augmentation. For all four datasets Caaaassts-mmaeeeeee.. (Deng et al., 2009). FFHQ (Karras et al., 2019), CelebA-HQ (Karras et al., 2017), and LSUN-Churches (Yu al., 2015) follow Llama Gen (Sun et al., 2024) by applying iterative box downsampling to resize all images to a 256x256 resolution.
Ginoder-Decoderr Architecture. For experiments within the VQ-Transplant framework, we maintain the identical encoder-decoder architecture used in the VAR tokenizer (Tian al., 2024). The encoder-a U-Net (Ronneberger et al., 20l 5o-ddwosampllls input images by a factor of 16, resulting latent features with 16 16 spatial resolution.
Training Details. All models were trained on two NVIDIA HI00 GPUs using the Adam W opti- mizer (Loshchilov & Hutter, 2019) with 31 0.9 and 81 0.95. During VQ module substitution, we used an initial learning rate of 10-1 with linear decay to 10-6 For decoder adaptation, the learn- ing rate remained constant 10 The number training epochs for each experiment configuration is shown in Table 11. The batch size for all experiments was 32 per GPU.
Table 11:
Training epochs across various dataset.
Datasets
ImageNet-Ik
FFHQ
CelebA-HQ
SSUN-Churches
Substitution Epochs
30
30
20
Adaptation Epochs
30
30
20
Loss Weight. For all experiments, Ap fixed 1. In multi-scale quantization experiments, Àg 0.5, while in fixed-scale quantization experiments, Ào 0.4. We set 7 0.2 for configurations employing Wasserstein distance (Wasserstein VAR and VQ) and y = 0.5 for for configurations using MMD distance (MMD VAR and VQ).
B THE MOTIVATION OF MMD VQ: FROM WASSERSTEIN DISTANCE TO MMD DISTANCE
The motivation for using Maximum Mean Discrepancy (MMD) arises from a well-establiseed theoretical understanding of vector quantization (VQ): VQ becomes near-optimal when the codebook distribution matches the underlying feature distribution, yielding both minimal quantization error and maximal codebook utilization. This principle is rigorously supported in prior work (Fang et al., 2025; Graf & Luschgy, 2000). Motivated by this insight, we aim to approximate the optimality condition during training by explicitly encouraging alignment between the encoder feature distribution and the learned codebook distribution. MMD provides an effective mechanism for this alignment through its use of characteristic kernels that guarantee universal distribution matching.
To motivate the choice of MMD, we first situate in relation the Wasserstein distance (WD), metric used in prior VQ research for distribution alignment (Fang et al., 2025). The Wasserstein distance provides geometric notion of discrepancy by measuring the minimal cost required to transport mass from one distribution another. Formally, the p- Wasserstein distance is defined as:
Wasserstein Distance: Definition and Context.
W W,PP,, = inf ((r,))y [||== 1/P (P.I((..PP))
(6)
where II.P,. Py) denotes the set of all couplings whose marginals are P, and Py. Intuitively, Wasserstein distance also known as the earth-mover distance measures the minimum amount of "work"" needed to transform P, into P,. The case = 2 corresponds to the quadratic Wasserstein distance.
14

---

Published as a conference paper at ICLR 2026
Table 12:
Quantization under varying levels of non-Gaussianity.
Methods
0.0
<= 0.5
1.0
(= 1.5
2
<
(= 3.0
(= 3.5
< 4.0
Wasserstein VQ
0.976
1.099
1.177
1.252
1.318
1.373
1.420
1.462
1,502
MMD VQ
0.968
1.088
1.142
1.155
1.171
1.186
1.198
1.217
1.240
Table 13: Codehook utilization under varying levels of non-Gaussianity.
Methods
<=
0.0
<= 0.5
< <=1.0
(
1.5
( (=2.0 2.0
= 2.5
(= 3.0
{= 3.5
( 4.0
Wasserstein VQ
99.9%
99.8%
97.0%
78.2%
62.7%
52.1% 88.9%
44.8% 85.7%
39.2%
34.8%
MMD VQ
99.9%
99.9%
99.8%
97.0%
92.5%
81.2%
75.6%
Log Qctriatee sme
Eng Coiriiatee tiee
Co0AA Intlorve: (5)
E 5
Tmining Skepu 1000)
Tranina Supp (k1000)
Tainz Steza (x1000)
Traning Slope (K1000)
(a) MMD VQ
(b) Wasserstein VQ
(c) MMD VQ
(d) Wasserstein VQ
Figure 7: Comparison between MMD-VQ and Wasserstein-Ve VQ on non-Gaussian data.
I. Limitation of Wasserstein Distance in VQ. A well-known practical limitation of Wasserstein distance is that it is computationally intractable when applied directly during neural network training. To address this, Wasserstein VQ (Fang al., 2025) assumes that both the feature distribution and the codebook distribution follow Gaussian forms,
Fg = N(n,, E,), = Nu2, E2),
Under this Gaussianity assumption, the intractable optimization in Equation 6 reduces a differen- tiable closed-fone form expression involving only means and covariances:
W2(N( (u2,, 2 Il r2lli + [1 + Σn -2==+
However, this simplification has an important implication: Wasserstein VQ aligns only first- and second-order moments of the distributions. While sufficient when encoder features are approx- imately Gaussian, this restriction becomes limiting for the multi-modal, highly Gaussian,, or heavy-tailed feature distributions commonly encountered in deep visual representations.
Advantage of MMD-VQ. In contrast, MMD leverages reproducing kernel Hilbert space (RKHS) embeddings with characteristic kernels (c.g., RBF), which guarantee that minimizing MMD corre- sponds to matching all moments of the distributions. This yields a substantially more expressive alignment mechanism and makes MMD-VE VQ inherently robust in scenarios where Gaussian assump- tions do not hold.
Observed Similar Empirical Performance In standard benchmarks (ImageNet, FFHQ, CelebA- HQ. LSUN-Churches), aucoder-producee latent features are typically approximately Gaussian, likely due to the aggregation of many weakly dependent components in deep representations, phenomenon broadly consistent with the Central Limit Theorem (CLT). In such cases, aligning first and second moments is sufficient, and the advantage of MMD-VQ is less pronounced.
Analyses on Non-Gaussian Data. To investigate the practical impact of the moment-matching limitations of Wasserstein VQ, conduct controlled experiments synthetic latent feature dis- tributions with varying degrees of non-Gaussianity. Specifically, we construct a bimodal mixture distribution:
Pg = 0.5 - (((,, 0.5 -
where ( E {0.0, 4.0} controls the separation between the two modes. When Ç = 0, the distribution reduces standard Gaussian; increases, it becomes progressively multi-modal and strongly non- Gaussian.
The codebook distribution P. is initialized standard Gaussian. We set the codebook size to 16,384 with code vectors of dimension and train both Wasserstein- VQ and MMD-VQ for 10,000 steps. At each step, we sample 20,000 feature vectors from Pg. and record the quantization error and
15

---

Published as a conference paper at ICLR 2026
Table 14: Reconstruction performance of multi-scale VQ algorithms on ImageNet-II I datasct. For cach training
strategy and codebook size, the best-perforning result is highlighted hold.
Methods
Training Stralegy
Tokens
Codebook Size K
E(U)
(4)
PSNR(1)
SSIM(1)
LPIPS
>FID(
r-1S(1)
Vanilla VAR
Decoder-nnly
680
4096
0.305
38.2%
23.22
60.1
0.126
1.25
185.6
EMA VAR
Decoder-only
680
4096
0.321
99.9%
22.90
59.1
0.139
1.69
177.4
Online VAR
Decoder-only
4096
0.276
99.0%
0.117
1.05
193.0
Wassersicin VAR
Decoder-nnly
680
1096
0.255
100% %
24,10
62.9
0.109
0.93
196.9
MMD VAR
Decoder-onde only
680
4096
0.255
100%
24.16
63.2
0.108
0.91
199.2
Vanilla
Decoder-only
0.128
1.30
EMA VAR
Decoder-anly
680
8192
0.312
99.8%
23.02
59.3
0.137
1.15
191
Online VAR
Decodr-onll
680
8192
0.269
73.9%
23.87
62.5
0,113
1,00
193
Wasserstein
Decodkr-only
100%
MMD VAR
Decoder-anly
680
8192
0:234
100%
24.37
63.8
0,104
0.81
201.0
Vanilla VAR
Joint Optimization
680
4096
0.403
26.4%
23.64
60.5
0,118
1.07
191.7
EMA VAR
Joint Optimization
680
1096
1.512
99.9%
22.36
55,5
0.154
1.94
172.5
Online
Joint Optimization
4096
100%
Wasserstein VAR
Joint Optimization
680
4096
0.273
100%
24.25
63.4
0,107
0.87
198. 7
MMD VAR
Joint Optimization
680
4096
0,264
100%
24.35
63.6
0,106
0.87
199.5
Vanilla VAR
Joint Optimization
680
8192
0.416
12.1%
23.59
60.4
0.119
1.09
190.7
EMA VAR
Joint Optimization
680
8192
1.384
99. 9%%
22.31
55.4
0.152
1.95
169.2
Online VAR
Joint Optimization
8192
0.262
85,4%
24.09
62.7
0,109
0.92
197.5
Wasserstein VAR
Joint Optimization
680
8192
0.229
100%
24.48
64.2
0.102
0.81
201.7
MMD VAR
Joint Optimization
680
8192
0.227
100%
24.51
64.4
0.102
0.79
201.0
codebook utilization every 100 steps. The evolution of these metrics throughout training is shown in Figure 7.
We evaluate the performance of both methods across different levels of non- -on-Gaussi,n,ty, as summa- rized in Tables and 13. For small Ç. where the latent distribution remains approximately Gaussian, Wasserstein- VQ and MMD- VQ perform similarly, consistent with the observation that matching only the first two moments is sufficient in this regime. As ( increases and the distribution becomes more strongly bimodal, the difference between the two methods becomes pronounced. Wasserstein- VQ experiences significant degradation in both quantization error and codebook utilization, reflecting its inability to capture higher-order structures beyond the first two moments. In contrast, MMD- VQ maintains robust performance, leveraging characteristic kernels to align all moments of the distributions.
These experiments empirically confirm the theoretical expectations discussed earlier: MMD-VQ provides a clear advantage whenever the latent feature distribution deviates from Gaussianity, while Wasserstein- VQ remains competitive only when the Gaussian assumption approximately holds. Therefore, MMD-VQ demonstrates superior generalization capability: in future research scenarios where the latent feature distribution deviates from Gaussianity, MMD-VQ is expected to provide a more robust and effective alignment, offering clear advantages over Wasserstein VVQ.
C STAGE ALTERNATIVE: AN JOINT OPTIMIZATION OF ENCODER, DECODDR, AND VQ
In the original Stage II in section 4.1, only the decoder D,p is updated while keeping the pretrained encoder Eg. and newly trained VQ module 2 frozen. This approach addresses the mismatch between the updated quantized latent space and the frozen decoder, but it does not allow the encoder to adapt to the new quantization or jointly refine the reconstruction capability.
As alternative, also introduce joint optimization scheme in Stage II, where the encoder, decoder, and VQ module are updated simultaneously. Let ze = E (z) denote the encoder's latent embedding, and 2g = 2, (ze) denote the quantized latent from the VQ module. The decoder reconstructs the input as T = VQ reconstruction loss and the decoder reconstruction loss:
The overall joint optimization objective integrates both the
Lroine (0, o. p) = [sg(2.) (z) - 21 + Bl - sg(( sg(z,))l2 + Cunique (
+ || ==12 + Xp dcLEAN,
where Lunique enforces codebook uniqueness (e.g., Wasserstein loss for Wasserstein VQ (Fang et al., 2025) MMD loss for MMD VQ). and Lrer and LGAN correspond to perceptual and adversarial losses that promote high-quality reconstruction. The parameter 3 is fixed to 0.25, while 7, Ap, and AG are hyperparameters balancing the respective terms, as detailed in Appendix A.
16

---

Published as a conference paper at ICLR 2026
Table 15: Training time comparison between decoder-only and joint optimization strategies.
Strategies
Stage Epuchs
Hours Per Epoch
Stage II Epochs
Hours Per Epuch
Total Hours
Decoder-Only
2.25
5
3.5
22 29.5
Joint Optimization
2.25
5
In this setup, all three components eerponents--enodder encoder Eg. decoder g. and VQ module 2s. are updated jointly. To initialize the training, we load all parameters from Stage ensuring that the encoder, decoder, and VQ module start from the previously optimized representations. Joint optimization enables the encoder to adapt the updated quantized space, allows the VQ module to refine the codebook representations, and improves the decoder's ability to reconstruct images accurately from the newly quantized latent features. For adversarial training, we follow prior works (Tian al., 2024; Chen et al., 2025a; Li et al., 2025) and employ a frozen DINO-S (Caron et al., 2021; Oquab et al., 2024) discriminator with StyleGAN-lik architecture (Karras al., 2020; 2019), augmented with DiffAug (Zhao et al., 2020), consistency regularization (Zhang et al., 2019), and LeCAM regularization (Tseng et al., 2021).
We conduct experiments on ImageNet-IK to compare the decoder-only and joint optimization strategies, as summarized in Table 14. Joint optimization maintains full codebook utilization (100%) for MMD-VAR VAR, and thus does not experience the mode collapse that the reviewer had anticipated, while providing slight improvements across most metrics, with the exception of EMA- However, as shown in Table 15, this approach entails additional training time. Thus, while joint optimization offers slightly stronger performance, we chose decoder-only adaptation in the main submission to emphasize efficiency, core design goal of VQ-Transplant.
D COMPATIBILITY OF VQ-TRANSPLANT WITH PRETRAINED LDM TOKENIZER
To evaluate the compatibility of VQ-Transplant with different tokenizers, we applied it to the pretrained LDM-16 continuous tokenizer (Rombach et al., 2022). As shown in Table 16, VQ- Transplant achieves reasonable performance on LDM-16. Nevertheless, its adaptability is lower compared VAR-based models, as compared in Table 3, particularly in terms of r-FID and r-IS metrics.
We identify two primary factors that likely contribute to this performance gap:
-ecoerr Model capacity differences: The VAR tokenizer employs a larger encoder decoder archi- tecture (104M parameters) compared to LDM-16 (68M), providing stronger reconstruction capabilities and greater flexibility during adaptation.
The VAR decoder is pretrained on quantized features 2g- making it inherently robust to the perturbations introduced by new VQ modules. In contrast, the L.DM decoder is trained exclusively on continuous features ze, which limits its ability to adapt to discrete VQ representations. As result, VAR-based VQ-Transplant achieves strong performance with only a few adaptation epochs, whereas L.M variants may require more extensive training.
Decoder adaptation behavior:
These observations indicate that while VQ, Transplant is generally compatible with different tokeniz- ers, the intrinsic properties of the base model as incoder-decoder capacity and the nature of pretrained features, play a critical role in determining adaptation efficiency.
17

---

Published as a conference paper at ICLR 2026
Table 16: Reconstruction performance of multi-scale VQ algorithms on ImageNet-IK IK dataset based on LDM-16 Tokenizer. For each codebook size and each phase, the best-performing result is highlighted in bold
Methods
Phose
Tokens
Codebook Size K
E(U)
PSNR(h
SSIM(1)
LPIPS
piSSh)
et
Vanilla VAR
Substiration
690
4096
0.424
97.0%%
22.89
55.5
0.165
9.74
120.7
EMA VAR
Substitution
680
4396
0.367
100%
23.24
57.0
0.153
7.71
134,4
Online VAR
Substitution
680
4096
0.299
100%
23.65
58.7
0.138
5.62
150.3
Wassertten
Subatitution
0.278
153.5
Substitution
4096
0.278
1001
Vanilla VAR
Substitution
680
8192
0,418
87. 5%
22. 86
55.4
0.166
9.83
120.4
FMA VAR
Substirution
680
8192
0.333
100%
23.40
$7.8
0. 0.146
6,63
142.2
Online VAR
Substrution
680
8192
0.283
94.7%
23.76
59.1
5.14
153.4
Substitution
8192
0.252
100%
189.8
MMD VAR
Substiruion
680
8192
0.254
100%
23.80
39.7
0.131
4.59
158.6
Adapeation
136.7
EMA VAR
Adaptation
690
4096
0.367
100%
22.88
57.0
0.147
4.11
144.9
Online VAR
Adaptation
690
4096
0.299
100%
23:45
59.1
0.134
3.18
158.8
Wasserstein VAR
Adaptation
680
4096
0.278
100%%
23.09
59.5
0.130
2.87
161.4
MMD
Adaptation
Vanilla VAR
Adaptation
8192
0,418
22.58
0.157
132.6
FMA VAR
Adaptation
680
8192
0.333
1001%
23.26
58.4
0.139
3.59
151.6
Online VAR
Adapaation
680
8192
0.283
94.77%
23.55
.9.66
0.130
2.98
139.9
Wasserstein VAR
Adaptation
680
8192
0.252
100%
23.59
2.58
166.9
Adaptation
Figure 8: Visualization of reconstructed ImageNct- 1k images. Top row: original 256 x 256 input images. Subsequent rows (top to bottom): reconstructions from the VAR tokenizer (Tian et al., 2024) and VQ-Transplant- trained Vamilla VAR, EMA VAR, Online VAR, Wasserstein VAR, and MMD VAR models.
18

---

Published as a conference paper at ICLR 2026
Figure 9: Visualization of reconstructed ImageNet- Ik images. Top row: original 256 256 input images. Subsequent rows (top to bottom): reconstructions from KQ-Transplant--rrill trained Vanilla VQ, EMA VQ. Online VQ, Wasserstein VQ. and MMD VQ models.
19


Reviews:
[{"review_id": "cLiszAI9yN", "reviewer": "Reviewer_DdHb", "summary": "This paper addresses the high computational cost of training quantization modules for state-of-the-art VQ-based visual tokenizers by proposing VQ-Transplant, a framework enabling plug-and-play replacement of VQ modules in pre-trained tokenizers without costly end-to-end retraining. it also introduces MMD-VQ, a novel VQ algorithm using Maximum Mean Discrepancy to align feature and codebook distributions, which eliminate Gaussian data assumption, for better compatibility with VQ-Transplant. VQ-Transplant achieves near state-of-the-art reconstruction fidelity on models like VAR, reduces training cost, and shows strong cross-dataset generalization on FFHQ, CelebA-HQ, and LSUN-Churches.", "questions": "1. In most experiments, Wasserstein VQ and MMD VQ exhibit quite comparable performance. Given this, what are the practical advantages of MMD VQ over the former? Or in another way, can you prove that MMD-VQ does provide a better performance on non-gaussian data than Wasserstein VQ?\n2.  Since VQ-Transplant and MMD VQ are two key contributions of this paper, consideration should be given to dividing these two components into separate subsections in Chapter 4. Although you claimed MMD-VQ is a method introduced within VQ-Transplant, MMD-VQ can be implemented without VQ-Transplant. Describing these two contributions in a more separate way can help understand the subsequent experimental results. \n3. In line 340 there seems to be a mistake. \"Table 3 tracks the progression of r-FID metrics throughout decoder adaptation on ImageNet-1K across training epochs.\", \"table 3\" should be \"table 4\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. VQ-Transplant gives efficiently an admirable performance with various VQ method. In Table 1 demonstrate a comparison over training cost among different work, which shows a large amount of time saving. \n2. MMD-VQ eliminate gaussian data assumption, which shows a certain of potential.\n3. MMD-VQ shows a competable performance as Wasserstein VQ and shows an ability of generalization to other domain.", "weaknesses": "1. VQ-Transplant is only experimentally valid on VAR. The framework is exclusively validated on the VAR pre-trained tokenizer, with no tests on other mainstream visual tokenizers (e.g., VQGAN, LDM) to verify its compatibility across architectures.\n2. While the VQ transplant framework with frozen encoder weights is designed to enable the plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers, it still lacks a full-scale training experiment. Without such an experiment, a direct performance-versus-time comparison between full-scale training and VQ transplant cannot be established. Consequently, it remains unclear whether the proposed framework delivers a superior performance-time ratio.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCxBrElqHX", "reviewer": "Reviewer_8aY8", "summary": "This paper presents an experimental approach to study the efficacy of different vector quantization (VQ) strategies in image tokenizers. The \"transplant\" aspect of their approach comes from the fact that they bootstrap from a pretrained image tokenizer. A sketch of their approach is as follows: (1) take a pretrained image tokenizer and replace the VQ module and learn a new codebook, (3) fine-tune the decoder of the pretrained image tokenizer with respect to the new VQ module. The authors demonstrate that they are able to improve a pretrained image tokenizer (VAR) using this approach with a new VQ algorithm MMD-VQ -- which is based on distributional similarity.", "questions": "Is the optimization objective for MMD-VQ: Y* = argmin_Y D^2(X, Y), where X is a batch of images? If so, please specify this more explicitly in the paper. I also can't find where you specify the batch size used in the experiments.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The general idea of being able to plug and play different components to explore a space of algorithmic inductive biases is welcome. MMD-VQ is an interesting proposal, but there is limited analysis of its properties provided for it in the paper.", "weaknesses": "In many ways this approach resembles a fairly standard practice of taking a pretrained vision model as a feature extractor and training a new classification layer. The representational biases of the features will still be biases to the pretraining task -- or in this case, the VQ method used by the original tokenizer. This is further exemplified by the fact that the decoder needs to be fine-tuned in order for this approach to work -- at this point, why not also finetune the encoder? How I see it, bootstrapping from a pretrained model is where all of the computational savings are coming from, not from freezing any part of the network.\n\nThe MMD approach is interesting and potentially novel. However, for me to be convinced that it is a good approach, the paper requires a more detailed analysis (empirical or mathematical) of its properties. The approach only seems to only work if the encoder features are fixed distribution -- joint optimization of the encoder I anticipate will result in mode collapse. This is usually where the Gaussian assumptions (KL loss) come in to stabilize training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yQhL889kYz", "reviewer": "Reviewer_auQc", "summary": "This paper proposes VQ-Transplant, a framework that allows plug-and-play replacement of vector quantization (VQ) modules in pre-trained visual tokenizers such as VAR, without retraining the full model.  \nThe method freezes the encoder and decoder, replaces the native VQ module, and applies a lightweight decoder adaptation for only five epochs on ImageNet-1k to mitigate decoder–quantization mismatch.  \nThe paper also introduces MMD-VQ, a new quantization method using Maximum Mean Discrepancy to align feature and codebook distributions without assuming Gaussianity, making it well-suited for real-world data.\n\nExperiments show that VQ-Transplant achieves comparable or even superior reconstruction quality to fully trained baselines (e.g., VAR, VQGAN) while reducing computational cost by up to 95%.  \nIt generalizes well across datasets (ImageNet-1K, FFHQ, CelebA-HQ, LSUN-Churches) and supports both fixed-scale and multi-scale quantization settings.", "questions": "1. Can MMD VQ be extended to video or multimodal quantization?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "### 1. Practical contribution  \nThe proposed idea of transplanting VQ modules into frozen tokenizers is both novel and highly practical. It enables flexible experimentation with quantization methods without the need for costly end-to-end retraining, significantly improving research efficiency.  \n\n### 2. Theoretical soundness  \nThe introduction of MMD-VQ offers a principled approach to aligning feature and codebook distributions without relying on Gaussian assumptions. This formulation enhances robustness and provides a clear theoretical improvement over prior methods such as Wasserstein VQ.  \n\n### 3. Empirical performance  \nExtensive experiments demonstrate strong empirical results and substantial reductions in training cost, with consistent improvements across multiple datasets and model architectures, validating both the effectiveness and efficiency of the proposed framework.", "weaknesses": "### 1. Limited scope of tokenizer architectures  \nThe experiments are limited to VAR-based tokenizers. Evaluating the proposed method on diffusion-based or hybrid tokenizers (e.g., LDM, SANA) would better demonstrate its generality and robustness across different visual tokenization paradigms.  \n\n### 2. Limited evaluation in image generation and multimodality  \nIn modern settings, visual tokenizers are widely applied not only to image generation but also to multimodal understanding and reasoning.  \nIt would strengthen the paper if the authors could evaluate their approach on multimodal comprehension benchmarks such as Unified-IO, 4M (Massively Multimodal Masked Modeling), or TokLIP, if feasible within the rebuttal period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HZor1NbrW3", "reviewer": "Reviewer_tdwt", "summary": "This paper proposes a new method named VQ-Transplant, a computationally efficient framework for discrete visual tokenization designed to enable cheap and rapid exploration of novel Vector Quantization (VQ) techniques. To improve the compatibility of VQ-Transplant, the authors also propose a new variant called MMD-VQ. The paper conducts extensive experiments on image reconstruction to evaluate the proposed methods.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper is well written.\n\n - The proposed method is plug-and-play and can be applied to any pretrained discrete image tokenizer.\n\n - This method effectively reduces the training time of the codebook.", "weaknesses": "- Table 1 is intended to demonstrate that VQ-Transplant requires very few resources for training. However, I have some concerns. First, it only compares training time without comparing performance, so we cannot tell how well VQ-Transplant actually performs. Second, this comparison is also unfair, because VQ-Transplant benefits from a pre-trained tokenizer — comparing post-training cost with pre-training cost together is not a fair evaluation.\n\n - Table 2 shows that when the codebook size is 4096, the performance of MMD-VAR is comparable to VAR, but does not surpass it.\n\n - I believe that after a VQ tokenizer is trained, the encoder–decoder parameters and the codebook are highly correlated (otherwise the model would collapse). In this case, if a strong tokenizer is provided, retraining the intermediate codebook will naturally make it very close to the original one. I somewhat suspect that VQ-Transplant might simply be replicating the original codebook. The authors should conduct verification on the codebook to check whether it is merely a simple copy of the original.\n\n - The paper lacks evaluation on generative capability. Since the tokenizer is ultimately intended for generation tasks, whether it can outperform the baseline in generation is also a crucial part of evaluating the tokenizer’s performance.\n\n - It is recommended that the authors also conduct experiments on continuous VAEs to further validate the effectiveness of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)