PQGAN: Product-Quantised Image Representation for High-Quality Image Synthesis
Authors: Denis Zavadski
Nikita Philip Tatsch
Carsten Rother
Venue: ICLR 2026 Poster (2026)
Keywords: Vector Quantisation
Representation Learning
Diffusion Models
Generative Models
Image Synthesis
Product Quantisation
Primary area: generative models

Abstract: Product quantisation (PQ) is a classical method for scalable vector encoding, yet it has seen limited usage for latent representations in high-fidelity image generation.
In this work, we introduce \textit{PQGAN}, a quantised image autoencoder that integrates PQ into the well-known vector quantisation (VQ) framework of VQGAN and adapts it to the regime of large-scale latent generative models.
PQGAN achieves a noticeable improvement over state-of-the-art methods in terms of reconstruction performance, including both quantisation methods and their continuous counterparts. We achieve a PSNR score of 37dB, where prior work achieves 27dB, 
and are able to reduce the FID, LPIPS, and CMMD score by up to 96\%. 
Our key to success is a thorough analysis of the interaction between codebook size, embedding dimensionality, and subspace factorisation, with vector and scalar quantisation as special cases. We obtain novel findings, such that the performance of VQ and PQ behaves in opposite ways when scaling the embedding dimension. Furthermore, our analysis shows performance trends for PQ that help guide optimal hyperparameter selection.
Finally, we demonstrate that PQGAN can be seamlessly integrated into pre-trained diffusion models. This enables either a significantly faster and more compute-efficient generation, or a doubling of the output resolution at no additional cost, positioning PQ as a strong extension for discrete latent representations in image synthesis.

Published as a conference paper at ICLR 2026
PQGAN: PRODUCT-QUANTISER IMAGE REPRESEN- TATION FOR HIGH-QUALITY IMAGE SYNTHESIS
Denis Zavadski, Nikita P. Tatsch, Carsten Rother
Heidelberg University, Zuse School ELIZA
{name. surname
ABSTRACT
Product quantisation (PQ) is classical method for scalable vector encoding, yet it has seen limited usage for latent representations high-fidelity image gener- ation. In this work, introduce PQGAN, quantised image autoencoder that integrates PQ into the well-known vector quantisation (VQ) framework of VQ- GAN and adapts it to the regime of large-scale latent generative models. PQ- GAN achieves a noticeable improvement over state-of-the-art methods in terms of reconstruction performance, including both quantisation methods and their con- tinuous counterparts, We achieve PSNR score of 37 dB, where prior work achieves 27 dB, and are able to reduce the FID, LPIPS, and CMMD score by up to 96%. Our key to success is thorough analysis of the interaction between codebook size, embedding dimensionality, and subspace factorisation, with vector and scalar quantisation special cases. We obtain novel findings, such that the performance of VQ and PQ behaves in opposite ways when scaling the embed- ding dimension. Furthermore, our analysis shows performance trends for PQ that help guide optimal hyperparameter selection. Finally, we demonstrate that PQ- GAN can be seamlessly integrated into pre- pre-trained diffusion models. This enables either a significantly faster and more compute-efficienn generation, or a doubling of the output resolution at additional cost, positioning PQ strong extension for discrete latent representations in image synthesis.
INTRODUCTION
Recent advancements in image processing in general and image generation in particular, have inten- sified the need for scalable solutions. Datasets grow larger (Schuhmann et al., 2022; Gadre et al. 2023), models become increasingly compute-intensive (Podell et al. [024| Rombach et al.) 2022) and resolutions increase (Chen et al., 2024b a). Addressing this trend typically requires either a proportional increase in computational resources, which is often impractical, or efficient data representations that reduce redundancy while preserving semantic content (Rombach et al.) 2022 Peebles & Xie 2023 Esser et al.] (02T). common strategy to operate in the latent space of an autoencoder trained for image reconstruction, under the assumption that this bottleneck repre- sentation discards perceptually negligible details while retaining the core visual structure (Rombach)
ct al. 2022 Lee et al. 2022).
2017) demonstrated that latent image representations can be quan- tised by learning a codebook of discrete vectors, where each latent vector is replaced by its nearest codebook entry. This mapping transforms the latent space into a discrete index space, where each spatial location is represented by single codebook index. Crucially, this makes the storage cost of representation independent of the dimensionality of the vectors themselves and only the spatial resolution and the codebook size determine the representation size. This property enabled new class of models for image compression EEl-Nouby et al. 20233 Mentzer et al. 2020: 2024) and au-
VQ-VAE VVan Den Oord
toregressive generation (Esser et al.) 2021 Ramesh et al. 2021 Yu al. 2022b). where discrete
latents offer both compactness and modelling flexibility.
While it is theoretically possible to arbitrarily high-dimensional latent vectors, learning such codebook entries becomes increasingly difficult as dimensionality grows (Zhu et al. (024). Because for VQ all dimensions are coupled within each codebook entry, the training signal becomes sparse

---

Published as a conference paper at ICLR 2026
Input
POGAN (F 8)
POGAN (F 16)
SDv2.1 VAE (F = 8)
Figure 1:
Reconstruction results for our quantised PQGAN vs. the continuous VAE in StableDiffu- sion2.1 (Rombach et al., (2022). Even with stronger spatial downsizing factor FF for the latent repre- sentation, our product quantised representation space shows higher fidelity (see details in faces).
in high-dimensional spaces, leading to slower convergence and limiting the effective size of the codebook. Moreover, redundancy often arises, as distinct entries may differ only partially across dimensions. We address both issues by applying product quantisation (PQ), originally introduced by JJgou et al. (2010), and factorising each codebook entry into S independent subspaces. Learning each subspace independently reduces the dimensionality of the learned features and ensures that training signals remain dense and well-conditioned. The factorisation enables our model to represent fictive codebook of size K.S where is the number of entries per subspace. Since entries from different subspaces can be composed freely, there is no need to store or learn redundant combinations explicitly. These properties allow us to achieve state-of-the-ee results on ImageNet (Russakovsky et al., 2015), with FID scores as low as 0.036 and PSNR values up to 37.4 dB, using only 512 codebook entries per subspace.
In this work, we conduct a thorough evaluation of PQ for latent image representation across its full hyperparameter spectrum, spanning the extremes vector and scalar quantisation. Our analysis reveals consistent trends that guide principled parameter selection. Furthermore, it gives novel find- ings, such that the performance of VQ and PQ behaves in opposite ways when scaling the embedding dimension.
applications such as image compression (Menter et al. 2020 El-Nouby et al.) 2023) and autore- gressive modelling (Esser et al. 2021), PQ introduces practical challenges: As the number of sub- spaces increases, one must either store separate codebook indices or construct a joint codebook, which becomes exponentially large and unfeasible store for compression learn for generative models. As a result, prior work has not fully explored the representational capacity of PQ. How- ever, we observe that the limitation of having small set of codebook indices does not apply the context of diffusion- and flow-based generation (Rombach el al.] 2022, Podell et al. [2024] Ho et al.) 2020 Esser et al., 2024), where the decoder operates directly the latent space without requiring an explicit autoregressive factorisation. This makes PQ particularly well-suited for diffusion- and flow-based models, and allows us to explore its full potential large-scale generative setting.
We demonstrate that large latent diffusion models, such as Stable Diffusion (Rombach et al. (022). can operate effectively on spatially restricted yet high-dimensional latent representations, maintain- ing comparable or even reduced computational cost. However, our evaluation reveals that as image resolution increases, the reconstruction quality of default latent representations can degrade signifi- cantly, introducing artefacts that propagate through the generative process while PQ representations manage to maintain a high reconstruction fidelity (compare Figure
Our contributions can be summarised follows:
We propose PQGAN, new latent representation for image generation, based on the con- cept of product quantisation (PQ).
We conduct a thorough analysis of PQ hyperparameters, with Vector and Scalar quantisa- tion as special cases. We observe novel findings, such that the performance of VQ and PQ
2

---

Published as a conference paper at ICLR 2026
behaves in opposite ways when scaling the embedding dimension and further performance trends that guide optimal hyperparameter selection.
We demonstrate that PQ enables state-of-the-an image representation quality, achieving PSNR of 37.4 dB and FID as low as 0.036 using only 512 codebook entries per subspace, surpassing even continuous counterparts.
We show that pre-trained diffusion models can be efficiently adapted to PQ-based repre- sentations, enabling either significantly faster generation or double resolution at equal cost.
RELATED WORK
Latent Image Representation for Generative Models. Modern generative models demand in- creasingly high computational resources, driven by the scaling of model parameters (Podell al., 2024 Peebles & Xiel 2023), the use of attention-heavy (Vaswani et al. 2017) architectures (Chen
2023 Touvron et al. 2021), and the pursuit of high-resolution image Chen et al. 2002aa). Many of the most successful methods, such as au- toregressive models and diffusion models, also require iterative inference, further compounding the cost of generation. Optimising the generative pipeline remains an active area of research, either by improving the generative process itself (Sauer et al. 2024b Kang et al. 2024 Salimans Ho 2022 Lin et al. 2024 Sauer et al. (024a) or by modifying the representation which the model operates. The latter strategy typically involves encoding high-resolution images into lower-resolution latent spaces, significantly reducing the computational burden during generation. Different generative models impose different constraints on their latent spaces. Autoregressive transformers, for exam- ple, require discrete and indexable representations (Esser et al. 2021), hereby limiting the capacity of the underlying representation. However, diffusion models are generally assumed to operate ef- fectively on continuous latent spaces (Rombach al., 2022 Podell al. 2024 Pernias al. 2024 Chen et al. (024a), provided they are sufficiently low-dimensional In practice, this often motivates the use of both low spatial resolutions and reduced channel dimensionality in the latent represen- tation to ensure tractable generation. Contrary to common practice (Rombach et al., 2022 Podell al. 2024), we find that diffusion models can efficiently operate on high-dimensional embeddings as long as the spatial resolution remains low. By leveraging PQ, we learn discrete representation that not only surpasses continuous latent spaces in reconstruction fidelity, but also allows for even lower spatial resolutions resulting in more efficient generation with no loss in quality.
al. 2024b| Peebles Xie
synthesis (Podell et al. 22024
Vector Quantisation and Codebook Models. Vector quantisation (VQ), popularised by VQ- VAE (Van Den Oord et al., learns discrete latent representations by replacing each continuous vector with its nearest entry from a learned codebook. This approach is well-suited to autoregressive models (Esser et al., 2021 Lee et al. 2022 Yu et al. 2022b), and has also been widely applied in image compression (Mentzer et al., 2020 2024), where discrete indices compactly represent high- dimensional latent vectors.
However, standard VQ suffers from limitations when applied to high--dimensional embeddings. Be- cause all dimensions are quantised jointly through single codebook lookup, the model receives sparse and entangled training signals, often Ieading to codebook collapse and slow convergence. To address this, various extensions have been proposed (Yu et al. 2022a) such as residual quantisa- tion (Kim et al. 2025 Lee et al.] 2022). hierarchical quantisation (Razavi et al. 2019). and other regularisation techniques (Zhang et al. 2023)- but these generally offer only incremental improve- ments and retain the disadvantage of high dimensional codebooks with sparse training signals.
One core issue remains: the joint modelling high- tigh-dimensional latent spaces via a single codebook imposes structural bottleneck. Multi-channel quantisation (Zheng et al. (022) made the first step into splitting the latent representation along the channel side and used one joined codebook for all splits thereby limiting expressiveness and the number of reasonable splits. In the context of image compression, scalar quantisation and product quantisation have been explored (El-Nouby al. 2023: Mentzer al. 2020), but their capacity was limited by the need to balance bitrate and distortion.
In contrast, we apply PQ in the context of latent representation learning for diffusion based gener- ative models. Freed from rate-distortion constraints, we explore the full potential of PQ and show that highly factorised latent space not only avoids the training difficulties of conventional VQ,
3

---

Published as a conference paper at ICLR 2026
but also enables reconstructions that surpass continuous autoencoders (Rombach et al. 2022 Podell et al. 2024) achieving state-of-the oftt----r---ttttt fidelity while operating at significantly lower spatial resolution.
3 METHOD
In this work, we isolate the effect of product quantisation on the representational capacity of latent autoencoders. To ensure controlled comparison, we adopt the VQGAN (Esser et al. (021] archi- tecture and training pipeline, widely regarded the standard backbone for vector- cettor-quattsed image representation, without modification. Our only change is to replace the original vector quantisation module with product quantisation scheme, as detailed in Section enabling direct evaluation of PQ's impact on reconstruction quality.
To further assess the utility of these representations, train StableDiifusion2.1 (Rombach al, 2022) to operate directly on our product quantised latent space and revisit two common assumptions diffusion-based image generation. First, we challenge the widespread misconception that latent representations must be kept low-dimensional to maintain computational efficiency. Second, we highlight critical oversight: as image resolution increases, the reconstruction quality of standard latent spaces deteriorates, significantly introducing artefacts that propagate through the generative process. We discuss both issues in detail Section 3.2
3.1
PRODUCT QUANTISATION
Encoder
Decoder
Latent Embedding
Quantised Embodding
Vestor Quantisatio
Product Quantisation
Lailen
Codebooks
Figure 2: Comparison of vector quantisation (VQ, top) and product quantisation (PQ, bottom) in the latent space of an autoencoder. VQ replaces each latent vector with the nearest entry from single codebook of size K. PQ splits the vector into subspaces, quantising each independently, resulting in combinatorially large fictive codebook of size K"S while maintaining low per-subspace complexity.
Preliminaries Vector Quantisation. We begin by formalising the notation for vector quantisation the context of latent image representations. Let E # denote an RGB image resolution H x W. Its latent embedding ze is obtained from the bottleneck of an autoencoder, typically with a spatial resolution reduced by downsampling factor F. The resulting latent representation W/F, and d is the embedding dimensionality. We refer to
=n E kxxxxxd where h = H/F, F, each vector at a spatial location as a latent "pixel", denoted by E Rdd.
To discretise this representation, vector quantisation learns codebook C (eI...,, cK} consisting of K codewords ex E Rd. Each latent vector Pe is then replaced by its nearest codebook entry under some distance metric (typically 62). However, learning such high-dimensional codebook is non-trivial: since all d dimensions must be learned jointly. the signal-to- tottttopppaattee ratio becomes increasingly sparse, which limits scalability and leads slower convergence (see Section 4.2).
Product Quantisation. To address the challenges of -igeedimensional codebook learning, PQ de- composes each latent vector pe Bdd into S disjoint subspaces of dimension d/S (assuming is divisible by S), and learns separate codebook for each subspace. That is, for S E {1,...,dh d], each
latent pixel is split as pt
(e) is quantised indepen- [p2 . (9)) with Pt (s) Rd/S and each p
(1)
4

---

Published as a conference paper at ICLR 2026
dently using its own codebook C(+) {el special cases of PQ when S = 1 and S = d, respectively.
6K ). Vector and scalar quantisation emerge as
A visual comparison between VQ and PQ is shown in Figure 2 Because the sub-codebooks trained independently, the signal per parameter is significantly denser, improving convergence and scalability. Moreover, since sub-codebook entries can be freely recombined across subspaces, PQ defines a fictive codebook of combinatorial size K$ without requiring the storage or learning of all combinations individually. This greatly increases the expressive capacity of the representation while keeping each sub-codebook manageable in size.
More formally, the advantages of PQ can be attributed to two closely related phenomena:
1. Training signal sparsity and sample efficiency. Vector quantisation operates the full entangled d-dimensional latent space, and learning meaningful codebook requires many training samples sufficiently populate that space, To achieve covering resolution e, the required number of samples grows as (((1/e)*), leading to severe data sparsity in high dimensions. This impairs codebook learning. as each centroid receives little supervision. In contrast, PQGAN factorises the space into S = =d/2 subspaces of dimension 2. each of which can learned independently with only (((1/e)²) samples to achieve the same latent coverage resolution of regardless of the latent dimension. This keeps the effective sample density constant as dl increases, allowing PQ to scale latent dimensionality without sacrificing codebook fidelity.
2. Quantisation error sealing. Classical quantisation theory (ador 1982) shows that mean squared quantisation error scales as O(K -2/d) in d-dimensional space with K centroids. For VQ. this implies that increasing latent dimension requires exponential growth in code- book size to maintain the same quantisation error, which is an impractical trade-off. PQ sidesteps this by operating fixed low-dimensional subspaces (with d/2), where the quantisation error remains constant 222) = O(K-1), independent of total latent di- mension. This allows PQ to increase total representational capacity (via higher d) without degrading quantisation error or requiring larger codebooks.
Together, these principles suggest clear difference in scaling behaviour. While VQ expected to degrade as latent dimensionality increases, PQ should benefit from higher-dimensional embeddings. These trends arise naturally from the differing scaling properties of VQ and PQ with respect to training signal efficiency and quantisation error.
As in standard VQ, the quantisation objective minimises the reconstruction loss between the original and quantised latents, and includes a commitment term to stabilise codebook usage. The training objective is:
Lpo 11 - sg(2,)112 + +1(sg(==)) - =2112 Liee + LGAN
(1)
where sg(.) denotes the stop-gradient operator, 2g the quantised latent, Cree the perceptual reconstruction loss, and LGAN is the adversarial loss used in VQGAN training. All components are inherited unchanged from the original VQGAN (Esser et al. 2021 framework.
3.2 LATENT ADAPTATION
Modern state-of of-the-rrt image generation models typically operate on latent representations rather than raw images to improve computational efficiency (Rombach et al. 2022 Podell et al. (024). The key factor behind this efficiency the reduced spatial resolution of the latent space. In diffusion models such as Stable Diffusion (Rombach et al.) (2022), the use of attention layers throughout the architecture causes computational cost to scale quadratically with spatial resolution. As result, generative models usually favour low-dimensional latents limit compute.
However, we highlight that in diffusion models, spatial resolution not channel dimensionality is the primary computational bottleneck. We argue that high-dimensional latent representations be used without compromising efficiency. To validate this, we adapt StableDiffusion2.1 to operate directly product-quantised latents in two-stage training procedure.
5

---

Published as a conference paper at ICLR 2026
We observe that the U-Net generator contains fixed projection from 4 to 512 channels at the input layer, and from 512 back to 4 channels at the output layer. These projections form an artificial bottleneck that restricts the usable dimensionality of the latent space. We increase these projections to match the dimensionality In the first stage, train only the modified input and output projections, keeping all other weights frozen. After 20k steps, we unfreeze the entire model and continue fine-tuning for an additional IM steps with the typical diffusion objective
our quantised latents, leaving the rest of the architecture unchanged.
L Ez [lle c0(2e, c.)
with the target image 20, the noisy image 2t, the timestep t, the text conditioning e and noise
4 EXPERIMENTS
The first part of our experimental analysis investigates how the key design choices in product quan- tisation namely, codebook size, subspace factorisation, and latent dimensionality influence re- construction quality and codebook behaviour. In Section[4.) we present structured evaluation that spans the quantisation spectrum between vector and scalar regimes, revealing consistent trends and points of saturation in model performance.
To assess the efficiency and stability of learned representations, we further analyse codebook utilisation via entropy and perplexity metrics in Section 4.3 Finally, in Section 4.4 compare models to state-of-the-arr discrete and continuous latent representations, demonstrating clear and consistent gains in both reconstruction fidelity and quantisation robustness.
In the second part of our evaluation (Section 4.5 we adapt StableDiffusion2.1 to operate on our product-quantiser latent space. This setup allows to examine underexplored assumptions in latent diffusion models: the need for low-dimensional latents and the presumed absence of degra- dation of reconstruction quality high resolution. We show that product-quantised latents offer practical advantages in both computational efficiency and fidelity, particularly in the high-resolution regime.
4.1 METRICS
To evaluate the quality of the reconstructed images, we report four complementary metries: FID (Heusel et al. (017), PSNR, CMMD (Jayasumana et al.) (024)), and LPIPS (Zhang et al.) 2018 While FID remains a widely adopted perceptual metric, it has known limitations due to its reliance on Gaussian assumptions and sensitivity to distributional shifts (Jayasumana et al. 2024). To address this, we additionally employ CMMD, which has been shown to provide a more faithful and unbiased comparison of image similarity.
For pixel-level accuracy and structural fidelity, we report PSNR and LPIPS, which respectively quantify exact reconstructions and perceptual consistency. Finally, assess the effectiveness of our product-quantised representations, we analyse codebook utilisation using two complementary metrics normalised entropy Hn, which captures the uniformity of code usage, and normalised perplexity Pa. which reflects the fraction of entries actively used:
(((p) H, log
EK Pr log log K
K
P(p) K
exp K
P.
(3)
where p is the empirical codeword distribution and (()) is its Shannon entropy. Both metries are bounded in [0, 1]. with H, indicating perfectly uniform usage over the active subset of the codebook, and P,, = denoting full codebook activation.
4.2 PRODUCT SPACE COMPOSITION
To study the interaction between key quantisation parameters, we systematically vary the latent dimensionality (4...., 2266), the number of subspaces E {1, 2 2 8 d), and the codebook

---

Published as a conference paper at ICLR 2026
S
S
d/8
S=d/4
S=d/2
S
10²
10²
10
10a
10*
FID
FTD
10t
10a
100
10
10a
10a Codebook Size
10
Latent Dimension
Number of Subspaces
(a) Trends with respect to performance.
1.0
1.0
0.8
Entropy Hn.
Entropy H.
Porplexity Pa
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
10a
10'
10
10
10a
10a
10*
Number of Splits
Codebook Size K
Codebook Size K
(b) Trends with respect to codebook utilisation.
Figure 3: Quantitative analysis of product quantisation configurations. (a) Performance across latent dimensionality d, number of subspaces S, and codebook size K measured by FID. (b) Codebook utilisation metrics (entropy and perplexity) for the same configurations, showing improved usage and efficiency with increasing factorisation.
size K E {128, 512, 1024, 8192, 16384). All models are trained on the ImageNet (Russakovsky al. 2015) dataset resolution of 256x256 for one million steps using a batch size 20.
Figure 3a presents FID scores on a log-log scale across different PQ configurations. Each marker represents configuration, with colour indicating the number of subspaces S. In the left plot, marker size reflects codebook size K; in the middle and right plots, it reflects latent dimensionality d.
We observe clear divergence in behaviour between vector quantisation (Esser al. 2021) (S 1) and PQ: increasing latent dimensionality degrades performance for VQ, while improving it for PQ (left). Increasing the number of subspaces S Slaads consistent performance gains, saturating around S = d/2, just before the scalar quantisation limit at S = d (middle).
FID appears only weakly dependent on codebook size, but is strongly influenced by the number of subspaces (right). The plot shows that even for small codebooks (K = 128 or 512), PQ models with higher dimensionality and greater factorisation outperform all VQ configurations.
These results indicate that transitioning from tQ PQ fundamentally changes the scaling behaviour of latent representations: PQ benefits from larger latent dimensions and learns high fidelity, whereas VQ fails scale effectively. Full results for PSNR, LPIPS, and CMMD are provided in the supple- ment C and confirm the same trends. However, maintaining a high number of subspaces requires higher codebook matching times that scale linearly with the number of subspaces which put practical limitation on the usability of PQ in high dimensional settings. The best performing PQ autoencoder in our experiments had a quantisation overhead of just 50% of walltime compared to when codebook matching is omitted. More details on the computational overhead are provided in the supplement B
4.3 CODEBOOK UTILISATION
One of the main challenges in vector quantisation is learning high-dimensional latent representations within a large codebook. Therefore we track codebook diagnostics to assess the effective utilisation and diversity of the quantised representations. Figure 3b shows the normalised entropy H,, and the normalised perplexity P, for all evaluated configurations, plotted on a linear-log scale. Marker colours indicate the number of subspaces S, and marker sizes correspond to latent dimensionality d.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of quantisation methods on ImageNet 256x256 validation set. Metrics include reconstruction FID (rFID), PSNR, CMMD, LPIPS and information about the downsampling factor F, the channel latent dimension d, codebook size K and the latent training method with KL, vector quantisation (VQ), residual quantisation (RQ) product quantisation (PQ) and multi-channel quanti- sation (MCQ).
Method
Latent
Latent Resolution
PSNR1
rFID]
CMMD
LPIPS]
VQGAN [Esser al. 2021
VQ
16
256
16 384
19.7
4.98
0.422
0.1633
VQ
256
0024
19.4
7.94
0.862
0.1834
VQGAN (LDM Rombach at al.
2022
VQ
16
16 x 16
16 384
20.6
5.51
0.621
0.1462
0.0815
VQ
4
64 64
256
26.4
0.47
VQGAN-LC Zhu
2.62
0.1280
VQ
8
32 x2
100.000
27.0
1.29
0.080
0.0712
VIT-VGGAN Yi 2022a
VQ
32
12
1.28
Mo-VQGAN Zheng el al 2022
MCQ
x 16
256
1024
22.4
1.12
0.1132
RO-VAE (Lee
RQ
8
256
16 384
2.69
LDM VAE Rombach et al
2022
K1.
18 x 16
16
24.0
0.87
0.198
0.0665
KL.
32
24.2
0.94
0.168
0.0651
x 61
28.
0.27
0.080
0.0282
SDv2. I VAE Rombach 0022
KL.
32 x 32
25.3
0.75
0.133
0.0610
Praeetee
0.0573
PQGAN (ours)
PQ
8 X
256
256
24.6
0.90
0.164
0.0621
128
28.3
0.41
0.0304
PQ
8
32 x 32
128
512
37.4
0.036
0.011
0.0024
VQ (S 1) exhibits the lowest entropy and fails scale with increasing latent dimensionality, as reflected in declining performance for larger d (larger marker sizes, left).
In contrast, increasing the number of subspaces S leads to more uniform code usage, with entropy Hn largely unaffected by codebook size K (middle). Instead, we observe that entropy clusters by subspace configuration, reflecting the structure imposed by PQ. The right plot shows that normalised perplexity Pn, unlike entropy, does depend on K: as K increases, high-dimensional PQ represen- tations gradually saturate their code usage. Even for very large codebooks (K" 16334), PQ models maintain H, > 0.8, suggesting robust and effective utilisation despite overparamcterisation.
These results reinforce our earlier findings: PQ not only improves reconstruction performance with increasing factorisation, but also makes more efficient use of the available codebook capacity, while VQ struggles to scale beyond small latent dimensions.
4.4 COMPARISON TO STATE OF THE ART
Building on our previous analysis, we select the best-performing configuration at downsampling factor F 16, which uses d 128 channels, S 64 subspaces, and codebook size of K 128.
For F = 8, we evaluate the top three configurations and retain the best (S = 64, d = 128, K = 512) final high-fidelity F quantisation (Lee et al., 2022). we also train a F 32 model with the same dimension d = 256 as the competitor but with S 64 and K 256. Table compares reconstruction performance on the ImageNet (Russakovsky et al. 2015) validation set against state-of-the-ae quantisation meth- ods as well as continuous latent spaces based on KL-regularised autoencoders within the VQGAN framework. We also include the latent representations employed by the production-gren -radde diffusion models 1 (Rombach et al. 2022) and SDXL (Podell et al. 2024).
model. To allow for fair comparison
state-of-the-art residual
For the ImageNet 256 x 256 comparison, we evaluate latent spaces downsized by factors F (8, 16, 32), resulting latent resolutions 32 32, x 16 and x 8, respectively. We also report results for F 4 (i.e., 64 x 64 latent resolution), but highlight these in grey indicate that such configurations are impractical for real-world applications and primarily serve as performance upper bounds for competing methods.
Our best perforrff model, PQGAN with outperforming all other quantised and continuous baselines across all evaluation metrics. According to rFID, CMMD, and LPIPS, reconstructions from this model are nearly indistinguishable from the original images. Notably, the second-best model is also PQGAN, operating F = 16, i.e. with half
8, achieves a PSNR 37.4 dB substantially
8

---

Published as a conference paper at ICLR 2026
Table Comparison of representation transferability for different encoding methods.
FFHQ
LSUN
Method
Latent Resolation
rFID
PSNR
CMMD1
LPIPSL
rFID
PSNR
CMMD
LPIPS.
VQGAN-LC Zhu 2026
32 32
3.35
29.6
0.094
0.0390
7.01
24.7
0.261
0.0711
SDv2.1
Kombach
29.9
SDXI. VAP
Rxalll of 81 21/4
32
32
1.77
30.0
0.122
0.024
6.00
24.9
0.317
0.0566
PQGAN (Ours)
16 16
0.51
33.3
0.190
0.0138
4.77
27.9
0.353
0.0315
32
Table 3: Quantitative comparison StableDiffusion (SD) and SD when adapted to PQ latent space. Sample time is measured A A100 GPU with 50 sampling steps. Sample time and required memory is measured for the generation of one sample.
Generator
Image Size
Latent Resolution
F
Samples/s1
Memory (GB)
SDv2.1 Rombach al. 2022
768 768
96
0.116
14.9
PQSD-HR (Ours)
1536 x 1536
96 x
128
0.112
14.9
POSD-Precise (Ours)
768 x 768
96
128
0.1
14.8
PQSD-Quick (Ours)
768 768
48 48
128
0.465
7.7
the spatial resolution, yet still outperforming all competing methods, including those with F 4, which use 16 times more latent pixels.
While other quantised approaches require large codebooks (up 100000 entries (Zhu 2024
to achieve competitive results, PQGAN attains superior performance with compact codebooks of K 128 512, depending on the downsampling factor. These results underscore the efficiency and scalability of product quantisation in high- high-fidelity latent representation learning.
To assess the generality of our learned representationss we evaluate their transferability to unseen datasets. Specifically, we compare PQGAN against the best-performing quantised and continuous baseline models (at 8) on FFHQ (Karras al. 2019)) (10k images) and LSUNN (Yu et al. 2015)
(3k images) resized to 256 256. Results are shown in Table? 2
PQGAN again achieves the strongest performance across all metrics, even when operating with smaller latent resolution. On FFHQ, PSNR increases for all models, with PQGAN (F 8) reaching 42.1 dB, while LPIPS also improves substantially. Although rFID and CMMD scores increase across the board, PQGAN remains top-ranked. Since both rFID and CMMD are global statistical metrics rather than image-pair comparisons, they are best interpreted for relative model ranking rather than absolute cross-domain comparison.
On LSUN, LPIPS values remain largely consistent with ImageNet results. The continuous baselines show almost unchanged PSNR, while VQGAN-LC (Zhu et al. 2024) experiences a slight drop. PQGAN, by contrast, maintains or even slightly improves its performance, demonstrating robustness under domain shift across all metrics, including FID and CMMD.
These results suggest that PQGAN learns stable and expressive representations that generalise well unseen data. Even when operating lower spatial resolutions, it consistently outperforms both quantised and continuous baselines highlighting the effectiveness of PQ for robust representations.
By training latent diffusion model from scratch on the PQGAN representation with the ImageNet data, the models achieve an improved FID score for generated ImageNet samples compared to the model trained on the StafleDiffusson2.11 VAE. More details are provided in the supplement
4.5 LATENT SPACE INTEGRATION
Having established the fidelity and generality of the learned representation, we now investigate its usability as a generative prior by integrating it into StableDiffusion2.1 (Rombach et al.) 2022) As discussed in Section 3.2 the default SD generator operates on a continuous latent space with downsampling factor F
and dimensionality d 4, mapping images from (H x W 3) to (H/8 x W/8 x 4). In our setup, we explore three PQGAN variants summarised in Table each offering different trade-offs between spatial resolution and generative efficiency.

---

Published as a conference paper at ICLR 2026
Figure 4: Generated examples for our PQSD-HR (left), PQSD-Precise (top), PQSD-Quick (bottom).
Because our PQGAN models were originally trained lower resolutions, we fine-tune both F and F = 16 variants on ImageNet (Russakovsky et al. (013) at 768 x 768 for 200,000 steps using batch size of 15. Figure a qualitative comparison against the native SD latent representation. The SD-VAE exhibits degradation at high resolution, including hallucinations, which are absent from our PQ-based reconstructions. More examples in the supplement
To adapt SD's U-Net architecture to higher channel dimensionality, last convolutional layers, increasing their channel width while reusing the pretrained weights. To preserve the pretrained model's capabilities, we first freeze all other layers and train only the newly added components first for 20k steps. We then train the full model and gradually increase the resolution up 768 768 for POSD-Quick and 1536 1536 for PQSD-HR. Full training details are provided in the supplement A.
modify only the first and
Figure shows generations from all three PQSD variants. Generating 1536 1536 samples with PQSD-HR (F = 16) requires approximately the same computational cost as 768 x 768 generation with SDv2.1, demonstrating the efficiency of operating at lower spatial resolutions (see Table 3 Alternatively, PQSD-Quick allows 768 x 768 samples to be generated with a speed-up of up to times, while maintaining higher reconstruction fidelity than the native SD-VAE. These results highlight that product-quantne also serve as drop-in replacements in generative pipelines, offering improved performance with minimal architectural modification.
latent spaces not only generalise well and scale efficiently, but can
5 CONCLUSION & DISCUSSION
We introduced PQGAN, product-quantised image representation model that achieves state-of-the- art latent reconstruction fidelity. By integrating product quantisation into the VQGAN framework, we obtain high-dimensional latent spaces reduced spatial resolution without sacrificing codebook efficiency.
We demonstrated that such representations can be seamlessly integrated into pre-trained diffusion models. Adapting Stable Diffusion to three distinct PQ configurations, we showed that large gener- ative models can operate on new latent spaces without full retraining. This enables either doubling of output resolution at comparable computational cost faster generation at fixed resolution.
While product-quantise spaces are not directly suited for index-based tasks due to the exponential growth of index combinations with increasing splits, generative methods that do not rely indexing but use quantisation (Kolesnikov et al., 2022) or image representations directly (Chen et al. 2024a
BFL et al. 2025) fully benefit from PQ representations.
Overall, our results establish product quantisation as practical and scalable alternative to both discrete and continuous latent representations in modern generative modelling.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Denis Zavadski and Nikita Philip Tatsch are supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Re- search. The authors gratefully acknowledge the support by the Ministry of Science, Research and (MWK) through bwHPC, SDS@ hd and the German Research Foun- dation (DFG) through the grants INST 35/1597-1 FUGG and INST 35/1503-1 FUGG. The authors gratefully acknowledge the computing time made available to them the rer-rperformance com- puter at the NHR Center of TU Dresden under the DFG grant number 498181233
the Arts
REFERENCES
BFL, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhom, Jack English, Zion English, Patrick Esser, et al. Flux. kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint wwXiv:2506.15744 2025.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to st-ron training of diffusion transformer for 4k text-to-image sont-to-imaee generation. In European Conference on Computer Vision, pp. 74-91. Springer, 2024a.
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-$ alpphaS Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024b. URL
net/ (arppennreeeeeeeeetooomm2222222
Alaaeldin El-Nouby, Matthew Muckley, Karen Ullrich, Ivan Laptev, Jakob Verbeek, and Herve Jegou. Image compression with product quantized masked image modeling. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httpp::/epperrview..cem
Eorum? 1dmmm11222222aaa55
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion, pp. 12873- 12883, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning. 2024.
Samir Yitzhak Gadre. Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances Neural Information Processing Systems, 36:27092-27112 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, (166800-8511 2020.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Pro- ceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 9307- 9315, 2024.
Herve Jegou. Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-12) 2010.

---

Published as a conference paper at ICLR 2026
Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shecht- man, Jun- Yan Zhu, and Taesung Park. Distilling diffusion models into conditional gans. In Euro- pean Conference on Computer Vision. pp. 428 447.. Springer. 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style- based generator architecture for generative adversarial networks. In Proceedings of IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Jachyeon Kim, Tachong Moon, Kcon Lee, and Jaewoong Cho. Efficient generative modeling with residual vector quantization based tokens. In secty- International Conference on Machine Learning, 2025. URL
net fott/foru??dd=fffo FurF ZuuzND
Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. Ad- vances Neural Information Processing Systems, 35:26295-26308, 2022.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Shii-SShnni Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Shanchuan Lin, Anran Wang, and Xiao Yang. Sixl-lightnin: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402 13929, 2024.
Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gen- erative image compression. Advances in neural information processing systems, 19:11911-11924, 2020.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: VQ-VAE made simple. In The Tweifth International Conference on Learning Representa- fions, 2024. httppttss////eeee
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4195-4205, 2023.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview net/ Deeereeviiewneee/fooouuddddsssss
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL
htttps://openrevieew.. net wowoooomidddiideeeeee
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmir, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image syuthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-221 2015.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL https: http:::///pp/piie../// het./Forum:11
12

---

Published as a conference paper at ICLR 2026
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIG- GRAPH Asia 2024 Conference Papers, pp. 1-11, 2024a.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2024b.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open lcage-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 35:25278-25294, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Unsplash. Unsplash full dataset 1.3.0, 2025, URL htttttss////p...e.. com/ /ata
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv:1500 506.00000000 03365, 2015.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In International Conference Learning Representations, 2022a. URL https:
/ T/openreview. net httt//orrummldd-d/yyyee.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022b. ISSN 2835-8856.
htwpe/eweweeeeeeeeeeeoooeeeeeeee net forum? dereiii===FFFyHHne
Featured Certification.
Paul Zador. Asymptotic quantization emor of continuous signals and the quantization dimension. IEEE Transactions on Information Theory, 2892):139-149, 1982.
Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18467-18476, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Chuanxia Zheng, Tung-Long Vuong, Jianfci Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 2022.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vq-gan to 100,000 with a utilization rate of 99%. Advances in Neural Information P Processing Systems, 37:12612- 12635, 2024.
13

---

Published as a conference paper at ICLR 2026
Table 4: Training Detail Summary.
Model
Training Data
Image Resolution
Training Sleps
Size
Learning
PQGAN (All)
ImageNet
256 256
IM
20
4.5 x 10-6
PQGAN Tuning
ImageNet
768 x 768
200k
15
4.5 x 10
PQSDD Precise
Ureppash-FFi Full
384 x 384
20k
32
8 10-5
Unsplast--uul
7G8 768
800k
1x
Unsplash-Full
708 x 768
200k
32
2 10
PQSD-Quick
Unsplash-Full
384 384
20k
8x 10
Unsplash Full
768 x 768
800k
32
8 10
Unsplash-Full
768 x 768
200k
2 10
PQSD-HR
Unsplash-Ful Full
384 x 384
20k
32
8 10
Unsplash-Full
768 x 768
400k
S
UnsplasttFuul Full
1024 x 1024
400k
32
80
Unsplash-Full
1536 x 1536
200k
32
2 10-3
A TECHNICAL DETAILS
In the following, we describe additional training details for the training of all our PQGAN configu- rations and the adaptation of Stable Diffusion Rombach et al. (2022) to the PQGAN image represen- tation space. Code, weights and models will be made available at GitHub for training and inference. We use the VQGAN Esser et al. (2021) framework, which is released under the MIT License. We modify StableDiffusion2.) |Rombach et al. (2022). which released under the CreativeML Open RAIL++-M License.
A.1 PQGAN TRAINING
All configurations of our PQGAN are trained within the VQGAN Esser et al. (2021) framework, using the original architecture and training schedule to preserve the purity of the comparison for the quantisation mechanism with product quantisation. All configurations are trained on ImageNet Rus- sakovsky et al. (2015) images of size 256 x 256 for one million training steps with an overall batch size of 20 using 4x A100 GPUs These models used for all comparison with kfat-oof-theaar For downstream applications like latent image representation for generative diffusion models, we further fine-tune the best configurations according PSNR, CMMD Jayasumana et al. (2024) and Jayasumaan LPIPS Zhang et al. (018). The best configuration for the used downscaling factors 8 and F 16 are:
FF = 16)
K = 128,
d = 256,
S = 128,
(4)
(F- 8)
K 512,
d = 128,
S 64,
with codebook sizes K, latent dimension d and number of sub-domains S. These two confi configurations are further fine-tuned for 200k training steps with an overall batch size of 15 on 3xH100 GPUs on ImageNet images with the increased resolution of 768 x 768.
A.2 LATENT ADAPTATION TRAINING
We adapt Stable Diffusion 2.1 Rombach et al. 2022) (SDv2.1) to the latent image representations of our PQGAN using different compression factors
and F 16. As training data, we use the Unsplash-Full dataset (Unsplash) 2025) consisting of 6.5 million high quality images. For the SDv2.1 generator, the input and output dimension increased from the new latent dimension (128 for F = 8 and 256 for F = 16) by increasing the dimension of the first and last convolution layer. The new parameters are initialised by copying the previous pre-trained weights. First, only the input and output layers are trained while the rest of the diffusion model is kept frozen to not disturb the pre-trained weights of the generative model. This phase goes on for 20k training steps with images of resolution 512 x 512, an overall batch size of 32 over H200 GPUs and learning rate of x 10 10-5 We then unfreeze the model and train for 800k more steps with a resolution of 768 x 768. For PQDS- Precise and POSD-Quick, we keep training with 768 x 768 images for the remaining 200k steps, while we increase the resolution every 400k steps for our PQSD-HR model. A summary is shown in Table
14

---

Published as a conference paper at ICLR 2026
B CODEBOOK MATCHING SPEED
S
S d/8
S d/4
S d/2
S
175
Reconstriction Walltime [soo]
0.18
Quantistionn Overheed [%)
125
100
0.19
0.08
150
25C
30"
Namber Splits
Codebook
Figure 5: Reconstruction time and quantisation overhead comparison for product quantisation figurations.
In Figure 5 we show the walltime and the quantisation overhead for the reconstruction an image for different product quantised configurations. We measure averaged times 1000 reconstruc- tions on a A100 GPU. Each marker represents a configuration, with colour indicating the number of subspaces S. Marker sizes reflect latent dimensionality d. The quantisation overhead Figure calculated as the ratio of the codebook matching time and the reconstruction time without quanti- sation. The codebook matching time depends mainly the number of splits and is mostly indepen- dent of the actual codebook size to 16 384 entries. The best performing configurations emerged to have d 128 latent dimension with S 64 subspaces and require therefore around 50% overhead for the codebook matching during the reconstruction process.
C PERFORMANCE TRENDS FOR PQ CONFIGURATIONS
In Figure 6 we show all trends for different PQ configurations with regards to PSNR, FID Heusel al. (2017). CMMD Jayasumana et al. (2024) and LPIPS Zhang al. (2018). Each marker rep- resents a configuration with colour indicating the number of subspaces S. In the left plot, marker size reflects codebook size K; in the middle and right plots, reflects latent dimensionality d. All metrics follow the same trend reported for FID in the main paper: product quantised representations with high factorisation consistently achieve the strongest performance.
D GENERATIVE TRAINING FROM SCRATCH
Albeit the focus of the generative part of our work the reparametrisation a pre-trained generative model towards new latent representation, we also train and show results for training a diffusion model from scratch on the PQGAN representation. We train a latent diffusion model (Rombach) et al. 2022) with 400M parameters ImageNet for 2M training steps and batch size of 64 with the latent representation of StableDiifusion2. VAE and our PQGAN representations with the down- sizing factors 8 and 16. The results are summarised in Table Both PQGAN representation result in better downstream generation performance in terms of FID (Heusel et al. 2017
E RECONSTRUCTION RESULTS
We reiterate the finding that current latent representations of in produucctiion text-to-image generation models like 1tafleDiffusion2..1 have clear limitations when it comes high-resolution image rep- resentation. Figure] 7 we show additional examples of reconstructed images for our quantised POGAN image representations that we use for the task of high resolution image generation. We
15

---

Published as a conference paper at ICLR 2026
S=1
5
S- d/4
d/2
PSNR Latent Dimansion
PSMR Number Sub-Demaing
PSMR Codebook Sizes
28
28
28
25
822
19
19
16
10'
10'
16 10"
10
10"
16
10a
10
Latent Dimension d
Number of Subspaces S
Codebook Size K
(a) Trends respect to PSNR.
10a
vs Latent Dimension
vs Number of Sub-Domains
vs Codebook Sizes
10a
10a
10
10'
10a
10
10"
10"
10
Latent Dimension
Number of Subspaces S
Codebook Size K
(b) Trends with respect to FID.
CMMD vs Latent Dimension
CMMD vs Number Sub- all-Domains
CMMD vs Codebook Sizes
10
109
10
CMIM
CHMD
10
10-1
10
10a
10a
10F
10c
10"
10a
10a
Latent Dimension
Number of Subspaces S
Codebook Size K
(c) Trends with respect CMMD
20
Dimension
10a
Sub D-Dmaains
LPIPS
10-
10-1
10
102
10"
10.
10'
107
10'
Latent Dimension d
Number of SSusspaces S
Codebook Size K
(d) Trends with respect to LPIPS.
Figure 6: Quantitative analysis of product quantisation configurations. Performance is measured across latent dimensionality. number of subspaces S. and codebook size K. All metrics show the same trend with regard configuration. Plots for FID, CMMD and LPIPS are on log-log scale. Plots for PSNR are on semi-log scale as PSNR is already in dB.
Table 5: Quantitative results for training latent diffusion model (Rombach et al. 2022) from scratch on the latent representations of the StableDiifusion2.) VAE and PQGAN at different downsampling factors F on ImageNet 256 x 256 (Russakovsky et al. 2015). The generative FID (gFID) is evalu- ated on 5k samples and the ImageNet validation set.
Latent Model
F
Generator
Batch Size
Training Steps
rFID.
gFID.
SDv2.1-VAR
8
LDM
64
2M
0.75
27.21
PQGAN (Ours)
8
LDM
64
2M
0.036
22.66
16
LDM
64
2M
0.41
21.08
16

---

Published as a conference paper at ICLR 2026
again compare with the unconstrained reconstruction from the representation provided by SDv2.1 VAE in StableDiffusion2.1. The SDv2.1 1122.1VVA VAE fails to faithfully preserve fine- ime-grained image details, often producing blurred reconstructions or hallucinatory artefacts. In contrast, PQGAN with the same downsampling factor (F 8) yields reconstructions that are visually near-indistinguishable from the original image. Even F 16, PQGAN retains high level of structural and perceptual fidelity, exhibiting only minor deviations while still outperforming SDv2.1-VAE, despite using spatially significantly more compressed representation.
F ADDITIONAL QUALITATIVE RESULTS
We include additional qualitative examples from our PQSD variants further reinforce the effec- tiveness of our latent representation for large-scale text-to-image diffusion models. Figure shows samples from our POSD-Precise model, using the 768 768 and 768 x 1536. Figures 9 TO show samples for our PQSD-HR and PQSI--uicck adapted models, using the F 16 compression space, for resolutions of 768 768 to 1536 3072.
shows compressed space, for resolutions of
17

---

Published as a conference paper at ICLR 2026
Input
PQGAN (F = 8)
POGAN (F 16)
SDv2.1 VAE (F = 8)
CB
GB
C8
N
Dhan CapT LLA
CAOL ILA
CAOL ILA
CAOL ILA
Figure 7: Reconstruction results for quantised PQGANs the continuous VAE in StableDif- fusion2.1 Rombach et al.) (2022). Even with stronger downsizing F. the reconstructions from our representation space show higher fidelity. The representation space of StableDiffusion2.1 has trou- ble preserving information for high resolutions (1536 x 1536, top image) but also loses information
for its native resolution (768 x 768, rest).
18

---

Published as a conference paper at ICLR 2026
Figure 8: Additional samples from PQSD-Preciss model adapted operate F
product quantised image representation space. Samples have the resolutions 768 x 768 and 768 x 1536.
19

---

Published as a conference paper at ICLR 2026
Figure 9: Additional samples from our PQSD-HR model (first two rows) and our PQSD-Quick model (last two rows) adapted to operate on our F = 16 product quantised image representation space. Samples have the resolutions 768 768 and 1536 x 1536.
20

---

Published as a conference paper at ICLR 2026
Figure 10: Additional samples from our PQSD-HR model (top three images) and our PQSD-Quick
model (bottom-right three images) adapted to operate on our F = 16 product quantised image
representation space. Samples have the resolutions 768 x 768, 1536 1 1536, 1536 x 768, 3072 x 1536 and 3072 x 1536.
21


Reviews:
[{"review_id": "9G6aJLsUiW", "reviewer": "Reviewer_Nh49", "summary": "This paper introduces PQGAN, a novel quantised image autoencoder that utilizes Product Quantisation (PQ) to achieve state-of-the-art fidelity for latent image representations, especially for high-quality image synthesis. PQGAN is integrated into pre-trained diffusion models, like Stable Diffusionl, which provides significantly faster and more compute-efficient generation, and improve the output resolution. \nThe paper also reports difference behavior of VQ and PQ with respect to embedding dimension.", "questions": "From Table 1, I understand that the big improvement in PSNR comes from using a resolution of 32x32, while the 16x16 patch size is only offering a PSNR of 28.3. \nHave you tried larger patch sizes? \n\nSimilarly, RVQ is not evaluated in the context of compression, and only reported with patch size of 8x8. So I wonder if the better performance of PQ in this context is simply due to a better hyper-parameter choice of the patch size. Have you tried replacing PQ by a RVQ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "PQGAN surpasses both existing quantisation methods and continuous autoencoders in reconstruction quality", "weaknesses": "This work is not the first to consider PQ (or RQ) for image compression, which is mentioned in the paper, yet the phrasing is still misleading in some places (abstract). That's being said, the method of this paper is much better. It would be worth ablating and analyzing more why. \n\nFor the mundane reader, it would be worth citing the paper that has introduced/popularized product quantization (\"Product quantization for nearest neighbor search\"). \n\ntypo L329: benifits", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q7PD6JQUcW", "reviewer": "Reviewer_gjYk", "summary": "This paper introduces PQGAN, a novel way to quantize the latents of the auto encoders using product quantized learning. This method factorizes the high dimensional latent channel into many smaller, independent subspaces, each quantized with their own codebook. This method improve the reconstruction quality of the VAE and achieves a state-of-the-art 37.4 dB PSNR. The authors argue that the spatial resolution is the main bottleneck and through their method, they can operate at a lower spatial resolution but with a much higher channel dimension. This enables the method to generate larger resolution images or get speed up upto 4x.", "questions": "1) The paper does not provide quantitative generative metrics like FID or CLIP score to compare their with other methods. This is the main limitation and it would be good to see these scores to validate the claims in the paper. Why are these evaluations not in the paper? \n\n2. Discuss more about the tradeoff in weakness 2). What is the cut-off?\n\n3. 50% increase in the inference cost seems high. What is the increase in the training time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)  PQGAN achieves very high reconstruction fidelity, 37.4 dB PSNR, which is higher than 25.3 dB of the standard Stable Diffusion VAE and other methods.\n\n2) The paper demonstrates a novel finding the product quantization improves the reconstruction quality in VAEs\n\n3) The method can either double the output image resolution or achieving a 4x generation speedup at the same resolution, This is achieved with the same cost.", "weaknesses": "1) The paper fails to provide  quantitative comparison (e.g., FID, CLIP Score) between the generations of its adapted PQSD model and the original Stable Diffusion.\n\n2) The independent learning of codebooks might not learn complex correlations as the number of subspaces increase. In the limit it is as if sampling independently from each dimension. The tend in the paper also shows that.\n\n3) Since multiple indices are associated with the same pixel, it is incompatible with the autoregressive models. Only Flow based models and diffusion models can benefit from this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GDErQ99zqQ", "reviewer": "Reviewer_Jhoa", "summary": "This paper introduces PQGAN, which incorporates product quantisation (PQ) into the VQ-VAE framework. PQGAN achieves state-of-the-art reconstruction performance, significantly improving metrics such as PSNR, FID, LPIPS, and CMMD compared to existing methods. Furthermore, the paper demonstrates that PQGAN can be seamlessly integrated with pre-trained diffusion models, resulting in faster or higher-resolution image generation.", "questions": "- The authors are suggested to highlight the contribution of this work.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- PQGAN demonstrates significant improvements in image reconstruction compared to state-of-the-art methods.", "weaknesses": "- The novelty of the overall approach is very limited. PQ has been used for some time in the field of VQ-VAE, e.g., UniTok [a]. This paper is more like a technical report than a research paper, and it proposes few new ideas or inspirations.\n\n- The proposed latent adaptation is also very straightforward; reducing the spatial resolution of the VAE and increasing the number of channels are common techniques for reducing computation when training diffusion models. This application does not necessarily demonstrate that PQ brings any additional benefits.\n\n- The comparison with previous works is unfair, as this method uses significantly more tokens than other VQ-VAEs.\n\n[a] Unitok: A unified tokenizer for visual generation and understanding. C Ma, Y Jiang, J Wu, J Yang, X Yu, Z Yuan, B Peng, X Qi. arXiv preprint arXiv:2502.20321", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bUDiknsLXL", "reviewer": "Reviewer_rVnn", "summary": "This paper introduces PQGAN, a novel quantised autoencoder that uses product quantisation (PQ) instead of the traditional vector quantisation (VQ) used in models like VQGAN. The simple yet effective idea is to splits the vector into S subspaces, each quantized separately. This allow a codebook which is exponentially bigger than the VQ codebook, depending on the dimension of the subspaces (with one subspace only, PQ reduces to VQ). \n\nThe motivation behind PQGAN is that standard VQ suffers from training sparsity, codebook collapse, and redundancy: PQGAN addresses this by factorising each latent vector into subspaces, quantising each independently.\n\nOn ImageNet 256×256, PQGAN outperforms the  baselines and achieves high fidelity with small codebooks (K = 128–512).\n\nThe paper carries on a detailed analysis on codebook usage (in terms of Perplexity and Entropy) and metrics are evaluated against size of the codebook, number of subspaces and latent embedding size.\n\nFinally, PQGAN was integrated into Stable Diffusion 2.1, in three variants improving generation results.", "questions": "Please refer to the \"Weakness\".\n\nI think the paper would benefit from more insights (possibly theoretical) on the reasons why PO obtains such successful results.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novelty: despite being simple, the idea is novel and effective.\n\n- The paper conducts a systematic analysis of how embedding dimension, number of subspaces, and codebook size interact, covering the full spectrum between scalar and vector quantisation.\n\n- Extensive experiments on ImageNet, FFHQ, and LSUN establish both quantitative and qualitative superiority. Reported improvements are substantial and consistent.\n\n- Integration with Stable Diffusion is a useful addition to the method and is carefully validated, showing also computational benefit.\n\n- I have appreciated the analysis of codebook utilisation, with entropy and perplexity evaulations. Codebook utilization is an issue in standard VQ-VAEs \n\n- The paper is clearly written and well-structured. The presentation of results is well-designed", "weaknesses": "I think the main weakness of the paper is the substantial lack of theoretical foundation: PQ seems to be a well-established technique in signal processing and vector encoding [1], and its application to autoencoders has appeared in other context contexts (e.g., El-Nouby et al., 2023; Mentzer et al., 2020) as highlighted in the paper itself.\n\nA deeper theoretical argument for why PQ improves latent scalability and codebook utilization (rather than relying only on empirical evidence) may strengthen the paper.\n\nThe paper also claims that PQ and VQ behave “in opposite ways when scaling the embedding dimension,” but this is again presented empirically without theoretical insight.\n\n[1] Product Quantization for Nearest Neighbor Search, Jegou et al. - TPAMI 2011", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)