Finite Scalar Quantization: VQ-VAE Made Simple
Authors: Fabian Mentzer
David Minnen
Eirikur Agustsson
Michael Tschannen
Venue: ICLR 2024 poster (2024)
Keywords: representation learning
vector quantization
quantization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Finite Scalar Quantization: VQ-VAE Made Simple

Abstract: We propose to replace vector quantization (VQ) in the latent representation of VQ-VAEs
with a simple scheme termed finite scalar quantization (FSQ), where we project the VAE representation down to a few dimensions (typically less than 10).
Each dimension is quantized to a small set of fixed values, leading to an (implicit) codebook given by the product of these sets.
By appropriately choosing the number of dimensions and values each dimension can take, we obtain the same codebook size as in VQ.
On top of such discrete representations,
we can train the same models that have been trained on VQ-VAE representations. For example, autoregressive and masked transformer models for image generation, multimodal generation, and dense prediction computer vision tasks.
Concretely, we employ FSQ with MaskGIT for image generation, and with UViM for depth estimation, colorization, and panoptic segmentation.
Despite the much simpler design of FSQ, we obtain competitive performance in all these tasks.
We emphasize that FSQ does not suffer from codebook collapse and does not need the complex machinery employed in VQ (commitment losses, codebook reseeding, code splitting, entropy penalties, etc.) to learn expressive discrete representations.

FINITE SCALAR QUANTIZATION: VAE MADE SIMPLE
Fabian Mentzer', David Minnen', Eirikur Agustsson', Michael Tschannen 2. Google Research Google DeepMind
ABSTRACT
We propose to replace vector quantization (VQ) in the latent representation of VA-VAEs with a simple scheme termed finite scalar quantization (FSQ), where we project the VAE representation down to few dimensions (typically less than 10). Each dimension quantized to a small set of fixed values, leading to an (implicit) codebook given by the product of these sets. By appropriately choosing the number of dimensions and values each dimension can take, we obtain the same codebook size in VQ. On top of such discrete representations, we can train the same models that have been trained on VQ-VAE representations. For example, autoregressive and masked transformer models for image generation, multimodal generation, and dense prediction computer vision tasks. Concretely, we employ FSQ with MaskGIT for image generation, and with UViM for depth estimation, colorization, and panoptic segmentation. Despite the much simpler design of FSQ, we obtain competitive performance in all these tasks. We emphasize that FSQ does not suffer from codebook collapse and does not need the complex machinery employed VQ (commitment losses, codebook reseeding, code splitting, entropy penalties, etc.) to learn expressive discrete representations, Colab on GitHub.
INTRODUCTION
Vector quantization (VQ), initially introduced by Gray (1984), has recently seen a renaissance in the context of learning discrete representations with neural networks. Spurred by the success of VQ- VAE (Van Den Oord et al., 2017), Esser et al. (2020) and Villegas et al. (2022) showed that training an autoregressive transformer on the representations of VQ-VAE trained with GAN loss enables powerful image and video generation models, respectively. At the same time, VQ has become popular component in image (Bao al., 2021: Li et al., 2023) and audio (Baevski et al., 2019) representation learning, and is a promising building block for the next generation of multimodal large language models (Aghajanyan et al., 2022; Kim al., 2023; Aghajanyan al., 2023).
When training VAE, the goal to learn a codebook C whose elements induce a compressed, semantic representation of the input data (typically images). In the forward pass, image is en- coded into a representation z (typically sequence of feature vectors), and each vector in quantized (i.e,, replaced with) the closest vector in C. The quantization operation is not differentiable. When training a VAE with VQ in the latent representation, Van Den Oord et al. (2017) use the straight- through estimator (STE) (Bengio al., 2013), copying the gradients from the decoder input to the encoder output, resulting in gradients to the encoder. Since this still does not produce gradients for the codebook vectors, they further introduce two auxiliary losses to pull the codeword vectors towards the (unquantized) representation vectors and vice-versa.
The above formulation challenging optimize, and leads to the well-documented problem of un- derutilized codebooks (Lancucki et al., 2020; Takida et al., 2022; Dhariwal et al., 2020; Huh et al., 2023); as the size of increased, many codewords will be unused. Subsequent works aimed to im- prove this with various tricks such as reinitializing the entire codebook some codewords Dhariwal et al. (2020); Lańcucki et al. (2020), stochastic formulations Takida et al. (2022), etc. (see Sec. 2).
Significant technical contributions.

---

FSQ
VQ
NN
010,41
FSQ (left): the final encoder layer projects to d dimensions (d 3 shown). We bound each dimension of the encoder output 2 L values (L shown), and then round to integers, resulting in the quantized s, the nearest point in this hypercube. VQ (rightJ: The final encoder layer projects to d dimensions (d = 7 shown, as d is typically much larger for VQ). The resulting vector is replaced with the closest vector from the codebook, 2, by nearest neighbor lookup.
Figure 1:
Here, we are interested in simplifying the original VQ- VAF formulation (Van Den Oord et al., 2017) with the following goals: i) remove auxiliary losses, ii) achieve high codebook utilization by design, and iii) keep the functional setup the same to the extent that we obtain a drop-in replacement for VQ.
To this end, we draw inspiration from the neural compression literature, where discrete codes are typically obtained with scalar quantization, following initial work (Ballé al., 2016; Theis et al., 2017): Each (scalar) entry in the representation z is independently quantized to the nearest integer by rounding. The majority of the current compression literature uses unbounded scalar quantization, where the range of integers is not limited by the encoder, only by constraining the entropy of the representation. Other compression work relied on bounding the range of the quantizer (Mentzer al., 2018; Tschannen al., 2018; Agustsson al., 2019).
We call this approach finite scalar quantization (FSQ). The important insight is that by care- fully choosing how to bound each channel, we can get implicit codebook of (almost) any de- sired size: Consider a vector with d channels. If we map each entry 2; to L values (e.g.. via 24 + L/2 tanh(z;) followed by rounding to integers). we obtain a quantized 2, where is one of L unique possible vectors. Fig. shows FSQ for d=3, L=3, implying a codebook ((-1, 1111,111-11 (-1, 0), (-1,-1, .).... (1, 1)), where |C] L 27.
To get gradients through the rounding operation, we use the STE like VQ-VAE. Thus, using FSQ in- side an autoencoder trained with reconstruction loss, we get gradients to the encoder that force the model to spread the information into multiple quantization bins, as that reduces the reconstruction loss. As a result, we obtain a quantizer that uses all codewords without any auxiliary losses.
To the best of our knowledge, considering the (product) codebook obtained from FSQ has not been done before, neither in neural compression nor in other tasks where VQ dominant. We aim to change this by revisiting FSQ in conjunction with powerful transformers/languueee models. In summary, our contributions are:
1.
We show that FSQ can serve drop-in replacement for VQ in various architectures, for dif- ferent datasets and tasks, by applying it to MaskGIT (Chang et al., 2022) for image generation, and in UViM (Kolesnikov et al., 2022) for depth estimation, colorization, and panoptic segmen- tation. We observe a reduction of only 0.5 3% in the respective metrics, and correspondingly get highly similar visual results. We emphasize that the two model families have very differ- ent designs (convolutional vs. transformer-batt autoencoders, masked vs. fully autoregressive transformers, decoder-only vs. encoder-decoder transformers, etc.).
2.
We analyze the trade-offs for VQ FSQ, characterize the scaling behaviors w.r.t. codebook size of the two models, and analyze the representation complexity from a compression angle. We find that FSQ is able to leverage large codebooks for better reconstruction metrics, and better sample quality. The codebook usage is very high for FSQ (s-100% for most models), without relying on any auxiliary losses.
3. We show that the full generality of the VQ formulation gives little benefits method (VQ is actually worse for large codebooks C). This can be attributed to VQ being difficult to optimize, whereas FSQ can be viewed the standard VQ formulation changed such that a)
simpler FSQ
2

---

VQ
FSQ
fuo raund_stode ste(fi2))
Quantization
arg minnec ||2 ell
round
Gradients
STE
STE
Aux. Losses
Commitment, codebook, entropy loss
Tricks
EMA codebook,
-1
codebook splitting projections,... Codebook
-2
Parameters
-3 -2
Figure 2: Left: VQ made simple: comparing implementation and optimization of VQ vs. FSQ. Right: Bounding x with f. and rounding the output, shown for a single channel with L = 5.
the encoder output is bounded and b) C is fixed. We note that the (implicit) FSQ C has much smaller dimensionality vs. VQ (typically d < 10 for FSQ, vs. d 2 512 for VQ).
RELATED WORK
VQ-VAE and improvements Van Den Oord et al. (2017) introduced the initial formulation in VQ-VAE, including commitment loss and EMA for improved codebook learning. Roy el. (2018) use soft expectation maximization (EM) to train VQ-VAE. They also report success in tuning the codebook size for the target tasks. Dhariwal al. (2020) VQ-VAE for audio generation. To prevent codebook collapse, they use "random restarts", where vectors are reset to encoder outputs when their usage becomes low. They also introduce multi-scale variant of VQ. Lańcucki et al. (2020) aim to improve codebook learning by periodically reinitializing it using offline clustering algorithms. Yu al. (2021) introduce a vision transformer (ViT) based VQ-GAN. To improve learning of the quantizer, they Iz-normalize all vectors and map codes to a lower dimensional space lookup. Takida et al. (2022) propose stochastic quantization approach to avoid codebook collapse, adding Gaussian noise to the encoder output to imitate quantization, which is annealed during training. Williams et al. (2020) also explore stochastic quantizers, in addition to a hierarchical representation. Huh et al. (2023) examines challenges in training the vanilla VQ formulation. They propose various improvements, including a re-parameterization, alternating optimization, and an improved commitment loss.
VQ Alternatives Residual quantization (RVQ) has been used for image (Lee et al., 2022) and audio (Zeghidour et al., 2021) generation. There, quantized codes are refined by additionally storing (quantized) residuals. In Product quantization (PQ) (El-Nouby et al., 2022), the codebook is factored into a product of smaller codebooks. In similar spirit, there is body of literature around reducing the number of tokens output by VQ-VAEs for more efficient inference, see, e.g., Huang et al. (2023).
Neural compression Many works (Ballé al., 2016: Minnen al., 2018: Lu al., 2019; Mentzer et al., 2020; Cheng et al., 2020) rely on unbounded scalar quantization and constrain the entropy of the quantized representation to prevent spreading to all integers. Bounded scalar quantization (i.e. FSQ), has been used to represent images with high fidelity (Mentzer et al. (2018) use d-16,, L-5), and for "extreme compression" (Agustsson et al. (2019) used d=5, L-5). To the best of our knowl- edge, FSQ has not been used outside of compression. We note that the neural image compression field generally targets "high bitrate" reconstructions, and the challenge is to reduce the entropy of the complex representations, whereas in representation learning with VQ-VAE, the goal is usually the opposite: increase the entropy of a heavily constrained representation to maximally use
3 METHOD
We start with some high-level intuition. VQ defines learnable Voronoi partition in the high- dimensional latent space VQ-VAE which leads to a complex non-linear partitioning of the VQ- VAE input space (e.g., images). FSQ. by contrast, relies simple, fixed grid partition in much lower- dive-dimensional space. Intuitively this is feasible because VAEs have a relatively high model ca- pacity in typical applications (see Sec. 2), and thus the non-linearity of VQ can be "absorbed" into
3

---

encoder and decoder. that FSQ enables partitions of the VAE input space of similar complexity as VQ. We note that like VQ-VAE based approaches, we train FSQ models in two stages: We first train FSQ-VA VAE, and then transformer frozen FSQ- VAE representations.
3.1 FINITE SCALAR QUANTIZATION
Given a d-dimensional representation E IR, our goal is to quantize z to a finite set of codewords. To this end, we first apply bounding function f. and then round integers. We chose f such that each channel/entry in 2 (fund(((z)) takes one of L unique values (e.g., f: + [ [tanh(2)) Thereby, we have
C, where C is the implied codebook, given by the product of these per-channel codebook sets, with [C] T,d_ The vectors in C can simply be enumerated leading to a bijection from any to an integer in [1....,L*) Therefore, VQ can be replaced with FSQ in any neural network- related setup where VQ is commonly used, e.g. train transformers, after appropriately adapting the output and input dimension the layers before and after VQ, respectively. We generalize the above exposition to the case where the i-th channel is mapped to L; values and get [C] II[ L.
We visualize FSQ Fig. (left) and in Fig. 2. Since quantization is performed by round integers, supporting even L requires an asymmetric f. We show the general f used throughout this paper as code in App. 1. To propagate gradients throughout the round operation, we use the STE throughout, replacing the gradients with 1. In ML frameworks, this can easily be implemented via the "stop gradient" (sg) operation as round ste: x x + sg(round(z) - z).
3.2 HYPERPARAMETERS
FSQ has the following hyper-parameters the number of channels d and the number of levels per channel, L= Li..... La]. In most of our experiments, to obtain fair comparisons, we will choose target codebook sizes CC based the VQ codebooks we aim replace with FSQ. However, various configurations of and L, can approximate given |C| (i.e., any L where II. L as |C] is a candidate). We explore various configurations in our study, and find that not all choices lead optimal results. However, we found a simple heuristic that performs well in all considered tasks: Use Li 2 Ví. In Table I tabulate L for common target [C].
3.3 PARAMETER COUNT
We note that FSQ has fewer parameters than VQ, since in VQ, codebook of size |C| d is learned. For example, for typical (CI-212-4096 and d=512, this results 2M parameters, which FSQ lacks. Additionally, since for FSQ. d tends to be much smaller than for VQ (e.g., d=5 for FSQ for this [C], see Tab. 1), the final encoder layer also has fewer parameters when training FSQ. To compensate for this, explored adding more dense layers the end of the VAE encoder, resp. at the start of the decoder, but found no further gains from doing so. Thus, in all models in this paper, FSQ with the same codebook size has fewer parameters.
4 EXPERIMENTS
4.1 REVIEW OF MASKGIT AND UVIM
We start with brief review of MaskGIT (Chang et al., 2022) and UViM (?). In MaskGIT, the authors first train a (convolutional) VQ-GAN autoencoder (Esser al., 2020) for reconstruction (Stage I). They then freeze the autoencoder, and train a masked transformer BERT-style (Devlin et al., 2018) to predict the quantized representations (Stage II): Given representation é. fraction of tokens is randomly "masked out", i.e., replaced with a special MASK token. The resulting sequence EM fed to a transformer in addition class token, and the transformer predicts a distribution for
Target Size |C]
2
210
22
2
216 [8, 8, 5,5,5
Proposed
[8, 6, 5]
88 5,5,5,5]
[7,5,5,5,5)
8 888,8,6,3
Table
Recommended sets of FSQ levels L to approximately match given codebook size (C).
4

---

Rcconstruction
Sampling
15
P5Q
vQ
14
13
12
11
6
2a
2
2
244
2
2
2.4
2.4
2
2
Codebosk Stze
Codebook Size
Usage
d) Compression [bits]
215
16
FSQ
VQ
14
23
Maximum
Uniform
12
24
10
21
22 2
2:00 2
2
2
2
2
2
2.2
2
2
Size
Figure 3: Characteristics and trade-offs for VQ and FSQ for 128 x 128 ImageNet. We see that Reconstruction FID correlates with codebook size for FSQ, and improves we scale the codebook size. FSQ gets better Sampling FID and higher codebook usage for codebook size exceeding 2 while the metries start deteriorating for VQ.
each masked token. During inference, initially only MASK tokens along with the class token fed to the transformer. Then, some of the token locations are selected based on prediction confidence, and corresponding tokens are sampled (see (Chang al., 2022, Sec 3.2)). These tokens are used to replace mask tokens at the input, and the model is ran again, until all input tokens have been uncovered.
UViM (Kolesnikov et al., 2022) is general architecture to tackle various (dense) prediction tasks in computer vision. In the first stage transformer-be based VQ-VAE trained to model the label space of the target task. Optionally, both the VQ-VAE encoder and decoder can rely on the task input (RGB image for depth estimation and segmentation, grayscale image for colorization) as side information or "context", which was found beneficial for some tasks. In the second stage, encoder-decoder transformer trained to predict the dense label as quantized tokens produced by the VQ-VAE en- coder, given the task input. For inference, a code is sampled autoregressively using the transformer conditioned on the input and then fed to the VQ-VAR decoder. The architecture is shared for the three tasks, but different weights are learned for each task.
4.2 CHARACTERISTICS AND TRADE-OFFS FOR VQ AND FSQ REPRESENTATIONS
We with study, where we train MaskGIT models on lower resolution 128 128 ImageNet images and for shorter time compared to the paper Chang et al. (2022) (100 epochs for Stage I, 200 epochs for Stage Please see Appendix A.4.1 for more hyperparameters) This allows to sweep the codebook size and other hyperparameters, For VQ, we use the auxiliary entropy loss from MaskGIT, that aims to increase the entropy of the codebook (to increase utilization). We only sweep the codebook size. For FSQ, we explore various d and L; match these codebook sizes.
We track the following metrics: toencoder when the 50k validation images are fed through the quantized autoencoder. This is the FID that the Stage II transformer would achieve if would perfectly model the data. We use the well established ADM TensorFlow Suite (Dhariwal & Nichol, 2023), which computes FID from 50k reconstructions w.r.t. the training set. Codebook Usage: The fraction of the codewords that are used least once when encoding the validation set.
Reconstruction FID, the FID obtained by the GAN-trained au-
5

---

Model
Source
CFG
Sampling FID
Precision
Recall
Usaget
MaskGIT (VQ)
Ours
0.1
4.509
0.860
0,465
81%
MaskGIT (FSQ)
Ours
0.2
4.534
0.864
0.453
100%
MaskGIT (VQ)
GitHub
4.916
0.836
0,489
ADM (Dhariwal &
Nichol, 2021)
1.5
4.59
0.83
0.52
0.90
5.6
0.85
5.4
Sampling FID
Precision
0.80
5.2
5.0
0.75
MaskGIT (FSQ)
MaskGIT (VQ)
4.8
0.70
MaskGIT (github)
ADM
4.6
0.65
0.45
0.50
0.55
0.60
0.00
0.05
0.10
0.15
0.20
0.25
Recall
CFG Weight a (inference)
Figure 4: MASKGIT results on ImageNet 256. Top: We show the best classifier-free guidance (CFG) setting for each MaskGIT model. As reference, we show the well established diffusion based ADM model (Dhariwal & Nichol, 2021). Bottom Left: Precision vs. Recall for various CFG weights. Bottom Right: Sampling FID for various CFG weights. We show ADM horizontal line. because the CFG weight 1.5 used for ADM is not comparable with our in absolute terms. TWe use the ADM TensorFlow Suite to evaluate all shown models, see text.
With the transformer trained in Stage II, we additionally report Sampling FID, the FID obtained when decoding representations sampled colas-coodittinnally)) with the transformer. We addition- ally propose studying Compression Cost as proxy for how hard it is to model the discrete distri- bution underlying the representations (ie.., modelling complexity): Note that any transformer that predicts a distribution over discrete codes can be used to losslessly compress the corresponding rep- resentation. For masked transformers, the only requirement that gradually uncovers the input. Using such schedule, we can compress any 2 to bits, by pairing the transformer outputs with entropy coding. We use the deterministic masking schedule employed in M2T (Mentzer et al., 2023) and refer to Section in that work for further details on the theory.
deterministic masking schedule,
4.3 MASKGIT
We train MaskGIT models on ImageNet 256 based on the public GitHub code, training Stage for 1M steps with batch size 512, and Stage for 2.5M steps with batch size 256. For inference, use 12 steps with the cosine to sample an image. Initial experiments with the public code showed slight instability in the Stage II transformer loss, which we were able to mitigate by lower bounding the minimal masking ratio used during training. Please see Appendix A.4.3 for further details and hyper parameters. We train VQ with codebook size 1024 (10 bits) and the entropy loss, in the published model. For FSQ, we use L = [8, 5, 5, 5] as suggested in Tab. 1.
Following the paper, we report Sampling FID as well as Precision and Recall (Sajjadi et al., 2018) to assess the quality of the generative model. Additionally, we also report Codebook usage. We again use the well-established ADM TensorFlow Suite, leading to an FFD--r-FID-trai of 4.916 for the official checkpoint published in the MaskGIT GitHub, 6.19 reported in the MaskGIT paper.
Early experiments showed that FSQ lands at a different Precision & Recall point compared VQ (FSQ had higher recall, lower precision). Inspired by the diffusion literature, thus add classifier free guidance (CFG) (Ho & Salimans, 2022) to MaskGIT: During training, we replace 10% of the class labels with the MASK token to let the model learn the unconditional distribution. During inference, we interpolate logits: Let I, be the logits obtained when conditioning on the class label and be unconditional logits. During inference, compute logits I' + aell. l)),

---

Figure 5: Non-cherry-picked samples from our FSQ (top) and VQ (bottom) MaskGIT models for imagenet classes (330, 320, 510, 454). We show two samples per model per category. Both models get very comparable sample quality, reflected by the metries in Fig. 4.
where is the CFG inference weight. Intuitively, this pulls the predicted distribution towards the unconditional one. We emphasize that this has previously been explored the context of masked transformers, egg. by (Chang et al., 2023, Sec. 2.7).
4.4
UVIM
We retrain the public UViM GitHub code for all three tasks (panoptic segmentation, depth estima- tion, colorization). As in the paper, we train each Stage II transformer times, and report averaged metrics. For VQ, we use 4096 codewords (12 bits), and we use the codebook splitting (described below), as the published results. We obtain similar metrics to what is reported in the GitHub repo, Sec. 5. For FSQ,
L 5, 5, 5, 5) from Tab. 1.
Following the UViM paper. we report panoptic quality (PQ) for panoptic segmentation, RMSE for depth estimation, and FID-5k for colorization. For all tasks, by the UViM github repository. We refer to (Kolesnikov et al., 2022) for more details on these tasks and corresponding data sets.
the evaluation suite provided
We ablate the effect of VAE context input (i.e., the RGB image, see above) on the performance of VQ and FSQ in the panoptic segmentation task. Further, we investigate the codebook splitting employed by UViM to avoid unused codewords in VQ- "Q-VAE. Specifically, they adopt the algorithm from Linde al. (1980), where throughout training, unused vectors are detected. These are then replaced by splitting most frequently used embeddings into two new embeddings, adding noise to each. Since we observe training instabilities when deactivating codebook splitting in the panoptic segmentation task, we use the depth estimation task for this ablation.
5 RESULTS
5.1 TRADEOFF STUDY
In Fig. 3 we show the results for the trade-off study. On the x-axis, we always show the codebook size [C), representing the maximal amount of information the codebook can store. We observe the following:
Codebook size correlates with Revonstruction FID for FSQ In Fig. a), we see that as increase the codebook size, the reconstruction FID for FSQ keeps improving. This is what one would expect from a compression perspective: as we have more bits store information, we should get better reconstruction metrics. However, (despite entropy regularization of the codes), and reconstruction FID achieves a minimum at 2 codes, co-inciding with the point where the codebook usage starts decreasing (ef. Fig. 3 c)). We note that for low codebook sizes (Fig. a), left), VQ marginally outperforms FSQ, likely owning to the its more expressive nature (see Contribution in the Section 1).
that VQ struggles with utilizing large codebooks
FSQ gets better Sampling FID A similar picture emerges in Fig. 3 b). where we see that the better Stage behavior of FSQ translates to better Sampling FID scale the codebook.

---

NYU Depth v2
Source
RMSE'
Codebook Usage
UViM (VQ)
Ours
0.468 0.012
99%
UViM (FSQ)
Ours
0.473 0.012
99%
UViM (VQ without splitting)
Ours
0.490 1 0.0037
0.78%
UViM (VQ)
GitHub
0.463
DenseDepth (Alhashim & Wonka, 2018)
0.465
COCO Panoptic
Source
PQ f
Codebook Usage
UViM (VQ)
Ours
43.4 +: 0.0008
100%
UViM (FSQ)
Ours
43.2 + 0.0014
100%
UViM (VQ without context)
Ours
39.0 + 0.0023
99%
UViM (FSQ without context)
Ours
40.2 + 0.0019
99%
UViM (VQ)
GitHub
43.1
DETR-RIA R101 (Carion et al., 2020)
45.1
ImageNet Colorization
Source
FID-5k 4
Codebook Usage
UViM (VQ)
Ours
16.90 0.056
100%
UVIM (FSQ)
Ours
17.55 + 0.057
100%
UViM (VQ)
Github
16.99 0.057
ColTran (Kumar et al., 2021)
19.37
Table UVIM results for the three tasks. For each, we show results in the corresponding metric averaged over three runs with std. dev. (as in UViM). We show the numbers reported by the reference GitHub repository, as well as one well established baseline per task. For our models, we show Codebook usage. For Depth Estimation, we train ablation where we do not employ the codebook splitting in VQ. Overall, FSQ obtains competitive but marginally worse results on all tasks. We use the UViM GitHub evaluation suite.
FSQ gets high codebook usage In Fig. 3 c) we see that FSQ uses almost all codewords for codebook size of 214=161; without employing any tricks. the same time, VQ starts dropping below 50% usage for codebooks larger than 2 and is not able to utilize more than 20 codewords for larger codebooks. contrast, for FSQ usage continues growing with more than 2 codewords utilized for codebook of size 26
Diminishing gains from codebook scaling One might wonder whether just scaling the codebook size more would lead to ever lower sampling FID. However, as shown in Fig. d), the compression cost of the representation keeps increasing. This indicates that the quantized representations get more complex model for the transformer. Indeed, we see in Fig. 3 b) that the Sampling FID saturates for FSQ starting when using about 212 codewords. We note that in general, for this task, the discrete distribution underlying the FSQ representations are slightly harder model (as seen by the higher Compression Cost when training the same transformer on different VAEs, Fig. 3 d)). We also note how the Compression Cost for VQ correlates with the codebook usage: when the usage drops, the code becomes easier to model again. Similarly, within model group (i.e., considering only FSQ or VQ models), the compression cost anti-correlated with sampling FID.
Selecting the number of levels per channel L In Appendix 1.4.1 we also show the effect of different on the Sampling FID. We find that T, < leads to subpar performance.
5.2 MASKGIT
In Fig. we show the metrics MaskGIT on 256x256 ImageNet. We sweep the CFG weight for both VQ and FSQ. The following be observed:
FSQ and VQ achieve comparable metrics and visual results Fig. shows that both quantizers achieve very comparable FID, as well precision and recall. To put the numbers in context, we show the well established diffusion based ADM model (Dhariwal Nichol, 2021). When inspect- that both quantizers lead to qualitatively similar samples.
ing the visual results in Fig. 5, Motivated by the tradeoff study (sec. 5.1), we explored a larger codebook for these models, but did not observe further gains.
8

---

Ingut
Ground Truth
FSQ
VQ
VQ w/o code splisting
Figure 6:
Samples from UViM for the depth estimation task. Other tasks in Appendix A.2. We observe that VQ and FSQ lead to comparable samples. VQ without splitting leads to jagged edges.
Semanties is commonly argued in the literature that the codebook in VAES and VQ-GANs learns semantically meaningful codes. Yet, we see that we get similar samples from both VQ and FSQ, even though FSQ does not learn an explicit codebook (and thus has less parameters). We performed small study to see whether either representation is more semantically meaningful than the other, shown in Appendix A.3. We found no evidence that a particular code represents a fixed visual concept in either quantizer. Indeed, both behave very similary in that study.
Precision-Recct trade-offs Note that precision is a measure for the "quality" of the samples, while recall measures the proportion of the true distribution that is covered by the samples (Sajjadi et al., 2018). When we sweep the CFG weight a during inference, we obtain models that cover a very similar space in Precision & Recall (bottom, left), and that obtain very similar minimal FID (bottom, right).
5.3
UVIM
Table 2 shows the results for the three tasks trained with UViM along with some baselines from the literature.
FSQ competitive with VQ on all tasks We can see that across tasks, FSQ obtains compet- itive metrics compared to VQ. This is also reflected in the visual results shown in Fig. 6 (for depth estimation) and App. A.2 (for panoptic segementation and colorization).
FSQ performs better in absence of side information (context) Table 2 also shows removing the VAE context in UViM (panoptic segmentation), i.e., removing the original RGB image input to the VAE encoder and decoder (see Sec. 4.1). In this setting, both the FSQ and VQ-based models obtain lower PQ numbers than with context, but the performance of the FSQ-based model degrades less.
FSQ does not rely codebook splitting We explore disabling the codebook splitting the NYU Depth task, and we observe signficantly worse RMSE, while Codebook usage drops by more than orders of magnitude to 0.78% In the predictions, (right most column). At the same time, FSQ does not rely on any auxiliary algorithms to obtain 99% codebook usage.
observe jagged edges, Fig. 6
CONCLUSION
In this work, we showed that we can replace the vector quantizer in VQ-VAEs with simple scalar quantization scheme, where the representation is projected to very few dimensions which are bounded and rounded. We studied and compared the behavior of FSQ and VQ as function of the codebook size and observed that FSQ achieves much better codebook utilization for large code- book sizes. Despite the much more constrained setup, we were able to obtain comparable metrics on image generation with MaskGIT, and dense computer vision tasks with UViM. We hope future work will explore FSQ in even more applications.

---

Reproducibility
We refer to Section A.1 for reference code.
Ethics Statement This work proposes a drop-in replacement for VQ. and can thus be applied in all domains where VQ is used. A domain where w.r.. biases has to be taken is generative models. However, no new ethical concern arises from our method that would not be a concern for VQ-based methods.
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multi- modal model of the internet. arXiv preprint arXiv:2201.0777 077220000 2022.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan,, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zeutlemoyer. Scaling laws for gen- erative mixed-modal language models. arXiv preprint arXiv:2301.03728 2023.
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Gener- ative adversarial networks for extreme learned image compression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 221-2331 2019.
Ibraheem Alhashim and Peter Wonka. High quality monocular depth estimation via transfer learn- ing. arXiv preprint 181iv:1812.1 11941, 2018.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2019.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. arXiv preprint arxiv 2016.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers.
In International Conference on Learning Representations, 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint a 3o..xv:3308.33333 3432, 2013.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Nummy programs, 2018. URL http:
//github. com/ google/jax
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229, Springer, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 11315-11255 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- cration via masked generative transformers. arXiv preprint arXiv:2301. 007044 2023.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jíro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7939-7948. 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 181000088 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
10

---

Prafulla Dhariwal and Alexander Nichol. ADM TensorFlow Suite, 2023. URL https://
github. http:///paa//oeaa...oaaaaaaaaa//
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint vrXiv:2005.00341 2020.
Alaaeldin El-Nouby, Matthew Muckley, Karen Ullrich, Ivan Laptev, Jakob Verbeek, and Hervé Jégou. Image compression with product quantized masked image modeling. arXiv preprint arXiv:22/2.073372 2022.
Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-resolution image synthesis. 2021 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12868-12878, 2020.
Robert Gray. Vector quantization. IEEE Assp Magazine, 1(2):4299, 1984.
Jonathan Ho and Tim Salimans. arXiv:2207. /2598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Mengqi Huang, Zhendong Mao, Quan Wang, and Yongdong Zhang. Not all image regions matter: Masked vector quantization for autoregressive image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2002-2011, 2023.
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- through estimator: Overcoming optimization challenges vector quantized networks. arXiv preprint wrXiv:2305.08822, 2023.
Sungwoong Kim, Daejin Jo, Donghoon Lee, and Jongmin Kim. Magvlt: Masked generative vision- and-language transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23338-23348, 2023.
Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. Ad- vances in Neural Information Processing Systems, 15:26295-26300 2022.
Manoj Kumar, Dirk Weissenbom, and Nal Kalchbrenner. Colorization transformer. arXiv preprint a1X:v:2102.04432, 2021.
Adrian Łańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alumäe, and Antoine Laurent. Robust training of vector quan- tized bottleneck models. 2020 International Joint Conference on Neural Networks (IJCNN),
pp. 1-7. IEEE, 2020.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532. 2022.
Tianhong Li. Huiwen Chang, Shlok Mishra, Han Zhang. Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2142-2152, 2023.
Yoseph Linde, Andres Buzo, and Robert Gray. An algorithm for vector quantizer design. IEEE Transactions on communications, 28(1):84 95, 1980.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11006-11015, 2019.
Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Condi- tional probability models for deep image compression. In Proceedings of the IEEE Conference 2018.
Computer Vision and Pattern Recognition, pp.
Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gen- erative image compression. Advances in Neural Information Processing Systems, 33:11913: 1913- 11924, 2020.
11

---

Fabian Mentzer, Eirikur Agustsson, and Michael Tschannen. M2t: Masking transformers twice for faster decoding. arXiv preprint arXiv:2304.03313, 2023.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv: 1805. 1063, 2018.
Mchdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chieh-Hsin Lai, Junki Ohmura, Toshimitsu Ue- saka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. Sq-vae: Vari- ational bayes on discrete representation with self-lnnealed stochastic quantization. arXiv preprint arXiv:2205.07547, 2022.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arXiv preprint arXiv: 7703.00395, 2017.
Michael Tschannen, Eirikur Agustsson, and Mario Lucic. Deep generative models for distribution- preserving lossy compression. Advances in Neural Information Processing Systems, 31, 2018.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in
neural information processing systems, 30, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernnn Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2022.
Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hier- archical quantized autoencoders. Advances in Neural Information Processing Systems, 33:4524- 4535, 2020.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 10.04627, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
12


Reviews:
[{"review_id": "yawQGjMcSl", "reviewer": "Reviewer_Hp3C", "summary": "This work considers replacing the quantization layer of vector quantized variational autoencoders (VQ-VAE) with a scalar quantization strategy. The goal of VQ-VAE is discrete representation learning, i.e., we want to embed a datapoint in the latent space using an encoder network, and we want the representation in the latent space to belong to a discrete set of finite values (referred to as the codebook). This codebook is learnt by jointly training the encoder and decoder network.\n\nTraditional VQ-VAE approaches consists of a quantization layer, in which the codebook (or the quantization points) is done using K-means clustering or exponential moving average. These approaches have some issues such as codebook collapse, in which the learnt codebook is a very small set of points compared to the latent space. The strategy proposed in this paper replaces the VQ layer by coordinate wise scalar quantization. This drastically reduces the number of hyper-parameters and the codebook size, while still maintaining the reconstruction quality. Quantization is done by first passing it through a bounding function such as $\\mathrm{tanh}$ which bounds the dynamic range of the entries to be quantized. This is followed by quantizing the $i^{\\mathrm{th}}$ coordinate of the latent representation to one out of $L_i$ values. Here, $i$ varies from $1$ to $d$, where $d$ is the number of channels. Numerical evaluations validate this claim.\n\nPlease correct me if I am mistaken in my understanding of the contributions of the paper. I am more than happy to rectify my review if that's the case.", "questions": "I have no critical questions right now, expect for the following.\n\nHow does the choice of the squeezing / bounding function, i.e., $\\mathrm{tanh}$ affect the performance?\n\nI'd be happy to edit my review post the discussion period.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "VQ-VAE has been successful in obtaining compact latent representations. However, the VQ layer has some issues such as code under-utilization as mentioned in the summary. This paper proposes a simple strategy that imposes the cubic lattice structure while obtaining the quantization codebook. This leads to much fewer parameters to store.\n\nThe simplicity of the proposed approach is quite commendable. It is surprising that this hasn't already been studied before. Even though quantization is a non-linear operation, the VAE network can be trained via back-propagation using the stop gradient estimator.", "weaknesses": "The idea is well described in the paper. My only concern is the novelty of the contribution of the proposed approach. It is surprisingly simple, which raises my question as to what are the limitations of FSQ, and if it has not been explored before, especially since VAEs have been around for a while. \n\nCan we have any guarantees as to why FSQ performs as good as VQ, even after imposing strict structural constraints? Is there anything particularly special about the tasks considered in the numerical simulations, or can we expect FSQ to perform at par for any application of VAE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "egDnYJUshP", "reviewer": "Reviewer_HYXB", "summary": "This paper proposes a new type of discrete tokens used in VQ-VAE named finite scalar quantization (FSQ). The dimension of the VQ-VAE representation is decreased to a low number and each dimension can only take a few scalars. With this design, FSQ is able to achieve the same size of the codebook used in original VQ-VAE with far less parameters. In addition, FSQ does not require complex design like entropy penalty as well. By using these FSQ representation, the models using VQ-VAE can also be trained with FSQ, such as image generation, multimodal generation, and dense prediction computer vision tasks. Experiment results show that FSQ has competitive performance with original VQ-VAE and prove the expressive discrete representations learned by FSQ.", "questions": "Since the paper does not contain the training process of FSQ, I am a little confused about the setting. Here are some questions:\n\n* Q1: As I mentioned in the weakness part, is FSQ used in MaskGIT and UViM trained from scratch? If yes, what is the model architecture and the other hyperparameters like training epochs? And what is the reconstruction and generation ability of FSQ used as a VAE?\n\n* Q2: There is another concurrent work named Lookup Free Quantization [1], which also aims to compress the codebook. Their method directly let the dimension of codebook features become zero, which can also support a larger vocabulary size. Can the authors compare the proposed method with theirs?\n\nI will be very happy to raise my score if these concerns are addresed.\n\n[1] Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation, arxiv 2310.05737", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* S1: The method is effective and easy to implement, since no extra complex design is needed.\n* S2: The writing of the FSQ design is easy to follow.\n* S3: The experiments are extensive. The authors demonstrate the efficiency of the proposed method under various settings.", "weaknesses": "* W1: The relation between VQ-VAE and FSQ is confusing. Specifically, I do not understand how the FSQ model is trained. Is the FSQ model trained from scratch using the VAE architecture, or is the FSQ model transformed from a pretrained VQ-VAE model in some way. Since MaskGIT is a two stage method, where the first stage is to train a discrete tokenizer, I will regard FSQ as a model trained from scratch.\n* W2: The principle of desigining the shape of the codebook is not given. It seems that the number of levels per channel of the recommended shape is uniform and larger than 5. But there is not a intuitive explanation on why.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d3KnfBdeBk", "reviewer": "Reviewer_fsTh", "summary": "The paper proposes Finite Scalar Quantization (FSQ) to replace the popular VQ-VAE.\nInstead of performing cumbersome VQ-VAE optimizations (multiple losses, handling codebook underutilization, etc.), FSQ performs a simple quantization (rounding) of the learned embedding, optimized via the standard STE.\nThe method is able to reach FSQ performance on several tasks.", "questions": "1. There is an obvious difference between the FSQ and VQ-VAE functionals (rounding vs L2 distance). It would be interesting to see the difference in the quality of the representations (e.g., same class samples should be close(r with FSQ?) in their representations) and not only regarding the optimized metrics.\n\n2. Following 1., it is not clear how the codebook is seen as implicit and not simply defined by the hypercube's samples.\n\n3. How do you explain that extending the model capacity (Sec 3.3) does not result in improvement, especially since the method is still worse than VQ-VAE?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clear\n2. The approach is somehow original for generative models\n3. The method is simple (appears to be much simpler than the VQ-VAE cumbersome training and optimization) with less model capacity and computational complexity\n4. Yet the method reaches good performance.", "weaknesses": "1. The main weakness lies in the originality/novelty of the method which appears to be a compressive autoencoder (e.g., [1,2]) applied to generative tasks. As the Method section demonstrates, there are very few (no) technical/theoretical contributions.\n\n2. The results are still (mostly slightly) worse than VQ.\n\n3. The method obviously lacks compression performance since the quantized vectors' indices are insufficient/irrelevant in this case. \n\n[1] *Lossy Image Compression with Compressive Autoencoders*\n\n[2] *Weakly Supervised Discovery of Semantic Attributes* (and this is not a compression method)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hf0gqBsIUs", "reviewer": "Reviewer_zMky", "summary": "This paper proposes a simple method called finite scalar quantization (FSQ) to replace vector quantization (VQ) in VQ-VAEs. FSQ projects the representation to very few dimensions (typically less than 10), with each dimension quantized to a fixed set of values. This implies a codebook of the same size as in VQ. Results show that as codebook size increases, FSQ achieves better reconstruction metrics and higher codebook utilization, while these metrics deteriorate for VQ.", "questions": "I hope the authors can add some intuitive numerical comparisons to demonstrate the inference speedup brought by FSQ, and emphasize that FSQ does not need the parameters for the codebook.\n\nAdditionally, from the results in the paper, we can see that the metrics of FSQ only show significant advantages when the codebook size is greater than 2^10. However, wouldn't such a large size have a low cost-effectiveness for the second stage (I mean the trade-off between parameter size and FID, where a larger codebook may require a larger model to handle)? This will affect the model's practicality.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is written clearly and proposes a straightforward solution FSQ.\n2. Using scalar quantization to replace vector quantization in VQ, FSQ method avoids the codebook collapse problem in VQ and does not need auxiliary losses. \n3. Sufficient and concrete experiments demonstrate the effectiveness of the proposed method.\n4. The authors make detailed comparisons between the performance of FSQ and VQ.", "weaknesses": "Although I believe this paper meets the bar of acceptance, I still have a few concerns:\n\n1. FSQ's expressive capability is slightly weaker than VQ, and its performance is slightly worse with small codebooks.\n\n2. The paper lacks analysis on the speedup and compression rate brought by the quantization of FSQ.\n\n3. FSQ also faces difficulties in finding proper quantization hyperparameters (dimensions and number of quantization levels).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)