Image and Video Tokenization with Binary Spherical Quantization
Authors: Yue Zhao
Yuanjun Xiong
Philipp Kraehenbuehl
Venue: ICLR 2025 Poster (2025)
Keywords: quantization
visual compression
visual generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ). BSQ projects the high-dimensional visual embedding to a lower-dimensional hypersphere and then applies binary quantization. BSQ is (1) parameter-efficient without an explicit codebook, (2) scalable to arbitrary token dimensions, and (3) compact: compressing visual data by up to 100×
with minimal distortion. Our tokenizer uses a transformer encoder and decoder with simple block-wise causal masking to support variable-length videos as input. The resulting BSQ-ViT achieves state-of-the-art visual reconstruction quality on image and video reconstruction benchmarks with 2.4× throughput compared to the best prior methods. Furthermore, by learning an autoregressive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable visual compression results with commonly used compression standards, e.g. JPEG2000/WebP for images and H.264/H.265 for videos. BSQ-ViT also enables masked language models to achieve competitive image synthesis quality to GAN and diffusion approaches.

Published as a conference paper at ICLR 2025
IMAGE AND VIDEO TOKENIZATION WITH BINARY SPHERICAL QUANTIZATION
Yue Zhao
Yuanjun Xiong
Philipp Krähenbühl
UT Austin
MThreads AI
UT Austin
yzhaoocs. cetooosssutexass... edu
bitxiong@gmail.comm com
philkrêcs utexas. edu
ABSTRACT
We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ). BSQ projects the high-dimensional visual embed- ding to liwer-dimeniional hypersphere and then applies binary quantization. BSQ is (1) parameter-efficien without an explicit codebook, (2) scalable to arbi- trary token dimensions, and (3) compact: compressing visual data by up to 100x with minimal distortion. Our tokenizer uses a transformer encoder and decoder with simple block-wise causal masking to support variable-length videos as input. The resulting BSQ-ViT achieves state-of-the-an visual reconstruction quality on image and video reconstruction benchmarks with 2.4x throughput compared to the best prior methods. Furthermore, by learning autoregressive prior for adap- tive arithmetic coding, BSQ-ViT ViT achieves comparable visual compression results with commonly used compression standards, e.g. JPEG2000/WebP for images and H.264/H.265 for videos. BSQ-ViT also enables masked language models to achieve competitive image synthesis quality GAN and diffusion approaches.
INTRODUCTION
Learned discrete image and video tokenization allows for state-of-the-art visual compression (Daede et al., 2016; Agustsson et al., 2017; El-Nouby et al., 2023), recognition (Yu et al., 2022; Bao et al., 2022; Zhou et al., 2022; Wang al., 2022) and generation (Van Den Oord et al., 2017; Esser et al., 2021; Chang et al., 2022). These models follow proven recipe from large language model- ing (Brown et al., 2020; Achiam et al., 2023; Touvron et al., 2023): Tokenize input and outputs into discrete units and learn an auto-regressive model to predict this tokenized stream one token at time. The most widely used approach for image encoding is cestor-Quantized Variational Auto-Encodes Encoder (VQ-VAE) (Van Den Oord et al., 2017). They encode inputs in continuous latent embeddings and map them to a learned codebook through nearest-neighbor lookup. However, VQ-VAE style ap- proaches have drawbacks: First, most image encoders built upon convolutional networks (CNN) (Esser et al., 2021; Podell et al., 2023). Adapting spatial convolution for images to spatial- temporal convolution for videos requires non-trivial architectural changes (Ge al., 2022; Yu al., 2023; 2024) with increased computational cost. Treating videos as a sequence of images leads to a suboptimal quantization (Yu et al., 2023). Second, vector quantization (VQ) scales poorly with the codebook size. The runtime scales linearly with the codebook size, and the codebook easily overfits on smaller datasets (Yu et al., 2024). This is especially troubling for video inputs, as they rely on larger codebooks to represent both static visual patterns and dynamic motion patterns.
This paper proposes a unified visual tokenizer based on a Vision Transformer and Binary Spher- ical Quantization (BSQ). The Transformer-based encoder-decoder leverages a block-wise causal mask and uses only visual tokens from the current or past timestamps for reconstruction (Figure 3). BSQ first projects the high-dimensional visual embedding of the encoder to lower-dimensional hypersphere and then applies binary quantization. The transformer encoder, decoder, and BSQ are seamlessly integrated into the VQ-GAN (Esser et al., 2021) framework and trained end-to-end.
Our proposed visual tokenizer features several advantages. First, the Transformer-basee encoder- decoder shows Pareto improvement in visual reconstruction quality and computational efficiency Now at Adobe Firefly

---

Published as a conference paper at ICLR 2025
compared standard CNNs. Second, the block-wise causal design unifies images and videos as input at training and supports variable-length videos at inference. BSQ constructs an implicit code- book whose effective vocabulary grows exponentially with the spherical dimension with no learned parameters. The increasing codebook size consistently yields better reconstruction results. Com- pared to Lookup-free Quantization (LFQ) (Yu et al., 2024), recent technique that also builds an implicit codebook based on scalar quantization (SQ), BSQ has a bounded quantization error and easier to train. Furthermore, we show that the soft quantization probability in BSQ reduces to simple product of multiple channel-independee regularization during training. Specifically, we show how a factorized approximation to the entropy for soft quantization of L bits reduces the theoretical computation complexity from O(2", x L) to O(L) with minimal approximation error, and negligible performance degradation in practice.
Bernoulli distributions, leading to efficient entropy
We validate the effectiveness of BSQ-ViT on visual reconstruction and compression benchmarks. On image reconstruction, our model archives a state-of-the-art visual reconstruction quality by both pixel-level and semantic metrics. In particular, our best-performing BSQ-ViT achieves recon- struction FID of 0.41 on ImageNet-Ik val, a 43% reduction compared to the runner-up (SDXL- best model
VAE (Podell et al., 2023)), while being 2.4x faster. On video reconstruction, reduces FVD on UCF-101 by more than half (8.62 - 4.10). By further learning an autoregres- sive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable results on video compres- sion with conventional compression standards, e-g. H.264 and HEVC. By learning a masked lan- guage model, BSQ enables image generation with similar quality to BigGAN (Brock et al., 2018) and ADM (Dhariwal & Nichol, 2021). Code is available at https: //////g //g/ggtuuut. com/ zhaoyue zephyrus/Dsq-vil
2 RELATED WORK
Visual Tokenization. VQ-VAE (Van Den Oord et al., 2017) introduced the concept of discrete tokenized bottlenecks in auto-encoder architectures. Recent improvements include better training objectives (Ramesh et al., 2021; Esser et al., 2021), increasing VQ codebook usage (Yu et al., 2022; Zheng& & Vedaldi, 2023), replacing VQ with product quantization (PQ) (EI-Nouby et al., 2023) or scalar quantization (SQ) (Mentzer et al., 2024), and employing stronger generative models (Esser et al., 2021; Chang et al., 2022). Image tokenizers are trivially extended to video by tokenizing individual frames (Blatmann et al., 2023). However, this ignores dynamic motions and leads to suboptimal tokenization: The same visual information is compressed repeatedly across frames. Video Tokenization. Dedicated video tokenizers make better use of temporal correlations in the input signal. Yan et al. (2021) proposes 3D (de-)convolutions in VQ-VAE for video generation. TATS (Ge et al., 2022) replaces zero padding with replicate padding to mitigate the temporal cor- ruption when video length varies. Yu et al. (2023) introduce central inflation of pretrained 2D convolutional filters to 3D and further make them causal (Yu et al., 2024). Phenaki (Villegas et al., 2022) adopts a factorized causal video vision Transformer (Araab al., 2021) (C-ViViT), which improves efficiency but sacrifices modeling complex motion across time. All these methods focus on generation while we demonstrate that good video tokenizer can perform well in compression. Neural Compression. Since Shannon established the fundamental source coding theorem (Shan- non, 1948), it has formed the basis of lossless compression (Huffman, 1952; Pasco, 1976; Rissanen & Langdon, 1979; Duda, 2009) with probabilistic models including RNN (Mikolov, 2012; Goyal et al., 2019). CNN (Van den Oord et al., 2016; Van Den Oord et al., 2017), VAE (Townsend et al., 2019; 2020), and Transformers (Bellard, 2019; Delétang et al., 2024). Mentzer et al. (2019) presents a fast hierarchical probabilistic model (L3C) for lossless image compression. Delétang et al. (2024) show that LLMs trained primarily on text, egg. Llama (Touvron et al., 2023) and Chinchilla (Hoff- mann et al., 2022), are general-purposs compressors for text, images, and audio. However, these LLMs are too big and slow to make this compression practical. Our tokenizer presents a lighter- weight alternative: Tokenization performs initial local lossy compression, while lightweight and thus computationally efficient sequence model (~~300M) compresses the global video structure. Video compression. Most high-performing modern video compression methods rely on hybrid coders that combine transform coding (Goyal, 2001; Ballé et al., 2020) and motion compensa- tion (Wiegand et al., 2003; Sullivan et al., 2012). Such belief continues in most of the recently popularized saaming-based solutions (Lu et al., 2019; Rippel et al., 2019; Agustsson et al., 2020; Li et al., 2021). VCT (Mentzer et al., 2022) proposes a Transformer-ban based temporal entropy model to learn motion implicitly. However, VCT requires a heavily-engineered image compression model (He
2

---

Published as a conference paper at ICLR 2025
et al., 2022) and has short temporal context window. In this work, we show that learned video tokenizer combined with an arithmetic coder modeled by a sequence model achieves competitive compression results without explicitly modeling motion.
3 PRELIMINARIES
A tokenization- based compression algorithm has three basic steps: A visual tokenizer, L.e. VQ-VAE or LFQ, translates raw visual inputs to a discrete set of tokens. A sequence model then predicts an auto-regressive probability distribution over these discrete tokens. Finally, arithmetic coding converts this distribution into a compressed representation.
Visual Tokenization. VQ-VAB (Van Den Oord et al., 2017) introduced the concept of learning discrete visual representation with auto- architecture and bottleneck module in between with vector quantization (VQ). Given a video X E T x3, an encoder & produces a set of d- dimensional latent embeddings Z E(X) R (= xd with a spatial-temporal downsample factor of r x p x The bottleneck module y then transforms the real-valued latent embeddings into some discrete tokens & q(z). In VQ. the quantizer q assigns each z E Z to the closest entry in learnable code in codebook C [cl... cKk]
IR lg.Ke
2 = Q=) = Ck arg min > cgll2.
(1)
c,
Here, K' is the vocabulary size of the codebook and the integer k is the discretized token represen- tation of z which can be stored in [log( K)] bits. A decoder G maps the discretized tokens back into a visual representation X = () The entire network (E, g, and g) is end-to-end trainable and minimizes an MSE loss LMSF. || X||2 using STE (Bengio et al., 2013) to propagate gradients through the quantization bottleneck. More recent quantizers rely on a perceptual LIPIPS and adversarial LGAN loss for better visual quality (Zhang et al., 2018; Esser al., 2021),
minimize Ex (E, + CCLPIPS(E, G, + AEGAN(E, E.G.q
(2)
where the quantization loss term Lvq emulates online clustering to learn Cs- The main issue with VQ-VAE is that Vector Quantization scales poorly with increasing vocabulary size K (Yu et al., 2024). Remedies include using a smaller code dimension (Yu et al., 2022), introducing stochastic- ity (Takida et al., 2022), reviving "dead" codevectors (Zheng & Vedaldi, 2023), and regularizing with a commitment loss (Van Den Oord et al., 2017):
Lcommit (2, = 11 sg(2) 2/1.
(3)
where sg()) denotes the stop-gradient operation.
Lookup-Free Quantization (LFQ) (Yu et al., 2024) uses a fixed implicit codebook Clrq 1- as corners of a hypercube in L dimensional space. The best vector quantizer for this implicit codebook the binary quantization QLFQ (z) sign(z). To optimize for an effective latent code and encourage code usage, they use an additional entropy objective (Jansen et al., 2020):
Centropy E (g(( - H E [x)]],
where both entropy terms rely a soft quantization (Agustsson et al., 2017)
exp(-T(c 2)))
((c/2) EceoLrg exp(- 2)2)
(5)
to guarantee the loss is differentiable. The final loss LIPF is combination of LMSE, Lcommit LLPIPS, LGAN, and Lentropy. The main computational bottleneck in LFQ is the entropy optimiza- tion of higher-dimensional codebook, as it involves summation over 24 implicit codebook entries.
Both VQ-VAE and LFQ lossily compress visual inputs into N discrete tokens [K, [kw] where k E (1.... K). in N[log KJ bits. Neither tokenization strategy exploits the global image or video structure well. A sequence model with lossless arithmetic coding better fits this global structure. Arithmetic Coding (AC) (Pasco, 1976; Rissanen & Langdon, 1979; Witten et al., 1987) offers way of constructing a bitstream with near-optimal length by leveraging the statistical property of the coding distribution. Given distribution over token streams P. {1,'.. K}" + (0, 1]. arithmetic
3

---

Published as a conference paper at ICLR 2025
Binary Spherical Quantization (BSQ)
encoder
binsry quambiie
2
decoder
project
project
g
x
(a) BSQ (Ours).
VQQ
Lookup Frre Quanciattinn (UPQ)
Grorder
decodex
(C) LFQ-VAE Figure 1: Variational Auto-Encoders (VAE) with different bottlenecks (BSQ, VQ, and LFQ).
(b) VQ-VAE.
coding looks to encode the token stream (-[log Pe(ki,... kN)1 + 1) bits. The most common token distribution is auto-regressive model
Pe(k1 kx) (((k))) k1) ... P.kkl. kN-1)
(6)
for which efficient incremental encoding and decoding algorithms exist (Mentzer et al., 2022).
TRANSFORMER-BE BASED VISUAL TOKENIZER WITH BSQ
Our video tokenizer follows encoder-decoder architecture with discretization bottleneck il- lustrated in Figure la. It combines a transformer- based encoder, a transformer-buie based decoder, and a Binary Spherical Quantization (BSQ) layer. BSQ projects the latent code into lower-dimensional spherical space, applies binary quantization, and then projects the result back up into the decoder's latent space. This projection onto dimenssiioo spherical space has several theoretical advan- lages: The approximation error of the quantizer is bounded and much of the entropy computation factorizes along individual dimensions. These advantages result in experimental improvements as well. BSQ converges quicker and to better tokenizer than other quantization schemes.
4.1 BINARY SPHERICAL QUANTIZATION
Binary Spherical Quantization (BSQ) optimizes over an implicit codebook CBSQ = VT. vT, a hypercube projected onto unit sphere. Each corner C2 E CBSQ of hypercube corresponds to unique token k. The quantizer works as follows: it projects some high-dimensional latent embedding to a lower-dimensional unit hypersphere u, applies binary quantization per axis á sign(u) (u), and back-projects to the quantized vector in the original latent space &, as shown in Figure la. Specifically, we start with an encoded visual input latent embedding to dimensions v = Linear(z) E R2,, where T. < Next, we project v onto the unit sphere u v [o] and perform binary quantization to each dimension of u independently
E(x) RR We first linearly project the
á = - (ug, where sign(z) is the sign function. To keep outputs on the unit sphere, we map sign(0) We use a Straight-Through Estimator (STE) (Bengio al., 2013) to make the operator differentiable, siguste(r) sg(sign(2)) x)) + where sg(-) denotes the stop-gradient operation. Finally, we back-project the quantized to the d-dimensional space = Linear (ú) e Rd.
BSQ has a few appealing properties: As with LFQ, the implicit codebook entry is parameter-fie free and easy to compute. Unlike LFQ, soft quantization of BSQ has a simple probabilistic interpretation, which leads efficient entropy computation in entropy loss -entropy- Finally, BSQ's 's quantization error bounded, which empirically leads to much faster and better convergence than LFQ. Efficient implicit code assignment. At inference time, we map a projected embedding e to a token through simply binarization 1: = Σte [v, >0)= 20 where 1|.] is the indicator function. The inverse mapping uses the bitshift and the bitwise AND operations.
Soft BSQ and entropy. To best the entire range of the implicit codebook CBsq, we the entropy loss Centropy = E,, (g()))) yH [E,, [q(w)]] (Jansen et al., 2020). To compute this entropy loss first derive soft quantization scheme (Agustsson et al., 2017). Since both codebook
4

---

Published as a conference paper at ICLR 2025
C3
C6
09
V
C2
61 =
C i
2
02
65
C8
Cg
C4 G1 Cg Cy C3
C4
(a) BSQ
(b) FSQ
(c) LFQ
Figure 2: Illustration of BSQ compared to LFQ and FSQ in the simplest case (L = 2). In FSQ. we consider each channel to have 3 possible values (=1,0)- 0). The Voronoi diagram for both FSQ and LFQ looks like hypercubes partitioning the entire space while BSQ's looks like hypersphere evenly divided by 2' centroids.
entries and inputs the quantizer are unit vectors, the soft quantization is a distribution
exp(Te v)
L
(cc/u) = EceOnso exp(re u)
II d=1
where a is a sigmoid function, and the overall soft quantizer is independent along each dimension. See Section C.I for a derivation. This allows for an efficient computation of the first entropy term
I Eu H (((c = Eeg Σ H(j(calua))
(8)
See Section C.2 for derivation. Instead of reasoning over distributions over the entire codebook, which is exponentially large, we instead treat each dimension independently. The resulting entropy computation is linear to the dimension L of the bottleneck.
Unfortunately, the second entropy term cannot directly use the same independence assumption, as dimensions in the expected value [[((()) are correlated through the distribution of u. We find the closest factorized distribution ((c) II ((ca) to E,, [((c/u)), and instead minimize the entropy of the approximate distribution. As we will show in Section C.3 the best approximation in terms of the KL-divergence q(ca) Eus (Ā(ca ca uz)]. The final approximate entropy term to maximize is
((Eulā(c)u)) = H((())) =
(9)
As we will show in Section C.3 this approximation is an upper bound to the true entropy, but empir- ically closely tracks the true entropy. This entropy term is again efficient for evaluation. Quantization error in BSQ. Most quantizers use straight-throur ing (Yu et al., 2024; Van Den Oord et al., 2017). Though simple to implement, it assumes that the gradients for an unquantized u and quantized bottleneck are almost the same, which only holds the quantization error d(u, ú = ||u úill is small. As we show in Section C.4, this is true for BSQ:
gradient estimates during train-
E, [d(u, ú) v 2/VL V2.
(10)
Relation to other quantization methods. BSQ is closely connected to many concepts introduced in information and coding theories. LFQ (Yu et al., 2024) uses the same binarization technique as BSQ but does not normalize its output. This leads to an unbounded quantization eπтоr and does not allow for as simple of soft quantization for entropy computation. A pictural comparison between LFQ and BSQ is shown in Figure 2 and a summary is provided in Table 7. Spherical Vector Quantization (SVQ) (Hamkins Zeger, 2002) also ensures all code vectors have a pre-defined radius. However, SVQ assumes variety of radii, which have to be encoded by additional gain quantizer. In our case, the source code is the output of a learned encoder E. Therefore, the unit radius assumption is sound, and the gain quantizer can be avoided. Pyramid Vector Quantization (PVQ) (Fischer, 1986) assumes all code vectors have constant ε norm, but the l normalized centroids partition the hypersphere less uniformly than l.
5

---

Published as a conference paper at ICLR 2025
4.2 TOKENIZATION NETWORK WITH CAUSAL VIDEO TRANSFORMER
We propose to use Vision Transformer (ViT) (Dosovitskiy et al., 2021) to model both the encoder and decoder due its better computational efficiency and higher reconstruction quality. C+2
Video Transformer. We start from ViT- (Yu ct al., 2022) and extend it to take videos as input. We divide an input video
X E T into non-overlapping patches of size 1 x p p. These patches are flattened into a ID sequence, linearly projected, and passed through a stack of N Transformer Encoder layers yield the latent representation, (21, : 22). The decoder takes the same architecture, maps the latent embeddings & back to the pixel space, and regroups them into the original shape. (#1.... .②n) = de-
SrranfformeDDeooddee _,,... ,2N)), where MLP coding head with a two-layer MIP,, ie. Linear o Tanh o Linear. Blockwise Causal Attention. During training, we always assume the input video has 7 frames, which might not hold at inference. Padding shorter video segments to T frames works but wastes a lot of bits, especially in the context of compression. To han- dle variable-length videos, we propose a simple blockwise causal masked attention analogous to causal attention in language model- ing ( (Vaswani et al., 2017). It specifies that only those tokens at time t or earlier are used to reconstruct the visual tokens at time l.
<ini0> BSQ Code
Figure 3: Given an input video, block-wise causal masked atten- tion enables the Transformer en- coder to only use patches from current past timestamps to code each visual patch and later translate it into BSQ code.
(2 4x W
2xx H. x W = TransformerEncoddr 21,
2x x W.
(21-11x xx
QLFQ (x(-1)x #
Ex
W.
(i xx # = MLP Trrnnformerr Decoder (1 ... 2 x
This can be efficiently implemented with a blockwise causal attention mask written in a blockwise lower triangle matrix in Figure 3. When T = 1, the proposed encoder-decoder reduces ViT with full attention mask. Therefore, we can easily train it using mixture of images and videos.
We use factorized spatial-temporaa position embedding to encode the temporal information. Specif- ically, we add set of zero-initialized temporal position embeddings PF, E TR. to the original spatial ones PF., E TR Nxd in the image tokenizer, i.e. PF : = PF, [i, None, :] PPE [None,;;) :] Training the Video Tokenizer from an Image Tokenizer. For training efficiency, we first train an image tokenizer on image data and then fine-tune it to be a video tokenizer. Though previous works (Wang et al., 2022; Blattmann et al., 2023) argue that a pre-trained image tokenizer can be used for videos as is, we observe that the video tokenizer after fine-tuning demonstrates much higher reconstruction quality on video benchmarks, see Section 5. The gain further magnified when the effective vocabulary size becomes larger. We hypothesize that such increased vocabulary size, enabled by the proposed BSQ, is handy for learning vidco-specific motion and blur. In contrast, vanilla VQ methods fail to maintain high codebook usage when the codebook size exceeds 16K.
Optimizing the Visual Tokenizer. Following VQGAN (Esser et al., 2021), we use perceptual loss (Zhang et al., 2018) and an adversarial loss (Goodfellow et al., 2014). We use StyleGAN (Karras et al., 2019) the discriminator since VIT VQGAN (Yu et al., 2022) reports it is much easier train than PatchGAN (Isola et al., 2017). When we fine-tuned the tokenizer on videos, unlike MAGVIT or TATS, we did not inflate StyleGAN be 3D discriminator. Instead, we pass all reconstructed frames individually to the vanilla StyleGAN and sum up the losses.
EXPERIMENTS
We train the image tokenization model on the training set of ImageNet ILSVRC2012 (Russakovsky ct al., 2015) and evaluate the image reconstruction result on the validation set of MS- AOSC (Lin et al., 2014) and ImageNet, denoted by COCO 2017val and ImageNet-Ik respectively. We fine- tune the video tokenization model on UCF-101 (Soomro et al., 2012) and conduct video compres- sion experiments on two standard benchmarks, i.e. MCL-JCV (Wang al., 2016) and UVG (Mereat et al., 2020). We leave dataset statistics and implementation details in Section E. Evaluation metrics. For tokenization, we report perceptual metric (LPIPS- AlexNN))) (Zhang et al., 2018), PSNR, SSIM (Wang et al., 2004), and Fréchet Inception Veeeooon/Video Distance (FID/FVI) (Heusel

---

Published as a conference paper at ICLR 2025
Table 1: Image reconstruction results on COC02017 and ImageNet-1K (256 x 256). The "data" column refers to the training data: CC for CC3M, YF for YFCC100M, Olmg for OpenImages, LAION for LAION-SB, IN for ImageNet, and "?" for unknown source. The "arch." column shows the encoder/decoder architecture: for ConvNets with Self-Attentio,, and T-B for BiaeB The "# bits" column refers to the effective number of bits per token defined in Section 5.1. is obtained by multiplying the latent dimension with the precision. The "TP" column means the inference throughput (images/second) per GPU. The number is taken from the paper. Note that STDs of PSNR, SSIM, and I.PIS are computed across samples instead of multiple runs.
COC02017 val
ImageNet- val
Method
Data
Arch.
Quant. Param. bits TP- PSNR; SSIM; LPIPS, (FID, PSNR; SSIM; LPIPS, iFID.
DALL-B dVAE (Komest et 2021)
CC+YF
VQ
98M
34.0
25.15
.7497
.3014
55.07
25.46
.7385
.127
36.84
$3.40
4.1124
4-1221
+3.95
#.1343
4.1480
MIsKGTT (Ching ctall, 2022)
IN-IK
VQ
54M
10
37.6
17.52
.194
.2157
8,90
17.93
.223
2018
2.23
13.95
VITAVQGAN (Y.4tal,,022
IN-IK Olmg
T-B
VQ
182M
13 17.5
1.55
AD-VAEE Lx (Rombach di 2023)
VQ
68M
21.78
.0422
6.79
22.12
.6046
1039
1.52
4:3.1
d:14
# 0345
+:3.70
#.1563
0400
3D-VAE Lx (Rombuch dl 102)
Olmg
VQ
68M
22.54
0905
6.07
22.82
.6154
.0122
1:1:5
1,140
4 0123
1:3.97
1.1664
0190
ADDV Ix (Rumbssh d. 2022)
Olmg
K1.
68M
*64 22.4
21.68
.6375
0985
5.94
21.99
.6275
.980
1.35
SD. VAB 2.x (huldlle, 2023
Olmg+
KL.
84M
*64 18.9
24.82
.7202
.0994
4.63
25.08
.7054
.0731
0.78
LAION
1:164
1:1341
1: 0344
1:411
4: 1468
0448
SDXL-VAL (Poki 2023)
Olmgt
KL
84M
*64 18.9
25.11
.7433
.0623
4.23
25.38
.7276
.6666
0.72
LAION+?
#1.91
4:1340
2 1789
1:4.41
+: 1414
2 0173
Ours
IN-IK
T-B
BSQ
174M
45.1
25.06
.7662
.0744
5.81
25.36
.7578
.761
1.14
#:.57
2:099
4: 12295
+:4.82
#:136
2.035
IN-IK
BSQ
174M
27.64
.4122
3.42
27.88
.4100
.0422
+:334
2:0700
:: 01.9
::4.26
2: 2,0821
2255
(w/, EMA)
IN-IK
BSQ
174M
27.92
.0380
3.34
28.14
.0814
.0400
1:378
0508
:: 0187
1:4.2
2,0814
0233
et al,, 2017; Unterthiner et al., 2019). To distinguish it from generation, we denote it as rFID/FVD. For generation, we report FID, Inception Score (IS) (Salimans et al., 2016), and improved precision and recall (IPR, Prec, and Rec) (Kynkäänniemi et al., 2019). For compression, we report PSNR and MS-SSIM (Wang et al., 2003) under different levels of bits per pixel (bpp).
5.1 MAIN RESULTS
Image Reconstruction. We first compare the image reconstruction result of BSQ on COCO and ImageNet (256 256) with ofte-oofthhearr art image tokenizers, including DALL-E dVAE (Ramesh et al., 2021), SD-VAE 1.x (Rombach et al., 2022), SD- VAE 2.x, SDXL-VAE (Podell et al., 2023), MaskGIT (Chang et al., 2022), and ViT- ViT-VQGAN (Yu et al., 2022). To perform a comprehensive and fair comparison, we rerun all models using the same augmentation on COCO 2017val and ImageNet- 1k val except the undisclosed ViT-VQGAN. From Table 1, we can see that our model outperforms prior works all metrics (PSNR, SSIM, LPIPS, and rFID), often by big margin. In Figure 4 and Figure 9, we show reconstructed images produced compared to the best prior work, SAXL-VAE. Our method preserves more details about high-frequency texture and fine-grained shape/geometry.
To compare the compression capability of different bottleneck modules, We study the effective number of bits per token (# bits). For VQ-based models, # bits equals log2(K), where K is the codebook size; For KL-regularized models (SD-VAE 2.x and XL). since the latent is continuous, we count # bits as the latent dimension multiplied by the numeric precision (here we use 16 since the checkpoint is stored in FP16). For our BSQ, # bits is L because each latent channel is binary. We summarize the key observations as follows. (1) BSQ efficiently compresses image patches into a small amount of bits. It reconstructs images better in all metrics using fewer bits per token than the second-best method (SDXL-VAE), (2) BSQ is also computationally efficient. Although the ViT- based backbone doubles the parameters, our method yields a 2.4x higher throughput than SDXL- VAE. MaskGIT runs at a comparable speed but reconstructs significantly worse because of a small codebook size and more spatial downsampling. (3) BSQ is generalizable across different domains of images. ImageNet is relatively object-centric while COCO is more scene-centric. Though trained on ImageNet only, our method does well the scene-centric COCO too. It even works better than SD- VAE 1./2.x trained on the similarly scene-centric OpenImages dataset (Kuznetsova et al., 2020). Video Reconstruction. We present the video reconstruction on both UCF-101 training and valida- tion subsets in Table 2. First,
the image tokenizer to reconstruct the video frame by frame. BSQ works slightly better than VQ but neither is comparable to the specialized video tokenizers fine-tuned on video data shown in the lower half of Table 2. Second, we finetune the image tok-
7

---

Published as a conference paper at ICLR 2025
Table 2:
Video reconstruction results on UCF-101 (split 1).
UCF-101 train
UCF-101 val
Method
Backbone Quantizer
Param.
bits PSNR; SSIM; LPIPS, #VVD, PSNR, SSIM- LPIPS, IFVD.
(IMAGE TOKENIZER, w/o ADAPTING TO VIDEDS)
VQ
25.64
.142
25.58
.8120
.1146
382
Ours
VIT
BSQ
174M
18
25.86
.273
.1089
326
25.83
0.8259
0.1108
342
(IMAGE TOKENIZER + VIDEO TOKENIZER)
MaskGIT 20022)
2D
VQ
.685
0.114
TATS (Ge et 202
3D CNN
VQ
32M
14
162
MAGVIT-L. M.GVET-L.CGeeal. 2m1))
3D CNN
VQ
158M
10
22.0
.701
.0990
25
MAGVIT-22 2000)
C.3D CNN
LFQ
158M
.0694
16.12
MAGVIT ( (1.4
C.3D CNN
LPQ
N/A
(-155M)
18
.0537
8.62
Ours
non-BC
VQ
174M
33.06
.9518
.0223
9.16
32.92
9506
02.28
12.79
Ours
BC VIT
174M
32.81
.9496
.0236
10.76
32.68
9484
0241
14.17
Ours
non-BC VIT
BSQ
174M
18
32.43
.9479
.0213
7.34
31.88
9410
0254
10.57
Ours
BSQ
8.08
31.49
9957
0276
11.62
Ours
BC VIT
BSQ
174M
36
33.80
.96066
.0159
4.10
33.55
.9588
.0677
6.21
Table 4: Visual compression results on Kodak and MCL-JCV (640 X 360).
(a) Image Compression results Kodak.
(b) Video compression results on MCL-JCV.
Method
BPP PSNR; MS-SSIM (dB), LPIPS,
Method
BPP PSNR, MS SSMM, LPIPS,
JPEG2000
0.2986
29.192
11.574
.1922
MAGVIT
0.0391
23.70
0.846
.144
WebP
0.2963
29.151
12.193
1655
MAGVIT-v2
0,0508
27.83
0.92
.104
MAGVIT2
0.2812
23.467
8.103
.220
H.264
0.1373
35.415
0.9796
.0949
VQ
0.2812
98.987
12,580
0944
H.265
0.1373
35.670
0,9807
.0908
Ours
0.2812
27.785
12,852
.0823
Ours (w/o, AC)
0.2333
33.698
0.9818
.0501
Ours (w/ AC) 0.2073
Ours (w/. AC)
0,1373
33.698
0.9818
.0501
enizer on videos and see significant improvements. For example, our 18-bit BSQ with causal ViT reduces rFVD from 342 to 11.62 and improves PSNR from 25.83 to 31.49 dB. The compared prior methods include: (1) MaskGIT (Chang et al., 2022) which is a fine-tuned 2D-CNN based tokenizer, (2) TATS (Ge et al., 2022) which uses a 3D CNN with replicated padding, (3) MAGVIT (Yu et al., 2022) whose 3D CNN initialized by pero-inflating a 2D filter, and (4) MAGVIT-v2 (Yu et al., 2024) which makes 3D CNN causal. We take their reported numbers directly. Our models with all configurations outperform MAGVIT-v2 with comparable number of parameters (174M vs. 158M) by large margin. The best-performing MAGVIT-v2 uses a larger backbone and achieves rFVD of 8.62. Our causal BSQ-ViT ViT with L = 18 achieves an 8.08 rFVD and halves the LPIPS. For BSQ with L 36, method further improves the reconstruction metries.
We also show the effect of using block-witd causal masks. The non-causal variant (non-BC) works slightly better all metrics because now the model can look at all visual patches within the temporal context window. This result resembles the observations in video compression that using bidirectional predicted pictures (B-frames) benefits compression quality given the same group of pictures (GoP). Image Generation. Our BSQ-ViT tokenizer can be seamlessly inte- grated into existing generative mod- els for visual generation. We fol- low MaskGIT (Chang al., 2022), a masked language modeling ap- proach. Unlike MaskGIT with a VQ- VAE with K = 1024, BSQ-ViT has an effective vocabulary size 2 and L = 18, resulting in a slow embed- ding lookup. We fix it by dividing each token into groups and treating sub-tokens independently with a similar rationale in Section 4.1. We increase the number of decoding steps accordingly. Table 3
Table 3: Image generation results on ImageNet- (128 x
128). 'The number is taken from the paper.
Category
Method
# steps
HID;
18: Precy
Recy
GAN
BigGAN
2018)
6.02
145.8
0.86 0.35
Diffusion
ADM (Dtusiont Niditt
1,000
5.91
93.3
0.70 0.65
VQ
12
19.4
Masked LM
FSQ
('teateer et did.,
12
18.5
BSQ (Ours)
12 5.69
48.5
0.85
0.42
BSQ (Ours)
32
5,44
139.6
0.80
0.50
shows that the masked LM with BSQ outperforms those with VQ and FSQ reported in (Mentzer et al., 2024). Our method achieves comparable results with other generation paradigms such as GaN-based (Brock et al., 2018) and diffusion- (Dhariwal & Nichol, 2021) approaches. We show qualitative results in Figure 5.
Image and Video Compression. We compare BSQ with two image compression standards includ- ing JPEG2000 and WebP on the Kodak image dataset in Table 4a. we can see that BSQ without entropy coding outperforms JPEG2000 and WebP in terms of MS- with a slightly smaller bpp. We show a detailed RD Curve in Figure 12 in See G.2. We compare the video compression result on MCL-JCV in Table 4b. More results with detailed discussions on MCL-JCV and UVG can be
8

---

Published as a conference paper at ICLR 2025
Table 5: Abalation studies on ImageNet-Ik val 128x128.
Method f2-norm bits (K d L)
PSNR,
SSIM,
LPIPS,
rFID.
Code usage
10 (1024x32)
23.61+3.21
6687341121
.2144.0034
7.05
57.5%
VQ
(16384x8)
25.7613346
.78344.0988
16692.082
4.27
100.0%
/
16 (65536x8)
25.67+3336
.7851
.0706+.0083
6.61
100.0%
/
24.11
.7250=.1121
.09191.0338
4.51
100.0%
BSQ
/
14
25.26:13.11
.77101.0992
.0784-1.0293
4.60
99.8%
/
25.9713.37
.79901.0906
.062911.266
2.66
93.8%
LFQ
18.5812.10
..8281.1340
229511.080
30.7
0.6%
Table 6: Ablation studies of the loss design.
(a) Leave-one-out ablations for training losses.
(b) Group size. (L = 18)
Ccommit
Lantropy
LLPIPS
rFID
Code usage
group
FID
Code usage
Speed
H(plclu)) (((pp())))
size
(ms)
2.95
45.6%
(0OM)
70.0
/
2.83
93.8%
=
2.83
93.8%
0.335
x
/
2.44
78.3%
2.76
95.2%
0.232
/
x
/
13.8
13.3%
3
3.32
96.0%
0.233
/
/
x
19.2
6.9%
Ours (g = 1)
2.86
95.1%
0.212
found in Figure 8 in Section G.3. Simply flattening the video token sequence bitstream achieves
an MS-SSIM of 0.9816 18 at 0.2333 bpp. Although this is not great, we use an auto-regressive model to predict the conditional probability such that the hpp reduced by 41% %. This leads to better tradeoff than standard video codecs including H.264 and HEVC
5.2 ABLATION STUDIES
For ablation studies, we train an ImageNet image tokenizer with resolution 128x 128 with p 8, although our conclusions generally hold for higher resolution, egg. 256x256 in Sec. 5.1.
BSQ vs VQ. Table 5 shows that BSQ and VQ follow a similar trend: better reconstruction for increased L. Since K 2 results in an out-of-memory issue, we try smaller K = 2 65536 for VQ. The gain for VQ already diminishes even though the small bottleneck dimension of 8 still guarantees full code usage. In contrast, BSQ consistently works better on all metrics when L = 18. Importance of lg normalization in BSQ. We remove the by normalization in BSQ, which is equiv- alent LFQ. and show results in the last rows of Table We see much lower code usage and worse rFID, indicating that LFQ does not work well with a ViT-based tokenization encoder.
Contribution of losses. We study the effect of each loss in Table 6a. Although it is computationally prohibitive enumerate all combinations of loss terms and their associative weights, we conduct a simple "leave-one-oue better after removing Lcommit and H (p(cc))) However, the code usage varies greatly. The best configuration is to keep the minimal entropy term while dropping the commitment loss. The commit loss unnecessary because the strictly bounded quantization error BSQ. On the contrary, the dataset entropy maximization term and perceptual term do matter. Without - H (E,, (p(cuu)) rFID increases to 13.8 while the code usage in the validation set significantly drops to 13.3%. We also observe that the perceptual loss is important for low FID and high code usage. However, deeper look into its role beyond the scope of this paper.
setting where one of the losses is removed at time. BSQ works slightly
Approximating the dataset entropy term. We now show the efficacy of approximating the dataset entropy term using Equation 9. We compare with the approximation method in (Yu et al., 2024) that computes entropy in sub-groups of dimensions with varying group size E {9, 6, 3]. Our approximation method also be interpreted as group size of g 1. From Table 6b, we conclude that our approximation achieves a similar level of rFID and code usage compared to other setups while running the fastest.
CONCLUSIONS
We present a new tranfoormr-basee image and video tokenizer with Binary Spherical Quantiza- tion (BSQ). The wansformer-based architecture effortlessly integrates image and video tokenization over an arbitrary time horizon. The Binary Spherical Quantization allows for efficient and effective training of the quantized bottleneck. Our results indicate that the proposed tokenizer at faster

---

Published as a conference paper at ICLR 2025
speed, reconstructs with higher fidelity, and in combination with sequence model offers strong baseline for lossy video compression and image synthesis.
iPod
iPod
iPod
Figure Reconstruction results BSQ- ViT (right) compared original image (left) SDXL.- VAB (Podell at., 2023) (middle). See Figure 9 for more visualization.
BigGAN
BSQ-VIT-Mesked-LM (Ours)
Groundtruth
Figure 5: Sampled generation results of BSQ-ViT USQ-VIT- + Masked-LM. LM (second column from left) compared BigGAN (Brock et al., 2018) (right), ADM (Dhariwal & Nichol, 2021) (sccond column from right) and original images (left). Classes are 1: goldfish, 279: arctic fox, 323: monarch butterfly, 417: balloon.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This material is based upon work in part supported by the National Science Foundation under Grant No. IIS-1845485.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc V Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. NeurlPS, 2017.
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle. Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In CVPR, 2020.
Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. VIVIT: video vision transformer. In ICCV, 2021.
Johannes Ballé, Philip A Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agustsson, Sung Jin Hwang, and George Toderici. Nonlinear transform coding. IEEE Journal of Selected Topics in Signal Processing, 15(2):339-353, 2020.
Hangbo Bao, Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transform- ers. In ICLR, 2022.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. CompressAl: a py- torch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.
Fabrice Bellard. Lossless data compression with neural networks. org/nncp/nnc. pdf, 2019.
URL: https://bellard
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:
13XXXv::300.333 3008333833333 3432, 2013.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprini arXiv:231 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. MaskGIT: Masked genera- tive image transformer. In CVPR, 2022.
Thomas J Daede, Nathan E Egge, Jean-Mare Valin, Guillaume Martres, and Timothy B Terriberry. Daala: A perceptually-driven 2016.
generation video codec. arXiv preprint arXiv:1603.03122
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein. Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. In ICLR, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. NeurIPS, 2021.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Jarek Duda. Asymmetric numeral systems, arXiv preprint arXiv:0902.0271. 2009.
Alaaeldin El-Nouby, Matthew Muckley, Karen Ullrich, Ivan Laptev, Jakob Verbeek, and Hervé Jégou. Image compression with product quantized masked image modeling. TMLR, 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Thomas Fischer. A pyramid vector quantizer. IEEE Transactions on Information Theory, 32(4): 568-583, 1986.
Songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic VQGAN and time-sensitive transformer. In ECCV, 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 2014.
Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa. Deepzip: Lossless data compression using recurrent neural networks. In DCC, 2019.
Vivek K Goyal. Theoretical foundations of transform coding. IEEE Signal Processing Magazine, (8(5):9-21,200
Jon Hamkins and Kenneth Zeger. Gaussian source coding with spherical codes. IEEE Transactions Information Theory, 18(11):2980-2980 2002.
Dailan He, Ziming Yang, Weikun Peng. Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In CVPR, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge a local Nash equilibrium. NeurIPS, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12590 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. In NeurlPS, 2022.
David Huffman. method for the construction of mininum-redundancy codes. Proceedings of the IRE, 4009): 1952.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imagee -mage translation with conditional adversarial networks. In CVPR, 2017.
Aren Jansen, Daniel PW Ellis, Shawn Hershey, Channing Moore, Manoj Plakal, Ashok C Popat, and Rif A Saurous. Coincidence, categorization, and consolidation: Learning to recognize sounds with minimal supervision. In /CASSP, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
12

---

Published as a conference paper at ICLR 2025
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurlPS, 2019.
Jiahao Li, Bin Li, and Yan Lu. Deep contextual video compression. In NeurIPS, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In 2CCVV.0144
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end- to-end deep video compression framework. In CVPR, 2019.
Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Practical full resolution learned lossless image compression. In CVPR, 2019.
Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, and Eirikur Agustsson. VCT: A video compression transformer. In NeurIPS, 2022.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: VQ-VAE made simple. In ICLR, 2024.
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
Tomáš Mikolov. Statistical language models based on neural networks. PhD thesis, Bmo University of Technology, 2012.
Richard Clark Pasco. Source coding algorithms for fast data compression. PhD thesis, Stanford University CA, 1976.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01152, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander Anderson, and Lubomir Bour- dev. Learned video compression. In ICCV, 2019.
Jorma Rissanen and Glen Langdon. Arithmetic coding. IBM Journal of research and development, 1362);149-162, 1979.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. ISC,, 115:211-252, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. NeurIPS, 2016.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 1948.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2/.:/222,000 2012.
Gary Sullivan, Jens-Rane Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. TCSVT, 2012.
13

---

Published as a conference paper at ICLR 2025
Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chich-Hsin Lai, Junki Ohmura, Toshimitsu Ue- saka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. SQ-VAE: Variational bayes discrete representation with self-annealed stochastic quantization. In ICML, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
James Townsend, Tom Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In ICLR, 2019.
James Townsend. Thomas Bird, Julius Kunze, and David Barber. Hilloe: Lossless image compres- sion with hierarchical latent variable models. In ICLR, 2020.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: new metric for video generation. In ICLR Workshop, 2019.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Condi- tional image generation with pixelenn decoders. In NeurlPS, 2016.
Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In NeurIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Ruben Villegas, Mohammad Babacizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2022.
Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchich Lin, Lina Jin, Longguang Song, Ping Wang, loannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset. In 2016.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. BEVT: BERT pretraining of video transformers. In CVPR, 2022.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In ACSSC, 2003.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. TCSVT, 2003.
Ian Witten, Radford M Neal, and John G Cleary. Arithmetic coding for data compression. Com- munications of the ACM, 3006):500-540, 1987.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using 2021.
vq-vae and transformers. arXiv preprint
Jiahui Yu, Xin Li, Jing Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQ- GAN. In ICLR, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsus Yang, Yuan Hao, Irfan Essa, et al. MAGVIT: Masked generative video transformer. In CVPR, 2023.
14

---

Published as a conference paper at ICLR 2025
Table 7: Comparing BSQ LFQ.
LFQ (Yu al., 2024)
BSQ (Ours)
Quantized oulput
sign(v)
vI sign(u) VI
sign(
STE gradient
80; = 1
au, VI - u// 16/10/?
deg
0v:
Eu [d(v, 6)] = 00
Eu [d(ue ui] < 2 12- vI.
< v2
Quantization Error
Unbounded CMSE. Leommit, CLPIPS, LGAN. Lentropy = H[p(c/o) 11 (Bu
Upper-boundets (Sce Scction C.4)
MMEE CLPIPS, LGAN.
Training objective
Leutropy = H[p(clu) [[c|u)]
Lijun Yu, José Lezama, Nitesh Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion- tokenizer key to visual generation. ICLR, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In ICCV, 2023.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT pre-training with online tokenizer. In ICLR, 2022.
A ARITHMATIC CODING DETAILS
Starting from the initial interval to = [0, 1), the AC encoder recursively partitions the interval into series of sub-interval I [L,, un such that I In-1 C ... C To. and In determined by In-1 and p(u c.n).
T, = ln-1 + (Wa-1-1, 11--
Σ ply)den), 1n-1 + (Un-1 - In plu (ea) (11)
y=1
Any number in the final interval IN can sufficiently represent the encoded sequence. To obtain the final bit stream, take binary fraction À E x 2-i, b, {0, in IN such that IN < X < uN- The bit stream fbo...., bc is the encoding result with a length of C bits.
The AC decoder takes in A, starts with To, and performs similar interval partitioning process. At the nith step, the decoder queries the model Pr. (v( calculate the sub-intervals for all possible values of y using Equation 1,, and decodes output H, that leads to À E (2,,). The decoder can recover the encoded token sequence by continuing with l based on the decoded and repeating for step n + for N steps.
In practice, the encoder and the decoder can be implemented efficiently with fixed-length integer numbers and operate incrementally for arbitrarily long input sequences.
B COMPARISON BETWEEN BSQ AND LFQ
In Section 4.1, we have introduced the mechanism of BSQ and briefly discussed the connections and differences with LFQ. We summarize them Table Note that STE gradient in BSQ is anisotropic and is more likely to be good estimation because of an upper-bounded quantization error regardless of L. This property explains why commitment loss like Lcommit (ù, u.) is not needed in BSQ but useful for LFQ.
C PROOFS
C.I PROOF OF EQUATION
Before proving Equation 7, will first prove the following identity:
15

---

Published as a conference paper at ICLR 2025
u R², C Ω2 e R
for I = 1--1 1
VI' vzl.
Σ
e =
Cuaa II Σ
(12)
CEC
CEC d=1
d=1 cgE
Proof. With r dropped for simplicity of notation.
L
Σ n
Σ
II
CEC
CEC k=1
Σ GEReeEE
Σ
I
Baca
T-1 II
Σ
Σ
Σ
uaca
CLE
d=1
L-1
E
Σ
Σ
II
baca
Σ
ELLOL
"L C1-
d=1
Σ
Σ
2202
Σ
excer
II
Σ
naca
Therefore, the probability of u being assigned to e; can written as:
eruTe
IT euuba
((c/u)= Exe
(Using Equation 12)
IT Eesel to vel erusas
L
etuada
II
(since cd
+2₂)
etugda + Tugaa
VI.
L
II d=1
(2ruaaa)
C.2 PROOF OF EQUATION
Since ((clu) A (2Tuaad) each variable ca is independent of each other. Thus by definition
I
H[((clw)] = Σ H(o(2ruaca)).
d=1
C.3 PROOF OF EQUATION
Now we look at H E,, [(cck)) We first compute Q(c) = Ey [[(c)e))).
Q(c) E,, [((clu) Σ ((clu)
EI (2ucce). N
1
Unlike c, does not factorize like Equation 12. This would require compute Q(c) full distribution over 24 states, which is slow (O x 24)) and easily overfits. Instead, we approximate
Q(c) by factorized distribution q(c) I (a(C)), where ca Ω for Ω {- V J ). using
16

---

Published as a conference paper at ICLR 2025
an M-projection. We again omit for notational brevity.
( ()) H(Q)
2'
((c,) log((() HQQ)
2
Σ ((1)) Σ log -
2
-H(Q)
log = Σ Q(es) log -1) -11 HQQ))
Σ
log (c()
E0(e) H(Q),, where c. sums over all dimensions except d.
(.E(-1.1)
The minimizer of the above projection
8 ((Q - 0 8ea
"a(ca)) Q(())= Eu
Σ ((clu)
Eu
ΣII (2uncH)
EE
Σ ((uuaca) II (2unck) kid
En
((uuaca) IIl a (2uncu) c-d kid
= En
(2uace)) II
Σ
o (2uncH)
= En (ouu
legtid C-n
Therefore, the entropy term is simplified to:
((T)) = Σ H(āa(ca)) = Σ H (Eu (2uaa(())))))).
By the nature of the above derivation the cross entropy H(Q, HH)) equals the entropy of the approximation. This means that D(Qllğ) ( (QQ 2 0, and the entropy of the approxima- tion is an upper bound (ÿ) 2 (Q) the entropy.
In practice, this bound is relatively tight. The most adversarial distribution P(u) is P( 1))
and P 11-Ti11 J. where all inputs are maximally correlated, but the factorized distribution not. Figure shows an empirical estimate of this approximation error for various values r. In practice, use T I00 which has little to no approximation error.
C.4 PROOF OF THE QUANTIZATION ERROR BOUND OF BSQ (EQUATION 10)
We consider di-distance d(u à ||u ii/. A simple (but loose) bound is:
2 2
En [d(u, ú)] = En [dmax (ue, Ki <
< V2,
(13)
VI
where dmax attained is any axis, 00
1,
17

---

Published as a conference paper at ICLR 2025
Upper bound KL divergence between and á
1.0
L-5
L-10
L-15
Approumsion eroe
0.0
0.2
Figure 6: Empirical estimation of the approximation error with respect to at different bottleneck
dimensions
Proof of Equation 13. The sketch derivation dmaz (u, ú) as follows. For an arbitrary vector u residing on the unit hypersphere (uu22 = 1) and its quantized counterpart á also on the unit hypersphere, we can write the quantization error as d(u, (u, ú v2 2u ū. The maximum of d(u, ú is therefore attained when the minimum of u ú; are elements of u and ú, respectively. In BSQ the quantization law is that ú; VI (ign(()).
Σ u, û, is achieved, where u, and
This leads to u.ũ; A [4] and á > Nil [w]] VL
VI" Minimizing u ú is then equivalent to
finding an l2 unit vector with the minimal & norm, which is known to be achieved only if the vector is one-hot vector and the minimal & norm is 1. This leads to the maximum of d(u, ú) being
2
To achieve a tighter bound, we first expand the definition,
dsu V ú)
En [d(u, úi]
(14)
dg V
where 56-1 = fx E TR-= || = denotes the unit T-sphere of radius 1 and ds V denotes its surface area element. We further define a hyperspherical coordinate system that analogous the spherical coordinate system for 3D Euclidean space or the polar coordinate system for 2D space to represent the surface area element.
= cos((p1)
U2 sin(p1) cos((22),
U3 sin(p11 sin (222 cos((ss)
U sin(p1) sin sin(pi.-))
71 sin(p1) ( sinn(pp-))sin(((_-
(surface area element) dga-1 1 V = sin L (p1) sin ((p) .in (in))))...ii(()L___-. sin(4--2)dp)p) d
1/2
(surface area) St-1
dga-1 r(()
SL
Due to symmetry, we assume the subarea 44-1 where Vi E {1,... L}. us > 0, and it will be quan- The unit hypersphere S has 2 of such subareas interchangeably.
tized to it +z T.
18

---

Published as a conference paper at ICLR 2025
Computing Equation 14 is equivalent
1 dsr V û)
Eu [d(u, u)]
A4
(15)
dsa-1V .
Ah-1
We expand the the numerator in Equation 15 as follows:
Jo² * ... //² dsa-1 V [[(os(())) VI 12 [sin((s1) cos(122)
1
12 +...
vT.
+ (sin((91) sin(pp2) ... sin((L--2 cos cos((9-11) 1
vL }
(16)
... sin( sin(pL-))sin((LL)) sin((p-1)
VL 12]
is composed of I. square terms. easy to see that the sum of constant terms leads to 1. Next, let's sum over all quadratic terms and keep on using sin sin²(0) co)
L-2 sin (p,) sin (ML-1) + II
L-2
(p1) + sin' (1P1)
II
sin (yp) ((L-1)
jml
So the distance function to be integrated simplifies to
L-2
sin(),) cos c0s(())1) 2 L-2
2
2 sin(p11 cos((p2)
2
cos((p1)
VL II j=1
II
sin({;) sin (iL-())
VL
VL
vL j=1
2
cos((21)
VI
Plug into the numerator in Equation 15 and continue simplifying:
ds v (2- VI
cos((z))
(17)
Al-11
2 f² {8 (2 (2-v) VI cos((p)) )) sin sin--2(pi)ddpp
--
(18)
A4-1
S1
74 1
Therefore, we have
2 L
Eu [d(u, 4)
<
(2
cos(
sin 2
(19)
VL
where RHS can be numerically computed and plotted in Figure 7.
D DATASET OVERVIEW
ImageNet-Il has 1.28M training images and 50,000 validation images; COCO 2017val has 5,000
images.
19

---

Published as a conference paper at ICLR 2025
140
Quantrstion Erro
136-
1.32
128
20
Figure 7: Quantization error with vocabulary size L.
UCF101 has 13,320 video clips and three train-val splits. Following prior works (Yu et al., 2023), we consider split-1 which has 9,537 clips for training and 3,783 for validation.
The MCL.JCVV dataset (Wang et al., 2016) consists of thirty 1080P (1,920 1,080) video sequences with 24--30 FPS. The Open Ultra Video Group (UVG) dataset (Mercat et al., 2020) consists of six- teen 4K x 160) test video sequences captured at 50/120 FPS. Following prior works (Agusts- son et al., 2020), we report the performance subset of seven videos in YUV 8bit format at 120 FPS under the resolution of 1,920x1,088
E IMPLEMENTATION DETAILS
Training Image Tokenizers. We train the image tokenizer with a batch size of 32 per GPU. We use AdamW optimizer (Loshchilov & Hutter, 2019) with (31, (0.9, 0.99) with x 10-4 weight decay. The base learning rate is 4 x 10-7 (or a total learning rate of x 10-2) and follows a half- period cosine annealing schedule. The model is trained for 1M steps which amounts 200 epochs over the entire ImageNet- lk training set. We did not heavily study the effect of loss weights. Instead, we keep y = 1 in the entropy terms. We use perceptual loss weight of 0.1 and an adversarial loss weight of 0.1 throughout the experiments.
Evaluating Image Tokenizers. We observe that reconstruction metrics vary with many factors, especially preprocessing (e.g. interpolation). input resolution, and downsample scales (Section D.2 in (Rombach et al., 2022)). we resize all images such that the smaller edge is 256 pixels using Lánczos interpolation in Table and bilinear interpolation Table 8, take the center crop (H x W) (256 256). and ensure all models have the same spatial downsample ratio of 8 (except for MaskGIT, p = 16). We rerun all models on COCO 2017val and ImageNet-Ik Ik val except the undisclosed ViT-VQGAN.
Training Video Tokenizers. We finetune the video tokenizer with batch size of 32 per GPU. The optimization schedule follows the image-based one but trains for fewer iterations. The network is initialized from the SeageNet--retraining checkpoint and undergoes another 500K steps which amounts to 1600 epochs over UCF-101 split-1 train.
Training a Masked Language Model for Generation. The masked LM is a standard Post-LN Transformer with 24 layers and a hidden dimension of 768 following MaskGIT (Chang et al., 2022). We train the masked LM on nodes of 8x GPUs (16 total) with a total batch size of 1024 for IM steps. We use AdamW optimizer with (81, (2) (0.9, 0.96) with 0.045 weight decay. At inference time, we use a cosine unmasking schedule temperature to 15. We classifier-free free guidance (Ho & Salimans, 2022): At training, 20% of the class condition labels with the mask token so that the model learns an unconditional distribution simultaneously. Let f. be class- inference, we interpolate logits using l = f, + a<f, - fg). where a = 0.5.
MaskGIT (Chang ct al., 2022) and set the sampling
replace
logits and lp be unconditional logits. During
Training an Regreseereeeivv Model for Arithmetic Coding. The auto regr-segees model a Transformer with 24 layers and hidden dimension 768. We train this model on 8x GPUs with a total batch size of 64. We use AdamW optimizer with (3, 32) = (0.9, 0.96) with 0.045 weight decay.
20

---

Published as a conference paper at ICLR 2025
Table Image reconstruction results on COC02017 ImageNet-1K (256 x 256). The settings strictly follow Table except that all images are resized with bilinear interpolation.
COC02017 val
ImageNet- Ik val
Method
Arch. Quant. Param. bits
PSNR, SSIM, LPIPS, (FID, PSNR, SSIM, LPIPS, (FID;
DALL-E dVAE (Runcei 2021)
CC+YF
VQ
98M
13
34.0
26.97
.0837
.2544
48.60
27.31
.7943
.2444
32.63
2341
2:0022
4. 1057
13.81
2:1144
1057
MaskGIT (Chang ctal, 2022)
IN-IK
VQ
54M
10
37.5
18.21
.4966
.1330
8.47
18.63
.4699
.1884
1.98
23.34
2: 0444
1:3.00
2:1812
2 6497
VII-VQGAN cta, 2022)
VQ
182M
17.5
1.55
SD-VAE Lx (Romischis al, 2023)
Olmg
VQ
68M
10
22.4
23.29
.6705
0949
6.49
23.65
.6155
0940
1.40
281.34
1313
1:1.0
1:150
0167
SD-VAE LX (Reabach
Olmg
VQ
68M
14
22.4
24.17
.7422
.814
5.75
24.48
.931
.814
1.13
+33.50
#:1276
#: 0280
+:3.98
#.1502
0289
SD-VAE Lx (Rambashet di,
Olmg
KL.
68M
64
22.4
23.21
.6930
0908
5.94
23.54
.6835
.0999
1.22
2:3.44
#:1249
.44222
+:3.62
#.1463
03337
SD-VAE 2.x (Psidle 2023)
Olmgt
KL.
84M
64
18.9
26.62
.7722
0584
4.26
26.90
.7592
.0609
0.70
LATON
11.64
1086
+: 1777
1:4.00
2:1
01:40
SDXL -AE 201))
Olmg+
KL
84M
64
.7953
.041
3,93
27.37
.7814
.0744
LAION+?
=:3,88
+: 0250
+14.36
#:1282
2.0320
BSQ
174M
27.78
.1711
1:347
#:0851
#: 0255
+:3999
#:0967
2.0307
BSQ
174M
29.85
30.12
.803
11.65
2500
1 0163
14.13
1.050
0700
(RV. EMA)
IN-IK
BSQ
174M
30.19
.0144
3.07
30.45
.843
.0329
Hardware. The hardware for training is GPU-serverss with NVIDIA A5000 (24GB). Pre-training an image tokenizer and fine-tuming a video tokenizer in the full schedule is done across two servers with distributed training and takes around days. Training the AR model for AC done on an GPU server and takes around I week. When measuring the tokenizer's 's throughput and the com- pression runtime, we use a server with 4x A5000 GPU and 1x AMD Ryzen Threadripper PRO 5975WX 32-Core CPU (64 threads).
F BASELINES FOR VIDEO COMPRESSION
Following SSF (Agustsson et al., 2020), we used FFmpeg' to produce the evaluation metrics for H.264 and HEVC. We use the commands provided in CompressAl (Bégaint et al., 2020).
f fmpeg -y -s:v $RESOLUTION -1 $FILE. yuv -c:v h264 -crf $CRF \
-preset medium -bf 0 -pix_fmt yuv420p -threads 4 $FILE.mp4
where $Resolution E ( (1920x10880) 640x360), and SCRF {17, 20, 37,
G MORE EXPERIMENTAL RESULTS
G.1 IMAGE RECONSTRUCTION
In Table 8, we compare the image reconstruction results under bilinear interpolation. We conclude that varying interpolation changes the values but unalts the order of methods.
G.2 IMAGE COMPRESSION
In Figure 12, we compare the rate-distortion (RD) curve in terms of MS-SSIM bpp (bits per pixel) on Kodak. We see BSQ consistently improves MS-SSIM with increasing BPP while VQ-based AE shows diminishing gain at higher BPP.
G.3 VIDEO COMPRESSION
We compare the compression result on MCL-JCV and UVG in Figure 8. On UVG 1080P, our model comparable H.264 while being worse than HEVC and VCT (Mentzer et al., 2022). Note that our
httpp:://// :/l:://ffff httee://ffffgggpore
21

---

Published as a conference paper at ICLR 2025
ELKA Prefees
VCT HAGGer oul GA37 1837
(a) PSNR MCL-JCV.
(b) MS-SSIM MCL-JCV.
(C) PSNR UVG.
(d) MS-SSIM UVG.
Figure 8: Video compression results on MCL-JCV 640 and UVG 1920 1080.
Table 9:
Comparisons of encoding/decoding speed. 'The number did not include the image encoder ac-
cording to (Mentzer al., 2022).
Method
Resolution
Encode
EC
Decode
FPS
VCT (Mentzer al, 2022)
x920 1080
1494 ms
30.5 ms
168 ms
1.4
H.264
1920 1080
2.6
Ours
x920 1080
55.8 ms
42,2 ms
64.8 ms
6.1
VCT (Mentzer et al, 2022)
640x360
122.2 ms
4.24 ms
10.1 ms
27.3
H.264
640x360
22.4
Ours
640x360
6.2 ms
4.69 ms
7.2 ms
55.2
model trains UCF-101 which only has 9K 320x240 video clips encoded in MPEG-4 while VCT has been trained on a million high-resolution Internet video clips. We hypothesize that the gap will be mitigated by adding more diverse videos and removing compression artifacts from the training videos. Nevertheless, we show the potential advantage of our method in encoding and decoding speed in Table 9. Due to the simplicity of the Transformer-basse encoder and decoder, our method runs faster than VCT.
H QUALITATIVE RESULTS
In Figure 9, we show reconstructed images produced by the proposed BSQ-ViT in comparison to the best prior work, SDXL- (Podell et al., 2023). We can see that our method able to preserve more details about high-frequency texture and fine-grained shape/geometry. BSQ-ViT ViT often shows better reconstruction results for characters.
In Figure 10, we show sampled results produced by Masked LM with the proposed BSQ-ViT in comparison to existing methods, BigGAN (Brock et al., 2018) and ADM (Dhariwal Nichol, 2021). We also plot the samples from the ground-truth ILSVRC2012 validation set for reference. Our method produces competitive results with ttate-ofthhe-ee methods.
In Figure 11. show the compressed image produced by different methods under the same BPP used Table 4a.
22

---

Published as a conference paper at ICLR 2025
iPod
iPod
iPod
60 BY SRAIN
60 BY TRAIN
Bv IDAIN
GO BY TRAIN
80 RV
GO BY
TRAIN
TRAIN
Figure 9: Reconstruction results of BSQ-VIT VIT (right) compared to the original image (left) and SDXL- VAB (Podell ct al., 2023) (middle). The three images taken from COCO 2017 which more scene- centric compared ImageNet data that our model trained on.
23

---

Published as a conference paper at ICLR 2025
BigGAN
BSQ MiT--a (Ours)
Groundtruth
Figure 10: Sampled generation results of BSQ-ViT Masked-LM (second column from left) compared BigGAN (Brock al., 2018) (right), ADM (Dhariwal & Nichol, 2021) (sccond column from right) and original images (left). Classes are 1: goldfish, 279: arctic fox, 323: monarch butterfly, 417: balloon.
24

---

Published as a conference paper at ICLR 2025
(e) Original image
(bUPEG2000
(6) WrbP
[6) (Ours)
Figure 11:
Compression results of BSQ-ViT (d) compared to JPEG2000 (b). WebP (c) and the original
images (a). images are from Kodak (kodim01, kodim15, and kodim23).
16
15
14
13
MS-SSIM (dB)
12
11
JPEG2000
10
WebP
VTM
BSQ-AE (Ours), W/o. AC
BSQ-AE (Ours), w/. AC
VQ-AE, w/o. AC
8
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
bpp
Figure 12: Image compression results on Kodak.
25


Reviews:
[{"review_id": "zGTQeAPzv2", "reviewer": "Reviewer_GS7J", "summary": "This paper proposes a novel image and video tokenizer, BSQ-ViT, based on Binary Spherical Quantization (BSQ) integrated with a transformer architecture. The proposed method achieves state-of-the-art performance in image and video reconstruction with significant improvements in computational efficiency. It introduces a block-wise causal masking for video tokenization, supports variable-length videos, and delivers competitive results in visual compression against standards like JPEG2000 and H.264. BSQ-ViT also shows promising results in image generation.", "questions": "see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents an innovative quantization method (BSQ) that addresses the limitations of existing vector quantization approaches by offering a more efficient and scalable solution.\n2. Extensive experiments on benchmarks such as ImageNet and UCF-101 demonstrate that BSQ-ViT significantly improves reconstruction quality, outperforming prior methods in terms of speed and fidelity.\n3. The methodology is clearly explained with detailed comparisons to related work, and the theoretical basis of BSQ is well-supported with mathematical derivations and experimental validation.\n4. BSQ-ViT's ability to handle both image and video tokenization and perform well on diverse tasks such as compression and generation showcases its generalizability.", "weaknesses": "1. While the transformer architecture is explored, the paper does not demonstrate the effectiveness of  BSQ within a CNN-based model.\n2. The paper provides limited comparative data in video reconstruction, reducing the robustness of the comparison. Additionally, while block-wise causal attention is noted to impact performance, the study lacks experiments on BSQ without this causal masking. \n3. The reported image and video compression results are better on MS-SSIM, potentially  due to the inclusion of perceptual losses like LPIPS and GAN loss in the loss function. However, the lack of subjective evaluation metrics limits the strength of the comparison.\n4. When BSQ and VQ use the same number of indexes, BSQ performs less effectively, and codebook utilization decreases as the bits increase.\n5. In the ablation experiments, the reproduction of the LFQ, which serves as the basis for this work, is too poor in terms of behavior and code usage. And it is premature to conclude that LFQ is less compatible with transformer-based codecs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BMMhiUBmJQ", "reviewer": "Reviewer_zX4y", "summary": "This paper propose a transformer-based image and video tokenizer with Binary Spherical Quantization, which projects the high-dimensional embedding to lower-dimensional hypersphere and applies binary quantization. BSQ is parameter-efficient without an explicit codebook, scalable to token dimensions and compact. The experiments show that the proposed method achieves comparable compression performance with JPEG2000/WebP for images and H.264/H.265 for videos. And it enables masked language models to achieve competitive image synthesis quality to GAN and diffusion methods.", "questions": "See the weakness part", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The Binary Spherical Quantization seems to show more effective training of the qunatization bottleneck. Analysis shows that the proposed method can provide fast speed and good performance.", "weaknesses": "- Lack of comparison at different bitrate range for visual compression results. Table 4 only provides BPP, PSNR and MS-SSIM for one bitrate point. However, visual compression tasks usually require showing a Rate-Distortion curves and compare at different bitrate points. Your can use BD-Rate metric for more reasonable comparison and analyze the results at low bitrate and high bitrate.\n\n- Test settings for ablation study. Please provide more experiment setting details. In Table 5, do VQ, LFQ and BSQ use the same tokenizers? I observed that the metrics vary significantly across different quantization methods. To alleviate the influence of training and focus on the quantization bottleneck, it is better to: 1. use the same tokenizer 2. freeze the other network parts and only train the information bottleneck parts.\n\n- Complexity. It is better to provide the computation complexity and encoding/decoding time for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sshM8GOHxM", "reviewer": "Reviewer_PNuG", "summary": "The paper introduces a transformer-based image and video tokenizer that uses Binary Spherical Quantization (BSQ). By projecting high-dimensional visual embeddings onto a lower-dimensional hypersphere and applying binary quantization, BSQ achieves a bounded quantization error. The authors demonstrate that the proposed BSQ outperforms previous methods in image and video reconstruction, image generation, and compression tasks.", "questions": "1. For image compression, why not use arithmetic coding to improve compression performance? Is intra-frame information used for video compression? If not, why not?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of projecting high-dimensional visual embeddings onto a lower-dimensional hypersphere is straightforward yet effective.\n\n2. The motivation is clear, and the overall presentation is coherent and easy to follow. The experiments are comprehensive and provide convincing evidence to support the approach.\n\n3. The BSQ-ViT model achieves competitive performance in diverse tasks such as image/video reconstruction, generation, and compression.", "weaknesses": "1. This method uses a transformer encoder and decoder, which limit the flexibility in resolution. How do the authors address this issue?\n\n2. For image and video compression results, it would be beneficial to include an LPIPS comparison to assess perceptual performance.\n\n3. There are a few minor issues. 1) In Eq. (7), $\\hat{q}(c|u) = \\frac{\\exp(\\tau c^{T}u)}{\\sum_{c \\in C_{BSQ}} \\exp(\\tau c^{T}u)}$ might need to be revised to $\\hat{q}(c|u) = \\frac{\\exp(2 \\tau c^{T}u)}{\\sum_{c \\in C_{BSQ}} \\exp(2 \\tau c^{T}u)}$ ? 2) On page 16, line 837, $\\frac{e^{\\tau u_d \\hat{c}_d}}{e^{\\tau u_d \\hat{c}_d} + e^{\\tau u_d \\hat{c}_d}}$ should be corrected to $\\frac{e^{\\tau u_d \\hat{c}_d}}{e^{\\tau u_d \\hat{c}_d} + e^{-\\tau u_d \\hat{c}_d}}$. 3) On page 3, line 118, it would be clearer to use different symbols for the downsample factor $q$ and the bottleneck module $q$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHqSZIReh4", "reviewer": "Reviewer_Aq9e", "summary": "This paper proposes a visual tokenizer based on a Vision Transformer and Binary Spherical Quantization (BSQ).   \nThe Transformer-based encoder-decoder leverages a block-wise causal mask and uses only visual tokens from the current or past timestamps for reconstruction.  \nBSQ first projects the high-dimensional visual embedding of the encoder to a lower-dimensional hypersphere and then applies binary quantization.   \nThe transformer encoder, decoder, and BSQ are seamlessly integrated into the VQ-GAN framework and trained end-to-end.  \n\nThe proposed visual tokenizer has several advantages in trading off  visual reconstruction quality and computational efficiency and supporting variable-length input. Experiments on visual reconstruction and compression are conducted to verify the performance.", "questions": "My major concerns in the weakness part can be summarized as the direct comparison between BSQ, VQ and LFQ on representative image and video tasks are not adequate. As a general tokenizer for image and video, it is important to make direct comparisons with strictly controlled variables.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Visual tokenizers are critical for visual modeling. The proposed Binary Spherical Quantization (BSQ) for unified image and video tokenization is novel and important.  \n\nTheoretical contribution is good. It is proved that the proposed BSQ has bounded quantization error.    \n\nThe manuscript is well written. BSQ is well placed agains previous VQ and LFQ.  \n\nExtensive experiments on visual reconstruction, compression and generation show the better performance of BSQ.", "weaknesses": "My major concern is the fair comparison with VQ and LFQ. Though the ablation study is provided in  tab 5 with image reconstruction task on imagenet-1k, the resolution is only 128, and more tasks should be tested:     \n\nIt seems to me the results on image is not as obvious as video. For example, in image reconstruction (tab 1), the proposed method has best image quality, but the parameter number is also larger. It is better to add Ours-VQ for better comparison, as done in tab 2 for video reconstruction.  \n\nIn image compression, no VQ or LFQ based method is compared.     \n\nFor image generation in tab 3, more steps are used for BSQ, what if we use the same steps as VQ and LFQ?   \n\nIn L389, we get a direct comparison between VQ and BSQ, BSQ has more bits (18 vs 14) and the video quality is only comparable with VQ.   \n\n \n\n\n\nMinor  \nThe derivation for eq. 13 is not given. Please provide a proof or give a reference here.  \nThere exist some typos in the derivations.  \nIn Line 878, the p shuld be Q.   \nIn line 886, the p should be q^hat.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)