Image and Video Tokenization with Binary Spherical Quantization

## Abstract

Abstract We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ).
BSQ projects the high-dimensional visual embedding to a lower-dimensional hypersphere and then applies binary quantization.
BSQ is (1) parameter-efficient without an explicit codebook, (2) scalable to arbitrary token dimensions, and (3) compact: compressing visual data by up to 100 × \times × with minimal distortion.
Our tokenizer uses a transformer encoder and decoder with simple block-wise causal masking to support variable-length videos as input.
The resulting BSQ-ViT achieves state-of-the-art visual reconstruction quality on image and video reconstruction benchmarks with 2.4 × \times × throughput compared to the best prior methods.
Furthermore, by learning an autoregressive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable results on video compression with state-of-the-art video compression standards.
BSQ-ViT also enables masked language models to achieve competitive image synthesis quality to GAN- and diffusion-based methods.

## 1 Introduction

Learned discrete image and video tokenization allows for state-of-the-art visual compression [, , ], recognition [, , , ] and generation [, , ].
These models follow a proven recipe from large language modeling [, , ]: Tokenize input and outputs into discrete units and learn an auto-regressive model to predict this tokenized stream one token at a time.
The most widely used approach for image encoding is Vector-Quantized
Variational Auto-Encoder (VQ-VAE) [].
They encode inputs in continuous latent embeddings and map them to a learned codebook through nearest-neighbor lookup.
However, VQ-VAE style approaches have two drawbacks:
First, most image encoders are built upon convolutional networks (CNN) [, ].
Adapting spatial convolution for images to spatial-temporal convolution for videos requires non-trivial architectural changes [, , ] with increased computational cost.
Treating videos as a sequence of images leads to a suboptimal quantization [].
Second, vector quantization (VQ) scales poorly with the codebook size.
The runtime scales linearly with the codebook size, and the codebook easily overfits on smaller datasets [].
This is especially troubling for video inputs, as they rely on larger codebooks to represent both static visual patterns and dynamic motion patterns.

This paper proposes a unified visual tokenizer based on a Vision Transformer and Binary Spherical Quantization (BSQ).
The Transformer-based encoder-decoder leverages a block-wise causal mask and uses only visual tokens from the current or past timestamps for reconstruction (Figure 3).
BSQ first projects the high-dimensional visual embedding of the transformer encoder to a lower-dimensional hypersphere and then applies binary quantization.
The transformer encoder, decoder, and BSQ are seamlessly integrated into the VQ-GAN [] framework and trained end-to-end.

Our proposed visual tokenizer features several advantages.
First, the Transformer-based encoder-decoder shows a Pareto improvement in visual reconstruction quality and computational efficiency compared to standard CNNs.
Second, the block-wise causal design unifies images and videos as input at training and supports variable-length videos at inference.
BSQ constructs an implicit codebook whose effective vocabulary grows exponentially with the spherical dimension with no learned parameters.
The increasing codebook size consistently yields better reconstruction results.
Compared to Lookup-free Quantization (LFQ) [], a recent technique that also builds an implicit codebook based on scalar quantization (SQ), BSQ has a bounded quantization error and is easier to train.
Furthermore, we show that the soft quantization probability in BSQ reduces to a simple product of multiple channel-independent Bernoulli distributions, leading to efficient entropy regularization during training.
Specifically, we show how a factorized approximation to the entropy for soft quantization of $L$ bits reduces the theoretical computation complexity from $O(2^{L}\times L)$ to $O(L)$ with minimal approximation error, and negligible performance degradation in practice.

We validate the effectiveness of BSQ-ViT on visual reconstruction and compression benchmarks.
On image reconstruction, our model archives a state-of-the-art visual reconstruction quality by both pixel-level and semantic metrics.
In particular, our best-performing BSQ-ViT achieves a reconstruction FID of 0.41 on ImageNet-1k val, a 43% reduction compared to the runner-up (SDXL-VAE []), while being 2.4$\times$ faster.
On video reconstruction, our best model reduces FVD on UCF-101 by more than half (8.62 $\rightarrow$ 4.10).
By further learning an autoregressive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable results on video compression with state-of-the-art video compression standards, e.g. H.264 and HEVC.
By learning a masked language model, BSQ-ViT enables image generation with similar quality to BigGAN [] and ADM [].
Code and models will be released at [https://github.com/zhaoyue-zephyrus/bsq-vit](https://github.com/zhaoyue-zephyrus/bsq-vit).

## 2 Related Work

Visual Tokenization.
VQ-VAE [] introduced the concept of discrete tokenized bottlenecks in auto-encoder architectures.
Recent improvements include better training objectives [, ], increasing VQ codebook usage [, ], replacing VQ with product quantization (PQ) [] or scalar quantization (SQ) [], and employing stronger generative models [, ].
Image tokenizers are trivially extended to video by tokenizing individual frames [, ].
However, this ignores dynamic motions and leads to suboptimal tokenization: The same visual information is compressed repeatedly across frames.

Video Tokenization.
Dedicated video tokenizers make better use of temporal correlations in the input signal.
VideoGPT [] proposes 3D (de-)convolutions in VQ-VAE for video generation.
TATS [] replaces zero padding with replicate padding to mitigate the temporal corruption when video length varies.
Yu et al. introduce central inflation of pretrained 2D convolutional filters to 3D  [] and further make them causal [].
Phenaki [] adopts a factorized causal video vision Transformer [] (C-ViViT), which improves efficiency but sacrifices modeling complex motion across time.

Neural Compression.
Since Shannon established the fundamental source coding theorem [] it has formed the basis of lossless compression [, , , ] with probabilistic models including RNN [, ], CNN [, ], VAE [, ], and Transformers [, ].
L3C [] presents a fast hierarchical probabilistic model for lossless image compression.
LMIC [] shows that LLMs trained primarily on text, e.g. Llama 2 [] and Chinchilla [], are general-purpose compressors for text, images, and audio.
However, these LLMs are too big and slow to make this compression practical.
Our tokenizer presents a lighter-weight alternative: Tokenization performs initial local lossy compression, while a lightweight and thus computationally efficient sequence model ($\sim$300M) compresses the global video structure.

Video compression.
Most high-performing modern video compression methods rely on hybrid coders that combine transform coding [, ] and motion compensation [, ].
Such belief continues in most of the recently popularized learning-based solutions [, , , ].
VCT [] proposes a Transformer-based temporal entropy model to learn motion implicitly.
However, VCT requires a heavily-engineered image compression model [] and has a short temporal context window.
In this work, we show that a learned video tokenizer combined with an arithmetic coder modeled by a sequence model achieves competitive compression results without explicitly modeling motion.

## 3 Preliminaries

A tokenization-based compression algorithm has three basic steps:
A visual tokenizer, i.e. VQ-VAE [] or LFQ [], translates raw visual inputs to a discrete set of tokens and back.
A sequence model then predicts an auto-regressive probability distribution over these discrete tokens.
Finally, arithmetic coding translates this distribution into a compressed representation.

Visual Tokenization.
VQ-VAE [] introduced the concept of learning discrete visual representation with an auto-encoder architecture and a bottleneck module in between with vector quantization (VQ).
Given a video $\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$, an encoder $\mathcal{E}$ produces a set of $d$-dimensional latent embeddings $\mathbf{Z}=\mathcal{E}(\mathbf{X})\in\mathbb{R}^{\left(\frac{T}{q}\times\frac{
H}{p}\times\frac{W}{p}\right)\times d}$ with a spatial-temporal downsample factor of $q\times p\times p$.
The bottleneck module $q$ then transforms the real-valued latent embeddings into some discrete tokens $\hat{\mathbf{z}}=q(\mathbf{z})$.

In Vector Quantization (VQ) the quantizer $q_{VQ}$ assigns each $\mathbf{z}\in\mathbf{Z}$ to the closet entry in a learnable code in a codebook $\mathbf{C}=[\mathbf{c}_{1}\cdots\mathbf{c}_{K}]\in\mathbb{R}^{K\times d}$

$$ $\displaystyle\hat{\mathbf{z}}=q_{VQ}(\mathbf{z})=\mathbf{c}_{k}=\operatorname* {arg\,min}_{\mathbf{c}_{\hat{k}}\in\mathbf{C}}\|\mathbf{z}-\mathbf{c}_{\hat{k} }\|_{2}.$ (1) $$

Here, $K$ is the vocabulary size of the codebook and the integer $k$ is the discretized token representation of $\mathbf{z}$ which can be stored in $\lceil\log(K)\rceil$ bits.
A decoder $\mathcal{G}$ maps the discretized tokens back into a visual representation $\hat{\mathbf{X}}=\mathcal{G}(\hat{\mathbf{Z}})$.
The entire network ($\mathcal{E}$, $\mathcal{G}$, and $q$) is end-to-end trainable and minimizes an MSE loss $\mathcal{L}_{\mathrm{MSE}}=\|\hat{\mathbf{X}}-\mathbf{X}\|_{2}$ using straight-through estimator [] to propagate gradients through the quantization bottleneck.
More recent quantizers rely on a perceptual $\mathcal{L}_{\mathrm{LPIPS}}$ and adversarial $\mathcal{L}_{\mathrm{GAN}}$ loss for better visual quality []

$$ $\displaystyle\operatorname*{minimize}_{\mathcal{E},\mathcal{G},q}\mathbb{E}_{ \mathbf{X}}\left[\mathcal{L}_{\mathrm{VQ}}(\mathcal{E},\mathcal{G},q)+\eta \mathcal{L}_{\mathrm{LPIPS}}(\mathcal{E},\mathcal{G},q)+\lambda\mathcal{L}_{ \mathrm{GAN}}(\mathcal{E},\mathcal{G},q)\right],$ (2) $$

where the quantization loss term $\mathcal{L}_{\mathrm{VQ}}$ emulates online clustering to learn $\mathbf{c}_{k}$.
The main issue with VQ-VAE is that Vector Quantization scales poorly with increasing vocabulary size $K$ [].
Remedies include using a smaller code dimension [], introducing stochasticity [], reviving “dead” codevectors [], and regularizing with a commitment loss []:

$$ $\displaystyle\mathcal{L}_{\mathrm{commit}}(\hat{\mathbf{z}},\mathbf{z})=\| \operatorname{sg}(\hat{\mathbf{z}})-\mathbf{z}\|,$ (3) $$

where $\operatorname{sg}(\cdot)$ denotes the stop-gradient operation.

Lookup-Free Quantization (LFQ) [] uses a fixed implicit codebook $\mathbf{C}_{LFQ}=\left\{-1,1\right\}^{L}$ as corners of a hypercube in $L$ dimensional space.
The best vector quantizer for this implicit codebook is the binary quantization $q_{LFQ}(\mathbf{z})=\operatorname{sign}(\mathbf{z})$.
To optimize for an effective latent code and encourage usage of the implicit codebook, Yu et al. [] use an additional entropy objective []:

$$ $\displaystyle\mathcal{L}_{\mathrm{entropy}}=\mathbb{E}\left[H(q(\mathbf{z})) \right]-\gamma H\left[\mathbb{E}\left[q(\mathbf{z})\right]\right],$ (4) $$

where both entropy terms rely on a soft quantization []

$$ $\displaystyle\hat{q}(\mathbf{c}|\mathbf{z})=\frac{\exp(-\tau(\mathbf{c}- \mathbf{z})^{2})}{\sum_{\mathbf{c}\in\mathbf{C}_{LFQ}}\exp(-\tau(\mathbf{c}- \mathbf{z})^{2})}$ (5) $$

to guarantee the loss is differentiable.
The final loss $\mathcal{L}_{LFQ}$ is a combination of $\mathcal{L}_{\mathrm{MSE}}$, $\mathcal{L}_{\mathrm{commit}}$, $\mathcal{L}_{\mathrm{LPIPS}}$, $\mathcal{L}_{\mathrm{GAN}}$, and $\mathcal{L}_{\mathrm{entropy}}$.
The main computational bottleneck in LFQ is the entropy optimization of a higher-dimensional codebook, as it involves summation over $2^{L}$ implicit codebook entries.

Both VQ-VAE and LFQ lossily compress visual inputs $\mathbf{X}$ into $N$ discrete tokens $[k_{1},\ldots,k_{N}]$, where $k_{i}\in\{1,\ldots K\}$, in $N\lceil\log{K}\rceil$ bits.
Neither tokenization strategy exploits the global image or video structure well.
A sequence model with lossless arithmetic coding better fits this global structure.

Arithmetic Coding (AC) [, , ] offers a way of constructing a bitstream with near-optimal length by leveraging the statistical property of the coding distribution.
Given a distribution over token streams $P_{t}:\{1,\cdots,K\}^{n}\mapsto(0,1]$, arithmetic coding looks to encode the token stream in $(-\lceil\log P_{t}(k_{1},\ldots,k_{N})\rceil+1)$ bits.
The most common token distribution is an auto-regressive model

$$ $\displaystyle P_{t}(k_{1},\ldots,k_{N})=P_{t}(k_{1})P_{t}(k_{2}|k_{1})\ldots P _{t}(k_{N}|k_{1},\ldots,k_{N-1})$ (6) $$

for which efficient incremental encoding and decoding algorithms exist [].

Figure: (a) BSQ-VAE (Ours).
Refer to caption: x1.png

## 4 Transformer-based Visual Tokenizer with Binary Spherical Quantization

Our video tokenizer follows an encoder-decoder architecture with a discretization bottleneck as illustrated in Figure 1a.
It combines a transformer-based encoder, a transformer-based decoder, and a Binary Spherical Quantization (BSQ) layer.
BSQ projects the latent code into a lower-dimensional spherical space, applies binary quantization, and then projects the result back up into the decoder’s latent space.
This projection onto a low-dimensional spherical space has several theoretical advantages: The approximation error of the quantizer is bounded and much of the entropy computation factorizes along individual dimensions.
These advantages result in experimental improvements as well.
BSQ converges quicker and to a better tokenizer than other quantization schemes.

### 4.1 Binary Spherical Quantization

Binary Spherical Quantization (BSQ) optimizes over an implicit codebook $\mathbf{C}_{BSQ}=\{-\frac{1}{\sqrt{L}},\frac{1}{\sqrt{L}}\}^{L}$, a hypercube projected onto a unit sphere.
Each corner $\mathbf{c}_{k}\in\mathbf{C}_{BSQ}$ of a hypercube corresponds to a unique token $k$.
The quantizer works as follows: it projects some high-dimensional latent embedding $\mathbf{z}$ to a lower-dimensional unit hypersphere $\mathbf{u}$, applies binary quantization per axis ${\hat{\mathbf{u}}=\operatorname{sign}(\mathbf{u})}$, and back-projects to the quantized vector in the original latent space $\hat{\mathbf{x}}$, as shown in Figure 1a.
Specifically, we start with an encoded visual input $\mathbf{z}=\mathcal{E}(\mathbf{x})\in\mathbb{R}^{d}$.
We first linearly project the latent embedding to $L$ dimensions $\mathbf{v}=\mathrm{Linear}(\mathbf{z})\in\mathbb{R}^{L}$, where $L\ll d$.
Next, we obtain project $\mathbf{v}$ onto the unit sphere $\mathbf{u}=\frac{\mathbf{v}}{|\mathbf{v}|}$, and perform binary quantization to each dimension of $\mathbf{u}$ independently $\hat{\mathbf{u}}=\frac{1}{\sqrt{L}}\operatorname{sign}(\mathbf{u})$, where $\operatorname{sign}(x)$ is the sign function.
To keep outputs on the unit sphere, we map $\operatorname{sign}(0)\to 1$.
We use a Straight-Through Estimator (STE) [] to make the operator differentiable, $\operatorname{sign}_{\mathrm{STE}}(x)=\operatorname{sg}(\operatorname{sign}(x)
-x)+x$, where $\operatorname{sg}(\cdot)$ denotes the stop-gradient operation.
Finally, we back-project the quantized $\hat{\mathbf{u}}$ to the $d$-dimensional space $\hat{\mathbf{z}}=\mathrm{Linear}(\hat{\mathbf{u}})\in\mathbb{R}^{d}$.

BSQ has a few appealing properties:
As with LFQ, the implicit codebook entry is parameter-free and easy to compute.
Unlike LFQ, a soft quantization of BSQ has a simple probabilistic interpretation, which leads to efficient entropy computation in an entropy loss $\mathcal{L}_{\mathrm{entropy}}$.
Finally, BSQ’s quantization error is bounded, which empirically leads to much faster and better convergence than LFQ.

Efficient implicit code assignment.
At inference time, we map a projected embedding $\mathbf{v}$ to a token through simply binarization $k=\sum_{i=1}^{L}1_{[v_{i}>0]}2^{i-1}$, where $1_{[\cdot]}$ is the indicator function.
The inverse mapping uses the bitshift and the bitwise AND operations.

Soft BSQ and entropy.
To best use the entire range of the implicit codebook $\mathbf{C}_{BSQ}$, we use the entropy loss $\mathcal{L}_{\mathrm{entropy}}=\mathbb{E}_{\mathbf{u}}\left[H(q(\mathbf{u}))
\right]-\gamma H\left[\mathbb{E}_{\mathbf{u}}\left[q(\mathbf{u})\right]\right]$ [].
To compute this entropy loss we first derive a soft quantization scheme [].
Since both codebook entries and inputs to the quantizer are unit vectors, the soft quantization is a distribution

$$ $\displaystyle\hat{q}(\mathbf{c}|\mathbf{u})=\frac{\exp(\tau\mathbf{c}^{\top} \mathbf{u})}{\sum_{\mathbf{c}\in\mathbf{C}_{BSQ}}\exp(\tau\mathbf{c}^{\top} \mathbf{u})}=\prod_{d=1}^{L}\sigma\left(2\tau c_{d}u_{d}\right),$ (7) $$

where $\sigma$ is a sigmoid function, and the overall soft quantizer is independent along each dimension.
See Sec. C.1 for a derivation.
This form allows for an efficient computation of the first entropy term

$$ $\mathbb{E}_{\mathbf{u}}\left[H(\hat{q}(\mathbf{c}|\mathbf{u}))\right]=\mathbb{ E}_{\mathbf{u}}\left[\sum_{d=1}^{L}H(\hat{q}(c_{d}|u_{d}))\right].$ (8) $$

See Sec. C.2 for a derivation.
Instead of reasoning over distributions over the entire codebook, which is exponentially large, we instead treat each dimension independently.
The resulting entropy computation is linear to the dimension $L$ of the bottleneck.

Unfortunately, the second entropy term cannot directly use the same independence assumption, as dimensions in the expected value $\mathbb{E}_{\mathbf{u}}[\hat{q}(\mathbf{c}|\mathbf{u})]$ are correlated through the distribution of $\mathbf{u}$.
We find the closest factorized distribution $\tilde{q}(\mathbf{c})=\prod_{d=1}^{K}\tilde{q}(c_{d})$ to $\mathbb{E}_{\mathbf{u}}[\hat{q}(\mathbf{c}|\mathbf{u})]$, and instead minimize the entropy of the approximate distribution.
As we will show in Sec. C.3 the best approximation in terms of the KL-divergence $\tilde{q}(c_{d})=\mathbb{E}_{\mathbf{u}_{d}}[\hat{q}(\mathbf{c}_{d}|\mathbf{u}
_{d})]$.
The final approximate entropy term to maximize is

$$ $H(\mathbb{E}_{\mathbf{u}}\left[\hat{q}(\mathbf{c}|\mathbf{u})\right])\approx H (\tilde{q}(\mathbf{c}))=\sum_{d=1}^{L}H(\mathbb{E}_{\mathbf{u}_{d}}[\hat{q}( \mathbf{c}_{d}|\mathbf{u}_{d})]).$ (9) $$

As we will show in Sec. C.3 this approximation is an upper bound to the true entropy, but empirically closely tracks the true entropy.
This entropy term is again efficient for evaluation.

Quantization error in BSQ.
Most quantizers use pass-through gradient estimates during training [, , ].
Though simple to implement, it assumes that the gradients for an unquantized $\mathbf{u}$ and quantized $\hat{\mathbf{u}}$ bottleneck are almost the same, which only holds if the quantization error $d(\mathbf{u},\hat{\mathbf{u}})=\|\mathbf{u}-\hat{\mathbf{u}}\|$ is small.
As we show in Sec. C.4,
this is true for BSQ

$$ $\mathbb{E}_{\mathbf{u}}\left[d(\mathbf{u},\hat{\mathbf{u}})\right]<\sqrt{2- \frac{2}{\sqrt{L}}}<\sqrt{2}.$ (10) $$

Figure: (a) BSQ
Refer to caption: x4.png

Relation to other quantization methods.
BSQ is closely connected to many concepts introduced in information and coding theories.
LFQ [] uses the same binarization technique as BSQ but does not normalize its output.
This leads to an unbounded quantization error and does not allow for as simple of a soft quantization for entropy computation.
A pictural comparison between LFQ and BSQ is shown in Figure 2 and a summary is provided in Table 7.
Spherical Vector Quantization (SVQ) [] also ensures all code vectors have a pre-defined radius.
However, SVQ assumes a variety of radii, which have to be encoded by an additional gain quantizer.
In our case, the source code is the output of a learned encoder $\mathcal{E}$.
Therefore, the unit radius assumption is sound, and the gain quantizer can be avoided.
Pyramid Vector Quantization (PVQ) [] assumes all code vectors have a constant $\ell_{1}$ norm, but the $\ell_{1}$ normalized centroids partition the hypersphere less uniformly than $\ell_{2}$.

### 4.2 Tokenization Network with Causal Video Transformer

We propose to use Vision Transformer (ViT) [] to model both the encoder and decoder due to its better computational efficiency and higher reconstruction quality.

Video Transformer.
We start from ViT-VQGAN [] and extend it to take videos as input.
We divide an input video $\mathbf{X}\in\mathbb{R}^{T\times H\times W\times 3}$ into non-overlapping patches of size $1\times p\times p$, $\mathbf{x}_{i}\in\mathbb{R}^{1\times p\times p\times 3}$.
The visual tokens are flattened into a 1D sequence, linearly projected, and passed through a stack of $N$ Transformer Encoder layers to yield the latent representation, $(\mathbf{z}_{1},\cdots,\mathbf{z}_{N})$.
The decoder takes the same architecture, maps the latent embeddings $\hat{\mathbf{z}}$ back to the pixel space, and regroups them into the original shape.
$(\hat{\mathbf{x}}_{1},\cdots,\hat{\mathbf{x}}_{N})=\mathrm{MLP}(\mathrm{
TransformerDecoder}(\hat{\mathbf{z}}_{1},\cdots,\hat{\mathbf{z}}_{N}))$, where $\mathrm{MLP}$ is a decoding head with a two-layer MLP, i.e. $\mathrm{Linear}\circ\mathrm{Tanh}\circ\mathrm{Linear}$.

Blockwise Causal Attention.
During training, we always assume the input video has $T$ frames, which might not hold at inference.
Padding shorter video segments to $T$ frames works but wastes a lot of bits, especially in the context of compression.
To handle variable-length videos, we propose a simple blockwise causal masked attention analogous to causal attention in language modeling [].
It specifies that only those tokens at time $t$ or earlier can be used for reconstructing the visual tokens at time $t\in\{1,\cdots,T\}$.

$$ $\displaystyle(\mathbf{z}_{(t-1)\times\frac{H}{p}\times\frac{W}{p}+1},\cdots, \mathbf{z}_{t\times\frac{H}{p}\times\frac{W}{p}})$ $\displaystyle=\mathrm{TransformerEncoder}\left(\mathbf{x}_{1},\cdots,\mathbf{x }_{t\times\frac{H}{p}\times\frac{W}{p}}\right),$ (11) $\displaystyle(\hat{\mathbf{z}}_{(t-1)\times\frac{H}{p}\times\frac{W}{p}+1}, \cdots,\hat{\mathbf{z}}_{t\times\frac{H}{p}\times\frac{W}{p}})$ $\displaystyle=q_{LFQ}\left(\mathbf{z}_{(t-1)\times\frac{H}{p}\times\frac{W}{p} +1},\cdots,\mathbf{z}_{t\times\frac{H}{p}\times\frac{W}{p}}\right),$ (12) $\displaystyle(\hat{\mathbf{x}}_{(t-1)\times\frac{H}{p}\times\frac{W}{p}+1}, \cdots,\hat{\mathbf{x}}_{t\times\frac{H}{p}\times\frac{W}{p}})$ $\displaystyle=\mathrm{MLP}\left(\mathrm{TransformerDecoder}\left(\hat{\mathbf{ z}}_{1},\cdots,\hat{\mathbf{z}}_{t\times\frac{H}{p}\times\frac{W}{p}}\right) \right).$ (13) $$

This can be efficiently implemented with a blockwise causal attention mask written in a blockwise lower triangle matrix in Figure 3.
When $T=1$, the proposed encoder-decoder reduces to a ViT with a full attention mask.
Therefore, we can easily train it using a mixture of images and videos.

We use factorized spatial-temporal position embedding to encode the temporal position information.
Specifically, we add a set of zero-initialized temporal position embeddings $\mathrm{PE}_{t}\in\mathbb{R}^{T\times d}$ and add it to the original spatial position embedding $\mathrm{PE}_{s}\in\mathbb{R}^{N\times d}$ in the image tokenizer, i.e. $\mathrm{PE}[i,:,:]=\mathrm{PE}_{t}[i,\texttt{None},:]+\mathrm{PE}_{s}[\texttt{
None},:,:]$.

Figure: Figure 3: Given an input video, block-wise causal masked attention enables the Transformer encoder to only use the flattened patches from current or past timestamps to encode each visual patch and later translate it into a BSQ code.
Refer to caption: x7.png

Training the Video Tokenizer from an Image Tokenizer.
Due to the lack of diversity in existing video datasets, we first train an image tokenizer on image data and then fine-tune it to be a video tokenizer.
Though previous works [, ] argue that a pre-trained image tokenizer can be used for videos as is, we observe that the video tokenizer after fine-tuning demonstrates much higher reconstruction quality on video benchmarks, see Sec. 2.
The gain is further magnified when the effective vocabulary size becomes larger.
We hypothesize that such increased vocabulary size, enabled by the proposed BSQ, is handy for learning video-specific motion and blur.
In contrast, vanilla VQ methods fail to maintain high codebook usage when the codebook size exceeds 16K.

Optimizing the Visual Tokenizer.
Following VQGAN [], we use a perceptual loss [] and an adversarial loss [].
We use StyleGAN [] as the discriminator since ViT-VQGAN [] reports it is much easier to train than PatchGAN [].
When we fine-tuned the tokenizer on videos, unlike MAGVIT or TATS, we did not inflate StyleGAN to be a 3D discriminator.
Instead, we pass all reconstructed frames individually to the vanilla StyleGAN and sum up the losses.

## 5 Experiments

We train the image tokenization model on the training set of ImageNet ILSVRC2012 [] and evaluate the image reconstruction result on the validation set of MS-COCO [] and ImageNet, denoted by COCO 2017val and ImageNet-1k respectively.
We fine-tune the video tokenization model on UCF-101 [] and conduct video compression experiments on two standard benchmarks, i.e. MCL-JCV and UVG.
We leave dataset statistics and implementation details in Sec. E.

Evaluation metrics.
For image/video tokenization, we report perceptual metric (LPIPS-AlexNet) [], PSNR, SSIM [], and Fréchet Inception/Video Distance (FID/FVD) [, ].
To distinguish it from generation, we denote it as rFID/rFVD.
For generation, we report FID, Inception Score (IS) [], and improved precision and recall (IPR, Prec, and Rec) [].
For compression, we report PSNR and MS-SSIM [] under different levels of bits per pixel (bpp).

### 5.1 Main Results

**Table 1: Image reconstruction results on COCO2017 and ImageNet-1K ($256\times 256$). The “data” column refers to the training data: CC for CC3M, YF for YFCC100M, OImg for OpenImages, LAION for LAION-5B, IN for ImageNet, and “?” for unknown source. The “arch.” column shows the encoder/decoder architecture: C for ConvNets with Self-Attention, and T-B for ViT-Base. The “# bits” column refers to the effective number of bits per token defined in Sec. 1. ^# is obtained by multiplying the latent dimension with the precision. The “TP” column means the inference throughput (images/second) per GPU. ^†The number is taken from the paper. Note that STDs of PSNR, SSIM, and LPIPS are computed across samples instead of multiple runs.**
|  | COCO2017 val | ImageNet-1k val |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Data | Arch. | Quant. | Param. | # bits | TP↑ | PSNR↑ | SSIM↑ | LPIPS↓ | rFID↓ | PSNR↑ | SSIM↑ | LPIPS↓ | rFID↓ |
| DALL-E dVAE [] | CC+YF | C | VQ | 98M | 13 | 34.0 | 25.15 | .7497 | .3014 | 55.07 | 25.46 | .7385 | .3127 | 36.84 |
|  |  |  |  |  |  |  | $\pm$3.49 | $\pm$.1124 | $\pm$.1221 |  | $\pm$3.93 | $\pm$.1343 | $\pm$.1480 |  |
| MaskGIT [] | IN-1k | C | VQ | 54M | 10 | 37.6 | 17.52 | .4194 | .2057 | 8.90 | 17.93 | .4223 | .2018 | 2.23 |
|  |  |  |  |  |  |  | $\pm$2.75 | $\pm$.1619 | $\pm$.0473 |  | $\pm$2.93 | $\pm$.1827 | $\pm$.0543 |  |
| ViT-VQGAN [] | IN-1k | T-B | VQ | 182M | 13 | ^†7.5 | - | - | - | - | - | - | - | ^†1.55 |
| SD-VAE 1.x [] | OImg | C | VQ | 68M | 10 | 22.4 | 21.78 | .6139 | .1042 | 6.79 | 22.12 | .6046 | .1039 | 1.52 |
|  |  |  |  |  |  |  | $\pm$3.41 | $\pm$.1430 | $\pm$.0345 |  | $\pm$3.79 | $\pm$.1663 | $\pm$.0409 |  |
| SD-VAE 1.x [] | OImg | C | VQ | 68M | 14 | 22.4 | 22.54 | .6470 | .0905 | 6.07 | 22.82 | .6354 | .0912 | 1.23 |
|  |  |  |  |  |  |  | $\pm$3.55 | $\pm$.1409 | $\pm$.0323 |  | $\pm$3.97 | $\pm$.1644 | $\pm$.0390 |  |
| SD-VAE 1.x [] | OImg | C | KL | 68M | ^#64 | 22.4 | 21.68 | .6375 | .0985 | 5.94 | 21.99 | .6275 | .0980 | 1.35 |
|  |  |  |  |  |  |  | $\pm$3.32 | $\pm$.1375 | $\pm$.0309 |  | $\pm$3.74 | $\pm$.1600 | $\pm$.0371 |  |
| SD-VAE 2.x [] | OImg+ | C | KL | 84M | ^#64 | 18.9 | 24.82 | .7202 | .0694 | 4.63 | 25.08 | .7054 | .0731 | 0.78 |
|  | LAION |  |  |  |  |  | $\pm$3.64 | $\pm$.1241 | $\pm$.0344 |  | $\pm$4.11 | $\pm$.1469 | $\pm$.0448 |  |
| SDXL-VAE [] | OImg+ | C | KL | 84M | ^#64 | 18.9 | 25.11 | .7433 | .0623 | 4.23 | 25.38 | .7276 | .0666 | 0.72 |
|  | LAION+? |  |  |  |  |  | $\pm$3.91 | $\pm$.1240 | $\pm$.0289 |  | $\pm$4.41 | $\pm$.1469 | $\pm$.0373 |  |
| Ours | IN-1k | T-B | BSQ | 174M | 18 | 45.1 | 25.08 | .7662 | .0744 | 5.81 | 25.36 | .7578 | .0761 | 1.14 |
|  |  |  |  |  |  |  | $\pm$3.57 | $\pm$.0993 | $\pm$.0295 |  | $\pm$4.02 | $\pm$.1163 | $\pm$.0358 |  |
| Ours | IN-1k | T-B | BSQ | 174M | 36 | 45.1 | 27.64 | .8485 | .0412 | 3.42 | 27.88 | .8410 | .0432 | 0.41 |
|  |  |  |  |  |  |  | $\pm$3.74 | $\pm$.0704 | $\pm$.0199 |  | $\pm$4.26 | $\pm$.0821 | $\pm$.0253 |  |
| Ours (w/. EMA) | IN-1k | T-B | BSQ | 174M | 36 | 45.1 | 27.92 | .8526 | .0380 | 3.34 | 28.14 | .0814 | .0400 | 0.45 |
|  |  |  |  |  |  |  | $\pm$3.78 | $\pm$.0698 | $\pm$.0187 |  | $\pm$4.32 | $\pm$.0814 | $\pm$.0237 |  |

Image Reconstruction.
We first compare the image reconstruction result of BSQ on COCO and ImageNet ($256\times 256$) with state-of-the-art image tokenizers, including
DALL-E dVAE [], SD-VAE 1.x [], SD-VAE 2.x, SDXL-VAE [], MaskGIT [], and ViT-VQGAN [].
We observe that reconstruction metrics vary with many factors, especially preprocessing (e.g. interpolation), input resolution, and downsample scales (Sec. D.2 in []).
To perform a comprehensive and fair comparison, we resize all images such that the smaller edge is 256 pixels using Lánczos interpolation(^1^11The reconstruction result of bilinear interpolation is computed in Table 8 for reference. In short, the conclusion is that varying interpolation changes the values but unalts the order of all methods.), take the center crop $(H\times W)=(256\times 256)$, and ensure all models have the same spatial downsample ratio of $p=8$ (except for MaskGIT, $p=16$).
We rerun all models on COCO 2017val and ImageNet-1k val except the undisclosed ViT-VQGAN.
From Table 1, we can see that our model outperforms prior works on all metrics (PSNR, SSIM, LPIPS, and rFID), often by a big margin.

To compare the compression capability of different bottleneck modules, We study the effective number of bits per token (# bits).
For VQ-based models, # bits equals to $\log_{2}(K)$, where $K$ is the codebook size;
For KL-regularized models (SD-VAE 2.x and XL), since the latent is continuous, we count # bits as the latent dimension multiplied by the numeric precision (here we use 16 since the checkpoint is stored in FP16).
For our BSQ, # bits is $L$ because each latent channel is binary.
We summarize the key observations as follows.
(1) BSQ efficiently compresses image patches into a small amount of bits.
It reconstructs images better in all metrics using fewer bits per token than the second-best method (SDXL-VAE).
(2) BSQ is also computationally efficient.
Although the ViT-based backbone doubles the parameters, our method yields a 2.4$\times$ higher throughput than SDXL-VAE.
MaskGIT runs at a comparable speed but reconstructs significantly worse because of a small codebook size (1024) and more spatial downsampling ($16\times$).
(3) BSQ is generalizable across different domains of images. ImageNet is relatively object-centric while COCO is more scene-centric.
Though trained on ImageNet only, our method does well on the scene-centric COCO too.
It even works better than SD-VAE 1.x/2.x trained on the similarly scene-centric OpenImages dataset [].

**Table 2: Video reconstruction results on UCF-101 (split 1).**
|  | UCF-101 train | UCF-101 val |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Backbone | Quantizer | Param. | # bits | PSNR↑ | SSIM↑ | LPIPS↓ | rFVD↓ | PSNR↑ | SSIM↑ | LPIPS↓ | rFVD↓ |
| (Image Tokenizer, w/o adapting to videos) |  |  |  |  |  |  |  |  |  |  |  |  |
| Ours | ViT | VQ | 174M | 14 | 25.64 | .8142 | .1120 | 357 | 25.58 | .8120 | .1146 | 382 |
| Ours | ViT | BSQ | 174M | 18 | 25.86 | .8273 | .1089 | 326 | 25.83 | 0.8259 | 0.1108 | 342 |
| (Image Tokenizer $\rightarrow$ Video Tokenizer) |  |  |  |  |  |  |  |  |  |  |  |  |
| MaskGIT [] | 2D CNN | VQ | 53M | 10 | 21.5 | .685 | 0.114 | 216 | - | - | - | - |
| TATS [] | 3D CNN | VQ | 32M | 14 | - | - | - | 162 |  |  |  |  |
| MAGVIT-L [] | 3D CNN | VQ | 158M | 10 | 22.0 | .701 | .0990 | 25 | - | - | - | - |
| MAGVIT-v2 [] | C.-3D CNN | LFQ | 158M | 18 | - | - | .0694 | 16.12 | - | - | - | - |
| MAGVIT-v2 [] | C.-3D CNN | LFQ | N/A (>158M) | 18 | - | - | .0537 | 8.62 | - | - | - | - |
| Ours | non-BC ViT | VQ | 174M | 14 | 33.06 | .9518 | .0223 | 9.16 | 32.92 | .9506 | .0228 | 12.79 |
| Ours | BC ViT | VQ | 174M | 14 | 32.81 | .9496 | .0236 | 10.76 | 32.68 | .9484 | .0241 | 14.17 |
| Ours | BC ViT | BSQ | 174M | 18 | 32.08 | .9421 | .0244 | 8.08 | 31.49 | .9357 | .0276 | 11.62 |
| Ours | BC ViT | BSQ | 174M | 36 | 33.80 | .9606 | .0159 | 4.10 | 33.55 | .9588 | .0167 | 6.21 |

Video Reconstruction.
We present the video reconstruction on both UCF-101 training and validation subsets in Table 2.
First, we use the image tokenizer to reconstruct the video frame by frame.
BSQ works slightly better than VQ but neither is comparable to the specialized video tokenizers fine-tuned on video data shown in the lower half of Table 2.
Second, we finetune the image tokenizer on videos and see significant improvements.
For example, our 18-bit BSQ with causal ViT reduces rFVD from 342 to 11.62 and improves PSNR from 25.83 to 31.49 dB.
The compared prior methods include: (1) MaskGIT [] which is a fine-tuned 2D-CNN based tokenizer, (2) TATS [] which uses a 3D CNN with replicated padding, (3) MAGVIT [] whose 3D CNN is initialized by zero-inflating a 2D filter, and (4) MAGVIT-v2 [] which makes 3D CNN causal.
Since most methods do not release checkpoints, we take their reported numbers directly.
Our models with all configurations outperform MAGVIT-v2 with a comparable number of parameters (174M vs. 158M) by a large margin.
The best-performing MAGVIT-v2 uses a larger backbone and achieves a rFVD of 8.62.
Our causal BSQ-ViT with $L=18$ achieves an 8.08 rFVD and halves the LPIPS.
For BSQ with $L=36$, our method further improves the reconstruction metrics.

We also show the effect of using block-wise causal masks.
The non-causal variant (non-BC) works slightly better on all metrics because now the model can look at all visual patches within the temporal context window.
This result resembles the observations in video compression that using bidirectional predicted pictures (B-frames) benefits compression quality given the same group of pictures (GoP).

**Table 3: Image generation results on ImageNet-1K ($128\times 128$). ^†The number is taken from the paper.**
| Category | Method | # steps | FID↓ |  | IS↑ | Prec↑ | Rec↑ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GAN | BigGAN [] | 1 | 6.02 |  | 145.8 | 0.86 | 0.35 |
| Diffusion | ADM [] | 1,000 | 5.91 |  | 93.3 | 0.70 | 0.65 |
| Masked LM | VQ | 12 | ^†9.4 |  | - | - | - |
| FSQ [] | 12 | ^†8.5 |  | - | - | - |  |
| BSQ (Ours) | 32 | 5.44 |  | 139.6 | 0.80 | 0.50 |  |

Image Generation.
Our BSQ-ViT tokenizer can be seamlessly integrated into existing generative models for visual generation.
We follow MaskGIT [], a masked language modeling approach.
At training time, the underlying masked language model (masked LM) learns to predict the masked tokens given a random proportion of unmasked tokens like BERT [].
At inference time, the model repeats decoding in a non-autoregressive way [] for several steps and progressively decodes from an all-masked canvas to visually plausible contents following a pre-defined unmasking schedule.
Unlike MaskGIT with a VQ-VAE with $K=1024$, BSQ-ViT has an effective vocabulary size of $2^{L}$ and $L=18$, resulting in a slow embedding lookup.
We fix it by dividing each token into groups and treating sub-tokens independently with a similar rationale in Sec. 4.1.
We increase the number of decoding steps accordingly.
Table 3 shows that the masked LM with BSQ outperforms those with VQ and FSQ reported in [].
Our method achieves comparable results with other generation paradigms such as GAN-based [] and diffusion-based [] approaches.
We leave qualitative results in Sec. F.

**Table 4: Comparisons of encoding/decoding speed. ^†The number did not include the image encoder according to [].**
| Method | Resolution | Encode | EC | Decode | FPS |
| --- | --- | --- | --- | --- | --- |
| VCT [] | 1920$\times$1080 | ^†494 ms | 30.5 ms | 168 ms | 1.4 |
| H.264 | 1920$\times$1080 | - | - | - | 2.6 |
| Ours | 1920$\times$1080 | 55.8 ms | 42.2 ms | 64.8 ms | 6.1 |
| VCT [] | 640$\times$360 | ^†22.2 ms | 4.24 ms | 10.1 ms | 27.3 |
| H.264 | 640$\times$360 | - | - | - | 22.4 |
| Ours | 640$\times$360 | 6.2 ms | 4.69 ms | 7.2 ms | 55.2 |

Video Compression.
We compare the compression result on MCL-JCV and UVG in Figure 4.
Simply flattening the video token sequence to a bitstream achieves an MS-SSIM of 0.9818 at 0.2333 bpp.
Although this is not great, we use an auto-regressive model to predict the conditional probability such that the bpp is reduced by 41%.
This leads to a better tradeoff than standard video codecs including both H.264 and HEVC.

Figure: (a) PSNR on MCL-JCV.

On UVG 1080P, our model is comparable to H.264 while being worse than HEVC and VCT [].
Note that our model trains on UCF-101 which only has 9K 320$\times$240 video clips encoded in MPEG-4 while VCT has been trained on a million high-resolution Internet video clips.
We hypothesize that the gap will be mitigated by adding more diverse videos and removing compression artifacts from the training videos.
Nevertheless, we show the potential advantage of our method in encoding and decoding speed in Table 4.
Due to the simplicity of the Transformer-based encoder and decoder, our method runs faster than VCT.

### 5.2 Ablation Studies

For ablation studies, we train an ImageNet image tokenizer with resolution 128$\times$128 with $p=8$, although
our conclusions generally hold for higher resolution, e.g. 256$\times$256 in Sec. 5.1.

BSQ vs VQ.
Table 5 shows that BSQ and VQ follow a similar trend: better reconstruction for increased $L$.
Since $K=2^{18}$ results in an out-of-memory issue, we try a smaller $K=2^{16}=65536$ for VQ.
The gain for VQ already diminishes even though the small bottleneck dimension of $8$ still guarantees full code usage.
In contrast, BSQ consistently works better on all metrics when $L=18$.

**Table 5: Abalation studies on ImageNet-1k val 128$\times$128.**
| Method | $\ell_{2}$-norm | # bits ($K\times d$ or $L$) | PSNR↑ | SSIM↑ | LPIPS↓ | rFID↓ | Code usage |
| --- | --- | --- | --- | --- | --- | --- | --- |
| VQ | ✓ | 10 (1024$\times$32) | 23.61$\pm$3.21 | .6873$\pm$.1211 | .1214$\pm$.0434 | 7.05 | 57.5% |
| ✓ | 14 (16384$\times$8) | 25.76$\pm$3.46 | .7834$\pm$.0988 | .0669$\pm$.0282 | 4.27 | 100.0% |  |
| ✓ | 16 (65536$\times$8) | 25.67$\pm$3.36 | .7851$\pm$.0962 | .0706$\pm$.0283 | 6.61 | 100.0% |  |
| BSQ | ✓ | 10 | 24.11$\pm$3.25 | .7250$\pm$.1121 | .0919$\pm$.0338 | 4.51 | 100.0% |
| ✓ | 14 | 25.26$\pm$3.31 | .7710$\pm$.0992 | .0784$\pm$.0293 | 4.60 | 99.8% |  |
| ✓ | 18 | 25.97$\pm$3.37 | .7990$\pm$.0906 | .0629$\pm$.0261 | 2.66 | 93.8% |  |
| LFQ | ✗ | 18 | 18.58$\pm$2.10 | .4828$\pm$.1340 | .2951$\pm$.0806 | 30.7 | 0.6% |

Importance of $\ell_{2}$ normalization in BSQ.
We remove the $\ell_{2}$ normalization in BSQ, which is equivalent to LFQ, and show results in the last rows of Table 5.
We see much lower code usage and worse rFID, indicating that LFQ does not work well with a ViT-based tokenization encoder.

Contribution of losses.
We study the effect of each loss in Table 6a.
Although it is computationally prohibitive to enumerate all combinations of loss terms and their associative weights, we conduct a simple “leave-one-out” setting where one of the losses is removed at a time.
BSQ works slightly better after removing $\mathcal{L}_{\mathrm{commit}}$ and $H(p(\mathbf{c}|\mathbf{u}))$.
However, the code usage varies greatly.
The best configuration is to keep the minimal entropy term while dropping the commitment loss.
The commit loss may be unnecessary because the quantization error in BSQ is already strictly bounded.
On the contrary, the dataset entropy maximization term and perceptual term do matter.
Without $-H(\mathbb{E}_{\mathbf{u}}[p(\mathbf{c}|\mathbf{u})])$, rFID increases to 13.8 while the code usage in the validation set significantly drops to 13.3%.
We also observe that the perceptual loss is important for low FID and high code usage.
However, a deeper look into its role is beyond the scope of this paper.

Approximating the dataset entropy term.
We now show the efficacy of approximating the dataset entropy term using Eq (9).
We compare with the approximation method in [] that computes entropy in sub-groups of dimensions with varying group size $g\in\{9,6,3\}$.
Our approximation method can also be interpreted as a group size of $g=1$.
From Table 6b, we conclude that our approximation achieves a similar level of rFID and code usage compared to other setups while running the fastest.

**Table 6: Ablation studies of the loss design.**
| $\mathcal{L}_{\mathrm{commit}}$ | $\mathcal{L}_{\mathrm{entropy}}$ | $\mathcal{L}_{\mathrm{LPIPS}}$ |  | rFID | Code |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | $H(p(\mathbf{c}|\mathbf{u}))$ | $-H(\mathbb{E}[p(\mathbf{c}|\mathbf{u})])$ |  |  |  | usage |
| ✓ | ✓ | ✓ | ✓ |  | 2.95 | 45.6% |
| ✗ | ✓ | ✓ | ✓ |  | 2.83 | 93.8% |
| ✓ | ✗ | ✓ | ✓ |  | 2.44 | 78.3% |
| ✓ | ✓ | ✗ | ✓ |  | 13.8 | 13.3% |
| ✓ | ✓ | ✓ | ✗ |  | 19.2 | 6.9% |

## 6 Conclusions

We present a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ).
The transformer-based architecture effortlessly integrates image and video tokenization over an arbitrary time horizon.
The Binary Spherical Quantization allows for efficient and effective training of the quantized bottleneck.
Our results indicate that the proposed tokenizer runs at a faster speed, reconstructs with higher fidelity, and in combination with a sequence model offers a strong baseline for lossy video compression and image synthesis.

## Appendix A Arithmatic Coding Details

Starting from the initial interval $I_{0}=[0,1)$, the AC encoder recursively partitions the interval into a series of sub-interval $I_{n}=[l_{n},u_{n})$ such that $I_{n}\subset I_{n-1}\subset\cdots\subset I_{0}$, and
$I_{n}$ is determined by $I_{n-1}$ and $\rho(y|x_{<n})$.

$$ $\displaystyle{I}_{n}(y)=\left[l_{n-1}+(u_{n-1}-l_{n-1})\sum_{y=1}^{x_{n}-1} \rho(y|x_{<n}),\quad l_{n-1}+(u_{n-1}-l_{n-1})\sum_{y=1}^{x_{n}}\rho(y|x_{<n}) \right).$ (14) $$

Any number in the final interval $I_{N}$ can sufficiently represent the encoded sequence. To obtain the final bit stream, we take a binary fraction $\lambda=\sum_{i=1}^{C}b_{i}\times 2^{-i},~{}b_{i}\in\{0,1\}$ in $I_{N}$ such that $l_{N}\leq\lambda<u_{N}$. The bit stream $\{b_{0},\ldots,b_{C}\}$ is the encoding result with a length of $C$ bits.

The AC decoder takes in $\lambda$, starts with $I_{0}$, and performs a similar interval partitioning process.
At the $n$-th step, the decoder queries the model $\rho_{n}(y|x_{<n})$, calculate the sub-intervals for all possible values of $y$ using Eq. (14), and decodes output $x_{n}$ that leads to $\lambda\in I_{n}(x_{n})$.
The decoder can recover the encoded token sequence by continuing with $I_{n+1}$ based on the decoded $x_{n}$ and repeating for step $n+1$ for $N$ steps.

In practice, the encoder and the decoder can be implemented efficiently with fixed-length integer numbers and operate incrementally for arbitrarily long input sequences.

## Appendix B Comparison between BSQ and LFQ

In Sec. 4.1, we have introduced the mechanism of BSQ and briefly discussed the connections and differences with LFQ.
We summarize them in Table 7.
Note that STE gradient in BSQ is anisotropic and is more likely to be a good estimation because of an upper-bounded quantization error regardless of $L$.
This property explains why a commitment loss like $\mathcal{L}_{\mathrm{commit}}(\hat{\mathbf{u}},{\mathbf{u}})$ is not needed in BSQ but useful for LFQ.

**Table 7: Comparing BSQ and LFQ.**
|  | LFQ [] | BSQ (Ours) |
| --- | --- | --- |
| Quantized output | $\hat{\mathbf{v}}=\operatorname{sign}(\mathbf{v})$ | $\hat{\mathbf{u}}=\frac{1}{\sqrt{L}}\operatorname{sign}(\mathbf{u})=\frac{1}{<br>\sqrt{L}}\operatorname{sign}(\frac{\mathbf{v}}{|\mathbf{v}|})$ |
| STE gradient | $\frac{\partial\hat{v}_{i}}{\partial v_{i}}=1$ | $\frac{\partial\hat{u}_{i}}{\partial v_{i}}=\frac{1}{\sqrt{L}}(1-{v_{i}^{2}}/{|<br>\mathbf{v}|^{2}})$ |
| Quantization Error | $\mathbb{E}_{\mathbf{v}}\left[d(\mathbf{v},\hat{\mathbf{v}})\right]=\infty$ | $\mathbb{E}_{\mathbf{u}}\left[d(\mathbf{u},\hat{\mathbf{u}})\right]<\sqrt{2-<br>\frac{2}{\sqrt{L}}}<\sqrt{2}$ |
| Unbounded | Upper-bounded (See Sec. C.4) |  |
| Training objective | $\mathcal{L}_{\mathrm{MSE}}$, $\mathcal{L}_{\mathrm{commit}}$, $\mathcal{L}_{\mathrm{LPIPS}}$, $\mathcal{L}_{\mathrm{GAN}}$, | $\mathcal{L}_{\mathrm{MSE}}$, $\mathcal{L}_{\mathrm{LPIPS}}$, $\mathcal{L}_{\mathrm{GAN}}$, |
| $\mathcal{L}_{\mathrm{entropy}}={H}[p(\mathbf{c}|\mathbf{v})]-{H}[\mathbb{E}_{<br>\mathbf{u}}[p(\mathbf{c}|\mathbf{v})]]$ | $\mathcal{L}_{\mathrm{entropy}}={H}[p(\mathbf{c}|\mathbf{u})]-\hat{H}[\mathbb{E<br>}_{\mathbf{u}}[p(\mathbf{c}|\mathbf{u})]]$ |  |

## Appendix C Proofs

### C.1 Proof of Eq ( 7 )

Before proving Eq (7), we will first prove the following identity:

Let $\mathbf{u}\in\mathbb{R}^{L}$, $\mathbf{C}=\Omega^{L}\in\mathbb{R}^{L\times 2^{L}}$ for $\Omega=\{-\frac{1}{\sqrt{L}},\frac{1}{\sqrt{L}}\}$,

$$ $\displaystyle\sum_{\mathbf{c}\in\mathbf{C}}e^{\tau\mathbf{u}^{\top}\mathbf{c}}=$ $\displaystyle\sum_{\mathbf{c}\in\mathbf{C}}\prod_{d=1}^{L}e^{\tau u_{d}c_{d}}= \prod_{d=1}^{L}\sum_{c_{d}\in\Omega}e^{\tau u_{d}c_{d}}.$ (15) $$

Proof.
With $\tau$ dropped for simplicity of notation.

$$ $\displaystyle\sum_{\mathbf{c}\in\mathbf{C}}e^{\mathbf{u}^{\top}\mathbf{c}}$ $\displaystyle=\sum_{\mathbf{c}\in\mathbf{C}}\prod_{k=1}^{L}e^{u_{k}c_{k}}$ $\displaystyle=\sum_{c_{1}\in\Omega}\sum_{c_{2}\in\Omega}\ldots\sum_{c_{L}\in \Omega}\prod_{d=1}^{L}e^{u_{d}c_{d}}$ $\displaystyle=\sum_{c_{1}\in\Omega}\sum_{c_{2}\in\Omega}\ldots\sum_{c_{L}\in \Omega}e^{u_{L}c_{L}}\prod_{d=1}^{L-1}e^{u_{d}c_{d}}$ $\displaystyle=\sum_{c_{1}\in\Omega}\sum_{c_{2}\in\Omega}\ldots\sum_{c_{L-1}\in \Omega}\left(\prod_{d=1}^{L-1}e^{u_{d}c_{d}}\right)\left(\sum_{c_{L}\in\Omega} e^{u_{L}c_{L}}\right)$ $\displaystyle=\ldots$ $\displaystyle=\left(\sum_{c_{1}\in\Omega}e^{u_{L}c_{L}}\right)\left(\sum_{c_{2 }\in\Omega}e^{u_{2}c_{2}}\right)\ldots\left(\sum_{c_{L}\in\Omega}e^{u_{L}c_{L} }\right)=\prod_{d=1}^{L}\sum_{c_{d}\in\Omega}e^{u_{d}c_{d}}.\square$ $$

Therefore, the probability of $\mathbf{u}$ being assigned to $\mathbf{c}_{i}$ can be written as:

$$ $\displaystyle\hat{q}(\hat{\mathbf{c}}|\mathbf{u})$ $\displaystyle=\frac{e^{\tau\mathbf{u}^{\top}\hat{\mathbf{c}}}}{\sum_{\mathbf{c }\in\mathbf{C}}e^{\tau\mathbf{u}^{\top}\mathbf{c}}}=\frac{\prod_{d=1}^{L}e^{ \tau u_{d}\hat{c}_{d}}}{\prod_{d=1}^{L}\sum_{c_{d}\in\{-\frac{1}{\sqrt{L}}, \frac{1}{\sqrt{L}}\}}e^{\tau u_{d}c_{d}}}$ (Using Eq ( 15 )) $\displaystyle=\prod_{d=1}^{L}\frac{e^{\tau u_{d}\hat{c}_{d}}}{e^{\tau u_{d} \hat{c}_{d}}+e^{\tau u_{d}\hat{c}_{d}}}$ (since $c_{d}=\pm\frac{1}{\sqrt{L}}=\pm\hat{c}_{d}$ ) $\displaystyle=\prod_{d=1}^{L}\sigma(2\tau u_{d}\hat{c}_{d}).$ $$

### C.2 Proof of Eq ( 8 )

Since $\hat{q}(\hat{\mathbf{c}}|\mathbf{u})=\prod_{d=1}^{L}\sigma(2\tau u_{d}\hat{c}_
{d})$ each variable $c_{d}$ is independent of each other.
Thus by definition

$$ $\displaystyle H[\hat{q}(\mathbf{c}|{\mathbf{u}})]=\sum_{d=1}^{L}H(\sigma(2\tau u _{d}c_{d})).\quad\square$ $$

### C.3 Proof of Eq ( 9 )

Now we look at $H[\mathbb{E}_{\mathbf{u}}[\hat{q}(\mathbf{c}|\mathbf{u})]]$.
We first compute $Q(\mathbf{c})=\mathbb{E}_{\mathbf{u}}\left[\hat{q}(\mathbf{c}|\mathbf{u})\right]$.

$$ $\displaystyle Q(\mathbf{c})=\mathbb{E}_{\mathbf{u}}\left[\hat{q}(\mathbf{c}| \mathbf{u})\right]$ $\displaystyle=\frac{1}{N}\sum_{\mathbf{u}}\hat{q}(\mathbf{c}|\mathbf{u})=\frac {1}{N}\sum_{\mathbf{u}}\prod_{k}^{L}\sigma(2u_{k}c_{k}).$ $$

Unlike $\mathbf{c}$, $\mathbf{u}$ does not factorize like Eq (15).
This would require us to compute $Q(\mathbf{c})$ as a full distribution over $2^{L}$ states, which is slow ($O(L\times 2^{L})$) and easily overfits.
Instead, we approximate $Q(\mathbf{c})$ by a factorized distribution $\tilde{q}(\mathbf{c})=\prod_{d=1}^{L}\tilde{q}_{d}(c_{d})$, where $c_{d}\in\Omega$ for $\Omega=\{-\frac{1}{\sqrt{L}},\frac{1}{\sqrt{L}}\}$, using an M-projection.
We again omit $\tau$ for notational brevity.

$$ $\displaystyle D(Q\|\tilde{q})$ $\displaystyle=H(Q,\tilde{q})-H(Q)$ $\displaystyle=-\sum_{i=1}^{2^{L}}Q(\mathbf{c}_{i})\log\tilde{q}(\mathbf{c}_{i} )-H(Q)$ $\displaystyle=-\sum_{i=1}^{2^{L}}Q(\mathbf{c}_{i})\sum_{d}\log\tilde{q}_{d}(c_ {d})-H(Q)$ $\displaystyle=-\sum_{d}\underbrace{\sum_{i=1}^{2^{L}}Q(\mathbf{c}_{i})\log \tilde{q}_{d}(c_{d})}-H(Q)$ $\displaystyle=-\sum_{d}\left(\log\tilde{q}_{d}(\mathbf{c}_{d}=1)\sum_{\mathbf{ c}_{-d}}p(\mathbf{c}_{i})+\log\tilde{q}_{d}(c_{d}=-1)\sum_{\mathbf{c}_{-d}}Q(c _{i})\right)-H(Q)$ $\displaystyle=-\sum_{d}\sum_{c_{d}\in\{-1,1\}}\log\tilde{q}_{d}(c_{d})\sum_{ \mathbf{c}_{-d}}Q(\mathbf{c})-H(Q)\text{, where $\mathbf{c}_{-d}$ sums over all dimensions except $d$}.$ $$

The minimizer of the above projection $\frac{\partial}{\partial\tilde{q}_{d}}D(Q\|\tilde{q})=0$

$$ $\displaystyle\tilde{q}_{d}(c_{d})^{*}$ $\displaystyle=\sum_{\mathbf{c}_{-d}}Q(\mathbf{c})=\mathbb{E}_{\mathbf{u}}\left [\sum_{\mathbf{c}_{-d}}p(\mathbf{c}|\mathbf{u})\right]$ $\displaystyle=\mathbb{E}_{\mathbf{u}}\left[\sum_{\mathbf{c}_{-d}}\prod_{k} \sigma(2u_{k}c_{k})\right]=\mathbb{E}_{\mathbf{u}}\left[\sum_{\mathbf{c}_{-d}} \sigma(2u_{d}c_{d})\prod_{k\neq d}\sigma(2u_{k}c_{k})\right]$ $\displaystyle=\mathbb{E}_{\mathbf{u}}\left[\sigma(2u_{d}c_{d})\sum_{\mathbf{c} _{-d}}\prod_{k\neq d}\sigma(2u_{k}c_{k})\right]=\mathbb{E}_{\mathbf{u}}\left[ \sigma(2u_{d}c_{d})\underbrace{\prod_{k\neq d}\sum_{\mathbf{c}_{-d}}\sigma(2u_ {k}c_{k})}_{=1}\right]=\mathbb{E}_{\mathbf{u}}\left[\sigma(2u_{d}c_{d})\right]$ $$

Therefore, the entropy term is simplified to:

$$ $\displaystyle H(\tilde{q})=\sum_{d}H(\tilde{q}_{d}(c_{d}))=\sum_{d}H\left( \mathbb{E}_{\mathbf{u}}[\sigma(2u_{d}c_{d})]\right).$ $$

By the nature of the above derivation the cross entropy $H(Q,\tilde{q})=H(\tilde{q})$ equals the entropy of the approximation.
This means that $D(Q\|\tilde{q})=H(\tilde{q})-H(Q)\geq 0$, and the entropy of the approximation is an upper bound $H(\tilde{q})\geq H(Q)$ to the true entropy.

In practice, this bound is relatively tight.
The most adversarial distribution $P(\mathbf{u})$ is $P(\frac{1}{\sqrt{L}}\vec{1})=\frac{1}{2}$ and $P(-\frac{1}{\sqrt{L}}\vec{1})=\frac{1}{2}$, where all inputs are maximally correlated, but the factorized distribution is not.
Figure 5 shows an empirical estimate of this approximation error for various values of $\tau$.
In practice, we use $\tau=\frac{1}{100}$, which has little to no approximation error.

Figure: Figure 5: Empirical estimation of the approximation error with respect to $\tau$ at different bottleneck dimensions $L$.
Refer to caption: x8.png

### C.4 Proof of the Quantization Error Bound of BSQ (Eq. ( 10 ))

We consider $\ell_{2}$-distance $d(\mathbf{u},\hat{\mathbf{u}})=\|\mathbf{u}-\hat{\mathbf{u}}\|$.
A simple (but loose) bound is:

$$ $\displaystyle\mathbb{E}_{\mathbf{u}}\left[d(\mathbf{u},\hat{\mathbf{u}})\right ]=\mathbb{E}_{\mathbf{u}}\left[d_{\mathrm{max}}(\mathbf{u},\hat{\mathbf{u}}) \right]<\sqrt{2-\frac{2}{\sqrt{L}}}<\sqrt{2},$ (16) $$

where $d_{\mathrm{max}}$ is attained if $\mathbf{u}$ is at any axis, $\mathbf{u}=[\underbrace{0,\cdots,0}_{n},1,\underbrace{0,\cdots,0}_{L-1-n}]$.
To achieve a tighter bound, we first expand the definition,

$$ $\displaystyle\mathbb{E}_{\mathbf{u}}\left[d(\mathbf{u},\hat{\mathbf{u}})\right]$ $\displaystyle=\frac{\underbrace{\int\cdots\int}_{S^{L-1}}d_{S^{L-1}}Vd(\mathbf {u},\hat{\mathbf{u}})}{\underbrace{\int\cdots\int}_{S^{L-1}}d_{S^{L-1}}V},$ (17) $$

where $S^{L-1}=\{x\in\mathbb{R}^{L}:\|x\|=1\}$ denotes the unit $L$-sphere of radius 1 and $d_{S^{L-1}}V$ denotes its surface area element.
We further define a hyperspherical coordinate system that is analogous to the spherical coordinate system for 3D Euclidean space or the polar coordinate system for 2D space to represent the surface area element.

$$ $\displaystyle u_{1}$ $\displaystyle=\cos(\varphi_{1}),$ $\displaystyle u_{2}$ $\displaystyle=\sin(\varphi_{1})\cos(\varphi_{2}),$ $\displaystyle u_{3}$ $\displaystyle=\sin(\varphi_{1})\sin(\varphi_{2})\cos(\varphi_{3}),$ $\displaystyle\cdots$ $\displaystyle u_{L-1}$ $\displaystyle=\sin(\varphi_{1})\sin{\varphi_{2}}\cdots\sin(\varphi_{L-2})\cos( \varphi_{L-1}),$ $\displaystyle u_{L}$ $\displaystyle=\sin(\varphi_{1})\sin{\varphi_{2}}\cdots\sin(\varphi_{L-2})\sin( \varphi_{L-1}),$ (surface area element) $\displaystyle d_{S^{L-1}}V=\sin^{L-2}(\varphi_{1})\sin^{L-3}(\varphi_{2}) \cdots\sin(\varphi_{L-2})d\varphi_{1}\cdots d\varphi_{L-1},$ (surface area) $\displaystyle S_{L-1}=\underbrace{\int\cdots\int}_{S^{L-1}}d_{S^{n-1}}V=\frac{ 2\pi^{L/2}}{\Gamma(\frac{L}{2})}.$ $$

Due to symmetry, we assume the subarea $A^{L-1}$ where $\forall i\in\{1,\cdots,L\}$, $u_{i}>0$, and it will be quantized to $\mathbf{c}_{1}=\hat{\mathbf{u}}_{1}=\frac{1}{\sqrt{L}}\overrightarrow{1}$.
The unit hypersphere $S^{L-1}$ has $2^{L}$ of such subareas interchangeably.
Computing Eq (17) is equivalent to

$$ $\displaystyle\mathbb{E}_{\mathbf{u}}\left[d(\mathbf{u},\hat{\mathbf{u}})\right]$ $\displaystyle=\frac{\underbrace{\int\cdots\int}_{A^{L-1}}d_{S^{L-1}}Vd(\mathbf {u},\hat{\mathbf{u}})}{\underbrace{\int\cdots\int}_{A^{L-1}}d_{S^{L-1}}V}.$ (18) $$

We expand the the numerator in Eq (18) as follows:

$$ $\begin{split}&=\int_{0}^{\frac{\pi}{2}}\cdots\int_{0}^{\frac{\pi}{2}}d_{S^{L-1 }}V\{[\cos(\varphi_{1})-\frac{1}{\sqrt{L}}]^{2}+[\sin(\varphi_{1})\cos(\varphi _{2})-\frac{1}{\sqrt{L}}]^{2}+\cdots\\ &+[\sin(\varphi_{1})\sin(\varphi_{2})\cdots\sin(\varphi_{L-2})\cos(\varphi_{L- 1})-\frac{1}{\sqrt{L}}]^{2}\\ &+[\sin(\varphi_{1})\sin(\varphi_{2})\cdots\sin(\varphi_{L-2})\sin(\varphi_{L- 1})-\frac{1}{\sqrt{L}}]^{2}\}^{\frac{1}{2}}\end{split}$ (19) $$

It is composed of $L$ square terms.
It is easy to see that the sum of constant terms leads to $1$.
Next, let’s sum over all quadratic terms and keep on using $\sin^{2}(\theta)+\cos^{2}(\theta)=1$:

$$ $\displaystyle\cos^{2}(\varphi_{1})+\sin^{2}(\varphi_{1})\cos^{2}(\varphi_{2})+ \cdots+\prod_{j=1}^{L-2}\sin^{2}(\varphi_{j})\sin^{2}(\varphi_{L-1})+\prod_{j= 1}^{L-2}\sin^{2}(\varphi_{j})\cos^{2}(\varphi_{L-1})=1$ $$

So the distance function to be integrated simplifies to

$$ $\displaystyle\left[2-\frac{2}{\sqrt{L}}\cos(\varphi_{1})-\frac{2}{\sqrt{L}} \sin(\varphi_{1})\cos(\varphi_{2})-\cdots-\frac{2}{\sqrt{L}}\prod_{j=1}^{L-2} \sin(\varphi_{j})\cos(\varphi_{L-1})-\frac{2}{\sqrt{L}}\prod_{j=1}^{L-2}\sin( \varphi_{j})\sin(\varphi_{L-1})\right]^{\frac{1}{2}}$ $\displaystyle<\left(2-\frac{2}{\sqrt{L}}\cos(\varphi_{1})\right)^{\frac{1}{2}}.$ $$

Plug into the numerator in Eq (18) and continue simplifying:

$$ $\displaystyle\underbrace{\int\cdots\int}_{A^{L-1}}d_{S^{L-1}}V\left(2-\frac{2} {\sqrt{L}}\cos(\varphi_{1})\right)^{\frac{1}{2}}$ (20) $\displaystyle=\underbrace{\underbrace{\int\cdots\int}_{A^{L-1}}d_{S^{L-2}}V}_{ \frac{S_{L-2}}{2^{L-1}}}\int_{0}^{\frac{\pi}{2}}\left(2-\frac{2}{\sqrt{L}}\cos (\varphi_{1})\right)^{\frac{1}{2}}\sin^{L-2}(\varphi_{1})d\varphi_{1}.$ (21) $$

Therefore, we have

$$ $\displaystyle\mathbb{E}_{\mathbf{u}}\left[d(\mathbf{u},\hat{\mathbf{u}})\right ]<\frac{2\Gamma(\frac{L}{2})}{\sqrt{\pi}\Gamma{\frac{L-1}{2}}}\int_{0}^{\frac{ \pi}{2}}\left(2-\frac{2}{\sqrt{L}}\cos(\varphi_{1})\right)^{\frac{1}{2}}\sin^{ L-2}(\varphi_{1})d\varphi_{1},$ (22) $$

where $\mathrm{RHS}$ can be numerically computed and plotted in Figure 6.

Figure: Figure 6: Quantization error with vocabulary size $L$.
Refer to caption: x9.png

## Appendix D Dataset Overview

ImageNet-1k has 1.28M training images and 50,000 validation images; COCO 2017val has 5,000 images.

UCF101 has 13,320 video clips and three train-val splits.
Following prior works [], we consider split-1 which has 9,537 clips for training and 3,783 for validation.

The MCL-JCV dataset [] consists of thirty 1080P (1,920$\times$1,080) video sequences with 24$\sim$30 FPS.
The Open Ultra Video Group (UVG) dataset [] consists of sixteen 4K (3,840$\times$2,160) test video sequences captured at 50/120 FPS.
Following prior works [], we report the performance on a subset of seven videos in YUV 8bit format at 120 FPS under the resolution of 1,920$\times$1,080.

## Appendix E Implementation Details

Training Image Tokenizers.
We train the image tokenizer with a batch size of 32 per GPU.
We use AdamW optimizer [] with $(\beta_{1},\beta_{2})=(0.9,0.99)$ with $1\times 10^{-4}$ weight decay.
The base learning rate is $4\times 10^{-7}$ (or a total learning rate of $1\times 10^{-4}$) and follows a half-period cosine annealing schedule.
The model is trained for 1M steps which amounts to 200 epochs over the entire ImageNet-1k training set.
We did not heavily study the effect of loss weights.
Instead, we keep $\gamma=1$ in the entropy terms.
We use a perceptual loss weight of 0.1 and an adversarial loss weight of 0.1 throughout the experiments.

Training Video Tokenizers.
We finetune the video tokenizer with a batch size of 32 per GPU.
The optimization schedule follows the image-based one but trains for fewer iterations.
The network is initialized from the ImageNet-pretraining checkpoint and undergoes another 500K steps which amounts to 1600 epochs over UCF-101 split-1 train.

Training a Masked Language Model for Generation.
The masked LM is a standard post-LN Transformer with 24 layers and a hidden dimension of 768 following MaskGIT [].
We train the masked LM on 2 nodes of $8\times$ GPUs (16 in total) with a total batch size of 1024 for 1M steps.
We use AdamW optimizer with $(\beta_{1},\beta_{2})=(0.9,0.96)$ with $0.045$ weight decay.
At inference time, we use a cosine unmasking schedule in MaskGIT [] and set the sampling temperature to 15.
We use classifier-free guidance []: At training, we replace 20% of the class condition labels with the mask token so that the model learns an unconditional distribution simultaneously.
Let $\ell_{c}$ be class-conditioned logits and $\ell_{\emptyset}$ be unconditional logits.
During inference, we interpolate logits using $\ell^{\prime}=\ell_{c}+\alpha(\ell_{c}-\ell_{\emptyset})$, where $\alpha=0.5$.

Training an Auto-Regressive Model for Arithmetic Coding.
The auto-regressive model is a Transformer with 24 layers and a hidden dimension of 768.
We train this model on 8$\times$ GPUs with a total batch size being 64.
We use AdamW optimizer with $(\beta_{1},\beta_{2})=(0.9,0.96)$ with $0.045$ weight decay.

Hardware.
The hardware for training is 8$\times$GPU-servers with NVIDIA A5000 (24GB).
Pre-training an image tokenizer and fine-tuning a video tokenizer in the full schedule is done across two servers with distributed training and takes around 5 days.
Training the AR model for AC is done on an 8$\times$GPU server and takes around 1 week.
When measuring the tokenizer’s throughput and the compression runtime, we use a server with 4$\times$ A5000 GPU and 1$\times$ AMD Ryzen Threadripper PRO 5975WX 32-Core CPU (64 threads).

**Table 8: Image reconstruction results on COCO2017 and ImageNet-1K ($256\times 256$). The settings strictly follow Table 1 except that all images are resized with bilinear interpolation.**
|  | COCO2017 val | ImageNet-1k val |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Data | Arch. | Quant. | Param. | # bits | TP↑ | PSNR↑ | SSIM↑ | LPIPS↓ | rFID↓ | PSNR↑ | SSIM↑ | LPIPS↓ | rFID↓ |
| DALL-E dVAE [] | CC+YF | C | VQ | 98M | 13 | 34.0 | 26.97 | .0837 | .2544 | 48.60 | 27.31 | .7943 | .2544 | 32.63 |
|  |  |  |  |  |  |  | $\pm$3.41 | $\pm$.0922 | $\pm$.1057 |  | $\pm$3.81 | $\pm$.1114 | $\pm$.1057 |  |
| MaskGIT [] | IN-1k | C | VQ | 54M | 10 | 37.5 | 18.21 | .4596 | .1930 | 8.47 | 18.63 | .4619 | .1884 | 1.98 |
|  |  |  |  |  |  |  | $\pm$2.74 | $\pm$0.1606 | $\pm$.0444 |  | $\pm$2.90 | $\pm$.1812 | $\pm$.0497 |  |
| ViT-VQGAN [] | IN-1k | T-B | VQ | 182M | 13 | ^†7.5 | - | - | - | - | - | - | - | ^†1.55 |
| SD-VAE 1.x [] | OImg | C | VQ | 68M | 10 | 22.4 | 23.29 | .6705 | .0949 | 6.49 | 23.65 | .6615 | .0940 | 1.40 |
|  |  |  |  |  |  |  | $\pm$3.34 | $\pm$.1316 | $\pm$.0313 |  | $\pm$3.69 | $\pm$.1540 | $\pm$.0367 |  |
| SD-VAE 1.x [] | OImg | C | VQ | 68M | 14 | 22.4 | 24.17 | .7042 | .0814 | 5.75 | 24.48 | .6931 | .0814 | 1.13 |
|  |  |  |  |  |  |  | $\pm$3.50 | $\pm$.1276 | $\pm$.0289 |  | $\pm$3.98 | $\pm$.1502 | $\pm$.0289 |  |
| SD-VAE 1.x [] | OImg | C | KL | 68M | 64 | 22.4 | 23.21 | .6930 | .0908 | 5.94 | 23.54 | .6835 | .0899 | 1.22 |
|  |  |  |  |  |  |  | $\pm$3.24 | $\pm$.1249 | $\pm$.04282 |  | $\pm$3.62 | $\pm$.1465 | $\pm$.0337 |  |
| SD-VAE 2.x [] | OImg+ | C | KL | 84M | 64 | 18.9 | 26.62 | .7722 | .0584 | 4.26 | 26.90 | .7592 | .0609 | 0.70 |
|  | LAION |  |  |  |  |  | $\pm$3.64 | $\pm$.1086 | $\pm$.0273 |  | $\pm$4.09 | $\pm$.1300 | $\pm$.0349 |  |
| SDXL-VAE [] | OImg+ | C | KL | 84M | 64 | 18.9 | 27.08 | .7953 | .0541 | 3.93 | 27.37 | .7814 | .0574 | 0.67 |
|  | LAION+? |  |  |  |  |  | $\pm$3.88 | $\pm$.1066 | $\pm$.0250 |  | $\pm$4.36 | $\pm$.1282 | $\pm$.0320 |  |
| Ours | IN-1k | T-B | BSQ | 174M | 18 | 45.1 | 26.89 | .8133 | .0652 | 5.41 | 27.78 | .8171 | .0633 | 0.99 |
|  |  |  |  |  |  |  | $\pm$3.47 | $\pm$.0851 | $\pm$.0255 |  | $\pm$3.99 | $\pm$.0987 | $\pm$.0307 |  |
| Ours | IN-1k | T-B | BSQ | 174M | 36 | 45.1 | 29.85 | .8862 | .0341 | 3.07 | 30.12 | .8803 | .0355 | 0.36 |
|  |  |  |  |  |  |  | $\pm$3.65 | $\pm$.0570 | $\pm$.0163 |  | $\pm$4.13 | $\pm$.0670 | $\pm$.0207 |  |
| Ours (w/. EMA) | IN-1k | T-B | BSQ | 174M | 36 | 45.1 | 30.19 | .8904 | .0314 | 3.07 | 30.45 | .8843 | .0329 | 0.42 |
|  |  |  |  |  |  |  | $\pm$3.69 | $\pm$.0561 | $\pm$.0153 |  | $\pm$4.19 | $\pm$.0661 | $\pm$.0194 |  |

## Appendix F Qualitative Results

In Figure 7, we show reconstructed images produced by the proposed BSQ-ViT in comparison to the best prior work, SDXL-VAE [].
We can see that our method is able to preserve more details about high-frequency texture and fine-grained shape/geometry.
BSQ-ViT often shows better reconstruction results for characters.

Figure: Figure 7: Reconstruction results of BSQ-ViT (right) compared to the original image (left) and SDXL-VAE [] (middle). The three images are taken from COCO 2017val which are more scene-centric compared to ImageNet data that our model is trained on.
Refer to caption: x10.png

In Figure 8, we show sampled results produced by a Masked LM with the proposed BSQ-ViT in comparison to existing methods, BigGAN [] and ADM [].
We also plot the samples from the ground-truth ILSVRC2012 validation set for reference.
Our method produces competitive results with state-of-the-art methods.

Figure: Figure 8: Sampled generation results of BSQ-ViT + Masked-LM (second column from left) compared to BigGAN [] (right), ADM [] (second column from right) and the original images (left). Classes are 1: goldfish, 279: arctic fox, 323: monarch butterfly, 417: balloon.
Refer to caption: x13.png

## Appendix G Baselines for Video Compression

Following SSF [], we used FFmpeg(^2^22[https://ffmpeg.org/](https://ffmpeg.org/)) to produce the evaluation metrics for H.264 and HEVC.
We use the commands provided in CompressAI [].

where $Resolution $\in\{\texttt{1920x1080},\texttt{640x360}\}$, and $CRF $\in\{17,20,22,27,32,37,42,47\}$.

## Appendix H Limitations

The proposed tokenizer has been tested on images with a 256$\times$256 or 128$\times$128 resolution and videos with a 128$\times$128 resolution.
Training a visual tokenizer on higher-resolution inputs and variable aspect ratio remains unexplored.
Also, the training dataset is limited to ImageNet-1k and UCF-101.
Scaling the proposed model to larger-scale visual contents remains an interesting problem to study.

## Appendix I Broader Impacts

The video compression application illustrated in the paper may be useful to reduce the storage and transmission cost of video data.
Also, the proposed visual tokenization model runs more efficiently than prior ones, resulting in potential energy savings.
Both of them will ultimately benefit society.

## Appendix J Licenses for Existing Assets

### J.1 Datasets

ImageNet The terms of service are available on [https://image-net.org/about](https://image-net.org/about).

COCO The terms of service are available on [https://cocodataset.org/#termsofuse](https://cocodataset.org/#termsofuse).

MCL-JCV
Copyright is available on the website [https://mcl.usc.edu/mcl-jcv-dataset/](https://mcl.usc.edu/mcl-jcv-dataset/).

UVG(^3^33[https://ultravideo.fi/dataset.html](https://ultravideo.fi/dataset.html)) The dataset is licensed under a CC BY-NC 3.0 Deed license.

### J.2 Evaluation Metrics

FID score is based on the PyTorch re-implementation(^4^44[https://github.com/mseitzer/pytorch-fid](https://github.com/mseitzer/pytorch-fid)).
The original implementation(^5^55[https://github.com/bioinf-jku/TTUR](https://github.com/bioinf-jku/TTUR)) is based on TensorFlow.
Both are licensed under an Apache-2.0 License.

LPIPS is based on the implementation(^6^66[https://github.com/richzhang/PerceptualSimilarity](https://github.com/richzhang/PerceptualSimilarity)) licensed under a BSD-2-Clause license.

SSIM and MS-SSIM are based on the PyTorch implementation(^7^77[https://github.com/VainF/pytorch-msssim](https://github.com/VainF/pytorch-msssim)) licensed under an MIT license.

Generation Metrics (FID, Inception Score, Precision, and Recall) are reported using a TensorFlow implementation(^8^88[https://github.com/openai/guided-diffusion/tree/main/evaluations](https://github.com/openai/guided-diffusion/tree/main/evaluations)) licensed under an MIT license.

### J.3 Baseline Methods

DALL-E dVAE(^9^99[https://github.com/openai/DALL-E](https://github.com/openai/DALL-E)) is licensed under a Modified MIT License.

SD-VAE 1.x(^10^1010[https://github.com/CompVis/latent-diffusion](https://github.com/CompVis/latent-diffusion)) is licensed under an MIT License.

SD-VAE 2.x(^11^1111[https://huggingface.co/stabilityai/sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)) is licensed under an MIT License.

SDXL-VAE(^12^1212[https://huggingface.co/stabilityai/sdxl-vae](https://huggingface.co/stabilityai/sdxl-vae)) is licensed under an MIT License.

ADM(^13^1313[https://github.com/openai/guided-diffusion](https://github.com/openai/guided-diffusion)) is licensed under an MIT License.

MaskGIT(^14^1414[https://github.com/google-research/maskgit/tree/main](https://github.com/google-research/maskgit/tree/main)) is licensed under an Apache-2.0 License.

CompressAI(^15^1515[https://github.com/InterDigitalInc/CompressAI](https://github.com/InterDigitalInc/CompressAI)) is licensed under a BSD-3-Clause-Clear license.

FFmpeg is licensed under the GNU LGPL version 2.1 or later. For more detail, please refer to [https://ffmpeg.org/legal.html](https://ffmpeg.org/legal.html).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: see the weaknesses.
soundness: 3
strengths: 1. The paper presents an innovative quantization method (BSQ) that addresses the limitations of existing vector quantization approaches by offering a more efficient and scalable solution.
2. Extensive experiments on benchmarks such as ImageNet and UCF-101 demonstrate that BSQ-ViT significantly improves reconstruction quality, outperforming prior methods in terms of speed and fidelity.
3. The methodology is clearly explained with detailed comparisons to related work, and the theoretical basis of BSQ is well-supported with mathematical derivations and experimental validation.
4. BSQ-ViT's ability to handle both image and video tokenization and perform well on diverse tasks such as compression and generation showcases its generalizability.
summary: This paper proposes a novel image and video tokenizer, BSQ-ViT, based on Binary Spherical Quantization (BSQ) integrated with a transformer architecture. The proposed method achieves state-of-the-art performance in image and video reconstruction with significant improvements in computational efficiency. It introduces a block-wise causal masking for video tokenization, supports variable-length videos, and delivers competitive results in visual compression against standards like JPEG2000 and H.264. BSQ-ViT also shows promising results in image generation.
weaknesses: 1. While the transformer architecture is explored, the paper does not demonstrate the effectiveness of  BSQ within a CNN-based model.
2. The paper provides limited comparative data in video reconstruction, reducing the robustness of the comparison. Additionally, while block-wise causal attention is noted to impact performance, the study lacks experiments on BSQ without this causal masking. 
3. The reported image and video compression results are better on MS-SSIM, potentially  due to the inclusion of perceptual losses like LPIPS and GAN loss in the loss function. However, the lack of subjective evaluation metrics limits the strength of the comparison.
4. When BSQ and VQ use the same number of indexes, BSQ performs less effectively, and codebook utilization decreases as the bits increase.
5. In the ablation experiments, the reproduction of the LFQ, which serves as the basis for this work, is too poor in terms of behavior and code usage. And it is premature to conclude that LFQ is less compatible with transformer-based codecs.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See the weakness part
soundness: 2
strengths: The Binary Spherical Quantization seems to show more effective training of the qunatization bottleneck. Analysis shows that the proposed method can provide fast speed and good performance.
summary: This paper propose a transformer-based image and video tokenizer with Binary Spherical Quantization, which projects the high-dimensional embedding to lower-dimensional hypersphere and applies binary quantization. BSQ is parameter-efficient without an explicit codebook, scalable to token dimensions and compact. The experiments show that the proposed method achieves comparable compression performance with JPEG2000/WebP for images and H.264/H.265 for videos. And it enables masked language models to achieve competitive image synthesis quality to GAN and diffusion methods.
weaknesses: - Lack of comparison at different bitrate range for visual compression results. Table 4 only provides BPP, PSNR and MS-SSIM for one bitrate point. However, visual compression tasks usually require showing a Rate-Distortion curves and compare at different bitrate points. Your can use BD-Rate metric for more reasonable comparison and analyze the results at low bitrate and high bitrate.

- Test settings for ablation study. Please provide more experiment setting details. In Table 5, do VQ, LFQ and BSQ use the same tokenizers? I observed that the metrics vary significantly across different quantization methods. To alleviate the influence of training and focus on the quantization bottleneck, it is better to: 1. use the same tokenizer 2. freeze the other network parts and only train the information bottleneck parts.

- Complexity. It is better to provide the computation complexity and encoding/decoding time for comparison.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. For image compression, why not use arithmetic coding to improve compression performance? Is intra-frame information used for video compression? If not, why not?
soundness: 3
strengths: 1. The idea of projecting high-dimensional visual embeddings onto a lower-dimensional hypersphere is straightforward yet effective.

2. The motivation is clear, and the overall presentation is coherent and easy to follow. The experiments are comprehensive and provide convincing evidence to support the approach.

3. The BSQ-ViT model achieves competitive performance in diverse tasks such as image/video reconstruction, generation, and compression.
summary: The paper introduces a transformer-based image and video tokenizer that uses Binary Spherical Quantization (BSQ). By projecting high-dimensional visual embeddings onto a lower-dimensional hypersphere and applying binary quantization, BSQ achieves a bounded quantization error. The authors demonstrate that the proposed BSQ outperforms previous methods in image and video reconstruction, image generation, and compression tasks.
weaknesses: 1. This method uses a transformer encoder and decoder, which limit the flexibility in resolution. How do the authors address this issue?

2. For image and video compression results, it would be beneficial to include an LPIPS comparison to assess perceptual performance.

3. There are a few minor issues. 1) In Eq. (7), $\hat{q}(c|u) = \frac{\exp(\tau c^{T}u)}{\sum_{c \in C_{BSQ}} \exp(\tau c^{T}u)}$ might need to be revised to $\hat{q}(c|u) = \frac{\exp(2 \tau c^{T}u)}{\sum_{c \in C_{BSQ}} \exp(2 \tau c^{T}u)}$ ? 2) On page 16, line 837, $\frac{e^{\tau u_d \hat{c}_d}}{e^{\tau u_d \hat{c}_d} + e^{\tau u_d \hat{c}_d}}$ should be corrected to $\frac{e^{\tau u_d \hat{c}_d}}{e^{\tau u_d \hat{c}_d} + e^{-\tau u_d \hat{c}_d}}$. 3) On page 3, line 118, it would be clearer to use different symbols for the downsample factor $q$ and the bottleneck module $q$.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: My major concerns in the weakness part can be summarized as the direct comparison between BSQ, VQ and LFQ on representative image and video tasks are not adequate. As a general tokenizer for image and video, it is important to make direct comparisons with strictly controlled variables.
soundness: 3
strengths: Visual tokenizers are critical for visual modeling. The proposed Binary Spherical Quantization (BSQ) for unified image and video tokenization is novel and important.  

Theoretical contribution is good. It is proved that the proposed BSQ has bounded quantization error.    

The manuscript is well written. BSQ is well placed agains previous VQ and LFQ.  

Extensive experiments on visual reconstruction, compression and generation show the better performance of BSQ.
summary: This paper proposes a visual tokenizer based on a Vision Transformer and Binary Spherical Quantization (BSQ).   
The Transformer-based encoder-decoder leverages a block-wise causal mask and uses only visual tokens from the current or past timestamps for reconstruction.  
BSQ first projects the high-dimensional visual embedding of the encoder to a lower-dimensional hypersphere and then applies binary quantization.   
The transformer encoder, decoder, and BSQ are seamlessly integrated into the VQ-GAN framework and trained end-to-end.  

The proposed visual tokenizer has several advantages in trading off  visual reconstruction quality and computational efficiency and supporting variable-length input. Experiments on visual reconstruction and compression are conducted to verify the performance.
weaknesses: My major concern is the fair comparison with VQ and LFQ. Though the ablation study is provided in  tab 5 with image reconstruction task on imagenet-1k, the resolution is only 128, and more tasks should be tested:     

It seems to me the results on image is not as obvious as video. For example, in image reconstruction (tab 1), the proposed method has best image quality, but the parameter number is also larger. It is better to add Ours-VQ for better comparison, as done in tab 2 for video reconstruction.  

In image compression, no VQ or LFQ based method is compared.     

For image generation in tab 3, more steps are used for BSQ, what if we use the same steps as VQ and LFQ?   

In L389, we get a direct comparison between VQ and BSQ, BSQ has more bits (18 vs 14) and the video quality is only comparable with VQ.   

 



Minor  
The derivation for eq. 13 is not given. Please provide a proof or give a reference here.  
There exist some typos in the derivations.  
In Line 878, the p shuld be Q.   
In line 886, the p should be q^hat.

Meta-review: The authors propose a novel image and video tokenizer based on a binary alphabet. It addresses difficulties training tokenizers based on larger alphabets, and shows solid performance and speed improvements while being scalable. The authors satisfactorily address concerns  regarding fair empirical evaluation.

Given the wide usage of VQ-like tokenizers in the larger ICLR community, I believe it may be of value to accept this paper as a spotlight, in order to make the existence of competitive alternatives more widely known. However, in the presence of time or space constraints, a poster would be appropriate as well, given the reviewer's ratings.

Decision: Accept (Poster)