OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation
Authors: Junke Wang
Yi Jiang
Zehuan Yuan
BINGYUE PENG
Zuxuan Wu
Yu-Gang Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: visual generation
tokenizer
language model
Primary area: generative_models

Abstract: Tokenizer, serving as a translator to map the intricate visual data into a compact latent space, lies at the core of visual generative models. Based on the finding that existing tokenizers are tailored to either image or video inputs, this paper presents OmniTokenizer, a transformer-based tokenizer for joint image and video tokenization. OmniTokenizer is designed with a spatial-temporal decoupled architecture, which integrates window attention and causal attention for spatial and temporal modeling, respectively. To exploit the complementary nature of image and video data, we further propose a progressive training strategy, where OmniTokenizer is first trained on image data on a fixed resolution to develop the spatial encoding capacity and then jointly trained on image and video data on multiple resolutions to learn the temporal dynamics. OmniTokenizer, for the first time, handles both image and video inputs within a unified framework and proves the possibility of realizing their synergy. Extensive experiments demonstrate that OmniTokenizer achieves state-of-the-art (SOTA) reconstruction performance on various image and video datasets, e.g., 1.11 reconstruction FID on ImageNet and 42 reconstruction FVD on UCF-101, beating the previous SOTA methods by 13% and 26%, respectively. Additionally, we also show that when integrated with OmniTokenizer, both language model-based approaches and diffusion models can realize advanced visual synthesis performance, underscoring the superiority and versatility of our method.

OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation
Junke Wang Yi Jiang Zehuan Yuan", Binyue Peng", Zuxuan Wu 12: Yu-Gang Jiang 1.2
Shanghai Key Lab of Intell. Info. Processing, School CS, Fudan University Shanghai Collaborative Innovation Center Intelligent Visual Computing, Bytedance Inc.
Code available http:///ihhaatcooommommaaitiiioo
Abstract
Tokenizer, serving translator to map the intricate visual data into a compact latent space, lies at the core of visual generative models. Based on the finding that existing tokenizers OmniTokenizer, a transformer- based tokenizer for joint image and video tokeniza- tion. OmniTokenizer is designed with a spatial-temporal decoupled architecture, which integrates window and causal attention for spatial and temporal modeling. To exploit the complementary nature of image and video data, further propose a progressive training strategy, where Omn'Tokenizer is first trained on image data fixed resolution develop the spatial encoding capacity and then jointly trained on image and video data on multiple resolutions to learn the temporal dynamics. OmniTooknnizer, for the first time, handles both image and video inputs within a unified framework and proves the possibility of realizing their synergy. Extensive experiments demonstrate that OmniTokenizer achieves state-of-the-art (SOTA) reconstruction performance on various image and video datasets, e.g. 1.11 reconstruction FID ImageNet and 42 reconstruction FVD on UCF-101, beating the previous SOTA methods by 13% and 26%, respectively. Additionally, we also show that when integrated with Omn'Tokenizer, both language model- ap- proaches and diffusion models can realize advanced visual synthesis performance, underscoring the superiority and versatility of our method.
tailored to image video inputs, this paper presents
Introduction
The development of generative models [25, 54, 14, 17, 10, 41] has been one of the most exhilarating developments in artificial intelligence, offering the potential revolutionize the way we gener- ate visual content. In recent years, visual generation approaches have emerged as two dominant paradigms: language model-based methods [54, 12, 66, 48] and diffusion models [17, 45]. The former exploits the superior sequence modeling capability of language models (LMs) [36, 37, 52] for visual generation by formulating it as a next-token prediction process, while the latter gradually transforms noise into coherent visual structures through a carefully crafted reverse diffusion process.
Core to both approaches is the tokenizer, which translates visual signals into latent representations, with LMM tokenizers, also known as VQVAF, discretizing inputs into sequences of latent codes [12, 64, 66, 27], and diffusion tokenizers, i.e., VAE, modeling their probability distributions within latent space [25, 41]. Analogous to the role of the lexicon in a written language, tokenizers for visual synthesis dictate the upper bound of the generative models, thus attracting increasing attention the community [12, 63, 19].
project leaders, t: corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Existing tokenizers are designed specifically for either image [12, 64] or video [63, 13, 66] inputs, resulting in inherent limitations regarding their application flexibility and data scalability for the following generative models. Although MAGVITv2 [67] have explored causal 3D convolution to process both modalities, they still have to train separate models for the image and video data, without achieving the synergy between them. This work highlights the critical need for joint image-video tokenizer with two primary considerations: firstly, a joint image-video tokenizer enables joint learning from image and video data [59], which mitigates the scarcity of data in single modality addition, (particularly video data) and facilitates the tokenizer to learn more general representations. unified tokenization framework inherently enjoys better versatility and scalability. For instance, its performance can be improved by incorporating the data from either modality for training. This further promotes the efficacy of generative models tailored to image or video generation.
With this in mind, we present OmniTokenizer, a transformer- based tokenizer for joint image-video tokenization. As intuitive as it may seem, the simple unification of image and video data could not lead to the reciprocal effects between both modalities. To address this challenge, we turn to a spatial-tempont decoupled architecture [2], where window attention [28] is employed in the spatial dimension owing to its local aggregation capacity and efficiency, and causal attention used in the temporal dimension to capture the motion in videos and ensure temporal coherence. Complementing the model design, we introduce a progressive training strategy that begins with image pretraining on fixed resolution to establish fundamental understanding of static visual information. After this, we integrate video data for joint training on variable resolutions to capture the dynamics in more complex scenes. The progressive training strategy allows our method to bridge the gap between disparate forms of visual input and capitalize on the rich spectrum of visual data.
To empirically validate the effectiveness of the proposed method, we separately implement the LM and diffusion tokenizers, i.e., OmmiTokenizer-VQVAT and OmniTokennzzer-Va VAE, and conduct experiments on wide range of datasets including ImageNet [9], CelebA-HQ [21], FFHQ [22], UCF-101 [46], Kinetics-600 [6], etc. The results demonstrate our model outperforms existing methods in terms of reconstruction FID on both image datasets (e.g. 1.11 rFID for OmniTokenizer- VQVAE and 0.69 rFID for Omni Tokenizer VAE on ImageNet) and video datasets (e.g.. 42 rFVD for
VQVAE and 23 rFVD for Omni employing our approach for tokenization, we also show that both language model-based generative models and diffusion models could achieve competitive results dass-condittonal, unconditional generation, and frame prediction tasks.
VAE on UCF-101). In addition,
In summary, work makes the following key contributions:
We introduce OmniTokenizer, transformer-base tokenizer for joint image and video tokenization. For the first time, Tokenizee employs a shared framework and weight to handle both types of visual data.
We propose a progressive training strategy that begins with image pre-training at a fixed resolution and then transits to image-video joint training at multiple resolutions. Such an approach capitalizes on the synergies between image and video data, facilitating OmniTokenizer achieve better performance than solo image video training.
We conduct extensive experiments across various datasets like ImageNet, CelebA-HQ, FFHQ. UOF-101, and Kinetics-600. The results showease the state-of-the-art reconstruction performance of OmniTokenizer both image and video datasets. Furthermore, equipped with OmniTok- enizer, both language model-based generative models and diffusion models could achieve superior generation results.
2 Related Work
2.1 Language Models for Visual Generation
Language models have emerged as powerful contenders in the visual generation field, drawing
inspiration from their unparalleled success in natural language processing [36, 336,37.5 37, 51, 52] and visual understanding [11, 5, 30, 58, 57]. These methods [12, 7, 13, 66) recast visual synthesis as sequence prediction problem, similar to constructing sentences in human language.

---

M!
Temporal Transformer Layers
Spatial Transformed Layers
Codebook
Patchify
Patch Embed
Look up LM Tokenizer
3D Patch Embed
Time
N( 1)=
Diffusion Tokenizer
Figure 1: Architecture of Omne'ookenizrr, which consists patch emhedding layers, and separate spatial-temporal attention blocks. To obtain the latent representations, OmniTokenizer- VQVAE looks up codebook to quantize the encoder embeddings, while OmniTokenizerr- VAE samples from Gaussian distribution. We omit the decoder and only show the tokenization process.
Depending on whether the tokens are predicted sequentially or in parallel, LM- LM-based methods can be further categorized into autoregressive models [12, 65] and non-autoregressive models [7, 67]. Autoregressive (AR) models have been the initial foray into visual generation, utilizing the inherent sequential nature of language models to generate images [64, 65] and videos [63, 13] in step fashion. These models, such as DALL-E [39] and its preceding variants, typically work by predicting token at time and characterized by their high-quality outputs and precise control over the generation process. IreeffJreesfines the autoregressive learning framework on images as coarse-to- fine "next-scale prediction" paradigm. con-autoregrrssive (Non-AR) models, on the other hand, have been developed to allow for faster generation process by predicting multiple tokens independently and in parallel. Models like MaskGIT [7] leverage this parallelism to significantly reduce generation time while maintaining high fidelity in synthesized images. The non-AR approaches have also demonstrated promise in video generation, featured by MAGVIT series [66, 67]. Both AR and non-AR methods have significantly advanced the field of visual generation, offering novel methods to synthesize high-quality images and videos.
2.2 Diffusion Models for Visual Generation
Diffusson models [17, 33, 3, 62, 61] represent alternative avenue for visual generation, benefiting from their probabilistic nature that iteratively denoise random signal into structured images or videos. These models stand out for their flexibility in generating visual outputs that not only exhibit coherent global structures but are also rich with intricate textures [32, 34]. Unlike language models that discretize visual inputs latent codes, diffusion models directly generate visual samples in continuous pixel space [45, 10]. While effective, this approach demands significant computational resources given the high dimensionality of visual data.
The advent of latent diffusion models (LDMs)) [41] seeks to mitigate these issues by compressing the high-dimensional visual data into latent space with pretrained Variational Autoencoder (VAE) [25, 41]. LDM preserves the desirable properties pixel-space diffusion models, such as high-quality image synthesis and the ability to incorporate conditional information, while drastically reducing the training and sampling overhead. After that, the rise of LDMs [71, 35, 34, 29] continues to push visual generation toward higher quality, larger resolution, and more complex scenes.
3 Methodology
3.1 Joint Image and Video Tokenization
We aim to enable image and video tokenization in unified framework and achieve mutual benefits between them. To accomplish this, we employ a transformer-based architecture with decoupled spatial and temporal blocks (Sec. 1.1. Complementing this, we also propose progressive training strategy consisting of two consecutive stages to learn the visual encoding in an incremental way (Sec. 3.2. The overall framework of our method is illustrated in Figure 1.

---

Image Tokenizers
Video Tokenizers
Image training from scratch, eg,, VQGAN, ViI-VQGAN
Video training from scratch, egg, TATS or Image as initialization, eg, MAGVITv2
OmniTokenizer
OmniTokenizer
OmniTokenizer with progressive training
Figure 2: Illustration of the proposed progressive training paradigm. With this, OmniTokenizer could tokenize both image and video inputs with the same architecture and weight.
3.1.1 Space-Time Transformer
Patchify. Given a visual input x E (R(+T)xHXXX) where (1 + T) is the number of frames (T' = for image) and H x W denotes the spatial resolution, always process the first frame Io E [xxxxxxx3 and following frames 21.7 E T x3 separately for the joint encoding of videos and static images [67]. Specifically, both zo and F1.r split into non-overlapping patches, with a patch size of x and x x p. respectively. After that, we project the image and video patches with two linear layers. obtaining the patch embeddings cu E IR 211x and 61:7 E R xxxx W. where L, = x W. and L2 = x !! x p ep and E are then concatenated along the sequence dimension as the spatial-temporal embedding e. In this way, we compress the input resolution from (1+ H W to (1 +) H W.
Encoder and Decoder. To have better compatibility with image and video inputs, adopt spatial-temporal factorized encoder consisting separate spatial and temporal blocks. In the spatial dimension, window attention [28] is employed
exhibits superior local aggregation capability and efficiency. While in the temporal dimension, we use causal attention to align with the autoregressive visual generation in the second stage. Next, the latent code could be obtained by looking up a codebook [54] for LM tokenizer (i.e., quantization in VQVAE), or sampling from a Gaussian distribution for diffusion tokenizer.
The architecture the decoder is symmetric with the encoder. Finally, we map the spatial-temporal tokens to the pixel space with two linear projection layers without any activation function.
3.1.2 Progressive Training
Unlike existing image tokenizers that conduct training on image data only [12, 64] or video tokeniz- ers that train with image counterparts as intialization [66, 67]. We leverage progressive training paradigm that involves two consecutive stages of VQ training to facilitate spatial-temporal represen- tation learning of our LM tokenizer diffusion tokenizer, OmniTokenizer VAE, with KL fine-tuning.
wow.TokeeizzeQQaaaa After this, it could be fine-tuned
Two-stage VQ Training, as depicted in Figure 2, aims to learn the visual reconstruction with the discrete latent codes. includes two stages, the initial stage focuses on fixed-resolution image data to lay foundation for spatial understanding. Building upon this, the second stage introduces video data learn the modeling of temporal dynamics alongside static image features. This image-vided joint training stage is critical for the model to learn universal embedding that accurately captures both spatial intricacies of individual frames and the temporal relationships sequential video data.
During both stages, the model is trained with vector-quantization objective:
Lvq A [{(ee][EE(e]] z111 Ao E(e) sgl,,]1l3,
(I)

---

Table 1: Reconstruction FID ImageNet validation split, Cel and FFHQ. denotes models trained with Cumbel-Softmax reparameterization [39]. For method, the results that are jointly trained with UCF-101 are reported.
Table 2: Reconstruction FVD UCF-101 and Moments in- Time val. split. denotes training image tokenizer with video loss.
Method
Dataset
Lat. shape Codebook
rFID
Method
Type
UCF MiT
VIT-VQGAN [64]
CelebAHQ
32
8192
4.66
MaskGIT [7]
Img
240
VQVA-
CelebAHQ
32 x 32
8192
1.93
VQGAN [12]
Img
299
306
ViT-VQGAN [64]
FFHQ
32 32
8192
3.13
VIT-VQGAN [64]
Img
VQVA-
FTHQ
32 x 32
8192
1.91
VIT-VQGAN" [64]
Img
173
DALL-E [39]
ImageNet
32 x 32
8192
32,01
CVIVIT [55]
Vid
66
VQGAN' [12]
ImageNet
32 x
8192
1.49
TATS [13]
Vid
162
VIT- [64]
ImageNet
32 x 32
8192
1.28
MAGVIT [66]
Vid
58
Ours- VQVAF
ImageNet
32
8192
Ours- VASS-QQAE
Joint
42
20
Ours-VAF
ImageNet
32 x 32
0.69
Ours-VAE
Joint
23
where sg denotes the stop-gradient operation, Å1 and X2 are the balancing hyperparameters, E and Zg represent the encoder of OmniTokenizer and codebook vectors, respectively. Factorized codes and l2-normalized codes [64] are also used to boost the codebook usage.
KL fine-tuning, After the VQ training, we further fine-tune model diffusion tokenizer (i.e., OnniTokenizer-Wo VAE) by replacing the above Lvq with Kullback-Leibler (KL) loss:
CKL >aD >
(2)
where P(z) is Gaussian distribution, Q(=(x) represents the inferred posterior configurations of the latent code given the observed input.
Besides LvQ or LKL, both VQ training and KL fine-tuning also employs L2 reconstruction loss Lrecon and GAN loss LGAN.
3.2 Visual Generation
As mentioned in Sec. 3.1.2, after the progressive training and KL fine-tuning, we can obtain two tokenizers: Ommiookenizer-VOV inputs into latent codes discrete codebook the continuous latent space. With this, we further train language models or diffusion models for visual generation.
and Omni'okeniier-Vall which separately encode the visual
Language models-based generation approaches formulate visual synthesis token prediction problem. Specifically, after Omni Tokenizer- VQVAE tokenizes image or video imputs into a sequence of discrete latent codes, first flatten them the raster order [8, 12] obtain the code indices Then a transformer language model [36] is trained to maximize the log-likelihood between the predicted tokens ý and the target tokens y with cross-entropy loss:
maximize (0gP(0 v1--1;0))
where e represents the condition (e.g., label for class-conditional image and video generation), 0 is the learnable parameters of the language model, P and L denote the softmax probability and the length of y. During inference, we predict each token according the model likelihood.
Latent diffusion models (LDMs) [41] perform diffusion process in the latent space to enable high- quality image synthesis with improved computational efficiency. Specifically, with the 2D latent representation from Omni Toneizekei VAF, the diffusion process gradually applies Gaussian noise to the latent code generate perturbed sample, while the denoising process trains diffusion model to predict the noise that has been added. During inference, the well-trained diffusion model could generate a coherent visual sample from the noise by iteratively reversing the noising process.
4 Experiments
Datasets. We evaluate the visual tokenization performance OmniTokenizer on both image and video datasets, including ImageNet [9], CelebA-HQ [21], FFHQ [22], Kinetics [23, 6]. UCF-101 [46],
5

---

Table 3: Comparions of class-conditional results on ImageNet 256 256 using language models. "4" ("f") indicates lower (higher) is better. Metrics include Fréchet inception distance (FID) and incep- tion score (IS). NAR and AR: non-autoregressive and autoregressive. ": taken from MaskGIT [7].
Table 4: Comparions of class-conditional gener- ation results on UCF-101 and frame prediction results on Kinetics-6000 Fréchet video distance (FVD) is reported.
FVD
Type
Method
# PPara
UCF K600
Type Method
# Param
FID4
1St
NAR
Phenaki [55]
227M
36.4
AR
VQGAN" [12]
227M
18.65
80.4
NAR MAGVIT [66]
306M
76
9.9
AR
RQ-Transformer [26]
488M
15.72
86.8
NAR MAGVITv2 [67]
307M
58
4.3
AR
Ours
227M
10.13
94.5
AR
LVT [38]
50M
224.7
AR
VITrans [60]
373M
170.0
AR
VQVAF-2" [40]
13.5B
31.11
m-5
AR
CogVides [19]
9.4B
626
109.2
AR
VQGAN [12]
1.4B
15.78
74.3
AR
ViVQVAE [56]
NA
64.3
AR
RQ- TQ-Transformer [26]
821M
13.11
104.3
AR
TATS [13]
321M
332
AR
ViT-VQGAN [64]
650M
8.81
110.8
AR
Ours
227M
314
34.2
AR
Ours
650M
7.45
146.7
AR
Ours
650M
191
32.9
Moments-in- Time (MiT) [31], and Something-Something v2 (SSV2) [15]. We adopt subset of the above datasets for visual generation to compare with previous works [12, 64, 55, 13].
Implementation Details. Tokenizer adopts a decoupled spatial-temporal architecture consisting of 4 window attention-based spatial layers (window size 8) and 4 causal attention-based temporal layers. The hidden dimension is 512 and the latent dimension is 8, following ViT-VQGAN [64]. \1. Ag. and λ are set 1, 1, 1c-6, respectively. As mentioned in Sec. 3.1.2, the training of OmniTokenizer follows progressive training strategy, where both stages last 500K iterations. The learning rate is warmed up to 1e-3 and decayed to 0 using cosine scheduler. Adam [24] is employed for optimization (81 = 0.9 and (82 = 0.99). During the image training stage, we train the model with fixed image resolution of 256x256, For the joint training stage, we forward the model with image and video data iteratively, with the video sequence length being 17 frames. The spatial resolutions are randomiy chosen from 128, 192, 256, 320, and 384 [49]. Only random horizontal flip is adopted for data augmentation. We train our model using 8 NVIDIA A A100 GPUs for 2 weeks. Unless otherwise stated, the results reported in this paper are jointly trained on ImageNet and UCF-101.
We try both the language models and diffiusin models for visual generation with OmniTokenizer as the tokenizer. The configuration for the language model follows VQGAN [12], and for a fair comparison with previous methods, we also scale up the model size by increasing the hidden dimension to 1535, following ViT VQGAN [64]. The training of image and video diffusion transformers follows DiT [34] and Latte [29], respectively.
4.1 Visual Tokenization
We first evaluate the visual tokenization capability of Omn'Tokenizer on ImageNet and two high- quality face datasets, CelebA-HQ and FFHQ Reconstruction FID is used following the previous methods [12, 64]. We can observe from Table that with the same compression rate and codebook size, Omniiokenizer outperforms existing methods by large margin on all these datasets. Especially, OmniTokenizer- VQVAE achieves 1.11 FID on ImageNet, beating ViT-VQGAN, the previous state- of-the-art method by 13%. When fine-tuned as OmniTokenizer VAE, the FID is further reduced to 0.69. We hypothesize the improved performance is because KL training provides smoother gradients than VQ training and avoids loss of information in the quantization process.
In addition, we also conduct video reconstruction experiments and report the results in Table 2. We can see that on both UCF-101 and Moments- -n--iii-eeee Time datasets, Omni'ookenin achieves the best results. The video reconstruction results on more datasets can be found in the ablation study.
4.2 Visual Generation with AutoRegressive Transformers
for tokenization, we train language models to predict latent code indices in the codebook in an autoregressive manner for image and video synthesis. The class- conditional x56x22 x56x256 generation results on ImageNet, presented in Table 3, demonstrate that our
Using Omni'Toeenizrr-VVVV

---

Table 5: Class-conditional results ImageNet
Table 6: Comparisons of unconditional re- sults on UCF-101 256x256 using GAN and diffusion models.
256x 256 using GAN and diffusion models.
Method
FID
1St
Prect
Reet
Method
Lat. Comp.
FVD1
BigGAN [4]
6.95
171.4
0.87
0.28
StyleGAN-XL [42]
2.30
265.12
0.78
0.53
MoCoGAN [53]
2886.9
VideoGPT [63]
4 4
2880.6
ADM [10]
10.94
100.98
0.69
0.63
MoCoGAN-HI [50]
1729.6
LDM-4
10.56
103.49
0.71
0.62
DIGAN [69]
1630.2
CDM [18]
4.88
158.71
StyleGAN-V [44]
1431.0
DiT-XL/2 [34]
9.62
121.50
0.67
0.67
PVDM [68]
1 4
1141.9
DIT-XL/2-CFG [34]
2.27
278.24
0.83
0.57
MoSIGAN-V [43]
1380.3
12.25
109.94
0.73
0.64
Latte [29]
1 X 8 x 8
478.0
Ours-DiT- DI-------/
3.48
244.23
0.89
0.52
Ours-Latte
4 x 8
8
209.2
model surpasses existing autoregressive image generation methods with significant margins. Re- markably, with a model comprising only 227M parameters, we achieve 10.13 FID and 94.5 IS, outperforming VQGAN [12] by 32% and 25%, respectively. Upon scaling up larger model with 650M parameters, the FID is further reduced to 7.45.
In the domain of video generation, the-art autoregressive model, TATS [13] for class-conditional video generation on UCCF-10 with much lower FVD (283 v.s. 3 314.. Moreover, for frame prediction tasks on the Kinetics-600 dataset, our model not only achieves the best performance compared to other autoregressive models but also surpasses Phenaki [55], aon-autoregressive method.
illustrated in Table 4, our model beats the previous state-of-
4.3 Visual Generation with Diffusion Models
In parallel language model-based methods, diffusion model [17, 45, 10], especially latent diffusion model [41], is another promising technique for visual synthesis. Therefore, we also evaluate the effec- tiveness of our method on diffusion model-based image and video generation with OmniTokenizer VAE as the tokenizer. Here we employ the same architecture of DìT [34] and Latte [29] and replace their VAE [1] with OmniTokenizer- VAE. DiT [34] first applies the transformer architecture to latent diffusion models and exhibits appealing scalability properties. Following this, Latte [29] extends the transformer the latent video diffusion model by alternating spatial and temporal attention blocks.
The experimental results, as depicted in Table 5, indicate that when equipped with OmniTokknnzerr VAE, DiT-XL/2 with classifier-free guidance (CFG) achieves competitive results in terms of FIDD and inception score, underscoring the efficacy of our tokenizer within diffusion model frameworks for image synthesis. For unconditional video generation on the UCF-101 dataset, our method not only offers the advantage of reduced training costs by realizing a higher compression rate, but also exhibits a much lower FVD than previous methods.
4.4 Ablation Study
Training Paradigms. To verify the effect of the proposed progressive training paradigm, we compare different training strategies and show the results in Table 7. The results in lines 3-4 and line 6 indicate that joint training outperforms video training on all video datasets remarkably, demonstrating the importance of image pre-training for the following video training. In addition, although joint training on a fixed resolution (line 5) could achieve much better results on video datasets than video training, the reconstruction FID on ImageNet gets worse, i.e., from 1.28 to 1.35. Comparatively, the progressive training paradigm leads to the best performance on video datasets and surprisingly improves the image reconstruction performance.
Deeper analysis on attention mechanisms. We conduct experiments on both image and video reconstruction using different attention variants to fully analyze its effect. As can be seen in Table 8, window attention outperforms plain attention for image reconstruction (ImageNet) since incor- porates the local modeling. While for video reconstruction (K600), using causal or not has little influence on the performance (Table 9), but causal attention is necessary since we train the following image video generation transformer in autoregressive manner.
7

---

Table 7: Comparison of rFID on ImageNet and rFVD on various video datasets.
ImageNet
K600
UCF
MiT
SSV2
Method
256
128
256
128
256
128
256
128
256
Ours- Jurs-Image (Fix)
1.28
2
Ours-Image (Multi)
1.44
3
Durs-Vides (Fix)
211.51
48.89
214.83
118.52
211.07
64.47
162.53
22.82
4
Ours- Video (Multi)
194.51
54.89
211.83
114.52
238.07
26.47
193.35
38.82
5
Ours-Joint (Fix)
1.35
113.51
26.89
186.83
62.52
140.07
21.47
108.35
20.82
6
Ours-Joint (Multi)
1.11
84.38
25.97
107.80
42.35
59.47
19.87
84.78
20.30
Table Spatial atten.
Table
Temporal atten.
Table 10: Ablation on the training order.
ImageNet VQVAE VAE
K600
Spatial Attention
rFID
Spatial
Temporal
rFVD
Training
VQVAE
VAE
Plain
1.55
Window
Plain
26.43
Joint (VQ-KI.)
1.11
0.69
25.97
13.02
Window
1.28
Window Causal
25.97
Joint (KL-VQ)
2.05
0.89
33.79
12.43
Effeets of training recipe. We change the training recipe by performing KL training first and then VQ training, and the results in Table 10 show that this will lead to performance drop both ImageNet and K600. We believe this is because going from discrete tokens to continuous embedding will make the learning of the model easier and more stable.
4.5 Visualizations
Visual Reconstruction. We visualize the reconstruction results by OmniTokenizer, VQGAN [12] and TATS [13] in Figure Our method works significantly better than baselines for face and text reconstruction, which are typically regarded as the most challenging reconstruction cases.
GT
VQGAN
Ours
GT
TATS
Ours
Figure 3: Image and video reconstruction results of VQGAN [12], TATS [13], and method.
Class-conditional Image and Video Generation. The class-conditional generation results are shown in Figure 4-7. Our model could synthesize visually coherent and contextually accurate images and videos, showcasing the strengths of OmniTokenizer in facilitating generative tasks.

---

Figure Class-conditional ImageNet generation results using language models, with OmniTokenizerr- VQVAE as tokenizer.
Figure 5: Class-cond. ImageNet generation using diffusion models, OmniTokenizer-Vo VAE as tokenizer.
Figure 6: Class-cond. UCF-101 generation using LMs, OmniTTokenize--VOn VQVAI as tokenizer.
Frame Prediction and Arbitrary Long Video Generation. The frame prediction results by our method are presented in Figure 8, from which we can see that our model could forecast subsequent frames with high clarity and temporal coherence. Moreover, we exhibit the potential of our method for generating videos of arbitrary lengths by employing cyclical process, where each newly generated frame is recursively used condition for the subsequent frame generation.

---

byp you
byf your
Figure 7: Unconditional UCF-101 generation using diffusion models (and OmniTookenizrr VAE).
Figure 8: Visualization of the frame prediction results by Omkiiokenizer. The frames marked in red are given during inference, while the following frames are generated.
Conclusion and Discussion of Broader Impact
This paper presented OmniTokenizer, a transformer- based tokenizer for joint image-viden tokeniza- tion. Omni Tokenizer adopts spatial-temporia decoupled architecture, employing the window and causal attention in the spatial and temporal dimensions. To realize the synergy between images and video data, we proposed progressive training strategy that starts with image training fixed resolution to acquire the spatial encoding capability and then incorporates video data for multi- resolution joint training to learn temporal modeling. Extensive experimental results substantiate the state-of-the-art performance of Tokenizke in visual reconstruction tasks. Further, when equipped with OmniTokenizer, both language model-based methods and diffusion models could achieve superior visual generation results.
Previous literature [20, 16, 70, 48, 47] has revealed that the performance of transformer models improves significantly as the model size increases, also known as scaling law. the future, we will explore scaling the model capacity of OmniToken for more advanced tokenization performance.
Acknowledgements. This work was supported in part by National Natural Science Foundation of China (#62032006).
10

---

References
[1] S. AL Stable diffusion v1-4.
https://huggingf.cccco//// http://g........ooo///ao
Crable-diffusion-Ve
2022.
[2] G. Bertasius, H. Wang. and L. Torresani. Is space-time attention all you need for video
understanding? In ICML, 2021.
[3] A. Blattmann, T. Dockhorn, S. Kulla, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English,
V. Voleti, A. Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[4] A. Brock, Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image
synthesis. In ICLR, 2019.
[5] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[6] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-66 600. arXiv preprint arXiv: 1e0..088800444 2018.
[7] H. Chang, H. Zhang. L. Jiang, C. Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
[8] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and Sutskever. Generative pretraining from pixels. In ICML, 2020.
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[10] P. Dhariwal and A. Nichol. Diffusion models beat gans image synthesis. In NeurlPS, 2021.
[11] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani,
M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image is worth 16x16 Transformers for image recognition at scale. In ICLR, 2021.
words:
[12] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis.
In CVPR, 2021.
[13] S. Ge, T. Hayes, H. Yang. X. Yin, G. Pang. D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In ECCV, 2022.
[14] I. Goodfellow, Pouget-Abadic M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 2020.
[15] R. Goyal, S. Ebrahimi Kahou, V. Michalski, Materzynska, S. Westphal, H. Kim, Haenel,
I. Fruend, P. Yianilos, M. Mueller-Freitage et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[16] Henighan, Kaplan, M. Katz, M. Chen, Hesse, J. Jackson, H. Jun, B. Brown, Dhari- wal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
[17] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[18] J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion
models for high fidelity image generation. JMLR, 2022.
[19] W. Hong, M. Ding, W. Zheng. X. Liu, and J. Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In ICLR, 2023.
[20] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint TTt::20010083666 2020.
[21] Karras, T. Aila, Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10. 10/96, 2017.
[22] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[23] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, Green, T. Back, Natsev, al. The kinetics human action video dataset. arXiv preprint 175.v.700..06950. 2017.

---

[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
[25] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In ICLR, 2014.
[26] D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
[27] T. Li, H. Chang. S. Mishra, H. Zhang, D. Katabi, and D. Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2142-2152, 2023.
[28] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[29] X. Ma. Y. Wang, G. Jia, X. Chen, Z. Liu, Y.-F. Li, C. Chen, and Y. Qiao. Latte: Latent diffusion
transformer for video generation. arXiv preprint arXiv:2401.0300
2024.
[30] I. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.G.. Jiang, and S.-N. Lim. Adavit: Adaptive
vision transformers for efficient image recognition. CVPR, 2022.
[31] M. Monfort, A. Andonian, B. Zhou, K. Ramakrishnan, A. Bargal, T. Yan, L. Brown, Q. Fan, D. Gutfreund, C. Vondrick, et al. Moments in time dataset: one million videos for event
understanding. TPAMI, 2019.
[32] A. Nichol, P. Dhariwal, A. Ramesh, Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, PMLR, 2022.
[33] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[34] W. Peebles and S. Xie. Scalable diffusion models with transformers. In CVPR, 2023.
[35] D. Podell, Z. English, K. Lacey, A. Blattmann, Т. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
[36] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding
by generative pre-training. OpenAl Blog, 2018
[37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models unsupervised multitask learners. OpenAl Blog, 2019.
[38] R. Rakhimov, D. Volkhonskiy, A. Artemov, D. Zorin, and E. Burnaev. Latent video transformer. arXiv preprint 20NY::2006.17704, 2020.
[39] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In JCML, 2021.
[40] A. Razavi, A. Van den Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurlPS, 2019.
[41] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[42] A. Sauer, K. Schwarz, and A. Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH. 2022.
[43] X. Shen, X. Li, and M. Elhoseiny. Mostgan-v: Video generation with temporal motion styles. In CVPR. 2023.
[44] I. Skorokhodov, S. Tulyakov, and M. Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In CVPR, 2022.
[45] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In ICIR, 2021.
[46] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:
2012.
[47] P. Sun. Y. Jiang, S. Chen, S. Zhang. B. Peng. P. Luo, and Z. Yuan. Autorcgressive model beats diffusion: Llama for scalable image generation. arXiv preprint S45i::2406006225, 2024.
[48] K. Tian, Y. Jiang, Z. Yuan, B. Peng, and I.. Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint S5ti::2004002005. 2024.
12

---

[49] R. Tian, Z. Wu, Q. Dai, H. Hu, Y. Qiao, and Y.-G. Jiang. Resformer: Scaling vits with multi-resolution training. In CVPR. 2023.
[50] Y. Tian, J. Ren, M. Chai, K. Olszewski, X. Peng, D. N. Metaxas, and S. Tulyakov. A good image generator is what you need for high-resolution video synthesis. In ICLR, 2021.
[51] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2332. 13971, 2023.
[52] H. Touvron, Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Illama 2: Open foundation and fine-tuned chat models. arXiv preprint arX2v22307.009288 2023.
[53] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, 2018.
[54] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017.
[55] R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro,
J. Kunze, and D. Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2022.
[56] Walker, A. Razavi, and A. v. d. Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021.
[57] J. Wang, D. Chen, C. Luo, B. He, L. Yuan, Z. Wu, and Y.-G. Jiang. Omnivid: A generative framework for universal video understanding. In CVPR, 2024.
[58] J. Wang, D. Chen, Z. Wu, C. Luo, C. Tang, X. Dai, Y. Zhao, Y. Xie, L. Yuan, and Y.-G. Jiang. Look before you match: Instance understanding matters in video object segmentation. In CVPR, 2023.
[59] J. Wang, D. Chen, Z. Wu, C. Luo, L. Zhou, Y. Zhao, Y. Xie, C. Liu, Y.GG. Jiang, and L. Yuan. Omnivl: One foundation model for image-language and video-language tasks. NeurIPS, 2022. [60] D. Weissenhorn, O. Täckström, and J. Uszkoreit. Scaling autoregressive video models. In ICLR, 2020.
[61] Z. Weng, X. Yang, Z. Xing, Z. Wu, and Y.-G. Jiang. Genree: Unifying video generation and recognition with diffusion models. In NeurlPS, 2024.
[62] Z. Xing. Q. Dai, H. Hu, Z. Wu, and Y.-G. Jiang Simda: Simple diffusion adapter for efficient video generation. In CVPR, 2024.
[63] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprini 2rxiv:2104. 10157, 2021.
[64] Yu, Li, J. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu. Vector-quantized image modeling with improved vggan. In ICLR, 2022.
[65] J. Yu, Y. Xu, J. Y. Koh, T. Luong. G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. In ICLR, 2024.
[66] I.. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang,
Y. Hao, I. Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023.
[67] L. Yu, Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann, et al. Language model beats liffusion-tokeniin is key to visual generation. In ICLR, 2024.
[68] S. Yu, K. Sohn, S. Kim, and J. Shin. Video probabilistic diffusion models in projected latent space. In CVPR, 2023.
[69] S. Yu, Tack, S. Mo, H. Kim, J. Kim,J.WW. J.-W. Ha, and Shin. Generating videos with dynamics- aware implicit generative adversarial networks. In ICLR, 2022.
[70] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In CVPR, 2022.
[71] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In CVPR, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We clearly and accurately state the contribution and scope in the abstract and introduction (Sec. 1).
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: We discuss the limitations by showing failure cases the experimental section (Sec. 4.5) and conclusion section (Sec. 5).
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: We specify the model configurations, hyperparameters, and data in Sec. 4, supporting the reproduction of main experimental results.
5. Open access to data and code
Question: Does the paper provide open tions to faithfully reproduce the main experimental results, as described in supplemental material?
the data and code, with sufficient instruc-
Answer: [Yes].
Justification: The code has been released.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes].
Justification: We specify the model configurations, hyperparameters, and data in Sec. 4.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No].
Justification: Error bars would be too computationally expensive to report. We claim that gains in our experimental results are consistent and significant.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes].
14

---

Justification: The paper specifies details about computer resources in Sec. 4.
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes].
Justification: We make sure to conduct the research strictly following the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: We discuss the broader impacts in Sec. 5.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No].
Justification: The paper poses such risks.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: properly credit the datasets used the paper and state the license in reference.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No].
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]-
Justification: The paper does not involve crowdsourcing nor research with human subjects.
15


Reviews:
[{"review_id": "kltC1ZHQTK", "reviewer": "Reviewer_pXcq", "summary": "The paper introduces OmniTokenizer, a transformer-based image-video tokenizer designed for visual generation tasks. It adopts a spatial-temporal decoupled architecture, integrating window attention for spatial modeling and causal attention for temporal dynamics, allowing it to process both image and video data within a unified framework. A progressive training strategy is proposed, starting with image data to develop spatial encoding and then incorporating video data for learning temporal features across multiple resolutions. Extensive experiments demonstrate OmniTokenizer's state-of-the-art performance in reconstruction tasks on various datasets, outperforming previous methods.", "questions": "See weakness", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed tokenizer can enhance the performance of both language model-based and diffusion model-based visual generation approaches. \n2. The key of this work lies in its potential to provide a versatile (image and video) and efficient tool for translating complex visual data into compact latent representations.\n3. This paper is generally well-written and clearly stated.", "weaknesses": "1. The novelty is limited. The proposed tokenizer architecture is not new and is basically the same as previous methods (e.g., MagVit). The only difference is further regarding images as a 1-frame video to pretrain the network. I don’t think that counts as enough contributions.\n2. The spatial-temporal decoupled architecture is a key aspect of OmniTokenizer. A deeper dive into the role and impact of different attention mechanisms on the model's performance could offer more clarity on design choices and potential improvements. \n3. While the paper mentions the potential for scaling up the model capacity, it does not provide a detailed analysis of how the model scales with increased data volume or model size. Providing insights into scalability, such as computational complexity, training time, and resource requirements, would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The novelty is limited.\n\n    =>Answer: Please refer to question 1 of our global rebuttal.\n\n\n2. The spatial-temporal decoupled architecture is a key aspect of OmniTokenizer. A deeper dive into the role and impact of different attention mechanisms on the model's performance could offer more clarity on design choices and potential improvements.\n\n    =>Answer: Please refer to question 3 of our global rebuttal.\n\n\n3. While the paper mentions the potential for scaling up the model capacity, it does not provide a detailed analysis of how the model scales with increased data volume or model size. Providing insights into scalability, such as computational complexity, training time, and resource requirements, would be valuable.\n\n    =>Answer: Great suggestion. We leave the model scaling as our future work, as previous literature indicates that transformer models always exhibit promising scalability. We will study how the model scales with increased data or model size when more resources are available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p5ka4YSF6a", "reviewer": "Reviewer_Nxg3", "summary": "The paper introduces OmniTokenizer, a transformer-based tokenizer designed for both image and video tokenization within a unified framework. This tokenizer employs a spatial-temporal decoupled architecture, using window attention for spatial and causal attention for temporal modeling. The approach leverages a progressive training strategy, starting with image data to develop spatial encoding skills and then extending to video data to handle temporal dynamics. Extensive testing on several datasets demonstrates that OmniTokenizer achieves good performance in reconstruction tasks and enhances the effectiveness of both AR-based and diffusion based generative models.", "questions": "Overall I believe this paper shows some interesting points like progressive training and KL fine-tuning. However, the unfair comparison with baselines (magvitv2 and others) prevents me from giving higher ratings, and it is difficult to evaluate the significance of the performance given the current results", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The progressive training strategy is intuitive, allowing the model to learn incrementally from simpler to more complex tasks. This strategy contributes to the model's performance and makes sense to me.\n- The KL fine-tuning for converting a VQ tokenizer to a continuous tokenizer seems interesting and few works explored that.", "weaknesses": "- The model does not compare with magvitv2 thoroughly, which should be a strong baseline in terms of reconstruction and generation. \n- The architecture for single frame mostly follows ViT-VQGAN and the improvement comes from joint ImageNet/UCF training, when compared with single frame baselines in Table 1/3/4 the comparison seems unfair as the baseline tokenizers are trained on ImageNet only, but the UCF data can also be used as a single frame data source.\n- The factorized spatial-temporal attention is not novel as many previous works also employed that technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The model does not compare with magvitv2 thoroughly, which should be a strong baseline in terms of reconstruction and generation.\n\n    =>Answer: Please refer to question 2  of our global rebuttal.\n\n    \n2. The architecture for single frame mostly follows ViT-VQGAN and the improvement comes from joint ImageNet/UCF training, when compared with single frame baselines in Table 1/3/4 the comparison seems unfair as the baseline tokenizers are trained on ImageNet only, but the UCF data can also be used as a single frame data source.\n\n    =>Answer: Good question. Our goal is to enable image and video tokenization in a unified framework and achieve mutual benefits between them. To achieve this, it is necessary and natural to use both image and video data to train our model. In addition, as shown in the Table 17(rebuttal PDF), simply training on image and video data cannot lead to performance gains on image and video reconstruction, highlighting the importance of the proposed training strategy.\n\n    |**Method**|**ImageNet**|**K600**|\n    |:--------|:--------------:|:---------:|\n    |Ours-Image (Fix)|1.28|-|\n    |Ours-Video (Multi)|-|54.89|\n    |Joint w/o Image (Multi)|2.62|67.77|\n    |Ours-Joint (Multi) |1.11|25.97|\n\n\n3. The factorized spatial-temporal attention is not novel as many previous works also employed that technique.\n\n    =>Answer: Please refer to question 1 of our global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTH621i1Cq", "reviewer": "Reviewer_hbMY", "summary": "This paper proposes OmniTokenizer, a transformer-based visual tokenizer model that processes both image and video input and achieves state-of-the-art reconstruction quality. OmniTokenizer's core designs are a decoupled spatial-temporal attention mechanism and a progressive training schedule. Two OmniTokenizers are trained for autoregressive generative models and diffusion models, respectively. Extensive experiments further show the new tokenizers effectively improve image and video generation performance, consolidating OmniTokenizer's effectiveness.", "questions": "The paper studies visual tokenizers and proposes an effective architecture with a joint image-video training framework. The thorough quantitative results convincingly demonstrate the advantages of the proposed OmniTokenizers, and further scaling up this framework could lead to more powerful tokenizers that benefit future research. However, the paper lacks several important information: i) qualitative comparisons (especially for videos), ii) details of the tokenizer architecture and insights on why it works better than other variants, and iii)  details of the image/video generative models. The lack of these details/analyses hurts the clarity and reproducibility. I will increase the score if these concerns are properly addressed in the rebuttal.", "limitations": "The limitations and broader impacts are discussed in Sections 4 and 5 of the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Visual tokenizers are essential components for representation learning and generation, laying the foundation for many cutting-edge techniques, including image/video diffusion models and multimodal large language models. This paper's efforts in improving the tokenization quality and reconstruction performance of visual tokenizers are appreciated.\n\n2. The proposed OmniTokenizer employs image and video data jointly with a properly designed two-stage training paradigm. Ablation studies show that the joint two-stage training strategy effectively improves the reconstruction of both image and video data.\n\n3. Two variants of OmniTokenizers are trained, one with VQ and one with the vanilla VAE's KL regularization. The paper further trains autoregressive and diffusion-based image/video generative models with the OmniTokenizers, demonstrating superior results across several image and video datasets.", "weaknesses": "1. Although there are extensive quantitative results on image/video reconstruction and generation, the qualitative comparisons are insufficient. Much more video reconstruction/generation comparisons can be provided in the supplementary materials to better demonstrate the effectiveness of OmniTokenizers. \n\n2. The paper does not elaborate on the intuitions for designing decoupled window attention and causal attention. Table 8 investigates different architectural design choices, but there are no analyses on why the proposed design leads to better reconstruction quality than the other two variants. \n\n3. Section 3.1.1 of the paper is too sketchy. To make the architecture reproducible and the paper self-contained, at least some concrete description or illustration of the window/causal attention mechanisms should be provided. Similarly, there are not many details about the image/video generative models trained with OmniTokenizers— the concrete model architecture/specifications and training setups could be provided in the appendix.\n\n4. How is the order of VQ and KL training determined? There is no sufficient explanation on why the KL fine-tuning is conducted after the VQ training, and it's also unclear if the two variants of OmniTokenizers use two separate decoders or share the same decoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Although there are extensive quantitative results on image/video reconstruction and generation, the qualitative comparisons are insufficient. Much more video reconstruction/generation comparisons can be provided in the supplementary materials to better demonstrate the effectiveness of OmniTokenizers.\n\n   =>Answer: Great suggestion! Due to the rebuttal policy, we cannot update the supplementary, but we have shown more qualitative results in Figure 11 and 12 of our rebuttal PDF. \n\n\n2. The paper does not elaborate on the intuitions for designing decoupled window attention and causal attention. Table 8 investigates different architectural design choices, but there are no analyses on why the proposed design leads to better reconstruction quality than the other two variants.\n\n   =>Answer: Please refer to question 3 of our global rebuttal.\n\n\n3. Section 3.1.1 of the paper is too sketchy. To make the architecture reproducible and the paper self-contained, at least some concrete description or illustration of the window/causal attention mechanisms should be provided. Similarly, there are not many details about the image/video generative models trained with OmniTokenizers— the concrete model architecture/specifications and training setups could be provided in the appendix.\n\n   =>Answer: Thanks for your suggestion. We illustrate the network architecture in detail in Figure 13 (rebuttal PDF). The image and video transformers both follow a GPT-style architecture, which consist of 12 transformer layers. The number of heads is 12, hidden dimension is 768, and no dropout is used. We will add this introduction to our appendix.\n\n\n4. How is the order of VQ and KL training determined? There is no sufficient explanation on why the KL fine-tuning is conducted after the VQ training, and it's also unclear if the two variants of OmniTokenizers use two separate decoders or share the same decoder.\n\n   =>Answer: Great question. OmniTokenizer-VQVAE and OmniTokenizer-VAE use two separate decoders and sorry for the confusion. We chose to perform VQ training before KL training because going from discrete tokens to continuous embedding is a process of information gain and vice versa is a process of information loss. Therefore, the current order will make the learning of the model easier and more stable. As shown in the Table 12 of rebuttal PDF, if KL training is performed before VQ training, the results of the latter will instead decrease. We will update this result in the main paper.\n\n    | **Training** | **ImageNet-VQVAE** | **ImageNet-VAE** |  **K600-VQVAE** |  **K600VAE** | \n    |:--------|:--------------:|:--------------:|:--------------:|:--------------:|\n    |Joint (VQ-KL) | 1.11 | 0.69 | 25.97 | 13.02 | \n    | Joint (KL-VQ) | 2.05 | 0.89 | 33.79 | 12.43 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AgIXyUqVUk", "reviewer": "Reviewer_ZuXX", "summary": "This paper proposes OmniTokenizer, a tokenizer that can be used to tokenize both image and video data. To train the OmniTokenizer, they devise an architecture that decouples spatial and temporal axis, offering efficiency. Next, they perform a progressive training scheme that initializes from image-only training at fixed resolution that is followed by a joint image & video training at multiple resolutions. The resulting VQ tokenizer can be used with autoregressive modeling objective to perform generation. Optionally, it can also be fine-tuned with KL loss to be used as a diffusion tokenizer for latent diffusion models. The evaluations are performed on several image and video datasets to demonstrate the effectiveness of the tokenizer.", "questions": "Please see the weaknesses for details. In particular, I am wondering an in-depth comparison against relevant baselines e.g. MagVITv2, MAGE, and MaskGIT. I am also wondering qualitative results on videos and some training & experimental clarifications I discussed in the weaknesses. I will update my final score accordingly.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Having a joint tokenizer for image and video modalities is an important step. From the results, the paper seems to do a decent job at it and the resulting tokenizers could be useful to the community. The ideas of decoupling spatial & temporal axis and progressive training are intuitive and seem effective from the provided ablation results. The paper is clearly written.", "weaknesses": "My main weakness point with the paper is the lack of comparisons to the relevant baselines. I think MagVITv2 is the closest baseline to your method, but there is no in-depth comparison to it. I can see in Table 4 that MagVITv2 was reported in NAR setting while yours is in AR setting. I think it's important to have an apples-to-apples comparison to understand the effectiveness of your method. Same applies to image generation results in Table 3. For the comparison, I wouldn't limit the comparison to only generation results but also other relevant & important aspects such as video compression and the quality of learned representations. \n\nOther important and missing baselines are MAGE and MaskGIT. There are also recent advancements in the literature such as VAR but I wouldn't penalize for them as they are quite recent. Still, having a discussion & comparison to them would be useful.\n\nOther questions and comments:\n- How the trends would look like for 512x512 resolution?\n- For the reconstruction results, having other metrics such as PSNR, SSIM, and IS would be helpful.\n- Seeing the qualitative video results on actual videos would be really helpful. From the provided visuals it's unclear.\n- Are you planning to release the checkpoints & code for OmniTokenizer?\n- How do you explain the small performance gap for generation results with diffusion models? Can we say that OmniTokenizer is more effective with an autoregressive Transformer?\n- It was mentioned that the decoder is symmetric with encoder. Could you clarify that and provide a full architecture visualization? Could you also clarify the 2D and 3D patch embeddings?\n- What would be the reconstruction results for diffusion models on Table 1 & 2? You reported SD1.4 in the appendix. There are some numerical differences though (e.g. yours rFID is 0.69 in the main paper and 0.68 in the appendix), what is the reason for this? What are the reconstruction results for other latent diffusion models like SD3?\n- In the ablation results (Table 7) what would be the performance if you skip the fixed resolution image training stage and perform only the second stage on image & video jointly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. My main weakness point with the paper is the lack of comparisons to the relevant baselines. I think MagVITv2 is the closest baseline to your method, but there is no in-depth comparison to it. I can see in Table 4 that MagVITv2 was reported in NAR setting while yours is in AR setting. I think it's important to have an apples-to-apples comparison to understand the effectiveness of your method. Same applies to image generation results in Table 3. For the comparison, I wouldn't limit the comparison to only generation results but also other relevant & important aspects such as video compression and the quality of learned representations.\n\n   =>Answer: Please refer to question 2 of our global rebuttal.\n\n2. Other important and missing baselines are MAGE and MaskGIT. There are also recent advancements in the literature such as VAR but I wouldn't penalize for them as they are quite recent. Still, having a discussion & comparison to them would be useful.\n\n   =>Answer: Thanks for your suggestion. We will add more discussion on MAGE, MaskGIT, and VAR in the main paper.\n\n\n3. How the trends would look like for 512x512 resolution?\n\n   =>Answer: Good point! We evaluate VQGAN and our model at 512x512 resolution, as shown in Table 13 of rebuttal PDF, both rFID and rFVD get lower and our method still outperforms VQGAN.\n\n\n    | **Method** | **ImageNet-256** | **ImageNet-256** |  **UCF-256** |  **UCF-512** | \n    |:--------|:--------------:|:--------------:|:--------------:|:--------------:|\n    | VQGAN | 1.49 | 0.82 | 94.49 | 32.56 | \n    | Ours (VQVAE) | 1.11 | 0.72 | 25.97 | 14.09 |\n\n4. For the reconstruction results, having other metrics such as PSNR, SSIM, and IS would be helpful.\n\n   =>Answer: Thanks for pointing this out. We evaluate the PSNR and SSIM of VQGAN and our method, and show the results in Table 14 (rebuttal PDF). On both image and video benchmarks, we surpass VQGAN.\n\n    | **Method** | **ImageNet-PSNR** | **ImageNet-SSIM** |  **UCF-PSNR** |  **UCF-SSIM** | \n    |:--------|:--------------:|:--------------:|:--------------:|:--------------:|\n    | VQGAN | 23.27 | 0.69 | 26.53 | 0.81 | \n    | Ours (VQVAE) | 24.96 | 0.77 | 28.89 | 0.90 |\n\n5. Seeing the qualitative video results on actual videos would be really helpful. From the provided visuals it's unclear.\n\n   =>Answer: Great suggestion! Due to the rebuttal policy, we cannot update the supplementary to provide qualitative video results on actual videos, but we have shown more qualitative results in Figure 11 and 12 of our rebuttal PDF.\n\n\n6. Are you planning to release the checkpoints & code for OmniTokenizer?\n\n   =>Answer: Yes, we will definitely open-source the checkpoints & code for OmniTokenizer.\n\n\n7. How do you explain the small performance gap for generation results with diffusion models? Can we say that OmniTokenizer is more effective with an autoregressive Transformer?\n\n   =>Answer: Good question. For the experiments on diffusion models, we adopt the same hyper-parameters with DiT and Latte, to demonstrate the effects of replacing SD VAE with our VAE on the image/video generation performance. Therefore, these hyper-parameters are tuned for the original VAE, and we find that after tuning the learning rate for our model, the performance could be boosted, as shown in Table 15 of the rebuttal PDF (due to time limit, we only conduct experiments on Latte). Therefore, we believe OmniTokenizer is effective with both transformer models and diffusion models.  \n\n    |**Method**|**Learning rate**|**FVD**|\n    |:--------|:--------------:|:--------------:|\n    |Latte| 1e-4 | 478.0 |\n    |Ours-Latte | 1e-4 | 525.6 |\n    |Ours-Latte | 2.5e-4 | 209.2 | \n\n\n8. It was mentioned that the decoder is symmetric with encoder. Could you clarify that and provide a full architecture visualization? Could you also clarify the 2D and 3D patch embeddings?\n\n   =>Answer: Thanks for the suggestion. The detailed architecture is illustrated in Figure 13 (rebuttal PDF).\n\n\n9. What would be the reconstruction results for diffusion models on Table 1 & 2? You reported SD1.4 in the appendix. There are some numerical differences though (e.g. yours rFID is 0.69 in the main paper and 0.68 in the appendix), what is the reason for this? What are the reconstruction results for other latent diffusion models like SD3?\n\n   =>Answer: Thanks for pointing this out. The rFID of our VAE model is 0.69 and sorry for the confusion. In Table 16 (rebuttal PDF), we compare the reconstruction performance of SD1.4 VAE and SD3 VAE with our tokenizer on ImageNet, K600, and UCF (Table 1&2). It is worth mentioning that although SD VAEs are trained on large-scale high-quality data, we still beat them on ImageNet and achieve on-par results on K600 and UCF using the publicly available academic datasets.\n\n    |**Method**|**ImageNet**|**K600**| **UCF**|\n    |:--------|:--------------:|:--------------:|:--------------:|\n    |SD1.4 VAE | 0.77 | 8.89 | 20.91 | \n    | SD3 VAE | 0.74 | 6.28 | 19.43 |\n    | Ours (VAE) | 0.69 | 13.02 | 23.44 |\n\n10. In the ablation results (Table 7) what would be the performance if you skip the fixed resolution image training stage and perform only the second stage on image & video jointly?\n\n    =>Answer: Great Suggestion. We perform joint training without image pretraining and compared the results in Table 17 (rebuttal PDF). It can be seen that both ImageNet and K600 reconstruction performance get dropped compared to the progressive training, which are even worse than the image-only and video-only training. This shows the importance of image pre-training for video reconstruction learning, and also indicates that simple joint training could not improve the results of image reconstruction and video reconstruction.\n\n    |**Method**|**ImageNet**|**K600**|\n    |:--------|--------------|---------|\n    |Ours-Image (Fix)|1.28|-|\n    |Ours-Video (Multi)|-|54.89|\n    |Joint w/o Image (Multi)|2.62|67.77|\n    |Ours-Joint (Multi) |1.11|25.97|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their valuable comments. We are happy the reviewers think the progressive training strategy is **intuitive** [Review pXcq, Reviewer ZuXX] and **effective** [Review hbMY, Reviewer ZuXX]. Below we respond to the common concerns of reviewers. 
1. Novelty of this work

    =>Answer: We would like to emphasize that our primary contribution is not an innovation in model architecture, but a novel progressive training strategy. It offers twofold significant advantages over existing methods. First of all, progressive training enables image and video tokenization with one model and one weight (casting the image and video inputs to the same codebook), which allows the joint training of image and video generation with one tokenizer. Secondly, compared to image-only and video-only training, the proposed progressive training could lead to better performance on both image and video reconstruction, as can be seen from Table 7 (main paper). This is not easy to achieve, since simply training an image tokenizer like ViTVQGAN on video data will hurt the performance on image reconstruction, as verified in  Table 7 (main paper) and Table 17 (rebuttal PDF).

    |**Method**|**ImageNet**|**K600**|
    |:--------|:--------------:|:---------:|
    |Ours-Image (Fix)|1.28|-|
    |Ours-Video (Multi)|-|54.89|
    |Joint w/o Image (Multi)|2.62|67.77|
    |Ours-Joint (Multi) |1.11|25.97|

2. Comparison with MAGVITv2

    =>Answer: We strongly agree that Magvitv2 is a strong baseline for our method. However, since it is not open-source, we can only look for the results in its paper and compare with it as thoroughly as possible. 
    For visual reconstruction, as shown in Table 10 (left) of our rebuttal PDF, our method is better than Magvitv2 on ImageNet, while on UCF, we are worse than MagViTv2 but outperforms previous SOTAs like TATS significantly. For visual generation, we adopt different generative model architectures, i.e., autoregressive v.s. non-autoregressive transformers, so it is not a fair comparison. We believe that the reconstruction performance can better reflect the capability of  different tokenizers.


   |**Method**|**ImageNet**|**UCF**|
   |:--------|:--------------:|:---------:|
   |TATS|-|162|
   |MAGVITv2|1.15|9|
   |Ours-VQVAE |1.11|42|
   |Ours-VAE |0.69|23|

    We also compare with Magvitv2 on video understanding tasks, i.e., action recognition on K600. We follow them to use tokens as the input to the ViViT transformer. It can be seen from Table 10 (right) that our methods achieve on-par performance with MagViTv2 and outperforms MagViT by a clear margin.
   |**Method**|**K600**|
   |:--------|:--------------:|
   |3D VQ-VAE|45.67|
   |MAGVIT|74.65|
   |MAGVITv2|77.93|
   |Ours (VQVAE)|77.34|

    In addition, it is worth mentioning that our work differs from MAGVITv2 in that they need to train separate models for image and video tokenization, but the proposed progressive training strategy allows us to tokenize image and video inputs with the same model and weight, and to the best of our knowledge we are the first to achieve this.

3. A deeper dive into the attention mechanisms

    =>Answer: We do ablation studies on attention mechanisms in Table 8 (main paper). In order to fully analyze its effect, we additionally conduct experiments on both image and video reconstruction using different attention variants. As can be seen in Table 11 (rebuttal PDF), window attention outperforms plain attention for image reconstruction (ImageNet) since it incorporates the local modeling. While for video reconstruction (K600), using causal or not has little influence on the performance, but causal attention is necessary since we train the following image or video generation transformer in an autoregressive manner.

    |**Spatial**| **rFID**|
    |:--------|:--------------:|
    |Plain|1.55|
    |Window | 1.28 |


    |**Spatial**| **Temporal** | **rFVD**|
    |:--------|:--------------:|:--------------:|
    |Window|Plain| 26.43|
    |Window | Causal | 25.97|

Meta-review: Reviewers initially had concerns about the paper, namely around the contribution and lack of comparison to magvitv2. In rebuttal authors justified that this baseline was not open source and would be difficult to reproduce for proper comparison. They also clarified the contribution and provided additional results requested by reviewers. All reviewers increased their scores to weak accept (one did not by lapse -- they told authors they would).

While the contribution is somewhat subtle (a "progressive training strategy" for learning a joint image-video tokenizer), all reviewers believe the results are strong enough that there may be something to it, and hence deserves acceptance.
Decision: Accept (poster)