OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation

Authors: junke  wang
yi  jiang
zehuan  yuan
binyue  peng
zuxuan  wu
yu-gang  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 08:22:51

Abstract: Tokenizer, serving as a translator to map the intricate visual data into a compact latent space, lies at the core of visual generative models. Based on the finding that existing tokenizers are tailored to either image or video inputs, this paper presents OmniTokenizer, a transformer-based tokenizer for joint image and video tokenization. OmniTokenizer is designed with a spatial-temporal decoupled architecture, which integrates window attention and causal attention for spatial and temporal modeling, respectively. To exploit the complementary nature of image and video data, we further propose a progressive training strategy, where OmniTokenizer is first trained on image data on a fixed resolution to develop the spatial encoding capacity and then jointly trained on image and video data on multiple resolutions to learn the temporal dynamics. OmniTokenizer, for the first time, handles both image and video inputs within a unified framework and proves the possibility of realizing their synergy. Extensive experiments demonstrate that OmniTokenizer achieves state-of-the-art (SOTA) reconstruction performance on various image and video datasets, e.g., 1.11 reconstruction FID on ImageNet and 42 reconstruction FVD on UCF-101, beating the previous SOTA methods by 13% and 26%, respectively. Additionally, we also show that when integrated with OmniTokenizer, both language model-based approaches and diffusion models can realize advanced visual synthesis performance, underscoring the superiority and versatility of our method.

Main paper: 
 introduction: 
 The development of generative models [25, 54, 14, 17, 10, 41] has been one of the most exhilarating developments in artificial intelligence, offering the potential to revolutionize the way we generate visual content. In recent years, visual generation approaches have emerged as two dominant paradigms: language model-based methods [54, 12, 66, 48] and diffusion models [17, 45] . The former exploits the superior sequence modeling capability of language models (LMs) [36, 37, 52] for visual generation by formulating it as a next-token prediction process, while the latter gradually transforms noise into coherent visual structures through a carefully crafted reverse diffusion process.Core to both approaches is the tokenizer, which translates visual signals into latent representations, with LM tokenizers, also known as VQVAE, discretizing inputs into sequences of latent codes [12, 64, 66, 27] , and diffusion tokenizers, i.e., VAE, modeling their probability distributions within a latent space [25, 41] . Analogous to the role of the lexicon in a written language, tokenizers for visual synthesis dictate the upper bound of the generative models, thus attracting increasing attention in the community [12, 63, 19] .Existing tokenizers are designed specifically for either image [12, 64] or video [63, 13, 66] inputs, resulting in inherent limitations regarding their application flexibility and data scalability for the following generative models. Although MAGVITv2 [67] have explored causal 3D convolution to process both modalities, they still have to train separate models for the image and video data, without achieving the synergy between them. This work highlights the critical need for a joint image-video tokenizer with two primary considerations: firstly, a joint image-video tokenizer enables joint learning from image and video data [59] , which mitigates the scarcity of data in a single modality (particularly video data) and facilitates the tokenizer to learn more general representations. In addition, a unified tokenization framework inherently enjoys better versatility and scalability. For instance, its performance can be improved by incorporating the data from either modality for training. This further promotes the efficacy of generative models tailored to image or video generation.With this in mind, we present OmniTokenizer, a transformer-based tokenizer for joint image-video tokenization. As intuitive as it may seem, the simple unification of image and video data could not lead to the reciprocal effects between both modalities. To address this challenge, we turn to a spatial-temporal decoupled architecture [2] , where window attention [28] is employed in the spatial dimension owing to its local aggregation capacity and efficiency, and causal attention is used in the temporal dimension to capture the motion in videos and ensure temporal coherence. Complementing the model design, we introduce a progressive training strategy that begins with image pretraining on a fixed resolution to establish a fundamental understanding of static visual information. After this, we integrate video data for joint training on variable resolutions to capture the dynamics in more complex scenes. The progressive training strategy allows our method to bridge the gap between disparate forms of visual input and capitalize on the rich spectrum of visual data.To empirically validate the effectiveness of the proposed method, we separately implement the LM and diffusion tokenizers, i.e., OmniTokenizer-VQVAE and OmniTokenizer-VAE, and conduct experiments on a wide range of datasets including ImageNet [9] , CelebA-HQ [21] , FFHQ [22] , UCF-101 [46] , Kinetics-600 [6] , etc. The results demonstrate our model outperforms existing methods in terms of reconstruction FID on both image datasets (e.g., 1.11 rFID for OmniTokenizer-VQVAE and 0.69 rFID for OmniTokenizer-VAE on ImageNet) and video datasets (e.g., 42 rFVD for OmniTokenizer-VQVAE and 23 rFVD for OmniTokenizer-VAE on UCF-101). In addition, employing our approach for tokenization, we also show that both language model-based generative models and diffusion models could achieve competitive results on class-conditional, unconditional generation, and frame prediction tasks.In summary, our work makes the following key contributions:• We introduce OmniTokenizer, a transformer-based tokenizer for joint image and video tokenization.For the first time, OmniTokenizer employs a shared framework and weight to handle both types of visual data.• We propose a progressive training strategy that begins with image pre-training at a fixed resolution and then transits to image-video joint training at multiple resolutions. Such an approach capitalizes on the synergies between image and video data, facilitating OmniTokenizer to achieve better performance than solo image or video training.• We conduct extensive experiments across various datasets like ImageNet, CelebA-HQ, FFHQ, UCF-101, and Kinetics-600. The results showcase the state-of-the-art reconstruction performance of OmniTokenizer on both image and video datasets. Furthermore, equipped with OmniTokenizer, both language model-based generative models and diffusion models could achieve superior generation results.2 Related Work 
language models for visual generation: 
 Language models have emerged as powerful contenders in the visual generation field, drawing inspiration from their unparalleled success in natural language processing [36, 37, 51, 52] and visual understanding [11, 5, 30, 58, 57] . These methods [12, 7, 13, 66] recast visual synthesis as a sequence prediction problem, similar to constructing sentences in human language. Depending on whether the tokens are predicted sequentially or in parallel, LM-based methods can be further categorized into autoregressive models [12, 65] and non-autoregressive models [7, 67] . Autoregressive (AR) models have been the initial foray into visual generation, utilizing the inherent sequential nature of language models to generate images [64, 65] and videos [63, 13] in a step-wise fashion. These models, such as DALL-E [39] and its preceding variants, typically work by predicting one token at a time and are characterized by their high-quality outputs and precise control over the generation process. VAR [48] redefines the autoregressive learning framework on images as coarse-tofine "next-scale prediction" paradigm. Non-autoregressive (Non-AR) models, on the other hand, have been developed to allow for a faster generation process by predicting multiple tokens independently and in parallel. Models like MaskGIT [7] leverage this parallelism to significantly reduce generation time while maintaining high fidelity in synthesized images. The non-AR approaches have also demonstrated promise in video generation, featured by MAGVIT series [66, 67] . Both AR and non-AR methods have significantly advanced the field of visual generation, offering novel methods to synthesize high-quality images and videos.Diffusion models [17, 33, 3, 62, 61] represent an alternative avenue for visual generation, benefiting from their probabilistic nature that iteratively denoise a random signal into structured images or videos. These models stand out for their flexibility in generating visual outputs that not only exhibit coherent global structures but are also rich with intricate textures [32, 34] . Unlike language models that discretize visual inputs as latent codes, diffusion models directly generate visual samples in continuous pixel space [45, 10] . While effective, this approach demands significant computational resources given the high dimensionality of visual data.The advent of latent diffusion models (LDMs) [41] seeks to mitigate these issues by compressing the high-dimensional visual data into latent space with a pretrained Variational Autoencoder (VAE) [25, 41] . LDM preserves the desirable properties of pixel-space diffusion models, such as high-quality image synthesis and the ability to incorporate conditional information, while drastically reducing the training and sampling overhead. After that, the rise of LDMs [71, 35, 34, 29] continues to push visual generation toward higher quality, larger resolution, and more complex scenes.3 Methodology 
joint image and video tokenization: 
 We aim to enable image and video tokenization in a unified framework and achieve mutual benefits between them. To accomplish this, we employ a transformer-based architecture with decoupled spatial and temporal blocks (Sec. 3.1.1) . Complementing this, we also propose a progressive training strategy consisting of two consecutive stages to learn the visual encoding in an incremental way (Sec. 3.1.2). The overall framework of our method is illustrated in Figure 1 . Patchify. Given a visual input x ∈ R (1+T )×H×W ×3 , where (1 + T ) is the number of frames (T = 0 for image) and H × W denotes the spatial resolution, we always process the first frame x 0 ∈ R 1×H×W ×3 and following frames x 1:T ∈ R T ×H×W ×3 separately for the joint encoding of videos and static images [67] . Specifically, both x 0 and x 1:T are split into non-overlapping patches, with a patch size of p × p and t × p × p, respectively. After that, we project the image and video patches with two linear layers, obtaining the patch embeddings e 0 ∈ R L1×c and e 1:T ∈ R L2×c , whereL 1 = H p × W p and L 2 = T t × H p × Wp . e 0 and e 1:T are then concatenated along the sequence dimension as the spatial-temporal embedding e. In this way, we compress the input resolution from(1 + T ) × H × W to (1 + T t ) × H p × W p .Encoder and Decoder. To have better compatibility with image and video inputs, we adopt a spatial-temporal factorized encoder consisting of separate spatial and temporal blocks. In the spatial dimension, window attention [28] is employed as it exhibits superior local aggregation capability and efficiency. While in the temporal dimension, we use causal attention to align with the autoregressive visual generation in the second stage. Next, the latent code z could be obtained by looking up a codebook [54] for LM tokenizer (i.e., quantization in VQVAE), or sampling from a Gaussian distribution for diffusion tokenizer.The architecture of the decoder is symmetric with the encoder. Finally, we map the spatial-temporal tokens to the pixel space with two linear projection layers without any activation function.Unlike existing image tokenizers that conduct training on image data only [12, 64] or video tokenizers that train with image counterparts as intialization [66, 67] . We leverage a progressive training paradigm that involves two consecutive stages of VQ training to facilitate spatial-temporal representation learning of our LM tokenizer OmniTokenizer-VQVAE. After this, it could be fine-tuned as a diffusion tokenizer, OmniTokenizer-VAE, with KL fine-tuning.Two-stage VQ Training, as depicted in Figure 2 , aims to learn the visual reconstruction with the discrete latent codes. It includes two stages, the initial stage focuses on fixed-resolution image data to lay a foundation for spatial understanding. Building upon this, the second stage introduces video data to learn the modeling of temporal dynamics alongside static image features. This image-video joint training stage is critical for the model to learn a universal embedding that accurately captures both the spatial intricacies of individual frames and the temporal relationships of sequential video data.During both stages, the model is trained with vector-quantization objective: where sg denotes the stop-gradient operation, λ 1 and λ 2 are the balancing hyperparameters, E and z q represent the encoder of OmniTokenizer and codebook vectors, respectively. Factorized codes and l 2 -normalized codes [64] are also used to boost the codebook usage.EQUATIONKL fine-tuning. After the VQ training, we further fine-tune our model as a diffusion tokenizer (i.e., OmniTokenizer-VAE) by replacing the above L V Q with Kullback-Leibler (KL) loss:EQUATIONwhere P (z) is Gaussian distribution, Q(z|x) represents the inferred posterior configurations of the latent code given the observed input.Besides L V Q or L KL , both VQ training and KL fine-tuning also employs L 2 reconstruction loss L recon and GAN loss L GAN .As mentioned in Sec. 3.1.2, after the progressive training and KL fine-tuning, we can obtain two tokenizers: OmniTokenizer-VQVAE and OmniTokenizer-VAE which separately encode the visual inputs into latent codes in a discrete codebook or the continuous latent space. With this, we further train language models or diffusion models for visual generation.Language models-based generation approaches formulate visual synthesis as a token prediction problem. Specifically, after OmniTokenizer-VQVAE tokenizes image or video inputs into a sequence of discrete latent codes, we first flatten them in the raster order [8, 12] to obtain the code indices y. Then a transformer language model [36] is trained to maximize the log-likelihood between the predicted tokens ŷ and the target tokens y with cross-entropy loss:EQUATIONwhere c represents the condition (e.g., label for class-conditional image and video generation), θ is the learnable parameters of the language model, P and L denote the softmax probability and the length of y. During inference, we predict each token according to the model likelihood.Latent diffusion models (LDMs) [41] perform diffusion process in the latent space to enable highquality image synthesis with improved computational efficiency. Specifically, with the 2D latent representation from OmniTokenizer-VAE, the diffusion process gradually applies Gaussian noise to the latent code to generate a perturbed sample, while the denoising process trains a diffusion model to predict the noise that has been added. During inference, the well-trained diffusion model could generate a coherent visual sample from the noise by iteratively reversing the noising process. 
experiments: 
 Datasets. We evaluate the visual tokenization performance of OmniTokenizer on both image and video datasets, including ImageNet [9] , CelebA-HQ [21] , FFHQ [22] , Kinetics [23, 6] , UCF-101 [46] , Moments-in-Time (MiT) [31] , and Something-Something v2 (SSV2) [15] . We adopt a subset of the above datasets for visual generation to compare with previous works [12, 64, 55, 13] .Implementation Details. OmniTokenizer adopts a decoupled spatial-temporal architecture consisting of 4 window attention-based spatial layers (window size = 8) and 4 causal attention-based temporal layers. The hidden dimension is 512 and the latent dimension is 8, following ViT-VQGAN [64] . λ 1 , λ 2 , and λ 3 are set to 1, 1, 1e-6, respectively. As mentioned in Sec. 3.1.2, the training of OmniTokenizer follows a progressive training strategy, where both stages last 500K iterations. The learning rate is warmed up to 1e-3 and decayed to 0 using a cosine scheduler. Adam [24] is employed for optimization (β1 = 0.9 and β2 = 0.99). During the image training stage, we train the model with a fixed image resolution of 256×256. For the joint training stage, we forward the model with image and video data iteratively, with the video sequence length being 17 frames. The spatial resolutions are randomly chosen from 128, 192, 256, 320, and 384 [49] . Only random horizontal flip is adopted for data augmentation. We train our model using 8 NVIDIA A100 GPUs for 2 weeks. Unless otherwise stated, the results reported in this paper are jointly trained on ImageNet and UCF-101.We try both the language models and diffusion models for visual generation with OmniTokenizer as the tokenizer. The configuration for the language model follows VQGAN [12] , and for a fair comparison with previous methods, we also scale up the model size by increasing the hidden dimension to 1535, following ViT-VQGAN [64] . The training of image and video diffusion transformers follows DiT [34] and Latte [29] , respectively.We first evaluate the visual tokenization capability of OmniTokenizer on ImageNet and two highquality face datasets, CelebA-HQ and FFHQ. Reconstruction FID is used following the previous methods [12, 64] . We can observe from Table 1 that with the same compression rate and codebook size, OmniTokenizer outperforms existing methods by a large margin on all these datasets. Especially, OmniTokenizer-VQVAE achieves 1.11 FID on ImageNet, beating ViT-VQGAN, the previous stateof-the-art method by 13%. When fine-tuned as OmniTokenizer-VAE, the FID is further reduced to 0.69. We hypothesize the improved performance is because KL training provides smoother gradients than VQ training and avoids loss of information in the quantization process.In addition, we also conduct video reconstruction experiments and report the results in Table 2 . We can see that on both UCF-101 and Moments-in-Time datasets, OmniTokenizer achieves the best results. The video reconstruction results on more datasets can be found in the ablation study.Using OmniTokenizer-VQVAE for tokenization, we train language models to predict latent code indices in the codebook in an autoregressive manner for image and video synthesis. The classconditional 256×256 generation results on ImageNet, presented in Table 3 , demonstrate that our In the domain of video generation, as illustrated in Table 4 , our model beats the previous state-ofthe-art autoregressive model, TATS [13] for class-conditional video generation on UCF-101 with much lower FVD (283 v.s. 314). Moreover, for frame prediction tasks on the Kinetics-600 dataset, our model not only achieves the best performance compared to other autoregressive models but also surpasses Phenaki [55] , a non-autoregressive method.In parallel to language model-based methods, diffusion model [17, 45, 10] , especially latent diffusion model [41] , is another promising technique for visual synthesis. Therefore, we also evaluate the effectiveness of our method on diffusion model-based image and video generation with OmniTokenizer-VAE as the tokenizer. Here we employ the same architecture of DiT [34] and Latte [29] and replace their VAE [1] with OmniTokenizer-VAE. DiT [34] first applies the transformer architecture to latent diffusion models and exhibits appealing scalability properties. Following this, Latte [29] extends the transformer to the latent video diffusion model by alternating spatial and temporal attention blocks.The experimental results, as depicted in Table 5 , indicate that when equipped with OmniTokenizer-VAE, DiT-XL/2 with classifier-free guidance (CFG) achieves competitive results in terms of FID and inception score, underscoring the efficacy of our tokenizer within diffusion model frameworks for image synthesis. For unconditional video generation on the UCF-101 dataset, our method not only offers the advantage of reduced training costs by realizing a higher compression rate, but also exhibits a much lower FVD than previous methods.Training Paradigms. Deeper analysis on attention mechanisms. We conduct experiments on both image and video reconstruction using different attention variants to fully analyze its effect. As can be seen in Table 8 , window attention outperforms plain attention for image reconstruction (ImageNet) since it incorporates the local modeling. While for video reconstruction (K600), using causal or not has little influence on the performance (Table 9 ), but causal attention is necessary since we train the following image or video generation transformer in an autoregressive manner. 10 show that this will lead to performance drop on both ImageNet and K600. We believe this is because going from discrete tokens to continuous embedding will make the learning of the model easier and more stable.Visual Reconstruction. We visualize the reconstruction results by OmniTokenizer, VQGAN [12] and TATS [13] in Figure 3 . Our method works significantly better than baselines for face and text reconstruction, which are typically regarded as the most challenging reconstruction cases.Figure 3 : Image and video reconstruction results of VQGAN [12] , TATS [13] , and our method. Frame Prediction and Arbitrary Long Video Generation. The frame prediction results by our method are presented in Figure 8 , from which we can see that our model could forecast subsequent frames with high clarity and temporal coherence. Moreover, we exhibit the potential of our method for generating videos of arbitrary lengths by employing a cyclical process, where each newly generated frame is recursively used as a condition for the subsequent frame generation. Previous literature [20, 16, 70, 48, 47] has revealed that the performance of transformer models improves significantly as the model size increases, also known as scaling law. In the future, we will explore scaling the model capacity of OmniTokenizer for more advanced tokenization performance. 
 
 Appendix:  


Decision: Reviewers initially had concerns about the paper, namely around the contribution and lack of comparison to magvitv2. In rebuttal authors justified that this baseline was not open source and would be difficult to reproduce for proper comparison. They also clarified the contribution and provided additional results requested by reviewers. All reviewers increased their scores to weak accept (one did not by lapse -- they told authors they would).

While the contribution is somewhat subtle (a "progressive training strategy" for learning a joint image-video tokenizer), all reviewers believe the results are strong enough that there may be something to it, and hence deserves acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces OmniTokenizer, a transformer-based image-video tokenizer designed for visual generation tasks. It adopts a spatial-temporal decoupled architecture, integrating window attention for spatial modeling and causal attention for temporal dynamics, allowing it to process both image and video data within a unified framework. A progressive training strategy is proposed, starting with image data to develop spatial encoding and then incorporating video data for learning temporal features across multiple resolutions. Extensive experiments demonstrate OmniTokenizer's state-of-the-art performance in reconstruction tasks on various datasets, outperforming previous methods.
strength_weakness: strength_weakness: 1. The novelty is limited. The proposed tokenizer architecture is not new and is basically the same as previous methods (e.g., MagVit). The only difference is further regarding images as a 1-frame video to pretrain the network. I don’t think that counts as enough contributions.
2. The spatial-temporal decoupled architecture is a key aspect of OmniTokenizer. A deeper dive into the role and impact of different attention mechanisms on the model's performance could offer more clarity on design choices and potential improvements. 
3. While the paper mentions the potential for scaling up the model capacity, it does not provide a detailed analysis of how the model scales with increased data volume or model size. Providing insights into scalability, such as computational complexity, training time, and resource requirements, would be valuable.
questions: questions: See weakness
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces OmniTokenizer, a transformer-based tokenizer designed for both image and video tokenization within a unified framework. This tokenizer employs a spatial-temporal decoupled architecture, using window attention for spatial and causal attention for temporal modeling. The approach leverages a progressive training strategy, starting with image data to develop spatial encoding skills and then extending to video data to handle temporal dynamics. Extensive testing on several datasets demonstrates that OmniTokenizer achieves good performance in reconstruction tasks and enhances the effectiveness of both AR-based and diffusion based generative models.
strength_weakness: strength_weakness: - The model does not compare with magvitv2 thoroughly, which should be a strong baseline in terms of reconstruction and generation. 
- The architecture for single frame mostly follows ViT-VQGAN and the improvement comes from joint ImageNet/UCF training, when compared with single frame baselines in Table 1/3/4 the comparison seems unfair as the baseline tokenizers are trained on ImageNet only, but the UCF data can also be used as a single frame data source.
- The factorized spatial-temporal attention is not novel as many previous works also employed that technique.
questions: questions: Overall I believe this paper shows some interesting points like progressive training and KL fine-tuning. However, the unfair comparison with baselines (magvitv2 and others) prevents me from giving higher ratings, and it is difficult to evaluate the significance of the performance given the current results
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes OmniTokenizer, a transformer-based visual tokenizer model that processes both image and video input and achieves state-of-the-art reconstruction quality. OmniTokenizer's core designs are a decoupled spatial-temporal attention mechanism and a progressive training schedule. Two OmniTokenizers are trained for autoregressive generative models and diffusion models, respectively. Extensive experiments further show the new tokenizers effectively improve image and video generation performance, consolidating OmniTokenizer's effectiveness.
strength_weakness: strength_weakness: 1. Although there are extensive quantitative results on image/video reconstruction and generation, the qualitative comparisons are insufficient. Much more video reconstruction/generation comparisons can be provided in the supplementary materials to better demonstrate the effectiveness of OmniTokenizers. 

2. The paper does not elaborate on the intuitions for designing decoupled window attention and causal attention. Table 8 investigates different architectural design choices, but there are no analyses on why the proposed design leads to better reconstruction quality than the other two variants. 

3. Section 3.1.1 of the paper is too sketchy. To make the architecture reproducible and the paper self-contained, at least some concrete description or illustration of the window/causal attention mechanisms should be provided. Similarly, there are not many details about the image/video generative models trained with OmniTokenizers— the concrete model architecture/specifications and training setups could be provided in the appendix.

4. How is the order of VQ and KL training determined? There is no sufficient explanation on why the KL fine-tuning is conducted after the VQ training, and it's also unclear if the two variants of OmniTokenizers use two separate decoders or share the same decoder.
questions: questions: The paper studies visual tokenizers and proposes an effective architecture with a joint image-video training framework. The thorough quantitative results convincingly demonstrate the advantages of the proposed OmniTokenizers, and further scaling up this framework could lead to more powerful tokenizers that benefit future research. However, the paper lacks several important information: i) qualitative comparisons (especially for videos), ii) details of the tokenizer architecture and insights on why it works better than other variants, and iii)  details of the image/video generative models. The lack of these details/analyses hurts the clarity and reproducibility. I will increase the score if these concerns are properly addressed in the rebuttal.
limitations: limitations: The limitations and broader impacts are discussed in Sections 4 and 5 of the main paper.
paper_summary: paper_summary: This paper proposes OmniTokenizer, a tokenizer that can be used to tokenize both image and video data. To train the OmniTokenizer, they devise an architecture that decouples spatial and temporal axis, offering efficiency. Next, they perform a progressive training scheme that initializes from image-only training at fixed resolution that is followed by a joint image & video training at multiple resolutions. The resulting VQ tokenizer can be used with autoregressive modeling objective to perform generation. Optionally, it can also be fine-tuned with KL loss to be used as a diffusion tokenizer for latent diffusion models. The evaluations are performed on several image and video datasets to demonstrate the effectiveness of the tokenizer.
strength_weakness: strength_weakness: My main weakness point with the paper is the lack of comparisons to the relevant baselines. I think MagVITv2 is the closest baseline to your method, but there is no in-depth comparison to it. I can see in Table 4 that MagVITv2 was reported in NAR setting while yours is in AR setting. I think it's important to have an apples-to-apples comparison to understand the effectiveness of your method. Same applies to image generation results in Table 3. For the comparison, I wouldn't limit the comparison to only generation results but also other relevant & important aspects such as video compression and the quality of learned representations. 

Other important and missing baselines are MAGE and MaskGIT. There are also recent advancements in the literature such as VAR but I wouldn't penalize for them as they are quite recent. Still, having a discussion & comparison to them would be useful.

Other questions and comments:
- How the trends would look like for 512x512 resolution?
- For the reconstruction results, having other metrics such as PSNR, SSIM, and IS would be helpful.
- Seeing the qualitative video results on actual videos would be really helpful. From the provided visuals it's unclear.
- Are you planning to release the checkpoints & code for OmniTokenizer?
- How do you explain the small performance gap for generation results with diffusion models? Can we say that OmniTokenizer is more effective with an autoregressive Transformer?
- It was mentioned that the decoder is symmetric with encoder. Could you clarify that and provide a full architecture visualization? Could you also clarify the 2D and 3D patch embeddings?
- What would be the reconstruction results for diffusion models on Table 1 & 2? You reported SD1.4 in the appendix. There are some numerical differences though (e.g. yours rFID is 0.69 in the main paper and 0.68 in the appendix), what is the reason for this? What are the reconstruction results for other latent diffusion models like SD3?
- In the ablation results (Table 7) what would be the performance if you skip the fixed resolution image training stage and perform only the second stage on image & video jointly?
questions: questions: Please see the weaknesses for details. In particular, I am wondering an in-depth comparison against relevant baselines e.g. MagVITv2, MAGE, and MaskGIT. I am also wondering qualitative results on videos and some training & experimental clarifications I discussed in the weaknesses. I will update my final score accordingly.
limitations: limitations: Yes.

Comments: title: Thanks for raising the rating to weak accept
Thanks for your positive comments!
Thanks for your positive comments!
Thanks, rebuttal was helpful
Help check whether questions are well answered.
Additional Notes on MAGVITv2 Comparison
comment: Dear reviewer Nxg3, thanks for your effort in reviewing our paper and choosing to raise your rating to weak accept!
I have read the authors' responses and other reviewers' comments. I appreciate the authors' efforts on providing additional results and choose to raise my rating to weak accept.
Thanks for your valuable suggestions and positive feedback. We will incorporate these experimental results and the architecture explanation into the paper.
Thank you for providing the new results and more details of the model architecture. The intuitive explanation of the order of VQ and KL training makes sense to me, and is well backed up by the new ablation study. All my questions are resolved, and I will increase my rating to 6. If the paper is accepted, please incorporate the additional qualitative/quantitative results and the detailed architecture explanation into the paper (most of them can go into the appendix with a reference in the main paper).
Thanks for your positive comments and we will follow your suggestion to add these results to our paper!
Thank you for the detailed response and the new results you provided, they were helpful. Please incorporate all of them to your paper. I still think having an apples-to-apples comparison with MagVITv2 on visual generation will be useful to the practitioners to understand the capabilities of two approaches. Consider adding this to your paper as well.

I'm raising my score to weak accept.
Dear reviewer Nxg3,

We would like to thank you again for your effort and valuable suggestions. Can you help find time to take a look at the response and check whether your questions are well answered. We are very happy to discuss with you and provide further clarification for any new question.
MAGVITv2 utilizes an exceptionally large codebook size (2^18), whereas we have used only 16k, which precludes a fair comparison. Additionally, since MAGVITv2 is not open-source, reproducing its work to achieve a fair comparison is challenging. In future work, we will compare performance using the same codebook size.