VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
Authors: Yecheng Wu
Zhuoyang Zhang
Junyu Chen
Haotian Tang
Dacheng Li
Yunhao Fang
Ligeng Zhu
Enze Xie
Hongxu Yin
Li Yi
Song Han
Yao Lu
Venue: ICLR 2025 Poster (2025)
Keywords: Unified Visual Language Model
Autoregressive Model
Primary area: foundation or frontier models, including LLMs
TL;DR: VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. It employs a single autoregressive next-token prediction framework for both tasks.

Abstract: VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.

Published as a conference paper at ICLR 2025
VILA-U: A UNIFIED FOUNDATION MODEL INTEGRAT- ING VISUAL UNDERSTANDING AND GENERATION
Yecheng Wu Wu Zhuoyang Zhang2+1
Junyu Chen 1.2 Haotian Tanga
Dacheng Li
Yunhao Fang Ligeng Zhu
Enze Xie
Hongxu Yin Yi Song Han
Yao Lu
Tsinghua University* MIT NVIDIA
UC Berkeley
UC San Diego
httpss//aal.bb.......errrrrrro mttt.edu/projects/vil..
wwt.
ABSTRACT
VILA-U is Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of offtthe-rtt performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using fully token-based autoregressive framework. Our code open sourced at https:// / thub. cwm/mittttcoooo hhan:/ntttttiitt-eeeeeeeee 1ab /vilaal..
INTRODUCTION
In recent years, large language models (LLMs) have demonstrated superior capabilities in various language tasks. Their appealing properties like instruction following, zero-shot generalization, and few-shot in-context learning motivate researchers to combine them with vision models build visual language models (VLMs) for multi-modal tasks. Many efforts (Dai et al., 2024; Liu et al., 2024b; Lin et al., 2023) in this field have achieved remarkable performance on visual language understanding. In these works, visual inputs are projected onto I.I.Ms' semantic space through vision model like CLIP (Radford et al., 2021) to bridge two modalities by including text-image alignment objectives.
In addition to visual understanding, another essential research direction in combining visual and language modalities visual generation. There are two popular approaches for text-guided image generation. One approach employs diffusion models (Rombach et al., 2022a), powerful tool for various generation tasks. The other line of work converts visual content into discrete tokens through vector quantization (VQ) and then leveraging autoregressive transformers for high- and diverse generation (Esser et al., 2021; Yu et al., 2021; Lee et al., 2022; Tian et al., 2024b; Sun et al., 2024).
Witnessing the rapid advancements in both visual understanding and generation, emerging trend to unify these techniques into a single multi-modal framework. Prior to VILA-U, there are two main approaches to achieving such unification: (1) One approach (Liu et al., 2024a; Yu al., 2023a; (Esser et al., 2021) tokenizer to convert visual inputs
Xie et al., 2024) utilizes VQGAN- into discrete tokens and leverages an autoregressive model for both understanding and generation. However, (Xie et al., 2024) has shown that visual tokens from VQGAN- information and usually results in a severe performance drop in downstream visual understanding tasks. (2) Another approach (Zhan et al., 2024; Ge et al., 2023b; Jin et al., 2023) utilizes codebook to quantize features produced by a pre-trained vision model like CLIP. Since CLIP features encode
encoder lack semantic
"Equal Contribution.
1 Part of the work done during an internship NVIDIA.

---

Published as a conference paper at ICLR 2025
rich semantic information, these approaches generally achieve significantly better performance on understanding tasks. However, these tokenizers lack decoding capability, requiring an external visual generation model, such as diffusion model, to use the generated visual tokens as conditions for producing visual outputs. This approach adds complexity to infrastructure design. Available large-scale foundation model training pipelines and deployment systems have already been highly optimized for language modeling with next-token prediction. Designing and maintaining an additional stack to support diffusion models would incur significant engineering costs.
In this work, we present VILA-U, an end-lo-end autoregressive framework with a unified next-token prediction objective for both visual and text inputs that can achieve competitive performance on both visual language understanding and generation tasks, without the help of external components like diffusion models. We identify two critical principles to unify vision and language modalities: (1) Existing unified end-to-end autoregressive VLMS cannot achieve competitive visual understanding performance because the discrete VQGAN tokens are trained solely on image reconstruction loss and are not aligned with textual inputs. Therefore, it is crucial to introduce text alignment during VQ vision tower pretraining to enhance perception capabilities. (2) Autoregressive image generation can attain similar quality as diffusion models if trained on high-quality data with sufficient size. Guided by these insights, VILA-U features a unified foundation vision tower that converts visual inputs into discrete tokens through vector quantization and aligns these tokens with textual inputs using contrastive learning. The multi-modal training of VILA-U takes advantage of a unified next-token prediction objective for both visual and textual tokens on a small-size high-quality image-text corpus.
We evaluate VILA-U on common visual language tasks, including image-langual understanding, video-language understanding, image generation and video generation. VILA-U significantly nar- rows the gap in visual understanding performance between end-to-end autoregressive models and continuous-tokk VLMs, while introducing competitive native visual generation capabilities.
2 RELATED WORK
Large Language Models (LLMs). LLMs based on pre-trained large-scale transformers (Vaswani al., 2017) has drastically revolutionized natural language processing field. Featuring gigantic model size and pre- pre-training data corpus, LLM has achieved remarkable performance on various linguistic tasks. The development of open-source LLMs such as LLaMA (Touvron et al., 2023a), Mixtral (Jiang et al., 2024) and Vicuna (Chiang et al., 2023) has furthered nourished research on how to adopt L.M for complex language tasks. Besides excellent zero-shot generalizability to diverse domains, LLM is commonly finetuned on custom datasets for better performance specific tasks. Instruction tuning (OpenAI, 2023; Chung et al., 2024; Ouyang et al., 2022) also stands as a key step for better outputs in applying LLMs. In this work, we adopt the LLaMA-2-7B (Touvron et al., 2023a) model as our basic LLM.
Visual Language Models (VLMs). Combining computer vision and natural language processing gives rise to VLM in this LLM era. In VLMs, researchers leverage vision foundation models such as CLIP (Radford et al., 2021), BLIP (Li et al., 2022) and CoCa (Yu et al., 2022) to extract visual features, align with texts, and feed them into L.M to achieve the cross-modality understanding between texts and visual content. Building upon such progress, many VLMs (Alayrac al., 2022; Li et al., 2023b; Liu et al., 2024b; Lin et al., 2023; Luo et al., 2024; Tian et al., 2024a) have been designed and trained on extensive vision-language data achieve remarkable performance on visual understanding and reasoning tasks. In this work, we aim to develop VLM with visual understanding capacities comparable to prior works, while also possessing the new capacity of visual generation.
Unified Visual Language Models. Numerous efforts have been made to develop unified visual language models capable of generating both text and visual content, including images and videos. There are two mainstream methods to generate visual content in VLMs. Many works (Sun et al., 2023b;a; Jin et al., 2023; Ge et al., 2023b; Li et al., 2023c; Ge et al., 2024; Jin et al., 2024; Ge et al., 2023a) combine VLMs with diffusion models like Stable Diffusion (Rombach et al., 2022a) for high-quality image generation. Other works (Liu et al., 2024a; Yu et al., 2023a; Lu et al., 2023; Team, 2024; Xie al., 2024) adopt VQGAN-based vision encoders to convert visual inputs into discrete tokens and make LLMs learn to predict them. For more details on the distinction between our method and other unified visual language models, please refer to Appendix A.
2

---

Published as a conference paper at ICLR 2025
Multi-modal Inputs
Multi-modal Token Sequence
Multi-modal Sequence Out
Multi-modal Inference Outputs
Text
Text Encoder
Text Decader
The mon
skoting
Text Tokens
Generative Multi-modal Model
Imoge
Text aligned Vision Encoder
Vision
Decoder
Text-aligned Discrete
Video
Visual Tokens
Next-token Prediction
Figure I: An overview of our framework's multi-modal training and inference process. Visual inputs are tokenized into discrete tokens and concatenated with textual tokens form a multi-modal token sequence. All tokens are involved in our next-token prediction process, enabling unified training objective. During inference, the output tokens are decoded by our text detokenizer or vision tower decoder to yield multi-modee content.
3
METHODS
This work proposes multi-modal framework that aims to unify visual and language modalities effectively. The key components enabling such unification are a unified foundation vision tower that converts visual inputs into discrete tokens aligned with text, and unified multi-modal generative training procedure. An overview of the main multi-modal training and inference process within our framework is depicted in Figure 1.
3.1 UNIFIED FOUNDATION VISION TOWER
To support diverse visual understanding and generation tasks, we first build unified foundation vision tower to provide appropriate visual features. We propose include text-image contrastive loss and VQ-based image reconstruction loss in our vision tower training, empowering the text alignment and discrete tokenization abilities for our vision tower. As depicted Figure the features extracted from images are primarily discretized through residual quantization. Then in one route, the discrete visual features are fed into decoder to reconstruct the image and compute the reconstruction loss; on the other route, we compute the image-text contrastive loss between the discrete visual features and the textual features provided by a text encoder. With this training procedure, the vision tower learns to extract discrete features suitable for both understanding and generation in our VLM.
Unified Training Recipe. Training the unified vision tower with two objectives from scratch would be diffieult, because alignment and reconstruction tasks require high-level semantic and low-level appearance features, respectively. Training the entire vision tower from scratch with both objectives could induce conflicting goals. practice, we observe that training the vector-quantized vision tower from scratch with both image reconstruction and contrastive loss results in mere 5% Top-1 accuracy for zero-shot image classification on ImageNet (Deng al., 2009a) after several epochs of training.
To address this issue, we experiment with different training recipes (failed recipes are listed in Appendix C) and find the following solution to most effective. Instead of learning both objectives simultaneously, we suggest first equipping the model with text-image alignment ability and then learning reconstruction while maintaining alignment ability. We initialize the vision encoder and text encoder with pretrained weights from the CLIP model to ensure good text-image alignment. Next, we freeze the text encoder and keep all vision components trainable using both contrastive and reconstruction loss. The contrastive loss maintains alignment ability, while the reconstruction loss develops reconstruction ability. This approach converges quickly and yields strong performance. The pre-trained CLIP weights contain learned high-level priors, which are difficult and computationally expensive to learn from scratch. Initializing with these weights enables the binding of low-level and high-level features much faster and more tractably for the vision encoder. With this recipe, we train a vision tower that exhibits both good text alignment and image reconstruction abilities. We use weighted combine the text-image contrastive loss and VQ-based image reconstruction loss:
Ltotal = Wcontra Loontra + Wrecon Erecon
(1)
In experiments, pick Woontra and Wrecon

---

Published as a conference paper at ICLR 2025
Reconstruction
Loss
Vision Encoder Pretromed
Residual Quantizer
Vision
Decoder
Discrete Visual Takens
This is en image o cot
Text Encoder Pretroined
Controstive
Frazen weights
Trainable weights
Figure 2: Overview of our unified foundation vision tower. Given input images the features extracted by the vision encoder are discretized using residual quantization. Then the discrete vision features are meanwhile put into the vision decoder to reconstruct images and used to perform the text-image alignment. During this process, the reconstruction loss and contrastive loss are computed to update the vision tower, endowing it to produce discrete visual features with text alignment.
Residual Vector Quantization. Our visual features are discretely quantized, so their representation ability heavily depends on the code size used in our quantizer. Since we hope they contain both high-level and low-level features, we need more capacities in their vector feature space, making larger code size necessary for good performance in downstream tasks. However, too many codes for each image will result in too many tokens for L.LM to produce in the visual generation process, incurring much latency. So in an attempt to increase the vector feature capacity and meanwhile maintain reasonable number of tokens for LLM, we adopt a residual vector quantization method following RQ-VAE (Lee et al., 2022) to discretize vector D discrete codes:
RQ(z; C, D) (ki,....
kD) [K]P,
(2)
where C is the codebook, |C| and k the code of z at depth d. Starting with To z, we recursively perform vector quantization by
2 Cr)
rd-1 - e (ka),
for each depth d 1.2,..., D, where is the codebook embedding table and Q is the standard vector quantization:
Q(z: C)
arg minl e(k)112 ke[K]
The quantized vector for z the sum over the depth dim: E e (k,)). Intuitively, in each depth we choose code to reduce the quantization error. So compared to the standard vector quantization methods, we have 1 codes to quantize one vector, allowing for finer approximation and larger feature space. During multi-modal training and inference, LLM only needs to predict the code embedding, with codes in different depth sequentially produced by a depth transformer taking the code embedding the initial input, enhance the representation capability our vision tower while incurring little latency.
will introduce in Section 3.2. So with this residual quantization,
3.2 UNIFIED MULTI-MODAL GENERATIVE PRE-TRAIN TRAIIRINNG
Figure 1 presents an overview of our unified multi-modal pre-training process. Our vision tower encoder processes visual inputs sequentially, generating ID token sequence. This sequence is then concatenated with text tokens form a multi-modal sequence. To distinguish between modalities and enable visual content generation, we insert special tokens: <image_start> start>> and <image_ent end> the start and end of image tokens, and <video siddeo_starr______diddiiddddda_e start> and <video_edt _nd__ddeddd>>__ddeee at the start and end of video tokens. Video tokens are the direct concatenation of multi-frame image tokens.
Pre-training data form. In terms of unified pre-training data, we leverage different concatenation forms between text and visual tokens to facilitate both understanding and generation. We use [image, text], [text, imagel, and [text, video] forms, with supervision loss added only on the latter modality in each pair to avoid unconditional content generation and promote modality alignment. We also employ an interleaved text and image concatenation form for enhanced understanding, with supervision loss applied solely to the text. Notably, exclude the [video, text] ] form during pre-training for efficiency reasons, as we find incorporating it during supervised fine-tuning effectively yields excellent video understanding ability.
4

---

Published as a conference paper at ICLR 2025
Training Objective. Since both visual tokens and text tokens discrete, can train LLM with the general language modeling next- prediction objective. However, due to the use of residual quantization for visual tokens, the training objectives for text and visual tokens differ slightly. For text tokens, the negative log-likelihood loss is calculated as
T Liext Σ
log Pe (uilves),
(5)
where T' is the length of the multi-modal sequence and only counts when the text token appears at position i. For visual tokens, residual quantization introduces depth- structure of codes at each visual position j. To address this, we leverage the depth transformer introduced in RQ-VAP (Lee et al., 2022). Specifically, given the code embedding h, generated by the LLM for visual tokens at position j. the depth transformer autoregressively predicts D residual tokens (kj1, p)). During training, the input of the depth transformer Uja depth d is defined the sum of the code embeddings of up to depth - 1 for d > 1 such that
d-1
Uja Σ e(k e(kjæ),
(6)
d'=1
and Uij1 = hj. Thus, the depth transformer predicts the next code for finer estimation of the feature 2j based on the previous estimations up to d 1. Then the negative log-likelihood loss for visual tokens is
D log Ps (kija Misea)
Livisual -- j=1d=
where T is the length of the multi-modal sequence and j only counts when visual token appears at position j. During the multi-modal pre-training, the weights of the depth transformer are randomly initialized and updated together with the LLM.
4 EXPERIMENTS
In this section, we introduce comprehensive experiments to evaluate our method on various visual understanding and generation tasks. Firstly, outline our experimental setup, including the model architecture, training datasets, and evaluation henchmarks. Subsequently, we evaluate the performance of our unified foundation vision tower. Then, we compare our method with other popular VLMs on various visual understanding and generation benchmarks. Finally, we give some qualitative results.
4.1 EXPERIMENTAL SETUP
In our experiments, we employ LLaMA-2-7B (Touvron et al., 2023b) as our base language model. For the vision tower, we choose Laciii--arrr-paachh-65 / S00-I---0400Mppachllll 14-344 (Zhai et al., 2023) as our vision encoder architecture, and adopt the residual quantizer, depth transformer well as the decoder architecture from RQ-VAE (Lee al., 2022). The quantizer codebook size is 16384. All images and videos are resized to a resolution of 256 x 256 384 x 384, with each image or video frame converted into 16 16 4/ 27 x 27 16 code with the residual depth D= 4/ 16. We train our vision tower on COYO-700M (Byeon et al., 2022) and evaluate for zero-shot classification and reconstruction performance on ImageNet (Deng et al., 2009b). For visual understanding, we leverage IM [image, text] data from ShareGPT4V (Chen et al., 2023), 6M interleaved text and image data from MMC4 (Zhu et al., 2024). For visual generation, incorporate 15M high-quality [text, image] data curated from internal dataset and 1M [text, video] data from Open Vid (Nan et al., 2024) datasets. Classifier free guidance (Ho & Salimans, 2022) is employed for visual generation with CFG value of 3.
For examining visual understanding ability, we evaluate our model on the widely adopted zero-shot image-based visual-language benchmarks including VQAv2 (Goyal al., 2017), GQA (Hudson &
5

---

Published as a conference paper at ICLR 2025
Manning, 2019). TextVQA (Singh et al., 2019), POPE (Li et al., 2023d), MME (Fu al., 2024), SEED (Li et al., 2023a), MM- Vet (Yu et al., 2023b) and video-based vasual-language benchmarks including ActivityNet (Caba Heilbron et al., 2015), MSVD (Chen & Dolan, 2011), MSRVTT (Xu et al., 2017), TGIF (Li et al., 2016).
To evaluate the visual generation capability, we MJHQ-30K (Li et al., 2024) and GenAl-Bench (Lin et al., 2024) for image generation and VBench (Huang et al., 2024) for video generation. MJHQ-30K adopts the FID between generated images and 30K high-quality images to reflect the overall capability of image generation. GenAl-Bench is challenging image-to-text generation benchmark that reflects the comprehensive generative abilities of image generation models. Vbench is a comprehensive benchmark suite for video generative models that decomposes the generation quality into multiple well-defined dimensions to facilitate fine- tne-graineed and objective evaluation.
4.2 UNIFTED FOUNDATION VISION TOWER
We present the commonly used metries reconstruction FID (rFID) and Top-1 accuracy for zero-shot image classification on ImageNet to measure the reconstruction and text alignment capabilities of the unified foundation vision tower in Table Please refer to the Appendix B.1 for the qualitative reconstruction results. Our model achieves significantly better reconstruction results than VQ-GAN. Our rFID is slightly inferior to that of RQ- VAE when using the same code shape. This is expected as the introduction of contrastive loss during training, aimed at enhancing image understanding, led to decrease in reconstruction quality. For the text alignment capability, our unified vision tower achieves Top-1 accuracy of 73.3 78.0 under 256 384 resolution. This demonstrates the exceptional text alignment capability of our unified vision tower. However, it is worth noting that both the rFID) and Top-1 accuracy of the vision tower only serves medium indicator. As the unified vision tower is an integral component of the entire autoregressive model, we believe that its performance on downstream tasks, such as visual understanding and generation, holds greater significance.
Table The reconstruction FID (rFID) and Top-1 accuracy for zero-shot image classification of our unified vision tower ImageNet.
Model
Pretrained Weights Resolution
Shape of Code
rFI.
Top-1 Accuracy?
VQ-GAN
256 256
16 x 16
4,98
RQ-VAE
256 256
x x
3,20
RQ-VAE
256 256
16 16 x
1.30
Ours
LigLie-Large SigLIP-SO400M
256 256
x 16 4
73.3
Ours
384 384
27
1.25
78.0
4.3 QUANTITATIVE EVALUATION
Visual Understanding Tasks. Table 2 and Table 3 summarize the comparison between our method and other leading VLMs on the image-language and video-language benchmarks respectively. Com- pared to the mainstream choice of continuous visual tokens produced by foundation models like CLIP, the VQGAN-based discrete visual tokens have less alignment with text, thus harming VLMs' performance on visual understanding tasks. With our unified foundation vision tower, our model can have a performance close to leading VLMs even with discrete visual tokens.
Visual Generation Tasks. As shown in Table 4, VILA-U can achieve a better FID than other autoregressive methods and have comparable performance with some diffu- sion based methods. This result shows the feasibility of our method for visual gener- ation. Table 5 summarizes the quantitative results of our method and other visual gener- Although
Method
Type Diffusion
#Images
FID.
SD v2.1
26.96
SD-XI.
Diffusion
2000M
9.55
PixArt
Diffusion
25M
6.14
Playground v2.5
Diffusion
4.48
LWM
Autoregressive
17.77
Show-o
Autoregressive
36M
15.18
ation methods on GenAl-Bende Our method is inferior to diffusion- based visual generation methods that have been trained billions-level image-text pairs,
(256)
Autoregressive
15M
12.81
(384)
Autoregressive
15M
7.69
Table 4: Comparison with other visual generation methods on MJHQ-30K evaluation benchmark.

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with leading methods on image-based visual language benchmarks. Our performance is close to leading VLMs, surpassing many methods by large margin under the same LLM size, even with discrete visual token type. indicates that images in the training split of these datasets are observed during VLM training.
Method
LLM
Visual Token
Res.
VQA+2
GQA
TexVVQA
POPE
MME
SEED
MM- Vet
LLaVA-15
Socuaa-1.5-25
Continuous
336
78.5*
62.0"
58.2
85.9
1510.7
58.6
30.5
2-7B
79.9*
62.3*
61.1
34.9
Unified-10 10 2
6,8B from scratch
Continuous
384
79.4
87.7
61.8
InstreeBLIIP
Vicuna-7B
Contimoous
224
49.2
50.1
53.4
26.2
IDEFICS-BB
LLaMA-7B
Continuous
224
50.9
38.4
25.9
Emu
LLaMA-13B
Continuous
224
LaVIT
LLMMA-BB
Continuous
224
66.0
46.8
DreamLLM
Vicuna-7B
Continuous
224
72.9'
41.8
36.6
Video-LaVIT
LLaMA-2-7B
Continuous
224
80.2"
63.6*
15815
64.4
35.0
Emu2-Chat
Emu2-37B
Continuous
448
84,9"
65.1"
66.6'
Interleaved
60.5*
DEEM
Vicuna-7B
Continuous
448
68.2"
55.7*
37.4
scratch
LWM
LLaMA-2-7B
Discrete
256
55.8
44,8
18.8
75.2
9.6
Show-o
P
Discrete
256
59.3*
48.7*
948.4
LEED-LLaMA
Vicuna-7B
Discrete
224
66.2
51.5
Ours
LLaMA-2-78
Discrete
256
75.3"
58.3"
48.3
83.9
1336.2
56.3
27.7
LLaMA-2-7B
33.5
Table 3: Comparison with leading methods video-based visual language benchmarks. The performance of our method is close to statc-of-the-art VLMs, surpassing many methods under the same LLM size, even with a discrete visual token type.
Method
Visual Token
MSVD-QA
MIRVTT-QA
TGIF-QA
Activity Net-QA
2
6.8B from seratch
Continuous
384
52.1
42.5
Emu
LLaMA-13B
Continuous
224
18.8
8.3
VideoChat
Vicuna-7h
Continuous
224
56.3
34.4
Video LLaMA
LLaMA 2 7B
Continuous
224
51.6
29.6
LLaMA-2-7B
Continuous
49,3
35.2
Video LLvvv
Vicuna 7B
Continuous
224
70.7
$9.2
70.0
45.3
Video-LaVIT
LLMMA-2-7B
Continuous
224
73.5
50.2
Emu2-Chat
Emu2-37B
Contiquous
448
49.0
31.4
LWM
LLaMA-2-7B
Discrete
256
55.9
44.1
40.9
SEED LLaMA
Vicuna 7B
Discrete
224
40.9
30.8
Ours
LLaMA
Discrete
256
73.4
58.9
$1.3
51.6
our method has comparable performance with SD v2.1 (Rombach et al., 2022b) and SD-XL (Podell et al., 2023) on advanced prompts even trained with magnitude-level less data. This further shows that VILA-U can learn the correlation among visual and textual modalities effectively with our unified training framework. For video generation, we evaluate our method on VBench (Huang et al., 2024) and compare it against Open-Sora (Zheng et al.), Cog Video (Hong et al., 2022), and Cog VideoX (Yang et al., 2024). The results, presented in Table 6, demonstrate that our method achieves performance that better than CogVides and comparable to Open-Sora, highlighting the effectiveness of our approach.
4.4 QUALITATIVE EVALUATION
Visual Understanding. To validate the effectiveness of VILA-U in comprehensive visual understanding tasks, we apply
Method
Total Score?
Quality Scoret
Semantic Scoret
Open-Sora
75.91
78.82
64.28
several under- standing and reasoning tasks, as some examples shown in Figure and Figure 4. From the results, can see the versatility of VILA-U in various tasks including visual captioning and visual question answering. Besides, model has inherited some important capabilities from VILA (Lin al.,
Cog Video
67.01
72.06
46.83
Cog VideoX
81.61
82.75
77.04
Ours (256)
74.01
76.26
65.04
Table 6: Comparison with other visual generation methods on VBench (Huang et al., 2024).

---

Published as a conference paper at ICLR 2025
Table 5: Comparison with other visual generation methods on GenAl-Bench (Lin et al., 2024). The results show that our method outperforms previous autoregressive visual generation methods. For advanced prompts that require better text following ability to generate, our method can have relatively small performance gap with diffusion-based methods, even with much less training data.
Relation
Method
Type
# #Training Images
Attributef
Scenet
Overall
Spatial
Action
Part
SD v2.1 SD-XL
Diffusion
2000M 2000M
0.80
0.79
0.76
0.77
0.80
0.78
Diffusion
0.84
0.84
0.82
0.83
0.89
0.83
Midjourney
Diffusion
0.88
0.91
0.87
3
Diffusion
0.91
0.90
0.92
0.89
0.91
0,90
LWM
Autoregressive
0.63
0.62
0.65
0.63
0.70
0.63
Show-o
Autoregressive
36M
0.72
0.72
0.70
0.70
0.75
0,70
Ours (256)
Autoregressive
I5M
0.78
0.78
0.77
0.78
0.79
0.76
Ours (384)
Autoregressive
15M
0.75
0.76
0.75
0.73
0.75
0.73
(a) VQAScores on basic prompts of GenAl-Bence
Logicalt
Method
Type
#Trainiig Images
Countt
Differt
Compare"
Overall
Negate
Universal
SD v2.1
Diffusion
2000M 2000M
0.68
0.70
0.68
0.54
0.64
0.62
SD-XLL
Diffusion
0.73
0.69
Midjoumey v6
Diffusion
0.78
0.78
0.79
0.50
0.76
0.69
DALL-E3
Diffusion
0.82
0.78
0.82
0.48
0,80
0.70
LWM
Autoregressive
0.59
0.58
0.54
Show-o
Autoregressive
36M
0.70
0.62
0.71
0.51
0.65
0.60
Ours (256)
Autoregressive
15M
0.70
0.71
0.74
0.53
0.66
0.64
Ours (384)
Autoregressive
15M
0.68
0.67
0.71
0.51
0.64
0.61
(b) VQAScores on advanced prompts of GenAI-Benchl
2023) including multi-image understanding, in-context learning, as shown in Figure 5 and Figure 6. More visualizations can be found in the Appendix B.2 and B.3.
Visual Generation. We present some examples of the visual generation results in Figure 7. Our model can be employed in both image generation and video generation, even trained with relatively small data corpus. In the given examples, our method can generate nice-looking images and continuous videos adhering to the user's input. More visualizations can found in the appendix B.4.
ABLATION STUDY
5.1
IMPACT OF CONTRASTIVE LOSS TO VISUAL UNDERSTANDING
We include contrastive loss in vision tower training, which endows it with the text alignment ability. During our multi-modal training, such text alignment ability crucial in enhancing modality fusion and performance downstream visual language tasks. We validate the importance of this alignment by training the vision tower with and without the contrastive loss, evaluating its impact on visual language understanding performance. For this ablation study, we randomly sample 25M data from
Prompt: <videc>\n
Elaborate visual namative elements the video detail.
VILA-U(Ours): The video feutures person wearing blue t-shin, khaki pants, and cup, executing series of breukdancing a concrete road. The background includes a clear sky, a bridge, and train passing by: The dancer's movements are dynamic, with spins, flips, and handstands. The lighting bright, suggesting daytime, and the shadows indicate and the setting appears to be quiet road with minimal traffie.
high. dancer's attire casual.
Figure 3: VILA-U can correctly caption videos and cover all the details, thanks to the text alignment of our vision encoder.
8

---

Published as a conference paper at ICLR 2025
Input image
Question
Output
ecamplas fanget Image
Ouiqui
is there water visible?
Yes
flying sky?
polar bears
postas
How mutorcycles picture?
How many people are on each motornyele?
Figure VILA-U has good visual question answering capability. The images and questions are from the test split of VQAv2 dataset.
Figure 5: VILA-U has good in-context learning capability. We feed two image-text pairs and third image as the context to prompt the VLM.
Prompt: Image I: <imgl> Vn Image 2: <img2>\n
What is the common thing and the difference between image I and image 27 VILA-U(Ours):
common thing both images presence of and panda bear. However, the difference between the panda bear image is lying down on rock, while the panda hear in image lying down on log.
that
Prompt: Image I: <<igg11 >> An Image 2: <img2>\n
What is the common thing the difference between image image VILA-U(Ours): The common thing in both inages the presence of trees, but the difference hetween the two images is the angle of the sunlight. In image I, the sun is shining through the trees, creating a bright and lively atmosphere. contrast, the darker and more subdued forest
visible image which shows
Figure 6: VILA-U can correctly reason over multiple images.
COYO-700M to train the vision tower. For multi-modal training, we use ShareGPT4V and MMC4 without text-image and text-video data. The results of the first two lines in Table 7 demonstrate the
crucial role of text alignment in achieving strong visual language understanding performance. Scaling the dataset size from 25M 700M further enhances performance, highlighting the importance of learning text alignment on a large-scale dataset.
Table 7: Impact of contrastive loss to visual understanding.
Pretrained Weights Data
Loss Type
Top-1 Accuracy
VQAv2
POPE
MME
SEED
MM-Vet
SigLIP-Large
25M
Recon.
$7.7
75.1
937.7
38.7
15.3
LigLiP-Large
Recon. + Contra.
62.9
83.7
20.8
SigLIP-Large
700M
Recon. Contra.
73.3
75.3
83,9
1336.2
56.3
27.7
5.2 IMPACT OF CONTRASTIVE Loss TO VISUAL GENERATION
We conduct two experiments demonstrate the influence of contrastive loss to generation perfor- mance. For efficiency, we conduct only text-to-image pretraining and utilize Sheared-LLaMa. 1.33 (Xia et al., 2023) instead LLaMA-2-11 2- as the LLM. the first experiment, we use the RQ VAE as the vision tower, which has an rFID of 1.30. In the second experiment, we employ our unified vision tower. Results are shown in Table 8. Our Unified Vision Tower yielded slightly worse FID results than the RQ- VAE on MJHQ-30K, possibly due to its inferior rFID resulting from the contrastive loss.
Table Impact of CFG.
Table 8: Impact of contrastive loss to visual generation.
CFG Value
FID
Vision Tower
LLM
Resolution
rFID
1.0
14.1
RQ- VQE (Lee el al., 2022)
Cheared-LLaMAs
256 x 256
1.30
12.0
2.0
13.0
Ours
256 256
13.2
3.0
12.8
5.0
13.2

---

Published as a conference paper at ICLR 2025
Happy dreamy monater arting tee branoh, colortu gittering partisles, Torest bookground, detaled feathers:
cute kitten aliding down slide, happy excited Vibrant colors, water splashing the lens:
A black dog
Crocodila sweaten
a
handsome 24 years old boy the sky color background with anime
realicticc landecape Lights dancing snavy
deep cleating
reflecting galaay-
exploding sky:
Figure 7: VILA-U can generate high-quality images and videos given input.
5.3 IMPACT OF CLASSIFIER-FREE GUIDANCE
We adopt classifier-free guidance during the visual content generation. We investigate the impact of the CFG value on our 256-resolution model. Results presented in Table 9 indicate that a CFG value of 3.0 yields the best FID score.
CONCLUSION AND LIMITATION
We present VILA-U, a novel and unified visual language model that integrates video, image and language understanding and generation tasks into one autoregressive next-token prediction framework. Our method is not only more concise than most VLMs that leverage additional components like diffusion models for unifying visual generation and understanding, but also demonstrates that autoregressive methods can achieve comparable performance to state-of-the-art VLMs. We believe VILAA-U can serve as a general- purpose framework for diverse visual language tasks.
As demonstrated in Section 5.2, the introduction of contrastive loss impacts the reconstruction ability of the vision tower. Balancing these two capabilities within the unified vision tower presents an interesting and complex challenge that requires further exploration. Additionally, currently do not observe significant synergy or mutual enhancement between understanding and generation tasks. In the future, aim to investigate and explore more effective methods enable these tasks to complement and reinforce each other, thereby fully realizing the untapped potential of a unified visual language model.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
lcan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. htttp:///ithub... wwtp:/cccoooaakkoae
coyo-daa daoo-adtttee.... 2022.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th anmual meeting of the association for computational linguistics: human language technologies, pp. 190-200, 2011.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311. 127993, 2023.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, et al. Scaling instruction-finetueed language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning, Advances in Neural Information Processing Systems, 36, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009a.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009b.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Yuying Ge. Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting seed of vision in large language model. arXiv preprint arXiv:2307.08441, 2023a.
Yuying Ge, Sijie Zhao, Ziyun Zeng. Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama see and draw with seed tokenizer. In The Twelfth International Conference on Learning Representations, 2023b.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi guatulgranullaiit comprehension and generation. arXiv preprint 4rXiv:2404.14330 2024.
11

---

Published as a conference paper at ICLR 2025
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint 2rXiv:2055.15800 2022.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Champaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11807-21818, 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709 2019.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv:2401.0444 .04088, 2024.
Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, CHEN Bin, Chengru Song, Di ZHANG, Wenwu Ou, et al. Unified language-vision pretraining in IIm with dynamic discrete visual tokenization. In The Twelfth International Conference on Learning Representations, 2023.
Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et Video-lavit: Unified Lide--langaae pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.03161, 2024.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11523-11532 2022.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint 24Xiv:2402.172255 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19740-19742. PMLR, 2023b.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403.1 18814, 2023c.
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large lisionllanguaee models. arXiv preprint arXhv:2305.10355 10355, 2023d.
Yuncheng Li. Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4641-4650, 2016.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
12

---

Published as a conference paper at ICLR 2025
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprins arXiv:2404.01291, 2024.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-lengtee video and language with ringattention. arXiv preprint arXiv:2402.08226. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint wrXiv:2312.17172 2023.
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232 2024.
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint ar arXiv:2407.0237// 2024.
OpenAl. Chatgpt. https: pestps://opente /tpp////pp/aaaa//////// wwt.ccocccccccccccccecoocoeococt cttpggg//atttttt..... 2023.
2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01192, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022b.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng. Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation, 2024. URL https: //arxiv. wwtpiii.orgg//... abs 2406. 06525.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjin Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. arXiv preprint 1rXiv:23/2.13286 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222. 2023b.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024.
13

---

Published as a conference paper at ICLR 2025
Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleaved: Interleaved image-text generative modeling via multi-moddl feature synchronizer. arXiv preprint arXiv:2401.10208, 2024a.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: /
Scalable image generation via next-scale prediction, 2024b. URL 2404. 02905.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv:2302. 13971, 2023a.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 1 19971, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: sttps://proceedingg nwurips. (tee3335ee243554755 f pdf.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694 2023.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408. 06072, 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 2rXiv:2110.04622 10.04627 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.
Coca: Contrastive captioners are image-text foundation models, 2022.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.025911 2(3), 2023a.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.024: 2023b.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 11975-1 1986, 2023.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint 200. 2024.
14

---

Published as a conference paper at ICLR 2025
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tranyi Li. and Yang You. Open-sora: Democratizing efficient vidco production for all, march 2024.
URL https://github. com/hppcateech/Open-SSttt 1(3):4.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A DIFFERENCE WITH RELATED WORKS
Prior to VILA-U, unified visual language models were dominated by two mainstream approaches:
(I) Represented by LWM, CM3Leon and Show-o which utilizes VQGAN- tokenizer to convert visual inputs into discrete tokens. However, as these tokenizers are trained solely with a reconstruction objective, the resulting tokens lack rich semantic information. This limitation leads to poor performance on multimodal understanding tasks. But can easily support autoregressive visual generation and the generated visual tokens be seamlessly decoded into visual outputs using the lightweight decoder of VQGAN.
(2) Represented by AnyGPT SEED-LLaMa and LaViT, which utilizes codebook to quantize features produced by a pre-trained ViT model like CLIP. Since CLIP features encode rich semantic information, these approaches generally achieve significantly better performance understanding tasks compared to VQGAN- based tokenizers. However, these tokenizers lack decoding capability, requiring an external visual generation model, such as a diffusion model, use the generated visual tokens as conditions for producing visual outputs.
Compared to these two mainstream approaches, VILA-U introduces a solution that addresses the limitations of both. We design unified vision tower that extracts features with rich semantic information, similar to CLIP, while also supporting image reconstruction capabilities akin to VQGAN, This is achieved by incorporating both reconstruction loss and contrastive loss into the autoencoder training process, along with utilizing residual quantization to enhance the representation capability of the visual features. Building this foundation, develop single end-to-end autoregressive framework that eliminates the need for external visual generation models required by approach 2 and significantly outperforms the understanding results of methods in approach 1.
B QUALITATIVE RESULTS
B. RECONSTRUCTION
Original Image
Reconstruction [256 resolution)
Reconstruction Image (384 resolution)
Figure Visualization of the reconstruction results from text-aligned discrete visual tokens.
We present qualitative reconstruction results in Figure 8 for our 256 / 384 resolution vision tower. These vision towers effectively reconstruct images in detail using text-aligned discrete visual tokens.
16

---

Published as a conference paper at ICLR 2025
B.2
VISUAL UNDERSTANDING
Input image
Question
Output
What animals are shown here?
Zebrus
What color they?
Black and white
What is the color of right board?
Bluc
How many men are in the picture?
cooking bread?
Yes
What this?
Kitchen
Did the man hit the but?
No
Is the man's legs together apart?
Apart
Figure 9: Image understanding results. Examples taken from the test split of VQAv2 dataset.
Prompt: decsir drops rimes?
VILA-U(Ours): drops down the window each time they hit the window, of drops
Prompt: cor deideo>>n many guys laughing
VOusA-(OOur): 2 guys are Imughing.
Prompt: evideo>in where is the man running and then be fall
VIurs- The man running on a rreadmill a gym, and then he falls off the trendmill.
Prompt: deideozAn many boy holds candy bar and opens mouth VOLAA-((Our)::
The video shows one boy holding the candy har and opening his mouth.
Figure 10: Video understanding results. Examples are taken from the test split of TGIF dataset.
We provide more image understanding and video understanding examples in Figure and Figure 10.
VILA-U successfully answers the questions accurately.
17

---

Published as a conference paper at ICLR 2025
B.3 IN-CONTEXT LEARNING EXAMPLES
Input few images target image
Output
UNDERGROUND
Underground
CONGRESS
Congress
Souiomee
2+1-3
5
5-6-11
3x6
3x6-18
Romanticiam
Surrealism
Impressioniem
Google
The
The iPhone and Mac.
The company for graphice processing unit.
search engine.
nVIDIA
dogs
giratfas
Les sanglots longs l'automnne blessent langueur monotone.
Pour sont cas serpents sifflent sur
Los flomants se sont formés couple, les cráatures de la tête, et leur tôte est touchée.
In-context learning examples. We try all in-context learning examples in Lin et al. (2023). The results demonstrate that VILA-U has inherited good in-context learning capabilties.
Figure 11:
We provide more qualitative results demonstrate in-context learning capabilities of VILA-U in Figure 11. VILA-U exhibits good in-context learning capabilties.
B.4 VISUAL GENERATION
snowy mountai::
painting of garden where every bloom, ahowcosing u
slephant warking under
groy-
Knolling a drawing
looks up staryy dky. othereal:
nugged
Figure 12: Image generation results. VILA-U can generate high-quality images given text input.
18

---

Published as a conference paper at ICLR 2025
Burning carpfire
Sunlight shining through Leaves:
Figure 13: Video generation results. VILA-U can generate high-quality videos given text input.
We provide more image generation and video generation examples in Figure 12 and Figure 13. VILA-U can generate high-quality images and videos given text input.
C FAILED TRAINING RECIPES.
experiment with numerous training recipes and find none to be as effective as our final approach. We list four alternative recipes and discuss their shortcomings compared to final recipe: 1) Load pre-trate tre-trained CLIP weights into the text encoder only; 2) Load pre-trained RQ- VAE weights for the vision encoder and decoder while training other parts from scratch; 3) Freeze the vision encoder; 4) Make the text encoder trainable.
Recipes 1) and 2) fail due the lack of pre-trained CLIP weights for the vision encoder. Training CLIP model from scratch typically requires numerous GPU days with large global batch size (e.g.. 32k). However, VQ-based reconstruction training necessitates relatively small global batch size (e.g., 512) for steady improvement. With such small batch size, training text-aligned vision tower from scratch would be prohibitively time-consuming and resource-intensse
Recipe 3) fails because freezing the vision encoder prevents it from learning the low-level features essential for reconstruction. In this case, the burden of reconstruction falls entirely on the vision decoder, but it is impossible to reconstruct images well using only semantic features.
Recipe 4) fails because the quantized features are chaotic during the initial training steps, and the contrastive loss disrupts the text encoder weights, slowing down the entire training process.
In contrast, our final training recipe leverages pre-trained CLIP weights for the vision encoder, enabling it to maintain learned semantic features rather than grasping them from scratch. This allows us to train with a small batch size while keeping the vision encoder trainable, facilitating the learning of low-level features for reconstruction during training.
19


Reviews:
[{"review_id": "cGas6kZlaM", "reviewer": "Reviewer_7Smq", "summary": "- The paper presents VILA-U, a unified model for language, image and video understanding + generation\n- The model is trained with an autoregressive next token prediction loss for all tasks\n- The paper explores vision encoder choices to ensure understanding and generation performance", "questions": "- Please share missing details as mentioned in the weaknesses\n- What are the number of image and video tokens going into the LLM? How many tokens are processed by the RQ-transformer and what is its size (the RQ-VAE paper has multiple different settings)?\n- It would be interesting to see if the vision tower training results hold for a general VAE setup instead of an RQ-VAE since that would make the results even more broadly applicable", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper's most interesting contribution is the unified vision tower exploration to unify generation and understanding and the appropriate ways to train such an encoder\n- The approach is quite straightforward and the application of RQ-VAE allows for token efficiency while preserving more information\n- VILA-U is close to SOTA on visual understanding tasks (image and video) with comparable models\n- The model also fares well on image generation tasks and comes close to diffusion models", "weaknesses": "- The method chooses RQ-VAE for efficiency, but there isn't a discussion / results around this. How would the results look if the vision tower didn't use RQ-VAE? How important is the RQ-VAE?\n- The generated images are relatively low-resolution (256 or 384px), especially since the RQ-VAE allows for increased efficiency in tokens\n- The paper doesn't really discuss video implementation details. Video understanding and generation have a mismatch in FPS / durations they usually support, what does VILA-U support? There isn't a discussion around this.\n- The paper claims to support video generation, but there are no quantitative results around this. The two qualitative examples are also very simplistic in Figure 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ERPUllpxWY", "reviewer": "Reviewer_ma7u", "summary": "The paper presents VILA-U, a unified foundation model for visual understanding and generation that integrates image and language processing into a single autoregressive next-token prediction framework. Unlike traditional visual language models that rely on separate modules or diffusion models for generation, VILA-U employs a unified vision tower to discretize visual inputs, aligning them with textual tokens through contrastive learning. From the experiments, the authors show that VILA-U can achieve state-of-the-art performance in both image generation and comprehension.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. VILA-U introduces a unified framework that handles both visual understanding and generation in a single autoregressive next-token prediction model. \n\n2. The model leverages a unified vision tower that uses contrastive learning to align discrete visual tokens with textual inputs, which enhances the model's visual perception and text-visual alignment capabilities.\n\n3. The experiments indicate the state-of-the-art performance of VILA-U in both image generation and understanding.", "weaknesses": "1. Missing the clarification between VILA-U and other tokenization-based multimodal models, like AnyGPT [1] and SEED-LLaMa [2]. Those models also used visual tokenizers to discrete the images and trained with causal language modeling loss. I noticed the authors cite the SEED-LLaMa in the line 102, but the claim of “In this work, we design our framework based on the autoregressive next-token prediction method for visual generation and make our VLM learn to generate visual content effectively.” does not the main difference between VILA-U and SEED-LLaMa.\n\n2. One of the claimed contributions of this paper is about proposing the training strategy for the unified foundation vision tower. However, the training strategy seems similar to SEED [3], which also used contrastive loss between image embeddings and text embeddings. Can authors clarify the difference between the unified foundation vision tower and SEED?\n\n3. Comparisons with other tokenization-based multimodal models [1,2] and Emu2 [4] are missing.\n\n4. The limitation section, which is required, is missing.\n\n[1] Zhan, Jun, et al. \"Anygpt: Unified multimodal llm with discrete sequence modeling.\" arXiv preprint arXiv:2402.12226 (2024).\n\n[2] Ge, Yuying, et al. \"Making llama see and draw with seed tokenizer.\" arXiv preprint arXiv:2310.01218 (2023).\n\n[3] Ge, Yuying, et al. \"Planting a seed of vision in large language model.\" arXiv preprint arXiv:2307.08041 (2023).\n\n[4] Sun, Quan, et al. \"Generative multimodal models are in-context learners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L9rXkxDShj", "reviewer": "Reviewer_X72f", "summary": "The paper, VILA-U presents a unified framework of autoregressive multimodal generation and understanding. It achieves this by first training a vision encoder (discretized via RQ codebook) for text-conditioned image tokens (initialized from CLIP) and then training image+text data using autoregressive modeling. It presents a complete training recipe for creating autoregressive multimodal models, and the resulting model is benchmarked against a wide range of existing models across tasks (generation and understanding)", "questions": "My biggest suggestion/question is related to the number 1 weakness described above. If the author could highlight the main contribution of the work that would make its positioning much easier. One positioning that was left out in the weakness section above is to position the work as the \"first\" in some regards. However, while autoregressive modeling of text + language is a burgeoning field, VILA-U is not the first model that performs autoregressive modeling of multiple modalities.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The unification of multiple modalities in the same architecture (with the same training objective) is a very important topic. The paper is a valuable contribution to this overall research program. In the current work, the choice of quantized image tokens for image representation makes the autoregressive modeling task more natural as the image modality is tokenized into discrete tokens much like language. This helps minimizes the amount of code development required for adapting existing LLM code bases to their multimodal counterparts.\n2. The paper performed fairly complete evaluations (image-text, video-text, text-image, ) and ablation studies that include model backbone and training objective.", "weaknesses": "1. It is not clear to me how to position the work in its novelty or effectiveness and this may be addressable with some rewriting. I see 3 potential angles\n    1. Training effectiveness by leveraging pretrained networks. The authors motivates the work by emphasizing that existing methods that attempt to unify multimodal generation and understanding either require significant architectural modifications to their uni-modal counterparts, or training from scratch. However, this comparison seems not to play a central role in the subsequent discussions. If the effectiveness of the proposed method is reflected in ease of training, then readers would expect to see comparison of training time/compute for comparable performances. \n    2. Effective token representation of image modality as discrete tokens: VILA-U differs from prior work in its adoption of RQ-VAE embedding for images. However, if this is the main innovation, the choice of RQ, its superiority over alternative methods, the important of discontinuous embedding of images (as compared to, for example, continuous embedding as in LaViT) will need to be elevated.\n    3. State-of-the-art performance: If the main contribution is instead just the shear effectiveness of the method. Then it should demonstrate this quantitative in the paper. Unfortunately, the comparison tables doesn’t seem to suggest that the VILA-U model is the state-of-the-art in most benchmarks. Perhaps it achieves Pareto frontier between understanding and generation tasks? Or outperforms other models for the same training compute/time? Either way I’m not clear what the main advantage of the current work is over others. \n2. The discussion around training recipe is very important and useful for practitioners. However, it lacks both quantitative and qualitative (with examples) comparisons of the different training recipes. With the conclusion seems to be use an aligned CLIP model for image encoder initialization, which doesn’t seem to be a novel finding. I would recommend either supporting the discussion with more evaluation (quantitive or qualitative, ideally both) or moving the discussion to the appendix.\n3. The paper suffers from unsubstantiated claims ( neither references nor experimental support). I've highlighted a few statements that are very important for the message in the paper below:\n    - \"replacing continuous tokens with VQ tokens in VLMs usually results in a severe performance drop\"\n    - \"A straightforward combination of contrastive and reconstruction loss cannot converge\"\n    - \"both the rFID and Top-1 accuracy of the vision tower only serves as a medium indicator instead of directly linearly correlated to the final performance of our whole multi-modal framework.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uMPyFz62XX", "reviewer": "Reviewer_n4tc", "summary": "Summary:\n\nVILA-U is a foundation model that unifies video, image, and language understanding and generation. Unlike traditional models that use separate components for different tasks, VILA-U simplifies this by employing a single autoregressive framework. This reduces misalignment and maintains near state-of-the-art performance in both understanding and generating visual language content. Key factors for its success include a unified vision tower that aligns visual and textual inputs, enhancing perception, and the ability to achieve high-quality image generation similar to diffusion models.\n\nContributions:\n\n1.  VILA-U strives for an end-to-end autoregressive model that handles both visual and textual inputs through a unified next-token prediction approach. This approach eliminates the need for external components like diffusion models, simplifying the infrastructure.\n2.  VILA-U is tested across a range of tasks, including image-language and video-language understanding, as well as image and video generation. It demonstrates notable improvements, particularly in narrowing the gap between autoregressive and continuous-token models in visual understanding, while also offering robust visual generation capabilities.", "questions": "1. The  solid experimental results of VILA-U have largely reignited my confidence in the autoregressive image-text unified modeling direction. However, why is there no comparison with other text-image unified modeling models such as \\textbf{MM-Interleaved, SEED, and DEEM} on image understanding tasks? Ignoring the contributions of pioneers is not advisable.\n\n2. The video generation experiments are insufficient. Why not compare with methods like \\textbf{OpenSora} and \\textbf{CogVideoX} on \\textbf{VBench}?\n\n3. The article is unclear in its expression; are the visual tokens features directly discretized by the visual encoder, or are they encoded by a large language model? I suspect it is the former.\n\n4. VILA-U claims to have lower computational complexity and to avoid misalignment. While I recognize the importance of addressing misalignment, the claim of lower complexity requires experimental support. Specifically, compared to unified autoregressive image-text modeling models, using separate models like fine-tuning Stable Diffusion can also construct end-to-end autoregressive image-text modeling, which is more efficient in training and performs better. Moreover, utilizing existing mature acceleration schemes offers fast speeds. VILA-U should emphasize more on data cleansing quality and misalignment.\n\n5. Lastly, and most critically, I hypothesize that the structural improvements of the model provide minimal benefits compared to previous autoregressive unified models, with the majority of improvements stemming from the engineered data cleansing. For instance, MMC4-Core contains 22.4M data while MMC4 has 375M, yet some research indicates that training with these two datasets yields similar outcomes. Large-scale datasets like MMC4 are of very low quality. However, using just 6M of data to achieve excellent results suggests that your data is meticulously filtered, yet the paper lacks any detail on the core contributions of data construction. Conducting experiments on the same data with other model structures like \\textbf{DreamLLM} is necessary to demonstrate the efficiency of \\textbf{VILA-U}. \n\nI will improve my rating score if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of VILA-U is very straightforward, and the experiments are solid. It significantly enhances the capabilities of end-to-end autoregressive multimodal models in visual-language tasks, bridging the gap between autoregressive multimodal models and the LLAVA series, while also excelling in image generation.\n\n2. The structure of the VILA-U paper is simple and easy to read, and the model implementation is very easy to follow.", "weaknesses": "1.Regarding the issue of missing in context learning assessments, VILA-U has undergone extensive training on image-text sequences and can accept any interleaved layouts of images and text. Therefore, it should possess excellent contextual learning abilities. This work could be enhanced by conducting tests on its ICT capabilities.\n\n2.The description of the data curation process is not sufficiently clear, making it uncertain whether the data was meticulously selected or randomly chosen. If it is the former, I suspect that most of the improvements stem from high-quality data engineering rather than advancements in model architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)