X-VILA: Cross-Modality Alignment for Large Language Models

## Abstract

Abstract We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities.
By aligning modality-specific encoders with LLM inputs and diffusion decoders with LLM outputs, X-VILA achieves cross-modality understanding, reasoning, and generation.
To facilitate this cross-modality alignment, we curate an effective interleaved any-to-any modality instruction-following dataset.
Furthermore, we identify a significant problem with the current cross-modality alignment method, which results in visual information loss.
To address the issue, we propose a visual alignment mechanism with a visual embedding highway module.
We then introduce a resource-efficient recipe for training X-VILA, that exhibits proficiency in any-to-any modality conversation, surpassing previous approaches by large margins. X-VILA also showcases emergent properties across modalities even in the absence of similar training data.
The project will be made open-source.

## 1 Introduction

Large language models (LLMs) provide an emerging foundation for enhancing various deep learning tasks beyond the realm of natural language processing. As an example, research community has been quickly extending the fast progress of LLMs [, , , , , , , , , , , , ] towards the computer vision (CV)
domain [, , , , , , , , ].
The introduction of LLMs in CV tasks enables vision models to perform many zero/few-shot and in-context learning tasks that are “promptable” through user questions, potentially empowering reasoning capabilities for the first time.
Despite remarkable progress, cross-modality alignment is still a challenging task as the joint training stage for cross-modality learning requires carefully designed feedback signal [, ] to guide the connected foundation models [, , ], backed by cross-modality datasets at scale [, , ].
Hence, the majority of existing studies revolve around a solitary input modality linked to LLMs, with the output being solely text. For example, Flamingo [], LLaVA [], and VILA [] delve into image input, while Video-LLaMA [] and LITA [] specifically concentrates on video input.
Exploring the integration of various modalities into a cohesive framework is a crucial yet relatively unexplored research challenge [, , ] in the domain of multi-modality LLMs, yet observed practical in proprietary GPT-4o [].

This study focuses on the development of a systematic approach to integrate multiple modalities, such as video, image, and audio, into an LLM at both the input and output stages. Our objective is to facilitate cross-modal conversations in an any-to-any modality (or “X-to-X”) LLM, allowing for generation in different modalities.
To accomplish the ambitious objective, we present a two-phase alignment mechanism:
(i) Textual alignment.
We align input and output representations of different modalities to the textual embedding space of the LLM [].
Specifically, in regard to the input of LLM, we use a unified embedding space that allows for the sharing of features extracted from encoders across diverse modalities. As for the output of LLM, we employ fine-tunable modality-specific diffusion models to convert the generated outputs of the LLM into content that aligns with the respective modalities.
(ii) Visual alignment.
We observe that the previous textual alignment alone fails to preserve visual features adequately in vision-to-vision generation tasks, such as image-to-video and video-to-image generation. This limitation can be attributed to the loss of information during the projection process from visual encoders to the LLM, as well as the LLM’s tendency to prioritize common concepts over specific visual details.
To address this issue, we propose a new module named Visual Embedding Highway (VEH). The VEH module facilitates the direct guidance of visual decoders by enabling visual features to bypass the LLM. By incorporating VEH, we have observed a notable enhancement in the correspondence of visual content between the input and output stages of our framework.

On the other hand, the scarcity of cross-modality instruction-following data poses a significant challenge in the development of any-to-any modality (or “X-to-X”) LLMs. This limitation severely restricts the progress in creating LLMs that can seamlessly handle multiple modalities in both input and output ends.
Existing datasets provide limited data, mostly in the form of X-to-text or text-to-X.
Therefore, we curate a new X-to-X dataset based on multi-modality data from WebVid [] and ActivityNet Captions [] to facilitate cross-modality interactions between text, audio, image, and video. Overall, we synthesize more than 1.5M multi-modality conversations, with each conversation containing at least one cross-modality question-and-answer pair.

To achieve the cross-modality input-output alignment of LLMs in our X-to-X LLM, we design three major training phases:
(i) A data-effective alignment phase that involves aligning the multi-modality encoders with the LLM inputs and the multi-modality decoders with the LLM outputs.
(ii) An interleaved multi-modality pre-training phase with interleaved instruction data across modalities for enhanced in-context learning performance.
(iii) An X-to-X cross-modality instruction tuning phase that includes a two-step alignment process: textual alignment and visual alignment mechanism.
Through our innovative approach to multi-modality alignment, we build a powerful X-to-X multi-modality LLM with the ability to comprehend and generate multi-modality content. We term our new model “X-VILA” for cross-modality understanding, reasoning, and generation in the domains of Video, Image, Language, and Audio.
For instance, as shown in Figure 1 and Figure 8, X-VILA demonstrates its capacity to recognize the subjects in the image, which results from our vision-language alignment training.
Then, it can retrieve its knowledge and make logical deductions to answer the user’s questions about the content in the image. Last but not least, it can generate aligned multi-modality output that matches the given context.

In summary, this work makes contributions in three aspects:

- •
A new family of any-to-any modality chat LLM that is able to conduct multi-modality conversations by understanding signals from different modalities and generating content in various formats, including video, audio, image, and text.
- •
A novel 2-step alignment mechanism that effectively aligns both semantic and visual details between the input and output spaces. This mechanism ensures a comprehensive and accurate correspondence between the input and output of our X-to-X LLM.
- •
The creation of a new X-to-X multi-modality instruction tuning dataset that is proven effective for cross-modality alignment. This dataset serves as a valuable resource for future research in the realm of multi-modality foundation models.

## 2 Methodology

Figure: Figure 2: X-VILA schematic diagram. X-VILA augments a pretrained LLM towards new modalities via (i) connecting pretrained encoders to LLM input textual embedding space and (ii) connecting pretrained diffusion decoders to the LLM output textual embedding space (Section 2.1). The system is jointly trained via a new cross-modality alignment procedure (Section A).
Refer to caption: x2.png

### 2.1 X-VILA Architecture

We consider four common modalities in this work: text, image, video, and audio.
The tenet of X-VILA is an alignment-based architecture to augment an LLM with the ability to “see/hear/read” multi-modality inputs and “draw/speak/write” multi-modality outputs, as shown in Figure 2.

Modality-specific encoders.
We adopt modality-specific encoders to handle inputs from different modalities. This strategy harvests the pre-trained understanding ability of the domain expert encoders and has been proven successful in many vision-language models [, , ]. To better align embeddings of different modalities, we use ImageBind encoders [], which unify features from different modalities, including image, video, and audio, into one feature space.
Technically, for each modality $m\in\{\text{`text', `image', `video', `audio'}\}$, we notate the encoders as $\text{{Enc}}_{m}$.
For text modality, the encoder is a text tokenizer [], while for other modalities they are ImageBind transformers [].
We then use modality-specific trainable linear layers, notated as $\textbf{P}^{\text{in}}_{m}$, to project $\text{{Enc}}_{m}$ output into embedding sequences $\mathbf{S}$ in the textual embedding space of the following LLM. We can formulate this process as:

$$ $\mathbf{S}^{\text{in}}=\{\textbf{P}^{\text{in}}_{m}(\textbf{Enc}_{m}(\mathbf{X }_{m}))\},$ (1) $$

where $\mathbf{X}_{m}$ is input from different modalities $m\in\{\text{`text', `image', `video', `audio'}\}$.

Large language model (LLM).
LLM serves as the “brain” of our framework. It processes information from the textual embedding space and predicts language outputs correspondingly. We adopt Vicuna-7B-1.5 [, ], which demonstrates state-of-the-art language understanding and generation ability. For easier understanding, we slightly abuse the annotation and write the autoregressive process of generating output embedding sequence $\mathbf{S}^{\text{out}}$ by the LLM as:

$$ $\mathbf{S}^{\text{out}}=\mathbf{LLM}(\mathbf{S}^{\text{in}}).$ (2) $$

Modality-specific decoders.
For generating multi-modality outputs other than text, we adopt the “modality-specific generation tokens” designed by [].
Other than common text tokens, there are three types of modality-specific generation tokens: image generation tokens {[IMGi], $i\in[1,N_{img}]\}$, video generation tokens {[VIDi], $i\in[1,N_{vid}]\}$, and audio generation tokens {[AUDi], $i\in[1,N_{aud}]\}$. $N_{img}$, $N_{vid}$, and $N_{aud}$ are the numbers of generation tokens for image, video, and audio, respectively.
These modality-specific generation tokens are added to the vocabulary of LLM. The LLM is trained to predict when to generate these modality-specific generation tokens, and these generation tokens will be translated for the synthesis of image, video, or audio, via a set of modality-specific decoders (i.e., generation models).
Technically, we extract the subset of output embedding sequence $\mathbf{S}^{\text{out}}$ corresponding to the aforementioned generation tokens of modality $m$. We name this subset the generation embedding sequence $\mathbf{S}^{\text{gen}}_{m}$.
We use modality-specific transformer layers, denoted as output projection layers $\textbf{P}^{\text{out}}_{m}$, to project $\mathbf{S}^{\text{gen}}_{m}$ to the feature space of the original pre-trained text encoder of the modality-specific decoder. As the resulting embedding will be used to control the modality-specific decoder via cross-attention, we name the resulting embedding vector as “textual controller embedding” $\mathbf{E}^{\text{text}}_{m}$. Thus we have:

$$ $\mathbf{E}^{\text{text}}_{m}=\textbf{P}^{\text{out}}_{m}(\mathbf{S}^{\text{gen }}_{m}).$ (3) $$

[] freezes the decoder models and only supervises the $\mathbf{E}^{\text{text}}_{m}$ to be similar to the original text encoders of the diffusion models. This behavior largely limits the synergy between generation models and the other parts of the model, as the learning target is essentially to mimic the pre-trained text encoder of the diffusion models.
Differently, we include the modality-specific decoder models in fine-tuning to better align them with the LLM and other parts of the unified generative framework. The training details will be discussed in a later section.
Specifically, to achieve a better multi-modality generation ability, we employ state-of-the-art generation models trained on large-scale data as modality-specific decoders. We adopt VideoCrafter2 [] for video generation, Stable Diffusion 1.5 [] for image generation, and AudioLDM [] for audio generation.

Figure: Figure 3: Illustration of the proposed visual embedding highway in X-VILA. The visual highway embedding $\mathbf{E}^{\text{vis}}$ is obtained from the visual encoder. The design incorporates a visual controller module responsible for processing $\mathbf{E}^{\text{vis}}$ and generating control signals. These signals are then incorporated into various layers of the U-Net in visual decoders. $\mathbf{E}^{\text{text}}_{m}$ is “textual controller embedding”, which is the subset of output embedding sequence $\mathbf{S}^{\text{out}}$ corresponding to the generation tokens of modality $m$. $z(t)$ is the latent at reverse step $t$. $\epsilon^{p}$ is the predicted noise by U-Net.
Refer to caption: x3.png

Visual embedding highway.
The weakness of the previously introduced text-space-based alignment is the inadequate visual features available at the output end, as can be seen in examples in Figure 4.
Intuitively, this stems from the one-to-many correspondence between text and visual semantic spaces, *e.g*let@tokeneonedot, “a dress” may relate to images varying in colors and styles.

To address this issue, we propose a visual embedding highway that bridges the visual encoders and decoders, built to alleviate the information loss when projecting high-dimensional visual content to the textual embedding space.
Specifically, we obtain the layer-wise feature maps from the ImageBind visual encoder and add up these features as visual highway embedding $\mathbf{E}^{\text{vis}}$. $\mathbf{E}^{\text{vis}}$ has shape $H\times W\times C$, where $H$ and $W$ are height and width of the feature maps, $C$ is the embedding vector.
To control the decoder using $\mathbf{E}^{\text{vis}}$, we design a light-weight visual controller (VisCtrl) module based on the philosophy of [, ] to process $\mathbf{E}^{\text{vis}}$.
The controller module comprises 4 stages, where each stage consists of two residual convolutional blocks. These blocks have cascading spatial dimensions that match the resolution settings in the U-Net encoder [] of image/video decoders.
In each stage, there is an additional convolutional block initialized with zero weights.
This block generates output control signals for the stage, which are initially zero at the start of the training.
These control signals are added to different stages of the U-Net, as shown in Figure 3. Inspired by [], we employ a conditioning rate $\alpha\in[0,1]$ to regulate the proportion of steps conditioned by visual features.
Therefore, the noise prediction process in each reverse step $t$ in the visual decoders can be written as:

$$ $\epsilon^{p}=\begin{cases}\text{U-Net}_{m}(z(t),\text{VisCtrl}_{m}(\mathbf{E}^ {\text{vis}}),\mathbf{E}^{\text{text}}_{m})&\text{if }t<T\times\alpha\\ \text{U-Net}_{m}(z(t),\text{Null},\mathbf{E}^{\text{text}}_{m})&\text{if }t \geq T\times\alpha\end{cases},m\in\text{\{`image', `video'\}}.$ (4) $$

where $\epsilon^{p}$ is the predicted noise given input latent $z(t)$, $T$ is the number of diffusion steps, $\text{U-Net}_{m}$ is the U-Net of the diffusion decoder for modality $m$, and $\text{VisCtrl}_{m}$ is the visual control module for modality $m$.“Null” means no VEH feature is passed to the U-Net at the corresponding timestep.
During instruction tuning process on X-to-X datasets, both the U-Net and the controller modules are fine-tuned together. This manner ensures a better synergy between decoders and the LLM.

The experimental results introduced in the later sections show that the proposed visual embedding highway can significantly increase the consistency between the generation results and the visual context of our multi-modality unified generation model.

### 2.2 X-VILA Training

The training process of X-VILA is divided into three phases, namely (i) encoder-LLM-Decoder alignment training, (ii) interleaved data pre-training, and (iii) X-to-X cross-modality instruction fine-tuning.
We describe the details of X-VILA training in Appendix A due to space limit.

## 3 Experiments

### 3.1 Datasets and Evaluation

Setup. In this work, we utilize different datasets for different training phases. For the first encoder-LLM-decoder alignment training, the X-text pairs from LLaVA-pretrain [], cc3m [], WebVid [], AudioCaps [], and WavCaps [] are utilized.
During the interleaved data pre-training phase, we construct interleaved multi-modality corpus from MMC4 [] and ActivityNet Captions [].

Figure: Figure 4: Effectiveness of the proposed visual embedding highway network. Given the visual reference image/video, we prompt the model with “Please generate an image similar to the semantics in the input.” Compared to textual alignment only (TA), our visual embedding highway (VEH) helps preserve visual details from the visual inputs.
Refer to caption: x4.png

In the final X-to-X cross-modality instruction tuning, we create a new X-to-X dataset from WebVid [] and ActivityNet Captions. We synthesize conversation samples in 6 types based on the modalities in input and output ends: video-to-image, video-to-video, image-to-video, video-to-audio, audio-to-video, image&audio-to-video.
Statistically, for ActivityNet Captions, we make 10009 image-to-video, 10009 video-to-image, and 10009 video-to-video conversations.
For WebVid, we randomly select 500k training samples and build 499,915 image-to-video, 499,915 video-to-image, 499,915 video-to-video, 32,874 audio-to-video, 32,874 video-to-audio, and 32874 image+audio-to-video conversations. Each conversation contains more than one pair of cross-modality Q&A pairs. Some conversation examples are shown in Figure 9.
We blend our X-to-X dataset with SFT datasets from LLaVA [], VideoChat [], NextGPT-instructions [], and Alpaca [].

Figure: Figure 5: We observe emergent abilities of X-VILA without training on similar data: (i) Long-context cross-modality generation ability. Combine multiple inputs from different modalities and generate consistent content. (ii) New types of cross-modality ability. Conduct image-to-audio and audio-to-image generation tasks. Conversations are continuous left-to-right within each green box.
Refer to caption: x5.png

Evaluation. For benchmarking the X-to-X alignment ability of different models, we randomly sample a validation subset of WebVid and ActivityNet Captions to build the cross-modality conversations for evaluation. Specifically, for ActivityNet Captions we generate 100 video-to-image, 100 video-to-video, and 100 image-to-video conversations.
For WebVid, we curate 100 video-to-image, 100 image-to-video, 100 video-to-video, 62 audio-to-video, 62 image+Audio-to-video, and 62 audio-to-video conversations for evaluation.
In order to evaluate the similarity between ground-truth annotations and model predictions across different modalities, we introduce a metric called the “X-to-X Alignment Score (X^2A Score)”. To compute this score, we employ the ImageBind transformer [] to extract embedding vectors from the audio, video, and image predictions as well as the corresponding ground truths. We then calculate the cosine similarity scores between these vectors. The resulting scores are presented as percentages, ranging from 0 to 100. Finally, we average the scores across all validation samples to obtain the X^2A scores for each type of data.

Baseline methods.
We conduct a comparison between our model and Next-GPT [], a recently introduced instruction-following LLM designed for multi-modality understanding and generation. Their method is restricted to textual alignment exclusively.

**Table 1: X^2A scores on ActivityNet Captions. “w/ X2X text” denotes using our X-to-X dataset for textual alignment only. “VEH (img)” denotes using the proposed visual embedding highway (VEH) for image decoder, while “VEH (img+vid)” denotes using VEH for both image and video decoders. We observe that image generation task is significantly improved after using VEH (img), and the video generation tasks are boosted after using VEH on video decoder.**
| Method | VID2IMG | VID2VID | IMG2VID |
| --- | --- | --- | --- |
| Next-GPT [] | 27.85 | 10.47 | 13.08 |
| X-VILA w/ X2X text | 36.09 | 46.18 | 45.93 |
| X-VILA w/ X2X text + VEH (img) | 44.06 | 46.68 | 45.94 |
| X-VILA w/ X2X text + VEH (img+vid) | 43.95 | 49.76 | 48.81 |

**Table 2: X^2A scores on WebVid. “w/ X2X text” denotes using our X-to-X dataset for textual alignment only. “VEH (img)” denotes using the proposed visual embedding highway (VEH) for image decoder, while “VEH (img+vid)” denotes using VEH for both image and video decoders. The effectiveness of visual embedding highway is solid for image and video generation.**
| Method | VID2IMG | IMGAUD2VID | VID2AUD | IMG2VID | VID2VID | AUD2VID |
| --- | --- | --- | --- | --- | --- | --- |
| Next-GPT [] | 15.31 | 44.63 | 8.17 | 38.23 | 31.81 | 37.13 |
| X-VILA w/ X2X text | 53.82 | 49.54 | 22.79 | 42.94 | 44.42 | 42.23 |
| X-VILA w/ X2X text + VEH (img) | 67.40 | 48.64 | 23.53 | 42.66 | 43.04 | 42.04 |
| X-VILA w/ X2X text + VEH (img+vid) | 67.94 | 59.71 | 23.87 | 57.01 | 57.39 | 49.44 |

Figure: Figure 6: Visual comparison to the recent any-to-any modality LLMs including Next-GPT [], CoDi [], and GPT-4o [] on the cross-modality alignment task to generate a video similar to the input image context. X-VILA demonstrates good generation quality and better visual cross-modality consistency. GPT-4o is only able to generate images but not videos.
Refer to caption: x6.png

Figure: Figure 7: (left, middle) Study of using different conditioning rates in VEH (image). Higher conditioning rates brings generally better X-to-X alignment. (right) An in-depth comparison of varying design choices of X-VILA on cross-modality alignment tasks. We observe that both Text-Aligned Decoding and Text-Embed-Aligned Decoding fall short in effectively capturing semantic details from visual inputs. However, with the incorporation of our Visual Embedding Highway (VEH), we witness a substantial improvement in visual consistency.
Refer to caption: x7.png

### 3.2 Quantitative Analysis and Ablation Study

Effectiveness of Visual Embedding Highway. We compute the aforementioned X^2A scores of different models on the X-to-X alignment benchmarks of both ActivityNet Captions and WebVid datasets, and present the results in Table 1 and Table 2.
Specifically, we study the X^2A scores of Next-GPT and different versions of our X-VILA model. We investigate the performance of our model under different scenarios: (i) utilizing only textual alignment, (ii) incorporating visual alignment through the proposed visual embedding highway (VEH) on the image decoder, and (iii) extending VEH to both the image and video decoders.
Our findings indicate that even by utilizing textual alignment alone with our carefully curated X-to-X datasets, our model demonstrates a substantial performance advantage over Next-GPT.
Moreover, as we progressively introduce the visual embedding highway to the image and video decoders, we observe consistent and significant improvements in visual understanding and generation tasks.
In summary, our X-VILA demonstrates significantly stronger cross-modality understanding, reasoning, and generation ability on all types of conversation data.
These results suggest the effectiveness of our X-to-X alignment strategy and the proposed visual embedding highway design.
Notably, both Next-GPT and X-VILA are based on the ImageBind model, making it fair to use ImageBind scores for both models.

**Table 3: X-VILA demonstrates comparable performance to domain experts when evaluated on extra multi-modality benchmarks.**
| Method | VQAv2 | VisWiz | MMMU-val |
| --- | --- | --- | --- |
| BLIP-2 13B [] | 65.0 | 19.6 | - |
| InstructBLIP 13B [] | - | 33.4 | - |
| Qwen-VL-Chat 7B [] | 78.2 | 38.9 | 35.9 |
| LLaVA 1.5 7B [] | 78.5 | 50.0 | 36.4 |
| X-VILA 7B | 72.9 | 50.9 | 33.9 |

Influence of conditioning rates.
We present the X^2A scores plotted with varying conditioning rates $\alpha$ (Equation 4) in VEH (image), as depicted in Figure 7. Our observations indicate that an increase in $\alpha$, corresponding to more reverse steps exposed to VEH features during image sampling, leads to improved multi-modality alignment. This outcome aligns with our intuitive expectations.

Extra multi-modality benchmarks.
To further evaluate the multi-modality understanding capabilities of X-VILA, we perform zero-shot experiments on several multi-modality VQA benchmarks, including VQAv2 [], VisWiz [], and MMMU-val [].
The results in Table 3 indicate that X-VILA is competitive with the leading domain-expert VLMs, while possessing the X-to-X capability.

### 3.3 Qualitative Analysis and Ablation Study

Qualitative X-to-X alignment measurement.
We provide a qualitative comparison to the state-of-the-art any-to-any LLMs, namely Next-GPT [], CoDi [], and GPT-4o [] on visual cross-modality alignment tasks in Figure 6.
We assess their performance by supplying an image to the models and prompting “Please generate a video (or an image in the case of GPT-4o which cannot generate video) similar
to the semantics in the input.”
X-VILA demonstrates significant improvements in visual correspondence over previous methods, thanks to the integration of the Visual Embedding Highway (VEH) into output diffusion models.

Emergent X-to-X ability.
During our experiments, we observe highly promising emergent abilities displayed by X-VILA following its training on our X-to-X datasets. As depicted in Figure 5, we have identified two key capabilities that have surfaced:
(i) Long-context cross-modality generation. X-VILA exhibits an impressive capacity for comprehending and combining diverse concepts from multiple iterations of input. Consequently, it produces natural and coherent output, as suggested by the users.
(ii) Unseen cross-modality ability. Remarkably, X-VILA showcases the ability to perform image-to-audio and audio-to-image tasks without any explicit training on similar data. This newfound competence emerges organically through the model’s exposure to our comprehensive X-to-X dataset.
These remarkable emergent abilities underscore the efficacy of our meticulously curated X-to-X dataset. Not only does it enable the model to excel in the specified data types as suggested in Section 3.2, but it also facilitates generalization across a wide range of multi-modality interactions between users and the model.

More insights on varying design choices on decoder alignment.
We next present our findings when aligning LLM output end to the modality-specific decoders.
We study different ways to bridge LLM output and the diffusion models:
(i) “Text-Aligned Decoding”: LLM generates text description for the expected image/video/audio predictions and then feeds the text description into pre-trained image/video/audio decoders.
(ii) “Text-Embed-Aligned Decoding”: LLM generates modality-specific generation tokens and then we use the corresponding high-dimensional textual embeddings to control the modality-specific decoders (as described in Section 2.1).
(iii) “Text-Embed-Aligned Decoding with VEH”: Building upon method (ii), we introduce the Visual Embedding Highway (VEH) to align the visual feature between encoders and decoders.
We conduct experiments on video-to-image and image-to-video cross-modality alignment tasks, and show the results on the right side of Figure 7.

The findings suggest that conveying specific details such as visual style, object appearance, and precise human actions from the input to the output is challenging for Text-Aligned Decoding. This difficulty arises due to the low-dimensional nature of pure text descriptions, which limits the amount of information they can contain. On the other hand, Text-Embed-Aligned Decoding offers a significantly greater “bandwidth” in the textual embedding space between the LLM and modality-specific decoders.
Consequently, Text-Embed-Aligned Decoding is capable of generating more consistent outcomes. Nevertheless, Text-Embed-Aligned Decoding alone is still not good enough for capturing visual details, as a substantial amount of visual information is lost during the projection from encoders to the LLM. This is where our Visual Embedding Highway demonstrates its performance and aids X-VILA in attaining notably enhanced visual consistency.

Conversation examples.
To thoroughly investigate the performance of our any-to-any modality LLM, we conducted extensive testing on X-VILA examining many use cases. We present conversation examples of X-VILA across varying tasks in Figure 1 and Figure 8. It can be observed that X-VILA provides users with a comprehensive set of multi-modality responses leveraging the encoders for perception, LLM for understanding and reasoning, and decoders for multi-modality content generation.
As shown in Figure 14, X-VILA not only exhibits its understanding of the visual input, including the scene and objects, but also predicts the actions of the person depicted in the image. This capability is a result of training on our extensive X-to-X dataset. Based on the visual input, it generates outputs visually consistent with the input, e.g. the snow mountain and red ski suit are presented in the generation output correctly.

Figure: Figure 8: Examples of X-VILA performing a multi-turn any-to-any modality conversation. Prompts are given left to right in a multi-round manner. Best viewed in color.
Refer to caption: x10.png

## 4 Related Work

The era of Large Language Models (LLM) arguably started with the introduction of transformers [] and a series of works that scaled them. Particularly, OpenAI introduced the Generative Pre-trained Transformer (GPT) models [], [], from GPT-2 (1.5B parameters) to GPT-4 [] (1.76T), and showed that parameter scaling, together with more high-quality data, can generate coherent and contextually relevant text across various domains. BERT [] introduced a paradigm of bidirectional text processing enabling stronger context understanding and boosted question answering. T5 [] converted language problem into a text-to-text format advancing translation and summarizing. Transformer-XL [] demonstrated the capability of extending the context window allowing for a better understanding of longer text. The application era of LLM was kickstarted by ChatGPT [] which showcased the unprecedented ability of LLM chatbots.

Current Vision-Language Models (VLM) benefited from the development of ViT [] that offers a unified way for vision models to communicate with other transformers from different modalities. Rapid progress has been shown in three streams []:
(i) textually prompted models that accept image and text as input (CLIP [], Frozen [], BLIP [], PaLI [], LLaVa [], VILA [], miniGPT4 []);
(ii) visually prompted models (CLIPSeg [], SAM []); and (iii) multi-modal input-output models (Painter [], ImageBind [], Palm-E [], Video ChatGPT [], RegionGPT [], mPLUG-owl [], PandaGPT [], CoDi [], NextGPT [], Unified-IO [, ]). Among the first, Frozen [] demonstrated that VLM can be constructed by linear projection of ViT features into LLM and only tuning ViT on image-text captioning data. They are the first that discover the few-shot capabilities of VLM without instruction. Flamingo [] used cross-attention for vision language binding, and for a first time demonstrated surpassing state-of-the-art finetuned models for multiple tasks. PALI [] created a universal model that can do vision and language tasks separately, they scaled ViT to 4B and demonstrated the importance of adding language-only data to the pretraining stage. Overall, VLM follows the pipeline of taking a pretrained LLM; adding a pretrained vision encoder; learning feature alignment at scale via a projector or cross-attention; followed by instruct-tuning (InstructBLIP [], FLAN []).
In close relation to our research, Next-GPT introduces an LLM that possesses the capability to comprehend multi-modality inputs and generate corresponding multi-modality outputs through textual alignment, yet it cannot effectively handle visual details present in the input.

## 5 Conclusion

This paper presents X-VILA, an any-to-any modality LLM that is able to understand, infer, and generate multi-modality contents. This ability is achieved through any-to-any modality alignment, for which we curate a dataset for any-to-any modality instruction tuning.
We further identify a significant drawback in the previous textual alignment method that leads to the loss of crucial visual details. Accordingly, we propose an innovative visual alignment mechanism that incorporates a visual feature highway module. This solution helps preserve essential visual details from the input.
The experimental results, both quantitative and qualitative, indicate the effectiveness of our data and methodology.
X-VILA’s performance can be further enhanced across various VLM benchmarks.

## Appendix A X-VILA Training

The training process of X-VILA is divided into three phases, namely (i) encoder-LLM-Decoder alignment training, (ii) interleaved data pre-training, and (iii) X-to-X cross-modality instruction fine-tuning.

### A.1 Encoder-LLM-decoder alignment training phase.

As the first step, we align the output of modality-specific encoders and the input of modality-specific decoders to the textual embedding space of LLM, as detailed in [].
To achieve this goal, we only train the input projection layers, output projection layers, and the vocabulary embedding layer of LLM, while keeping all other parameters frozen.
We use corpus with “X”-text pairs to train the model, where “X” is one of the video, image, or audio modalities.

For this stage, we design two primary tasks to train the projection layers: X-to-text generation and text-to-X generation.
(a) X-to-text generation includes video, image, and audio captioning tasks. The model is supervised to generate text based on the multi-modality inputs. During this process,
the input projection layers are trained to align the output embedding of modality-specific encoders and the textual embedding space of pre-trained LLM.
(b) Text-to-X generation aims at aligning the output textual embedding space of LLM and the input end of modality-specific decoders. We use video, image, and audio generation tasks to train the model, where only the output projection layers are optimized.
As previously mentioned, the training objective here is pure textual alignment: minimizing the feature distance between the textual controller embedding $\mathbf{E}^{\text{text}}_{m}$ generated by the output projection layers and the embedding generated by the original pre-trained text encoder of diffusion model. This training strategy ensures that $\mathbf{E}^{\text{text}}_{m}$ shares a distribution similar to that of the pre-trained text encoder in the diffusion model. After training, $\mathbf{E}^{\text{text}}_{m}$ replaces the diffusion text encoder feature to control the U-Nets of the modality-specific decoders via cross-attention.

### A.2 Interleaved data pre-training phase.

Interleaved data training has been proven to be an effective strategy for vision-language models in alleviating the catastrophic forgetting issue after training on only visual-text pairs, and obtaining long-context understanding ability [, ].
Therefore, we introduce a dedicated phase for pre-training X-VILA using a multi-modality interleaved corpus. In addition to interleaved image-text pairs as in MMC4 [], we further construct a new dataset from ActivityNet Captions []. The main idea is to exploit the nature of video that contains sequential flow of text (*e.g*let@tokeneonedot, captions), audio, short video, and image. This enables us to put the images/videos and texts in an interleaved manner, and use the corpus to pre-train X-VILA.

Specifically, we construct interleaved multi-modality data sequences from each target video clip as:

$$ $\displaystyle\scriptsize\underbrace{\texttt{\{<img. 1>, <aud. 1>, <vid. 1>, < txt 1>\}}}_{\textrm{sampled from video chunk }1},...,\underbrace{\texttt{\{< img. n>, <aud. n>, <vid. n>, <txt n>\}}}_{\textrm{sampled from video chunk }n},$ $$

where the video chunks are sampled from an entire video clip that offers natural sources of interleaved cross-modality data structure. Once constructed, the modalities are sampled during training to align varying targets for gradient computation and network projector alignment. In this work, we observe the even sampling method and $n=3$ are sufficient for the task, namely constructing cross-modality tasks for the beginning, middle stage, and ending of video clips.
During this stage, we jointly train the input and output projection layers, and use LoRA [] on LLM for fine-tuning.

### A.3 X-to-X cross-modality instruction tuning phase.

After the previous two phases, we have textually aligned different components of X-VILA in a unified framework. However, the model is still not ready for understanding and generating multi-modality content in a proper manner.
To achieve this goal, we curate a comprehensive “X-to-X dataset” for cross-modality generation instruction tuning.
As video captioning datasets are inherently multi-modal and provide abundant corpus in video, audio, image, and text forms, we build our X-to-X dataset based on two video captioning datasets: Webvid [] and ActivityNet Captions [].
Our X-to-X dataset features six different types of cross-modality generative conversations, namely video-to-image, video-to-video, image-to-video, video-to-audio, audio-to-video, and image+audio-to-video.
We show examples of different types of conversations in Figure 9.
Each conversation contains one or more rounds of cross-modality conversation.
More details about the X-to-X dataset are described in the experiment section.

We further divide the X-to-X cross-modality instruction tuning phase into two distinct steps, each based on different alignment methods: textual alignment and visual alignment.

(a)
To achieve textual alignment, we first project the multi-modality inputs into the textual embedding space of LLM. Then, LLM generates textual embeddings that are subsequently converted into the corresponding modality’s content.
We follow a process similar to phases (i) and (ii). Firstly, for image, video, or audio outputs, we generate embeddings using the text encoders of corresponding diffusion models. We then optimize the distance between these embeddings and the $\mathbf{E}^{\text{text}}_{m}$ generated by our model. During this step, we keep all the decoder weights frozen and train the input projection layers, output projection layers, and vocabulary embedding layer as well as LoRA parameters of LLM.
For training data, we blend our X-to-X dataset with common SFT datasets used by other VLM models [, ] (more details in the experiment section).

(b)
As mentioned earlier, relying solely on textual alignment is inherently insufficient to retain the visual details of the input when generating visual outputs.
To address such an issue, we design a novel visual alignment method. We propose a visual embedding highway (VEH) module as introduced in Section 2.1, which is utilized for the image and video decoders when there is a visual modality in the input. During training, we update the parameters of the visual decoders and the visual controller module. Meanwhile, we keep all other network parameters fixed, including the input and output projection layers and LLM. In this way, the model’s ability to conduct tasks in other modalities is not influenced by the visual alignment process.

Figure: Figure 9: Examples of different types of conversations in our X-to-X dataset. They cover six types of cross-modality understanding and generation tasks.
Refer to caption: x11.png

## Appendix B More Qualitative Results

### B.1 Examples of our X-to-X dataset.

To provide an intuitive understanding of the six types of conversations in our curated X-to-X dataset, we visualize the conversation samples of the dataset in Figure 9.
The design of the dataset focuses on building any-to-any modality connection through various conversation templates.

### B.2 Visual comparison with CoDi on cross-modality alignment.

To further examine the visual alignment advantage of X-VILA, we compare it with the state-of-the-art any-to-any model CoDi [] in Figure 10. We observe that CoDi fails to capture the real semantics and details of the input. Notably, CoDi is unable to perform X-to-X chatting, unlike X-VILA, which is specifically designed for omni-modality chatting while being able to produce superior visually aligned generation results.

### B.3 Human-model interaction demonstration.

To conduct a comprehensive assessment of our any-to-any modality LLM’s performance, we undertake more testing on X-VILA, meticulously examining different use cases. We present a collection of human-model conversation examples in Figure11, 12, 13 and 14, showcasing the versatility of X-VILA across diverse tasks.
These results demonstrate the effectiveness of X-VILA in addressing the needs of users by offering comprehensive and generative multi-modality capabilities.

Figure: Figure 10: Visual comparison to the recent work CoDi [] on cross-modality alignment. X-VILA demonstrates largely improved generation quality and cross-modality consistency.
Refer to caption: x12.png

Figure: Figure 11: Example of X-VILA performing cross-modality chat. X-VILA shows strong multi-understanding and generation ability.
Refer to caption: x13.png

Figure: Figure 12: Example of X-VILA performing cross-modality chat. X-VILA shows strong multi-understanding and generation ability.
Refer to caption: x14.png

## Appendix C More Implementation Details

As introduced in Section A, X-VILA training is separated into three phases.
(i) In the initial phase, referred to as encoder-LLM-decoder alignment training, the model undergoes 20,000 iterations using an Adam optimizer. The base learning rate is set to $4\times 10^{-4}$, and a learning rate warm-up strategy is employed. The batch size for this phase is set to 200.
(ii) During the second phase, known as interleaved data pre-training, a batch size of 192 is utilized. The base learning rate is set to $1\times 10^{-4}$, and the training is conducted for 10,000 iterations.
(iii) The final phase, called cross-modality instruction tuning, involves separate training for textual and visual alignment. For textual alignment, a batch size of 192 is maintained, and the model is trained for 30,000 iterations using a base learning rate of $1\times 10^{-4}$.
Regarding visual alignment, both the Visual Embedding Highway (VEH) and modality-specific decoders are trained for 20,000 iterations. The batch size for this phase is set to 64, and the learning rate is adjusted to $1\times 10^{-6}$.
For data amount,
Our training pipeline is highly efficient compared to many previous vision-language models [, , ].
We utilize a total of 4 NVIDIA A100 80GB server nodes in the training process.

Figure: Figure 13: More examples of X-VILA performing cross-modality chatting.
Refer to caption: x15.png

Figure: Figure 14: Conversation examples of X-VILA performing cross-modality conversation. X-VILA demonstrates a remarkable capability to comprehend the visual input and perform reasoning based on it. Our proposed visual alignment technique plays a crucial role in establishing visual consistency between the input and output.
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * I personally appreciate the architecture that supports multimodal perception and generation. However, it is true that vision and audio alone do not fully encompass the concept of Any-to-Any; other modalities are also prevalent. It appears that the current architecture lacks sufficient scalability for additional modalities. Introducing new modalities often requires re-training the LLM, which poses a limitation that may not be conducive to building a truly Any-to-Any architecture.

* I would like to understand the nature of the interactions between modalities. For instance, how does modeling and learning within the audio modality affect that of the image modality?
soundness: 2
strengths: This paper represents a commendable attempt in the field of multimodal integration. The proposed Vehicle for Enhancing Harmony (VEH) offers valuable insights into facilitating the interaction between perception modality encoding and decoding. Overall, the manuscript is well-written and provides a thorough performance evaluation.
summary: This paper introduc X-VILA, an omni-modality model designed to extend the capabilities of large language models by incorporating image, video, and audio modalities.  X-VILA achieves cross-modality understanding, reasoning, and generation within one model. And related capabilities are evaluated on extensive benchmarks.
weaknesses: * **Confusion in Structural Design**: According to the authors, the Embedding Highway design is intended to be applicable across various modalities. However, it is only applied to the visual modality, which appears to contradict the authors' objective of constructing a truly Any-to-Any multimodal model.

* **Novelty of the Model**: To my knowledge, the architecture that employs modality-independent encoding and decoding, integrated through a LLM, has been extensively explored in several prior works (e.g., Next-GPT [1]). This architecture does not seem to demonstrate clear novelty. It would be beneficial for the authors to systematically compare their work with these existing studies to highlight its innovative aspects.

[1] Wu S, Fei H, Qu L, et al. NExT-GPT: Any-to-Any Multimodal LLM[C]//Forty-first International Conference on Machine Learning.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Why use Imagebind as all modality encoders? It seems all tasks performs bad, If we change "unbind encoders", such as clip for vision, whisper/beats for audio, the performance could definitely be better. Imagebind only sounds better instead of performance better.
2. Number of compared methods are too small. Only NextGPT? 
3. " As depicted in Figure 4, we have identified two key capabilities that have surfaced Long-context cross-modality generation. X-VILA exhibits an impressive capacity for comprehending and combining diverse concepts from multiple iterations of input. Consequently, it produces natural and coherent output, as suggested by the users."  I don't think the context is long enough to be called "long-context" in the example of Figure 4.
4. The first stage training, so-called "encoder-LLM-Decoder alignment training" utilizes the "X-text pairs" from academic datasets as in prior work, so what does the decoder train? It is so confused.
soundness: 2
strengths: 1.The instructional tuning dataset may be helpful for multi-modality generation research.
2. Proposed method surpasses Next-GPT on compared benchmarks 
3. Proposed Visual Embedding Highway sounds make sense.
summary: This paper propose a multi-modality large language model which can model the understanding and generation between audio, image, video and text. It also propose a X-to-X generation instructional tuning dataset. The method is evaluated on multimodal generation benchmark and some general visual/audio QA/caption benchmarks.
weaknesses: 1. Novelty is limited. Unifing text and visual generation have been studied for a lot, such as Cogview or Emu series.  Adding audio modality also have been researched by AnyGPT or NextGPT. So this paper seems more like some performance incresements istead of some new insights, which is not excited enough to me.
2. Architecture design does not have new insights. Multimodal encoders + projectors + LLM + diffusion decoders are a  common way for unifying multimodal understanding and generation. Even though authors  emphasize that diffusion models are tuned together which make difference from earlier works, but the architecture still seem not novel and elegant enough. In addition, the visual high way design seems serving for edit-like tasks targetedly instead of a general design. 
3. Performances on visual QA or audio caption benchmarks are too weak.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Is there a planned timeline for releasing the training code and datasets for X-VILA?
soundness: 3
strengths: 1. The introduction of the Visual Embedding Highway (VEH) to preserve and enhance visual features represents a significant advancement over existing methods in visual alignment.
2. The paper provides a large X-to-X dataset for multi-modal instruction, which is a valuable resource for advancing research in multi-modal modeling.
3. The authors plan to open-source X-VILA and the dataset, promoting academic and engineering research in multi-modal foundation models.
summary: The paper introduces X-VILA, a model capable of handling various input and output modalities, including video, image, language, and audio. X-VILA employs a two-step alignment mechanism involving textual alignment and a novel visual alignment component called the Visual Embedding Highway (VEH). This mechanism addresses common challenges in multi-modality alignment, such as loss of visual information. The authors contribute a large-scale X-to-X multi-modal instruction dataset to support further research in multi-modal models.
weaknesses: 1. Missing Example Component: The example in Figure 1 lacks the "video" input component, which disrupts the clarity and completeness of the illustration.
2. High Computational Requirements: The model’s training process is resource-intensive, potentially limiting accessibility and reproducibility, particularly for smaller research teams.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section.
soundness: 2
strengths: The introduction of the Visual Embedding Highway (VEH) module to preserve visual details is an original contribution that enhances the model's performance in visual tasks. 

The paper is of high quality, as evidenced by its thorough methodology, comprehensive experiments, and rigorous evaluation. 

The paper is well-structured and clearly written.
summary: The paper introduces X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities. X-VILA achieves cross-modality understanding, reasoning, and generation by aligning modality-specific encoders with LLM inputs and diffusion decoders with LLM outputs. The key contributions of the paper are:

- It presents a new family of any-to-any modality chat LLMs capable of conducting multi-modality conversations, understanding signals from different modalities, and generating content in various formats, including video, audio, image, and text.
- The paper introduces a new X-to-X multi-modality instruction tuning dataset, which has proven effective for cross-modality alignment. 
- To address the issue of visual information loss, a visual alignment mechanism with a visual embedding highway module is introduced, which allows visual features to bypass the LLM, enhancing the correspondence of visual content between input and output stages.
- X-VILA demonstrates emergent properties across modalities even in the absence of similar training data, showcasing abilities like long-context cross-modality generation and new types of cross-modality ability.
weaknesses: - How do the authors ensure the data quality of the proposed X-to-X dataset, and what is the specific process for its creation?

- Why is the additional module VEH implemented solely in the visual component?

- Compared to Next-GPT, the method in this paper utilizes a larger instruction tuning dataset. Could the authors provide experimental results demonstrating the impact of scaling up the instruction data?