MIO: A Foundation Model on Multimodal Tokens

## Abstract

Abstract In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. While the emergence of large language models (LLMs) and multimodal large language models (MM-LLMs) propels advancements in artificial general intelligence through their versatile capabilities, they still lack true any-to-any understanding and generation. Recently, the release of GPT-4o has showcased the remarkable potential of any-to-any LLMs for complex real-world tasks, enabling omnidirectional input and output across images, speech, and text. However, it is closed-source and does not support the generation of multimodal interleaved sequences. To address this gap, we present MIO, which is trained on a mixture of discrete tokens across four modalities using causal multimodal modeling. MIO undergoes a four-stage training process: (1) alignment pre-training, (2) interleaved pre-training, (3) speech-enhanced pre-training, and (4) comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. Our experimental results indicate that MIO exhibits competitive, and in some cases superior, performance compared to previous dual-modal baselines, any-to-any model baselines, and even modality-specific baselines. Moreover, MIO demonstrates advanced capabilities inherent to its any-to-any feature, such as interleaved video-text generation, chain-of-visual-thought reasoning, visual guideline generation, instructional image editing, etc.

## 1 Introduction

The advent of Large Language Models (LLMs) is commonly considered the dawn of artificial general intelligence (AGI) [, ],
given their generalist capabilities such as complex reasoning [], role playing [], and creative writing [].
However, original LLMs lack multimodal understanding capabilities. Consequently, numerous multimodal LLMs (MM-LLMs) have been proposed, allowing LLMs to understand images [, ], audio [, , , ], and other modalities [, , ]. These MM-LLMs typically involve an external multimodal encoder, such as EVA-CLIP [] or CLAP [], with an alignment module such as Q-Former [] or MLP [] for multimodal understanding. These modules align non-textual-modality data features into the embedding space of the LLM backbone.

**Table 1: The comparison between previous models and MIO (ours). I/O Consistency indicates whether the model ensures that the input and output representations for the same data remain consistent. Uni. Bi. SFT refers to whether the model undergoes a unified (Uni.) supervised fine-tuning (SFT) for both multimodal understanding and generation (Bi.=Bidirectional). Multi-Task SFT assesses whether the model undergoes a comprehensive SFT that includes diverse tasks, with at least visual question answering tasks. MM. Inter. Output evaluates whether the model supports the generation of multimodal interleaved (MM. Inter.) sequences. We refer readers to §1 for the definitions of the different modeling approaches.**
| Models | Emu1 [] | Emu2 [] | SEED-LLaMA [] | AnyGPT [] | CM3Leon [], Chameleon [] | Gemini [] | Transfusion [] | MIO (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| I/O Consistency | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| Uni. Bi. SFT | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Multi-Task SFT | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ |
| Speech Input/Output | ✗/✗ | ✗/✗ | ✗/✗ | ✓/✓ | ✗/✗ | ✓/✗ | ✗ | ✓/✓ |
| Video Input/Output | ✓/✓ | ✓/✓ | ✓/✓ | ✗/✗ | ✗/✗ | ✓/✗ | ✗ | ✓/✓ |
| Voice Output | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| MM. Inter. Output | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Modeling | CICO | CICO | DIDO | DIDO | DIDO | CIDO | AR+Diff | DIDO |

Another line of work involves building any-to-any and end-to-end MM-LLMs that can input and output non-textual modality data. Typically, there are four approaches:
(1) Discrete-In-Discrete-Out (DIDO): Non-textual modality data is discretized using vector quantization techniques [, ] and then fed into LLMs [, , ].
(2) Continuous-In-Discrete-Out (CIDO): The LLM backbones intake densely encoded non-textual modality data features and generate their quantized representations [, ].
(3) Continuous-In-Continuous-Out (CICO): The LLMs both understand and generate non-textual modality data in their densely encoded representations [, , , , ].
(4) Autoregression + Diffusion (AR + Diff): The autoregressive and diffusion modeling are integrated in a unified LLM [, , ].
Although these works have succeeded in building MM-LLMs unifying understanding and generation, they exhibit some drawbacks, as illustrated in Table 1.
For example, Emu1 [] and Emu2 [] explore the autoregressive modeling of three modalities: text, images, and videos. SEED-LLaMA [] proposes a new image quantizer aligned with LLMs’ embedding space and trains the MM-LLMs on images and videos.
However, neither considers the speech modality, which is heterogeneous from visual modalities like videos and images.
Although AnyGPT [] has explored settings involving four modalities, including text, image, speech, and music, it lacks video-related abilities, voice synthesis, and comprehensive multi-task supervised fine-tuning, leading to limited multimodal instruction-following and reasoning capabilities.
Furthermore, AR + Diff approaches, such as Transfusion [], suffer from limited multimodal understanding capabilities because the multimodal inputs are noised for denoising modeling, and the image tokenizer used is suitable for generation rather than understanding.

Moreover, most of current MM-LLMs are typically dual-modal, combining text with another modality, such as images. Although previous works, such as Meta-Transformer [] and Unified-IO 2 [], have explored omni-multimodal understanding settings with more than two non-textual modalities, they still lag significantly behind their dual-modal counterparts, especially in terms of multimodal instruction-following capabilities.
Moreover, these MM-LLMs are typically focused on understanding only, neglecting the important aspect of multimodal generation.
Several works have enabled LLMs to call external tools to address this issue.
For example, HuggingGPT [] generates textual image descriptions for external diffusion models to synthesize images.
GPT-4 [] can utilize either an image generator like DALL-E 3 [] or a text-to-speech (TTS) tool like Whisper [] to support multimodal generation.(^1^11[https://openai.com/index/chatgpt-can-now-see-hear-and-speak/](https://openai.com/index/chatgpt-can-now-see-hear-and-speak/))
However, these methods are not end-to-end, relying on the text modality as an interface.

Recently, the release of GPT-4o has demonstrated the capabilities of any-to-any and end-to-end foundation models.(^2^22[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/))
It is the first foundational model to accept multimodal tokens as inputs and generate multimodal tokens within a unified model while also demonstrating strong abilities in complex multimodal instruction-following, reasoning, planning, and other generalist capabilities.
Furthermore, as the continuous scaling up of LLMs in the community depletes high-quality language tokens, GPT-4o verifies a new source of data for LLM training: multimodal tokens. This approach suggests that the next generation AGI could derive more knowledge from multimodal tokens when language tokens are exhausted.
However, GPT-4o is closed source and focuses primarily on end-to-end support for speech I/O, image I/O, 3D generation, and video understanding.
Its recent open-source “alternatives”, such as VITA [], still lack the ability to *generate* data of all supported modalities, particularly for the generation of multimodal interleaved sequences.

To address the aforementioned issues, we introduce MIO (Multimodal Input and Output, or Multimodal Interleaved Output), the first open-source any-to-any foundation model that unifies multimodal understanding and generation across four modalities–text, image, speech (with voice), and video, while enabling the generation of multimodal interleaved sequences.
Specifically, MIO is built on discrete multimodal tokens that capture both semantic representations through contrastive loss and low-level features via reconstruction loss [, ] from raw multimodal data.
Due to the consistent data format shared with textual corpora, the model can treat non-textual modalities as “foreign languages”, allowing it to be trained with the next-token-prediction paradigm.
Note that since the representation of an image remains the same whether it is used as an input or an output, our model flexibly supports multimodal interleaved sequence generation, where an image functions simultaneously for both understanding and generation.
Moreover, we employ a three-stage pre-training process, along with an additional SFT stage, to effectively train the model for modality expansion.

Our experimental results show that MIO, trained on a mixture of four modalities, demonstrates competitive performance compared to its dual-modal counterparts and previous any-to-any multimodal language model baselines.
Additionally, MIO is the first model to demonstrate interleaved video-text generation, chain-of-visual-thought reasoning, and other emergent abilities relying on any-to-any and multimodal interleaved output features (c.f.,§3.4).

## 2 Method

Figure: Figure 1: The framework of MIO and its training recipe.
Refer to caption: x1.png

Firstly, we elaborate on our modeling approach, which supports multimodal token input and output, as well as causal language modeling (CausalLM), in §2.1. Secondly, we describe our three-stage pre-training procedures in §2.2. Thirdly, we provide details of our comprehensive supervised fine-tuning on diverse multimodal understanding and generation tasks in §2.3.

### 2.1 Modeling

As illustrated in Figure 1, the framework of MIO involves three parts: (1) multimodal tokenization, (2) causal multimodal modeling, and (3) multimodal de-tokenization.

#### Multimodal Tokenization.

In our work, we use SEED-Tokenizer [] as our image tokenizer and SpeechTokenizer [] as our speech tokenizer.
SEED-Tokenizer encodes images using a ViT [] derived from BLIP-2 [], and then converts the encoded features into fewer tokens with causal semantics via Q-Former [].
These features are subsequently quantized into discrete tokens that are well-aligned with the language model backbone’s textual space. The codebook size for these discrete image tokens is 8192. SEED-Tokenizer transforms each image into a 224x224 resolution and quantizes it into 32 tokens. We use two special tokens, <image> and </image>, to indicate the start and end of the image tokens per image, respectively.

As for videos, we first apply specific frame-cutting methods to convert videos into image sequences. In our training data processing procedures, the number of frames for each video is dynamically determined by its duration, the length of its context, or its scene switching(^3^33[https://github.com/Breakthrough/PySceneDetect](https://github.com/Breakthrough/PySceneDetect)) to (1) avoid exceeding the LLM backbone’s context window limit, and (2) capture complete but concise information of the video. Each frame is then tokenized in the same manner as an image.

In terms of speech, SpeechTokenizer [] leverages an 8-layer RVQ [] to tokenize speech into tokens with 8 codebooks, with each codebook derived from one layer.
Since the first layer’s quantization output is distilled from HuBERT [], which encodes more semantic information, SpeechTokenizer can separate content tokens and timbre tokens from a quantized speech. The first-layer quantization is treated as content quantization, while the remaining layers’ quantization is treated as timbre quantization.
SpeechTokenizer encodes speech into 50 tokens per second for each codebook, resulting in 400 tokens per second with all eight codebooks. To improve context efficiency, we drop the last four layers’ codebooks and only use the content codebook and the first three timbre codebooks. Our vocabulary size for the speech modality is $1024\times 4=4096$.

Since the open-source pretraining-level speech data is collected from individuals with diverse voices, the timbre tokens exhibit a relatively random and noisy pattern, while the content tokens are more fixed-pattern and better aligned with the corresponding transcriptions. Given these priors in speech tokens, it is important to choose the proper interleaving mode of speech tokens [].
We denote the four codebooks as $\mathcal{A}$, $\mathcal{B}$, $\mathcal{C}$, and $\mathcal{D}$, where $\mathcal{A}$ is the codebook for content tokens and the remaining three are for timbre tokens. For simplicity, assuming that we have only two tokens for each codebook in a tokenized speech sequence (i.e., $a_{1}a_{2}$, $b_{1}b_{2}$, $c_{1}c_{2}$, and $d_{1}d_{2}$), there are two interleaving patterns for causal multimodal modeling: (1) sequential interleaving pattern: $a_{1}a_{2}b_{1}b_{2}c_{1}c_{2}d_{1}d_{2}$ and (2) alternating interleaving pattern: $a_{1}b_{1}c_{1}d_{1}a_{2}b_{2}c_{2}d_{2}$.

In our preliminary experiments, we observed that text-to-speech generation (TTS) training is difficult to converge when using the alternating interleaving pattern because the noisy and random timbre tokens ($b_{1}c_{1}d_{1}$) tend to mislead the continuations. Moreover, the speech-to-text understanding (ASR) performance improves much more slowly during training with the alternating interleaving pattern due to the sparsity of semantic information in the timbre tokens. As a result, we drop the timbre tokens for speech understanding and use the sequential interleaving pattern for speech generation. We use <spch> and </spch> as special tokens to indicate the start and end of the speech token sequence.

#### Causal Multimodal Modeling.

As illustrated in Figure 1, the speech and images, including video frames, are tokenized by SpeechTokenizer [] and SEED-Tokenizer [], respectively. We add the 4096 speech tokens and 8192 image tokens to the LLM’s vocabulary. In addition, we introduce four new special tokens, namely <image>, </image>, <spch>, and </spch>, to the vocabulary. Consequently, the embedding layer of the LLM backbone and the language modeling head are extended by $4096+8192+4=12292$ to support the embedding and generation of these new tokens.
The image tokens contain *causal* semantics due to the use of a *Causal* Q-Former [], and the speech tokens are intrinsically causal due to their temporal nature. Therefore, these multimodal tokens are as suitable for autoregressive training as textual tokens, allowing us to unify the training objectives for understanding and generation of multimodal tokens into next-token-prediction with cross-entropy loss. The training objective is thus:

$$ $\mathcal{L}=-\sum_{t=1}^{T}\log P(x_{t}\mid x_{<t};\theta)$ (1) $$

where $x_{t}$ represents the discrete multimodal tokens, and $\theta$ denotes the parameters of the LLM backbone. We use the pre-trained model, Yi-6B-Base [], for initialization.

Furthermore, to eliminate the computational inefficiency caused by <pad> tokens, we use the masked packing strategy [, , ].
Specifically,
the samples are concatenated along the sequence length dimension until the context window is full.
Then, we construct the causal attention mask for the tokens of each sample and mask out all the tokens of the other samples.

#### Multimodal De-Tokenization.

After the generation of multimodal tokens, it is essential to use modality-specific decoders to reconstruct the images or speech from the codes. Specifically, for image tokens, we directly utilize SEED-Tokenizer’s decoder, which involves an MLP projection to convert the discrete codes into dense latents. These latents condition an off-the-shelf diffusion model [] to generate the images in the pixel space [].
The vanilla SpeechTokenizer [] involves generating timbre tokens through a *non-autoregressive* model outside the language model, and then feeding the concatenated content and timbre tokens into the SpeechTokenizer decoder to synthesize speech. In our work, to inject the timbre priors into the multimodal language model itself, the timbre tokens are also generated by the *autoregressive* language model.

### 2.2 Pre-Training

As shown in Table 2, we use a three-stage strategy for pre-training, with each stage targeting different objectives. The three stages are:
(1) Alignment Pre-training: This stage focuses on learning a multimodal representation more aligned with the language space.
(2) Interleaved Pre-training: This stage aims to obtain a multimodal representation with richer contextual semantics.
(3) Speech-enhanced Pre-training: This stage specifically enhances the model’s speech-related capabilities, while concurrently replaying data from other modalities.
For more details on the pre-training data and its processing procedures, we refer the readers to Appendix B.

**Table 2: Pre-training stages and their details. We use “Inter” to denote “Interleaved” for short. We provide batch sizes for each data type per GPU in image-text pair data:language-only data:(image-text interleaved data + video data):speech-text pair data. See Appendix B and Appendix C for more details including pre-training data sources, data cleaning procedures, pre-training hyperparameters, etc.**
| Pre-training Stage | Stage I | Stage II | Stage III |
| --- | --- | --- | --- |
| Objective | Multimodal Alignment | Multimodal Interleaving | Speech Enhancement |
| Image-Text Pair | SBU, CC3M,<br>LAION-COCO,<br>JourneyDB | SBU, CC3M,<br>LAION-COCO,<br>JourneyDB | CC3M<br>LAION-COCO |
| Language-Only | RefinedWeb | RefinedWeb | RefinedWeb |
| Image-Text Inter | - | OBELICS,<br>MMC4-core-ff | MMC4-core-ff |
| Video-Text Pair | - | WebVid-10M | WebVid-10M |
| Video-Text Inter | - | HowTo-100M,<br>YT-Temporal-180M | HowTo-100M,<br>YT-Temporal-180M |
| Speech-Text Pair | Libriheavy | Libriheavy | Libriheavy |
| GPUs | 128 A800-80GB | 128 A800-80GB | 8 A800-80GB |
| Training Steps | 24,800 | 12,800 | 32,200 |
| Batch Size | 12:2:0:2 | 2:2:6:6 | 2:1:1:12 |

#### Stage I: Alignment Pre-Training.

To fully leverage the superior capabilities of the pre-trained LLM backbone, it is essential to align the non-textual modality data representations with text. There are two types of pre-training data for image-text multimodal learning:
(1) Image-text paired data: This data has well-aligned dependencies between images and text.
(2) Image-text interleaved data: This data features more natural and contextual dependencies but is less aligned.
Note that in our setting, video-text paired and interleaved data can be treated as image-text interleaved data, with videos being sequential images interleaved with text. Therefore, in this stage, we exclude the image-text interleaved data and video data to ensure the most aligned pattern between images and text.

#### Stage II: Interleaved Pre-Training.

In this stage, we extend the data used for pre-training to include image-text interleaved data (including video-text data) as a novel image-text dependency pattern.
The image-text interleaving pattern has a different nature compared to pairing patterns.
Although [] and [] argued that interleaved image-text data mainly serves for *multimodal in-context learning*,
we argue that it is also essential for context-aware image generation where images are generated based on specific context, rather than a precise description of the image content.
For example, in image-text interleaved data, the text might serve as the image’s preceding or continuing context, rather than its description.
This pattern significantly differs from the previous descriptive image generation demonstrated in image-text paired data, where images are generated based on precise and detailed text that clearly describe the content of the images [].
Therefore, context-aware image generation is essential for tasks such as *chain-of-visual-thought reasoning* or *visual storytelling* [, ], where images are generated without textual descriptions.
Due to the lack of benchmarks and evaluation metrics for context-aware image generation, we provide some demonstrations in §3.4 to showcase the potential of our model in visual storytelling, interleaved video-text generation, instructional image editing, chain-of-visual-thought reasoning, multimodal in-context learning, etc.

Moreover, in this stage, due to the extensive training on image-text paired data in Stage I, we can reduce its mixing ratio to the minimal essential scale for replay to avoid catastrophic forgetting. This allows us to increase the batch size for image-text interleaved data, video data, and speech data.

#### Stage III: Speech-Enhanced Pre-Training.

The speech tokenizer that we use generates 200 tokens for each second of audio. Given that the duration of a speech sample can be 15 seconds, this results in around 3,000 tokens per sample. In comparison, the image tokenizer produces only 32 tokens per image. This creates a significant disparity in the number of tokens among different modalities. Consequently, our training data is dominated by speech tokens. If we mix all the different modalities according to their original proportions for training, the model would likely become overly focused on speech, at the expense of other modalities.

To address this issue, we implement a three-stage strategy that gradually increases the proportion of speech tokens. In Stage I, speech-text data accounts for 12.5% of the training tokens, which rises to 37.5% in Stage II, and finally reaches 75.0% in Stage III. This incremental increase in the proportion of speech tokens ensures that the model’s performance in non-speech modalities is not compromised by the speech modality, while also allowing for the optimization of the model’s speech capabilities.

Furthermore, we keep the data mixing ratio for other modalities of pre-training data at the minimal essential scales for replay, and we only use the high-quality subsets of them in this stage. This stage requires significantly fewer compute resources, due to the foundation laid in the previous stages.

We refer the reader to Appendix C for more details about the hyperparameters and prompt templates used for pre-training.

### 2.3 Supervised Fine-Tuning

As shown in Table 7,
our model undergoes comprehensive and systematic supervised fine-tuning (SFT) with 16 different tasks and 34 diverse open-source datasets.
The chat template used for SFT is the same as that used for Yi-6B-Chat [], and only the assistant responses are supervised. We refer the reader to Appendix D for more details about the hyperparameters and prompt templates.

## 3 Experiments

In this section, we present our quantitative evaluation results across various domains: image-related tasks (§3.1), speech-related tasks (§3.2), and video-related tasks (§3.3).
Due to the lack of benchmarks for several advanced and emergent abilities of any-to-any multimodal LLMs, we also provide numerous qualitative demonstrations (§3.4) to demonstrate these capabilities.
We refer the reader to Appendix E for more details, including the decoding hyperparameters and prompt templates.

### 3.1 Image-Related Tasks

**Table 3: Experimental results for image understanding abilities. “Imagen” denotes whether the model is capable of generating images. “Speech” denotes whether the model supports speech modality. “I” denotes the instruction tuned version. The metrics used are CIDEr for COCO, MCQ accuracy for the SEED Bench, and VQA accuracy for the other tasks, following the standard procedures. In all cases, higher scores indicate better performance.**
| Models | Imagen | Speech | COCO | VQAv2 | OKVQA | VizWiz | SEED Bench |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Emu-Base (14B) | ✓ | ✗ | 112.4 | 52.0 | 38.2 | 34.2 | 47.3 |
| Emu-I (14B) | ✗ | ✗ | 120.4 | 57.2 | 43.4 | 32.2 | 58.0 |
| SEED-LLaMA-I (8B) | ✓ | ✗ | 124.5 | 66.2 | 45.9 | 55.1 | 51.5 |
| AnyGPT (8B) | ✓ | ✓ | 107.5 | - | - | - | - |
| Flamingo (9B) | ✗ | ✗ | 79.4 | 51.8 | 44.7 | 28.8 | 42.7 |
| Flamingo (80B) | ✗ | ✗ | 84.3 | 56.3 | 31.6 |  | - |
| Kosmos-1 (1.6B) | ✗ | ✗ | 84.7 | 51.0 | - | 29.2 | - |
| MetaLM (1.7B) | ✗ | ✗ | 82.2 | 41.1 | 11.4 | - | - |
| IDEFICS-I (80B) | ✗ | ✗ | 117.2 | 37.4 | 36.9 | 26.2 | 53.2 |
| CM3Leon (7B) | ✓ | ✗ | 61.6 | 47.6 | 23.8 | 37.6 | - |
| InstructBLIP (8.1B) | ✗ | ✗ | - | - | - | 34.5 | 58.8 |
| MIO-Instruct (7B) | ✓ | ✓ | 120.4 | 65.5 | 39.9 | 53.5 | 54.4 |

#### Image Understanding.

We compare our models with Emu [], SEED-LLaMA [], AnyGPT [], Flamingo [], Kosmos-1 [], MetaLM [], IDEFICS [], CM3Leon [], and InstructBLIP [].
We evaluate our models in diverse tasks, including: (1) image captioning on MS-COCO [] Karpathy test split with CIDEr score [] as the metric, (2) three visual question-answering benchmarks, i.e., VQAv2 [] (test-dev split), OK-VQA [] (val split), and VizWiz [], with VQA accuracy as the metric, and (3) SEED-Bench [], a comprehensive visual question-answering benchmark including 9 dimensions with MCQ accuracy as the metric.
The scores for all baselines are copied from their reports.
As shown in Table 3,
our MIO-Instruct is ranked in the top group among all baselines, demonstrating its competitive image understanding performance.
Although SEED-LLaMA achieved better scores compared to our model, we additionally support the speech modality. It is also noteworthy that MIO, with a size of approximately 7 billion parameters, outperforms several larger models such as Emu-14B and even IDEFICS-80B.

**Table 4: Image generation evaluation by CLIP-I score. “I” denotes the instruction tuned version. Higher values are better.**
| Models | MS-COCO | Flickr30K |
| --- | --- | --- |
| Emu-Base | 66.46 | 64.82 |
| SEED-LLaMA-Base | 69.07 | 65.54 |
| SEED-LLaMA-I | 70.68 | 66.55 |
| GILL | 67.45 | 65.16 |
| AnyGPT | 65.00 | - |
| MIO-Base | 64.15 | 62.71 |
| MIO-Instruct | 67.76 | 68.97 |

#### Image Generation.

We compare our models with Emu [], SEED-LLaMA [], GILL [], and AnyGPT [] for image generation.
We use two benchmarks, i.e., MS-COCO [] Karpathy test split and Flickr30K [].
Following GILL [] and SEED-LLaMA [], we use CLIP-I as the metric that evaluates the similarity between the generated images and the ground-truth images with the image encoder in CLIP [].
As shown in Table 4, the pre-trained model and instruction-tuned model of MIO both have competitive image generation capabilities.
Note that beyond single image generation abilities, our model can also exhibit multi-image generation capabilities such as generating visual stories, image sequences, and even visual thoughts as illustrated in §3.4.

### 3.2 Speech-Related Tasks

**Table 5: Speech ability evaluation. “WER” denotes word error rate. Lower values are better.**
| Models | ASR<br>WER | Models | TTS<br>WER |
| --- | --- | --- | --- |
| Wav2vec | 2.7 | VALL-E | 7.9 |
| Whisper | 2.7 | USLM | 6.5 |
| AnyGPT | 8.5 | AnyGPT | 8.5 |
| MIO-Base | 6.3 | MIO-Base | 12.0 |
| MIO-Instruct | 10.3 | MIO-Instruct | 4.2 |

We evaluate the speech understanding and generation abilities of MIO on ASR and TTS tasks.
Wav2vec 2.0 [], Whisper Large V2 [], and AnyGPT [] are the baselines for ASR tasks, while VALL-E [], USLM [] , and AnyGPT [] are the baselines for TTS tasks.
The test set used for ASR evaluation is LibriSpeech [], while the test set used for TTS evaluation is VCTK [] following AnyGPT []’s practice.
The Whisper medium model is used to transcribe the speech generated for the TTS task.
The WER (word error rate) is computed by comparing the generated transcribed text with the ground-truth transcription after text normalization(^4^44[https://github.com/openai/whisper/blob/main/whisper/normalizers/english.py](https://github.com/openai/whisper/blob/main/whisper/normalizers/english.py)).
As shown in Table 3.2,
our models exhibit speech performance comparable to the speech-specific baselines and outperform the AnyGPT baseline. It is important to note that although AnyGPT is capable of generating content tokens for speech, it lacks the ability to generate timbre tokens, which necessitates the use of an additional voice cloning model. In contrast, our models generate both content and timbre tokens, making the TTS tasks more challenging for our models compared to AnyGPT.
Nonetheless, after instruction tuning, our model still achieves better TTS performance.

**Table 6: Video understanding evaluation using top-1 accuracy for both benchmarks. “I” denotes the instruction-tuned version.**
| Models | MSVDQA | MSRVTT-QA |
| --- | --- | --- |
| Flamingo (9B) | 30.2 | 13.7 |
| BLIP-2 (4.1B) | 33.7 | 16.2 |
| InstructBLIP (8.1B) | 41.8 | 22.1 |
| Emu-Instruct (14B) | 32.4 | 14.0 |
| SEED-LLaMA-I (8B) | 40.9 | 30.8 |
| MIO-Instruct | 42.6 | 35.5 |

### 3.3 Video-Related Tasks

We compare MIO with Flamingo [], BLIP-2 [], InstructBLIP [], Emu [], and SEED-LLaMA [] for video understanding. The models are evaluated on the MSVDQA [] and MSRVTT-QA []. The results are presented in Table 6. Our model achieves the highest scores compared to all baselines.
Due to the lack of video (frame sequence) generation benchmarks in our setting, we provide video generation examples in §3.4.
These results demonstrate the superior performance of our models in both video understanding and video generation.

### 3.4 Demonstrations

Figure: Figure 2: Demonstrations for the basic abilities of MIO.
Refer to caption: x2.png

Figure: Figure 3: Demonstrations for the advanced abilities of MIO.
Refer to caption: x3.png

We illustrate the basic and advanced abilities of MIO in Figure 2 and 3.
The basic abilities of MIO involve image understanding and generation, video understanding and generation, ASR, and TTS.
The advanced abilities of MIO are based on its any-to-any and multimodal interleaved sequence generation features.
These abilities involve visual storytelling (i.e., interleaved video-text generation), chain of visual thought, speech-in-speech-out, instructional image editing, visual guideline generation, etc. We refer the readers to Appendix F for more demonstrations including multimodal chain of thought and multimodal in-context learning.

## 4 Related Works

### 4.1 Multimodal LLMs

With the rapid success of Large Language Models (LLMs), current multimodal LLMs (MM-LLMs) are typically built on a pre-trained LLM backbone and are endowed with the ability to understand multiple modalities [, , , , , , ]. Generally, these MM-LLMs align the representations of images obtained from visual encoders with the text embedding space, thereby leveraging the powerful capabilities of the foundational models. For example, BLIP-2 [] uses CLIP-ViT [] to extract high-level features from images and then employs a Q-Former to compress the number of image tokens and further align image tokens with the LLM embeddings. In contrast, LLaVA [, ] utilizes a simple linear projection or MLP as the connector between the image encoder and the LLM backbone. These models demonstrate strong multimodal understanding abilities, achieving significant progress in tasks such as visual question answering, visual commonsense reasoning, chart understanding, etc.

Additionally, beyond images, other MM-LLMs have also focused on modalities such as speech and video. For instance, LLaSM [] and InternVideo [, ] are MM-LLMs designed for speech and video understanding, respectively. These models adopt a similar architectural design to BLIP-2 or LLaVA but redesign modality-specific encoders tailored for speech and video.

Recently, increasing attention has been given to unifying multiple modalities within a single MM-LLM. For example, ImageBind [] develops encoders suited for multiple modalities such as images, videos, audio, heat maps, among others, while OmniBind [] trains an omni-representation model by aligning encoders across four modalities: audio, language, images, and 3D objects.

However, these models focus primarily on multimodal understanding and often overlook the important aspect of multimodal generation.

### 4.2 Any-to-Any MM-LLMs

To enable multimodal generation in MM-LLMs, a straightforward approach is to allow these models to call external multimodal generation tools, such as Stable Diffusion [] or text-to-speech (TTS) tools [, , ].
However, as highlighted in the Gemini technical report [], relying on an intermediate natural language interface can limit the model’s ability to express images.
If a model cannot natively output images, it will not be able to generate images with prompts of interleaved sequences of image and text.
This claim is in line with our distinction between descriptive image generation and context-aware image generation, as discussed in §2.2.

As a result, recent works focus on the unification of multimodal understanding and generation in a single model (i.e., any-to-any MM-LLMs), enabling the generation of multimodal tokens without natural language as an interface.
These models typically follow different approaches, depending on how images are represented in both input and output sides.
For example, the Discrete-In-Discrete-Out (DIDO) approach has been explored in works such as SEED-LLaMA [], AnyGPT [], and Chameleon [].
Continuous-In-Discrete-Out (CIDO) methods have been implemented in models like DaVinCi [], Gemini [], and Unified-IO 2 [].
The Continuous-In-Continuous-Out (CICO) approach is used in models such as Emu [, ], DreamLLM [], and MiniGPT-5 [].
Another approach, the integration of autoregression and diffusion (AR + Diff), can be seen in models like Transfusion [], Show-o [], and []’s.

However, these models face specific limitations.
DreamLLM (CICI,  []) and CIDO models suffer from inconsistencies between input and output forms for multimodal data, making it difficult for them to natively support the generation of interleaved multimodal sequences where an image functions in a coupled way as both input and output.
Emu2 (CICO,  []) struggles with the challenges of the mean square error (MSE) loss used for training continuous output representations, as well as with the uni-modal assumption of the Gaussian distribution in the MSE loss.
Transfusion (AR + Diff,  []) applies noise to images from the input side to support multimodal generation with diffusion modeling, and relies on VAE [] features rather than CLIP [] features for denoising, which largely trade off the multimodal understanding abilities.

To mitigate these issues, we adopt the DIDO approach in our work, despite the information loss caused by vector quantization. A comprehensive comparison of our models with other any-to-any MM-LLMs is presented in Table 1.

## 5 Conclusion

In conclusion, MIO represents an advancement in the realm of multimodal foundation models. By employing a rigorous four-stage training process, MIO successfully integrates and aligns discrete tokens across text, image, video, and speech modalities. This comprehensive approach enables MIO to understand and generate multimodal content in an end-to-end, autoregressive manner, addressing the limitations of current multimodal large language models. Our experimental results showcase its competitive performance across a variety of benchmarks compared to the dual-modality baselines and other any-to-any multimodal large language models. With the any-to-any and multimodal interleaved output features, MIO exhibits novel emergent abilities such as interleaved video-text generation, chain-of-visual-thought reasoning, etc.

## Appendix A Limitations

Our work has several limitations.
First, due to the information loss caused by vector quantization procedures, our model is not applicable for OCR-related images, or other images with highly detailed features. This is also a vital issue faced by CM3Leon [] and Chameleon [].
Second, due to the lack of voice-controlled speech data, our model cannot control the generated timbre yet.
Third, although our model can generate frame sequences as video representation, it cannot generate the raw video data which is in the continuous form.
We leave these issues as the research questions for the future work.

## Appendix B Pre-training Data

#### Pre-training Data Sources.

The pre-training data sources involve six types:

- 1.
Image-text paired data: SBU [], CC3M [], LAION-COCO [], and JourneyDB [], where JourneyDB only serves for image generation.
- 2.
Language-only data: RefinedWeb [].
- 3.
Image-text interleaved data: OBELICS [], MMC4-core-ff [].
- 4.
Video-text paired data: WebVid-10M [].
- 5.
Video-text interleaved data: HowTo-100M [], Youtube-Temporal-180M [].
- 6.
Speech-text paired data: Libriheavy [].

#### Pre-training Data Processing.

We have different data processing procedures for different data types illustrated in §B following Emu [] and Qwen-VL []:

- 1.
Image-text paired data: we remove pairs with more than 2:1 aspect ratio or smaller than $224\times 224$ resolution of the image. We remove pairs with more than 0.27 CLIP scores. We remove non-English pairs. We randomly place the image or text at the forefront for generating captions based on images and vice versa.
- 2.
Language-only data: we use the same data processing pipeline as used in Yi [].
- 3.
Image-text interleaved data: we filter the data using a CLIP score threshold of 0.25, and follow the same procedure as illustrated in Emu [].
- 4.
Video-text paired data: we randomly place the frames or text at the forefront for generating captions based on frames and vice versa. 60% of the pairs are text-to-video, while 40% of the pairs are video-to-text. We sample 4 to 8 frames of each video for training according to the text lengths.
- 5.
Video-text interleaved data: We first use PySceneDetect to extract key frames from the video based on scene changes, following the practice of Stable Video Diffusion []. Then, for each video clip between two key frames, we extract a central frame for textual caption generation with BLIP-2 []. Additionally, the video clips between key frames are processed using ASR (automatic speech recognition) tools to extract subtitles. The ASR text and captions are then integrated and refined using Yi-34B-Chat [], resulting in a single text segment. These text segments, along with the key frames and central frames, form the video-text interleaved data.
- 6.
Speech-text paired data: we remove speechs with more than 15 seconds.

## Appendix C Pre-training Details

#### Hyperparameters.

We enable Flash Attention [, ] during pre-training.
Gradient clipping is set to 1.0 for all stages.
The maximum sequence length for training is 2800 tokens.
We use a cosine learning rate scheduler with a peak learning rate of 3e-5 and a warmup ratio of 0.03.
The optimizer used is AdamW [].

#### Prompt Templates.

The prompt template is only necessary for paired datasets.
For image-text paired data, we use the prompt templates of “{image} The caption of this image is: {caption}” and “Please generate an image of “{caption}”: {image}”.
For video-text paired data: we use the prompt templates of “Please describe the following video: {image} {description}” and “Please generate a video for “{description}”: {video}”.
For speech-text paired data: we use the prompt templates of “{speech} Transcribe this speech: {transcription}” and “Please generate a speech of “{transcription}”: {speech}” during Stage I and Stage II.
While for Stage III, we change the ASR prompt template into ‘{speech} The transcription of this speech is: {transcription}”.

## Appendix D Supervised Fine-Tuning Details

**Table 7: Supervised Fine-Tuning Data. “ICL” denotes In-Context Learning, and “CoT” denotes Chain of Thought.**
| Task | Dataset |
| --- | --- |
| Language Only | OpenHermes [] |
| Multimodal ICL | MMICL [] |
| Multimodal CoT | ScienceQA [] |
| Chart Understanding | Geo170K [] |
| Instructional Image Generation | InstructPix2Pix [], MagicBrush [] |
| ASR | LibriSpeech [], GigaSpeech [], Common Voice [] |
| Video Dialogue | VideoChat2-IT [] |
| Image QA | Vision-Flan [], VizWiz [], LAION-GPT4V(^5^55[https://huggingface.co/datasets/laion/gpt4v-dataset](https://huggingface.co/datasets/laion/gpt4v-dataset)), LLaVAR [], OCR-VQA [], VQA [], TextVQA [], OK-VQA [], Mantis-Instruct [] |
| Speech Generation | SpeechInstruct [] |
| Speech Understanding | SpeechInstruct [] |
| Image Captioning | Flickr30K [], MS-COCO [] |
| Descriptive Image Generation | Flickr30K [], MS-COCO [] |
| TTS | GigaSpeech [], Common Voice [] |
| Video Generation | MSR-VTT [], MSVD [] |
| Video Understanding | MSR-VTT [], MSVD [], MSVD-QA [], MSRVTT-QA [] |
| Visual Storytelling | VIST [] |

#### Supervised Fine-Tuning Data.

As shown in Table 7, we use 16 tasks with 34 datasets for a comprehensive supervised fine-tuning.

#### Prompt Templates.

The chat template is the same as used in Yi [].
The system prompt is unified as: “You are MIO, an AI assistant capable of understanding and generating images, text, videos, and speech, selecting the appropriate modality according to the context.” except for speech generation and TTS whose system prompts are “You are MIO, an AI assistant capable of understanding images, text, videos, and speech, and generating speech. Please respond to the user with speech only, starting with <spch> and ending with </spch>.” to avoid randomness of the output modality.

#### Hyperparameters.

Similar to pre-training (c.f., Appendix C), we enable Flash Attention [, ] during supervised fine-tuning.
Gradient clipping is set to 1.0.
The maximum sequence length for training is 2800 tokens.
We use a cosine learning rate scheduler with a peak learning rate of 3e-5 and a warmup ratio of 0.03.
The optimizer used is AdamW [].

## Appendix E Evaluation Details.

**Table 8: Decoding Hyperparameters.**
| Output Modality | Text | Image | Speech | Video |
| --- | --- | --- | --- | --- |
| Beam size | 5 | 1 | 1 | 1 |
| Do Sampling | False | True | True | True |
| Top-P | - | 0.7 | 0.7 | 0.7 |
| Repetition Penalty | 1.0 | 1.0 | 1.15 | 1.15 |
| Temperature | 1.0 | 1.0 | 1.0 | 1.0 |
| Guidance Scale | 1.0 | 1.0 | 1.0 | 1.0 |

#### Hyperparameters.

The decoding strategies and hyperparameters are quite important for a superior performance. As shown in Table 8, we use different sets of parameters for different output modalities.

**Table 9: Prompt templates used for evaluating instruction-tuned models.**
| Task | Prompt Template |
| --- | --- |
| Image Captioning | Provide a one-sentence caption for the provided image. {image} |
| Image QA | (We use the prompt templates in LMMs-Eval []). |
| Image Generation | Please generate an image according to the given description. {description} |
| ASR | Please transcribe this speech.{speech_token} |
| TTS | Please generate a speech according to the given transcription. Start with <spch>. {transcription} |
| Text-only | The following are multiple choice questions (with answers) about {subject} {question} |
| Video QA | The goal is to use the visual information available in the image to provide an accurate answer to the question. This requires careful observation, attention to detail, and sometimes a bit of creative thinking.{video} Question: {question}<br>Answer: |

#### Prompt Templates.

The prompt templates used for evaluating pre-training checkpoints are the same as used during pre-training.
For SFT checkpoint evaluation, we list the prompt templates in Table 9.

## Appendix F More Demonstrations.

Figure: Figure 4: Multimodal Chain-of-Thought and Multimodal In-Context Learning Demos.
Refer to caption: x4.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: - The paper explores the potential of unifying four different modalities in a DIDO manner using existing tokenizers within a single causal MLLM.
- It also enables the generation of multimodal output in interleaved sequences.
summary: This paper proposes a discrete token-based foundation model to unify understanding and generation capability in four modalities: speech, text, images, and videos with a proposed four-stage training recipe.
weaknesses: - The paper demonstrates limited novelty in comparison to existing work within the research community.
- The overall performance of the proposed model is less competitive and lacks comprehensiveness. For example:
  - In the image understanding task, the dual-modal baselines used for comparison are relatively weak, with some considered obsolete.
  - In the image generation task, the reliance on the CLIP score metric is limiting, as it primarily focuses on text alignment and overlooks important aspects like structural integrity and aesthetics. Additionally, it has been noted to be less aligned with human preferences [1-3].
- There is a lack of both qualitative and quantitative ablation or analysis to justify key model design choices, such as the tokenizer selection, data composition, and training setup.

[1] Kirstain, Yuval, et al. "Pick-a-pic: An open dataset of user preferences for text-to-image generation." Advances in Neural Information Processing Systems 36 (2023): 36652-36663.

[2] Xu, Jiazheng, et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation." Advances in Neural Information Processing Systems 36 (2024).

[3] Wu, Xiaoshi, et al. "Human preference score: Better aligning text-to-image models with human preference." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weaknesses.
soundness: 3
strengths: 1. This paper systematically analyzes the limitations of current MLLM models and focuses on their critical aspect: any-to-any understanding and generation capabilities.
2. This paper examines images, videos, and speech to explore how to create a generalized model for all modalities while noting differences among them, such as variations in token length distribution.
3. This paper compares MIO with various existing MLLM models across different tasks to showcase its effectiveness.
summary: This paper introduces a new multimodal foundation model called MIO. Multimodal foundation models are essential tools for related research, similar to the role of SD in text-to-image tasks. Although this resembles a systems engineering process rather than a novel discovery, the pipeline is clear and informative. The bottleneck, such as the token length gap between image and audio, is intriguing and has been partly resolved in this paper.
weaknesses: 1. This paper uses speech-enhanced pretraining to address variations in token length distribution. However, this approach is more of a patch than a well solution.
2. This paper states that "GPT-4o has showcased... However, it is closed-source." Therefore, will MIO be made publicly available? We do not require an additional closed-source copy of GPT-4o for research purposes.
3. This paper aims to address any-to-any understanding and generation. It should possess capabilities beyond existing methods. For example, can this model generate a talking-hand video in a few-shot setting that encompasses text understanding, as well as video and speech generation?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weakness.
soundness: 2
strengths: 1. Similar to GPT-4o, the proposed "MIO" model is capable of understanding and generating multimodal contents across text, image, speech, and video modalities, which is a notable advancement.

2. To address the disparity of different modalities, the paper proposes a three-stage pretraining process, including alignment, interleaving, and speech ability enhancement. Experiments show that with such a design, the model can generate various modality contents in a unified model.

3. MIO exhibits advanced features such as interleaved video-text generation and visual guideline generation.

4. The authors will release the model, which has the potential to contribute to society.
summary: This paper proposes a novel multimodal foundation model named "MIO" to understand and generate speech, text, images, and videos in an end-to-end, autoregressive manner. MIO is built on multimodal tokens across four modalities and undergoes a four-stage training process, including alignment pre-training, interleaved pre-training, speech-enhanced pre-training, and comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. The model demonstrates competitive performance compared to previous models and showcases advanced capabilities such as interleaved video-text generation and chain-of-visual-thought reasoning.
weaknesses: 1. This paper heavily relied on each modality's existing tokenizer or detokenizer, so it is restricted by their drawbacks. For example, SpeechTokenizer, although it provides a good semantic and acoustic discrete representation of a waveform, is RVQ-based and has many codebooks. So speech generation is inefficient (with 200hz).

2. As the paper claims multi-modal tokens for LLM, although it focuses on video/speech/image understanding and generation, it is necessary to evaluate the text-related benchmarks, i.e., MMLU, MMLU-Pro, GPQA etc. Please follow the practice of Llama[1] and Qwen[2] and compare your models with similar open-source models (e.g., Llama3 8B).
- [1] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).
- [2] Yang, An, et al. "Qwen2 technical report." arXiv preprint arXiv:2407.10671 (2024).

3. The authors propose a three-stage pretraining mechanism with a low ratio to a high ratio of speech tokens to tackle the disparity among different modalities.
* 3.1 Please conduct ablation experiments to demonstrate the effectiveness of the proposed method. In detail, you can compare the three-stage pretraining with one-stage pretraining. Then the one-stage pretraining's speech ratio can vary from low to high. If pretraining on the whole dataset is unaffordable, you can conduct it on a reasonable subset.
* 3.2 Usually changing the training data ratio may cause unstable results. For example, from stage 1 to stage 3, the text ratios decrease largely, thus leading to the degradation of text generation ability. Can you provide the detailed training/validation loss curve of each modality for each stage? Some evaluation results such as teacher-forcing ppl (or you can follow the practice of llama3[1] to evaluate the performance of pretrained model) are also helpful.
- [1] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

4. For speech, the paper only conducts experiments on TTS/ASR. But as the paper claims to address the gap of GPT-4o, the most interesting and important ability, speech dialogue, is not included. In TTS/ASR tasks, the model is given textual/speech content mapping, so they can not demonstrate speech ability like GPT-4o. So I think it is an over-claim.

Furthermore, you can conduct speech-to-speech task (an example is llama-omini[1]), and evaluate it on speech-to-speech benchmarks [1]
- [1] Fang, Qingkai, et al. "Llama-omni: Seamless speech interaction with large language models." arXiv preprint arXiv:2409.06666 (2024).

5. For ASR experiments, the results are not good enough. WER 6.3 is relatively high[1]. I think the first layer of SpeechTokenizer may lose some semantic information. Why did the instructed model perform worse (6.3 --> 10.3)?

- [1] https://github.com/espnet/espnet/blob/master/egs/librispeech/asr1/RESULTS.md

6. For TTS experiments, the experiments are not convincing. I recommend following the evaluation practice of zero-shot TTS experiments[1,2,3], including 1) speaker similarity, 2) speech quality, 3) robustness (WER), and some other aspects. I also recommend evaluating on more than one benchmark.

- [1] Chen, Sanyuan, et al. "VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers." arXiv preprint arXiv:2406.05370 (2024).
- [2] Le, Matthew, et al. "Voicebox: Text-guided multilingual universal speech generation at scale." Advances in neural information processing systems 36 (2024).
- [3] Ju, Zeqian, et al. "Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models." arXiv preprint arXiv:2403.03100 (2024).
- [4] Jiang, Ziyue, et al. "Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis." The Twelfth International Conference on Learning Representations. 2024. 

7. It is difficult to understand the demonstration cases, especially for speech (i.e., I can not listen to the real samples). I recommend preparing an anonymous demo page instead of only several pictures for the case study. An example is https://aka.ms/valle2 for VALL-E 2.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. One related work, NextGPT [1], is missing, which also can do text, speech, and video generation. Can authors please claim the difference between MIO and this work?

2. Why is Emu2 missing in the results?

[1] Wu, Shengqiong, et al. "Next-gpt: Any-to-any multimodal llm." arXiv preprint arXiv:2309.05519 (2023).
soundness: 3
strengths: 1. MIO introduces a novel approach as a foundation model tailored for interleaved cross-modality generation, advancing current capabilities in multimodal large language models.

2. By utilizing pretrained multimodal tokenizers, MIO enables scalable and efficient parallel training on interleaved multimodal data, enhancing its adaptability across diverse input types.

3. As a foundation model, MIO offers substantial potential for further research and development in multimodal large language models and related downstream tasks.
summary: The paper introduces MIO, a novel open-source, any-to-any multimodal foundation model designed for understanding and generating content across four modalities: text, images, speech, and video keyframes. Built with a Discrete-In-Discrete-Out (DIDO) approach, MIO leverages pretrained multimodal tokenization and a four-stage training pipeline to ensure robust cross-modality performance.
weaknesses: 1. Qualitative comparisons with other baselines are missing. One of the main differences between MIO and Emu is the change from CICO to DIDO, and Table 4 shows that MIO performs better in image generation. The paper can be enhanced by showing the qualitative comparisons with other models, like Emu.

2. Need more clarification on multimodal inference. During the inference time, how does the model output fix-length image tokens (like one image should have 32 tokens) and speech tokens, especially for videos? Furthermore, do authors use any strategies to promise the multimodal tokens within their vocabulary?

3. Need more explanation on results. From table 3, MIO has better performance on image understanding with fewer parameters (7B vs 80B) even compared with some works that mainly focus on image understanding, like InstructBLIP and Flamingo. Can authors analyze which part led to this outperformance? Also, MIO outperforms Emu on several benchmarks. Does this indicate DIDO is better than CICO in both image generation and understanding?

4. The video generation in MIO seems more like video keyframe generation, which only maintains semantic and style consistency.

Meta-review: This paper introduces MIO, a new multimodal foundation model designed for understanding and generating content across speech, text, images, and videos. MIO uses a discrete token-based approach (Discrete-In-Discrete-Out or DIDO). The model employees a multi-stage training recipe from different pre-trainings to sft on tasks from different modalities to ensure the robust cross-modality performance. 

The reviewers agrees this is an important direction of investigating the unified LLM for any-to-any understanding and generation capabilities. The paper provides a good systematical analysis of the limitations of the current approach. The proposed multi-stage training is also reasonable. Reviewers raised two major concern. The first is the novelty is limited heavily relays on the existing work and mostly connecting them together, and second the performance is less competitive such as vision understanding tasks and TTS tasks.

Decision: Reject