MIO: A Foundation Model on Multimodal Tokens

Authors: Noah Wang, Kang Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jessie Wang, Ning Shi, Siyu Li, Yizhi LI, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Multimodal Language Models

Abstract:
In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. While the emergence of large language models (LLMs) and multimodal large language models (MM-LLMs) propels advancements in artificial general intelligence through their versatile capabilities, they still lack true any-to-any understanding and generation. Recently, the release of GPT-4o has showcased the remarkable potential of any-to-any LLMs for complex real-world tasks, enabling omnidirectional input and output across images, speech, and text. However, it is closed-source and does not support the generation of multimodal interleaved sequences. To address this gap, we present MIO, which is trained on a mixture of discrete tokens across four modalities using causal multimodal modeling. MIO undergoes a four-stage training process: (1) alignment pre-training, (2) interleaved pre-training, (3) speech-enhanced pre-training, and (4) comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. Our experimental results indicate that MIO exhibits competitive, and in some cases superior, performance compared to previous dual-modal baselines, any-to-any model baselines, and even modality-specific baselines. Moreover, MIO demonstrates advanced capabilities inherent to its any-to-any feature, such as interleaved video-text generation, chain-of-visual-thought reasoning, visual guideline generation, instructional image editing, etc.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730530680450

thread_type:
official_review

<thread_object>
creator: reviewer_gvgf

<content>
summary:
The paper introduces MIO, a novel open-source, any-to-any multimodal foundation model designed for understanding and generating content across four modalities: text, images, speech, and video keyframes. Built with a Discrete-In-Discrete-Out (DIDO) approach, MIO leverages pretrained multimodal tokenization and a four-stage training pipeline to ensure robust cross-modality performance.

strengths:
1. MIO introduces a novel approach as a foundation model tailored for interleaved cross-modality generation, advancing current capabilities in multimodal large language models.

2. By utilizing pretrained multimodal tokenizers, MIO enables scalable and efficient parallel training on interleaved multimodal data, enhancing its adaptability across diverse input types.

3. As a foundation model, MIO offers substantial potential for further research and development in multimodal large language models and related downstream tasks.

weaknesses:
1. Qualitative comparisons with other baselines are missing. One of the main differences between MIO and Emu is the change from CICO to DIDO, and Table 4 shows that MIO performs better in image generation. The paper can be enhanced by showing the qualitative comparisons with other models, like Emu.

2. Need more clarification on multimodal inference. During the inference time, how does the model output fix-length image tokens (like one image should have 32 tokens) and speech tokens, especially for videos? Furthermore, do authors use any strategies to promise the multimodal tokens within their vocabulary?

3. Need more explanation on results. From table 3, MIO has better performance on image understanding with fewer parameters (7B vs 80B) even compared with some works that mainly focus on image understanding, like InstructBLIP and Flamingo. Can authors analyze which part led to this outperformance? Also, MIO outperforms Emu on several benchmarks. Does this indicate DIDO is better than CICO in both image generation and understanding?

4. The video generation in MIO seems more like video keyframe generation, which only maintains semantic and style consistency.

questions:
1. One related work, NextGPT [1], is missing, which also can do text, speech, and video generation. Can authors please claim the difference between MIO and this work?

2. Why is Emu2 missing in the results?

[1] Wu, Shengqiong, et al. "Next-gpt: Any-to-any multimodal llm." arXiv preprint arXiv:2309.05519 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730560676225

thread_type:
official_review

<thread_object>
creator: reviewer_uas7

<content>
summary:
This paper proposes a novel multimodal foundation model named "MIO" to understand and generate speech, text, images, and videos in an end-to-end, autoregressive manner. MIO is built on multimodal tokens across four modalities and undergoes a four-stage training process, including alignment pre-training, interleaved pre-training, speech-enhanced pre-training, and comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. The model demonstrates competitive performance compared to previous models and showcases advanced capabilities such as interleaved video-text generation and chain-of-visual-thought reasoning.

strengths:
1. Similar to GPT-4o, the proposed "MIO" model is capable of understanding and generating multimodal contents across text, image, speech, and video modalities, which is a notable advancement.

2. To address the disparity of different modalities, the paper proposes a three-stage pretraining process, including alignment, interleaving, and speech ability enhancement. Experiments show that with such a design, the model can generate various modality contents in a unified model.

3. MIO exhibits advanced features such as interleaved video-text generation and visual guideline generation.

4. The authors will release the model, which has the potential to contribute to society.

weaknesses:
1. This paper heavily relied on each modality's existing tokenizer or detokenizer, so it is restricted by their drawbacks. For example, SpeechTokenizer, although it provides a good semantic and acoustic discrete representation of a waveform, is RVQ-based and has many codebooks. So speech generation is inefficient (with 200hz).

2. As the paper claims multi-modal tokens for LLM, although it focuses on video/speech/image understanding and generation, it is necessary to evaluate the text-related benchmarks, i.e., MMLU, MMLU-Pro, GPQA etc. Please follow the practice of Llama[1] and Qwen[2] and compare your models with similar open-source models (e.g., Llama3 8B).
- [1] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).
- [2] Yang, An, et al. "Qwen2 technical report." arXiv preprint arXiv:2407.10671 (2024).

3. The authors propose a three-stage pretraining mechanism with a low ratio to a high ratio of speech tokens to tackle the disparity among different modalities.
* 3.1 Please conduct ablation experiments to demonstrate the effectiveness of the proposed method. In detail, you can compare the three-stage pretraining with one-stage pretraining. Then the one-stage pretraining's speech ratio can vary from low to high. If pretraining on the whole dataset is unaffordable, you can conduct it on a reasonable subset.
* 3.2 Usually changing the training data ratio may cause unstable results. For example, from stage 1 to stage 3, the text ratios decrease largely, thus leading to the degradation of text generation ability. Can you provide the detailed training/validation loss curve of each modality for each stage? Some evaluation results such as teacher-forcing ppl (or you can follow the practice of llama3[1] to evaluate the performance of pretrained model) are also helpful.
- [1] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

4. For speech, the paper only conducts experiments on TTS/ASR. But as the paper claims to address the gap of GPT-4o, the most interesting and important ability, speech dialogue, is not included. In TTS/ASR tasks, the model is given textual/speech content mapping, so they can not demonstrate speech ability like GPT-4o. So I think it is an over-claim.

Furthermore, you can conduct speech-to-speech task (an example is llama-omini[1]), and evaluate it on speech-to-speech benchmarks [1]
- [1] Fang, Qingkai, et al. "Llama-omni: Seamless speech interaction with large language models." arXiv preprint arXiv:2409.06666 (2024).

5. For ASR experiments, the results are not good enough. WER 6.3 is relatively high[1]. I think the first layer of SpeechTokenizer may lose some semantic information. Why did the instructed model perform worse (6.3 --> 10.3)?

- [1] https://github.com/espnet/espnet/blob/master/egs/librispeech/asr1/RESULTS.md

6. For TTS experiments, the experiments are not convincing. I recommend following the evaluation practice of zero-shot TTS experiments[1,2,3], including 1) speaker similarity, 2) speech quality, 3) robustness (WER), and some other aspects. I also recommend evaluating on more than one benchmark.

- [1] Chen, Sanyuan, et al. "VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers." arXiv preprint arXiv:2406.05370 (2024).
- [2] Le, Matthew, et al. "Voicebox: Text-guided multilingual universal speech generation at scale." Advances in neural information processing systems 36 (2024).
- [3] Ju, Zeqian, et al. "Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models." arXiv preprint arXiv:2403.03100 (2024).
- [4] Jiang, Ziyue, et al. "Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis." The Twelfth International Conference on Learning Representations. 2024. 

7. It is difficult to understand the demonstration cases, especially for speech (i.e., I can not listen to the real samples). I recommend preparing an anonymous demo page instead of only several pictures for the case study. An example is https://aka.ms/valle2 for VALL-E 2.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648951426

thread_type:
official_review

<thread_object>
creator: reviewer_dfgg

<content>
summary:
This paper introduces a new multimodal foundation model called MIO. Multimodal foundation models are essential tools for related research, similar to the role of SD in text-to-image tasks. Although this resembles a systems engineering process rather than a novel discovery, the pipeline is clear and informative. The bottleneck, such as the token length gap between image and audio, is intriguing and has been partly resolved in this paper.

strengths:
1. This paper systematically analyzes the limitations of current MLLM models and focuses on their critical aspect: any-to-any understanding and generation capabilities.
2. This paper examines images, videos, and speech to explore how to create a generalized model for all modalities while noting differences among them, such as variations in token length distribution.
3. This paper compares MIO with various existing MLLM models across different tasks to showcase its effectiveness.

weaknesses:
1. This paper uses speech-enhanced pretraining to address variations in token length distribution. However, this approach is more of a patch than a well solution.
2. This paper states that "GPT-4o has showcased... However, it is closed-source." Therefore, will MIO be made publicly available? We do not require an additional closed-source copy of GPT-4o for research purposes.
3. This paper aims to address any-to-any understanding and generation. It should possess capabilities beyond existing methods. For example, can this model generate a talking-hand video in a few-shot setting that encompasses text understanding, as well as video and speech generation?

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730764501480

thread_type:
official_review

<thread_object>
creator: reviewer_mgdu

<content>
summary:
This paper proposes a discrete token-based foundation model to unify understanding and generation capability in four modalities: speech, text, images, and videos with a proposed four-stage training recipe.

strengths:
- The paper explores the potential of unifying four different modalities in a DIDO manner using existing tokenizers within a single causal MLLM.
- It also enables the generation of multimodal output in interleaved sequences.

weaknesses:
- The paper demonstrates limited novelty in comparison to existing work within the research community.
- The overall performance of the proposed model is less competitive and lacks comprehensiveness. For example:
  - In the image understanding task, the dual-modal baselines used for comparison are relatively weak, with some considered obsolete.
  - In the image generation task, the reliance on the CLIP score metric is limiting, as it primarily focuses on text alignment and overlooks important aspects like structural integrity and aesthetics. Additionally, it has been noted to be less aligned with human preferences [1-3].
- There is a lack of both qualitative and quantitative ablation or analysis to justify key model design choices, such as the tokenizer selection, data composition, and training setup.

[1] Kirstain, Yuval, et al. "Pick-a-pic: An open dataset of user preferences for text-to-image generation." Advances in Neural Information Processing Systems 36 (2023): 36652-36663.

[2] Xu, Jiazheng, et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation." Advances in Neural Information Processing Systems 36 (2024).

[3] Wu, Xiaoshi, et al. "Human preference score: Better aligning text-to-image models with human preference." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733194668653

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Seeking further feedback

comment:
Dear Reviewers and Area Chairs,

We sincerely appreciate the valuable feedback and constructive suggestions provided by the reviewers. In response, we have carefully addressed the comments, offering detailed clarifications and additional experimental results.

As the discussion phase is nearing its conclusion (**less than 10 hours remain**), we kindly request any further feedback you might have at your earliest convenience.

Thank you for your time and consideration.

Sincerely,
MIO Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733306230501

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Rebuttal

comment:
Dear Reviewers and Chairs,

We sincerely thank all the reviewers for their insightful and constructive feedback. We appreciate that you recognized the significance of our work in advancing multimodal foundation models. We have carefully considered all comments and have made revisions to strengthen our paper.

Our key contributions are summarized as follows and listed in Table 1:

1. **Support for Multimodal Interleaved Sequence Generation**: MIO uniquely enables the generation of multimodal interleaved sequences, a capability absent in prior works. This feature allows for advanced tasks such as interleaved video-text generation, chain-of-visual-thought reasoning, instructional image editing, etc.
2. **Addressing Limitations in Existing Models**: MIO surpasses the scopes of prior models by supporting both understanding and generation across four modalities in a unified manner.  
3. **Emergent Abilities and Generalist Features**: Combining multiple modalities with both understanding and generation capabilities, MIO demonstrates emergent abilities arising from its any-to-any foundation.

Following the insightful suggestions of the reviewers, we have made the following revisions:

- **Reviewer MGdu:**
    - **Added Strong Baselines**: We included two strong baselines, Qwen-VL-Chat (13B) and LLaVA 1.5 (7B), in our image understanding experiments (Table 3) to address concerns about outdated baselines.
    - **Expanded Metrics for Image Generation**: To address limitations in evaluation metrics for image generation, we added SSIM and Aesthetic Predictor v2.5 scores and conducted human evaluations to assess structural integrity, aesthetics, and human preferences (Table 12).
    - **Provided Qualitative and Quantitative Ablations**: We added analyses on tokenizer selection, data composition, and training setup to justify key design choices (A4 for Reviewer MGdu).
- **Reviewer Uas7**:
    - **Evaluated on Text-Only Benchmarks**: We assessed our models on text-related benchmarks like MMLU to demonstrate MIO's language capabilities (Table 7).
    - **Ablations and Loss Curves**: We provided ablation experiments on the three-stage pretraining strategy and discussed the similarities between our strategy and one-stage pretraining with speech ratio warmup strategy (A3.1 for Reviewer Uas7). We also provided the loss curves for the three pretraining stages (Figure 3).
    - **Evaluated Speech Dialogue Performance**: We conducted evaluations on speech-to-speech tasks and compared our model with concurrent works like LLaMA-Omni (Table 13).
    - **Enhanced TTS Evaluation**: We added more metrics and benchmarks for TTS evaluation, including speaker similarity, WER, and human evaluation, on GLOBE, LibriSpeech, and VCTK benchmarks, comparing MIO with state-of-the-art speech models and AnyGPT (Table 5, Table 14, and A6 for Reviewer Uas7).
    - **Provided Audio Demos**: We included non-cherry-picked audio demonstrations in the supplementary materials to enhance understanding of the model's capabilities.
- **Reviewer dfgg**:
    - **Clarified the Three-Stage Pretraining Strategy**: We provided detailed explanations and experimental results to demonstrate that the speech-enhanced pretraining stage is an effective solution, not just a patch (A1 for Reviewer dfgg).
    - **Open-Sourced the Model**: We have open-sourced both the models and codes (anonymized for review), addressing concerns about accessibility for research purposes.
    - **Highlighted Advanced Capabilities**: We included demonstrations and explanations of MIO's capabilities beyond existing methods, such as advanced multimodal tasks (Figure 4).
- **Reviewer GVGf**:
    - **Provided Qualitative Comparisons**: We added qualitative comparisons between MIO and Emu for image generation in the supplementary materials to illustrate the advantages of our model.
    - **Clarified Multimodal Inference**: We explained how the model outputs fixed-length image, speech, and video tokens during inference without special decoding constraints (A2 for Reviewer GVGf).
    - **Analyzed Model Performance**: We provided insights into why MIO outperforms other models in image understanding and generation, discussing the benefits of the DIDO approach over CICO (A3 for Reviewer GVGf).
    - **Added More Baselines**: We included results for Emu2 in our experiments for a more comprehensive comparison (A6 for Reviewer GVGf).
    - **Emphasized the Importance of Video Keyframe Generation over Full Video Generation and Discussed Differences Between MIO and NeXT-GPT** (A4 and A5).

We are glad that our responses addressed most of your concerns (Reviewer Uas7 and Reviewer dfgg), even though the scores weren't raised. We still look forward to further feedback from Reviewer GVGf and Reviewer MGdu (by editing the initial reviews or scores).

Thank you all once again for your valuable feedback and engagement. Your comments have significantly helped us improve our paper.

Best regards,

MIO Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces MIO, a new multimodal foundation model designed for understanding and generating content across speech, text, images, and videos. MIO uses a discrete token-based approach (Discrete-In-Discrete-Out or DIDO). The model employees a multi-stage training recipe from different pre-trainings to sft on tasks from different modalities to ensure the robust cross-modality performance. 

The reviewers agrees this is an important direction of investigating the unified LLM for any-to-any understanding and generation capabilities. The paper provides a good systematical analysis of the limitations of the current approach. The proposed multi-stage training is also reasonable. Reviewers raised two major concern. The first is the novelty is limited heavily relays on the existing work and mostly connecting them together, and second the performance is less competitive such as vision understanding tasks and TTS tasks.