Multi-turn Interleaved Multimodal Instruction-following in the Wild (2024)

Abstract: Large language models with instructionfollowing abilities have revolutionized the field of artificial intelligence. These models show exceptional generalizability to tackle various real-world tasks through their natural language interfaces. However, their performance heavily relies on high-quality exemplar data, which is often difficult to obtain. This challenge is further exacerbated when it comes to multimodal instruction following. We introduce TEXTBIND, an almost annotation-free framework for empowering LLMs with multi-turn interleaved multimodal instruction-following capabilities. Our approach requires only imagecaption pairs and generates multi-turn multimodal instruction-response conversations from a language model. To accommodate interleaved image-text inputs and outputs, we devise MIM, a language model-centric architecture that seamlessly integrates image encoder and decoder models. Extensive quantitative and qualitative experiments demonstrate that MIM trained on TEXTBIND achieves remarkable generation capability in multimodal conversations compared to recent baselines.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 9053–9076 August 11-16, 2024 ©2024 Association for Computational Linguistics
Large language models with instructionfollowing abilities have revolutionized the field of artificial intelligence. These models show exceptional generalizability to tackle various real-world tasks through their natural language interfaces. However, their performance heavily relies on high-quality exemplar data, which is often difficult to obtain. This challenge is further exacerbated when it comes to multimodal instruction following. We introduce TEXTBIND, an almost annotation-free framework for empowering LLMs with multi-turn interleaved multimodal instruction-following capabilities. Our approach requires only imagecaption pairs and generates multi-turn multimodal instruction-response conversations from a language model. To accommodate interleaved image-text inputs and outputs, we devise MIM, a language model-centric architecture that seamlessly integrates image encoder and decoder models. Extensive quantitative and qualitative experiments demonstrate that MIM trained on TEXTBIND achieves remarkable generation capability in multimodal conversations compared to recent baselines.

heading: 1 Introduction
text: Artificial intelligence (AI) has experienced a significant paradigm shift with the rise of large language models (LLMs). These models are capable of processing a wide range of natural language processing (NLP) applications through natural language interactions with users (OpenAI, 2022, 2023).
Recently, a number of efforts have been made to augment LLMs with visual perception and understanding abilities. Prior work uses templatebased instruction-following datasets for training (Xu et al., 2023b; Dai et al., 2023; Li et al., 2023c).
*Equal Contribution. Work done during HL and SL’s internships at Tencent AI Lab. Correspondence to DC (thisisjcykcd@gmail.com).
These datasets comprise a variety of classic computer vision (CV) tasks, e.g., object detection, with each task being converted into an instructional format using a handful of human-written natural language instructions. However, classic CV tasks often represent manageable and focused abstractions or simplifications of real-world tasks (Marr, 2010), they generally fall short in representing the true variety and complexity of real-world tasks and capturing the lexical diversity of human language. For example, most of them are single-turn inquiries about a single input image, albeit a small fraction supports multi-turn textual interactions or multiple image inputs. Consequently, the instruction-following capabilities of models trained on these datasets remain limited in open-world scenarios (Xu et al., 2023a). This is reminiscent of the early development of instruction tuning in NLP, where public NLP tasks were initially adopted but eventually superseded by high-quality, diverse open-world instruction data (Ouyang et al., 2022). Nevertheless, collecting such data for multimodal models can be extremely costly.
In this paper, we address the above challenge by introducing TEXTBIND, an almost annotationfree framework for augmenting LLMs with multiturn interleaved multimodal instruction-following capabilities. The main idea is to represent images through their textual descriptions, e.g., captions, and utilize an LLM to generate multi-turn instructions and responses. To ensure the coherence and meaningfulness of the constructed multi-turn conversations, we propose a series of strategies such as topic-aware image sampling and humanin-the-loop refinement of in-context demonstrations. TEXTBIND can harvest large-scale datasets given the abundance of public image-caption pairs. TEXTBIND provides examples of processing and generating arbitrarily interleaved image-text content. To accommodate interleaved image-text inputs and outputs, we devise MIM, a multimodal
9053
model that emphasizes the reasoning abilities of LLMs and seamlessly integrates image encoder and decoder models.
For a comprehensive evaluation of open-world multimodal instruction following, we also construct TEXTBINDEVAL, an evaluation set of TEXTBIND validated and refined by human annotators. To verify the effectiveness of TEXTBIND, we compare MIM models trained on TEXTBIND and various existing datasets such as MiniGPT-4 (Zhu et al., 2023) and LLaVA (Liu et al., 2023b). The results demonstrate that TEXTBIND substantially outperforms these datasets across multiple evaluation metrics. Additional evaluations on conventional vision-language, including MME (Fu et al., 2023), MMBench (Liu et al., 2023c), and MM-Vet (Yu et al., 2023), further confirm that TEXTBIND provides complementary benefits to existing multimodal datasets. For image generation quality, we compare the proposed MIM with GILL (Koh et al., 2023a) and SDXL (Podell et al., 2023), showcasing improved performance (e.g., 0.673 vs. 0.608 CLIP scores). The advantage is particularly pronounced in multi-turn scenarios. Moreover, a holistic human evaluation further substantiates the superior performance of MIM trained on TEXTBIND in contrast to baseline models in the context of open-world multiturn interleaved multimodal instruction-following. Most interestingly, rather than requiring users to provide explicit commands and direct descriptions of the desired image, our model can spontaneously generate images in proper conversation contexts. Our qualitative experiments show that MIM trained on TEXTBIND can perform a wide range of tasks, including composing engaging stories inspired by a set of images, identifying commonalities and differences across multiple images, explaining concepts with vivid images, generating long coherent stories with illustrations, etc. Examples can be found in Appx. A. We hope TEXTBIND serves as an initial step towards building AGI that can interact with humans flexibly in different modalities and broad real-world scenarios.

heading: 2 Related Work
text: Multimodal Datasets Existing multimodal datasets can be broadly classified into two categories: (1) Conventional datasets for specific vision-language tasks such as image captioning (Chen et al., 2015; Agrawal et al., 2019; Young et al., 2014) and visually-grounded question answering (Hudson and Manning, 2019; Marino
et al., 2019; Singh et al., 2019; Lu et al., 2022; Zhou et al., 2018; Goyal et al., 2017; Gurari et al., 2018). (2) Recent datasets for general instruction following. For instance, MultiInstruct (Xu et al., 2023b), InstructBLIP (Dai et al., 2023), and M3IT (Li et al., 2023c) convert existing vision-language datasets into a unified instructional format with handcrafted templates. This approach is reminiscent of the early explorations on instruction tuning in NLP (Wei et al., 2022; Sanh et al., 2022), where existing NLP tasks were phrased as instructions. However, it has been reported that such instruction-tuned multimodal models still generalize poorly to open-world scenarios (Xu et al., 2023a). This finding also aligns with the observations in NLP (Ouyang et al., 2022), where template-based instruction tuning is less effective than instruction tuning data collected from real-world scenarios due to its restricted diversity. There are also some attempts to convert the output of existing vision-language models into natural language answers for constructing instruction-tuning data (Liu et al., 2023b; Zhu et al., 2023; Chen et al., 2023a).
Compared to existing instruction-tuning data, the examples in TEXTBIND (1) generally exhibit greater task and lexicon diversity; (2) typically involve multiple images scattered throughout a multiurn conversation; (3) support multimodal output (image generation).
Multimodal Models To augment existing LLMs with visual abilities, one straightforward approach is to employ off-the-shelf vision models as external tools. That is, the LLM calls expert vision models through their language interfaces for completing specific visual tasks when needed (Wu et al., 2023a; Shen et al., 2023; Chen et al., 2023b; Zou et al., 2022; Yang et al., 2023; Surís et al., 2023).However, these approaches may suffer from cross-modal information loss and lack of generality.
Recently, end-to-end multimodal language models have garnered significant interest. Flamingo (Alayrac et al., 2022) and OpenFlamingo (Alayrac et al., 2022) are among the pioneering work to extend LLMs to vision-language pretraining. Different from training from scratch, subsequent research efforts have focused on integrating pretrained vision and language models. BLIP-2 (Li et al., 2023b) proposes Qformer to align the feature spaces of vision models and language models. To date, various network architectures and training
strategies have been proposed (Zhu et al., 2023; Liu et al., 2023b; Ye et al., 2023; Li et al., 2023a; Zhang et al., 2023; Du et al., 2022; Chen et al., 2023a; Dai et al., 2023; Liu et al., 2023a). However, these models are limited to the use of visual content as input. Our work is inspired by recent work on LLM-empowered image retrieval or generation (Koh et al., 2023b,a) and the pioneer work of (Sun et al., 2022) for chitchat in the context of single photo sharing. Contrary to prior work, we aim to present the first instruction-following model capable of processing and generating arbitrarily interleaved image-text inputs and outputs.
Numerous contemporary studies also exist in this field (Team, 2023; Yao et al., 2023; Dong et al., 2023; Zheng et al., 2023; Ge et al., 2023). A unique characteristic of our work lies in our emphasis on the aspect of data creation, whereas other studies primarily concentrate on architecture design or training algorithms. Pan et al. (2023) and Chen et al. (2023c) focus on image editing, while Wu et al. (2023b) and Moon et al. (2023) place emphasis on incorporating additional modalities, such as audio and video.
Evaluation Conventional vision datasets designed for specific tasks and scenarios may suffer from data contamination issues for evaluating LLMs. Recently, efforts have been made to provide systematic evaluations with a broader coverage of diverse visual abilities. MME (Fu et al., 2023) is an evaluation dataset containing visually-grounded Yes/No questions. OwlEval (Ye et al., 2023) is a benchmark comprising 82 questions based on 50 images and relies on human feedback evaluation. The test size is limited, and the results may suffer from subjective bias. In response to these challenges, MMbench (Liu et al., 2023c) and MM-Vet (Yu et al., 2023) are two recent benchmarks aiming to offer more comprehensive evaluations by incorporating the use of ChatGPT/GPT4 for answer verification. LVLM Arena (Xu et al., 2023a), an online evaluation framework that ranks different models using human judgment, is also introduced. However, the above benchmarks primarily focus on question answering based on a single image at the beginning of a conversation.

heading: 3 TEXTBIND
text: In this work, we seek to enhance the multi-turn instruction-following capabilities of a language model in the context of arbitrarily interleaved im-
ages and text. Constructing such datasets poses significant challenges: 1) it demands inventive thinking for devising high-quality visually-grounded instructions and their responses; 2) it requires specialized expertise to craft appropriate images. To tackle these issues, we introduce TEXTBIND, a method that predominantly resorts to existing textonly language models to produce the desired data.

heading: 3.1 Definition of Data
text: The goal of TEXTBIND is to construct a collection of multi-turn conversation such as [x1u,x 1 a, . . . ,x T u ,x T a ], where T is the number of turns, xiu denotes the i-th instruction from the user, and xia represents the i-th response from the assistant. The conversation is also accompanied by an image set {m1, . . . ,mn}, where n is the number of unique images in this conversation. Each instruction xiu or response x i a is a sequence of tokens in Vlang ∪Vimg, where Vlang is the ordinary vocabulary of a language model and Vimg contains n distinct pointers to the images m1, . . . ,mn respectively. It is worth noting that every image can appear at any point within the conversation.

heading: 3.2 Automatic Data Generation
text: TEXTBIND consists of a three-step pipeline: 1) topic-aware image sampling for ensuring the coherence of each conversation and the diversity across conversations; 2) LLM-empowered multiturn instruction-response generation to create natural and practical conversations; 3) post-processing and filtering to eliminate low-quality data. An overview of the pipeline is shown in Fig. 1. Topic-Aware Image Sampling The initial step of TEXTBIND entails assembling groups of images that will serve as the foundation for generating multi-turn conversations. In order to facilitate coherent, meaningful, and practical conversations, the images within each group should exhibit meaningful interconnections. Furthermore, to guarantee a comprehensive representation of real-world scenarios, the topics of images across different conversations should demonstrate a wide range of diversity.
Following the above inspirations, we employ unsupervised clustering algorithms to group the images in our dataset into clusters and execute a two-step image sampling process for each conversation. Concretely, we use the image encoder of the CLIP model (Radford et al., 2021) to obtain vector representations of images. Then, we execute the k-means algorithm to classify all images
into K clusters (topics). Examples of such clusters are given in Fig. 1. For each conversation, we randomly sample a cluster from the available K clusters, then sample n ∈ {2, 3, 4} images from the chosen cluster. We want to higlight that the clustered images are semantically relevant, rather than visually similar.
Generation of Multi-turn Conversations After selecting a list of images, we proceed to leverage a text-only LLM, such as GPT-4, to simulate a conversation between a user and an assistant based on the chosen images. The core idea is to let LLMs receive and process the textual descriptions of the images as if they see the actual images. Given the abundance of publicly available image-caption pairs, we propose representing an image with an XML-like string <imgX> DESCRIPTION </imgX>, where DESCRIPTION serves as a placeholder for the image caption, <imgX> and </imgX> mark the caption boundaries, and X denotes the image index in the input image list. After generating the conversation, we replace the XML-like strings in the conversation with the original images. Importantly, to ensure that a caption faithfully describes its corresponding image, we employ the CLIP model (Radford et al., 2021) to filter out image-caption pairs with matching scores below a high threshold.
The detailed prompt can be found in Appx. B, and examples of generated conversations before mapping the textual descriptions back to visual images are shown in Appx. C. In the prompt, we also provide in-context examples to improve the generation quality. We collect the in-context examples through a human-in-the-loop refinement process, which is elaborated in §3.3.
Post-processing and Low-quality Filtering To ensure data quality, we filter out conversations where there is a pair of input and output image descriptions with an edit distance higher than 0.1. We also exclude conversations containing image descriptions not present in the provided image list and conversations containing formatting errors such as co-reference errors and invalid image tags.

heading: 3.3 Human-in-the-loop Refinement
text: In-context learning has been demonstrated to be crucial for enhancing the generation quality of LLMs (Brown et al., 2020; Wang et al., 2023). Therefore, we also construct a seed set of highquality in-context examples S. The seed set S begins as an empty set and is iteratively updated
with human feedback. In each iteration, we follow the steps detailed below:
1. We employ the latest S and the template in Appx. B, and generate 100 new conversations using TEXTBIND (§3).
2. We manually analyze the generated conversations. Each conversation is assigned a quality label (“Excellent”, “Satisfactory”, or “Poor”). Besides, we label the visual abilities required for each conversation.The detailed annotation guideline for quality labels and visual abilities is outlined in Tab. 8 (Appx. E).
3. We add the generated conversations with “Excellent” or “Satisfactory” labels to S.
To ensure diversity across different conversations, we randomly sample three in-context examples from the seed set for each generation. We further require that at least one in-context example is labeled “Excellent” and the three examples must cover all four visual abilities. After three iterations, we fix the seed set and employ it to generate the remaining data. The percentage of “Poor” data annotated by humans declines from 30% to 9%.

heading: 4 TEXTBIND Data from GPT4
text: We apply TEXTBIND to GPT4 and the Conceptual Captions (CC3M) dataset (Sharma et al., 2018; Changpinyo et al., 2021) as a case study. The details of the construction process can be found in Appx. F. In this section, we present comprehensive analyses of the constructed dataset.
Statistics As depicted in Tab. 1, our constructed dataset comprises 25, 629 conversations. The average number of turns per conversation is 3.36 (each turn is defined as a pair of instruction and response).
The mean number of images in each conversation is 2.46. The length of each dialogue is about 285.90.
Diversity To understand the lexical and task diversity of our constructed data, we identify four types of required visual abilities: image creation, image comparison, intrinsic understanding, extrinsic understanding (See Tab. 8 for detailed definitions). The distribution is shown in Fig. 2b. We observe that a significant portion of conversations in our dataset focuses on more insightful and informative tasks, such as extrinsic understanding and image comparison. For topic diversity, we display three randomly sampled clusters in Fig. 1. The distribution of images across different turns is depicted in Fig. 2c. We also compare the lexical diversity of our dataset and existing datasets in Tab. 2. It can be seen that our datasets exhibit higher lexical diversity than existing datasets.
Quality To check the quality of the generated data, we randomly sample 100 conversations and perform an in-depth error analysis. The percentages of “Excellent”, “Good”, and “Poor” data are 57%, 34%, and 9% respectively. Note that we label the whole conversation as “Poor” if any of its turns has a problem. We identify three major sources: image-caption mismatch, incoherence, and hallucination. Detailed analysis is in Appx. G.

heading: 5 Augmenting LLMs with Visual I/O

heading: 5.1 Model
text: To support interleaved multimodal inputs and outputs, we supplement LLMs with visual input and output modules. Specifically, LLama2-Chat1 (Touvron et al., 2023) is employed as the backbone LM. For visual input, we use the vision encoder from BLIP2 (Li et al., 2023b)2, followed by a pretrained Q-Former model (Li et al., 2023b) that maps the features from the vision model into the embedding space of the LM. Inspired by GILL (Koh et al., 2023a), we attempt to learn a mapping from the output space of the LM to the input space of a stable diffusion (SD) model (Rombach et al., 2022) (in this work, the embeddings produced by the text encoder of Stable Diffusion XL (Podell et al., 2023)). To this end, we explore three model variants in our preliminary experiments. Q-Former as Medium. We add a special token <IMG> to the vocabulary of the LM, indicating that an image should be generated when it is emitted. We then use a Q-Former (Li et al., 2023b) that takes all previous hidden states of the LM as input and outputs the SD embeddings. Q-Former with Prompt Tokens as Medium. To further leverage the reasoning abilities of the LM,
1 https://huggingface.co/meta-llama/Llama-2-7b-chat-hf 2 https://huggingface.co/Salesforce/blip2-flan-t5-xxl
9%
57%
34%
Excellent Satisfactory Poor
Img Create
Img Compare
Intrinsic Understand.
Extrinsic Understand.
Percentage 0% 20% 40% 60% 80%
AV G
. I m
ag e
N um
0.0
0.2
0.4
0.5
0.7
Turn Num
0 1 2 3 4 5
User Assistant
(a)
9%
57%
34%
Excellent Satisfactory Poor
Img Create
Img Compare
Intrinsic Understand.
Extrinsic Understand.
Percentage 0% 20% 40% 60% 80%
AV G
. I m
ag e
N um
0.0
0.2
0.4
0.5
0.7
Turn Num
0 1 2 3 4 5
User Assistant
(b)
9%
57%
34%
Excellent Satisfactory Poor
Img Create
Img Compare
Intrinsic Understand.
Extrinsic Understand.
Percentage 0% 20% 40% 60% 80%
AV G
. I m
ag e
N um
0.0
0.2
0.4
0.5
0.7
Turn Num
0 1 2 3 4 5
User Assistant
(c)
Figure 2: Statistics of data quality and diversity. The results in Fig. 2a and 2b are based on the human annotations on 100 randomly sampled conversations.
Large Language Model
Hi, Do you know the breed of this cat? Yes, the cat in the image appears to be a Ragdoll, a breed known for its affectionate nature. Here is another one:
Vision Encoder Q-Former
Linear
Wow, I see, they are so adorable. Do you know any other breeds of cats?
Stable Diffusion
<IMG> … <Assis> … <start> Ragdoll<user> … <start> … <start> …Persian<Assis>Wow …<user>…
Stable Diffusion
Sure! The Maine Coon is a large domestic cat breed from the U.S.
and the Persian cat is a longhaired breed with a distinctive flat face and large eyes, like this:
Figure 3: The architecture of MIM. It integrates a vision model, a language model, and a stable diffusion model. MIM is able to process multi-turn interleaved multimodal inputs and outputs. Notably, the images generated by stable diffusion model will be fed back to the MIM model to enhance the image understanding.
we incorporate a series of special tokens (<img1>, ..., <IMG{r}>), instead of a single token (<IMG>), to the LM. When <img1> is emitted, the generation of the special token sequence is enforced, serving as additional reasoning steps for predicting the forthcoming image. Subsequently, the Q-Former only accepts the hidden states of special tokens as input.
Language Description as Medium. The previous two variants try to align the continuous hidden spaces of different models. An alternative is to use discrete language descriptions for information exchange, as depicted in Fig. 3. Specifically, we add two special tokens, <start> and <end>, and
encode the generated text between these two tokens using the text encoder in the SD model.
The first two variants are optimized by minimizing the mean squared error (MSE) loss between the output and SD embeddings. For the third variant, we employ the standard cross-entropy loss. We empirically find that only the last method demonstrates satisfactory performance on multi-turn interleaved multimodal instruction-following, for which we name it MIM. The key difference between MIM and GILL is that MIM uses discrete text as medium rather than continuous embeddings. See §6.3 for detailed comparisons.

heading: 5.2 Training
text: Our training process consists of two stages, namely, the multimodal alignment stage and the multimodal instruction tuning stage.
Multimodal Alignment The first stage aims to align the feature spaces of the vision model and the language model. We utilize massive imagecaption pairs for training, drawing from datasets such as Conceptual Captions (Changpinyo et al.,
2021; Sharma et al., 2018) and SBU (Ordonez et al., 2011). During training, only the Q-Former connecting the vision and language models is optimized while other model components remain frozen.
Multimodal Instruction Following The second stage further trains the joint model on multimodal instruction tuning data to improve its instructionfollowing capabilities. The Q-Former model and LLM are optimized in this stage. In addition to TEXTBIND, we also explore existing multimodal instruction data, including MultiInstruct (Xu et al., 2023b), MiniGPT-4 (Zhu et al., 2023), LLaVA (Liu et al., 2023b), and Shikra (Chen et al., 2023a).

heading: 6 Experiments
text: To verify the effectiveness of the proposed methods, we carry out quantitative evaluations against a set of recent baselines. Our quantitative evaluations are divided into three parts: textual response generation, image generation, and a holistic evaluation of multimodal instruction-following.

heading: 6.1 TEXTBINDEVAL
text: To facilitate comprehensive and dedicated evaluation for instruction-following in realistic scenarios, we construct a new dataset named TEXTBINDEVAL. TEXTBINDEVAL is initially generated through the automatic pipeline of TEXTBIND (§3) and subsequently refined by human annotators. These annotators are tasked with discarding lowquality examples or rectifying amendable issues such as revising incoherent or hallucinated content. After a rigorous review, we establish an evaluation dataset comprising 278 conversations in total.

heading: 6.2 Textual Response Generation
text: Setup We consider each assistant turn of each conversation in TEXTBINDEVAL as a test point. All its preceding context is treated as input (which may contain interleaved images and text), and the goal is to generate a coherent and helpful response. We measure the response quality using a set of reference-based evaluation metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2020). We also report the Diversity (Su et al., 2022) scores of the generated responses. For simplicity, we replace any image in the responses with a special token <image>.
For a fair comparison, we compare different MIM models trained on different datasets (Xu et al., 2023b; Zhu et al., 2023; Liu et al., 2023b;
Chen et al., 2023a)4 and GILL (Koh et al., 2023a)5. The implementation details are shown in Appx. H.
Results As shown in Tab. 3, the MIM model trained on TEXTBIND outperforms all other baselines by wide margins across all evaluation metrics. The results suggest that more realistic and diverse training data such as TEXTBIND is necessary for tackling open-world tasks, which cannot be well-supported by existing template-based and VQA-like datasets. Nevertheless, we also find that the performance can be further improved when combining different datasets (i.e., the mixture of MultiInstruct, MiniGPT-4, LLaVA, Shikra, and TEXTBIND, denoted by MIX), indicating that there is a complementary relationship between TEXTBIND and existing datasets.
Additional Results on Conventional Benchmarks Finally, we report the results on three popular multimodal benchmarks, MME (Fu et al., 2023), MMBench (Liu et al., 2023c), and MMVet (Yu et al., 2023). As shown in Tab. 4, TEXTBIND gets a relatively lower score than other datasets. The reason stems from the intrinsic difference between TEXTBIND and the two benchmarks. TEXTBIND focuses more on realistic instructions (i.e., free-form text generation based on arbitrarily interleaved images and text across diverse userassistant conversations). In contrast, MME, MMBench and MM-Vet focus more on VQA questions (e.g., who is this person, what is the color of the object), which are more similar to the data in MultiInstruct, LLaVA, and Shikra. For example, the model trained on MultiInstruct achieves the best performance on MME, though it displays the worst performance in open-world scenarios in Tab. 3. Another interesting observation is that the mix of all datasets (MIX) attains the best overall performance on MMBench and MM-Vet, indicating that different datasets are complementary. This again confirms that the capabilities that TEXTBIND can bring are almost orthogonal to existing multimodal instruction-following datasets.

heading: 6.3 Image Generation
text: Setup The models trained on existing datasets, i.e., the baselines in §6.2 except for GILL, are in-
4The original papers of these datasets used distinct model architectures such as different pretrained language models. Nevertheless, all of them do not support image generation.
5For a fair comparison, we replicate GILL using the same image-captioning data to train by our models.
capable of generating images. To showcase the image generation capabilities of our model, we compare it with SDXL (Podell et al., 2023) and GILL (Koh et al., 2023a). In addition, we present the results of the two model variants described in §5.1, namely, Q-former as Medium and Q-former with Prompt Tokens as Medium.
We take each image from the assistant in TEXTBINDEVAL as a test point. All its preceding context is taken as input, and the models are enforced to output an image. We take the original images in TEXTBINDEVAL as references. Following Koh et al. (2023a), we evaluate image generation with two reference-based metrics: (1) CLIP Similarity. We use the CLIP vision encoder to produce image representations and compute the cosine similarity between generated images and reference images. A higher score means better semantic similarity. (2) Learned Perceptual Image Path Similarity (LPIPS). LPIPS (Zhang et al., 2018) measures the distance between generated images and reference images. A lower score means that images are more similar in perceptual space. (3) Frechet Inception Distance (FID). FID measures the distributional difference between the generated images and reference images. A lower score indicates better resemblance to reference images.
Results To gain further insights into the multiturn instruction-following abilities, we group different test points by the number of previous conversation turns. The results are shown in Tab. 5. As seen,
MIM generally achieves better performance than SDXL and GILL across different turns and evaluation metrics. Importantly, the performance gaps are enlarged as the number of turns increases. This indicates that our model exhibits a better understanding ability of multi-turn conversations. Compared to the two model variants, MIM is substantially better. Our case study reveals that the disparity stems from the one-to-many nature of image generation in real-world conversations. Unlike generating images for explicit descriptions, there can exist numerous distinct images for a given conversation context. Operating in the hidden space may inadvertently average all possibilities, resulting in ambiguous or noisy images. However, MIM mitigates the one-to-many issue by taking full advantage of the autoregressive generation of language models for decision-making.

heading: 6.4 Holistic Evaluation
text: In addition to the above automatic evaluation, we also conduct a holistic evaluation of instructionfollowing abilities through human annotation.
Setup We randomly sample 100 contexts from TEXTBINDEVAL and evaluate the responses generated by MIM and two representative baselines, LLaVA (Liu et al., 2023b) and GILL (Koh et al., 2023a). We instruct three human annotators to score the quality of each generated response on a Likert scale from 1 to 4 (major error, minor error, acceptable, and excellent). The details of evaluation guideline are in Appx. I. For inter-annotator agreement, the Krippendorff’s α = 0.75, which indicates a high consistency across annotators.
Results As depicted in Table 6, MIM achieves substantially higher human scores than GILL and LLaVA, in terms of both average score and acceptance rate, indicating its remarkable superiority in open-world multimodal conversations. To further elucidate where the derived dataset and training helps, we ask human annotators to evaluate the
quality of the generated responses across three finegrained dimensions: intent recognition, context understanding and response informativeness. As seen, MIM is more adept at following human instructions and leveraging the multimodal context, while the informativeness scores of models trained on TEXTBIND and LLaVA remain comparable.

heading: 7 Conclusion
text: In conclusion, the introduction of the TEXTBIND framework has opened new doors for enhancing large language models with multi-turn interleaved multimodal instruction-following capabilities. By requiring only image-caption pairs, our approach significantly reduces the need for high-quality exemplar data, making it a more accessible and scalable solution for various real-world tasks. The MIM architecture seamlessly integrates image encoder and decoder models, enabling the model to effectively handle interleaved image-text inputs and outputs. Comprehensive quantitative and qualitative experiments demonstrate the remarkable performance of MIM, trained on TEXTBIND, when compared to recent baselines in open-world multimodal conversations.
Limitations
Despite the amazing instruction-following capabilities demonstrated by our models trained on TEXTBIND, there are many avenues for further improvement.
1. Our best-performing models use textual descriptions as a medium for image generation. Nevertheless, it is impossible to describe everything in an image through words. Thus, our current models have difficulties completing tasks such as precise image editing. Investigating fine-grained multimodal feature combinations may enhance performance in these areas.
2. The multimodal capabilities of TEXTBIND can be further enriched by incorporating additional modalities. For example, future work may use audio-text pairs for data synthesis.
3. Similar to existing multimodal language models, models trained on TEXTBIND exhibit some common deficiencies such as hallucination, toxicity, and stereotypes.

heading: A Demonstrations
text: The four high-level characteristics of the TEXTBIND dataset equips MIM with a variety of capabilities. We demonstrate those capabilities with concrete user cases.
Image Creation One core innovation of TEXTBIND is that it enables the model to create images based on the conversation context without explicit dictations from the users. This characteristic is extremely useful for open-world scenarios, because in many cases people may just have an implicit intention and have no clear thoughts about what the images should be. We observe that our model can explain concepts and ideas for users with vivid images (Figure 5a), creating images with correct emotions (Figure 5b), and editing images based on the whole context (Figure 5c and 5d). Furthermore, as shown in Figure 4, we discover that our model is proficient in generating long stories featuring interleaved text and images while maintaining exceptional coherence.
Image Comparison Another interesting feature of TEXTBIND is that it can compare or relate the information in multiple images. For example, our model can correctly explain the different and common parts in multiple images in Figure 6.
Intrinsic & Extrinsic Image Understanding The model trained on TEXTBIND can understand the content in images precisely in a multi-turn conversation. In all the three sub-figures of Figure 7, the model precisely follows the human instructions and explains the details of the images to users. Moreover, TEXTBIND also enables the model to explore the meaning of an image beyond the symbols in it. For example, the model also explains the the influence of Bob Dylan’s album in Figure 7b and the impact of iPhone in Figure 7c.

heading: B Prompt of TEXTBIND
text: The prompt is shown in Fig. 8.

heading: C Examples of Constructed Conversations
text: The examples are given in Fig. 9.

heading: D Comparison with previous datasets.
text: We further investigate the lexical diversity of generated data, including both instructions and responses. The diversity of text in a dataset is defined as
∑4 n=2( #Unique n-grams #Total n-grams ), in line with previous works (Su et al., 2022). As shown in Tab. 2, our dataset achieves better diversity scores than most existing visual-language datasets, indicating that the language used in our dataset is more informative.

heading: E Human annotation guideline.
text: The comprehensive guideline for human evaluation is shown in Table 8.
F Implementation Details (Data)
We construct our TEXTBIND dataset based on the CONCEPTUAL CAPTIONS 3M (CC3M) (Sharma et al., 2018; Changpinyo et al., 2021) dataset, which only provides image-caption pairs. In our experiments, we employ the clip-vit-base-patch16 model released by OpenAI6 (Radford et al., 2021) to filter out image-caption pairs with matching scores lower than 30. We use the k-means clustering algorithm implemented by FAISS (Johnson et al., 2019) toolkit to classify the cleaned CC3M dataset into 4096 clusters. The features used for k-means clustering are the hidden representations of images encoded by clip-vit-base-patch16 model. In addition, clusters with less than 32 images are regarded as outliers and will not be considered. The number of images in each conversation is sampled from {2, 3, 4}. We access the GPT-4 model through the OpenAI API7, and set top_p and temperature hyper-parameters to 1.0.

heading: G Constructed Conversations with “Poor” Label
text: In Table 9, we identify three typical errors present in the constructed dataset. Despite setting a high threshold to filter out mismatched image-caption pairs, some mismatched cases cannot be detected by the CLIP model (Radford et al., 2021). A few conversations suffering from incoherence and hallucinations may be attributed to the GPT-4 model. Overall, while a small number of conversations are affected by errors that are difficult to detect using rules, most generated conversations exhibit high quality. We present several cases labeled with “Poor”. We can find that most of those “Poor” cases only have minor and non-obvious problems.
H Implementation Details (Model)
Our experiments are based on Huggingface Transformers8 (Wolf et al., 2020) and DeepSpeed9
6https://huggingface.co/openai/ clip-vit-base-patch16
7https://openai.com/blog/openai-api 8https://huggingface.co/docs/transformers/
index 9https://github.com/microsoft/DeepSpeed
(a) Hallucination Case: The topic is about rain boot. However, not all the boots shown in the second image are rain boots.
(b) Incoherence Case: The first turn discusses about relaxing while the second turn is about outdoor activities. There are no connections.
(Rasley et al., 2020). We use the filtered synthetic captions given by BLIP10 (Li et al., 2022), including Conceptual Captions (Changpinyo et al., 2021; Sharma et al., 2018) and SBU (Ordonez et al., 2011), totally 12M image-caption pairs. We employ the same visual encoder and Q-Former as used in BLIP-2 (Li et al., 2023b) and use their weights for initialization. LLama2-Chat11 (Touvron et al., 2023) is utilized as the backbone language model. For the image generation model, we use Stable Diffusion XL12 (Podell et al., 2023). The training configures are shown in Table 10. We use 8 NVIDIA A100 (40G) GPUs for all experiments.

heading: I Human Evaluation Guideline
text: For overall quality assessment, we adopt the following criteria.
• Score 4: The response is excellent.
• Score 3: The response is acceptable but may not be very informative and interesting.
• Score 2: The response has minor issues, such as slight hallucinations when describing the images in the context.
• Score 1: The response is invalid and has significant drawbacks, e.g., irrelevant to the context.
For fine-grained annotations, intent recognition evaluates whether the response fulfills the the users’ intent, context understanding examines whether the response correctly comprehends the information in text and images, and the response informativeness measures the usefulness of the response. For each dimension, a human annotator will assign a score in {1, 2, 3, 4}. The four scores ranging from 1 to 4 indicate “major error”, “minor error”, “acceptable”, and “perfect”, respectively.

heading: J Data Format of Model Input
text: The data format of model input is given in Fig. 12.
10https://github.com/salesforce/BLIP 11https://huggingface.co/meta-llama/
Llama-2-7b-chat-hf 12https://github.com/Stability-AI/ generative-models

heading: K TEXTBIND Usage
text: TEXTBIND is completely open-source and can be used for academic research and commercial purposes.

heading: Future Work
text: For example, future work may use audio-text pairs for data synthesis. 3. Similar to existing multimodal language models, models trained on TEXTBIND exhibit some common deficiencies such as hallucination, toxicity, and stereotypes