Vision-Language Instruction Tuning: A Review and Analysis
Authors: Chen Li
Yixiao Ge
Dian Li
Ying Shan
Venue: Accepted by TMLR ()

Abstract: Instruction tuning is a crucial supervised training phase in Large Language Models (LLMs), aiming to enhance the LLM's ability to generalize instruction execution and adapt to user preferences. With the increasing integration of multi-modal data into LLMs, there is growing interest in Vision-Language Instruction Tuning (VLIT), which presents more complex characteristics compared to pure text instruction tuning. In this paper, we systematically review the latest VLIT settings and corresponding datasets in multi-modal LLMs and provide insights into the intrinsic motivations behind their design. For the first time, we offer a detailed multi-perspective categorization for existing VLIT datasets and identify the characteristics that high-quality VLIT data should possess. By incorporating these characteristics as guiding principles into the existing VLIT data construction process, we conduct extensive experiments and verify their positive impact on the performance of tuned multi-modal LLMs. Furthermore, we discuss the current challenges and future research directions of VLIT, providing insights for the continuous development of this field. The code and dataset related to this paper have been open-sourced at \url{https://github.com/palchenli/VL-Instruction-Tuning}.

Published in Transactions on Machine Learning Research (07 (07/2024)
Vision-Language Instruction Tuning: A Review and Analysis
Chen Li
palchenli@tencent.com
Yixiao Ge
picnaage
Dian Li $
goodlisttemcent.com
Ying Shan
yingshhan@tencett.com
ARC Lab, Tencent PCG
Foundation Technology Center, Tencent PCG
Reviewed on OpenReview: https: // openreveee.......... net/ formm? id id=ut2tbUPt1Q
Abstract
Instruction tuning is a crucial supervised training phase in Large Language Models (LLMs), aiming to enhance the LLM's ability to generalize instruction execution and adapt to user preferences. With the increasing integration of multi-modal data into LLMs, there is growing interest Lision-Language Instruction Tuning (VLIT), which presents more complex characteristics compared to pure text instruction tuning. this paper, we systematically review the latest VLIT settings and corresponding datasets in multi-modal LLMs and provide insights into the intrinsic motivations behind their design. For the first time, we offer a detailed multi-perspective categorization for existing VLIT datascts and identify the characteristics that high-quality VLIT data should possess. By incorporating these characteristics as guiding principles into the existing VLIT data construction process, we conduct extensive experiments and verify their positive impact on the performance of tuned multi-modal LLMs. Furthermore, we discuss the current challenges and future research directions VLIT, providing insights for the continuous development of this field. The code and dataset related to this paper have been open-sourced at hhttp::/// frggthuu..... wnwm:/aalccellliLIInnnnneeeeeee
1 Introduction
Recently, there has been growing dissatislaction with the limited capability of Large Language Models (LLMs) process only discrete textual information (Brown al.) 2020 Chowdhery al. 2022 Zhang et al., 2023c As result, rescarchers have been exploring techniques to enhance LLMs by enabling them to process additional modalities of information, primarily visual signals, allowing them to "see" beyond text (Huang et al. 2023a Liu et al., 2023c Zhu et al. 2023a). Integrating visual encoding modules into LLMSs is currently the primary method for achieving this functionality, which improves the versatility of Multi-modal Large Language Models (MLLMs]) | by facilitating the perception of visual data. Although the strategies employed by these MLLMS for visual signal conversion vary, their training paradigms follow a consistent. pattern (i.c., pre-training and fine-tuning) (Liu et al., 2023c Dai et al., 2023 Awadalla ct al. 2023 Among these, fine-tuning with vision-language instructions plays vital role throughout the training process.
Instruction tuning is supervised training procedure that follows the pre-training stage in LLM training, typically encompassing a variety of tasks OOyyag et al.) 2022: Wei et al. 2021). This process offers a twofold benefit: enhancing the ability of LLMs lo generalize and execute diverse task instructions while also
In this paper, MLLLSs specifically refer LLMs capable of processing visual and signals.

---

Published in Transactions on Machine Learning Research (07 2224))
LLM training paradigm
MLLM training paradigm
MLLM
VLIT Setting
LLM
Viai tusée
Adigter
(lune)
MLLM A rchitecture
Tuning Setting
Pre- Pre-training
Pictuee Aligument
Pre triitinng
VLIT Data
Instruction Tuning
Vanue L Langage Instrucioo Taning
D Data Construction
Data Tendency
Data Quality Optimization
Tuned LLM
Tuned MLLM
Figure 1: Different training paradigms of LLM and MLLM (left), and the VLlT-related key points collected and discussed in this paper (right).
bridging the gap between user preferences and model output. Vision-Language Instruction Tuning (VLIT) is specialized adaptation of instruction tuning in multi-modal scenario, tailored to the overall architecture design and training objectives of MLLMs. Compared to traditional instruction tuning, VLIT carries more responsibilities (Liu et al. 2023c Dai et al. 2023 Zhu et al.) 2023a Numerous state-of-the-art MLLMs now demonstrate strong performance on zero-shot downstream visual tasks and boast exceptional user interaction capabilitics, largely attributable their efficient VLIT.
More specifically, shown in Figure [ VLIT facilitates the efficient tuning of MLLM, primarily due to two crucial components:
VLIT Setting refers to the decision- making of whether each module in the MLLM architecture is tunable during the VLIT stage. Different MLLMs often integrate their unique frameworks and diverse understandings and requirements of VLIT to establish corresponding VLIT settings, guiding MLLMs to develop the desired capabilities.
VLIT Data, the VLIT process. On one hand, VLIT data must continuously guide MLLM in comprehending task instructions and user preferences. On the other hand, it also assumes the responsibility of directing MLLM to enhance cross-modal correlations further and understand more refined information basod on instructions.
element directly influencing MLLM performance, is of paramount importance in
However, existing reviews of instruction tuning [Zhang et al. 2023d: Wang et al., 2023g are typically limited to language-only LLMs. Discussions of instruction tuning multi-modal scenarios remain scattered throughout various MLLM papers, lacking systematic summaries and comprehensive solutions (Zhou et al. Wei et al. 2023 Wang et al.) 2023f Xu et al., 2023a). Therefore, in this paper, organize and summarize the latest MLLMs with VLIT stages and corresponding VL.IT data, providing a reasonable classification and analysis of VLIT settings and data from different perspectives. First, we explore the underlying motivations hidden behind different VLIT settings. Simultaneously, for the first time, we analyze and outline the main characteristics that high-quality VLIT data should possess, formalizing them into a complete set of quantitative evaluation indicators for different perspectives. By thoroughly considering these principles various stages (data collection, instruction-rssposee generation, and quality control), we propose pipeline capable of generating higher-quality VLIT data. To verify the effectiveness of these principles and the entire pipeline, we use common public annotation data to construct VLIT dataset according to the aforementioned design and perform VLIT on three widely used MLLMs with different architectures (ALaVA (Liu et al. 2023ch, BLIP-2 (Li et al. 2023h), and OpeuFlaming: (Awadalla et al. 2023)) alongside two other classic VLIT datasets (LLa VA (Liu et al. 2023c) and MIMIC-IT (Li et al.) 2023b)). Empirical results demonstrate that the VLIT data generated using the method proposed this paper surpasses existing work. Moreover, the various evaluation indicators proposed in this paper exhibit strong correlation with the output of tuned MLLMS in downstream tasks, and the quality control module implemented through these
2023
2

---

Published in Transactions on Machine Learning Research (07 2224)
Pure Text Instruction
Visual-Languuag Instruction
<B0 S> <conlex>>
Instruction: <instruction>
#*# Instruction:
<instruction>
Inpul: <est>
### Input: (<images>, <text>
Response: <oulpul>
### Response: <output> <EOS>
Figure 2: Instruction data format examples of LLM and MLLM.
indicators can further enhance the performance of the tuned MLLMs. The code and VLIT data are both open source and will be regularly updated with new VLIT data.
In summary, the main contributions of this paper can be outlined as follows: 1) We present a systematic review of all relevant settings and datasets of VLIT in MLLMs. For the first time, we summarize the principles that should be considered when constructing VLIT data and highlight the challenges and directions that still need exploration. 2) We propose comprehensive pipeline for constructing high-quality VLIT data and provide corresponding implementation details. 3) Based on the proposed construction pipeline and publicly available annotated data, we construct VLIT dataset and conduct experiments comparing it with existing VLIT datasets on multiple MLLMs with different architectures. The results demonstrate the validity of the summarized principles and the effectiveness of the construction pipeline.
The remainder of this paper is organized as follows: Section
2 introduces the relevant background and preliminaries of VLIT. Section 3 organizes recent MLLMs and corresponding VLIT settings and datasets. Section
4 summarizes the VLIT data construction principles based on the aforementioned information and proposcs VLIT data construction pipeline and example implementation based these principles. Section 5 primarily includes the design, implementation, and discussion of the verification experiment. Section presents current challenges and future research directions.
2 Background and Preliminary
In this section, provide brief overview of the background and preliminaries of VLIT.
Recently, LLMs have developed the ability to perform diverse tasks based different instruction descriptions, similar to humans. Instruction tuning is the primary technique employed to achieve this functionality. It introduces pre-defined instructions during the fine-tuning stage after pre-training, guiding LLMs to complete
tasks while understanding the instructions. Compared with traditional supervised finc-tuning (Tovrron
et al. 2023a), which limited to solving single lask, and prompt tuning (Lester el al. 2021 Liu al. 2022], which guides LLMs only to strengthen a certain type of task, instruction tuning is not restricted to adapting to specific tasks but rather enhances its ability solve any task. Many mature LLMs are equipped with instruction tuning, such InstructGPT (Ouyang et al.) 2022), ChatGPT (OpenAI 2023a). and LLaMA (Touvron et al. 2023a.). They follow pre-defined instructions effectively and support unseen new instructions, significantly improving zero-shot performance and greatly enhancing their generalization ability. In addition, some LLMs used as chatbots incorporate instruction tuning to make the output LLMs more aligned with human preferences. They can adapt to real-world conversation scenarios while solving tasks and significantly improve their performance when directly interacting with users. This makes instruction tuning a critical training phase for enhancing the generalization and usability of LLMs.
There has been surge in efforts to develop MLLMs due to the desire to go beyond accomplishing only discrete text-based tasks (Liu et al. 2023c Zhu et al. 2023a Awadalla et al.) 2023 Dai ct al. 2023). The introduction of visual signals has led to MLLMs being generally equipped with visual encoders and image-to-text adapters, and the training paradigm has been adjusted accordingly. Similar to the pre-training stage in LLM, MLLMs first prepare a large number of image-text pairs as training data to achieve fcature alignment between modalities and convert visual signals into representations or tokens that LLMs can understand. Then, in the instruction tuning stage, there is additional requirement for understanding visual information and
3

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: The VLIT settings and VLIT data of MLLMs. The model architecture of MLLMs can be roughly divided into three parts (i.e., visual encoder, projector, and LLM), and the colors of the modules represent the two different settings that be trainable frozen during the VLIT process. When adapting to multi-modal scenarios, the LLM needs to undergo two training stages: Pre-training (i.e., Feature Alignment (FA)) between modalities and Vision-Tanguage Instruction Tuning (VLIT). Annotation Adaption (AA); Self-Instruct (ST), and Data Mixing (DM) represent the three paradigms for constructing VLIT data. Datascts marked with indicate that they have undergone quality optimization. '_" represents that the module is not explicitly mentioned in the paper or does not exist. More detailed model and data statistics are provided in Table and Table g in the appendix.
Setting
MLLM
MLLLM Architecture
Tuning
Datases
DC
Viansl
Projector
Satsing
Plaminge
A Alvrac es Nl.
Drzz
CLIP
Percniver Recamplar
LLM(XATTN)
PA
DM
KOSMOR
CLIF
Pereeiver Resempler
MAGNETO
VLIT
DM
CLI
Linear
Vieuns
FA
VLIT
Instruct-150h
SI
Eus-CLIP
Vieuna
PA
VLAT
AA
CLIP
Linear
LLaMA
FA
VLIT
DM
InteraVideo
Q-Former | Linear
LLaMA
FA
VLIT
VidcoChu
DM
Vintoni- 2-3
22/22e
RaeNat/Inseee /Insern-H
BEItT Bare--------TTT
Alpaca
RA
VLIT
DM
X-1-1-1
CLI-
Q-Former Limear
ChatGLM
PA
VLIT
X-LLM
AA
Eve- CLIP
Q-Former Linear
Vieuna
FA VLIT
SI
22020
Eva-CLIP
Perceiver
Vieuna
PA VLIT
AA
Imagellind
Linear
Vieuna
VIET
vid
DM
(PMC-ACLIP
Linear
(PMG-)LLAMA
VLIT
PME-VQA
DM
VQ-CAN
Yals
Tools
OPT/LLSMA/.
GPT4Tools
CLIP
Linear
VicuRa
VingVVIM
at 1202
Eos-CLIP
Q-Pormer++
Ziya
M
Dster [L. et
CLIP
Percaiver Rocampler
LEaMA
Video- ("hatii" Dass
2025
CLIP
Linexr
Vienna
VLIT
ViccCEast
(14b7A1
Eva.CLIP
(Vid)Q- Pormer Linese
Vicuna
Rocampler
L&MM
CLIP
Transformer
Eea-CLIP
Q-Forauc Lincar
FlanT5/
FlacT5/Vicuna Viecma
VLIT
Lav
CLIP
MMAAdapter
LLaMA
VLFT
DM
2029
CLIP
MHSA
LLaMA
VLIT
MACAW
SI
CLIP
Linear
VLIT
SI
Lyare 2 2e 111
ChatGLM
VLIT
OERGEMM
ST
Travar
Panayc B. d 20
CLPPP
Linear
Vienn
PA VLIT
LIaVAR
S1
CLIP
Linear
Vienns
PA VLIT
S1
202gh
CLIP
Percnivar Resamplar
LLaMA HaMAA/Gaaaee /Gaanaco
VLIT
P1-200
ST
CLIP
Linear
LLaMA
FA VLIT
DM
LPPT 4itol Ye 121755
CLIP
Linear
Vieuna
PA VLIT
GPT4Rol
AA
CL.PP
Einear
Viemn
VLAT
CLIP
Percaiver Recampler
MAGNETO
VLAT
GILT CILT-20M
BusoCPT
Eca-CLIP
Q-Former Lincar
Vicuna
BabeCFT
Chatsior
CLIP
Lincur
Vicuna
YOLOX
Eva-CLIP
Linear
FLANTS
MIC_MM
Bind Network
LLMMA
VLIT
DM
RVA
Resampler
Vieunai Adapter)
PA VLTT
Lvnx
DM
1123m
CLIPP
Linear
OPTT//ooomx/
PA VLIT
Multimodal
SI
SI
A- DWlliWangee en 12025
Linear
Husky
FA VLIT
AS-1B
DM
Elrecby bh oi
CLIP
g-Former+ Linear
Viouna/LLMMe
FA VLIT
DM
Eva-CLIP
G- Q-Former+Lhnaa
Vicons
FA VLIT
DM
Fii pf
20721
CIRM/CTT BtabieLLAVA
SI
In et
20251
CLIP
Lincue
Vicuna
FA VLIT
AA
Mastic V
Bay 1E
CLIP
VL Adapter
Qwen
FA VLIT
DM
VIGL Waal
CLIP
Q-Focmert Linear
Vicuna
YLIT
VIGCT
DM
Pointt LM
Polnt- BER
Linear
Vicuna
PA VLIT
Poincl.1M
SI
20244
CLP
Q-Rormer+Tihngg
Viemn
PA VLIT
MILEM-FIP
SI
11
Imagelind
Linaar
Vieuna
EA VLIT
SI
2m Am
CLIP /aegionLIPP
Linox
Vieuns
FA VLIT
S1
Toat
2012/11
Eva-CLIP
Q-Former Lincar
LLaMA2
FA VLIT
ToxtBiad
SI
Dicami-n))
CL.1P
Linear
Vicuna
FA VLIT
DIGOWLLM
AnystAt
CLIP/DING
Linnar
LLaMA2
FA VLIT
AnyMAL
InteralMCXX
Eca-CLiP
Pereuive Sampler
InternLM
FA VLIT
IntsenLL-X
Eog-CLIP
Q-Former Lincar
Vicuna
FA
SD-VC/VC
CLIP
Ees-CLiP
SwIGLU
Vicuns+
CLIP+ CLIP= SAVSampler
Linear
Vicuna
VLUT
Forret
DM
Linear
PaYu Transformer
PA VLUT
9023r
CoPP
Linear
LIaMA2
PA VLTT
MinGPPTw3 ComVint
DM
Di ct
2029
SE
C1-10151
ml 2023
CLIP SAM
Lincur
Vicuna
FA VLIT
GraD
SI
CLIP
Camaa Q- QoFormer CB
Vioune/LLaMaN
PA VLIT
DM
Detienn 14
Linear
PuYu Transformar
VLIT
DM
1023a
Exa-CLIP
Q- Q-Former Linear
LLaMA2
FA - VLIT
DM
(Wang et
iNIS-INSTVV
SI
VideoPool
Kondestyuk al.
2024
MAGVIT-22
FA VLIT
DM
instructions beyond the original responsibilities. As shown Figure 2 the main difference between current mainstream language instructions and pure text instructions that images are included as one of the input information sources.
4

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Currently, recently emerged and rapidly evolving research area, some recent works have started to summarize existing VLII studies. Among them, due to the consistent training objectives of VLIT with the pre-training stage, the differences in its settings are primarily reflected in the update strategies of various module parameters within the MLLM framework. Comparatively, more work concentrates on the relationship between VLIT data quality and the performance of tuned MLLMs. However, research on pure lext instructions is limited methodological discussions, lacking both quantitative and qualitative experimental validation, and does not provide guidance for constructing and filtering instructional tuning data (Zhang al. 2023d Wang et al. 2023g) Lou et al., 2023). Furthermore, the analysis of VLIT data is only sporadically present in some MLLM literature, without offering a comprehensive and reliable mature experience (Li et al. 2023k
Du et al. 2023
Here, we give a formal definition of VLIT. Given a vision-language instruction sample (Instruction, Input, Response), where Instruction is the instruction sentence, Input = {< images >, <, text > is the input images and context text, Response is the response of the given instruction and input. The MLLM predicts the answer of the given instruction and input, i.e., Predicted = MLLM (Isttuucoooooon,,, Impul; 0), where (.LM(() denotes the MLLM, and 0 is the parameters of MLLM. Further, in line with the autoregressive training sctting in the pre-training stage, the instruction tuning stage still needs to correctly predict the next token in Response as the training objective.
Then, the loss function of the instruction tuning stage can be written as
N
[(0) =
Σ log pl [Reeponsec Instructiinnn Input, Rosponsc(o:)::0 0).
(1)
3 A Review of Vision-Language Instruction Tuning
In this section, we thoroughly sort out all VLIT-related work, analyzing the MLLM architecture, instruction luning paradigm, and the corresponding luning data. For relevant statisties, please refer to Table I and corresponding detailed information can be found in Table S and Table in the appendix.
3.1 Setting for Vision-Langune Instruction Tuning
The VLIT stage is an important step for adjusting MLLM, especially for further linking visual signals with the LLM. In this section, we first bricfly introduce the basic architecture and cxisting characteristics MLLM. Then, we summarize the settings for different components in the architecture during the VLIT process.
3.1.1 MLLM Architecture
As illustrated in Figure and Table the current MLLM structure primarily comprises three parts.
Visual encoder, serving as the visual signal entry point for MLLM, is mainly responsible for encoding the image. As shown in Table CLIP [Radford et al.) 2021 and its variant version: EVA-CLIP (Fang et al.) 2023) are selected the visual encoder for the vast majority of MLLMs due to its powerful visual representation and text alignment capabilitios. In addition, classic or novel visual encoders such as ResNet (He et al. 2015) and ImageBind (Han et al., 2023) are also selected by some MLLMs due to model compatibility. Overall, as the entry point for MLLM to receive visual signals, the visual encoder usually only needs to able to convert visual information such as images and vidcos into vector representations moet the requirements.
LLM processos various input information and generates response content. Recently, it is the gradual maturity of LLMs that has attracted a lot of allempts to understand multi-modal informati ii (especially visual) using LLMs. Although both are seq2seq models, these LLMs can be divided into three categorics based on their architectures. Among them, Encoder-only architecture LLMs are better at analyzing and classifying text content, such BERT (Devlin al. 2019) and RoBERTa (Liu et al. 2019) adopt this framework. L.LMs that use Decoder-only architecture classic autoregressive
5

---

Published in Transactions on Machine Learning Research (07/2024)
models that consider previous inputs context and output subsequent content. As shown in Table [ LLaMA (Touvron et al. 2023a), vicuna (a version of LLaMA that uses more text data for tuning) (Zheng et al. 2023), OPT (Zhang et al. 2022). and LLaMA2 (Touvron et al. 2023b), which use this architocture, are the choices for most MLLMs. In addition, LLMs based on the Encoder-Decoder framework, such as T5FLAN (Chung et al. 2022) and ChatGLM (Zng et al.) 2023ap, use new technologies and architecture adjustments to optimize performance, especially for tasks with complex mapping relationships between input and output sequences. Therefore, they are also chosen by some MLLMs.
Projector is tasked with connecting the above two relatively mature modules by querying and projecting the representation of the input visual signal, enabling the information output by the visual encoder to be processed by LLM. Considering the explicit function of the projector and its unrestricted implementation methods, there are many ways to implement it As shown in Table D many methods use the most direct linear projection layer (Liu et al.) 2023c] Su et al.
2023 or
projection matrix (Wu al.) 2023) or pixcl-level linear projection (Bavishi et al. 2023
Li ct al.
2023a lo map visual semantics and have achieved good results. Some methods (Zhu al. 2023a
et al. 2023 directly employ Q-Former (stacked transformers) trained three different visual-text.
tasks in BLIP2 (Li et al. 2023h and concatenate them to the visual eucoder to selectively extract visual information in the form of queries. Similarly, the Perceiver Resampler (Huang et al. 2023a Li
et al. 2023b) in Flamingo (Alayrac et al.
2022 and some adapters designed based on the transformer
structure (Wang et al. 2023e| Lyu et al.
2023 Han et al. 2023 Wang et al. 2023c are also used to
selectively map input visual representations.
3.2 Tuning Setting
To equip the aforementioned MLLM with the capability to understand and process multi-modal data, it undergoes approximately two training processes, similar to LLM: pre-training and fine-tuning, shown in Table [ In contrast to LLM pre-training, which is based on massive text, the 'pre-training' stage of MLLMM typically relies on image-text pairs to achieve the initial projection from visual semantics to text, semantics, which can be understood as cross-modal feature alignment. In the fine-tuning stage (i.e., VLIT stage), MILM assumes more responsibilities than LLM. On one hand, it needs to continue guiding LLM to understand additional visual data and instructions. On the other hand, it must further direct LLM associate instructions with different granularity visual data to provide support for successfully executing corresponding instructions. As demonstrated in Table g all MLLMs essentially follow the above training paradigm. Some methods have also made fine adjustments to the training process to enhance the performance of VLIT. For instance, some work further decomposes the tuming steps by subjectively defining the before and after relationships of tasks, aiming to progressively endow MLLM with new capabilitics (Wang et al. 2023d Bai et al. 2023: Zhang et al. 2023b Chen el al. 2023d
Additionally, as shown in Table | is worth noting that not all module parameters will be updated within
the entire VLIT framework. Some MLLMs [Liu et al.
2023c
Wang et al. 2023e Li et al.
2023e
Luo et al.
2023b
Zhao et al.
2023c b Bai et al. 2023 Xu et al.
2023b Chen et al. 2023a Li et al.
2023
Dong et al.
2023
Chen et al. 2023d represented by LLaVA, typically make their adapters and LLM tunable during
VLIT to achieve the main goal of VLIT. That involves continuing to tune the adapter to further adapt to
the information conversion between modalities and opening LLM tuning to improve its understanding of instructions, associating instructions with fine-grained visual information, and ultimately completing the ability to execute instructions. Simultaneously, some methods employ LoRA (Hu al.
2022) instead of full
tuning to control the scale of LLM updates, preventing the impact of VLIT on the original capabilities of
LLM (Su et al. 2023 Ye et al. 2023b Yang et al. 2023 Li et al. 2023k Gong et al.
2023 Yin et al. 2023
Han et al. 2023
Zhao et al. 2023d
Tu et al. 2023
Wu et al. 2023 Moon ct al. 2023
Morcover, some other
MLLMs directly freeze the LLM
parameters and only update the adapter parameters Chen et al.) 2023c Pi
lt al.) 2023 Zhang et al.] 2023f Maaz et al.) 2023 Zhang et al.) 2023aa Lyu et al. 2023 Liu et al.) 2023a
Zeng et al.
2023b Huang et al. 2023b), as the adapter possosscs the ability to understand instructions (Dai
et al. 2023
Li ct al. [202gg Hu et al.) 2023b Wang et al. 2023a) or directly insert visual information into the
LLM structure through the adapter (Zhao et al. 2023e
Zhang et al. 2023b Wang et al. 2023c You et al.
6

---

Published in Transactions on Machine Learning Research (07/2024)
Expand
Anmotation Adaption
type
Revriie
Candidate Instruction Pool
Instrutionn
Based
Original A Annotation Data:
A Annolation
f<images; <test)
Hlw seagulls theee pictur? Ai Two
Inpat
Data
Expand
Gemeraled VLII Data:
labebbbox/nnwwer....
Reurrti
Response
Instraction: What is the anower the following questio??
[LM Based
Input: [Image), "How mony soggi them the ylctune?
type of annotation data
Standard VLIT data
Outgut:
Self-Instruct
System Prompt
Original A Anotation Data:
Posprocessngg
osficce
Pa
iretrnos anirol had 125.06, noan, in s0t 90.13) vige many sugals thnn Twe.
An
In-Context
LLM
A annotation
Examples
birl 22.. 20, 90.35) (Myigg)
Data
Tomplate
Convertaion
G Geserated VLIT Data:
Detailed
LLM
Description
User: What's the picture? hands flying Whit birds picture they
the pecture.
Oze sesgall feeding. while the other eppeess jujt flying
Different of annolation data for the same object
Flextble VLIT
Figure 3: Two different construction paradigms for VLIT data: Annotation Adaption (AA) and Self-Instruee (SI). Specifically, AA usually only considers one type of annotation data as input, while SI considers multiple types of annotation data as input. AA independently considers attributes such as instruction and reply diversity to output standard VLIT data, while SI considers all different types of annotated information jointly generale interactive VLIT data, such as QA pairs or multi-turn conversations. The toy examples on the right are the outputs of different construction paradigms.
2023 Lastly, there are some MLLMs that aim to achieve VLIT from the visual encoder Li al. 20023 and even open the luning of all modules (Peng et al. 2023 which can also yield positive application results.
3.3 Data for Vision-Language Instruction Tuning
As the key to guiding MLLMs to enhance their ability to understand instructions and generalize instruction execution, VLIT data has consistently been the focus of all MLLM-related work. In this section, we categorize existing work from two independent perspectives: data construction strategies and instruction tendencies. The overall classification of the tuning data can be observed in Figure
3.3.1 Construction Strategy
VLIT data generally comes from a variety of publicly annotated visual task data (such as object detection, VQA, elc.). When constructing VLIT data, it is usually designed based on the type of inpul annolated data, the way the data organized, and the style the output data. Specifically, as illustrated in Figure 3 it can be divided into two categories: Annotation Adaption and Self-Instruct. In addition to the above two, mixing existing data and newly constructed data (i.e., Data Mixing) of different styles is also common way to expand VLIT data.
Annotation Adaption In recent years, the rapid advancement of visual models has led to the creation of large-scale, diverse, and high-quality annotated data (Lin et al. 2014 Patterson Hays 2016 Veit al.,
2016 Antol et al.] 2015). This data is suitable for wide range of downstream tasks and can be easily adapted for instructional data. As shown in Figure many studies specifically tailor the annotated

---

Published in Transactions on Machine Learning Research (07/2024)
data to adhere to the standard format found in instructional data. The Instruction component, which serves formal statement explaining the task's nature, is typically derived from manually written candidate
instruction pools (Zhu et al.) 2023a; Wang et al.) 2023e Chen et al. 2023c
Zhang et al. 2023f Li et al.
2023k
Xu et al.
2022: Dai et al. 2023: Ye et al.
2023a
Zhang et al. 2023c
Peng et al.) 2023 Zhao et al.,
2023b
Zeng et al.
2023b Moon et al. 2023 You el al. 2023). , Some researchers use manual instructions as
seeds to guide L.Ms in rewriting (Zhu et al. 2023a or expanding (Wang et al. 2023c Zhao et al., 2023e
Gong et al., 2023 Zeng et al.) 2023b} the instruction pool and enhancing instructional diversity. The Imput
consists of images and optional context, generally obtained from the original annotation data, and forms integral part of the instructional data. The Response corresponds to the result found in the annotation data and represents the output portion of the instructional data. If the original annotation data meets the user's needs, it is usually transferred without any alterations. However, it is crucial to recognize that for specific downstream tasks, such as classification, judgment, and brief captions, which typically generate outputs of only words or short sentences, the original answer may need to be expanded based on input information using LLMs as necessary to prevent polential overfitting and other related issues (Zhu al., 2023a; Zhang et al., 2023f Zeng et al. 2023b Notably, some studies consider such concise responses as a form of instruction and thus choose to incorporate suitable limiting language in the instruction to address this concern.
The scale, diversity, and creativity of instructions are often constrained by existing tasks and corresponding annotation data, hindering the universality of MLLMs that are tuned using this data. The standard instruction format faces challenges in adapting to evolving real-world application scenarios and addressing user needs. Therefore, to obtain additional instructional data that is more relevant to real-world situations, Self-Instruct is initiating the integration of a broader varicty of annotation data (Wang et al. 2022 This approach will prompt LLMs to generate more diverse and content-rich range of mnsrrutioon-flloww data, Specifically, such methods leverage the powerful understanding and generation capabilities of LLMs. As illustrated in Figure 3 they organize the annotation information (e.g., caption, bbox, label, etc.) into a detailed description of the image based on the template, LLM, etc., and then guide the language-only LLM to generate QA pairs or directly into multi-round conversations. Among them, to guide LLM correctly, these methods will manually customize detailed system prompts and some examples and further improve the quality of generated content through in-contextual learning (Liu et al. 2023c Pi et al.
Self-Instruct
2023 Li et al.
2023e b Yin et al.] 2023 Liu et al. 2023a; Zhang et al. 2023g Li el al. 2023] Zhao et al.] 2023d Wu el al.
2023 Chen et al. 2023a Li et al. 2023f You et al. 2023) (Zhao et al. believes that this will limit diversity).
Self-Instruct eliminates strict restrictions on the data, allowing for increased diversity and complexity in both
instructions and responses.
Data Mixing Of course, to expand the scale of VLIT data while ensuring that VLIT data has good diversity and other properties, many methods use the above two methods at the same time and construct large-scale VLJT data. Recently, some MLLMs have even merged VL.IT data from multiple sources with different origins to further improve the performance of MLLMs, as shown in Figure 0
3.3.2 Instruction Tendency
Due to the excellent ability of MLLM to process multi-modal information, some works have begun to explore its capacity to understand and execute more granular and domain-speciff data. As shown in Figure G we have conducted statistics based on the design of specific objects, forms, or domains in different VLIT datasots.
Among them, the main focus of the VLIT data related to specific objects is on different granularity or forms of visual information. For example, for data with different granularity, some works explore the performance of MLLM on data types such as single region, multiple regions, and image-text interleaving. Some works, represented by (Zhang et al. 2023g), focus on images with rich text information and enhance their performance on text-related tasks by constructing VLIT data. There is also a large number of video-related data, which introduces the modeling of temporal information into the VLIT process.
In addition there are corresponding VLIT data and MLLMs in fields such as medicine (Li et al.
2023e
Zhang et al.] 2023 Gao et al. 2023). documents Ye et al. 2023a), and point clouds {Xu et al. 2023b
8

---

Published in Transactions on Machine Learning Research (07/2024)
These works combine their domain-specific data and instruction characteristics to propose VLIT data that is more suitable for their scenarios.
3.3.3 Data Quality Optimization
In general, the VLIT data constructed by the methods outlined in Sec. 3.3.1 may contain a significant number of defects, particularly inherent issues (such as hallucination and expression ambiguity) introduced by LLM during the construction process. These issues tend to be concentrated in the generated VLIT data. As shown in Table 0 to mitigate these problems as much as possible, enhance data quality, and further improve the performance of the tuned MLLM, numerous researchers have optimized the construction of VLIT data based on their understanding.
Firstly, to ensure the accuracy of individual VLIT data samples, some methods employ manual annotation (Zhu et al. 2023a Li et al. 2023k Xu et al. 2022 Wang et al. 2023d), while others utilize classification based the causes of correctness issues (Zhao et al. 2023d) (such as hallucination (Wang et al. 2023a), recognition errors (Zhang et al.] 2023c), external knowledge (Zhang et al. 2023f), etc.). Socondly, from a broader perspective, some methods emphasize the diversity of VLIT data from various angles. For instance, numerous
methods enrich the types of tasks by designing additional tasks (Dai et al. 2023 Li et al. 2023k). Some
methods introduce pure text instructions (Ye al. 2023b Luo ct al.) 2023a Lyu ct al. 2023 Peng al.,
2023 Zeng et al.) 2023b Bai et al.) (023)) and rewrite or expand existing instructions (as mentioned
Sec. 3.3.1) enhance the diversity of instructions within single task.
Other methods also control the
length of the output text to consider the diversity of responses (Gong al. 2023 Liu et al. 2023a Thirdly, many rescarchers have started to recognize the importance of instruction complexity. Consequently, some methods attempt to increase the complexity of VLIT data by introducing multi-round dialogues (T.yu et al., 2023), adding negative samples (Yang et al. 2023 Liu et al. 2023a: Zhao et al. 2023c You et al. 2023) , and incorporating more complex task formats (Xu et al. 2023b). Lastly, some studies acknowledge the significance of balance in VLIT and introduce methods for balancing distributions within VLIT data (Zhao
al. 2023a Zeng et al. 2023b).
However, there are two main problems in the construction of the current VLIT dataset.
Most existing VLIT data construction methods are based on intuition and some scattered unverified practical experience, and lack reasonable, reliable, and systematic construction principles to guide the construction of VLIT data (especially in visual data).
The current instruction data filtering strategy is still very rough and cannot quantitatively and qualitatively evaluate a single data sample and the overall data distribution. As result, the existing VLIT data generally very large and noisy, which greatly affects the downstream performance and tuning efficiency of MLLMs.
4 How to Construct High-Quality VLIT Data?
In this section, we begin by meticulously summarizing the characteristics of high-quality VLIT data. Subse- quently, integrate these characteristics guiding principles into the existing construction process, thereby forming a pipeline that generates superior-quality VLIT data. Simultancously, we also provide a sample implementation of this pipeline and construct a VLIT dataset.
4.1 Characteristics and Principles
Combining the discussion in Sec. 3.3.3 we describe in detail the characteristics that high-quality VLIT data should possess and organize them into several principles as follows:
First of all, as the data input to MLLM to train MLLM, it must usable, so that it will not have a negative impact on MILM. Then the principles of usable can be summarized as:
Principle 1. Readability: The text and images in VLIT data need support MLLM obtaining sufficient information, which means that the text and images need to have semantic readability.
9

---

Published in Transactions on Machine Learning Research (07/2024)
Principle 2. Correctness: The visual information and tertual content in each VLIT data (t.., instruction- response pair) must be perfeclly malched, ensuring that there are no inaccuracies in the descriplions of visual information or external knowledge within the generated content.
Diversity has long been local point in natural language generation and a crucial aspect of evaluating content. quality (Tevet & Berant 2020 Zhang et all )020). In particular, diversity plays a critical role in the tuning
of LLM instructions,
it determines the LLM's ability to generalize instructions (Zhang et al. 2023d
Specifically, in the field of VLIT, diversity exhibits more distinct characteristics, which can be summarized into the following three aspects:
Principle 3. Tosk Diversity: VLIT data should encomposs an adequate variety of tasks, encompassing both teztual and visual tasks.
Principle 4. Instruction Diversity: In VLIT data, is essential for each task to comprise a sufficient voriety of distinct instruction sentences.
Principle 5. Response Diversity: In VLIT data, the response text for each task needs to maintain sufficient distinguishabliity.
In particular, VLIT, the diversity of visual information will also directly affect the performance of MLLM in executing instructions in different scenarios. Therefore, the diversity of visual information should also be taken into consideration.
Principle 6. Visual Semantic /Objett Diversity: The visual information (i.e., visual semantics and objects) between different VLIT data needs to have sufficient differences to ensure that VLIT has sufficient diversity as a whole.
LLM with text instructions is the task's complexity, which has resulted in the development of the chain of thought tuning. In a multi-modal scenario, high-quality VLIT data with appropriate complexity can further guide MLLMs to jointly comprehend visual and textual semantics at varying gramularities. Cousequently, when constructing VLIT, two complexity-related principles must be taken into account:
Principle 7. Instruction Complezity; In VLIT dala, it is essential to ensure that some instructions necessilale sufficiently complex logic for completion.
Principle 8. /rrnnularity Complerity: In VLIT data, the granularities of the instruction's object operation must eahibit a sufficient level of complexitty
At the same time, since complex visual information often increases the difficulty of understanding and executing instructions, is also necessary to consider the complexity of visual information in VLIT data to ensure that it can assist MLLMs in achieving efficient VLIT.
Principle 9. Visual Semantic Complexity: The visual information contained in VLIT data needs to include more complex scenes so that VLIT can guide MLLMs to process instructions at different semantic granularities.
Maintaining data balance is crucial for both traditional machine learning and LLMs. Research has shown that uneven data settings may lead to forgetting phenomena in large models, seriously hindering their overall performance (Wei et al. 2023 Zhai et al.) 2023). Therefore, the following principle related to balance is equally important.
Principle 10. In VLIT data, the distribution of tasks needs to be uniform and there no obvious long tail.
4.2 Principle Guided VLIT Data Construction
Based on the aforementioned principles, we incorporate corresponding considerations into each step of the classic VLIT data generation pipcline and added filtering mcasures that cover all principles in the final quality control module. In the following sections, we provide a comprehensive description and an example implementation of the VL.IT data generation pipeline guided by these principles.
10

---

Published in Transactions on Machine Learning Research (07/2024)
Data Collection
Indircctinn Response Generation
Candidate Instraction-Re Responee Pairs
"Ewer: "Pouee deertbe the glvm bauge in dita." aboier
COCCan1
Nadienaal
Pronpt Desig
pert the piiture io rod with exadle red Bookl parked side
GP1-33
STOP
"AP": "The dom petter vent cxmmer" tht dicknng a "Bod" stidor under a "Stpp agg, the vandale ouerosey aakis
owty
ariifiial leadlyncce
the pulltts Eidta people trati Beartllon i' that verdadds uanda vateed mo pubiical
peacrate conversatins teage
Here an euuppá, the gfren tintnal decorptiie and geoerated converaation
Quality Control
Iedividoll Evottotion
stop
Captin
from our.
Reabalilyy
Genuration Priteallly
Teag keudalll
No 10 Aseoscnce
stret dig cood step
randorssua
Degre
Alggnee
[28.52 158.61, 52.2% (3,62)
uebide, cur: parkted
Cavas testrall desccpttins Tidn plctare which on tem booch Diyyyle with alt Innthhmal Conmated
Carretuoe
11 luvel Dircinttate
Attribate
vebille cur andalinntalle.
Overall Evalaation
Task Dounsity
LLMilusel Teggins
l'quael: Whhr wason is this phote?"
msutiax U' "Mlll tell sheet "A1"; tmage slow
batradion Diumat
Comeunndy Delactin
sssae: [1 [[auuseer 7), Espringg 20,
Vieaa U|A
fon
Repeese Diverity
Dusaby hand Cluderzq
Uquane:: Whet coker in thes our the
Soeeeattter blandty
Senants Chaker
background? "anne: Irved' 1011
surd tleme valaacart the aree.)
ObyatiCrunkaitt Cumpheite
Sematt Bakk Pase
OCR
Inttrcion Compleitty
L.M--towd Scortag
Viaal Elhoss
Vieaa Semartic
IC-haad Amoozust
trmplate
Blesr
Iakn Clas Variase
Figure 4: The complete pipeline for VLIT data construction. Data collection (refer to Sec. 4.2.1 is the stage of collecting annotated data; Instruction Resspesss Generation (refer to Sec. 4.2.2 uses GPT3.5 to generate multi-turn conversations based on preset system prompts, manually edited in-context examples, and collected visual clues; Quality control (refer to Sec. 4.2.3 filters the generated data based on the principles proposed in Sec.
4.2.1 Data Collection
Multi-modal annotation data used to generate VLIT data typically originates from various visual tasks, such as recognition, segmentation, and classification. These tasks often serve as the source of supervision for training traditional machine learning models, providing large-scale, high-quality data, which makes them highly practical.
When collecting data, it is essential to thoroughly consider factors such as the type, source, scale, and quality of the annotation data. This can ensure that the collected information generates VLIT data that meets the requirements of correctness, diversity, and complexity. Specifically, annotation data that has been manually reviewed and widely used has higher credibility and will not cause subsequent errors due to annotation inaccuracies. Simultancously, different types of annotation data can usually be extended to define various lasks, significantly enhancing the diversity of the final VLIT data. When multi-modal data from the same source contains different types of annotations, it can even generate more complex contexts and new task types on top of the original simple visual tasks, ensuring complexity.
In this paper, we provide a simple implementation to quickly construct VLIT data, as shown in Figure 4 Specifically, in data collection, we first select COCO 2014 (Lin et al.) 2014] as the image source, and (caption, object, altribule, OCR, visualQA] as the selected sources of annotation data (Antol et al. 2015 Patterson & Hays 2016 Veit al. 2016).
4.2.2
Respoons Generation
Currently, using LLMs (such as ChatGPT, GPT4, and LLaMA) to assist in generating VLIT data has become mature. contrast, annotation adaption is significantly less flcxible, so we refer to self-instruct to generate VLIT data. Specifically, when providing an image and corresponding annotation data of various types, it is necessary to design appropriate templates for each type of annotation data to ensure that the inserted
11

---

Published in Transactions on Machine Learning Research (07 2204))
annotation data conforms to standard language representation. By combining all annotation data sources, detailed and accurate description of the image can be obtained, which is an understandable visual clue for language-only LLMs. To ensure the diversity and complexity of the generated content, it is necessary to write sufficiently detailed prompts to guide the LLMs, and manually write a sufficient number of example instruction-respouee pairs. In the system prompt design, we follow the principles proposed in Sec. [.3.3 and guide the generator to generate more diversified and complex using system prompts with different additional constraints (e.g., extended knowledge, negative samples, multi-round dialogues, etc.). At the same time, the context examples we design are sufficiently sufficient, meet the characteristics of comprehensiveness and randomness, correspond to the boundaries involved, and provide appropriate examples of all possible scenarios so that the generator can correctly understand the meaning of the prompls. In addition, lo ensure sulficient diversity during generalion, we also redundantly set the key points of the system prompts and context examples, that is, repeat certain key points multiple times (such as not generating uncertain content, avoiding fuzzy answers, etc.).
pairs by flexibly
In implementation, we employ GPT-3.5 (OpenAl 2023a) as the generator for LLM, and formulate corre- sponding system cues and in-context example: 5 Specifically, based on the annotated data and different system prompts, we introduce different granularities of tasks, add negative samples, generate multi-turn complex dialogues, and extend external knowledge in the generation process to enhance the diversity and complexity of the generated VLIT data. In addition, to further enhance diversity, we set the temperature value of GPT-335 to 0.8 and the number of generations to
4.2.3 Quality Control
Owing to the potential errors in LLM itself and the uncontrollable nature of the generated data distribution, it is crucial to conduct quality control on the VLIT data direetly output by LLM. Recently, numerous studies have started to address this issue and have filtered the dala based on established experience (Li el al. 2023b k) In this paper, we design appropriate data filtering strategies based on the principles proposed in Sec.
and carry out local and global quality control on the generated VLIT data.
Usability. Regarding the readability mentioned in Principle | we need to conduct rcadability analysis separately for the text and images in VLIT data. For text readability, we use the overall generation probability of the text calculated by a trained language model (e.g., Word2Vec (Mikolov et al. 2013) to achieve quantitative evaluation.
I N
Readabililytent
Σ log(P.). 1=1
(2)
N
For image readability, we focus on the overall clarity of the image, so we refer to a related mature visual task (i.c., no-refcrence image quality assessment, which aims to assess the perceptual quality of images by human subjective perception) and corresponding method (i.e., MANIQA ( (Yang et al., 2022)) for quantitative evaluation. In accordance with the two aspects mentioned in Principle 2 it vital to assess the degree of alignment between visual and textual information in the given candidate VLIT data, as well as the overall
accuracy. In this paper, we opt for the classic pre-training model, CLIP (Radford et al. 2021 b, to implement the matching function. Specifically, we calculate the representations of visual and textual information separately and evaluate the degree of alignment through similarity. For accuracy evaluation, we introduce MLLM (i.e., LLaVA 1.5 (Liu et al. 2023b) and manually written guiding examples to conduct multiple accuracy assessments on the given VLIT data, taking the highest frequency as the final accuracy score.
Diversity. To prevent the influence of the inherent correlation of original tasks themselves (c.g., detection and classification) when evaluating task diversity, we attempt to establish new task categories for the generated instructions. In practice, we employ a zero-shot labeler TagGPT [Li et al. 2023d] to assign category labels to the instructions based on their content. Among them, TagGPT is labeler that can be cold-started. It. generates labels to samples by introducing the semantic understanding capability of LLM, retains high-quality labels based on indicators such as label coverage and redundancy, and finally assigns labels to samples. When
5 Seee the Appendix for all system prompts used during the generation process
12

---

Published in Transactions on Machine Learning Research (07/2024)
evaluating instruction diversity, since there is still some semantic similarity in the instruction text for tasks with the same task type, it is not possible to directly use semantic calculations to evaluate instruction diversity. Although some methods try to avoid this problem by counting predicate verbs, some noun information also affects the calculation of diversity. In this paper, we use a graph- basph-baeed method to calculate instruction diversity, namely, converting instruction texts with the same task label into an element graph based on co-occurrence relationships after removing stop words and using the overlapping community partitioning method (i.e., Ego-Splitting (Epasto et al., 2017)) to count the number of clusters to evaluate instruction diversity. Ego-Splitting is a generative model with overlapping clusters in which it can reconstruct most of the clusters exactly by using the connected component algorithm as non-overlapping partition algorithm both in the ego-net clustering phase and in the persona graph clustering phase. When evaluating response diversity, we directly utilize an efficient. unsupervised lext represenlation method (i.e., SimCSE (Gao el al., 2021 to represent response texts and employ a density-based clustering algorithm (i.e., DBScan (Schubert ct al. 2017)) to mine semantic clusters as indicator. SimCSE generates unsupervised positive samples by randomly sampling dropout masks to improve contrastive learning performance. That is, during the training phase, the same sample is imput into the same encoder twice to obtain two different representation vectors 2, 2', 2² will be used as positive sample, then the training target of the model is:
log Σ
(3)
sim (//)/T 1/r
f==
DBScan is clustering method that treats clusters dividing high-density areas by low-density areas in the data space, and divides sufficiently high-density areas into clusters. It cycles through labeling candidate samples to corresponding clusters and deleting them, and finally achieves clustering after all samples are labeled. To ensure that images can also provide rich and diverse semantic information as mentioned in Principle G we first refer to the evaluation method of response diversity. We employ a mature visual encoder (i.e., CLIP) to obtain image representations, and evaluate visual semantic diversity by clustering (e.g., DBScan) these visual representations and counting the number of clusters. In addition, to evaluate the number of different object categorics in the images, we utilize an open-set object detection algorithm (e.g., YOLO-Worke (Cheng et al. 2024 to count the number of objects and evaluate the object diversity of the images.
Complexity. To evaluate complexity, we first use classical semantic dependency analysis to count the number of clements in the instructions (Märquez et al., 2008). and then utilize I.LaM--2 as the discriminative model to provide a quantitative difficulty score under the guidance of the given sample. Specifically, semantic dependency analysis refers to the task of extracting important semantic elements in a text and assigning correspouding roles. In this paper, we use mature tool for semantic role extraction as a data source for counting the number of instruction elements. When evaluating the complexity of instructions, similar to existing work, we inpul given instruction-reply pairs into LLM (i.e., LLaMA-2) and rely on manually scored examples to give all candidate pairs complexity score of 1-10. For the evaluation of visual semantic complexity mentioned in Principle g we also refer to classic visual task (i.e., automatic image complexity assessment, which is a task that simulates human subjective evaluation of image complexity.) and use a corresponding mature solution for visual complexity evaluation (Feng et al. 2022).
Balance. When assessing balance, we can utilize the obtained task labels to calculate the variance of sample sizes across different tasks, serving as a balance indicator. Simultaneously, by randomly and uniformly sampling from various task labels, we can acquire VLIT data with category balance.
5
Experiments
In this section, we will thoroughly examine the correlation between the evaluation indicators proposed in Sec 4.2.3 and the performance of tuned MLLMs. Specifically, we select two common VLIT data sets and
httpps:/hanlpphanmt hankss. www.:haanccccccnnnci...e. com/demos/sdp..tee
13

---

Published in Transactions on Machine Learning Research (07/2024)
the generated VLIT data in Sec 1.1 amounting to total of three types of VLIT data, which used for instruction luning of MLLMs with varying architectures. Subsequently, by jointly analyzing the evaluation indicators of the VLIT data and their performance in different tuned MLLMs, we can ascertain the validity and applicability of these ovaluation indicators.
5.1
Implementation Details
We list the relevant settings these experiments in detail as follows.
5.1.1
Multi-Modal Large Language Model
To fairly verify the influence of instruction data on tuning results, we choose three MLLMs with different frameworks, respectively:
LLaVA (Liu et al. 2023c). is a representative MLLM consisting of visual encoder, an LLM, and a trainable linear projection layer. It shows good session performance after two-stage training (i.c., pro-trained feature alignment and end-to-end instruction tuning). In this paper, use LLaVA library / for implementation. We select LLaVA-7B as MLLM, in which the visual encoder is CLIP (ViT-I-14) (Radford et. al. 2021 and the LLM is ficuna-LL And we refer to the original paper setting and directly use the open-source checkpoint" as the pro-trained MLLM.
BLIP-2 (Li et al. 2023h), a new structure Q-former is proposed to connect visual information with textual information. After Q-former has been trained on three pre-defined image-text tasks (i..., matching, contrastive, and generation), visual information can be extracted by trainable queries and combined with the fully connected layer to realize the conversion of visual features to textual features. In this paper, we use its official library, Le., LAVIST and utilize checkpoints pre-trained on the COCO-Caption as the initial MLLM. Its visual encoder is CLIP and its language model OPT-2.7E ZZang et al. 2022
OpenFlamingo (Aaadalla et al. 2023); is an open-source version of Flamingo (Alayrac et al. 2022 It includes visual information conversion module composed of visual encoder and perceiver resampler, which convert images into few visual tokens and put them into LLM. In this paper, use its official version: OpenFlamingo-33, in which the visual encoder is CLIP (ViT-I-14) and the LLM is OPT-1.3 [Zhang et al.
2022 And we utilize the checkpoin 14 of theenllaming--3B. which is pre-trained on MMC4 (Zhu et al. 2023b and LAION (Schuhmann et al. 2022), as the initial MLLM.
These MLLMs are all trained utilizing Telsa V100 (32G) GPUs with the Python environment and other detailed settings (e.g. hyperparameters) of the three models can be found in Section A.3 in the Appendix.
5.1.2 Vision-Language Instruction Tuning Data
To validate the correlation between the quality of VL.IT data and the proposed characteristics, as well as the utility of the suggested VLIT data construction pipeline, we utilize the example VLIT data constructed in Sec44.1 and select LLa VA ([LaVA-Instruet15500) (Liu al. 2023c) and MIMIC-IT (Li et al.) 2023b) comparative data. In the quality evaluation process, to ensure fairness, we the smallest dataset size the scale for all the test VLIT data and randomly sample VLIT datasots larger than this scale. In addition to incorporating insights on high-quality VLIT data during construction, we also rely on the designed quality control strategy to filter the sample data constructed in this paper, further verifying the accuracy and validity of the conclusions presented in this paper
https::/gtthubb.co///
http:::/ggthua...ooo///ttttttttt flamingo
hhttps:/hhuggingee
See Sec.
the Appendix for more generated instruction-responee examples
14

---

Published in Transactions on Machine Learning Research (07/2024)
Table Performance of different VI.IT data in three MLLMs. w/o means without.
MLLM
VLIT Data
Overall
SU
11
IL
LA
IC
SR
I1R
VR
TR
LLAVA MIMIC-IT
28.0
25.3
28.3
33.9
24.3
25.8
27.6
10.0
10.3
18.2
26.3
24.4
25.4
30.7
24.3
23.1
25.7
10.0
14.8
21.5
Ours
28.3
25.0
27.0
35.0
24.3
25.8
26.3
10.0
17.2
45.5
Ours(QC)
28.7
26.3
29.1
36.1
26.5
26.4
26.3
20.0
18.3
47.1
LLaVA
-w/o usability
25.7
28.7
35.8
26.5
26.9
26.3
20.0
17.8
46.3
-w/o diversity
28.6
25.0
28.7
35.8
26.5
26.4
25.7
20.0
19.1
47.1
ww/ complexrity
28.3
25.7
27.0
36.1
25.2
26.4
25.7
20.0
19.1
47.1
-w/o balance
28.3
26.3
28.7
33.9
22.7
26.4
24.6
20.0
20.6
40.3
LLaVA
27.3
25.7
25.3
33.9
22.7
22.1
22.4
20.0
10.3
8.6
MIMIC-IT
26.5
24.1
24.4
31.2
20.1
21.7
21.8
20.0
13.8
9.1
Ours
27.5
26.3
26.5
33.9
25.2
23.6
23.7
20.0
13.8
9.1
Ours(QC)
27.7
27.0
35.0
25.2
25.8
25.7
20.0
19.4
17.5
BLIP-2
-w/o usabilily
27.8
26.3
27.0
34.4
25.2
25.8
24.6
20.0
17.2
15.9
-w/o diversity
26.3
26.5
35.0
25.2
26.4
25.7
20.0
17.8
17.5
ww/o complerity
28.0
25.7
27.0
34.4
26.5
26.4
25.7
20.0
18.3
17.5
-w/ balance
27.1
27.7
28.7
33.9
24.3
23.1
23.7
20.0
19.4
11.7
LLaVA
25.5
25.7
27.0
30.7
22.7
23.6
25.2
10.0
13.8
21.5
MIMIC-IT
23.9
24.4
33.9
22.7
24.0
24.6
20.0
10.8
20.4
Ours
28.1
28.2
26.0
33.9
21.4
25.5
22.4
17.2
27.3
Ours(QC)
29.1
30.5
29.6
37.9
26.5
27.5
24.6
20.0
20.6
30.7
OpenFlamingo
-w/o usability
29.4
29.6
37.1
26.5
26.4
24.6
20.0
19.1
28.9
w/o diversity
28.9
29.4
29.1
37.9
26.5
28.1
23.7
20.0
19.8
30.7
w/o complexity
28.6
28.7
28.7
37.1
26.5
26.4
23.7
20.0
19.8
27.3
w/o balance
28.4
31.7
29.6
36.1
24.3
25.5
22.4
20.0
20.6
15.9
5.1.3 Metrics
To more comprehensively evaluate the understanding and generation capabilities of MLLMs, we employ SEED-Bench (Li et al. 2023c) the evaluation metric for MLLMs after instruction tuning. SEED-Bench comprises large number of multiple-choice questions, and we have selected nine tasks related to vision- language tasks for evaluation. Specifically, as shown in Table "SU" represents Scene Understanding, 'II' denotes Instance Identity, 'IL' signifies Instance Location, 'IA' stands for Instance Attribute, 'IC" indicates Instance Counting, 'SR' refers to Spatial Relation, 'IIR" corresponds to Instance Interaction, "VR" represents Visual Reasoning, and "TR' denotes Text Recognition.
5.1.4 Comparison of VLIT Data
The quality assessment of three different VLIT data and the performance of different MLLMs tuned with them can be seen in Table 2 and Table respectively.
Firstly, by observing the results in Table 2 we can discern that the example VLIT data constructed this paper significantly outperforms the other two in the majority of metrics. Simultaneously, the VLIT data, after being filtered by the quality control strategy, exhibit better performance, demonstrating that the characteristies of high-qualin VLIT data summarized in this paper accurate, and the quality control scheme designed in the text is reasonable. Second, we perform ablation experiments scparately by ignoring different filtering metrics (i.e., usability to Principle 1 2h, diversittyreefer to Principle 3 complexity (refer lo Principle 7 and balance(refes to Principle I0 in quality control filtering. The experimental results of three MILMs show that filtering based on the proposed metrics can indoed improve the overall performance of MILM. When carefully observing the changes in each sub-task indicator in Table 2 it can be found that when different types of filter indicators removed, the corresponding related task evaluation indicators will
15

---

Published in Transactions on Machine Learning Research (07/2024)
Table 3: Quality assessment results of different VLIT data. Among them, the Text Readability (TR), the Matching Degree (MD) and Correctness (C) of a single sample the average of all samples VLIT data; the Task (T) diversity index is the number of lasks, the Instruction (I) diversity index is the number of clusters in the element graph, and the Response (R) diversity index is the log value of the mumber of spatial semantic clusters; the
(O/G) complexity index is the number of objects, the Instruction (I) complexity is the average score given by LLM; the balance index is the variance of the number of samples in different. tasks. t represents the higher the better, while 4 represents the lower the better. ICC is the inter-class correlation coefficient, and Single Measurement (SM) and Average Measurement (AM) are two kinds of mcasurements.
Quality Evaluation
ICC
Single
Overall
Dataset
Usability
Correctness
Diversity
Complexity
Balance
SM
AM
TR t
MD t
C
T
I
R f
O/G
If
LLaVA MIMIC-IT
0.84
30.7
91.1
9
14.5
2.2
1.6
6.5
14.6
0.8083
0.9267
0.83
30.9
93.4
9
13.3
2.3
1.3
4.4
10.5
0.7268
0.8887
Ours
0.84
33.2
94.6
10
20.5
2.3
1.9
6.8
16.8
0.3036
0.5667
Ours(QQ (QC)
0.84
34.1
94.8
10
22.6
2.0
7.1
12.3
0,4281
0.6920
-w/ usability
0.84
33.1
94.6
10
22.5
2.5
2.0
7.1
12.7
0,4084
0.6744
-w/o diversity
0.84
34.1
94.9
10
20.5
2.3
1.9
7.0
12.4
0.4374
0.6999
m//o complexity
0.84
34.1
94.8
10
21.8
2.4
1.9
6.9
12.4
0.3950
0.6620
w/o balance
0.84
34.3
94.9
10
22.8
2.5
2.3
7.1
16.7
0.3212
0.5867
Table 4: Performance of VLIT data under different visual information qualities in three MLLMs.
VD
Overall
Setting
IR
VSC
SD
OD
LLaVA
BLIP-2
OpenFalmingo
Original
0.5343
44
5.5
48.9
27.6
27.0
27.8
Readability
0.2245
44
5.5
48.9
24.3
23.9
24.7
Diversily 4
0.5339
27
3.3
48.6
26.8
26.3
27.2
Complexity 0.5295
43
5.3
42.2
27.4
26.7
27.7
change. The more obvious ones are "diversity & instruction recognition' and 'complexity & scene recognition". In addition, due to the removal of the "balance" filter indicator, some sub-tasks (e.g., visual reasoning and sense understanding) account for larger proportion of the VLIT data, which indiroctly Icads to significant improvement in the performance of the corresponding tasks. At the same time, the performance of the tasks corresponding to a smaller proportion of data decreases. To ensure the robustness and credibility of the experimental results, we evaluate the consistency (i.e., the inter-class correlation coefficient (Koo & Lij 2016)) of the performance of the same VLIT data in different MLLM frameworks, and the results are shown in Table 3. It. can be observed that the performance of the same dataset in different MLLMs is essentially or highly consistent. This further ensures the reliability of the conclusions we derive.
In particular, since the three VLIT datasets selected are all based high-quality data sets mainly including COCO, the evaluation results of their visual data (i.e., image readability, diversity, and complexity) are basically the same, making them unsuitable for direct comparative analysis. Therefore, we choose to discuss it independently. To investigale the impact of images with different qualities on VLIT data, we filtered the dataset constructed in the paper and manually constructed VLIT data containing images of different qualitics, shown Table Spocifically, we used image distortion (corresponding to sotting 1), adjusted the number of clusters and objects (corresponding to setting 2), and re-splitter based on a predetermined visual semantic complexity threshold (corresponding to setting 3) to form several equally sized subsets of VLIT
16

---

Published in Transactions on Machine Learning Research (07 (07/2022 2024)
Text Readability
Matching Degree
Correctness
0.6
Task Diversity
0.4
Instruction Diversity
Response Diversity
0.2
Object Complexity
0.0
Instruction Complexity
-0.2
Balance
overall
Scene Understanding
Instruction Identity
Instruction Location
Instruction Attributes
Instance Counting
Spatial Relation
Instance Interaction
Visual Reasoning
Text Recognition
Figure 5: Correlation analysis between the quality control indicators of VLIT data and the performance of tuned MLLM in different downstream lasks.
data for training three baseline MLLMs. The experimental results based on VLIT data of different qualities are shown in Table g It can be observed that VLIT data of higher quality have improved VLIT performance to varying degrees, dernonstrating the crucial role of high-quality visual data in VLIT performance. Among them, the most important the readability of the image, followed by the diversity the image, and the complexity of the image has relatively little impact on VLIT.
Simultaneously, as shown in Table data under the evaluation indicators proposed in Sec 4.1 To further verify the rationality of the proposed evaluation indicators, we conduct a correlation analysis between each quality control indicator and the performance of MLLM on different tasks after utilizing its tuning. Since it has been demonstrated above that all VLIT data exhibit good consistency across three different MLLMs, we use the performance mean of VLIT data on three MLLMs for analysis, and the results are displayed in Figure 3 Upon observation, we can discorn that the results of most quality evaluation indicators are directly proportional to the performance of the tuned MLLM rewnsttream tasks (balance inversely proportional because lower indicators indicate better quality). Except for the spatial relationship indicator, there is only certain correlation between object/ sojject/raann ggraullariy complexity and instruction complexity. This may be attributed to the fact that spatial positional relationships typically only have certain correlation with the number of elements appearing in the instruction and the related instructions.
compile statistics the performance of the mentioned VLIT
6 Challenges and Future Directions
Despite the substantial amount of data and research available concerning vision-language instruction tuning, the field remains in its nascent stages. Consequently, there is still much to explore and numerous challenges to overcome. We highlight several pertinent directions for future investigation.
Mature Multi-Modal Large Language Model. Recently, with the disclosure of GPT-4V (OpenAl 2023b), a large number of MLLMs have matured visual understanding capabilities, and they no longer need use text describe visual signals for language-only LLMs generate VLIT. However, we still need note that hallucinations, information loss, and other phenomena more serious in MLLM than in LLM.
17

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
PROTECTING WILDLIFE FOR A HEALTHY PLANET
tageteer
The image diephays serene underwater soene with focus on macine consrvation. The backdrop constats of a gradianc of bine hues, representing the occant's is deysh, with a lighter blue al the top transisioning to darker shades toward she bottom, suggesting the increasing depth of the aco. Centrally positioned in the image is a sea turtle, angled such that it appeare to be swioming upward toward the left side the frame. The tuetle is captured in clear detall, showing Overlaid on the image, in the utatemeut emplonising she theue of wildlife peotection being crucial for the bealth of our planet.
GPT4V:
patterned shell, Aippers, and head, all charasteriatic of portion, is bold while tex: that ronds "PROTECTING WILDLIFE FOR A HEALTHY PLANET." Thio is clous
turtle's anatomy.
Below the headline, in smaller white text. is an additional mesage that states, "WWF works in unity with many to achieve lasting conservacion results. Together, we can protect and rostors species and their babitats." This further emphastza the purpose of the World Wide Fund for Nacure (indicated by the WWF an)nyya), highlighting their collaborative efforts in conservatinn and restoration of wildlife and habisut..
the the the posttion the turtle, the direction the turtle's movement.
Table 5: An example of GPT4V incorrect description of the given image. The location of the error has been marked with an underline in the original text.
Instruction: Please indicate what is on the license plate of the car in the picture.
Mini-GPT4:
The license plate on the car says "GO". X LLaVA 1.5: The license plate on the car says "TIGO." x
Qwen- VL: The license plate of the car not visible in the image, so cannot determine what it says. X
Instruction: Please answer the question: "Are there more birds flying the picture birds standing on the ground?".
There are three seagulls slanding on the sandy beach, with one of them holding its beak open and the other two looking at The water the background calm and blue. There are no other birds in the picture. X
Mini- GPT4:
LLaVA 1.5: There are more birds standing on the ground the picture than flying. /
Qwen- VL: There are more birds flying in the sky than standing on the ground in the picture. X
Table Difficult examples (fine- gaane-rained content understanding and complex reasoning) for existing commonly used MLLMs.
For example, shown in the examples in Table although GPT4V can describe the image very well and in detail, it still has some errors. Therefore, how to reasonably use only MLLM to guide the generation of VLIT data and achieve quality control is direction worth studying.
Difficult Sample. Existing MLLMs have demonstrated strong instruction execution and generalization abilitics in some simple tasks. However, shown in Table 0 they generally have shortcomings when dcaling with some special difficult samples, such as fine-grained content understanding, complex reasoning, and multi-modal content editing (i.e., image/vider editing). The thouugt (Wei et al. 2022) and in-context learning (Dong et al., 2022) can alleviate the occurrence of the above problems to some extent, but the cost of obtaining nugh-tuality annotated data too high, and the model computation cost brings also unacceptable. Therefore, how to efficiently solve such difficult samples is a problem worthy of attention.
18

---

Published in Transactions on Machine Learning Research (07/2024)
Instruction: Please describe this image in detail.
This image shows a tree with several ripe peaches hanging from its branches. The tree is in field with mountains in the background. The peaches are bright red color and appear to be ripe and ready be picked. The leaves on the tree are deep green color and there are some small flowers on the branches. The sky is a bright blue and there are a few clouds the distance. The overall mood of the image is peaceful and serene.
Mini-GPT4:
Table 7:
An example of hallucination phenomena that MLLMs experience when describing an image (mistaking apples for peaches and hallucinating flowers and clouds).
Selective Forgetting. As pointed out by Zhai et al., fine-tuning LLMs can easily cause them to forget their original capabilities (e.g., the classification ability of the original CLIP) or instructions, which is catastrophic. In fact, this phenomenon is prevalent in LLM, and abundant research has demonstrated that supervised fine-tuning in downstream tasks frequently causes overfitting of the LLM in the corresponding lownstream tasks, leading to the loss of some or all of its original capabilities. Currently, existing studies generally address this issuc by regulating the learning rate and minimizing the overall loss, achieving certain outcomes. The training data in instructional tuning serves the learning object of the model. It is equipped with appropriate settings, such as the repetition of old samples and the balancing of task ratios, to mitigate the forgetting phenomenon to a certain extent. Nonetheless, further research is necessary to tackle this matter.
Hallucination and Bias. Similar to LLM AOpenAl 2023a Touvron et al. 2023a), the MLLM model is prone to hallucinations and biases. As shown in Table 7 Hallucinations occur when the MLLM generates non-existent content, as unconsciously imitates data in its training corpus. In the multi-modal environment, this issue has resulted in the creation of non-existent visual targets, which can significantly mislead users. The issue of bias arises when content generated caters towards specific groups or delivers prejudiced outcomes, compromising user experience, impartiality, and equality, among other aspects. This is typically the result of partial training data distribution. To address the problem, this paper suggests employing additional, more detailed image-text fully aligned samples, numerous negative samples, as well as data quality modules. However, obtaining corresponding data on a large scale can often be difficult due to the absence of detailed annotations.
Limited Emergence. As significant purpose of instruction tuning, instruction generalization is an advantage that. LLM (and MLLM) should possess, stemming from the emergence phenomenon. Tn the multimodal scenario, emergence shows more characters, such as supporting visual objects that have not appeared. However, recently, a large number of MLLMs have not shown good performance in either of the above two scenarios. Therefore, deeper consideration is needed on how to design MLLMs and corresponding instruction tuning data.
7 Conclusion
In this paper, we have compiled recent research vision-language instruction tuning, presenting succinct summarics and analyses, along with catcgorization, details, and shortcomings of existing literature. To construct high-quality vision-language instruction dala, we suggested several construction principles and a complete set of construction processes based on the existing research summaries, Extensive experimentation has demonstrated that MLLMs, when tunod on the instruction data constructed via our proposed method, yield better overall performance. To conclude, we lay out the remaining issues and potential future research directions.
19

---

Published in Transactions on Machine Learning Research (07 (274))
Table MLLMs with VLIT stage. "Training Paradigm" refers the training phase included in the MLLM, where FA is short for feature alignment (MLLMs lacking this phase either the pre-trained ckpt directly do not have this phase). 'Pure Text' refers to whether the VLIT data contains pure text instructions. 'Data Source' refers to whether the data used for instruction tuning contains instruction data from other existing VLIT data. 'Self Construction" refers to whether this MLLM is tuned by VLII data constructed autonomously.
Name
Training Paradigm
Pure
Data
Self
URL
Text
Construction
Model
LaVV [Ta et X 20236
PA
VLIT
Pooe
Mini-GPT4 M0
FA
VLIT
Pure
mPLUG-OW
FA
VLIT
Hybrid
VideoChat
PA
VLIT
Hybrid
VisionLLM
PA[Object Detection) VLIT
Hybrid
X-LLM Clisn
Hybrid
ChatBinde
PandaGITT
VLIT
MedVisTTTT
20231
VLIT
Pure
GPTMTuule
Yang
VLIT
LLaVA-Med
EA VLIP
Pare
Ying-Via VLM Lig (POPIR))
FA VLIT
Hybrid
Otiar Li el KUr-0n
VLIT
Hybrid
Video-C Care-Caaree MA3 Ht
VLTT
Pure
OFAmaltiinee
VLIT
Pure
Vide--LaakA
FA VLIT
Hybrid
VALLEY [Tuo
PA VLIT
Pare
MulsiModd
2022
Hybrid
Hybrid
LaVIN
MACAWaRRR
LAMM 1t
OpkGLa
VLIT
LaVAR
PA VLIT
Pore
Shikes Piri 211
PA VLIP
Hybrid
Clewer Pismmrge Loinis
20:4h
FA VLIP
Pure
mPLUG-DD.d Ne..
1022m
EA VLIT
Hybrid
GPT4Rol Yo.et
(Region Adaption) VLIT
Pure
SVIT (Zhah Mi
FA VLIT
Pure
KOSMUS-<4 Femin
PWPM
FA VLIT
Hybrid
BuboCPT Minoo 1i 1il1Pe5
FA VLIT
Hybrid
UFF
ChatSpet
ImageBind
(A(Tarameter--fRtt Tuning) + VLIT
Hybrad
Lyas ZZeng (Pa5510
(A Matching)
VLJTT(Multi-Taak Tuxing)
BLIVA
pin
Hybrid
StableLixeV
FA VLPP
Prrr
Qwen-Vi [eat
FA VLLT(Muues-Tae Tuning)
Pure
VIGC (Waher 11
VLIT
Hybrid
PaintL.b6 Xu mt
FA VLIT
Pure
MI-IM-Dae-Ti Daraaaaiiiinn mucunne
(ala m a1 2023d
VLIT
Hybrid
MM [uu el 61 2023
FA VLIT
Hybrid
NEx1->P Win 2099
FA FA VLIT
Pure
PVTT (Chen 00 al 2022m
Pure
LLaVA
2023c
Hybrid
TixtBid
1i al. P0leey
FA
(laterloavee Generative)
AnyMAI.
(MuT((MulsI-Sak Tuning)
SpacklosChal
Hybrid
LLaVA
Hybrid
CogVLM (Weang
FA VEPE
Hybrad
Purret (Yoh (riiiie
FA VEPT
Hybrad
MinGPP-y2 1- m1 m
20236
FA (Mull-uui Tuning)
Hybrid
GLaMM REvdledd 28
PA VLPP
Pure
11
SRED-LI- L02
PA VLIP
Hybrid
OrterlID
[L A11
PA VLIT
Hybrid
mPLUG--Wwz
3023
FA VLIT
Hybrid
A
Appendix
A.1 Details of VLIT Setting and Data
For more details, please see Table Table and Figure
A.2 Vision-Language Tasks
We summarise the visual tasks covered in all current MLLMs and datasets. Since the names used describe tasks are inconsistent across different tasks, we unified similar tasks as much as possible. All statistical results are shown in Table T0
20

---

Published in Transactions on Machine Learning Research (07/2024)
Table VLIT datasets. "G" (Global), 'R' (Regional), and 'T' (Interleaved) represent the different. compositions of the image, and 'I-R' is the abbreviation of Instruction-Respee "Type' refers the stratogy used for data construction, where 'AA' indicates annotation adaption, and 'SI" indicates relf-Instruct. indicates the absence or absence of clear information.
Name
Model
Inst
Tmog
#5Tmge
Vid
#Vidd
# #1--R
G
Tt
VA-nnttrudt-100 Lines
2023
LLaVA
B0K
168K
alag
Min--GPTA
3.5K
3.5K
VhaeoChat (1
VideoChat
7K
18K
X-LLM (Chii Marrils (airirss
X-LLM
3.5K
10K
paudagpi
PundaGPT
AA
161 SK
PMC-VQA
MedVLuT-TE/TH
149K
227K
GPT477un'
GPT-Tools
71K
M
Ying-VLM
MIMICIT
8.1M
Video-Chut
POPH
Videa-ChatCPP
MULTHNNTTRUCE
235K
Video-LLaMA (2hary
Video-
AA
63.KK
11K
164K
Valley
Suu-IInatuuct-77
VALLEY
S1
TSK
75K
MuitiModdal-GP Commsyya.. 1:10
MultiModal-GPT
AA
250 3K
Dall m0 (2112))
InstructRiip
AA
MACAW-LLM
20724
MACAW-LLM
91
LOK
9.8K
1IRK
LAMM YYu et RUPS)
LAMM
31
106K
OpEGLM
OphCLMM
20K
LLaYAR
LLaVAR
Shikra-
PF-1M
Flamingo
GPTaRel
SVIT(MMMLLM)
XOSMO8-2
BuboGPT
BaboOPT
AA
SK
MOVLID Mrrn
ChatSpos
AA-SI
1.2M
3M
Lyux Zenk
202391
Lyrxx
AA
Multicrnii
2023
LMPN)(IPNN
31
7,38
IS Work Suw.eee 20250
ASM
AA SE
LIDI
8.8B
1a
Chestor
AA-SI
1.77M
4KT.
II al. 20/411
StableLLaVA
AAA
128K
MeHalDDtecd ConleT
2023
31
4K
16K
VIGC Wang Mr KIZA
VIGC
A A
Pointhemm
PoIntI.LM
730K
72.9X
Ainoya 20230
HLLLM-Datenngin
23K
NEET-GPT
NETT-GPT
PVIT
PVIT
ToxsBind
DrsamLI..
21/2
DrewmL..M
120K
120K
AsyMAL
(Moon
(275)
AnyMAL
AA
210R
InternLM.X. MootnMMMXCCoie
PUPE
InternLM
A
X
SparklesChatt
31
7K
4.4K
SparklesChat
31
12:9K
2K
Ferret You mi. 20211
Furzet
AA=S5
95X
MiniGP
2023
MinGGPTw3
AA-S1
ComVinr
12 Aaraap
32K
Granl)
Woog
110K
A.3 Experimental Details
A.3.1
Settings
In this paper, we used three different MLLMs to tune on VLIT data. Table 11 Table 12 and Table respectively list all their hyperparameter settings during instruction tuning.
A.3.2 Generation Details
System Prompts.
Depending on the requirements for the generated command-response pairs, we use the following system prompts as setting input GPT3.5.
You are artificial intelligence assistant. You'll be given text describing image, and based on that text you'll be asked to generate conversation around the object circled by square box in the image. The conversation context is a question-and-answar session between the visual AI assistant and the user around the image. Specifically, the generated dialogue needs to meet the following requirements: 1. Answer confidently to the content described in the text. 2. The conversation must about the object in the circle. 3. The visual artificial assistant's tone that sees the picture, not the text description. It necessary ask more complex questions about the content in the image, appropriately combine it with external knowledge that does not appear in the image, and expand the discussion the people, places, brands, and other information
21

---

Published in Transactions on Machine Learning Research (07/2024)
B.g. MinGCPT-4 (Zun ct ml.
2023a
X-LLMM (Chen et al.
10236
MULTIS Zhao ct al
2023e
PMC-VQA (hwng mi
20233
S--T [IA
20235
INSTRUCT
Xu cl a
MaiGPT-v2 Chen
20230
MultihonaFGG"
Goung
2025
Anactation Adaption
ILstr
Dar et nl.
2124
PF-IM (Chen et al.
20236 Gl Kitl Zhang et al.
2025e
GRIT-2IM Pring
20253.
MGVLII Zaan ni.
212266
Lynx Zenn ot nl.
14 [T4 e6 al
[[ al 20231)
Any.
Morn et ml 2022
InternLM-
ommpueer
(Zhanng al.
20230
Ferrel
You al.
Egg.. LLaVA [Luu al [2023e
VideoChal eL
20123]
DeIGPT Pi
2023
MIMIC-IT [L et al (2123 V
Clao-C (Jinez et ql.
2023
VALLEY Lito et
2023bb
Macaw-LLAA Lini a
LAMM Vin nia (0224
OpECIM (Cao
ILaVAR Zhany et, nl. Maria
Shilkra
CTeen of mi 21727e
SVTT (Zhao ot 2024m
4331
MGVLID Zago
20225
LMEge
ni. 2122
et 212
Constrastion
Strateggy
Solf-Instrret
M-HallDesoT Gii nd at
2023
PoinTLLI Xi mu
2022mm CTPM (TT) vt m).
20223
GPTVQA Zino el al; 20250]
T2M
et all 20125
Moell (Wa al.
PVTT (Chen at. (2223
"TeetBind
[Lt
Dreamm L.
Dong al 2025
InternI omsooser Khinng et nl.
2122l y Sparales Dinlogge VG/OC [inan et 2025b
Forret You nl. 2010 MinGPTSZ (Crii
2023d] ComVins
Di 2023
GranD Rashevd es al, 2023
INTS- INSTUCSTTU
Waing el al.
2023b
Egg. mPLUG-Owl (Ye es al
20233b
PandGGPT Su el al. 20223
VideoLLaMA [Zaanic al 12125
LaVIN (Lav et a 12ee3
ButoGPT ZZho
2023kc
Data Mixing
Imageliind LL.t [lan et ni.
2024
AS-IB Wang et al 20125
BLIVA [Tu et m. 2024b
VIGC (ann et 2i
Owen- VI. (Ba
2025
Llavii.5
VI.I Data
CogVL.II Wang es al. 20123e
OtterilD La al.
2123k
SEED LLaMX (Ge ot ml. 2023
Egg, ChatSpot (Zhmo et al.
CPT4Rol (Zhung at
20236
SVIT (Zlao et all 20255
GRTT
(Peng el Rl. 2029
Region
MGVLID Maso
20226
AS IB
(Wang
212ld
Shikra
Clea et al
20223)
PVIT
Cleea es al 20133
Ferret
You at 2023
GLaMM Rassheed
2012
Egg. 14 [T) al. A023x]
Boud [L
20233
DreamLLM
Dong
2023
Interkeave
Sparkles Dallogii VC/CC
(uunng et al. 20133
Object/ Speciie
Egg. VideoClat eLi al
2023
X-LLM Cheea el al
2023c
MULTIS Zhag et 1212e
AIMIC-IT
(La et a.. AApa
Video
VideoLLaMIX ( (Zhang et ni.
2023a
VALLEY Lit os al Macawe LLb1 Lyy
2023b
Pidee-ChncGP11 Manz
3.3.2
Tnatruction Tendency
Text
Rgg, LLaVAR Zhang of
2023
PVTT
Chen al. 20236
BLIVA Hu el 20236
Mediciue
Bgg PMC-VQA (Zhagg et al.
20231
LLs Medl
2023
Shikra
Cheaa et al. 20236)
Donnain-Specille
Document
Egg. mPLUG-DocOWi Ye et
2123g
PrinsCloud
Egg, Po Xu at al.
2023h
Figure 6: Tree category diagram of existing VLIT Data (The dataset without name is replaced by its MLLM name).
known to the public in the image. The answers in the conversation need to be as detailed as possible, and examples reasoning steps need to be given when necessary to make the answer more convincing. 6. A certain number of negative answers need to be included in the generated dialogue, such as questions about non-occurrng objects, and misdescriptions. 7. The generated conversation starts with the user asking a question. 8. Don't have anything like 'the description', or "the given text" in the generated conversation. 9. The format of the output dialogue must refer to ("User": "Question?", "Al': "Answer.").
In-Context Examples. In this paper, we generally obtain about 10 generated samples through manual writing, randomly select 2 of them as context, and input them GPT3.5 in advance. The following is an example:
Here is an example, with the given textual description and the generated conversation. Given textual description: This is a picture which can be described as Two women preparing food in a kitchen, one at the sink and one at the table. A set table with silverware, glasses and a bottle of wine. A woman having a slice of bread with jelly. A woman sitting at a table with win and bread. A women who tearing bitchen_botttle,
off a piece of bread. The image contains kitchen bottle, furniture harniture_dining table, 2 person__pesoon sitchen_spoon,
kitchen cup, 3 kitchen fork, 4 kitchen knife, 2 kitchen spoon, 1 food banana, 2 food orange, 2 food apple,
22

---

Published in Transactions on Machine Learning Research (07/2024) /024)
Figure 7: Image of in-context example.
furniture_ chhii. The image also contains a number of question and answer pairs: 'Q: IS there flowers on the table? A: no., 'Q: How many bottles are on the table? A: 22, 'Q: Is he woman making sandwiches? A: yes or no?, 'Q: How many glasses on the table are ready receive liquid immediately? A: 4:, 'Q: What type of patterns are featured? A: stripes.. The following characters can be recognized in the image: BARDOLING. Generated conversation: ("User": "Are there any flowers the table?", "AI": 'No, there are no flowers the table.', "User": "Is one of the women making sandwiches?', 'AT': "One of the two women in the picture is actually making something, but it's not clear if it's a sandwich. The other person in the picture is handling bread.", 'User': "Who do you think the women are in the picturo?', "AI:: "Based on the image alone, cannot determine the identity the women. However, it's possible that they could be friends or family members preparing a meal together.", 'User': "What kind of fruit can you see the picture?', "AT': 'I see some bananas, oranges, and apples in the picture.", "User": "Can you tell me more about the furniture in the picture?', 'AI': 'The furniture in the picture includes dining table and chairs.')
B Examples of Generated Instruction- Pair
Below, shown in Table 14 Table 15
Table 16 Table
17 Table
18 and Table 19
give several examples
of instruction-response pairs generated in this paper.
C
Correlation Analysis
The data in Table 20 is the detailed numerical content of the data in Figure 5
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Newral Information Processing Systems, 35:23716 15:23716-23736 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pp. 2425 2433, 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessol, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openftamingo: An open-source framework for training large autoregressive vision-language models. Xix preprint arXiv:2308.0133 2023.
23

---

Published in Transactions on Machine Learning Research (07/2024)
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-laguuage model with versatile abilities. or Xie preprinl arXiv:2308.1 12966, 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşılar. Introducing our multimodal models, 2023. URL https://www.
adept. http::/wwwwwadeppaa//abbgo/// ai/blog/fuyuu-8
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. Position-enhanced visual instruction tuning for multimodal large language models. arXie preprint arXiv:2308. 13437, 2023a.
Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. Visual instruction tuning with polite flamingo. arXiv preprint orXi::2307.0000 01003, 2023b.
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-lIm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint ar 2iv: 04 06,6 2023c.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXic preprint or Xrxi::3110009/ 2023d.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Iim's referential dialogue magic. arXiv preprint orXiv:2306. 151995 2023e.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXic preprint arXiv:2204. 2XXx:2204.02311,, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha. Chowdhery, Alex Castro-Ros Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finettundd language models, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- lision-lagguag models with instruction tuning, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidiroctional transformers for language understanding, 2019.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint NNi::2301.0022/ 2022.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXie preprint ar XrX:::2309. 11499, 2023.
Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingehen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning, 2023.
24

---

Published in Transactions on Machine Learning Research (07/)024)
Alessandro Epasto, Silvio Lattanzi, and Renato Paes Leme. Ego-splitting framework: From non-overlapping to overlapping clusters. In Proceedings of the 23rd ACM SIGKDD Intervational Conference on Knowledge Discovery and Data Mining, pp. 145 154, 2017.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: exploring the limits of masked visual representation learning at scale. In TEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2028, pp. 19358-19360 IEEE, 2023.
Tinglei Feng, Yingjie Zhai, Jufeng Yang, Jie Liang, Dong-Ping Fan, Jing Zhang, Ling Shao, and Dachong Tao. Ic9600: benchmark dataset for automatic image complexity assessment. IEEE Transactions on Paltern Analysis and Machine Intelligence, 2022.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings.
arXiv preprinl ar Xrx:m:110 04. 08821, 2021.
Weihao Gao, Zhuo Deng, Zhiyuan Niu, Fuju Rong, Chucheng Chen, Zheng Gong, Wenze Zhang, Daimin Xiao, Fang Li, Zhenjie Cao, et al. Ophglm: Training an ophthalmology large language-and-viste assistant based on instructions and dialogue. arXiv preprint arXiv: 12174, 2023.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. Xrxiv preprint Xii:2310.012211 2023.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint 23x5 04790, 2023.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. arXiv preprint arXiv:2308.06:: 063,,, 2023.
Jiamiig Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-Ilm: Multi-modality instruction tuning. orXiv preprint XrXi::23090030 03905, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreviext
Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning, arXiv preprint ar arv::23009. 02301, 2023a.
Wenbo Hu, Yifan Xu, Y Li, W Li, Z Chen, and Z Tu. Bliva: A simple multimodal Ilm for better handling of text-rich visual questions. arXic preprint arXiv:230.. 23000000000 2023b.
Shaohan Huang, Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengehao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023a.
Yupan Huang, Zaiqiao Meng, Fangyu Liu, Yixuan Sn, Nigel Collier, and Yutong Lu. Sparkles: Unlocking chats across multiple images for multinodal instruction-followngg models. arXiv preprint ar Xiv::23008.160 2023b.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Auja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming-Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold, and Lu Jiang. Videopoet: A large language model for zero-shot video generation, 2024.
25

---

Published in Transactions on Machine Learning Research (07/2024)
Terry K Koo and Mac Y Li. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chrropractic medicine, 15(2):155- 2016.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effcient prompt tuning.
arXiv preprint (xXx:2104.000000 2021.
Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model, 2023a.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei
Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint wrXiv22366.54225 2023b.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Iims with generative comprehension. arXiv preprint orXit:2307.16192 2023c.
Chen Li, Yixiao Ge, Jiayong Mao, Dian Li, and Ying Shan. Taggpt: Large language models are zero-shot multimodal taggers. arXiv preprint arXiv:2904.0000 2023d.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large auguuage-and-vision assistant for biomedicine in one day. arXiv preprint ar Xiv:23306 00890, 2023e.
Huayang Li, Siheng Li, Deng Cai, Longyue Wang, Lemao Lin, Taro Watanabe, Yujiu Yang, and Shuming Shi. Textbind: Multi-turn interleaved multimodal instruction-folloing arXiv preprint arXiv:2309. 08637, 2023f.
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Song Chua, Siliang Tang, and Yueting Zhuang. Empowering lsson-language models to follow interleaved vision-language instructions. arXie preprint arXi::2088.00 04 152, 2023g.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. ar preprint Xrv:22301. 12597, 2023h.
Junyan Li, Delin Chen, Yining Hong, Zhenfang Chen, Peihao Chen, Yikang Shen, and Chuang Gan. Covim: Composing visual entities and relationships in large language models via communicative decoding, 2023i.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0033 2023j.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M A large-scale dataset towards multi-modal multilingual instruction tuning, arXic preprint XrXio:2306. 04387, 2023k.
Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yunchao Wei. Stablellava: Enhanced visual instruction tuning with syuthesized image-dialogue data. arXiv preprint arXi::2308. 10253, 20231.
Yunxin Li, Baotian Hu, Xinyu Chen, Lin Ma, and Min Zhang. Lmeye: An interactive perception network for large language models. arXiv preprint ar arXiv:2305.03701, 2023m.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 19th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755, Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiu:2306.1455; 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved basclines with visual instruction tuning, 2023b.
26

---

Published in Transactions on Machine Learning Research (07/2024)
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. ar preprint arXiv:2304.08488, 2023c.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Renze Lou, Kai Zhang, and Wenpeng Yin. Is prompt all you need? no. comprehensive and broader view of instruction learning, 2023.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. CoRR, abs/2312 0bss22312.11772 2023a.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b.
Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, and Yelong Shen. An empirical study of scaling instruct-tuned large multimodal models. arXiv preprint arxiv:2309, 09958, 2023c.
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. arXiv preprint ar Xix:22305 1 15023, 2023a.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.074 2023b.
Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arxx:22006.000 09009,, 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arX1m:2306.05/2) 2023.
Lluís Märquez, Xavier Carreras, Kenneth C Litkowski, and Suzanne Stevenson. Semantic role labeling: an introduction to the special issue, 2008.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013.
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, et al. Anymal: An efficient and scalable any-modality augmented language model. arXie preprint arxiv:3009 16058, 2023.
AT Chatgpt. htttss///opnaal.com/////////a 2023a.
2023a.
OpenAL. Gpt-4v(ision).
2023b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730 27744, 2022.
Genevieve Patterson and James Hays. Coco attributes: Attributes for people, animals, and objects. In Computer
2016: 14th European Conference, Amsterlom, The Netherlands, October 11-14, 2016, Proceedings, Part VT 14, pp. 85-100. Springer, 2016.
27

---

Published in Transactions on Machine Learning Research (07/2024)
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furn Wei. Kosmos-22 2: Grounding multimodal large language models to the world. arXiv preprint Xrv:22000. 14824, 2023.
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. Detgpt: Detect what you need via reasoning. arXiv preprint 230: 14 14167, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsual Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. ArXio 2311.03356, 2023.
Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu. Dbscan revisited, revisited: why and how you should (still) use dbscan. ACM Transactions on Database Systems (TODS), 42(3):1-21, 2017.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Newral Information Processing Systems, 35:25278- 25222277-25294, 2022.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpl: One model to instruction- follow them all. arXiv preprint arXit:2305.1635, 2023.
Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. arXiv preprint ar xrx:i:2004.0000 029900, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Aune Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1999 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXie preprint *xXi:2307.009288 2023b.
Haoqin Tu, Bingchen Zhao, Chen Wei, and Cihang Xie. Sight beyond text: Multi-modal training enhances Ilms in truthfulness and ethics. arXiv preprint arXiv:2309.071 07120, 2023.
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint 160101160..71100 2016.
Bin Wang, Fan WU, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, el al. Vige: Visual instruction generation and correction. arXiv preprint or (xxx:22088.1211 2023a.
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning, 2023b.
Weihan Wang, Qingsong Lv, and Wenmeng Yu. Cogvlm: Visual expert for large language models. In Github, 2023c.
Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjic Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-secing project: Towards panoptic visual recognition and understanding of the open world. aixio preprint ar Xiv:2308.01907, 2023d.
28

---

Published in Transactions on Machine Learning Research (07/2024)
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint WrXi:2305.11175, 2023e.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, pp. 1-36, 2023f.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Sell-instruct: Aligning lauguage model with self generated instructions. arXiv preprinl arXiv:2212 2022.
Yufei Wang. Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey, arXiv preprint wxiv:2307.12966, 2023g.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint ar (tt:2209.01652 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. of prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:200s. 2023.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arxiv preprint Xixe:22309.0 05519, 2023.
Peng Xu, Xiatian Zhu, and David A Clifion. Multimodal learning with transformers: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308.16911 2023b.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint ar Kiv:2212.1077: 2022.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arX7::230.11872, 2023.
Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1191-1200, 2022.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. docowl: Modularized multimodal large language model for document understanding. arXie preprint xxXi:2337.02249 2023a.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXin preprint ar 2rX::22304.14 14178, 2023b.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: Lansuage-assisted multi-modal instruction-tumne dataset, framework, and benchmark. Xixiv preprint ar xxX::23366.06687, 2023.
29

---

Published in Transactions on Machine Learning Research (07/2024)
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arviv preprinl arXiv:2310.07704, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130B: an open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net net, 2023a.
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training a gpt4-style language model with multimodal inputs? ar preprint 0rXiv:2307.02468 2023b.
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309. 10813, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprinl arXiv:2306. 02858, 2023a.
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. orXiv preprinl arXiv:2004.1044 2020.
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm-xcomposee A vision-language large model for advanced text-image comprehension and composition. Xix preprint arXiv:2309.15 (an::3309.11122 2023b.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and
Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint wxXii:2303.16100 2023c.
Shengyu Zhang, Linfeng Dong, Xiaoya. Li, Sen Zhang, Xiaofci Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint 23v: 10792, 2023d.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gptároi: Instruction tuning large language model region-of-interee arXiv preprint arXoo:2307.0300 03601, 2023c.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100
2022.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xic. Pme-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXtc:2905.14415 2023f.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. ar preprint arXin:2306.17107, 2023g.
Bo Zhao, Boya Wu, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04085 2023a.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering langua- model with multi-modal in-context. learning, 2024.
Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Rumpei Dong, Chunrui Han, et al. Chatspot: Bootstrapping multimodal 1lms via precise referring instruction tuning. Xin preprint ar 2rX1::2307.09474, 2023b.
30

---

Published in Transactions on Machine Learning Research (07/2024)
Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal llms. Xiv preprint wxXi:2307.08581, 2023c.
Zhiyuan Zhao, Linke Ouyang, Bin Wang, Siyuan Huang, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Milm-dataengine: An iterative refinement approach for mllm. arXiv preprint Xin:2308.13560 2023d.
Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint WXXu:2305.16103, 2023c.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Chunting Zhou, Penglei Liu, Puxin Xu, Srini Tyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint S.kt:23355112000 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint 21K1::2344115922, 2023a.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint orXiv:2304.06952 2023b.
31

---

Published in Transactions on Machine Learning Research (07/2024)
Table 10: Visual task statistics in VLIT data.
Name
Visual-Langage Task
LLaVA [Lu al. 2023c
conversation, detailed description, complex reasoning
Mini-GPT4 Tau ot al 2023n
detailed description
mPLUG-OWl Ye
conversation, detailed description, complex reasoning
VideoChat el et aLl 20231
(video) conversation, (video) detailed description, complex reasoning object detection, instance segmentation, visual grounding, image captioning, visual question auswering
VisionLLM
Wang et al
2023e
X-LLM (Chen et au. 20253
(videa) detailed description, visual question answering, dialogue generation image captioning, ohject detection
De
m. [2023
ChatBrrdge Zhao et al: 2023e
image captioning, visual question answering, (video) conversation, (video) detail description, (video) complex reasoning conversation, detailed description, complex reasoning
PandaGPT
Su et al.
2023
MedVInT-TE
E7TD Zhang et al
20231
visual question answering
GPTTToolls
Yang et (024)
object grounding, object segmentation, image generating, visual question answering visual question answering image captioning, visual question answering, reasoning, classifieation, image genera- tion
LLdVA-Med
LL. ct a. 20255
Ying- VLM
[T et 202310
Otter
Li et
20235
lava-interleaved, spot the difference, visual storytelling. deuse captions, TV show captions, indour event planning, ego4d
Video- ChaiGP1 Video-ChaIGP1
Manz et al. 2023
video reasoning, creation and generation, action recoguition, video understauding, video conversation, visual question answering, temporal understanding vismal question answering, visual relationship, temporal ordering, grounded generation, grounded matching, commonsense reasoning, miscellaneous, image understanding, region understanding, image text matching
OFA onttttiiintt Xu et al. 2022
MLdeo (Zhiig al. VALLEY [Luo et Bl 20255
2023a
(video) conversation, (video) detailed description, (video) complex reasoning (video) conversation, (video) detailed description, complex reasoning conversation, detailed description, complex reasoning, image captioning, visual ques- tion answering
MultiModa--PP GPT Gong et al. 2023
Br.I Dal of al. 2023
image captioning, visual reasoning, conversation, visual question answering, image question generation, image classification
LaVIN Luo et al 20258
conversation, detailed description, complex reasoning, visual question answering image captioning, video captioning, visual question answering
MACAW LLM
LAMM Yin ct. hig 2022
conversation, complex reasoning, detailed description, visual task dialogue medical imaging description, causes and symptoms, diaguosis and examination, treatment and prevention, prognosis and lifestyle visual question answering (text-rich)
OphGLM Gao et al.
2023
LLaVAR Zhang et n 2023g
Shikra (Chen aL 2023e)
conversation, detailed description, complex reasoning, visual question answering, image captioning, grounding caption
Clever Planingo Ghen
202
image captioning, image classification, change captioning, visual question answering visual question answering, information extraction, natural language inference, image captioning, conversation, detailed description, complex reasoning single-region captioning, multi-region captioning, single-region reasoning, multi-region reasoning,
.PLUG--DeOwl Yo Al.
20236
GPT4RoI Yo 20234
SVIT Zhao 120233
conversation, detailed description, complex reasoning conversation, detailed description, complex reasoning conversation, detailed description, complex reasoning image captioning, visual question answering, object detection, OCR, region caption conversation, detailed description, complex reasoning visual question answering, image captioning, video captioning, classification, conver- sation, complex reasoning
KOSMOS-2 Peng ds al
2023
BahoGPT Zhao pl.
ChatSpot Zhao éf al.
20236
ImageBind LLM Ban EL K1.
2023
Lynx (Zeng al. 20236
LME
2023m
conversation, detailed description, complex reasoning, image-text matching visual question answering, visual captioning
ASM Wangg ot al. 2023c
Chentor LT ot. al. 2023g
conversation, visual relation inference, visual storytelling, multi-modal close, grounded question auswering, text-rich question answering, complex ressoning image captioniug, visual question answering, conversation, OCR, detailed description, complex reasoning
BLIVA [H et al 20256)
StableLLAVA [Li et al [2023]
visual reasoning
Qwen- VL (Bai et al (002
visual question answering, grounding, OCR conversation, detailed description, complex reasoning brief description, conversation visual question answering conversation, detailed description, complex reasoning conversation
VIGC (Wang et al. 2212
PointLLMI Xu ét all
20233
MLLM-DataEn
Zitao el al. 2023d
MM (Tu et al. 2023
NExT=GPT WD
2023
PVIT CChen et al. 2023a
small object recoguition, same-category object discrimination, object relationship- based reasoning, object attribute- based reasoning, OCR image creation, image comparison, intrinsic & extrinsic image understanding conversation, detailed description, complex reasoning visual question answering
TextBind et all 20/231
DreamLLM Dong all
2023
AnyMAL Moon 2023
InternI. XCotapaser Zhanng et al. 2023b
image captioning, visual question answering, image question generation, conversation, detailed description, complex reasoning conceptual caption, visual genome
SparklesChat (Hang et al 20236
LLaVA (Lau ald 20246
convensation, detailed description, complex reasoning conversation, detailed description, complex reasoning, visual question answering conversation, detailed description, complex reasoning
CogVLM Wang éf al 20236
Ferret (You el al. 2028
MiniGPT-vz (Chch
2023d
visual question answering, captioning, grounded captioning, referring expression comprehension, referring expression generation, object parsing and grounding
32

---

Published in Transactions on Machine Learning Research (07/2024)
Table 11: Hyperparameters of LLaVA
version
v1
vision tower
dip-vii-large-allchl
pretrain mm mlp adapter
caption bin
mm_ vision _electt_e____eeeee sclect_layer mm_use__it aass__m__ttte imm_star___n_ _nn mm use im patch token
-2
False
False
fp16
True
num train arain_epochs
1
per device train batch size per_ per_device_evee @oval_batch_site size
4
2
gradient accumulation steps learning_raa rate
2
2e-5
weight__dss decay
warmup ratio Ir scheduler lype
0.03
cosine
tf32
False
model max length
2048
dataloader num workers preppoccesssss
4
True
Table 12: Hyperparameters of BLIP-2
model type
caption_ cttpl__occoo____111_11 opt2.7b
load finetured
False
use
lse__rad__ss _ccdd__hccpointtt
True
freeze vit Ir sc_ee init min Ir warmup Ir warmup steps weight_deces max epoch
False
linear iiner__wrrmup
le-5
0
1e-8
1000
0.05
batch_ssie sicob_sse_____tee_e train
num workers
4
accum arcum_grad iters len min len num beams num ans ca_diddaddssssss inference inferonce__mehhod method seed
50
1
3
128
rank
42
33

---

Published in Transactions on Machine Learning Research (07/202) /204)
Table 13: Hyperparameters of OpenFlamings
cross attn _verry _layerss dataset_resamplee
True
batch_sizee
train mum samples
158000
loss jos__mutiplicr
0.2
workers
freeze Im em__ediddin run rum_name num num_epocchs
True
OpeaFlaningg-BBiiitntttttltuuu
warmup steps
1000
34

---

Published in Transactions on Machine Learning Research (07 (27/2024)
Table 14: An example of a conversation generated from an image as a whole.
Image
User
Can you tell me more about the child in the picture?
AI Assistant
Certainly, the image showcases small child, possibly a baby, who is engaging with toothbrush. The child appears to be inspecting or playing with the toothbrush, which is colored green and yellow. The child seems to be foeling comfortable and cozy, and there's a sense adorable curiosity about him. There's also a hint of seriousness in his actions, as if he's truly trying to figure out the purpose of the toothbrush. How does the baby feel about the toothbrush?
User AI Assistant
Based the image, the baby scems intrigued by the toothbrush. The child doesn't show any signs of fear discomfort; instead, he appears curious, if he's trying to understand what the toothbrush is. It's a common behavior for children of this age to be curious about the objects around them.
User AT Assistant
the baby being fed the photo?
No, the image doesn't show any signs of the child being fed. The primary focus is on the child's interaction with the toothbrush. Any foeding utensils like spoons or bottles are not visible the picture.
User
You mentioned the child looks serious. Can you elaborate that?
AI Assistant
Sure, the term 'scrious' here refers to the child's focused attention on the toothbrush. Even though he is just a baby, his intent gaze and the way he's holding the toothbrush suggest a sense of earnest exploration, as he's trying to understand how this object works or what its purpose might be. It's an adorable demonstration of child's natural curiosity and learning process.
User Al Assistant
Can you tell me about the toothbrush?
Yes, the child is holding toothbrush that is green and yellow in color. Given the size and the color combination, it seems like typical toothbrush designed for children. It's quite common for children's toothbrushes to be colorful and attractive to help make the process of learning to brush their teeth more enjoyable.
User AI Assistant
Does the image suggest anything about the baby's mood?
Yes, based on the attributes present in the image, the baby seems to be comfortable and curious. They look serious while inspecting the toothbrush, which suggests they are deeply engaged in the activity. Although there's no definitive way to tell, it's reasonable to say that the baby appears content and intrigued by their surroundings.
User AI Assistant
Is there anything else of note in the photo?
The focus this picture is mostly on the baby and their interaction with the toothbrush. There's no additional information about the surrounding environment. However, the fact that the baby feels comfortable playing naked suggests that they are probably a familiar and safe space, most likely their home.
35

---

Published in Transactions on Machine Learning Research (07 (224))
Table 15: An example of conversation generated from an image a whole.
Image
User
Can you tell me more about the scene in this image? Absolutely! The image shows an office that is quite bit cluttered. There is desk equipped with two computer monitors, all of which are displaying various data. There's also additional computer equipment scattered around, including a laptop, a keyboard and couple of mice. There is a presence of other electronic devices as well, such as a TV. Interestingly, it's not just an office space, as I also see a sink and a refrigerator, and kitchenware like a cup, which suggests this might be a home office or small startup office. The workspace seems to be quite disordered or messy.
AI Assistant
User
Are the monitors up-to-date or old?
AI Assistant
The monitors in the image appear to be of an older model. What can you tell me about the lighting for the workspace on the left?
User
Al Assistant
The workspace on the left seems to be using a lamp for illumination. It's quite common in home offices or personal workspaces to have such setup.
User
Is there any music-related equipment From what I can observe, there might be CD present on the tabletop, suggesting possible interest in music or use of software stored on CDs by the person working here. Would you describe this as an organized workspace?
the desk?
AI Assistant
User AT Assistant
Yes, based on the attributes present in the image, the baby seems to be comfortable and curious. They look serious while inspecting the toothbrush, which suggests they are deeply engaged in the activity. Although there's no definitive way to tell, it's reasonable to say that the baby appears content and intrigued by their surroundings.
User Is there anything else of note in the photo?
AI Assistant
Based the scene in the image, this workspace does not appear to be organized. There are various pieces of equipment strewn about haphazardly, which suggests that the area is fairly messy.
User AT Assistant
Any idea what kind of work is being done in this office?
Given the presence of soldering equipment and multiple computers, it could be an office where hardware development or repair is done. This might include tasks like assembling fixing computer parts, or working with other electronic components. However, without more specific details in the image, it's difficult to say for certain.
User
Can you identify any other items in the picture?
AI Assistant
Sure, beyond the computer hardware, I can also see two chairs, indicating that there might be space for another person to work or meet. There a keyboard and two mouses, suggesting that at least two computer systems are use. There's a laptop present as well, which adds to the versatility of the workspace. Additionally, there's a lamp offering light, which can be particularly useful for detailed work like soldering.
36

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Table 16: An example of generated conversation with negative responses.
Image
User
Can you tell me what the people in the picture are doing?
AI Assistant
The people in the picture, who seem to be family, are gathered around a television. They seem to be watching TV together in their living room.
User
What time of year does this scene seem to be set in?
Assistant
Given that there a lit Christmas tree the room, strongly suggests that this scene is set during the holiday scason, most probably around Christmas time. Are there any books in the picture?
User
AT Assistant
Yes, there indeed two books in the picture.
User AI Assistant
What's on top of the Christmas tree?
The top of the Christmas tree is adorned with bow. It's a common tradition to place a decorative item on the top of a Christmas tree, such as a star, angel, or in this case, a bow.
User
Are there any pets in the room?
AI Assistant
No, the description the image doesn't mention any animals or pets being present in the room.
User
What the kids looking at?
AI Assistant
The kids are looking out of window. It's unclear what they're looking at specifically, that detail isn't provided in the image.
User
Do you know what show they are watching on the TV?
AI Assistant
Unfortunately, the image doesn't provide specific information about what's being displayed on the lelevision. They could be watching any number things, such as a holiday-themed movie, Christmas special of television show, or even the news. Is the Christmas
User AI Assistant
the main focus of the image?
While the Christmas tree is a significant element in the picture, it's not the main focus. The main focus appears to be the group of people, particularly the three kids looking out of a window and the family gathered around the television.
User Assistant
Are there any other significant objects in the picture?
Yes, besides the Christmas tree and the television, the image also contains two books and a vase. However, it's not clear from the image how these objects relate to the people or the overall scene.
User AI Assistant
Are there any adults in the picture?
From the information provided, it's not clear whether there are any adults in the picture. It mentions three people, who could be adults, children, or a combination of both.
37

---

Published in Transactions on Machine Learning Research (07 2024)
Table 17: An example of generated conversation with negative responses.
Image
User
What the women doing in the picture?
Al Assistant
The women in the picture are enjoying their time in field with tall grass where they are flying a kite that resembles dragon.
User
Can you describe the kite in more detail?
AT Assistant
The kite in the picture has a design that makes it look like a dragon. Interestingly, its wings are reported to be pink. sounds quite vibrant and playful. Are the women in the image friends?
User
AT Assistant
Based on the image, both women appear be friendly and enjoying their time toget It's quite likely they friends, though the image docsn't provide explicit confirmation of their relationship.
User AI Assistant
Does the field look like public place?
Yes, based on the image, it seems that the field is a public place. The women are dressed casually which also suggests that this is a relaxed, informal setting.
User Do the women seem to be experts al flying kites?
Al Assistant
The image doesn't 't provide enough information to determine if they are experts at kite flying. However, since they seem to be enjoying themselves, it's clear they know at least the basics of how to fly a kite.
User Is there any sign of a crowd or other people in the image?
Al Assistant
No, the available information doesn't mention the presence of other people or crowd. The focus seems to be solely on the two women and their kite flying activity.
User
Do you know what season it based on the image?
A1 Assistant
The image doesn't give specific information about the season, though the presence of tall, dry grass might suggest it's late summer or early autumn. However, this is simply a conjecture based the given image.
User
Are they in park or private property?
AI Assistant
The picture doesn't provide clear information about whether the field is part of public park or private property.
User AI Assistant
What the condition of the grass in the field?
The grass in the field quite tall and appears to be dry. It's probably need of being cut, based on the information given.
User
Is there any wildlife in the picture? The image does not mention any wildlife. It primarily focuses the two women and their activities with the kite in the field.
AI Assistant
38

---

Published in Transactions on Machine Learning Research (07 2224)
Table 18: A generation conversation for regions where the image is marked.
GEORGE
WAY
C
Tan Noiton
TiMBiTS
Image
User
What is the person in the circle doing? The person appears to be sitting and holding something, perhaps a utensil or one of the food items on the table. Can we tell what the person is holding? From the image, it's uncertain exactly what the person is holding, It might be spoon knife.
AI Assistant
User
Al Assistant
User
Can you tell me something about the brand mentioned in the image? Yes, the brand Tim Hortons or Timbits is mentioned in the context of the image. Tim Hortons is a famous coffee shop chain in Canada known for its coflee and doughnut holes called 'Timbits' It seems that the box of doughnut holes on the table might be from this brand.
AI Assistant
User Is the person alone are there other people in the picture?
AT Assistant
From the given image, it appears that there's only one person visible in the image. However, without seeing the entire picture, it's hard to conclusively say if there are others present but not visible within the frame.
User AI Assistant
Do you think the person is about to eat the food?
While it's difficult to definitively confirm without more context, the fact that the person is sitting the table with a variety of food items in front of them, and appears to be holding something, suggests that they might be about to eat the food. However, they could also be in the process of setting the table or packing up leftovers. Can we tell who this person is?
User AI Assistant
As an AI, I can't identify individuals unless they are publicly recognizable and their information part of my built-in knowledge. However, in this picture, the individual appears to be just a rogular person sitting and holding something. Are they eating healthy?
User AT Assistant
The meal includes a sandwich and a soup, both of which have been described as healthy. However, they also have donut balls which are typically high in sugar and fats. So, while parts of their meal are healthy, the overall meal could be considered mix of healthy and less healthy options.
User AI Assistant
Is there any beverage visible in the picture?
No, there isn't it any visible beverage in the picture. The image mostly showcases the food on the table, such as the sandwich, soup, and donuts, but doesn't 't show any drinks.
39

---

Published in Transactions on Machine Learning Research (07 (07/2024) /024)
Table 19: gencration conversation for regions where the image is marked.
Image
User
What is marked in the image? The marked word in the image is 'PROCTOR'
AI Assistant
User
Could 'Proctor' denote the name of the baseball team? Yes, considering the background, 'Proctor' could indeed be the name of the baseball team depicted the photograph.
AI Assistant
User
Can you tell me more about the significance of 'Proctor' in relation to baseball? Unfortunately, I don't have any specific information about baseball team named 'Proctor'. It could be school leam, a local community leam, even professional team from the past. The name could also possibly refer to a location or sponsor. Without additional context, it's challenging to be certain.
AI Assistant
User AI Assistant
Are there any well-known baseball players in the photo?
Based on the given description, it not possible to determine if there are any well-known players in this black-and-white photograph. The image features 14 individuals, presunably members of the 'Proctor' team, but without clear identification or further context, it's impossible to specify if any of them are famous.
User
Are all the people in the picture players? No, not everyone in the picture is player. They may be coaches or team staff rather than players.
AI Assistant
User
there any indication the picture of the era or time period this game was played in? The fact that the photograph in black and white suggests that it could be quite old, possibly from the early 20th century. However, without more context specific details, it's hard to pinpoint
AI Assistant
exact era.
User
Are there any spectators or fans visible in the picture?
AI Assistant
No, there are no spectators or fans visible in the picture. The image focuses on the baseball
team named 'PROCTOR:
40

---

Published in Transactions on Machine Learning Research (07 (0722
Table 20: Correlation coefiiiiet data. The values in the table are correlation coefficients and p-values, respectively. Where *** ** * represent significance levels of 1%, 5%, and 10%, respectively.
900 00000 1 ***
02800022000 +]
MaAaSS (0. (0.188)
0.000000 ( ne0r"
(0.780) (600
(0.4(00.70))
TR.
0. 197222
0.9830 01.0111330 (*)
( nours
0.52% 028 0 (8ss
( (UUT
1 (20)
0. (0..2 10115111100
(
VR.
(0 10
0.821 179)
( (00-485))
0.:3800 800
0919110 058)
ITR
0.700
0.32800 6721
00000000 8058
0. 438411 5823
2215) 0.6830 9935)
0.51220 3811
4834
(0.0000.000
(0.065(0.90))
SR
10000
-0.0631
0.113
) 0.445000000000000
1811 (0.
-511
(0.
1C
0.9200
0.009 0.24990
8,859 102) 00000000000000000000000000000000
052 (552(0.144) 11488
74100 239)
0. (414
0.538(0 (289))
(
0. 787 (0
TA
0.946(01.000) 0.044))
(0 116)
0.963 (0.50
0.8237 (00333
177)
0.937(0 1985
0.205(0.700) 796)
(1
TI
0.802/10
0.705 295)
03310.3800 389)
0.77%(0.200 220
0.6(() 400)
0.80300 131)
0. 893 (093(00100) 1071
0.25(0.75) 750)
IT
0. 03110
0.891 (0 1000
0 122210 158)
0. 0.9257 (0 (sss
0.7261 (0:0002277)))) 274)
0. 0. 11
0.301 (0:699)
SU
0.952(0 (0.038 )
0 (0.000000000))00000 098*)
ID 0.986 (006(0 0.014*))
(
0.725(0.22) 276)
0.963(0.017*)) (0.017**)
0. 0.861(0..) (0.1111 186)
0.377(0.622) (6255)
6233)
Overall
MD
TD
RD
DC
IC
131
41


Reviews:
[{"review_id": "UGEP80gSbI", "reviewer": "Reviewer_UyUM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n+ This paper is well-organized and easy to follow.\n+ The investigated VLIT is vital for building MLLMs, as previous studies mostly focus on the architecture.\n+ The review is quite comprehensive (e.g., Tables 1&2), containing both models and datasets.\n\n**Weakness**\n+ We understand this paper focuses on VLIT. However, it is still required to highlight the model architecture (e.g., which LLM/vision-encoder/adapter is used) in Table 1, as they are all essential to make VLIT work and may influence performance. Besides, more discussion should be covered more clearly in the draft, instead of just a brief description (the first paragraph of Sec. 3.1).\n+ Sec. 4 dives into the construction of VLIT datasets. As the proposed principles and quality aspects, why not have an experiment to evaluate all used datasets? This can be valuable to guide future research during dataset selection.\n+ For the experiments, they involve LLaVA, BLIP-2, and OpenFlamingo. Though they attempt to use the same VLIT datasets, their inherent architectures are different (different LLM sizes or types of adapters). Furthermore, the pre-training data of their used LLMs are quite different (e.g., LLaMA in LLaVA is obviously better than OPT in BLIP-2). Is this comparison fair? Does the result actually reflect the quality of different VLIT datasets?\n+ The discussed limitations (Sec. 6) are quite brief. It will be better to involve qualitative examples for each item, which can help them be more concrete and easy to understand.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims at the literature review of multimodal large language models (MLLMs), specifically for vision-language instruction tuning (VLIT). They cover various VLIT settings as well as different constructions of VLIT datasets. Built upon this, they conduct experiments and demonstrate performance impacts from different VLIT settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness for details.\n\n+ Highlight different architectures of MLLM models.\n+ An evaluation/comparison for each used VLIT dataset.\n+ Concrete examples for the discussed limitations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cBretJl46p", "reviewer": "Reviewer_ahPg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n-The paper provides a comprehensive review of the latest developments in Vision-Language Instruction Tuning (VLIT) settings and datasets, offering valuable insights for both the community and researchers interested in the current state of multi-modal LLMs.\n\n-It presents a detailed categorization of VLIT datasets from multiple perspectives, enhancing understanding of data requirements for effective multi-modal LLMs. From this insight, the authors propose a pipeline to create high-quality VLIT data that could improve visual instruction tuning for these models.\n\n-The thorough experiments conducted demonstrate the effectiveness of the VLIT data produced by their proposed pipeline.\n\n[Weaknesses]\n\n-While the paper covers a wide range of VLIT settings and datasets, the paper doesn't significantly differ from the survey on language-only instruction tuning, which hinders the novelty of the paper. Given the modality gap between language and image, I think further discussion on topics such as the distribution and mutual influence of multimodal data and unimodal data, the organization and composition of vision data, the impact of high-quality vision data on visual instruction tuning, the influence of image data clarity and diversity, etc., would be beneficial.\n\n-While much emphasis is placed on discussing the setting of Vision-Language Instruction Tuning, it appears that some aspects are still missing from the discussion. Specifically,  although the authors have divided the training process of Vision-Language Instruction Tuning into pretraining and finetuning stages, the pretraining stage described in the paper more closely resembles cross-modality alignment rather than pretraining. Given that there is extensive work focusing on pretraining Vision-Language models from scratch and treating Vision-Language Instruction Tuning as finetuning(e.g. Gemini[4], VideoPoet[5], Unified-IO1,2[7,8], Flamingo, etc). It seems that these works have been overlooked by the author.\n\n-While the author discusses the correctness and diversity of instruction tuning data, the focus primarily remains on the language aspect. Further discussion of the fidelity and diversity of images would solidify this paper.\n\n-The authors employ clip similarity as a metric to eliminate \"incorrect\" image-text pairs. Nonetheless, as suggested in [3], clip embedding similarity is already known to have limitations in capturing fine-grained, ambiguous information. Additionally, the instruction prompts generated by LLMs vary significantly from the training data of the clip model. Therefore, it necessitates further discussion and experimentation on the efficiency of filtering using clip similarity.\n\n-Lack of ablation studies: Given the significant differences in the data organization and data sources between the LLava, MIMIC-IT datasets, and the generated VLIT dataset, it is challenging to draw a convincing conclusion about the superiority of the VLIT dataset without any ablation study.\n\n-Some references are missing:\n\nPage 5 Section 3.1 a  pixel-level linear projection [1]\nPage 8 Section 3.2 A method of Multimodal in-context learning [2]\nPage 12 Section 4 Correctness with clip  [3]\n\n[1]  Rohan et al. fuyu-8b: Introducing our Multimodal Models\n\n[2] Zhao et al. Mmicl: Empowering vision-language model with multi-modal in-context learning[J]. arXiv preprint arXiv:2309.07915, 2023.\n\n[3] Tong et al. Eyes wide shut? exploring the visual shortcomings of multimodal llms[J]. arXiv preprint arXiv:2401.06209, 2024.\n\n[4] Gemini: A Family of Highly Capable Multimodal Models\n\n[5] Kondratyuk et al. Videopoet: A large language model for zero-shot video generation[J]. arXiv preprint arXiv:2312.14125, 2023.\n\n[7] Lu et al. Unified-io: A unified model for vision, language, and multi-modal tasks[C]//The Eleventh International Conference on Learning Representations. 2022.\n\n[8] Lu et al. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action[J]. arXiv preprint arXiv:2312.17172, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors provide an in-depth analysis of the design and characteristics of recent VLIT settings and datasets in multi-modal LLMs, providing insights into the motivations behind the dataset design. They give a detailed multi-perspective categorization of existing VLIT datasets.  Furthermore, they outline the qualities that high-standard VLIT data should possess.\nThis paper's significant contributions include:\n\n-A systematic review of all relevant settings and datasets of VLIT in MLLMs. The authors describe the principles that are essential during the construction of VLIT data. They further highlight the ongoing challenges and future research areas that warrant further exploration.\n\n-The paper proposes a detailed pipeline for creating high-quality VLIT data with explicit implementation details. This roadmap includes understanding the intrinsic motivations behind VLIT settings and pinpointing the qualities that high-standard VLIT data must have.\n\n-The authors develop a VLIT dataset using the outlined construction pipeline and public annotation data. They compare this new dataset with prior VLIT datasets on varied MLLMs with distinct architectures. Their results substantiate the effectiveness of the principles and the usefulness of the construction pipeline.\n\n-The authors make the code and dataset publicly available.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-**Expansion on Multimodal Data Discussion, Specially for Vision data** (Critical): The paper would benefit from a more nuanced discussion on the distribution and mutual influence of multimodal versus unimodal data, the organization and composition of vision data, and the impact of high-quality vision data on visual instruction tuning. \n\n-**Clarification on Pretraining Stage Description** (Critical): There is a need to clarify the pretraining stage description and discuss the relevant works that train the multi-model LLM from scratch.\n\n-**Evaluation of CLIP Similarity Metric** (Strengthening): Given the limitations of clip similarity, the authors should provide further discussion and experimentation on the efficiency of using clip similarity as a metric for filtering instruction data. \n\n-**Ablation Studies** (Critical): This would strengthen the paper's claims and provide clearer evidence of the benefits of their dataset construction pipeline.\n\n-**Updating References** (Strengthening): Adding the missing discussions and references to some recent work could strengthen the impact and depth of this submission.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BkbndzGexE", "reviewer": "Reviewer_hRhr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n1. the paper is well motivated and certainly timely and relevant due to the recent this massive interest in large vision-language models.\n2. the inclusion of experiments in a review paper is a big plus. there are careful ablations to test whether each of the 'recommended' strategies for VLIT actually translate to performance improvements, and these results can serve as useful insights for practitioners.\n3. the paper is generally well-written and was nice to read, but some of the tables and figures can be improved to make them more readable and intuitive. see below.\n\nweaknesses:\n1. the writing and presentation of the paper can be improved. there should be an overview figure at the beginning highlighting the main taxonomy/categorization which is the main contribution of the paper. some of the tables and figures are too densely packed (such as table 1, table 2, and figure 2). these are quite hard for the reader to parse and is more suitable for the appendix. i am wondering if there can be simplified versions of table 1, table 2, and figure 2 that are more suitable for the main paper.\n2. another aspect of presentation that can be improved is to have more visual depictions of the categories, instead of in purely words. for example, in table 2, the meaning of the categories 'annotation adaptation', 'self-instruct', 'data-mining' and so on are not immediately intuitive. is it possible to have some intuitive figures illustrating what they mean? and how they mainly differ from each other? also, why was this categorization chosen, and why is it the most suitable to categorize in this way? there are some attempts to draw these out in figure 3 and figure 4, but these figures are too dense and not clear at first glance. the figure captions should also be more detailed to assist the reader in parsing the figure. also, 'data-mining' is missing as a concept that is illustrated in a figure.\n3. the coloring in figure 5 needs to be fixed. also, from the results in figure 5, does it mean that 'Balance' as a principle is not as important as the other concepts? even leading to negative correlation in some cases? also is there something about spatial relation that causes methods to not work? can the hypothesis 'This may be attributed to the fact that spatial positional relationships typically only have a certain correlation with the number of elements appearing in the instruction and the related instructions' be tested to see whether new guidelines are needed to improve in this case?\n4. also, were the trials run several times and averaged/reported with error bars? how sensitive are some of these numbers to different runs?\n5. is the only area of study in VILT in the type of data? are there are no other considerations such as in the training objectives or architectures of the models? if so, these should also be categories in the review paper and worth discussing. if not, please justify/argue why training objectives or architectures or other considerations are not included in the study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reviews the latest advances in vision-language instruction tuning (VLIT) to train multi-modal LLMs. It provides a detailed categorization for existing VLIT datasets and identifies several major design decisions necessary for training to succeed. The authors also supplement the review paper with experiments to verify their summarized insights on the performance of tuned multi-modal LLMs. Finally, they end with current challenges and future research directions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "please see weaknesses above.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a comprehensive survey of visual language instruction tuning. The paper is well organized, including both methods and datasets.
Decision: Accept (poster)