Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
Authors: Juncheng Li
Kaihang Pan
Zhiqi Ge
Minghe Gao
Wei Ji
Wenqiao Zhang
Tat-Seng Chua
Siliang Tang
Hanwang Zhang
Yueting Zhuang
Venue: ICLR 2024 spotlight (2024)
Keywords: Multimodal Large Language Models
Demonstrative Instruction
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize. This is achieved by training the VPGs on millions of image-caption pairs, where the VPG-generated tokens of images are fed into a frozen LLM to generate the corresponding captions. However, this image-captioning based training objective inherently biases the VPG to concentrate solely on the primary visual contents sufficient for caption generation, often neglecting other visual details. This shortcoming results in MLLMs’ underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task. To address this issue, we introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C), which can infer and complete the missing details essential for comprehending demonstrative instructions. Further, we propose a synthetic discriminative training strategy to fine-tune VPG-C, eliminating the need for supervised demonstrative instructions. As for evaluation, we build DEMON, a comprehensive benchmark for demonstrative instruction understanding. Synthetically trained with the proposed strategy, VPG-C achieves significantly stronger zero-shot performance across all tasks of DEMON. Further evaluation on the MME and OwlEval benchmarks also demonstrate the superiority of VPG-C. The code and models are available at https://github.com/DCDmllm/Cheetah.

Published as a conference paper at ICLR 2024
FINE-TUNING MULTIMODAL LLMS TO FOLLOW ZERO-SHOT DEMONSTRATIVE INSTRUCTIONS
Juncheng Li Kaihang Pan! Zhiqi Ge Minghe Gao!
Wei Ji Wenqiao Zhang'
Tat-Seng Chua² Siliang Tang! Hanwang Zhang' Yueting Zhuang'
'Zeejiang University, 'National University of Singapore, 'Nanyang Technological University
ABSTRACT
Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize. This is achieved by training the VPGs on millions of image-caption pairs, where the VPG-generated tokens of images are fed into a frozen LLM to generate the corresponding captions. However, this image-captioning based training objective inherently biases the VPG concen- trate solely on the primary visual contents sufficient for caption generation, of- ten neglecting other visual details. This shortcoming results in MILMs' under- performance in comprehending demonstrative instructions consisting of multi- ple, interleaved, and multimodal instructions that demonstrate the required con- text to complete task. To address this issue, we introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C), which can in- fer and complete the missing details essential for comprehending demonstra- tive instructions. Further, we propose synthetic discriminative training strat- egy to fine-tune VPG-C, eliminating the need for supervised demonstrative in- structions. As for evaluation, we build DEMON, a comprchensive benchmark for demonstrative instruction understanding. Synthetically trained with the proposed strategy, VPG-C achieves significantly stronger zero-shot performance across all tasks of DEMON. Further evaluation the MME and OwlEval benchmarks also demonstrate the superiority of VPG-C. C. The code and models are available at https://githunt wtt. hn//hhetaa
INTRODUCTION
Recent advances in Multimodal Large Language Models (MLLMs) (Li al. 2023c Liu et al. 2023
Zhu et al., 2023a) have exhibited promising capabilities processing single-image instructions, such as producing detailed image descriptions and answering questions about the image. However, they fall short in demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task. For instance, the instruction in Figure [ contains interleaved visual and textual context, requiring the model to determine the authenticity of the milk in the second image based the official image provided in the first.
An MLLM should at least have the following two capabilities to comprehend demonstrative instruc- tions effectively:
1) Not just the primary subject: Beyond focusing on the primary visual content, it should be able to meticulously discern the details within the demonstrations. These details, complementing the primary content, play a crucial role in semantically connecting the instructions. case in point is Figure wherein accurate discernment relies recognizing the logo detail milk carton.
2) Reasoning-aware details: How to decide what details are complementary to the reasoning? We expect that an MLLM may "think twice", that is, given preliminary reasoning using the primary contents, it would know what additional contents are needed as complementary details. For example, in Figure after preliminary reasoning, the model should re-attend details such the logo and
"Equal Contribution. 1 'Corresoondigg Authors.

---

Published as a conference paper at ICLR 2024
Demonstrative
Compared in ahr gensine milk diplloyed
pleuee raumier parkaging shoigs based mulk doose
Ae the parchaed wilh gomuine?
InstructbLLIP Anewer: les, partuaeed
Ours- Ours-Answer:
red dianond shuped parclused
auticaally ditforent from aill.
Visualzed anentins maps of ListrsdLLP'ss VPG.
Stagel: initial attestion
Stage2: utentton mapo afer prelimuarry reasoning
Figure 1: An example of InstructBLIP Dai et al. 2023 and our MLLM enhanced by VPG-C.
brand name on the milk carton, thereby discerning its authenticity. However, to follow zero-shot demonstrative instructions, this "reasoning-aware" capability should be acquired without the need for supervised demonstrative instructions.
Unfortunately, find that the reason why existing MLLMS not effective in demonstrative instructions is due to the lack of the above capabilities. More specifically, the crux lies in the Visual Prompt Generator (VPG) in MLLMs. VPG, such as Q-former (Li al. 2023c) and Re- sampler (Alayrac et al. 2022), translates visual features into tokens recogmizable by LLMs, and the translation is trained on millions of image-caption pairs by feeding the VPG-generatedd tokens of images into frozen LLM which generates the corresponding captions. However, this image captioning training strategy inevitably introduces the inductive bias that VPG only focuses on the primary visual contents which are just enough for the captioning task, but tends omit other visual details. For example Figure | the averaged attention map of InstructBLIP (Dai et al. 2023) (Fig- ure I shows dominant focus the primary contents, neglecting the logo detail, which however the key to answering the question.
To this end, propose lightweight Visual Prompt Generator Complete module (VPG-C), which infer and complete the missing de- tails essential for comprehending demonstra- tive instructions (Section 2.1). As shown Fig- ure 2 I) VPG-C first derives the instruction- specific guidance by intercepting the interme- diate LLM's output of the primary contents extracted by conventional VPG, and then
LLE
LLM Layar
Image Erooder
Gurdoace
LLM
Figure 2:
An overview of VPG-C,
2) guides the VPG to recover the missing visual residual details. Finally, 3) these residual details are then seamlessly reintegrated into the intermediate L.MM's s layer via a skip connection. Together with improved comprehension of the
the original intermediate output, VPG-C is expected to provide demonstration instructions. Yet, VPG-C is not ready to follow zero-shot demonstrative instructions because the "Guide" step requires fine-tuning specialize in missing detail recovery. Therefore, we propose a synthetic discriminative training strategy to fine-tune VPG-C, without the need for the expensive data collection of "detail-caption" pairs (Section 2.2).
To evaluate VPG-C and diagnose existing MILMM,, we build DEMON, a comprehensive benchmark for demonstrative instruction understanding, covering 31 diverse tasks across categories, as shown in Figure (Section)) Systematic evaluation on DEMON confirms the limitation of existing MLLMS in demonstrative instructions. Without additional demonstrative instruction data, the lightweight VPG-C module can be effectively tuned by the synthetic training strategy in several hours with a single A100 GPU. While computation- and data- efficient, VPG-C significantly outperforms ex- isting MLLMs on the DEMON benchmark. Zero-shot evaluation on other multimodal instruction benchmarks (Fu et al. 2023
(Section3).
Ye al. 2023) also indicates considerable improvement by VPG-C
METHOD
2.1 VISUAL PROMPT GENERATOR COMPLETE
As illustrated in Figure VPG-C C built upon the frozen LLM (Vicuna-7B (Chiang et al., 2023)
and vision encoder -LLIP Fang al. 2023 We adopt the widely used Q-Foreer from BL.IP.
2

---

Published as a conference paper at ICLR 2024
2 (Li et al. 2023c as visual prompt generator. VPG-C first uses the intermediate output of the LLM to infer instruction-specifee guidance. This then assists the VPG in attending to the missing visual details from the images. By merging these residual details back via skip connection, VPG-C achieves a thorough grasp of the demonstrative instruction.
Given a demonstrative instruction, we first adopt the Q-former generate general visual prompts for each image in the instruction. Q-former takes a fixed number of K query vectors to in- teract with image features by several cross-attention layers, and the output query representa- tions are used as visual prompts, which are inserted into the position of their corresponding im- ages in the instruction. We denote the input instruction for the language decoder as H0 (h]. h2. ..., vfl. ..., Vir..., hi...... VJI "i...., jK. .... hy J. where hς represents the i-th text token and vg. [V] ....... v2x) } represents the K visual prompts for the j-th interleaved image. Taking the instruction as input to the L-layer language decoder, we then extract the hidden representation of the last input token h at the 4-th layer, which can fully perceive the whole multimodal context during the first 1 layers and contains comprehensive instruction-aware semantics. Next, we infer the instruction-speeific guidance g from h via linear projection layer: g Lincar(h 3/3
After obtaining the instruction-speiifi guidance from the language decoder, we compose with a new set of learnable queries: g ++ 2, where Q R xd and g is added to each query of Q. Then, we reuse the same Q-former with the above conditionally generated queries to attend to residual visual details, thus obtaining the visual prompts V, {V,I,-.. ...; V5x) for each image j. which contains the complementary details missed by the original visual prompts. Finally, the transformed V, is reintegrated with the corresponding original intermediate representations of D//? via skip connection: 17.12 v/./2 Linear(V,), which is taken as the input the (5 + 1) layer.
Efficient training. Our VPG-C module is parameter-efficien as the Q-former is frozen and only set of query embeddings and two linear projection layers need to be fine-tuned, which only account for 0.09% (~6.3M) of the entire model. To stabilize the training process (Zhang & Agrawala, 2023), we initialize the linear projection layers with zeros. Thus, at the early training stage, the input to the
( + 1) layer can be converted to: D///2 = vL//2 =>///2 which will not cause any influence on LLMs.
Analysis on inserting VPG-C in the intermediate layer (: 1) Guidance generation. Previous studies have shown that features provided by the intermediate layer may suffice to preliminarily understand the given input samples (Xin et al. 2020) and can serve as guidance hints improve training (Romero et al.] 2014). Thus, generating guidance in the intermediate layer allows the model to form a preliminary understanding of the given instruction. Generating guidance too early could be problematic, as the model might not have gathered sufficient contextual information to generate effective guidance. Conversely, generating guidance too late could result in the model's attention being narrowly focused on what it perceives as the final answer, hindering its ability to guide the Q-former to extract relevant details from the images. Therefore, placing the guidance generation step in the intermediate layer strikes a balance. 2) Detail reintegration. Intermediate-layer rein- tegration of residual visual details preserves prior knowledge and allows subsequent layers to in- tegrate new information effectively. Reintegrating residual details too early in the pipeline might overwrite important context, while reintegrating it too late could limit the impact on the model's reasoning. Therefore, the intermediate layer offers a strategic position for residual details reinte- gration, enabling the model to reason effectively and arrive at the correct answers by leveraging the complemented visual residual details. We further provide quantitative analysis in Section 4.4
2.2 SYNTHETIC DISCRIMINATIVE TRAINING STRATEGY
The proposed training strategy diagnoses the areas initially ignored by Q-former according to its cross-attention maps between the queries and the image features, and generates synthetic image by performing several types of editing on the ignored areas. Then, an inter-image discriminative task is formulated deseribing the subtle difference between the original and the synthetic images. Considering the edits are performed in the mostly ignored areas, VPG-C is challenged to recover the missing details to describe the difference. An overview is illustrated in Figure R
Editing target identification. The Q-former takes the queries to interact with frozen image fea- tures through several cross-attention layers and uses the output query representations the visual
3

---

Published as a conference paper at ICLR 2024
Doleting Object Modifying Objoct
Instructio:
differance
imugee
Caremb descrppton
Segment Anything
Difktiin
Input Image
Segment Map
Adding
Swappin Object
Edlted
Igmarad
MLLM
Q Former
erea3
Rasporsa:
In virggn the cer on rooe yellow wherods the imoge. red."
Attention Map
Editing Target
Cross-Attention Guided Synthetic Image Editing
stones....
Inter-Image Discriminative Training
Figure 3: Pipeline demonstration of synthetic discriminative training strategy for VPG-C.
prompts. Therefore, the cross-attention maps between queries and image features reflect the interest of queries. We average the cross-attention maps across all layers and all queries to obtain the global cross-attention map A, where the value Aij indicates the significance degree of the corresponding image feature by the original task-agnostic Q-former queries. After that, we employ the advanced vision foundation model (Kirillov et al.) 2023) to obtain all the objects with segmentation masks in the image. Then, the significance degree of each object ((;;) computed based on the cross- attention map A with RolAlign (He et al. (017). where average the values of A within the mask m, by interpolation. ((.) reflects what degree the visual features of object og extracted by the Q-former. Thus, select the most ignored objects based on the ((o,) value.
Editing description generation. We define four types editing: modifying objects, swapping objects, deleting objects, and adding objects. Given the selected object, we instruct ChatGPT (Ope- nAI 2023a) to generate a suitable editing description that is harmony with the context, where ChatGPT prompted with the corresponding image caption and detailed object information (i.e. labels, positions). For adding objects, we only select BACKGROUND objects to add objects.
Synthetic image generation. After obtaining the editing description, generate the synthetic image using text-to-image latent diffusion model (i.e., Blended Diffusion (Avrahami et al. 2022)) )- Blended Diffusion performs local editing the image according the target object mask and the editing description, thus rendering the synthetic image. To ensure quality, we filter the edited images using CLIP similarity (Radford al. 2021b).
Inter-image discriminative training. Given the original image and the synthetic image pair, along with the task instruction ("Describe the difference between the images") the inter-image discrimi- native training task is defined as generating sentences to describe the subtle difference between the images. We convert the editing description acquire the ground-truth sentences.
DEMON BENCHMARK
Data format. All task instances are transformed into unified instruction-response form for zero- shot evaluation. Formally, each instance in DEMON consists of the following components:
Dask_Instruction provides a complete natural language definition of a given task, includ- ing the input/output format and the task objective.
Task_ Jask_Instance: is concrete instance of given task that consists of demonstrative image- text sequential context (e.g., visually-rich textbooks, specific questions about the context).
Response: represents the target output in natural language for given task instruction and task instance. For classification tasks, convert the class labels as options into the instruction and ask the model to output the option index in natural language as the response.
Without any specific emphasis, we use the term "instruction" to refer to the combination of Task_instruetion and Task Task_Inst Instance. For each task, manually design 10 Task_Instruction templates natural language to increase the diversity.
Task collection and categorization. To comprehensively benchmark the demonstrative instruc- tion following ability, we extensively gather a wide variety multimodal datasets from different fields and scenarios. As illustrated in Figure DEMON has three important properties: 1) Demon- strative vision-language context, all the instructions contain sequences of inter-related images and texts, such storyboards with scripts, and texthooks with diagrams. 2) Diverse forms of complex
4

---

Published as a conference paper at ICLR 2024
Category: Visual Storyteling
Category: Muts Mon--Moaall Dialogue
Instruction: Génan alctess pazse secerall trapaes
fresh the akory baned rmsoe??
Inatructions Responses
As on inlligent assizint you usmal: dbjcclive place soap container counter Blat by carelilyy utsersing ging your surounddage moterg) by
ush Mile your
1pt dhe Virill Bai onls.
prattae of Vieer Meelts varee
Syppriattt wtl ride
unique pait
Category: Visual Relstion Inference
Category: Text Rich Images QA
Turn left and walk the fight
Insmuction: Please andyzo the data these sides and accordingl proposed
What mocifoatioss con bo observed
Ies hecheeett thecee han pistuned?
Live
Open betton lelt cabénst and jane out the dioseui soup container
OK: Whet followe??
Response:
There thoee perzple phatts left and none the photo on h gitt.
user hus 242 connect
350 mill members whern
Response: Over 25 million
Category: Multi Reasoning
Category: Multi Modal Cloze
Instruction:
Incractio:: With sequence comii panels providedt could Hondly the most relevant nest pand from the avallable cholces?
pietunns
2eii stya??
Yos. They are in simiirr style
Category: Knowledge
Instruatio:: Provided with seriex diegnams from Mcbbck your negporabblly comecily amooee the follwing quealion
The dissocaation turve for hemoolootén Based the grash what lkdly
A. decrease in corbor dixx ido, Introase
pel, or decreoso temperature
increase carton dioadda,
carbon di, decreeso
the shoowm distedd pirt lise?
pit or decrneoe semperatue
D. decroase in carbor dlox de. inproase
pri or inerease in lempor
Resporse:
disaate,
Figure 4: Demonstrations and task taxonomy of the proposed DEMON benchmark.
instructions, the instructions range from designing panels for comics, discovering differences between surveillance images, and to conversational embodied tasks. 3) Vast range of instruction- following scenarios, the benehmark covers multiple practical scenarios, including cartoons, indus- trial visuals, driving recordings, recipes, etc.
Evaluation protocols. Thanks the unified task format of DEMON, all tasks can be evaluated in zero-shot manner. For the open-ended generation tasks, we adopt ROUGE-L L for evaluation. For the tasks that require the models to output option indexes, we take Accuracy as the evaluation metric. While well-formated options are provided, we empirically observe that many MLLMs struggle to strictly follow instructions output the option indexes but generate free-form text. Thus, when models do not exactly output the required options, match their outputs to of the given options based on the TF-IDF distance, which we find is more robust than model- based methods (OpenAl, 2023al Reimers & Gurevych| 2019). Since explore large number of tasks, take maximum of 500 instances per task for evaluation efficiency and exclude several datasets that are difficult to obtain and are subject strict copyright restrictions (referred to DEMON DEMON-Core), Meanwhile, we report the full version of the benchmark to facilitate future research on large-scale multimodal DEMON-Full Unless specifically stated, we DEMON refer
instruction tuning (referred to DEMON-Core in the following.
Table 1: Detailed statistics of DEMON benchmark.
Tasks
Scenarios Images Instructions Avg. Images Instruction Avg. Words Instruction
DEMON Core
29
62.8IK 18.18K
3.46
92.69
DEMON-Full
31
20
1.77M
477.72K
3,70
97.58
details the statistics. DEMON benchmark covers 31 tasks cat- egories across 20 scenarios. In total, DEMON-Full includes 477.72K instruction-responsee pairs,
Benchmark analysis. Table
5

---

Published as a conference paper at ICLR 2024
Table 2: Average results of zero-shot evaluation on each task category DEMON Benchmark.
Multinodal Dialogue
Visual Storytellng
Visual Relation
Multimodal Cloze
Knowledge Grounded QA
Text- Text-Rich
Multi-Inage
Inference
Images QA
Reasoming
BLIP Lt R0z3
26.12
21:31
17.94
39.23
35.53
39.65
Instractiir Dai e 2025
33.38
24.41
11.49
21:20
47.40
44.41]
48.55
LLaMA-Adipa
2023
14,22
17.57
13,51
18.00
44.80
32.00
44,03
LLaV (Lis
MiniGPT (Zar 20236
13.69
17.07
7.95
16.60
30.27
26.40
43.50
mPLUGGOOT 18 20343
12.67
19.33
5.40
16.25
33.27
32.47
42,50
Otier (L. ct al. 10258
15.37
15.57
11.39
16.00
11.67
27.73
43.85
VBG-C
37,50
25.20
25,90
22.15
48.60
44.93
50.28
serving as large-scale benchmark for demonstrative instruction following. On average, each in- struction contains 3.70 images and 97.58 words. Please refer Appendix for more details.
EXPERIMENTS
4.1 ZERO-SHOW EVALUATION ON DEMON BENCHMARK
Comparison with advanced MLLMs. In this section, conduct comprehensive evaluation of our VPG-C and the recent advanced MLLMs on the proposed DEMON benchmark. For all methods, we choose versions with parameter sizes less than 10B. Please refer to Appendix D for details. The average results of each task category are summarized in Table which indicates the following.
VPG-C consistently outperforms existing models by large margin across all categories, which demonstrates the stronger generalizability to follow such complicated demonstrative instructions.
While previous works mainly fine-tune on massive multimodal instruction data, VPG-C still achieves the highest performance using synthetic training data with much lower computation cost. This validates the effectiveness of the proposed VPG-C module and its synthetic training strategy. Compared with previous works that fine-tune the large-scale language decoder or visual en- coder (i.e., LLaVA, mPLUG- our model only tunes the lightweight VPG-C module with 6.3M parameters and achieves significant performance gain.
VPG-C exhibits significant superiority in several challenging tasks. For instance, VPG-C sur- passes SOTA methods by 3.92% multimodal dialogue, which requires models effectively associate the interleaved images mentioned in different turns of the dialogue.
Innovative findings. The extensive evaluation on DEMON benchmark reveals several key findings.
Poor performance on demonstrative instructions. While several models (e.g., OpenFlamingo, Otter, mPLUG-owl) have been trained on interleaved lisguanlanguage data, such mme4 (Zhu jet al. 2023b), they still struggle to perform well on the demonstraive instructions. We suppose that while mmc4 contains sequences of interleaved images as context, the web-crawled images are often weakly related. In contrast, the images and text in demonstrative instructions are highly related, requiring models deeply associate them to comprehend the task intents.
Limited instruction following ability. Despite existing vision-language models leveraging state- of-the-art art LLMs, which have demonstrated impressive ability in following language instructions, this competence seems falter when dealing with complex multimodal instructions. For instance, when tasked with selecting the correct answer from choice list given the context of images and texts, we observed some models inclining more towards describing the contents of the images instead of addressing the posed questions. This is perceived as deficiency in the image-text alignment training process, to which we attribute the discrepancy.
Failing to process image-choice questions. When dealing with multimodal cloze tasks, all mod- els are limited to processing instructions that involve images options. We hope future work to utilize the new benchmark to make progress on this type of demonstrative instructions.
4.2 ZERO-SHOT EVALUATION ON MME BENCHMARK
We evaluate our VPG-C on the concurrently proposed MME benchmark (Fu et al., 2023) to further illustrate its strong generalizability follow diverse range of single-image instructions. MME benchmark measures both perception and cognition abilities on a total of subtasks. We report the averaged results of perception tasks and cognition tasks in Table 3 respectively. While

---

Published as a conference paper at ICLR 2024
Table 3: Zero-shot evaluation of perception and cognition abilities on MME benchmark.
BLIP-2
InstructBLIP
LA-V2
LLaVA
MinGPPT4
mPLUG-Owl
Otter
VPG-C
Perception 1293.84
1212.82
972.67
502.82
866.57
967.34 276.07
1292.26
1299.24
Cogmition 290.00
291.79
248.93
214.64
292.14
306.43
321.07
do use massive multimodal instruction data fine-tune VPG-C, VPG-C still achieves superior performance, compared with the supervised instruction-tuned models. This indicates our method effectively overcomes the inherent limitation of VPGs and the completed residual details are also essential for single-image instructions. Please refer to Appendix for detailed results.
4.3 HUMAN EVALUATION ON GENERAL-PURPONT LANGUAGE GENERATION
We further conduct human evaluation on the OwlEval benchmark (Ye et al., 2023), which contains 82 open- ended questions including advertisement and poem cre- ation, diagram and flowchart comprehension, and teach- ing, etc. Specifically, we recruit well-educated people to rank the randomly shuffled responses from VPG-C, MiniGPT-4, mPLUG-OWl, OpenFlamingo, and Instruct- BLIP. The scores range from to 5 (5 means best) and are allowed to be equal for comparable instances. As shown in Figure 5 VPG-C also demonstrates better open-ended language generation ability various practical cases.
4.19
3.62
3.24
2.69
1.89
OpenFlamino InstuciBLIP MiniSPT4
VP6CC
Figure 5: Human evaluation.
4.4 IN-DEPTH ANALYSIS
Effectiveness of individual components. We investigate the effectiveness of each component in Table We start with the backbone model that uses the Q-former to generate visual prompts. 1) Instead of applying VPG-C to capture missing details, we first attempt simple heuristic-based method that directly extracts the less attended visual features according to the cross-attention maps of Q-former and reintegrates them to the intermediate layer the LLM as ours. We fine-tune linear layer before reintegrating with 0.5 million image-captin pairs. The results of Row 2 show that such a sample heuristic can bring some improvement. This validates the importance of re-extracting missing visual features from images for comprehending demonstrative instructions. 2) Then, we replace with VPG-C C and train only using the image-caption pairs without synthetic training. The results of Row 3 demonstrate that VPG-C can more accurately complete the required missing details by leveraging the intermediate inference results of the LLM. 3) However, solely using common image-caption data not fully unleash the power of VPG-C. Comparing Row 3 and Row 4, observe a significant improvement for all tasks, indicating that the proposed synthetic discriminative training strategy can methodically empower VPG-C to extract missing visual details.
VPG-C can better guide VPGs. Since InstructBLIP can perform conditional visual feature ex- traction, implement variant version that concatenates its initially generated answer with the instruction condition to re-extract features. The initial generated answer serves as an additional heuristic from the LLM for guiding feature extraction. Then, the newly extracted visual prompts are used to re-generate answers. For fair comparison, we provide a zero-shot version (Row 5) and a fine-tuned version (Row 6) using synthetic training as ours. As shown in Table 4 directly using synthetic data and inferred answers as heuristic conditions fails to yield notable improvement. In contrast, VPG-C can better guide the VPG to complete the missing visual details by intercepting it only
intermediate representations of the LLM. Further, VPG-C is more computation-efficien requires one full forward pass of the LLM, while the InstructBLIP variants require twice.
VPG-C works well various language backbones. Table also validates that approach can well cooperate with language backbones of different families (LLaMA2) and sizes (Vicuna-13B)
VPG-C can be implemented with very simple VPG. As generic method, VPG-C can be im- plemented with different VPGs. Beyond the widely used Q-former that is composed of multiple Transformer blocks, further probe the effectiveness of VPG-C with simpler VPG, i.e., Linear Projection, as used in LLaVA (please refer to Appendix C for implementation details). Table Row 7 shows promising results. VPG-C can also significantly bolster the performance of simple

---

Published as a conference paper at ICLR 2024
Table 4: Ablation results on DEMON Benchmark.
Multimodal Dialogae
Visual Storytelling
Visual Relation Inference
Multimodal Cloze
Knowledge Grounded QA
Text-Rich Images QA
Mult--lmae
Reasoning
Backhone
25.65
21.72
9.33
17.06
37.21
32.42
41.30
*Houristtc Details
28.13
22.76
12.69
18.81
38.75
34.14
43.26
+VPG-C
31.76
23.62
19.12
20.09
42.53
39.68
46.71
-Synthetic Training
25.20
25.90
22.15
48.
44.93
50.28
InstructBLIP
33.58
24.41
11.48
21.20
47.40
44,40
48.55
*Answer Condition
32
23.76
11.02
42.08
-Synthetic Training
31,76
24.32
12.78
19.87
46.58
42.36
49.82
10.70
8.27
15.85
36,20
28.35
7
Lincar VPG
16.43
19.48
14.75
18.54
41.32
36,87
46:02
8
VPG-C- LLL4M22277
42.70
24.76
25,50
22.95
$1.00
44.93
48.68
9
VPS
Vicuna 130
38.14
26.59
27.15
27.15
52.93
49.33
53.65
linear VPG, verifying the transferability of VPG-C. It is promising adapt our generic VPG-C and corresponding low-resource synthetic training strategy to different VPGs in the future.
Analysis the inserted layer of vPG-C. We investi- gate the impact of inserting VPG-C C into different layers
50
Arcuney
45
of LLMs. We report the averaged accuracy for multiple- choice tasks and averaged ROUGE-L for open-ended generation tasks in Figure 6
40
35
We observe that the per- formance is low when we insert VPG-C too early (i.e. 4, 8) as the model might not have gathered sufficient con- textual information to generate effective guidance. Mean- while, inserting VPG-C C too late (i.e., 24, 28) degenerates the performance. We speculate this is due the gener- ated guidance being too concentrated and there not being enough layers integrate the residual details.
25
20
16 Layer VPG-C
Figure 6: Performance on DEMON with different insertion layers.
Synthetic training is data-efficient. Since our proposed synthetic training strategy can construct challenging discriminative tasks in a targeted manner, enhancing VPG-C's ability to complete missing details, it avoids the need for large amount of supervised demon- strative instruction data. We further investigate the impact of differ- numbers of synthetic training data. As illustrated in Table the performance keeps increasing when the number of data is increased from 16K to 64K. Beyond this, escalating the data count from 64K to 80K yields only marginal enhancement. Further amplification of data eventually triggers per- formance dip as excessive data leads model overfitting to the synthetic training task.
Table 5: Efficiency analysis of synthetic training.
Accuracy
ROUGE-L
16K
38.93
25.67
32K
39.62
27.38
48K
40.45
28.81
80K
41.62
29.73
96K
40.12
28.31
Image order sensitivity. The order of inter- leaved images demonstrative instructions is pivotal for the compositional semantics of the instruction. Intuitively, altering the order of images within demonstrative instruction can significantly shift its semantics. Consequently, reveal the
20
Mos-GPT4
Opan gorgg
10
variations in model performance model's 's sensitivity to the instruction semantics. An ideal model should keenly capture changes in instruction semantics. Therefore, we visu- alize the performance variations of models randomly shuffling the order of interleaved im- ages within the demonstrative instructions. Ac- cording to Figure 7 we surprisingly find that SOTA models are less sensitive to the image order. In contrast, VPG-C can keenly capture the semantic changes caused by the shuffled image order. Particularly, performance varies dramatically in multimodal dialogue, as the order of images within these tasks is closely intertwined with the dialogue content.
Indrueglii
Muli
Modil Dialog
Reaaion
Modal
Grounded
Rich
Te
Inference
Cloze
QA
InegesReasaning
QA
Figure Analysis on image order sensitivity.
Qualitative examples. As illustrated in Figure VPG CC demonstrates strong abilities to perform reasoning over complicated demonstrative instructions. For instance, in (a), VPG-C can keenly identify the connections between the images and thereby infer the reason that causes this unusual phenomenon. (b, c), VPG-C exhibits the ability to comprehend absurd objects through mul- timodal conversations with humans. In (d, e), VPG-C reasonably infer the relations among
8

---

Published as a conference paper at ICLR 2024
og temees Giglts dedecas anagge
Rivey ell unal no. Moree Fo baggog
ciera podiaaty dig the wate meree
AA Lar ofageol
Dhe pulfiul voluri uries
para oud pacuad paruil
dest That somicaa colatios
bnty the roles orlaniss unile sion brreed crdee.
Lsed bualing pers!
(b)
Google
Thes alc came saap shore change di
Uo pioduc Clean cagges aia
rergen
and very dear in thiii picuus some hus come can and aronee well denu ofa car the the buiiiiy slraiig thil our
no
conurdaiion ven ven in fumor las
(c)
Figure 8: Qualitative examples generated by our TTPG-C-Vicune 7B model.
the images and understand the metaphorical implications they want to convey. In Appendix G we provide more practical examples as well as comparisons with other MILMss, where we find that baseline models fail to correctly associate multiple images and comprehend demonstrative context.
5 RELATED WORK
MLLMS Yin et al. 2023) aim to serve as a general-purpose assistant to perform various vision- language tasks by free-text generation. Flamingo (Alayrac et al. [002) and BLIP-2 eLi (023c)
bridge LLMs with powerful pre-trained visual encoders and demonstrate strong zero-shot ability
by aligning visual features with LLMs. Follow-up works of LLaVA (Liu et al. 2023), MiniGPT-
4 (Zhu et al., 2023a). InstructBLIP (Dai et al., 2023). Hallucidoctor (Yu et al.) 2024), mPLUG-
Owl (Ye et al. 2023) propose fine-tune MLLMs with multimodal instruction tuning data. To
effectively benchmark the recent progress in MLLMs, concurrent works of L-elMe (Xu et al.
2023) and MME Benchmark (Fu et al.] 2023) are proposed, while they mainly focus on instructions that only involve single image with limited instruction diversity. In this paper, we propose the
first demonstrative instruction-following benchmark, covering various tasks of diverse scenarios. Further, we propose a lightweight and generic VPG- C module to address the inherent limitation of current VPGs. Our VPG-C is efficiently tuned by our synthetic discriminative training strategy, which demonstrates powerful potentials of text-to-image diffusion models (He et [022]] Lin et al.)
2023 Prabhu et al., 2023] Bansal & Grover 2023
Yu et al.
2023b) to facilitate vision-language
understanding (Radford et al.] 2021b Tia et al. 2021
et al.
2022b
CONCLUSION
In this paper, we propose VPG-C. generic and parameter-efffcient approach that infers and com- pletes the missing visual details for MI.LMs to comprehend demonstrative instructions with inter- leaved multimodal context. Meanwhile, we present a synthetic discriminative training strategy to fine-tune VPG-C, eliminating the need for supervised demonstrative instruction data. To foster the research on demonstrative instruction understanding, we build DEMON, a comprehensive benchmark for multimodal large language models, consisting of 31 tasks with complicated vision-language demonstrative context, covering a wide range of scenarios. Through synthetic training, VPG-C showcases notable zero-shot performance on the DEMON benchmark. Its superior performance on other established benchmarks like MME and OwlEval further underscores its effectiveness.
Acknowledgment. This work was supported by the NSFC (No. 62272411). Key Research and Development Projects in Zhejiang Province (No. 2024C01 106), the National Key Research and De- velopment Project of China (2018AAA010190) the Tencent WeChat Rhino-Bird Special Research Program (Tencent WXG-FR-2023-10), and Research funding from Fin Volution Group.

---

Published as a conference paper at ICLR 2024
REFERENCES
Lean-Baptistt Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, March 2023. URL https:
10. 5281/ /enooo. ctttoddo.7735555 7733589
Haoping Bai, Shancong Mou, Tatiana Likhomanenko, Ramazan Gokberk Cinbis, Oncel Tuzel, Ping Huang, Jiulong Shan, Jianjun Shi, and Meng Cao. Vision datasets: A benchmark for vision-based industrial inspection. arXiv preprint Xxix:2306.078%) 2023.
Ankan Bansal, Yuting Zhang, and Rama Chellappa. Visual question answering on image sets. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pp. 51-67. Springer, 2020.
Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. arXiv preprint arXiv:2302.02503, 2023.
Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different an- swers? In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 2019.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Wehqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16495-16504, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, 2023. httpst //vi cuna. Imsys org
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee language mod- els. arXiv preprint arXiv:2210. 11416, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose viniun-language models with instruction tuning. arXiv preprint arXiv:2305.06533 2023.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19358-19369, 2023.
Maxwell Forbes, Christine Kaeser-Chert Piyush Sharma, and Serge Belongie. Neural naturalist: generating fine-grained image comparisons. arXiv preprint arXiv: 1009: 04/01, 2019.
10

---

Published as a conference paper at ICLR 2024
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprchensive evaluation benchmark for multi- modal large language models. arXiv preprint arXiv: 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, al. Llama-adapter v2: Parameter-effiieet visual instruction model. arXiv preprint X31v:2304.15010, 2023.
Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to compositions videos. In Proceedings of the European conference computer vision (ECCV), pp. 598-613, 2018.
Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Automatic spatially-awarr fashion concept discovery. In Proceedings of the IEEE international conference on computer vision, pp. 1463-1471, 2017.
Darryl Hanman, Akshay Jain, and Mohit Bansal. Manymodalqa: Modality disambiguation and qa over diverse inputs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7879-7886, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiao- juan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.0757. 2022.
Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learning from an auxiliary task. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2725-2734, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09663 2021.
Ting- Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for compu- 2016.
tational linguistics: Human language technologies, pp.
Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations image collections. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1383-1391, 2015.
Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Graber, Hal Daume, and Larry S Davis. The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp. 7186-7195, 2017.
Harsh Jhamtani and Taylor Berg-Kirkpattrn Learning describe differences between pairs of similar images. arXiv preprint arXiv:1808 10584, 2018.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- laiguage representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp. 4999-5007, 2017.
11

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint (rXXi:2334400264. 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li. and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven CH Hoi. Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.09019 2022a.
Juncheng Li, Xin Wang. Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang. and William Yang Wang. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
12123-12132, 2020.
Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned languale pre-training. Advances in neural information processing systems, 35:7290-7303, 2022b.
Juncheng Li, Junlin Xie, Long Qian, Linchao Zhu, Siliang Tang, Fei Wu, Yi Yang, Yueting Zhuang, and Xin Eric Wang. Compositional temporal grounding with structured variational cross-graph correspondence learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3032-3041, 2022c.
Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat-Seng Chua, and Yueting Zhuang. Gradient-regulated meta-prompt learning for generalizable vision-language models. 2023b.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 2rXiv:2301.12597 2023c.
Mengze Li, Tianbao Wang, Haoyu Zhang, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang, Wenming Tan, Jin Wang, Peng Wang. et al. End-to-end modeling via information tree for one- shot natural language spatial video grounding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8707-8717, 2022d.
Mengze Li, Haoyu Zhang, Juncheng Li, Zhou Zhao, Wenqiao Zhang, Shengyu Zhang, Shiliang Pu, Yucting Zhuang, and Fei Wu. Unsupervised domain adaptation for video object grounding with cascaded debiasing learning. In Proceedings of the 31st ACM International Conference Multimedia, pp. 3807-3816, 2023d.
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carl- son, and Jianfeng Gao. Storygan: A sequential conditional gan for story visualization. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6329-6338, 2019.
Yongqi Li, Wenjie Li, and Liqiang Nie. Mmcoqa: Conversational question answering over text, tables, and images. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pp. 4220-4231, 2022e.
Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the power of synthetic data on few-shot object detection. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 638-647, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304 08485, 2023.
Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Storydall-e: Adapting pretrained text-to- image transformers for story continuation. In Computer Vision-ECCV 2022: 17th European Con- ference, Tel Aviy, Israel. October 23-27, 2022, Proceedings, Part XXXVII, pp. 70-87. Springer, 2022.
12

---

Published as a conference paper at ICLR 2024
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
OpenAl. Chatgpt: A language model for conversational ai. Technical report, OpenAL, 2023a. URL
OpenAl. Gpt-4 technical report. arXiv:2303.08774, 2023b.
2023b
Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman. Lance: Stress- testing visual models by generating language-guided counterfactual images. arXiv preprint UrXiv:2305.19166 2023.
Alec Radford, Jong Wook Kim. Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR, 2021b.
Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, and Mubbasir Kapadia. Aesop: Ab- stract encoding of stories, objects, and pictures. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2052-2063, 2021.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.633: 2014.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning, In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Longg Papers), pp. 2556-2565, 2018
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 10740-10749, 2020.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv: 1.00491, 2018.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang. Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images. arXiv preprint arXiv: 104.06044 2021.
Hao Tan, Franck Dernnncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relation- ships via language. arXiv preprint arXiv: 0tt0::/990007770 07689, 2019.
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida. Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. arXiv preprint arXiv:2301. 044883 2023.
13

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Amthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojmic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models. arXiv:2307.09228, 2023b.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9068-9079, 2018.
Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint ar 200: 12993, 2020.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-20577 PMLR, 2015.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lylm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09266 2023.
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler Cinbis. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. arXiv preprini arXiv: 1889.008/22 2018.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2xXx:2334414178 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. Visually-prompted language model for fine-grained scene graph generation in an open world. In Proceedings of the IEEE/CVF International Conference Computer Vision pp.
21
Qifan Yu, Juncheng Li, Wentao Ye, Siliang Tang, and Yueting Zhuang. Interactive data synthesis for systematic vision adaptation via llms-aiges collaboration. arXiv preprint arXiv:2305. 12799, 2023b.
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity visual instruction data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models.
arXiv preprint arXiv:2302.05544 2023.
Wenqiao Zhang, Siliang Tang, Yanpeng Cao, Shiliang Pu, Fei Wu, and Yueting Zhuang. Frame augmented alternating attention network for video question answering. IEEE Transactions on Multimedia, 22(4): 2019.
14

---

Published as a conference paper at ICLR 2024
Wenqiao Zhang, Haochen Shi, Jiannan Guo, Shengyu Zhang, Qingpeng Cai, Juncheng Li, Sihui Luo, and Yueting Zhuang. Magic: Multimodal relational graph adversarial inference for diverse and unpaired text-based image captioning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp- 3335-3343, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.00592 2023a.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. arXiv preprint arXiv:2304.08 2023b.
15

---

Published as a conference paper at ICLR 2024
A OVERVIEW
In this appendix, we present:
Detailed information of the proposed DEMON benchmark (Section B
Implementation details of our VPG- (Section C
Implementation details of existing MLLMs on the DEMON benchmark (Section D
Detailed zero-shot performance on MME benchmark (Section E
Detailed zero-shot performance on DEMON benchmark (Section)
Qualitative comparison with existing MLLMs (Section
B BENCHMARK DETAILS
Task
Sccnario
Dataset
Metire
Multimodal Dialogue
Conversational Embodied Dialogue
Embodied
ALFRED Shridhar et al. 2020
ROUGE-L
Multimodal Dialogue
Conversation
MMCoQA (Li ct al.
ROUGE-L
Visual Storytelling
Animated Story Complction
Cartoon
AESOP Ravi et al. 2021
ROUGE-L
Animated Story Completion
Cartoon
PororoS
ROUGE-L
Animated Story Completion
Cartoon
FlintstonesSV Gupta VIST (Huang et al. 2016
2018
ROUGE-L
Sequential Photo Storytelling
Album
ROUGE-L
Sequential Photo Storytelling
Cartoon
DiDeMoSV Mabarana et al. 2022
ROUGE-L
Visual Relation Inference
Visual Change Captioning
Surveillance
Spot-the-Die Diff' Jhamtuni & Berg- Kirkpatrick 2018
ROUGE-L
Visual Change Captioning
Synthetic
CLEVR- R-Chaang (loosemnzaueh & Wang 2021
ROUGE-L
Visual Relationship Expressing
General
IEdit Tan el al. 2019
ROUGE-L
Suhtle Difference Expressing
Fine-Grained
Birds-to- Wordy Forhes
2019
ROUGE-L
Multimodal Cloze
Comic Dialogue Identification
Cartoon
COMICS-Dialogue [yyer al 2017
Accuracy
Comic Panel Identification
Cartoon
COMMCC--Pance (yyer et al 2017
Accuracy
Recipe Completion
Recipe
RecipeQA-TexiClove TextCloze
Yagcioglu lu el al.
2018
Accuracy
Visual Step Cloze
Recipe
RecipeQA-
VisualCloze RecipeQA-VisualClozze
Yagcrogiu et 2018
Accuracy
Knowledge Grounded QA
Webpage QA
Wehpage
WehQA (Chang et al.
2022
Accuracy
Textbook QA
Textbook
TQA (embitav al.
Accuracy
Complex Multimodal
Wikipedia
MMQA [almor
Accuracy
Complex Multimodal QA*
Wikipedia
MANYMODALQA Hannan
2020
Accuracy
Text-Rich Images QA
Slide QA
Slide Book Cover
SlideVQA Tanaka al. 2023
Accuracy
OCR QA
OCR-VQA Mishra et al. 2019
Accuracy
Document QA
Document Image
DocVQA Maanew e al. 2121
Accuracy
Multi-Image Reasoning
Image-Set QA*
Indoor Egocentric
Gibson
(Bansal et al.
2020 Xia al.
2018
Accuracy
Image-Set QA
Driving Recording
nuScenes
Bansal
2120
Accuracy
Industrial Inspection
Industrial
VISION Ba
Accuracy
Fashion QA
Fashion
Fashion200K (Hán et 2017
Accuracy
Property Coherence
General
MIT Staates-PropertyCCffeee [ssola al
2015
Accuracy
State Transformation Coherence
General
Www-Stttte--StteChheeeeee [Isola et 2015
Accuracy
Visual Step Matching
Recipe
(Yagcioglu el al, 2018
Accuracy
Multi-Image Visual Entailment
General
NLVR2 (Suhr al. 2018
Accuracy
Ambiguity Analysis
Mobile Photo
WizWiz Bhaltacharya
Accuracy
Table 6: Summary of the demonstrative instruction-following tasks in DEMON benchmark. indi-
cates the tasks that are not included in TEMON-Cores
C IMPLEMENTATION DETAILS
Model. We choose ViT-G/14 from EVA-CLIP (Fang al. 2023) visual encoder and pre-
trained Q-former from BLIP-2 without instruction tuning as the task-agnostic visual prompt gen- erator. For the large language model, implement three versions: LLaMA2-7B (Touvron al.
16

---

Published as a conference paper at ICLR 2024
Step1: Reasoning with Task-Agnestie Visual Prompts (VP)
Step2: Visual Prompt Generator Complete (VPG-C)
Step3: Reasoning with
Froen Guenes
Quediss
Ineruction-Spaciie
Lineer
Langgage Respora
Flatfonss Gerwrllge
LLM Layer +
Image Enceder
#Q-Former
LLM Layer 7
LLM L Layer L/2
FQrmerer
Resiboal Detalo Echoction
LLM Layer
Nittigrahoo
Linear
Vistens ierguege Instructioe
JnstuctionSpwesll Vasv Pronppte
AP
tokken
Vitual
Mi roused Forser tales the sane spou Son She srage onceadr
Figure 9: Detailed framework of our MLLM enhanced with VPG-C. C.
2023b), Vicuna-7B (Chiang et al. 2023 Vicuna-13B, with 32, 32, 48 Transformer layers, respec- tively. We derive instruction-specile conditions from the 16th 24th layer and re-inject the condi- tional visual knowledge into the 17th / 25th layer. Furthermore, we provide detailed framework of our MLLM enhanced with VPG-C in Figure
Choice of Q-former. Recently, InstructBLIP (Dai et al. 2023) proposes to take the instruction as additional input the Q-former and fine-tune the Q-former to extract visual features according to structions using 16M multimodal instruction tuning data. While achieving outstanding performance on in-domain tasks, a recent study (Xu et al., 2023) indicates that fine-tuning on massive in-domain data severely undermines its generalizability on open-world scenarios. Instead of directly relying on the Q-former to achieve task-specific feature extraction by massive instruction tuning, we aim to utilize the sophisticated reasoning ability of LLMs to guide the Q-former to conditionally attend to residual visual details. Thus, we use the Q-former without instruction data tuning from BLIP-2 et al. 2023c), which extracts the task agnostic primary visual contents at the first time.
Training. We implement VPG-C in LAVIS library (Li et al. 2022a). We keep the visual backbone, visual prompt generator, and the language model frozen, and tune the VPG-C module using the proposed training strategy. Since BL.IP-2 models do not include pre-trained Q-former that matches Vicuna and LLaMA2, we reuse the Q-former that matches F5aXT5-XXL and fine-tune the last linear projection layer with 5 million image-text pairs to align it with Vicuna/LLaMA2. All the tunable parameters of our VPG-C module are set of query embeddings and two linear projection layers, which only accounts for 0.09% (~6.3M) of the entire model. As for synthetic training, we select about 30k images from CC3M (Sharma et al. (2018) that contain significantly ignored objects and perform different types of editing on them. Totally, we generate approximately 64k synthetic images with suitable modifications. To stablize the training and avoid overfitting, we use 500k image- caption pairs from CC3M to jointly train the VPG-C module. We tune the VPG-C module for 18k steps using a batch size of 24 for synthetic training and 64 for image captioning, which takes about adopt the AdamW optimizer with 3 = (0.9, 0.999), and set the learning rate and weight decay to 0.00002 and 0.05, respectively. We warm up the training with 2k warm-up steps, followed by learning rate decay mechanism with the cosine schedule.
7 hours complete with a single A A100 GPU. Additionally,
Implementation of VPG-C with the linear VPG. As a generic method, VPG-C can be implemented with different VPGs. Beyond widely used Q-former that is composed of multiple Transformer blocks, we further probe the effectiveness of VPG-C with simpler VPG, i.c. Linear Projection, as used in LLaVA (Liu et al., (002).. LLaVA trains a simple linear layer as the VPG to connect image features into the word embedding space. To implement VPG-C with the linear VPG, we first linearly project the generated guidance g and then take it as filter to perform element-wise Hadamard product with the visual features X
from the image encoder:
V = (Wigr") @ (Wax")
where Wi and W2 are linear projection matrixes, 1 is the transpose of an all-ones vector, and O represents Hadamard product. The output represents the newly-extracted missing visual details according to the inferred guidance. And y is reintegrated into the L.LM in the same manner.
17

---

Published as a conference paper at ICLR 2024
D MODEL DETAILS IN DEMON BENCHMARK
Recent advancements in LLMs (OpenAl 2023a b) have heralded significant achievements across various domains. Inspired by this success, many MLLMS (Li et al.
2023c Liu et al. 2023 Zhu
et al., 2023a; |Aaayrac al., 2022 Ye et al. 2023 Gao et al., 2023
Li et al. 2023a have been Li et al., [202bb] 2020; Yu
proposed to foster generalist vision-lanee language reasoning (Xu et al. 2015
ct al.
2023a
Li et al., 2022d] Zhang et al. 2022 Li et a.. 2022c 2023d Zhang et al.) | Antol
et al. 2015). In our experiments, we conducted comparisons with some of the most recent and representative MILLMs in the following.
LLaVA (Liu et al., 2023 establishes a connection between the visual encoder ViT-L/14 from CLIP (Radford et al. 2021a) and the language decoder LLaMA (Touvron et al. 2023a), utilizing a lightweight, fully-connected (FC) layer. Initially, the system trains this FC layer using 595K image-text pairs, while keeping both the visual encoder and LLM static. Following this, LLaVa VA fine-tunes both the FC layer and LLM using dataset com- prising 158K instructional Dision-language pairs. The tested version "LLaVA-7B-voe
LLaMA-Adapter V2 (Gao et al. 2023 stands as a model of parameter efficiency within the realm of visual instruction. Despite maintaining the visual encoder (ViT-L/14) and the LLM in a static state, LA-V2 distributes the instruction-folloiigg capacity of the entire LLaMA system via bias-tuning. This method allows for the refinement of scale, bias, norm, and prompt parameters on diverse data sets. These include 200M image captioning data, 158K visual inslroction-follooiing data, and an additional 52K language instruction- following data, the latter of which was assembled by GPT-4 (OpenAl, 2023b). The tested version "LLaVA-7B""
MiniGPT-4 Zhuu et al. 2023a) bridges the gap between the visual encoder and text en- coder using fully-connected (FC) layer. Initially, this model trains the FC layer dataset comprised of 5M image-text pairs before fine-tuning it on 3.5K instructional vision- language data. Notwithstanding its simplicity, MiniGPT-4 requires the loading of pre- trained vision encoder from BLIP2, as well as a Vicuna LLM (Chiang et al. 2023). The tested version iminigpt--aligneed ali--ad-wiitttte wiiggeddwwit---vcmaa7"""
BLIP2 (Li et al.) 2023c) employs a dual-stage strategy to seamlessly bridge the modal- ity gap, utilizing lean Q-Former pre-trained 129 million image-text pairs. The initial stage kick-starts the learning process of vision-language representation, leveraging a frozen image encoder, the V/T-g/14 from EVA-CLIP (Fang al. (023). Subsequently, the sec- ond stage harnesses a frozen LLM, the FlanTS (Chung el al. 2022). to initiate the vision- to-language generative learning. This innovative strategy effectively facilitates zero-shot instructed image-to-text generation. The tested version is pranp---ettrii-fllmaxxx"
mPLUG-Owl (Ye et al. 2023) introduces visual abstractor, fundamentally close the Per- ceiver Resampler in Flamingo (Alayrac et al. (2022), as a bridge between the pre-trained vi- sual encoder ViT-L/14 and the LLM (LLMMA (Touvron et al. (023aa). This model adopts a two-stage fine-tuning procedure. In the initial phase, both the visual encoder and the visual abstractor undergo comprehensive fine-tuning using dataset of 204M image-text pairs. Subsequently, in the second phase, mPLUG-OWl applies the 158K LLa dataset to fine-tune the pre-trained LLM in a parameter--fffcieett manner through the use of The tested version is "mplug-on owl-llama-7b".
LoRA (Hu et al. 2021
2023a> is multimodal model that applies in-context instruction tuning based on OpwnFlamingee lamingo (Alayrac et al., 2022). This model integrates a LLaMA-7B (Tou- vron al. 2023a language encoder and CLIP ViT-L/14. While the visual and text encoders remain static, Otter refines an additional 1.3 billion parameters. These parame- ters are derived from adaptation modules and are trained using 158K instruction-follooinng data. The tested version is "OTTER- Laamg--LLMM77------AAA
Otter (Li et al.
InstructBLIP (Dai al. 2023) originates from pre-trained BLIP-2 model, which con- sists of a ViT-g/14 14 image encoder, a Vicuna, and a Q-Former to act as the bridge between these two components. During the process of dision-language instruction tuning, only the Q- Q-Former undergoes fine-tuning tine-tunigg, with the training process leveraging data from 13 distinct visual question-answering datasets. The tested version is "blip2- Siin---viunn--nstuue-777
18

---

Published as a conference paper at ICLR 2024
OpenFlamingo (Alayrac al. 2022 Awadalla al. 2023 represents of the pioneer- ing efforts to incorporate Language Model Learning (LLMs) into the domain of vision- language pretraining. To optimize its conditioning visual features, Flamingo strategi- cally integrates a number of gated cross-attention dense blocks amidst the layers of the pre-trained language encoder. OpenFlamingo offers open-source rendition of this ad- vanced model. The tested version is "llama-7b".
The DEMON benchmark predominantly features interleaved vision-language instructions, distin- guishing it from the traditional single-image datasets. While our innovative method, VPG-C, along with OpenFlamingo and MiniGPT-4, inherently accommodates interleaved image-text sequences, other models like BLIP-2, InstructBlip, LLaVA, mPLUG-Owl, Otter, and LLaMA-Adapter V2 do not. For these, employed strategy where we concatenate the embeddings of all images. This approach can be amalogized to treating images as frames within a video. To maintain the positional context of each image an interleaved image-text instruction, we explicitly indicate the location of each image within the context.
E DETAILED FERO-SHOT PERFORMANCE ON MME BENCHMARK
In this section, we report the detailed performance on the 14 subtasks of MME benchmark in Table
Table 7: Detailed shot performance MME benchmark.
BLIP-2
InstructBLIP
LA-V2
LLaVA
MiniGPT4
mPLUG-Owl
Otter
VPG-C
Existence
160.00
185.00
120.00
50.00
115.00
120.00
195.00
180.00
Count
135.00
143.33
50.00
50.00
123.33
88.33
50.00
96.67
Position
73.33
66.67
48.33
50.00
81.67
50.00
86.67
80.00
Color
148.33
153.33
75.00
55.00
110.00
55.00
113.33
116.67
Poster
141.84
123.81
99.66
50.00
55.78
136.05
138.78
147.28
Celebrity
101.18
48.82
65.29
172.65
164.12
Scene
145.25
153.00
148.50
50.00
95.75
135.50
158.75
156.00
Landmark
138.00
79.75
150.25
50.00
69.00
159.25
137.25
145.00
Artwork
136.50
134.25
69.75
49.00
55.75
96.25
129.00
113.50
OCR
110.00
72.50
125.00
50.00
95.00
65,00
72.50
100,00
Perception
1293.84
1212.82
972.67
502.82
866.57
967.34
1292.26
1299.24
Commonsense
110.00
129.29
81.43
$7.14
72.14
78.57
106.43
98.57
Numerical
40.00
40.00
62.50
50.00
55.00
60.00
72.50
77.50
Text Translation
65.00
65.00
50.00
$7.50
55.00
80.00
57.50
57,50
Code Reasoning
75.00
57.50
55.00
50.00
110.00
57.50
70.00
87.50
Cognition
290.00
291.79
248.93
214.64
292.14
276.07
306.43
321,07
F DETAILED ZERO-SHOT PERFORMANCE ON DEMON BENCHMARK
Table Zero-shot evaluation on multimodal dialogue.
Conversational Embodied Dialogue
Multimodal Dialogue
BLIP-2
16.75
35.49
InstructBLIP
18.07
49.09
LLaMA-Adapte V2
19.04
9.40
LLaVA
10.19
5.39
MiniGPT-4
16.82
10.57
mPLUG-OWl
11.07
14.27
OpenFlamingo
24.27
9.49
Otter
16.06
14.68
VPG-C-LLaMA2-78
48.31
37.04
VPG-C-Vicuna-7B
41.02
33.99
VPG-C-Vicuna-133
42.25
34.02
19

---

Published as a conference paper at ICLR 2024
Table 9: Zero-shot evaluation on visual storytelling.
Animated Story Completion-AESO
Animated Siory Completion ParoroSV
Animated Sioty Completion PlinktoneeSS
Sequenial Photo Storytelling VIST
Sequenial Photo
Storytelling DiDeMeSV
InstruciBLIP
18.80
28.20
33.32
16.92
24.80
LLaMA-Adapeer V2
18.01
20.15
21.22
10.89
|4.57
LLaVA
13.56
11.44
12.77
8.00
7.71
MiniGPT-4
mPLUG-Oww
18.28
20.19
32.12
10.82
11.94
OpenPlamingo
23.32
32.35
37.79
15.14
12.50
Other
13.94
17.52
22.21
9.96
14.23
VDG 1B
VeG--3-vi cuna-Ta
19.93
28. 36
39.19
17.34
21.27
VP cuna-1 38
20.53
29.81
41.32
19.04
22.26
Table 10: Zero-shot evaluation on visual relation inference.
Visual Change Captoning
Visual Change Captioning
Visual Relationship Expressing
Subtle Ditterence Expressing
SSpot-th-DDiff
-CLEVR-Change
BLIP-
17.48
3.21
12.37
InstructBLIP
19.71
4.61
10.70
10,92
LLaMA-Adipter V2
16.72
15.52
7.88
13.92
LLaVA
8.50
8.76
6.72
9.11
MiniGPT-4
7.50
7.49
7.84
8.97
mPLUG-Owl
6.06
1.46
6.22
7.86
OpenFlamingo
13.01
11.90
12.57
17.90
12.69
VPG-C- UUG-C-LLaMA2-78
21.02
42.05
14.10
24.81
VPG-C-Vicuna-
20.01
41.60
16.35
25.64
VPG-C-Vicuna- -33
21.56
40.67
20.27
26.08
Table 11: Zero-shot evaluation on multimodal cloze.
Comic Dialogue Identification
Comic Panel Identification
Recipe Completion
Step
BLIP-2
39,70
0.00
30.46
1.60
InstructBLIP
40.60
0.00
27.40
16,80
Adapter
38.20
LLaVA
30.60
0.00
32.80
0.00
MiniGIT-4
33,00
31,60
0.80
APLUG-OW
36.60
0.00
27.60
0.80
OpenFlamingo
38.40
1.20
29.00
18.00
V-G-C- LLaMA2-73
36.80
1,80
51.80
1.40
V2G C Vicuna 73
39, 20
3.60
30.40
15.40
12G-C-Vicuna-
42.20
8.20
39.80
18.40
For tasks with images as options, only responses that hegin with the correct answer will be evaluated as correct.
Table 12: Zero-shot evaluation knowledge grounded QA.
Webpage QA
Textbook QA
Complex Multimodal QA
BLIP-2
47.60
29.73
40.36
InstructBLIP
45.20
30.20
66.80
LLaMA-Adapter V2
44.60
46.00
43.80
LLaVA
39.40
39.60
29.60
MiniGPT-4
27.40
28.60
34.80
mPLUG-Owl
34.20
30.00
35,60
OpenFlamingo
37.80
32.40
25,80
Otter
45.00
39.00
41.00
VPG C- LLLLMMM-277
49.40
42.40
61.20
VPG-C-Vicuna-78
50.00
33.40
62.40
VPG-C-Vicuna-131
50.60
43.40
64.80
20

---

Published as a conference paper at ICLR 2024
Table 13: Zero-shot evaluation on text-rich images QA.
Slide QA
OCR QA
Document QA
BLIP.2
43.80
10,40
46.40
InstructBLIP
42.00
44.20
47.00
LLaMA-Adapter V2
43.00
3.40
49.60
LLaVA
38.80
2.60
43.60
MiniGPT-4
35.20
7.20
36.80
mPLUG-OWl
35.60
22.60
39.20
OpenFlamingo
35.60
3.80
$2.40
Otter
38.40
2.20
42.60
VPG-C-LLaMA2-2
45.80
39.60
49.40
VPG-C-Vicuna-78
46.80
39.40
48.60
VPG-C-Viccun-133p
48.80
46.60
52.60
Table 14: Zero-shot evaluation on multi- iulti-image reasoning.
Image-Set
Property Colecrnnce
Transtormation
Step
Image Visual Entailment
Ambeguity
QA
Inspoction
QA
Coherence
Matching
Analysis
42,80
43.20
59.00)
20)
0.20
45,80
intricthipphh
LLaMA-Adapier V2
41.60
55.00
45.66
48.80
63.00
0.00
$4.80
43-40
LLaVA
29.60
53.00
45.20
50.40
59.20
0.80
50.80
43.20
MinGGPT4
mPLUG-Ow]
29.20
$4.20
45.80
50,00
60.50
0.00
$5.00
45.20
OpenFlamingo
25.80
$2.20
44 44.20
59.60
$1,40
2,20
$3.60
44.00
VVG-C--LLaEE -B
62.60
61,40
36,50
37.80
$3.50
31.20
VDG-C-V1 cuna-78
67.20
48.90
50.00
60.80
60.00
0.20
$7.80
$7.40
FPG-C-Vicuree
images options, only responses begin
correct.
G QUALITATIVE COMPARISON
In this section, we compare our model with existing MLLMs on some complicated demonstrative instructions.
21

---

Published as a conference paper at ICLR 2024
explain interesting point piotures.
expaan interestin three pictures.
The image
perfectly
associated dime oulderr
first indicates that
fael amasing piztonged suniight.
The is shining the person's Saet.
the suriight resulting
the remaina
under holes reaches the skin
tanned dirdes.
VPG-C
InstructBLIP
Please expizin the Interesting point based on the three pictures.
Please explain the interesting point based on the three protures.
The pictura the blue sky Thehe picture sky sky
picture sky and the
and clouds. dlouds. picture
Please explain the interesting g point basoc on the Ihree pictures
dlouds.
sky and douds.
pictura blue siyy
The picture
sky
sky
OpenFlamingo
MiniGPT-4
Figure 10: Qualitative comparison between our VPG-C C and existing MLLMs.
22

---

Published as a conference paper at ICLR 2024
What do thise four pictures want convey us?
theis four pictures want convey
images is creative advertiaement adoption, encourages people
Dan't eot dog meat
cage behind people cages.
adopted
VPG-C
InstructBLIP
pictures convey
convey
What What
What these pictures? the message? Whate the these picturee?? What
The poster ia of dog that has been attacked predator la lying on the ground with its eyes clesed and its ears bent down. The poster is placed on a brick wall with a message saying i"please take one home.
messcage? What the these pictures?
What the messugge
OpenFlamingo
MiniGPT-4
Figure 11: Qualitative comparison between our VPG- C and existing MLLMs.
changes piclures
changes
chargges??
changeo??
This pictures mainly shows the
only beginning
very clean. picture, appeat
by
image
anifa. fol showing that will
after time.
VPG-C
InstructBLIP
What changes have ocoumed this wall the piclures over time, and what does the set pictumes hope to showo us through these changeos?
What changes have med to this wall in the
X
pictures over time and what does this set pictume hoga to show us through these changeo??
one carves, artst
one
<dmg>
The the warl The wasl the the srree artiet the
OpenFlamingo
MiniGPT-4
Figure 12: Qualitative comparison between our VPG-C and existing MILMs.
23

---

Published as a conference paper at ICLR 2024
Google
Google
What between
Whst
gray Muggle
Google'a logo that
connection between these two
google
their tont scheme
What makeeo this set of pictures
What makes this ser of pictures look Interwetinng?
lodi internelin??
Tiecuuse the word "margger uses the sarn form and color scheme as Google logo, and thair aronuncaaion is very similar: its humor les using hamopphnnn making the person wesrtigg look meggle
This can be seer ss clever play on wards or suetie referenioe the peju Harry Potter franchisc, where the torn "MMugglle" is used describe non magical people strarding wods.
VPG-C
InstructBLIP
Google
Google
What is the connection between theoe fom pict.rea?
What belaeeen these picturea?
Google Muggle, Hany Potler J.K. Rowling. Wizarding World Wizard, Whthh Wicarding Wezaringg World of Harry Potter. Universal Ortando Resert, Universal Stucks Florica, Unieersal Criando,
writien ecommple of how oolor schwmm be changeed in difiement
The blue,
+
What interesting?
piclures
Whal makes es this sei piclures look mereesingg
mage
mage
Googlets
</mggn-xa>>>
similar.
OpenFlamingo
MiniGPT-4
Figure 13: Qualitative comparison between our VPG C and existing MLLMs.
24

---

Published as a conference paper at ICLR 2024
NeaSSh
31ral 20hn
What is this in picture?
What is this picture?
image aawvery
"The
lynng
3
12.20 inches
Is sill the fish you thought it Vose noww
Is still the fish you thoughtit now?
con't are many
pres packerd pencil
proture.
pers
Yes, it stil the fish.
black
What can this pencil case holting pens?
can penci case holsiig pens?
froid slickers, other stationery, bullita Fish-sl aypearance
Hare are acamee possible Loves Iriet Deek scoeeesary The par cil cass can selve ai cecorative dest accessory, adding a touch of sommay and semonnaily to your worsspace.
deceptive!
VPG-C
InstructBLIP
NaaSS
31.0/12.00
What in this picture?
3er
What is this pirtum??
an image Iile lying
cm 2 13 am-4. 5. 13 13
fisped up open, image
13 am om 8. 13 cm 10. cm
Is the Fish you thoughtil now?
Is stll the fish you thought was now?
This imaga
lying is
not fish. a large, caftish been placed top pilo colored pens
sico visitle. The fish
5.5 inches and 20 inches wide
cann this puncil cass holding
things. sharpanee
The feeh is black and sibwery with smrlll mouth and styos. it lying on side, with its sail and fins visible. The Eah is about. 0.57 nchhee long and 12.20 inches wice.
pancil unseer, pancil sharpener, pencil sharpener.
OpenFlamingo
MiniGPT-4
Figure 14: Qualitative comparison between our VPG-C C and existing MLLMs.
25

---

Published as a conference paper at ICLR 2024
What Is and how
C
What in how to use it
Use 7
lans designed type of typically combinat on imagas designed form
image
Camara
fllm
captures imago.
you camera image
you this image camera lens?
I'm sorry. The image shows a large silver with handle, featuring a lent-ilke design. The cup made stanness steel and there also spoon resting inedd the cup.
image
Can this lons take photos?
Can this lens lake photos?
No, It's just designed to look lik a camera lens, but It's not real camera lans.
VPG-C
Yes, camara can photos.
InstructBLIP
What
What and
The EOS 10000 as 10.1 megapteel entry level DSUR camera announced by Canon 25 June 2005 it the aucoeesor the EOS 450D and the peedecsesor in the FOS 1100. The E0S 1000D is the frrst DSL R camera to saw the new EF-S 18s 56mm 05 5-5 8 18 lana.
Tris is an image a canerre lero. It is lypa camera lans that is des ignod to caplure images and viceos. is typically ly used in combination with a ca camora body to form a camera system. The lans designed to locus light onto a film or digital aeriwcr. wi captares the image.
Are you sure this an Image of camera lens?
Are you sure this image camera lens?
looks more coffee mus. I'm sure you trying say. not what you trying say. not sure what you are trying to say.
sorry. The image shows large cup with featuring lens-like design. cup the cup.
spoon reating
this lems phrtlee??
Can take photos?
Yes, can.
designed capture videos.
OpenFlamingo
1
No. It's just designed look
It's
MiniGPT-4
Figure 15: Qualitative comparison between our VPG-C and existing MLLMs.
26


Reviews:
[{"review_id": "ysMNhimnqn", "reviewer": "Reviewer_HGkc", "summary": "This paper aims to improve reasoning capabilities of Multi-modal Large Language Models (MLLMs) for demonstrative instructions. The authors highlight that most of MLLMs have been over-optimized on the image-captioning objective which has led to the use of visual features that could only describe the captions while neglecting its focus on minor yet discriminative features important for fine-grained reasoning.  \n\nFirstly, this work proposes VPG-C which is a lightweight adaption on top of VPG which aims to reuse the information from the intermediate LLM layer and is used to modulate the VPG features through guidance. The modified VPG features are integrated into LLM intermediate layer which effectively improves the fine-grained reasoning performance.\n\nSecondly, to train the VPG-module, this work proposes a automatic way to generate synthetic data used to improve fine-grained discriminative performance of MLLMs. \n\nLastly, DEMON benchmark is proposed to evaluate the demonstrative instruction understanding of the proposed technique and other MLLMs.\n\nThe proposed approach is fairly motivated with analysis and ablation studies.", "questions": "Please refer to weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n\n1) This paper identifies and aims to address a crucial limitation of lack of reasoning capabilities of Multi-modal Large Language Models (MLLMs) for demonstrative instructions. Improving MLLMs for demonstrative instructions will pave more rapid growth for building human-friendly AI assistants.\n\n2) The proposed VPG-C design is fairly motivated and it is compute friendly.\n\n3) The idea of generating synthetic data with automatic pipeline to improve fine-grained discriminative capabilities of MLLMs is encouraging.\n\n4) The authors have proposed a suitable benchmark demon which would enable more systematic developments in improving MLLMs.", "weaknesses": "I could not observe any significant weaknesses. However I have a concern regarding the inference compute efficiency of the proposed approach. \n\n1) As the model is reusing its intermediate features via a feedback loop system, this will significantly increase the training and testing time and might not be batch friendly during inference. How does the throughput of this technique compares against previous methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EQePdHSTEY", "reviewer": "Reviewer_cxqK", "summary": "This paper studies the instruction tuning in multimodal language models. In particular, it tries to improve the bottleneck of visual prompt generator (VPG), aka the visual feature converter which converts a generic visual embedding into LLM-interpretable inputs. It hypothesizes that the bottleneck comes from the lacking of attention to details in VPG. To address this, the paper proposed two components: (1) a VGP-C architecture which additionally generate features for intermediate LLM layers with attention to LLM intermediate features, (2) a synthetic data generation procedure to generate training data to teach VPG to attend to details. In addition to these, the paper also introduced a new evaluation benchmark.", "questions": "NA.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed two methods for multimodal instruction following, the VPGC architecture and the data synthesis technique. Both of them are novel and inspiring.\n2. Plenty of ablation studies are provided to support the effectiveness of the method. Comprehensive experiments also demonstrate the superiority of the method compared to existing models.\n3. The paper also introduced a benchmark for future research.", "weaknesses": "1. The method contains several steps and is thus quite complicated. It may be hard to reproduce the whole framework in different settings and code bases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rQnmwWT5XG", "reviewer": "Reviewer_gX7w", "summary": "This paper presents a new training method for Q-Former/Resampler, aiming to provide richer visual representations for lora-based Multimodal Language Models (MLLMs). In addition, the paper proposes a synthetic training dataset for training VPG-C. After training, VPG-C achieves surprising results on the benchmark proposed in this paper and other open-source benchmarks.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors are the first (at least to my knowledge) to propose using the latent features of the intermediate layers of the llm as guidance for the q-former, providing directed detail supplementation for the LLM.\n\n2. The provided dataset/training method may inspire future research.\n\n3. The provided benchmark can better diagnose the capabilities of MLLMs.", "weaknesses": "1. The paper does not mention the setting for instruction tuning. My understanding is that after using the synthetic discriminative training strategy, the model automatically acquires the ability to follow instructions without needing an instruction tuning phase.\n\n2. The ablation in the paper validates the effectiveness of several proposed modules. But can VPG-C be applied to a finetuning setting, such as llava/minigpt4?\n\n3. In multimodal dialogues, does the model need to update guidance multiple times when providing multiple answers? In other words, for each new additional question, does the model need to run the whole model to obtaion all hidden states and can't use the existing qkv cache?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GR9OjOEYbR", "reviewer": "Reviewer_kFPK", "summary": "This paper aims to address the issue that the image-captioning based training objective often leads the visual prompt generators (VPGs) to neglect visual details. It proposes a VPG-C(omplete) module to complete the missing details and a synthetic discriminative training strategy to train VPG-C without the need for supervised instructions. The experiments are conducted on the proposed DEMON benchmark, the MME, and OwlEval benchmarks.", "questions": "The primary concerns are related to experimental evaluation, and the rating would improve if the experimental evaluation were comprehensive.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The idea of completing the missing details for visual content is reasonable. The method of synthetic discriminative training is interesting and straightforward for training the VPG-C without the need for supervised instructions.\n- The proposed DEMON benchmark encompasses a wide range of tasks spanning multiple categories, offering the potential for the evaluation of future research efforts.", "weaknesses": "- The effectiveness of the proposed VPG-C has not been fully validated. On the proposed DEMON benchmark, the improvement of VPG-C compared to InstructBLIP is quite limited, which does not align with the expectation that completed details would significantly enhance VPG. The improvement on the MME benchmark is also not significant.\n- The experimental evaluations on many common benchmarks are missing, such as the evaluation protocols of InstructBLIP (”divide the 26 datasets into 13 held-in datasets and 13 held-out datasets” including NoCaps, Flickr30K, GQA, VSR, IconQA, TextVQA, Visdial, HM, VizWiz, SciQA Image, MSVD QA, MSRVT QA, iVQA), the Mini-GPT4 dataset, the LLaVA-Instruct-150K benchmark, and the MIMIC-IT dataset. Evaluating on the same benchmarks as other MLLMs is important for a comprehensive and fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)