Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning
Authors: Mustafa Shukor
Alexandre Rame
Corentin Dancette
Matthieu Cord
Venue: ICLR 2024 poster (2024)
Keywords: large multimodal models
in-context-learning
evaluation
alignment to human preferences
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: LMMs suffer from several limitations. ICL can reduce some of these limitaions, and amplifies others. More complex ICL approaches are promising to address this problem.

Abstract: Following the success of Large Language Models (LLMs), Large Multimodal Models (LMMs), such as the Flamingo model and its subsequent competitors, have started to emerge as natural steps towards generalist agents. However, interacting with recent LMMs reveals major limitations that are hardly captured by the current evaluation benchmarks. Indeed, task performances (e.g., VQA accuracy) alone do not provide enough clues to understand their real capabilities, limitations, and to which extent such models are aligned to human expectations. To refine our understanding of those flaws, we deviate from the current evaluation paradigm, and (1) evaluate 10 recent open-source LMMs from 3B up to 80B parameter scale,  on 5 different axes; hallucinations, abstention, compositionality, explainability and instruction following. Our evaluation on these axes reveals major flaws in LMMs. While the current go-to solution to align these models is based on training, such as instruction tuning or RLHF, we rather (2) explore the training-free in-context learning (ICL) as a solution, and study how it affects these limitations. Based on our ICL study, (3) we push ICL further and propose new multimodal ICL variants such as; Multitask-ICL, Chain-of-Hindsight-ICL, and Self-Correcting-ICL. Our findings are as follows; (1) Despite their success, LMMs have flaws that remain unsolved with scaling alone. (2) The effect of ICL on LMMs flaws is nuanced; despite its effectiveness for improved explainability, answer abstention, ICL only slightly improves instruction following, does not improve compositional abilities, and actually even amplifies hallucinations. (3) The proposed ICL variants are promising as post-hoc approaches to efficiently tackle some of those flaws. The code is available here: https://github.com/mshukor/EvALign-ICL.

Published as a conference paper at ICLR 2024
BEYOND TASK PERFORMANCE: EVALUATING AND REDUCING THE FLAWS OF LARGE MULTIMODAL MOD- ELS WITH IN-CONTEXT LEARNING
Mustafa Shukor
Alexandre Rame
Corentin Dancette'
Matthieu Cord 1.2
Sorbonne University
2 Valeo.ai
ABSTRACT
Following the success of Large Language Models (LLMs), Large Multimodal Models (LMMs), such as the Flamingo model and subsequent competitors, have started to emerge natural steps towards generalist agents. However, interacting with recent LMMs reveals major limitations that are hardly captured by the current evaluation benchmarks. Indeed, task performances (e-g., VQA accuracy) alone do not provide enough clues to understand their real capabilities, limitations, and which extent such models are aligned human expectations. To refine our understanding of those flaws, we deviate from the current evaluation paradigm, and (1) evaluate 10 recent open-source LMMs from 3B up to 80B parameter scale, on 5 different axes; hallucinations, abstention, compositionality, explainability and instruction following. Our evaluation on these axes reveals major flaws LMMs. While the current go-to solution to align these models is based on training, such as instruction tuning or RLHF, we rather (2) explore the training- frre in-context learning (ICL) as a solution, and study how it affects these limitations. Based on our ICL study, (3) we push ICL further and propose new multimodal ICL. variants such as; Multitask-ICL, Chain-of-Hindsigtt-CCCC Hindsight-ICL, and Self-Correcting Correcting-ICL. Our findings are as follows; (1) Despite their success, LMMs have flaws that remain unsolved with scaling alone. (2) The effect of ICL LMMs flaws is nuanced; despite its effectiveness for improved explainability, answer abstention, ICL only slightly improves instruction following, does not improve compositional abilities, and actually even amplifies hallucinations. (3) The proposed ICL variants are promising as post-hoc approaches to efficiently tackle some of those flaws. The code is available here:
INTRODUCTION
The quest for building generalist assistants has garnered significant attention and effort (OpenAI, 2023; Gao et al., 2023). The recent breakthroughs in Large Language Models (LLMs) (Brown et al., 2020; Chowdhery al., 2022; Touvron al., 2023b) represent promising initial step towards this goal, achieving near-human performance across numerous NLP tasks. However, their confinement to the single textual modality remains significant limitation in developing universal models. Consequently, the focus has shifted to building multimodal models that transcend generation and understanding across text and images (Huang al., 2023; Yu et al., 2022; Wang al., 2022a). The prevailing approach to develop Large Multimodal Models (LMMs),, is to build on top of LLMs, bridging the gap between language and the other modalities. Those "augmented language models" (Alayrac et al., 2022; Mialon et al., 2023; Shukor et al., 2023a) beat previous models (Chen et al., 2020; Li et al., 2021; Dou et al., 2021: Shukor et al., 2022) on almost all benchmarks.
Although LMMs have achieved remarkable scores, measuring the task performance alone, such as their prediction accuracy on general benchmarks (e.g.. VQA accuracy or CIDEr for captioning). is insufficient to assess their genuine capabilities. For example, performances on those tasks may artificially increase simply by exploiting dataset biases and shortcuts, without truly understanding and
"Contact: (firstname.t Jasttame .n

---

Published as a conference paper at ICLR 2024
Hallucinations
Compostionalty
Explainability
following
Onsor th ings
What oolrr are Teltue broe??
Dow the betwing seftenoo
In Flos pizza
Kior can Ieferne shout
Answer keaile with
ourge
Arawar
Arower: yoa becasse ter
Arawer: The giallii hustla
Zaro-shot
Zaro-shot
Zaro
Zero-shel
Zero-shol ICL
ICE X4CL
ICL X-CL (Duns)
ICL X-ICL
Performance
Low
Moderate
Satisfaclory
LMM
25:
X (Kinage>, .T)
ICL:
X (cinage?;, T, R:) (cimage>; TT
X-CL
X (<image>;, fTT,, R.) (<ixage>
Figure 1: Evaluation framework. We study LMMs following strategics, different axes; hallucinations, abstention, compositionality. explainability and instruction following. In addition to an image <image> and a question T used in zero-shot (ZS), in-context learning (ICL) considers N demonstrations of images-questions- answers (<image>,, T,, R.) as input X, augmented function f in X-ICL.
generalization (Geirhos et al., 2020; Dancette et al., 2021; Du et al., 2022). While evaluating LLMs (Chang al., 2023; Li et al., 2023d) and small multimodal models (Ma et al., 2023; Dai et al., 2023b) has received attention, the evaluation of recent LMMs has been comparatively overlooked. This is becoming increasingly important as recent works (Alayrae al., 2022; Shukor et al., 2023b;a), in preliminary investigations, have highlighted qualitatively several major flaws (e.g., hallucinations), showing that LMMs are still not aligned with the needs for deployment in real-world applications.
As argued in Askell et al. (2021), LLMs should be helpful, honest, and harmless to align with human preferences. Similarly, we argue that this should also be the case for LMMs, which becomes an urgent requirement with the exponential performance improvements. Thus, LMMs must be helpful (e.g.. provide explanations. follow user instructions), honest (e.g., abstention the ability to say I don't know, no hallucinationss, truthful and harmless (e.g., no hallucinations, especially in critical applications), generalize well and understand semantics (e.g., compositionaliiyy. Thus, we start by asking the following question: to which extent LMMs are aligned with human expectations?
To provide an answer, we propose a different set of experiments, evaluating LMMs on axes (illustrated in Figure 1). (1) Object hallucinations (OH) (honest, harmless), where LMMs generate text predictions referring to objects not present the input image (Rohrbach et al., 2018; Dai et al., 2023b). (2) Abstention (honest), the ability to abstain from amswering, avoid incorrect responses when the input image cannot provide the required information (Whitchead et al., 2022). (3) Compositionality (helpful, generalization) wherein the meaning of the sentence depends only on its constituents (Werning et al., 2012; Lake et al., 2017) allowing to generalize to an infinite number of compositions. Users might ask the model (4) explain (helpful) its answers understand the underlying rationale. In addition, true assistant should engage in conversations with users and (5) precisely follow their complex instructions (helpful) (Liu al., 2023b). The conclusion of our study is that current LMMs lack proficiency in these aspects, revealing that scaling alone is not enough. Specifically. LMMs generate plausible and coherent answers instead of faithful and truthful ones (Section 2.1), provide answers when they do not know (Section 2.2), lack compositionality (Section 2.3), struggle to provide good explanations (Section 2.4) or precisely follow user instructions (Section 2.5).
means to
We then investigate how to tackle these limitations. The current go-to solution to align these models is with training (e.g. instruction tuning, RLHF). Here, we rather focus efficient approaches. For LLMs, a cheap, and effective alternative to finetuning is In-Context Learning (ICL), which used to adapt the model to particular task, recently have been used align LLMs (Lin et al., 2023). While ICL has been extensively investigated for LLMs (Lu et al., 2022; Liu et al., 2022; Wei et al., 2022), its application to LMMs has received less attention and mainly focuses adaptation to new
2

---

Published as a conference paper at ICLR 2024
image-text tasks (Tsimpoukelli et al., 2021; Alayrac et al., 2022). In this work, we explore which extent we can efficiently tackle LMMs flaws using different variants of multimodal ICL. Our main contributions are:
We evaluate 10 recent LMMs (from 3B to 80B) and show important flaws on 5 axes; object hallucinations, answer abstention, compositionality, explainability and instruction following.
We explore Multimodal ICL remedy, and study its effect on these abilities. We show that while ICL. can help on some aspects (explainability, abstention). it has marginal effect (instruction following), no effect (compositionality) or even worsen hallucinations.
Based on our ICL study, we propose simple and novel ICL variants such as; Multitask- In-Context-Mullee MultlMMsltttttttttee Leaaninn (MT-ICL), Chain-of- Hi--iiiiight-ICLL (CoH-ICL), and Self- Correcting-ICa ICL (SC-ICL). We show the effectiveness of these variants on several abilities.
Table 1: Evaluated LMMs. We evaluate 10 models that differ size, training data, and LLM initialization. Tr: training/trainabee. (I): instruction. P/D: image-text pairs/web documents. use additional ChatGPT data.
Model
# Tr. params.
# Th: samples (P/D)
Language model
Vision Model
( Tuning
OBv2-3B
1.05D
60M/1 20M
MPT-ID (Team. 2023)
CLIP WITL/14
OPv2-38 (l)
1,05H
60M//20M
MPT-18 (Instruct) (Team. 2023)
CLIP VIT-1/14
OF5y2.4B
60M/1
RedPajama. 3B (logetherai,
CLIP VIT-J14
Olv2-448 (1)
1,098
60M" /120
RedPajoma-38 (Instract) (logether.ak 2023)
CLIP VIT-1/14
OF42-9B
1,38B
60M'
MPT-7B (Team, 2023)
CLIP VIT-L/14
OFv1-98
1,318
SM/I0M
LlaMAv1-78 (Touwron et al., 2023a)
CLIP VIT-L114
IDEFICS-9B
2B
141M+/182B
LlaMA+1-7B (Touszon et al., 2023a)
OpenCLIP VIT-H/144
IDEFICS-9B
141M+/1
LlaMAv1-78 (Touvron
OpenCLIP VIT-H/14
IDEFICS-80B
15B
141M+/1882
LlaMAv
(Touvron al., 2023a)
OpenCLIP WT-H1/14
IDERICS-S0B (1)
80B
141Ms/1.82B
LlaMA116650 (Touvton al., 2023a)
OpenCLIP WT-H/14
2 LMMS EVALUATION AND MULTIMODAL ICL
Background on LMMs and ICL. We refer by LMMs (Chen et al., 2022b; Alayrac et al., 2022; Huang et al., 2023; Li et al., 2023c) to multimodal models (beyond one modality) that train large number of parameters (beyond IB) on large datasets (hundreds of millions of examples). The typical development of such models builds on top of pretrained LLMs and vision encoders, with additional trainable adaptation modules. This strategy was used in the Flamingo (Alayrac et al., 2022) model, showing impressive performance on a myriad of rision-language tasks. This has driven significant efforts in the community to build similar open-source models such as Open Flamingo (OF) (Awadalla et al., 2023) and IDEFICS (Laurençon et al., 2023). The architecture of those models consists of a frozen decoder-only LLM (e.g., LLaMA, MPT), frozen vision encoder (e.g., CLIP-ViT) followed by perceiver resampler, and gated cross-attention injected between LLM blocks. An interesting aspect of those LMMs is the ICL ability (Brown et al., 2020; Dong et al., 2022), allowing adaptation to new tasks with only a few demonstrations in context. Despite being heavily investigated for LLMs, as a way to solve new tasks or enhance reasoning (Wei et al., 2022: Zhang et al., 2022; Chen et al., 2022a), little work (Tsimpoukelli et al., 2021; Alayrac et al.. 2022: Huang et al., 2023) addressed ICL for LMMs, which usually focus on solving general benchmarks like VQA, captioning, or classification. For multimodal ICL. (M-ICI.) LMMs take an input 7 (e.g., an image < magee gage and a question/instruction T), preceded by a Context C (e.g.. N task demonstrations of images and text with responses R) and generate output o. M-ICL can be written as follows:
C = [<<image>, TT, < endo fchunk {>) N. I = (<i (<image>T), = LMM((C, I])-
(1)
Implementation details. We consider 10 different models from OpenFlamingo (OF) (Awadalla et al.. 2023) and IDEFICS (up to 80B parameters) (Laurençon et al., 2023) as described in Table 1. The models mainly change in size, initialization (LLMs), and training data. For ICL, we follow the standard way and randomly select the demonstration examples (without an explicit task instruction, results with task instructions are Appendix I). We repeat each experiment 3 times and report the averaged results. For zero-shot, we follow other approaches and use 2 examples without images as context (ă la Flamingo). We provide more details in Appendix C. The details for each benchmark are provided in Appendix E.
2.1 HALLUCINATION
Hallucinations in text is the tendency of LLMs to generate coherent plausible responses, over factual ones. By analogy, when considering multiple modalities, (Rohrbach et al., 2018) define as object
3

---

Published as a conference paper at ICLR 2024
Object hallucinations COCO
Abstention
IDEFICS-808
IDEFICS-BI0
IDEFICS-R08
4-shot
4 shot
IDEFICS-800 808
DEFICS-90
B-shoi
8-shot
IDEFICS-98 01
70
EDEFICS-91
10-shot
IDEFICS-98
12
32-shot
OFV1-9B
DPV2-98
OFv2-98
OFV2-48 4B
OFv2-4B
10
OFV2-48
DFV2-38
CHAIRS
OFv2:38 ())
OFv2-38
SoTA
70
80
90
100
110
120
130
140
35
40
45
50
55
60
CIDEr
Accuracy
(a) Object hallucination. CIDEr (f) for captioning and CHAIR, (4) for hallucination on COCO dataset.
(b) Abstention. Overall VQA accuracy (1) and ab- stention Fi-score (f) on TDIUC dataset.
Figure 2: Evaluation of LMMs OH (left) and abstention (right). A refers zero-shot and the size refers to the number shots in ICL.
hallucinations (OH) the textual description by multimodal models of objects not present in the input image. Addressing OH is critical avoid any harm, especially in critical applications (e.g. autonomous driving or medical imaging).
Benchmark. We evaluate the LMMs on COCO captioning dataset. The performance is measured with CIDEr. In addition, to capture OH, we report the CHAIR, metric (Rohrbach et al., 2018)
comparing the objects referred in the generated captioning those actually the image.
LMMs suffer from object hallucinations. Figure 2a compares the various LMMs. In zero-shot setup, all LMMs suffer from OH, as seen the high CHAIR, scores, and in comparison the much smaller SoTA captioning models (OFA ( (Wang et al., 2022b) from Shukor et al. (2023b)). This reveals that simply scaling LMMs is not enough reduce hallucinations. For IDEFICS models, we noticed high hallucinations with zero-shot. More details and comparisons can be found in Appendix F.l.
ICL does not reduce hallucination, but instead amplifies it. We investigate if ICL can reduce hallucinations. We can notice (Figure 2a) that adapting models to the captioning task COCO with 4-shots reduces OH. Yet, more than 4 shots actually amplify ballucinations, as the CHAIR, metric then increases with the number of shots. This reveals that while the overall metric (CIDEr) is improved with ICL, the generated captions contain more hallucinations. This is less the case for the largest models (IDEFICS-80B) which suffer less from such amplification.
What reduces hallucinations? First, pretraining on more multimodal data seems to reduce hallucina- tions, all OFv2 models better than OFv1. Second, training all model parameters (including the language model) on multimodal instruction datasets significantly reduces hallucinations (IDEFICS- 9B (I) IDEFICS-9B), Third, instruction-tur models (OFv1-3B (I) OFv1-3B and OFv1-4B (I) vs OFy]-4B) tend to hallucinate less with higher number of ICL shots.
Finding 1. LMMs suffer from severe hallucinations. small number ICL shots partially alleviate it, while increasing them exacerbates the problem, especially for small models (<9B params.). Pretraining on more high-tuality data and unfreezing the LLM weights helps to reduce hallucinations.
2.2 ABSTENTION
LMMs should know when they do not know, and abstain instead of providing incorrect answers. Here study scenario where the question not be answered from the image.
Benchmark. We evaluate on TDIUC (Kafle & Kanan, 2017), a VQA dataset containing absurd questions (~ 22% of a total number of questions), that not related to the image and thus should not
4

---

Published as a conference paper at ICLR 2024
be answered. In of abstention, the model should generate specific keyword ("doesnotapply") We report the overall accuracy in addition to the Fl-score abstention metric (absurd question or not).
LMMss tend to always give an answer. Figure 2b shows comparison between different LMMs. From the low zero-shot Fl-scores, we can notice that models are hardly able to abstain from answering to absurd questions. Adding explicit instruction for abstention help get additional improvements (as further shown in Appendix I).
ICL significantly improves abstention. Increasing the number of context examples (and thus the number of absurd examples), significantly helps abstention. However, even with the best performant model (IDEFICS-99 (I)), the Fi-score is still low.
What helps the model to abstain? First, instruction tuning while unfreezing the language model parameters seems significantly increase the abstention score (IDEFICS IDEFICS (D)) Second, increasing model size up to certain scale (9B) improves abstention (OFv2-3B vs OFv2-4B vs. OFvl- 9B). general, we notice a positive correlation between accuracy and abstention performances.
Finding 2. LMMs give more likely incorrect answers than abstaining. ICL helps them abstain. Larger models, better quality data, and unfreezing LM weights improve abstention.
2.3 COMPOSITIONALITY
Compositionality (Hupkes et al., 2020) exists when the meaning of a sentence is determined by its elements, and the rules compose them. To study this, evaluate if LMMs' understanding of caption is changed when changing its constituents.
Benchmark Appendix E. We evaluate the CREPE benchmark (Ma et al., 2023) that focuses on systematicity (Fodor & Pylyshyn, 1988) and productivity (von Humboldt et al., 1988; Chomsky, 1956).
It is an image-text retrieval dataset with hard negatives, constructed by changing the composition of the ground truth captions. Instead of retrieval, we create the task of Tenage-Text Matching (ITM) (Appendix F.2 for other choices). The model is given one caption and asked decide whether describes the image not. We use the positive and negative captions provided by the benchmark. When evaluated on systematicity, we consider 2 types of negative captions: HN-Atom (replacing atoms, such as objects, attributes, or relations with atomic foils e.g., A grill underneath the porch instead of A grill on top of the porch) and HN-Comp (composing two negative captions constructed with HN-Atom e.g., blue and pink toy instead of A pink car). We noticed similar observations with productivity. To complete our evaluation, we similarly evaluate on SugarCREPE (Hsich et al., 2023) and put more details and results in Appendix F.
Sompl
029C5-200
120
DDPFC3-9922
(Porkc HMl
OF12 40
2'12-60
OP/2 21/244111
CPv2-18
Randem
DFv2-48
00
B5
ETM MCUFTCC
60
of
(a) CREPE (HN-Atom)
(b) CREPE (HIN-Comp)
(c) SugarCREPE (Average)
Figure 3: Compositionality. Models are evaluated on the CREPE and SugarCREPE with the ITM task.
LMMs are only slightly better than random chance on compositionality. Zero-shot performances in Figure shows that LMMs are close to random on the categories, with only slightly better performance the HN-Comp. This reveals that, despite scaling the number of model parameters and of training examples, LMMs still lack compositional abilities. The baseline in Figure 3 refers to ITM without hard negative examples (Appendix F.2).
5

---

Published as a conference paper at ICLR 2024
ICL has almost effect atomic foils. Interestingly, providing more demonstrations with positive and hard negative examples does not increase accuracy on the HN-Atom split. The models seem unable to detect fine- fine-graine grained changes to the sentence, despite changing completely its meaning.
ICL seems to help on compound foils. On HN-Comp, ICL significantly increases the accuracy, especially with OFv1-9B and IDEFICS-9B (I).
models really good at compositionality? In Figure 3, we can notice that the largest models (8OB) seem perform better on the CREPE benchmark. However, is not clear this gain is coming from really improving compositionality or exploiting biases in this benchmark, where the hard negative examples are usually longer (Ma et al., 2023), not always make logical sense, and lack fluency Hsieh et al. (2023). Our study suggests that this improvement is coming rather from biases, which supported in the poor performance of all LMMs on SugarCREPE Appendix
Are
Finding 3. LMMs lack compositional ability and struggle to acquire them even with ICL.
2.4
EXPLAINABILITY
VOA-X
110
90
600
80
IDEFICS 808
1DEFICS-08
IDEFICS 98
DFV1-90
DFv2-96
50
OFv2
0FV2-38
0Fv2-36
So7A
16
shots
Figure 4: Explainability. Models are asked to generate an explanation for image, question and answer triplets from the VQA-X dataset
Despite the impressive abilities of LMMs, it is still unclear if generations are caused by some underlying complex reasoning based on the input image, rather on some memorization or bias exploitation. Instead of looking at internal activations and learned features as means of output explanation, we try another and more explicit approach; by asking the model itself for an explanation.
Benchmark. We consider VQA-X (Park et al., 2018), VQA dataset with human-annotated explanations for each mmage-qusstion-assee metric to measure the syntactic similarity between the generated explanations and the ground truths.
triplets, and CIDEr (Vedantam et al., 2015) as the
LMMs struggle to provide good quality explanations. To assess to which extent LMMs can explain their answers, we evaluate LMMs in a zero-shot manner. We give the model image, question, and the correct answer and ask it to provide a possible explanation. Figure 4 shows that LMMs can provide explanations, however, the explanation quality very limited and significantly far from existing smaller and finetuned SoTA (Sammani et al., 2022b) (filtered scores).
ICL significantly improves model explanations. We evaluate the effectiveness of ICL improve model explainability. The context consists of few demonstrations, each one containing an image, question, correct answer, and human written explanation. Figure shows that CIDEr is significantly improved by increasing the number of context demonstrations, Interestingly, while most of LMMs are still lagging. IDEFICS-SS SOB succeed to surpass SoTA.
Large scale models are better at explanations. We find clear positive correlation between model size and the quality of the generated explanation. In addition, training on more and better quality data (IDEFICS vs OF) helps improve the performance, well as instruction tuning with language model parameters unfrozen (IDEFICS-9B vs IDEFICS-P91 (I)) However, for 80B parameter models this is not the case, which might be due overfitting when training the I.I.M.

---

Published as a conference paper at ICLR 2024
Finding 4. LMMs still fail to provide good explanations, yet ICL can improve performances. Bigger models explain better.
28: Graftes
The imeg depicis a group givlles slanding in sevarine The gi ghaffes are laciig the same diection. with their necks craned upwards, the graffes are standing in the middle the mmge, while the tackground grass
descriition
IC1:
Z5: He wearing holmet
itte
nolewarty
he is wearing holmna while riing Nis broycle Helmets are an essentill piece of protective goar
precaution noteworty??
ICL:
for bicyciets as they can techue the risk heed
60
iquiiss the event of creah full. Additinnllyy
halimats can also protect the b sycliia head from the sur's hormiul UV mys
Qr Whei aeject are insorponted Arhhe incaliation inducess pidure trames potled sberts lange elements, stradive and Lraque larchcape dheigh. intent buinid Ins araato rala batao??
Z5:
The parpoes or trer betlnn the seleii indtalationn amn owele atracton and unique bestecape design
The purpose or Intent behend this artetic ixtidiation madd be create a whirricai and visually appealing landscape cesign. The uso picture trames, podted plants, and large white vasu could
0rv2-48
shots
exche
playfalesss cruatliity,
poorporation eloments as plants
Figure 5: Instruction following. Evaluation on the LlaVA benchmark on types of instructions: detailed descriptions, complex questions and conversations. Left: example with OFv2-9B. Right: average scores (over instruction types) given by GPT-4. Detailed scores for each type in Appendix
2.5 INSTRUCTION FOLLOWING
Existing multimodal models are trained to solve relatively simple tasks, such as providing shallow image descriptions or providing 1-word answers. These capabilities are not enough build general assistants that can engage in conversation with humans. Helpful assistants should help humans answer complex questions, precisely following specific instructions and engaging in conversations. Current approaches (Liu et al., 2023b; Dai et al., 2023a) to integrate such abilities are based on instruction tuning, wherein the model is fine-tuned curated instruction datasets. In this section, evaluate if LMMs lack this ability and qualitatively investigate if ICL can help. Here we focus on IDEFICS and OFv2-9B, and provide more qualitative results Appendix F to support
findings.
Benchmark. We evaluate the models on the LlaVA dataset (Liu et al., 2023b), which contains types of instructions; giving detailed image deseriptions, and answering complex questions and conversations. These instructions generated with GPT-4 (text-only). For ICL, the demonstrations are selected randomly from the dataset with the same instruction type as the query. We report both qualitative and quantitative evaluation with GPT-4. (Liu et al., 2023b), GPT-4 evaluates the response and gives a score with respect to the ground truth, given also by GPT-4.
LMMs are unable to precisely follow user instructions. For models that not instruction tuned, Figure shows that zero-shot (ZS) LMMs lack the ability to follow user instructions. For example, generated even when detailed ones explicitly asked; the simple answers do
short descriptions not fully answer complex questions; and the responses in the conversation are unhelpful. This is also reflected by the low ZS scores given these models by GPT-4.
ICL can marginally help to adapt LMMs to follow instructions. ICL adapts the model to follow user instructions. This can noticed Figure where the scores increase with the number of ICL shots. Qualitatively, the descriptions are more detailed; the answers to complex questions are richer and more elaborate: and the responses in conversation are more engaging. However, we also confirm here that ICL increases hallucinations, previously shown in Section 2.1 and further discussed in Appendix J. Interestingly, we show the scores with 2-shots but without images (shown as V), the relatively high scores raises more concerns on the effectiveness of ICL for instruction following.
Finding 5. LMMs do not precisely follow user instructions, and small number of ICL demonstrations makes them more helpful, especially for models without instruction tuning.
7

---

Published as a conference paper at ICLR 2024
3 RECTIFYING THE FLAWS OF LMMS WITH MULTIMODAL ICL (X-ICL)
In the previous section, we show that ICL is effective in improving LMMs on some axes, such as explainability and abstention. Motivated by this, here we push ICI. further and propose new improved variants to address these limitations (Appendix H for more quantitative and qualitative results).
ICL (CoH-ICL). Chain of Hindsight (CoH) (Liu et al.. 2023a) is an alternative approach for aligning LLMs to human preferences. It transforms the feedback into sentences and trains L.MS to generate this feedback. Specifically, the model is trained to generate both helpful and unhelpful responses, and during evaluation, it is prompted with the helpful prompt. Inspired by this, and to avoid costly training, we propose CoH-ICL; a training-free approach that leverages both good and bad responses as kind of in-context demonstrations. Here, we are not limited to human preferences as feedback and use positive and negative responses in general (e.g., ., from human annotation, previous model generation, random text.... ...). With T* /R* and T /R referring to positive and negative demonstrations respectively, Equation (1) for CoH-ICL can be written as:
C = [<<image>,T.T7) T, R [endofcuukk>>)) n and I (<image>TT*).
Table 2: Explainability. Overall task accuracy and CIDEr for explanations on VQA-X. ICL here refers to single-task ICL (answer or explain).
Midel
Mahed
4-chot
S-het
16-shet
32-4her
CIDBr
CIDBr
0F-2-98
Cofl ICL
IC3
69.52
(1936619,33)
61.41
23.31
78.97 (14.20
94.71
73.11
(11.8
80441
12.91
73,22
80.51
MT-ICI.
(-566)))))))
(6662(66.17)
75.79 (-3.08)
74.85 (40.37)
74.99 (488 753
77.36 (-3.3)
(+2.49)
(-1.59
74.63
75.30
83.45
76.12
$8.50
C-H-ICT MT-KC
62
(-0)-11
74.80(10.117
(10.6))
76.516122
(1000
76.75
7800 (12.0)
Explainability. We leverage CoH-ICL to improve model explainability. The context consists of; an image, question, answer, human annotation as the good response, and previous model's s generation (with ICI. 32-shot) as the bad response. Table 2 shows significant improvements over ICL (which uses only the positive human annotations as context).
Coff-Correcting ICL (SC-ICI... Recently, self-correction in LLMs has received large attention (Pan et al., 2023; Madaan et al., 2023; Raunak et al., 2023). The idea is to use the model itself to automatically correct its generated answers.
Abstention. We explore a similar approach to help LMMs abstain from answering. Specifically, we first ask the model the question using ICL. Then, for each question, we ask the model decide whether the question is answerable based on the image or not. In case the model recognizes that the question is not answerable, the previous answer is ignored and replaced with an abstention keyword. The correction is with 32-shot in this step 2 (we consider smaller number of shots in Appendix H.2). Following Equation (1), the steps 1 and 2 of SC-ICI, can be written as: where T2 is a fixed question
G =((<image>> & l 1>)) = ( LMM((C, RI), C2 =(<<iimgg>> "R"""","""""""" "R*x /ndofchhnk >) Jx, I2 (<image>T"*T")
(3)
LMM( I2),,
to ask the model if the following question T₃ is relevant to the image, and R² is yes or no. The final answer is given as function F of 0₁ and 02, i.c.. o = F(o,, 02). Table 3 shows the results with SC-ICL (32shot). We notice that SC-ICL. improves significantly over ICI. for both models.
Table 3: Abstention. Abstaining from answering unanswerable questions. We report the overall accuracy
(Acc), and abstention Fi-score (Abs F1) on the TDIUC dataset.
Meddl
4-
Sobtrr
Abse
Abot F1
Fl
Abut
KL.
40.91
35.37
4421
42.00
46.51
S1.00
66
55.45
0Pv2-96
(-7.0)
47.34 (-1.0))
(-2331
$2.85 (000
20151-2.10
(41.6)
(x2.20
XL SC SCICLLSSboo (5bllo
45.41
32.00
$1.89
47.51
51.01
60.22
61.94
67.45
IDEFICS
XIACI
40.66 (-..))
(-299)
37,22
(-3
(47.50
$4.75 64
$1.50
$3,76
(0025
(-1.1))
54.7663325)
(-120
(4.04)
39.55 (0.57)
(3.55
(1085)
(-.11
(-1.5))
Multitask ICL (MT-ICL). Multitask learning (Caruana, 1997) aims at leveraging the synergy
between tasks, usually by training one model on different related tasks. Different from this, we propose to do multitask learning in context, without changing the model's weights. Our objective is to benefit from information from other tasks to reduce LMMs flaws. With T/R/ referring to task j. the context C in Equation (1) for MT-ICI. can be written as:
8

---

notice some Published as a conference paper at ICLR 2024
C (
and I == ( (<image>T")
(4)
For explainability, ask the model to simultancously; answer the question and explain its answers preceded with the prompt "because" (we find it better to provide the answer first). With MT-ICL (Table 2) both VQA accuracy and CIDEr are better than single task (ICL). However, degradation in CIDEr with a higher number of shots. For abstention, the main task is to answer the question and the second auxiliary task is decide whether the question is relevant to the image. Table 3 shows a significant improvement compared to single task ICL (only answering the question).
RELATED WORK
Limitations of multimodal models. Efforts have been made to address object hallucinations (Rohrbach al., 2018) by designing better training objectives (Dai et al., 2023b), incorporating object labels as input (Biten et al., 2022) or costly multi-turn reasoning (Xu et al., 2023). To abstain from answering, recent work has attempted tackle this problem by training selection functions on top of a VQA model (Whitehead et al., 2022: Dancette et al., 2023). The challenge of compositionality has received significant attention, and multiple evaluation benchmarks have been proposed (Ma et al., 2023; Thrush et al., 2022; Zhao et al., 2022). Some solutions involve training on hard negative examples (Yuksekgonul et al., 2022) or employing improved architectures (Ray et al., 2023). The issue of explainability has been tackled in various ways, such as training auxiliary models to provide explanations (Kayser et al., 2021; Marasović et al., 2020; Wu & Mooney, 2019), or training models that generate both answers and explanations (Sammani et al., 2022a). Furthermore, multimodal models also struggle to follow complex user instructions, as shown in recent work (Liu et al., 2023b; Shukor al., 2023b). To address this, previous work fine-tune models instruction tuning datasets (Liu et al., 2023b; Xu et al., 2022; Dai et al., 2023a; Li et al., 2023a; Zhu et al., 2023a). However, current approaches to address these limitations are focused mostly on small specialized multimodal models, and based on expensive finetuning; our ICL solutions are easier and cheaper.
Evaluation of LMMs. To achieve more nuanced evaluation of different model abilities, concurrent works have proposed several benchmarks (Xu et al., 2023; Li et al., 2023b; Yu et al., 2023: Liu al., 2023c; Yin al., 2023). These works span evaluating multimodal models on modality comprehension (Li et al., 2023b), different capabilities (Xu et al.. 2023) fine-grained tasks (Liu et al., 2023c), complicated tasks (Yu et al., 2023) or high-level 3D tasks (Yin et al., 2023). However, these benchmarks remain focused on task performance, with novelty in creating more fine-grained tasks. Besides, we differ from these benchmarks, as we consider different LMMs with ICL ability, and focus more on limitations/allgnmmett in the context of ICL. In general, there is still notable lack of work evaluating the limitations of LMMs.
5 DISCUSSION
Reproducibility statement. Each experiment is repeated 3 times with different context demonstra- tions. We use public datasets and official open-source implementations provided by respective authors. We release the code and detailed technical instructions to reproduce the results (Appendix D).
Limitations. The work has some limitations, further discussed in Appendix and Appendix A, such as the limited range of abilities that we evaluate and the limited effectiveness of ICL as a partial solution for the studied flaws and models.
We evaluate the limitations of recent LMMs on different axes; object hallucination, answer abstention, compositionality. explainability and instruction following. Despite their scale, we find that LMMs still struggle most of these axes. Besides, we study how ICL can affect these limitations, and find that while it might help on some abilities (c.g., abstention and explainability and instruction following) can amplify the flaws of LMMs (e.g., hallucination) or has almost no effect all (e.g., compositionality). We also propose simple ICI. variants that help reducing some of the flaws. Yet, we find that the improvements coming from ICL are limited, and more complex ICL variants or other strategies, such as RLHF might be required. Finally, we hope this provides more insights about the limitations of current LMMs, and offer promising directions towards efficiently aligning foundation models (Lin et al., 2023; Li et al., 2023e) to human preferences and expectations.
Conclusion.
Acknowledgments This work was partly supported by ANR grant VISA DEEP (ANR-20- CHIA-0022), and HPC resources of IDRIS under the allocation 2022-[ADO) [AD011013415] and 2023- (AD01 1013415R1 made by GENCI. We thank Hugo Laurencon for fruitful discussions.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 35: 23716-23736, 2022.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.0088 2021.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. OpenFlamingo: An open-source framework for training large autoregressive Vision-language models. arXiv preprint 23Xi::2308.01300 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0558 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 08073, 2022b.
Ali Furkan Biten, Lluis Gomez, and Dimosthenis Karatzas. Let there be clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1381-1390, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901, 2020.
Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, al. survey on evaluation of large language models. arXiv preprint arXiv:2307 00.00999,,,,,,, 2023.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211. 12588, 2022a.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. PaLI: A jointly-scaled multilingual language-image model. arXiv preprint (tx.::22090907944 2022b.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprini arXiv:2305.188 2023.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), pp. 104- 120. Springer, 2020.
N. Chomsky. Three models for the deseription of language. IRE Transactions on Information Theory, 2(3):113-124, 1956. doi: 10.1109//T111 195110999111155111681111
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023:: 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems (NeurIPS). 30, 2017.
10

---

Published as a conference paper at ICLR 2024
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose models with instruction tuning. arXiv preprint arXiv:2305..000 06500, 2023a.
purpose vision-lane languala
Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallucination in vision- -ainunlanguage pre-training. In Proceedings of the 17th Conference the European Chapter of the Association for Computational Linguistics, pp. 2136-2148, May 2023b.
Corentin Dancette, Rémi Cadène, Damien Teney, and Matthieu Cord. Beyond question-bused biases: Assessing multimodal shortcut learning in visual question answering. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 1574-1583, October 2021.
Corentin Dancette, Spencer Whitchead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. Improving selective visual question answering by learning from your peers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 94049-24055 2023.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint wrXiv:2301.00234 2022.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang. Chenguang Zhu, Zicheng Liu, Michael Zeng, al. An empirical study of training end-to-end vision-and-languaae transformers. arXiv preprint arXiv:2 11.0387, 2021.
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding: A survey. arXiv preprint arXiv:2208. 1857, 2022.
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A crit- ical analysis. Cognition, 28(1):3-71, 1988. ISSN 0010-0277. doi: https://doi.orr/100
101660000-2277(839003155 URL http::/wwww.sceenee.... com..ccccccc
artiicle/pp/0000027889000
Wikimedia Foundation. Wikimedia downloads. htLps/t/:/ /ttpp///uppppp///////// wi wilkkieeia.... org.
Difci Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. AssistGPT: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint ar Xi1x:2336608640 2023.
Robert Geirhos, Jörn Hörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1
2020.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15180-15190, 2023.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https:///ithut /t/gttgttt//t////// http:/ttth..mmooogggeeggee www.so//wwggggggnnoooooogoo
2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sug- arCrepe: Fixing hackable benchmarks for arXiv:2306. 14610, 2023.
compositionality. arXiv preprint
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint a302v:2302. 14045, 2023.
11

---

Published as a conference paper at ICLR 2024
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Brumi. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE/CVF Interaational Conference on Computer Vision (ICCV), 2017.
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. e-vil: A dataset and benchmark for natural language explanations in vision rision-language tasks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 1244-1254, 2021.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Hugo Laurengon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al. OBELISC: An open web-scale filtered dataset of interleaved image-text documents. arXiv preprint arX1v:2306.16527, 2023.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanprect Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge. Yixiao Ge, and Ying Shan. JEED-Bench. Benchmarking multimodal iims with generative comprehension. arXiv preprint arX1v:2307.16125, 2023b.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified lasion-language understanding and generation. arXiv preprint arXiv:2201. 12000 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint 2.ti::22301122999 2023c.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian Yun Nie, and Ji-Rong Wen. HELMA: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint 175577 2023d.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision (ECCV), pp. 121-137. Springer, 2020.
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. arXiv preprint X3v:2309.00724 2023e.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. arXiv preprint arXiv:23/2.015.. 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision (ECCV), pp. 740-755. Springer, 2014.
12

---

Published as a conference paper at ICLR 2024
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02666, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.084255 2023b.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, May 2022.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? arXiv preprint ar arXiv:2377.0688/ 2023c.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8086-8098 May 2022.
Zixian Ma, Jerry Hong, Mustafa Omer Gul. Mona Gandhi, Irena Gao, and Ranjay Krishna. CREPE: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10910-10921, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. / 17651, 2023.
Ana Marasović, Chandra Bhagavatula, Jae sung Park, Ronan Le Bras, Noah A. Smith, and Yejin Choi. Natural language rationales with full-stack visual reasoning: From pixels to semantic frames to commonsense graphs. In Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 2810-2829, November 2020.
Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, and Dilek Hakkani-Tür. Using in-context learning to improve dialogue safety. arXiv preprint arXiv:2302.0087/ 2023.
Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint wrXiv:2302.078422 2023.
OpenAl. Gpt-4 technical report. arXiv,, 2023.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188. 2023.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Mareus Rohrbach. Multimodal explanations: Justifying decisions and pointing the evidence. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 8779-8788, 2018.
Vikas Raunak, Amr Sharaf, Hany Hassan Awadallah, and Arul Menezes. Leveraging GPT-4 for automatic translation post-editing. arXiv preprinl arXiv: 14878, 2023.
Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A Plummer, Ranjay Krishna, and Kate Saenko. COLA: How to adapt vision- language models to compose objects localized with attributes? arXiv preprint arXiv:2305.02 03689, 2023.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.
2018
13

---

Published as a conference paper at ICLR 2024
Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deligiannis. Nix-gpt: A model for natural language explanations in vision and vision- -anguuge tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8322 8332, 2022a.
Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deligiannis. Nix-pp: A model for natural language explanations in vision and vision-languag tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6322-8332, 2022b.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of flip-filtred 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 35:25278-222 25294,, 2022.
Mustafa Shukor, Guillaume Couairon, and Matthieu Cord. Efficient vision-language pretraining with visual concepts and hierarchical alignment. In 33rd British Machine Vision Conference (BMVC), 2022.
Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual augmentation of language models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 22066-22069, October 2023a.
Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unified model for image, video, audio and language tasks. arXiv preprint arXiv:2307. /6/84, 2023b.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15633-15650,, 2022.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable 1lms, 2023. URL www www.mosa www.iiisiii.... cml cww/ccoo/oooooooooooooo
mo
Accessed: 2023-05-05.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- asso-lingguistic composition- ality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5248, 2022.
together.ai. Releasing 3b and 7b redpajama- incite family of models including base, instruction-tuned and chat models, 2023. URL https: Sttps://wwxz /www. www..ttetteerrtttctoooowwwr.. xyz/blog/ redpaj ammdaaaaaammmmee-eee
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 200.00928, 2023b.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems (NeurIPS), 34:200-212, 2021.
Ramakrishna Vedantam, C Lawrence Zitncck and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 4566 4575, 2015.
14

---

Published as a conference paper at ICLR 2024
W.F. von Humboldt, P. Heath, and H. Aarsleff. On Language: The Diversity of Human Language- Structure and its Inftuence on the Mental Development of Mankind. Texts in German Philosophy. Cambridge University Press, 1988. ISBN 9780521315135. URL hattps://boolt google.
fr/books? SdD.SSXEAADEXEEAAAA...
Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and lan- guage. Transactions on Machine Learning Research (TMLR), 2022a. ISSN 2835-8856. URL https: wwwp/eveeeeeeeeee///// net / (errm?id=b4tMhpNOO
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequeenceto--sequenee learning framework. arXiv preprint urXiv:2202.03052, 2022b.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and laiinnla tasks. arXiv preprint arXiv:2208. 10442, 2022c.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurlPS), 35:24824 24837, 2022.
M. Werning, W. Hinzen, and (Eds.) Machery. The oxford handbook of compositionality. Oxford University Press, 2012.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In European Conference on Computer Vision (ECCV), pp. 148-166. Springer, 2022.
Jialin Wu and Raymond Mooney. Faithful multimodal explanation for visual question answering. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. pp. 103- 2019.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvim-ehub: comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXivv2306. 09265, 2023.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multinstrucc: Improving multi-modal zero-shot learning via instruction tniig. arXiv preprint arXiv:22 10773, 2022.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. LAMM: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. urXiv preprint arXiv:230k 06687, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinehao Wang, and Lijuan Wang. MM-Vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint 20000023000 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bag-of-words models, and what to do about it? arXiv preprint arXiv:2210.019333 2022.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision languae models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5579-5588, 2021.
Yiyuan Zhang, Kaixiong Gong. Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint ar0iv:2307. 10802, 2023.
15

---

Published as a conference paper at ICLR 2024
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint vrXiv:2207.002211 2022.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18-24 Jul 2021.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint MrXiv:2304.1052 2023a.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjac Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint (tt.iiv:33440693333 2023b.
16

---

Published as a conference paper at ICLR 2024
Supplementary material
This supplementary material is organized as follows:
Appendix A: discussion about the work and future directions.
Appendix B extends our related work section.
Appendix C: background on LMMs and multimodal ICL.
Appendix D: more implementation details about the evaluation setup.
Appendix E: details about the different datasets and benchmarks that use.
Appendix F: additional evaluation results.
Appendix provides additional details and results with CoH-ICL, SC-ICL and MT-ICL.
Appendix we investigate adding task instructions can help ICL.
Appendix we discuss the limitations of the work.
A DISCUSSION
Other limitations and evaluation axes. The work does not consider all existing limitations. For instance, other kinds of hallucinations, beyond objects (e.g., relations, actions, attributes). For answer abstention, consider the case when the question is not relevant to the image, but not for example when the question is relevant but unanswerable, or when it requires external knowledge that the model does not know. Other important axes include evaluating the reasoning ability of these models, especially in real situations (e.g., embodiment) and to which extent the model prediction is grounded in the real world.
ICL way to address foundation model limitations. Despite being effective in some bench- marks, ICI. is still limited in addressing some flaws. The different variants that we propose bring additional improvements. However, more effort should be put into devising more effective variants to obtain reasonable performance. In addition, results, thus more prompt engineering work can help to get additional improvement. The importance of such training-fres post-hoc approaches is, in addition to being efficient, they be complementary to other training-based ones, such RLHF (Christiano et al., 2017: ;; Bai et al., 2022a) and RLAIF (Bai et al., 2022b). Finally, more effort should be put into understanding why and when ICL works, to help develop better approaches.
noticed that the design of the prompt affects the
Other LMMss and foundation models. The work addresses one kind of LMMs that are based on the Flamingo architecture. We choose these models, they obtain the best performance several multimodal benchmarks, they are open source and exist with different scales. The work can straightforwardly be extended other multimodal models that have ICL abilities. For the broader family of multimodal models, especially the instruction-tuned ones, we believe that these models are also flawed, and is important to quantitatively assess their limitations. Besides LMMs, the proposed ICL variants might be also effective in tackling the limitations of LLMs, which have received great attention in recent years.
Beyond 9B parameters. In this work, we only consider models up to parameters. The effec- tiveness of ICL. is limited on some benchmarks probably due to the model size. In fact, the ICL performance of OF models not very stable as shown in the original paper (Awadalla et al., 2023) VQA). Thus, it will
(e.g., sometimes increasing the number of shots decreases the performance be interesting to evaluate larger and more powerful models. In addition, as ICL becomes more effective with larger models, X-ICL approaches must be also the case, especially on benchmarks where we noticed positive correlations between scaling and performance. On harder problems such compositionality, hallucinations it is uncertain if ICI. will become more effective.
17

---

Published as a conference paper at ICLR 2024
Beyond image-text modalities. While this work addresses image-text models, we argue that similar limitations also exist in models trained on other modalities. We believe the extension of this work, especially the ICL part, is straightforward to models tackling other modalities (e.g.. videos-text or audio-text) and have ICI. abilities. In fact, we argue that most of the findings on image-text models also hold on other modalities, which is supported by recent works (Shukor al., 2023a; Girdhar et al., 2023; Shukor et al., 2023b; Zhang et al., 2023) demonstrating the feasibility of extending image-text models or using almost the same image-text techniques to address other modalities.
Performance saturation after large number of ICL demonstrations. In our study, we notice that the performance start to saturate after large number of shots (16/32) on most of the benchmarks. This issue can be seen in several previous work, in particular. the original work of OpenFlamingo (Awadalla et al., 2023) and IDEFICS (Laurençon et al., 2023). For example, in (Awadalla et al., 2023); the VQA accuracy saturates or even degrades after 4/8 shots. Similarly for IDEFICS, but slightly better. There is multiple possible reasons for why multimodal ICL is not as effective as in LLMs, such as: (a) the multimodal datasets are still an order of magnitude smaller than those for LLMs. In addition, the web documents used to train such models do not contain many interleaved image-text pairs (a lot less than 32), which might hinder the ability of the model to generalize to larger number of in-context demonstrations during test. b) The trainable parameters during pretraining, relatively small (;15B), and acquiring better ICL ability might require training more parameters for more iterations. Finally, would like highlight the lack of in depth analysis of ICL in the context of LMMs, which we keep for future work.
B RELATED WORK
LMMs. The success of Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023b) has spurred considerable efforts to extend the potential of these models to more modalities (Chen et al., 2022b; 2023; Huang et al., 2023; Li et al., 2023c; Wang et al., 2022c). In particular, Large Multimodal Models (LMMs) (Alayrac et al., 2022), or multimodal models (beyond one modality) that train large number of parameters (beyond 1B parameter) on large datasets (hundreds of millions of examples). Typical LMMS build on top of LLMs, with additional adaptation modules. These models mainly differ in the adaptation modules (Shukor et al., 2023a; Li et al., 2023c), pretraining data (Schuhmann et al., 2021: Zhu et al., 2023b; Laurençon al., 2023), and initialization (LLMs). These LMMs surpass the performance of traditional finetuned multimodal models (Li et al., 2021; Shukor et al., 2022; Dou et al., 2021). Recently, proprictary model called Flamingo (Alayrac et al., 2022), has been proposed. followed by several open source models such as Open Flamingo (OF) (Awadalla et al., 2023) and IDEFICS (Laurençon et al., 2023). While most LMMs are currently tailored to image-text tasks, many works have demonstrated the potential for extension to other modalities (Shukor et al., 2023a; Girdhar al., 2023; Shukor et al., 2023b; Zhang et al., 2023).
ICL. One of the emerging abilities when scaling LLMs is In Context Learning (ICL. (Brown et al., 2020; Dong et al., 2022); the ability to adapt the model from demonstrations. Several works target the design of the context prompt to enhance ICL effectiveness (Lu et al., 2022; Liu et al., 2022; Zhao et al., 2021), and improve the model's reasoning ability (Wei et al., 2022; Zhang et al., 2022: Chen et al., 2022a). Few works have used ICL for aligning LLMs with human preferences, such as generating safer dialogue (Meade et al., 2023) and producing harmless, honest, and helpful text (Askell et al., 2021). However, the investigation of ICL in the realm LMMs remains limited, where previous studies (Tsiipoukellli et al., 2021; Alayrac et al., 2022; Huang et al., 2023) mainly focused on adapting pretrained LMMs to solve general benchmarks like VQA, captioning, or classification.
C BACKGROUND ON LMMS AND BASELINE MODELS
We consider 10 different LMMs from OpenFlamingo (OF) (Awadalla et al., 2023) and IDEFICS (Laurençon et al., 2023) as described in Table 1. For OF models; the multimodal pretraining of all models are done on part of the documents from the Multimodal-C4 dataset (Zhu et al., 2023b) and image-text pairs from the english LAION 2B (Schuhmann et al., 2022). OFv2-4B models are trained additionally on ChatGPT--generan data. Note that, the first version of OF (OFv1-9B) is trained
18

---

Published as a conference paper at ICLR 2024
on less data compared OFv2 models. For IDEFICS; the multimodal pretraining is done data from OBELICS (Laurençon et al., 2023), LAION (Schuhmann et al., 2022), Wikipedia (Foundation) and PMD (Singh et al., 2022). IDEFICS (I) is trained additionally on several instruction tuning datasets. The architectures of all models are similar, with the main difference in the model size and initialization (which LLLM). Specifically, these models consist of frozen decoder-only LLM (e.g., LLaMA, MPT), frozen vision encoder followed by a perceiver resampler (e.g. CLIP-ViT) and gated cross-attention injected between LLM blocks. The learnable gate cross-altentions helps to stabilize the early stage of the training.
D EVALUATION SETUP
The evaluation of all models are done with zero-shot (a la Flamingo: 2-shot without images) or few-shot ICL, without any finetuning. In the paper, when we refer to evaluation we usually mean to the zero-shot setup. For efficient inference, we the accelerate library (Gugger et al., 2022) from transformers, and run all OF models with floatl6 (which leads to very small degradation in performance compared running with float32). For IDEFICS the inference is done with Bfloat1 For ICL, we follow the standard approach and randomly select the examples from the corresponding datasets. For each benchmark, randomly sample subset of examples and divide them into separate query and context examples. Each score that we report is the average of scores after repeating the experiment times. We use the official open-source implementation provided by the models' authors.
E BENCHMARKS AND METRICS
COCO (Lin et al., 2014) (object hallucination) is widely used image captioning dataset. It consists of 118K images for training and 5K for validation and testing. Each image human- annotated with 5 different captions. We use 5K examples from the validation set. This dataset is used to evaluate object hallucinations with the CHAIR metrics (Rohrbach et al., 2018). These metries are based comparing the textual objects in the generated captions to the actual objects present in the image (from the segmentation annotation of COCO images).
TDIUC (Kafle & Kanan, 2017) (abstention) is a VQA dataset with 168K images and 1.6M questions divided into 12 types. The questions are imported from COCO, VQA, and Visual Genome in addition to some annotated questions. One type them is absurd questions (366K nonsensical queries about the image). We sample 8K examples (2%% of them absurd questions) for evaluation. To report the abstention metries, we use the same metrics used in binary classification; accuracy and Fl-score which is the harmonic mean of the precision and recall.
CREPE (Ma et al., 2023) (compositionalityy is a large-scale benchmark to evaluate composi- tionality (productivity and systematicity) in vision- languaee models. Based on the visual genome dataset, they propose an automated pipeline to generate hard negative captions. In this work, we focus on systematicity. For HN-Atom, the hard negatives are created by replacing the objects, attributes, and relationships in the ground truth captions with an atomic foil (e.g., antonyms). For HN-Comp, they concatenate two compounds, and each one of them contains an atomic foil. We evaluate on 5K examples, randomly sampled from a test set designed for LAION (as the evaluated models use LAION during pretraining). The main difference to our work is that instead of image-text retrieval, consider this benchmark as image-tex: matching (TTM) or image-tex selection (ITS; where the model is given a correct and incorrect caption and the task to select which one describes the image). For these created tasks, report the binary classification accuracy (e.g.. for ITM if the caption describes the image or not). We stick to the accuracy as we sample balanced context demonstrations.
SugarCREPE (Hsieh et al., 2023). Is a benchmark to remedy the previous hackable datasets, by reducing the biases and shortcuts that can be exploited when evaluating compositionality. This is mainly due to using LLMs instead of rule-based templates to create hard negative examples. It covers 7 types of hard negatives; replace (objects, attributes, and relations), swap (objects and attributes) and add (objects and attributes). Each image is associated with a positive description (image caption) and several hard negative descriptions.
19

---

Published as a conference paper at ICLR 2024
VQA-X (Park et al., 2018) (explainability) is based the VQA and VQAv2 datasets, and contains 32K question/answer pairs and 41K explanations annotated by humans. The explanations are intended explain the ground truth answer for the question, based on the corresponding image. We use the test set of this benchmark (1.9K pairs and 5.9K explanations). To evaluate the explainability performance, we consider captioning metrics such as CIDEr that are based on the syntactic similarity between the generated explanations and ground truth ones (annotated by humans).
LlaVA (Liu et al., 2023b) (instruction following) consists of synthetically generated instructions of images from the COCO dataset. The authors GPT-4 (OpenAI, 2023) to generate intricate instructions that can be categorized into categories; 23K detailed descriptions, 77K complex questions, and 58K examples of conversations between humans and AI agent. To generate the instruction, GPT-4 (text-only) is prompted with several handcrafted examples (ICL). To make understand images, the image transformed into set of bounding boxes and captions, passed as a sequence of textual tokens to GPT-4. For each category, we sample randomly some examples from the dataset of the same category. GPT-4 is used evaluate models quantitatively (Liu al., 2023b). Specifically, we ask text-only GPT-4 to evaluate the performance and give an overall score. However, evaluation based on LLMs are biased and might contain some flaws.
F ADDITIONAL EVALUATION EXPERIMENTS
F.1 HALLUCINATION
Table 4: Hallucinations. Comparison with other image captioning models. ": heros-hot without context (in
contrast a la Flamingo used in the paper). SoTA results from (Dai al., 2023b; Shukor et al., 2023b).
Method
CIDEr f
CHAIRs -v
CHAIR, +
BLIP Large (Li cl al., 2022)
136.70 70
8.8
4.7
VinVL (Zhang et al., 2021)
130.8
10.5
5.5
OSCAR Base (Li al., 2020)
117.6
13.0
7.1
OFA (Wang et al., 2022b)
75.27
4.36
3,98
UnIVAL (Shukor et al., 2023b)
91.04
4.44
3.64
LMMs: Zero-shot
OFv1-9B
65.64
17.38
OFv2-3B
73.93
6.85
OFv2-3B (I)
73.54
9.07
8.61
OFv2-4B
73.14
8.35
7.69
OFv2-4B (I)
77.89
7.32
6,58
OFv2-9B
78.10
7.21
6.63
IDEFICS 9B
63. 63.22/40.22 22
31.42/4.95'
28.35/5.55
IDEFICS-9B (I)
103.42/52.31
18.25/5.61
16.96/4.19*
In Table 4, we provide a comparison with other multimodal models. Most of these models are finetuned on COCO dataset, except for OFA and VnIVAL (that use COCO only during pretraining). Despite being an order of magnitude larger, LMMs generally hallucinate more than other baseline models. This might be due mainly to training COCO dataset and not relying LLMs. For IDEFICS models, we noticed very high hallucination when evaluated in zero-shot a la Flamingo.
F.2 COMPOSITIONALITY
CREPE, Figure 6, we complete our evaluation on the CREPE benchamrk by adding the results for HN-Atom HN-Comp.
SugarCREPE.
We evaluate LMMs on SugarCREPE. Figure 7 shows that all LMMs suffer on this benchmark. revealing that previous improvements on CREPE is coming mainly from biases in the dataset, rather than acquiring compositional ability.
Comparison between ITM and ITS. Figure 8 provide a comparison between ITS (HN-ITS) and ITM (HN-ITM) the CREPE benchmark. We notice that ITS is much harder than ITM with hard negatives. We also include two baselines (ITM and ITS) where the negative caption is sampled randomly from the COCO dataset. Without hard negatives, LMMs perform very well, revealing
20

---

Published as a conference paper at ICLR 2024
Systematisiiyyy.o Aoom)
Systersniity (omo)
Systematichy 1Ht Itftetoon HHSSompl
100)
EDSPEC
100
ID2PE3
110
02715-40 60
0112000
CP/1-00
70
50
Figure 6: Compositionality. Models are evaluated on the CREPE benchmark with the ITM task.
Replace (OSJect)
Regiace (AMt/RUUC))
Replace (Seettion!
INFRCS
ESEFKSS BER
EIERSS
ICPPICS 06
EEPICS-
ESEPICS
1CFFIC--00
92
GEFICS-BD
13C7125-90
40
60
Acd [0bjec))
Add (AIL)bAA))
Seap [Objet1)
12EFIC2-200
EEFIC2
12EPCS 902
IDETICS-8
DETICS-00 1D
(XC715440 01
ESFICS
IEERCS
00
Cha2-18
50
The excumcc
THE
20
93
Smap (Allritate)
Avariage
100
(D5F-C5 918
1031 CS-923
IDEFIC5-300
90
OCd1 50
Hander
Rardien
119 AcunE))
36
Figure 7: Compositionality. Models are evaluated on the SugarCREPE benchmark with the ITM task.
that the poor results with (HN-ITM/ITS) are mostly due to a lack of compositionality and not the difficulty of the task itself.
21

---

Published as a conference paper at ICLR 2024
Systemalicity (HN-Atom)
Systematicty (HN-Comp)
Systematicty (Hi-Atom + HN-Comp)
Random (TM [T5
Random
Actumay
Acounccy
TN
Acsumay
-M4TTH
52
Figure 8: Compasitinality. Comparison between ITM and ITS the CREPE benchmark.
Detta-ee Description
LLVVlComplex reasoning
Conversavnon
80
80
60
70
5.0
IDEFICS-0RR
IDFTICS-R0n
IDFFICS-a0R
40
[DEFICS 808
IDEFICS 808
OF-2 BCB
IDEMCS-98 10
IDEFKCS-98 ()
IDEFICS-08 (0)
IDETICS-00
IDETICS-00
IDETICS-90
0142-93
40
DPv2-90
OF-2--5 96
of shots
shots
shots
Figure Instruction following. Quantitative evaluation on the the LlaVA benchmark on 3 types instructions (from left to right): detailed descriptions, complex questions and conversations. V: 2-shots without images
F.3 INSTRUCTION FOLLOWING
We provide additional quantitative Figure 9 and qualitative results for instruction following; detailed descriptions (Figure I),, answering complex questions (Figure 12), and conversation with humans (Figure 13) from the LlaVA benchmark. Discussion about the limitations can be found in Appendix J.
Table 5: Mean (AVG) and Standard deviation (STD).. We show that STD of our evaluation is not significant.
Mode
Yakk
4-bbet
8-da::
10-due
33-4bol
CIDE,CHAIR, CHAIR
STD
0.590 591122
0136220013
63800.230.166
AsolAbard Acc/Absad
.M.666999
SAR
3406
Computrratty TRIIE)
AMG
$18890 7R/58
Fompliii
Eisplanabuls (VGA-3)
ANG
26.77
ALAS
74.22
MLS1
STD
LBB
CUM
2.94
1.35
2
(COO
ANg
40 2022590 956.52
100 10L5440300000
18
IALI BOLLIBOOSNLL
CIDE HARHCCARR.CHAAR JCHAIR
STD
0.559 257.31
0.7MOAAALLC 06
R22A 21/012
N.41 3611
(VOA-X
AN0
50 $9/6
51.9/77 1347
75167
IDSFICS MB
Acs/Abaan Acc/Abaad F1
STD
0,370 (0X159
O ELENG 19
0 0.N0M 546.75
0.280 150.3)
Corpuatrettty
STO
Esplarabitn (VD4-3)
MILTS
22

---

Published as a conference paper at ICLR 2024
Welhi
didics derds
cresing
25:
Ho wealtng Feiree
medilation to
design.
QP/248
ICL:
The mage dupictu group craftir stand with care cosnd specnds protiss withyoos roee
The iis bo procu ton rota eoteeet becuus he
The paproeu inter bwtein tio redullution botieattoooooooo could be
encoll
cas
con paturo tonns. pritor plbers large will plantr
fall Aedin setronaall feireis can anker sepelter
coudiii dy anita comoortioe voluee eireriia ducf
28:
greille
in werve and
soaring hermer
parseee trkent put ii ertaiic mililaate
piiils
ligs The pialle ling legi. grodie
10EFICS
IGL:
The image stowe hart groffie sainding granry
The ina bay titta mage tnig helleet riling
The parpose or intert bahlrr tii an restalation could
tait rede emachhed upo The caffee appsar bn
thttee is seare the raks aetocland Kiing bayya
diggn An gardee cutiior saace The inseoonttc of
the aforiull giafts the hord. The gi sallion unt
uraa lite insuraaiii to wai hal whil edingia
could usedlo crant cohuull and hamoorouu landraap
out booioll of Bair.
ball collaion saan Weal hulrrel bubu roduce the
Figure 10: Instruction following. Qualitative evaluation results of IDEFICS and OFv2-9B on the LlaVA benchmark on 3 types of instructions (from left to right): detailed descriptions, complex questions and conversations.
23

---

Published as a conference paper at ICLR 2024
Instnctioo::
Welee Berellie
Pep
wrinkd
por an
ribie the sred The cigets tu milere the aheel and
youn men kapphi to wadling but noarm
04-246
IOLI
eyen
paage gaolls
In rage, airan aranda daarrrggn in laga acappie WiiE
rage balkis poar graa
The
maso
sacie und ricies ellire of
mearg abdl decck loking
and
Uor vis uitar in Boroutt da
The paste seck
robde
ot duria mamer cal
Fus kive resk indlogs
eeceeno cod uomre conninng wittatee
garsen and somar la searing seriing-....
Iar radi arecher certt. The acoverts to
vabide used as cnperpprraio 1h Thalend bitull
indhals hoding and tame this right
ica de joe beal la wili thai logi shaltwe
entte sile The vraad Lase dul aogus thae
ena and hargle fick There cluf
But There seviral
woart scpeasy dasting aulicss for ang took
nariing de Blanals
weggraer that aede
Figure 11: Illustration of instruction following on detailed descriptions.
Inttnction::
Vir rapp was chosn ta Baar
Eaplar raad conter rage grazt
aitali doys paocudee staurtty?
to flor nit
piofts wabel Pesmurra
07/288
ICL:
The lav clicann to lay cnw flor nis
The praftcs saliac arland hooe aus to aralinet Savrman
Doy's pina roberi Encias
dop
iin luse erg may thasen
dees para evaeted
in soodi farancess.
alp shap cojicil casae jary to
bagg head baraku UV
appears
koutre leme foem de- them vod
IDL:
The tar chipen to coy the floor rit
The aoe shoure govfle sanding gieee Red viich
Th liile soy image in neartig yeiret
alrcc and ai Adiloonda dee foor cass
ciber ulaa sasad focll de kand
daladda with ridng lake
pal eeter
dog may drase dovtt of fiedl.
aich alco han' to ees rech high-iooi g partt.
nil of newd Wodee sind 1il collace.
Figure 12: Illutratioon of instruction following on complex questions.
Instrcttoo:
Hord
pary sany susdhee
mening le arlee dec deig
darta deri
oucare make
28:
Mdeiiiated te di dlnn be
mume
07-248
ICL-
ceeoeiiire of
and
the ses wes aodd
ned egunn dene dessan moce rikding
inecr briting paenn poriiii
28:
roperie The seveelee codiy bunk alt
cpalice
en emartie and
There mas mepore Entieaa
wal ensecin hotter
meddeeaan regitiiis and
The Meee Masicaale 1i sndi
Thes teraae bishaan
mecage oo di
demluct puaat code anig and hel comeiiidly plorlgg
Figure 13: Illustration of instruction following on conversations.
24

---

Published as a conference paper at ICLR 2024
G ADDITIONAL DETAILS FOR X-ICL
Content:
Contoct:
yewr the home deig? Sumping Deplastion The 8.3
Cc Wiar ror man Sqparadoe: Thwee feepliee
Eoes the may tas ratccet & you Expla NNe
whee the van ssing:: ceeping Raporatiior
Whar and ofven chown:: sar Expioration is beys
ein hor soors you puuger tas oboert ai you
Quary:
Quary::
Query:
perood vaaley Exptanation
Whinkls mech Eepeasatee
Reploration
twoe to of paoks grow
there seiee goound
Hokkdt Node and Novan Opeaovi are teo the
CoH-ICL:
G: the hons dong? A jumping Good Explanatir:
What rpam tent Au seg mon Good Explanator:: Thwrs roort
Coos the man have tatbear? A you Good Explanation
Explorations ia jump over barnd
latoc on his are halding licit scttbnuis
layingg bed and ecrrovee eng Bidd Explanation The maniss
lrgg de whicr has assiit sish Bad Expraradariit
Equlasion he in much ane and has par cs sing Bax
Quarr:: Good Explanation
Quany:
ofece mich
the appooria to bas cnktiday"
What senon doing vidled Eaytorationn
Figure 14: Explainability with COH-ICL. The model is prompted with good (written by humans) and bad explanations (from previous model generations).
Slain-of-Hindsigee ICL (CoH-ICL). Chain of Hindsight (CoH) (Liu et al., 2023a) consists of training the model to generate both helpful and unhelpful answers, by providing both answers as input to the LLM, each preceded by a the corresponding prompt (e.g. "helpful answer:". "unhelpful answer:"). Inspired by this, we propose CoH-ICL. Specifically, for each image we collect positive and negative description. The good/positive description (image caption) is annotated by humans and the bad/negative description is generated by the model itself. As illustrated in Figure 14, during ICL the context consists of several examples as follows; an image, question, answer, human annotation as the good response, and previous model's generation (with ICL. 32-shot) as the bad response. More formally, Equation (I) for CoH-ICL can be written as:
= [<<<image>,TT1]]
</doofchunk |>)] and (<image>TT**)
(5)
where + /R+ and T /R- refer positive and negative demonstrations respectively.
Q: W/wari the wortn dolg with the oel phone?
What otlr ae tine?
(hot see the Hig deing?
Whalls
the
Ct: What color
Ot: What coleg?
taking
orange
aating
SC-ICL:
meay?? Whete woren sengwer phooeet
mngge Vihct 0001 to aroe?
beard inage? wis are byg shrap doing?
Fioll knewer syart cvverttpply
Flaxl Anoowe egeft dccenotapply
Final Answar syarf dowenotogaly
Figure 15: Illustration of SC-ICL for answer abstention.
Self-Correcting ICL (SC-ICL). Self-correction (SC) (Pan al., 2023; Madaan et al., 2023; Raunak et al., 2023), consists of using the model itself to automatically correct its generated answers. We explore similar approach help the model abstain from answering. illustrated in Figure 15, our SC-ICL consists of the following steps:
We first simply ask the model the question Q using ICL, and the model gives an answer A. This is the typical ICI. approach used to evaluate the model different VQA benchmarks.
25

---

Published as a conference paper at ICLR 2024
2. Then, provide the same question Q input and ask the model if it is relevant answerable given the image.
3. case the model recognizes that the question is not answerable, the previous answer is ignored and replaced with abstention keyword. Note that, in case of SC, usually the model itself correct the answers, but here we employ this heuristic mechanism.
Formally, the SC-ICL steps can be compressed in 2 steps as follows:
C1 ((<<(((mae>;TT) R (eneoo/kkkkddooc
(6)
endofchunk) >) /2 <<image>?""T").. LM ( J2)), C2 =(<<image>>112m: 2
where T is fixed question ask the model if the following question T; is relevant to the image, and R² is yes or no. The final answer is given as a function F (heuristics) of 01 and 02, i.e., 0 = F(01, 02).
Question:
drsap
GT:
stapely
ICL:
Aseaer hot
Anaowe Mice
Asswer
MT-ICL:
Ansner: dbcorstapply Uuay th pewwArrranns
Areoner: docortharlyy Utao prsnics ganition visntt the
Answer no Dona pashoo gandon dhache
Figure 16: Illustration of MT-ICL for answer abstention.
Multitask ICL (MT-ICL). Multitask learning (Caruana, 1997) consists of training the same model on different tasks. We propose to do multitask learning in context. Specifically, the demonstrations contain two tasks, such as simultaneously answering the question and deciding whether the question answerable not. Figure 16 illustrate MT-ICL for model abstention. More formally, With T/R referring to task j, the context C Equation (1) for MT-ICL. can be written as:
tinnccch C= ((<(inngge>; T/RRTR endoffhhnk 1>) {>)) and / (<image>T")
H ADDITIONAL X-ICL EXPERIMENTS
Here provide additional experiments with different X-ICL variants to address hallucinations, abstention, compositionality. and explainability. We skip the instruction following ability as we do not have quantitative metrics measure the improvements over ICL.
H.1 EXPLAINABILITY
Table 6: Explainability. Overall task accuracy and CIDEr for explanations on VQA-X. ICT. here refers to single task ICL (answer or explain).
Model
Mithol
CCible
4-shor
S-toor
16-bot
30-too
OPV1-98
Gall-Its.
67.41 76.5
90.48 (wd
74,52
83.15
$1.29 (43.19)
MTECCI.
66.02 (+) 05)
71.75 Saj
0006 (43000
95.26-1.32
12.07 (+2.19)
72.89 72.81-2669 (2.64)
73.22 (42.1)
79.22 (4.57)
Cal--E3
70 (-l)
78.97 (wd
82.27 (-1
71 27 (d.75)
MELCL
74.36 (1566)
07.02
(1.79)
74.38 (10.17)
74.59 (10)))
77.24 (3.50)
74,42 (12,49)
76.40 (-409)
75.30
38.50
JCL MTTCC)
74.50
76.51
(40
76.75 (1653)
53.56 (-1
74
85 (-1.32)
ID0FTCS-9000 (1)
$3.03
90026
$4.35
94.54
1.99
$4.35
96.33
$2.00
94.27
Table 6 provides additional results with CoH-ICL. CoH-ICL significantly improves the scores over ICL with all models. We also provide some qualitative results in Figure 14 to illustrate the approach.
CoH-ICL.
26

---

Published as a conference paper at ICLR 2024
Explainability VQA-X
IDEFICS 98
4-shot
100
0Fv1-98
8-shot
0FV2-4B
32 shot
0Fv2-48
80
OFv2:38
SoTA
60
50
30
Accuracy
Figure 17: Explainability with MT-ICL. The model is asked answer the question and explain its answer. We report the CIDEr (f) for explainability and the overall VQA accuracy (L).
MT-ICL. Figure 17 shows a comparison between different LMMs. LMMs answer the question and then provide explanation. Increasing the number of shots in ICL significantly improves both tasks. Interestingly, IDEFICS (I) is able to surpass the current SoTA (NLX-GPT (Sammani et al., 2022b) unfiltered scores). In addition, Table provide results for different models. Compared to ICI., the overall accuracy is increased with OFvl, OFv2, and IDEFICS models. For CIDEr, the improvement mostly with a small number shots, except IDEFICS (I).
H.2 ABSTENTION
SC-ICL. In Table 7, we provide the results with SC-ICL (correction with the same number of shots in both SC steps) and SC-ICL (32 shots) (correction with 32 shots). SC-ICL (32shot) is significantly better than SC-ICL which is expected classifying questions (relevant the image not) is better with more shots. We illustrate SC-ICL in Figure 15. With IDEFICS (I) model, the model tends to answer the question instead of deciding if is relevant not (which might also be the reason why the improvement margin with IDEFICS is generally smaller than OF models). More adapted prompts should fix that, which we keep for future work.
The model here, simultancously answers the question and decides whether the question relevant the image not. Table shows that MT-ICL is better than ICL on answer abstention, especially with small number of shots. We illustrate MT-ICL in Figure 16.
MT-ICL.
Table 7: Abstention. We evaluate the ability the model to abstain on the TDIUC dataset.
Muddl
Methed
Acc. Absard Acc. Absunt Fi
4-clat
Solist
16-laat
32-duo
JCL
$7.14
67.32
31.02
46.71
69.95
43.90
52.87
76.54
$7.80
$7.16
79.40
63.87
0Pv1 9B
ITLICL
SC- KCL
39.82
42.09
72.35
62.52
41.61
36.17
47.33
46.49
74.34
65.01
49.64
47.5
52.63
53.53
75.10
76.63
59.35
57.51
55.83
57.23
78.13
77.49
64,90
62.85
SC (32shoo)
45.34
70.32
52.00
50.53
73.30
57.11
54.95
76.35
02.46
$7.22
75.10
64.56
ICL.
00993
73:46
25.27
46.71
75.50
42.00
65.83
77.54
$1.80
46.83
79.13
56.44
01v2 90
MTHICI
SC ICL
47.99
43.32
77.18
70.93
42.50
20.90
47.26
48:41
72.76
75.95
$2.57
48.00
40.13
47.75
75.56
76.40
$6.57
$4.58
48.5
48.25
78.00
77.7
60.16
59, 14
SC ICL (32shoo)
64.38
73.2
27.34
46.92
74.95
$2.55
03.35
76.51
$7.41
47.95
77.49
$9.93
IC1. MT-ICI.
45.41
74.33
32.00
51.5 50
77.12
47.51
53.01
$0.30
66.22
61 04
81.75
67:45
IDEFICS-OB
SC ICL
45.30
45.13
68.19
76.61
43.25
37.82
51.1 10.50
52.27
74.67
78.90
$3.41
48.60
54.75
53.75
79.64
$1.36
62.55
59.55
62.56
58.51
$1.84
92.67
67.57
68.62
SC ICL (32ahor)
49.56
77.06
99.55
34.75
78.89
$7.96
59.21
$0.73
64.16
62.77
82.01
68.96
IDFFICS-OR())
MTHICL
JCL.
60.24
50.57
79.37
79.34
20.91
35.38
63. 301
63.64
82.65
83.30
46.23
50.52
66.94
68.20
56.17
55.85
61.16
64.88
68.56
70.50
85.35
86.91
68.39
72.75
27

---

Published as a conference paper at ICLR 2024
H.3 OBJECT HALLUCINATIONS
For object hallucinations, we use object recognition (listing existing objects in the image without localization) as an auxiliary task (using the prompt "There only these objects:"". The motivation is that recognizing objects in the image might push the model to describe only seen objects. From Table 8, we noticed that this approach reduces object hallucinations when the hallucinations is significant (OFv1-99) 1-9B and IDEFICS).
MT-ICL.
Table Hallucinations. We evaluate object ballucinations on the COCO dataset.
Model
Mathod
4-sbot
S-dtot
CIDE: (CHAIR. CHAIRR
16-thol
32-shor
OEV1-9B
MT-ICI.
ICL
73.88
75.36
13.53
12.58
10.82
10.34
77.60
78.98
13.75
12.68
10.90
10.49
50:5747
81.38
13.32
13.94
11.08
10.59
$1.57
53,82
14,08
13.04
11.31
10.41
0PV2-9B
MT-ICI.
ICL
90.46
$7.43
5.02
5.64
4.54
4.35
94.13
96.29
6.93
6.43
5.28
5.03
98.09
96.01
8.03
7.99
6.16
6.05
94.60
99.55
10.74
9.00
8.25
6.70
IDEFICS-9
MT-ICI.
ICL
100.54
96.44
9.39
7.76
6.08
0.96
102.15
99.70
8.08
9.27
6.81
6.15
101.72
102.09
7.73
9.37
5.94
6.88
109.90 90
103.15
9.56
7,66
5.85
0.99
IDEFICS-B
MT-ICI
ICI.
129.84
133.89
4.79
3.76
3.15
2,56
136.12
132.55
4.42
3.90
2.65
2.96
134.25
136.81
4.36
3,88
2.7
2.62
136.56
135.99
3.78
3.89
2.62
2,60
H.4 COMPOSITIONALITY
Table 9: Compasitionality. We evaluate compositionality on the CREPE benchmark.
Model
Method
[IN-Atom
JIN Comp
[IN- ANo-Atom-
IIN Comp
<-shot
S-sbot
16-shoi
32 shot
ICT.
56.48
63.55
59.54
57.4
68:21
60
57.57
79.77
66.51
56.33
$5.44
67.39
OF 9B
MT ICL
57.57
61.88
60.58
$6.47
69.89
61.83
58.31
77.55
64.80
59.62
$1.28
67.99
ICT.
55.70
58.93
56.64
53.32
61.06
56.63
54.32
69.67
18.71
$2.30
75.61
60.59
OFv2-90
MTHCC
57.18
68.54
61.00
55.67
78.74
63.94
54.52
88.53
68.45
52.88
84.85
66.96
ICT. MT-CCL
58.98
64.31
63.11
80.24
60.54
96.39
IDEFICS-9B
56.86
65.25
57.32
62.60
78.56
IDEFICS-OB.01 (1)
MTHICL
56.26
56.92
56.32
56.26
59.03
57.10
58.09
61.68
58.83
55.13
58.70
57.18
MT-ICL Here, we also consider object detection as an auxiliary task, if the model able to detect the objects in the image, it should be able to recognize when the caption description is false (when randomly replacing objects in the caption with atomic foils). In Table 9, MT-ICL seems to have a positive effect on HN-Comp, where the ITM accuracy is significantly improved. We notice that this approach works when the performance on compositionality is lower (OFv2 and IDEFICS (I))
I CAN TASK INSTRUCTIONS HELP ICL?
Table 10: Task instructions used different benchmarks (Appendix I).
Rencmarioo
Task inatrctions
Object hallacteeiiinn (0000))
Desorbe following de inclode
Here few illarzaton currlles
Alsieating (TDIUC)
Answay the folowing quesioon about the ingge give shott asswers you do net know the snows opestion relevart to image sy docinntagyyy Here ills stiin cumples.
Compusiionaliy (CICEIE)
You roed to find prosided ummmon acousstl Amsethe the the image then the annrmor des are dramib the imag. You also neod ceneii thet can help dodide Her fow ilsaamation eusmpies
composiinn of the sentence dree mucch
Explabublly (VQA-X)
explasation be given quentio szawer, need give Box
In practice LLMs are augmented with relatively long instruction, explicitly describing the task. In this section, we investigate if giving the model an explicit instruction (illustrated in Table 10) can help. We show the results in Table 11. We can notice that the added instructions can bring significant improvements with a small number of shots. However, when adding more demonstrations (8/16/32- shot) the effect of the instructions starts to be negligible. This is expected, as more demonstrations will help the model infer more easily the task from the context examples.
28

---

Published as a conference paper at ICLR 2024
Table 11: ICL with task instructions. Adding explicit task instructions can help get additional improvements with small number of ICL shots.
Model
Teanation
In-dee
24/22-3
33
CIDECCHAIR.CO CCAARR
82115331112 57
75 SW1555311
76.7614.111.0
82. 82.8815..6/11.
(VQA-3
44
57.15/9945003.1
Ancibburd
6877
6969
68HE8
4675 78457
Compoatroeaty
HN-Awesit Aacp
57,51688
Explarahisy (vgA-X)
C11
29

---

Published as a conference paper at ICLR 2024
J LIMITATIONS
J.1 INSTRUCTION FOLLOWING WITH ICL
Instruction:
Asalyoa
camlled
Whande
anaperoar?
What' happeiing acans?
braah
OF-248
of the and has meti foons The silcasea la browe
A4 Icau has anal on
Thes voage fot sisage carcoss wit
Suaae
The sorge cxpture a boaes withi withinn
The image wi bouthouus ek tee nouth
Forms sousue revearing nn warity attene
wnsinng couata surnnn recking oag
Te trsken dearriinn frrd weila will becihinuun accears vit
also anudicts
aiveaal peopla apara zone ofttarn
indicatig
lnad with treas, adding
Z5:
The inaggn gen sean Thes casse maak gur honddaa fandl
The Beach undiods
bears.
wood
hor die
IDERCS
made sood. The fas hondls
KEAL
of wen cesooa eatioes reencing
urdtnnis card andansi opis
ippwas soor call snaiucto
wottts wareri onipn Baro
High sooe betheture with des
subiaie thes civerca Farms
touig there and
ssund plosed witl loofl pails sike and
suggnes ththe inda brushing
Figure 18: Limitations of instruction following with ICL.
Despite being effective in adapting the model to follow user instructions, we noticed that ICL has several limitations. Mainly, while the responses are longer and richer, they sometimes include significant hallucinations. contradict what is seen in the image. Some of these limitations can be seen in Figure 18.
addition, they might include inaccurate and wrong statements that
J.2 X-ICL
X-ICL variants partially solve the flaws of LMMs, here we highlight X-ICL limitations to provide more inspiration for the community devise better variants.
STOR
1 A
Bo bepoard whelas? you
O Should sibes be the raad7 A
anticr eaanng
lipippee leybourd
ofter
CoH-ICL:
Quary: Good Beraneter:
Query: Deed Explanatoen
Quarry:
Good Explevation
thenis 053 dgn Pariium.
Pein i siriau synbol kaybeand
the sebis tnts vidile of raad
Figure 19: CoH-ICL limitations with explainability VQA-X. The generated explanations more like image descriptions (left), include hallucinations (middle) and be unhelpful (right).
CoH-ICL, This variant also suffers from several limitations as illustrated Figure 19. In the case of explainability, the generated output is more like an image description than actual explanation. ICL can introduce some hallucinations and provide unhelpful explanations.
SC-ICL. As illustrated in Figure 20, SC-ICL can fail on some abstention cases such as not recognizing the question absurd or relevant. In addition, correct only in case we classify the
30

---

Published as a conference paper at ICLR 2024
|||II
Question:
Tene shorp letty to koding or ooof? Azawer commrstapply
Whats Be st oolor
Wiai lo made
GT:
Acawert
mobi
ICL:
shoup
cour?
What ludk cuke? dossnotappty
Whale made
wood
All dneenstagpyy
woue
irage? Are hass stres Boayy be fesling hot or cod??
inage? "Wai la the tais calon"
tased imags? What's tack made off
Firal
answer: desegalfpply docanotasply
Findl cocanotapply
Figure 20: SC-ICL limitations. Some failure cases on TDIUC abstention benchmark.
question as irrelevant, thus we do not consider the case when the model abstains in step and then classify the question as relevant in step
Question:
GT:
skiabbaadingg
Answer dostrctaply
ICL:
Ancear: anontcardin
Arswar: blus
Anseer orange
MT-ICL:
Anseer: unowbcardigg Ooss te prwcbat quartio
Answer: Eoeanotapply, Doar previuur qaetion susetbe Ve
Anseee dosunclapply Dees tee pawdice quaato
Figure 21: MT-ICL limitations. Some failure
TDIUC abstention benchmark.
MT-ICL. Figure 21 shows some failure cases with MT-ICI. answer abstention. Sometimes there an inconsistency between the output of the main and auxiliary task such as replying "doesnotapply" and classifying the image relevant. MT-ICL does not seem to help correctly respond to the question and fails detect some abstention cases.
31


Reviews:
[{"review_id": "QgTd7nyLvG", "reviewer": "Reviewer_wHgE", "summary": "The paper presents a series of experiments assessing Large Multimodal Models (LMMs) in five areas: object hallucinations, abstention, compositionality, explainability, and instruction following. It shows that scaling LMMs doesn't fully address their deficiencies. In-Context Learning (ICL) is investigated as a remedial measure, with mixed results: it helps in some areas but not others, and can even increase hallucinations. Despite these insights, the paper suggests innovative ICL variants with potential for improvement.", "questions": "1. How is the selection process for demonstrations in context $C$ determined?\n1. Could you specify the evaluation metrics used for each of the five axes and explain why the metrics can measure the axes?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper provides a thorough critique of LMMs by examining their performance beyond standard benchmarks and introducing new ICL approaches to overcome their limitations. It show some insight for understanding LMMs' applicability.", "weaknesses": "While the paper addresses an essential aspect of LMMs' functionality, it falls short in pioneering novelty. It does not introduce new datasets or unique evaluation methodologies, which limits its contribution to the field. The potential impact of the proposed ICL variants is not sufficiently analyzed, and the paper overlooks a discussion on the selection process for demonstrations in ICL. This lack of methodological detail, particularly when compared to existing demonstration selection strategies based on similarity, raises concerns about the practical implementation and reproducibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MudZbpThlJ", "reviewer": "Reviewer_PTKm", "summary": "This paper proposes to study the issues of large multimodal models and proposes a way to improve their performance through in-context learning. The paper proposes an axis of different ways to evaluate models (hallucinations, abstention, compositionally, explainability and instruction following) and illustrates how multimodal models typically struggle with this, with in-context learning further hurting performance. The paper proposes a new way to do in-context learning which improves performance along each of these axises", "questions": "- Can the authors provide psuedocode for each of the in-context learning methods that are proposed?\n- Why does the modification to in-context learning of presenting both positive and negative demonstrations  correspond to CoH in context learning?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper analyzes the performance of multimodal models and illustrates a variety fo axises on how they might fail\n- This paper proposes new methods to improve the in context learning performance of methods", "weaknesses": "- The results lack confidence intervals -- in general there is very large variance between the numbers. For instance in Table 2 and 3, there is large variance of a single method between a different number of shots which seems larger than the reported gains of the proposed method . \n- It would be good to report quantitative results for each result in the paper.\n- I had difficulty understanding the main differences between the new in-context learning methods presented in the paper and the performances seem minor\n-  The existing evaluated models seem out of date -- it would be good to compare with models such as BLIP or LLava, MiniGPT-4 or GPT4V", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXwXfOzwtd", "reviewer": "Reviewer_dAEJ", "summary": "This study assesses the shortcomings of open-sourced LMMs (3B-9B) in areas such as object hallucination, abstention, compositionality, explainability, and instruction following. The authors show that LMMs face challenges in most areas despite having ~9B parameters. The impact of In-context Learning (ICL) on these limitations is also examined, showing mixed effects; it helps in some aspects, worsens others, or has minimal impact. Variants of ICL are proposed to mitigate some flaws, aiming to continually improve benchmarks.", "questions": "With three random trials conducted, it remains unclear whether the reported results in the tables represent mean or median values.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written, clearly highlighting essential results. The use of colored words effectively guides the reader to the relevant axes under examination.\n2. By offering novel insights, the paper contributes meaningfully to the future development of LMMs.\n3. A commendable effort has been made to cover multiple open-sourced LMMs\n4. The authors thoughtfully propose modifications to ICL, aiming to induce improvements along various axes.", "weaknesses": "1. While the modifications to ICL yield improvements in open-sourced LMMs, it would be advantageous to evaluate these findings in the context of models with >9B parameters, as these larger models are prevalent in the state-of-the-art (SOTA) and also have been shown to exhibit different trends than the smaller models.\n2. A more comprehensive discussion on why longer CL (32 & 64) does not yield improvements across various axes would enhance the understanding of the models' behaviors.\n3. The text in all illustration figures is very difficult to read, and Table 3 is also challenging to interpret due to the small font size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6lGVpGJ781", "reviewer": "Reviewer_sfnT", "summary": "This paper produces a detailed analysis of various open-source Large Multimodal Models (LMMs) across 5 different axes: hallucinations, abstention, compositionality, explainability, and instruction following. The paper identifies how each model’s performance changes (or does not change) with increased ICL examples, and proposes variants of ICL prompting methods to improve performance on some of the identified categories.", "questions": "See above question about automated evaluations for instruction following.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The categorization of various LMM failures and the thorough analysis of how different model variants and sizes perform at each task is well done. The main takeaways from this paper, highlighting crucial limitations of current LMMs and identifying where ICL is and is not useful, and proposed methods for improving ICL in the LMM context (X-ICL) are all meaningful insights for the research community. Overall, the work seems significant, experiments are thorough and diverse, and the writing is clear.", "weaknesses": "- Only qualitative results are given for the Instruction Following evaluations. It would be interesting to see if other forms of automated evaluations assessing instruction following can be used (e.g. checking that detailed answers are indeed more detailed or checking whether the original question was indeed answered by passing the LMM generated text and original instruction into a LLM like GPT-4). While LLM-based evaluations can still be flawed, it can also provide a more thorough view (looking at a wider range of questions rather than the small sample that was presented qualitatively in this work). Have the authors experimented with automatic evaluations for the instruction following tasks? \n- Nit: the table and figure fonts are very small and hard to see.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)