VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models
Authors: Yonatan Bitton
Hritik Bansal
Jack Hessel
Rulin Shao
Wanrong Zhu
Anas Awadalla
Joshua P Gardner
Rohan Taori
Ludwig Schmidt
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Vision-language models
Instruction-following
Benchmark
Multimodal chatbots
ELO rating system
Automatic evaluation metric
Dataset collection
TL;DR: Introducing VisIT-Bench, a dynamic benchmark and evaluation metric for vision-language models. It provides rankings, diverse tasks, and enhances model evaluations. VisIT-Bench highlights performance gaps and supports future research.

Abstract: We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluating instruction-following vision-language models for real-world use. Our starting point is curating 70 "instruction families" that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at https://visit-bench.github.io/.

VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use
Yonatan 1.t Hritik Bansal#3
Jack Hessel*# Rulin Shao
Wanrong Zhu
Anas Awadalla Josh Gardner Rohan Taori" Ludwig Schimdt
Abstract
We introduce VisIT-Bench (Visual InsTruction Benchmark), benchmark for evaluating instruction-folloiing vision-language models for real-world use. Our starting point is curating 70 "instruction families" that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and cre- ative generation. Following curation, our dataset comprises 592 test queries, each with a authore instruction instruction-cconitionne caption. These descriptions surface instruction-specine factors, e.g., for an instruction asking about the accessibility of storefront for wheelchair users, the instruction-conditione caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multi- modal generations using a text-only LLM, aligning with human judgment. We quan- tify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-followii model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to partic- ipate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available https: //visitt//itttii bench. gi thub. io/.
Reasoning
Knowledge
Recognition
Plots
Understanding
Teach me about this pavring.
Where this?
you gainng for plonic location. what items should
Here photo bathrom. How nicer?
Confextual Knowledge of Evants
Figurative Speech Explanation
Chamical Identification
Hazard Identification
Game Playing
Someone said event about anpet Why?
Which chemical this
I you and should
What hand
hand?
you
Figure 1: A sample from the 70 instruction families in VisIT-Bench representing tasks we envision instruction- following vision-lange lingua-angu models should be able to follow.
Equal contribution. HHebrew University *Google Research "UCLA *Allen Institute for AI Unniversity of Washington UCCBB
Stanford "LAION
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
long-standing challenge for artificial intelligence is to build general-purpont assistants that can, in collaboration with humans, solve diverse and weverr-eforr-see seen tasks [1]. For textual tasks, several recent works (2, 3, 4,5, 6, 7] have shown that fine-tuning language models such as GPT-3 and LLaMA with supervised instructtion-respnns examples [8, 9, 10] enables them to respond to imperative requests and questions without task-specific training. Zero-shot generalization is promising not only for standard academic benchmarks, but perhaps more-so for creative, useful, and real-world queries that downstream users of language technologies are likely to make.
On the multimodal side, recent instraction-following vision-language models also provide zero-shot interface. Given an image (or multiple images) and a query (c.g., "how many apples are in this image?" "What is this?" "Write poem in the style of Robert Frost about this scene.") textual response is provided. Recent works like OpenFlamingo [11, 12], LLaVA [13] and others [14, 15, 16, 17, 18), have implemented this interface with promising initial results. Although standard benchmarks like VQAv2 [19] and COCO captioning [20] are commonly used to assess performance, less is known about how models perform broader, open-ended queries that resemble real-world user behavior. Evaluations of such queries typically rely on informal and qualitative approaches.
To support quantitative evaluation for this setting, we present VisIT-Bench (Visual Ins Tuscuuction Benchmark), a dynamic benchmark consisting of 592 challenging vision- laniuae instructions. Each instance contains an instruction, imput image(s), cestruction-condtionn caption (a human-craftee caption for the (anngc(s)iistruction) and a human verified reference (Figure 2). Instructions are image-contextual imperative requests questions, c.g., for image of pancakes, user asks "how can I cook this in a healthy way?". Different from existing zero-shot evaluations, many of the instructions focus on open-ended generation requests (e.g.. "write poem..." or "what should bring if were to visit here?").
We created VisIT-Bench to cover wide array of "instruction families". Our starting point was set of 70 "wish-list" tasks such as "home renovation" and "gardening tips" collected by the authors: each requiring varied high-level skills from recognition to complex reasoning (Figure 1). We derived 25/70 instruction families from benchmark tasks such as Visual Question Answering (VQA) [21] and robust change captioning [22] into chatbot-style format (this reformatting differs from prior work [14, 17, 13]. as we focus on onen-ended chatbot style responses.). Notably, 10 of these repurposed tasks involve multiple images.
We started with 10 images for each instruction family. Our annotators, guided by an example, create a new instruction, and provide (permissively licensed) image. For each instruction, we next collect instructioncconditionn
captions- - unlike prior work [23, 24] these descriptions are designed not only to describe the image in general, but also, surface information targeted to the instruction. Finally, captions to generate a reference candidate output from GPT-4; an additional human verification step discards GPT-4 references deemed to be incorrect.
we use
We conduct large-scale empirical comparison of multimodal cneructioo-follwiinn models using VisIT-Bench ($4). We first gather predictions for each instance from 7 candidate models. Then, we collect 5K human judgements of output quality by pitting model outputs read-to-head, and (in a forced-choice setup) crowd- crowd-sourcing pairwise preference judgements. This analysis not only reveals
Instruction
Instruction-Conditionnl Caption
Hluman-Verifiel GPT4 Response
Create a catchy title country song based upon the advice printed on the wheel cover.
There is a jeep parked on the street with top down. There wheel cover on spare tire in the back with an inscription that "Quit your Job, Buy Ticket, Get Tan, Fall in Love, Never Return."
"Tan Lines & Ticket Stubs Love Story Unbound" country song about leaving it all behind, finding love, and embracing the freedom of the open road.
Figure 2: An example from VisIT-Bench displays an image, instruction, an instruction-conniiinn
caption based on the instruction, GPT-4 suggested response, and a label confirming its accuracy. All 678 entries in VisIT-Bench have such labels, with 592 confirming accurate GPT-4 responses. These components aid in assessing multimodal chatbots and updating a dynamic leaderboard.

---

Table Comparison with related works for evaluating instruction-followigg languae models. Win-rates* refers to the model win-rates against reference output/model.
Muliilssiuutt
[w] InstrecdbLIPPLA
M9T[331 LVLA[[S4]
GAVIE 1351
VisII-Bench
Namber of Models
4
8
10
Namber of Skills Tosted
13
47
70
Multiplo-Images
X
Video
X
X
V
X
X
X
Multi Tum Conversations
/
/
Multilingual Conversations
Instruction conditioned Captions
X
X
Chathot-spp Responses
Datasc-specilee Evaluation
Human Evaluation
X
/
X
AttrGGPT-4 Evaluation
Win-rates
X
Elo Kating
X
significant differences between models (e.g., that LLa [13] is generally preferred Panda [18]). but also, that the human verified references in our corpus are preferred significantly more than the ones generated using multimodal models. We summarize head-to-head comparisons with two metrics: 1) Elo ratings [25, 26), which provide relative "skill" rating estimates encoding the probability that model A will be preferred model B; and 2) win rate versus our references, which provides an absolute metric. The best model according to human judgement is LLaMA-Adapter- v2 [16], yet it only wins in a pairwise setting against the reference 27.4% of cases.
Finally, we design an automated evaluation for VisIT-Bench, utilizing GPT-4 to rank pairs of model responses based on factors like correctness, relevance, and fluency. Using the instruction-conditonne caption and the instruction, GPT-4 determines the better response between two options, expediting iteration compared to human preferences. We explore reference-free and reference-backer versions of this metric. Compared to various metrics (BLEU-4 [27], ROUGE-L [28], METEOR [29], CIDEr [30], and BERTScore [31]) }), our evaluation aligns best with human preferences. For example, achieves 94% agreement rate in the cases where all five annotators agree. Figure 6 illustrates the process.
While is difficult a priori envision all possible scenarios under which more performant multimodal chatbots might be used, we hope VisIT-Bench can provide a path to improving vision-languee models "in the wild." Table 1 presents a summary of our contributions in comparison the recent works [32, 14, 17, 33, 34, 35] in the evaluation of multimodal chatbots. We publicly release VislT-Bench data, code, and automatic metrics https //visitvittttteeeeee wwwrisstteennhhh.. .ei..igthhn attuub.io//
2 VisIT-Bench: A World Inspired VL Instruction-Folloving Benchmark
VisIT-Bench was built to emulate real-world applications of multimodal models through image-text tasks, creating an extensive and practical benchmark. These tasks, or 'instruction families' are seen key capabilities of high-performing vision-ane vision-and-langeee lan-lang model. Although selections not exhaustive, they provide a broad basis for evaluating beyond academic benchmarks. We prioritize family coverage vs. number of instances-per-take task. The final corpus, comprising 678 instances and 1.159 public images, can be found at VislT-Bench Sheet Multi- Images. VisIT-Bench instances are either from 45 newly assembled instruction families or reformatted from 25 existing datasets (see Table 5). Notably, 10 instruction families cater to multi-image query scenarios (e.g., Figure 4).
2.1 Data Collection
The authors of this work perform an initial annotation step of curating instruction families. For each instruction family not derived from an existing task (45 out of 70), we designate a name for the family (e.g.. "Contextual Knowledge of Events") and identify an image-instruction pair that exemplifies the category, along with a sample response ("Martin Luther King Jr. is waving to acknowledge and greet the crowd of protesters [...]")) 10 sample familes are in Figure 1.
We work with crowdworkers at $18/hour to execute the annotation steps, as outlined Figure (1) taking the image/instruction example as guiding seed task crowdworkers formulate new instruction that examines the same instruction family ("instruction generation")) (2) crowdworkers create detailed image captions that describe the image and allow entity, relying solely on this

---

Instruction Generation
IntrructoonConditionnen Caption Generation
Model Output Evaluation
"Create a new, challenging Instruction inspired by fask that famil"
"Provide highly deseriptive cuption fext-only recjolent to follow the irstruction"
"Evaluste model predictions by answering the following questions...
Task
Contextual Knowlodge Events
Instruction
Thore and color of candy in the bowl that missing spoon. Whish is not represented in the spoon full of candies?
Instruction
am the player with the while chess pieces.
Family
Seed
Why he waving?
this turn capture one of my opponent's pieces?
Instruction
hssppeeed in this event?
Instruction- Conditioned
There a woodon choss board [..] The black rocks are at al and hü. The black knights c8 and
Image
Image
Caption
Image
Output
warving
to scknowledge and greet the
crowd of profesters who have gathered hear him speak [...]
GPT-4 Output
Yes, you can capture the black pawn at ed with your white pown e4.
New
Why guy relaing hand?
Annotated Caption
Brightly colored candles, with colors of red. orange, yallow, green, pink and dark spoan is raised above the bin of candy, bringing them in for close-up. spcon red, orange, yollow, green and dark brown candy it.
Output
correctly follow instruction?
Instruction
Eval
Yes [X] No
ananswered 'No' for the above, mark following:
Selected
purple
Image
N/A
dense
detailed enough,
the probiem GPT-4's enswer
J The dense caption underspeciffed, not
GP7-4's fault
Figure 3: Data collection steps: (1) Instruction Generation from a seed task (left). (2) Caption Generation creates rich intrructio-condiiionn captions for GPT-4 (middle). (3) Model Evaluation with human- -valdddd GPT-4 responses (right). Top: rater instructions; bottom: outputs.
text, interpret and execute the given instruction successfully ("instruction-conniitone caption generation"), (3) crowdworkers assess the correctness of GPT-4's response to the instruction ("model output evaluation"). We further elaborate on these steps using human annotators below.
Re-formatting existing datasets. 25/70 instruction families (corresponding to 25 10=250 in- stances) are re-formatted versions of existing vision-language tasks (See Appendix D for full list). This process involves re- re-formatting tasks into chatbot-style instruction/response versions. In re-formatting, we re-write instructions to retain the original task's goal while maintaining the original images, see Figure 4. These repurposed tasks are integrated into our data collection process, ensuring uniformity between the chatbot-style style answers the full VisIT-Bench instances and the reinterpreted tasks.
Instruction Generation. Here, annotators create new instance from the same instruction family as a given example, along with an instruction and corresponding image. For instance, in Figure (left), the instruction family is "Contextual Knowledge of Events", and the example instruction is "Why he waving? What happened in this event?" alongside an image Martin Luther King, Jr. To collect images, annotators were instructed use openverse for Creative Commons licened images.
Conictiioon-oniiionnnedee Caption Generation. Annotators are provided with the image and instruction, and tasked to construct caption that is rich enough to allow entity, solely receiving the text they author, to follow the instruction. These captions, termed instruction-con instructio-conditionee conditionnd captions, aid GPT-4 reference generation and text-only evaluation. See Figure (middle) for example: an annotator doesn't t just mention the skittles and a spoon, but, given the query regarding specific colors, they indicate the exact colors in detail.
Model Output Evaluation. The goal of this stage is to gather responses for each multimodal instruction query. We initially obtain response candidates from GPT-4 given the instruction and the instruction- instuution-coondiiioned caption. GPT4's 's prompt is: "Consider an image depicted by: <caption>' Now, briefly follow this instruction, and you can add short explanation: <instruction> Response: This prompt is employed for both single and multiple image instances, with appropriate modifications for the latter. Then we verify each response with human annotators. response is marked incorrect, the annotator identifies whether the issue lies with the detail level of the instrectio-cooditionn captions or with GPT-4's response itself. For VislT-Bench, we discard any case marked as incorrect for either reason. An example is given in Figure 3 (right), where GPT-4's
reference chatbot

---

The this task you will be provided with two individual images i.e., BEFORE and AFTER. Please study them carefully and determine the following sentence is correct (Answer "Yes"/"No"):
NLVR2
None of the dogs in these images wearing clothing.
small chibuahuas, one
standing and looking straight at camera. lifs wearing pink and white polka dotted dress, which same white teat Undor dag coment walloway, and there is grass to the dog's right and the rightt the
fur, and the other with light brows coot of for back but white for its small pillow that has large white wards praclaining, "BE NINE on it. The all white chiltuolua resting heed on laying
1
2
shirt
MINE
behiidd
No. The second image has dog wearing pink and white polka-dotted dress and pink shirt with white text on it.
GPT-4
Figure 4: An example multi-image task from VislT-Bench. sourced from NLVR2 [36], tests visual reasoning. While NLVR2 uses sentence, two images, and binary answer, we add zero-shot prompt, a instruction-conditione caption per image, and verified GPT-4 reply. This chatbot-style design aids automatic evaluation of future chatbot interactions.
candidate reference response aims to answer question about chess position (which it does so incorrectly, and thus, the instance discarded).
2.2 Data Collection Annotation and Results
We conduct the data collection steps Figure using Amazon's Mechanical Turk (MTurk) platform. Prior to annotating, each MTTrrk worker passed qualification test, which involved five to ten sample tasks designed to assess their ability to generate high-quality annotations. More detailed information about the execution process and full user interface examples can be found in Appendix C.
Annotation results are in Table 2. We assess our collection and filtra- tion efficiency. For single-image tasks, our pipeline's yield was 91.5% from the original candidate set. However, the success rate dropped 63.0% for multi-image tasks, accompanied by an uptick in issues either the captions (6.0% or GPT-4's responses (30.0%). This drop suggests that multi-image queries may pose a more difficult data collection challenge.
Table Human rating metrics for the VisIT-Bench dataset: overall, single-, and multi-image tasks.
Metrics
Overall
Single
Multi
GPT-4 Correct (%)
87.3
91.5
63,0
Problem in Caption (%)
4.0
3,6
6.0
Problem in GPT-4 (%)
7.7
3.8
30.0
3 VislT-Bench Analysis
We analyze the tasks, images, and instruction-cooditioone captions of VisIT-Bench.
Are instruction-com conditioon- captions necessary? To elucidate instruction-condiioe captions' role, we conduct an experiment 150 single-image instances. We replace our instruction-conditione captions with BLIP2 [15] captions, a leading image captioning model, and feed them to GPT-4 for chatbot response. See Figure for this process. We manually evaluate whether the resulting output captions yielded 91 % correct
accurately followed the instructions. We find results, but with BLIP2 captions, success dropped 31% (Table 2). This underscores the importance of instruction-condition in our dataset are sophisticated enough such that most are not solvable by using simple Socratic model [37] baseline of caption - LLM.
captions in the construction of VisIT-Bench, and shows that the instances
5

---

BLIP2 Caption
It net possible deternine disabled wheolshair- buuud individual would find easy ge into the store based en image sign on brick wall. Additional information about the store's entrance, accessibilitt features, needed make that assessment.
a sign on wall
6PT4
Instruction- Canioo--Codiiieeed Caption
Overall
Success:
This image of an exterior stare front and sidewalk, showing portable wedge ramps,
31%
green The ramps are positioncd against the step to two retail deorways, and soem bridge the vertical distance between the level of sidewalk and step up the doorway, The ramps are printed with "STOPGAP.O GA the top surface.
Yes, disabled aheelchair-bpune individual would likely find to store, as partable wedga ramps provide smooth transition from the sidewalk the doorway, making accessible them.
Would disabled wheolchair bound individual it easy store?
GP14
Overall
Sucsess:
99%
Figure 5: This experiment evaluates the value of instruction-conniitonne captions in accurate instraction-following tasks. Given an image and instruction, GPT-4 generates responses using both instruction-conditinee caption and less detailed BLIIP-2 [15] caption. The latter's impreci- sion leads to an error, emphasizing the need for detailed, task-specific captions.
What skills are required for VisIT-Bench? The full list of instruction families we cover are in Appendix Table 6. Following [38], for the VisIT-Bench instructions, we extract the most frequent root verbs and their direct nouns (a full plot is in Figure 10). The most common include: 'answer question', 'write story/poem', 'create title'. etc. There's also long-tail of diverse requests that demand comprehension, commonsense, and cross-mes modal understanding, e.g.. 'identifying objects' to 'need ingredient' to "connect device'. Additional examination reveals a range of underlying skills required ranging from 'emotion identification' to complex reasoning tasks such 'paper folding'.
What is contained in VisIT-Bench images? We detect all the COCO [20] objects present in the images from our dataset using Yolov5-L [39]; The most common detected objects in VislT-Bench are "person" (~ 900 detections), chair, and car (~ 100). But, long tail of rarer objects exists well: full distribution in Appendix Figure 9. Overall, to perform well at VislT-Bench, a model must account for broad range of scenes and objects.
4 Experiments
We evaluate range of state-of-the-art publicly accessible vision-and-language chatbots on the 592 instances in BesT.BBenc. In $4.1, we provide the details of the iistruction-oollwwee models in our benchmark. Following this, we collect the human preferences for pairwise model generations to achieve a comannguudded Elo ranking and the win-rates against the reference of the models in $4.2. We then develop automatic evaluation on VislT-Bench in $4.3, that can be scaled and improved given new and improved models. Finally, we establish the trustworthiness of our automatic evaluation method by performing agreement analysis with the human judgments in $4.3
PandaGPT
Instnccias- Concitioned Caption A cave man a headband is holding
The unusual aspezt this image is that appears scenes images The main frcus as cartoon drawing a mas halding sign that says 'ART"" Hawover. there are alsa elements a kitchen sush as a refrigarater. micrewave, and a clack which de not seem le be rebitd to the "ART" theme This andiapssitinn an of unrelated subjects in the same image creales pazullar and uneapecled siaual effect.
combination of two different
An unssual aspect of this image the man's appearancs He has hang a heard, and searing liinclutt, which is ancient form of clotting worn by man some cultures. Additionall, he is holding a slack money his hand which is modem day itens. This combination of ancient and modern elements creates inferesting centraet and raises guestions about the contact and purpese the image.
InstractBLIP
GPT4
money bill that has the face of cave
InstructRUP
What unusual about this image?
Figure 6: ELO-based evaluation for VisIT-Bench: Our reference-free approach uses a GPT4 evaluator compare two coaruction-oollwinng models with an instruction and instruction-condae conditioned caption. The instance is obtained from an existing dataset, WHOOPS! [40].

---

Table Human scores for models are displayed ELO ratings and win-rates against the reference, summarizing 5.0K pairwise judgments. The 'matches' column shows each model's 's participation count, and wwin-rate vs. reference" denotes its win rate against reference outputs.
Model
Elo
matches
Win-rate vs. reference (w/ ratings)
Single Image
Human Verified GPT-4 Reference
1223
14399
LLaVA (13B)
1085
1462
26.23% (n=244) 27.41 % (n=259)
LlamaAdapier--2 (7B)
1061
1507
mPLUG-Owl (7B)
995
1345
14.95% (n=214)
InstructBIIIP (13B)
957
1315
12.37% (n=194)
MiniGPT-4 (7B)
893
1513
14.72% (n=
PandaGPT (13B)
786
1441
10.48% (n=229)
Multiple Images
Human Verified GPT-4 Reference mPIJUGOwl
1193
210
997
190
15.38% (n=78)
Otter OpenFlamingo
917
147
3.17% (n=63)
893
4.35% (n=69)
4.1 Models
We evaluate VLa [13], InstructB1IP--15. 13B [14], MiniGPT4-7B [41], mPLUU-OWL-- [17), LlamaAdapter-v22777 [16]. PandaGPT-13B [18], VisualChatGPT [42]. Multimodal GPT [43], Open- Flamingo v1 [11, 44] and Otter v1 [45]. For the execution-based VisualChatGPT [42], we implement a chat window for each sample, hold inputs and intermediate chains of thoughts and actions in memory, and feed the images and the instruction sequentially. For OpenFlamingo [11] and Otter [45], we feed the image(s) and the instruction in an interleaved format. For the others, we feed the image to the vision feature extractor and feed the instruction a prompt to the text encoder.
4.2 Human Evaluation
We collect 5K pairwise human preference judgements across an initial set of 6 models and the human-verified references. For 1K uniformly randomly sampled tuples of (query, model A, model B), we collect 5 crowdworker judgements each. Preferences are collected a "forced choice" setting, annotators are instructed to decide based accuracy, helpfulness, and detail. We provide the template for the human annotation process in Appendix Figure 15. We summarize the results with two metrics:
Elo We follow [26] and compute Elo ratings, treating each pairwise human judgement as a "match."The difference between the Elo ratings of two different models provides an estimate for the win probability when pitting model A vs. model B. More details are in Appendix E.
Relative metric:
Absolute metric: Win rate vs. reference. We provide a win-rate vs. the human- verified reference. We use the 1.4K pairwise human judgments where one of A or B is the reference. We report the percent of cases where the human judge prefers the output from that model vs. the human-verifier GPT-4 reference output. Because we do not allow for ties in our forced-choice setup, if the annotator believes the responses are of equal quaity, they choose one arbitrarily.
Results Table contains the Elo and win-rate vs. reference. In terms of Elo, the Human Verified GPT-4 reference achieves a higher rating than all alternatives, validating the quality of our reference set: concretely, for our Elo settings, the reference (Elo =1223) has an estimated win-rate over one of the best performing models, LLaVA, (Elo 10055) of 69%, and an estimated win rate of 93% against the lowest performing model in this setup. PandaGPT (Elo =786). This result can partly be explained by the training process of the underlying models: The improved performance of LLaVA (13B) might be attributed to its fine-tuning process, which utilized 150K instruction- tuninng data that is rich in both diversity and quality. Interestingly, despite achieving slightly lower Elo (the computation of which is based on all head-to-head "matches", rather than just ones against the human reference), Llama v2 (7B) wins with the highest rate against the reference. However, the complexity of models and tasks in ViscT-Bench makes it challenging to definitively pinpoint the factors influencing performance. We conduct an initial exploration of this result in Section 4.3.
7

---

Table 4: As of July 19th, 2023, reference-free Elo rankings summarize 12K matches between models,
each with 2 GPT-4 queries. With the dynamic VislT-Bench rankings update as more models join the leaderboard and more head-to-head evaluations oceur.
Model
Elo
matches
Win vs. Reference (w/ # ratings)
Single Image
Human Verified GPT-4 Reference
1370
5442
LLaVA (13B)
1106
5446
17.81 % (n=494)
IlamaAdapter--2 (7B)
1082
5445
13.75% (n=502)
mPLUG-OWl (7B)
1081
5452
15.29% (n=497)
InstructBLIP (13B)
1011
5444
13.73% (n=517)
Otter v1 (9B)
991
5450
6.84% (n=512)
VisualGPT (Da Vinci 003)
972
5445
1.52% (n=527)
MiniGPT-4 (7B)
921
5442
3.26% (n=522)
OpenFlamingo v1 (9B)
877
$449
2.86% (m=524)
PandaGPT (13B)
826
5441
2.63% (n=533)
Multimodal GPT
763
5450
0.18% (n=544)
Multiple Images
Human Verified GPT-4 Reference
1192
180
mPLUG-Owl
995
6.67% (n=60)
Otter vl
911
180
1.69% (n=59)
OpenFlamingo vl
902
180
1.67% (n=60)
4.3
Automatic Evaluation and Leaderboard
Because it is costly to gather human pairwise preference judgements for new model submissions, to support faster model development, we seek an automatic evaluation procedure that produces high correlation with our human evaluation setup.
Automatic evaluation metric candidates. We consider several existing seerence--bccke evaluation metries: BLEU-4 [27] ROUGE- I. [28]. METEOR [29], CIDEr [30], and BERTScore [31], we use the RoBERTa-Large
nore
% Agres with Human (Higher-bette)
98
GPT-4el
BERTS
upper bound
METEOR
english version [46], treating the human-verified GPT-4 reference as the evaluation reference. We additionally report two baseline metrics: ran- dom, which assigns a random score without ac- counting for the candidate, and length, which assigns a score equal to the number of non- whitespace tokens in the candidate. Beyond existing metrics and baselines, following the re- cent line of work utilizing APl-accessed L.MSs with a prompt for automatic evaluation [6, 47). consider two GPT-4! [7] backed evaluation metrics.
ROUGE
BLEU
Lenghh
72
CIDE
58
random
5/5 2e-3300
315
(n=200)
(n4104)
Agraament
Figure 7: Correlations between evaluation metrics and human preferences ranked by performance, with our reference free evaluation (GPT-4-no-ref) showing the strongest alignment. Bottom: random chance (50%), top: upper performance bound.
Specifically, we provide the LLM with: 1) system prompt describing the desired evaluation behavior; 2) the instruction condetiionn-niiiiineeededee caption for the image; 3) the instruction to be followed;
and 4) two candidate generations dubbed "Response A" and "Response B". We also consider reference-backen backed version where the human-verified reference is provided as well. We provide our prompts in Appendix F. To mitigate potential biases in "A" and "B" positioning, for all pairs of candidates, we run two queries covering both possible orderings. Our prompt encourages the model to think step-by-step that its chain-of-though process is made explicit [48, 49]. Despite strongly encouraging the model to select between the two references in a forced-choice setup, it sometimes
'OpenAl [7] hosts several API versions of GPT-4 and updates them over time, we use the versions they host interchangeably (specifically, our evaluations mix their models named: apt-4-0314 (which became depreciated during the development of this work) and gpt-4 (which underwent update during experiments).

---

From-the-wild
Existing datasets
25
reructiuu (::00
LLakk (328)
Win rate vs GPT-4
netu Ow (3R)
Otter
15
History foneeli
"Bccht cat Sucoii
Cestunes Understanding Figurative Speeeh eeeee Astronomy
Home Renovation
Thatii Signt
Catchy Tilles
Phisica Knowledge
Human Emetion
Who to cail
Ruscor
Chemical Courting
Game Playin
Write a Story
Gardening Exerces
Events knowledge
Animali
Rachion products
Art anowledde
Directiont
Dressing canddia
Location Medical
Geomnety Problems
Climate unders undertain staddinn
Caltural Knowledlp
Food abled
Architectural Styles
Pop culture
Capadty Gues Guesrtimate
Hazedd dentificaaion
Tour maskeiil
DCR Wath
Paper folding
Anagra Aiagrrms
Board Games
Write Poem
vicwic
toltssps
oloqa
ankoop
nowwork
kilogr
whoops
hammol_mmmee sciencega
memees
sokvege
clour general
hammoll
House
Figure 8: Reference-fie assesment win rate vs. duman-verified GPT4 response for each instruction category. Axes: win rate (Y), instruction categories (X). Categories are from-the- sool-dhe-wild or existing datasets. VislT-Bench facilitates analysis of diverse instruction tuning tasks.
refuses and outputs "tie" which we account for later. We call the reference- free version of this metric "GPT4-no-ref", and the piteeence-backer version of this metric "GPT4-ref".
Evaluating evaluation metrics. We measure the correlation between the candidate metrics and human judgements using a pairwise framework. Specifically, we use a subset of the 5K pairwise human judgements in $ 4.2. For 690 pairwise instances where both candidate instances are model- generated (rather than human-verified references), we have 5 pairwise judgements from crowd- workers. For 336 pairs, there is 5/5 agreement, for 200 pairs, there is 4/5 agreement, and for 154 pairs, there is 3/5 agreement. For each metrie, we measure the percent of time the metric able to accurately reconstruct a majority vote judgement from the 5 crowdworkers. The newly proposed GPT-4 based metrics sometimes outputs "tie" (this happens in 10-15% of cases overall) for fair comparison with the other metrics in forced choice setting, we randomly choose one of the two options when GPT-4 reports tie.
The results are in Figure with GPT-4-no-ref best aligns with human correlation. The best perform- ing metric our newly proposed GPT-4 based metric, which accurately reconstructs majority-vote pairwise human judgments better than alternatives (p < .05; binomial proportion CI nonover- lapping). For example, for instances where 5/5 annotators agree, GPT4-no-ref, with refer- ence, accurately reconstructs human judgment 93% of the time, whereas the next best metrics DERTSOrC/METTOORRUGGE
reconstruct accurately 80%/78%/70% of the time; length base- line metric achieves only 60%. Notably, the reference-backed version of the newly proposed GPT-4 based metric achieves comparable (but slightly worse) performance compared the reference-free version. Thus, we adopt the reference- frre version, which additionally enables us to place the references themselves into the the Elo setup, because they are not used in the prompts.
System-level Correlation. We summarize the LLM's pairwise judgements using the same metrics as introduced in $4.2, Elo ratings and win rate vs. reference, but instead of using human judge, our reference-free GPT-4 based metric. The results are in Table 4. Notably, among the 7 systems for which we gathered human ratings for, the automatic metric produces the same ordering compared to human evaluation (p = 1.0, p < .01).
Shortcomings of proposed metric. While the relative ranking of models produced by the automatic metric correlates strongly with the ranking produced by human judgements, the win rate vs. reference according human judgement (Table 3) higher overall compared to the win-rate according to the automatic metric Table 4. One plausible explanation for this discrepancy is that GPT-4, evaluation model, may prefer responses that closely match its own response distribution.
reference
Per-category results. In Figure 8, we plot the win-rate vs reference for the models across all the single-image instruction families. We find that there is model that performs the best and

---

worst across all the instruction families. Thus. VisIT-Bench aids in highlighting the strengths and weaknesses of the instruction-folle following models along various real-world use-cases.
Related Work
Our work builds on prior multimodal image-text models and -ollowinn benchmarks in machine learning. We provide detailed overview of related work in ŞB. Multi-model modals for image-text understanding have recently emerged as powerful and useful methods for many image-language reasoning tasks [12, 15, 13, 50, 18. 14, 17, 11, 7]. Both language and multimodal models are often trained to follow language instruction, paradigm known as "instruction following" existing vision-language datasets
[5, 16, 51, 14, 13, 17]. Despite the success of these approaches (GQA, Image Captioning [21, 52, 20]), there is no quality benchmarking dataset for multimodal cnrtuction-following tasks that reliably replicates the way in which humans would interact with multimodal chatbots in the wild. The absence of benchmarking data impedes reliable progress assessments [53] and limits empirical evaluations of multimodal LLMMs[5)..
Conclusion
We present VisIT-Bench, benchmark assessing multimodal chatbot skills. Going beyond prior efforts, VisIT-Bench's collection process centers potential real-world use cases, and 70 diverse instruction families encompassing a range of tasks from recognition to complex reasoning. Besides human-verified outputs, it features an Elo ranking aligning with human judgments, Our data reveals performance gap between models and humans. Releasing data, code, and metries, we aim for community engagement and believe VisIT-Bende will quantify progress and gaps in multimodal Al.
7 Limitations
Although VisIT-Bench covers wide spectrum of potential use-cases, it does not incorporate every possible vision-language task. We hope to add more categories of tasks over time. In terms of dialogue, VisIT-Bench concentrates single-turn instances with one instruction and response. This does not encompass multi-turn interactions between users and chatbots, which presents a direction for future research. Our study focuses image-text modalities. Future extensions could expand the scope to include other modalities like audio and video, enabling a more comprehensive evaluation. Additionally, while the dataset offers wide variety of tasks, larger number of examples per category could provide more depth. Finally, while our GPT-4 based metric correlates well with human judgement at instance and system level, has a stronger preference for GPT-4 based generations compared to humans. Thus, models which train, e.g., by distilling from GPT-4 outputs, may have unfair advantage evaluation.
evidence that the GPT-4 based metric
References
[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant laboratory for alignment. arXiv preprint arXiv:2112.0088 2021.
[2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[3] Jason Wei, Maarten Bosma, Vincent Y Zhao. Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0165) 2021.
[4] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Suppernnaturalinstrattiisss Generalization via declarative instructions 1600+ nlp tasks. In
10

---

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 5085-5109, 2022.
[5] Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. https:// g gigtthu..com cww/totsu tatttttttuuuaaaa lab/stanf ord alpaca, 2023.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL
https://lmsys... http:////sssssssoooo.ooereeeeeee 200l/lg/22220000 100--000300-vvaaa.-
[7] OpenAl. Gpt-4 technical report. arXiv, 2023.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
[10] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne I Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397) 2023.
[11] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, March 2023. URL https: //doi.org/11 5281/zemodo. 7733589.
[12] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204. 14198, 2022.
[13] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[14] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arv preprint arXiv:2305.0650 2023.
[15] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 1225977711..977 2023.
[16] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adaaper v2: Parameter-efficient visual instruction model. arXiv preprint 23Xiv:2304.15014 2023.
[17] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304 14178. 2023.
[18) Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-folllw them all. arXiv preprint arXiv:2305. 16355, 2023.
[19] Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the V in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904- 6913, 2017.
11

---

[20] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[21] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Ziticck, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425- 2015.
[22] Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4624 4633, 2019.
[23] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Sorieut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 647-664. Springer, 2020.
[24] Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:221 1.09699, 2022.
[25] Arpad E Elo. The proposed usef rating system. its development, theory, and applications. Chess Life, (2())242-247, 1967.
[26] Lianmin Zheng, Ying Sheng, Wei-Lin Chiang, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: Benchmarking ilms in the wild with elo ratings. 2023. URL https: / //tmsys.org//log//// bttg/blog//033-/2///gg 05- 11--003-rree aren//
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
[28] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
[29] Satanjeev Banerjee and Alon Lavie. METEOR: automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop on Evaluation Measures for MT and Summarization, 2005.
[30] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015.
[31] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinherger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In ICLR, 2020.
[32] Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint arXiv:2212.10773 2022.
[33] Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387 2023.
[34] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng. Siyuan Huang, Yu Qiao, and Ping Luo. Lvim-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023.
[35] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565. 2023.
[36] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded photographs. arXiv preprint arXiv: 111::811000099 1.00491, 2018.
12

---

[37] Andy Zeng, Maria Attarian, Brian lchter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.0059% 2022.
[38] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions, arXiv preprint arXiv:2212. 10560, 2022.
[39] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[40] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! vision-and-language benchmark of synthetic and compositional images. arXiv preprint arXiv:2303.07274, 2023.
[41] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing issionllanguge understanding with advanced large language models. arXiv preprint arXiv:2304. 11tt22204..092,, 2023.
[42] Chenfei Wu, Shengming Yin, Weizhen Qi. Xiaodong Wang. Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04677 2023.
[43] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.047900 2023.
[44] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01300 2023.
[45] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. arXiv preprint arXiv: 2023,
[46] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[47] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback, 2023.
[48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thouue prompting elicits reasoning in large language models. In NeurIPS, 2022. URL https: / /rxiv. org/ ab////////// abs///as/222.../ 11903.
[49] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022. URL
2205. 11916.
[50] Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Cross-modal attention congruence regularization for language relation alignment. arXiv preprint arXiv:2212. 120400999 2022.
[51] Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://balie berkeley http/////o////////3.333
[52] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Proceedings of the IERE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
13

---

[53] Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? meta review of evaluation failures across machine learning. In Thirty-fftth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[54] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.1335 2023.
[55] Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. arXiv preprint NxXiv:2203.08244 2022.
[56] Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint arXiv:2305. 2023.
[57] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.88402 2022.
[58] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277, 2023.
[59] Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. Dynosaur: A dynamic growth paradigm for instruction-tuingg data curation. arXiv preprint arXiv:2305.1433 14327, 2023.
[60] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11 1206, 2023.
[61] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[62] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695 2022.
[63] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 1381, 2023.
[64] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08 1,88, 12800 2023.
[65] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953-14962. 14962, 2023.
[66] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Kei-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: slug-and-plat compositional reasoning with large language models, arXiv preprint arXiv:2304.09842 2023.
[67] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans, 2023.
[68] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint mxXv:2302.140045 2023.
[69] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
14

---

[70] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[71] Pranav Rajpurkar, Robin Jia, and Perey Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint 100i:::80060082222 2018.
[72] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[73] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.074611 2018.
[74] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
[75] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept, 2021.
[76] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09 100110,, 2022.
[77] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distri- bution shift question answering models. In International Conference on Machine Learning, pages 69166 PMLR, 2020.
[78] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar- 10? arXiv preprint arXiv: 1801.800600111 2018.
[79] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[80] Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich- Keil, Moritz Hardt, John Miller, and Ludwig Schmidt. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems, 32, 2019.
[81] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6720-6731, ,,,,-6731,2111
[82] Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. Broaden the vision: Geo-diverse visual commonsense reasoning. arXiv preprint arXiv:2 (et.:21199.0880 2021.
[83] Jack Hessel, Ana Marasović, Jena D Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh electric sheep? humor" understanding" benchmarks from the new yorker caption contest. arXiv preprint arXiv:2209.0629 2022.
[84] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017
[85] Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D Hawkins, and Yoav Artzi. Abstract visual reasoning with tangram shapes. arXiv preprint arXiv:2211.16492. 2022.
[86] Shivam Sharma, Siddhant Agarwal, Tharun Suresh, Preslav Nakov, Md Shad Akhtar, and Tanmoy Charkraborty. What do you meme? generating explanations for visual semantic role labelling in memes. arXiv preprint 1xXiv:2212.00713 2022.
15

---

[87] Pan Lu, Swaroop Mishra, Tony Xia. Liang Qiu, Kai- Wei Chang, Song- Cong-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS). 2022.
[88] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok- A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[89] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022.
[90] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real- time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333-342, 2010.
[91] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 742- 758. Springer, 2020.
[92] Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. arXiv preprint 22X.::2203.15867. 2022.
[93] Harsh Jhamtani and Taylor Kirn-Kirkpatrick. Learning to describe differences between pairs of similar images. arXiv preprint 18Xiv:1808.100 10584, 2018.
[94]
Yonatan Bitton, Ron Yosef, Eli Strugo, Dafna Shahaf, Roy Schwartz, and Gabriel Stanovsky, Vasr: Visual analogies of situation recognition. arXiv preprint arXiv:2212.0454 2022.
[95] Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. Winogavil: Gamified association benchmark to challenge lision-and-language models. Advances in Neural Information Processing Systems, 35:26549- 26564, 2022.
[96] Ron Yosef, Yonatan Bitton, and Dafna Shahaf. Irfl: Image recognition of figurative language. arXiv preprint 130.::233.11444 2023.
[97] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia. Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint 1rXiv:2110.13214, 2021.
[98] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01566 2023.
16

---

Appendix
A License and Intended Use
The VisIT-Bench dataset, along with its various contributions such as instructions, reference outputs, and model ranking annotations, is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license applies to all the images we have directly contributed, each of which carries a public license specification in the "public images metadata" field within the dataset sheets. However, the dataset also incorporates images sourced from pre-existing collections. For these images, the original licensing terms are respected and remain applicable.
VisIT-Bench's primary purpose is to function as dynamic benchmark that continuously evolves and evaluates instruction-followin vision-language models. In the current landscape, commercial chatbots are often trained on non-disclosed and non-public datasets, which raises concerns about potential data contamination and inadvertent training on our evaluation data [8]. This risk is further highlighted by recent studies [55, 56]. To mitigate such concerns, we have chosen to withhold the complete VislT-Bench test set from public disclosure, while still making the images and instructions available for direct download. Researchers, however, can utilize VisIT-Bench to its full potential as dynamic benchmark by submitting their model predictions for evaluation. We will assess their models using the undisclosed test set, ensuring the ongoing evolution of the benchmark. Moreover, we are open to releasing the test data upon receiving reasonable and justified requests, particularly when additional analysis is necessary, provided that requesters agree to our non-contamination policy which prohibits the use of this data for training commercial chatbots. This approach strikes a balance between the need for robust model evaluation and the mitigation of potential data contamination.
A Dataset Analysis
10a
Count (log scale)
10A
R3300al geve
tran
refrigrator
carret
keyboard
couch
tolee
stop sign
moteroycic
surfboard
scissors
girafte
knife
bicycle
horee
boot
traffic light
sprom
truck
baseball bat
apple
sultcaae
bench
tv
crange
call phrons
umbreii
dag
sports ball
petted plant
dining table
backpartk
varce
rermote
wiie glass
handing
clock
cake
bowl
tie
book
run
bottle
car
cheir
person
Objects
Figure 9: Count of the various COCO objects present the VisIT-Bench images on log-scale. The object detection was performed by Yolov5-Large [39]. We observe that the object 'person' occurs most of the time the dataset images i.e., 875 times.
17

---

httle
word
question
poem
ereate
doserile
story
write
have
resii
denuisy
epinn
make
artist
need
Dse
obsec
colo
Hape
materiaal
device
volu
apesiion
poem
sta
tela
bau
mola
molor
da
su
Figure 10: Most frequently occurring verbs (inner circle) and their top 4 direct nouns (outer circle) in the TislT-Bench instructions.
B Related Work
Multimodal Models for Image-Test Text Understanding: Recently, the field of machine learning has experienced rapid proliferation of new models which can perform various image-text tasks [12, 15, 13, 50, 18, 14]. This growth has been driven by several factors, including the emergence of large-scale multimodal datasets (e.g. LAION-5B [57], Multimodal C4 [11]), improved software and hardware frameworks, and advances in modality-specife models such as language models (e.g.. [10].. Our work specifically evaluates models which can generate textual outputs. given
more images, and text. Recent examples of such models include LLLVA [13], mPLUG-Owl [17], InstructBL.IP LLaMA-Adapter, Flamingo [12] and OpenFlamingo
[11], PandaGPT [18], and GPT-4 [7] (which reports multimodal capabilities but has not yet seen a release of the multimodal variant).
Instruction Following: "Instruction-follovingg is emerging paradigm for training models via language, where instead of being trained to complete only a single, fixed task (such as image classification captioning), models trained to follow textual instructions that describe arbitrary task, with the aim of generalizing to novel instructions. Examples of wow.etiionnooooiinggg models include Alpaca [5], ALLaMA-Adaple [16], Koala [51], InstructBLIP [14], LLaVA [13], and mPLUG-oWl [17]. As the downstream capabilities of these models are influenced by the quality of the training dataset, there has also been extensive work on developing instruction-followin datasets [38, 58, 59, 13, 60].
To build powerful these models, two broad approaches have been shown to be effective. One approach focuses on leveraging existing pretrained task-specific tools such as image captioners [15], object detectors [61] and text-to-image generators [62] by either creating multimodal prompt interfaces [42, 63] or by executing LLM- L.M--geeeatted programs [64, 65, 66]. The other approach [13, 16, 67, 45, 68, 17, 11] focuses building single pretrained model that can follow instructions by supervised finetuning on multimodal vision-language data.
Despite the success of both these approaches the existing vision-language datasets e.g., VQA, GQA, Image Captioning [21, 52, 20], there is a lack of a high-quality benchmarking dataset for multimodal instruction-followiin interact with multimodal chatbots in the wild. Similar to the image-text models discussed above, many instruction-folllee peer review or thorough evaluation. As a result, the effectiveness of these models for many tasks is not well-understood.
tasks that reliably replicates the way in which humans would
models have been released directly open-source without undergoing
Benchmarks for Machine Learning: High-quality evaluation datasets have served both to (re)assess, and to accelerate, progress many machine learning tasks [53]. For example, our work draws particularly from the fields of computer vision and natural language processing, where benchmarking datascts have been critical drivers of progress. On the vision side, datasets such ImageNet [69] and
18

---

CIFAR [70] have proven to be critical yardsticks of progress. On the language side, benchmarks such as SQuAD [71]. SST [72], GLUE/SuperGLUE [73, 74] and more [75. 76] seen wide use. Recent work has indicated that improvements on these high-quality benchmark datasets is not the result of overfitting, and a reliable indicator of genuine progress beyond the benchmark data [77, 78, 79, 80).
However, high-quality benchmarking datasets and evaluation methods do not yet exist for multimodal cestruction-folloiing As result, difficult to assess progress in this direction, which both reduces the field's ability to identify true breakthroughs and increases vulnerability to potential pitfalls of evaluation that have hampered progress in other areas of machine learning [53, 54].
C Interfaces for Collecting Human Annotations
In this section, provide the templates used to collect human annotations for the instruction generation (Figure II), the dense caption generation (Figure 12), model verification (Figure 13 and Figure 14), and the model rating (Figure 15).
Expected Output
Instruction
appears Massaman Cur in large pot, add onion and saut, av curry paste, ginger and garlc. Saute chicken and potatoes are cooked throu minutes. Serve with (white, brow top.
What is this dish and how can I make it?
Find an Image to Annotate to test Food Recipe, and create a Visual instruction, that textual-only model/human won't
Foel free skip the HIT if you don't feel creative with this one
LCCESSE*** image URL.
0/200
Provide **creative and challenging** model instruction that would blow your mind if an Al model could answer it.
Previca
Showing
Next
Figure 11: An interface that collects the image and corresponding instruction.
Instrcction (Tacted ScllCCerrce
Seard cemmecee sbout hode se wotth homer whice opettor trads compettton
Magg lesua dess atae iici 94 for accutton wot-
Figure 12: An interface that collects the dense caption given an image-and and-instruction pair.
19

---

GPT4 inserction?
Dense Caption:
Jenan weiiing and one sinng athethae witi danue and unc the that ciher drmalled
cho
anowered NNo' for the above, mark one of following:
Instructions
aggit Deea
cetailed probicm GPT4's arswer
GPT4 Prediction:
The de pin is undersees fiees not GPTM's foutt
Anca no vosi vioci vusttal
offerannw
Figure 13: A sample of the model verification where the GPT4 follows the instruction correctly.
Coss GFTT4 sarnecly falica seruc
Dene Costere
(x)s Nxtt -1 F(() x.3
Tenevean Nor oen, the Scttwingg
GPiCC wer
stough
C1 1 of swe desne prblenusk
Figure 14: A sample of the model verification where the GPT4 failed to follow the instruction due to its incorrect reasoning.
instructoon What person doing looking the lase
The ceamining the posably sying fgure purpose how might
The person is standing front a laser with their bact to the comera. They are looking at the boee, poesitly
vmwensing or examining festume.
instruction, factuel, helpful athering the instruction debals?
Model
Model
Figure 15: An interface that collects the feedback of the model rating.
20

---

D Existing Datasets incorporated in VisIT-Bench
In Table 5, we listed the existing datasets that are incoprated in our VisIT-Bench. Among these datasets, 15 contain single image in each sample pair, and 10 require reasoning hased multiple images.
Table 5: List of existing datasets in VisIT-Bench categorized as single and multiple image datasets.
Dataset
Topic
VQA [21]
Visual Question Answering
VCR [81]
Cognition-level Visual Understanding
GD-VCR [82]
Geo-Diverse Commonsense Reasoning
WHOOPS [40]
What Makes this Image Strange
Newyork Caption [83]
Humor Understanding
CLEVR [84]
Visual Question Answering
Single
Kilogram [85]
Tangrams Identification
Harmful Memes [86]
Memes Understanding
ScienceQA [87]
Science Question Answering
OK-VQA [88]
Outside Knowledge Visual Question Answering
AOK-VQA [89]
Outside Knowledge Visual Question
AOK-VQA [89]
Question Generation
VizWiz [90]
Visual Question Answering
GQA [52]
Visual Question Answering on Scene Graphs
TextCaps [91]
Visual Question Answering Texts
Robust Change Captioning [22]
Describing What has Change in a Scene
NLVR2 [36]
Testing Visual Language Bias
ImageCoDE [92]
Image Retrieval
Spot-the-Diff [93]
Identifying Differences
VASR [94]
Visual Analogies
Multiple
WinoGavil [95]
Visual Associations
IRFL (Metaphor) [96]
Figurative Speech Understanding
IRFL (Idioms) [96]
Figurative Speech Understanding
IconQA [97]
Abstract Diagram Understanding
Fick-a-Pic|P9
Text-to-Image User Preferences
Table 6: List of skills and existing datasets in VisIT-Bench
"scienceqa", "ocr math', 'recognition', "okvqa', "house plan understanding', 'nlvr2', 'gardening tips', 'textcaps', 'architectural styles', 'dressing sense". 'winoground', 'oood recipe', 'paper folding', 'whoops', "spot the diff', "wino- gavil', 'imagecode', 'exercise', 'art knowledge', 'gqa', 'physical knowledge', 'contextual knowledge of events', 'home renovation', 'aokvqa', 'animals', 'vasr', 'counting', 'board games', 'solving geometry problems', 'who to call?', 'clevr', 'building materials', 'hazard identification', 'pickapick', 'astronomy', 'figurative speech explanation', 'write story', 'gestures understanding', 'newyork', 'cul- tural knowledge', 'aokvqg', 'traffic sign identification', 'pop culture', 'fashion products', 'harmful memes', 'write a poem', 'vizwiz', 'guesstimate of capacity', 'location understanding', 'graph reasoning', 'vqa', 'game playing', 'differently abled', 'chemical identification', 'history knowledge', 'climate and weather understanding', 'irfl metaphor', 'human emotion recognition', 'medical', 'gd ver', 'ver', 'technical support', 'catchy titles', 'kilogram', 'anagrams', 'color', 'tour guide', 'directions', 'irfi idiom', 'ree'
21

---

E
Elo Rating
For many years, the Elo rating has been popular in ranking players zero-sum games such chess [25]. Recently, it has been adopted to rate large language models (LLMs) against each other on the user instructions. In this work, we adopt the same strategy to rank set of mnstuctiionfollooinng vision- models, that can grow dynamically with further advances in the field.
Given multimodal chatbots C, and C. with their absolute Elo rating R. and Rs. respectively. Simply put, the probability of C, winning over C. in a head-to-head battle is given by:
P(Ca wins over Cb) = 1 + 10(R.-RR1/4 R21/400
(1)
In practice, calculating the Elo rating requires us to set hyperparameters to decide the weightage for each win and loss head-to-head battle between models. In our work, use the open implementation of Elo for LLMs by FastChat at https://gi htt//ggtttb.././////////t www.cmmmmmmmaeeee Ltt////ysy//a/aaatttteeeaa mainn/fastchaa/eeeeree//mottttt http:///onttt/ooooooaaaaassaaaaa py.
F
GPT-4 Pairwise Evaluation Prompts
The specific prompts we use extract pairwise judgements from our language model are provided in Table 16 (reference-free version) and Table 17 (reference-backe) version). When applied to GPT-4 [7], these prompts usually solicit definitive pairwise response by the model. But, in some cases, the model either produces a pairwise judgement in unexpected format, or, refuses to issue judgement at all. For cases like these, we issue additional query to ChatGPT to extract an answer (or decide there is no answer) using an additional prompt, given in Table 18. If after this step there still no definitive pairwise judgment, we call the result a tie.
head-to-head comparison involves querying GPT-4 two times. Each query has approximately 800 input/250 output tokens (output includes chain of thought; see Supplementary for full prompt/response example). Running a head-to-hhee head comparison between new model and randomly sampled existing model across the 700 instances is usually sufficient for stable Elo estimate. In total, at current GPT-4 prices, the 700 head-to-head comparisons required to assess a new model costs 50-70 dollars.
22

---

syatem proupt (human authored)
You ImageTask Evannnntti expert langunge model judging whether response adequately addreesses instruction in the context of an image. More specifically, you will be given the following:
An image context: This will describe the contents of an image with sufficient detail to address the instruction.
instraction: This is question, imperative request, something similar about the image which requires response.
3. Two responses, response A and response B: These two responses attempt to address the instruction in the context of the image.
Your job is to judge whether response response better. A and B randomly ondered.
Some things remember:
Even though you accurale.
language model, image description will be sufficienlly detailed that your judgements be
You cupable of judging response quality, accounting for important factors like correctness, relevance, fluency, specificity, You think step siep-by-step bt-step, but ultimately respond with 'Respunse A" or "Response B
user authored)
I will deseribe an image to you, and provide an instruction. Then, will give you two candidate responses that address the instruction the context of the image: these will he lahelled "Response A' and "Response B'. Your job is first step-hy-ste step about which response is best in terms of accuracy, specificity. fluency, etc. After reasoning stop-by-stop and comparing the prosicons of each response, the end, respond with "Overall, Response X is better." where either B.
asaistant: (human authored)
Sure, please provide the image cuntext, the instruction, and the two candidute respanses, Response A and Respoose B. Then, I will think step-by-step and provide my ultimate judgement to which response is better.
user (from human- authored template)
OK. Here is the image description, the instruction, and the two response options. Response A and Response B. Image context: skier wearing green jacket, burgundy pants, white shocs, white gloves, and gray beanie. There are people sitting the buckground with their snow boards....
Instruction: Answer the following question bused on the provided image: Are the trousers the same color us the shoes?
Response A: no
Response B: No, the trousers are the same color the shoes. The trousers are burgundy, while the shoes are white.
Think step-by-stop and finish your response with "Overall, Response is better." where X cither or B.
assistant (machine generated; used derive prodiction)
Step 1: Identify the key information from the image description.
The skier wearing:
Rurgundy pants
White sbocs
Step 2: Assess the accuracy of cach response in addressing the instruction.
Both Response A and Response B convey that the trousers and shoes are not of the same color.
Step 3: Compare specificity and detail of each response.
Response A is concise and provides straightforward answer, hut lacks detail.
Response provides a clear, comprehensive answer includes the color both the trousers and shoes.
Step 4; Consider the fluency and correctness.
responses are fluent grammutically correct.
Overall, Response better.
Figure 16: The prompt we for reference-free evaluation, along with sample instance/completion from GPT-4 [7].
23

---

system prompt (human authored)
an expert language model at judging whether or not response adequately addresses an instruction in the context of image. More specifically, you will he given the following:
You are ImageTaskEvahuttions
An image context: This will describe the contents of image with sufficient detail to address the instruction.
2. An instruction: This is a question, an imparative roquest, or something similar about the image which requires a response.
A reference oulput: This high high-quality example output that humans have judyed he accurate response for the input instruction.
4. Two responses, response A and response B: These two responses attempt to address the instruction in the context of the image.
Your job is to judge whether response response better. A and B randomly ordered. Some things to remembher: Even though you are just a language model, the image description will be sufficienlly detailed so that your judgements can be accurate.
Take the high quality reference into account when making your judgerments, but remember: some instuctions are more open ended
than others, for those cases, high quality response can differ from the reference.
You are capable of julginn respone quality, accouming for important factors like curreciness, relevance, fluency, specificity, elc. in You think step-by-step. but ultimately respond with 'Response A" or 'Response B"
user (human authored)
I will describe image you, and provide instruction. Then, I will provide reference output which is example of high quality outpur for that instruction in the context of the image. Then, I will give you two candidate responses that address the instruction the context of the image: these will be labelled 'Response A" and "Response B". Your job is first reason step by step ahout which response is best in terms of accuracy, specificity, Bluency, elc. After rensoning slep by step and comparing the prosicons of each response, the end, respond with "Overall, Response X is hetter." where X is either or B.
asaistant: (human authored)
Sure. please provide the image context, the instruction, the reference, and the two candidate responsex, Response A and Response B. Then, will think step-by-step and provide my ultimate judgement to which response better.
user (from human- authored template)
OK. Here is the image description, the instruction, the hugh-quulity reference, and the two response options, Response A and Response B. Image context: 'hhee is a white plate that has on it a small pizza with many lopics, mostly unrecognizable, due to the pie being very well done, charred to the point of almost being burnt, and the crust blackened in many places.
Instruction: Someone told the creator of this item that they are very talented. What is the implied meaning? ?
High quality reference: The implied meaning that the person complimenting the creator is being sarcastic, as the pizza overcooked and almost burot. which is not sign of talent.
When someone says that they are "very talented" the context of the image. it implics that the person responsible for creating the pizza skilled in the craft and able to create an appealing and tasty dish. The person praising the pizza's is creator recognizes their culinary skills, which can be seen through the combination of ingredients and the overall look of the finished dish. The praise forn of approciation for their efforts and the enjoyable result they achieved.
Response A:
Response B: is difficult to desermine the implied meaning without further context information about the image. It appears be a pizza that has been burnt or charred on the surfaee and has various toppings, including pepperoni, onion, and mushrooms. The image is shown on white plate or surface. The implied meaning of this image could be that it is dish that has heen cooked badly that it is dish with a unique or unusual combination of ingredients.
Think stop-by-stop and finish your response with "Overall, Response is better." where X cither A or B.
sesistant (machine generated; used to derive the prodiction)
Step I: Comparing both responses with the high-quality reference, it's clear that Response A descrihes the pizza as well made and appealing, while Response more focused on the fact that burnt or charred, somewhat aligning it with the reference. Step 2: Evaluating specificity, Response A defines the creator as skilled, even though the image description contradicts this by stating that the pizza overcooked. Response B, while not as specifiic acknowledges the overcooked nature of the pizza.
Step 3: In terms of fluency, both responses are easily understood and written in coherent language.
Step 4: Considering accuracy. Response A is inaocurate relation to the context and instruction because implics a positive meaning when it should he sarcastic. Response B doesn't fully capture the sarcasm hut is closer in the truth about the pizza being budly cuoked.
Step 5: Understanding relevance, Response A is not entirely relevant due to the inaccuracy in the creator's 's talent, while Response B. although less direct, more relevant by mentioning the burni or churred nature of the pizza. Overall, Response is hetter.
Figure 17: The prompt we use for reference-backed evaluation, along with a sample in- stance/completion from GPT-4 [7].
24

---

system prompt (human authored)
an expert language model at extracting multiple choice answers from writlen out from longer explanations, You will he several sentences describing thought process which should eventually contain final answer, either A or B. Your job is to extract the final answer that the explanation arrives at.
You are Tinal//nserFSttrattooooo
Some things to remember:
Keep your unswers short: only oulpul "Final Amswer: where A or If the explanation doesn't mention linal choice explicitly, you ouiput "Unknown"
in-context examples os (human authored))
Example One: GPT-4 gives definitive answer, but in the wrong format
user (GPT-4's raw pairwise judgement)
Step Compare the responses the instruction
- The instruction asks people are roquired to stop and roquests an oxplanation.
Stop 2: Assess Response A
Response indicates that people should stop is pressed against the bus, and the LED lights are off.
presence stop sign. However, doesn't consider fact that sign
Step 3: Assess Response Ressonse simply says "yes," acknowledging that people should stop, but it doesn't 't provide the explanation required the instruction.
Step 4: Compare specificity, fluency, and other factoes
Response A is specific and fluent but misconstrues the image description, Response B is accurate in that ledgesledges the stop sign but lacks an explamation.
Overall, since neither response is fully correct, difficult to declare a better response. However, given the fact that Response acknowledges prominent stop sign, I would say that Response B slightly better, should also explain how the situation indicates that people should stap. Please extract the linal
from the above text.
assistant (machine generated; used derive the prediction)
Final Answer: Response B slightly better, but should also explain how situation indicates that people should stop.
Example GPT-4 to answer.
user raw pairwise judgementt
Step Consider the image context, which states that the original thirteen colonies are shown New Hampshire specifically highlighted in darker green.
Step 2:
Assess the accuracy of Response A, which claims that the calony shown is "Virginia." " This is not accurate since the image context indicates New Hampshire being highlighted.
Step Assess accuracy of Response which claims the colony shown Massachusetts Bay Colony. This response also not accurate, as the image context states New Hampshire is the highlighted colony.
Step 4: In terms of accurncy, both responses are incorrect. Neither response is consistent with the image context.
Step 5: Consider other factors like fluency and specificity. Both responses seem fluent and specific in nature.
fter considering these factors, none of the responses accurate given the information provided in the image context. However, I am unable provide 'Overall, Response X better." judgement case both responses are incurrect neither out as heing better than the other.
Please extract the final answer from the above text.
assistant (mnachine generated; used to derive the prediction)
Final Unknown.
Figure 18: The prompt we use to extract answers from ill-formatted GPT-4 [7] responses, along with two example outputs. In one case, GPT-4's pairwise judgement is given, but in format different than requested. In the other case, GPT-4 "defies" the prompt, and refuses to issue a judgement because both options are bad.
25


Reviews:
[{"review_id": "72a8C3kMUk", "reviewer": "Reviewer_Fmx9", "summary": "", "questions": "", "limitations": "See above.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper incorporates an Elo-based ranking system to establish a leaderboard for multimodal chatbots. This system, utilizing GPT-4 to provide pairwise preferences based on instructions and candidate outputs, enables the evaluation of multimodal models in an automated manner.\n2. The instances in VisIT-Bench are categorized into specific \"skills,\" which better captures the ability of models to follow instructions of various types.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3DkIOaP1Q5", "reviewer": "Reviewer_ywz9", "summary": "", "questions": "", "limitations": "None", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It provides both automatic and Elo-based human evaluation for evaluating the instruction-following ability of multimodal large language models. The benchmark covers diverse real-life vision-language instructions across 70 tasks.\n2. The way to add detailed dense caption and verified reference outputs using language-only GPT4 model for evaluation is good and proves to be more effective.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xXY3FR7Yzq", "reviewer": "Reviewer_AvKi", "summary": "", "questions": "", "limitations": "see the above\n\n", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "S1: VisIT-Bench introduces an open and unique benchmark dataset for evaluating instruction-following vision-language models, providing challenging and diverse test examples that represent human intents in multimodal chatbots.\n\nS2: Through an empirical study, publicly available model outputs are generated, accompanied by in-depth analysis of dataset diversity and assessment of model strengths and weaknesses.\n\nS3: An automatic evaluation method utilizing human and automatic rankings based on Elo rating is employed, establishing trustworthiness through instance-level agreement analysis with human preferences.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "AmmFqLgyND", "reviewer": "Reviewer_UGFt", "summary": "", "questions": "", "limitations": "This paper provides a limitation section to discuss the potential limitations of this work. ", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A good research problem! It's very important to quantify the actual progress on vision-language modeling, given that most papers nowadays only provide qualitative examples. \n\n2. A benchmark dataset has been constructed, consisting of hundreds of examples that can evaluate the instruction-following ability of vision-language models. The benchmark construction process is reasonable, including several concrete steps, like asking human annotators to provide dense captions for future evaluation. \n\n3. Comprehensive evaluation is conducted for measuring various vision-language models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts with mixed scores. AC feels that this work makes valuable and timely contributions to benchmark the performances of VLM models. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper.
Decision: Accept (Poster)