SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yonatan Bitton\*1,2 Hritik Bansal\*3 Jack Hessel\*4 Rulin Shao5 Wanrong Zhu6 Anas Awadalla5 Josh Gardner5 Rohan Taori7 Ludwig Schimdt4,5,8
We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluating instruction-following vision-language models for real-world use. Our starting point is curating 70 "instruction families" that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just  $27\%$  of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at https://visit-bench.github.io/.
Reasoning Over
How much is the browser usage for Firefox and Safari?
Art Knowledge
Teach me about this painting.
Recognition
Where is this?
Location
Understanding
If you are going for a picnic at this location, what items should you carry with you?
Home Renovation
Here is a photo of my bathroom. How can I design it nicer?
Contextual
Knowledge of Even
Tell me what is notable or important about the event in this photo.
Explanation
Someone said that this man is an angel. Why?
Figurative Speech
Chemical identification
Which chemical compound does this image represent?
Hazard Identification
If you are driving and come across this scenario, what should you do?
Figure 1: A sample from the 70 instruction families in VisIT-Bench representing tasks we envision instruction-following vision-language models should be able to follow.
Game Playing
What is the poker hand shown in the picture? Is this a good hand?
$^{*}$ Equal contribution.  $^{1}$ Hebrew University  $^{2}$ Google Research  $^{3}$ UCLA  $^{4}$ Allen Institute for AI  $^{5}$ University of Washington  $^{6}$ UCSB  $^{7}$ Stanford  $^{8}$ LAION,
A long-standing challenge for artificial intelligence is to build general-purpose assistants that can, in collaboration with humans, solve diverse and never-before-seen tasks [1]. For textual tasks, several recent works [2, 3, 4, 5, 6, 7] have shown that fine-tuning language models such as GPT-3 and LLaMA with supervised instruction+response examples [8, 9, 10] enables them to respond to imperative requests and questions without task-specific training. Zero-shot generalization is promising not only for standard academic benchmarks, but – perhaps more-so – for creative, useful, and real-world queries that downstream users of language technologies are likely to make.
On the multimodal side, recent instruction-following vision-language models also provide a zero-shot interface. Given an image (or multiple images) and a query (e.g., "how many apples are in this image?") or "What is this?" or "Write a poem in the style of Robert Frost about this scene.") a textual response is provided. Recent works like OpenFlamingo [11, 12], LLaVA [13] and others [14, 15, 16, 17, 18], have implemented this interface with promising initial results. Although standard benchmarks like VQAv2 [19] and COCO captioning [20] are commonly used to assess performance, less is known about how models perform on broader, open-ended queries that resemble real-world user behavior. Evaluations of such queries typically rely on informal and qualitative approaches.
To support quantitative evaluation for this setting, we present VisIT-Bench (Visual InsTruction Benchmark), a dynamic benchmark consisting of 592 challenging vision-language instructions. Each instance contains an instruction, input image(s), a instruction-conditioned caption (a human-crafted caption for the image(s)/instruction), and a human verified reference (Figure 2). Instructions are image-contextual imperative requests or questions, e.g., for an image of pancakes, a user asks "how can I cook this in a healthy way?". Different from existing zero-shot evaluations, many of the instructions focus on open-ended generation requests (e.g., "write a poem..." or "what should I bring if I were to visit here?").
We created VisIT-Bench to cover a wide array of "instruction families". Our starting point was a set of 70 "wish-list" tasks such as "home renovation" and "gardening tips" collected by the authors: each requiring varied high-level skills from recognition to complex reasoning (Figure 1). We derived 25/70 instruction families from benchmark tasks such as Visual Question Answering (VQA) [21] and robust change captioning [22] into a chatbot-style format (this reformatting differs from prior work [14, 17, 13], as we focus on open-ended chatbot style responses.). Notably, 10 of these repurposed tasks involve multiple images.
We started with 10 images for each instruction family. Our annotators, guided by an example, create a new instruction, and provide a (permissively licensed) image. For each instruction, we next collect instruction-conditioned captions – unlike prior work [23, 24] these descriptions are designed not only to describe the image in general, but also, surface information targeted to the instruction. Finally, we use instruction-conditioned captions to generate a reference candidate output from GPT-4; an additional human verification step discards GPT-4 references deemed to be incorrect.
We conduct a large-scale empirical comparison of multimodal instruction-following models using VisIT-Bench (§4). We first gather predictions for each instance from 7 candidate models. Then, we collect 5K human judgements of output quality by pitting model outputs head-to-head, and (in a forced-choice setup) crowd-sourcing pairwise preference judgements. This analysis not only reveals
Instruction
Instruction-Conditioned Caption
Human-Verified GPT4 Response
Figure 2: An example from VisIT-Bench displays an image, instruction, an instruction-conditioned caption based on the instruction, a GPT-4 suggested response, and a label confirming its accuracy. All 678 entries in VisIT-Bench have such labels, with 592 confirming accurate GPT-4 responses. These components aid in assessing multimodal chatbots and updating a dynamic leaderboard.
Create a catchy title for a country song based upon the advice printed on the wheel cover.
There is a jeep parked on the street with the top down. There is a wheel cover on the spare tire in the back with an inscription that says, "Quit your Job, Buy a Ticket, Get a Tan, Fall in Love, Never Return."
"Tan Lines & Ticket Stubs: A Love Story Unbound" - A country song about leaving it all behind, finding love, and embracing the freedom of the open road.
Table 1: Comparison with related works for evaluating instruction-following vision-language models. Win-rates* refers to the model win-rates against a reference output/model.
MultiInstruct [32]	Owl [17]	InstructBLIP [14]	M3IT [33]	LVLM [34]	GAVIE [35]	VisIT-Bench
Number of Models	1	5	3	4	8	5	10
Number of Skills Tested	9	6	13	13	47	16	70
Multiple-Images	X	✓	X	X	X	X	✓
Video	X	X	✓	✓	X	X	X
Multi-Turn Conversations	✓	✓	✓	✓	✓	X	X
Multilingual Conversations	X	✓	X	✓	X	X	X
Instruction-conditioned Captions	X	X	X	X	X	X	✓
Chatbot-style Responses	X	X	X	X	X	X	✓
Dataset-specific Evaluation	✓	✓	✓	✓	✓	X	X
Human Evaluation	X	✓	X	X	✓	X	✓
Auto/GPT-4 Evaluation	X	✓	X	✓	X	✓	✓
Win-rates*	X	✓	X	✓	X	✓	✓
Elo Rating	X	X	X	X	✓	X	✓
significant differences between models (e.g., that LLaVA-13b [13] is generally preferred to Panda [18]), but also, that the human verified references in our corpus are preferred significantly more than the ones generated using multimodal models. We summarize head-to-head comparisons with two metrics: 1) Elo ratings [25, 26], which provide relative "skill" rating estimates encoding the probability that model A will be preferred to model B; and 2) win rate versus our references, which provides an absolute metric. The best model according to human judgement is LLaMA-Adapterv2 [16], yet it only wins in a pairwise setting against the reference in  $27.4\%$  of cases.
Finally, we design an automated evaluation for VisIT-Bench, utilizing GPT-4 to rank pairs of model responses based on factors like correctness, relevance, and fluency. Using the instruction-conditioned caption and the instruction, GPT-4 determines the better response between two options, expediting iteration compared to human preferences. We explore reference-free and reference-backed versions of this metric. Compared to various metrics (BLEU-4 [27], ROUGE-L [28], METEOR [29], CIDEr [30], and BERTScore [31]), our evaluation aligns best with human preferences. For example, it achieves a  $94\%$  agreement rate in the cases where all five annotators agree. Figure 6 illustrates the process.
While it is difficult to a priori envision all possible scenarios under which more performant multimodal chatbots might be used, we hope VisIT-Bench can provide a path to improving vision-language models "in the wild." Table 1 presents a summary of our contributions in comparison to the recent works [32, 14, 17, 33, 34, 35] in the evaluation of multimodal chatbots. We publicly release VisIT-Bench data, code, and automatic metrics in https://visit-bench.github.io/.
2 VisIT-Bench: A Real-World Inspired VL Instruction-Following Benchmark
VisIT-Bench was built to emulate real-world applications of multimodal models through image-text tasks, creating an extensive and practical benchmark. These tasks, or 'instruction families', are seen as key capabilities of a high-performing vision-and-language model. Although our selections are not exhaustive, they provide a broad basis for evaluating beyond academic benchmarks. We prioritize family coverage vs. number of instances-per-task. The final corpus, comprising 678 instances and 1,159 public images, can be found at VisIT-Bench Sheet Multi-Images. VisIT-Bench instances are either from 45 newly assembled instruction families or reformatted from 25 existing datasets (see Table 5). Notably, 10 instruction families cater to multi-image query scenarios (e.g., Figure 4).
2.1 Data Collection
The authors of this work perform an initial annotation step of curating instruction families. For each instruction family not derived from an existing task (45 out of 70), we designate a name for the family (e.g., "Contextual Knowledge of Events") and identify an image-instruction pair that exemplifies the category, along with a sample response ("Martin Luther King Jr. is waving to acknowledge and greet the crowd of protesters [...]"). 10 sample families are in Figure 1.
We work with crowdworkers at $18/hour to execute the annotation steps, as outlined in Figure 3: (1) taking the image/instruction example as a guiding seed task crowdworkers formulate a new instruction that examines the same instruction family ("instruction generation"); (2) crowdworkers create detailed image captions that describe the image and allow an entity, relying solely on this
Instruction Generation "Create a new, challenging instruction inspired by the seed task for that family"
Instruction-Conditioned Caption Generation "Provide a highly descriptive caption to allow a text-only recipient to follow the instruction"
Model Output Evaluation "Evaluate model predictions by answering the following questions...".
Figure 3: Data collection steps: (1) Instruction Generation from a seed task (left). (2) Caption Generation creates rich instruction-conditioned captions for GPT-4 (middle). (3) Model Evaluation with human-validated GPT-4 responses (right). Top: rater instructions; bottom: outputs.
text, to interpret and execute the given instruction successfully ("instruction-conditioned caption generation"); (3) crowdworkers assess the correctness of GPT-4's response to the instruction ("model output evaluation"). We further elaborate on these steps using human annotators below.
Re-formatting existing datasets. 25/70 instruction families (corresponding to  $25^{*}10 = 250$  instances) are re-formatted versions of existing vision-language tasks (See Appendix D for full list). This process involves re-formating tasks into chatbot-style instruction/response versions. In re-formatting, we re-write instructions to retain the original task's goal while maintaining the original images, see Figure 4. These repurposed tasks are integrated into our data collection process, ensuring uniformity between the chatbot-style answers in the full VisIT-Bench instances and the reinterpreted tasks.
Instruction Generation. Here, annotators create a new instance from the same instruction family as a given example, along with an instruction and corresponding image. For instance, in Figure 3 (left), the instruction family is "Contextual Knowledge of Events", and the example instruction is "Why is he waving? What happened in this event?" alongside an image of Martin Luther King, Jr. To collect images, annotators were instructed to use openverse for Creative Commons licensed images.
Instruction-Conditioned Caption Generation. Annotators are provided with the image and instruction, and are tasked to construct a caption that is rich enough to allow an entity, solely receiving the text they author, to follow the instruction. These captions, termed instruction-conditioned captions, aid GPT-4 reference generation and text-only evaluation. See Figure 3 (middle) for an example: an annotator doesn't just mention the skittles and a spoon, but, given the query regarding specific colors, they indicate the exact colors in detail.
Model Output Evaluation. The goal of this stage is to gather human-validated reference chatbot responses for each multimodal instruction query. We initially obtain response candidates from GPT-4 given the instruction and the instruction-conditioned caption. GPT4's prompt is: "Consider an image depicted by: <caption>'. Now, briefly follow this instruction, and you can add a short explanation: <instruction>. Response: This prompt is employed for both single and multiple image instances, with appropriate modifications for the latter. Then we verify each response with human annotators. If a response is marked incorrect, the annotator identifies whether the issue lies with the detail level of the instruction-conditioned captions or with GPT-4's response itself. For VisIT-Bench, we discard any case marked as incorrect for either reason. An example is given in Figure 3 (right), where GPT-4's
NLLVR2
The In this task you will be provided with two individual images i.e., BEFORE and AFTER. Please study them carefully and determine if the following sentence is correct (Answer "Yes"/"No"):
Two small chihuahua, one with all white fur, and the other with a light brown coat of fur on its back, but white fur on its head and neck, are near a small pink pillow that has large white words proclaiming, 'BE MINE' on it. The all white chihuahua is resting its head on the pillow, while the other one is laying behind it.
A small all white chihuahua is standing and looking straight at the camera. It's wearing a pink and white polka-dotted dress, over which is a pink shirt with some white text on it. Under the dog is a cement walkway, and there is grass to the dog's right and to the right of the walkway.
Figure 4: An example multi-image task from VisIT-Bench, sourced from NLVR2 [36], tests visual reasoning. While NLVR2 uses a sentence, two images, and a binary answer, we add a zero-shot prompt, a instruction-conditioned caption per image, and a verified GPT-4 reply. This chatbot-style design aids automatic evaluation of future chatbot interactions.
No. The second image has a dog wearing a pink and white polka-dotted dress and a pink shirt with white text on it.
candidate reference response aims to answer a question about a chess position (which it does so incorrectly, and thus, the instance is discarded).
2.2 Data Collection Annotation and Results
We conduct the data collection steps in Figure 3 using Amazon's Mechanical Turk (MTurk) platform. Prior to annotating, each MTurk worker passed a qualification test, which involved five to ten sample tasks designed to assess their ability to generate high-quality annotations. More detailed information about the execution process and full user interface examples can be found in Appendix C.
Annotation results are in Table 2. We assess our collection and filtration efficiency. For single-image tasks, our pipeline's yield was  $91.5\%$  from the original candidate set. However, the success rate dropped to  $63.0\%$  for multi-image tasks, accompanied by an uptick in issues either in the captions  $(6.0\%)$  or GPT-4's responses  $(30.0\%)$ . This
Table 2: Human rating metrics for the VisIT-Bench dataset: overall, single-, and multi-image tasks.
Metrics	Overall	Single	Multi
GPT-4 Correct (%)	87.3	91.5	63.0
Problem in Caption (%)	4.0	3.6	6.0
Problem in GPT-4 (%)	7.7	3.8	30.0
drop suggests that multi-image queries may pose a more difficult data collection challenge.
3 VisIT-Bench Analysis
We analyze the tasks, images, and instruction-conditioned captions of VisIT-Bench.
Are instruction-conditioned captions necessary? To elucidate instruction-conditioned captions' role, we conduct an experiment on 150 single-image instances. We replace our instruction-conditioned captions with BLIP2 [15] captions, a leading image captioning model, and feed them to GPT-4 for a chatbot response. See Figure 5 for this process. We manually evaluate whether the resulting output accurately followed the instructions. We find instruction-conditioned captions yielded  $91\%$  correct results, but with BLIP2 captions, success dropped to  $31\%$  (Table 2). This underscores the importance of instruction-conditioned captions in the construction of VisIT-Bench, and shows that the instances in our dataset are sophisticated enough such that most are not solvable by using a simple Socratic model [37] baseline of caption  $\rightarrow$  LLM.
Figure 5: This experiment evaluates the value of instruction-conditioned captions in accurate instruction-following tasks. Given an image and instruction, GPT-4 generates responses using both a instruction-conditioned caption and a less detailed BLIP-2 [15] caption. The latter's imprecision leads to an error, emphasizing the need for detailed, task-specific captions.
What skills are required for VisIT-Bench? The full list of instruction families we cover are in Appendix Table 6. Following [38], for the VisIT-Bench instructions, we extract the most frequent root verbs and their direct nouns (a full plot is in Figure 10). The most common include: 'answer question', 'write story/phon', 'create title', etc. There's also a long-tail of diverse requests that demand comprehension, commonsense, and cross-modal understanding, e.g., 'identifying objects' to 'need ingredient' to 'connect device'. Additional examination reveals a range of underlying skills required ranging from 'emotion identification' to complex reasoning tasks such as 'paper folding'.
What is contained in VisIT-Bench images? We detect all the COCO [20] objects present in the images from our dataset using Yolov5-L [39]; The most common detected objects in VisIT-Bench are "person" ( $\sim$  900 detections), chair, and car ( $\sim$  100). But, a long tail of rarer objects exists as well: full distribution in Appendix Figure 9. Overall, to perform well at VisIT-Bench, a model must account for a broad range of scenes and objects.
We evaluate a range of state-of-the-art publicly accessible vision-and-language chatbots on the 592 instances in VisIT-Bench. In §4.1, we provide the details of the instruction-following models in our benchmark. Following this, we collect the human preferences for pairwise model generations to achieve a human-guided Elo ranking and the win-rates against the reference of the models in §4.2. We then develop automatic evaluation on VisIT-Bench in §4.3, that can be scaled and improved given new and improved models. Finally, we establish the trustworthiness of our automatic evaluation method by performing agreement analysis with the human judgments in §4.3
Figure 6: ELO-based evaluation for VisIT-Bench: Our reference-free approach uses a GPT4 evaluator to compare two instruction-following models with an instruction and a instruction-conditioned caption. The instance is obtained from an existing dataset, WHOOPS! [40].
Table 3: Human scores for models are displayed as ELO ratings and win-rates against the reference, summarizing 5.0K pairwise judgments. The 'matches' column shows each model's participation count, and 'win-rate vs. reference' denotes its win rate against reference outputs.
Model	Elo	matches	Win-rate vs. reference (w/# ratings)
Single Image	Human Verified GPT-4 Reference	1223	1439	-
LLaVA (13B)	1085	1462	26.23% (n=244)
LlamaAdapter-v2 (7B)	1061	1507	27.41% (n=259)
mPLUG-Owl (7B)	995	1345	14.95% (n=214)
InstructBLIP (13B)	957	1315	12.37% (n=194)
MiniGPT-4 (7B)	893	1513	14.72% (n=299)
PandaGPT (13B)	786	1441	10.48% (n=229)
Multiple Images	Human Verified GPT-4 Reference	1193	210	-
mPLUG-Owl	997	190	15.38% (n=78)
Otter v1	917	147	3.17% (n=63)
OpenFlamingo v1	893	171	4.35% (n=69)
We evaluate LLaVA-13B [13], InstructBLIP-13B [14], MiniGPT4-7B [41], mPLUG-Owl-7B [17], LlamaAdapter-v2-7B [16], PandaGPT-13B [18], VisualChatGPT [42], Multimodal GPT [43], OpenFlamingo v1 [11, 44] and Otter v1 [45]. For the execution-based VisualChatGPT [42], we implement a chat window for each sample, hold inputs and intermediate chains of thoughts and actions in memory, and feed the images and the instruction sequentially. For OpenFlamingo [11] and Otter [45], we feed the image(s) and the instruction in an interleaved format. For the others, we feed the image to the vision feature extractor and feed the instruction as a prompt to the text encoder.
4.2 Human Evaluation
We collect 5K pairwise human preference judgements across an initial set of 6 models and the human-verified references. For 1K uniformly randomly sampled tuples of (query, model A, model B), we collect 5 crowdworker judgements each. Preferences are collected in a "forced choice" setting, annotators are instructed to decide based on accuracy, helpfulness, and detail. We provide the template for the human annotation process in Appendix Figure 15. We summarize the results with two metrics:
Relative metric: Elo We follow [26] and compute Elo ratings, treating each pairwise human judgement as a "match." The difference between the Elo ratings of two different models provides an estimate for the win probability when pitting model A vs. model B. More details are in Appendix E.
Absolute metric: Win rate vs. reference. We provide a win-rate vs. the human-verified reference. We use the 1.4K pairwise human judgments where one of A or B is the reference. We report the percent of cases where the human judge prefers the output from that model vs. the human-verified GPT-4 reference output. Because we do not allow for ties in our forced-choice setup, if the annotator believes the responses are of equal quality, they choose one arbitrarily.