SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results Table 3 contains the Elo and win-rate vs. reference. In terms of Elo, the Human Verified GPT-4 reference achieves a higher rating than all alternatives, validating the quality of our reference set: concretely, for our Elo settings, the reference (Elo =1223) has an estimated win-rate over one of the best performing models, LLaVA, (Elo =1085) of  $69\%$ , and an estimated win rate of  $93\%$  against the lowest performing model in this setup, PandaGPT (Elo =786). This result can partly be explained by the training process of the underlying models: The improved performance of LLaVA (13B) might be attributed to its fine-tuning process, which utilized 150K instruction-tuning data that is rich in both diversity and quality. Interestingly, despite achieving a slightly lower Elo (the computation of which is based on all head-to-head “matches”, rather than just ones against the human reference), LlamaAdapter-v2 (7B) wins with the highest rate against the reference. However, the complexity of models and tasks in VisIT-Bench makes it challenging to definitively pinpoint the factors influencing performance. We conduct an initial exploration of this result in Section 4.3.
Table 4: As of July 19th, 2023, reference-free Elo rankings summarize 12K matches between models, each with 2 GPT-4 queries. With the dynamic VisIT-Bench, rankings update as more models join the leaderboard and more head-to-head evaluations occur.
Model	Elo	matches	Win vs. Reference (w/# ratings)
Single Image	Human Verified GPT-4 Reference	1370	5442	-
LLaVA (13B)	1106	5446	17.81% (n=494)
LlamaAdapter-v2 (7B)	1082	5445	13.75% (n=502)
mPLUG-Owl (7B)	1081	5452	15.29% (n=497)
InstructBLIP (13B)	1011	5444	13.73% (n=517)
Otter v1 (9B)	991	5450	6.84% (n=512)
VisualGPT (Da Vinci 003)	972	5445	1.52% (n=527)
MiniGPT-4 (7B)	921	5442	3.26% (n=522)
OpenFlamingo v1 (9B)	877	5449	2.86% (n=524)
PandaGPT (13B)	826	5441	2.63% (n=533)
Multimodal GPT	763	5450	0.18% (n=544)
Multiple Images	Human Verified GPT-4 Reference	1192	180	-
mPLUG-Owl	995	180	6.67% (n=60)
Otter v1	911	180	1.69% (n=59)
OpenFlamingo v1	902	180	1.67% (n=60)
4.3 Automatic Evaluation and Leaderboard
Because it is costly to gather human pairwise preference judgements for new model submissions, to support faster model development, we seek an automatic evaluation procedure that produces high correlation with our human evaluation setup.
Automatic evaluation metric candidates. We consider several existing reference-backed evaluation metrics: BLEU-4 [27], ROUGE-L [28], METEOR [29], CIDEr [30], and BERTScore [31], we use the RoBERTa-Large english version [46], treating the human-verified GPT-4 reference as the evaluation reference. We additionally report two baseline metrics: random, which assigns a random score without accounting for the candidate, and length, which assigns a score equal to the number of non-whitespace tokens in the candidate. Beyond existing metrics and baselines, following the recent line of work utilizing API-accessed LLMs with a prompt for automatic evaluation [6, 47], we consider two GPT-4[7] backed evaluation metrics.
Specifically, we provide the LLM with: 1) a system prompt describing the desired evaluation behavior; 2) the instruction-conditioned caption for the image; 3) the instruction to be followed;
and 4) two candidate generations dubbed "Response A" and "Response B". We also consider a reference-backed version where the human-verified reference is provided as well. We provide our prompts in Appendix F. To mitigate potential biases in "A" and "B" positioning, for all pairs of candidates, we run two queries covering both possible orderings. Our prompt encourages the model to think step-by-step so that its chain-of-thought process is made explicit [48, 49]. Despite strongly encouraging the model to select between the two references in a forced-choice setup, it sometimes
Figure 7: Correlations between evaluation metrics and human preferences ranked by performance, with our reference free evaluation (GPT-4-no-ref) showing the strongest alignment. Bottom: random chance  $(50\%)$ , top: upper performance bound.
<sup>1</sup>OpenAI [7] hosts several API versions of GPT-4 and updates them over time, we use the versions they host interchangeably (specifically, our evaluations mix their models named: gpt-4-0314 (which became depreciated during the development of this work) and gpt-4 (which underwent an update during our experiments).
Figure 8: Reference-free assessment win rate vs. human-verified GPT4 response for each instruction category. Axes: win rate (Y), instruction categories (X). Categories are from-the-wild or existing datasets. VisIT-Bench facilitates analysis of diverse instruction tuning tasks.
refuses and outputs "tie" which we account for later. We call the reference-free version of this metric "GPT4-no-ref", and the reference-backed version of this metric "GPT4-ref".
Evaluating evaluation metrics. We measure the correlation between the candidate metrics and human judgements using a pairwise framework. Specifically, we use a subset of the 5K pairwise human judgements in § 4.2. For 690 pairwise instances where both candidate instances are model-generated (rather than human-verified references), we have 5 pairwise judgements from crowdworkers. For 336 pairs, there is 5/5 agreement, for 200 pairs, there is 4/5 agreement, and for 154 pairs, there is 3/5 agreement. For each metric, we measure the percent of time the metric is able to accurately reconstruct a majority vote judgement from the 5 crowdworkers. The newly proposed GPT-4 based metrics sometimes outputs "tie" (this happens in  $10 - 15\%$  of cases overall) – for fair comparison with the other metrics in forced choice setting, we randomly choose one of the two options when GPT-4 reports a tie.
The results are in Figure 7, with GPT-4-no-ref best aligns with human correlation. The best performing metric is our newly proposed GPT-4 based metric, which accurately reconstructs majority-vote pairwise human judgments better than alternatives ( $p < .05$ ; binomial proportion CI nonoverlapping). For example, for instances where 5/5 annotators agree, GPT4-no-ref, with no reference, accurately reconstructs human judgment  $93\%$  of the time, whereas the next best metrics BERTScore/Meteor/ROUGE-L reconstruct accurately  $80\%/78\%/70\%$  of the time; A length baseline metric achieves only  $60\%$ . Notably, the reference-backed version of the newly proposed GPT-4 based metric achieves comparable (but slightly worse) performance compared to the reference-free version. Thus, we adopt the reference-free version, which additionally enables us to place the references themselves into the the Elo setup, because they are not used in the prompts.
System-level Correlation. We summarize the LLM's pairwise judgements using the same metrics as introduced in §4.2, Elo ratings and win rate vs. reference, but instead of using a human judge, we use our reference-free GPT-4 based metric. The results are in Table 4. Notably, among the 7 systems for which we gathered human ratings for, the automatic metric produces the same ordering compared to human evaluation ( $\rho = 1.0$ ,  $p < .01$ ).
Shortcomings of proposed metric. While the relative ranking of models produced by the automatic metric correlates strongly with the ranking produced by human judgements, the win rate vs. reference according to human judgement (Table 3) are higher overall compared to the win-rate vs. reference according to the automatic metric Table 4. One plausible explanation for this discrepancy is that GPT-4, as an evaluation model, may prefer responses that closely match its own response distribution.
Per-category results. In Figure 8, we plot the win-rate vs reference for the models across all the single-image instruction families. We find that there is no model that performs the best and
worst across all the instruction families. Thus, VisIT-Bench aids in highlighting the strengths and weaknesses of the instruction-following models along various real-world use-cases.
Our work builds on prior multimodal image-text models and instruction-following benchmarks in machine learning. We provide a detailed overview of related work in §B. Multi-model modals for image-text understanding have recently emerged as powerful and useful methods for many image-language reasoning tasks [12, 15, 13, 50, 18, 14, 17, 11, 7]. Both language and multimodal models are often trained to follow language instruction, a paradigm known as "instruction following" [5, 16, 51, 14, 13, 17]. Despite the success of these approaches on existing vision-language datasets (GQA, Image Captioning [21, 52, 20]), there is no quality benchmarking dataset for multimodal instruction-following tasks that reliably replicates the way in which humans would interact with multimodal chatbots in the wild. The absence of benchmarking data impedes reliable progress assessments [53] and limits empirical evaluations of multimodal LLMs[54].
We present VisIT-Bench, a benchmark assessing multimodal chatbot skills. Going beyond prior efforts, VisIT-Bench's collection process centers potential real-world use cases, and 70 diverse instruction families encompassing a range of tasks from recognition to complex reasoning. Besides human-verified outputs, it features an Elo ranking aligning with human judgments. Our data reveals a performance gap between models and humans. Releasing data, code, and metrics, we aim for community engagement and believe VisIT-Bench will quantify progress and gaps in multimodal AI.
Although VisIT-Bench covers a wide spectrum of potential use-cases, it does not incorporate every possible vision-language task. We hope to add more categories of tasks over time. In terms of dialogue, VisIT-Bench concentrates on single-turn instances with one instruction and response. This does not encompass multi-turn interactions between users and chatbots, which presents a direction for future research. Our study focuses on image-text modalities. Future extensions could expand the scope to include other modalities like audio and video, enabling a more comprehensive evaluation. Additionally, while the dataset offers a wide variety of tasks, a larger number of examples per category could provide more depth. Finally, while our GPT-4 based metric correlates well with human judgement at instance and system level, we see some evidence that the GPT-4 based metric has a stronger preference for GPT-4 based generations compared to humans. Thus, models which train, e.g., by distilling from GPT-4 outputs, may have an unfair advantage on our evaluation.
[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021.
[2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[3] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
[4] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on  $1600+$  nlp tasks. In
[87] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
[88] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[89] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022.
[90] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333-342, 2010.
[91] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020.
[92] Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. arXiv preprint arXiv:2203.15867, 2022.
[93] Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. arXiv preprint arXiv:1808.10584, 2018.
[94] Yonatan Bitton, Ron Yosef, Eli Strugo, Dafna Shahaf, Roy Schwartz, and Gabriel Stanovsky. Vasr: Visual analogies of situation recognition. arXiv preprint arXiv:2212.04542, 2022.
[95] Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. Winogavil: Gamified association benchmark to challenge vision-and-language models. Advances in Neural Information Processing Systems, 35:26549-26564, 2022.
[96] Ron Yosef, Yonatan Bitton, and Dafna Shahaf. Irf: Image recognition of figurative language. arXiv preprint arXiv:2303.15445, 2023.
[97] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021.
[98] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569, 2023.
A License and Intended Use
The VisIT-Bench dataset, along with its various contributions such as instructions, reference outputs, and model ranking annotations, is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license applies to all the images we have directly contributed, each of which carries a public license specification in the "public images metadata" field within the dataset sheets. However, the dataset also incorporates images sourced from pre-existing collections. For these images, the original licensing terms are respected and remain applicable.
VisIT-Bench's primary purpose is to function as a dynamic benchmark that continuously evolves and evaluates instruction-following vision-language models. In the current landscape, commercial chatbots are often trained on non-disclosed and non-public datasets, which raises concerns about potential data contamination and inadvertent training on our evaluation data [8]. This risk is further highlighted by recent studies [55, 56]. To mitigate such concerns, we have chosen to withhold the complete VisIT-Bench test set from public disclosure, while still making the images and instructions available for direct download. Researchers, however, can utilize VisIT-Bench to its full potential as a dynamic benchmark by submitting their model predictions for evaluation. We will assess their models using the undisclosed test set, ensuring the ongoing evolution of the benchmark. Moreover, we are open to releasing the test data upon receiving reasonable and justified requests, particularly when additional analysis is necessary, provided that requesters agree to our non-contamination policy which prohibits the use of this data for training commercial chatbots. This approach strikes a balance between the need for robust model evaluation and the mitigation of potential data contamination.
A Dataset Analysis
Figure 9: Count of the various COCO objects present in the VisIT-Bench images on a log-scale. The object detection was performed by Yolov5-Large [39]. We observe that the object 'person' occurs most of the time in the dataset images i.e., 875 times.
Figure 10: Most frequently occurring verbs (inner circle) and their top 4 direct nouns (outer circle) in the VisIT-Bench instructions.
Multimodal Models for Image-Text Understanding: Recently, the field of machine learning has experienced a rapid proliferation of new models which can perform various image-text tasks [12, 15, 13, 50, 18, 14]. This growth has been driven by several factors, including the emergence of large-scale multimodal datasets (e.g. LAION-5B [57], Multimodal C4 [11]), improved software and hardware frameworks, and advances in modality-specific models such as language models (e.g., [10]). Our work specifically evaluates models which can generate textual outputs, given one or more images, and text. Recent examples of such models include LLaVA [13], mPLUG-Owl [17], InstructBLIP, LLaMA-Adapter, Flamingo [12] and OpenFlamingo [11], PandaGPT [18], and GPT-4 [7] (which reports multimodal capabilities but has not yet seen a release of the multimodal variant).
Instruction Following: "Instruction-following" is an emerging paradigm for training models via language, where instead of being trained to complete only a single, fixed task (such as image classification or captioning), models are trained to follow textual instructions that describe an arbitrary task, with the aim of generalizing to novel instructions. Examples of instruction-following models include Alpaca [5], LLaMA-Adapter [16], Koala [51], InstructBLIP [14], LLaVA [13], and mPLUG-owl [17]. As the downstream capabilities of these models are influenced by the quality of the training dataset, there has also been extensive work on developing instruction-following datasets [38, 58, 59, 13, 60].
To build powerful these models, two broad approaches have been shown to be effective. One approach focuses on leveraging existing pretrained task-specific tools such as image captioners [15], object detectors [61] and text-to-image generators [62] by either creating multimodal prompt interfaces [42, 63] or by executing LLM-generated programs [64, 65, 66]. The other approach [13, 16, 67, 45, 68, 17, 11] focuses on building a single pretrained model that can follow instructions by supervised finetuning on multimodal vision-language data.
Despite the success of both these approaches on the existing vision-language datasets e.g., VQA, GQA, Image Captioning [21, 52, 20], there is a lack of a high-quality benchmarking dataset for multimodal instruction-following tasks that reliably replicates the way in which humans would interact with multimodal chatbots in the wild. Similar to the image-text models discussed above, many instruction-following models have been released directly as open-source without undergoing peer review or thorough evaluation. As a result, the effectiveness of these models for many tasks is not well-understood.
Benchmarks for Machine Learning: High-quality evaluation datasets have served both to (re)assess, and to accelerate, progress on many machine learning tasks [53]. For example, our work draws particularly from the fields of computer vision and natural language processing, where benchmarking datasets have been critical drivers of progress. On the vision side, datasets such as ImageNet [69] and
CIFAR [70] have proven to be critical yardsticks of progress. On the language side, benchmarks such as SQuAD [71], SST [72], GLUE/SuperGLUE [73, 74] and more [75, 76] seen wide use. Recent work has indicated that improvements on these high-quality benchmark datasets is not the result of overfitting, and is a reliable indicator of genuine progress beyond the benchmark data [77, 78, 79, 80].
However, high-quality benchmarking datasets and evaluation methods do not yet exist for multimodal instruction-following. As a result, it is difficult to assess progress in this direction, which both reduces the field's ability to identify true breakthroughs and increases vulnerability to potential pitfalls of evaluation that have hampered progress in other areas of machine learning [53, 54].
C Interfaces for Collecting Human Annotations
In this section, we provide the templates we used to collect human annotations for the instruction generation (Figure 11), the dense caption generation (Figure 12), the model verification (Figure 13 and Figure 14), and the model rating (Figure 15).
Expected Output
This dish appears to be Massaman Cur oil in a large pot, add onion and saut, air curry paste, ginger and garlic. 3. Saute chicken and potatoes are cooked through minutes. 5. Serve with rice (white, brown cilantro on top.
Find an Image to Annotate to test Food Recipe, and create a Visual instruction, that a textual-only model/human won't
Feel free to skip the HIT if you don't feel creative with this one
Provide a \*\*PUBLICALLY LICENSED\*\* image URL.
0/200
Provide a \*\*creative and challenging\*\* model instruction that would blow your mind if an AI model could answer it.
Previous HIT Showing Task 23 of 70 Next HIT
Figure 11: An interface that collects the image and corresponding instruction.
Figure 12: An interface that collects the dense caption given an image-and-instruction pair.
Instruction (Tested Skill=Exercise) Based on commonly held beliefs about upper body strength in men and women which competitor is likely to win this competition.
A highly visual description that will allow TEXT-ONLY models *MULTIPLE* answers for the instructions WITHOUT THE IMAGE
0/2000
Figure 13: A sample of the model verification where the GPT4 follows the instruction correctly.
Dense Caption:
A woman wearing a red gown is sitting in a wheelchair but dancing with a man in a black tuxedo and white dress shirt. She is dressed in her best hand and this right hand while posing with her other hand. He has left arm stretched out. They are performing on a stage with a highlighter.
Why is seeing this couple compete in this event an uncommon sight? Does this convey a positive message?
GPT-4 Prediction:
Seeing this couple compete is an uncommon sight because the woman is in a wheelchair, which is not typical in traditional care. However, this conveys a positive message as it showcases inclusivity, determination, and the ability to overcome challenges.
Does GPT4 correctly follow the instruction?
If answered 'No' for the above, mark one of the following:
1.  $\odot$  N/A
2. The dense caption is detailed enough; the problem is in GPT4's answer
3. O The dense caption is underspecified; not GPT4's fault
$\square^{*}$  Might be offensive/bad advise/problematic
Figure 14: A sample of the model verification where the GPT4 failed to follow the instruction due to its incorrect reasoning.
There are two equations on a white paper. 1)  $f(x) = \sqrt{2x + 6} - 1$  and 2)  $f(x) = \frac{1}{3x + 2} - 1$
For the following exercises, graph the functions by translating, stretching, and/or compressing a toolkit function.
To graph the functions, follow these steps: 1) For  $|x| = \operatorname{sqrt}(x + 8) - 1$ , start with the toolkit function  $y = \operatorname{sqrt}(x)$ . Translate it 6 units to the left and 1 unit down. 2) For  $|x| = 1 \operatorname{sqrt}(x + 1) - 1$ , start with the toolkit function  $x = 1 \operatorname{sqrt}x$ . Translate it 2 units to the left and 1 unit down.
1. N/A 2.OT
2. The dense caption is detailed enough; the problem is in
GPT4's answer 3. O The dense caption is underspecified: not GPT4's fault
* Might be offensive/bad advise/problematic
Figure 15: An interface that collects the feedback of the model rating.
Model A
Model B
D Existing Datasets incorporated in VisIT-Bench
In Table 5, we listed the existing datasets that are incorporated in our VisIT-Bench. Among these datasets, 15 contain a single image in each sample pair, and 10 require reasoning based on multiple images.