SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: List of existing datasets in VisIT-Bench, categorized as single and multiple image datasets.
Dataset	Topic
Single	VQA [21]	Visual Question Answering
VCR [81]	Cognition-level Visual Understanding
GD-VCR [82]	Geo-Diverse Commonsense Reasoning
WHOOPS [40]	What Makes this Image Strange
Newyork Caption [83]	Humor Understanding
CLEVR [84]	Visual Question Answering
Kilogram [85]	Tangrams Identification
Harmful Memes [86]	Memes Understanding
ScienceQA [87]	Science Question Answering
OK-VQA [88]	Outside Knowledge Visual Question Answering
AOK-VQA [89]	Outside Knowledge Visual Question
AOK-VQA [89]	Question Generation
VizWiz [90]	Visual Question Answering
GQA [52]	Visual Question Answering on Scene Graphs
TextCaps [91]	Visual Question Answering on Texts
Multiple	Robust Change Captioning [22]	Describing What has Change in a Scene
NLVR2 [36]	Testing Visual Language Bias
ImageCoDE [92]	Image Retrieval
Spot-the-Diff [93]	Identifying Differences
VASR [94]	Visual Analogies
WinoGavil [95]	Visual Associations
IRFL (Metaphor) [96]	Figurative Speech Understanding
IRFL (Idioms) [96]	Figurative Speech Understanding
IconQA [97]	Abstract Diagram Understanding
Pick-a-Pic[98]	Text-to-Image User Preferences
Table 6: List of skills and existing datasets in VisIT-Bench
'scienceqa', 'ocr math', 'recognition', 'okvqa', 'house plan understanding', 'nlvr2', 'gardening tips', 'textcaps', 'architectural styles', 'dressing sense', 'winoground', 'food recipe', 'paper folding', 'whoops', 'spot the diff', 'winogavil', 'imagecode', 'exercise', 'art knowledge', 'gqa', 'physical knowledge', 'contextual knowledge of events', 'home renovation', 'aokvqa', 'animals', 'vasr', 'counting', 'board games', 'solving geometry problems', 'who to call?', 'clevr', 'building materials', 'hazard identification', 'pickapick', 'astronomy', 'figurative speech explanation', 'write a story', 'gestures understanding', 'newyork', 'cultural knowledge', 'aokvqg', 'traffic sign identification', 'pop culture', 'fashion products', 'harmful memes', 'write a poem', 'vizwiz', 'guesstimate of capacity', 'location understanding', 'graph reasoning', 'vqa', 'game playing', 'differently abled', 'chemical identification', 'history knowledge', 'climate and weather understanding', 'irfl metaphor', 'human emotion recognition', 'medical', 'gd vcr', 'vcr', 'technical support', 'catchy titles', 'kilogram', 'anagrams', 'color', 'tour guide', 'directions', 'irfl idiom', 'rcc'
E Elo Rating
For many years, the Elo rating has been popular in ranking players in zero-sum games such as chess [25]. Recently, it has been adopted to rate large language models (LLMs) against each other on the user instructions. In this work, we adopt the same strategy to rank a set of instruction-following vision-language models, that can grow dynamically with further advances in the field.
Given two multimodal chatbots  $\mathcal{C}_a$  and  $\mathcal{C}_b$  with their absolute Elo rating  $\mathcal{R}_a$  and  $\mathcal{R}_b$ , respectively. Simply put, the probability of  $\mathcal{C}_a$  winning over  $\mathcal{C}_b$  in a head-to-head battle is given by:
$$
P \left(\mathcal {C} _ {a} \text {w i n s o v e r} \mathcal {C} _ {b}\right) = \frac {1}{1 + 1 0 ^ {\left(\mathcal {R} _ {a} - \mathcal {R} _ {b}\right) / 4 0 0}} \tag {1}
$$
In practice, calculating the Elo rating requires us to set hyperparameters to decide the weightage for each win and loss in a head-to-head battle between two models. In our work, we use the open implementation of Elo for LLMs by FastChat at https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/monitor/elo_analysis.py.
F GPT-4 Pairwise Evaluation Prompts
The specific prompts we use to extract pairwise judgements from our language model are provided in Table 16 (reference-free version) and Table 17 (reference-backed version). When applied to GPT-4 [7], these prompts usually solicit a definitive pairwise response by the model. But, in some cases, the model either produces a pairwise judgement in an unexpected format, or, refuses to issue a judgement at all. For cases like these, we issue an additional query to ChatGPT to extract an answer (or decide there is no answer) using an additional prompt, given in Table 18. If after this step there is still no definitive pairwise judgment, we call the result a tie.
A head-to-head comparison involves querying GPT-4 two times. Each query has approximately 800 input/250 output tokens (output includes chain of thought; see Supplementary for full prompt/response example). Running a head-to-head comparison between a new model and a randomly sampled existing model across the 700 instances is usually sufficient for a stable Elo estimate. In total, at current GPT-4 prices, the 700 head-to-head comparisons required to assess a new model costs 50-70 dollars.
system prompt (human authored)
You are ImageTaskEvaluationGPT, an expert language model at judging whether or not a response adequately addresses an instruction in the context of an image. More specifically, you will be given the following:
1. An image context: This will describe the contents of an image with sufficient detail to address the instruction.
2. An instruction: This is a question, an imperative request, or something similar about the image which requires a response.
3. Two responses, response A and response B: These two responses attempt to address the instruction in the context of the image. Your job is to judge whether response A or response B better. A and B are randomly ordered.
Some things to remember:
- Even though you are just a language model, the image description will be sufficiently detailed so that your judgements can be accurate.
- You are capable of judging response quality, accounting for important factors like correctness, relevance, fluency, specificity, etc.  
- You think step-by-step, but ultimately respond with "Response A" or "Response B"
user (human authored)
I will describe an image to you, and provide an instruction. Then, I will give you two candidate responses that address the instruction in the context of the image: these will be labelled "Response A" and "Response B". Your job is to first reason step-by-step about which response is best in terms of accuracy, specificity, fluency, etc. After reasoning step-by-step and comparing the pros/cons of each response, in the end, respond with "Overall, Response X is better." where X is either A or B.
assistant
assistant (human authored)
Sure, please provide the image context, the instruction, and the two candidate responses, Response A and Response B. Then, I will think step-by-step and provide my ultimate judgement as to which response is better.
user (from human-authored template)
OK. Here is the image description, the instruction, and the two response options. Response A and Response B. Image context: A skier is wearing a green jacket, burgundy pants, white shoes, white gloves, and a gray beanie. There are people sitting on the background with their snowboards.
Instruction: Answer the following question based on the provided image: Are the trousers the same color as the shoes?
Response A: no
Response B: No, the trousers are not the same color as the shoes. The trousers are burgundy, while the shoes are white.
Think step-by-step and finish your response with "Overall, Response X is better." where X is either A or B.
Step 1: Identify the key information from the image description.
The skier is wearing:
Burgundy pants
-White shoes
Step 2: Assess the accuracy of each response in addressing the instruction.
Both Response A and Response B convey that the trousers and shoes are not of the same color.
Step 3: Compare specificity and detail of each response.
Response A is concise and provides a straightforward answer, but lacks detail.
Response B provides a clear, comprehensive answer that includes the color of both the trousers and shoes.
Step 4: Consider the fluency and correctness.
Both responses are fluent and grammatically correct.
Overall, Response B is better.
Figure 16: The prompt we use for reference-free evaluation, along with a sample instance/completion from GPT-4 [7].
1. An image context: This will describe the contents of an image with sufficient detail to address the instruction.
2. An instruction: This is a question, an imperative request, or something similar about the image which requires a response.
3. A reference output: This is a high-quality example output that humans have judged to be an accurate response for the input instruction.
4. Two responses, response A and response B: These two responses attempt to address the instruction in the context of the image.
Your job is to judge whether response A or response B better. A and B are randomly ordered. Some things to remember:
- Even though you are just a language model, the image description will be sufficiently detailed so that your judgements can be accurate.
- Take the high-quality reference into account when making your judgements, but remember: some instructions are more open-ended than others, so for those cases, a high quality response can differ from the reference.
- You are capable of judging response quality, accounting for important factors like correctness, relevance, fluency, specificity, etc.
- You think step-by-step, but ultimately respond with "Response A" or "Response B".
I will describe an image to you, and provide an instruction. Then, I will provide a reference output which is an example of a high quality output for that instruction in the context of the image. Then, I will give you two candidate responses that address the instruction in the context of the image: these will be labelled "Response A" and "Response B". Your job is to first reason step-by-step about which response is best in terms of accuracy, specificity, fluency, etc. After reasoning step-by-step and comparing the pros/cons of each response, in the end, respond with "Overall, Response X is better." where X is either A or B.
Sure, please provide the image context, the instruction, the reference, and the two candidate responses, Response A and Response B. Then, I will think step-by-step and provide my ultimate judgement as to which response is better.
OK. Here is the image description, the instruction, the high-quality reference, and the two response options, Response A and Response B.
Image context: There is a white plate that has on it a small pizza with many topics, mostly unrecognizable, due to the pie being very well done, charred to the point of almost being burnt, and the crust blackened in many places.
Instruction: Someone told the creator of this item that they are very talented. What is the implied meaning?
High quality reference: The implied meaning is that the person complimenting the creator is being sarcastic, as the pizza is overcooked and almost burnt, which is not a sign of talent.
Response A: When someone says that they are "very talented" in the context of the image, it implies that the person responsible for creating the pizza is skilled in the craft and able to create an appealing and tasty dish. The person praising the pizza's creator recognizes their culinary skills, which can be seen through the combination of ingredients and the overall look of the finished dish. The praise is a form of appreciation for their efforts and the enjoyable result they achieved.
Response B: It is difficult to determine the implied meaning without further context or information about the image. It appears to be a pizza that has been burnt or charred on the surface and has various toppings, including pepperoni, onion, and mushrooms. The image is shown on a white plate or surface. The implied meaning of this image could be that it is a dish that has been cooked badly or that it is a dish with a unique or unusual combination of ingredients.
Step 1: Comparing both responses with the high-quality reference, it's clear that Response A describes the pizza as well-made and appealing, while Response B is more focused on the fact that it is burnt or charred, somewhat aligning it with the reference.
Step 2: Evaluating specificity, Response A defines the creator as skilled, even though the image description contradicts this by stating that the pizza is overcooked. Response B, while not as specific, acknowledges the overcooked nature of the pizza.
Step 3: In terms of fluency, both responses are easily understood and written in coherent language.
Step 4: Considering accuracy, Response A is inaccurate in relation to the context and instruction because it implies a positive meaning when it should be sarcastic. Response B doesn't fully capture the sarcasm but is closer to the truth about the pizza being badly cooked.
Step 5: Understanding relevance, Response A is not entirely relevant due to the inaccuracy in the creator's talent, while Response B, although less direct, is more relevant by mentioning the burnt or charred nature of the pizza.
Overall, Response B is better.
Figure 17: The prompt we use for reference-backed evaluation, along with a sample instance/completion from GPT-4 [7].
system prompt (human authored)
You are FinalAnswerExtractionGPT, an expert language model at extracting multiple choice answers from written out from longer explanations. You will be given several sentences describing a thought process which should eventually contain a final answer, either A or B. Your job is to extract the final answer that the explanation arrives at.
Some things to remember:
- Keep your answers short: only output "Final Answer: X" where X is A or B
- If the explanation doesn't mention a final choice explicitly, you can output "Unknown"
(+ 2 in-context examples (human authored))
Example One: GPT-4 gives a definitive answer, but in the wrong format
user (GPT-4's raw pairwise judgement)
Step 1: Compare the two responses to the instruction
- The instruction asks if people are required to stop and requests an explanation.
Step 2: Assess Response A
- Response A indicates that people should stop due to the presence of a stop sign. However, it doesn't consider the fact that the sign is pressed against the bus, and the LED lights are off.
Step 3: Assess Response B
- Response B simply says "yes," acknowledging that people should stop, but it doesn't provide the explanation required by the instruction.
Step 4: Compare specificity, fluency, and other factors
- Response A is more specific and fluent but misconstrues the image description.
- Response B is accurate in that it acknowledges the stop sign but lacks an explanation.
Overall, since neither response is fully correct, it is difficult to declare a better response. However, given the fact that Response B acknowledges the prominent stop sign, I would say that Response B is slightly better, but it should also explain how the situation indicates that people should stop.
Please extract the final answer from the above text.
assistant
assistant
Final Answer: Response B is slightly better, but it should also explain how the situation indicates that people should stop.
Example Two: GPT-4 refuses to answer.
user (GPT-4's raw pairwise judgement)
Step 1: Consider the image context, which states that the original thirteen colonies are shown with New Hampshire specifically highlighted in a darker green.
Step 2: Assess the accuracy of Response A, which claims that the colony shown is "Virginia." This is not accurate since the image context indicates New Hampshire being highlighted.
Step 3: Assess the accuracy of Response B, which claims that the colony shown is the Massachusetts Bay Colony. This response is also not accurate, as the image context states New Hampshire is the highlighted colony.
Step 4: In terms of accuracy, both responses are incorrect. Neither response is consistent with the image context.
Step 5: Consider other factors like fluency and specificity. Both responses seem fluent and specific in nature.
After considering these factors, none of the responses are accurate given the information provided in the image context. However, I am unable to provide an "Overall, Response X is better." judgement in this case as both responses are incorrect and neither stands out as being better than the other.
Please extract the final answer from the above text.
assistant
Final Answer: Unknown.
Figure 18: The prompt we use to extract answers from ill-formatted GPT-4 [7] responses, along with two example outputs. In one case, GPT-4's pairwise judgement is given, but in a format different than requested. In the other case, GPT-4 "defies" the prompt, and refuses to issue a judgement because both options are bad.