SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Ablation study on important components of the proposed method.
Model	Accuracy
TOA-full	60.6
TOA (w/o hypothesis)	52.5
TOA (w/o multi-round)	57.8
Visual Programming	40.9
keyword matching and nearest-neighbor projection. Specifically, we first match the keywords of the predicted answer with the answer candidates. For the synonym paraphrasing without overlap of keywords, we project the prediction into the vocabulary using cosine similarity in the space of GloVe [50] word embeddings, which is similar to the multiple-choice evaluation of A-OKVQA. After this post-processing, we follow the conventional soft accuracy to report our results. It is worthwhile to note that the evaluation of open-ended answers is still an open problem and we believe that more efforts should be dedicated to addressing this problem given its increasing demand in the future.
4.2 Results on OK-VQA and A-OKVQA
We compare our proposed method with the state-of-the-art methods on OK-VQA dataset in Table 1. The compared methods are categorized into two classes. Methods in the first category [9, 51, 52, 4, 5, 6, 10] leverage explicit knowledge sources other than the implicit knowledge of LLM. The commonly used external knowledge source include ConceptNet [9, 51, 4], Wikipedia [51, 4, 5, 6] and Google [4, 52]. Note that KAT [5] and REVIVE [6] use both explicit knowledge sources and LLM as implicit knowledge. Methods in the second category [11, 13] only use LLM as the implicit knowledge source to obtain the answer. The results show that the introduction of LLM as an implicit knowledge source brings significant improvement compared to explicit knowledge retrieval methods. PromptCap [13] trains a task-aware captioning model with synthesized data. It is the primary baseline of our proposed method since the caption given to our task scheduler at the initial step is generated by PromptCap. The improvement in performance on our method indicates that we can acquire more useful visual information other than the caption through multi-round dialogue.
We also report the results on the A-OKVQA dataset in Table 2, where we outperform the compared methods on direct answer settings while achieving the second-best result on multiple choices settings. This implies that our proposed method is better at open-ended questions than multiple choices questions, since it generates open-ended answers of free-form natural language.
We conduct extensive ablation studies to have an in-depth view of each component of our proposed method. Due to the expensive price of calling the OpenAI API and the limited visit frequency allowed to the public, we conducted our ablation studies on a randomly selected subset of OK-VQA.
Question: Which furnishing seen here is of a type reminiscent of what is seen in some classrooms?
Caption: a person walking in a messy room.
Reasoning: Since the captioning does not provide any information about the furnishings, we need to locate them first.
Hypothesis: None Verification: locate(furnishing) Answer: None
Response: Located furnishing.
Reasoning: We can use visual attributes to compare the furnishings to those seen in classrooms. Desks and chairs are often seen in classrooms. We can verify if there exists a desk or chair.
Hypothesis: [desk, chair] Verification: exist(desk) Answer: None
Response: True
Reasoning: Since the vision model verifies the hypothesis, we can decide the answer.
Hypothesis: None Verification: None Answer: desk
Question: What is the white screen on the back for?
Figure 3: Examples of multi-round dialogue between task scheduler and vision executor.
Caption: a classroom with laptops on the desks and a white screen on the back.
Reasoning: We can infer from the captioning that the white screen is a projection screen, We can verify if there exists a projector in the image. Hypothesis: [projection] Verification: exist projector) Answer: None
Response: False
Reasoning: Since there is no projector in the image, we assume that the white screen is used for writing or drawing with markers.
Hypothesis: [writing, drawing]
Verification: filter(white screen writing) Answer: None
Reasoning: Since the verification conflicts with the hypothesis, we call the simple query function to obtain the answer.
Hypothesis: None Verification: simple query(What is the white screen on the back for?) Answer: None
Response: projector
Reasoning: The answer obtained from the simple query function is projection screen, which implies that the white screen is used for projecting images or videos from a projector.
Hypothesis: None Verification: None Answer: projector
Question: What media was used to create the tigers face?
Caption: a group of people sitting on a sidewalk next to two tigers.
Reasoning: Since the question asks about the tigers face, we first locate it in the image.
Hypothesis: None Verification: locate(tiger face) Answer: None
Response: Located tiger face.
Reasoning: The media used to create the tigers face could be paint, pencil, or digital. We can verify if the tigers face is painted.
Hypothesis: [paint] Verification: filter(tiger face, painted)
Answer: None
Reasoning: Since the verification agrees with the hypothesis, we can decide the answer.
Hypothesis: None Verification: None Answer: paint
Question: What design is on the headboard of the bed?
Caption: a bed with a design on the headboard.
Reasoning: Since the question asks about the headboard of the bed, we first locate it in the image.
Hypothesis: None Verification: locate(headboard) Answer: None
Response: Located headboard
Reasoning: We can use vision model to detect the design on the headboard. Hypothesis: None
Verification: query_attribute(headboard, design) Answer: None
Response: It is a gold patterned headboard.
Reasoning: Since the result shows that the headboard has a pattern, we can further query about the pattern. Hypothesis: [gold pattern]
Verification: query_attribute(headboard, pattern) Answer: None
Response: a floral pattern.
Reasoning: The vision model specifies the pattern on the headboard, so we can decide the answer. Hypothesis: None Verification: None Answer: flower
Table 4: Results of OK-VQA dataset with a different number of in-context examples and selection strategy.
8 examples	16 examples	24 examples
Random	54.9	58.8	60.2
Similarity-based	55.3	60.6	60.6
There are two important components in our proposed method, i.e., the multi-round dialogue strategy, and the hypothesis-verification process. To quantify the benefit of these two main strategies, we remove each of them from our full model and compare the results. To evaluate the impact of the hypothesis-verification strategy, we compare an ablative model which removes the REASONING and HYPOTHESIS parts from the instruction of the task scheduler. At each step, the scheduler directly specifies the vision function without explicit reasoning. To evaluate the benefit of multi-round dialogue, we let the task scheduler generate the whole program according to the initial hypothesis. We also compare the results of Visual Programming [19] since it is similar to our proposed method but generates the whole programming at one time. The results are shown in Table 3, and they demonstrate that the hypothesis-verification process with multi-round dialogue is essential to the performance. The clear output format based on the idea of hypothesis-verification better activates the reasoning ability of large language models. Multi-round dialogue allows the model to decide the next step dynamically based on the previous results, hence having better efficiency of visual information collection and tolerance of the misleading results of the vision model.
Another factor that influences the performance is the in-context example selection. Previous works [45, 11] have discussed the influence of in-context example selection strategies and the number of examples. We show the results of random selection and the heuristic selection described in Section 3.4, each with respect to a different number of examples in Table 4. We can observe from the results that more in-context examples often bring better results, while the marginal impact is diminishing. The similarity-based in-context selection strategy can achieve better results than random selection since it can pair the input question with the most similar examples. However, as the number of examples increases, the advantage is not as significant. This is probably because the model already has enough similar examples selected from the fixed available example pool.
We show a few qualitative results in Figure 3 to further illustrate our proposed task-oriented active VQA approach. In the upper left example, the captioning does not provide relevant information to answer the question. For the upper right example, the captioning fails to distinguish between real tigers and tiger-shaped statues. However, the scheduler makes a reasonable hypothesis based on common-sense knowledge, and the hypothesis is verified by the vision model. For the bottom left example, the vision model fails to detect the projection screen, and the responses conflict with the initial hypothesis. In this scenario, the scheduler makes a new reasonable hypothesis and actively acquires more information. Despite the failure of the vision model, it makes the correct decision considering both the common-sense knowledge and the visual information. Finally, the bottom right example shows how the scheduler progressively acquires the essential visual information based on the previous dialogue to give a more accurate answer.
We propose task-oriented active VQA (TOA), which uses LLM as an implicit knowledge source and answers the question through a sequential hypothesis-verification process. In TOA, the LLM as the task scheduler actively acquires visual evidence from a set of vision models in a task-oriented manner, and hence can attend to the essential information in the image more accurately and efficiently. Our proposed method dynamically decides the next step through multi-round dialogue. It can better deal with situations when the vision models are imperfect, though still limited by the performance of them. A potential limitation of our method is that the design of the prompting examples is subjective, which may bring uncertainty to the reasoning process. Besides, addressing the evaluation of open-ended question answering remains an open problem for future works.
This work was supported in part by National Science Foundation grant IIS-1815561 and IIS-2007613.
A Demonstration of our prompt instructions and in-context examples
We illustrate the prompt instructions and in-context examples we used in our method in Figure 4.
B Human evaluation and Using GPT for evaluation
Since the final answers are generated open-ended and unaware of the answer vocabulary in our proposed method, it is improper to evaluate the results by conventional exact matching. To handle the semantically equivalent answers, we proposed a method combining keyword matching and nearest-neighbor projection. To further analyze our evaluation strategy, we conduct human evaluation on the predictions of OKVQA dataset and also leverage ChatGPT to judge whether the predicted answer and the ground truth can be considered consistent, given the original question. The results of different evaluation strategies are shown in Table 5.
These results further demonstrate that although our initial attempt on open-ended answer evaluation may not be perfect, it did not over-claim the performance of our proposed method. Thus we can affirm the effectiveness of our method. Moreover, this analysis further validates the necessity of addressing the evaluation of open-ended question answering.
Table 5: Comparison of results on OKVQA dataset by different evaluation methods.
Matching and projection	Human	ChatGPT
Accuracy	60.6	65.0	64.7
C Ablation study on the design choice of our prompting strategy
To demonstrate the effectiveness of our design principles, we conduct ablation study on different components of the prompting. We removed the task description, vision tools description and hypothesis guidance in the prompt instruction and in-context examples, respectively. The results are shown in Table C
The results show that these concise yet essential design principles have proven to be effective to guide the LLM for knowledge-based visual reasoning. However, we also recognize that our design might not be optimal. Further refinement in prompt design might yield improvements, which will be explored in our future work.
Table 6: Ablation study on prompting strategy.
Model	Accuracy
TOA-full	60.6
w/o task description	57.2
w/o vision tools description	58.6
w/o hypothesis guidance	55.1
Task description	I have a question about an image. Each time you need to make a hypothesis and verify by collecting visual information. You can use the following visual functions to gather information.
Constraints	You can only call one vision function at a time. The answer should be very concise.
Vision tools description	exist(object_name:str)->bool
Returns True if the object specified by object_name is found in the image, and False otherwise.
attribute(object_name:str, visual_attribute:str)->str
Returns the attribute of the object specified by object_name. Presupposes the existence of the object. The parameter visual_attribute is a nominal class name, such as color, shape, material.
filter(object_name:str, property:str)->bool
Returns True if the object possesses the property, and False otherwise.
Presupposes the existence of the object.
...
In-context examples
Question: Which of these appliances would you heat your noodles in?
Caption: A kitchen with a refrigerator and a dishwasher
Assistant: Reasoning: We know that there is a refrigerator and dishwasher in the kitchen.
None of them can heat. We often use microwave for heating. We can verify if there exists a microwave. Hypothesis: [microwave]
Verification: exist(microwave) Answer: None
User: True
Assistant: Reasoning: Since the hypothesis is verified by the vision model, we can decide the answer. Hypothesis: None Verification: None Answer: microwave
Question: Name the material used to make this car seat shown in this picture?
Caption: A cat sitting on the back of a car seat
Assistant: Reasoning: Since the question asks about the car seat, we first locate it in the image. Hypothesis: None Verification: locate(car seat) Answer: None
User: Located car seat
Assistant: Reasoning: Car seats are often made of leather, fabric or vinyl. We can directly verify each of them. Hypothesis: [leather, fabric, vinyl] Verification: filter(car seat, leather) Answer: None
User: False
Assistant: Reasoning: The car seat is not made of leather. Next we verify if it is fabric.
Hypothesis: [fabric, vinyl] Verification: filter(car seat, fabric) Answer: None
User: True
Assistant: Reasoning: Since the vision model verifies the hypothesis, the answer is fabric.
Hypothesis: None Verification: None Answer: fabric
Figure 4: The prompting instructions and in-context examples which guide the large language model to follow the hypothesis-verification reasoning stream. The instructions consist of task description, format constraints and vision tools description.