Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
Authors: Deepanway Ghosal
Navonil Majumder
Roy Ka-Wei Lee
Rada Mihalcea
Soujanya Poria
Venue: EMNLP 2023 Findings (2023)
Keywords: VQA
Multimodal Language Models
Question Answering

Abstract: Visual question answering (VQA) is the task of answering questions about an image. The task assumes an understanding of both the image and the question to provide a natural language answer. VQA has gained popularity in recent years due to its potential applications in a wide range of fields, including robotics, education, and healthcare. In this paper, we focus on knowledge-augmented VQA, where answering the question requires commonsense knowledge, world knowledge, and reasoning about ideas and concepts not present in the image. We propose a multimodal framework that uses language guidance (LG) in the form of rationales, image captions, scene graphs, etc to answer questions more accurately. We benchmark our method on the multi-choice question-answering task of the A-OKVQA, Science-QA, VSR, and IconQA datasets using CLIP and BLIP models. We show that the use of language guidance is a simple but powerful and effective strategy for visual question answering. Our language guidance improves the performance of CLIP by 7.6% and BLIP-2 by 4.8% in the challenging A-OKVQA dataset. We also observe consistent improvement in performance on the Science-QA, VSR, and IconQA datasets when using the proposed language guidances. The implementation of LG-VQA is publicly available at https://github.com/declare-lab/LG-VQA.

Language Guided Visual Question Answering: Elevate Your Multimodal
Language Model Using Knowledge-Enriched Prompts
Deepanway Ghosal Navonil Majumder', Roy Ka-Wei Leel,
Rada Mihalcea Soujanya Poria
Singapore University of Technology and Design, Singapore University of Michigan, USA
[deepanway_ghosa] @mymamlm......... suteaa.iissu..e..m edu.sg (navonil_mm _a seavonil_mjjudeer, jumder,, rоy roy_lee, sporia] @sut edu. sg mihalcea www.ceee@uiichheee
Abstract
Visual question answering (VQA) the task of answering questions about an image. The task assumes an understanding of both the im- age and the question provide natural lan- guage answer. VQA has gained popularity in recent years due its potential applications in wide range of fields, including roboties, education, and healthcare. In this paper, we focus on knowledge mnowledge-augmented VQA, where answering the question requires commonsense knowledge, world knowledge, and reasoning about ideas and concepts present in the image. We propose multimodal framework uses language guidance (I.G) the form of rationales, image captions, scene graphs, etc to answer questions more accurately. We benchmark our method on the multi-choice task of the A-OKVQA, Science-QAn VSR, and IconQA datasets using CLIP and BLIP models. We show that the use of language guidance is simple but powerful and effective strategy for visual question an- swering. Our language guidance improves the performance of CLIP by 7.6% and BLIP-2 by 4.8% in the challenging A-OKVQA dataset. We also observe consistent improvement in performance on the Science-QA, VSR, and IconQA datasets when using the proposed language guidances. The implementation of LG- VQA is publicly available at https:// github. com/ pecelare-lab/I---
Figure 1: time of day are the skateboarders probubly skating on the beach? given. In our LG-VQA framework, we generate the caption: a group of skate boarders in front of palm trees at sunset and the question constrained ra- tionale: the sun is just beginning to set behind the sand to provide the answer: sunset.
The above image and the question At what
of the existing approaches focus solely on analyz-
ing the visual features associated with the image or the linguistic content or representation of the ques- tion (and possibly candidate answers). without uti- lizing additional sources of guidance. However, incorporating external guidance into these models has the potential to improve their performance and enhance their understanding of visual content. In this paper, propose a multimodal frame- work, LG-VQA that leverages language guid- ance to improve the accuracy of image question- answering systems. Language guidance is sourced from elements such as image captions, scene graphs, and rationales created in response to ques- tions. These sources serve to enrich textual in- structions with valuable knowledge. Our method is evaluated the multi-choice A-OKVQA dataset (Schwenk al., 2022). We choose A- OKVQA specifically because the questions in this dataset require broad base of commonsense and world knowledge to answer. We also evaluate our method on the challenging ScienceQA (Lu et al.,
1
Introduction
Visual understanding one of the most complex tasks in artificial intelligence. Among the many challenges associated with it, image question an- swering has been formulated as task that tests the ability of a system to understand the elements of image in a way similar to how humans in- teract with images. This task involves creating models that can accurately answer questions based on the content of an image. While significant progress has been made in image question answer- ing (Wang et al., 2022a; Chen et al., 2022), most

---

2022), Visual Semantic Reasoning (VSR) (Liu al., 2023), and IconQA (Lu al., 2021) datasets. With the advent of large-scale multi- modal pre-training (Wang et al., 2022a,b), the per- formance in the commonly used VQA and VQA v2 datasets (Antol et al., 2015; Goyal et al., 2017) has saturated.
normalize (ICLI
T 12 normalize Cormalie(ToLIP((( (CLIP((tt) score e' (I.T) e Im Tm
(1)
m
where / is a learned temperature parameter for scaling the dot product similarity. We use this for- mulation of the (image, text) matching methodol- ogy for CLIP models in our VQA framework.
The datasets considered in this paper provide a more challenging test bed for VQA where fac- tual, commonsense,, physical, scientific, and visual knowledge are required, going beyond the usual image recognition, attribute detection tasks that are generally found in the VQA, VQA v2 datasets. Previously Shah al. (2019); Wu et al. (2017); Zheng et al. (2021) have used concrete knowledge bases such DBPedia, Freebase, or named entity knowledge graphs in VQA. We instead show that simple language guidance without using knowl- edge graphs is also a powerful technique for im- proving VQA performance in various challenging datasets.
BLIP-2 (Li et al., 2023) bootstraps joint pre- training over image and language data with off- the-shelf frozen image encoders and frozen large language models (LLMs). A small transformer network called Query Transformer Q-Former is trained to model the interaction between the frozen image and language models.
In the first stage of BLIP-2 pre-training, the Q-Former learns extract visual representations from the image that most relevant to the using image-text contrastive learning, and image- grounded text generation tasks. In the second stage, the trained Q-Former is used to extract a set of query embeddings about the image. The query embeddings are converted into the input embed- ding space of the LLMs through a linear projec- tion. The projected vector functions the soft vi- sual prompts that condition the LLM generate the desired text. BLIP-2 uses paired (image, text) data for both stages of pre-training.
We benchmark our approach with the CLIP (Radford et al., 2021) and BLIP-2 (Li al., 2023) models and show significant improvements over those, demonstrating that incorporating language guidance can be an effective strategy for visual question answering.
2 Background
We denote the image encoder and Q-Former of BLIP2 as JBLIP and QBLIP. The image-text matching score is computed as follows:
We use the following two strong multi-modal models in our framework:
IBLIP(igg)
features = QILIP(T txt,
(2)
CLIP (Radford et al., 2021) is model for learning transferable visual representations from natural language supervision. CLIP is trained 400 million (image, text) pairs learn a joint em- bedding space where image representations and their corresponding text representations are close to each other. We use the CLIP model with the ViT-L/14 image encoder (Dosovitskiy et al., 2020) and the GPT-2 text encoder (Radford et al., 2019). CLIP encodes the image and the text separately through the corresponding encoders. Then, nor- malized dot product between the encoded vectors provides the image-tett matching score. We de- note the image and text encoder as TCLIP and TCLIP. Given an image img and text instance txt, the matching score is computed as follows:
score = Proj (features)
where q is the leaned embeddings during pre- training and Proj is the projection head. Addi- tionally, we use the FlanT5 (Chung al., 2022) model as the LLM for image-constrained text gen- eration. Here, the Q-Former does not take any text as input, but the LLM encoder may use a prefix text prefiz on which the output text is condi- tioned:
Upromppt QBIJ((I,
encoded FlanT5 Encoder(U prefíx) (3)
output FlanT5 Decoder(encoded)
We use the image-text matching methodology

---

and the (visual prompt, prefix) conditioned text generation in various stages of our framework.
C L= (s) (log(8k)
(6)
3 LG-VQA Framework
where we denote p; to be the class label of the answer choices. The value of pk is as og is the correct answer, whereas the values of the other ps are zero as they are the incorrect answers. The loss thus simplifies to the -log(5i) (k.) as shown Eq. (6). The loss equivalent to the cross-entropy loss used in multi-class classification problems.
3.1 Zero-Shot Visual Question Answering
The CLIP and BLIP-2 models show impressive zero-shot performance in various computer vision tasks. For ImageNet type classification tasks, the method generally works by creating sentence in- stances from the labels: A photo of a dog, A photo of cat, photo of a bird, etc. Then, similarity scores are computed between the image and all the sentence instances created from the labels. The la- bel (e.g. bird) is then predicted from the sentence (e.g. A photo of a bird) which provided the maxi- mum similarity score.
3.3 Constained Inference Generation
We use the BLIP-2 model with FLAN-T5 to gen- erate inferences (such as rationales, explanations and captions) constrained on the image img and question question that could potentially be help- ful for the VQA task. We follow the setting speci- fied earlier in Eq. (3). We do not use any prefix, and thus the FLAN-T5 model is constrained en- tirely upon the visual prompt extracted from the question and the image with the Q-Former.
We apply similar method for the zero- shot multi-choice visual question-answering task. Given image img, question question, multiple answer choices 01,02, 02........ , an, we create sen- tences by concatenating the question and the an- swer {question, a;}. The concatenated sentence providing the highest image-text matching score corresponds to the predicted answer:
We use the rationales in the A-OK VQA dataset and the explanations for the VQA and VQA v2 provided in (Li et al., 201 8; Park et al., 2018) to fine-tune the BLIP-2 with FLAN-T5 model. We also COCO Captions (Chen al., 2015) for image captioning. We keep the visual encoder and the FLAN-T5 frozen and only tune the Q-Former network for the tasks rationales or explanation or caption generation. We use the generated ra- tionales, explanations, and captions as language guidance in our main model ($3.5).
txt; = {question, ai) vi=11 1......
match(img, txt:)
(4)
best = argmar([ss,..... 8n])
answer 0best
3.2 Unguided Visual Question Answering
We use the term unguided to denote the con- ventional visual question-answering task with full fine-tuming without using any additional guidance. Here, we train a model with the image img, question question, and multiple answer choices U1...., an to predict the correct answer choice ag- We obtain the image-text matching score for each (image, question, answer) triplet from CLIP BLIP-2. We then normalize the scores with softmax layer across the n choices:
3.4 Scene Graphs and Object Detection
We use the Relation Transformer (RelTR) network network (Cong et al.. 2023) for scene graph gener- ation from the images. ReITR uses transformer model based on DETR (Carion al., 2020) pre- trained on the Visual Genome dataset (Krishna et al., 2017) to predict the scene graph triplets about an image. We use the ReITR model to pre- dict scene graphs for the images. We concatenate the triplets with [SEP] token to create the scene graph guidance. We also use the UniDet (Zhou et al., 2022) and DETR models for object detec- tion in the images. After detection, we aggregate the objects using their counts: two dogs, one girl, three toys. The objects detected are potentially helpful for questions that require counting or nu- merical reasoning.
lz4 {q,a;} =1=1....
8; match(img, lxl,)
(5)
5; Σラ=1
Finally, we use the cross-entropy loss to train the underlying CLIP or BLIP-2 model. Assuming that as is the correct answer, the loss is as follows:

---

3.5 Language Guided Visual Question Answering
CLIP
BLTP-2
Mode
Overall
Easy
Hard
Overall
Easy
Hard
Zero-Shot
58.52
58.98
51.43
64.98
65.95
50.00
We use the notation guide to denote the various guidance strategies their combinations. Con- cretely it could be either the rationales, explana- tions, captions, scene graphs, objects, or any of their combinations. The combination achieved by simply concatenating the various guidances. We use the notation txt__gidde to denote the con- catenation of (question, 04, guide). We follow two strategies for using the guide in CLIP or BLIP-2. For CLIP, the strategy for com- puting the matching score is similar to Eq. (5) with text_ guiddg
No Guidance
68.30
68.74
61.43
75.02
76.09
IG-VQA
Rationale
74.11
74.72
64.71
76.77
77.77
61.43
Explanation
68.
68.93
57.14
76.16
77.02
62.86
Captions
69.08
69.67
60.00
76.68
71.77
60.00
Scene Graph
68.03
66.23
65.71
76.61
77.5 51
62.86
Ohjects
67.77
55.71
75,94
77.07
All
75.98
76.65
65.71
80,47
70.00
Table I: Results the A-OKVQA validation set. We report VQA accuracy scores for zero-shot, unguided, and various kinds of guidance for the overall, easy, and hard sets, Scores average of three
We also experiment with the ScienceQA (Lu et al., 2022), Visual Semantic Reasoning (VSR) (Liu al., 2023), and IconQA (Lu al.. 2021) datasets. ScienceQA is collected from elementary and high school science curricula containing 10k questions with image context. The number of answer choices varies between 2 and 5, among which only one is correct. Visual Spatial Reasoming (VSR) is a corpus of caption- image pairs with true/false labels. Each caption describes the spatial relation of two individual objects in the image. The task is to determine if the caption correctly describes the image. We use the true/false labels as the two answer choices for multi-choice question answering. IconQA is a dataset for diagram understanding and cognitive reasoning in real-world diagram word problems. We use the multi-text-choice subtask which has a total of 31k instances.
8; match(img, tat_ aut_g)dde;)
Then, we use the softmax normalization with enoss-yn loss specified in Eq. (6) for train- ing. CLIP was originally trained maximum length of 77 input tokens. We extend the maxi- mum length to 512 tokens to incorporate guide as part of the input and update these positional em- beddings during training.
For BLIP-2, we found that passing trl, and txt guide, through two forward passes followed by feature merging to be the optimal strategy:
1 = QBLIP (OLIP(I,Lxl4.00)
I2 = QBLIP txt (I,LdiL_guideee,,, q)
f features = 11,, T2, 11 - x2, # x22
= Proj{featuren
The model is trained with the cross-entropy loss with softmax normalized scores over the choices. We train the Q-Former of BLIP-2, the projection layer Proj and keep the visual encoder frozen. We found that fine-tuning the visual en- coder is computationally expensive, with marginal improvement in performance.
Results for A-OKVQA We report the results for the overall set and the easy, hard subsets for A-OKVQA in Table 1. In the first two rows of the Table 1, we report the results in the baseline set- tings: i) zero-shot performance using the image- text matching scores without any training ($3.1), and ii) training the models the conventional VQA setting without any guidance ($3.2). We ob- tain a zero-shot overall accuracy of 58.52% which improves to 68.30% when trained without guid- ance for the CLIP model. In comparison the BLIP- model shows zero-shot accuracy of 64.98% and without guidance accuracy of 75.02%.
4 Experiments
Dataset Used We use the multi-choice ques- tion answering setup of the A-OKVQA (Schwenk et al., 2022) dataset for evaluating our VQA framework. Each instance has four possible an- swer choices, among which only one is correct. We report the scores in the validation dataset as the test set labels are not available. We also report results for the easy and hard subset as specified in the dataset. The dataset has a total of 25K in- stances.
The lower part of Table shows the results with various kind of guidances. Most of the guidances improve performance when used individually on their own. For individual guidance, we obtain the highest performance with rationales for both CLIP

---

ScienceQA
VSR
IconQA
Model
Mode
What
Which
Why
How
Where
Mode
CLIP
BLIP-2
CLIP
BLIP-2
CLIP
BLIP-2
Zero-Shot
60.14
53.13
55.74
45.00
66.0
No Guidance
85.32
84.28
63.99
76.76
82.36
86.21
No Guidance
68.96
56.00
77.05
58.33
86.0
CLIP
LG-VQA
Rationale
75,60
64.00
90.16
63.33
86.0
Captions
86.22
84.88
64.10
76.68
83,98
86.47
76.40
88.52
Scene Graph
86.32
63.75
77.00
82.54
86.18
Zero Sbot
66.67
72.13
Objects
87.22
85.28
63.95
77.40
83.25
86.52
CSO
86.02
86.27
78.15
84.44
BLII--2
No Guidance
75.49
60.00
88.52
65.00
90.0
Lecture
86.92
86.27
Rationale
77.32
65.33
86.89
63.33
90.0
CSOL
86.37
86.56
All
80.53
66.67
90.16
70,00
94.0
Table 2: Results on the ScienceQA, VSR, and IconQA test sets. CSO denotes captions, scene graphs, and objects guid- ance. The CSOL denotes additional lecture guidanee. The lecture guidance is available only for the ScienceQA dataset. Scores are average of three runs.
Table 3: Results of CLIP and BLIP--2 with different guid- ance across question types. Scores are average of three runs.
Analysis For A-OKVQA, we analyze the per- formance of our models across various wh- question types in Table 3. In A-OKVQA, 'What' type questions are the most prevalent type. The incorporation of all guidance helps both CLIP and BLIP-2 to improve on the 'What' question types in addition to all the other question types. For CLIP, the rationale-based guidance outperforms the guidance across all the question types. Notably, the 'Why' accuracy jumps to 90% from 77%.
and BLIP-2. In particular, the improvement over no guidance is close 6% for CLIP, which is also somewhat expected as the rationale generator BLIP-2 model trained on the A-OKVQA train- ing set itself. The performance with all the guid- ances is significantly better compared to no guid- ance for both models, with 7.6% improvement in CLIP and 4.8% improvement in BLIP-2 in the overall set. We also report the scores for the easy and hard subsets in A-OKVQA. We note that the improvement in all guidance setting comes from improvement in both the easy and hard subsets. Notably, BLIP-2 achieves a hard set accuracy of 70% with all guidance compared to 58.57% with no guidance.
Error Analysis We found some common error patterns from our analysis: (i) questions about very small or tiny objects present in the image are erroneously predicted as the visual encoders of CLIP or BLIP-2 are not sensitive enough de- tect those objects; (ii) questions about optical char- acters embedded the image are often wrongly predicted (which number birthday is being cele- brated? for a cake with the number 10 drawn with cream); (iii) questions about abduction (Given the position of the bat and ball the batter most likely did what?) visual occlusions (What is likely in front of the rug?) are also incorrectly predicted because of insufficient world knowledge.
Results for other datasets We report results for the other three datasets in Table 2. For ScienceQA without guidance the CLIP model achieves an ac- curacy of 85.32% which is a percent higher than the BI.IP-2 accuracy of 84.28%. We observe 1-2 increase in accuracy with the guidances for both models. The objects guidance in CLIP provides the maximum accuracy of 87.22%.
Conclusion
In this work, we presented simple method for vi- sual question answering with language guidance. Our language guidance consists of rationale and explanations constrained upon the image and the question, image captions, scene graphs, and de- scription of the objects present in the image. We propose an effective strategy of fusing the lan- guage guidance with the pre-trained multi-modal CLIP and BLIP-2 models. Our language guidance improves the performance of CLIP by 7.6% and BLIP-2 by 4.8% in the challenging A-OKVQA dataset. We also observe consistent improvement in performance with guidance on the ScienceQA, VSR, and IconQA datasets.
In VSR, the BLIP-2 models heavily outperform the CLIP models. We observe a minor improve- ment in performance for CLIP with the various guidances. For BLIP-2, the improvement is close to 1.5%, when use the captions, scene graphs, and objects (CSO) guidance together.
We observe a similar trend in performance for IconQA. The use of various guidances helps in im- proving the accuracy of both models. However, the margin of increase is not as high in BLIP-2 as it is in CLIP. We observed 2% increase in accu- racy for CLIP and 0.5% in increase in accuracy for BLIP-2.

---

6 Limitations
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, al. 2022. Pali: A jointly-scaled mul- tilingual language-image model. arXiv preprint 2rX9v:2209.000 00794 (2022).
As mentioned before, our proposed model under- performs for specific categories of questions where tiny object recognition or optical charac- ter recognition the images is necessary for an- swering the questions. We also found that the baseline models CLIP and BLIP-2, and our sub- sequently developed models with guidance can- not adequately model complex world knowledge and commonsense knowledge for answering dif- ficult questions about the image. In the future, we would like to incorporate explicit world and commonsense knowledge as part of the guidance to improve the VQA models.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language mod- els. arXiv preprint arXiv:2210. 11416 (2022).
Yuren Cong. Michael Ying Yang, and Bodo Rosen- hahn. 2023. Reltr: Relation transformer for scene graph generation. IEEE Transactions on Puttern Analysis and Machine Intelligence (2023).
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.
7 Acknowledgments
We thank the anonymous reviewers for their con- structive feedback. We also thank Anubhav Jan- gra for providing feedback on the paper. This project is supported by the AcRF MoF Tier-2 grant (Project no. T2MOE2008, and Grantor ref- erence no. MOE-T2EPP0220-0001 titled: "CSK-
Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image under- standing in visual question answering. In Proceed- ings of the IEEE conference on computer vision and pattern recognition. 6904-6913.
NLP: Leveraging Commonsense Knowledge for NLP", and the SRG grant id: TISRIS19149 titled "An Affective Multimodal Dialogue Sys- tem". This project was partially funded by a grant from the Michigan Automotive Research Center ("ARC"). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Michigan Automotive Research Center.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- Kenji Hata. Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li. David Shamma, al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International journal of computer vision 123 (2017), 32-73.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint 3xXiv22301.1259) (2023).
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Ziticck and Devi Parikh. 2015. Vqa: Visual question an- swering. In Proceedings of the IEEF international conference on computer vision. 2425-2433.
Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. 2018. Vqa-e: Explaining, elaborating, and en- hancing your answers for visual questions. In Pro- ceedings of the European Conference on Computer Vision (ECCV). 552-567.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer EsionnEECV222 2020:
Fangyu Liu, Guy Emerson, and Nigel Collier. 2023.
Visual spatial reasoning. Transactions of the As- sociation for Computational Linguisties 11 (2023), 635-651.
16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. Springer, 213- 229.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint 171 (2017).
Xinlei Chen, Hao Fang, Tsung-Yl Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015, Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:7504.00325 (2015).
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Pe- Clark, and Ashwin Kalyan. 2022. Learn to Ex- plain: Multimodal Reasoning via Thought Chains

---

for Science Question Answering. In The 36th Con- ference on Neural Information Processing Systems (NeurIPS).
and external knowledge. IEEE transactions on pat- tern analysis and machine intelligence 40, (2017), 1367-1381.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song- Chun Zhu. 2021. IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Lan- guage Reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks.
Wenfeng Zheng. Lirong Yin, Xiaobing Chen, Zhiyang Ma, Shan Liu, and Bo Yang. 2021. Knowledge base graph embedding module design for Visual question answering model. Pattern recognition 120 (2021), 108153.
Xingyi Zhou, Vladlen Koltun, and Philiip Krähenbühl. 2022. Simple multi-dataset detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition. 7571-7580.
Dong Huk Park. Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schicle, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing the evidence. In Proceedings of the IEFE conference on computer vi- sion and pattern recognition.
A Experimental Details and Resources
We train our models with the AdamW opti- mizer (Loshchilov and Hutter, 2017) with a batch size of 8 and epochs. We used a learning rate be- tween [1e-6, 3e-6, 5e-6). We train our models on single RTX A6000 GPU. The main task training on A-OKVQA takes around 4 hours for CLIP and 8 hours for BLIP-2 with 8 epochs.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In Inter- national conference machine learning. PMLR, 8748-8763.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Lan- guage models are unsupervised multitask learners. (2019).
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-OKVQA: A Benchmark for Visual Ques- tion Answering Using World Knowledge. In Com- puter Vision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Pro- ceedings, Part VIII. 146-162.
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. Kvqa: Knowledge- aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence. Vol. 33. 8876-8884,
Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b. Ofa: Unifying ar- chitectures, tasks, and modalities through simple sequence-to-sequence learning framework. Inter- national Conference on Machine Learning. PMLR, 233400.
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subho- jit Som, et al. 2022a. Image as foreign language: Beit pretraining for all vision and vision- languag tasks. arxiv preprint 2r02 10442 (2022).
Qi Wu, Chunhua Shen, Peng Wang, Anthony Dick, and Anton Van Den Hengel. 2017. Image captioning and visual question answering based on attributes


Reviews:
[{"review_id": "Hkpqw1sZDM", "reviewer": "Reviewer_ssgE", "summary": "", "questions": "How many parameters/layers are tuned in the BLIP when fine-tuning to generate rationales?\nLook at the questions above. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review of our work. Please find the responses below:\n\n**Reject 1:** Yes, the rationales generated using the AOK-VQA dataset are key towards the improvement in performance. We have also tried using all the other components except the rationale as the guidance. We achieved the following results:\n\n| Mode                |  CLIP | BLIP-2 |\n|---------------------|-------|--------|\n| No Guidance         | 68.30 |  75.02 |\n| All except Rationale| 71.28 |  78.43 |\n| All                 | **75.98** |  **79.83** |\n\nWe observe around 3% improvement over the baseline if we use all components except the rationales for guidance. For other VQA datasets, the explanations as annotated in Li et al., 2018, and Park et al., 2018 (line 195) could be as crucial as the guidance. We will benchmark the VQA v2 dataset and report the results in the updated draft of our work.\n\n**Reject 2:** We tested the frozen InstructBLIP model to generate the rationales in a zero-shot setting and found that it is substantially less suitable for the task. We achieved a ROUGE-L score of 42.9 w.r.t to the gold rationales with our trained rationale generator model in the paper. In comparison, the ROUGE-L score is 24.2 for the InstructBLIP zero-shot setting.\n\n**Reject 3:** We couldn’t add them to the paper due to page limitations. We will add examples and visualizations in the paper, given the additional page if the paper is accepted.\n\nSome examples:  \n**Image**: https://cocodataset.org/#explore?id=163640  \n**Question**: What type of pants is the man on the right wearing?  \nChoices: [linen, corduroy, silk, denim]  \nCorrect Answer: denim  \n**Guidances**:  \nCaptions: men posing in front of a door holding a snack  \nObjects: three person, one bowl, one cabinet/shelf, one hat, one picture or frame, one plate  \nRationales: the man is wearing a pair of blue jeans  \n\n**Image**: https://cocodataset.org/#explore?id=182202  \n**Question**: What type of device is sitting next to the laptop?  \nChoices: [mouse, mobile phone, pen, keyboard]  \nCorrect Answer: mobile phone  \n**Guidances**:  \nCaptions: a silver laptop on a desk  \nObjects: two mobile phone or cell phone, one computer keyboard, one laptop  \nRationales: there is a silver mobile phone on the desk next to the laptop  \n\n**Question 1:** The visual encoder and the language model are kept frozen for the rationale generation task. We only train the Q-Former component of the BLIP-2 model (lines 196-201). This results in a total of ~107 million trainable parameters.\n\n**Missing References:** Thanks for the pointers. We will include them in the related work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Shows evidence in experiments that the extra context helps improve performance for VQA.\n2. Nice ablations are done with the type of guidance and the type of questions. ", "reasons_to_reject": "1. The approach requires rationale supervision, which is expensive and might not scale. Authors train and test on the A-OK-VQA dataset, so it is unclear how the rationales trained on that scale on other VQA datasets.\n2. Related to the above, it would be interesting to see how this works if you do not train the rationale generation on the OK-VQA dataset. Is there enough knowledge the frozen VLMs to generate a rationale zero-shot that helps for the task?\n3. Some qualitative results of the kind of guidance the BLIP-2 would be nice. \n", "questions_to_authors": "How many parameters/layers are tuned in the BLIP when fine-tuning to generate rationales?\nLook at the questions above. ", "paper_topic_and_main_contributions": "The authors use contextual guidance in addition to the image and text to improve performance on VQA that requires outside knowledge to answer. The contextual guidance comes from VLM models tuned on rationales for answering the question in the OK-VQA dataset and from scene graphs predicted from the images. They train the pipeline on the A-OKVQA dataset. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Minor: line 234 - \"text_guide\" and \"txt_guide\" mismatch in equation 7, and the description above it\n"}, "extra_text": {"Missing_References": "Some relevant related work that might be good. to look at:\n- https://openaccess.thecvf.com/content/CVPR2021/papers/Marino_KRISP_Integrating_Implicit_and_Symbolic_Knowledge_for_Open-Domain_Knowledge-Based_VQA_CVPR_2021_paper.pdf \n- https://arxiv.org/abs/1909.04696 -EMNLP 2019\n- https://arxiv.org/abs/2001.06927 - CVPR 2020"}}, {"review_id": "7XQt03X3mi", "reviewer": "Reviewer_PMF9", "summary": "", "questions": "1. I want to see if the caption of the image can improve the VQA accuracy, and the compared result with guidance generated by BLIP2.\n2. If you ask the model step by step, for example, you first ask the model to generate captions and then ask the model to answer the question, if the accuracy can be improved?\n3. Why did you choose CLIP as your model to generate guidance, as far as I know, CLIP is not a model designed for text generation.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review of our work. Please find the responses below:\n\n**Reject 1:** Yes, we will add examples and visualizations in the paper, given the additional page if the paper is accepted. We couldn’t add them to the original submission due to space constraints.\n\nSome examples:  \n**Image**: https://cocodataset.org/#explore?id=163640  \n**Question**: What type of pants is the man on the right wearing?  \nChoices: [linen, corduroy, silk, denim]  \nCorrect Answer: denim  \n**Guidances**:  \nCaptions: men posing in front of a door holding a snack  \nObjects: three person, one bowl, one cabinet/shelf, one hat, one picture or frame, one plate  \nRationales: the man is wearing a pair of blue jeans  \n\n**Image**: https://cocodataset.org/#explore?id=182202  \n**Question**: What type of device is sitting next to the laptop?  \nChoices: [mouse, mobile phone, pen, keyboard]  \nCorrect Answer: mobile phone  \n**Guidances**:  \nCaptions: a silver laptop on a desk  \nObjects: two mobile phone or cell phone, one computer keyboard, one laptop  \nRationales: there is a silver mobile phone on the desk next to the laptop \n\n**Reject 2:** Thanks for the suggestion. We will add the results of more baseline models on the Visual7W dataset in our updated draft. Due to computational constraints and the time limitation of the rebuttal period, we are unable to provide the results now. We promise to include the results on Visual7W in the camera ready.\n\nHowever, we experimented on some other smaller datasets -- **ScienceQA, Visual Semantic Reasoning (VSR), and IconQA** --- with captions, scene-graph, and objects (CSO) guidance. The ScienceQA dataset also provides a lecture for each question that can be used as a guidance. The test set accuracies are as follows:\n\n| Dataset and Guidance |  CLIP | BLIP-2 |\n|----------------------|-------|--------|\n| Science QA           |       |        |\n|    No Guidance       | 85.32 |  84.28 |\n|    Lecture           | 86.02 |  86.12 |\n|    CSO               | 86.02 |  86.27 |\n|    CSO + Lecture     | **86.37** |  **86.56** |\n|----------------------|-------|--------|\n| VSR                  |       |        |\n|    No Guidance       | 63.99 |  76.76 |\n|    CSO               | **64.24** |  **78.15** |\n|----------------------|-------|--------|\n| IconQA               |       |        |\n|    No Guidance       | 82.36 |  86.21 |\n|    CSO               | **84.44** |  **86.72** |\n|----------------------|-------|--------|\n\n\n**Question 1:** We are unsure what you meant. In the 6th row of Table 1, we have already provided the results when only the caption is used as the guidance. The VQA accuracy is improved when the caption is used for both CLIP and BLIP-2.\n\n**Question 2:** We have only used the (image, text) matching contrastive CLIP and BLIP-2 models as the main VQA model in our paper. We agree with you that the caption generator BLIP-2 model can also be used to generate both the caption and the answer. We think that this setting could be more useful for generative QA tasks, as opposed to the multi-choice QA tasks that we have reported in the paper. \n\nWe have tried a very similar setting with the InstructBLIP Vicuna model where the guidance (including captions) are provided in the input to the model. The model sees the (image, question, caption + other guidances) and generates the answer. As Vicuna is a decoder-only model, this is equivalent to your proposed setup of generating the caption and then using the caption to generate the answer.\nWe found that the InstructBLIP model is significantly outperformed by the contrastive BLIP-2 models on all the datasets:\n\n| Dataset              |  InstructBLIP Generative | BLIP-2 Contrastive |\n|----------------------|--------------------------|--------------------|\n| AOKVQA               |          77.40           |    **79.83**       |\n| Science QA           |          62.29           |    **86.20**       |\n| VSR                  |          64.52           |    **78.15**       |\n| IconQA               |          55.19           |    **86.72**       |\n\n\n\n**Question 3:** We have not used CLIP for guidance generation. The different guidelines are generated using the various generative models as specified in Sections 3.3 and 3.4.\n\nThe CLIP model is only used for the main VQA task (Table 1), where we assume that the different guidelines are provided to it. We show that the CLIP model can effectively use the guidance to improve the VQA accuracy.\n\nAll the new reported scores are an average of three runs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper uses guidance to improve VQA accuracy.", "reasons_to_reject": "1. Visualization results should be provided.\n2. More baseline models (such as ofa, flamingo) and datasets (such as visual 7w) should be tested.", "questions_to_authors": "1. I want to see if the caption of the image can improve the VQA accuracy, and the compared result with guidance generated by BLIP2.\n2. If you ask the model step by step, for example, you first ask the model to generate captions and then ask the model to answer the question, if the accuracy can be improved?\n3. Why did you choose CLIP as your model to generate guidance, as far as I know, CLIP is not a model designed for text generation.", "paper_topic_and_main_contributions": "This paper proposes a multimodal framework that uses language guidance generated by Sota models to answer questions more accurately. The most contribution of this paper is to prove that VQA accuracy can be improved if more information is provided.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xn485vB7GW", "reviewer": "Reviewer_WdmU", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review of the work. Please find the responses below:\n\n**Results on other datasets:**\n\nWe experimented on some other datasets -- **ScienceQA (with image context subset), Visual Semantic Reasoning (VSR), and IconQA**. We used the captions, scene graphs, and objects (CSO) as guidance. The ScienceQA dataset also provides a lecture context that can be used as additional guidance. The test set accuracies are as follows:\n\n| Dataset and Guidance |  CLIP | BLIP-2 |\n|----------------------|-------|--------|\n| Science QA           |       |        |\n|    No Guidance       | 85.32 |  84.28 |\n|    Lecture           | 86.02 |  86.12 |\n|    CSO               | 86.02 |  86.27 |\n|    CSO + Lecture     | **86.37** |  **86.56** |\n|----------------------|-------|--------|\n| VSR                  |       |        |\n|    No Guidance       | 63.99 |  76.76 |\n|    CSO               | **64.24** |  **78.15** |\n|----------------------|-------|--------|\n| IconQA               |       |        |\n|    No Guidance       | 82.36 |  86.21 |\n|    CSO               | **84.44** |  **86.72** |\n|----------------------|-------|--------|\n\nThe scores are an average of three runs.\n\n**Results of other baselines:**\n\nWe performed more experiments with the InstructBLIP Vicuna-7B model and found the results as specified below:\n\nAOKVQA: We were unable to reproduce the original results reported in the InstructBLIP paper with the Vicuna-7B model. However, in our experimental setup, the use of guidance improved the results by 4.6%.\n\n| Setup                    |  InstructBLIP Vicuna 7B |\n|--------------------------|-------------------------| \n| Original Reported Result |         75.7            |\n| Our Reproduced Result    |         72.8            |\n|--------------------------|-------------------------|\n|   + All  Guidance        |         **77.4**        |\n|--------------------------|-------------------------|\n\n\nWe also benchmarked InstructBLIP Vicuna-7B on ScienceQA (with image context subset), Visual Semantic Reasoning (VSR), and IconQA datasets. We were not able to exactly reproduce the zero-shot results on these datasets as originally reported in the InstructBLIP. In Science QA we obtained slightly lesser scores than what was originally reported and in VSR, IconQA we obtained higher scores than what was originally reported. Instruct-BLIP used a restricted vocabulary method for calculating the log-likelihood of the answer candidates for the multi-choice question answering tasks. However, some of the key details were unclear to us (restriction over single or multiple tokens, use of temperature, etc.) We thus had our own assumptions which might explain the difference in the reproduced results.\n\nIn ScienceQA instances, a lecture context is provided that can be helpful to answer the question. The original results and our reproduced results use this lecture context to obtain accuracies of 60.50 and 59.79, respectively. \n\nIn the no guidance setup, we do not use this lecture which results in a drop of accuracy to 54.98. After using the captions, scene-graph, and objects (CSO) guidance the zero-shot accuracy increases to 55.94. Additionally, if we fine-tune only the Q-Former and keep the rest of the model frozen, then the accuracy improvement is observed to be over 5%. **Note that we do not use the Lecture context at all in the CSO guidance setup.** So a steady and consistent performance improvement is observed using our proposed guidance strategies.\n\n| Dataset and Guidance  |  Zero-Shot | QFormer FineTune |\n|-----------------------|------------|------------------|\n| Science QA            |            |                  |\n|  Originally Reported  |    60.50   |       n/a        |\n|  Our Reproduced Result|    59.79   |       62.05        |\n|  No Guidance          |    54.98   |      57.09       |\n|  CSO Guidance         |    **55.94**   |      **62.29**       |\n|-----------------------|------------|------------------|\n| VSR                   |            |                  |\n|  Originally Reported  |    54.30   |       n/a        |\n|  Our Reproduced Result|    59.35   |      60.06       |\n|  CSO Guidance         |    **62.58**   |      **64.52**       |\n|-----------------------|------------|------------------|\n| IconQA                |            |                  |\n|  Originally Reported  |    43.10   |       n/a        |\n|  Our Reproduced Result|    51.12   |      54.68       |\n|  CSO Guidance         |    **52.91**   |      **55.19**       |\n|-----------------------|------------|------------------|\n\nFor VSR and IconQA, we observe that the CSO guidance improves over our reproduced results (which is the setup without using any guidance). We observe this improvement in performance in both the zero-shot and QFormer finetune settings.\n\nAll the reported scores are an average of three runs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed language guidance improves two popular VQA baselines, CLIP and BLIP-2, for the recent A-OKVQA benchmark.", "reasons_to_reject": "1. The technical novelty of this work is unclear. \n(1) The proposed idea sems to simply supplement the outputs of existing techniques (e.g. RelTR for scene graphs and UniDet for detected objects) into the text input of VQA models, CLIP and BLIP-2.\n(2) It is not a surprisingly novel idea in recent popular uses of zero-shot models. \n\n2. Experimental evaluation is highly limited.\n(1) This work deals with only two baselines, CLIP and BLIP-2. Admittedly, they are two of the most popularly used vision and language models. However, in order to clearly justify the effectiveness of the proposed guidance, more thorough empirical results with more baselines are required. \n(2) This work does not compare with SOTA models for the A-OKVQA benchmark. For example, the proposed guidance can be applied to InstructBLIP, and see whether it improves InstructBLIP too. \n(3) Only a single multi-choice question-answering dataset (i.e., A-OKVQA) is tested. It is limited to conclude that the proposed idea works generally. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This work contributes a way of infusing language guidance into knowledge-based VQA models, using various information of rationales, image captions, and scene graphs. \n\n2. The proposed language guidance improves the performance of CLIP by 7.6% and BLIP-2 by 4.8% in the A-OKVQA dataset.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on knowledge-augmented VQA, which requires common sense knowledge and reasoning among VQA problems. The paper aims to improve the performance of VQAs by providing a variety of relevant knowledge in the form of language guidance; performance gains have been observed when language guidance is introduced for CLIP and BLIP-2-based baselines.

Pros:
Introducing a variety of relevant knowledge in the form of language guidance is a convincing approach.
Performance improvements have been observed.

Cons: 
As two reviewers questioned, reviewer PMF9's comments include a misunderstanding, but there are some questions about whether the baseline should only be CLIP and BLIP-2.
As reviewer WdmU points out, there are several baselines and experimental settings, and the range of effects this study demonstrated is limited. 
Although Rebuttal shows experimental results for different data sets, it is not a justification for the baseline setting.
CLIP is not good at focusing on the finer points of an image and can only understand the broad features represented in the image. A more detailed discussion is needed regarding what type of instruction is effective. Also, authors may discover cases where the proposal would work better by using a different VQA model (e.g., T5-based) that can capture different image features.
Decision: Accept-Findings