A Simple Baseline for Knowledge-Based Visual Question Answering
Authors: ALEXANDROS XENOS
Themos Stafylakis
Ioannis Patras
Georgios Tzimiropoulos
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge-based Visual Question Answering (KB-VQA)
Commonsense Reasoning
Few-shot learning

Abstract: This paper is on the problem of Knowledge-Based Visual Question Answering (KB-VQA).
Recent works have emphasized the significance of incorporating both explicit (through external databases) and implicit (through LLMs) knowledge to answer questions requiring external knowledge effectively. A common limitation of such approaches is that they consist of relatively complicated pipelines and often heavily rely on accessing GPT-3 API. Our main contribution in this paper is to propose a much simpler and readily reproducible pipeline which, in a nutshell, is based on efficient in-context
learning by prompting LLaMA (1 and 2) using question-informative captions as contextual information. Contrary to recent approaches, our
method is training-free, does not require access to external databases or APIs, and yet achieves state-of-the-art accuracy on the OK-VQA and A-OK-VQA datasets. Finally, we perform several ablation studies to understand important aspects of our method. Our code is publicly available at https://github.com/alexandrosXe/ASimple-Baseline-For-Knowledge-Based-VQA

A Simple Baseline for Knowledge-Based Visual Question Answering
Alexandros Xenos Themos Stafylakis 2.3 Ioannis Patras! Georgios Tzimiropoulos'
'Qeen Mary University of London
2 Athens University of Economics and Business 3millia Conversational Intelligence, Athens, Greece
(a xenos,nosssss i. patras, twwmiroooouuaee 3 ac. uk, istafylakis@aueb..g
Abstract
Recently, several works on KB-VQA (Gui al., 2022; Lin al., 2022) have emphasized the signif- icance incorporating both explicit and implicit knowledge. However, such approaches usually re- quire complicated pipelines. Firstly, KB (e.g, wikidata) covering world knowledge needs to be maintained and used for knowledge retrieval which is time-consuming and very sensitive to noise. Sec- ondly, powerful LLMs such as GPT-3 (Brown al., 2020) or OPT-175B (Zhang al., 2022) are lever- aged due to the huge amount of implicit knowl- edge stored in their parameters and their powerful reasoning capabilities through few-shot in-context learning. However, the computational or even ac- tual monetary cost (e.g. cost for API access) as- sociated with accessing such models renders them unaffordable for many researchers. Thirdly, it is crucial to train fusion mechanism that can effec- tively reason by combining the retrieved explicit and implicit knowledge.
This paper is on the problem of Knowledge- Based Visual Question Answering (KB-VQA). Recent works have emphasized the significance of incorporating both explicit (through external databases) and implicit (through L.LMs) knowl- edge answer questions requiring externl knowledge effectively. A common limitation of such approaches that they consist of rela- tively complicated pipelines and often heavily rely on accessing GPT-3 API. Our main contri- bution in this paper is to propose much sim- pler and readily reproducible pipeline which, a nutshell, is based on efficient in-context learning by prompting LLaMA (I and 2) using question-informative captions as contextual in- formation. Contrary to recent approaches, our method is training-free free, does not require access to external databases or APIs, and yet achieves state-of-the-un accuracy on the OK-VQA and A-OK-VOC VQA datasets. Finally, we perform sev- eral ablation studies to understand important aspects of our method. Our code is publicly available at Simple-Base Baseline- Boo--Knwledgge-Ban Basege---aaddd--e
Main contributions: We present a simple yet powerful pipeline for KB- BB-VQA which by-passes the need for using most of the components of the above-mentioned systems. Specifically, the pro- posed system is simply based on few-shot prompt- ing of LLaMA-13B (Touvron al., 2023a,b). The key component of our method is the implementa- tion of effective in-context learning using question- informative captions as contextual information which, as we show, results in large accuracy boosts.
1 Introduction
Knowledge-based VQA (KB-VQA) is a recently introduced VQA task (Wang et al., 2017, 2018; Marino et al., 2019; Shah et al., 2019) where the im- age alone is not sufficient to answer the given ques- tion, but effective utilization of external knowledge resources additionally required. To solve such a task, model would need not only strong visual per- ception but also reasoning capabilitics while also being able to effectively incorporate world knowl- edge from external KBs (e.g. Wikipedia, etc) and LLMs.. Systems capable of answering general and diverse questions about the visual world find wide range of applications: from personal assistants to aids for the visually impaired and robotics !
The proposed system features several advan- tages: (1) it is entirely training-free, requiring only few examples for in-context learning; (2) it is based on the open-source LLaMA-13B (Tou- al., 2023a,b) (considerably smaller than the widely-used GPT-3): (3) it is straightforward to produce; and (4) achieves state-of-the thet--rrt (SOTA) accuracy on the widely-used OK-V OK-VQA (Marino et al., 2019) and A-OK-VOL VQA datasets (Schwenk et al., 2022).
*Corresponding author.
hhrps//wwww.delaide htt..a/aii////oorreeeeee///eccie htwp:////wguuiiiiionnnaaaaanniie

---

2 Related Work on KB-VQA
vious work (Yang al., 2022: Gui et al., 2022: Lin et al., 2022) we leverage the open-source LLM LLaMA-13B (Touvron et al., b023a,b) instead of GPT-3 as an implicit language knowledge base and treat VQA as an open-ended text generation task. Our method builds upon the pipeline of PICa, which is the pioneering work that utilizes GPT-3 for few-shot in-context learning in order to address the KB-VQA task. GPT-3 is decoder-only au- toregressive LLM of 175B parameters, trained on a diverse range of data sourees, including Common Crawl, webtexts, books, and Wikipedia (Brown et al., 2020). During inference, in-context few-shot learning involves formulating a novel downstream task as text sequence generation task using the frozen GPT-3 model. When provided with a testing imput æ, the target is predicted based on format- ted prompt p(h, C, E, c, 2r). In this prompt, h repre- sents a prompt head or instruction that describes the task, while E {er, C22 ..., en) represents a set of n in-context examples (shots), where e; = (xx, Mk) represents an input-target pair of the task, where x; and are the input and target, respectively. These pairs are constructed manually or sampled from the training set. C = {c1, C2 ..... Cn represents a set of generic image captions describing each o; since images cannot be inputted GPT-3. The caption for the test input is labeled as c. The tar- get y is denoted as a text sequence consisting of L tokens, expressed as y = (y' .v?,..., y²). At each decoding step L, the following conditions apply:
Methods Without LLMs: Several methods have been proposed including KRISP (Marino et al., 2021) which uses multi-modal pretrained BERT (Devlin et al., 2019), MAVEX (Wu et al., 2022) which proposes validate promising answer candidates based answer-specific knowledge retrieval, and DPR which uses pseudo-relevance labels integrated with answer generation for end- to-end training. Typically, these systems are not as competitive as the ones based on LLMs.
Methods based on LLMs: PICa (Yang et al., 2022) is the first method to adopt GPT-3 for solv- ing the KB-VQA task in few-shot manner by just providing few in-context VQA examples. Gui et al. (2022) proposed to use both implicit (i.e. GPT-3) and explicit (i.e. KBs) knowledge based on CLIP retrieval (Radford et al., 2021) which are combined by a novel fusion module called KAT (based on T5 or Bart). Lin et al. (2022) proposed to integrate local visual features and positional infor- mation (bounding box coordinates), retrieved exter- nal and implicit knowledge (using GPT-3) into transformer-bant question-answering model. Hu al. (2023) proposed PrompiCap, a novel task- aware captioning model that uses a natural lan- guage prompt to control the generation of the visual content that can be used in conjunction with GPT-3 in-context learning. Img2Prompt Guo et al. (2023) is a zero-shot VQA method that generates image- relevant exemplar prompts for the LLM. Their key insight is that synthetic question-ansee answer pairs can be generated using image captioning and question- generation techniques as in-context exemplars from the provided image. Prophet Shao et al. (2023) proposes to prompt GPT-3 with answer heuristics (answer candidates and answer-aware examples) that are encoded into the prompts to enable GPT-3 to better comprehend the task, thus enhancing its capacity.
j' ArgmaxpLLas(( y'
(1)
In order to utilize any LLM for the knowledge- based VQA task, the crucial step is to design suit- able prompts. When given question qi and an image U; as inputs, the VQA task's objective is to predict the corresponding answer a; However, since LLMs do not inherently comprehend images, becomes necessary convert the image into a caption & using a pre-existing captioning model. While SOTA pretrained captioning models have demonstrated impressive performance, they are pri- marily optimized to generate generic image cap- tions. Unfortunately, these captions often fail to capture all the specific details required to accu- rately answer given question about the image. In this work, instead of generic captions, we generate drestion-guiderd informative image captions using the Blug-and-Plas VQA (PNPVQA) framework (Tiong et al., 2022) which identifies the most re-
3 Methodology
While explicit knowledge retrieval focuses on se- mantic matching between an image and knowledge entries, it lacks implicit commonsense knowledge (c.g. Lemons are sour) which can be found in LLMs (Gui et al., 2022). LLMs are critical in ex- tracting implicit knowledge due to the vast amount of implicit information embedded in their parame- ters, and their powerful reasoning capacity through shw-shot in-context learning. Different from pre-

---

lated image patches to the question with saliency map-based interpretability technique and generates captions from these patches only.
Method
Knowledge Resources
KRISP
38.15
Wikigediaa ConcepoNer (Sooole Images
39.4
Unilied--OO (2
Mullinakal Pretraining
54
Flaningo
Muhimodal Pretraining
57.8
PICs-Full
Fresen (175))
48.0
KAT (single)
WroscntsFroeen (1758)
50.58
KAT_large (single)
Wilkdana: Prosen GPT:3
JAr_aage (cusemblle)
WissdattsPPooenn (P7:3
54.41
Cuttie st
REVIVE_ large (single)
Wilsiana-Prooee (1753)
56.6
REVIVE lagge (omsembbe)
Wilsian-FFoeen (175))
58.0
28 es Pugge-S-/e rtritng
Prophct
Frozen (1758)
61.1
Ours
Frozen LLaMA (138)
58.69
Ours MCAN
Fromz E.1 ..A (13R)
60.02
Qurs
Frozen L.J.MA (133))
59.07
Ours MCAN
Frozen LLaMA 2 (138)
61.2
Figure I: Inference-time of our method for n-shot VQA. The imput prompt to LLaMA consists of a prompt head h (blue box), n in-context examples (fea, 22; M )==)) (red boxes), and the VQA input {c,z} (green box). The answer y is produced an open-ender text gen- eration manner. In this example we use two question- informative captions per example (separated by com- mas).
Table Comparison with other methods on the OK- VQA dataset: Our method with 9 question-informative captions achieves statc-of-the-ar performance.
more examples via multi-query ensemble. In-context Example Selection tries search for the best examples for each inference-time time input x among all available examples ( (Yang et al., 2022). We consider in in-context examples that have similar question features as x. More specifically, given an inference-time question, we use BLIP's text en- coder to obtain its textual feature and compute its cosine similarity with the questions in all available in-context examples. We then average the ques- tion text similarity with the image visual similarity to guide the example selection similarly Yang et al. (2022). We select the top-n questions with the highest similarity and use the corresponding examples as the in-context examples.
For each imagc-question pair, we first gener- ate 50 question-gutie guided informative image captions from the image o using PNPVQA. We then em- ploy BLIP's (Li et al., 2022) text encoder to encode all the image captions and BLIP's image encoder to encode the image U/. We rank the image captions per image vj according to their cosine similarity with the image u, and keep the top-m most simi- lar captions G per example. After extracting the top-m most similar captions per image i; we con- struct a carefully designed text prompt consisting of a general instruction sentence, the captions C, the question, the test input's captions c, and a set of context- weatt--uuuttionnasssve triplcts (shots) taken from the training dataset that are semantically most similar to the current image-question pair (see Fig. 1). Then this text prompt is passed frozen LLaMA- 13B model and in-context few-shot Iearn- ing performed in order to obtain its output as promising answer candidate to the current image- question pair.
Multi-query ensemble: Given an inference- time example x, use k X n in-context examples to generate l prompts. This way, we prompt LLaMA- 13B for k times and obtain k answer predictions instead of 1 similar Yang et al. (2022), where the number of queries to ensemble. Finally, among the k answer predictions, we select the one with the most occurrences (majority vote).
4 Experimental Results
3.1 Selecting Informing Examples For Few-Shot In-Context Learning
Comparative results on OK-VQA: Table sum- marizes the results of various methods on OK-VQA including our best method (last row) which uses 9 question-inforrative captions and 5 query ensem- bles. When using LLaMA our approach outper- forms all methods and achieves comparable results with Prophet especially when using the same shot selection strategy based on MCAN (Yu al., 2019). Morcover, it performs better than Unified-IO and the 80B Flamingo which have been pre-trained
As Yang et al. (2022) notes, feeding more in- context examples to GPT-3 yields better fho- performance. However, the maximum input length of the model constrains the maximum number of examples n in the prompt. To better use these avail- able examples we: (i) improve the example quality by careful in-context example selection (Liu et al., 2022; Gui et al., 2022; Shao et al., 2023), and (ii)

---

Method
DA
MC
Captions
A:
Acc (%)
Test
Val
Test
Generic
14
43.35
ClipCap
30.9
25.9
56.9
51.4
Question-informative
57.56
ViLBERT
30.6
25.9
49.1
41.5
LXMERT
30.7
25.9
51.4
41.6
Table 3: Generic vs. investion-informann captions.
KRISP
33.7
51.9
42.2
GPV-2
48.6
60.3
53.7
Unified-IO
45.2
shows the performance of our method when using generic captions vs question-informatiio captions for in-context learning which is the key component of system. Following Yang al. (2022); Shao et al. (2023) we leverage the OSCAR+ (Zhang et al., 2021) as the captioning model. The results suggest using question-infoo informative captions results in huge accuracy boosts (43.35% vs 57.56%)
Prophet
58.2
55.7
59.3
57.3
Ours (LLaMA)
54.4
53.8
Ours MCAN (LLaMA)
57.4
55.0
Ours (LLaMA 2)
57.1
55.4
Ours MCAN (LLaMA
58.6
57.5
Table 2: Comparison with other methods on the A-OK- VQA dataset: Our method with guestion-inforrmtiiv captions achieves state-of-the-an performance the direct answer (DA) setting. Note that our method does not support multiple-choice (MC).
Selection Strutegy
Captions
Acc
Random
Dueeticn-infemmmiiee
14
53.19
Avg. Question and Image Sim.
Question informative
14
$6,50
MCAN lsient
Queticn-infommativ
14
57,56
with multimodal objectives. When compared to methods that rely on GPT-3 for implicit knowl- edge extraction, our approach outperforms PICa- Full which only uses generic image captions by 12.02% while outperforming the SOTA supervised methods KAT and REVIVE by 5.61% and 2.02% respectively. Finally, when using LLaMA 2 and MCAN-based shot selection strategy, our method achieves state-of-the-arr accuracy of 61.2%.
Table 4: Accuracy when using different shot selection strategies. Avg. question and image sim. strategy trieves shots based on the average cosine similarity be- tween the test sample's 's question and image, and the training examples' question and image. MCAN latent space strategy retrieves shots that are closer to the test sample in the trained MCAN's latent space.
Effect of shot selection strategy: Table 4 shows that selecting random shots during in-context learn- ing hurts the accuracy, confirming the findings of Yang al. (2022). Retrieving shots based on the similarity between the test sample and the train- ing examples yields significant accuracy boost. Prophet's shot selection strategy based on MCAN also seems to be effective but we note that is based on pre-training a vanilla VQA model on a different dataset (VQA-v2).
Comparative results on A-OK-VQA: Table summarizes the results of various methods on A- OK-VQA including our best method (last row) which uses question-informatii captions and 5 query ensembles. We compare our method to the strong baselines in (Schwenk et al., 2022) and the current state-of-the-art method Prophet (Shao et al., 2023). When employing LLaMA, our approach surpasses all other methods on the DA setting and achieves comparable results to Prophet, par- ticularly when employing the same shot selection strategy based on MCAN. Finally, with LLaMA 2 and MCAN our method attains state-of-the- art performance on both the validation and test sets, achieving 58.6% and 57.5% accuracy respectively, demonstrating the effectiveness and robust general- ization of our proposed method.
Effect of number of question-informativ captions: Fig. 2 (a) shows the accuracy when we increase the number of captions per sample in the prompt during in-context learning. Here, we are using k = 5, and n = 10 when using 1-10 captions, and n = 5 when using more than 10 captions due to max. sequence length constraints. More captions provide more information for each example helping the model to make more accurate prediction based context. As shown in the figure, the validation accuracy keeps increasing up to 60.02%. When using more than 10 captions, the accuracy decreases but this also can be attributed to the fact that we are also decreasing
5 Ablation Studies
We conduct several ablations OK- BK-VQA to better understand the key components of method.
Effect of question-informative captions: Table

---

LLMs apart from LLaMA, which presents a lim- itation of our study. Lastly, due limitations in resources, we were unable conduct experiments with larger sizes beyond 13B. However, would indeed be intriguing to observe the performance when employing LLaMA models of sizes such as 30B or 65B.
Figure 2: (a) Accuracy vs number of question informa- tive captions used per shot during few shot in-context learning. (b) Accuracy vs number of prompts & used during in-context learning.
Ethics Statement
The authors of this paper recognize the importance of responsible AI in both research and development endeavors. We are committed to ensuring that the model we develop is not only accurate but also fair and unbiased. We understand the potentially significant impact of VQA technology on society and, therefore, pledge maintain transparency by sharing our findings and progress with relevant rescarchers and stakcholders.
the number shots Effect of multi-query ensemble: Fig. 2 (b) shows the accuracy as the number of prompts, k, increases. As anticipated, employing multiple prompts of LLaMA instead of just one yields improved ac- curacy. However, beyond & = 6, the accuracy begins to fluctuate. It is important to note that this fluctuation could be attributed to the retrieval of noisy (irrelevant the question) context examples as the value of & increases.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss.. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Effeet of explicit knowledge: We also tried to use KAT's (Gui et al., 2022) KB and trained a T5 (Raffel et al., 2020) in order to integrate explicit model. For each image, used BLIP to extract explicit knowledge via image- to-text retrieval. We used 40 retrieved passages and LLaMA predictions as explicit and implicit knowledge, respectively. We achieved an accuracy of 58.70% which shows that our model does not benefit from such an approach.
knowledge into
Effect of size of LLM: We also used a LLaMA- ceessiooniiffommativ captions, n = 10 and = 5. Reducing the size of the LLM leads to decreased accuracy but the drop is not large, still obtaining 57.99% accuracy.
7B model using
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
6 Conclusions
We proposed a simple yet effective baseline for KB-VQA. Our training-free method is based on in-context few-shot learning of the open-source LLaMA using cetestionniffomaaiiv captions. We show that this is sufficient achieve SOTA re- sults on the widely used OK-VQL and A-OK-VOA datasets.
Liangke Gui, Borui Wang, Qiuyuan Huang. Alexan- der Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT: A knowledge augmented transformer for vision-and-language 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 156-968, Seattle, United States. Association for Computational Lin- guistics.
In Proceedings the
Limitations
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven C. H. Hoi. 2023. From images to textual
It is important to acknowledge that we have not explored the utilization of any other medium-sized

---

prompts: Zero-shot vqa with frozen large language models.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023, Prompting large language models with answer heuris- tics for knowledge-based visual question answering.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, and Jiebo Luo. 2023. Promptcap: Prompt-guided task-aware image captioning.
Anthony Meng Huat Tiong, Junnan Li, Boyang Li. Sil- vio Savarese, and Steven C.H. Hoi. 2022. Plug-and- play VQA: Zero-shot VQA by conjoining large pre- trained models with zero training. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 951 951-967, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified lision-language understanding and generation.
Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022. Revive: Re- gional visual representation matters in knowledge- based visual question answering.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal. Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100- 1-1111141 114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, An- thony Hartshorn. Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta. Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2021. Krisp: Inte- grating implicit and symbolic knowledge for open- domain knowledge-basee vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1411 1-14
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. -qa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Henge. 2017. Explicit knowledge- based reasoning for visual question answering. In Proceedings of the 26th International Joint Con- ference on Artificial Intelligence, IJCAP 17, page 1290-1296 AAAI Press.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Peng Wang. Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2018 Fvqa: Fact-based vi- sual question answering. IEEE Transactions on Pat- tern Analysis and Machine Intelligence,400110) 24113 2427.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answer- ing using world knowledge. In Computer Vision ECCV 2022, pages 146-162, Cham. Springer Nature Switzerland.
Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. 2022. Multi-modal answer validation for knowledge-based vqa. Proceedings of the AAAI Con- ference on Artificial Intelligence, 36(3):7112229
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. Kvqa: Knowledge- aware visual question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 3380718886-8884
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge- based vqa. 08:3081-3089

---

Zhou Yu, Jun Yu. Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co-attention networks for visual question answering. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recog- nition (CVPR).
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei 7hang, Lijuan Wang, Yejin Choi, and Jian- feng Gao. 2021. Vinvi: Revisiting visual representa- tions in vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5579-5588.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer. Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Example Appendix
A.1 Implementation Details
We used the Huggingface Transformers library? in order to run LLaMA models. We used beam search with beam size during generation and max new tokens = while using the default values for all the other parameters in the generate method. We run our model on a 40-GB VRAM A-100 GPU card.


Reviews:
[{"review_id": "YJxmn4BbPZ", "reviewer": "Reviewer_P6LA", "summary": "", "questions": "- If possible, can you provide peformance after some fine-tuning steps? This can prove wider applicability of the proposed framework.\n- [1] propose a new ensemble scheme for QA with few-shot examples. Can you provide performance of your framework equipped with idea of [1] in multi-query ensemble?\n- If I understood correctly, we may replace LLaMA-13B model in the overall process of this paper into GPT-4, which is the state-of-the-art large language model. Since, this work utilizes LLM without any finetuning, I expect that GPT-4 can show upper-bound performance which is higher than LLaMA-13B. Even though GPT-4 is black-box model, I want to see the upper bound performance of the framwork.\n\n[1] Replug: Retrieval-augmented black-box language models, Shi et al", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1.1: “If possible, can you provide performance after some fine-tuning steps? This can prove wider applicability of the proposed framework”\n\nA1.1: We have tried to finetune LLaMA using parameter-efficient techniques but we still got an OOM (out-of-memory) error because the prompts are very lengthy (1856 subtokens per average). Thank you we will discuss this in the paper.\n\nQ1.2 “Replug: Retrieval-augmented black-box language models, Shi et al,  propose a new ensemble scheme for QA with few-shot examples. Can you provide performance of your framework equipped with idea of [1] in multi-query ensemble?”\n\nA1.2: The authors of the mentioned paper haven’t made the code available yet. We didn’t have the time to reproduce the code but we will try to address this for the final version of our paper!\n\nQ1.3: “If I understood correctly, we may replace LLaMA-13B model in the overall process of this paper into GPT-4, which is the state-of-the-art large language model.”\n\nA1.3: Yes this is true, we can replace LLaMA-13B model with any Large Language Model in the overall process of the paper.\n\nQ1.4: “Since, this work utilizes LLM without any finetuning, I expect that GPT-4 can show upper-bound performance which is higher than LLaMA-13B. Even though GPT-4 is black-box model, I want to see the upper bound performance of the framwork.”\n\nA1.4: Due to the high cost of openAI’s API we were able to run experiments using Instruct-GPT-3 (text-davinci-003 model) using only k = 1(number of prompts in the multi-query ensemble) and using 9 question-informative captions. The result we got was 55.34% validation accuracy which is smaller than the respective of our LLaMA-13B model in paper 58.97%. Importantly, our work focuses on open-sourced small-medium size LLMs. \nThank you for this we will add it to the paper.\n\nQ1.5: “ Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.”\n\nA1.5: We kindly disagree. The results are 100% reproducible and we will open-source the code for this framework. The data and all the models used are 100% open-sourced.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- If I understood correctly, this work is the first LLM-based reproducible open-source VQA framework. Since VQA is one of the most practical problem in NLP era, this framework can be employed as basis for the future research on VQA. \n- The empirical results show that the proposed VQA framework achieves SOTA peformance which is comparable with existing black-box framework based on GPT-3 for OK-VQA dataset.\n- The proposed method is simple and intuitive.\n- Nice ablation results (Section 5).", "reasons_to_reject": "- I think novelty of the proposed method is not so significant. (However, it is not important for this kind of research.)\n- Please refer to questions.", "questions_to_authors": "- If possible, can you provide peformance after some fine-tuning steps? This can prove wider applicability of the proposed framework.\n- [1] propose a new ensemble scheme for QA with few-shot examples. Can you provide performance of your framework equipped with idea of [1] in multi-query ensemble?\n- If I understood correctly, we may replace LLaMA-13B model in the overall process of this paper into GPT-4, which is the state-of-the-art large language model. Since, this work utilizes LLM without any finetuning, I expect that GPT-4 can show upper-bound performance which is higher than LLaMA-13B. Even though GPT-4 is black-box model, I want to see the upper bound performance of the framwork.\n\n[1] Replug: Retrieval-augmented black-box language models, Shi et al", "paper_topic_and_main_contributions": "- The paper deal with the problem of knowledge-based visual question answering (KB-VQA).\n- The authors propose fully white-box framework for KB-VQA based on LLaMA-13B model.\n- The proposed framework achieves comparable performance with GPT-3 API-based VQA baselines and does not require any addional training steps.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V52lfFCPQV", "reviewer": "Reviewer_3YRF", "summary": "", "questions": "A. As the research of LLM advances, LLaMA series model had reached similar performance with older GPT-3, what would the method perform with the proposed prompting mechanisms applied on GPT-3 (as used in compared methods) for fair comparison?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q2.1: “Furthermore, the reviewer anticipates a comparative analysis across other KBVQA datasets, such as A-OKVQA, to broaden the evaluation of the method's applicability.”\n\nA2.1: We ran our model (without MCAN) to the newer  KB-VQA dataset A-OK-VQA and got a validation accuracy of 54.4% and a test accuracy of 53.8%. Our model again shows comparable performance to PROPHET which has a validation accuracy of 58.2% and a test accuracy of 55.7%. Thank you for this we will add it to the paper.\n\nQ2.2: “The reviewer also expects the analysis of the inability of retrieved external knowledge to improve the model performance.”\n\nA2.2: From the analysis we performed we found that adding explicit knowledge to the pipeline (through KAT) affects only a small sample of the predictions (486 out of the 5046 (9.63%) total predictions). In this small sample, the LLaMA model (implicit only) is most of the time correct while KAT is wrong (156 out of the 486 (32.1%) samples). Only in  82 out of 486 (16.87%) KAT was correct while LLaMA was wrong. Finally, in 98 out of 486 (20.16%) predictions, both models were correct with different predictions and in the last 150 out of the 486 total different predictions (30.86%), both models were wrong.\n\nQ2.3: “ As the research of LLM advances, LLaMA series model had reached similar performance with older GPT-3, what would the method perform with the proposed prompting mechanisms applied on GPT-3 (as used in compared methods) for fair comparison?”\n\nA2.3: Due to the high cost of openAI’s API we were able to run experiments using Instruct-GPT-3 (text-davinci-003) using only k = 1(number of prompts in the multi-query ensemble) and using 9 question-informative captions. The result we got was 55.34% validation accuracy which is smaller than the respective of our LLaMA-13B model in paper 58.97%. Importantly, our work focuses on open-sourced small-medium size LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduced a reproducible KB-VQA approach using a LLM (i.e., LLaMA) as knowledge source. They designed interesting caption generation and in-context demonstration selection mechanisms to improve the few-shot performance from LLM, especially on generating question-related captions. Despite the manuscript's concise length, the conducted experiments are thorough, effectively substantiating the efficacy of the devised mechanisms. Remarkably, the performance proves competitive even when compared to methodologies leveraging larger LLMs, such as GPT-3.", "reasons_to_reject": "While the few-shot prompting mechanisms demonstrate notable efficacy, it is worth noting their resemblance to the methodology employed in Prophet, which also utilizes a similarity-based sample selection mechanism.\nIt may be beneficial for the author to confirm the effectiveness of their approach using GPT-3, as is the case in other comparative methodologies. This could provide a more accurate assessment of the benefits gained from the proposed prompting mechanism. Furthermore, the reviewer anticipates a comparative analysis across other KBVQA datasets, such as A-OKVQA, to broaden the evaluation of the method's applicability. The reviewer also expects the analysis of the inability of retrieved external knowledge to improve the model performance.\nThe rebuttal addressed some concerns above (A-OKVQA result and GPT-3 based result). However, if LLaMA is already better than GPT-3 (on KBVQA as demonstrated in performance comparison in rebuttal, and on pure NLP tasks as demonstrated in LLaMA paper), the performance improvement might simply come from a better (although less parameters) LLM used.", "questions_to_authors": "A. As the research of LLM advances, LLaMA series model had reached similar performance with older GPT-3, what would the method perform with the proposed prompting mechanisms applied on GPT-3 (as used in compared methods) for fair comparison?", "paper_topic_and_main_contributions": "This paper focuses KB-VQA. Despite the current emphasis on using both explicit and implicit knowledge to answer questions needing external data, there are often complex pipelines and reliance on LLM API. This paper introduced a simpler, easily replicable process centered on efficient in-context learning and open-source LLaMA, negating the need for training or access to external databases or APIs. Despite this, the method maintains a competitive accuracy rate on the OK-VQA dataset. The paper concludes with several ablation studies for a deeper understanding of the method's key components.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ck44LT5eAb", "reviewer": "Reviewer_z51R", "summary": "", "questions": "What is the real algorithmic novelty of this work since it is a straightforward combination of existing methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q3.1: “Results on larger models and now LLaMA 2 would be nice, to see if this approach could actually achieve SOTA with a larger more modern model.”\n\nA3.1: When we were submitting the paper LLaMa 2 was not out yet. We were able to run LLaMA 2 now and got impressive results even surpassing the state-of-the-art model PROPHET. More specifically, using LLaMa 2 + MCAN we got 61.2% validation accuracy, while using LLaMa 2 without MCAN resulted in 59.7% validation accuracy. We will add these results to the updated version of our paper. Thank you.\n\nQ3.2: “What is the real algorithmic novelty of this work since it is a straightforward combination of existing methods?”\n\nA3:2 Our method is simple but not simplistic and it is proposed for the first time to address this particular problem. We are also the first to show that such a simple approach can be used to produce competitive results (or even SOTA results with Llama2; see A3.1) to recent SOTA approaches based on much more complicated pipelines. The method is based on in-context learning but the context is not just a generic image caption, it contains rich question-informative captions (after ranking them using text-to-image similarity) that describe the image with respect to the question. Furthermore, adding more captions per shot results in better describing the image for the LLM to perform the QA task, while encouraging diversity of captions and coverage of visual content.\n\nQ3.3 “It is odd that the KAT model is not introduced in the related work but that presented in the table of results.\nThe paper uses MCAN but does not explain or provide a reference for this technique\nThe idea of using captions and specifically question-focused captions for VQA were first introduced by papers in ACL 2019 by Kim and Bansal, and by Wu and Mooney (question specific), and it would have been nice to reference this earlier related work.”\n\nA3.3: We would like to thank you for pointing this out, we will of course address this in the updated version of our paper. Thank you again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper quite clearly presents a fairly simple new baseline system for the important problem of KB-VQA that achieves near SOTA performance on the OK-VQA standard benchmark for this task, almost tying the recent PROPHET system that uses GPT-3.  It combines a smaller open LLM (LLaMA) and a question focused captioner from the PNPVQA system.  The model is simpler and more open than current SOTA models for OK-VQA that use a closed, larger GPT-3 model.  Therefore, it provides a new more accessible and efficient model for KB-VQA.  The paper provides a series of ablation studies exploring the contribution of various aspects of the model such as the question-focusing of the captioner, few shot sample selection, the size of the few-shot training set, model size, and number of provided captions. Overall, a solid short paper on an important topic. The fact than a modest-sized model without explicit knowledge retrieval can do near-SOTA on this problem is somewhat surprising and interesting.  \n\nThe rebuttal results with LLaMA that achieve now SOTA results is a nice addition, I raised my excitement score to a 4, but if it was an option, I would have given a 3.5, since my excitement has only increased a relatively small amount based on this.", "reasons_to_reject": "The paper does not really introduce any new methods or algorithms, just combining existing methods like LLaMA and PNPVQA. So there is very little if any algorithmic novelty.  The results are close but do not actually surpass the SOTA results from Prophet.  So the only real advantages are simplicity, openness and efficiency.  \n\nResults on larger models and now LLaMA 2 would be nice, to see if this approach could actually achieve SOTA with a larger more modern model.\n\nThe paper is solid but not very novel or exciting", "questions_to_authors": "What is the real algorithmic novelty of this work since it is a straightforward combination of existing methods?", "paper_topic_and_main_contributions": "The paper presents a simpler model that relies on a smaller opensource LLM that achieves close to SOTA performance on the problem of Knowledge-Based Visual Question Answering (KB-VQA)  using few-shot learning that exploits question-focused captioning to provide image info to the LLM", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall the paper reads well"}, "extra_text": {"Missing_References": "It is odd that the KAT model is not introduced in the related work but that presented in the table of results.  \n\nThe paper uses MCAN but does not explain or provide a reference for this technique\n\nThe idea of using captions and specifically question-focused captions for VQA were first introduced by papers in ACL 2019 by Kim and Bansal, and by Wu and Mooney (question specific), and it would have been nice to reference this earlier related work."}}]

Author rebuttal:

Meta-review: The authors present a fairly simple new baseline system for the important problem of KnowledgeBased-VisualQA that achieves near SOTA performance on a popular benchmark. 

Reasons to accept:
- The proposed method is simple and intuitive.
- The conducted experiments are thorough, effectively substantiating the efficacy of the devised mechanisms.
- The analysis and the ablation are robust.
Decision: Accept-Main