SlowGuess/NAACL_2022_OCR huggingface dataset:
respectively. Our model uses contrastive-learning-based model to extract knowledge, leaving headroom by incorporating supervised pre-trained models, such as pre-trained object detectors. It should be noted that our proposed model is working on a more challenging problem. As the generated answer could contain an arbitrary number of words from the entire vocabulary. Our model is slightly better than PICa-Base which is a plain version of PICa-Full without example engineering. It implies that our single, unified architecture can effectively associate images with the explicit knowledge base. (2) If we take the implicit knowledge from GPT-3 as the additional input, our model outperforms PICa-Full by  $6.41\%$  which indicates it is important to integrate knowledge of different types when generating answers. The detailed comparison can be found in Table 3.
To unpack the performance gain and understand the impact of different components, we ablate and compare different model architectures, types of knowledge and the number of explicit knowledge.
Model architecture	Knowledge	Accuracy (%)
Base	Large	Explicit	Implicit
✓	18.56
✓	✓	40.93
✓	✓	44.25
✓	✓	47.60
✓	✓	49.72
✓	✓	✓	50.58
✓	✓	✓	54.41
Specifically, as shown in Table 3, our KAT-large shows a consistent improvement over using KAT-base. This larger model has more capacity for implicit knowledge reasoning. The integration of explicit and implicit knowledge achieves a performance gain of  $\sim 4\%$ , supporting the intuition that these two types of knowledge provide complementary pieces of knowledge.
5.1 Effectiveness of Knowledge Reasoning
To verify the effectiveness of our knowledge reasoning module, we use a KAT without the knowledge reasoning module which is denoted as KAT (w/o reasoning). This model concatenates explicit and
Table 3: Ablation study on model architectures and types of knowledge. Our experiments show that larger model has more capacity for implicit knowledge reasoning and jointly reasoning over both knowledge sources has a consistent improvement with baselines.
Method	Accuracy (%)
KAT (w/o reasoning)	51.97
KAT	54.41
Table 4: Comparison with KAT (w/o reasoning) which uses the concatenated knowledge as inputs without the knowledge reasoning module.
implicit knowledge as a sentence and adopts a maximum length of 256 tokens. We train this variant with the same parameter settings. As shown in Table 4, simply concatenating knowledge sources is  $2.43\%$  lower than our proposed model. It indicates that KAT (w/o reasoning) may introduce noise to relevant knowledge during encoding. Our model adaptively attend different knowledge sources for answer generation that can reduce the influence of irrelevant knowledge.
5.2 Extracting Explicit Knowledge
Figure 3: Our model achieves consistent improvement when aggregating more knowledge entries from an explicit knowledge base. However, as CLIP-ViT/16 and RN50 are very different explicit knowledge retrieval backbones we see the choice of backbone and number of sources to include are intimately related. Here we use KAT-base for demonstration.
From Figure 3 we can see, the performance of our model is directly affected by the size of retrieved explicit knowledge. When only considering the implicit knowledge (i.e., the number of retrieved entities is 0), our model achieves  $47.6\%$  which is slightly worse than PICa-Full baseline. It indicates that solely increasing model complexity cannot improve the performance. This also demonstrates the importance of explicit knowledge. Our model shows a consistent improvement by incorporating more explicit knowledge. While a more
Question:
What is painted on the bench?
Category:
Brands, Companies and Products
Answer:
Exp: strand
KAT (w/o KRM): red
KAT: Coca cola
Explicit Knowledge:
Tactile paving: system of textured ground surface indicators to assist pedestrians who are blind or visually impaired.
Coca Cola: carbonated brown-colored soft drink.  
Bench: piece of furniture on which several people may sit at the same time.  
Street furniture: collective term for objects and pieces of equipment installed outdoors for various purposes.
Implicit Knowledge:
Red: the bench is painted red.
Question:
What kind of board is this?
Category:
Sports and recreation
Exp: wakeboard
KAT(w/o KBM): surfboard
Imp: surfboard
KAT: surfboard
Explicit Knowledge:
Implicit Knowledge:
Figure 4: Two examples from OK-VQA dataset that our model generates correct answers by jointly reasoning over both implicit and explicit knowledge. (exp: predictions by using explicit knowledge only and imp: predictions by using implicit knowledge only). More examples and analysis can be found in Appendix C.
Wakeboard boat: boat designed to create a w for wakeboarding.
Wakeboarder: someone practicing wakeboarding.  
Kitesurfer: practitioner of kitesurfing.
Boardsport: sports that are practiced with some sort of board as the primary equipment.
Skiboarding: freestyle skiing using short skis and no poles.
Surboard: This sport is surfboard.
extensive knowledge set may include more distracting knowledge, retrieved knowledge entries can share either visually or semantically similar knowledge as the relevant ones. Thus this can massively reduce the search space and/or reduce spurious ambiguity.
We compare different explicit knowledge retrieval module. Though ViT/16 has a large classification improvement over ResNet-50 (e.g.,  $6.9\%$  on ImageNet) (Radford et al., 2021), there is a less gap between these two backbones. As the number of retrieved entities increases, our knowledge reasoning module can further migrate this gap by adaptively attending to different explicit knowledge.
5.3 Category Results on OK-VQA
Here we present quantitative analyses to illustrate how explicit and implicit knowledge influence the final predictions. Based on the types of knowledge required, questions in OK-VQA are categorized into 11 categories and the accuracy results of each category are reported in Table 5. We re-train our model under the same settings with only either explicit or implicit knowledge, denoted as "exp" and "imp" respectively.
For most categories, the model using only explicit knowledge performs worse than that using only implicit knowledge. As implicit knowledge comes from the results of state-of-the-art object detection, image captioning models and supporting evidence by prompting GPT-3. While explicit knowledge is retrieved based on semantic matching between images and entities from knowledge bases, it contains richer but more distracting knowledge. Note that using explicit knowledge performs better for category "Brands, Companies, and Prod
ucts" and "Weather and Climate". It indicates that accurately recognizing objects with fine-grained descriptions in the images is important for these categories to answer corresponding questions.
Question Type	Exp	Imp	Ours	Δ
Plants and Animals	42.2	51.5	54.7	+3.2
Science and Technology	44.4	43.3	52.8	+8.3
Sports and Recreation	49.7	53.8	60.4	+6.7
Geo, History, Lang, and Culture	45.6	45.4	55.8	+10.2
Brands, Companies, and Products	41.7	38.2	48.5	+6.8
Vehicles and Transportation	41.5	42.9	51.3	+8.4
Cooking and Food	47.9	47.7	52.7	+4.8
Weather and Climate	51.7	46.3	54.8	+3.1
People and Everyday	43.1	44.4	51.5	+7.1
Objects, Material and Clothing	42.9	45.4	49.3	+3.9
Other	41.5	50.2	51.2	+1.0
Table 5: Accuracy (%) of question types in OK-VQA full testing set. Our models outperforms exp and imp models by a large margin on all categories. (exp: explicit-only model and imp: implicit-only model)
Analyzed in previous sections, jointly reasoning over both knowledge sources during answer generation improves the explicit-only and implicit-only models by large margins. Figure 4 shows two examples comparing answers generated by different models along with retrieved knowledge. The left example shows that while explicit knowledge retrieved from the knowledge base contains the necessary knowledge entries for reasoning, it fails to generate the answer which requires the relation between bench and Coca Cola logos. On the other side, implicit knowledge retrieved from GPT-3 can only infer the bench is painted red, failing to recognize its logo. By jointly considering both knowledge sources, our model can associate the color of
Coca Cola logo with the painted color of the bench which derives the correct answer. The right example shows that though explicit knowledge does not contain the right knowledge entries, it provides visually similar descriptions of this sport which further constrains the search space of our model and verifies the correctness of the implicit knowledge.
This paper takes a step towards understanding the complementary role of implicit knowledge gained from continuing to scale models and explicit knowledge from structured knowledge bases. Importantly, it appears that there is headroom in both directions (i.e. improving retrieval and reasoning). Our conceptually simple yet effective approach for knowledge-based VQA makes these relationships explicit while still achieving a significant improvement against state-of-the-art results. Additional challenges remain, for example how best to align image regions with meaningful external semantics deserves and how to efficiently and accurately integrate multiple knowledge bases.
We are especially grateful to Jianwei Yang, Daniel McDuff, Dragomir Radev, Harkirat Behl, Hao Chen, Chunyuan Li, Baolin Peng, Kezhen Chen, Tejas Srinivasan for their for the early insightful discussions, suggestion, and their pointers to the modeling generation and literature. We thank Zhe Gan, Zhengyuan Yang, Lijuan Wang from cognition service team of Microsoft for their work and their generous helps and feedback for the project. We appreciate Subhojit Som from Turing team of Microsoft for his enormous support and encouragement. The authors gratefully acknowledge Kenneth Marino from DeepMind, and Roozbeh Mottaghi from the AllenAI for their comments, supporting and helps of the work. This research was supported in part by the Defense Advanced Research Projects Agency (DARPA) under contract number D17PC00340 and also supported by the US DARPA KAIROS Program No. FA8750-19-2-1004.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In CVPR.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In *The semantic web*, pages 722-735. Springer.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051.
Kezhen Chen, Qiuyuan Huang, Yonatan Bisk, Daniel McDuff, and Jianfeng Gao. 2021a. Kb-vlp: Knowledge based vision and language pretraining. In ICML, workshop.
Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, and Jianfeng Gao. 2021b. Nice: Neural image commenting with empathy. In EMNLP.
Kezhen Chen, Qiuyuan Huang, Paul Smolensky, Kenneth Forbus, and Jianfeng Gao. 2020. Learning inference rules with neural tp-reasoner. In NeurIPS, workshop.
François Garderes, Maryam Ziaeefard, Baptiste Abelloos, and Freddy Lecue. 2020. Conceptbert: Concept-aware representation for visual question answering. In EMNLP.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. In EACL.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020a. Bart:
Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. 2020a. Unicoder-vl: A universal encoder for vision and language by cross-modal pretraining. Proceedings of AAAI.
Guohao Li, Xin Wang, and Wenwu Zhu. 2020b. Boosting visual question answering with context-aware knowledge aggregation. In ACM MM.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020c. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV.
Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211-226.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS.
Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. 2021. Weakly-supervised visual-retriever-reader for knowledge-based question answering. In EMNLP.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2021. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In CVPR.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR.
Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. 2018. Out of the box: Reasoning with graph convolution nets for factual visual question answering. NeurIPS.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1-67.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Denny Vrandecic and Markus Krotzsch. 2014. Wiki data: a free collaborative knowledgebase. Communications of the ACM.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017a. Fvqa: Fact-based visual question answering. TPAMI, 40(10):2413-2427.
Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. 2017b. Explicit knowledge-based reasoning for visual question answering. In *IJCAI*.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. Symbolic knowledge distillation: from general language models to commonsense models. In ArXiv.
Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. 2022. Multi-modal answer validation for knowledge-based vqa. In AAAI.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI.
A Figure of Explicit Knowledge
In this section, we show one example Figure 5 to extract explicit knowledge from an image, which use the CLIP model to conduct the explicit knowledge retrieval with the image and a wiki knowledge base.
Overview of Extracting Explicit Knowledge
Figure 5: Overview of the explicit knowledge extraction. We use a sliding window to crop image regions and retrieve knowledge entries from an explicit knowledge base by CLIP.
Figure 7: An example of the prompts that we use to query GPT-3 in our knowledge-aunted GPT-3 query system.
B Examples of Prompts of Implicit Knowledge
In this Section B of the Appendix, we show two concrete examples (Figure 6 and Figure 7) for the prompts that constructed to query GPT-3 for implicit knowledge in our experiments:
1 Query Prompt:
2 What is the shape? circle. This is because
3
4 Response from GPT-3:
5 'the circle is the only shape that has no corners.'
Figure 6: An example of the evidence of rationale that we obtain from GPT-3 by using a combination of question and answer candidate to query it.
C Analysis on More Examples
In this section, we showcase more predictions from variants of our model. As shown in Figure 8, we analyze the predictions based on different type of knowledge from several aspects:
Effectiveness of explicit knowledge retriever. Our explicit knowledge retriever can retrieve fine-grained knowledge entries from the explicit knowledge base, such as golden retriever (a fine-grained breed of dogs), cucumber sandwich (a specific type of sandwich) and Macbook Pro (a specific model
1 Please answer the question according to the above context and the provided knowledge if given.
2
3 Context: a keyboard sitting in front of a computer monitor
4
5 Q: Is this a laptop or desktop?
6 A: desktop
7
8
9
10 Context: This picture shows one man on ski's and one youngster on a snowboard.
11
12 Q: Who is a famous participant in this sport?
13 A: bode miller
14
Context: a black motorcycle parked in a parking lot. motorcycle, parked, parking, ground, outdoor, wheel, fender, brake, vehicle, exhaust system, land vehicle, tire, muffler, motorcycling, rim, suspension, automotive exhaust, auto part, motorcycle racing, motorcycle fairing, automotive tire, fuel tank, vehicle brake, disc brake, cruiser, motorbike, bike
16
17 Knowledge:
18 A Benelli Adiva is a motor scooter model.
19 A Yamaha TMAX is a scooter.
20 A mofa is a moped with a maximum speed of 20 to  $25\mathrm{km / h}$
21 A motorcycle based vehicle is a motor vehicle with two or more wheels, based on cycle technology.
22 A sport bike is a motorcycle designed for sporty riding regardless of other characteristics.
23 A motorcycle development rider is a person who for their profession participates in tests to develop a motorcycle.
24 A Bravo is a Piaggio moped.
25 A Suzuki X-90 is a motor vehicle.
26 A Polaris Industries is a Designs, engineers and manufactures powersports vehicles.
27 A Monark is a Bicycle, moped and motorcycle manufacturer.
28
29 Q: What sport can you use this for?
30 A:
of Apple products). These fine-grained entities are hardly obtained from existing object detection models, which can constraint the search space of our model and are beneficial to our answer generation process.
Effectiveness of implicit knowledge retriever. Our implicit knowledge retriever can retrieve supporting evidence from GPT-3, such as Thomas: the train is named after the man who designed it. and Refrigerator: the refrigerator is used to keep food cold. These kinds of knowledge are highly related to commonsense knowledge which needs further inference based on entities and provide complementary explanation to explicit knowledge.
Answer generation & classification. As most previous work on OK-VQA task, such as KRISP or MAVEx method, implement OK-VQA as a classification task. The prediction vocabulary is dataset-specific and assumes the training and test set are sharing a similar vocabulary. The limitation of these methods is the generalization ability. Our proposed KAT model treats OK-VQA as an open-end generation task. From these examples we found, our model can generate answers like Iphone or Hercules that are visually and semantically reasonable. Our proposed novel KAT model using the explicit
Can you guess the material used to make the bag shown in this picture?
Objects, Material and Clothing
Exp: canvas
answer:
KAT (w/o KRM): leather  
KAT
entity:description)
Acer Aspire one: line of
notebooks by Acer Inc.
Drawing Instrument: tool used
for drawing or drafting.
Writing Implement: tool used for writing
Book bag: a bag, usually a
backpack, used by students to
carry their textbooks.
candidate:evidence)
leather: the bag is made of
leathe
Explicit Knowledge:
Question
What breed are the dogs?
Category:
Exp: golden retriever
KAT (w/o KRM): husky
KAT: golden retriever
Explicit Knowledge:
entity:descriiption)
Snow pillow: measuring
device for snowpack.
Search and rescue dog : dog
trained to locate or retrieve
a missing or trapped person. Golden retriever: dog breed.
Mushing: Sport or sport?
powered transport method.
Candidate: evidence
Husky: The husky is a very
intelligent dog. They are
what they want to do.
Category:
Question:
Category:
What type of sandwich is being served?
Cooking and Food
umber
KAT:cucumber
Salad: dish consisting of a mixture of small pieces of food, usually vegetables or fruit.
Cucumber sandwich: the
traditional cucumber
sandwich is composed of
tith slices of cucumber
placed between two thin slices of crustless, lightly
butterd white bread.
Vegetable chip: cooked chip
prepared using vegetables.
(candidate: evidence)
Sub: the sub is a type of
What sort of phone would you associate with
his computer?
Brands, Companies and Products
Exp: lphone
KAT (w/o KRM): cell
KAT:iphone
entity:description
Floor lamp: lamp standing on
the floor, often with a nightlight.
clipping up to the vertical middle of the room.
ebook Pro : laptop made by
MacOS: operating system for
Apple computers, launched in
2001 as Mac OS X
Smart mattress: Mattress monitoring sleep patterns
smartphone : the e - not a smartphone
Question:
Category:
What is the name of the famous train pictured?
Vehicles and Transportation
Exp: Smoot
Imp: Thomas
KAT (w/o KRM): Thomas
KAT: Thomas
Fog machine: device that
emits a dense vapor that appears similar to
Draiping small powered rail
vehicle used by track
maintenance workers
Oast house: buildings
designed for klining (drying) hops as part of the brewing process.
Clouding agent: type of
emulsifier used to make
beverage such as fruit juice
to look more cloudy.
Thomas: the train is named
after the man who designed
What is this dog running after?
Exp: person
Imp: ball
KAT (w/o KRM): ball
KAT: ball
Sighthound: dog breed.
American Staffordshire
Merrrier: dog breed.
Greyhound racing: canine
racing sport involving the
Whipper racing : Dog sex
$\therefore \overrightarrow{PB} \cdot  \left( {\overrightarrow{PA} - \overrightarrow{PC}}\right)  = 0$
Ball: the dog is chasing after the ball.
How often should someone use this?
Exp: twice
Imp: daily
KAT (w/o KRM): daily
KAT: daily
Bathroom linen: household
linen used specifically for
the bathroom.
toothbrush: oral hygiene
the close the teeth , runs
the clean the teeth, gums;
and tongue: Toothbrush holder:
containers or rack for
toothbrushes.
Laubwerk: delicate foliage
ornament with interlacing
straps.
Daily: the product is made
with natural ingredients.
This is why it is safe to use
What hobby might this depict?
Exp: paper craft
Imp: scrapbook
KAT (w/o KRM): painting
KAT: scrapbook
Embroidery workshop:
workshop where:
Embroidery is created.
Issors: hand-operated