SlowGuess/NAACL_2022_OCR huggingface dataset:
Liangke Gui $\S \ddagger$  Alex Hauptmann $\S$  Borui Wang $\dagger \ddagger$  Qiuyuan Huang $\ddagger$  Yonatan Bisk $\S \ddagger$  Jianfeng Gao $\ddagger$  Carnegie Mellon University  $\dagger$  Yale University  $\ddagger$  Microsoft Research {liangkeg, alex, ybisk}@cs.cmu.edu borui.wang@yale.edu, {qihua, jfgao} @microsoft.com
The primary focus of recent work with large-scale transformers has been on optimizing the amount of information packed into the model's parameters. In this work, we ask a complementary question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6% absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. Additionally, explicit knowledge integration improves interpretability of model predictions in our analysis. Code and pre-trained models are released at https://github.com/guilk/KAT.
There has been a revival of interest in knowledge-intensive tasks which require an external knowledge source for humans to perform. Many applications in real-world scenarios, such as autonomous AI agents, need to seamlessly integrate implicit (i.e., commonsense) and explicit knowledge (e.g., Wikidata) to answer questions. In this work, we investigate how to effectively integrate implicit and explicit knowledge for reasoning. Tasks like Outside Knowledge Visual Question Answering (OKVQA) (Marino et al., 2019) require that models use knowledge not present in the input to answer ques
Work done when Liangke and Borui interned at Microsoft Research.
Figure 1: Examples of knowledge-based VQA that requires external knowledge. Success on this task requires not only visual recognition, but also logical reasoning to incorporate external knowledge about the world.
tions, making it an ideal test bed for investigating this implicit-explicit knowledge trade-off.
Consider the examples from OK-VQA shown in Figure 1. To answer the question in the left example, the system needs to both ground organism to bird through explicit knowledge and then apply the implicit knowledge birds evolved from reptiles to answer the question. Similarly for the question in the right example, the system needs to recognize boats and harbor and requires the implicit knowledge anchors are used to stop boats from moving. A key challenge here is to accurately link image content to abstract external knowledge. There have been a number of recent developments demonstrating the feasibility of incorporating external knowledge into Question Answering models (Wang et al., 2017b; Li et al., 2020b; Marino et al., 2021; Wu et al., 2022; Garderes et al., 2020). Existing methods first retrieve external knowledge from external knowledge resources, such as DBPedia (Auer et al., 2007) and ConceptNet (Liu and Singh, 2004) before jointly reasoning over the retrieved knowledge and image content to predict an answer.
However, most existing approaches have several drawbacks. First, explicit knowledge retrieved using keywords from questions or image tags may be
Human Language Technologies, pages 956 - 968
too generic, which leads noise or irrelevant knowledge during knowledge reasoning. Second, existing work mainly focuses on explicit knowledge which is often in the form of encyclopedia articles or knowledge graphs. While this type of knowledge can be useful, it is insufficient to answer many knowledge-based questions. As shown in Figure 1, questions require the system to jointly reason over explicit and implicit knowledge, which is analogous to the way humans do. To address these challenges, we propose an approach, KAT, to effectively integrate implicit and explicit knowledge during reasoning. The main contributions of our work are as follows:
i) Knowledge extraction. We adopt two novel methods for knowledge extraction that significantly improve the quality and relevance of extracted knowledge: for implicit knowledge, we design new prompts to extract both tentative answers and supporting evidence from a frozen GPT-3 model; for explicit knowledge, we design a contrastive-learning-based explicit knowledge retriever using the CLIP model, where all the retrieved knowledge are centered around visually-aligned entities.
ii) Reasoning in an encoder-decoder transformer. We design a novel reasoning module in  $KAT$  to perform jointly reasoning over explicit and implicit knowledge during answer generation, which is trained by using an end-to-end encoder-decoder transformer architecture.
iii) OK-VQA performance.  $KAT$  sets a new state of the art on the challenging OK-VQA (Marino et al., 2019) benchmark, and significantly outperforms existing approaches.
Vision-Language Transformer. Multimodal transformers have made significant progress over the past few years, by pre-trained on large-scale image and text pairs, then finetuned on downstream tasks. VisualBERT (Li et al., 2019), Unicoder-VL (Li et al., 2020a), NICE (Chen et al., 2021b), and VL-BERT (Su et al., 2020) propose the single-stream architecture to work on both images and text. ViLBERT (Lu et al., 2019) and LXMERT (Tan and Bansal, 2019) propose a two-stream architecture to process images and text independently and fused by a third transformer in ta later stage. While these models have shown to store in-depth cross-modal knowledge and achieved impressive performance
on knowledge-based VQA (Marino et al., 2021; Wu et al., 2022; Luo et al., 2021), this type of implicitly learned knowledge is not sufficient to answer many knowledge-based questions (Marino et al., 2021). Another line of work for multimodal transformers, such as CLIP (Radford et al., 2021) or ALIGN (Jia et al., 2021), aligns visual and language representations by contrastive learning. These models achieve state-of-the-art performance on image-text retrieval tasks. Different from existing work that uses multimodal transformers as implicit knowledge bases, we focus primarily on how to associate images with external knowledge. Importantly, our model only relies on multimodal transformers learned by contrastive learning which do not require any labeled images. This makes our model more flexible in real-world scenarios.
Knowledge-based VQA. Some Knowledge-based visual language tasks require external knowledge beyond the image to answer a question. Early exploration, such as FVQA (Wang et al., 2017a), creates a fact-based VQA dataset by selecting a fact (e.g., <Cat, CapableOf, ClimbingTrees>) from a fixed knowledge base. A recent Outside Knowledge VQA (OK-VQA) dataset is a more challenging dataset, covering a wide range of knowledge categories. In our work, we focus on OK-VQA due to its large-scale knowledge-based questions as well as its open-ended nature.
Recent approaches have shown a great potential to incorporate external knowledge for knowledge-based VQA. Several methods explore aggregating the external knowledge either in the form of structured knowledge graphs (Garderes et al., 2020; Narasimhan et al., 2018; Li et al., 2020b; Wang et al., 2017a,b), unstructured knowledge bases (Marino et al., 2021; Wu et al., 2022; Luo et al., 2021), and neural-symbolic inference based knowledge (Chen et al., 2020; West et al., 2021). In these methods, object detectors (Ren et al., 2015) and scene classifiers (He et al., 2016) are used to associate images with external knowledge. Further, external APIs, such as Google (Wu et al., 2022; Luo et al., 2021), Microsoft (Chen et al., 2021a; Yang et al., 2022) and OCR (Luo et al., 2021; Wu et al., 2022) are used to enrich the associated knowledge. Finally, pre-trained transformer-based language models (Chen et al., 2021a; Yang et al., 2022), or multimodal models (Wu et al., 2022; Luo et al., 2021; Wu et al., 2022; Garderes et al., 2020; Marino et al., 2021) are leveraged as
implicit knowledge bases for answer predictions.
Different from previous approaches, Our work aims to develop a single, unified architecture, by jointly reasoning over explicit and implicit knowledge to augment generative language models. While part of our approach is similar to PICa (Yang et al., 2022) which considers GPT-3 as implicit knowledge base, our model takes one step further by showing that how explicit and implicit knowledge can be integrated during knowledge reasoning. Another similar work Vis-DPR (Luo et al., 2021) collects a knowledge corpus from training set by Google Search which is specific to a certain dataset. Our proposed model is more generic by collecting entities from Wikidata and not limited to the training set.
Open-Domain Question Answering (ODQA).
ODQA is the NLP task of answering general domain questions, in which the evidence is not given as input to the system. Several approaches (Chen et al., 2017; Karpukhin et al., 2020) propose to predict the answers by first retrieving support document from Wikipedia, before extracting answers from the retrieved document. Recent works (Izacard and Grave, 2020; Lewis et al., 2020b) combine text retrieval models with language generative models which achieve state-of-the-art performance on knowledge-intensive natural language processing tasks. Similar to these works as part of our method, we extend this framework to VQA domain and show the effectiveness of aggregating explicit and implicit knowledge for knowledge-based VQA.
When humans reason about the world, they process multiple modalities and combine external and internal knowledge related to these inputs. Inspired by this idea, we introduce a new  $KAT$  approach. The overview of the proposed  $KAT$  model is shown in Figure 2. We define the knowledge from explicit knowledge bases as the explicit knowledge, and the knowledge stored in large-scale language models as the implicit knowledge (i.e., implicit commonsense knowledge). We describe the retrieval method of our explicit knowledge (§3.2) and the retrieval method of our implicit knowledge (§3.3). Next, we introduce the details of our knowledge reasoning module which jointly reasons over both explicit and implicit knowledge (§3.4).
Problem Formulation. We apply our  $KAT$  on OK-VQA task in this paper. Formally, given a training dataset  $\mathbb{D} = \{(v_i,q_i,a_i)\}_{i = 1}^s$ , where  $v_{i}$  denotes the  $i^{th}$  training image;  $s$  is the total number of the training images;  $q_{i}$  and  $a_{i}$  represent the  $i^{th}$  question and its corresponding answer, respectively. We use a sequence-to-sequence model that is composed of an encoder and a decoder, which is a comparison method of T5 (Raffel et al., 2020) or BART (Lewis et al., 2020a). Let  $\theta$  be the parameters of the model  $p$  that needs to be trained. Unlike previous approaches that treat this task as a classification problem (Wu et al., 2022; Marino et al., 2021), our model is to take  $v_{i}$  and  $q_{i}$  as inputs and generate the answer  $a_{i}$  in an auto-regressive manner. It should be noted that our proposed model tackles a more challenging problem. As the generated answer may contain an arbitrary number of words from the entire vocabulary.
3.2 Explicit Knowledge Retrieval
3.2.1 Explicit Knowledge Extraction
Given an image  $v_{i}$  and corresponding question  $q_{i}$ , it is important to ground image regions with fine-grained descriptions, which is conducive to understanding both the image content and the question with the referred items. Existing approaches (Radford et al., 2021; Jia et al., 2021) on OK-VQA apply object detectors to generate image tags which are used for explicit knowledge retrieval. Such image tags can be generic and have a limited vocabulary size, leading noise or irrelevant knowledge. Motivated by the recent progress of visual-semantic matching approaches (Radford et al., 2021; Jia et al., 2021), we leverage a contrastive-learning-based model to associate image regions with external knowledge bases.
Similar to the previous work (Marino et al., 2021; Luo et al., 2021) which uses a subset of external knowledge, we construct an explicit knowledge base that covers the 8 categories of animals, vehicles and other common objects from Wikidata (Vrandecic and Krotzsch, 2014). The details can be found in Section 3.2.2. We denote the constructed knowledge base as  $\kappa$ . Each knowledge entry  $e$  from  $\kappa$  is a concatenation of the entity and its corresponding description.
The goal of our explicit knowledge retriever is to index all knowledge entries in  $d_r$ -dimensional dense representations by a dense encoder  $E_{ent}(\cdot)$ , such that it can efficiently retrieve the top  $m$  knowl-
Figure 2: Our  $KAT$  model uses a contrastive-learning-based module to retrieve knowledge entries from an explicit knowledge base, and uses GPT-3 to retrieve implicit knowledge with supporting evidence. The integration of knowledge is processed by the respective encoder transformer, and jointly with reasoning module and the decoder transformer as an end-to-end training with the answer generation.
edge entries relevant to each input image. Given an image  $v_{i}$ , we use a sliding window with a stride to generate  $N$  image regions  $\{v_{i}^{1},\dots,v_{i}^{N}\}$ . Then an image encoder  $E_{img}(\cdot)$  is applied to map each patch to a  $d_r$ -dimensional dense representation, and retrieves  $k$  knowledge entries from  $\mathcal{K}$  whose representations are closest to the patch-level representation. To define the similarity score between the image region  $v_{i}^{j}$  and the entity  $e$ , we use the inner product of their normalized representations:
$$
s i m (v _ {i} ^ {j}, e) = E _ {e n t} (e) ^ {T} E _ {i m g} (v _ {i} ^ {j}). \qquad (1)
$$
In total, we retrieve the top  $N \times k$  knowledge entries relevant to image  $v_{i}$ . We keep top- $m$  knowledge entries ranked by similarity scores as explicit knowledge source  $x^{exp}$ .
In principle, the image and knowledge entry encoders can be implemented by any multimodal transformer. We use the CLIP model (ViT-B/16 variant) (Radford et al., 2021) in our work and take the [CLS] as representations. We pre-extract representations of the knowledge entries in the knowledge base  $\mathcal{K}$  using the entity encoder  $E_{ent}$  and index them using FAISS (Johnson et al., 2019). The qualitative example for the extracting explicit knowledge model is presented in Appendix A.
3.2.2 Knowledge Base Construction
We use the English Wikidata (Vrandecic and Krotzsch, 2014) dump from Sep. 20, 2021 as the explicit knowledge source base which contains 95,870,584 entities. Each data item is stored in
a structured format constituted of property-value pairs. Properties are objects and have their own Wikidata pages with labels, aliases, and descriptions. We extract a subset that covers common objects in real-world scenarios. We remove all entities whose string labels or corresponding descriptions are empty or non-English. This results in a total of 423,520 entity triplets in the end (e.g.,  $<Q2813$ , Coca-Cola, carbonated brown colored soft drink>) (See Table 1).
Subclass	Number
Role	(Q214339)	162,027
Point of interest	(Q960648)	85,900
Tool	(Q39546)	78,621
Vehicle	(Q42889)	44,274
Animal	(Q729)	18,581
Clothing	(Q11460)	17,711
Company	(Q891723)	12,173
Sport	(Q349)	4,233
Total	423,520
Table 1: We collect a subset of Wikidata that covers common objects in real-life scenarios as our explicit knowledge base. Above are statistics of these subclasses.
3.3 Implicit Knowledge Retrieval
While our explicit knowledge retriever focuses on semantic matching between image regions and knowledge entries, it lacks implicit commonsense knowledge (e.g., Lemons are sour) which is usually stored in large-scale language models (Brown et al., 2020). In this section, we retrieve implicit
knowledge with supporting evidence by prompting from a large-scale pre-trained language model.
We design our implicit knowledge retriever with inspirations from the previous work (Yang et al., 2022). We leverage GPT-3 as an implicit language knowledge base and treat VQA as an open-ended text generation task. For each image-question pair, we first convert the image  $v_{i}$  into a textual description  $C$  via a state-of-the-art image captioning model (Li et al., 2020c), and then construct a carefully designed text prompt consisting of a general instruction sentence, the textual description  $C$ , the question, and a set of context-question-answer triplets taken from the training dataset that are semantically most similar to the current image-question pair (see Figure 7 in Appendix B for a concrete example). We then input this text prompt to the GPT-3 model in its frozen version and obtain the output from GPT-3 as the tentative answer candidate to the current image-question pair.
To gain deeper insights from the implicit knowledge coming out of GPT-3 and its rationale, we design another prompt to query GPT-3 for supporting evidence behind the tentative answer candidate that it generates. More specifically, for each image-question pair  $(v_{i},q_{i})$ , and for a tentative answer  $a$  generated by GPT-3, we construct the prompt in the form of: “(question  $q_{i}$ )? (answer  $a$ ). This is because” to query GPT-3 for supporting evidence (see Figure 6 in Appendix B for a concrete example). We finally compile both the tentative answers and the corresponding supporting evidence from GPT-3 as implicit knowledge source  $x^{imp}$ .
3.4 KAT Model
As showed in the Figure 2, the explicit knowledge entries are from an image, which are concerned with semantic matching of the image regions. These knowledge entries could be noisy or irrelevant to its corresponding question. Moreover, some of the supporting evidence prompted from GPT-3 is generic or not related to image content. Simple concatenation of different knowledge may introduce noise during model training. We design a knowledge reasoning module with inspirations from the previous work (Karpukhin et al., 2020). Our knowledge reasoning module encodes each question and knowledge pair separately, and jointly reason over both explicit and implicit knowledge when generating an answer.
Encoder. We concatenate question  $q_{i}$  with each knowledge as a question-knowledge pair. Firstly, we add sentinel tokens question:, entity: and description: before the question, the retrieved entity, and its description separately. Similarly, we add sentinel tokens question:, candidate: and evidence: before the question, the tentative answer, and its evidence. Secondly, we use an embedding layer followed by a sequence of encoder layers to encode the question-knowledge pairs separately. We average the token embeddings of each question-knowledge pair from the last encoder layer, which results in an embedding matrix of explicit knowledge  $X^{exp}\in \mathbb{R}^{m\times d}$  and implicit knowledge  $X^{imp}\in \mathbb{R}^{p\times d}$ , where  $d$ ,  $m$  and  $p$  are the embedding dimension, the number of explicit knowledge  $x^{exp}$ , and the number of implicit knowledge  $x^{imp}$ , respectively.
Reasoning Module. To jointly reason over implicit and explicit knowledge, we concatenate the embeddings of explicit and implicit knowledge form a global representation  $X \in \mathbb{R}^{(m + p) \times d}$ . The cross-attention module takes the global representation  $X$  of the encoder as the input. Let  $H \in \mathbb{R}^d$  be the output of the previous self-attention layer of the decoder. By definition (Vaswani et al., 2017), the scaled dot-product attention can be expressed as:
$$
Q _ {v} = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d}}\right) V, \tag {2}
$$
where queries  $Q$ , keys  $K$ , and values  $V$  are computed by applying linear transformations:  $Q = W_{Q}H, K = W_{K}X, V = W_{V}X$ . The attended representation  $Q_{v}$  is a weighted sum of the values, and implies that our model performs a joint reasoning over explicit and implicit knowledge when generating answers.
Decoder. We feed the embeddings of explicit and implicit knowledge to a sequence of decoder layers for answer generation. We train our model with a cross-entropy loss:
$$
\mathcal {L} _ {C E} = - \sum_ {t = 1} ^ {n} \log p _ {\theta} \left(y _ {t} \mid y _ {<   t}, x ^ {e x p}; x ^ {i m p}\right), \tag {3}
$$
where  $y_{t}$  is predicted autoregressively.
Method	Knowledge Resources	Acc (%)
No knowledge	Q only (Marino et al., 2019)	-	14.93
Vanilla T5	-	18.56
MLP (Marino et al., 2019)	-	20.67
BAN (Marino et al., 2019)	-	25.1
MUTAN (Marino et al., 2019)	-	26.41
With knowledge	BAN+AN (Marino et al., 2019)	Wikipedia	25.61
BAN+KG-AUG (Li et al., 2020b)	Wikipedia+ConceptNet	26.71
MUTAN+AN (Marino et al., 2019)	Wikipedia	27.84
ConceptBERT (Garderes et al., 2020)	ConceptNet	33.66
KRISP (Marino et al., 2021)	Wikipedia+ConceptNet	38.35
Vis-DPR (Luo et al., 2021)	Google Search	39.2
MAVEx (Wu et al., 2022)	Wikipedia+ConceptNet+Google Images	39.4
GPT-3	PICa-Base (Yang et al., 2022)	Frozen GPT-3 (175B)	43.3
PICa-Full (Yang et al., 2022)	Frozen GPT-3 (175B)	48.0
KAT-explicit (w/ reasoning)	Wikidata	44.25
KAT-implicit (w/ reasoning)	Frozen GPT-3 (175B)	49.72
KAT (w/o reasoning)	Wikidata+Frozen GPT-3 (175B)	51.97
KAT (single)	Wikidata+Frozen GPT-3 (175B)	53.09
KAT (ensemble)	Wikidata+Frozen GPT-3 (175B)	54.41
Table 2: Results of OK-VQA comparing to standard baselines show that our KAT (large size) model achieves state-of-the-art performance on OK-VQA full testing set. It is important (see table sections) to compare methods based on their access to increasingly large implicit sources of knowledge and utilization of explicit knowledge sources. Our five KAT models variants make the relative importance of these decisions explicit. We train our model with 3 random seeds and the result is denoted as ensemble.
OK-VQA (Marino et al., 2019) is currently the largest knowledge-based VQA dataset. The questions are crowdsourced from Amazon Mechanical Turkers and require outside knowledge beyond the images in order to be answered correctly. The dataset contains 14,031 images and 14,055 questions covering a variety of knowledge categories. We follow the standard evaluation metric recommended by the VQA challenge (Antol et al., 2015).
For the knowledge reasoning module, we initialize our model with the pre-trained T5 model (Raffel et al., 2020). We compare two model sizes, base and large, each containing  $220M$  and  $770M$  parameters respectively. We fine-tune the models on OK-VQA dataset, using AdamW (Loshchilov and Hutter, 2019). We use a learning rate of  $3e - 5$  to warm up for  $2K$  iterations and train for  $10K$  iterations. Limited by the computational resources,
we set the number of retrieved entities to 40. The model is trained with a batch size of 32, using 16 V100 GPUs with 32Gb of memory each. Unless otherwise specified, all results reported in this paper as KAT use this model which we found to perform best. We evaluate our predictions with ground-truth after normalization. The normalization step consists of lowercasing, and removing articles, punctuation and duplicated whitespace (Chen et al., 2017; Lee et al., 2019). To be consistent with previous work (Marino et al., 2021), we train our model with 3 different random seeds and use the average results for the leaderboard submission.
4.3 Comparison with Existing Approaches
We compare our model against existing approaches on the OK-VQA dataset and the results are summarized in Table 2. Our model outperforms state-of-the-art methods by significant margins. We compare our model with existing approaches from two aspects. (1) If we only consider using explicit knowledge, our model achieves  $44.25\%$  which is  $4.85\%$  and  $5.9\%$  higher than MAVEx and KRISP,