REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering
Authors: Leroy Lin
Yujia Xie
Dongdong Chen
Yichong Xu
Chenguang Zhu
Lu Yuan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Knowledge-based VQA

Abstract: This paper revisits visual representation in knowledge-based visual question answering (VQA) and demonstrates that using regional information in a better way can significantly improve the performance. While visual representation is extensively studied in  traditional VQA, it is under-explored in knowledge-based VQA even though these two tasks share the common spirit, i.e., rely on visual input to answer the question. Specifically, we observe in most state-of-the-art knowledge-based VQA methods: 1) visual features are  extracted either from the whole image or in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; 2) visual features are not well utilized in the final answering model, which is counter-intuitive to some extent. Based on these observations, we propose a new knowledge-based VQA method REVIVE, which tries to utilize the explicit information of object regions not only in the knowledge retrieval stage but also in the answering model. The key motivation is that object regions and inherent relationship are important for knowledge-based VQA. We perform extensive experiments on the standard OK-VQA dataset and achieve new state-of the-art performance, i.e., 58.0 accuracy, surpassing previous state-of-the-art method by a large margin (+3.6%). We also conduct detailed analysis and show the necessity of regional information in different framework components for knowledge-based VQA. Code is publicly available at https://github.com/yzleroy/REVIVE.

REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering
Yuanze Lin
Yujia Xie Chenguang Zhu
Dongdong Chen
Yichong Xu
Lu Yuan
University of Washington (yujiaxie, dochen, y picxu)@microsoot.com
Microsoft
eua
Abstract
This paper revisits visual representation knowledge-based visual question an- swering (VQA) and demonstrates that using regional information in better way can significantly improve the performance. While visual representation is ex- tensively studied in traditional VQA, it is under-explored in knowledge- VQA even though these two tasks share the common spirit, i.e., rely on visual input to answer the question. Specifically, knowledge-based VQA methods: 1) visual features are extracted either from the whole image in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; 2) visual features are not well utilized in the final answering model, which is counter- counter-intuitive some extent. Based on these observations, we propose a new knowledge- -baedd VQA method REVIVE, which tries utilize the explicit information of object regions not only in the knowledge retrieval stage but also in the answering model. The key motivation is that object regions and inherent relationship important for knowledge-based VQA. We perform extensive experiments on the standard OK-VQA dataset and achieve new state-of-the performance, i.e., 58.0% ac- curacy, surpassing previous state-of-the-arl method by a large margin (43.6%). We also conduct detailed analysis and show the necessity of regional information in different framework components for knowledge-based VQA. Code is publicly available https: /gtthbb.. com/yzleroy REEVIEE.
observe that in most state-of-the-ar
1 Introduction
Many vision-based decision making processes in our daily life go beyond perception and recognition. For example, if we
salad bowl in the deli bar, decision whether to buy it does not only depend on what is in the bowl, but also the calories in each of the item. This motivates the knowledge-basse based Visual Question Answering (VQA) task [22], which extends traditional VQA task [2] to solve more complex problems, i.e., where commonsense knowledge is required to answer the open-domain questions.
By definition, knowledge-based VQA takes three different information sources to predict the answer: input visual information (image), input question, and the external knowledge. While existing research on knowledge-basee
VQA mainly focuses on improving the incorporation of external knowledge, this paper focuses on improving the object-centric visual representation and presents a comprehensive empirical study to demonstrate that visual features matter this task.
Intuitively, visual information should be well used for both knowledge retrieval and final answering. However, we find existing state-of-the-att -fetttrrrt (SOTA) methods [37, 8] such domain have not fully
Work done during internship at Microsoft Redmond.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Object Regions
Image
Sliding Window
Knowledge
Answering Model
Prediction
Question
(b)
Knowledge (Batiy: Description)
Question:
Trainera: Traditional
Image
Object Regions
Knowledge
What the furniture item directly to the the people?
type rowing boats
Answering Model
Prediction
Question
Answer: couch (a)
(c)
Figure 1: (a) An example from OK- VQQ dataset, our method utilizes the retrieved knowledge and object-centric regions to solve the question. (b) The pipeline of previous ofte-off-the-ar arrt method KAT [8]. (c) The pipeline of our proposed REVIVE.
utilized it. On the one hand, they simply use either the whole image or sliding window on the image to retrieve the external knowledge. On the other hand, they ignore the essential visual information (i.e., object-centric representations) in the final answering model. other words, they fuse only the retrieved knowledge and the question as pure natural language processing (NLP) model to obtain the answer, typical method [8] is illustrated Figure (b).
In this paper, we revisit visual representation in knowledge-based VQA, and argue that the information of object regions and their relationship should be considered and used in a dedicated way. The underlying motivation is shown in Figure 1 (a), which demonstrates that understanding the objects and their relationship is necessary. To this end, we propose REVIVE to better utilize REgional VIsual Representation for knowledge-basee Visual quEstion answering. It not only exploits the detailed regional information for better knowledge retrieval, but also fuses the regional visual representation into the final answering model. Specifically, first use the object detector GLIP [16] to locate the objects, and then use the cropped region proposals to retrieve different types of external knowledge. Finally, integrate the knowledge together with the regional visual features into unified transformer based answering model for final answer generation.
We perform extensive experiments the OK-VQA dataset [22], and the proposed REVIVE achieves the SOTA performance 58.0% accuracy, a 3.6 % absolute improvement from the results of previous SOTA method [8].
We summarize our contribution as follows:
(a) We systematically explore how to better exploit the visual feature to retrieve knowledge. The empirical results suggest the region-based approach performs the best, compared to whole image-based and sliding window-basee approaches.
(b) We integrate the regional visual representation, retrieved external and implicit knowledge into transformer-base question answering model, which can effectively leverage the three information sources for solving knowledge-based VQA.
(c) Our proposed REVIVE achieves the state-of-the-ar performance on OK-VQA dataset, iee. 58.0% % accuracy, surpassing the previous methods by large margin.
2 Related Work
Knowledge-Based VQA. Knowledge-based VQA [22] aims to predict answers for general questions by leveraging external knowledge beyond image content. Early works [34, 33] introduce external knowledge to solve visual question answering (VQA) tasks. OK-VQA dataset [22] is the first large- scale dataset with questions that need be answered using external knowledge instead of provided fixed knowledge base [33]. Recent studies [34, 33, 24, 23, 40, 36, 21, 37, 8] integrate different knowledge from various external knowledge resources, e.g., ConceptNet [28], Wikipedia [31], etc, for solving knowledge-based VQA. Later, PICa [37] regards large language models, e.g., GPT-3 [3] as an implicit knowledge source and employs [3] to get answer prediction based on textual prompts. Inspired by the recent success of knowledge-retrieved methods [11. 10] that leverage

---

Wikidata
Regions
Regional Features
Explicht Knowlelge
Gaddin reteowr hrudd
Image:
Regional Positions
Encoder Decoder
GLIP
Visual Encoder
Object-centric Region Fastures
Regional
Answer: Golden retriever
Tags
Captioner
Context:
GPT-3
Implicit Knowledge Retrlever: dog
brown dogs,gghting Righting red frishee.
Question:
"What type of dog is pictured the left?"
The illustration of REVIVE. It exploits regional information (i.e., features, positions and tags), question and context to retrieve different types of knowledge. addition, it also incorporates learned object-centric region features with retrieved knowledge for answer generation.
Figure 2:
external knowledge retrieval with language generative models for open-domain question answering. KAT [8] exploits the Fil reader [11] to perform knowledge reasoning over retrieved implicit and explicit knowledge. Our work instead emphasizes revisiting the visual representation for knowledge retrieval, i.e., resorting to regional visual representation. In addition, we propose to incorporate object- centric regional visual representation together with retrieved knowledge in the answer generative model. Several works [23, 6, 5, 27, 24] have incorporated visual embeddings or captions predicting the final answers. However, these works target at different settings and they haven't fully explored how to better use regional representations to retrieve knowledge.
Vision-Language Models. Recent years have witnessed the rapid development of vision- laagun- models [30, 12, 38, 29, 17, 39, 32, 35]. Those works usually first pre-train neural network on large-scale image-text dataset and then finetune the models for solving specific vision-language tasks. Among them, VIn [39] aims learn the object-centric representation. CLIP [25] pre-trains the models with large-scale text-image pairs by contrastive learning. GLIP [38] reformulates the pre-training process by unifying object detection and phrase grounding. Our method uses the three models sub-modules to identify object-centric regions and retrieve knowledge for knowledge-based VQA task.
Proposed Method
Knowledge-based VQA task [22] seeks answer questions based external knowledge beyond images. Specifically, let us denote a knowledge-based VQA dataset as D = {(i, Qi, ) where I, Q, and A, denote the input image, question and answer of the i-th sample respectively, and N is the number of total samples. Given the dataset, the goal is to train a model with parameter 0 to generate the answer A, with input 1, and Q.
In this section, we introduce our method REVIVE. Figure 2 shows an overview the method. We leverage the detected regions of the input image to obtain the object-centric region features and retrieve explicit knowledge. Meanwhile, we prompt GPT-3 [3] by regional tags, question and context to retrieve implicit knowledge. After that, the regional visual features, retrieved knowledge, and the text prompt consists of regional tags, question and context will then be fused into a encoder-decoder module to generate the answer. We explain more details in Section 3. 3.2 and 3.3.
3.1 Regional Feature Extraction Module
Given an image 1, we first adopt a object detector to give us the positions of region proposals,
B = (b, (b,)) D(T),
(1)
M is the set of bouding boxes, M is the number of detected boxes, and D(-) is the where B (b,) object detector.
Here, we adopt ()-) as the visual grounding model GLIP [16]. We use the text prompt "Detect: person, bicycle, car, ... toothbrush", ..othba..oo which contains all object categories of MSCOCO

---

dataset [18]. In this way, the model can provide us with all bounding boxes associated with those categories.
After we get the bounding boxes B of interested objects from GLIP, We crop the image / according to B to obtain region proposals R = fry} [ry] j=1 We then extract the object-centric visual features from the proposals: V/ F)r;); where Bj E IR$ the visual embedding of the j-th proposal, is the embedding dimension and E(-) represents the image encoder. Inspired by the strong transferring capability of recent contrastively trained vision-language models, we adopt the visual encoder of CLIP [25] as our image encoder E(.). We use the encoding of [CLS] token the final embedding.
To understand the relationship between/among the objects, we find also important introduce the position information B along with its regional visual features.
In addition to the embeddings, explicitly obtaining the description of each region proposal in the textual format is also helpful for knowledge retrieval. For the contrastively trained vision-language models, the training loss explicitly encourages inner product between the image embedding and the text embedding to be larger the image and the text are well-aligned. Therefore, such a model is capable of selecting the tags that describe the image from set of customized tags T by computing the inner product. Denote the language encoder of CLIP T(.). Given set of tags T = {14);=) Ni is the number of total tags, we compute the inner product between the region proposals and all tags, and adopt the tags with the top-P similarities the description of the region proposals,
H (hpppm arg (opp j= M,
GET
where ( the inner product, P denotes the number of the obtained regional tags and means the retrieved regional tags.
In complement the localized textual description H. adopt a caption model explicitly describe the relationships between the major objects and provide more context,
c c= C(I)
(3)
where C(-) is the caption model. For example, in Figure 2, the context "Two brown dogs fi fighting over red frisbee" provides with the essential relationships between the objects, c.g., fighting over a red frisbee. Here, we adopt Vinvl [39] as the caption model C'(-).
In summary, we extract regional visual and positional information as {u,} J. and {by {b and textual descriptions for the objects and the relationship between the objects as 76 and c. In the next section, will elaborate how knowledge.
these regional information sources to retrieve external
3.2 Object-Centric Knowledge Retrieval Module
Inspired by KAT [8], we consider both the explicit knowledge and implicit knowledge. But different from it, we utilize regional visual information to help boost the final performance.
3.2.1 Explicit Regional Knowledge
Since the questions from knowledge-based VQA [22] are general and open-ended, introducing external knowledge is important for model to generate accurate answers by providing extra and complementary knowledge beyond visual contents of input images.
External Knowledge Base. We construct external knowledge base Q by constructing subset from Wikidata [31] following KAT [8]. Specifically, we extract commonly appeared cate- gories, i.e., Role, Point of interest, Tool, Vehicle, Animal, Clothing Company, Sport, to form the subset Q. Each item in Q consists of an entity and a corresponding description, e.g., one entity and its description can be "pegboard" and "board wall covering with reqularly-spaces holes for insertion of pegs or hooks" respectively.
Regional Knowledge Retrieval. As mentioned earlier, vision-language models like CLIP are capable of selecting the most relevant text from a set of texts. We reformat the entries knowledge base Q as "(entity) is [description]", and denote the reformatted text set as T. We retrieve the

---

top-K K most relevant knowledge entries among all the regional proposals as explicit knowledge &, j 1,
= { = arg Top)
(4)
d,T
where K denotes the number of retrieved explicit knowledge samples. In implementation, use FAISS [13] to speed up the computation of Equation (2) and (4),
3.2.2 Implicit Knowledge with Regional Descriptions
Large language models, e-g. GPT-3 [3], not only excel in many language tasks, but also memorize lots of commonsense knowledge from its training corpus [37]. Therefore, we exploit GPT-3 [3] as our implicit knowledge base by reformulating the task as open-domain question answering.
Context-Aware Prompt with Regional Descriptions. We design the textual prompt based on question Q, caption c, and tags H. Different from PICa [37] and KAT [8] that use whole-image feature to get the tags, we utilize fine-grained regional features to extract regional tags. Specifically, we adopt the prompt X be "context: : {caption] + [tags]. question: [question]". In this way, the language model is also supplemented with regional visual information.
Implicit Knowledge Retrieval. Finally, we query GPT-3 model [3] which takes the reformulated prompt X as imput, and obtain predictive answer. Since some of the questions may have ambiguity, follow the prompt tuning procedure of PICa [37] and get answer candidates {o,, " In addition to answer prediction, we also aim for acquiring corresponding explanation en from GPT-3 model to obtain more context information. To be feeding the text prompt "[question) [answer candidate). This is because" " into GPT-3. Note that "I "[question]" )" and "(answer candidate) are input question Q and GPT-3's answer Oy for image respectively. The final retrieved implicit knowledge can be denoted as I {(ow, e )
specific, the corresponding explanation is acquired by
3.3 Encoder-Decoder Module with Object-Centric Visual Features
Once we've retrieved the explicit and implicit knowledge and the regional information, we utilize the FiD network structure [11] to encode and decode retrieved knowledge and regional information.
Knowledge Encoder. For the explicit knowledge, reformat the input text as "entity: (entity) J description. ion: [description)" where the entity and the description is from the entries in the retrieved explicit knowledge &. We denote this text h4, where k = 1, K.
For implicit knowledge, we adopt input format as "candidate" [answer) evidence: (explana- tion]", where answer is the retrieved answer o,, and explanation is ea. Here, = I,... U, where U the number of answers provided by GPT-3. We denote the input text as 8u.
We then encode the knowledge in textual format by the FiD's encoder [30], which is denoted as Fe,
0 = Fo(hx,, Ba = Fe(su),
(5)
in which ag E ER E R and D means the embedding dimension.
Regional Visual Encoder. We introduce visual encoder for the regional visual embeddings M and positional coordinates {b, 1--1- We feed ey and b, into two different fully connected {u, [
layers, stack the outputs into a sequence of vectors, and then feed them into a transformer encoder Fv,
(C,b)))) where f E (22M)x (-) and F-2 are two different fully-connected layers, Concat(-) is the concatenation operation along a new dimension.
f (Coonattt,FC FC,(),
Context-aware Question Encoder. To better leverage the context information, we replace the input question Q by the context-aware prompt X, then encode it by the same transformer encoder F.(...
q Fx(X,,
where y D and y means encoded context-aware question.
K Generative Decoder. We have obtained the knowledge encoding (au k= and ( visual encoding f, and context-aware question encoding q. Note that as the outputs of the encoder Fg. they
5

---

are all sequences of vectors. We then concatenate these vectors along the first dimension, and feed them into the FiD's decoder Fa.
y = F((Concat((11"" , .a,, $1, ...
f.),
(8)
where y means the generated answer. The cross entropy loss function is adopted to train the model,
L Σ t=1 log Po
(9)
in which L is the length of the ground truth answer text. De is ground truth text at the position l and is the model parameters.
Model Ensemble. To generate more accurate answers, one promising method is to leverage multiple trained models, ie,, model ensemble. our experiments, we just train three models whose initialized seeds are different, and then the most frequent result among the generated results from these three models selected as final answer prediction for each sample.
3.4 Relationship to Existing Works
Inspired by KAT [8], REVIVE also retrieves two types of knowledge, i.e. implicit and explicit knowledge. Different from KAT, we explore how to better use visual features to retrieve knowledge. Motivated by the fact that the retrieved knowledge should also corresponds to individual concepts in the images in addition to the global theme, knowledge, and use regional descriptions to obtain the implicit knowledge. Moreover, we integrate the visual representation of object regions with retrieved knowledge in the answer generative model. The pipeline differences between KAT [8] and our method can be explained in Figure 1.
extracted regional features to retrieve external
There're re two works [36, 21] that leverage visual regions for knowledge-
VQA well. However, MAVEx [36] considers object regions as kind knowledge without using their visual representation to retrieve other knowledge, KRISP [21] utilizes object regions to learn implicit knowledge by transformer-base model and retrieve external knowledge by the text symbols of these regions, while proposed REVIVE explores how to better leverage visual representation to retrieve knowledge and integrate their visual features with retrieved knowledge into the answering model.
Experiments
4.1 Experimental Setup
Dataset. OK- VKA dataset [22] is selected for evaluation, which is currently the largest knowledge- based VQA dataset. OK-VQA dataset includes 14055 questions associated with 14031 images from MSCOCO dataset [18]. Its questions cover variety of knowledge categories, and are annotated by Amazon Mechanical Turkers. The training and testing split consist of 9009 and 5046 samples respectively. Each data sample is made up of one question, one corresponding image and 10 ground- truth answers. To construct the general domain tag set T, we collect the most frequently searched 400K queries in Bing Search the tags.
Pre-processing. We utilize the pre-trained visual grounding model GLIP-T [16] to detect object- centric region proposals by using its default prompt "Detects person, bi cycle, car, toothbrush", which contains all object categories of MS-COCO dataset [18]. The captions of images are obtained by the pre-trained Vinvl-Large model [39]. For explicit knowledge and regional tag retrieval, we choose CLIP model (ViT-B/16 variant)
[25]. In our experiments, we adopt U, K, M and P as 5, 40, 36 and 30 respectively. Note that the models of CLIP, GLIP, Vinvi and GPT-3 all frozen during usage.
Implementation Details. We use 4 x NVIDIA V100 32Gb to train models for 10K steps, with batch size of 8. The learning rate is Se and AdamW W [19] is chosen as optimizer. The warm-up steps are IK and the trained models are evaluated every 500 steps. We initialize our model with the pre-trained T5 model [26], i.e., Targer following KAT [8]. The encoder F, in Equation (6) consists of 9 transformer layers [30]. Note that evaluate the prediction results after normalization, and the normalization process mainly includes removing articles, punctuation and duplicated whitespace and lowercasing [4, 14].
Evaluation Metric. In our experiments, we choose the soft accuracy of VQAv2 [2] as evaluation metric for comparison.

---

Table Results comparison with existing methods on OK- VQA dataset [22], the evaluation metric
(i.e., accuracy) is in %.
Method
Knowledge Resources
Accuracy (%)
Q only [22]
14.9
MLP [22]
20.7
BAN [22]
25.1
BAN+AN [22]
Wikipedia
25.6
MUTAN [22]
26.4
BAN+KG-AUG [15]
Wikieedia+-onnceptNee
26.7
MUTAN+AN [22]
Wikipedia
27.8
ConceptBERT [7]
ConceptNet
33.7
KRISP [21]
Wikipedia ConceptNet
38.4
Visual Retriever-Readet [20]
Google Search
39.2
MAVEx [36]
Cikppedi= Images
39.4
PICa-Base [37]
Frozen GPT-3 (175B)
43.3
PICa-Full [37]
Frozen GPT-3 (175B)
48.0
KAT (Single) [8]
Wikidata+Frozeen GPT-3 (175B)
53.1
KAT (Ensemble) [8]
Wikidata+Frozeen GPT-3 (175B)
54.4
REVIVE (Single)
Wikidata+FFrozen GPT-3 (175B)
56.6
REVIVE (Ensemble)
Wikidata+Frozeen GPT-3 (175B)
58.0
4.2 Comparison with State-of-the-art Methods
As shown in Table I, we
that previous works (e.g., KRISP [21], Visual Retriever-Reader [20]
and MAVEx (36]) achieve similar performances, about 38.4% to 39.4% accuracy. Until recently. PICa [37] is the first one that exploits the pre-trained language model GPT-3 [3] knowledge base for knowledge-basc based VQA task and KAT [8] further introduces Wikidata [31] as an external knowledge resource, these two works obtain significant performances compared with previous ones.
The proposed REVIVE can outperform all existing methods by large margins. Specifically, even using the same knowledge resources (i.e., Wikidata [31] and GPT-3 [3]). our single model can achieve 56.6% accuracy versus previous state-of state-of-the-ar the-art method KAT's 53.1 % accuracy, when using model ensemble, method achieve 58.0% % accuracy compared with KAT's 54.4% accuracy. These results demonstrate the effectiveness of the proposed approach.
4.3 Ablation Study
Next, conduct extensive ablation studies the single model to figure the influence of each component of REVIVE.
Effect of Region Proposal Number. We perform the ablation study figure out the effect of using different region proposal numbers. The results are displayed Table 2. It can be observed that when the region proposal number is 36, the model achieves optimal performance. We conjecture that when the number of region proposals is too large, there are some meaningless and noisy region proposals, while if the number of region proposals is too small, many essential object-centric regions are ignored, which both hurt the model's performance.
Different Knowledge Retrieval Methods. The way of utilizing visual representation for retrieving knowledge plays an important role in knowledge-based VQA. We show the results of using three kinds of knowledge retrieval methods, i.e., image-based, sliding window-based and region-based, in Table 5. Note that sliding sindow-based approach follows KAT [8]. Specifically, we first resizes input images to 384 x 384 and then crop the images with a sliding window whose size is 256 x 256 and stride size is 128. We can observe that the proposed region-based approach achieves best performance and surpasses sliding window-bat baaed method by 1.8% points, which can validate the effectiveness of exploiting region- region-based visual representation for retrieving knowledge.
Effect of Regional Tag Number. In order to introduce more semantics into contexts, propose to add region-aware descriptions (i.e., regional tags) behind given contexts. We report the results of using different regional tag number for text prompt X in Table 3. The results show that when the
7

---

Table 2: Ablation study on using different region proposal number.
Table Ablation study on using different regional tag number.
region proposals
Accuracy (%)
of regional tags
Accuracy (%)
5
54.7
8
56.2
18
55.8
24
56.4
36
56.6
30
56.6
50
56.2
50
56.3
Table 5: Ablation study adopting different meth- ods for retrieving knowledge.
Table 4: Ablation study on adopting bounding box coordinates.
Method
Accuracy (%)
Positional coordinates
Accuracy (%)
Image- based
53.2
55.8
Sliding window-based Region-base
54.8
56.6
56.6
number of regional tags is 30, it achieves optimal performances. fact, when the number of regional tags large, we'll retrieve relatively irrelevant object tags, sacrificing the model's performance.
Effect Positional Coordinates. In addition to incorporating visual representation of object-centric region proposals into the model, we also adopt the position information (i.e., positional coordinates). The results of whether using positional coordinates are reported in Table 4. Introducing regional coordinates improve the performance by 0.8 % points.
Effect of Each Component. Finally, we showcase the results of using different components of REVIVE in Table 6. We can observe that the introduced components can consistently improve the model's performance. Especially for knowledge retrieval, using the regional descriptions can improve the performance of implicit knowledge by 1.2 %, while adopting the regional features can boost the performance of explicit knowledge retrieval by 1.1 %.
The object-centric region features can achieve 1.4 % points improvement, feeding context-aware questions, which can be denoted as prompt "context: [caption]- question: (question)" into the answer generative model attain 0.5 % points gain, further introducing regional descriptions (i.e... regional tags) into contexts, i.e.. prompt X, has 0.7 points improvement. These results can validate the efficiencies of our proposed components.
Table 6: Ablation study different components of REVIVE. Note that "Imp." and "R-Imp." mean implicit knowledge retrieved without and with the proposed regional descriptions, "Exp." " and "R- Exp." " mean explicit knowledge retrieved without and with regional features, "Visual" represents object-centric region features, "Contexr" and "Tag" mean introducing the contexts and regional descriptions (i.e. tags) into the final answering model respectively. "Acc." means accuracy.
Imp. R-Imp.
Exp.
R-Exp.
Visual
Context
Tag
Acc. (%)
51.2
52.4
52.9
54.0
55.4
55.9
56.6
4.4 Quantitative Result Analysis
Finally, present the quantitative results and provide analysis for enror cases, that can have clear insight into the proposed approach.
Visualizing Results. The success cases of our approach are shown in Figure 3. We can observe that our approach can accurately retrieve implicit and explicit knowledge, which corresponds the detected object regions, and deal with the relationship among these object areas. For example, in

---

Object Regions
Object Regions
Explicit Knowledge
Explicit Knowledge
Shelpok:
Brazilian Turrier:
Kazaka flsthread, using hutter. milk and sugar
Dog
What on this sundwich?
Q: What breed this?
Conaforter:
sandwich, sandwich, stack feod, food, pecson, cundour
C:
C: brown dog laying conch with
Type bedoover, often not thick as devet
Hambbrger topped cheese
baanetts, mummal, wall, dog, sofa, floor
A: Choese
Implicit Knowledge
A: Tenior
Implicit Knowledge
GT:: ("Chocse'. "Chocso", "Chocae". 'Cheese', "Cheese', CCheese Choese'
impoctant purt the sundktic
The cleasu uaad
GT: l'Terrier' "Torier', 'Terrier', 'Terrie''
'Crossbreed' 'Crussroedd; ; bull', 'Pit bull', 'Shepurd", "Shopind"]
Poodle: The dog is poodle
"Omelel", 'Omelet')
Acc.: 1.0
masn ingoedient
Cheddari
Acc.: 1.D
Tervier: The doy in terricr
Figure 3: Representative success cases the proposed REVIVE on OK-VQA dataset [22]. "Q", "C", "A and "GT" denote question, context, predictive answer, ground truth answers respectively. Note that the underlined text represents regional tags and five tags are selected for illustration. We rescale all the object regions to the same size for clearer view. "Acc." means accuracy.
Object Regions
Object Regions
Explicit Knowledge
Explicit Knowledge
Sami-4rsil mrni--rrilerr truck:
TPP Alle 24++Belle 2/4:
Regional clustrc mulliple uni muttic-gang seitiii-gange FIP moxur cruch
more
tn cerry fright
Q: What you build with these?
Q: What temperature is this?
C: red track canying logs road. tree.
Tree trnk uscal construction (whoke or can up into lumber)
Timber Log:
C: A red and white tmin on the tracks with traveling, free
Track Ballast:
Trackbed upon
neld, gnass purk, red
people. tmin,
A: Cabin
Implicit Kndwledge
A: Warm
Implicit Knowledge
GT: ['House', 'House', 'House", 'House'. 'House, 'House', 'House", 'House',
House: The house the world
GT: I'Cal", "Cold", "Cold', "Cold", 'Cold', "Cold', 'Cold, 'Cold',
Hot: The lemperatire above the normal temperature
important
'Log cabin', 'Log cabin')
Log Cabin: The logs are nonched toguther the
'Pleasant', ; 'Pleasant")
Acc::00
lempocature
Figure 4: Representative failure cases of the proposed REVIVE OK- OK-VQA dataset [22].
the left example of Figure 3, our method can recognize the potential referred objects and retrieve useful knowledge (e.g., cheeseburger and cheese), thus generating the correct answer, while in the right example, our method can also retrieve important knowledge (e.g., brazilian terrier) to answer the breed of the referred dog.
Failure Cases Analysis. We showcase the failure examples in Figure 4. As shown in the left example, even though the prediction result Cabin doesn't appear in the ground truth answers, the generated answer of our approach is still reasonable for such scenario. For the right example, predicted result is wrong due to the difficulty of answering such a general question. From figure 4, we can also observe that method can generate useful object-centric regions and accurately retrieve corresponding knowledge, especially explicit knowledge, which can demonstrate the potential of the proposed method.
Limitations and Broader Impact
The quality of constructed Wikidata subset and designed textual prompt can influence final retrieved knowledge. In addition, the detector for obtaining region proposals also affect retrieved knowledge and visual features, all these factors affect the models' performances.
This paper proposes novel approach REVIVE for knowledge-baid VQA. REVIVE can help models to efficiently use visual and language information sources to answer open- domain questions. It can also generalize real-life products, e.g., dialogue robot. However, the failure cases of REVIVE will be negative to the society when using it as the educational technique. There may also exist certain VQA
forms of bias, i.e,, the model may predict biased answers if the training data of knowledge-basse contain certain bias. For example, [1] suggests the model may be driven by superficial correlations in the training data, and [9] shows the VQA datasets may contain gender and racial bias that may cause the models to learn harmful stereotypes.

---

References
[1] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971-4980, 2018.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Cawrrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. arXiv preprint arXivv 170.00005, 2017.
[5] Noa Gareia and Yuta Nakashima. Knowledge-baid based video question answering with unsupervised scene descriptions. In European Conference on Computer Vision, pages 581-598. Springer, 2020.
[6] Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. Knowit vqa: Answering knowledge-based questions about videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages (0826-10834, 2020.
[7] François Gardères, Maryam Ziacefard, Baptiste Abeloos, and Freddy Lecue. Conceptbert: Concept-aware representation for visual question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 489-498. 2020.
[8] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for vision-and-language. arXiv preprint arXiv:21 12.iv:211.008114 2021.
[9] Yusuke Hirota, Yuta Nakashima, and Noa Garcia. Gender and racial bias in visual question answering datasets. arXiv preprint arXiv:2205.08144 2022.
[10] Gautier Izacard and Edouard Grave. Distilling knowledge from reader retriever for question answering. arXiv preprint arXiv:2012.0455 045844, 2020.
[11] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282. 2020.
[12] Haojun Jiang, Yuanze Lin, Dongchen Han, Shiji Song, and Gao Huang. Pseudo-q: Generating pseudo language queries for visual grounding. arXiv preprint 1xXiv:220300843) 2022.
[13] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scas similarity search with gpus. IEEE Transactions Big Data, 7(3):535-54) 547, 2019.
[14] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv: 1906.00300 2019.
[15] Guohao Li, Xin Wang, and Wenwu Zhu, Boosting visual question answering with context- aware knowledge aggregation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1227-1235, 2020.
[16] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jeng-Neng Hwang, al. Grounded language-image pre-training, arXiv preprint arXiv:2112.0388 2021.
[17] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for lanion-language tasks. In European Conference on Computer Vision, pages 121-137. Springer, 2020.
[18] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[19] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint axXiv:171.05101, 2017.
10

---

[20]
Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. Weakly-supervised vistal-retriever- reader for knowledge-baase question answering. arXiv preprint arXiv:2109.04014 2021.
[21] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111-14121, 2021.
[22] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3195-3220003 53043204,22 2019.
[23] Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. Out of the box: Reason- ing with graph convolution nets for factual visual question answering. Advances in neural information processing systems, 31, 2018.
[24] Medhini Narasimhan and Alexander Schwing. Straight to the facts: Learning knowledge base retrieval for factual visual question answering. In Proceedings of the European conference on computer vision (ECCV), pages 451-468, 2018
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763 PMLR, 2021.
[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.100 10683, 2019.
[27] Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar: Kvqa: Knowledge- aware visual question answering. In Proceedings of the AAAI conference on artificial intelli- gence, volume 33, pages 8876-8884, 2019.
[28] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference on artificial intelligence, 2017.
[29] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: 1908.098.000 08530, 2019.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[31] Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Com- munications of the ACM, 57(10):78-85, 2014.
[32] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for vision-language representation learning. arXiv preprini 1
2021.
[33] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40(10):2413-2427) 2017.
[34] Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. Explicit knowledge-based reasoning for visual question answering. arXiv preprint 2015.
[35] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 2021.
[36] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. multi-modal answer validation for knowledge-based vqa. arXiv preprint arXiv:2103.12244 2021.
[37] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Li- juan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. arXiv preprint 2rX1v:2109.05014, 2021.
11

---

[38] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:21 11.1111111 1432, 2021.
[39] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.
[40] Zihao Zhu, Jing Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu. Mucko: multi-layer cross-modal knowledge reasoning for fact-based visual question answering. arXiv preprint arXiv:2006.09077 2020.
12


Reviews:
[{"review_id": "TScHQ1ueEom", "reviewer": "Reviewer_5EMT", "summary": "This paper addresses the task of knowledge-based visual question answering (VQA). Given a question and an image, the aim is to answer by using external knowledge bases. The proposed model is built on top of large pre-trained models (GLIP, CLIP, GPT3, Vinyl, etc.) to extract and encode visual features, generate captions, retrieve external knowledge, encode implicit knowledge and predict an answer. The main contribution of the paper is to make better use of the visual information than in previous work, showing in the experiments that visual features can contribute to improving performance on the OK-VQA dataset.", "questions": "- Did you compare GLIP and Faster R-CNN object detectors and how they affect the final performance?\n- Will the code and models be made publicly available?", "limitations": "A more in-depth discussion about bias (both distributional [6] and societal [7]) in knowledge-based VQA and how it affects the proposed model would be interesting, but not necessary. \n- [6] Agrawal et al. Don't just assume; look and answer: Overcoming priors for visual question answering. CVPR 2018.\n- [7] Hirota et al. Gender and Racial Bias in Visual Question Answering Datasets. ACM FAccT 2022.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper and the proposed model are well-motivated. Previous work on knowledge-based VQA did not make full use of the image signal for answering prediction, relying mostly on external knowledge.\n\n- Showing that the information contained in the image is important for answering knowledge-based visual questions is an important contribution to the field. Until now, image features were dismissed and most of the attention was put on the language and knowledge signal. The paper shows that more attention should be paid to visual information.\n\n- The proposed method outperforms previous work on the OK-VQA dataset by a large margin, showing the efficacy of incorporating image information in the answer prediction.\n\n- Ablation study is conducted showing results when different parameters of the model are modified. According to the results, all of the components of the proposed model have a positive contribution to the overall performance.\n\nWeaknesses\n\n- Although previous work did not make full use of the image information on knowledge-based VQA, there are previously proposed methods that incorporate the information in the prediction module (as opposed to L36 “only use visual information for knowledge retrieval but ignore it in the final answering model”). A few examples: [1][2][3] (and others). Also, some work on video knowledge-based VQA also uses captions and object relationships in the prediction module [4][5]. I don’t think this changes the main point of the paper, as these models could not make the visual features improve significantly the final performance, but it would be better to accurately discuss it in the paper.\n\n- Text in the figures looks blurred. The resolution should be improved.\n\nReferences\n- [1] Narasimhan et al. Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering. NeurIPS 2018\n- [2] Narasimhan et al. Straight to the facts: Learning knowledge base retrieval for factual visual question answering. ECCV 2018\n- [3] Shah et al. KVQA: Knowledge-Aware Visual Question Answering. AAAI 2019\n- [4] Garcia et al. KnowIT VQA: Answering Knowledge-Based Questions about Videos. AAAI 2020.\n- [5] Garcia et al. Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions. ECCV 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H7lKXbcBSh_", "reviewer": "Reviewer_uNM2", "summary": "The authors observe that in most state-of-the-art knowledge-based VQA methods: \n1) visual features are extracted either from the whole image or in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; \n2) visual features are not well utilized in the final answering model, which is counter-intuitive to some extent. \nBased on these observations, they propose a new knowledge-based VQA method REVIVE, which tries to utilize the explicit information of object regions not only in the knowledge retrieval stage but also in the answering model.\nThe authors perform several experiments on the standard OK-VQA dataset and achieve new state-of-the-art performance. \n", "questions": "It would be better to further emphasize the necessity of each of the proposed components in the methodology section. What are the technical reasons why each of the components is required for better training and performance improvement?", "limitations": "The authors addressed the limitations of the proposed method and the potential negative social impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- The proposed method shows favorable performance compared to the recent baselines.\n\n- The authors provide various analyses on the design choices including the hyper-parameters that affect the performance improvements.\n\nWeakness\n\n- The technical novelty in this paper is too weak. This paper is an engineering paper that combines several engineering tricks without a unified motivation or a theoretical background. Most of the components of the proposed method are hackish tricks such as changing the input cues of some modules in their model.\n\n- In the ablation study in Table 6, the most effective components that give the largest performance gain are explicit knowledge (which is already introduced in the KAT method) and ensembling (which is a well-known trick to boost the performance). The other components such as changing or adding input cues show almost negligible performance gain which is less than 1%.\n\n\n\n========== ------- Comments after the rebuttal ------========\nI have read the other reviewer's comments and the author's rebuttal, which addresses most of my concerns. Therefore, I would like to raise my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D3bIQfmZAjK", "reviewer": "Reviewer_3J5a", "summary": "This paper augments the KAT model with regional visual representations for Outside-knowledge Visual Question Answering (OK-VQA) and achieves new state-of-the-art results. Precisely, the regional tags are used for retrieving better implicit knowledge using GPT-3. Regional features are used to retrieve explicit features from WikiData and serve as an additional hint to the FiD answer generator. ", "questions": "See weakness.  Also, the content of the explicit knowledge may not be relevant to the question, it is also good to do the ablation that only encodes the entities from CLIP without the knowledge sentences to prove the contribution of the regional description.", "limitations": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper presents a critical weakness of the existing SOTA approach that the regional features are missing. The authors address this issue by introducing the regional descriptions in different formats for different stages of the REVIVE framework including implicit knowledge retrieval and the answer generation stage. The experiments verify the value of introducing the regional features.\n\nWeakness: \nThe experiement part is not clear in that we do not understand how the regional features benefit the performance. Concretely, as the regional descriptions are injected into different stages of the framework, how do these descriptions influence the performance of implicit knowledge retrieval? Also, the left qualitative example in Figure 3 confuses me as none of the knowledge retrieved mentions the right answer ``battery'' and the regional tags introduce additional misleading objects say the desktop. \n\n\nOriginality: The paper is original.\nQuality: The paper presents an interesting idea of using regional features to augment the prompts for better OK-VQA results.\nClarity: The paper is well written.\nSignificance: Ok,", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept