Why Does the VQA Model Answer No?: Improving Reasoning through Visual and Linguistic Inference
Authors: Seungjun Jung
Junyoung Byun
Kyujin Shim
Changick Kim
Venue:  (2020)
Keywords: Image Captioning
Visual Question Answering
Explainable A.I
Beam Search
Constrained Beam Search

Abstract: In order to make Visual Question Answering (VQA) explainable, previous studies not only visualize the attended region of a VQA model but also generate textual explanations for its answers. However, when the model’s answer is ‘no,’ existing methods have difficulty in revealing detailed arguments that lead to that answer. In addition, previous methods are insufficient to provide logical bases, when the question requires common sense to answer. In this paper, we propose a novel textual explanation method to overcome the aforementioned limitations. First, we extract keywords that are essential to infer an answer from a question. Second, for a pre-trained explanation generator, we utilize a novel Variable-Constrained
Beam Search (VCBS) algorithm to generate phrases that best describes the relationship between keywords in images. Then, we complete an explanation by feeding the phrase to the generator. Furthermore, if the answer to the question is “yes” or “no,” we apply Natural Langauge Inference (NLI) to identify whether contents of the question can be inferred from the explanation using common sense. Our user study, conducted in Amazon Mechanical Turk (MTurk), shows that our proposed method generates more reliable explanations compared to the previous methods. Moreover, by modifying the VQA model’s answer through the output of the NLI model, we show that VQA performance increases by 1.1% from the original model.

Under review as a conference paper at ICLR 2020
WHY DOES THE VQA MODEL ANSWER No?: IMPROVING REASONING THROUGH VISUAL AND LIN- GUISTIC INFERENCE
Anonymous authors Paper under double-bline review
ABSTRACT
order to make Visual Question Answering (VQA) explainable, previous studies not only visualize the attended region of VQA model but also generate textual explanations for its answers. However, when the model's answer is 'no," existing methods have difficulty in revealing detailed arguments that lead to that answer. In addition, previous methods are insufficient to provide logical bases, when the question requires common sense answer. In this paper, propose novel textual explanation method to overcome the aforementioned limitations. First, we extract keywords that are essential infer answer from question. Second, for pre-trained explanation generator, we utilize novel Variable-Constrained Beam Search (VCBS) algorithm generate phrases that best describes the re- lationship between keywords in images. Then, we complete an explanation by feeding the phrase the generator. Furthermore, if the answer the question is "yes" or "no," we apply Natural I Langauge Inference (NLI) to identify whether contents of the question can inferred from the explanation using common sense. Our user study, conducted in Amazon Mechanical Turk (MTurk), shows that our proposed method generates more reliable explanations compared to the previous methods. Moreover, by modifying the VQÅ model's answer through the output the NLI model, we show that VQA performance increases by 1. % from the original model.
INTRODUCTION
Recently, the interpretability and reliability of Artificial Intelligence (AI) are being seriously dis- cussed (Kim et al. 2018b), since the opacity of its internal structure makes it hard to figure out the exact reason when it malfunctions (Gunning 2017 Kurakin et al. 2016). Previous works show that
decision process of conventional machine learning models (e.g.. decision tree) can be modified to be transparent while maintaining their original performance (Si & Zhu 2013 Letham et al. 2015). For deep learning models, Ribeiro al.) (2016) suggest method for reasoning their decisions by visualizing critical parts of the input for the image classification. However, it is harder to infer the decision process for more complicated tasks such as VQA.
VQA is one of the most active research fields in AI with strong expectation for real applications. It can be used for either assisting radiologists (Letham al. 2015) help visually impaired people to understand their environment more flexibly (Antol et al. 2015). Therefore, ofate-of-the-art VQA models have been continually proposed, but their internal processes are yet opaque. Displaying attended region might be used for indirect reasoning. However, it has limitation on describing what the model perceives in detail. The most spotlighted recent research for endowing interpretability to VQA model is making the model to produce answer and its explanation at the same time (Park et al. 2018: Li et al. 20188) Hudson & Manning 2011)). To make possible, they provide a ground truth explanation dataset for each image, question and answer pairs, and fine-tune the model to generate explanations in an end-to-end process.
Previous explanation models can successfully describe relationships between objects in the image. However, when the answer to the question is "no," the previous methods have difficulty on providing logical bases. Also, even though there questions that require common sense, existing models

---

Under review as a conference paper at ICLR 2020
vegetabdes on the plate?
There are vegeraties on the plats, siic of pizza sitting on plate.
Constraints set
VCBS decoding
Ao
Qo this Brokan Inae??
Exsden
Thrs broonn trass a train that siting
Constraints
VCBS decoding
on the traco..
Sample results using the proposed method. We first identify several constraints (blue nodes) from the question and answer. Then, the proposed Conable-Constrained Beam Search (VCBS) algorithm searches the best possible sentence which satisfies the given constraints far as possible.
Figure 1:
do not utilize it: previous models memorize explanations appropriate to each situation in training datasets (Park al. 2018 Zellers et al. 2019).
In this paper, we propose novel method that generates explanations for a VQA model's answer. First, the proposed method selects keywords that play an essential role in inferring the answer the question. We extract these keywords in the same way as Wang et al. (2019). Next, given a pre-ttriined explanation generator, we use the proposed Variable-Connsraineed Beam Search (VCBS) algorithm to generate a phrase that best describes the relationship between the keywords by inferring the image (See Fig. D. The phrase is fed to the generator to complete the remaining part of the explanation. Finally, we use Natural Language Inference (NLI) (Wang et al. 2019 Talman al. 2019) to check whether the contents of the question can be inferred from the explanation using common sense. For the overview of our method, please refer Fig.
Our main contributions are:
We propose the novel method generate textual explanations for the VQA model's 's answer. The proposed method not only produces more appropriate explanations (Kim et al.) 2018b) but can also be used as post-processing of the VQA model boost the performance. We validate our method on one hundred randomly sampled data from VQA 2.0 test-std split, through a user study conducted MTurk. By modifying the predicted answer using the generated explanation, we also compare the prediction accuracy of the method on complete VQA 2.0 test-std split. Our proposed method has gained MOS score higher than previous methods, and get VQA performance improvement of about 1.1 1.1%
We propose the the visual relationship between objects and attributes of interest in the image. Unlike the previous Constrained Beam Search (CBS) algorithm (Sipser et al. 2006), the VCBS has the variable accepting states. The variable accepting states help satisfy the given constraint as much as possible but allow the final beam to exclude some constraints necessary.
Beam Search (VCBS) algorithm that helps to identify
2 RELATED WORKS
We first summarize recent research VQA. Then we review the explanatory model for VQA and discuss the distinguishing characteristics of our work.
Visual Question Answering: The VQA challenge was first launched in 2016, and representative
studies that time (Fukui et al. 2016 Lu al. 2016 Kim et al. 2017) first extract image features
2

---

Under review as a conference paper at ICLR 2020
Spovet
So
Sg
Suit
Gaar
St
$+
Girit
Grree
The state diagram of the proposed VCBS algorithm: the E is the complete vocabulary, Estriat is set of strict constraints, Csaft is a set of soft constraints, C Cstrict U Cooft is the whole constraints. Cear = { [1y (fwo. ... 11-1),, wx) ncI 2 |C| - p} is a set of the variable constraints, where (fwo. ... w w/) = ww.... -.. } We is the next output word from the captioner, and p is the constraint loosening parameter. Note that Sg and Sp set of states (See Fig. 4.. Only the beams in accepting states in S can used as the feeding phrase (Sipser et al. 2006 Anderson et al. 2018 8a 2017).
Figure 2:
Quare there vegotables on the plate?
phraas
regpeiiiin plate: jomeeingggne regetablesee plute
vegetables on plate
vepetables veggeaalle ivluted
vegetables piatai vegetabla on something(nnt plans)
suggeiables in plate. here plate in image (noi vegetabble, no ling on plate)
Figure 3: Our method can specify the logical basis for the question when the answer is "no."
using residual networks (He et al. 2016) and established joint learning through question features and attention obtained from gated recurrent units (GRUs). the VQA 2017 challenge, Anderson et al. (018b) proposed a method that exploits Faster-RCNN (Ren et al. 2015) to extract image features the attended object candidates. Following its scheme, most the subsequent VQA studies also used Faster-RCNN, In the VQA 2018 challenge, Kim et al. 220 8a) presented bi-linear attention so that both question features and visual features were considered together when generating attention. They have shown that bilinear attention improves the residuality of the attention presented in (Kim) ct al. 2016) successfully.
Although various attention- based VQA models have shown excellent performance, the models heav- ily rely on linguistic bias rather than visual information most cases (Ramakrishnan et al. 2018). For this reason, recent studies are interested in identifying relationships among objects in images and using them to predict answers (Hudson & Manning 2019).
Explanatory model for VQA: Visualizing attention (Lu et al. 2016) utilizing Grad-CAM (Sel- varaju et al. 2017) is one the most widespread approach to understand the decision process of the VQA model. However, as mentioned in Li et al. (2018), we need detailed textual explanations rather than merely visualizing these areas. For this reason, recent research not only released ground truth explanation datasets (VQA-X (Park al. 2018) and VQA-E (Li et al. 2018)) but also pro- posed models that generates an answer and its textual explanation at the same time. While VQA-X made up of more elaborate and detailed explanations, VQA-E E composed of more extensive training and validation data. According to the results of the Li et al. (2018), fine-tuning through the VQA-E dataset increases the prediction accuracies of VQA models noticeably.
While both of the models are good at justifying the answer for the given question and image, they are poor at providing introspective explanations of the VQA model. First, the PJ-X model proposed by Park et al. (2018) consists of the sequence of VQA model and explanatory model. Since the PJ-X does not refer to the attention generated in the VQA model, the explanation does not reflect the decision process of the target VQA model. On the other hand, the multi-tase task VQA-E model proposed by Li et al. (2018) directly utilize the attention distribution of pre-trained VQA models. Although the model generates explanations based visual information about the attended area,
3

---

Under review as a conference paper at ICLR 2020
Q: what is color the horse?
Will deccding
A brown horse
broen
oafice
coloo horoo
Greedy decoding
8:
S
brown horse pullings cart.
fu Tayle
A: brown
(a)
Q: does the comfortable?
The look comfortable
hypothesis
LSTM
look
Groedy decoding
liyln
+
Entailment
comfortuale
CommpiiNer KG
cat is leying on couch.
premise
A: yes
Figure 4: Overview of the proposed method. (a) If the answer is not yes/no, set the answer as a strict constraint in VCBS decoding. (b) Else, we do not use strict constraint. Note that we use linguistic inference only if the answer is yes/no.
often produces sentences that are not related to the question. This because multi-task VQA-E does not directly specify the relationship between the objects related to the question.
Since is important for VQA task grasp the relationship between specific objects and attributes in an image, recent studies infer an answer through a scene graph a modular network formed for image (Hudson & Manning. 2019 Hu et al. 2018). Although both them provides explanatory insights into the model, the methods lack logical basis for the explanation in situations where the answer is "no." (See Fig. 3).
The goal of NLI is classify the relationship between two sen- tences: "premise" and "hypothesis." " If the "hypothesis" can be inferred from the "premise," we classify the relationship between the two sentences "entailment." While most NLI models LSTM to classify the relationship between the two sentences (Talman et al.] 2019), Wang et al. (2019) proposed method to grasp the relation by using ConceptNet (Speer al. (017) (a common
Natural Language Inference:
sense knowledge graph obtained through crowd-sourcing) as external knowledge. We use state-of-
art NLI models (the HBMP (Talman al. 2019), and the ConSeqNet (Wang al. 2019 identify the question could be answered using common sense and our explanation (See Fig 4(b)
APPROACH
For an input image, the Recurrent Neural Network (RNN) in the captioner forms the Variable Order Markov (VOM) model between words (Begleiter et al. 2004). This VOM model can be represented as directed graph G (E, E), where Σ is the complete vocabulary, and E is probability distri- bution at each node (See Fig. [ If the captioner ideal, the goal of VQA can be regarded as (i) checking whether path that contains specific nodes be found (for the question with yes/no answers. See Fig. [] (ii) or completing the path that pass through the specific nodes (for the other
type of questions. (Lin et al. 2019)).
4

---

Under review as a conference paper at ICLR 2020
Algorithm Variable-Coonstrainee Beam Search decoding
1: procedure VCBS(O,,,I,,,AA (, (p),
2: M <-
P for Memoization
3: for $ E S do
4:
B* t {e} if So clse ø
5:
for
t - to do
6:
E + Uvest(y w)ly' B² E, =
7:
B f- arg max max |F bb Ey EF O(y
> Beam extension
8:
s fs, (p) then
> for Beams in accepting states b keep beams satisfying more
9:
M - arg MuB EMUB |Cn y
constraints
10:
M t arg M Ey Ogy
only keeps the best beam
11:
return E M M + Φ else ø
3.1
CONSTRAINTS
First, identify Part-Of- Part-Of-Speecl (POS) tags (Márquez & Rodriguez 1998) of each word in the question. Then we take all detected nouns and verbs (excluding auxiliary verbs) as keywords. the VQA model's answer is "yes" "no," we add adposition to keywords (See Fig. Otherwise, use the answer as additional keywords (See Fig. 4(a)..
As in the work of Anderson et al. (2188)), construct Wordnet synsets (Fellbaum 2010) contain- ing plural and singular forms for each keyword. Thus, the nodes in Fig. correspond to the synsets of each keyword. We use a of these synsets as constraints simplicity in notation, we assume that C' is composed of keywords, not synsets, in the rest of this paper. The set C is further divided into two subsets, strict constraints (Cstrict) and soft constraints (Cooft). Cstrict is set of keywords that the final beam must satisfy in order to be in accepting state (See Fig. 22. Csaft a set keywords that are recommended to satisfy order for the final beam to be accepted, but not necessarily required. We use the answer as strict constraints only if the answer not "yes" "no" (See Fig.
C in our VCBS algorithm. For
3.2 VARIABLE-CONSTRAINE BEAM SEARCHING
For the RNN image captioner with parameter word w. at time step predicted given previous sequence of words (wo, w and the image 7.
Wi arg max pe (w, W wo, 1),
To naturally place the keywords into the sequence, we need partially specify sequence with Beam Search decoding (Anderson et al. 2018a). During the completion of beam, whenever the beam satisfies constraint, function as 5. The completed beam for given image may not satisfy all constraints: no clue is found in the image that satisfies the constraint (See Fig. [ or the constraint is given as a hyponym (See Fig. {(a).. Therefore, the satisfaction of constraints for each beam is not compulsory, but recommended (except the Cstriat).
change the state of the beam (See Fig. 4. We denote this state-transition
We express this condition as a set of accepting states Sr fs, (p) with variable p. where fs, (p) + 25, and S is a set of all possible states. The loosening parameter p denotes the number of keywords in C that can be neglected. The overview of our VCBS decoding is depicted in Algoirthm Note that = log (-).). b is the beam size, is the maximum sequence length. Since the completed beams may not stem from the phrase that satisfies the constraint, we keep the best phrase presents in Sp through Memoization. Finally, we complete the phrase y E M through the greedy decoding in Equation
5

---

Under review as a conference paper at ICLR 2020
league
glove
Icague
little
baseball
game
Irague
little
baseball
game
uniform
player
little
baseball
game
uniform
player
Soccer
football
uniform
player
(a)
(b)
(c)
Figure 5: An example illustration of subgraphs in Fig. 6(c) (a) Concepts Only, (b) One-Hop, (c) One-Hop Direct. Green nodes are concepts in the "premise," and the blue nodes concepts in the "hypothesis." Gray nodes are concepts in the ConceptNet KG (Speer et al. 2017) that do not belongs both cases.
3.3 LINGUISTIC INFERENCE
Some of the questions in VQA require inferences based background knowledge in addition visual information. Most of these types of questions require complex inference processes (Lin et al.) 2019) but can be reasoned relatively quickly when the answer is yes/no by checking whether the contents of the question can be inferred from visual information. Consequently, we first convert the question statement by moving the auxiliary verb behind the subjective noun (c.g.. Are dogs on the grass? = Dogs are on the grass). Using this statement as "hypothesis" and the complete sentence from y as "premise," we checked whether they are the "entailment" relationship (See Fig. (6C))
For the knowledge extension, we extract concepts from "premise" and "hypothesis." and identified their relationship in ConcepNet Knowledge Graph (ConceptNet KG). As in Wang et al. (2019), we use nouns and verbs (excluding auxiliary verbs) as concepts for "premise." However, for "hypoth- esis," we only use elements of remaining constraints C of concepts (See Fig. 4(b) Finally, we use Concepts Only (Wang et al.) 2019) and One-Hop Direct subgraphs with knowledge graph embedding (Wang et al.] 2014) for NLI. The One-Hop Direct subgraph only connects concepts from "premise" to "hypothesis" in one hop (See Fig. [5). These approaches reduce trivial subject- predicate-object (SPO) triples (Lu et al. 2019) significantly, hence, increase the performance of the NLI model for VQA.
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION
VQA model: We use two-glimpse BAN (Kmm et al. 2018a) for the target VQA model. All hyperpa- rameter settings follow the previous work [Kim al. (018a). The input image features are extracted from the Faster-RCNN (Ren et al., 2015). pre-trained using Visual Genome dataset (Krishna et al., 2017]. We adaptively extract 10 to 100 objects per image, as in (Kim al.) 2018a). For the question embedding in VQA, we initialize 300 dimension word embeddings with pre-trained GloVe vectors (Pennington et al. 2014).
We use the Up-Down captioning model (Anderson et al., 2018b) as our explanation generator. For training the model, we use both train and validation splits of MS COCO captioning dataset (Chen et al. 2013). While building the vocabulary, we drop any words if they occur less than five times in the training dataset, resulting in 10,512 words in Σ.
Explanation Generator:
VCBS decoding: Since all questions have different number of constraints |C), repeat the VCBS decoding with increasing values of 6 and until is not empty or = IC|: we initiate with b 3 and 0 (i.e., (b,p) {(3, 0), (4, 1), (5. 2), (6,3). (6,4), (6, (CD)). Note that we set an upper limit of as 6: beam size larger than 6 decreases the decoding speed drastically.
We train both the HBMP (Talman et al. 2019) and the ConSeqNet (Wang al. 2019) on the combination multiple datasets: SNLI dataset (Bowman et al.) 2015). SciTail dataset (Khot al. 2018), and MultiNI. II dataset (Williams et al. 2018). While generating explanations, we only
NLI model:

---

Under review as a conference paper at ICLR 2020
what is lying, next the oake?
PI-X
cake with knife the of
Mult
white plate with cut in half sandwich on too of it
VQA
Ours
close up of plate of food with a knife
knile
(a)
PI-
group people riding bikes down
Multi VQA-E
group riding ekateboard
Ours
the frees do shade the street, person skalebos street corner under trees.
A:
(b)
Q thas little leugue game?
X
ittic lcaguc game with child playing bascball
Mutt
ittic boy standing the dirt with basobal glove.
VoA-E
Ours
ittle langue player baseball uniform - this in a (I (l'bateball" -'tsA''game ['player'-' Related o'-ggmme"
te league
A: yes
(c)
Figure 6: Sample examples of VQA explanations using PJ-X (Park et al. 2018), Multi-task VQA- (Multi VQA-E) (Li et al. 2018), and the proposed method. We compare explanations for three cases: (a) for the usual questions and answers, (b) when the answer to the question is "no," (c) when answer requires linguistic inference. the "premise" and "hypothesis" are an "entailment" relationship, we denote it with "premise" + "hypothesis." We also visualize the SPO triples (Lu jet al. 2019) in the One-Hop Direct subgraph (i.c., ['subject' "rrrectt''prediiii 'predicate" 'object") I) to improve the explanability of the proposed method.
use the ConSeqNet (Wang et al. 2019) to visualize the linguistic inference process by One-Hop Direct subgraph (See Fig. 6(c).
Evaluation of VQA Explanations: Park et al. (2018) and Li et al. 2018 evaluated the performance by measuring the similarity between the generated deseription and its ground truth, using ordinary metrics in image captioning (i.e., CIDEr (Vedantam et al. 2015). Measuring the CIDEr score for VQA-X or VQA-E is measure of how well the justified explanation is reproduced. Since we focus on generating explanations through visual and linguistic inference, we use the Mean Opinion Score (MOS) to compare with these methods on one hundred random samples from VQA 2.0 test-std split (ee al. 2016 Antol al. 2015).
To minimize the bias in the survey, we conduct an online user study in MTurk (Paolacci et al, 2010), using hundred random samples from VQA 2.0 test-std split. According to preliminary survey within our research group, survey participants were likely to lost focus on the study if they asked hundred questions without rest. For this reason, divide 100 questions into five splits (i.e., 20 questions for each split). The study involved 52 Amazon Mechanical Turk workers (Turkers) for each split, who spent average time of about 12 minutes complete split. For each explanation, we ask its score for suitability as reasoning, ranging from I to 5 (i.e., indicates a very inappropriate explanation, and corresponds very appropriate explanation).
4.2 QUALITATIVE RESULTS
We compare the performance of the proposed method with state-of-the-an methods (PJ-X Park al. 2018), Multi-task VQA-E (Multi VQA-E) (Li et al. 2018),, and visualize the difference

---

Under review as a conference paper at ICLR 2020
between the previous methods and the proposed method in Fig. 6 When the answer to the question "no," previous methods only describe the image, but our method clarifies the reason (i.e., the model cannot find shade from the image) (See Fig. 6(b). Figure [(() shows example when the output of the NIJ model is "entailment." " The proposed method first creates a description of the objects and attributes of interest in the image. Then it shows how the answer the question is inferred from the description.
4.3 QUANTITATIVE RESULTS
Table 1: Mean opinion score (MOS) comparison on one hundred random samples from VQA 2.0 test-std split.
Method
split
split
split
split 4
split
overall
PI-X (Park al.
2018
2.941
3.2749
2.9987
2,9410
2.6095
2.9529
Multi VQA-E Li ct al
2018)
2.9038
3.0625
2.794
2,9108
2.5125
2.8367
Ours+ConSeqNer (Wang et al.
2019)
3,5194
3,4532
3.8415
3,4992
3.691
3.6089
Table 2: MOS score categorization by answer types.
Model
Yes/no
Number
Other
PJ-X (Park et al. 2018)
2.5154
3.3476
3,5145
Multi VQA-E (Liet al.
2018
2.4983
3.3754
3.4189
Ous++ConSeqNet Wang et al. 2019)
4.1321
3.3571
3.3909
Table 3: Test-dev scores of models VQA 2.0 dataset. We use two-glimpse BAN (Kim et al.
2018a) for the experiment. Note that the proposed methods use more data in training (i.c., SNLI dataset (Bowman et al. 2015), etc), and not fairly compared with the BAN.
Model
Overall
Yes/no
Number
Other
BAN (Kim et al.) 2018a)
68.13
84.44
53.38
57.50
IBP Ta Talman et al. 2019)
65.48
78.04
53.38
57.50
Ours+ConSeq et (Wang et al. 2019
69.23
87.10
53.38
57.50
We show the MOS results in Table The proposed method attained the highest score, while the PJ-X model (Park et al., 2018) attained similar score the Multi VQA-E model (Li et al.) 2018). We also categorize the MOS scores according to the answer types in Table 2 The result shows that our method for questions of yes/no answers has distinct advantages over the previous methods.
Finally, if the VQA model's answer yes/no, we change the answer to "yes" if the output of the NLI model is "enttalment" and to "no" otherwise (if [C]. we do not change the answer). Table shows the VQA performance comparison of models on the test server submission. The VQA perfor- mance is reduced when using the HBMP (Talman et al. 2019) for NLI, but the VQA performance is increased when the ConSeqNet (Wang et al. 2019) is used for NLI. Since the ConSeqNet trained using broader background knowledge (the HBMP does not use the ConceptNet KG (Speer et al. 2017), this result shows that background knowledge is essential for answering yes/no in VQA.
5 CONCLUSION
In this paper, we have suggested a novel method that generates explanations for VQA using visual and linguistic inference. Firstly, we describe the relationship between objects and attributes of in- terest in the image using the proposed VCBS decoding. The VCBS algorithm helps to generate explanations that contain essential concepts from the question. Next, if the answer is yes/no, we use NLI to check whether the content from questions can be inferred from the explanations. If it can be inferred, we visualize a One-Hop Direct subgraph between the explanation and the question. We evaluate the performance of the proposed method by measuring MOS. For the fair comparison, we conduct an online user study using MTurk. The result shows that our model attains better per- formance than the previous models. Finally, we modified the VQA model's 's answer based on the
8

---

Under review as a conference paper at ICLR 2020
NLI output and measured the performance. For the state-of-art NLI model (Wang al. 2019) using ConceptNet KG (Speer et al. 2017), we show an increase in the VQA model's performance.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. EMNLP, 2017.
Peter Anderson, Stephen Gould, and Mark Johnson. Partially-supervised image captioning. In Advances in Neural Information Processing Systems, pp. 1875-1886, 2018a.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In CVPR, volume pp. 6, 2018b.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Ron Begleiter, Ran El-Yaniv, and Golan Yona. On prediction using variable order markov models. Journal of Artificial Intelligence Research, 22:385-421, 2004.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2015.
Xinlei Chen, Hao Fang, Tsung-Yr Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint urXiv:1504.00325 2015.
Christiane Fellbaum. Wordnet. In Theory and applications of ontology: computer applications, pp. 231-243. Springer, 2010.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Mareus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. 2016.
David Gunning. Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. Proceedings of the European conference on computer vision (ECCV), pp. 53-69, 2018.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700-6709, 2019.
Tushar Khot, Ashish Sabharwal, and Peter Clark. SciTail: textual entailment dataset from science question answering. In AAAI, 2018
Jin-Hwa Kim, Sang-Woo Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung- Woo Ha, and Byoung- Tak Zhang. Multimodal residual learning for visual qa. In Advances in Neural Informa- tion Processing Systems, pp. 361-369, 2016.
Jin-Hwa Kim, Kyoung-Wao Woon On, Woosang Lim, Jeonghee Kim, Jung-Wer Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. 2017.
Jin-Hwa Kim, Jachyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. 2018a.

---

Under review as a conference paper at ICLR 2020
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explana- tions for self-driving vehicles. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 563-578, 2018b.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(():32-73, 20177
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1.XXv:6607.025333 2016.
Benjamin Letham, Cynthia Rudin, Tyler H McCormick, David Madigan, et al. Interpretable classi- fiers using rules and bayesian analysis: Building a better stroke prediction model. The Annals of Applied Statistics, 93)):3550-1371 2015.
Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. Vqa-e: Explaining. elaborating, and enhancing your answers for visual questions. 2018.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning. EMNLP-IJCNLP, 2019.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pp. 289- 297, 2016.
Xiaolu Lu, Soumajit Pramanik, Rishiraj Saha Roy, Abdalghani Abujabal, Yafang Wang, and Gerhard Weikum. Answering complex questions by joining multi-document evidence with quasi knowl- edge graphs. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 105-114. ACM, 2019.
Lluís Mårquez and Horacio Rodríguez. Part-of-speech tagging using decision trees. In European Conference on Machine Learning, pp. 25-36. Springer, 1998.
Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. Running experiments amazon mechanical turk. 2010.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Dar- rell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In 31st IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems, pp. 1541-1551, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144 ACM, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, et al. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, pp. 618-626, 2017.
Zhangzhang Si and Song-Chun Chun Zhu. Learning and-or templates for object recognition and detection. IEEE transactions on pattern analysis and machine intelligence, (5(9):2189-2200 2013.
10

---

Under review as a conference paper at ICLR 2020
Michael Sipser al. Introduction to the Theory of Computation, volume 2. Thomson Course Technology Boston, 2006.
Robert Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Aame Talman, Anssi Yli-Jyrä, and Jörg Tiedemann. Sentence embeddings in nli with iterative refinement encoders. Natural Language Engineering. 25(4))467-482 2019.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- bassdd image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang. Achille Fokoue, Bassem Makni, Nicholas Mattei, et al. Improving natural language inference using external knowledge in the science questions domain. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7208-7215 2019.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by trans- lating on hyperplanes. In Swenty-Fightle AAAI conference on artificial intelligence, 2014.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122 Association for Computational Linguistics, 2018. URL org/ antho 1N11 1101
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6720-6731, 2019.
A APPENDIX
A.1 QUANTITATIVE RESULTS
We used the BUTD model trained COCO captioning data (Chen al. 2013) explanation generator. The reasons for using it are as follows: 1) The COCO dataset is much larger than the VQA-X (Park et al.] 2018) and VQA-E (Li al.] 2018), we believe that the model trained on the COCO dataset can better understand the relationship between objects in an image. 2) We cannot train the Multi VQAE (Li al., 20188) (or PJ-X (Park al. 2018)) model on the COCO captioning dataset, because the dataset does not include VQA answers. However, we conduct additional exper- iments to confirm that our method can be used to improve the performance of the existing textual explanation model.
For this purpose, we extract 100 additional experimental samples from the VQA 2.0 test-std split. These samples are chosen as not duplicate the samples used in previous experiments. Similar to the experiment in Table [ we divided 100 samples into five splits. Unlike before, this survey involved 40 AMT workers for each split. Note that we used two different explanation generators in this experiment: the Up-Down (Anderson et al. 2018b) model trained on COCO captioning dataset (Chen et al. 2015), and the Multi (Li et al.] 2018) trained on VQA-E dataset (Li al. 2018). The results in Table 4 and Table Table show that our method helps to generate more appropriate textual explanations, especially when the VQA model's answer is "yes" "no."
A.2 QUALITATIVE RESULTS
We show qualitative results from the experiment conducted in Table 4 (See Fig. and Fig.
11

---

Under review as a conference paper at ICLR 2020
Table 4: Mean opinion score (MOS) comparison hundred random samples from VQA 2.0 test-std split. Note that we used two different explanation generators: the Up-Down (Anderson et al.
2018b), and the Multi-VQAE (Li al. 2018). Method
split
split
split 3
split 4
split 5
overall
Multi VQA-E (Li ct al. 2018)
2.9589
3.2089
3.1339
3.0835
3.0292
3.0829
Ours (Explanation Generator:
Up- Up-Down)+ConSeqNe
3.2284
3,4495
3,4712
3,3818
3.2932
3.3648
Ours (Explanation Generator: Multi NoConSeee
3.0146
3.3130
3.1450
3.1090
2,9795
3.1122
Table 5: MOS score (in Table
categorization by answer types.
Model
Yes/no
Number
Other
Multi VQA-E (Li et al. 2018
2.9875
2.8625
3.1951
Ours (Explanation Generator: Uo-DDown)+ConSeqNNet
3.3511
2.9990
3,4304
Ours (Explanation Generator Multi VoosEeeConSeqNNet
3.0465
2.8544
3.2122
is this profesioonl
baseball
basebal seball player ls swinging bat.
No,, boy that
VQJA
No. profoccional bascball game.
protessienel baseball game. young ay baseball game playing baseball
Multi vaJ-E
(a)
aebras
Multi
Yes, two zebras are standing next to each other.
stripes
VQA-E
Ours-UD
Yes. stancing top dry grass field
the zebras does have stripes on them. ('zebrs.. Ha sñ, 'stripes')
Yes, two zebra standing on s diri ground next to a zabra the zebras does have strices them. l'z HasA, 'stripes))
+Multi VOA-E
(b)
Q this room fully furnished?
Multi VQALE
Yes. there is tv mounted on the wall of an apartment.
this fully lumishad. room with coie h, chair, and television.
No, this fully fumished. The with emall tv, lamm. and few lights are on.
VQA-E
(c)
Multi
Yes, train pulling into the train station.
OurseUD
No, semeene is wailing on the train. A train on a with
the backpround
No. someone wailing on the train. Train an the tracks station with people walking around
MMuli VOA-E
Figure 7: Sample examples of VQA explanations using Multi-task VQA-E (Multi VQA-E) eLi al. 2018), our method using the Up-Down model (Anderson al.) 2018b) as the explanation generator, and our method using Multi VQA-E as the explanation generator.
12

---

Under review as a conference paper at ICLR 2020
Q is the boar a banker?
No. brown teddy bear tiingg front of laptop,
VQA-E
Ous+UD
No. the besr banker, small stuffed bear is ting on a desk
No, the bear benker, Having bear with a. book, bottle gatorade and bottle..
-Multi VQA-E
Q. this expensive
Multi VOA-E
No, there is stove with pots inside it.
Ours+UD
No. exgensive pot, siove with a on top of it.
Ours
No, this expensive pot. The pot being cooked in oven.
VQA-E
(b)
Figure 8: Sample examples VQA explanations using Multi-task VQA-E (Multi VQA-E) (Li et al. 2018), our method using the Up-Down model (Anderson et al. 2018b) as the explanation generator, and our method using Multi VQA-E as the explanation generator. It is shown that the Up-Down model trained on COCO captioning dataset better understands relationships between objects in the image.
A.3 STRICT ENTAILMENT AND WEAK ENTAILMENT
We conducted additional experiments with using the VQA model's answer as "yes," if the class probability of the NLI model for "Entail" is greater than "Contradictory." result "Neutral" (Weak Entailment). In this experiment, we found that the answer is often "yes," even when there is little correlation between premise and hypothesis. For this reason, answer as "yes" only if the NLI result is "Entail" (Strict Entailment) (See Fig. D
though the NLI
the
this picture 2o0?
premise
An image of giraffe and a giraffe in the grass.
hypothesis
This picture is taken at zoc.
One-Hop
('giraffe', AtLocation, 'zoo')
Direct
NLI result
(probability)
Entall:32% :22% Neutral 449% Contraclctory:
Entailment
X
Entailment
Figure 9: Comparison between strict entailment and weak entailment. In this paper, we regard the premise and hypothesis are in the "entailment" relationship only if the NLI result is "Entail."
13


Reviews:
[{"review_id": "rJlJyunRKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Interesting results and analysis but lack of novelty and details. \n\nThis paper proposed a novel text explanation method which extracts keywords that are essential to infer an answer from question. The authors proposed VCBS based on CBS and use Natural language inference to identify the entailments when the answer is yes or no. Experiment results show better mean opinion score 100 random samples results compared with the previous method and better VQA binary classifications when flipping based on the NLI results. \n\nDifferent from [Park et. al. 2018], who collected the explanations, this paper directly use the coco captions dataset as the source for the explanations. One of my major concern about this paper is it actually generate relevant captions with respect to the question and answer instead of real explanations. It's true that correlated caption can sometimes serve as the explanation when they cover the same concept coincidently but, we can not use these captions to explain the reasoning process of VQA. \n\nThe technique novelty of the proposed paper is also limited, the major novelty is the VCBS, which seems very similar to CBS. The only difference is VCBS adds relaxed parameters, which seems no technique novelty. Most annotations in Algorithms 1 is also not explained, making the readers hard to follow the actual content. The proposed model, although tied the vqa words with the explanation words, it suffers the same problems as PJ-X model, which didn't consider the VQA attention at all. \n\nThe experiment is also weak, considering the results is conduct on 100 samples, there might be significant variance. It's interesting the compared approach is learned based on a different dataset, which makes the results harder to compare. The NLI model results are interesting but for a more fair comparison, I would expect the proposed method compare with a model trained with VQA and coco caption dataset, such as VQA-E. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gXpfnhFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a novel method for explaining VQA systems. Different from most previous work, the proposed approach generates textual explanations based on three steps. First, it extracts keywords from the question. Then an explanation sentence is decoded (based on an RNN image captioner) through the proposed Variable-Constrained Beam Search (VCBS) algorithm to satisfy the keyword constraints. Finally, 3) checking through linguistic inference whether the explanation sentence can be used as a premise to infer the question and answer.\n\nI would recommend for acceptance. The paper proposes an alternative approach to VQA explanations, together with a few supporting algorithms such as VCBS. It is potentially helpful to future work on textual explanations and explainable AI in general.\n\nAt a high level, it is ambiguous to decide what is a reasonable explanation for many “no” answers. For example, one usually cannot provide stronger justification than “there is indeed no one” or “I don’t see anyone” to the question “Is there anyone in the room” with an answer “no.” The paper frames this explanation generation task as a linguistic inference task and checks entailment between the explanation and the question-answer pair. While it is debatable whether this is optimal, the proposed approach provides valuable insights on what constitutes a good explanation.\n\nHowever, the proposed approach also has noticeable weaknesses. \n\nIt relies on external models or tools for natural language inference, and such inference does not take into account the visual context of the image. Also, the explanations generated from the proposed model only justify the answer but are not introspective, and they do not reflect the decision process of the target VQA model.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1x8vBV6tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n I thank the authors for their response.  I would keep my score unchanged (i.e., 6 Weak Accept). \n\n-----------------------------------------------\n\nStrengths: \n- The paper enhances the beam search approach to generate explanations for answers to visual questions. The explanations are further used for verifying the yes/no answers.\n- The paper constructs a VCBS algorithm with novelties in allowing soft constrains of the generated beams. Since I have not worked on the constrained beam search before, thus it is hard for me to measure the novelty of this method.\n- The results of VQA 2.0 is pretty good. The accuracy of the Yes/No questions almost achieves the SotA systems. \n\n\nWeakness:\n- I have a question about the NLI system. Since the NLI is a three-way classifier where the answers would be \"Entail\", \"Contradictory\", and \"Neutral\".  What would the system do when the relationship is \"Neutral\"? For now, I think that it just give an answer of \"no\" but I am not sure whether it is correct.  For example, in Fig. 3, it shows an explanation (premise) of \"something (not the vegetable) on a plate\" and the hypo is \"there are vegetables on the plate\". Since the hypo is not necessary to contradict the premise, the relationship should be neutral. It does not directly provide evidence of the answer.\n\nComments:\n- Since more data are involved in training the explanation system, the proposed methods are not fairly compared with the BAN method. It would be better to mention this detail in the paper. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is good, with relatively positive support from the reviewers. However, there were also several legitimate issues raised, for example regarding the semantics of a negative answer and associated explanations. Though this paper cannot be accepted at this time, we hope the feedback here can help improve a future version, as all reviewers agree this is a valuable line of work.
Decision: Reject