VQABQ: Visual Question Answering by Basic Questions

Abstract: Taking an image and question as the input of our method, it can output the text-based answer of the query question about the given image, so called Visual Question Answering (VQA). There are two main modules in our algorithm. Given a natural language question about an image, the first module takes the question as input and then outputs the basic questions of the main given question. The second module takes the main question, image and these basic questions as input and then outputs the text-based answer of the main question. We formulate the basic questions generation problem as a LASSO optimization problem, and also propose a criterion about how to exploit these basic questions to help answer main question. Our method is evaluated on the challenging VQA dataset and yields state-of-the-art accuracy, 60.34% in open-ended task.

Visual Question Answering (VQA) is a challenging and young research field, which can help machines achieve one of the ultimate goals in computer vision, holistic scene understanding BIBREF1 . VQA is a computer vision task: a system is given an arbitrary text-based question about an image, and then it should output the text-based answer of the given question about the image. The given question may contain many sub-problems in computer vision, e.g.,
Besides, in our real life there are a lot of more complicated questions that can be queried. So, in some sense, VQA can be considered as an important basic research problem in computer vision. From the above sub-problems in computer vision, we can discover that if we want to do holistic scene understanding in one step, it is probably too difficult. So, we try to divide the holistic scene understanding-task into many sub-tasks in computer vision. The task-dividing concept inspires us to do Visual Question Answering by Basic Questions (VQABQ), illustrated by Figure 1 . That means, in VQA, we can divide the query question into some basic questions, and then exploit these basic questions to help us answer the main query question. Since 2014, there has been a lot of progress in designing systems with the VQA ability BIBREF2 , BIBREF0 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . Regarding these works, we can consider most of them as visual-attention VQA works because most of them do much effort on dealing with the image part but not the text part. However, recently there are some works BIBREF7 , BIBREF8 that try to do more effort on the question part. In BIBREF8 , authors proposed a Question Representation Update (QRU) mechanism to update the original query question to increase the accuracy of the VQA algorithm. Typically, VQA is a strongly image-question dependent issue, so we should pay equal attention to both the image and question, not only one of them. In reality, when people have an image and a given question about the image, we usually notice the keywords of the question and then try to focus on some parts of the image related to question to give the answer. So, paying equal attention to both parts is a more reasonable way to do VQA. In BIBREF7 , the authors proposed a Co-Attention mechanism, jointly utilizing information about visual and question attention, for VQA and achieved the state-of-the-art accuracy.
The Co-Attention mechanism inspires us to build part of our VQABQ model, illustrated by Figure 2 . In the VQABQ model, there are two main modules, the basic question generation module (Module 1) and co-attention visual question answering module (Module 2). We take the query question, called the main question (MQ), encoded by Skip-Thought Vectors BIBREF9 , as the input of Module 1. In the Module 1, we encode all of the questions, also by Skip-Thought Vectors, from the training and validation sets of VQA BIBREF0 dataset as a 4800 by 215623 dimension basic question (BQ) matrix, and then solve the LASSO optimization problem, with MQ, to find the 3 BQ of MQ. These BQ are the output of Module 1. Moreover, we take the MQ, BQ and the given image as the input of Module 2, the VQA module with co-attention mechanism, and then it can output the final answer of MQ. We claim that the BQ can help Module 2 get the correct answer to increase the VQA accuracy. In this work, our main contributions are summarized below:
The rest of this paper is organized as the following. We first talk about the motivation about this work in Section 2. In Section 3, we review the related work, and then Section 4 shortly introduces the proposed VQABQ dataset. We discuss the detailed methodology in Section 5. Finally, the experimental results are demonstrated in Section 6.
The following two important reasons motivate us to do Visual Question Answering by Basic Questions (VQABQ). First, recently most of VQA works only emphasize more on the image part, the visual features, but put less effort on the question part, the text features. However, image and question features both are important for VQA. If we only focus on one of them, we probably cannot get the good performance of VQA in the near future. Therefore, we should put our effort more on both of them at the same time. In BIBREF7 , they proposed a novel co-attention mechanism that jointly performs image-guided question attention and question-guided image attention for VQA. BIBREF7 also proposed a hierarchical architecture to represent the question, and construct image-question co-attention maps at the word level, phrase level and question level. Then, these co-attended features are combined with word level, phrase level and question level recursively for predicting the final answer of the query question based on the input image. BIBREF8 is also a recent work focusing on the text-based question part, text feature. In BIBREF8 , they presented a reasoning network to update the question representation iteratively after the question interacts with image content each time. Both of BIBREF7 , BIBREF8 yield better performance than previous works by doing more effort on the question part.
Secondly, in our life , when people try to solve a difficult problem, they usually try to divide this problem into some small basic problems which are usually easier than the original problem. So, why don't we apply this dividing concept to the input question of VQA ? If we can divide the input main question into some basic questions, then it will help the current VQA algorithm achieve higher probability to get the correct answer of the main question.
Thus, our goal in this paper is trying to generate the basic questions of the input question and then exploit these questions with the given image to help the VQA algorithm get the correct answer of the input question. Note that we can consider the generated basic questions as the extra useful information to VQA algorithm.
Recently, there are many papers BIBREF0 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 have proposed methods to solve the VQA issue. Our method involves in different areas in machine learning, natural language processing (NLP) and computer vision. The following, we discuss recent works related to our approach for solving VQA problem.
Sequence modeling by Recurrent Neural Networks.
Recurrent Neural Networks (RNN) can handle the sequences of flexible length. Long Short Term Memory (LSTM) BIBREF17 is a particular variant of RNN and in natural language tasks, such as machine translation BIBREF18 , BIBREF19 , LSTM is a successful application. In BIBREF14 , the authors exploit RNN and Convolutional Neural Network (CNN) to build a question generation algorithm, but the generated question sometimes has invalid grammar. The input in BIBREF3 is the concatenation of each word embedding with the same feature vector of image. BIBREF6 encodes the input question sentence by LSTM and join the image feature to the final output. BIBREF13 groups the neighbouring word and image features by doing convolution. In BIBREF20 , the question is encoded by Gated Recurrent Unit (GRU) BIBREF21 similar to LSTM and the authors also introduce a dynamic parameter layer in CNN whose weights are adaptively predicted by the encoded question feature.
Sentence encoding.
In order to analyze the relationship among words, phrases and sentences, several works, such as BIBREF22 , BIBREF9 , BIBREF23 , proposed methods about how to map text into vector space. After we have the vector representation of text, we can exploit the vector analysis skill to analyze the relationship among text. BIBREF22 , BIBREF23 try to map words to vector space, and if the words share common contexts in the corpus, their encoded vectors will close to each other in the vector space. In BIBREF9 , the authors propose a framework of encoder-decoder models, called skip-thoughts. In this model, the authors exploit an RNN encoder with GRU activations BIBREF21 and an RNN decoder with a conditional GRU BIBREF21 . Because skip-thoughts model emphasizes more on whole sentence encoding, in our work, we encode the whole question sentences into vector space by skip-thoughts model and use these skip-thought vectors to do further analysis of question sentences.
Image captioning.
In some sense, VQA is related to image captioning BIBREF24 , BIBREF25 , BIBREF26 , BIBREF27 . BIBREF27 uses a language model to combine a set of possible words detected in several regions of the image and generate image description. In BIBREF26 , the authors use CNN to extract the high-level image features and considered them as the first input of the recurrent network to generate the caption of image. BIBREF24 proposes an algorithm to generate one word at a time by paying attention to local image regions related to the currently predicted word. In BIBREF25 , the deep neural network can learn to embed language and visual information into a common multi-modal space. However, the current image captioning algorithms only can generate the rough description of image and there is no so called proper metric to evaluate the quality of image caption , even though BLEU BIBREF28 can be used to evaluate the image caption.
Attention-based VQA.
There are several VQA models have ability to focus on specific image regions related to the input question by integrating the image attention mechanism BIBREF10 , BIBREF11 , BIBREF29 , BIBREF8 . In BIBREF8 , in the pooling step, the authors exploit an image attention mechanism to help determine the relevance between original questions and updated ones. Before BIBREF7 , no work applied language attention mechanism to VQA, but the researchers in NLP they had modeled language attention. In BIBREF7 , the authors propose a co-attention mechanism that jointly performs language attention and image attention. Because both question and image information are important in VQA, in our work we introduce co-attention mechanism into our VQABQ model.
We propose a new dataset, called Basic Question Dataset (BQD), generated by our basic question generation algorithm. BQD is the first basic question dataset. Regarding the BQD, the dataset format is $\lbrace Image,~MQ,~3~(BQ + corresponding~similarity~score)\rbrace $ . All of our images are from the testing images of MS COCO dataset BIBREF30 , the MQ, main questions, are from the testing questions of VQA, open-ended, dataset BIBREF0 , the BQ, basic questions, are from the training and validation questions of VQA, open-ended, dataset BIBREF0 , and the corresponding similarity score of BQ is generated by our basic question generation method, referring to Section 5. Moreover, we also take the multiple-choice questions in VQA dataset BIBREF0 to do the same thing as above. Note that we remove the repeated questions in the VQA dataset, so the total number of questions is slightly less than VQA dataset BIBREF0 . In BQD, we have 81434 images, 244302 MQ and 732906 (BQ + corresponding similarity score). At the same time, we also exploit BQD to do VQA and achieve the competitive accuracy compared to state-of-the-art.
In Section 5, we mainly discuss how to encode questions and generate BQ and why we exploit the Co-Attention Mechanism VQA algorithm BIBREF7 to answer the query question. The overall architecture of our VQABQ model can be referred to Figure 2 . The model has two main parts, Module 1 and Module 2. Regarding Module 1, it takes the encoded MQ as input and uses the matrix of the encoded BQ to output the BQ of query question. Then, the Module 2 is a VQA algorithm with the Co-Attention Mechanism BIBREF7 , and it takes the output of Module 1, MQ, and the given image as input and then outputs the final answer of MQ. The detailed architecture of Module 1 can be referred to Figure 2 .
There are many popular text encoders, such as Word2Vec BIBREF23 , GloVe BIBREF22 and Skip-Thoughts BIBREF9 . In these encoders, Skip-Thoughts not only can focus on the word-to-word meaning but also the whole sentence semantic meaning. So, we choose Skip-Thoughts to be our question encoding method. In Skip-Thoughts model, it uses an RNN encoder with GRU BIBREF21 activations, and then we use this encoder to map an English sentence into a vector. Regarding GRU, it has been shown to perform as well as LSTM BIBREF17 on the sequence modeling applications but being conceptually simpler because GRU units only have 2 gates and do not need the use of a cell.
Question encoder. Let $w_{i}^{1},...,w_{i}^{N}$ be the words in question $s_{i}$ and N is the total number of words in $s_{i}$ . Note that $w_{i}^{t}$ denotes the $t$ -th word for $s_{i}$ and $\mathbf {x}_{i}^t$ denotes its word embedding. The question encoder at each time step generates a hidden state $\mathbf {h}_{i}^{t}$ . It can be considered as the representation of the sequence $w_{i}^{1},..., w_{i}^{t}$ . So, the hidden state $\mathbf {h}_{i}^{N}$ can represent the whole question. For convenience, here we drop the index $s_{i}$0 and iterate the following sequential equations to encode a question: 
$$\mathbf {r}^{t}~=~\sigma (\mathbf {U}_{r}\mathbf {h}^{t-1}+\mathbf {W}_{r}\mathbf {x}^{t})$$   (Eq. 12) 
$$\mathbf {z}^{t}~=~\sigma (\mathbf {U}_{z}\mathbf {h}^{t-1}+\mathbf {W}_{z}\mathbf {x}^{t})$$   (Eq. 13) 
, where $\mathbf {U}_{r}$ , $\mathbf {U}_{z}$ , $\mathbf {W}_{r}$ , $\mathbf {W}_{z}$ , $\mathbf {U}$ and $\mathbf {W}$ are the matrices of weight parameters. $\bar{\mathbf {h}}^{t}$ is the state update at time step $t$ , $\mathbf {r}^{t}$ is the reset gate, $\odot $ denotes an element-wise product and $\mathbf {U}_{z}$0 is the update gate. These two update gates take the values between zero and one.
Our idea is the BQ generation for MQ and, at the same time, we only want the minimum number of BQ to represent the MQ, so modeling our problem as $LASSO$ optimization problem is an appropriate way: 
$$\min _{\mathbf {x}}~\frac{1}{2}\left\Vert  A\mathbf {x}-\mathbf {b} \right\Vert _{2}^{2}+\lambda \left\Vert  \mathbf {x} \right\Vert _{1}$$   (Eq. 17) 
, where $A$ is the matrix of encoded BQ, $\mathbf {b}$ is the encode MQ and $\lambda $ is a parameter of the regularization term.
We now describe how to generate the BQ of a query question, illustrated by Figure 2 . Note that the following we only describe the open-ended question case because the multiple-choice case is same as open-ended one. According to Section 5.2, we can encode the all questions from the training and validation questions of VQA dataset BIBREF0 by Skip-Thought Vectors, and then we have the matrix of these encoded basic questions. Each column of the matrix is the vector representation, 4800 by 1 dimensions, of a basic question and we have 215623 columns. That is, the dimension of BQ matrix, called $A$ , is 4800 by 215623. Also, we encode the query question as a column vector, 4800 by 1 dimensions, by Skip-Thought Vectors, called $\mathbf {b}$ . Now, we can solve the $LASSO$ optimization problem, mentioned in Section 5.3, to get the solution, $\mathbf {x}$ . Here, we consider the elements, in solution vector $\mathbf {x}$ , as the weights of the corresponding BQ in BQ matrix, $A$ . The first element of $\mathbf {x}$ corresponds to the first column, i.e. the first BQ, of $A$ . Then, we rank the all weights in $\mathbf {x}$ and pick up the top 3 large weights with corresponding BQ to be the BQ of the query question. Intuitively, because BQ are important to MQ, the weights of BQ also can be considered as importance scores and the BQ with larger weight means more important to MQ. Finally, we find the BQ of all 142093 testing questions from VQA dataset and collect them together, with the format $\lbrace Image,~MQ,~3~(BQ + corresponding~ similarity~score)\rbrace $ , as the BQD in Section 4.
In this section, we propose a criterion to use these BQ. In BQD, each MQ has three corresponding BQ with scores. We can have the following format, $\lbrace MQ,(BQ1,~score1),(BQ2,~score2),(BQ3,~score3)\rbrace $ , and these scores are all between 0 and 1 with the following order, 
$$score1\ge score2\ge score3$$   (Eq. 20) 
and we define 3 thresholds, $s1$ , $s2$ and $s3$ . Also, we compute the following 3 averages ( $avg$ ) and 3 standard deviations ( $std$ ) to $score1$ , $score2/score1$ and $score3/score2$ , respectively, and then use $avg \pm std$ , referring to Table 3 , to be the initial guess of proper thresholds. The BQ utilization process can be explained as Table 1 . The detailed discussion about BQ concatenation algorithm is described in the Section 6.4.
There are two types of Co-Attention Mechanism BIBREF7 , Parallel and Alternating. In our VQABQ model, we only use the VQA algorithm with Alternating Co-Attention Mechanism to be our VQA module, referring to Figure 2 , because, in BIBREF7 , Alternating Co-Attention Mechanism VQA module can get the higher accuracy than the Parallel one. Moreover, we want to compare with the VQA method, Alternating one, with higher accuracy in BIBREF7 . In Alternating Co-Attention Mechanism, it sequentially alternates between generating question and image attention. That is, this mechanism consists of three main steps:
First, the input question is summarized into a single vector $\mathbf {q}$ .
Second, attend to the given image depended on $\mathbf {q}$ .
Third, attend to the question depended on the attended image feature.
We can define $\hat{\mathbf {x}}$ is an attention operator, which is a function of $\mathbf {X}$ and $\mathbf {g}$ . This operator takes the question (or image) feature $\mathbf {X}$ and attention guider $\mathbf {g}$ derived from image (or question) as inputs, and then outputs the attended question (or image) vector. We can explain the above operation as the following steps: 
$$\mathbf {H}~=~\rm {tanh}(\mathbf {W}_{x}\mathbf {X}+(\mathbf {W}_{g}g)\mathbf {1}^{T})$$   (Eq. 26) 
$$\mathbf {a}^{x}~=~\rm {softmax}(\mathbf {w}_{hx}^{T}\mathbf {H})$$   (Eq. 27) 
, where $\mathbf {a}^{x}$ is the attention weight of feature $\mathbf {X}$ , $\mathbf {1}$ is a vector whose elements are all equal to 1, and $\mathbf {W}_{g}$ , $\mathbf {W}_{x}$ and $\mathbf {w}_{hx}$ are matrices of parameters.
Concretely, at the first step of Alternating Co-Attention Mechanism, $\mathbf {g}$ is 0 and $\mathbf {X} = \mathbf {Q}$ . Then, at the second step, $\mathbf {X} = \mathbf {V}$ where $\mathbf {V}$ is the image features and the guider, $\mathbf {g}$ , is intermediate attended question feature, $\hat{s}$ , which is from the first step. At the final step, it uses the attended image feature, $\hat{v}$ , as the guider to attend the question again. That is, $\mathbf {X} = \mathbf {Q}$ and $\mathbf {g} = \hat{v}$ .
In Section 6, we describe the details of our implementation and discuss the experiment results about the proposed method.
We conduct our experiments on VQA BIBREF0 dataset. VQA dataset is based on the MS COCO dataset BIBREF30 and it contains the largest number of questions. There are questions, 248349 for training, 121512 for validation and 244302 for testing. In the VQA dataset, each question is associated with 10 answers annotated by different people from Amazon Mechanical Turk (AMT). About 98% of answers do not exceed 3 words and 90% of answers have single words. Note that we only test our method on the open-ended case in VQA dataset because it has the most open-ended questions among the all available dataset and we also think open-ended task is closer to the real situation than multiple-choice one.
In order to prove our claim that BQ can help accuracy and compare with the state-of-the-art VQA method BIBREF7 , so, in our Module 2, we use the same setting, dataset and source code mentioned in BIBREF7 . Then, the Module 1 in VQABQ model, is our basic question generation module. In other words, in our model ,the only difference compared to BIBREF7 is our Module 1, illustrated by Figure 2 .
VQA dataset provides multiple-choice and open-ended task for evaluation. Regarding open-ended task, the answer can be any phrase or word. However, in multiple-choice task, an answer should be chosen from 18 candidate answers. For both cases, answers are evaluated by accuracy which can reflect human consensus. The accuracy is given by the following: 
$$Accuracy_{_{VQA}}=\frac{1}{N}\sum _{i=1}^{N}\min \left\lbrace  \frac{\sum _{t\in T_{i}}\mathbb {I}[a_{i}=t]}{3},1 \right\rbrace $$   (Eq. 36) 
, where $N$ is the total number of examples, $\mathbb {I}[\cdot ]$ denotes an indicator function, $a_{i}$ is the predicted answer and $T_{i}$ is an answer set of the $i^{th}$ example. That is, a predicted answer is considered as a correct one if at least 3 annotators agree with it, and the score depends on the total number of agreements when the predicted answer is not correct.
Here, we describe our final results and analysis by the following parts:
Does Basic Question Help Accuracy ?
The answer is yes. Here we only discuss the open-ended case. In our experiment, we use the $avg\pm std$ , referring to Table 3 , to be the initial guess of proper thresholds of s1, s2 and s3, in Table 1 . We discover that when s1 = 0.43, s2 = 0.82 and s3 = 0.53, we can get the better utilization of BQ. The threshold, s1 = 0.43, can be consider as 43% of testing questions from VQA dataset which cannot find the basic question, from the training and validation sets of VQA dataset, and only 57% of testing questions can find the basic questions. Note that we combine the training and validation sets of VQA dataset to be our basic question dataset. Regarding s2 = 0.82, that means 82% of those 57% testing questions, i.e. 46.74%, only can find 1 basic question, and 18% of those 57% testing questions, i.e. 10.26%, can find at least 2 basic questions. Furthermore, s3 = 0.53 means that 53% of those 10.26% testing question, i.e. around 5.44%, only can find 2 basic questions, and 47% of those 10.26% testing question, i.e. around 4.82%, can find 3 basic questions. The above detail can be referred to Table 2 .
Accordingly to the Table 2 , 43% of testing questions from VQA dataset cannot find the proper basic questions from VQA training and validation datasets, and there are some failed examples about this case in Table 6 . We also discover that a lot of questions in VQA training and validation datasets are almost the same. This issue reduces the diversity of basic question dataset. Although we only have 57% of testing questions can benefit from the basic questions, our method still can improve the state-of-the-art accuracy BIBREF7 from 60.32% to 60.34%, referring to Table 4 and 5 . Then, we have 142093 testing questions, so that means the number of correctly answering questions of our method is more than state-of-the-art method 28 questions. In other words, if we have well enough basic question dataset, we can increase accuracy more, especially in the counting-type question, referring to Table 4 and 5 . Because the Co-Attention Mechanism is good at localizing, the counting-type question is improved more than others. So, based on our experiment, we can conclude that basic question can help accuracy obviously.
Comparison with State-of-the-art.
Recently, BIBREF7 proposed the Co-Attention Mechanism in VQA and got the state-of-the-art accuracy. However, when we use their code and the same setup mentioned in their paper to re-run the experiment, we cannot get the same accuracy reported in their work. The re-run results are presented in Table 5 . So, under the fair conditions, our method is competitive compared to the state-of-the-art.
In this paper, we propose a VQABQ model for visual question answering. The VQABQ model has two main modules, Basic Question Generation Module and Co-Attention VQA Module. The former one can generate the basic questions for the query question, and the latter one can take the image , basic and query question as input and then output the text-based answer of the query question. According to the Section 6.4, because the basic question dataset generated from VQA dataset is not well enough, we only have the 57% of all testing questions can benefit from the basic questions. However, we still can increase 28 correctly answering questions compared to the state-of-the-art. We believe that if our basic question dataset is well enough, the increment of accuracy will be much more.
According to the previous state-of-the-art methods in VQA, they all got the highest accuracy in the Yes/No-type question. So, how to effectively only exploit the Yes/No-type basic questions to do VQA will be an interesting work, illustrated by Figure 3 . Also, how to generate other specific type of basic questions based on the query question and how to do better combination of visual and textual features in order to decrease the semantic inconsistency? The above future works will be our next research focus.
This work is supported by competitive research funding from King Abdullah University of Science and Technology (KAUST). Also, we would like to acknowledge Fabian Caba, Humam Alwassel and Adel Bibi. They always can provide us helpful discussion about this work.

Figure and table captions:
Figure 1. Examples of basic questions. Note that MQ denotes the main question and BQ denotes the basic question.
Figure 2. VQABQ working pipeline. Note that all of the training and validation questions are only encoded by Skip-Thoughts one time for generating the basic question matrix. That is, the next input of Skip-Thoughts is only the new main question. Here, ”⊕” denotes the proposed basic question concatenation method.
Table 2. We only show the open-ended case of VQA dataset [1], and ”# Q” denoted number of questions.
Table 3. ”avg” denotes average and ”std” denotes standard deviation.
Table 1. Note that appending BQ means doing the concatenation with MQ.
Table 4. Evaluation results on VQA dataset [1]. ”-” indicates the results are not available, and the Ours+VGG(1) and Ours+VGG(2) are the results by using different thresholds. Note that our VGGNet is same as CoAtt+VGG.
Table 5. Re-run evaluation results on VQA dataset [1]. ”-” indicates the results are not available. Note that the result of [14] in Table 5 is lower than in Table 4, and CoAtt+VGG is same as our VGGNet. According to the re-run results, our method has the higher accuracy, especially in the counting-type question.
Table 6. Some failed examples about finding no basic question.
Figure 3. Some future work examples.

Questions about this paper:
In which setting they achieve the state of the art?

Answers:
in open-ended task esp. for counting-type questions