Introduction
In open-ended visual question answering (VQA) an algorithm must produce answers to arbitrary text-based questions about images BIBREF0 , BIBREF1 . VQA is an exciting computer vision problem that requires a system to be capable of many tasks. Truly solving VQA would be a milestone in artificial intelligence, and would significantly advance human computer interaction. However, VQA datasets must test a wide range of abilities for progress to be adequately measured.
VQA research began in earnest in late 2014 when the DAQUAR dataset was released BIBREF0 . Including DAQUAR, six major VQA datasets have been released, and algorithms have rapidly improved. On the most popular dataset, `The VQA Dataset' BIBREF1 , the best algorithms are now approaching 70% accuracy BIBREF2 (human performance is 83%). While these results are promising, there are critical problems with existing datasets in terms of multiple kinds of biases. Moreover, because existing datasets do not group instances into meaningful categories, it is not easy to compare the abilities of individual algorithms. For example, one method may excel at color questions compared to answering questions requiring spatial reasoning. Because color questions are far more common in the dataset, an algorithm that performs well at spatial reasoning will not be appropriately rewarded for that feat due to the evaluation metrics that are used.
Contributions: Our paper has four major contributions aimed at better analyzing and comparing VQA algorithms: 1) We create a new VQA benchmark dataset where questions are divided into 12 different categories based on the task they solve; 2) We propose two new evaluation metrics that compensate for forms of dataset bias; 3) We balance the number of yes/no object presence detection questions to assess whether a balanced distribution can help algorithms learn better; and 4) We introduce absurd questions that force an algorithm to determine if a question is valid for a given image. We then use the new dataset to re-train and evaluate both baseline and state-of-the-art VQA algorithms. We found that our proposed approach enables more nuanced comparisons of VQA algorithms, and helps us understand the benefits of specific techniques better. In addition, it also allowed us to answer several key questions about VQA algorithms, such as, `Is the generalization capacity of the algorithms hindered by the bias in the dataset?', `Does the use of spatial attention help answer specific question-types?', `How successful are the VQA algorithms in answering less-common questions?', and 'Can the VQA algorithms differentiate between real and absurd questions?'
Prior Natural Image VQA Datasets
Six datasets for VQA with natural images have been released between 2014–2016: DAQUAR BIBREF0 , COCO-QA BIBREF3 , FM-IQA BIBREF4 , The VQA Dataset BIBREF1 , Visual7W BIBREF5 , and Visual Genome BIBREF6 . FM-IQA needs human judges and has not been widely used, so we do not discuss it further. Table 1 shows statistics for the other datasets. Following others BIBREF7 , BIBREF8 , BIBREF9 , we refer to the portion of The VQA Dataset containing natural images as COCO-VQA. Detailed dataset reviews can be found in BIBREF10 and BIBREF11 .
All of the aforementioned VQA datasets are biased. DAQUAR and COCO-QA are small and have a limited variety of question-types. Visual Genome, Visual7W, and COCO-VQA are larger, but they suffer from several biases. Bias takes the form of both the kinds of questions asked and the answers that people give for them. For COCO-VQA, a system trained using only question features achieves 50% accuracy BIBREF7 . This suggests that some questions have predictable answers. Without a more nuanced analysis, it is challenging to determine what kinds of questions are more dependent on the image. For datasets made using Mechanical Turk, annotators often ask object recognition questions, e.g., `What is in the image?' or `Is there an elephant in the image?'. Note that in the latter example, annotators rarely ask that kind of question unless the object is in the image. On COCO-VQA, 79% of questions beginning with `Is there a' will have `yes' as their ground truth answer.
In 2017, the VQA 2.0 BIBREF12 dataset was introduced. In VQA 2.0, the same question is asked for two different images and annotators are instructed to give opposite answers, which helped reduce language bias. However, in addition to language bias, these datasets are also biased in their distribution of different types of questions and the distribution of answers within each question-type. Existing VQA datasets use performance metrics that treat each test instance with equal value (e.g., simple accuracy). While some do compute additional statistics for basic question-types, overall performance is not computed from these sub-scores BIBREF1 , BIBREF3 . This exacerbates the issues with the bias because the question-types that are more likely to be biased are also more common. Questions beginning with `Why' and `Where' are rarely asked by annotators compared to those beginning with `Is' and 'Are'. For example, on COCO-VQA, improving accuracy on `Is/Are' questions by 15% will increase overall accuracy by over 5%, but answering all `Why/Where' questions correctly will increase accuracy by only 4.1% BIBREF10 . Due to the inability of the existing evaluation metrics to properly address these biases, algorithms trained on these datasets learn to exploit these biases, resulting in systems that work poorly when deployed in the real-world.
For related reasons, major benchmarks released in the last decade do not use simple accuracy for evaluating image recognition and related computer vision tasks, but instead use metrics such as mean-per-class accuracy that compensates for unbalanced categories. For example, on Caltech-101 BIBREF13 , even with balanced training data, simple accuracy fails to address the fact that some categories were much easier to classify than others (e.g., faces and planes were easy and also had the largest number of test images). Mean per-class accuracy compensates for this by requiring a system to do well on each category, even when the amount of test instances in categories vary considerably.
Existing benchmarks do not require reporting accuracies across different question-types. Even when they are reported, the question-types can be too coarse to be useful, e.g., `yes/no', `number' and `other' in COCO-VQA. To improve the analysis of the VQA algorithms, we categorize the questions into meaningful types, calculate the sub-scores, and incorporate them in our evaluation metrics.
Synthetic Datasets that Fight Bias
Previous works have studied bias in VQA and proposed countermeasures. In BIBREF14 , the Yin and Yang dataset was created to study the effect of having an equal number of binary (yes/no) questions about cartoon images. They found that answering questions from a balanced dataset was harder. This work is significant, but it was limited to yes/no questions and their approach using cartoon imagery cannot be directly extended to real-world images.
One of the goals of this paper is to determine what kinds of questions an algorithm can answer easily. In BIBREF15 , the SHAPES dataset was proposed, which has similar objectives. SHAPES is a small dataset, consisting of 64 images that are composed by arranging colored geometric shapes in different spatial orientations. Each image has the same 244 yes/no questions, resulting in 15,616 questions. Although SHAPES serves as an important adjunct evaluation, it alone cannot suffice for testing a VQA algorithm. The major limitation of SHAPES is that all of its images are of 2D shapes, which are not representative of real-world imagery. Along similar lines, Compositional Language and Elementary Visual Reasoning (CLEVR) BIBREF16 also proposes use of 3D rendered geometric objects to study reasoning capacities of a model. CLEVR is larger than SHAPES and makes use of 3D rendered geometric objects. In addition to shape and color, it adds material property to the objects. CLEVR has five types of questions: attribute query, attribute comparison, integer comparison, counting, and existence.
Both SHAPES and CLEVR were specifically tailored for compositional language approaches BIBREF15 and downplay the importance of visual reasoning. For instance, the CLEVR question, `What size is the cylinder that is left of the brown metal thing that is left of the big sphere?' requires demanding language reasoning capabilities, but only limited visual understanding is needed to parse simple geometric objects. Unlike these three synthetic datasets, our dataset contains natural images and questions. To improve algorithm analysis and comparison, our dataset has more (12) explicitly defined question-types and new evaluation metrics.
TDIUC for Nuanced VQA Analysis
In the past two years, multiple publicly released datasets have spurred the VQA research. However, due to the biases and issues with evaluation metrics, interpreting and comparing the performance of VQA systems can be opaque. We propose a new benchmark dataset that explicitly assigns questions into 12 distinct categories. This enables measuring performance within each category and understand which kind of questions are easy or hard for today's best systems. Additionally, we use evaluation metrics that further compensate for the biases. We call the dataset the Task Driven Image Understanding Challenge (TDIUC). The overall statistics and example images of this dataset are shown in Table 1 and Fig. 2 respectively.
TDIUC has 12 question-types that were chosen to represent both classical computer vision tasks and novel high-level vision tasks which require varying degrees of image understanding and reasoning. The question-types are:
The number of each question-type in TDIUC is given in Table 2 . The questions come from three sources. First, we imported a subset of questions from COCO-VQA and Visual Genome. Second, we created algorithms that generated questions from COCO's semantic segmentation annotations BIBREF17 , and Visual Genome's objects and attributes annotations BIBREF6 . Third, we used human annotators for certain question-types. In the following sections, we briefly describe each of these methods.
Importing Questions from Existing Datasets
We imported questions from COCO-VQA and Visual Genome belonging to all question-types except `object utilities and affordances'. We did this by using a large number of templates and regular expressions. For Visual Genome, we imported questions that had one word answers. For COCO-VQA, we imported questions with one or two word answers and in which five or more annotators agreed.
For color questions, a question would be imported if it contained the word `color' in it and the answer was a commonly used color. Questions were classified as activity or sports recognition questions if the answer was one of nine common sports or one of fifteen common activities and the question contained common verbs describing actions or sports, e.g., playing, throwing, etc. For counting, the question had to begin with `How many' and the answer had to be a small countable integer (1-16). The other categories were determined using regular expressions. For example, a question of the form `Are feeling ?' was classified as sentiment understanding and `What is to the right of/left of/ behind the ?' was classified as positional reasoning. Similarly, `What <OBJECT CATEGORY> is in the image?' and similar templates were used to populate subordinate object recognition questions. This method was used for questions about the season and weather as well, e.g., `What season is this?', `Is this rainy/sunny/cloudy?', or `What is the weather like?' were imported to scene classification.
Generating Questions using Image Annotations
Images in the COCO dataset and Visual Genome both have individual regions with semantic knowledge attached to them. We exploit this information to generate new questions using question templates. To introduce variety, we define multiple templates for each question-type and use the annotations to populate them. For example, for counting we use 8 templates, e.g., `How many <objects> are there?', `How many <objects> are in the photo?', etc. Since the COCO and Visual Genome use different annotation formats, we discuss them separately.
Sport recognition, counting, subordinate object recognition, object presence, scene understanding, positional reasoning, and absurd questions were created from COCO, similar to the scheme used in BIBREF18 . For counting, we count the number of object instances in an image annotation. To minimize ambiguity, this was only done if objects covered an area of at least 2,000 pixels.
For subordinate object recognition, we create questions that require identifying an object's subordinate-level object classification based on its larger semantic category. To do this, we use COCO supercategories, which are semantic concepts encompassing several objects under a common theme, e.g., the supercategory `furniture' contains chair, couch, etc. If the image contains only one type of furniture, then a question similar to `What kind of furniture is in the picture?' is generated because the answer is not ambiguous. Using similar heuristics, we create questions about identifying food, electronic appliances, kitchen appliances, animals, and vehicles.
For object presence questions, we find images with objects that have an area larger than 2,000 pixels and produce a question similar to `Is there a <object> in the picture?' These questions will have `yes' as an answer. To create negative questions, we ask questions about COCO objects that are not present in an image. To make this harder, we prioritize the creation of questions referring to absent objects that belong to the same supercategory of objects that are present in the image. A street scene is more likely to contain trucks and cars than it is to contain couches and televisions. Therefore, it is more difficult to answer `Is there a truck?' in a street scene than it is to answer `Is there a couch?'
For sport recognition questions, we detect the presence of specific sports equipment in the annotations and ask questions about the type of sport being played. Images must only contain sports equipment for one particular sport. A similar approach was used to create scene understanding questions. For example, if a toilet and a sink are present in annotations, the room is a bathroom and an appropriate scene recognition question can be created. Additionally, we use the supercategories `indoor' and `outdoor' to ask questions about where a photo was taken.
For creating positional reasoning questions, we use the relative locations of bounding boxes to create questions similar to `What is to the left/right of <object>?' This can be ambiguous due to overlapping objects, so we employ the following heuristics to eliminate ambiguity: 1) The vertical separation between the two bounding boxes should be within a small threshold; 2) The objects should not overlap by more than the half the length of its counterpart; and 3) The objects should not be horizontally separated by more than a distance threshold, determined by subjectively judging optimal separation to reduce ambiguity. We tried to generate above/below questions, but the results were unreliable.
Absurd questions test the ability of an algorithm to judge when a question is not answerable based on the image's content. To make these, we make a list of the objects that are absent from a given image, and then we find questions from rest of TDIUC that ask about these absent objects, with the exception of yes/no and counting questions. This includes questions imported from COCO-VQA, auto-generated questions, and manually created questions. We make a list of all possible questions that would be `absurd' for each image and we uniformly sample three questions per image. In effect, we will have same question repeated multiple times throughout the dataset, where it can either be a genuine question or a nonsensical question. The algorithm must answer `Does Not Apply' if the question is absurd.
Visual Genome's annotations contain region descriptions, relationship graphs, and object boundaries. However, the annotations can be both non-exhaustive and duplicated, which makes using them to automatically make QA pairs difficult. We only use Visual Genome to make color and positional reasoning questions. The methods we used are similar to those used with COCO, but additional precautions were needed due to quirks in their annotations. Additional details are provided in the Appendix.
Manual Annotation
Creating sentiment understanding and object utility/affordance questions cannot be readily done using templates, so we used manual annotation to create these. Twelve volunteer annotators were trained to generate these questions, and they used a web-based annotation tool that we developed. They were shown random images from COCO and Visual Genome and could also upload images.
Post Processing
Post processing was performed on questions from all sources. All numbers were converted to text, e.g., 2 became two. All answers were converted to lowercase, and trailing punctuation was stripped. Duplicate questions for the same image were removed. All questions had to have answers that appeared at least twice. The dataset was split into train and test splits with 70% for train and 30% for test.
Proposed Evaluation Metric
One of the main goals of VQA research is to build computer vision systems capable of many tasks, instead of only having expertise at one specific task (e.g., object recognition). For this reason, some have argued that VQA is a kind of Visual Turing Test BIBREF0 . However, if simple accuracy is used for evaluating performance, then it is hard to know if a system succeeds at this goal because some question-types have far more questions than others. In VQA, skewed distributions of question-types are to be expected. If each test question is treated equally, then it is difficult to assess performance on rarer question-types and to compensate for bias. We propose multiple measures to compensate for bias and skewed distributions.
To compensate for the skewed question-type distribution, we compute accuracy for each of the 12 question-types separately. However, it is also important to have a final unified accuracy metric. Our overall metrics are the arithmetic and harmonic means across all per question-type accuracies, referred to as arithmetic mean-per-type (Arithmetic MPT) accuracy and harmonic mean-per-type accuracy (Harmonic MPT). Unlike the Arithmetic MPT, Harmonic MPT measures the ability of a system to have high scores across all question-types and is skewed towards lowest performing categories.
We also use normalized metrics that compensate for bias in the form of imbalance in the distribution of answers within each question-type, e.g., the most repeated answer `two' covers over 35% of all the counting-type questions. To do this, we compute the accuracy for each unique answer separately within a question-type and then average them together for the question-type. To compute overall performance, we compute the arithmetic normalized mean per-type (N-MPT) and harmonic N-MPT scores. A large discrepancy between unnormalized and normalized scores suggests an algorithm is not generalizing to rarer answers.
Algorithms for VQA
While there are alternative formulations (e.g., BIBREF4 , BIBREF19 ), the majority of VQA systems formulate it as a classification problem in which the system is given an image and a question, with the answers as categories. BIBREF1 , BIBREF3 , BIBREF2 , BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 , BIBREF9 , BIBREF27 , BIBREF28 , BIBREF8 , BIBREF19 , BIBREF29 . Almost all systems use CNN features to represent the image and either a recurrent neural network (RNN) or a bag-of-words model for the question. We briefly review some of these systems, focusing on the models we compare in experiments. For a more comprehensive review, see BIBREF10 and BIBREF11 .
Two simple VQA baselines are linear or multi-layer perceptron (MLP) classifiers that take as input the question and image embeddings concatenated to each other BIBREF1 , BIBREF7 , BIBREF8 , where the image features come from the last hidden layer of a CNN. These simple approaches often work well and can be competitive with complex attentive models BIBREF7 , BIBREF8 .
Spatial attention has been heavily investigated in VQA models BIBREF2 , BIBREF20 , BIBREF28 , BIBREF30 , BIBREF27 , BIBREF24 , BIBREF21 . These systems weigh the visual features based on their relevance to the question, instead of using global features, e.g., from the last hidden layer of a CNN. For example, to answer `What color is the bear?' they aim emphasize the visual features around the bear and suppress other features.
The MCB system BIBREF2 won the CVPR-2016 VQA Workshop Challenge. In addition to using spatial attention, it implicitly computes the outer product between the image and question features to ensure that all of their elements interact. Explicitly computing the outer product would be slow and extremely high dimensional, so it is done using an efficient approximation. It uses an long short-term memory (LSTM) networks to embed the question.
The neural module network (NMN) is an especially interesting compositional approach to VQA BIBREF15 , BIBREF31 . The main idea is to compose a series of discrete modules (sub-networks) that can be executed collectively to answer a given question. To achieve this, they use a variety of modules, e.g., the find(x) module outputs a heat map for detecting $x$ . To arrange the modules, the question is first parsed into a concise expression (called an S-expression), e.g., `What is to the right of the car?' is parsed into (what car);(what right);(what (and car right)). Using these expressions, modules are composed into a sequence to answer the query.
The multi-step recurrent answering units (RAU) model for VQA is another state-of-the-art method BIBREF32 . Each inference step in RAU consists of a complete answering block that takes in an image, a question, and the output from the previous LSTM step. Each of these is part of a larger LSTM network that progressively reasons about the question.
Experiments
We trained multiple baseline models as well as state-of-the-art VQA methods on TDIUC. The methods we use are:
For image features, ResNet-152 BIBREF33 with $448 \times 448$ images was used for all models.
QUES and IMG provide information about biases in the dataset. QUES, Q+I, and MLP all use 4800-dimensional skip-thought vectors BIBREF34 to embed the question, as was done in BIBREF7 . For image features, these all use the `pool5' layer of ResNet-152 normalized to unit length. MLP is a 4-layer net with a softmax output layer. The 3 ReLU hidden layers have 6000, 4000, and 2000 units, respectively. During training, dropout (0.3) was used for the hidden layers.
For MCB, MCB-A, NMN and RAU, we used publicly available code to train them on TDIUC. The experimental setup and hyperparamters were kept unchanged from the default choices in the code, except for upgrading NMN and RAU's visual representation to both use ResNet-152.
Results on TDIUC for these models are given in Table 3 . Accuracy scores are given for each of the 12 question-types in Table 3 , and scores that are normalized by using mean-per-unique-answer are given in appendix Table 5 .
Easy Question-Types for Today's Methods
By inspecting Table 3 , we can see that some question-types are comparatively easy ( $>90$ %) under MPT: scene recognition, sport recognition, and object presence. High accuracy is also achieved on absurd, which we discuss in greater detail in Sec. "Effects of Including Absurd Questions" . Subordinate object recognition is moderately high ( $>80$ %), despite having a large number of unique answers. Accuracy on counting is low across all methods, despite a large number of training data. For the remaining question-types, more analysis is needed to pinpoint whether the weaker performance is due to lower amounts of training data, bias, or limitations of the models. We next investigate how much of the good performance is due to bias in the answer distribution, which N-MPT compensates for.
Effects of the Proposed Accuracy Metrics
One of our major aims was to compensate for the fact that algorithms can achieve high scores by simply learning to answer more populated and easier question-types. For existing datasets, earlier work has shown that simple baseline methods routinely exceed more complex methods using simple accuracy BIBREF7 , BIBREF8 , BIBREF19 . On TDIUC, MLP surpasses MCB and NMN in terms of simple accuracy, but a closer inspection reveals that MLP's score is highly determined by performance on categories with a large number of examples, such as `absurd' and `object presence.' Using MPT, we find that both NMN and MCB outperform MLP. Inspecting normalized scores for each question-type (Appendix Table 5 ) shows an even more pronounced differences, which is also reflected in arithmetic N-MPT score presented in Table 3 . This indicates that MLP is prone to overfitting. Similar observations can be made for MCB-A compared to RAU, where RAU outperforms MCB-A using simple accuracy, but scores lower on all the metrics designed to compensate for the skewed answer distribution and bias.
Comparing the unnormalized and normalized metrics can help us determine the generalization capacity of the VQA algorithms for a given question-type. A large difference in these scores suggests that an algorithm is relying on the skewed answer distribution to obtain high scores. We found that for MCB-A, the accuracy on subordinate object recognition drops from 85.54% with unnormalized to 23.22% with normalized, and for scene recognition it drops from 93.06% (unnormalized) to 38.53% (normalized). Both these categories have a heavily skewed answer distribution; the top-25 answers in subordinate object recognition and the top-5 answers in scene recognition cover over 80% of all questions in their respective question-types. This shows that question-types that appear to be easy may simply be due to the algorithms learning the answer statistics. A truly easy question-type will have similar performance for both unnormalized and normalized metrics. For example, sport recognition shows only 17.39% drop compared to a 30.21% drop for counting, despite counting having same number of unique answers and far more training data. By comparing relative drop in performance between normalized and unnormalized metric, we can also compare the generalization capability of the algorithms, e.g., for subordinate object recognition, RAU has higher unnormalized score (86.11%) compared to MCB-A (85.54%). However, for normalized scores, MCB-A has significantly higher performance (23.22%) than RAU (21.67%). This shows RAU may be more dependent on the answer distribution. Similar observations can be made for MLP compared to MCB.
Can Algorithms Predict Rare Answers?
In the previous section, we saw that the VQA models struggle to correctly predict rarer answers. Are the less repeated questions actually harder to answer, or are the algorithms simply biased toward more frequent answers? To study this, we created a subset of TDIUC that only consisted of questions that have answers repeated less than 1000 times. We call this dataset TDIUC-Tail, which has 46,590 train and 22,065 test questions. Then, we trained MCB on: 1) the full TDIUC dataset; and 2) TDIUC-Tail. Both versions were evaluated on the validation split of TDIUC-Tail.
We found that MCB trained only on TDIUC-Tail outperformed MCB trained on all of TDIUC across all question-types (details are in appendix Table 6 and 7 ). This shows that MCB is capable of learning to correctly predict rarer answers, but it is simply biased towards predicting more common answers to maximize overall accuracy. Using normalized accuracy disincentivizes the VQA algorithms' reliance on the answer statistics, and for deploying a VQA system it may be useful to optimize directly for N-MPT.
Effects of Including Absurd Questions
Absurd questions force a VQA system to look at the image to answer the question. In TDIUC, these questions are sampled from the rest of the dataset, and they have a high prior probability of being answered `Does not apply.' This is corroborated by the QUES model, which achieves a high accuracy on absurd; however, for the same questions when they are genuine for an image, it only achieves 6.77% accuracy on these questions. Good absurd performance is achieved by sacrificing performance on other categories. A robust VQA system should be able to detect absurd questions without then failing on others. By examining the accuracy on real questions that are identical to absurd questions, we can quantify an algorithm's ability to differentiate the absurd questions from the real ones. We found that simpler models had much lower accuracy on these questions, (QUES: 6.77%, Q+I: 34%), compared to more complex models (MCB: 62.44%, MCB-A: 68.83%).
To further study this, we we trained two VQA systems, Q+I and MCB, both with and without absurd. The results are presented in Table 3 . For Q+I trained without absurd questions, accuracies for other categories increase considerably compared to Q+I trained with full TDIUC, especially for question-types that are used to sample absurd questions, e.g., activity recognition (24% when trained with absurd and 48% without). Arithmetic MPT accuracy for the Q+I model that is trained without absurd (57.03%) is also substantially greater than MPT for the model trained with absurd (51.45% for all categories except absurd). This suggests that Q+I is not properly discriminating between absurd and real questions and is biased towards mis-identifying genuine questions as being absurd. In contrast, MCB, a more capable model, produces worse results for absurd, but the version trained without absurd shows much smaller differences than Q+I, which shows that MCB is more capable of identifying absurd questions.
Effects of Balancing Object Presence
In Sec. "Can Algorithms Predict Rare Answers?" , we saw that a skewed answer distribution can impact generalization. This effect is strong even for simple questions and affects even the most sophisticated algorithms. Consider MCB-A when it is trained on both COCO-VQA and Visual Genome, i.e., the winner of the CVPR-2016 VQA Workshop Challenge. When it is evaluated on object presence questions from TDIUC, which contains 50% `yes' and 50% `no' questions, it correctly predicts `yes' answers with 86.3% accuracy, but only 11.2% for questions with `no' as an answer. However, after training it on TDIUC, MCB-A is able to achieve 95.02% for `yes' and 92.26% for `no.' MCB-A performed poorly by learning the biases in the COCO-VQA dataset, but it is capable of performing well when the dataset is unbiased. Similar observations about balancing yes/no questions were made in BIBREF14 . Datasets could balance simple categories like object presence, but extending the same idea to all other categories is a challenging task and undermines the natural statistics of the real-world. Adopting mean-per-class and normalized accuracy metrics can help compensate for this problem.
Advantages of Attentive Models
By breaking questions into types, we can assess which types benefit the most from attention. We do this by comparing the MCB model with and without attention, i.e., MCB and MCB-A. As seen in Table 3 , attention helped improve results on several question categories. The most pronounced increases are for color recognition, attribute recognition, absurd, and counting. All of these question-types require the algorithm to detect specified object(s) (or lack thereof) to be answered correctly. MCB-A computes attention using local features from different spatial locations, instead of global image features. This aids in localizing individual objects. The attention mechanism learns the relative importance of these features. RAU also utilizes spatial attention and shows similar increments.
Compositional and Modular Approaches
NMN, and, to a lesser extent, RAU propose compositional approaches for VQA. For COCO-VQA, NMN has performed worse than some MLP models BIBREF7 using simple accuracy. We hoped that it would achieve better performance than other models for questions that require logically analyzing an image in a step-by-step manner, e.g., positional reasoning. However, while NMN did perform better than MLP using MPT and N-MPT metric, we did not see any substantial benefits in specific question-types. This may be because NMN is limited by the quality of the `S-expression' parser, which produces incorrect or misleading parses in many cases. For example, `What color is the jacket of the man on the far left?' is parsed as (color jacket);(color leave);(color (and jacket leave)). This expression not only fails to parse `the man', which is a crucial element needed to correctly answer the question, but also wrongly interprets `left' as past tense of leave.
RAU performs inference over multiple hops, and because each hop contains a complete VQA system, it can learn to solve different tasks in each step. Since it is trained end-to-end, it does not need to rely on rigid question parses. It showed very good performance in detecting absurd questions and also performed well on other categories.
Conclusion
We introduced TDIUC, a VQA dataset that consists of 12 explicitly defined question-types, including absurd questions, and we used it to perform a rigorous analysis of recent VQA algorithms. We proposed new evaluation metrics to compensate for biases in VQA datasets. Results show that the absurd questions and the new evaluation metrics enable a deeper understanding of VQA algorithm behavior.
Additional Details About TDIUC
In this section, we will provide additional details about the TDIUC dataset creation and additional statistics that were omitted from the main paper due to inadequate space.
Questions using Visual Genome Annotations
As mentioned in the main text, Visual Genome's annotations are both non-exhaustive and duplicated. This makes using them to automatically make question-answer (QA) pairs difficult. Due to these issues, we only used them to make two types of questions: Color Attributes and Positional Reasoning. Moreover, a number of restrictions needed to be placed, which are outlined below.
For making Color Attribute questions, we make use of the attributes metadata in the Visual Genome annotations to populate the template `What color is the <object>?' However, Visual Genome metadata can contain several color attributes for the same object as well as different names for the same object. Since the annotators type the name of the object manually rather than choosing from a predetermined set of objects, the same object can be referred by different names, e.g., `xbox controller,' `game controller,' `joystick,' and `controller' can all refer to same object in an image. The object name is sometimes also accompanied by its color, e.g., `white horse' instead of `horse' which makes asking the Color Attribute question `What color is the white horse?' pointless. One potential solution is to use the wordnet `synset' which accompanies every object annotation in the Visual Genome annotations. Synsets are used to group different variations of the common objects names under a single noun from wordnet. However, we found that the synset matching was erroneous in numerous instances, where the object category was misrepresented by the given synset. For example, A `controller' is matched with synset `accountant' even when the `controller' is referring to a game controller. Similarly, a `cd' is matched with synset of `cadmium.' To avoid these problems we made a set of stringent requirements before making questions:
The chosen object should only have a single attribute that belongs to a set of commonly used colors.
The chosen object name or synset must be one of the 91 common objects in the MS-COCO annotations.
There must be only one instance of the chosen object.
Using these criteria, we found that we could safely ask the question of the form `What color is the <object>?'.
Similarly, for making Positional Reasoning questions, we used the relationships metadata in the Visual Genome annotations. The relationships metadata connects two objects by a relationship phrase. Many of these relationships describe the positions of the two objects, e.g., A is `on right' of B, where `on right' is one of the example relationship clause from Visual Genome, with the object A as the subject and the object B as the object. This can be used to generate Positional Reasoning questions. Again, we take several measures to avoid ambiguity. First, we only use objects that appear once in the image because `What is to the left of A' can be ambiguous if there are two instances of the object A. However, since visual genome annotations are non-exhaustive, there may still (rarely) be more than one instance of object A that was not annotated. To disambiguate such cases, we use the attributes metadata to further specify the object wherever possible, e.g., instead of asking `What is to the right of the bus?', we ask `What is to the right of the green bus?'
Due to a these stringent criteria, we could only create a small number of questions using Visual Genome annotations compared to other sources. The number of questions produced via each source is shown in Table 4 .
Answer Distribution
Figure 3 shows the answer distribution for the different question-types. We can see that some categories, such as counting, scene recognition and sentiment understanding, have a very large share of questions represented by only a few top answers. In such cases, the performance of a VQA algorithm can be inflated unless the evaluation metric compensates for this bias. In other cases, such as positional reasoning and object utility and affordances, the answers are much more varied, with top-50 answers covering less than 60% of all answers.
We have completely balanced answer distribution for object presence questions, where exactly 50% of questions being answered `yes' and the remaining 50% of the questions are answered `no'. For other categories, we have tried to design our question generation algorithms so that a single answer does not have a significant majority within a question type. For example, while scene understanding has top-4 answers covering over 85% of all the questions, there are roughly as many `no' questions (most common answer) as there are `yes' questions (second most-common answer). Similar distributions can be seen for counting, where `two' (most-common answer) is repeated almost as many times as `one' (second most-common answer). By having at least the top-2 answers split almost equally, we remove the incentive for an algorithm to perform well using simple mode guessing, even when using the simple accuracy metric.
Train and Test Split
In the paper, we mentioned that we split the entire collection into 70% train and 30% test/validation. To do this, we not only need to have a roughly equal distribution of question types and answers, but also need to make sure that the multiple questions for same image do not end up in two different splits, i.e., the same image cannot occur in both the train and the test partitions. So, we took following measures to split the questions into train-test splits. First, we split all the images into three separate clusters.
Manually uploaded images, which includes all the images manually uploaded by our volunteer annotators.
Images from the COCO dataset, including all the images for questions generated from COCO annotations and those imported from COCO-VQA dataset. In addition, a large number of Visual Genome questions also refer to COCO images. So, some questions that are generated and imported from Visual Genome are also included in this cluster.
Images exclusively in the Visual Genome dataset, which includes images for a part of the questions imported from Visual Genome and those generated using that dataset.
We follow simple rules to split each of these clusters of images into either belonging to the train or test splits.
All the questions belonging to images coming from the `train2014' split of COCO images are assigned to the train split and all the questions belonging to images from the `val2014' split are assigned to test split.
For manual and Visual Genome images, we randomly split 70% of images to train and rest to test.
Additional Experimental Results
In this section, we present additional experimental results that were omitted from the main paper due to inadequate space. First, the detailed normalized scores for each of the question-types is presented in Table 3 . To compute these scores, the accuracy for each unique answer is calculated separately within a question-type and averaged. Second, we present the results from the experiment in section "Can Algorithms Predict Rare Answers?" in table 6 (Unnormalized) and table 7 (Normalized). The results are evaluated on TDIUC-Tail, which is a subset of TDIUC that only consists of questions that have answers repeated less than 1000 times (uncommon answers). Note that the TDIUC-Tail excludes the absurd and the object presence question-types, as they do not contain any questions with uncommon answers. The algorithms are identical in both Table 6 and 7 and are named as follows:

Question: From when are many VQA datasets collected?
Answer: late 2014