Visual Question Answering using Deep Learning: A Survey and Performance Analysis

Abstract: The Visual Question Answering (VQA) task combines challenges for processing data with both Visual and Linguistic processing, to answer basic `common sense' questions about given images. Given an image and a question in natural language, the VQA system tries to find the correct answer to it using visual elements of the image and inference gathered from textual questions. In this survey, we cover and discuss the recent datasets released in the VQA domain dealing with various types of question-formats and enabling robustness of the machine-learning models. Next, we discuss about new deep learning models that have shown promising results over the VQA datasets. At the end, we present and discuss some of the results computed by us over the vanilla VQA models, Stacked Attention Network and the VQA Challenge 2017 winner model. We also provide the detailed analysis along with the challenges and future research directions.

Visual Question Answering (VQA) refers to a challenging task which lies at the intersection of image understanding and language processing. The VQA task has witnessed a significant progress in the recent years by the machine intelligence community. The aim of VQA is to develop a system to answer specific questions about an input image. The answer could be in any of the following forms: a word, a phrase, binary answer, multiple choice answer, or a fill in the blank answer. Agarwal et al. BIBREF0 presented a novel way of combining computer vision and natural language processing concepts of to achieve Visual Grounded Dialogue, a system mimicking the human understanding of the environment with the use of visual observation and language understanding.
The advancements in the field of deep learning have certainly helped to develop systems for the task of Image Question Answering. Krizhevsky et al BIBREF1 proposed the AlexNet model, which created a revolution in the computer vision domain. The paper introduced the concept of Convolution Neural Networks (CNN) to the mainstream computer vision application. Later many authors have worked on CNN, which has resulted in robust, deep learning models like VGGNet BIBREF2, Inception BIBREF3, ResNet BIBREF4, and etc. Similarly, the recent advancements in natural language processing area based on deep learning have improved the text understanding prforance as well. The first major algorithm in the context of text processing is considered to be the Recurrent Neural Networks (RNN) BIBREF5 which introduced the concept of prior context for time series based data. This architecture helped the growth of machine text understanding which gave new boundaries to machine translation, text classification and contextual understanding. Another major breakthrough in the domain was the introduction of Long-Short Term Memory (LSTM) architecture BIBREF6 which improvised over the RNN by introducing a context cell which stores the prior relevant information.
The vanilla VQA model BIBREF0 used a combination of VGGNet BIBREF2 and LSTM BIBREF6. This model has been revised over the years, employing newer architectures and mathematical formulations. Along with this, many authors have worked on producing datasets for eliminating bias, strengthening the performance of the model by robust question-answer pairs which try to cover the various types of questions, testing the visual and language understanding of the system. In this survey, first we cover major datasets published for validating the Visual Question Answering task, such as VQA dataset BIBREF0, DAQUAR BIBREF7, Visual7W BIBREF8 and most recent datasets up to 2019 include Tally-QA BIBREF9 and KVQA BIBREF10. Next, we discuss the state-of-the-art architectures designed for the task of Visual Question Answering such as Vanilla VQA BIBREF0, Stacked Attention Networks BIBREF11 and Pythia v1.0 BIBREF12. Next we present some of our computed results over the three architectures: vanilla VQA model BIBREF0, Stacked Attention Network (SAN) BIBREF11 and Teney et al. model BIBREF13. Finally, we discuss the observations and future directions.
The major VQA datasets are summarized in Table TABREF2. We present the datasets below.
DAQUAR: DAQUAR stands for Dataset for Question Answering on Real World Images, released by Malinowski et al. BIBREF7. It is the first dataset released for the IQA task. The images are taken from NYU-Depth V2 dataset BIBREF17. The dataset is small with a total of 1449 images. The question bank includes 12468 question-answer pairs with 2483 unique questions. The questions have been generated by human annotations and confined within 9 question templates using annotations of the NYU-Depth dataset.
VQA Dataset: The Visual Question Answering (VQA) dataset BIBREF0 is one of the largest datasets collected from the MS-COCO BIBREF18 dataset. The VQA dataset contains at least 3 questions per image with 10 answers per question. The dataset contains 614,163 questions in the form of open-ended and multiple choice. In multiple choice questions, the answers can be classified as: 1) Correct Answer, 2) Plausible Answer, 3) Popular Answers and 4) Random Answers. Recently, VQA V2 dataset BIBREF0 is released with additional confusing images. The VQA sample images and questions are shown in Fig. SECREF2 in 1st row and 1st column.
Visual Madlibs: The Visual Madlibs dataset BIBREF15 presents a different form of template for the Image Question Answering task. One of the forms is the fill in the blanks type, where the system needs to supplement the words to complete the sentence and it mostly targets people, objects, appearances, activities and interactions. The Visual Madlibs samples are shown in Fig. SECREF2 in 1st row and 2nd column.
Visual7W: The Visual7W dataset BIBREF8 is also based on the MS-COCO dataset. It contains 47,300 COCO images with 327,939 question-answer pairs. The dataset also consists of 1,311,756 multiple choice questions and answers with 561,459 groundings. The dataset mainly deals with seven forms of questions (from where it derives its name): What, Where, When, Who, Why, How, and Which. It is majorly formed by two types of questions. The ‘telling’ questions are the ones which are text-based, giving a sort of description. The ‘pointing’ questions are the ones that begin with ‘Which,’ and have to be correctly identified by the bounding boxes among the group of plausible answers.
CLEVR: CLEVR BIBREF16 is a synthetic dataset to test the visual understanding of the VQA systems. The dataset is generated using three objects in each image, namely cylinder, sphere and cube. These objects are in two different sizes, two different materials and placed in eight different colors. The questions are also synthetically generated based on the objects placed in the image. The dataset also accompanies the ground-truth bounding boxes for each object in the image.
Tally-QA: Very recently, in 2019, the Tally-QA BIBREF9 dataset is proposed which is the largest dataset of object counting in the open-ended task. The dataset includes both simple and complex question types which can be seen in Fig. SECREF2. The dataset is quite large in numbers as well as it is 2.5 times the VQA dataset. The dataset contains 287,907 questions, 165,000 images and 19,000 complex questions. The Tally-QA samples are shown in Fig. SECREF2 in 2nd row and 1st column.
KVQA: The recent interest in common-sense questions has led to the development of world Knowledge based VQA dataset BIBREF10. The dataset contains questions targeting various categories of nouns and also require world knowledge to arrive at a solution. Questions in this dataset require multi-entity, multi-relation, and multi- hop reasoning over large Knowledge Graphs (KG) to arrive at an answer. The dataset contains 24,000 images with 183,100 question-answer pairs employing around 18K proper nouns. The KVQA samples are shown in Fig. SECREF2 in 2nd row and 2nd column.
The emergence of deep-learning architectures have led to the development of the VQA systems. We discuss the state-of-the-art methods with an overview in Table TABREF6.
Vanilla VQA BIBREF0: Considered as a benchmark for deep learning methods, the vanilla VQA model uses CNN for feature extraction and LSTM or Recurrent networks for language processing. These features are combined using element-wise operations to a common feature, which is used to classify to one of the answers as shown in Fig. FIGREF4.
Stacked Attention Networks BIBREF11: This model introduced the attention using the softmax output of the intermediate question feature. The attention between the features are stacked which helps the model to focus on the important portion of the image.
Teney et al. Model BIBREF13: Teney et al. introduced the use of object detection on VQA models and won the VQA Challenge 2017. The model helps in narrowing down the features and apply better attention to images. The model employs the use of R-CNN architecture and showed significant performance in accuracy over other architectures.
Neural-Symbolic VQA BIBREF23: Specifically made for CLEVR dataset, this model leverages the question formation and image generation strategy of CLEVR. The images are converted to structured features and the question features are converted to their original root question strategy. This feature is used to filter out the required answer.
Focal Visual Text Attention (FVTA) BIBREF24: This model combines the sequence of image features generated by the network, text features of the image (or probable answers) and the question. It applies the attention based on the both text components, and finally classifies the features to answer the question. This model is better suited for the VQA in videos which has more use cases than images.
Pythia v1.0 BIBREF27: Pythia v1.0 is the award winning architecture for VQA Challenge 2018. The architecture is similar to Teney et al. BIBREF13 with reduced computations with element-wise multiplication, use of GloVe vectors BIBREF22, and ensemble of 30 models.
Differential Networks BIBREF19: This model uses the differences between forward propagation steps to reduce the noise and to learn the interdependency between features. Image features are extracted using Faster-RCNN BIBREF21. The differential modules BIBREF29 are used to refine the features in both text and images. GRU BIBREF30 is used for question feature extraction. Finally, it is combined with an attention module to classify the answers. The Differential Networks architecture is illustrated in Fig. FIGREF5.
The reported results for different methods over different datasets are summarized in Table TABREF2 and Table TABREF6. It can be observed that VQA dataset is very commonly used by different methods to test the performance. Other datasets like Visual7W, Tally-QA and KVQA are also very challenging and recent datasets. It can be also seen that the Pythia v1.0 is one of the recent methods performing very well over VQA dataset. The Differentail Network is the very recent method proposed for VQA task and shows very promising performance over different datasets.
As part of this survey, we also implemented different methods over different datasets and performed the experiments. We considered the following three models for our experiments, 1) the baseline Vanilla VQA model BIBREF0 which uses the VGG16 CNN architecture BIBREF2 and LSTMs BIBREF6, 2) the Stacked Attention Networks BIBREF11 architecture, and 3) the 2017 VQA challenge winner Teney et al. model BIBREF13. We considered the widely adapted datasets such as standard VQA dataset BIBREF0 and Visual7W dataset BIBREF8 for the experiments. We used the Adam Optimizer for all models with Cross-Entropy loss function. Each model is trained for 100 epochs for each dataset.
The experimental results are presented in Table TABREF7 in terms of the accuracy for three models over two datasets. In the experiments, we found that the Teney et al. BIBREF13 is the best performing model on both VQA and Visual7W Dataset. The accuracies obtained over the Teney et al. model are 67.23% and 65.82% over VQA and Visual7W datasets for the open-ended question-answering task, respectively. The above results re-affirmed that the Teney et al. model is the best performing model till 2018 which has been pushed by Pythia v1.0 BIBREF12, recently, where they have utilized the same model with more layers to boost the performance.
The Visual Question Answering has recently witnessed a great interest and development by the group of researchers and scientists from all around the world. The recent trends are observed in the area of developing more and more real life looking datasets by incorporating the real world type questions and answers. The recent trends are also seen in the area of development of sophisticated deep learning models by better utilizing the visual cues as well as textual cues by different means. The performance of even best model is still lagging and around 60-70% only. Thus, it is still an open problem to develop better deep learning models as well as more challenging datasets for VQA. Different strategies like object level details, segmentation masks, deeper models, sentiment of the question, etc. can be considered to develop the next generation VQA models.

Figure and table captions:
Fig. 1. The timeline of major breakthrough in Visual Question Answering (VQA) in last 5 years, ranging from DAQUAR in 2014 to Differential Networks in 2019.
TABLE I OVERVIEW OF VQA DATASETS DESCRIBED IN THIS PAPER.
Fig. 3. Vanilla VQA Network Model [1].
Fig. 4. Differential Networks Model [20].
TABLE II OVERVIEW OF MODELS DESCRIBED IN THIS PAPER. THE PYTHIA V0.1 IS THE BEST PERFORMING MODEL OVER VQA DATASET.
TABLE III THE ACCURACIES OBTAINED USING VANILLA VQA [1], STACKED ATTENTION NETWORKS [12] AND TENEY ET AL. [14] MODELS WHEN TRAINED ON VQA [1] AND VISUAL7W [9] DATASETS.

Questions about this paper:
What are remaining challenges in VQA?
What was the architecture of the 2017 Challenge Winner model?
What is an example of a common sense question?

Answers:
 object level details, segmentation masks, and sentiment of the question
Region-based CNN
Can you park here?
Is something under the sink broken?
Does this man have children?