Related Work
Datasets: Over the past few years several large scale datasets for Visual Question Answering have been released. These include datasets such as COCO-QA BIBREF3, DAQUAR BIBREF4, VQA BIBREF5, BIBREF6 which contain questions asked over natural images. On the other hand, datasets such as CLEVR BIBREF7 and NVLR BIBREF8 contain complex reasoning based questions on synthetic images having 2D and 3D geometric objects. There are some datasets BIBREF9, BIBREF10 which contain questions asked over diagrams found in text books but these datasets are smaller and contain multiple-choice questions. FigureSeer BIBREF11 is another dataset which contains images extracted from research papers but this is also a relatively small (60,000 images) dataset. Further, FigureSeer focuses on answering questions based on line plots as opposed to other types of plots such as bar charts, scatter plots, etc. as seen in FigureQA BIBREF0 and DVQA BIBREF1.
Models: The availability of the above mentioned datasets has facilitated the development of complex end-to-end neural network based models (BIBREF2, BIBREF12, BIBREF13, BIBREF14, BIBREF15). These end-to-end networks contain (a) encoders to compute a representation for the image and the question, (b) attention mechanisms to focus on important parts of the question and image, (c) interaction components to capture the interactions between the question and the image, and (d) a classification layer for selecting the answer from a fixed vocabulary. By design, these algorithms cannot be used in situations where the answer does not come from a fixed vocabulary but needs to be computed.
The PlotQA dataset
In this section, we describe the PlotQA dataset and the process to build it. Specifically, we discuss the four main stages, viz., (i) curating data such as year-wise rainfall statistics, country-wise mortality rates, etc., (ii) creating different types of plots with a variation in the number of elements, legend positions, fonts, etc., (iii) crowd-sourcing to generate questions, and (iv) extracting templates from the crowd-sourced questions and instantiating these templates using appropriate phrasing suggested by human annotators.
The PlotQA dataset ::: Data Collection and Curation
We considered online data sources such as World Bank Open Data , Open Government Data , Global Terrorism Database , etc. which contain statistics about various indicator variables such as fertility rate, rainfall, coal production, etc. across years, countries, districts, etc. We crawled data from these sources to extract different variables whose relations could then be plotted (for example, rainfall v/s years across countries, or movie v/s budget, or carbohydrates v/s food_item and so on). Some statistics about the crawled data are of interest. There are a total of 841 unique indicator variables (CO2 emission, Air Quality Index, Fertility Rate, Revenue generated by taxes, etc.) with 160 unique entities (cities, states, districts, countries, movies, food items, etc.). The data ranges from 1960 to 2016, though not all indicator variables have data items for all years. The data contains positive integers, floating point values, percentages, and values on a linear scale. These values range from 0 to $3.50\mathrm {e+}15$.
The PlotQA dataset ::: Plot Generation
We included 3 different types of plots in this dataset, viz., bar plots, line plots and scatter plots. Within bar plots, we have grouped them by orientation as either horizontal or vertical. Within the data sources we explored, we did not find enough data to create certain other types of plots such as Venn diagrams and pie charts which are used in specific settings. We also do not consider composite plots such as Pareto charts which have line graphs on top of bar graphs. Lastly, all the plots in our dataset contain only 2-axes. Figure FIGREF9 shows one sample of each plot type in PlotQA. Each of these plots can compactly represent 3-dimensional data. For instance, in Figure FIGREF9, the plot compares the indicator variable diesel prices across years for different countries. To enable the development of supervised modules for various sub-tasks we provide bounding box annotations for legend boxes, legend names, legend markers, axes titles, axes ticks, bars, lines, and plot title. By using different combination of indicator variables and entities (years, countries, etc.) we created a total of $224,377$ plots.
To ensure that there is enough variety in the plots, we randomly chose the following parameters: grid lines (present/absent), font size, notation used for tick labels (scientific-E notation or standard notation), line style (solid, dashed, dotted, dash-dot), marker styles for marking data points (asterisk, circle, diamond, square, triangle, inverted triangle), position of legends (bottom-left, bottom-centre, bottom-right, center-right, top-right), and colors for the lines and bars from a set of 73 different colors. The number of discrete elements on the $x$-axis varies from 2 to 12. Similarly, the number of entries in the legend box varies from 1 to 4. In other words, in the case of line plots, the number of lines varies from 1 to 4 and in the case of grouped bars the number of bars grouped on a single $x$-tick varies from 1 to 4. For example, for the plots in Figure FIGREF9, the number of discrete elements on the $x$-axis is 6 and the number of legend names (i.e., number of lines) is 3.
The PlotQA dataset ::: Sample Question Collection by Crowd-sourcing
Since the underlying data of the PlotQA dataset is much richer in comparison to FigureQA and DVQA, we found it necessary to ask a wider set of annotators to create questions over these plots. However, creating questions for all the plots in our dataset would have been prohibitively expensive. We sampled $1,400$ plots across different types and asked workers on Amazon Mechanical Turk to create questions for these plots. We showed each plot to 5 different workers resulting in a total of $7,000$ questions. We specifically instructed the workers to ask complex reasoning questions which involved reference to multiple plot elements in the plots. We also gave the workers a list of simple questions such as “Is this a vertical bar graph?”, “What is the title of the graph?”, “What is the value of coal production in 1993?” and asked them to not create such questions as we had already created such questions using hand designed templates. We paid the workers $0.1\$$ for each question.
The PlotQA dataset ::: Question Template Extraction & Instantiation
We manually analyzed the questions collected by crowdsourcing and divided them into a total of 74 templates (including the simple templates that we had manually designed as mentioned earlier). These templates were divided into 3 question categories. These question categories along with a few sample templates are shown below. We refer the reader to the Supplementary material for further details.
Structural Understanding: These are questions about the overall structure of the plot and do not require any quantitative reasoning. Examples: “How many different coloured bars are there?”, “How are the legend labels stacked?”.
Data Retrieval: These questions seek data item for a single element in the plot. Examples: “What is the number of tax payers in Myanmar in 2015?”, “How many bars are there on the 4th tick from the top?”.
Reasoning: These questions either require numeric reasoning over multiple plot elements or a comparative analysis of different elements of the plot, or a combination of both to answer the question. Examples: “In which country is the number of threatened bird species minimum?”, “What is the median banana production?”, “What is the difference between the number of deaths in Bulgaria and Cuba in the year 2005?”, “In how many years, is the rice production greater than the average rice production over all years?”.
We abstracted the questions into templates such as “In how many $<$plural form of X_label$>$, is the $<$Y_label$>$ of/in $<$legend_label$>$ greater than the average $<$Y_label$>$ of/in $<$legend_label$>$ taken over all $<$plural form of X_label$>$?”. We could then generate multiple questions for each template by replacing X_label, Y_label, legend_label, etc. by indicator variables, years, cities etc. from our curated data. However, this was a tedious task requiring a lot of manual intervention. For example, consider the indicator variable “Race of students” in Figure FIGREF1. If we substitute this indicator variable as it is in the above template, it would result in a question, “In how many cities, is the race of the students(%) of Asian greater than the average race of the students (%) of Asian taken over all cities ?”, which sounds unnatural. To avoid this, we asked in-house annotators to carefully paraphrase these indicator variables and question templates. The paraphrased version of the above example was “In how many cities, is the percentage of Asian students greater than the average percentage of Asian students taken over all cities ?'. Such paraphrasing for every question template and indicator variable required significant manual effort. Using this semi-automated process we were able to generate a total of $8,190,674$ questions. As shown in Table TABREF27, the answer could either be Yes/No or from Fixed Vocabulary, or Open Vocabulary. blackWe believe that this approach of creating questions based on templates extracted from complex human generated questions is a good middle ground between (i) the expensive and time consuming process of creating questions with the help of humans and (ii) the inexpensive and fast process of creating questions from very simple templates (as in FigureQA and DVQA).
Proposed Model
Existing models for VQA treat it as a multi-class classification problem, i.e., they assume that the answer needs to be picked from a fixed vocabulary. Such models work well for datasets such as DVQA where indeed all answers come from a fixed vocabulary (global or plot specific). However, in our dataset, for roughly 23% of Data Retrieval questions and 46% of Reasoning questions, the answers do not come from a fixed vocabulary but need to be computed by reasoning over one or more visual elements in the plots. To address such complex questions, we seek to leverage existing results on QA over tables BIBREF16. However, this requires the intermediate step of translating the plot image into a structured table (potentially similar to the one from which the plot was generated). To this end, we propose a pipelined method which separates the tasks of visual element detection and reasoning for QA. More specifically, our pipeline contains modules for (i) detecting visual elements in the plot such as bars, bounding boxes around axes label, etc. (ii) performing optical character recognition within these bounding boxes (iii) converting this data into a structured table and (iv) answering questions using this structured table.
Proposed Model ::: Visual Elements Detection (VED)
The data bearing elements of a plot are of 10 distinct classes: the title, the labels of the $x$ and $y$ axes, the tick labels or categories (e.g., countries) on the $x$ and $y$ axis, the data markers in the legend box, the legend names, and finally the bars and lines in the graph. Following existing literature (BIBREF17,BIBREF1), we refer to these elements as the visual elements of the graph. The first task is to extract all these visual elements by drawing bounding boxes around them and classifying them into the appropriate class. We can treat this as (i) an object detection + classification task or (ii) an instance segmentation task. If we take the former view then we can use existing object detection models such as RCNN, Fast-RCNN BIBREF18, YOLO BIBREF19, SSD BIBREF20, etc. and if we take the latter view we can use instance segmentation models such as Mask-RCNN. We tried both these approaches and found that instance segmentation based methods perform better for this task and hence we use Mask-RCNN as our VED module. Figure FIGREF21 shows the expected output of this stage with all the visual elements detected.
Proposed Model ::: Object Character Recognition (OCR)
Some of the visual elements such as title, legends, tick labels, etc. contain numeric and textual data. For extracting this data from within these bounding boxes, we use a state-of-the-art OCR model BIBREF21. More specifically, we crop the detected visual element to its bounding box, convert the cropped image into grayscale, resize and deskew it, and then pass it to an OCR module. Existing OCR modules perform well for machine-written English text, and indeed we found that a pre-trained OCR module works well on our dataset.
Proposed Model ::: Semi-Structured Information Extraction (SIE)
The next stage of extracting the data into a semi-structured table is best explained with an example shown in Figure FIGREF21. The desired output of SIE is shown in the table where the rows correspond to the ticks on the $x$-axis (1996, 1997, 1998, 1999), the columns correspond to the different elements listed in the legend (Brazil, Iceland, Kazakhstan, Thailand) and the $i$,$j$-th cell contains the value corresponding to the $x$-th tick and the $y$-th legend. The values of the $x$-tick labels and the legend names are available from the OCR module. The mapping of legend name to legend marker or color is done by associating a legend name to the marker or color whose bounding box is closest to the bounding box of the legend name. Similarly, we associate each tick label to the tick marker whose bounding box is closest to the bounding box of the tick label. For example, we associate the legend name Brazil to the color “Dark Cyan” and the tick label 1996 to the corresponding tick mark on the $x$-axis. With this we have the 4 row headers and 4 column headers, respectively. To fill in the 16 values in the table, there are again two smaller steps. First we associate each of the 16 bounding boxes of the 16 bars to their corresponding $x$-ticks and legend names. A bar is associated with an $x$-tick label whose bounding box is closest to the bounding box of the bar. To associate a bar to a legend name, we find the dominant color in the bounding box of the bar and match it with a legend name corresponding to that color. Second, we need to find the value represented by each bar. We extract the height of the bar using bounding box information from the VED module and then search for the $y$-tick labels immediately above and below that height. We then interpolate the value of the bar based on the values of these bounding ticks. With this we have the 16 values in the cells and thus have extracted all the information from the plot into a semi-structured table.
Proposed Model ::: Table Question Answering (QA)
The final stage of the pipeline is to answer questions on the semi-structured table. As this is similar to answering questions from the WikiTableQuestions dataset BIBREF22, we adopt the same methodology as proposed in BIBREF22. In this method, the table is converted to a knowledge graph and the question is converted to a set of candidate logical forms by applying compositional semantic parsing. These logical forms are then ranked using a log-linear model and the highest ranking logical form is applied to the knowledge graph to get the answer. Note that with this approach the output is computed by a logical form that operates on the numerical data. This avoids the limitation of using a small answer vocabulary for multi-class classification as is done in existing work on VQA. blackThere are some recent neural approaches for answering questions over semi-structured tables such as BIBREF23, BIBREF24 which take an ensemble of many models and outperform the relatively simpler model of BIBREF22 only by a small margin (1-2%). In the absence of an ensemble, these neural methods do not perform better than the method proposed in BIBREF22. To the best of our knowledge, there is one neural method BIBREF25 which performs better than BIBREF22 but the code for this model is not available which makes it hard to reproduce their results. Hence we chose the model of BIBREF22 for this stage which is relatively simpler and readily available.
Experiments
In this section we detail the data splits, baseline models, hyperparameter tunnig and evaluation metrics.
Experiments ::: Train-Valid-Test Splits
As mentioned earlier, by using different combinations of 841 indicator variables and 160 entities (years, countries, etc), we created a total of $224,377$ plots. Depending on the context and type of the plot, we instantiated the 74 templates to create meaningful (question,answer) pairs for each of the plots. The number of questions per plot varies from 17 to 44. We created train (70%), valid (15%) and test (15%) splits from this data. These statistics are summarized in Table TABREF27. The dataset, crowd-sourced questions and the model will be made available on the acceptance of this paper.
Experiments ::: Models Compared
We compare the performance of the following models:
- IMG-only: This is a simple baseline where we just pass the image through a VGG19 and use the embedding of the image to predict the answer from a fixed vocabulary.
- QUES-only: This is a simple baseline where we just pass the question through a LSTM and use the embedding of the question to predict the answer from a fixed vocabulary.
- SANBIBREF2: This is a state of the art VQA model which is an encoder-decoder model with a multi-layer stacked attention BIBREF26 mechanism. It obtains a representation for the image using a deep CNN and a representation for the query using LSTM. It then uses the query representation to locate relevant regions in the image and uses this to pick an answer from a fixed vocabulary.
- SANDYBIBREF1: This is the best performing model on the DVQA dataset and is a variant of SAN. Unfortunately, the code for this model is not available and the description in the paper was not detailed enough for us to reimplement it. Hence, we report the numbers for this model only on DVQA (from the original paper).
- VOES: This is our model as described in section SECREF3 which is specifically designed for questions which do not have answers from a fixed vocabulary.
- VOES-Oracle: blackThis is our model where the first three stages of VOES are replaced by an Oracle, i.e., the QA model answers questions on a table that has been generated using the ground truth annotations of the plot. With this we can evaluate the performance of the WikiTableQA model when it is not affected by the VED model's errors.
- SAN-VOES: Given the complementary strengths of SAN-VQA and VOES, we train a hybrid model with a binary classifier which given a question decides whether to use the SAN or the VOES model. The data for training this binary classifier is generated by comparing the predictions of a trained SAN model and a trained VOES model on the training dataset. For a given question, the label is set to 1 (pick SAN) if the performance of SAN was better than that of VOES. We ignore questions where there is a tie. The classifier is a simple LSTM based model which computes a representation for the question using an LSTM and uses this representation to predict 1/0. At test time, we first pass the question through this model and depending on the output of this model use SAN or VOES.
Experiments ::: Training Details
SAN: We used an existing implementation of SAN for establishing the initial baseline results. Image features are extracted from the last pooling layer of VGG19 network. Question features are the last hidden state of the LSTM. Both the LSTM hidden state and 512-d image feature vector at each location are transferred to a 1024-d vector by a fully connected layer, and added and passed through a non-linearity (tanh). The model was trained using Adam BIBREF27 optimizer with an initial learning rate of $0.0003$ and a batch size of 128 for 25000 iterations.
Proposed Pipeline: Of the four stages of the pipeline described in Section 4.2 only two require training, viz., Visual Elements Detection (VED) and Table Question Answering (QA). As mentioned earlier, for VED we train an instance segmentation model (MaskRCNN BIBREF28) using the bounding box annotations available in our dataset. We trained each model with a batch size of 32 for $200,000$ steps, beyond which no further training benefit was seen. We used RMSProp as the optimizer with an initial learning rate of $0.004$. For Table QA, we trained the model proposed in BIBREF22 using questions from our dataset and the corresponding blackground truth tables. Since this model is computationally expensive with a high training time, we could train it using only $400,000$ questions from our training set.
SAN-VOES: The binary question classifier in this hybrid model contains a 50-dimensional word embedding layer followed by an LSTM with 128 hidden units. The output of the LSTM is projected to 256 dimensions and this is then fed to the output layer. The model is trained for 10 epochs using RMSProp with an initial learning rate of 0.001. Accuracy on the validation set is $87.3\%$.
Experiments ::: Evaluation Metric
We used accuracy as the evaluation metric. Specifically, for textual answers (such as India, CO2, etc.) the model's output was considered to be correct only if the predicted answer exactly matches the true answer. However, for numeric answers which contain floating point values such an exact match is a very strict evaluation metric (for example, if the predicted answer is 10.5 and the true answer is 10 then in most cases it would be acceptable). Hence, we relax the accuracy measure to consider the predicted answer to be correct as long as it is within 5% of the correct answer.
Observations and Results
1. Evaluating models on PlotQA dataset (Table TABREF35): blackThe baselines IMG-only and QUES-only performed poorly with an accuracy of $14.84\%$ and $15.35\%$ respectively. We then evaluate SAN, VOES, VOES-Oracle, and SAN-VOES on each of the 9 question-answer types of the PlotQA dataset. SAN performs very well on Yes/No questions and moderately well on Fixed vocab. questions with a good baseline aggregate accuracy of 46.54%. SAN performs poorly on Open vocab. question, failing to answer almost all the 319,000 questions in this category. On the other hand, VOES fails to answer correctly any of the Yes/No questions, performs moderately well on Fixed vocab. questions, and answers correctly some of the hard Open vocab. questions. SAN-VOES combines the complementary strengths of SAN and VOES with the highest accuracy of 53.96%. In particular, the performance improves significantly for all Fixed Vocab. questions, while retaining the high accuracy of SAN on Yes/No questions and VOES' performance on Open vocab. There is a significant difference in the performance of VOES and VOES-Oracle across multiple question-answer types. This implies that the visual element detection in the VOES pipeline can be further improved.
2. Analysis of the VOES pipeline We analyze the performance of the of visual element detection (VED) and OCR.
- Table TABREF36 shows that the VED module performs reasonably well at an Intersection Over Union (IOU) of 0.5. For higher IOUs of 0.8 and 0.9, the accuracy falls drastically. For instance, at IOU of 0.9, dotlines are detected with an accuracy of under 5%. Clearly, such inaccuracies would lead to incorrect table generation and subsequent QA. This brings out an interesting difference between this task and other instance segmentation tasks where the margin of error is higher (where IOU of 0.5 is accepted). A small error in visual element detection as indicated by mAP scores of 80% is considered negligible for VQA tasks, however for PlotQA small errors can cause significantly misaligned table generation and subsequent QA. blackWe illustrate this with an example given in Figure 4. The predicted red box having an IOU of 0.58 estimates the bar size as 760 as opposed to ground truth of 680, significantly impacting downstream QA accuracy.
Retraining VED model with a higher IOU of 0.75 only resulted in a small increase in accuracy (last row). blackThus, inverting the plot generation function in going from the plot image to the structured table is a difficult CV task.
- In Table TABREF37 we evaluate the performance of the OCR module in standalone/oracle mode and pipeline mode. In the oracle mode, we feed ground truth boxes to the OCR model whereas in the pipeline model we perform OCR on the output of the VED module. We observe only a small drop in performance, which indicates that the OCR module is robust to the reduction in VED module's accuracy at higher IOU as it does not depend on the class label or the exact position of bounding boxes.
- In summary, a highly accurate VED for structured images is an open challenge to improve reasoning over plots.
3. Evaluating new models on the existing DVQA dataset (Table TABREF34): The proposed model VOES performs better than the existing models (SAN and SANDY-OCR) on DVQA. The higher performance of VOES in comparison to SAN (in contrast to the PlotQA results) suggests that the extraction of the structured table is more accurate on the DVQA dataset. This is because of the limited variability in the axis and tick labels and shorter length (one word only) of labels. The hybrid model, SAN-VOES, improves on the individual models and establishes a new SOTA result on DVQA.
Conclusion
We introduce the PlotQA dataset to reduce the gap between existing synthetic plot datasets and real-world plots and question templates. Analysis of an existing model for VQA for plots, SAN, on PlotQA reveals that it performs poorly for Open Vocabulary questions. We proposed the VOES model as a pipelined approach that combines visual element detection and OCR with QA over tables, specifically for the Open Vocabulary questions. A hybrid model, VOES-SAN, that combines SAN and VOES for different question types, generates state-of-the-art results on both the DVQA and PlotQA datasets. Detailed analysis of the VOES pipeline reveals the need for more accurate visual element detection to improve reasoning over plots.

Question: How big are FigureQA and DVQA datasets?
Answer: Unanswerable

Question: What models other than SAN-VOES are trained on new PlotQA dataset?
Answer: IMG-only, QUES-only, SAN, SANDY,  VOES-Oracle, VOES