Visual Question: Predicting If a Crowd Will Agree on the Answer

Visual question answering (VQA) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response. However, close examination of the VQA problem reveals an unavoidable, entangled problem that multiple humans may or may not always agree on a single answer to a visual question. We train a model to automatically predict from a visual question whether a crowd would agree on a single answer. We then propose how to exploit this system in a novel application to efficiently allocate human effort to collect answers to visual questions. Specifically, we propose a crowdsourcing system that automatically solicits fewer human responses when answer agreement is expected and more human responses when answer disagreement is expected. Our system improves upon existing crowdsourcing systems, typically eliminating at least 20% of human effort with no loss to the information collected from the crowd.

section_name: Introduction
paragraphs: What would be possible if a person had an oracle that could immediately provide the answer to any question about the visual world? Sight-impaired users could quickly and reliably figure out the denomination of their currency and so whether they spent the appropriate amount for a product BIBREF0 . Hikers could immediately learn about their bug bites and whether to seek out emergency medical care. Pilots could learn how many birds are in their path to decide whether to change course and so avoid costly, life-threatening collisions. These examples illustrate several of the interests from a visual question answering (VQA) system, including tackling problems that involve classification, detection, and counting. More generally, the goal for VQA is to have a single system that can accurately answer any natural language question about an image or video BIBREF1 , BIBREF2 , BIBREF3 .
Entangled in the dream of a VQA system is an unavoidable issue that, when asking multiple people a visual question, sometimes they all agree on a single answer while other times they offer different answers (Figure FIGREF1 ). In fact, as we show in the paper, these two outcomes arise in approximately equal proportions in today's largest publicly-shared VQA benchmark that contains over 450,000 visual questions. Figure FIGREF1 illustrates that human disagreements arise for a variety of reasons including different descriptions of the same concept (e.g., “minor" and “underage"), different concepts (e.g., “ghost" and “photoshop"), and irrelevant responses (e.g., “no").
Our goal is to account for whether different people would agree on a single answer to a visual question to improve upon today's VQA systems. We propose multiple prediction systems to automatically decide whether a visual question will lead to human agreement and demonstrate the value of these predictions for a new task of capturing the diversity of all plausible answers with less human effort.
Our work is partially inspired by the goal to improve how to employ crowds as the computing power at run-time. Towards satisfying existing users, gaining new users, and supporting a wide range of applications, a crowd-powered VQA system should be low cost, have fast response times, and yield high quality answers. Today's status quo is to assume a fixed number of human responses per visual question and so a fixed cost, delay, and potential diversity of answers for every visual question BIBREF2 , BIBREF0 , BIBREF4 . We instead propose to dynamically solicit the number of human responses based on each visual question. In particular, we aim to accrue additional costs and delays from collecting extra answers only when extra responses are needed to discover all plausible answers. We show in our experiments that our system saves 19 40-hour work weeks and $1800 to answer 121,512 visual questions, compared to today's status quo approach BIBREF0 .
Our work is also inspired by the goal to improve how to employ crowds to produce the information needed to train and evaluate automated methods. Specifically, researchers in fields as diverse as computer vision BIBREF2 , computational linguistics BIBREF1 , and machine learning BIBREF3 rely on large datasets to improve their VQA algorithms. These datasets include visual questions and human-supplied answers. Such data is critical for teaching machine learning algorithms how to answer questions by example. Such data is also critical for evaluating how well VQA algorithms perform. In general, “bigger" data is better. Current methods to create these datasets assume a fixed number of human answers per visual question BIBREF2 , BIBREF4 , thereby either compromising on quality by not collecting all plausible answers or cost by collecting additional answers when they are redundant. We offer an economical way to spend a human budget to collect answers from crowd workers. In particular, we aim to actively allocate additional answers only to visual questions likely to have multiple answers.
The key contributions of our work are as follows:
section_name: Paper Overview
paragraphs: The remainder of the paper is organized into four sections. We first describe a study where we investigate: 1) How much answer diversity arises for visual questions? and 2) Why do people disagree (Section SECREF4 )? Next, we explore the following two questions: 1) Given a novel visual question, can a machine correctly predict whether multiple independent members of a crowd would supply the same answer? and 2) If so, what insights does our machine-learned system reveal regarding what humans are most likely to agree about (Section SECREF5 )? In the following section, we propose a novel resource allocation system for efficiently capturing the diversity of all answers for a set of visual questions (Section SECREF6 ). Finally, we end with concluding remarks (Section SECREF7 ).
section_name: VQA - Analysis of Answer (Dis)Agreements
paragraphs: Our first aim is to answer the following questions: 1) How much answer diversity arises for visual questions? and 2) Why do people disagree?
section_name: Visual Question - Predicting If a Crowd Will (Dis)Agree on an Answer
paragraphs: We now explore the following two questions: 1) Given a novel visual question, can a machine correctly predict whether multiple independent members of a crowd would supply the same answer? and 2) If so, what insights does our machine-learned system reveal regarding what humans are most likely to agree about?
section_name: Prediction Systems
paragraphs: We pose the prediction task as a binary classification problem. Specifically, given an image and associated question, a system outputs a binary label indicating whether a crowd will agree on the same answer. Our goal is to design a system that can detect which visual questions to assign a disagreement label, regardless of the disagreement cause (e.g., subjectivity, ambiguity, difficulty). We implement both random forest and deep learning classifiers.
A visual question is assigned either an answer agreement or disagreement label. To assign labels, we employ 10 crowdsourced answers for each visual question. A visual question is assigned an answer agreement label when there is an exact string match for 9 of the 10 crowdsourced answers (after answer pre-preprocessing, as discussed in the previous section) and an answer disagreement label otherwise. Our rationale is to permit the possibility of up to one “careless/spam" answer per visual question. The outcome of our labeling scheme is that a disagreement label is agnostic to the specific cause of disagreement and rather represents the many causes (described above).
For our first system, we use domain knowledge to guide the learning process. We compile a set of features that we hypothesize inform whether a crowd will arrive at an undisputed, single answer. Then we apply a machine learning tool to reveal the significance of each feature. We propose features based on the observation that answer agreement often arises when 1) a lay person's attention can be easily concentrated to a single, undisputed region in an image and 2) a lay person would find the requested task easy to address.
We employ five image-based features coming from the salient object subitizing BIBREF22 (SOS) method, which produces five probabilities that indicate whether an image contains 0, 1, 2, 3, or 4+ salient objects. Intuitively, the number of salient objects shows how many regions in an image are competing for an observer's attention, and so may correlate with the ease in identifying a region of interest. Moreover, we hypothesize this feature will capture our observation from the previous study that counting problems typically leads to disagreement for images showing many objects, and agreement otherwise.
We employ a 2,492-dimensional feature vector to represent the question-based features. One feature is the number of words in the question. Intuitively, a longer question offers more information and we hypothesize additional information makes a question more precise. The remaining features come from two one-hot vectors describing each of the first two words in the question. Each one-hot vector is created using the learned vocabularies that define all possible words at the first and second word location of a question respectively (using training data, as described in the next section). Intuitively, early words in a question inform the type of answers that might be possible and, in turn, possible reasons/frequency for answer disagreement. For example, we expect “why is" to regularly elicit many opinions and so disagreement. This intuition about the beginning words of a question is also supported by our analysis in the previous section which shows that different answer types yield different biases of eliciting answer agreement versus disagreement.
We leverage a random forest classification model BIBREF23 to predict an answer (dis)agreement label for a given visual question. This model consists of an ensemble of decision tree classifiers. We train the system to learn the unique weighted combinations of the aforementioned 2,497 features that each decision tree applies to make a prediction. At test time, given a novel visual question, the trained system converts a 2,497 feature descriptor of the visual question into a final prediction that reflects the majority vote prediction from the ensemble of decision trees. The system returns the final prediction along with a probability indicating the system's confidence in that prediction. We employ the Matlab implementation of random forests, using 25 trees and the default parameters.
We next adapt a VQA deep learning architecture BIBREF24 to learn the predictive combination of visual and textual features. The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer.
We train the system to predict (dis)agreement labels with training examples, where each example includes an image and question. At test time, given a novel visual question, the system outputs an unnormalized log probability indicating its belief in both the agreement and disagreement label. For our system's prediction, we convert the belief in the disagreement label into a normalized probability. Consequently, predicted values range from 0 to 1 with lower values reflecting greater likelihood for crowd agreement.
section_name: Analysis of Prediction System
paragraphs: We now describe our studies to assess the predictive power of our classification systems to decide whether visual questions will lead to answer (dis)agreement.
We capitalize on today's largest visual question answering dataset BIBREF2 to evaluate our prediction system, which includes 369,861 visual questions about real images. Of these, 248,349 visual questions (i.e., Training questions 2015 v1.0) are kept for for training and the remaining 121,512 visual questions (i.e., Validation questions 2015 v1.0) are employed for testing our classification system. This separation of training and testing samples enables us to estimate how well a classifier will generalize when applied to an unseen, independent set of visual questions.
To our knowledge, no prior work has directly addressed predicting answer (dis)agreement for visual questions. Therefore, we employ as a baseline a related VQA algorithm BIBREF24 , BIBREF2 which produces for a given visual question an answer with a confidence score. This system parallels the deep learning architecture we adapt. However, it predicts the system's uncertainty in its own answer, whereas we are interested in the humans' collective disagreement on the answer. Still, it is a useful baseline to see if an existing algorithm could serve our purpose.
We evaluate the predictive power of the classification systems based on each classifier's predictions for the 121,512 visual questions in the test dataset. We first show performance of the baseline and our two prediction systems using precision-recall curves. The goals are to achieve a high precision, to minimize wasting crowd effort when their efforts will be redundant, and a high recall, to avoid missing out on collecting the diversity of accepted answers from a crowd. We also report the average precision (AP), which indicates the area under a precision-recall curve. AP values range from 0 to 1 with better-performing prediction systems having larger values.
Figure FIGREF8 a shows precision-recall curves for all prediction systems. Both our proposed classification systems outperform the VQA Algorithm BIBREF2 baseline; e.g., Ours - RF yields a 12 percentage point improvement with respect to AP. This is interesting because it shows there is value in learning the disagreement task specifically, rather than employing an algorithm's confidence in its answers. More generally, our results demonstrate it is possible to predict whether a crowd will agree on a single answer from a given image and associated question. Despite the significant variety of questions and image content, and despite the variety of reasons for which the crowd can disagree, our learned model is able to produce quite accurate results.
We observe our Random Forest classifier outperforms our deep learning classifier; e.g., Ours: RF yields a three percentage point improvement with respect to AP while consistently yielding improved precision-recall values over Ours: LSTM-CNN (Figure FIGREF8 a). In general, deep learning systems hold promise to replace handcrafted features to pick out the discriminative features. Our baselines highlight a possible value in developing a different deep learning architecture for the problem of learning answer disagreement than applied for predicting answers to visual questions.
We show examples of prediction results where our top-performing RF classifier makes its most confident predictions (Figure FIGREF8 b). In these examples, the predictor expects human agreement for “what room... ?" visual questions and disagreement for “why... ?" visual questions. These examples highlight that the classifier may have a strong language prior towards making predictions, as we will discuss in the next section.
We now explore what makes a visual question lead to crowd answer agreement versus disagreement. We examine the influence of whether visual questions lead to the three types of answers (“yes/no", “number", “other") for both our random forest (RF) and deep learning (DL) classification systems. We enrich our analysis by examining the predictive performance of both classifiers when they are trained and tested exclusively with image and question features respectively. Figure FIGREF9 shows precision-recall curves for both classification systems with question features alone (Q), image features alone (I), and both question and image features together (Q+I).
When comparing AP scores (Figure FIGREF9 ), we observe our Q+I predictors yield the greatest predictive performance for visual questions that lead to “other" answers, followed by “number" answers, and finally “yes/no" answers. One possible reason for this finding is that the question wording strongly drives whether a crowd will disagree for “other" visual questions, whereas some notion of common sense may be required to learn whether a crowd will agree for “yes/no" visual questions (e.g., Figure FIGREF7 a vs Figure FIGREF7 g).
We observe that question-based features yield greater predictive performance than image-based features for all visual questions, when comparing AP scores for Q and I classification results (Figure FIGREF9 ). In fact, image features contribute to performance improvements only for our random forest classifier for visual questions that lead to “number" answers, as illustrated by comparing AP scores for Our RF: Q+I and Our RF: Q (Figure FIGREF9 b). Our overall finding that most of the predictive power stems from language-based features parallels feature analysis findings in the automated VQA literature BIBREF2 , BIBREF21 . This does not mean, however, that the image content is not predictive. Further work improving visual content cues for VQA agreement is warranted.
Our findings suggest that our Random Forest classifier's overall advantage over our deep learning system arises because of counting questions, as indicated by higher AP scores (Figure FIGREF9 ). For example, the advantage of the initial higher precision (Figure FIGREF8 a; Ours: RF vs Ours: DL) is also observed for counting questions (Figure FIGREF9 b; Ours: RF - Q+I vs Ours: DL - Q+I). We hypothesize this advantage arises due to the strength of the Random Forest classifier in pairing the question prior (“How many?") with the image-based SOS features that indicates the number of objects in an image. Specifically, we expect “how many" to lead to agreement only for small counting problems.
section_name: Capturing Answer Diversity with Less Effort
paragraphs: We next present a novel resource allocation system for efficiently capturing the diversity of true answers for a batch of visual questions. Today's status quo is to either uniformly collect INLINEFORM0 answers for every visual question BIBREF2 or collect multiple answers where the number is determined by external crowdsourcing conditions BIBREF0 . Our system instead spends a human budget by predicting the number of answers to collect for each visual question based on whether multiple human answers are predicted to be redundant.
section_name: Answer Collection System
paragraphs: Suppose we have a budget INLINEFORM0 which we can allocate to collect extra answers for a subset of visual questions. Our system automatically decides to which visual questions to allocate the “extra" answers in order to maximize captured answer diversity for all visual questions.
The aim of our system is to accrue additional costs and delays from collecting extra answers only when extra responses will provide more information. Towards this aim, our system involves three steps to collect answers for all INLINEFORM0 visual questions (Figure FIGREF11 a). First, the system applies our top-performing random forest classifier to every visual question in the batch. Then, the system ranks the INLINEFORM1 visual questions based on predicted scores from the classifier, from visual questions most confidently predicted to lead to answer “agreement" from a crowd to those most confidently predicted to lead to answer “disagreement" from a crowd. Finally, the system solicits more ( INLINEFORM2 ) human answers for the INLINEFORM3 visual questions predicted to reflect the greatest likelihood for crowd disagreement and fewer ( INLINEFORM4 ) human answers for the remaining visual questions. More details below.
section_name: Analysis of Answer Collection System
paragraphs: We now describe our studies to assess the benefit of our allocation system to reduce human effort to capture the diversity of all answers to visual questions.
We evaluate the impact of actively allocating extra human effort to answer visual questions as a function of the available budget of human effort. Specifically, for a range of budget levels, we compute the total measured answer diversity (as defined below) resulting for the batch of visual questions. The goal is to capture a large amount of answer diversity with little human effort.
We conduct our studies on the 121,512 test visual questions about real images (i.e., Validation questions 2015 v1.0). For each visual question, we establish the set of true answers as all unique answers which are observed at least twice in the 10 crowdsourced answers per visual question. We require agreement by two workers to avoid the possibility that “careless/spam" answers are treated as ground truth.
We collect either the minimum of INLINEFORM0 answer per visual question or the maximum of INLINEFORM1 answers per visual question. Our number of answers roughly aligns with existing crowd-powered VQA systems, for example with VizWiz, “On average, participants received 3.3 (SD=1.8) answers for each question" BIBREF0 . Our maximum number of answers also supports the possibility of capturing the maximum of three unique, valid answers typically observed in practice (recall study above). While more elaborate schemes for distributing responses may be possible, we will show this approach already proves quite effective in our experiments. We simulate answer collection by randomly selecting answers from the 10 crowd answers per visual question.
We compare our approach to the following baselines:
 
As in the previous section, we leverage the output confidence score from the publicly-shared model BIBREF24 learned from a LSTM-CNN deep learning architecture to rank the order of priority for visual questions to receive redundancy.
 
The system randomly prioritizes which images receive redundancy. This predictor illustrates the best a user can achieve today with crowd-powered systems BIBREF0 , BIBREF5 or with current dataset collection methods BIBREF2 , BIBREF4 .
We quantify the total diversity of answers captured by a resource allocation system for a batch of visual questions INLINEFORM0 as follows: DISPLAYFORM0 
where INLINEFORM0 represents the set of all true answers for the INLINEFORM1 -th visual question, INLINEFORM2 represents the set of unique answers captured in the INLINEFORM3 answers collected for the INLINEFORM4 -th visual question, and INLINEFORM5 represents the set of unique answers captured in the INLINEFORM6 answers collected for the INLINEFORM7 -th visual question. Given no extra human budget, total diversity comes from the second term which indicates the diversity captured when only INLINEFORM8 answers are collected for every visual question. Given a maximum available extra human budget ( INLINEFORM9 ), total diversity comes from the first term which indicates the diversity captured when INLINEFORM10 answers are collected for every visual question. Given a partial extra human budget ( INLINEFORM11 ), the aim is to have perfect predictions such that the minimum number of answers ( INLINEFORM12 ) are allocated only for visual questions with one true answer so that all diverse answers are safely captured.
We measure diversity per visual question as the number of all true answers collected per visual question ( INLINEFORM0 ). Larger values reflect greater captured diversity. The motivation for this measure is to only give total credit to visual questions when all valid, unique human answers are collected.
Our system consistently offers significant gains over today's status quo approach (Figure FIGREF11 b). For example, our system accelerates the collection of 70% of the diversity by 21% over the Status Quo baseline. In addition, our system accelerates the collection of the 82% of diversity one would observe with VizWiz by 23% (i.e., average of 3.3 answers per visual question). In absolute terms, this means eliminating the collection of 92,180 answers with no loss to captured answer diversity. This translates to eliminating 19 40-hour work weeks and saving over $1800, assuming workers are paid $0.02 per answer and take 30 seconds to answer a visual question. Our approach fills an important gap in the crowdsourcing answer collection literature for targeting the allocation of extra answers only to visual questions where a diversity of answers is expected.
Figure FIGREF11 b also illustrates the advantage of our system over a related VQA algorithm BIBREF2 for our novel application of cost-sensitive answer collection from a crowd. As observed, relying on an algorithm's confidence in its answer offers a valuable indicator over today's status quo of passively budgeting. While we acknowledge this method is not intended for our task specifically, it still serves as an important baseline (as discussed above). We attribute the further performance gains of our prediction system to it directly predicting whether humans will disagree rather than predicting a property of a specific algorithm (e.g., confidence of the Antol et al. algorithm in its answer prediction).
section_name: Conclusions
paragraphs: We proposed a new problem of predicting whether different people would answer with the same response to the same visual question. Towards motivating the practical implications for this problem, we analyzed nearly half a million visual questions and demonstrated there is nearly a 50/50 split between visual questions that lead to answer agreement versus disagreement. We observed that crowd disagreement arose for various types of answers (yes/no, counting, other) for many different reasons. We next proposed a system that automatically predicts whether a visual question will lead to a single versus multiple answers from a crowd. Our method outperforms a strong existing VQA system limited to estimating system uncertainty rather than crowd disagreement. Finally, we demonstrated how to employ the prediction system to accelerate the collection of diverse answers from a crowd by typically at least 20% over today's status quo of fixed redundancy allocation.
section_name: Acknowledgments
paragraphs: The authors gratefully acknowledge funding from the Office of Naval Research (ONR YIP N00014-12-1-0754) and National Science Foundation (IIS-1065390). We thank Dinesh Jayaraman, Yu-Chuan Su, Suyog Jain, and Chao-Yeh Chen for their assistance with experiments.

Figures and tables:
caption: Fig. 1: Examples of visual questions and corresponding answers, when 10 different people are asked to answer the same question about an image (“visual question”). As observed, the crowd sometimes all agree on a single answer (top) and at other times offer different answers (bottom). Given a visual question, we aim to build a prediction system that automatically decides whether multiple people would give the same answer.
caption: TABLE I: Correlation between answer agreement and visual questions that elicit three different types of answers for VQAs on real images and abstract scenes. Shown for each answer type is the percentage of visual questions that lead to at most one disagreement (row 1), unanimous agreement (row 2), and exactly one disagreement (row 3) from 10 crowdsourced answers. On average, across all answer types for both datasets, the crowd agrees on the answer for nearly half (i.e., 53%) of all VQAs. Moreover, we observe crowd disagreement arises often for all three answer types, highlighting that the significance of crowd disagreement is applicable across various types of visual questions.
caption: Fig. 2: Summary of answer diversity outcomes showing how frequently different numbers of unique answers arise when asking ten crowd workers to answer a visual question for (a) 369,861 visual questions about real images and (a) 90,000 visual questions about abstract scenes. Results are shown based on different degrees of answer agreement required to make an answer valid: only one person has to offer the answer, at least two people must agree on the answer, and at least three people must agree on the answer. The visual questions most often elicit exactly one answer per visual question but also regularly elicit up to three different answers per visual question.
caption: Fig. 3: Illustration of visual questions that lead humans to (dis)agree on a single answer. As observed, (a) unanimous answer agreement arises when images are simple, questions are precise, and questions are visually grounded. (b-i) Answer disagreement arises for a variety of reasons: (b) expert skill needed, (c) human mistakes, (d) ambiguous question, (e) ambiguous visual content, (f) insufficient visual evidence, (g) subjective question, (h) answer synonyms, and (i) varying answer granularity.
caption: Fig. 4: (a) Precision-recall curves and average precision (AP) scores for all benchmarked systems. Our random forest (RF) and deep learning (DL) classifiers outperform a related automated VQA baseline, showing the importance in modeling human disagreement as opposed to system uncertainty. (b) Examples of prediction results from our top-performing RF classifier. Shown are the top three visual questions for the most confidently predicted instances that lead to answer disagreement and agreement along with the observed crowd answers. These examples illustrate a strong language prior for making predictions. (Best viewed on pdf.)
caption: Fig. 5: Precision-recall curves and average precision (AP) scores for our random forest (RF) and deep learning (DL) classifiers with different features (Question Only - Q; Image Only - I; Q +I) for visual questions that lead to (a-c) three answer types.
caption: Fig. 6: We propose a novel application of predicting the number of redundant answers to collect from the crowd per visual question to efficiently capture the diversity of all answers for all visual questions. (a) For a batch of visual questions, our system first produces a relative ordering using the predicted confidence in whether a crowd would agree on an answer (upper half). Then, the system allocates a minimum number of annotations to all visual questions (bottom, left half) and then the extra available human budget to visual questions most confidently predicted to lead to crowd disagreement (bottom, right half). (b) For 121,512 visual questions, we show results for our system, a related VQA algorithm, and today’s status quo of random predictions. Boundary conditions are one answer (leftmost) and five answers (rightmost) for all visual questions. Our approach typically accelerates the capture of answer diversity by over 20% from today’s Status Quo selection; e.g., 21% for 70% of the answer diversity and 23% for 86% of the answer diversity. This translates to saving over 19 40-hour work weeks and $1800, assuming 30 seconds and $0.02 per answer.

Questions and answers:
question: What is the model architecture used?
answers: answer: unanswerable: no
free_form_answer: LSTM to encode the question, VGG16 to extract visual features. The outputs of LSTM and VGG16 are multiplied element-wise and sent to a softmax layer.
answer: unanswerable: no
extractive_spans: random forest
The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer.
question: How is the data used for training annotated?
answers: answer: unanswerable: no
free_form_answer: The number of redundant answers to collect from the crowd is predicted to efficiently capture the diversity of all answers from all visual questions.