Question: How do they assign answer (dis)agreement labels?
Evidence: We pose the prediction task as a binary classification problem. Specifically, given an image and associated question, a system outputs a binary label indicating whether a crowd will agree on the same answer. Our goal is to design a system that can detect which visual questions to assign a disagreement label, regardless of the disagreement cause (e.g., subjectivity, ambiguity, difficulty). We implement both random forest and deep learning classifiers.
A visual question is assigned either an answer agreement or disagreement label. To assign labels, we employ 10 crowdsourced answers for each visual question. A visual question is assigned an answer agreement label when there is an exact string match for 9 of the 10 crowdsourced answers (after answer pre-preprocessing, as discussed in the previous section) and an answer disagreement label otherwise. Our rationale is to permit the possibility of up to one “careless/spam" answer per visual question. The outcome of our labeling scheme is that a disagreement label is agnostic to the specific cause of disagreement and rather represents the many causes (described above).
For our first system, we use domain knowledge to guide the learning process. We compile a set of features that we hypothesize inform whether a crowd will arrive at an undisputed, single answer. Then we apply a machine learning tool to reveal the significance of each feature. We propose features based on the observation that answer agreement often arises when 1) a lay person's attention can be easily concentrated to a single, undisputed region in an image and 2) a lay person would find the requested task easy to address.
We employ five image-based features coming from the salient object subitizing BIBREF22 (SOS) method, which produces five probabilities that indicate whether an image contains 0, 1, 2, 3, or 4+ salient objects. Intuitively, the number of salient objects shows how many regions in an image are competing for an observer's attention, and so may correlate with the ease in identifying a region of interest. Moreover, we hypothesize this feature will capture our observation from the previous study that counting problems typically leads to disagreement for images showing many objects, and agreement otherwise.
We employ a 2,492-dimensional feature vector to represent the question-based features. One feature is the number of words in the question. Intuitively, a longer question offers more information and we hypothesize additional information makes a question more precise. The remaining features come from two one-hot vectors describing each of the first two words in the question. Each one-hot vector is created using the learned vocabularies that define all possible words at the first and second word location of a question respectively (using training data, as described in the next section). Intuitively, early words in a question inform the type of answers that might be possible and, in turn, possible reasons/frequency for answer disagreement. For example, we expect “why is" to regularly elicit many opinions and so disagreement. This intuition about the beginning words of a question is also supported by our analysis in the previous section which shows that different answer types yield different biases of eliciting answer agreement versus disagreement.
We leverage a random forest classification model BIBREF23 to predict an answer (dis)agreement label for a given visual question. This model consists of an ensemble of decision tree classifiers. We train the system to learn the unique weighted combinations of the aforementioned 2,497 features that each decision tree applies to make a prediction. At test time, given a novel visual question, the trained system converts a 2,497 feature descriptor of the visual question into a final prediction that reflects the majority vote prediction from the ensemble of decision trees. The system returns the final prediction along with a probability indicating the system's confidence in that prediction. We employ the Matlab implementation of random forests, using 25 trees and the default parameters.
We next adapt a VQA deep learning architecture BIBREF24 to learn the predictive combination of visual and textual features. The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer.
We train the system to predict (dis)agreement labels with training examples, where each example includes an image and question. At test time, given a novel visual question, the system outputs an unnormalized log probability indicating its belief in both the agreement and disagreement label. For our system's prediction, we convert the belief in the disagreement label into a normalized probability. Consequently, predicted values range from 0 to 1 with lower values reflecting greater likelihood for crowd agreement.
Answer (Dis)Agreement Labels: A visual question is assigned either an answer agreement or disagreement label. To assign labels, we employ 10 crowdsourced answers for each visual question. A visual question is assigned an answer agreement label when there is an exact string match for 9 of the 10 crowdsourced answers (after answer pre-preprocessing, as discussed in the previous section) and an answer disagreement label otherwise.
Answer: When there is an exact string match for 9 of the 10 crowdsourced answers, the visual question is assigned an answer agreement label, or it is assigned an answer disagreement label.