Venue: ACL 2020

# Benefits of Intermediate Annotations in Reading Comprehension

Dheeru Dua

University of California,

Irvine, CA, USA ddua@uci.edu &Sameer Singh

University of California,

Irvine, CA, USA sameer@uci.edu &Matt Gardner

Allen Institute for Artificial Intelligence, Irvine, CA, USA mattg@allenai.org

###### Abstract

Complex, compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer. A large combinatorial space of possible decision paths that result in the same answer, compounded by the lack of intermediate supervision to help choose the right path, makes the learning particularly hard for this task. In this work, we study the benefits of collecting intermediate reasoning supervision along with the answer during data collection. We find that these intermediate annotations can provide two-fold benefits. First, we observe that for any collection budget, spending a fraction of it on intermediate annotations results in improved model performance, for two complex compositional datasets: DROP and Quoref. Second, these annotations encourage the model to learn the correct latent reasoning steps, helping combat some of the biases introduced during the data collection process.

+
Footnote †: _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5627–5634

+
Footnote †: _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5627–5634

## 1 Introduction

Recently many reading comprehension datasets requiring complex and compositional reasoning over text have been introduced, including HotpotQA Yang et al. (2018), DROP Dua et al. (2019), Quoref Dasigi et al. (2019), and ROPES Lin et al. (2019). However, models trained on these datasets Hu et al. (2019); Andor et al. (2019) only have the final answer as supervision, leaving the model guessing at the correct latent reasoning. Figure 1 shows an example from DROP, which requires first locating various operands (i.e. relevant spans) in the text and then performing filter and count operations over them to get the final answer "3". However, the correct answer can also be obtained by extracting the span "3" from the passage, or by adding or subtracting various numbers in the passage. The lack of intermediate hints makes learning challenging and can lead the model to rely on data biases, limiting its ability to perform complex reasoning.

In this paper, we present three main contributions. First, we show that annotating relevant context spans, given a question, can provide an easy and low-cost way to learn better latent reasoning. To be precise, we show that under low budget constraints, collecting these annotations for up to 10% of the training data (2-5% of the total budget) can improve the performance by 4-5% in F1. We supervise the current state-of-the-art models for DROP and Quorerf, by jointly predict the relevant spans and the final answer. Even though these models were not designed with these annotations in mind, we show that they can still be successfully used to improve model performance. Models that explicitly incorporate these annotations might see greater benefits. Our results suggest that future dataset collection efforts should set aside a fraction of budget for intermediate annotations, particularly as the reasoning required becomes more complex.

Figure 1: Example from DROP, showing the intermediate annotations that we collected via crowd-sourcing.



Second, these annotations can help combat biases that are often introduced while collecting data Gururangan et al. (2018); Geva et al. (2019). This can take the form of label bias--in DROP, 18% of questions have answers 1, 2, or 3--or annotator bias, where a small group of crowd workers creates a large dataset with common patterns. By providing intermediate reasoning steps explicitly, the annotations we collect help the model overcome some of these biases in the training data.

Finally, the intermediate annotations collected in this work, including 8,500 annotations for DROP and 2,000 annotations for Quoref, will be useful for training further models on these tasks. We have made them available at [https://github.com/dbua/Intermediate_Annotations](https://github.com/dbua/Intermediate_Annotations).

## 2 Intermediate Annotations

Intermediate annotations describe the right set of context spans that should be aggregated to answer a question. We demonstrate their impact on two datasets: DROP and Quoref. DROP often requires aggregating information from various events in the context (Figure 1). It can be challenging to identify the right set of events directly from an answer when the same answer can be derived from many possible event combinations. We annotate the entire event span including all the attributes associated with the specific event. Quoref requires understanding long chains of coreferential reasoning, as shown in Figure 2, which are often hard to disentangle, especially when the context refers to multiple entities. We specifically annotate the coreference chains which lead to the entity being queried.

Collection process:We used Amazon Mechanical Turk to crowd-source the data collection. We randomly sample 8,500 and 2,000 QA pairs from the training set for DROP and Quorerf respectively. We showed a QA pair and its context to the workers and asked them to highlight "essential spans" in the context. In case of DROP, crowd workers were asked to highlight complete events with all their corresponding arguments in each span. For Quoref, they were asked to highlight the coreference chains associated with the answer entity in the context.

Cost of gathering intermediate annotations:Each HIT, containing ten questions, paid $1, and took approximately five minutes to complete. Overall, we spent $850 to collect 8,500 annotations for DROP and $200 to collect 2,000 annotations for Quoref. If these annotations are collected simultaneously with dataset creation, it may be feasible to collect them at a lower cost, as the time taken to read the context again will be avoided.

## 3 Experiments and Results

In this section, we train multiple models for the DROP and Quoref datasets, and evaluate the benefits of intermediate annotations as compared to traditional QA pairs. In particular, we will focus on the cost vs benefit tradeoff of intermediate annotations, along with evaluating their ability to mitigate bias in the training data.

### Setup

We study the impact of annotations on DROP on two models at the top of the leaderboard: NABERT1 and MTMSN Hu et al. (2019). Both the models employ a similar arithmetic block introduced in the baseline model Dua et al. (2019) on top of contextual representations from BERT Devlin et al. (2019). For Quoref, we use the baseline XLNet Yang et al. (2019) model released with the dataset. We supervise these models with the annotations in a simple way, by jointly predicting intermediate annotation and the final answer. We add two auxiliary loss terms to the marginal log-likelihood loss function. The first is a cross-entropy loss between the gold annotations (\(g\)) and predicted annotations, which are obtained by passing the final BERT representations through a linear layer to get a score per token \(p\), then normalizing each token's score of being selected as an annotation

Figure 2: Example collected annotation from Quoref, showing the intermediate steps.

 

with a sigmoid function.

\[{\cal L}_{1}(\theta)=\alpha_{1}CE(g,\sigma(p)) \tag{1}\]

The second is an \(L_{1}\) loss on the sum of predicted annotations, encouraging the model to only select a subset of the passage.

\[{\cal L}_{2}(\theta)=\alpha_{2}\sum^{|tokens|}_{\ell=0}\sigma(p_{l})\]

The hyper-parameters \(\alpha_{1}\) and \(\alpha_{2}\) were used to balance the scale of both auxiliary loss terms with the marginal log-likelihood.

### Cost vs Benefit

To evaluate the cost-benefit trade-off, we fix the total collection budget and then vary the percentage of budget that should go into collecting intermediate annotations. As shown in Figure 3, the model achieves better performance (+1.7% F1) when spending $7k where 2% budget is used for collecting intermediate reasoning annotations as compared to model performance when spending $10k for collecting only QA pairs. Overall, from Figure 3 we can see that allocating even 1% of the budget to intermediate annotations provides performance gains. However, we observe that allocating a large percentage of the budget to intermediate annotations at the expense of QA pairs reduces performance. In our experiments, we find that the sweet-spot percentage of the budget and training-set that should be allocated to intermediate annotations is 2% and \(\sim\)10% respectively.

### Bias Evaluation

Unanticipated biases Min et al. (2019); Manjunatha et al. (2019) are often introduced during dataset collection due to many reasons (eg., domain-specific contexts, crowd-workers distributions, etc.). These "dataset artifacts" can be picked up by the model to achieve better performance without learning the right way to reason. We explore two examples of such dataset artifacts in DROP and Quoref.

In DROP, around 40% of the passages are from NFL game summaries. The frequency of counting and arithmetic questions from this portion of the data resulted in the answers 1, 2, and 3 making up 18% of the entire training set. To study the effect of biased answer distribution on model performance, we sample 10k QA pairs with answers \(\in\) [0,9] from the training set _randomly_ as a biased training set. We also sample QA pairs from the validation set _uniformly_ for each answer \(\in\) [0,9] thus ensuring that each answer has equal representation in the unbiased validation set.

In Quoref, we found that around 65% of the answers are entity names present in the first sentence of the context. Similar to DROP, we create a biased training set with 5k QA pairs from the original training data, and an unbiased validation set with equal representation of answers from the first sentence and the rest of the context.

We investigate the effects of spending a small additional budget, either by adding more QA pairs (from the biased data distribution) or by collecting intermediate annotations, on this bias. We use two metrics to measure the extent to which bias has been mitigated. The first is the original metric for the task, i.e. \(F_{1}\), that measures how accurate the model is on the unbiased evaluation. Further, we also want to evaluate the extent to which the errors made by the model are unbiased; in other words, how much is the error _diffused_ over all possible answers, rather than only over the biased labels. We compute _confusion loss_Machart and Ralaivola (2012) as the metric for this, which measures error diffusion by computing the highest singular value of the unnormalized confusion matrix after setting the diagonal elements (i.e. true positives), to zero Koco and Capponi (2013) (lower confusion loss implies more diffusion). In an ideal scenario, all labels should have an equally likely probability of being a mis-prediction. Higher confusion loss implies that if we consider mis-classifications of a model we see that it has a tendency of overpredicting a specific label, making it biased towards that specific class.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{2}{c}{**Baseline**} & \multicolumn{2}{c}{**More QA pairs**} & \multicolumn{2}{c}{**Annotations**} \\ \cline{2-7}  & F1 (\%) & Conf. & F1 (\%) & Conf. & F1 (\%) & Conf. \\  & (\%) & loss (\%) & (\%) & loss & (\%) & loss \\ \hline DROP & 24.6 & 101.5 & 25.5 & 107.5 & 28.1 & 94.5 \\ Quorerf & 61.8 & 103.0 & 62.7 & 109.0 & 64.3 & 97.0 \\ \hline \hline \end{tabular}
\end{table}
Table 1: F1 performance and confusion loss (lower is better) of models in three settings: baseline with 10k(DROP) and 5k(Quoref) QA pairs, additional QA pairs worth $250 and $100 for DROP and Quoref respectively, and additional annotations worth $250 and $100 for DROP and Quoref respectively. To put confusion loss in perspective, the _best_ confusion loss, i.e. perfect diffusion, is \(90.1\) for DROP and \(87.0\) for Quoref.

 

Table 1 shows that along with higher improvements in on providing annotations as compared to more QA pairs, we also see a reduction in the confusion loss with annotations indicating bias mitigation.

Further, we also find that for DROP, the false positive rate for top-3 common labels fell down from 47.7% (baseline) to 39.6% (with annotations), while the false positive rate for the bottom-7 increased from 30.4%(baseline) to 36.3%(with annotations), further demonstrating mitigation of bias. The confusion matrices are included in Appendix.

### Qualitative Result

Figure 4 shows a DROP example where the model trained without annotations is not able to determine the right set of events being queried, returning an incorrect response. The model trained with annotations can understand the semantics behind the query terms "first half" and "Cowboys", to arrive at the correct answer. The curves depicting quanti

Figure 4: Predicted relevant spans for question answered correctly with annotation (prediction: “3”) and incorrectly without annotations (prediction: “2”) by MTMSN model trained on DROP

Figure 3: Performance of model for varying percentage of budget invested in collecting intermediate annotation. The calculation were done with cost as §0.4 and §0.7 for a QA pair in DROP and Quoref respectively.

 

tative performance gains with varying amounts of annotations and QA pairs are in the appendix.

## 4 Related Work

Similar to our work, zaidan2007multi studied the impact of providing explicit supervision via _rationales_, rather than generating them, for varying fractions of training set in text classification. However, we study the benefits of such supervision for complex compositional reading comprehension datasets. In the field of computer vision, donahue2011learning collected similar annotations, for visual recognition, where crowd-workers highlighted relevant regions in images.

Within reading comprehension, various works like HotpotQA Yang et al. (2018) and CoQA Reddy et al. (2019) have collected similar reasoning steps for entire dataset. Our work shows that collecting intermediate annotations for a fraction of dataset is cost-effective and helps alleviate dataset collection biases to a degree. Another line of work Ning et al. (2019) explores the cost vs. benefit of collecting full vs. partial annotations for various structured predictions tasks. However, they do not focus on intermediate reasoning required to learn the task.

Our auxiliary training with intermediate annotations is inspired by extensive related work on training models using _side information_ or _domain knowledge_ beyond labels Mann and McCallum (2008); Chang et al. (2007); Ganchev et al. (2010); Rockatschel et al. (2015). Especially relevant is work on supervising models using explanations Ross et al. (2017), which, similar to our annotations, identify parts of the input that are important for prediction Lei et al. (2016); Ribeiro et al. (2016).

## 5 Conclusion

We show that intermediate annotations are a cost-effective way to not only boost model performance but also alleviate certain unanticipated biases introduced during the dataset collection. However, it may be unnecessary to collect these for entire dataset and there is a sweet-spot that works best depending on the task. We proposed a simple semi-supervision technique to expose the model to these annotations. We believe that in future they can be used more directly to yield better performance gains. We have also released these annotations for the research community at [https://github.com/dbua/Intermediate_Annotations](https://github.com/dbua/Intermediate_Annotations).

## Acknowledgements

This work was supported in part by Allen Institute of Artificial Intelligence, in part by Amazon, and in part by the National Science Foundation (NSF) grant #CNS-1730158.

## References

* D. Andor, L. He, K. Lee, and E. Pittler (2019)Giving bert a calculator: finding operations and arguments with reading comprehension. Annual Meeting of the Association for Computational Linguistics (ACL). Cited by: SS1.
* M. Chang, L. Ratinov, and D. Roth (2007)Guiding semi-supervision with constraint-driven learning. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 280-287. External Links: Document Cited by: SS1.
* P. Dasigi, N. F. Liu, A. Marasovic, N. A. Smith, and M. Gardner (2019)Quoref: a reading comprehension dataset with questions requiring coreferential reasoning. Annual Meeting of the Association for Computational Linguistics (ACL). Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)Bert: pre-training of deep bidirectional transformers for language understanding.. NAACL. Cited by: SS1.
* J. Donahue and K. Grauman (2011)Annotator rationales for visual recognition. In 2011 International Conference on Computer Vision, pp. 1395-1402. Cited by: SS1.
* D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner (2019)Drop: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, Cited by: SS1.
* K. Ganchev, J. Graca, J. Gillenwater, and B. Taskar (2010)Posterior regularization for structured latent variable models. Journal of Machine Learning Research (JMLR). Cited by: SS1.
* M. Geva, Y. Goldberg, and J. Berant (2019)Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. Annual Meeting of the Association for Computational Linguistics (ACL). Cited by: SS1.
* S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith (2018)Annotation artifacts in natural language inference data. In NAACL, Cited by: SS1.
* M. Hu, Y. Peng, Z. Huang, and D. Li (2019)A multi-type multi-span network for reading comprehension that requires discrete reasoning. Annual Meeting of the Association for Computational Linguistics (ACL) 

Sokol Koco and Cecile Capponi. 2013. On multi-class classification through the minimization of the confusion matrix norm. In _Asian Conference on Machine Learning_, pages 277-292.
* Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. _Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Lin et al. (2019) Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. _MRQA Workshop_.
* Machart and Ralaivola (2012) Pierre Machart and Liva Ralaivola. 2012. Confusion matrix stability bounds for multiclass classification. _arXiv preprint arXiv:1202.6221_.
* Manjunatha et al. (2019) Varun Manjunatha, Nirat Saini, and Larry S Davis. 2019. Explicit bias discovery in visual question answering models. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 9562-9571.
* Mann and McCallum (2008) Gideon S. Mann and Andrew McCallum. 2008. Generalized expectation criteria for semi-supervised learning of conditional random fields. In _Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 870-878.
* Min et al. (2019) Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishrizi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. _Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Ning et al. (2019) Qiang Ning, Hangfeng He, Chuchu Fan, and Dan Roth. 2019. Partial or complete, that's the question. _Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. _Transactions of the Association for Computational Linguistics_, 7:249-266.
* Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1135-1144. ACM.
* Rocktaschel et al. (2015) Tim Rocktaschel, Sameer Singh, and Sebastian Riedel. 2015. Injecting logical background knowledge into embeddings for relation extraction. In _Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
* Ross et al. (2017) Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. _IJCAI_.
* Yang et al. (2019) Zhilin Yang, Zhang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. _NeurIPS_.
* Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. _Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Zaidan et al. (2007) Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using annotator rationales to improve machine learning for text categorization. In _Human Language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference_, pages 260-267.

 

Figure 5: Performance of model trained on varying amount of annotations used in training

Figure 6: For the same cost intermediate annotations helps diffuse biased over-representation of number 3 as compared to adding more question-answer pairs



* [25] M. C. Bader, J. D. Kiefer, and R. G. Hinton. Learning to learn a model for the classification of natural images. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 1-40, 2016.
* [26] A. L. Balcan, S. P. Salakhutdinov, and V. Savona. Deep learning with deep convolutional neural networks. In _Advances in Neural Information Processing Systems_, volume 389 of _Lecture Notes in Computer Science_, pages 77-96, 2015.

[MISSING_PAGE_POST]