Venue: NAACL 2021

# Identifying inherent disagreement in natural language inference

Xinliang Frederick Zhang

Department of Computer Science and Engineering

The Ohio State University

zhang.9975@osu.edu Marie-Catherine de Marneffe

Department of Linguistics

The Ohio State University

demarneffe.l@osu.edu

###### Abstract

Natural language inference (NLI) is the task of determining whether a piece of text is entailed, contradicted by or unrelated to another piece of text. In this paper, we investigate how to tease systematic inferences (i.e., items for which people agree on the NLI label) apart from disagreement items (i.e., items which lead to different annotations), which most prior work has overlooked. To distinguish systematic inferences from disagreement items, we propose Artificial Annotators (AAs) to simulate the uncertainty in the annotation process by capturing the modes in annotations. Results on the CommitmentBank, a corpus of naturally occurring discourses in English, confirm that our approach performs statistically significantly better than all baselines. We further show that AAs learn linguistic patterns and context-dependent reasoning.

## 1 Introduction

Learning to effectively understand unstructured text is integral to Natural Language Understanding (NLU), covering a wide range of tasks such as question answering, semantic textual similarity and sentiment analysis. Natural language inference (NLI), an increasingly important benchmark task for NLU research, is the task of determining whether a piece of text is entailed, contradicted by or unrelated to another piece of text (i.a., Dagan et al., 2005; MacCartney and Manning, 2009).

Pavlick and Kwiatkowski (2019) observed inherent disagreements among annotators in several NLI datasets, which cannot be smoothed out by hiring more people. They pointed out that to achieve robust NLU, we need to be able to tease apart systematic inferences (i.e., items for which most people agree on the annotations) from items inherently leading to disagreement. The last example in Table 1, from the CommitmentBank (de Marneffe et al., 2019), is a typical disagreement item: some annotators consider it to be an entailment (3 or 2), while others view it as a contradiction (-3). A common practice to generate an inference label from annotations is to take the average (i.a., Pavlick and Callison-Burch, 2016). In this case, the average of the annotations is 0.25 and the gold label for this item would thus be "Neutral", but such label is not accurately capturing the annotation distribution. Alternatively, some work simply ignores items on which annotators disagree and only studies systematic inference items (Jiang and de Marneffe, 2019, 2019).

Here, we aim at teasing apart systematic inferences from inherent disagreements. In line with what Kenyon-Dean et al. (2018) suggested for sentiment analysis, we propose a finer-grained labeling

\begin{table}
\begin{tabular}{p{42.7pt} p{142.3pt}}  & **Xinliang Frederick Zhang** \\ Department of Computer Science and Engineering & **Marie-Catherine de Marneffe** \\ The Ohio State University & **Marie-Catherine de Marneffe** \\ zhang.9975@osu.edu & Department of Linguistics \\ \end{tabular}
\end{table}
Table 1: Examples from CommitmentBank, with finer-grained NLI labels. The labels in parentheses come from Jiang and de Marneffe (2019). Scores in brackets are the raw human annotations.



for NLI: teasing disagreement items, labeled "Disagreement", from systematic inferences, which can be "Contradiction", "Neutral" or "Entailment". To this end, we propose Artificial Annotators (AAs), an ensemble of BERT models Devlin et al. (2019), which simulate the uncertainty in the annotation process by capturing modes in annotations. That is, we expect to utilize simulated modes of annotations to enhance finer-grained NLI label prediction.

Our results, on the CommitmentBank, show that AAs perform statistically significantly better than all baselines (including BERT baselines) by a large margin in terms of both F1 and accuracy. We also show that AAs manage to learn linguistic patterns and context-dependent reasoning.

## 2 Data: The CommitmentBank

The CommitmentBank (CB) is a corpus of 1,200 naturally occurring discourses originally collected from news articles, fiction and dialogues. Each discourse consists of up to 2 prior context sentences and 1 target sentence with a clause-embedding predicate under 4 embedding environments (negation, modal, question or antecedent of conditional). Annotators judged the extent to which the speaker/author of the sentences is committed to the truth of the content of the embedded clause (CC), responding on a Likert scale from +3 to -3, labeled at 3 points (+3/speaker is certain the CC is true, 0/speaker is not certain whether the CC is true or false, -3/speaker is certain the CC is false). Following Jiang and de Marneffe (2019b), we recast CB by taking the context and target as the premise and the embedded clause in the target as the hypothesis.

Common NLI benchmark datasets are SNLI Bowman et al. (2015) and MultiNLI Williams et al. (2018), but these datasets have only one annotation per item in the training set. CB has at least 8 annotations per item, which permits to identify items on which annotators disagree. Jiang and de Marneffe (2019b) discarded items if less than 80% of the annotations are within one of the following three ranges: [1,3] Entailment, 0 Neutral, [-3,-1] Contradiction. The gold label for example 3 in Table 1 would thus be "Disagreement". However, this seems a bit too stringent, given that 70% of the annotators all agree on the 0 label and there is only one annotation towards the extreme. Likewise, for example 5, most annotators chose a negative score and the item might therefore be better labeled as "Contradiction" rather than "Disagreement". To decide on the **finer-grained NLI labels**, we therefore also took variance and mean into account, as follows:1

Footnote 1: Compared with the labeling scheme in Jiang and de Marneffe (2019b), our labeling scheme results in 59 fewer Disagreement items, 48 of which are labeled as Neutral.

* **Entailment:** 80% of annotations fall in the range [1,3] OR the annotation variance \(\leq\) 1 and the annotation mean \(>\) 1.
* **Neutral:** 80% of annotations is 0 OR the annotation variance \(\leq\) 1 and the absolute mean of annotations is bound within 0.5.
* **Contradiction:** 80% of annotations fall in the range [-3, -1] OR the annotation variance \(\leq\) 1 and the annotation mean \(<\) -1.
* **Disagreement:** Items which do not fall in any of the three categories above.

We randomly split CB into train/dev/test sets in a 7:1:2 ratio.2 Table 2 gives splits' basic statistics.

Footnote 2: We don’t follow the SuperGLUE splits Wang et al. (2019) as they do not include disagreement items. The data splits and codes are available at [https://github.com/FrederickXZhang/fgNLI](https://github.com/FrederickXZhang/fgNLI).

## 3 Model: Artificial Annotators

We aim at finding an effective way to tease items leading to systematic inferences apart from items leading to disagreement. As pointed out by Calma and Sick (2017), annotated labels are subject to uncertainty. Annotations are indeed influenced by several factors: workers' past experience and concentration level, cognition complexities of items, etc. They proposed to simulate the annotation process in an active learning paradigm to make use of the annotations that contribute to uncertainty. Likewise, for NLI, Gantt et al. (2020) observed that directly training on raw annotations using annotator

Figure 1: Artificial Annotators setup.

 

identifier improves performance. Essentially, Gantt et al. (2020) used a mixed-effect model to learn a mapping from an item and the associated annotator identifier to a NLI label. However, annotator identifiers are not always accessible, especially in many datasets that have been there for a while. Thus, we decide to simulate the annotation process instead of learning from real identifiers.

As shown by Pavlick and Kwiatkowski (2019), if annotations of an item follow unimodal distributions, then it is suitable to use aggregation (i.e., take an average) to obtain a inference label; but such an aggregation is not appropriate when annotations follow multi-modal distributions. Without loss of generality, we assume that items are associated with n-modal distributions, where n \(\geq\) 1. Usually, systematic inference items are tied to unimodal annotations while disagreement items are tied to multi-modal annotations. We, thus, introduce the notion of Artificial Annotators (AAs), where each individual "annotator" learns to model one mode.

### Architecture

AAs is an ensemble of \(n\) BERT models (Devlin et al., 2019) with a primary goal of finer-grained NLI label prediction. \(n\) is determined to be 3 as there are up to 3 relationships between premise and hypothesis, excluding the disagreement class. Within AAs, each BERT is trained for an auxiliary systematic inference task which is to predict entailment/neutral/contradiction based on a respective subset of annotations. The subsets of annotations for the three BERT are mutually exclusive.

A high-level overview of AAs is shown in Figure 1. Intuitively, each BERT separately predicts a systematic inference label, each of which represents a mode3 of the annotations. The representations of these three labels are further aggregated as augmented information to enhance final fine-grained NLI label prediction (see Eq. 1).

Footnote 3: It’s possible that three modes collapse to (almost) a point.

If we view the AAs as a committee of three members, our architecture is reminiscent of the Query by Committee (QBC) (Seung et al., 1992), an effective approach for active learning paradigm. The essence of QBC is to select unlabeled data for labeling on which disagreement among committee members (i.e., learners pre-trained on the same labeled data) occurs. The selected data will be labeled by an oracle (e.g., domain experts) and then used to further train the learners. Likewise, in our approach, each AA votes for an item independently. However, the purpose is to detect disagreements instead of using disagreements as a measure to select items for further annotations. Moreover, in our AAs, the three members are trained on three disjoint annotation partitions for each item (see Section 3.2).

### Training

We first sort the annotations in descending order for each item and divide them into three partitions.4 For each partition, we generate an auxiliary label derived from the annotation mean. If the mean is greater/smaller than +0.5/-0.5, then it's entailment/contradiction; otherwise, it's neutral. The first BERT model is always enforced to predict the auxiliary label of the first partition to simulate an entailment-biased annotator. Likewise, the second and third BERT models are trained to simulate neutral-biased and contradiction-biased annotators.

Footnote 4: For example, if there are 8 annotations for a given item, the annotations are divided into partitions of size 3, 2 and 3.

Each BERT produces a pooled representation for the [CLS] token. The three representations are passed through a multi-layer perceptron (MLP) to obtain the finer-grained NLI label:

\[P(y|{\bf x})={\rm softmax}({\bf W_{s}}\tanh({\bf W_{t}}[{\bf e};{\bf n};{\bf c }])) \tag{1}\]

\begin{table}
\begin{tabular}{l r r r r r r r r} \hline \hline  & \multicolumn{2}{c}{Dev} & \multicolumn{6}{c}{Test} \\ \cline{2-10}  & Acc. & F1 & Acc. & F1 & Entail & Neutral & Contradict & Disagree \\ \hline Always 0 & 55.00 & 39.03 & 45.42 & 28.37 & 0.00 & 0.00 & 0.00 & 62.46 \\ CBOW & 55.25 & 40.54 & 45.09 & 28.37 & 0.00 & 0.00 & 0.69 & 62.17 \\ Heuristic & 65.00 & 62.08 & 54.17 & 50.60 & 22.54 & **52.94** & 64.46 & 58.20 \\ Vanilla BERT & 63.71 & 63.54 & 62.50 & 61.93 & 59.26 & 49.64 & 69.09 & 61.93 \\ Joint BERT & 64.47 & 64.28 & 62.61 & 62.07 & 59.77 & 47.27 & 67.36 & 63.21 \\ \hline AAs (ours) & **65.15** & **64.41** & **65.60*** & **64.97*** & **61.07** & 51.27 & **70.89** & **66.49*** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Baselines and AAs overall performance on CB dev and test sets, and F1 scores of each class on the test set (average of 10 runs). * indicates a statistically significant difference (t-test, p \(\leq\) 0.01).

 

with \([{\bf e};{\bf n};{\bf c}]\) being the concatenation of three learned representations out of entailment-biased, neutral-biased and contradiction-biased BERT models. \({\bf W_{s}}\) and \({\bf W_{t}}\) are parameters to be learned.

The overall loss is defined as the weighted sums of four cross-entropy losses:

\[loss=r*loss^{f}+\frac{1-r}{3}(loss^{e}+loss^{n}+loss^{c}) \tag{2}\]

where \(r\in[0,1]\) controls the primary finer-grained NLI label prediction task loss ratio.

## 4 Experiment

We include five baselines to compare with:

* **"Always 0"**: Always predict Disagreement.
* **CBOW** (Continuous Bags of Words): Each item is represented as the average of its tokens' GLOVE vectors Pennington et al. (2014).
* **Heuristic baseline**: Linguistics-driven rules (detailed in Appendix A), adapted from Jiang and de Marneffe (2019b); e.g., conditional environment discriminates for disagreement items.
* **Vanilla BERT**: Devlin et al. (2019) Straightforwardly predict among 4 finer-grained NLI labels.
* **Joint BERT**: Two BERT models are jointly trained, each of which has a different speciality. The first one (2-way) identifies whether a sentence pair is a disagreement item. If not, this item is fed into the second BERT (3-way) which carries out systematic inference.

For all baselines involving BERT, we follow the standard practice of concatenating the premise and the hypothesis with [SEP].

Table 3 gives the accuracy and F1 for each baseline and AAs, on the CB dev and test sets. We run each model 10 times, and report the average.

CBOW is essentially the same as the "Always 0" baseline as it keeps predicting Disagreement regardless of the input. The Heuristic baseline achieves competitive performance on the dev set, though it has a significantly worse result on the test set. Not surprisingly, both BERT-based baselines outperform the Heuristic on the test set: fine-tuning BERT often lead to better performance, including for NLI Peters et al. (2019); McCoy et al. (2019). These observations are consistent with Jiang and de Marneffe (2019b) who observed a similar trend, though only on systematic inferences. Our proposed AAs perform consistently better than all baselines, and statistically significantly better on the test set (t-test, p \(\leq\) 0.01). Also, AAs achieve a smaller standard deviation on the test set within the 10 runs, indicating that it is more stable and potentially more robust to wild environments.

## 5 Analysis

Table 3 also gives F1 for each class on the test set. AAs outperform all BERT-based models under all classes. However, compared with the Heuristic, AAs show an inferior result on "Neutral" items mainly due to the lack of "Neutral" training data. The first 4 examples in Table 4 show examples for which AAs make the correct prediction while other baselines might not. The confusion matrix in Table 5 shows that the majority (\(\sim\)60%) of errors come from wrongly predicting a systematic inference item as a disagreement item. In 91% of

\begin{table}
\begin{tabular}{p{78.3pt}} \hline  & \\ _Premise_: B: Yeah, it is. A: For instance, B: ’I’m a historian, and my father had kept them, I think, since nineteen twenty-seven ub, but he burned the ones from twenty-seven to f i-, A: My goodness. B: I could not believe he did that, \\ _Hypothesis_: his father turned the ones from twenty-seven \\ Heuristics: C: V: BERT: D J. BERT: E As: R [E, E, E] Gold: E [3, 3, 3, 3, 2, 2, -1] \\ \hline   2 _Premise_: ‘She was about to tell him that was his own stupid fault and that she wasn't here to wait on him –particularly since he had proved to be insophiophile. But she bat back the words. Perhaps if she made herself useful meht might decide she could stay “for a white at least just until she got something else sorted out. \\ _Hypothesis_: she could stay \\ Heuristics: D V. BERT: D J. BERT: O AAs: N [N, N, N] Gold: N [3, 0, 0, 0, 0, 0, 0, 0, 0] \\ \hline   3 _Premise_: A: but that is one of my solutions. Uh.. B: I know here in Dallas that they have just instinated in the last couple of years, uh, a real long period of time that you can desmetre vote before the elections. And I do not think they have seen a really high improvement. \\ _Hypothesis_: there have been actually high improvement. \\ Heuristics: C: V. BERT: C. J. BERT: C AAS: C [C, C, C] Gold: C [-1, -2, -2, -2, -2, -2, -2, -3, -3] \\ \hline   4 _Premise_: So old your commute everyday then or, A: No. B: Oh, okay. A: No. no, us was six hour drive: B: Oh, okay, when you said it wague a way away, I dth root know that meant you had to drive like an hour \\ _Hypothesis_: speaker A hand to drive like an hour \\ Heuristics: C. V. BERT: D. J. BERT: E As: D [E, C, C] Gold: D [3, 2, 2, 1, 0, 0, -1, -1, -3] \\ \hline   5 _Premise_: The assassis ” tone” and bearing were completely confident. If he noticed that Zakow was now edging further to the side widening the arc of fire the dd not appear to be troubled. \\ _Hypothesis_: Zikov was eding further to the side heuristics: V. BERT: D. J. BERT: D ASA: D [E, E, N] Gold: E [3, 3, 3, 3, 2, 2, 1, 1] \\ \hline   6 _Premise_: By: Yeah, and EDS is very particular about this, hair cuts, A: Now. B: I mean it was like you can’t have, you dono, such and such facial hair, no beards, you knows, and just really detailed. A: A: Idon’t know that that would be a good argument to work in \\ _Hypothesis_: that would be a good environment to work in \\ Heuristics: C: V. BERT: C. J. BERT: D ASA: C [C, C, C] Gold: D [2, 0, 0, 0, 0, -1, -2, -3] \\ \hline   7 _Premise_: “Wilky did mention it. I was puzzled, I ’ll admit, but now I understand.” How did you know Hather had been there? \\ _Hypothesis_: Heather had been there \\ Heuristics: N: V. BERT: E. J. BERT: E As: E [E, E, E] Gold: D [3, 3, 3, 2, 1, 1, 0, 0, 0] \\ \hline  & \\ \end{tabular}
\end{table}
Table 4: Models’ predictions for CB test items. Labels in { } are predictions by individual AAs. E: entailment, C: contradiction, N: neutral, D: disagreement.

 

such errors, AAs predict that there is more than one mode for the annotation (i.e., the three labels predicted by individual "annotators" in AAs are not unanimous), as in example 5 in Table 4. AAs are thus predicting more modes than necessary when the annotation is actually following a uni-modal distribution. On the contrary, when the item is supposed to be a disagreement item but is missed by AAs (as in example 6 and 7 in Table 4), AAs mistakenly predict that there is only one mode in the annotations 78% of the time. It thus seems that a method which captures accurately the number of modes in the annotation distribution would lead to a better model.

We also examine the model performance for different linguistic constructions to investigate whether the model learns some of the linguistic patterns present in the Heuristic baseline. The Heuristic rules are strongly tied to the embedding environments. Another construction used is one which can lead to "neg-raising" reading, where a negation in the matrix clause is interpreted as negating the content of the complement, as in example 3 (Table 4) where _I do not think they have seen a really high improvement_ is interpreted as _I think they did not see a really high improvement_. "Neg raising" readings often occur with _know_, _believe_ or _think_ in the first person under negation. There are 85 such items in the test set: 41 contradictions (thus neg-raising items), 39 disagreements and 5 entailments. Context determines whether a neg-raising inference is triggered An and White (2019).

Table 6 gives F1 scores for the Heuristic, BERT models and AAs for items under the different embedding environments and potential neg-raising items in the test set. Though AAs achieve the best overall results, it suffers under conditional and question environments, as the corresponding training data is scarce (9.04% and 14.17%, respectively). The Heuristic baseline always assigns contradiction to the "I don't know/believe/think" items, thus capturing all 41 neg-raising items but missing disagreements and entailments. BERT, a SOTA NLP model, is not great at capturing such items either: 71.64 F1 on contradiction vs. 52.84 on the others Vanilla BERT); 71.69 F1 vs. 56.16 Joint BERT). Our AAs capture neg-raising items better with 77.26 F1 vs. 59.38, showing an ability to carry out context-dependent inference on top of the learned linguistic patterns. Table 7, comparing performance on test items correctly predicted by the linguistic rules vs. items for which context-dependent reasoning is necessary, confirms this: AAs outperform the BERT baselines in both categories.

## 6 Conclusion

We introduced finer-grained natural language inference. This task aims at teasing systematic inferences from inherent disagreements, overlooked in prior work. We show that our proposed AAs, which simulate the uncertainty in annotation process by capturing the modes in annotations, perform statistically significantly better than all baselines. However the best performance obtained (\(\sim\)66%) is still far from achieving robust NLU, leaving room for improvement.

### Acknowledgments

We thank the anonymous reviewers for their valuable feedback. This material is based upon work supported by the National Science Foundation under Grant No. IIS-1845122.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & negation & modal & conditional & question & negR \\ \hline Heuristic & 51.29 & 48.02 & 37.69 & 44.64 & 54.16 \\ V. BERT & 60.91 & 73.98 & 44.84 & 53.02 & 61.91 \\ J. BERT & 60.94 & 73.95 & 46.02 & 51.68 & 63.67 \\ \hline AAs & **65.96** & **80.18** & **48.05** & **54.95** & **68.00** \\ \hline \hline \end{tabular}
\end{table}
Table 6: F1 for CB test set under the embedding environments and “I don’t know/believe/think” (“negR”).

\begin{table}
\begin{tabular}{l|c c c|c c} \hline \hline \multirow{2}{*}{Correct inference} & \multicolumn{2}{c}{Yes (130)} & \multicolumn{2}{c}{No (110)} \\ \cline{2-5}  & Acc. & F1 & Acc. & F1 \\ \hline V. BERT & 80.00 & 80.45 & 41.51 & 42.48 \\ J. BERT & 79.74 & 80.04 & 42.73 & 44.15 \\ AAs & **84.37** & **84.85** & **46.97** & **48.75** \\ \hline \hline \end{tabular}
\end{table}
Table 7: BERT-based models performance on test items correctly predicted by vs. items missed by linguistic rules. Numbers next to Yes/No denote the size.

\begin{table}
\begin{tabular}{l|r r r r|r} \hline \hline PredictGold & E & N & C & D & Total \\ \hline E & 37 & 2 & 0 & 13 & 52 \\ N & 1 & 10 & 0 & 3 & 14 \\ C & 0 & 0 & 34 & 13 & 47 \\ D & 20 & 7 & 20 & 80 & 127 \\ \hline Total & 58 & 19 & 54 & 109 & 240 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Confusion matrix for the test set. E: entailment, N: neutral, C: contradiction, D: disagreement.

 



## References

* A. Ar and S. White (2019)The lexical and grammatical sources of neg-raising inferences. In Proceedings of the Society for Computation in Linguistics (SCiL 2020), pp. 220-233. Cited by: SS4.
* S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pp. 632-642. Cited by: SS1.
* A. Calma and B. Sick (2017)Simulation of annotators for active learning: uncertain oracles. In Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017), pp. 49-58. Cited by: SS1.
* I. Dagan, O. Glickman, and B. Magnini (2005)The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, pp. 177-190. External Links: Link Cited by: SS1.
* M. de Marneffe, M. Simons, and J. Tonhauser (2019)The commitmentBank: investigating projection in naturally occurring discourse. In Simn und Bedeutung 23, Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pp. 4171-4186. Cited by: SS1.
* W. Gantt, B. Kane, and A. Steven White (2020)Natural language inference with mixed effects. In The Ninth Joint Conference on Lexical and Computational Semantics (*SEM 2020), Cited by: SS1.
* N. Jiang and M. de Marneffe (2019a)Do you know that Florence is packed with visitors? evaluating state-of-the-art models of speaker commitment. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pp. 4208-4213. Cited by: SS1.
* N. Jiang and M. de Marneffe (2019b)Evaluating BERT for natural language inference: a case study on the Commitmentbank. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP 2019, pp. 6085-6090. Cited by: SS1.
* K. Kenyon-Dean, E. Ahmed, S. Fujimoto, J. Georges-Filteau, C. Glasz, B. Kaur, A. Lalande, S. Bhanderi, R. Belfer, N. Kanagasabai, R. Sarrazingendron, R. Verma, and D. Ruths (2018)Sentiment analysis: it's complicated!. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, pp. 1886-1895. Cited by: SS1.
* D. P. Kingma and J. Ba (2015)Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, Cited by: SS1.
* B. MacCartney and C. D. Manning (2009)An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, IWCS 2009, pp. 140-156. Cited by: SS1.
* T. McCoy, E. Pavlick, and T. Linzen (2019)Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pp. 3428-3448. Cited by: SS1.
* E. Pavlick and C. Callison-Burch (2016)Most "babis" are "little" and most "problem" are "huge". Compositional entailment in adjective-noouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Cited by: SS1.
* E. Pavlick and T. Kwiatkowski (2019)Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, pp. 677-694. Cited by: SS1.
* J. Pennington, R. Socher, and C. D. Manning (2014)Glove: global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, pp. 1532-1543. Cited by: SS1.
* M. E. Peters, S. Ruder, and N. A. Smith (2019)To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019, pp. 7-14. Cited by: SS1.
* C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2019)Exploring the limits of transfer learning with a unified text-to-text transformer. CoRRabs/1910.10683. Cited by: SS1.
* H. Sebastian Seung, M. Opper, and H. Sompolinsky (1992)Query by committee. In Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT 1992, Pittsburgh, PA, USA, July 27-29, 1992, pp. 287-294. Cited by: SS1.
* A. Wang, Y. Pruksachatkun, N. Nagia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2019)SuperGLUE: a stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference

on Neural Information Processing Systems 2019, NeurIPS 2019_, pages 3261-3275.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018_, pages 1112-1122.

## Appendix A Linguistic Rules

Our linguistic rules are inspired by and adapted from Jiang and de Marneffe (2019b) to explicitly include the most discriminating expressions for disagreement items. We utilize three linguistic features which are provided in CB: entailment-canceling environment (negation, modal, question, antecedent of conditional), matrix verb and its subject person.

1. Items under conditional are disagreement.
2. Items under question and with second person are neutral.
3. Items under question and with non-second person are disagreement.
4. Items of the form "I don't know/think/believe" are contradiction (i.e., negRaising structure).
5. Items with factive verbs are entailment.
6. Items under negation and with non-factive verbs are disagreement.
7. Items under modal and with non-third person are entailment.

When this policy is executed, there are two additional auxiliary rules: Items not falling in any group above are assigned a disagreement label as it is the dominant class in CB; For items satisfying more than one rule, the label will be determined by the higher-ranked rule (i.e., a smaller number indicates a higher rank). Note that the rules above also reveal the most discriminating expressions for each class.

## Appendix B Impact of \(r\)

We experiment with three different \(r\) values, 0.25, 0.4, 0.7. Intuitively,

* 0.25: each module contributes equally;
* 0.4: finer-grained NLI predictor is the main component and three artificial annotators are to complement the main predictor;
* 0.7: over-amplify the role of main predictor and suppress three artificial annotators.

Table A1 shows the results on the dev set in terms of both accuracy and F1 under different \(r\) values. We set \(r\) to 0.4 as it achieves the best accuracy on the dev set.

## Appendix C Reproducibility Checklist

### Implementation Details

We set \(r\) in eq. 2 at 0.4 (see Appendix B). For all experiments, we use the Adam optimizer Kingma and Ba (2015) with a learning rate of 1e-5 and fine-tune up to 10 epochs. The batch size of first three baselines is 8 and that of Ensemble BERT baseline and our AAs is 2. The gradient is clipped when its norm exceeds 5. We select the best model for each method using the accuracy on the dev set, and the average performance on dev set is shown in Table 3 as well. Hyperparameters for underlying BERT (bert-base-uncased)5 are the default.

Footnote 5: [https://huggingface.co/transformers/](https://huggingface.co/transformers/)

### Summary Statistics of Results

For the results reported in Table 3, we run each baseline (except for "Always 0" and Heuristic rules) and AAs 10 times, and report the average. For the results reported in Table 6 and 7, we randomly select 3 trained models for each baseline and AAs, and report the average. We also share the checkpoints of AAs at [https://github.com/FrederickXZhang/FeNLI](https://github.com/FrederickXZhang/FeNLI) to help reproduce the results given in Section 5.

### Computational Resources

All experiments are conducted using one single GeForce GTX 2080 Ti 12 GB GPU (with significant CPU resources). The overall running time of 

baselines and AAs are listed in Table 14.

### Dataset

The characteristics of the CommitmentBank (CB) are detailed in Section 2. The original version is available at [https://github.com/mcdm/CommitmentBank](https://github.com/mcdm/CommitmentBank), and the recast version used in this work is available at [https://github.com/FrederickXZhang/FgNLI](https://github.com/FrederickXZhang/FgNLI).