SlowGuess/ACL_2020_OCR huggingface dataset:
The category context misinterpretation refers to cases where the sentential context of the arguments is misinterpreted by the model. We identify the following context problems: (1) Inverted arguments: the prediction is inverse to the correct relation, i.e. the model's prediction would be correct if head and tail were swapped. (2) Wrong arguments: the model incorrectly predicts a relation that holds between head or tail and an un-annotated entity mention in the context, therefore misinterpreting one annotated argument. (3) Linguistic distractor: the example contains words or phrases related to the predicted relation, however they do not connect to any of the arguments in a way justifying the prediction. (4) Factuality: the model ignores negation, speculation, future tense markers, etc. (5) Context ignored: the example does not contain sufficient linguistic evidence for the predicted relation except for the matching entity types. (6) Relation definition: the predicted relation could be inferred from the context using common sense or world knowledge, however the inference is prohibited by the guidelines (e.g. the spokesperson of an organization is not a top member/employee, or a work location is not a pointer to the employee's residence). (7) No Relation: the model incorrectly predicts no Relation even though there is sufficient
linguistic evidence for the relation in the sentential context.
Discussion The relation label predicted most frequently across the 49 models disagreed with the ground truth label of the re-annotated Challenging and Control Test groups in 1017 (43.3%) of the cases. The inter-annotator agreement of error categories assigned to these examples is high at  $\kappa_{Test} = 0.83$  ( $\kappa_{Test} = 0.67$  if the category No Relation is excluded).
Argument errors accounted for only 43  $(4.2\%)$  misclassifications, since the entities seem to be mostly correctly assigned in the dataset. In all entity type misclassification cases except one, the errors originate from false annotations in the dataset itself.
Context misinterpretation caused 974  $(95.8\%)$  false predictions. No relation is incorrectly assigned in 646  $(63.6\%)$  of misclassified instances, even though the correct relation is often explicitly and unambiguously stated. In 134  $(13.2\%)$  of the erroneous instances the misclassification resulted from inverted or wrong argument assignment, i.e. the predicted relation is stated, however the arguments are inverted or the predicted relation involves an entity other than the annotated one. In 96  $(9.4\%)$  instances the error results from TAC KBP guidelines prohibiting specific inferences, affecting most often the classification of the relations per: cities_of_residence and
Figure 2: Error rates for different groups (example subsets) on the revised TACRED test set, for four different models. The bars show the number and fraction of correctly (blue) and incorrectly (orange) predicted examples in the given group. KnowBert, as the best-performing model, has the lowest error rates across most groups. Error rates for per:loc, same_nertag&positive are higher for all models than the model error rate on the complete test set (all), highlighting examples for further error analysis and potential model improvements.
Figure 2 shows the error rates of different groups on the revised TACRED test set. The plot shows
org:top_member/employee. Furthermore, in 52  $(5.1\%)$  of the false predictions models seem to ignore the sentential context of the arguments, i.e. the predictions are inferred mainly from the entity types. Sentences containing linguistic distractors accounted for 35  $(3.4\%)$  incorrect predictions. Factuality recognition causes only 11 errors  $(1.1\%)$ . However, we assume that this latter low error rate is due to TACRED data containing an insufficient number of sentences suitable for extensively testing a model's ability to consider the missing factuality of relations.
5.2 Automatic Model Error Analysis
For the automatic analysis, we defined the following categories and error groups:
- Surface structure - Groups for argument distance ( $argdist = 1$ ,  $argdist > 10$ ) and sentence length ( $sentlen > 30$ )
- Arguments - Head and tail mention NER type (same_nertag, per:*, org:*, per:loc), and pronominal head/tail (has_coref)
- Context - Existence of distracting entities (has_distractor)
- Ground Truth - Groups conditioned on the ground truth (positive, negative, same_nertag&positive)
error rates across four representative models. Each chart displays groups on the y-axis, and the fraction and number of correct (blue) vs. incorrect (orange) instances in the respective group on the x-axis. The average error rate of each model on the full test set is shown for reference in the top-most column titled all. Groups with higher than average error rate may indicate characteristics of examples that make classification difficult. On the other hand, groups with lower than average error rate comprise examples that the given model performs especially well on.
What is the error rate for different groups? In Figure 2, we can see that KnowBERT has the lowest error rate on the full test set (7.9%), and the masked CNN model the highest (11.9%). Span-BERT's and TRE's error rates are in between the two. Overall, all models exhibit a similar pattern of error rates across the groups, with KnowBERT performing best across the board, and the CNN model worst. We can see that model error rates e.g. for the groups has distractor, argdist  $>10$ , and has coref do not diverge much from the corresponding overall model error rate. The presence of distracting entities in the context therefore does not seem to be detrimental to model performance. Similarly, examples with a large distance between the relation arguments, or examples where co-referential information is required, are generally predicted correctly. On the other hand, we can see that all models have above-average error rates for the group positive,
its subgroup same_nertag&positive, and the group per:loc. The above-average error rate for positive may be explained by the fact that the dataset contains much fewer positive than negative training instances, and is hence biased towards predicting no Relation. A detailed analysis shows that the groups per:loc and same_nertag&positive are the most ambiguous. per:loc contains relations such as per:cities_of_residence, per:countries_of_residence and per:origin, that may be expressed in a similar context but differ only in the fine-grained type of the tail argument (e.g. per:city vs. per:country). In contrast, same_nertag contains all person-person relations such as per:parents, per:children and per:other_family, as well as e.g. org:parent and org:subsidiaries that involve the same argument types (per:per vs. org:org) and may be only distinguishable from context.
How important is context? KnowBERT and SpanBERT show about the same error rate on the groups per:loc and same_nertag&positive. They differ, however, in which examples they predict correctly: For per:loc,  $78.6\%$  are predicted by both models, and  $21.4\%$  are predicted by only one of the models. For same_nertag&positive,  $12.8\%$  of the examples are predicted by only of the models. The two models thus seem to identify complementary information. One difference between the models is that KnowBERT has access to entity information, while SpanBERT masks entity spans.
To test how much the two models balance context and argument information, we apply rewriting to alter the instances belonging to a group and observe the impact on performance. We use two strategies: (1) we remove all tokens outside the span between head and tail argument (outside), and (2) we remove all tokens between the two arguments (between). We find that SpanBERT's performance on per:loc drops from 62.1 F1 to 57.7 (outside) and 43.3 (between), whereas KnowBERT's score decreases from 63.7 F1 to 60.9 and 50.1, respectively. On same_nertag&positive, we observe a drop from 89.2 F1 to 58.2 (outside) and 47.7 (between) for SpanBERT. KnowBERT achieves a score of 89.4, which drops to 83.8 and 49.0. The larger drop in performance on same_nertag&positive suggests that SpanBERT, which uses entity masking, focuses more on the context, whereas KnowBERT focuses on the entity content because the model has access to the arguments. Surprisingly, both models show similar
Model	Original	Revised	Weighted
CNN, masked	59.5	66.5	34.8
TRE	67.4	75.3	48.8
SpanBERT	70.8	78.0	61.9
KnowBERT	71.5	79.3	58.7
Table 5: Test set F1 score on TACRED, our revised version, and weighted by difficulty (on revised). The weight per instance is determined by the number of incorrect predictions in our set of 49 RE models. The result suggests that SpanBERT better generalizes to more challenging examples, e.g. complex sentential context.
performance on the full test set (Table 5). This suggests that combining both approaches may further improve RE performance.
Should instance difficulty be considered? Another question is whether the dataset contains instances that can be solved more easily than others, e.g. those with simple patterns or patterns frequently observed during training. We assume that these examples are also more likely to be correctly classified by our baseline set of 49 RE models.
To test this hypothesis, we change the evaluation setup and assign a weight to each instance based on the number of correct predictions. An example that is correctly classified by all 49 baseline models would receive a weight of zero – and thus effectively be ignored – whereas an instance misclassified by all models receives a weight of one. In Table 5, we can see that SpanBERT has the highest score on the weighted test set (61.9 F1), a  $16\%$  decrease compared to the unweighted revised test set. KnowBERT has the second highest score of 58.7,  $3\%$  less than SpanBERT. The performance of TRE and CNN is much worse at 48.8 and 34.8 F1, respectively. The result suggests that SpanBERT's span-level pre-training and entity masking are beneficial for RE and allow the model to generalize better to challenging examples. Given this observation, we propose to consider an instance's difficulty during evaluation.
Relation Extraction on TACRED Recent RE approaches include PA-LSTM (Zhang et al., 2017) and GCN (Zhang et al., 2018), with the former combining recurrence and attention, and the latter leveraging graph convolutional neural networks.
Many current approaches use unsupervised or semi-supervised pre-training: fine-tuning of language representations pre-trained on token-level (Alt et al., 2019; Shi and Lin, 2019) or span-level (Joshi et al., 2019), fine-tuning of knowledge enhanced word representations that are pre-trained on entity-linked text (Zhang et al., 2019; Peters et al., 2019), and "matching the blanks" pre-training (Baldini Soares et al., 2019).
Dataset Evaluation Chen et al. (2016) and Barnes et al. (2019) also use model results to assess dataset difficulty for reading comprehension and sentiment analysis. Other work also explores bias in datasets and the adoption of shallow heuristics on biased datasets in natural language inference (Niven and Kao, 2019) and argument reasoning comprehension (McCoy et al., 2019).
Analyzing trained Models Explanation methods include occlusion or gradient-based methods, measuring the relevance of input features to the output (Zintgraf et al., 2017; Harbecke et al., 2018), and probing tasks (Conneau et al., 2018; Kim et al., 2019) that probe the presence of specific features e.g. in intermediate layers. More similar to our approach is rewriting of instances (Jia and Liang, 2017; Ribeiro et al., 2018) but instead of evaluating model robustness we use rewriting to test explicit error hypotheses, similar to Wu et al. (2019).
In this paper, we conducted a thorough evaluation of the TACRED RE task. We validated the 5k most challenging examples in development and test set and showed that labeling is a major error source, accounting for  $8\%$  absolute F1 error on the test set. This clearly highlights the need for careful evaluation of development and test splits when creating datasets via crowdsourcing. To improve the evaluation accuracy and reliability of future RE methods, we provide a revised, extensively relabeled TACRED. In addition, we categorized model misclassifications into 9 common RE error categories and observed that models are often unable to predict a relation, even if it is expressed explicitly. Models also frequently do not recognize argument roles correctly, or ignore the sentential context. In an automated evaluation we verified our error hypotheses on the whole test split and showed that two groups of ambiguous relations are responsible for most of the remaining errors. We also showed that models adopt heuristics when en
tities are unmasked and proposed that evaluation metrics should consider an instance's difficulty.
We would like to thank all reviewers for their thoughtful comments and encouraging feedback, and Matthew Peters for providing KnowBERT predictions on TACRED. We would also like to thank Elif Kara, Ulli Strohriegel and Tatjana Zeen for the annotation of the dataset. This work has been supported by the German Federal Ministry of Education and Research as part of the projects DEEPLEE (01IW17001) and BBDC2 (01IS18025E), and by the German Federal Ministry for Economic Affairs and Energy as part of the project PLASS (01MD19003E).
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Improving relation extraction by pre-trained language representations. In Proceedings of the 2019 Conference on Automated Knowledge Base Construction, Amherst, Massachusetts.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905, Florence, Italy. Association for Computational Linguistics.
Jeremy Barnes, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment analysis is not solved! assessing and probing sentiment classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12-23, Florence, Italy. Association for Computational Linguistics.
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367, Berlin, Germany. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ 8$ !  $\# *$  vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
David Harbecke, Robert Schwarzenberg, and Christoph Alt. 2018. Learning explanations from language data. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 316-318, Brussels, Belgium. Association for Computational Linguistics.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghda, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38, Uppsala, Sweden. Association for Computational Linguistics.
Heng Ji and Ralph Grishman. 2011. Knowledge base population: Successful approaches and challenges. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 1148-1158, Stroudsburg, PA, USA. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke S. Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. ArXiv, abs/1907.10529.
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 39-48, Denver, Colorado. Association for Computational Linguistics.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. available as a preprint.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling Relations and Their Mentions without Labeled Text. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD '10).
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. ArXiv, abs/1904.05255.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2019. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747-763, Florence, Italy. Association for Computational Linguistics.
Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on freebase via relation extraction and textual evidence.
In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2326-2336. Association for Computational Linguistics.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. International Conference on Learning Representations.
CNN For training we use the hyperparameters of Zhang et al. (2017). We employ Adagrad as an optimizer, with an initial learning rate of 0.1 and run training for 50 epochs. Starting from the 15th epoch, we gradually decrease the learning rate by a factor of 0.9. For the CNN we use 500 filters of sizes [2, 3, 4, 5] and apply  $l_{2}$  regularization with a coefficient of  $10^{-3}$  to all filter weights. We use tanh as activation and apply dropout on the encoder output with a probability of 0.5. We use the same hyperparameters for variants with ELMo. For variants with BERT, we use an initial learning rate of 0.01 and decrease the learning rate by a factor of 0.9 every time the validation F1 score is
plateauing. Also we use 200 filters of sizes [2, 3, 4, 5].
LSTM/Bi-LSTM For training we use the hyperparameters of Zhang et al. (2017). We employ Adagrad with an initial learning rate of 0.01, train for 30 epochs and gradually decrease the learning rate by a factor of 0.9, starting from the 15th epoch. We use word dropout of 0.04 and recurrent dropout of 0.5. The BiLSTM consists of two layers of hidden dimension 500 for each direction. For training with ELMo and BERT we decrease the learning rate by a factor of 0.9 every time the validation F1 score is plateauing.
GCN We reuse the hyperparameters of Zhang et al. (2018). We employ SGD as optimizer with an initial learning rate of 0.3, which is reduced by a factor of 0.9 every time the validation F1 score plateaus. We use dropout of 0.5 between all but the last GCN layer, word dropout of 0.04, and embedding and encoder dropout of 0.5. Similar to the authors we use path-centric pruning with  $\mathrm{K} = 1$ . We use two 200-dimensional GCN layers and similar two 200-dimensional feedforward layers with ReLU activation.
A.2 Relation Extraction Performance
Table 6 show the relation extraction performances for the models on TACRED and our revised version. Models with 'w/synt/sem' use named entity and part-of-speech embeddings in addition to the input word embeddings.