SlowGuess/ACL_2020_OCR huggingface dataset:
Christoph Alt Aleksandra Gabryszak Leonhard Hennig
German Research Center for Artificial Intelligence (DFKI) Speech and Language Technology Lab
{christoph.alt, aleksandra.gabryszak, leonhard.hennig}@dfki.de
Despite the recent progress, little is known about the features captured by state-of-the-art neural relation extraction (RE) models. Common methods encode the source sentence, conditioned on the entity mentions, before classifying the relation. However, the complexity of the task makes it difficult to understand how encoder architecture and supporting linguistic knowledge affect the features learned by the encoder. We introduce 14 probing tasks targeting linguistic properties relevant to RE, and we use them to study representations learned by more than 40 different encoder architecture and linguistic feature combinations trained on two datasets, TACRED and SemEval 2010 Task 8. We find that the bias induced by the architecture and the inclusion of linguistic features are clearly expressed in the probing task performance. For example, adding contextualized word representations greatly increases performance on probing tasks with a focus on named entity and part-of-speech information, and yields better results in RE. In contrast, entity masking improves RE, but considerably lowers performance on entity type related probing tasks.
Relation extraction (RE) is concerned with extracting relationships between entities mentioned in text, where relations correspond to semantic categories such as org:founded_by, person:spouse, or org:subsidiaries (Figure 1). Neural models have shown impressive results on this task, achieving state-of-the-art performance on standard datasets like SemEval2010 Task 8 (dos Santos et al., 2015; Wang et al., 2016; Lee et al., 2019), TA-CRED (Zhang et al., 2018; Alt et al., 2019b; Peters et al., 2019; Joshi et al., 2019), and NYT (Lin et al., 2016; Vashishth et al., 2018; Alt et al., 2019a). The majority of models implement an encoder architec
Figure 1: Example relation from TACRED. The sentence contains the relation org:subsidiaries between the head and tail entities 'Aerolineas' and 'Austral'.
ture to learn a fixed size representation of the input, e.g. a sentence, which is passed to a classification layer to predict the target relation label.
These good results suggest that the learned representations capture linguistic and semantic properties of the input that are relevant to the downstream RE task, an intuition that was previously discussed for a variety of other NLP tasks by Conneau et al. (2018). However, it is often unknown which exact properties the various models have learned. Our aim is to pinpoint the information a given RE model is relying on, in order to improve model performance as well as to diagnose errors.
A general approach to model introspection is the use of probing tasks. Probing tasks (Shi et al., 2016; Adi et al., 2017), or diagnostic classifiers, are a well established method to analyze the presence of specific information in a model's latent representations, e.g. in machine-translation (Belinkov et al., 2017), language modeling (Giulianelli et al., 2018), and sentence encoding (Conneau et al., 2018). For each probing task, a classifier is trained on a set of representations, and its performance measures how well the information is encoded. The probing task itself is typically selected in accordance with the downstream task, e.g. an encoder trained on RE may be probed for the entity type of a relation argument. If the classifier correctly predicts the type, it implies the encoder retains entity type information in the representations, which also directly inform the relation prediction. The simplicity of this ap
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1534-1545 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
proach makes it easier to pinpoint the information a model is relying on, as opposed to probing the downstream task directly.
Our goal in this paper is to understand which features of the input a model conditioned on relation extraction has learned as useful for the task, in order to be able to better interpret and explain model predictions. Relation extraction literature is rich with information about useful features for the task (Zhou et al., 2005; Mintz et al., 2009; Surdeanu et al., 2011). Consequently, our initial question is whether and how good the sentence representations learned by state-of-the-art neural RE models encode these well-known features, such as e.g. argument entity types, dependency path or argument distance features. Another question is how the prior imposed by different encoding architectures, e.g. CNN, RNN, Graph Convolutional Network and Self-Attention, affects the features stored in the learned sentence representations. Finally, we would like to understand the effect of additional input features on the learned sentence representations. These include explicit semantic and syntactic knowledge like entity information and grammatical role, and as recently proposed, contextualized word representations such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018). We therefore significantly extend earlier work on probing tasks as follows:
- Following the framework of Conneau et al. (2018), we propose a set of 14 probing tasks specifically focused on linguistic properties relevant to relation extraction.
- We evaluate four encoder architectures, also in combination with supporting linguistic knowledge, on two datasets, TACRED (Zhang et al., 2017) and SemEval 2010 Task 8 (Hendrickx et al., 2010), for a total of more than 40 variants.
- We follow up on this analysis with an evaluation on the proposed probing tasks to establish a connection between task performance and captured linguistic properties.
- To facilitate further research and wider adoption, we open-source our relation extraction framework<sup>1</sup> based on AllenNLP (Gardner et al., 2018), and REval<sup>2</sup>, a framework extending the SentEval toolkit (Conneau and Kiela, 2018) with our probing tasks.
2 Probing Tasks
This section introduces the probing tasks we use to evaluate the learned sentence representations. We base our work on the setup and tasks introduced by Conneau et al. (2018), but focus on probing tasks related to relation extraction. We therefore adopt some of the tasks they propose, and introduce new probing tasks specifically designed for RE. As in their work, the probing task classification problem requires only single sentence embeddings as input (as opposed to, e.g., sentence and word embeddings, or multiple sentence representations). This fits the standard RE setup quite well, where the task is typically to classify the relation(s) expressed between a pair of entity mentions in a single sentence. While we focus on supervised relation extraction, this setup is also applicable in a distantly supervised RE setting, where state-of-the-art approaches are often based on passing sentence representations to a bag-level classifier that computes classification label(s) over all sentences for a given entity pair (Mintz et al., 2009; Lin et al., 2016). Similar to Conneau et al. (2018), we also aim to address a set of linguistic properties related to relation extraction ranging from simple surface phenomena (e.g. relation argument distance) to syntactic information (e.g. parse tree depth and argument ordering) and semantic information (e.g. the entity types of relation arguments). We use the standard training, validation, test split of the original TACRED dataset for RE and probing task experiments. For SemEval we reuse test and use  $10\%$  of the training set for validation. For TACRED we use the provided named entity, part-of-speech, and dependency parsing information, and parse SemEval with the Stanford Parser (2018-10-05 version) (Manning et al., 2014).
Surface information These tasks test whether sentence embeddings capture simple surface properties of sentences they encode. The sentence length (SentLen) task, introduced by Adi et al. (2017), predicts the number of tokens in a sentence. We group sentences into  $n = 10$  bins (TACRED, 7 bins for SemEval) by length, selecting bin widths so that training sentences are distributed approximately uniformly across bins, and treat SentLen as a n-way classification task. Our next probing task, argument distance (ArgDist), predicts the number of tokens between the two relation arguments. Similar to SentLen, we group sentences into 10 bins
https://github.com/DFKI-NLP/RelEx
$^{2}$ https://github.com/DFKI-NLP/REval
(5 for SemEval) by relative distance. Inspired by a common feature in classical RE (Surdeanu et al., 2011), we also test if any named entity exists between the two relation arguments (EntExist), treating it as a binary classification problem. Addressing this task requires the encoder to produce a sentence embedding that (at least partially) represents the inner context of the relation arguments.
Syntactic information Syntactic information is highly relevant for relation extraction. Many RE approaches utilize e.g. dependency path information (Bunescu and Mooney, 2005; Krause et al., 2012; Mintz et al., 2009), or part-of-speech tags (Zhou et al., 2005; Surdeanu et al., 2011). We therefore include the tree depth task (TreeDepth) described by Conneau et al. (2018). This task tests whether an encoder can group sentences by the depth of the longest path from root to any leaf. We group tree depth values into 10 (TACRED, SemEval 7) approximately uniformly distributed classes, ranging from depth 1 to depth 15. To account for shortest dependency path (SDP) information, we include an SDP tree depth task (SDPTreeDepth), which tests if the learned sentence embedding stores information about the syntactical link between the relation arguments. Again, we group SDP tree depth values into bins, in this case only 6 (4) classes, since the SDP trees are generally more shallow than the original sentence dependency parse tree. The argument ordering task (ArgOrd) tests if the head argument of a relation occurs before the tail argument in the token sequence. An encoder that successfully addresses this challenge captures some information about syntactic structures where the order of a relation's arguments is inverted, e.g. in constructions such as "The acquisition of Monsanto by Bayer", as compared to default constructions like "Bayer acquired Monsanto". We also include 4 tasks that test for the part-of-speech tag of the token directly to the left or right of the relation's arguments: PosHeadL, PosHeadR, PosTailL, PosTailR. These tasks test whether the encoder is sensitive to the immediate context of an argument. Some relation types, e.g. per:nationality or org:top_member, can often be identified based on the immediate argument context, e.g. "US president-NN Donald Trump", or "Google 's-POSS CEO-NN Larry Page". Representing this type of information in the sentence embedding should be useful for the relation classification.
Argument information Finally, we include probing tasks that require some understanding of what each argument denotes. The argument entity type tasks (TypeHead, TypeTail) ask for the entity tag of the head, and respectively the tail, argument. Entity type information is highly relevant for relation extraction systems since it strongly constrains the set of possible relation labels for a given argument pair. We treat these tasks as multi-class classification problems over the set of possible argument entity tags (see Section 3.3).
Our last task concerns the grammatical function of relation arguments. The grammatical role tasks (GRHead, GRTail) ask for the role of each argument, as given by the dependency label connecting the argument and its syntactic head token. The motivation is that the subject and object of verbal constructions often correspond to relation arguments for some relation types, e.g. "Bayer acquired Monsanto". We currently test for four roles, namely nsubj, nsubjpass, dobj and iobj, and group all other dependency labels into the other class. Note that there are other grammatical relations that may be of interest for relation extraction, for example possessive modifiers ("Google's Larry Page"), compounds ("Google CEO Larry Page"), and appositions ("Larry Page, CEO of Google").
3 Experiment Setup
This section first introduces the four sentence encoding architectures we consider for evaluation (§3.1), followed by a description of the supporting linguistic knowledge we evaluate: entity masking and contextualized word representations (§3.2). We also introduce the two datasets we use for training the relation extraction models and probing the sentence representations (§3.3).
3.1 Sentence Encoders
Generally, methods in relation extraction follow the sequence to vector approach, encoding the input (often a single sentence) into a fixed-size representation, before applying a fully connected relation classification layer (Figure 2). A single input is represented as a sequence of T tokens  $\{w_{t}\}_{t = 1,\dots,T}$ , and the spans (headstart, headend) and (tailstart, tailend) of the two entity mentions in question. We focus our evaluation on four widely used approaches that have shown to perform well on RE. For all architectures we signal the position of head and tail by the relative offset to each to
Figure 2: Probing task setup. In the first step, we train a RE model (sentence encoder and relation classifier) on a dataset  $D$ . In the second step, we fix the encoder and for each probing task train a classifier on the encoder representations  $\{s_j\}_{j=1,\dots|D|}$  of all sentences in  $D$ . The probing classifier performance indicates how well the sentence representations encode the information probed by the classifier, e.g. the entity type of the tail relation argument.
ken  $w_{i}$  as a positional embedding  $p_i^h\in \mathbb{R}^c$  and  $p_i^t\in \mathbb{R}^c$  concatenated to the input token representation  $e_i^t = [e_i^w,p_i^h,p_i^t ]$  , where  $e_i^w\in \mathbb{R}^d$  is the token embedding.
CNN We follow the work of Zeng et al. (2014) and Nguyen and Grishman (2015), who both use a convolutional neural network for relation extraction. Their models encode the input token sequence  $\{w_{t}\}_{t = 1,\dots,T}$  by applying a series of 1-dimensional convolutions of different filter sizes, yielding a set of output feature maps  $M_{f}$ , followed by a max-pooling operation that selects the maximum values along the temporal dimension of  $M_{f}$  to form a fixed-size representation.
Bi-LSTM max Similar to Zhang and Wang (2015) and Zhang et al. (2017), we use a Bi-LSTM to encode the input sequence. A Bi-LSTM yields a sequence of hidden states  $\{h_t\}_{t=1,\dots,T}$ , where  $h_t$  is a concatenation  $[h_t^f, h_t^b]$  of the states of a forward LSTM  $h^f$  and a backward LSTM  $h^b$ . Similar to the CNN, we use max pooling across the temporal dimension to obtain a fixed-size representation<sup>3</sup>.
GCN Graph convolutional networks (Kipf and Welling, 2016) adapt convolutional neural networks to graphs. Following the approach of Zhang et al. (2018), we treat the input token sequence  $\{w_{t}\}_{t = 1,\dots,T}$  as a graph consisting of T nodes, with an edge between  $w_{i}$  and  $w_{j}$ , if there exists a dependency edge between the two tokens. We
convert the dependency tree into a  $T \times T$  adjacency matrix, after pruning the graph to the shortest dependency path between head and tail. A L-layer GCN applied to  $\{w_t\}_{t=1,\dots,T}$  yields a sequence of hidden states  $\{h_t\}_{t=1,\dots,T}$  contextualized on neighboring tokens with a graph distance of at most L. Forming a fixed size representation is done by max pooling over the temporal dimension and local max pooling over the tokens  $\{w_t\}$ , for  $t \in [head_{start},\dots,head_{end}]$  and similar for  $t \in [tail_{start},\dots,tail_{end}]$ .
Multi-Headed Self-Attention Similar to the Transformer (Vaswani et al., 2017), we compute a sequence of contextualized representations  $\{h_t\}_{t=1,\dots,T}$  by applying L layers of multi-headed self-attention to the input token sequence  $\{w_t\}_{t=1,\dots,T}$ . The representation  $h_t$  of  $w_t$  is computed as a weighted sum of a projection  $V$  of the input tokens, with respect to the scaled, normalized dot product of  $Q$  and  $K$ , which are also both linear projections of the input with the procedure repeated for each attention head. A fixed-size representation is obtained by taking the final state  $h_T$  at the last layer L.
3.2 Supporting Linguistic Knowledge
Adding additional lexical, syntactic, and semantic input features to neural RE approaches has been shown to considerably improve performance (Zeng et al., 2014; Zhang et al., 2017, 2018). Features include e.g. casing, named entity, part-of-speech and dependency information. Most recently, pre
<sup>3</sup>We considered taking the final hidden state but found max pooling to perform superior.
learned contextualized word representations (deep language representations) emerged, capturing syntactic and semantic information useful to a wide range of downstream tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018). We therefore evaluate the effect of adding explicit named entity and grammatical role information (through entity masking) on our pre-learned sentence representations, and compare it to adding contextualized word representations computed by ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) as additional input features.
Entity Masking Entity masking has been shown to provide a significant gain for RE performance on the TACRED dataset (Zhang et al., 2017) by replacing each entity mention with a combination of its entity type and grammatical role (subject or object). It limits the information about entity mentions available to a model, possibly preventing overfitting to specific mentions and forcing the model to focus more on the context.
ELMo Embeddings from Language Models, as introduced by Peters et al. (2018), are an approach to compute contextualized word representations by applying a pre-learned, two-layer Bi-LSTM neural network to an input token sequence  $\{w_{t}\}_{t = 1,\dots,T}$ . ELMo operates on a character level and is pretrained with the forward and backward direction as a separate unidirectional language model. It yields a representation  $h_i = [h_i^f,h_i^b ]$  for each token  $w_{i}$  with  $h_i^f$  conditioned on the preceding context  $\{w_{t}\}_{t = 1,\dots,i - 1}$  and independently  $h_i^b$ , conditioned on the succeeding context  $\{w_t\}_{t = i + 1,\dots,T}$ .
BERT Bidirectional Encoder Representations from Transformers (Devlin et al., 2018) improves upon methods such as ELMo and the OpenAI Generative Pre-trained Transformer (GPT) (Radford et al., 2018) by using a masked language model that allows for jointly training forward and backward directions. Compared to ELMo, BERT operates on word-piece input and is based on the self-attentive Transformer architecture (Vaswani et al., 2017). It computes a representation for a token  $w_{i}$  jointly conditioned on the preceding  $\{w_{t}\}_{t = 1,\dots,i - 1}$  and succeeding context  $\{w_{t}\}_{t = i + 1,\dots,T}$ .
Table 1 shows key statistics of the TACRED and SemEval datasets. TACRED is approximately 10x the size of SemEval 2010 Task 8, but contains a
much higher fraction of negative training examples, making classification more challenging.
Dataset	# Relations	# Examples	Neg. examples
SemEval	19	10,717	17.4%
TACRED	42	106,264	79.5%
Table 1: Comparison of datasets used for evaluation
TACRED The TAC Relation Extraction  $Dataset^4$  (Zhang et al., 2017) contains 106k sentences with entity mention pairs collected from the TAC KBP5 evaluations. Sentences are annotated with person- and organization-oriented relation types, e.g. per:title, org:founded and no_relation for negative examples. In contrast to the SemEval dataset the entity mentions are typed with subjects classified into person and organization and objects categorized into 16 fine-grained classes (e.g., date, location, title). As per convention, we report our results as micro-averaged F1 scores.
SemEval 2010 Task 8 The SemEval 2010 Task 8 dataset $^6$  (Hendrickx et al., 2010) is a standard benchmark for binary relation classification, and contains 8,000 sentences for training and 2,717 for testing. Sentences are annotated with a pair of untyped nominals and one of 9 directed semantic relation types, such as Cause-Effect, Entity-Origin as well as the undirected Other type to indicate no relation, resulting in 19 distinct types in total. We follow the official convention and report macroaveraged F1 scores with directionality taken into account.
Table 2 and Table 3 report the accuracy scores of the probing task experiments for models trained on the TACRED and SemEval dataset. We did not include the ArgOrd and EntExists task in the SemEval evaluation, since SemEval relation arguments are always ordered in the sentence as indicated by the relation type, and entity types recognizable by standard tools such as Stanford CoreNLP that might occur between head and tail are not relevant to the dataset's entity types and relations.
$^{4}$ https://catalog.ldc.upenn.edu/LDC2018T24 $^{5}$ https://tac.nist.gov/2017/KBP/index.html $^{6}$ http://www.kozareva.com/downloads.html