Fine-grained Entity Typing through Increased Discourse Context and Adaptive Classification Thresholds

Abstract: Fine-grained entity typing is the task of assigning fine-grained semantic types to entity mentions. We propose a neural architecture which learns a distributional semantic representation that leverages a greater amount of semantic context -- both document and sentence level information -- than prior work. We find that additional context improves performance, with further improvements gained by utilizing adaptive classification thresholds. Experiments show that our approach without reliance on hand-crafted features achieves the state-of-the-art results on three benchmark datasets.

Named entity typing is the task of detecting the type (e.g., person, location, or organization) of a named entity in natural language text. Entity type information has shown to be useful in natural language tasks such as question answering BIBREF0 , knowledge-base population BIBREF1 , BIBREF2 , and co-reference resolution BIBREF3 . Motivated by its application to downstream tasks, recent work on entity typing has moved beyond standard coarse types towards finer-grained semantic types with richer ontologies BIBREF0 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . Rather than assuming an entity can be uniquely categorized into a single type, the task has been approached as a multi-label classification problem: e.g., in “... became a top seller ... Monopoly is played in 114 countries. ...” (fig:arch), “Monopoly” is considered both a game as well as a product.
The state-of-the-art approach BIBREF8 for fine-grained entity typing employs an attentive neural architecture to learn representations of the entity mention as well as its context. These representations are then combined with hand-crafted features (e.g., lexical and syntactic features), and fed into a linear classifier with a fixed threshold. While this approach outperforms previous approaches which only use sparse binary features BIBREF4 , BIBREF6 or distributed representations BIBREF9 , it has a few drawbacks: (1) the representations of left and right contexts are learnt independently, ignoring their mutual connection; (2) the attention on context is computed solely upon the context, considering no alignment to the entity; (3) document-level contexts which could be useful in classification are not exploited; and (4) hand-crafted features heavily rely on system or human annotations.
To overcome these drawbacks, we propose a neural architecture (fig:arch) which learns more context-aware representations by using a better attention mechanism and taking advantage of semantic discourse information available in both the document as well as sentence-level contexts. Further, we find that adaptive classification thresholds leads to further improvements. Experiments demonstrate that our approach, without any reliance on hand-crafted features, outperforms prior work on three benchmark datasets.
Fine-grained entity typing is considered a multi-label classification problem: Each entity INLINEFORM0 in the text INLINEFORM1 is assigned a set of types INLINEFORM2 drawn from the fine-grained type set INLINEFORM3 . The goal of this task is to predict, given entity INLINEFORM4 and its context INLINEFORM5 , the assignment of types to the entity. This assignment can be represented by a binary vector INLINEFORM6 where INLINEFORM7 is the size of INLINEFORM8 . INLINEFORM9 iff the entity is assigned type INLINEFORM10 .
Given a type embedding vector INLINEFORM0 and a featurizer INLINEFORM1 that takes entity INLINEFORM2 and its context INLINEFORM3 , we employ the logistic regression (as shown in fig:arch) to model the probability of INLINEFORM4 assigned INLINEFORM5 (i.e., INLINEFORM6 ) DISPLAYFORM0 
and we seek to learn a type embedding matrix INLINEFORM0 and a featurizer INLINEFORM1 such that DISPLAYFORM0 
At inference, the predicted type set INLINEFORM0 assigned to entity INLINEFORM1 is carried out by DISPLAYFORM0 
with INLINEFORM0 the threshold for predicting INLINEFORM1 has type INLINEFORM2 .
As shown in fig:arch, featurizer INLINEFORM0 in our model contains three encoders which encode entity INLINEFORM1 and its context INLINEFORM2 into feature vectors, and we consider both sentence-level context INLINEFORM3 and document-level context INLINEFORM4 in contrast to prior work which only takes sentence-level context BIBREF6 , BIBREF8 . 
The output of featurizer INLINEFORM0 is the concatenation of these feature vectors: DISPLAYFORM0 
 We define the computation of these feature vectors in the followings.
Entity Encoder: The entity encoder INLINEFORM0 computes the average of all the embeddings of tokens in entity INLINEFORM1 .
Sentence-level Context Encoder: The encoder INLINEFORM0 for sentence-level context INLINEFORM1 employs a single bi-directional RNN to encode INLINEFORM2 . Formally, let the tokens in INLINEFORM3 be INLINEFORM4 . The hidden state INLINEFORM5 for token INLINEFORM6 is a concatenation of a left-to-right hidden state INLINEFORM7 and a right-to-left hidden state INLINEFORM8 , DISPLAYFORM0 
 where INLINEFORM0 and INLINEFORM1 are INLINEFORM2 -layer stacked LSTMs units BIBREF10 . This is different from shimaoka-EtAl:2017:EACLlong who use two separate bi-directional RNNs for context on each side of the entity mention.
Attention: The feature representation for INLINEFORM0 is a weighted sum of the hidden states: INLINEFORM1 , where INLINEFORM2 is the attention to hidden state INLINEFORM3 . We employ the dot-product attention BIBREF11 . It computes attention based on the alignment between the entity and its context: DISPLAYFORM0 
 where INLINEFORM0 is the weight matrix. The dot-product attention differs from the self attention BIBREF8 which only considers the context.
Document-level Context Encoder: The encoder INLINEFORM0 for document-level context INLINEFORM1 is a multi-layer perceptron: DISPLAYFORM0 
 where DM is a pretrained distributed memory model BIBREF12 which converts the document-level context into a distributed representation. INLINEFORM0 and INLINEFORM1 are weight matrices.
In prior work, a fixed threshold ( INLINEFORM0 ) is used for classification of all types BIBREF4 , BIBREF8 . We instead assign a different threshold to each type that is optimized to maximize the overall strict INLINEFORM1 on the dev set. We show the definition of strict INLINEFORM2 in Sectionsubsec:metrics.
We conduct experiments on three publicly available datasets. tab:stat shows the statistics of these datasets.
OntoNotes: gillick2014context sampled sentences from OntoNotes BIBREF13 and annotated entities in these sentences using 89 types. We use the same train/dev/test splits in shimaoka-EtAl:2017:EACLlong. Document-level contexts are retrieved from the original OntoNotes corpus.
BBN: weischedel2005bbn annotated entities in Wall Street Journal using 93 types. We use the train/test splits in Ren:2016:LNR:2939672.2939822 and randomly hold out 2,000 pairs for dev. Document contexts are retrieved from the original corpus.
FIGER: Ling2012 sampled sentences from 780k Wikipedia articles and 434 news reports to form the train and test data respectively, and annotated entities using 113 types. The splits we use are the same in shimaoka-EtAl:2017:EACLlong.
We adopt the metrics used in Ling2012 where results are evaluated via strict, loose macro, loose micro INLINEFORM0 scores. For the INLINEFORM1 -th instance, let the predicted type set be INLINEFORM2 , and the reference type set INLINEFORM3 . The precision ( INLINEFORM4 ) and recall ( INLINEFORM5 ) for each metric are computed as follow.
Strict: INLINEFORM0 
Loose Macro: INLINEFORM0 
Loose Micro: INLINEFORM0 
We use open-source GloVe vectors BIBREF14 trained on Common Crawl 840B with 300 dimensions to initialize word embeddings used in all encoders. All weight parameters are sampled from INLINEFORM0 . The encoder for sentence-level context is a 2-layer bi-directional RNN with 200 hidden units. The DM output size is 50. Sizes of INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are INLINEFORM4 , INLINEFORM5 , and INLINEFORM6 respectively. Adam optimizer BIBREF15 and mini-batch gradient is used for optimization. Batch size is 200. Dropout (rate=0.5) is applied to three feature functions. To avoid overfitting, we choose models which yield the best strict INLINEFORM7 on dev sets.
We compare experimental results of our approach with previous approaches, and study contribution of our base model architecture, document-level contexts and adaptive thresholds via ablation. To ensure our findings are reliable, we run each experiment twice and report the average performance.
Overall, our approach significantly increases the state-of-the-art macro INLINEFORM0 on both OntoNotes and BBN datasets.
On OntoNotes (tab:ontonotes), our approach improves the state of the art across all three metrics. Note that (1) without adaptive thresholds or document-level contexts, our approach still outperforms other approaches on macro INLINEFORM0 and micro INLINEFORM1 ; (2) adding hand-crafted features BIBREF8 does not improve the performance. This indicates the benefits of our proposed model architecture for learning fine-grained entity typing, which is discussed in detail in Sectionsec:ana; and (3) Binary and Kwasibie were trained on a different dataset, so their results are not directly comparable.
On BBN (tab:bbn), while C16-1017's label embedding algorithm holds the best strict INLINEFORM0 , our approach notably improves both macro INLINEFORM1 and micro INLINEFORM2 . The performance drops to a competitive level with other approaches if adaptive thresholds or document-level contexts are removed.
On FIGER (tab:figer) where no document-level context is currently available, our proposed approach still achieves the state-of-the-art strict and micro INLINEFORM0 . If compared with the ablation variant of the Neural approach, i.e., w/o hand-crafted features, our approach gains significant improvement. We notice that removing adaptive thresholds only causes a small performance drop; this is likely because the train and test splits of FIGER are from different sources, and adaptive thresholds are not generalized well enough to the test data. Kwasibie, Attentive and Fnet were trained on a different dataset, so their results are not directly comparable.
tab:cases shows examples illustrating the benefits brought by our proposed approach. Example A illustrates that sentence-level context sometimes is not informative enough, and attention, though already placed on the head verbs, can be misleading. Including document-level context (i.e., “Canada's declining crude output” in this case) helps preclude wrong predictions (i.e., /other/health and /other/health/treatment). Example B shows that the semantic patterns learnt by our attention mechanism help make the correct prediction. As we observe in tab:ontonotes and tab:figer, adding hand-crafted features to our approach does not improve the results. One possible explanation is that hand-crafted features are mostly about syntactic-head or topic information, and such information are already covered by our attention mechanism and document-level contexts as shown in tab:cases. Compared to hand-crafted features that heavily rely on system or human annotations, attention mechanism requires significantly less supervision, and document-level or paragraph-level contexts are much easier to get.
Through experiments, we observe no improvement by encoding type hierarchical information BIBREF8 . To explain this, we compute cosine similarity between each pair of fine-grained types based on the type embeddings learned by our model, i.e., INLINEFORM3 in eq:prob. tab:type-sim shows several types and their closest types: these types do not always share coarse-grained types with their closest types, but they often co-occur in the same context.
We propose a new approach for fine-grained entity typing. The contributions are: (1) we propose a neural architecture which learns a distributional semantic representation that leverage both document and sentence level information, (2) we find that context increased with document-level information improves performance, and (3) we utilize adaptive classification thresholds to further boost the performance. Experiments show our approach achieves new state-of-the-art results on three benchmarks.
This work was supported in part by the JHU Human Language Technology Center of Excellence (HLTCOE), and DARPA LORELEI. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.

Figure and table captions:
Figure 1: Neural architecture for predicting the types of entity mention “Monopoly” in the text “... became a top seller ... Monopoly is played in 114 countries. ...”. Part of document-level context is omitted.
Table 1: Statistics of the datasets.
Table 2: Examples showing the improvement brought by document-level contexts and dot-product attention. Entities are shown in the green box. The gray boxes visualize attention weights (darkness) on context tokens.
Table 3: Results on the OntoNotes dataset.
Table 5: Results on the FIGER dataset.
Table 4: Results on the BBN dataset.
Table 6: Type similarity.

Questions about this paper:
What is the architecture of the model?
What fine-grained semantic types are considered?

Answers:
Document-level context encoder, entity and sentence-level context encoders with common attention, then logistic regression, followed by adaptive thresholds.
/other/event/accident, /person/artist/music, /other/product/mobile phone, /other/event/sports event, /other/product/car