Venue: NAACL 2018

# Neural Fine-Grained Entity Type Classification

with Hierarchy-Aware Loss

 Peng Xu

Department of Computing Science

University of Alberta

Edmonton, Canada

pxu4@ualberta.ca &Denilson Barbosa

Department of Computing Science

University of Alberta

Edmonton, Canada

denilson@ualberta.ca

###### Abstract

The task of Fine-grained Entity Type Classification (FETC) consists of assigning types from a hierarchy to entity mentions in text. Existing methods rely on distant supervision and are thus susceptible to noisy labels that can be _out-of-context_ or _overly-specific_ for the training sentence. Previous methods that attempt to address these issues do so with heuristics or with the help of hand-crafted features. Instead, we propose an end-to-end solution with a neural network model that uses a variant of cross-entropy loss function to handle _out-of-context_ labels, and hierarchical loss normalization to cope with _overly-specific_ ones. Also, previous work solve FETC a multi-label classification followed by ad-hoc post-processing. In contrast, our solution is more elegant: we use public word embeddings to train a single-label that jointly learns representations for entity mentions and their context. We show experimentally that our approach is robust against noise and consistently outperforms the state-of-the-art on established benchmarks for the task.

## 1 Introduction

Fine-grained Entity Type Classification (FETC) aims at labeling entity mentions in context with one or more specific types organized in a hierarchy (e.g., **actor** as a subtype of **artist**, which in turn is a subtype of **person**). Fine-grained types help in many applications, including relation extraction Mintz et al. (2009), question answering Li and Roth (2002), entity linking Lin et al. (2012), knowledge base completion Dong et al. (2014) and entity recommendation Yu et al. (2014). Because of the high cost in labeling large training corpora with fine-grained types, current FETC systems resort to distant supervision Mintz et al. (2009) and annotate mentions in the training corpus with _all types_ associated with the entity in a knowledge graph. This is illustrated in Figure 1, with three training sentences about entity _Steve Kerr_. Note that while the entity belongs to three fine-grained types (**person**, **athlete**, and **coach**), some sentences provide evidence of only some of the types: **person** and **coach** from **S1**, **person** and **athlete** from **S2**, and just **person** for **S3**. Clearly, direct distant supervision leads to noisy training data which can hurt the accuracy of the FETC model.

One kind of noise introduced by distant supervision is assigning labels that are _out-of-context_ (**athlete** in **S1** and **coach** in **S2**) for the sentence. Current FETC systems sidestep the issue by either ignoring _out-of-context_ labels or using simple pruning heuristics like discarding training examples with entities assigned to multiple types in the knowledge graph. However, both strategies are integant and hurt accuracy. Another source of noise introduced by distant supervision is when the type is _overly-specific_ for the context. For instance, example **S3** does not support the inference that Mr. Kerr is either an **athlete** or a **coach**. Since existing knowledge graphs give more attention to notable entities with more specific types, _overly-specific_ labels bias the model towards popular subtypes instead of generic ones, _i.e._, preferring **athlete** over **person**. Instead of correcting for this bias, most existing FETC systems ignore the problem and treat each type equally and independently, ignoring that many types are semantically related.

Besides failing to handle noisy training data there are two other limitations of previous FETC approaches we seek to address. First, they rely on hand-crafted features derived from various NLP tools; therefore, the inevitable errors introduced by these tools propagate to the FETC systems via the training data. Second, previous systems treat FETC as a multi-label classification problem: during type inference they predict a plausibility score for each type, and, then, either classify types

with scores above a threshold (Mintz et al., 2009; Gillick et al., 2014; Shimaoka et al., 2017) or perform a top-down search in the given type hierarchy (Ren et al., 2016a; Abhishek et al., 2017).

Contributions:We propose a neural network based model to overcome the drawbacks of existing FETC systems mentioned above. With publicly available word embeddings as input, we learn two different entity representations and use bidirectional long-short term memory (LSTM) with attention to learn the context representation. We propose a variant of cross entropy loss function to handle _out-of-context_ labels automatically during the training phase. Also, we introduce hierarchical loss normalization to adjust the penalties for correlated types, allowing our model to understand the type hierarchy and alleviate the negative effect of _overly-specific_ labels.

Moreover, in order to simplify the problem and take advantage of previous research on hierarchical classification, we transform the multi-label classification problem to a single-label classification problem. Based on the assumption that each mention can only have one _type-path_ depending on the context, we leverage the fact that type hierarchies are forests, and represent each _type-path_ uniquely by the terminal type (which might not be a leaf node). For Example, _type-path_**root-person-coach** can be represented as just **coach**, while **root-person** can be unambiguously represented as the non-leaf **person**.

Finally, we report on an experimental validation against the state-of-the-art on established benchmarks that shows that our model can adapt to noise in training data and consistently outperform previous methods. In summary, we describe a single, much simpler and more elegant neural network model that attempts FETC "end-to-end" without post-processing or ad-hoc features and improves on the state-of-the-art for the task.

## 2 Related Work

Fine-Grained Entity Type ClassificationThe first work to use distant supervision (Mintz et al., 2009) to induce a large - but noisy - training set and manually label a significantly smaller dataset to evaluate their FETC system, was Ling and Weld (2012) who introduced both a training and evaluation dataset FIGER (GOLD). They used a linear classifier perceptron for multi-label classification. While initial work largely assumed that mention assignments could be done independently of the mention context, Gillick et al. (2014) introduced the concept of context-dependent FETC where the types of a mention are constrained to what can be deduced from its context and introduced a new OntoNotes-derived (Weischedel et al., 2011) manually annotated evaluation dataset. In addition, they addressed the problem of label noise induced by distant supervision and proposed three label cleaning heuristics. Yogatama et al. (2015) proposed an embedding-based model where user-defined features and labels were embedded into a low dimensional feature space to facilitate information sharing among labels. Ma et al. (2016) presented a label embedding method that incor

Figure 1: With distant supervision, all the three mentions of _Steve Kerr_ shown are labeled with the same types in oval boxes in the target type hierarchy. While only part of the types are correct: **person** and **coach** for **S1**, **person** and **athlete** for **S2**, and just **person** for **S3**.

 

portates prototypical and hierarchical information to learn pre-trained label embeddings and adpated a zero-shot framework that can predict both seen and previously unseen entity types.

Shimaoka et al. (2016) proposed an attentive neural network model that used LSTMs to encode the context of an entity mention and used an attention mechanism to allow the model to focus on relevant expressions in such context. Shimaoka et al. (2017) summarizes many neural architectures for FETC task. These models ignore the _out-of-context noise_, that is, they assume that all labels obtained via distant supervision are "correct" and appropriate for every context in the training corpus. In our paper, a simple yet effective variant of cross entropy loss function is proposed to handle the problem of _out-of-context noise_.

Ren et al. (2016a) have proposed AFET, an FCTC system, that separates the loss function for _clean_ and _noisy_ entity mentions and uses label-label correlation information obtained by given data in its parametric loss function. Considering the noise reduction aspects for FETC systems, Ren et al. (2016b) introduced a method called LNR to reduce label noise without data loss, leading to significant performance gains on both the evaluation dataset of FIGER(GOLD) and OntoNotes. Although these works consider both _out-of-context noise_ and _overly-specific noise_, they rely on handcrafted features which become an impediment to further improvement of the model performance. For LNR, because the noise reduction step is separated from the FETC model, the inevitable errors introduced by the noise reduction will be propagated into the FETC model which is undesirable. In our FETC system, we handle the problem induced from _irrelevant noise_ and _ overly-specific noise_ seamlessly inside the model and avoid the usage of hand-crafted features.

Most recently, following the idea from AFET, Abhishek et al. (2017) proposed a simple neural network model which incorporates noisy label information using a variant of non-parametric hinge loss function and gain great performance improvement on FIGER(GOLD). However, their work overlooks the effect of _overly-specific noise_, treating each type label equally and independently when learning the classifiers and ignores possible correlations among types.

**Hierarchical Loss Function**: Due to the intrinsic type hierarchy existing in the task of FETC, it is natural to adopt the idea of hierarchical loss function to adjust the penalties for FETC mistakes depending on how far they are in the hierarchy. The penalty for predicting **person** instead of **athlete** should less than the penalty for predicting **organization**. To the best of our knowledge, the first use of a hierarchical loss function was originally introduced in the context of document categorization with support vector machines (Cai and Hofmann, 2004). However, that work assumed that weights to control the hierarchical loss would be solicited from domain experts, which is inapplicable for FETC. Instead, we propose a method called hierarchical loss normalization which can overcome the above limitations and be incorporated with cross entropy loss used in our neural architecture.

Table 1 provides a summary comparison of our work against the previous state-of-the-art in fine grained entity typing.

## 3 Background and Problem

Our task is to automatically reveal the type information for entity mentions in context. The input is a knowledge graph \(\Psi\) with schema \({\mathcal{Y}}_{\Psi}\), whose types are organized into a type hierarchy \({\mathcal{Y}}\), and an automatically labeled training corpus \({\mathcal{D}}\) obtained by distant supervision with \({\mathcal{Y}}\). The output is a _type-path_ in \({\mathcal{Y}}\) for each named entity mentioned in a test sentence from a corpus \({\mathcal{D}}_{t}\).

More precisely, a labeled corpus for entity type classification consists of a set of extracted _entity mentions_\(\{m_{i}\}^{N}_{i=1}\) (_i.e._, token spans representing entities in text), the _context_ (_e.g._, sentence, paragraph) of each mention \(\{c_{i}\}^{N}_{i=1}\), and the _candidate

\begin{table}
\begin{tabular}{l|c c|c||c||c} \hline  & Attentive & AFET & LNR & AAA & **NFETC** \\ \hline no hand-crafted features & — & — & — & ✓ & ✓ \\ \hline uses attentive neural network & ✓ & — & — & — & ✓ \\ \hline adopts single label setting & — & — & — & — & ✓ \\ \hline handles _out-of-context_ noise & — & ✓ & ✓ & ✓ & ✓ \\ \hline handles _overly-specifc_ noise & — & ✓ & ✓ & – & ✓ \\ \hline \end{tabular}
\end{table}
Table 1: Summary comparison to related FETC work. FETC systems listed in the table: (1) **Attentive**(Shimaoka et al., 2017); (2) **AFET**(Ren et al., 2016a); (3) **LNR**(Ren et al., 2016b); (4) **AAA**Abhishek et al., 2017).

 

type sets_\(\{\mathcal{Y}_{i}\}_{i=1}^{N}\) automatically generated for each mention.

We represent the training corpus using a set of mention-based triples \(\mathcal{D}=\left\{(m_{i},c_{i},\mathcal{Y}_{i})\right\}_{i=1}^{N}\).

If \(\mathcal{Y}_{i}\) is free of _out-of-context noise_, the type labels for each \(m_{i}\) should form a single _type-path_ in \(\mathcal{Y}_{i}\). However, \(\mathcal{Y}_{i}\) may contain _type-paths_ that are irrelevant to \(m_{i}\) in \(c_{i}\) if there exists _out-of-context noise_.

We denote the type set including all terminal types for each _type-path_ as the target type set \(\mathcal{Y}_{i}^{t}\). In the example type hierarchy shown in Figure 1, if \(\mathcal{Y}_{i}\) contains types **person**, **athlete**, **coach**, \(\mathcal{Y}_{i}^{t}\) should contain **athlete**, **coach**, but not **person**. In order to understand the trade-off between the effect of _out-of-context noise_ and the size of the training set, we report on experiments with two different training sets: \(\mathcal{D}_{filtered}\) only with triples whose \(\mathcal{Y}_{i}\) form a single _type-path_ in \(\mathcal{D}\), and \(\mathcal{D}_{raw}\) with all triples.

We formulate fine-grained entity classification problem as follows:

**Definition 1**: _Given an entity mention \(m_{i}=(w_{p},\ldots,w_{t})\) (\(p,t\in[1,T],p\leq t\)) and its context \(c_{i}=(w_{1},\ldots,w_{T})\) where \(T\) is the context length, our task is to predict its most specific type \(\hat{y}_{i}\) depending on the context._

In practice, \(c_{i}\) is generated by truncating the original context with words beyond the context window size \(C\) both to the left and to the right of \(m_{i}\). Specifically, we compute a probability distribution over all the \(K=|\mathcal{Y}|\) types in the target type hierarchy \(\mathcal{Y}\). The type with the highest probability is classified as the predicted type \(\hat{y}_{i}\) which is the terminal type of the predicted _type-path_.

## 4 Methodology

This section details our Neural Fine-Grained Entity Type Classification (NFETC) model.

### Input Representation

As stated in Section 3, the input is an entity mention \(m_{i}\) with its context \(c_{i}\). First, we transform each word in the context \(c_{i}\) into a real-valued vector to provide lexical-semantic features. Given a word embedding matrix \(W^{word}\) of size \(d_{w}\times|V|\), where \(V\) is the input vocabulary and \(d_{w}\) is the size of word embedding, we map every \(w_{i}\) to a column vector \(\mathbf{w}_{i}^{d}\in\mathbb{R}^{d_{w}}\).

To additionally capture information about the relationship to the target entities, we incorporate word position embeddings Zeng et al. (2014) to reflect relative distances between the \(i\)-th word to the entity mention. Every relative distance is mapped to a randomly initialized position vector in \(\mathbb{R}^{d_{p}}\), where \(d_{p}\) is the size of position embedding. For a given word, we obtain the position vector \(\mathbf{w}_{i}^{p}\). The overall embedding for the \(i\)-th word is \(\mathbf{w}_{i}^{E}=[(\mathbf{w}_{i}^{d})^{\top},(\mathbf{w}_{i}^{p})^{\top}]^ {\top}\).

### Context Representation

For the context \(c_{i}\), we want to apply a non-linear transformation to the vector representation of \(c_{i}\) to derive a context feature vector \(h_{i}=f(c_{i};\theta)\) given a set of parameters \(\theta\). In this paper, we adopt bidirectional LSTM with \(d_{s}\) hidden units as \(f(c_{i};\theta)\). The network contains two sub-networks for the forward pass and the backward pass respectively. Here, we use element-wise sum to combine the forward and backward pass outputs. The output of the \(i\)-th word in shown in the following equation:

\[h_{i}=[\overrightarrow{h_{i}}\oplus\overleftarrow{h_{i}}] \tag{1}\]

Following Zhou et al. (2016), we employ word-level attention mechanism, which makes our model able to softly select the most informative words during training. Let \(H\) be a matrix consisting of output vectors \([h_{1},h_{2},\dots,h_{T}]\) that the LSTM produced. The context representation \(r\) is formed by a weighted sum of these output vectors:

\[G=\tanh(H) \tag{2}\] \[\alpha=softmax(w^{\top}G)\] (3) \[r_{c}=H\alpha^{\top} \tag{4}\]

where \(H\in\mathbb{R}^{d_{s}\times T}\), \(w\) is a trained parameter vector. The dimension of \(w,\alpha,r_{c}\) are \(d_{s},T,d_{s}\) respectively.

### Mention Representation

**Averaging encoder:** Given the entity mention \(m_{i}=(w_{p},\dots,w_{t})\) and its length \(L=t-p+1\), the averaging encoder computes the average word embedding of the words in \(m_{i}\). Formally, the averaging representation \(r_{a}\) of the mention is computed as follows:

\[r_{a}=\frac{1}{L}\sum_{i=p}^{t}\mathbf{w}_{i}^{d}\] 

This relatively simple method for composing the mention representation is motivated by it being less prone to overfitting (Shimaoka et al., 2017).

**LSTM encoder:** In order to capture more semantic information from the mentions, we add one token before and another after the target entity to the mention. The extended mention can be represented as \(m_{i}^{*}=(w_{p-1},w_{p},\dots,w_{t},w_{t+1})\). The standard LSTM is applied to the mention sequence from left to right and produces the outputs \(h_{p-1},\dots,h_{t+1}\). The last output \(h_{t+1}\) then serves as the LSTM representation \(r_{l}\) of the mention.

### Optimization

We concatenate context representation and two mention representations together to form the overall feature representation of the input \(R=[r_{c},r_{a},r_{l}]\). Then we use a softmax classifier to predict \(\hat{y}_{i}\) from a discrete set of classes for a entity mention \(m\) and its context \(c\) with \(R\) as input:

\[\hat{p}(y|m,c)=\text{softmax}(WR+b) \tag{6}\] \[\hat{y}=\arg\max_{y}\hat{p}(y|m,c) \tag{7}\]

where \(W\) can be treated as the learned type embeddings and \(b\) is the bias.

The traditional cross-entropy loss function is represented as follows:

\[J(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\log(\hat{p}(y_{i}|m_{i},c_{ i}))+\lambda\lVert\Theta\rVert^{2} \tag{8}\]

where \(y_{i}\) is the only element in \(\mathcal{Y}_{i}^{t}\) and \((m_{i},c_{i},\mathcal{Y}_{i})\in\mathcal{D}_{filtered}\). \(\lambda\) is an L2 regularization hyperparameter and \(\Theta\) denotes all parameters of the considered model.

In order to handle data with _out-of-context noise_ (in other words, with multiple labeled types) and take full advantage of them, we introduce a simple yet effective variant of the cross-entropy loss:

\[J(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\log(\hat{p}(y_{i}^{*}|m_{i}, c_{i}))+\lambda\lVert\Theta\rVert^{2} \tag{9}\]

where \(y_{i}^{*}=\operatorname*{arg\,max}_{y\in\mathcal{Y}_{i}^{t}}\hat{p}(y|m_{i}, c_{i})\) and \((m_{i},c_{i},\mathcal{Y}_{i})\in\mathcal{D}_{raw}\). With this loss function, we assume that the type with the highest probability among \(\mathcal{Y}_{i}^{t}\) during training as the correct type. If there is only one element in \(\mathcal{Y}_{i}^{t}\), this loss function is equivalent to the cross-entropy loss function. Wherever there are multiple elements, it can filter the less probable types based on the local context automatically.

Figure 2: The architecture of the NFETC model.

 



### Hierarchical Loss Normalization

Since the fine-grained types tend to form a forest of type hierarchies, it is unreasonable to treat every type equally. Intuitively, it is better to predict an ancestor type of the true type than some other unrelated type. For instance, if one example is labeled as **athlete**, it is reasonable to predict its type as **person**. However, predicting other high level types like **location** or **organization** would be inappropriate. In other words, we want the loss function to penalize less the cases where types are related. Based on the above idea, we adjust the estimated probability as follows:

\[p^{*}(\hat{y}|m,c)=p(\hat{y}|m,c)+\beta*\sum_{t\in\Gamma}p(t|m,c) \tag{10}\]

where \(\Gamma\) is the set of ancestor types along the _type-path_ of \(\hat{y}\), \(\beta\) is a hyperparameter to tune the penalty. Afterwards, we re-normalize it back to a probability distribution, a process which we denote as _hierarchical loss normalization_.

As discussed in Section 1, there exists _overly-specific noise_ in the automatically labeled training sets which hurt the model performance severely. With _hierarchical loss normalization_, the model will get less penalty when it predicts the actual type for one example with _overly-specific noise_. Hence, it can alleviate the negative effect of _overly-specific noise_ effectively. Generally, _hierarchical loss normalization_ can make the model somewhat understand the given type hierarchy and learn to detect those overly-specific cases. During classification, it will make the models prefer generic types unless there is a strong indicator for a more specific type in the context.

### Regularization

Dropout, proposed by Hinton et al. (2012), prevents co-adaptation of hidden units by randomly omitting feature detectors from the network during forward propagation. We employ both input and output dropout on LSTM layers. In addition, we constrain L2-norms for the weight vectors as shown in Equations 8, 9 and use early stopping to decide when to stop training.

## 5 Experiments

This section reports an experimental evaluation of our NFETC approach using the previous state-of-the-art as baselines.

### Datasets

We evaluate the proposed model on two standard and publicly available datasets, provided in a pre-processed tokenized format by Shimaoka et al. (2017). Table 2 shows statistics about the benchmarks. The details are as follows:

* **FIGER(GOLD):** The training data consists of Wikipedia sentences and was automatically generated with distant supervision, by mapping Wikipedia identifiers to Freebase ones. The test data, mainly consisting of sentences from news reports, was manually annotated as described by Ling and Weld (2012).
* **OntoNotes:** The OntoNotes dataset consists of sentences from newswire documents present in the OntoNotes text corpus (Weischedel et al., 2013). DBpedia spotlight (Daiber et al., 2013) was used to automatically link entity mention in sentences to Freebase. Manually annotated test data was shared by Gillick et al. (2014).

Because the type hierarchy can be somewhat understood by our proposed model, the quality of the type hierarchy can also be a key factor to the performance of our model. We find that the type hierarchy for FIGER(GOLD) dataset following Freebase has some flaws. For example, **software** is not a subtype of **product** and **government** is not a subtype of **organization**. Following the proposed type hierarchy of Ling and Weld (2012), we refine the Freebase-based type hierarchy. The process is a one-to-one mapping for types in the original dataset and we didn't add or drop any type or sentence in the original dataset. As a result, we can directly compare the results of our proposed model with or without this refinement.

Aside from the advantages brought by adopting the single label classification setting, we can see one disadvantage of this setting based on Table 2. That is, the perfo

\begin{table}
\begin{tabular}{l|l|l}  & FIGER(GOLD) & OntoNotes \\ \hline \# types & 113 & 89 \\ \hline \# raw training mentions & 2009898 & 253241 \\ \hline \# raw testing mentions & 563 & 8963 \\ \hline \% filtered training mentions & 64.46 & 73.13 \\ \% filtered testing mentions & 88.28 & 94.00 \\ \hline Max hierarchy depth & 2 & 3 \\ \hline \end{tabular}
\end{table}
Table 2: Statistics of the datasets

our proposed model are no longer \(100\%\): for example, the best strict accuracy we can get in this setting is \(88.28\%\) for **FIGER(GOLD)**. However, as the strict accuracy of state-of-the-art methods are still nowhere near \(80\%\) (Table 3), the evaluation we perform is still informative.

### Baselines

We compared the proposed model with state-of-the-art FETC systems 1: (1) **Attentive**Shimaoka et al. (2017); (2) **AFET**Ren et al. (2016a); (3) **LNR+FIGER**Ren et al. (2016b); (4) **AAA**Abhishek et al. (2017).

Footnote 1: The results of the baselines are all as reported in their corresponding papers.

We compare these baselines with variants of our proposed model: (1) **NFETC(f)**: basic neural model trained on \({\cal D}_{filtered}\) (recall Section 4.4); (2) **NFETC-nier(f)**: neural model with hierarchical loss normalization trained on \({\cal D}_{filtered}\). (3) **NFETC(r)**: neural model with proposed variant of cross-entropy loss trained on \({\cal D}_{raw}\); (4) **NFETC-dier(r)**: neural model with proposed variant of cross-entropy loss and hierarchical loss normalization trained on \({\cal D}_{raw}\).

### Experimental Setup

For evaluation metrics, we adopt the same criteria as Ling and Weld (2012), that is, we evaluate the model performance by strict accuracy, loose macro, and loose micro F-scores. These measures are widely used in existing FETC systems Shinmaoka et al. (2017); Ren et al. (2016b, a); Abhishek et al. (2017).

We use pre-trained word embeddings that were not updated during training to help the model generalize to words not appearing in the training set. For this purpose, we used the freely available 300-dimensional cased word embedding trained on 840 billion tokens from the Common Crawl supplied by Pennington et al. (2014). For both datasets, we randomly sampled \(10\%\) of the test set as a development set, on which we do the hyperparameters tuning. The remaining \(90\%\) is used for final evaluation. We run each model with the well-tuned hyperparameter setting five times and report their average strict accuracy, macro F1 and micro F1 on the test set. The proposed model was implemented using the TensorFlow framework. 2

Footnote 2: The code to replicate the work is available at: [https://github.com/billy-inn/NFETC](https://github.com/billy-inn/NFETC)

### Hyperparameter Setting

In this paper, we search different hyperparameter settings for FIGER(GOLD) and OntoNotes separately, considering the differences between the two datasets. The hyperparameters include the learning rate \(lr\) for Adam Optimizer, size of word position embeddings (WPE) \(d_{p}\), state size for LSTM layers \(d_{s}\), input dropout keep probability \(p_{i}\) and output dropout keep probability \(p_{o}\) for LSTM layers 3, L2 regularization parameter \(\lambda\) and parameter to tune hierarchical loss normalization \(\beta\). The values of these hyperparameters, obtained by evaluating the model performance on the development set, for each dataset can be found in Table 4.

Footnote 3: Following TensorFlow terminology.

### Performance comparison and analysis

Table 3 compares our models with other state-of-the-art FETC systems on FIGER(GOLD) and OntoNotes. The proposed model performs better than the existing FETC systems, consistently on both datasets. This indicates benefits of the proposed representation scheme, loss function and hierarchical loss normalization.

Discussion about _Out-of-context Noise_:For dataset FIGER(GOLD), the performance of our model with the proposed variant of cross-entropy loss trained on \({\cal D}_{raw}\) is significantly better than the basic neural model trained on \({\cal D}_{filtered}\), suggesting that the proposed variant of the cross-entropy loss function can make use of the data with _out-of-context noise_ effectively. On the other hand, the improvement introduced by our proposed variant of cross-entropy loss is not as significant for the OntoNotes benchmark. This may be caused by the fact that OntoNotes is much smaller than FIGER(GOLD) and proportion of examples without _out-of-context noise_ are also higher, as shown in Table 2.

\begin{table}
\begin{tabular}{c|c|c} Parameter & FIGER(GOLD) & OntoNotes \\ \hline \(lr\) & 0.0002 & 0.0002 \\ \hline \(d_{p}\) & 85 & 20 \\ \hline \(d_{s}\) & 180 & 440 \\ \hline \(p_{i}\) & 0.7 & 0.5 \\ \hline \(p_{o}\) & 0.9 & 0.5 \\ \hline \(\lambda\) & 0.0 & 0.0001 \\ \hline \(\beta\) & 0.4 & 0.3 \\ \hline \end{tabular}
\end{table}
Table 4: Hyperparameter Settings 



**Investigations on _Overly-Specific Noise:** With hierarchical loss normalization, the performance of our models are consistently better no matter whether trained on \({\cal D}_{raw}\) or \({\cal D}_{filtered}\) on both datasets, demonstrating the effectiveness of this hierarchical loss normalization and showing that _overly-specific noise_ has a potentially significant influence on the performance of FETC systems.

### T-SNE Visualization of Type Embeddings

By visualizing the learned type embeddings (Figure 3), we can observe that the parent types are mixed with their subtypes and forms clear distinct clusters without hierarchical loss normalization, making it hard for the model to distinguish subtypes like **actor** or **athlete** from their parent types **person**. This also biases the model towards the most popular subtype. While the parent types tend to cluster together and the general pattern is more complicated with hierarchical loss normalization. Although it's not as easy to interpret, it hints that our model can learn rather subtle intricacies and correlations among types latent in the data with the help of hierarchical loss normalization, instead of sticking to a pre-defined hierarchy.

### Error Analysis on FIGER(GOLD)

Since there are only 563 sentences for testing in FIGER(GOLD), we look into the predictions for all the test examples of all variants of our model. Table 5 shows 5 examples of test sentence. Without hierarchical loss normalization, our model will make too aggressive predictions for S1 with **Politician** and for S2 with **Software**. This kind of mistakes are very common and can be effectively reduced by introducing hierarchical loss normalization leading to significant improvements on the model performance. Using the changed loss function to handle multi-label (noisy) training data can help the model distinguish ambiguous cases. For example, our model trained on \({\cal D}_{filtered}\) will misclassify S5 as **Title**, while the model trained on \({\cal D}_{raw}\) can make the correct prediction.

However, there are still some errors that can't be fixed with our model. For example, our model cannot make correct predictions for S3 and S4 due to the fact that our model doesn't know that \(U\!W\) is an abbreviation of _University of Washington_ and _Washington state_ is the name of a province. In addition, the influence of _overly-specific noise_ can only be alleviated but not eliminated. Sometimes, our model will still make too aggressive or conservative predictions. Also, mixing up very ambiguous entity names is inevitable in this task.

\begin{table}
\begin{tabular}{l l l l l l l} \hline  & \multicolumn{4}{c}{FIGER(GOLD)} & \multicolumn{4}{c}{OntoNotes} \\ \cline{2-7} Model & Strict Acc. & Macro F1 & Micro F1 & Strict Acc. & Macro F1 & Micro F1 \\ \hline
**Attentive** & \(59.68\) & \(78.97\) & \(75.36\) & \(51.74\) & \(70.98\) & \(64.91\) \\
**AFET** & \(53.3\) & \(69.3\) & \(66.4\) & \(55.1\) & \(71.1\) & \(64.7\) \\
**LNR+FIGER** & \(59.9\) & \(76.3\) & \(74.9\) & \(57.2\) & \(71.5\) & \(66.1\) \\
**AAA** & \(65.8\) & \(\mathbf{81.2}\) & \(77.4\) & \(52.2\) & \(68.5\) & \(63.3\) \\ \hline
**NFECTC(f)** & \(57.9\pm 1.3\) & \(78.4\pm 0.8\) & \(75.0\pm 0.7\) & \(54.4\pm 0.3\) & \(71.5\pm 0.4\) & \(64.9\pm 0.3\) \\
**NFECTC-hier(f)** & \(68.0\pm 0.8\) & \(\mathbf{81.4\pm 0.8}\) & \(77.9\pm 0.7\) & \(59.6\pm 0.2\) & \(\mathbf{76.1\pm 0.2}\) & \(69.7\pm 0.2\) \\
**NFECTC(r)** & \(56.2\pm 1.0\) & \(77.2\pm 0.9\) & \(74.3\pm 1.1\) & \(54.8\pm 0.4\) & \(71.8\pm 0.4\) & \(65.0\pm 0.4\) \\
**NFECTC-hier(r)** & \(\mathbf{68.9\pm 0.6}\) & \(\mathbf{81.9\pm 0.7}\) & \(\mathbf{79.0\pm 0.7}\) & \(\mathbf{60.2\pm 0.2}\) & \(\mathbf{76.4\pm 0.1}\) & \(\mathbf{70.2\pm 0.2}\) \\ \hline \end{tabular}
\end{table}
Table 3: Strict Accuracy, Macro F1 and Micro F1 for the models tested on the FIGER(GOLD) and OntoNotes datasets.

\begin{table}
\begin{tabular}{p{113.8pt}|p{113.8pt}} \hline Test Sentence & Ground Truth \\ \hline S1: _Hopkins_ said four fellow elections is curious , considering the ... & Person \\ S2: ...for WiFi communications across all _the SD cards._ & Product \\ S3: A handful of professors in the _UW_ Department of Chemistry ... & Educational Institution \\ S4: Work needs to be done and, in _Washington state_, ... & Province \\ S5: _ASC_ Director Melvin Taing said that because the commission is ... & Organization \\ \hline \end{tabular}
\end{table}
Table 5: Examples of test sentences in FIGER(GOLD) where the entity mentions are marked as bold italics.





## 6 Conclusion and Further Work

In this paper, we studied two kinds of noise, namely _out-of-context noise_ and _overly-specific noise_, for noisy type labels and investigate their effects on FETC systems. We proposed a neural network based model which jointly learns representations for entity mentions and their context. A variant of cross-entropy loss function was used to handle _out-of-context noise_. Hierarchical loss normalization was introduced into our model to alleviate the effect of _overly-specific noise_. Experimental results on two publicly available datasets demonstrate that the proposed model is robust to these two kind of noise and outperforms previous state-of-the-art methods significantly.

More work can be done to further develop hierarchical loss normalization since currently it's very simple. Considering type information is valuable in various NLP tasks, we can incorporate results produced by our FETC system to other tasks, such as relation extraction, to check our model's effectiveness and help improve other tasks' performance. In addition, tasks like relation extraction are complementary to the task of FETC and therefore may have potentials to be digged to help improve the performance of our system in return.

## Acknowledgments

This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC).

## References

* Abhishek et al. (2017) Abhishek Abhishek, Ashish Anand, and Amit Awekar. 2017. Fine-grained entity type classification by jointly learning representations and label embeddings. _In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ pages 797-807.
* Cai and Hofmann (2004) Lijuan Cai and Thomas Hofmann. 2004. Hierarchical document categorization with support vector machines. In _Proceedings of the thirteenth ACM international conference on Information and knowledge management_. ACM, pages 78-87.
* Daiber et al. (2013) Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. _Proceedings of the 9th International Conference on Semantic Systems_ pages 121-124.
* Dong et al. (2014) Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ pages 601-610.
* Gillick et al. (2014) Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. 2014. Context-dependent fine-grained entity type tagging. _arXiv preprint arXiv:1412.1820_ .
* Hinton et al. (2012) Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. _arXiv preprint arXiv:1207.0580_ .
* Li and Roth (2002) Xin Li and Dan Roth. 2002. Learning question classifiers. In _Proceedings of the 19th international conference on Computational linguistics-Volume 1_. Association for Computational Linguistics, pages 1-7.
* Lin et al. (2012) Thomas Lin, Oren Etzioni, et al. 2012. No noun phrase left behind: detecting and typing unlinkable entities. _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ pages 893-903.

Figure 3: T-SNE visualization of the type embeddings learned from FIGER(GOLD) dataset where subtypes share the same color as their parent type. The seven parent types are shown in the black boxes. The below sub-figure uses the hierarchical loss normalization, while the above not.



Xiao Ling and Daniel S Weld. 2012. Fine-grained entity recognition. _AAAI_ .
* Ma et al. (2016) Yukun Ma, Erik Cambria, and Sa Gao. 2016. Label embedding for zero-shot fine-grained named entity typing. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. pages 171-180.
* Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ pages 1003-1011.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. _EMNLP_ 14(1532-1543).
* Ren et al. (2016a) Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. 2016a. Afet: Automatic fine-grained entity typing by hierarchical partial-label embedding. _EMNLP_ 16(17).
* Ren et al. (2016b) Xiang Ren, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, and Jiawei Han. 2016b. Label noise reduction in entity typing by heterogeneous partial-label embedding. _KDD_ .
* Shimaoka et al. (2016) Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2016. An attentive neural architecture for fine-grained entity type classification. _arXiv preprint arXiv:1604.05525_ .
* Shimaoka et al. (2017) Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2017. Neural architectures for fine-grained entity type classification. _In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ .
* Weischedel et al. (2011) Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. _Handbook of Natural Language Processing and Machine Translation. Springer_ .
* Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. _Linguistic Data Consortium, Philadelphia, PA_ .
* Yogatama et al. (2015) Dani Yogatama, Daniel Gillick, and Nevena Lazic. 2015. Embedding methods for fine grained entity type classification. _ACL (2)_ pages 291-296.
* Yu et al. (2014) Xiao Yu, Xiang Ren, Yizhou Sun, Quanquan Gu, Bradley Sturt, Urvashi Khandelwal, Brandon Norick, and Jiawei Han. 2014. Personalized entity recommendation: A heterogeneous information network approach. _Proceedings of the 7th ACM international conference on Web search and data mining_ pages 283-292.
* Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. _COLING_ pages 2335-2344.
* Zhou et al. (2016) Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. _The 54th Annual Meeting of the Association for Computational Linguistics_ .