Venue: NAACL 2019

# Improving Distantly-supervised Entity Typing

with Compact Latent Space Clustering

Bo Chen\({}^{1}\), Xiaotao Gu\({}^{2}\), Yufeng Hu\({}^{1}\), Siliang Tang\({}^{1*}\), Guoping Hu\({}^{3}\),

**Yueting Zhuang\({}^{1}\) & Xiang Ren\({}^{4}\)**

\({}^{1}\)Zhejiang University, \({}^{2}\)University of Illinois at Urbana Champaign

\({}^{3}\)iFLYTEK Research, \({}^{4}\)University of Southern California,

{chenbo123, xiaofeeem, siliang, yzhuang}@zju.edu.cn,xiaotao2@illinois.edu, gphu@iflytek.com,xiangren@usc.edu

###### Abstract

Recently, distant supervision has gained great success on Fine-grained Entity Typing (FET). Despite its efficiency in reducing manual labeling efforts, it also brings the challenge of dealing with false entity type labels, as distant supervision assigns labels in a context-agnostic manner. Existing works alleviated this issue with partial-label loss, but usually suffer from confirmation bias, which means the classifier fit a pseudo data distribution given by itself. In this work, we propose to regularize distantly supervised models with Compact Latent Space Clustering (CLSC) to bypass this problem and effectively utilize noisy data yet. Our proposed method first dynamically constructs a similarity graph of different entity mentions; infer the labels of noisy instances via label propagation. Based on the inferred labels, mention embeddings are updated accordingly to encourage entity mentions with close semantics to form a compact cluster in the embedding space, thus leading to better classification performance. Extensive experiments on standard benchmarks show that our CLSC model consistently outperforms state-of-the-art distantly supervised entity typing systems by a significant margin.

+
Footnote †: Corresponding Author.

## 1 Introduction

Recent years have seen a surge of interests in fine-grained entity typing **(FET)** as it serves as an important cornerstone of several nature language processing tasks including relation extraction Mintz et al. (2009), entity linking Raiman and Raiman (2018), and knowledge base completion Dong et al. (2014). To reduce manual efforts in labelling training data, distant supervision Mintz et al. (2009) has been widely adopted by recent FET systems. With the help of an external knowledge base (KB), an entity mention is first linked to an existing entity in KB, and then labeled with all possible types of the KB entity as supervision. However, despite its efficiency, distant supervision also brings the challenge of **out-of-context noise**, as it assigns labels in a context agnostic manner. Early works usually ignore such noise in supervision Ling and Weld (2012); Shimaoka et al. (2016), which dampens the performance of distantly supervised models.

Towards overcoming out-of-context noise, two lines of work have been proposed to distantly supervised FET. The first kind of work try to filter out noisy labels using heuristic rules Gillick et al. (2014). However, such heuristic pruning significantly reduces the amount of training data, and thus cannot make full use of distantly annotated data. In contrast, the other thread of works try to incorporate such imperfect annotation by partial-label loss (**PLL**). The basic assumption is that, _for a noisy mention, the maximum score associated with its candidate types should be greater than the scores associated with any other non-candidate types_Ren et al. (2016a); Abhishek et al. (2017); Xu and Barbosa (2018). Despite their success, **PLL**-based models still suffer from _Confirmation Bias_ by taking its own prediction as optimization objective in the next step. Specifically, given an entity mention, if the typing system selected a wrong

Figure 1: T-SNE visualization of the mention embeddings generated by NFETC (left) and CLSC (right) on the BBN dataset. Our model (CLSC) clearly groups mentions of the same type into a compact cluster.



type with the maximum score among all candidates, it will try to further maximize the score of the wrong type in following optimization epoches (in order to minimize **PLL**), thus amplifying the confirmation bias. Such bias starts from the early stage of training, when the typing model is still very suboptimal, and can accumulate in training process. Related discussion can be also found in the setting of semi-supervised learning Lee et al. (2006); Laine and Aila (2017); Tarvainen and Valpola (2017).

In this paper, we propose a new method for distantly supervised fine-grained entity typing. Enlightened by Kamnitsas et al. (2018), we propose to effectively utilize imperfect annotation as model regularization via **C**ompact **L**atent **S**pace **Clustering (CLSC)**. More specifically, our model encourages the feature extractor to group mentions of the same type as a compact cluster (dense region) in the representation space, which leads to better classification performance. For training data with noisy labels, instead of generating pseudo supervision by the typing model itself, we dynamically construct a similarity-weighted graph between clean and noisy mentions, and apply label propagation on the graph to help the formation of compact clusters. Figure 1 demonstrates the effectiveness of our method in clustering mentions of different types into dense regions. In contrast to **PLL**-based models, we do not force the model to fit pseudo supervision generated by itself, but only use noisy data as part of regularization for our feature extractor layer, thus avoiding bias accumulation. Extensive experiments on standard benchmarks show that our method consistently outperforms state-of-the-art models. Further study reveals that, the advantage of our model over the competitors gets even more significant as the portion of noisy data rises.

## 2 Problem Definition

Fine-grained entity typing takes a corpus and an external knowledge base (KB) with a type hierarchy \({\cal Y}\) as input. Given an entity mention (i.e., a sequence of token spans representing an entity) in the corpus, our task is to uncover its corresponding type-path in \({\cal Y}\) based on the context.

By applying distant supervision, each mention is first linked to an existing entity in KB, and then labeled with all its possible types. Formally, a labeled corpus can be represented as triples \({\cal D}=\{(m_{i},c_{i},{\cal Y}_{i})\}^{n}_{i=1}\), where \(m_{i}\) is the \(i\)-th mention, \(c_{i}\) is the context of \(m_{i}\), \({\cal Y}_{i}\) is the set of candidate types of \(m_{i}\). Note that types in \({\cal Y}_{i}\) can form one or more type paths. In addition, we denote all terminal (leaf) types of each type path in \({\cal Y}_{i}\) as the target type set \({\cal Y}^{t}_{i}\) (_e.g._, for \({\cal Y}_{i}=\{artist,teacher,person\}\), \({\cal Y}^{t}_{i}=\{artist,teacher\}\)). This setting is also adopted by Xu and Barbosa (2018).

As each entity in KB can have several type paths, _out-of-context_ noise may exist when \({\cal Y}_{i}\) contains type paths that are irrelevant to \(m_{i}\) in context \(c_{i}\). In this work, we argue triples where \({\cal Y}_{i}\) contains only one type path (i.e., \(|{\cal Y}^{t}_{i}|=1\)) as **clean data**. Other triples are treated as **noisy data**, where \({\cal Y}_{i}\) contains both the true type path and irre

Figure 2: The overall framework of CLSC. We calculate classification loss only on clean data, while regularize the feature extractor with CLSC using both clean and noisy data.

 

evant type paths. Noisy data usually takes a considerable portion of the entire dataset. The major challenge for distantly supervised typing systems is to incorporate both clean and noisy data to train high-quality type classifiers.

## 3 The Proposed Approach

**Overview.** The basic assumptions of our idea are: (1) all mentions belong to the same type should be close to each other in the representation space because they should have similar context, (2) similar contexts lead to the same type. For clean data, we compact the representation space of the same type to comply (1). For noisy data, given assumption (2), we infer the their type distributions via label propagation and candidate types constrain.

Figure 2 shows the overall framework of the proposed method. Clean data is used to train classifier and feature extractor end-to-endly, while noisy data is only used in CLSC regularization. Formally, given a batch of samples \(\{(m_{i},c_{i},\mathcal{Y}^{t}_{i})\}_{i=1}^{B}\), we first convert each sample \((m_{i},c_{i})\) into a real-valued vector \(z_{i}\) via a feature extractor \(z((m_{i},c_{i});\theta_{z})\) parameterized by \(\theta_{z}\). Then a type classifier \(g(z_{i};\theta_{g})\) parameterized by \(\theta_{g}\) gives the posterior \(P(y|z_{i};\theta_{g})\). By incorporating CLSC regularization in the objective function, we encourage the feature extractor \(z\) to group mentions of the same type into a compact cluster, which facilitates classification as is shown in Figure 1. Noisy data enhances the formation of compact clusters with the help of label propagation.

### Feature Extractor

Figure 3 illustrates our feature extractor. For fair comparison, we adopt the same feature extraction pipeline as used in Xu and Barbosa (2018). The feature extractor is composed of an embedding layer and two encoders which encode mentions and contexts respectively.

**Embedding Layer:** The output of this layer is a concatenation of word embedding and word position embedding. We use the popular 300-dimensional word embedding supplied by Pennington et al. (2014) to capture the semantic information and random initialized position embedding Zeng et al. (2014) to acquire information about the relation between words and the mentions.

Formally, Given a word embedding matrix \(W_{word}\) of shape \(d_{w}\times|V|\), where \(V\) is the vocabulary and \(d_{w}\) is the size of word embedding, each column of \(W_{word}\) represents a specific word \(w\) in \(V\). We map each word \(w_{j}\) in \((m_{i},c_{i})\) to a word embedding \(\mathbf{w_{j}^{d}}\in R^{d_{w}}\). Analogously, we get the word position embedding \(\mathbf{w_{j}^{p}}\in R^{d_{p}}\) of each word according to the relative distance between the word and the mention, we only use a fixed length context here. The final embedding of the j-th word is \(\mathbf{w_{j}^{E}}=[\mathbf{w_{j}^{d}},\mathbf{w_{j}^{p}}]\).

**Mention Encoder:** To capture lexical level information of mentions, an averaging mention encoder and a LSTM mention encoder Hochreiter and Schmidhuber (1997) is applied to encode mentions. Given \(m_{i}=(w_{s},w_{s+1},\cdots,w_{e})\), the aver

Figure 3: The architecture of feature extractor \(z((m_{i},c_{i});\theta_{z})\)



aging mention representation \(r_{a_{i}}\in R^{d_{w}}\) is :

\[r_{a_{i}}=\frac{1}{e-s+1}\sum_{j=s}^{e}\mathbf{w_{j}^{d}} \tag{1}\]

By applying a LSTM over an extended mention \((w_{s-1},w_{s},w_{s+1},\cdots,w_{e},w_{e+1})\), we get a sequence \((h_{s-1},h_{s},h_{s+1},\cdots,h_{e},h_{e+1})\). We use \(h_{e+1}\) as LSTM mention representation \(r_{l_{i}}\in R^{d_{l}}\). The final mention representation is \(r_{m_{i}}=[r_{a_{i}},r_{l_{i}}]\in R^{d_{w}+d_{l}}\).

**Context Encoder:** A bidirectional LSTM with \(d_{l}\) hidden units is employed to encode embedding sequence \((\mathbf{w_{s-W}^{E}},\mathbf{w_{s-W+1}^{E}},\cdots,\mathbf{w_{e+W}^{E}})\):

\[\overrightarrow{h_{j}}=LSTM(\overrightarrow{h_{j-1}},\mathbf{ w_{j-1}^{E}}) \tag{2}\] \[\overleftarrow{h_{j}}=LSTM(\overrightarrow{h_{j-1}},\mathbf{ w_{j-1}^{E}})\] \[h_{j}= [\overrightarrow{h_{j}}\oplus\overleftarrow{h_{j}}]\]

where \(\oplus\) denotes element-wise plus. Then, the word-level attention mechanism computes a score \(\beta_{i,j}\) over different word \(j\) in the context \(c_{i}\) to get the final context representation \(r_{c_{i}}\):

\[\alpha_{j}= w^{T}tanh(h_{j}) \tag{3}\] \[\beta_{i,j}= \frac{exp(\alpha_{j})}{\sum\limits_{k}exp(\alpha_{k})}\] \[r_{c_{i}}= \sum_{j}\beta_{i,j}h_{i,j}\]

We use \(r_{i}=[r_{m_{i}},r_{c_{i}}]\in R^{d_{z}}=R^{d_{w}+d_{l}+d_{l}}\) as the feature representation of \((m_{i},c_{i})\) and use a Neural Networks \(q\) over \(r_{i}\) to get the feature vector \(z_{i}\). \(q\) has \(n\) layers with \(h_{n}\) hidden units and use ReLu activation.

### Compact Latent Space Clustering for Distant Supervision

The overview of CLSC regularization is exhibited in Figure 4, which includes three steps: dynamic graph construction (Figure 4c), label propagation (Figure 4d, e) and Markov chains (Figure 4g). The idea of compact clustering for semi-supervised learning is first proposed by Kamnitas et al. (2018). The basic idea is to encourage mentions of the same type to be clustered into a dense region in the embedding space. We introduce more details of CLSC for distantly supervised FET in following sections.

**Dynamic Graph Construction:** We start by creating a fully connected graph \(G\) over the batch of samples \(\mathbf{Z}=\{z_{i}\}_{i=1}^{B}\), as shown in Figure 4c1. Each node of \(G\) is a feature representation \(z_{i}\), while the distance between nodes is defined by a scaled dot-product distance function Vaswani et al. (2017):

Footnote 1: \(\mathbf{Z}=\{z_{i}\}_{i=1}^{B}\) is a small subsample of the entire data, we didn’t observe significant performance gain when the batch size increases.

\[A_{ij}= exp(\frac{z_{i}^{T}z_{j}}{\sqrt{d_{z}}}),\forall z_{i},z_{j}\in \mathbf{Z} \tag{4}\] \[A= exp(\frac{Z^{T}Z}{\sqrt{d_{z}}})\]

Each entry \(A_{ij}\) measures the similarity between \(z_{i}\) and \(z_{j}\), \(A\in R^{B\times B}\) can be viewed as the weighted adjacency matrix of \(G\).

**Label Propagation:** The end goal of CLSC is to cluster mentions of the same type to a dense region. For mentions which have more than one labeled types, we apply label propagation (**LP**) on \(G\) to estimate their type distribution. Formally, we denote \(\boldsymbol{\Phi}\in R^{B\times K}\) as the label propagation posterior of a training batch.

The original label propagation proposed by Zhu and Ghahramani (2002) uses a transition matrix \(H\) to model the probability of a node \(i\) propagating its type posterior \(\phi_{i}=P(y_{i}|x_{i})\in R^{K}\) to the other nodes. Each entry of the transition matrix \(H\in R^{B\times B}\) is defined as:

\[H_{ij}=A_{ij}/\sum_{b}A_{ib} \tag{5}\]

The original label propagation algorithm is defined as:

1. Propagate the label by transition matrix \(H\), \(\boldsymbol{\Phi}^{(t+1)}=H\boldsymbol{\Phi}^{(t)}\)
2. Clamp the labeled data to their true labels. Repeat from step 1 until \(\boldsymbol{\Phi}\) converges In this work \(\Phi^{(0)}\) is randomly initialized2. Unlike unlabeled data in semi-supervised learning, distantly labeled mentions in FET have a limited set of candidate types. Based on this observation, We assume that \((m_{i},c_{i})\) can only transmit and receive probability of types in \(\mathcal{Y}_{i}^{t}\) no matter it is noisy data or clean data. Formally, define a \(B\times K\) indicator matrix \(M\in R^{B\times K}\), where \(M_{ij}=1\) if type j in \(\mathcal{Y}_{i}^{t}\) otherwise \(0\), where \(B\) is the batch size and 

is the number of types. Our clamping step relies on \(M\) as is shown in Figure 4d:

\[\Phi_{ij}^{(t+1)}\leftarrow\Phi_{ij}^{(t+1)}M_{ij}/\sum_{k}\Phi_{ik}^{(t+1)}M_{ ik} \tag{6}\]

For convenience, we iterate through these two steps \(S_{lp}\) times, \(S_{lp}\) is a hyperparameter.

**Compact Clustering:** The **LP** posterior \(\mathbf{\Phi}=\mathbf{\Phi}^{(S_{lp}+1)}\) is used to judge the label agreement between samples. In the desired optimal state, transition probabilities between samples should be uniform inside the same class, while be zero between different classes. Based on this assumption, the desirable transition matrix \(T\in R^{B\times B}\) is defined as:

\[T_{ij}=\sum_{k=1}^{K}\Phi_{ik}\frac{\Phi_{jk}}{m_{k}},m_{k}=\sum_{b=1}^{B}\Phi_ {bl} \tag{7}\]

\(m_{k}\) is a normalization term for class \(k\). Transition matrix \(H\) derived from \(z((m_{i},c_{i});\theta_{z})\) should be in keeping with \(T\). Thus we minimize the cross entropy between \(T\) and \(H\):

\[\mathcal{L}_{1-step}=-\frac{1}{B^{2}}\sum_{i=1}^{B}\sum_{j=1}^{B}T_{ij}log(H_ {ij}) \tag{8}\]

For instance, if \(T_{ij}\) is close to 1, \(H_{ij}\) needs to be bigger, which results in the growth of \(A_{ij}\) and finally optimize \(\theta_{z}\) (Eq.4). The loss \(\mathcal{L}_{1-step}\) has largely described the regularization we use in \(z((m_{i},c_{i});\theta_{z})\) for compression clustering.

In order to keep the structure of existing clusters, Kamnitsas et al. (2018) proposed an extension of \(\mathcal{L}_{1-step}\) to the case of **Markov chains** with multiple transitions between samples, which should remain within a single class. The extension maximizes probability of paths that only traverse among samples belong to one class. Define \(E\in R_{B\times B}\) as:

\[E=\mathbf{\Phi}^{T}\mathbf{\Phi} \tag{9}\]

\(E_{ij}\) measures the label similarities between \(z_{i}\) and \(z_{j}\), which is used to mask the transition between different clusters. The extension is given by:

\[\begin{split} H^{(1)}=& H\\ H^{(s)}=&(H\odot E)^{(s-1)}H\\ =&(H\odot E)H^{(s-1)},\end{split} \tag{10}\]

where \(\odot\) is Hadamard Product, and \(H^{(s)}_{ij}\) is the probability of a Markov process to transit from node \(i\) to node \(j\) after \(s-1\) steps within the same class. The extended loss function models paths of different length \(s\) between samples on the graph:

\[\mathcal{L}_{clsc}=-\frac{1}{S_{m}}\frac{1}{B^{2}}\sum_{s=1}^{S_{m}}\sum_{i=1 }^{B}\sum_{j=1}^{B}T_{ij}log(H^{(s)}_{ij}). \tag{11}\]

For \(S_{m}=1\), \(\mathcal{L}_{clsc}=\mathcal{L}_{1-step}\). By minimizing the cross entropy between \(T\) and \(H^{(s)}\) (Eq.11), \(\mathcal{L}_{clsc}\) compact paths of different length between samples within the same class. Here, \(S_{m}\) is a hyper-parameter to control the maximum length

Figure 4: A demonstration of the CLSC process. (a) represents the feature extraction step; (b)\(\rightarrow\)(h) shows the traditional type classification process (each color represents one candidate type), where suboptimal classifiers make predictions for each mention and misclassifies A into the Blue type; (c)\(\rightarrow\)(d)\(\rightarrow\)(e)\(\rightarrow\)(f)\(\rightarrow\)(g) demonstrates the process of CLSC as described in Section 3. Through label propagation and compact clustering, our model is able to group mentions of the same type into a dense region and leaves clear separation boundaries in sparse regions.

 

of Markov chain. \({\cal L}_{clsc}\) is added to the final objective function as regularization to encourage compact cluttering.

### Overall Objective

Given the representation of a mention, the type posterior is given by a standard softmax classifier parameterized by \(\theta_{g}\):

\[P(\hat{y_{i}}|z_{i};\theta_{g})=softmax(W_{c}z_{i}+b_{c}), \tag{12}\]

where \(W_{c}\in R^{K\times d_{z}}\) is a parameter matrix, \(b\in R^{K}\) is the bias vector, where \(K\) is the number of types. The predicted type is then given by \(\hat{t_{i}}=argmax_{y_{i}}P(\hat{y_{i}}|z_{i};\theta_{g})\).

Our loss function consists of two parts. \({\cal L}_{sup}\) is supervision loss defined by KL divergence:

\[L_{sup}=-\frac{1}{B_{c}}\sum_{i=1}^{B_{c}}\sum_{k=1}^{K}y_{ik}log(P(y_{i}|z_{ i};\theta_{g}))_{k} \tag{13}\]

Here \(B_{c}\) is the number of clean data in a training batch, K is the number of target types. The regularization term is given by \({\cal L}_{clsc}\). Hence, the overall loss function is:

\[{\cal L}_{final}={\cal L}_{sup}+\lambda_{clsc}\times{\cal L}_{clsc} \tag{14}\]

\(\lambda_{clsc}\) is a hyper parameter to control the influence of CLSC.

## 4 Experiments

### Dataset

We evaluate our method on two standard benchmarks: OntoNotes and BBN:

* **OntoSues:** The OntoNotes dataset is composed of sentences from the Newswire part of OntoNotes corpus Weischedel et al. (2013). Gillick et al. (2014) annotated the training part with the aid of DBpedia spotlight Daiber et al. (2013), while the test data is manually annotated.
* **BBN:** The BBN dataset is composed of sentences from Wall Street Journal articles and is manually annotated by Weischedel and Brunstein (2005). Ren et al. (2016a) regenerated the training corpus via distant supervision.

In this work we use the preprocessed datasets provided by Abhishek et al. (2017); Xu and Barbosa (2018). Table 2 shows detailed statistics of the datasets.

### Compared Methods

We compare the proposed method with several state-of-the-art FET systems3:

Footnote 3: The baselines result are reported on Abhishek et al. (2017); Xu and Barbosa (2018) in addition to performance of NFETC on BBN, we search the hyper parameters for it. Xu and Barbosa (2018) didn’t report the results on BBN

* **Attentive**Shimaoka et al. (2016) uses an attention based feature extractor and doesn't distinguish clean from noisy data;
* **AFET**Ren et al. (2016a) trains label embedding with partial label loss;
* **AAA**Abhishek et al. (2017) learns joint representation of mentions and type labels;
* **PLE+HYENA/FIGER**Ren et al. (2016b) proposes heterogeneous partial-label embedding for label noise reduction to boost typing systems. We compare two PLE models with HYENA Yogatama et al. (2015) and FIGER Ling and Weld (2012) as the base typing system respectively;
* **NFETC**Xu and Barbosa (2018) trains neural fine-grained typing system with hierarchy-aware loss. We compare the performance of the NFETC model with two different loss functions: partial-label loss and **PLL+**hierarchical loss. We denote the two variants as **NFETC** and **NFETC\({}_{hier}\)** respectively;
* **NFETC-CLSC** is the proposed model in this work. We use the NFETC model as our base model, based on which we apply Compact Latent Space Clustering Regularization as described in Section 3.2; Similarly, we report results produced by using both KL-divergense-based loss (**NFETC-CLSC**) and **KL+**hierarchical loss (**NFETC-CLSC\({}_{hier}\)**).

### Evaluation Settings

For evaluation metrics, we adopt strict accuracy, loose macro, and loose micro F-scores widely used in the FET task Ling and Weld (2012). To fine tuning the hyper-parameters, we randomly sampled 10% of the test set as a development set for both datasets. With the fine-tuned hyperparameter as mentioned in 4.4, we run the model five times and report the average strict accuracy, macro F1 and micro F1 on the test set.

 



### Hyper Parameters

We search the hyper parameter of Ontonotes and BBN respectively via Hyperopt proposed by [3]. Hyper parameters are shown in Appendix A. We optimize the model via Adam Optimizer. The full hyper parameters includes the learning rate \(lr\), the dimension \(d_{p}\) of word position embedding, the dimension \(d_{l}\) of the mention encoder's output (equal to the dimension of the context encoder's output), the input dropout keep probability \(p_{i}\) and output dropout keep probability \(p_{o}\) for LSTM layers (in context encoder and LSTM mention encoder), the L2 regularization parameter \(\lambda\), the factor of hierarchical loss normalization \(\alpha\) (\(\alpha>0\) means use the normalization), BN (whether using Batch normalization), the max step \(S_{lp}\) of the label propagation, the max length \(S_{m}\) of Markov chain, the influence parameter \(\lambda_{clsc}\) of CLSC, the batch size \(B\), the number \(n\) of hidden layers in \(q\) and the number \(h_{n}\) of hidden units of the hidden layers. We implement all models using Tensorflow4.

Footnote 4: The code for experiments is available at [https://github.com/herbertchen1/NFETC-CLSC](https://github.com/herbertchen1/NFETC-CLSC)

### Performance comparison and analysis

Table 1 shows performance comparison between the proposed CLSC model and state-of-the-art FET systems. On both benchmarks, the CLSC model achieves the best performance in all three metrics. When focusing on the comparison between **NFETC** and CLSC, we have following observation:

* Compact Latent Space Clustering shows its effectiveness on both clean data and noisy data. By applying CLSC regularization on the basic **NFETC** model, we observe consistent and significant performance boost;
* Hierarchical-aware loss shows significant advantage on the OntoNotes dataset, while showing insignificant performance boost on the BBN dataset. This is due to different distribution of labels on the test set. The proportion of terminal types of the test set is \(69\%\) for the BBN dataset, while is only \(33\%\) on the OntoNotes dataset. Thus, applying hierarchical-aware loss on the BBN dataset brings little improvement;
* Both algorithms are able to utilize noisy data to improve performance, so we would like to further study their performance in different noisy scenarios in following discussions.

\begin{table}
\begin{tabular}{|l l c c c c c c|} \hline \multirow{2}{*}{**Method**} & \multicolumn{5}{c}{**OntoNotes**} & \multicolumn{5}{c|}{**BBN**} \\ \cline{2-8}  & **Strict Acc.** & **Macro F1** & **Micro F1** & **Strict Acc.** & **Macro F1** & **Micro F1** \\ \hline
**AFET**[7] & 55.3 & 71.2 & 64.6 & 68.3 & 74.4 & 74.7 \\
**AAA**[1] & 52.2 & 68.5 & 63.3 & 65.5 & 73.6 & 75.2 \\
**Attentive**[1] & 51.7 & 71.0 & 64.91 & 48.4 & 73.2 & 72.4 \\
**PLE+HYENA**[7] & 54.6 & 69.2 & 62.5 & 69.2 & 73.1 & 73.2 \\
**PLE+FIGER**[7] & 57.2 & 71.5 & 66.1 & 68.5 & 77.7 & 75.0 \\ \hline \hline \multirow{2}{*}{**NFETC**} & \(clean\) & 54.4\(\pm\)0.3 & 71.5\(\pm\)0.4 & 64.9\(\pm\)0.3 & 71.2\(\pm\)0.2 & 77.1\(\pm\)0.3 & 76.9\(\pm\)0.3 \\ \cline{2-8}  & \(+noisy\) & 54.8\(\pm\)0.4 & 71.8\(\pm\)0.4 & 65.0\(\pm\)0.4 & 73.8\(\pm\)0.6 & 78.4\(\pm\)0.6 & 78.9\(\pm\)0.6 \\ \hline \multirow{2}{*}{**NFETC\({}_{bier}\)**} & \(clean\) & 59.6\(\pm\)0.2 & 76.1\(\pm\)0.2 & 69.7\(\pm\)0.2 & 70.3\(\pm\)0.3 & 76.8\(\pm\)0.3 & 76.6\(\pm\)0.2 \\ \cline{2-8}  & \(+noisy\) & 60.2\(\pm\)0.2 & 76.4\(\pm\)0.1 & 70.2\(\pm\)0.2 & **73.9\(\pm\)1.2** & 78.8\(\pm\)1.2 & **79.4\(\pm\)1.1** \\ \hline \hline \multirow{2}{*}{**NFETC-CLSC**} & \(clean\) & 59.1\(\pm\)0.4 & 75.3\(\pm\)0.3 & 69.1\(\pm\)0.3 & 73.0\(\pm\)0.3 & 79.0\(\pm\)0.3 & 78.8\(\pm\)0.3 \\ \cline{2-8}  & \(+noisy\) & 59.6\(\pm\)0.2 & 75.5\(\pm\)0.4 & 69.3\(\pm\)0.4 & **74.7\(\pm\)0.3** & **80.7\(\pm\)0.2** & **80.5\(\pm\)0.2** \\ \hline \multirow{2}{*}{**NFETC-CLSC\({}_{bier}\)**} & \(clean\) & 61.5\(\pm\)0.3 & 77.4\(\pm\)0.3 & 71.4\(\pm\)0.4 & 70.5\(\pm\)0.2 & 78.2\(\pm\)0.2 & 78.0\(\pm\)0.2 \\ \cline{2-8}  & \(+noisy\) & **62.8\(\pm\)0.3** & **77.8\(\pm\)0.4** & **72.0\(\pm\)0.4** & 71.9\(\pm\)0.3 & 79.8\(\pm\)0.4 & 79.5\(\pm\)0.3 \\ \hline \end{tabular}
\end{table}
Table 1: Performance comparision of FET systems on the two datasets.

\begin{table}
\begin{tabular}{|l|l|l|} \hline  & **OntoNotes** & **BBN** \\ \hline
**#types** & 89 & 47 \\ \hline
**Max hierarchy depth** & 3 & 2 \\ \hline
**\#mentions-train** & 253241 & 86078 \\ \hline
**fmentions-test** & 8963 & 12845 \\ \hline
**\%clean mentions-train** & 73.13 & 75.92 \\ \hline
**\%clean mentions-test** & 94.00 & 100 \\ \hline
**Average**\(|Y^{t}_{i}|\) & 1.40 & 1.26 \\ \hline \end{tabular}
\end{table}
Table 2: Detailed statistics of the two datasets.





### How robust are the methods to the proportion of noisy data?

By principle, with sufficient amount of clean training data, most typing systems can achieve satisfying performance. To further study the robustness of the methods to label noise, we compare their performance with the presence of \(25\%,20\%,15\%,10\%\) and \(5\%\) clean training data and all noisy training data. Figure 5 shows the performance curves as the proportion of clean data drops. As it reveals, the CLSC model consistently wins in the comparison. The advantage is especially clear on the BBN dataset, which offers less amount of training data. Note that, with only \(27.9\%\) of training data (when only leaving \(5\%\) clean data) on the BBN dataset, the CLSC model yield a comparable result with the **NFETC** model trained on full data. This comparison clearly shows the superiority of our approach in the effectiveness of utilizing noisy data.

### Ablation: Do Markov Chains improve typing performance?

Table 3 shows the performance of CLSC with one-step transition (\({\cal L}_{1-step}\)) and with Markov Chains (\({\cal L}_{dsc}\)) as described in Section 3.2. Results show that the use of Markov Chains does bring improvement to the overall performance, which is consistent with the model intuition.

## 5 Related Work

Named entity Recognition (NER) has been excavated for a long time Collins and Singer (1999); Manning et al. (2014), which classifies coarse-grained types (e.g. person, location). Recently, Nagesh and Surdeanu (2018a, b) applied ladder network Rasmus et al. (2015) to coarse-grained entity classification in a semi-supervised learning fashion. Ling and Weld (2012) proposed Fine-Grained Entity Recognition (FET). They used distant supervision to get training corpus for FET. Embedding techniques was applied to learn feature representations since Yogatama et al. (2015); Dong et al. (2015). Shimaoka et al. (2016) introduced attention mechanism for FET to capture informative words. Xin et al. (2018) used the TransE entity embeddings Bordes et al. (2013) as the query vector of attention.

Early works ignore the out-of-context noise, Gillick et al. (2014) proposed context dependent FET and use three heuristics to clean the noisy labels with the side effect of losing training data. To utilize noisy data, Ren et al. (2016) distinguished the loss function of noisy data from clean data via partial label loss (**PLL**). Abhishek et al. (2017); Xu and Barbosa (2018) proposed variants of **PLL**, which still suffer from confirmation bias. Xu and Barbosa (2018) proposed hierarchical loss to handle over-specific noise. On top of **AFET**, Ren et al. (2016) proposed a method **PLE** to reduce the label noise, which lead to a great success in FET. Because label noise reduction is separated from the learning of FET, there might be error propagation problem. Recently, Xin et al. (2018) proposed utilizing a pretrained language model measures the compatibility between context and type names, and use it to repel the interference of noisy labels. However, the compatibility got by language model may not be right and type information is defined by corpus and annotation guidelines rather than type names as is mentioned in Azad et al. (2018). In addition, there are some work about entity-level typing which aim to figure out the types of entities in KB Yaghoobzadeh and Schutze (2015); Jin et al. (2018).

Figure 5: Performance comparison between NFETC-CLSC and NFETC by removing \(75\%\)-\(95\%\) clean data.





## 6 Conclusion

In this paper, we propose a new method for distantly supervised fine-grained entity typing, which leverages imperfect annotations as model regularization via Compact Latent Space Clustering (CLSC). Experiments on two standard benchmarks demonstrate that our method consistently outperforms state-of-the-art models. Further study reveals our method is more robust than the former state-of-the-art approach as the portion of noisy data rises. The proposed method is general for other tasks with imperfect annotation. As a part of future investigation, we plan to apply the approach to other distantly supervised tasks, such as relation extraction.

## 7 Acknowledgments

This work has been supported in part by NSFC (No.61751209, U1611461), Zhejiang University-iFLYTEK Joint Research Center, Chinese Knowledge Center of Engineering Science and Technology (CKCEST), Engineering Research Center of Digital Library, Ministry of Education. Xiang Ren's research has been supported in part by National Science Foundation SMA 18-29268.

## References

* Abhishek et al. (2017) Abhishek Abhishek, Ashish Anand, and Amit Awekar. 2017. Fine-grained entity type classification by jointly learning representations and label embeddings. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, volume 1, pages 797-807.
* Azad et al. (2018) Amar Prakash Azad, Balaji Ganesan, Ashish Anand, Amit Awekar, et al. 2018. A unified labeling approach by pooling diverse datasets for entity typing. _arXiv preprint arXiv:1810.08782_.
* Bergstra et al. (2013) James Bergstra, Dan Yamins, and David D Cox. 2013. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In _Proceedings of the 12th Python in Science Conference_, pages 13-20. Citeseer.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In _Advances in neural information processing systems_, pages 2787-2795.
* Collins and Singer (1999) Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In _1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora_.
* Daiber et al. (2013) Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In _Proceedings of the 9th International Conference on Semantic Systems_, pages 121-124. ACM.
* Dong et al. (2015) Li Dong, Furu Wei, Hong Sun, Ming Zhou, and Ke Xu. 2015. A hybrid neural model for type classification of entity mentions. In _IJCAI_, pages 1243-1249.
* Dong et al. (2014) Xin Dong, Evgeniy Gabrilovich, Jeremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 601-610. ACM.
* Gillick et al. (2014) Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. 2014. Context-dependent fine-grained entity type tagging. _arXiv preprint arXiv:1412.1820_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Jin et al. (2018) Hailong Jin, Lei Hou, Juanzi Li, and Tiansi Dong. 2018. Attributed and predictive entity embedding for fine-grained entity typing in knowledge bases. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 282-292.
* Kamnitsas et al. (2018) Konstantinos Kamnitsas, Daniel Castro, Loic Le Folgoc, Ian Walker, Ryutaro Tanno, Daniel Rueckert, Ben Glocker, Antonio Criminisi, and Aditya Nori. 2018. Semi-supervised learning via compact latent space clustering. In _Proceedings of the 35th International Conference on Machine Learning_, volume 80 of _Proceedings of Machine Learning Research_, pages 2459-2468, Stockholmsmassan, Stockholm Sweden. PMLR.
* Laine and Aila (2017) Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In _Proc. International Conference on Learning Representations (ICLR)_.
* Lee et al. (2006) Changki Lee, Yi-Gyu Hwang, Hyo-Jung Oh, Soojong Lim, Jeong Heo, Chung-Hee Lee, Heyon-Jin Kim, Ji-Hyun Wang, and Myung-Gil Jang. 2006. Fine-grained named entity recognition using conditional random fields for question answering. In _Asia Information Retrieval Symposium_, pages 581-587. Springer.

\begin{table}
\begin{tabular}{|l|c|} \hline  & Strict Acc. \\ \hline
**CLSC(e)/\(\mathcal{L}_{1-step}\)** & 72.0\(\pm\)0.1 \\ \hline
**CLSC(e)/\(\mathcal{L}_{close}\)** & 73.0\(\pm\)0.3 \\ \hline
**CLSC(e+n)(\(\mathcal{L}_{1-step}\))** & 73.0\(\pm\)0.1 \\ \hline
**CLSC(c+n)(\(\mathcal{L}_{csc}\))** & 74.7\(\pm\)0.3 \\ \hline \end{tabular}
\end{table}
Table 3: The comparison of \(\mathcal{L}_{1-step}\) and \(\mathcal{L}_{clsc}\) on BBN.



Xiao Ling and Daniel S Weld. 2012: Fine-grained entity recognition. In AAAI, Vol. 12, pp. 94-100. Cited by: SS3.
* C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky (2014)The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pp. 55-60. Cited by: SS3.
* M. Rintz, S. Bills, R. Snow, and D. Jurafsky (2009)Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pp. 1003-1011. External Links: Link, Document Cited by: SS3.
* A. Nagesh and Mihai Surdeanu (2018a)Are exploration of three lightly-supervised representation learning approaches for named entity classification. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 2312-2324. Cited by: SS3.
* A. Nagesh and M. Surdeanu (2018b)Keep your bearings: lightly-supervised information extraction with ladder networks that avoids semantic drift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Vol. 2, pp. 352-358. Cited by: SS3.
* J. Pennington, R. Socher, and C. D. Manning (2014)Glove: global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543. Cited by: SS3.
* J. Raiman and O. Raiman (2018)Deep-type: multilingual entity linking by neural type system evolution. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artifical Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 5406-5413. Cited by: SS3.
* A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko (2015)Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, pp. 3546-3554. Cited by: SS3.
* X. Ren, W. He, M. Qu, L. Huang, H. Ji, and J. Han (2016)Afet: automatic fine-grained entity typing by hierarchical partial-label embedding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1369-1378. Cited by: SS3.
* X. Ren, W. He, M. Qu, C. R. Voss, H. Ji, and J. Han (2016)Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1825-1834. Cited by: SS3.
* S. Shimaoka, P. Stenetorp, K. Inui, and S. Riedel (2016)An attentive neural architecture for fine-grained entity type classification. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, pp. 69-74. Cited by: SS3.
* A. Tarvainen and H. Valpola (2017)Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204. Cited by: SS3.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008. Cited by: SS3.
* R. Weischedel and A. Brunstein (2005)Bbn pronoun coreference and entity type corpus. Linguistic Data Consortium, Philadelphia. Cited by: SS3.
* R. Weischedel, M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, N. Xue, A. Taylor, J. Kaufman, M. Franchini, et al. (2013)Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia PA. Cited by: SS3.
* J. Xin, Y. Lin, Z. Liu, and M. Sun (2018a)Improving neural fine-grained entity typing with knowledge attention. Cited by: SS3.
* J. Xin, H. Zhu, X. Han, Z. Liu, and M. Sun (2018b)Put it back: entity typing with language model enhancement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 993-998. Cited by: SS3.
* P. Xu and D. Barbosa (2018)Neural fine-grained entity type classification with hierarchy-aware loss. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Vol. 1, pp. 16-25. Cited by: SS3.
* Y. Yaghoobzadeh and H. Schutze (2015)Corpus-level fine-grained entity typing using contextual information. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 715-725. Cited by: SS3.
* D. Yogatama, D. Gillick, and N. Lazic (2015)Embedding methods for fine grained entity type classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Vol. 2, pp. 291-296.

 

Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_, pages 2335-2344.
* Zhu and Ghahramani (2002) Xiaojin Zhu and Zoubin Ghahramani. 2002. Learning from labeled and unlabeled data with label propagation.

## Appendix A Hyper parameters

\begin{table}
\begin{tabular}{|l r|r|r|} \hline  & **Ont.(C)** & **BBN(C)** & **BBN(N)** \\ \hline \(lr\) & 0.0006 & 0.0007 & 0.0007 \\ \hline \(d_{p}\) & 70 & 40 & 20 \\ \hline \(d_{l}\) & 1000 & 1000 & 240 \\ \hline \(p_{i}\) & 0.7 & 0.3 & 0.5 \\ \hline \(p_{o}\) & 0.6 & 1.0 & 0.4 \\ \hline \(\lambda\) & 0.0000 & 0.0000 & 0.0002 \\ \hline \(\alpha\) & 0.25/0.0 & 0.4/0.0 & 0.4/0.0 \\ \hline BN & FALSE & FALSE & TRUE \\ \hline \(S_{lp}\) & 200 & 200 & - \\ \hline \(S_{m}\) & 8 & 12 & - \\ \hline \(\lambda_{clsc}\) & 2.0 & 1.5 & - \\ \hline \(B\) & 512 & 512 & 512 \\ \hline \(n\) & 2 & 1 & - \\ \hline \(h_{n}\) & 700 & 560 & - \\ \hline \end{tabular}
\end{table}
Table 4: Hyper parameters of our experiments: (C) denotes CLSC, (N) denotes the hyper parameter is used for NFETC.