Venue: EMNLP 2023

MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition

Shuhui Wu\({}^{1}\) and Yongliang Shen\({}^{1\dagger}\) &Zeqi Tan\({}^{1}\)

Wenqi Ren\({}^{2}\) &Jietian Guo\({}^{2}\) &Shiliang Pu\({}^{2}\) &Weiming Lu\({}^{1\dagger}\)

\({}^{1}\)College of Computer Science and Technology, Zhejiang University, China

\({}^{2}\) Hikvision Research Institute

{shwu, syl,zqtan}@zju.edu.cn

{renwenqi,guojjietian,pushiliang.hri}@hikvision.comluwm@zju.edu.cn

###### Abstract

Distantly supervised named entity recognition (DS-NER) aims to locate entity mentions and classify their types with only knowledge bases or gazetteers and unlabeled corpus. However, distant annotations are noisy and degrade the performance of NER models. In this paper, we propose a noise-robust prototype network named MProto for the DS-NER task. Different from previous prototype-based NER methods, MProto represents each entity type with multiple prototypes to characterize the intra-class variance among entity representations. To optimize the classifier, each token should be assigned an appropriate ground-truth prototype and we consider such token-prototype assignment as an optimal transport (OT) problem. Furthermore, to mitigate the noise from incomplete labeling, we propose a novel denoised optimal transport (DOT) algorithm. Specifically, we utilize the assignment result between Other class tokens and all prototypes to distinguish unlabeled entity tokens from true negatives. Experiments on several DS-NER benchmarks demonstrate that our MProto achieves state-of-the-art performance. The source code is now available on Github1.

Footnote 1: [https://github.com/XiPotatonium/mproto](https://github.com/XiPotatonium/mproto)

+
Footnote †: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2361–2374

December 6-10, 2023 ©2023 Association for Computational Linguistics

## 1 Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing and is essential in various downstream tasks Li and Ji (2014); Miwa and Bansal (2016); Ganea and Hofmann (2017); Shen et al. (2021). Most state-of-the-art NER models are based on deep neural networks and require massive annotated data for training. However, human annotation is expensive, time-consuming, and often unavailable in some specific domains. To mitigate the reliance on human annotation, distant supervision is widely applied to automatically generate annotated data with the aid of only knowledge bases or gazetteers and unlabeled corpus.

Despite its easy accessibility, the distantly-annotated data is rather noisy, severely degrading the performance of NER models. We observe two types of issues in the distantly-supervised named entity recognition task. The first issue is **incomplete labeling**. Due to the limited coverage of dictionaries, entities not presented in dictionaries are mistakenly labeled as Other class (denoted as O class). For instance, in Figure 1, "Micheal" is not covered in the dictionary and thus cannot be correctly labeled by distant supervision. These unlabeled entities will misguide NER models to overfit the label noise, particularly hurting recall. The second issue is **intra-class variance**. As illustrated in Figure 1, tokens of the same class (e.g., "MainStay" and "Funds") may fall into different sub-clusters in feature space due to semantic differences. Traditional single-prototype classifiers do not consider the semantic difference within the same entity type. They set only one prototype for each type, which suffers from intra-class variance.

Various methods have been proposed to mitigate the noise of incomplete labeling. For example, AutoNER Shang et al. (2018) tries to modify the standard CRF-based classifier to adapt to the noisy NER datasets. Self-learning-based methods Liang

Figure 1: The distant annotations are marked in orange, and the human annotations are marked in blue. Here we illustrate two types of issues in the DS-NER. First, unlabeled entities will introduce incomplete labeling noise in O class. Second, tokens of the same class may fall into different sub-clusters due to semantic differences.

 

et al., 2020; Meng et al., 2021; Zhang et al., 2021c) learn by soft labels generated from a teacher model for denoising. Positive-unlabeled learning methods treat tokens of 0 class as unlabeled samples and optimize with an unbiasedly estimated empirical risk Peng et al. (2019); Zhou et al. (2022). Some other studies utilize negative sampling to avoid NER models overfitting label noise in 0 class Li et al. (2021, 2022). Despite these efforts, the issue of intra-class variance has not yet been studied in previous DS-NER research.

Unlike previous methods, we propose a novel prototype-based network named MProto for distantly-supervised named entity recognition. In MProto, each class is represented by multiple prototypes so that each prototype can represent a certain sub-cluster of an entity type. In this way, our MProto can characterize the intra-class variance. To optimize our multiple-prototype classifier, we assign each input token with an appropriate ground-truth prototype. We formulate such token-prototype assignment problem as an optimal transport (OT) problem. Moreover, we specially design a denoised optimal transport (DOT) algorithm for tokens labeled as 0 class to alleviate the noise of incomplete labeling. Specifically, we perform the assignment between 0 tokens and all prototypes and regard 0 tokens assigned to 0 prototypes as true negatives while tokens assigned to prototypes of entity classes as label noise. Based on our observation, before overfitting, unlabeled entity tokens tend to be assigned to prototypes of their actual classes in our similarity-based token-prototype assignment. Therefore, unlabeled entity tokens can be discriminated from clean samples by our DOT algorithm so as not to misguide the training.

Our main contributions are three-fold:

* We present MProto for the DS-NER task. MProto represents each entity type with multiple prototypes for intra-class variance. And we model the token-prototype assignment problem as an optimal transport problem.
* To alleviate the noise of incomplete labeling, we propose the denoised optimal transport algorithm. Tokens labeled as 0 but assigned to non-0 prototypes are considered as label noise so as not to misguide the training.
* Experiments on various datasets show that our method achieves SOTA performance. Further analysis validates the effectiveness of our multiple-prototype classifier and denoised optimal transport.

## 2 Method

In this section, we first introduce the task formulation in Section 2.1. Then, we present our MProto network in Section 2.2. To compute the cross-entropy loss, we assign each token with a ground-truth prototype, and the token-prototype assignment will be discussed in Section 2.3. Besides, to mitigate the incomplete labeling noise in 0 tokens, we propose the denoised optimal transport algorithm which will be specified in Section 2.4. Figure 2 illustrates the overview of our MProto.

### Task Formulation

Following the tagging-based named entity recognition paradigm, we denote the input sequence as \(X=[x_{1},\cdots,x_{L}]\) and the corresponding distantly-annotated tag sequence as \(Y=[y_{1},\ldots,y_{L}]\). For each token-label pair \((x_{i},y_{i})\), \(x_{i}\) is the input token and \(y_{i}\in\mathcal{C}=\{c_{1},\dots,c_{K}\}\) is the class of the token. Here we let \(c_{1}\) be 0 class and others be predefined entity classes. We denote the human-annotated tag sequence as \(\hat{Y}=[\tilde{y}_{1},\ldots,\tilde{y}_{L}]\). Human annotations can be considered as true labels. In the distantly-supervised NER task, only distant annotations can be used for training while human annotations are only available in evaluation.

### Multi-Prototype Network

Token Encoding.For each input token sequence \(X\), we generate its corresponding features with a pretrained language model:

\[\mathbf{H}=f_{\theta}(X) \tag{1}\]

where \(f_{\theta}\) is the pretrained language model parameterized by \(\theta\) and \(\mathbf{H}=[\mathbf{h}_{1},\ldots,\mathbf{h}_{L}]\) is the features of the token sequence.

Prototype-based Classification.MProto is a prototype-based classifier where predictions are made based on token-prototype similarity. Visualization in Figure 4 shows that tokens of the same class will form multiple sub-clusters in the feature space due to the semantic difference. However, previous prototype-based classifiers which represent each entity type with only a single prototype cannot characterize such intra-class variance. To this end, we represent each entity type with multiple prototypes. Specifically, for each class \(c\), let 

\({\cal P}_{c}=\{{\bf p}_{c,1},\ldots,{\bf p}_{c,M}\}\) be the set of \(M\) prototypes representing class \(c\). For the classification, we compare the similarity between token features and all prototypes, and the class of the most similar prototype is chosen as the prediction:

\[\hat{c}_{i}=\operatorname*{arg\,max}_{c}{\rm s}({\bf h}_{i},{\bf p}_{c,m}) \tag{2}\]

where \({\rm s}({\bf h},{\bf p})=\frac{{\bf h}\cdot{\bf p}}{||{\bf h}||\cdot||{\bf p}|}\) is the similarity metric and here we adopt cosine similarity. In the inference process, consecutive tokens of the same type are considered a singular entity.

Loss.To update the parameters of our MProto, we calculate the loss between token features and their corresponding ground-truth prototypes. First, we should assign each token with an appropriate ground-truth prototype. Specifically, for each token \(i\) and its annotation \(y_{i}=c_{i}\), one of the prototypes of class \(c_{i}\) will be assigned as the ground truth based on the similarity between the token feature and prototypes. Such token-prototype assignment is considered as an optimal transport problem, and the detail of solving the token-prototype assignment will be discussed later (in Section 2.3). Based on the assigned ground-truth prototype \({\bf p}_{c_{i},m}\) for token \(i\), we can compute the cross-entropy loss:

\[\ell^{\text{CE}}=-\sum_{i}\log\frac{\exp({\rm s}({\bf h}_{i},{\bf p}_{c_{i}, m}))}{\sum_{c^{\prime},m^{\prime}}\exp({\rm s}({\bf h}_{i},{\bf p}_{c^{\prime} ,m^{\prime}}))} \tag{3}\]

However, optimizing the model only through CE loss guides tokens to be **relatively** close to their ground-truth prototypes in feature space, while the compactness of the token features within the same sub-cluster is not considered. To this end, we further optimize the absolute distance between token features and ground-truth prototypes as follows:

\[\ell^{\text{c}}=\sum_{i}{\rm d}^{2}({\bf h}_{i},{\bf p}_{c_{i},m})=\sum_{i}(1 -{\rm s}({\bf h}_{i},{\bf p}_{c_{i},m}))^{2} \tag{4}\]

here we define the distance based on cosine similarity as \({\rm d}({\bf h}_{i},{\bf p}_{c_{i},m})=1-{\rm s}({\bf h}_{i},{\bf p}_{c_{i},m})\). The overall loss can be calculated as follows:

\[\ell=\ell^{\text{CE}}+\lambda^{\text{c}}\ell^{\text{c}} \tag{5}\]

where \(\lambda^{\text{c}}\) is the weight of the compactness regularization.

Prototype Updating.We update each prototype with the token features assigned to that prototype. For convenience, we denote the set of tokens assigned to prototype \(p_{c,m}\) as \({\cal T}\). At each training step \(t\), we update prototypes with exponential moving average (EMA):

\[{\bf p}_{c,m}^{t}=\alpha{\bf p}_{c,m}^{t-1}+(1-\alpha)\frac{\sum_{i\in{\cal T }}{\bf h}_{i}}{|{\cal T}|} \tag{6}\]

where \(\alpha\) is the EMA updating ratio. With EMA updating, the learned prototypes can be considered

Figure 2: The overview of MP Proto. For the clarity, we only show tokens of three classes (PER, ORG and 0) and MProto with two prototypes per class. MProto is a prototype-based classifier where predictions are made based on the similarity between token features and prototypes. To optimize the multiple-prototype classifier, we assign each token an appropriate ground-truth prototype. The top right of the figure illustrates the token-prototype assignment for entity tokens which is considered as an OT problem. The bottom right of the figure illustrates the token-prototype assignment for 0 tokens which is solved by our DOT algorithm where 0 tokens assigned with non-0 prototypes (in dashed line) are considered as label noise. The assignment is done based on cost matrix (the tables shown in the figure) where each cell is the distance between a token and a prototype.

 

as the representative of a certain sub-cluster in the feature space.

### Token-Prototype Assignment for Entity Tokens

In our MProto, each class is represented by multiple prototypes. Therefore, how to assign a token with an appropriate ground-truth prototype is an essential issue for optimization. First, we denote the set of tokens labeled as class \(c\) as \({\mathcal{T}_{c}}\). We aim to compute the assignment matrix \(\gamma^{c}\in\mathbb{R}^{|{\mathcal{T}_{c}}|\times M}\) between \({\mathcal{T}_{c}}\) and \({\mathcal{P}_{c}}\) (prototypes of class \(c\)). Then the assigned prototype \(p_{c_{i},m}\) for token \(i\) can be obtained by \(m=\arg\max_{j}\gamma^{c}_{i,j}\). We consider such token-prototype assignment problem as an optimal transport problem:

\[\hat{\gamma^{c}}=\operatorname*{arg\,min}_{\gamma^{c}}\sum_{i\in {\mathcal{T}_{c}}}\sum_{j=1}^{M}\gamma^{c}_{i,j}{\mathbf{C}^{c}_{i,j}},\] (7) s.t. \[\gamma^{c}{\bf 1}={\bf a},\gamma^{c}{\bf 1}={\bf b}\]

where \(C^{c}_{i,j}=\mathrm{d}({\bf h}_{i},{\bf p}_{c,j})\) is the cost matrix which is composed of the distances between features of the token set \({\mathcal{T}_{c}}\) and prototype set \({\mathcal{P}_{c}}\), \({\bf a}={\bf 1}\in\mathbb{R}^{|{\mathcal{T}_{c}}|}\) is the assignment constraint for each token which guarantees that each token is expected to be assigned to exactly one prototype, \({\bf b}\in\mathbb{R}^{M}\) is the assignment constraint for each prototype which prevents the model from the trivial solution where all tokens are assigned to a single prototype. The constraint can be set based on prior knowledge. However, to keep our method simple, we simply choose even distribution (\({\bf b}=\frac{|{\mathcal{T}_{c}}|}{M}{\bf 1}\)). The experiments also show that such simple constraint can already achieve satisfactory performance.

By optimizing Equation 7, we aim to obtain an assignment where each token is assigned a similar prototype. The token-prototype assignment problem can be solved by the sinkhorn-knopp algorithm Cuturi (2013), which is detailed in Appendix A.

### Token-Prototype Assignment for O Tokens

Incomplete labeling noise is a common issue in the distantly supervised named entity recognition task. If we directly assign all 0 tokens to 0 prototypes, features of unlabeled entity tokens will be pulled closer to 0 prototypes and farther to prototypes of their actual entity type, which leads to overfitting. To mitigate the noise of incomplete labeling, we specially design the denoised optimal transport algorithm for 0 tokens. Specifically, we allow all prototypes to participate in the assignment of 0 tokens. Here we denote the set of tokens labeled as 0 class as \({\mathcal{T}_{o}}\) and the assignment matrix between \({\mathcal{T}_{o}}\) and all prototypes as \(\gamma^{o}\in\mathbb{R}^{|{\mathcal{T}_{o}}|\times KM}\). The denoised optimal transport is formulated as follows:

\[\begin{split}\hat{\gamma^{o}}&=\operatorname*{ arg\,min}_{\gamma^{o}}\sum_{i\in{\mathcal{T}_{o}}}\sum_{j=1}^{KM}\gamma^{ o}_{i,j}{\mathbf{C}^{\text{all}}_{i,j}},\\ \text{s.t.}&~{}~{}~{}\gamma^{o}{\bf 1}={\bf a}, \gamma^{o}{\bf 1}={\bf b}\end{split} \tag{8}\]

where \({\mathbf{C}^{\text{all}}}=[{\mathbf{C}^{c_{1}}},\ldots,{\mathbf{C}^{c_{K}}}] \in\mathbb{R}^{|{\mathcal{T}_{o}}|\times KM}\) is the cost matrix composed of the distances between features of the token set \({\mathcal{T}_{o}}\) and all prototypes, and \([\cdot]\) is the concatenation operation. The first constraint \({\bf a}={\bf 1}\in\mathbb{R}^{|{\mathcal{T}_{o}}|}\) indicates that each token is expected to be assigned to exactly one prototype. The second constraint is formulated as:

\[{\bf b}=[\frac{\beta|{\mathcal{T}_{o}}|}{M}{\bf 1},\underbrace{\frac{(1- \beta)|{\mathcal{T}_{o}}|}{(K-1)M}{\bf 1},\cdots,\frac{(1-\beta)|{\mathcal{T}_{o}}|}{( K-1)M}{\bf 1}}_{K-1}] \tag{9}\]

where \(\beta\) is the assignment ratio for 0 prototypes. It indicates that: (1) we expect \(\beta|{\mathcal{T}_{o}}|\) tokens to be assigned to 0 prototypes, (2) the remaining tokens are assigned to non-0 prototypes, (3) the token features are evenly assigned to prototypes of the same type.

Intuitively, before the model overfits the incomplete labeling noise, unlabeled entity tokens are similar to prototypes of their actual entity type in feature space. So these unlabeled entity tokens tend to be assigned to prototypes of their actual entity type in our similarity-base token-prototype assignment. Therefore, tokens assigned to 0 prototypes can be considered as true negatives while others can be considered as label noise. We then modify the CE loss in Equation 3 as follows:

\[\begin{split}\ell^{\text{CE}}=&-\sum_{i\in{\mathcal{ T}_{o}}}\log\frac{\exp(\mathrm{s}({\bf h}_{i},{\bf p}_{c_{i},m}))}{\sum_{ c^{\prime},m^{\prime}}\exp(\mathrm{s}({\bf h}_{i},{\bf p}_{c^{\prime},m^{ \prime}}))}\\ &-\sum_{i\in{\mathcal{T}_{o}}}w_{i}\log\frac{\exp(\mathrm{s}({\bf h }_{i},{\bf p}_{i}))}{\sum_{c^{\prime},m^{\prime}}\exp(\mathrm{s}({\bf h}_{i},{ \bf p}_{c^{\prime},m^{\prime}}))}\end{split} \tag{10}\]

where \({\mathcal{T}_{o}}=\bigcup_{i=2}^{M}{\mathcal{T}_{ci}}\) is the set of tokens not labeled as 0, \(w_{i}=\mathds{1}({\bf p}_{i}\in{\mathcal{P}_{c_{1}}})\) is the indicator with value \(1\) when the assigned prototype \({\bf p}_{i}\) for token \(i\) is of class \(c_{1}\) (0 class). The first term of Equation 10 is the loss for entity tokens which is identical to traditional CE loss. The second term is the loss for 0 tokens where only these true negative 

samples are considered. In this way, unlabeled entity tokens will not misguide the training of our MProto, and the noise of incomplete annotation can be mitigated.

## 3 Experiments

### Settings

NER Dataset.The distantly-annotated data is generated from two wildly used flat NER datasets: (1) CoNLL03 Tjong Kim Sang (2002) is an open-domain English NER dataset that is annotated with four entity types: PER, LOC, ORG and MISC. It is split into 14041/3250/3453 train/dev/test sentences. (2) BC5CDR Wei et al. (2016) is a biomedical domain English NER dataset which consists of 1500 PubMed articles with 4409 annotated Chemicals and 5818 Diseases. It is split into 4560/4579/4797 train/dev/test sentences.

Distant Annotation.We generate four different distantly-annotated datasets based on CoNLL03 and BC5CDR for the main experiment: (1) CoNLL03 Dict is generated by dictionary matching with the dictionary released by Peng et al. (2019). (2) CoNLL03 KB is generated by KB matching follows Liang et al. (2020). (3) BC5CDR (Big Dict) is generated by dictionary matching with the dictionary released by Shang et al. (2018). (4) BC5CDR (Small Dict) is generated by the first 20% of the dictionary used in BC5CDR (Big Dict). We follow the dictionary matching algorithm presented in Zhou et al. (2022) and the KB matching algorithm presented in Liang et al. (2020).

Evaluation Metric.We train our MProto with the distantly-annotated train set. Then the model is evaluated on the human-annotated dev set. The best checkpoint on the dev set is tested on the human-annotated test set, and the performance on the test set is reported as the final result. Entity prediction is considered correct when both span and category are correctly predicted.

Implementation Details.For a fair comparison, we use the BERT-base-cased Devlin et al. (2019) for the CoNLL03 dataset, and BioBERT-base-cased Lee et al. (2020) for the BC5CDR dataset. We set \(M=3\) prototypes per class and the hyperparameter search experiment can be found in Appendix E. More detailed hyperparameters can be found in Appendix B.

### Baselines

Fully-Supervised.We implement a fully-supervised NER model for comparison. The model is composed of a BERT encoder and a linear layer as the classification head. Fully-supervised model is trained with human-annotated data, and the performance can be viewed as the upper bound of the DS-NER models.

AutoNER.AutoNER Shang et al. (2018) is a DS-NER method that classifies two adjacent tokens as break or tie. To mitigate the noise of incomplete labeling, the unknown type is not considered in loss computation. For a fair comparison, we re-implement their method to use BERT-base-cased as the encoder for the CoNLL03 dataset and use BioBERT-base-cased for the BC5CDR dataset.

Early Stoping.Liang et al. (2020) apply early stopping to prevent the model from overfitting the label noise. For a fair comparison, we re-implement their method to use BERT-base-cased as the encoder for the CoNLL03 dataset and use BioBERT-base-cased for the BC5CDR dataset.

Negative Sampling.Li et al. (2021, 2022) use negative sampling to eliminate the misguidance of the incomplete labeling noise. We re-implement their method and set the negative sampling ratio to be \(0.3\) as recommended in Li et al. (2021).

MPU.Multi-class positive-unlabeled (MPU) learning Zhou et al. (2022) considers the negative samples as unlabeled data and optimizes with the unbiasedly estimated task risk. Conf-MPU incorporates the estimated confidence score for tokens being an entity token into the MPU risk estimator to alleviate the impact of annotation imperfection.

### Overall Performance

Table 1 shows the overall performance of our MProto compared with various baselines in four distantly-annotated datasets. We observe that our MProto achieves state-of-the-art performance on all four datasets. Specifically, our MProto achieves +1.48%, +0.39%, +0.28%, and +0.60% improvement in F1 score on BC5CDR (Big Dict), BC5CDR (Small Dict), CoNLL03 (KB), and CoNLL03 (Dict) compared with previous state-of-the-art methods. We also notice that our MProto achieves consistent improvement on all datasets. In contrast, previous SOTA methods usually achieve promising results on some datasets while performing poorly in other



cases. We attribute this superior performance to two main factors: (1) The denoised optimal transport algorithm significantly mitigates the noise of incomplete labeling in O tokens by leveraging the similarity-based token-prototype assignment result, leading to a performance improvement across varying annotation quality (with differences in dictionary coverage or Distant Supervision matching algorithms). (2) Our multiple prototype classifier can characterize the intra-class variance and can better model rich semantic entity classes, which improves the robustness across diverse data domains.

### Ablation Study

We conduct the ablation study in the following three aspects. The results are shown in Table 2.

Multiple Prototypes.By representing each entity type with multiple prototypes, the F1 score of MProto improves by +0.95% on BC5CDR (Big Dict) and by +1.26% on CoNLL03 (Dict). It confirms that the multiple-prototype classifier can greatly help the MProto characterize the intra-class variance of token features.

Compactness Loss.By applying compactness loss as elaborated in Equation 4, the F1 score of MProto improves by +2.23% on BC5CDR and by +0.29%. Simply optimizing the Cross-Entropy loss in Equation 3 only encourages the token features to be relatively close to the assigned ground-truth prototypes. While the compactness loss directly optimizes the distance between the token feature and the prototype to be small, making the token features of the same prototype more compact.

Denoised Optimal Transport.Compared with assigning all O tokens to O prototypes, the denoised optimal transport elaborated in Section 2.4 improves the F1 score of MProto by +21.89% on BC5CDR and by +34.14% on CoNLL03. The improvement indicates that the denoised optimal transport significantly mitigates the incomplete labeling noise in the DS-NER task.

### Experiments on Dictionary Coverage

To analyze the performance of our method with different coverage of dictionaries, we generate distant annotations with dictionaries of different proportions following Zhou et al. (2022). The distant annotation quality can be found in Table 3 and the experiment result is shown in Figure 3.

The result shows that the performance of Negative Sampling and Early Stopping drops significantly when the coverage of the dictionary decreases. When the dictionary size drops from 100% to 20%, the F1 score of the Negative Sampling drops by -54.48% on the BC5CDR dataset and by -47.44% on the CoNLL03 dataset. The F1 score of the Early Stopping drops by -56.13% on the BC5CDR dataset and by -51.02% on the CoNLL03 dataset. This indicates that these methods suffer severely from incomplete labeling noise. However, our MProto only suffers from a slight performance drop compared with Negative Sampling and Early Stopping. The F1 score drops only by -7.17% on BC5CDR and by -9.29% on CoNLL03. We can conclude that with multiple prototypes and denoised optimal transport, our MProto is more robust to the incomplete labeling noise and can achieve better performance, especially on the DS

\begin{table}
\begin{tabular}{l c c c} \hline \hline Model & BC5CDR (Big Dict) & BC5CDR (Small Dict) & CoNLL03 (KB) & CoNLL03 (Dict) \\ \hline BERT (Fully Sup.) & 87.45 (85.86/89.10) & 91.30 (90.82/91.78) \\ \hline DS Matching & 64.32 (86.39/51.24) & 15.69 (80.02/8.70) & 71.40 (81.31/63.75) & 63.93 (93.12/48.67) \\ AutoNER & 73.02 (78.33/68.39) & 19.90 (68.43/11.64) & 64.03 (78.17/54.21) & 59.47 (81.89/46.69) \\ Early Stopping & 73.66 (80.43/67.94) & 17.21 (75.60/ 9.71) & 77.06 (84.03/71.16) & 67.74 (86.34/55.74) \\ Neg Sampling & 78.73 (79.30/78.17) & 24.25 (78.93/14.32) & 79.30 (83.16/75.78) & 74.90 (83.71/67.78) \\ MPU & 68.22 (56.50/86.05) & 73.91 (70.08/78.18) & 65.75 (58.79/74.58) & 67.65 (63.63/72.22) \\ Conf-MPU & 77.22 (69.79/86.42) & 71.85 (81.02/64.54) & 79.16 (68.58/79.75) & 81.89 (81.71/82.08) \\ \hline MPProto & **81.47** (77.53/85.84) & **74.30** (73.41/75.22) & **79.58** (79.80/79.37) & **82.49** (84.27/80.79) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overall performance. The results are reported in “F1 (precision / recall)” format.

Figure 3: Results on the distantly-annotated dataset with different proportions of the dictionaries. “NegSamp.” stands for Negative Sampling. “ES” stands for Early Stopping. “DS” stands for dictionary matching.

 

NER datasets with low-coverage dictionaries.

### Visualization of Tokens and Prototypes

We get the token features and the embedding of the prototypes from the best checkpoint on the dev set. To visualize high-dimensional features, we convert token features and prototype embeddings to 2-dimensions by t-SNE.

As illustrated in Figure 4, we can conclude that: (1) The intra-class variance is a common phenomenon in the NER task. The token features of Disease class form two sub-clusters. Representing each entity type with only a single prototype cannot cover token features in all sub-clusters. Therefore, single-prototype networks cannot characterize the intra-class variance and the performance will suffer. (2) Our MProto can model the intra-class variance in the NER task. For each sub-cluster of Disease, there exists at least one prototype to represent that sub-cluster. In this way, the performance of the NER method can benefit from the multiple-prototype network.

More visualizations on different datasets and MProto with different prototypes can be found in Appendix C.

### Effectiveness of Denoised Optimal Transport

To validate the efficacy of our DOT algorithm, we present the similarity between tokens and prototypes over training steps in Figure 5. The token-prototype similarity of each entity type is obtained by:

\[{\text{sim}_{c}=\frac{1}{|\chi_{c}|}\sum_{\textbf{x}\in\chi_{c}}\max_{m}s( \textbf{x},\textbf{p}_{c,m})} \tag{11}\]

where \(\chi_{c}\) is the set of tokens whose actual class is

Figure 4: The t-SNE visualization of token features and prototypes on BC5CDR (Big Dict). We omit the enormous 0 tokens for the clarity of demonstration.

Figure 5: The cosine similarity curve between tokens and their actual prototypes over training steps



\(c\) (the actual class of token is obtained from human annotation of CoNLL03) and \(s\) denotes the cosine similarity function. As can be observed from the figure, when the model is trained using the DOT, the similarity between entity tokens and their actual prototypes exhibits a consistent increase throughout the training, thereby bolstering the entity token classification performance. In contrast, the model trained without DOT exhibits a decrease in similarity, suggesting that the model overfits the incomplete labeling noise. Regarding O tokens, the similarity between O tokens and O prototypes is higher when trained without DOT. This implies the network tends to predict the O class, which leads to false negative. While with our DOT algorithm, this issue can be mitigated. Based on these observations, we can conclude that the DOT algorithm plays a important role in alleviating the noise of incomplete labeling, and thereby significantly enhancing the performance in the DS-NER task.

## 4 Related Work

Named entity recognition is a fundamental task in natural language processing and has been applied to various downstream tasks Li and Ji (2014); Miwa and Bansal (2016); Ganea and Hofmann (2017); Wu et al. (2020); Shen et al. (2021); Wu et al. (2022b). NER methods can be divided into two main categories: tagging-based and span-based. Tagging-based methods Lample et al. (2016); Huang et al. (2015) predict a label for each token, which perform well at detecting flat named entities while failing at detecting nested named entities. Span-based methods Sohrab and Miwa (2018); Yu et al. (2020); Wang et al. (2020); Shen et al. (2021) perform classification over span representations, which performs well on the nested NER task but is inferior in computational complexity. Tan et al. (2021); Shen et al. (2022); Wu et al. (2022a) design a query-based NER framework that optimizes entity queries using bipartite graph matching. Recently, some generative NER methods Yan et al. (2021); Shen et al. (2023); Ba et al. (2022) have been proposed with superior performance on various NER tasks. These supervised NER methods require a large amount of annotated data for training.

Ds-ner.To mitigate the need for human annotations, distant supervision is widely used. The main challenge of the DS-NER task is the label noise, of which the most widely studied is the incomplete labeling noise. Various methods have been proposed to address the noise in distant annotations. AutoNER Shang et al. (2018) design a new tagging scheme that classifies two adjacent tokens to be tied, broken, or unknown. Token pairs labeled unknown are not considered in loss computation for denoising. Negative sampling methods Li et al. (2021, 2022) sample O tokens for training to mitigate the incomplete labeling noise. Positive-unlabeled learning (PU-learning) Peng et al. (2019); Zhou et al. (2022) treats tokens labeled with O class as unlabeled samples and optimizes with an unbiasedly estimated empirical risk. Self-learning-based methods Liang et al. (2020); Zhang et al. (2021); Meng et al. (2021) train a teacher model with distant annotations and utilize the soft labels generated by the teacher to train a new student. Other studies also adopt causal intervention Zhang et al. (2021) or hyper-geometric distribution Zhang et al. (2021) for denoising. In our MProto, we propose a novel denoised optimal transport algorithm for the DS-NER task. Experiments show that the denoised optimal transport can significantly mitigate the noise of incomplete labeling.

Prototypical Network.Our work is also related to prototypical networks Snell et al. (2017). Prototypical networks learn prototype for each class and make predictions based on the similarity between samples and prototypes. It is widely used in many tasks such as relation extraction Ding et al. (2021) and the few-shot named entity recognition FS-NER task Fritzler et al. (2019); Ma et al. (2022). Previous prototypical networks for the NER task are mostly single-prototype networks, which do not consider the semantic difference within the same entity type. Tong et al. (2021) try to incorporate multiple prototypes for O class in the FS-NER task. However, they do not consider the intra-class variance in entity classes and their clustering strategy will introduce huge computational complexity when training data is large. In our work, we utilize multiple prototypes to represent each entity type and solve the token-prototype assignment efficiently with the sinkhorn-knopp algorithm. Experiments confirm that our MProto can successfully describe the intra-class variance.

## 5 Conclusion

In this paper, we present MProto, a prototype-based network for the DS-NER task. In MProto, each category is represented with multiple prototypes to 

model the intra-class variance. We consider the token-prototype assignment problem as an optimal transport problem and we apply the sinkhorn-knopp algorithm to solve the OT problem. Besides, to mitigate the noise of incomplete labeling, we propose a novel denoised optimal transport algorithm for 0 tokens. Experiments show that our MProto has achieved SOTA performance on various DS-NER benchmarks. Visualizations and detailed analysis have confirmed that our MProto can successfully characterize the intra-class variance, and the denoised optimal transport can mitigate the noise of incomplete labeling.

## Limitations

The intra-class variance is actually a common phenomenon in the NER task. Currently, we only try to utilize the MProto on the distantly supervised NER task. Applying MProto to the few-shot NER task and the fully supervised NER task can be further investigated.

## Acknowledgements

This work is supported by the Fundamental Research Funds for the Central Universities (No. 226-2023-00060), National Natural Science Foundation of China (No. 62376245), Key Research and Development Program of Zhejiang Province (No. 2023C01152), National Key Research and Development Project of China (No. 2018AAA0101900), Joint Project DH-2022ZY0013 from Donghai Lab, and MOE Engineering Research Center of Digital Library.

## References

* Cutri (2013) Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_, pages 2292-2300.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ding et al. (2021) Ning Ding, Xiaobin Wang, Yao Fu, Guangwei Xu, Rui Wang, Pengjun Xie, Ying Shen, Fei Huang, Hai-Tao Zheng, and Rui Zhang. 2021. Prototypical representation learning for relation extraction. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Fritzler et al. (2019) Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named entity recognition task. In _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_, SAC '19, pages 993-1000, New York, NY, USA. Association for Computing Machinery.
* Ganea and Hofmann (2017) Octavian-Evgen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2619-2629, Copenhagen, Denmark. Association for Computational Linguistics.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. _ArXiv preprint_, abs/1508.01991.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 260-270, San Diego, California. Association for Computational Linguistics.
* Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics (Oxford, England)_, 36(4):1234-1240.
* Li and Ji (2014) Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 402-412, Baltimore, Maryland. Association for Computational Linguistics.
* Li et al. (2021) Yangming Li, Lemao Liu, and Shuming Shi. 2021. Empirical analysis of unlabeled entity problem in named entity recognition. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Li et al. (2022) Yangming Li, Lemao Liu, and Shuming Shi. 2022. Rethinking negative sampling for handling missing entity annotations. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7188-7197, Dublin, Ireland. Association for Computational Linguistics.
* Liang et al. (2020) Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. BOND: bert-assisted open-domain named entity recognition 

with distant supervision. In _KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_, pages 1054-1064. ACM.
* Lu et al. (2022) Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified structure generation for universal information extraction. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5755-5772, Dublin, Ireland. Association for Computational Linguistics.
* Ma et al. (2022) Tingting Ma, Huiqiang Jiang, Qianhui Wu, Tiejun Zhao, and Chin-Yew Lin. 2022. Decomposed meta-learning for few-shot named entity recognition. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 1584-1596, Dublin, Ireland. Association for Computational Linguistics.
* Meng et al. (2021) Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang, Yu Zhang, Heng Ji, and Jiawei Han. 2021. Distantly-supervised named entity recognition with noise-robust learning and language model augmented self-training. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10367-10378, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1105-1116, Berlin, Germany. Association for Computational Linguistics.
* Peng et al. (2019) Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. 2019. Distantly supervised named entity recognition using positive-unlabeled learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2409-2419, Florence, Italy. Association for Computational Linguistics.
* Shang et al. (2018) Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2054-2064, Brussels, Belgium. Association for Computational Linguistics.
* Shen et al. (2021a) Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021a. Locate and label: A two-stage identifier for nested named entity recognition. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2782-2794, Online. Association for Computational Linguistics.
* Shen et al. (2021b) Yongliang Shen, Xinyin Ma, Yechun Tang, and Weiming Lu. 2021b. A Trigger-Sense Memory Flow Framework for Joint Entity and Relation Extraction. In _Proceedings of the Web Conference 2021_, WWW '21, pages 1704-1715, New York, NY, USA. Association for Computing Machinery.
* Shen et al. (2023a) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023a. Diffusion-NER: Boundary diffusion for named entity recognition. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3875-3890, Toronto, Canada. Association for Computational Linguistics.
* Shen et al. (2023b) Yongliang Shen, Zeqi Tan, Shuhui Wu, Wenqi Zhang, Rongsheng Zhang, Yadong Xi, Weiming Lu, and Yueting Zhuang. 2023b. PromptNER: Prompt locating and typing for named entity recognition. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12492-12507, Toronto, Canada. Association for Computational Linguistics.
* Shen et al. (2022) Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yueting Zhuang. 2022. Parallel instance query network for named entity recognition. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 947-961, Dublin, Ireland. Association for Computational Linguistics.
* Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 4077-4087.
* Sohrab and Miwa (2018) Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2843-2849, Brussels, Belgium. Association for Computational Linguistics.
* Tan et al. (2021) Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, and Yueting Zhuang. 2021. A Sequence-to-Set Network for Nested Named Entity Recognition. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence_, pages 3936-3942, Montreal, Canada. International Joint Conferences on Artificial Intelligence Organization.
* Sang (2002) Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In _COLING-O2: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
* Tong et al. (2021) Meihan Tong, Shuai Wang, Bin Xu, Yixin Cao, Minghui Liu, Lei Hou, and Juanzi Li. 2021. Learning from miscellaneous other-class words for few-shot named entity recognition. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference 

on Natural Language Processing (Volume 1: Long Papers)_, pages 6236-6247, Online. Association for Computational Linguistics.
* Wang et al. [2020] Jue Wang, Lidan Shou, Ke Chen, and Gang Chen. 2020. Pyramid: A layered model for nested named entity recognition. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5918-5928, Online. Association for Computational Linguistics.
* Wei et al. [2016] Chih-Hsuan Wei, Yifan Peng, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Jiao Li, Thomas C. Wiegers, and Zhiyong Lu. 2016. Assessing the state of the art in biomedical relation extraction: Overview of the BioCreative V chemical-disease retalion (CDR) task. _Database: The Journal of Biological Databases and Curation_, 2016:baw032.
* Wu et al. [2022a] Shuhui Wu, Yongliang Shen, Zeqi Tan, and Weiming Lu. 2022a. Propose-and-Refine: A Two-Stage Set Prediction Network for Nested Named Entity Recognition. In _Thirty-First International Joint Conference on Artificial Intelligence_, volume 5, pages 4418-4424.
* Wu et al. [2020] Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court s view generation with causality. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 763-780.
* Wu et al. [2022b] Yiquan Wu, Yifei Liu, Weiming Lu, Yating Zhang, Jun Feng, Changlong Sun, Fei Wu, and Kun Kuang. 2022b. Towards interactivity and interpretability: A rationale-based legal judgment prediction framework. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 4787-4799.
* Wu et al. [2023] Yiquan Wu, Weiming Lu, Yating Zhang, Adam Jatowt, Jun Feng, Changlong Sun, Fei Wu, and Kun Kuang. 2023. Focus-aware response generation in inquiry conversation. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 12585-12599.
* Yan et al. [2021] Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various NER subtasks. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5808-5822, Online. Association for Computational Linguistics.
* Yu et al. [2020] Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6470-6476, Online. Association for Computational Linguistics.
* Zhang et al. [2021a] Wenkai Zhang, Hongyu Lin, Xianguei Han, and Le Sun. 2021a. De-biasing distantly supervised named recognition via causal intervention. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4803-4813, Online. Association for Computational Linguistics.
* Zhang et al. [2021b] Wenkai Zhang, Hongyu Lin, Xiapei Han, Le Sun, Huidan Liu, Zhicheng Wei, and Nicholas Yuan. 2021b. Denoising Distantly Supervised Named Entity Recognition via a Hypergeometric Probabilistic Model. _Proceedings of the AAAI Conference on Artificial Intelligence_, 35(16):14481-14488.
* Zhang et al. [2021c] Xinghua Zhang, Bowen Yu, Tingwen Liu, Zhenyu Zhang, Jiawei Sheng, Xue Mengge, and Hongbo Xu. 2021c. Improving distantly-supervised named entity recognition with self-collaborative denoising learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10746-10757, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Zhou et al. [2022] Kang Zhou, Yuepei Li, and Qi Li. 2022. Distantly supervised named entity recognition via confidence-based multi-class positive and unlabeled learning. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7198-7211, Dublin, Ireland. Association for Computational Linguistics.

 

Sinkhorn-Knopp Algorithm

We apply the sinkhorn-knopp algorithm Cuturi (2013) to solve the optimal transport and denoised optimal transport elaborated in equation 7 and equation 8. In order to apply the sinkhorn-knopp algorithm, we add an entropy regularizer as follows:

\[\begin{split}&\hat{\gamma}=\operatorname*{arg\,min}_{\gamma}\sum_{i} \sum_{j}\gamma_{i,j}\mathbf{C}_{i,j}+\lambda^{r}\operatorname{H}(\gamma),\\ &\text{s.t.}~{}~{}~{}\gamma\boldsymbol{1}=\mathbf{a},\gamma^{ \top}\boldsymbol{1}=\mathbf{b}\end{split} \tag{12}\]

where \(\lambda^{r}\) is the weight of the regularization and \(\operatorname{H}(\gamma)=\sum_{i,j}\gamma_{i,j}\log(\gamma_{i,j})\) is the entropy of the assignment matrix.

We specify the pseudo-code for the sinkhorn-knopp algorithm in Algorithm 1. Here the \(\oslash\) corresponds to the element-wise division, \(\mathbf{a}\) and \(\mathbf{b}\) are vectors that represent the weights of each sample in the source and target distributions, \(\mathbf{C}\) is the cost matrix, \(\lambda^{r}\) is the regularization weight, and \(\gamma\) is the assignment matrix.

The sinkhorn-knopp algorithm mainly consists of several matrix operations which can be easily accelerated by GPU devices. And we empirically find that the sinkhorn-knopp algorithm can obtain satisfying results in a few iterations in our work. Therefore, applying the sinkhorn-knopp algorithm to solve the token-prototype assignment problem only has a slight impact on the speed of the training.

```
0:\(\mathbf{a},\mathbf{b},\mathbf{C},\lambda^{r}\) \(\mathbf{u}^{0}=\boldsymbol{1},\mathbf{K}=\exp(-\mathbf{C}/\lambda^{r})\) for\(i\) in \(1,\dots,n\)do \(\mathbf{v}^{i}=\mathbf{b}\circ\mathbf{K}^{\top}\mathbf{u}^{i-1}\) \(\mathbf{u}^{i}=\mathbf{a}\circ\mathbf{K}\mathbf{v}^{i}\) endfor return\(\gamma=\mathrm{diag}(\mathbf{u}^{n})\mathbf{K}\,\mathrm{diag}(\mathbf{v}^{n})\)
```

**Algorithm 1** Sinkhorn-Knopp Algorithm

## Appendix B Implementation Details and Hyperparameters

We implement our MProto with Pytorch2 and Huggingface Transformers3. All experiments are carried out on a single RTX-3090 with 24G graphical memory. And the training of the model can be finished in approximately 2 hours.

Footnote 2: [https://pytorch.org/](https://pytorch.org/)

Footnote 3: [https://huggingface.co/transformers](https://huggingface.co/transformers)

We report the hyperparameters used in different datasets in Table 4. When training, we use an AdamW optimizer with weight decay \(0.0001\) and maximum gradient norm \(1.0\). The maximum learning is \(0.0001\), and the learning rate is warmed up linearly in the first \(100\) steps and decayed linearly afterward. The batch size is set to \(32\). We train the model for \(10\) epochs in all experiments.

For the sinkhorn-knopp algorithm used in computing the token-prototype assignment, we set the regularization weight \(\lambda^{r}=0.001\) and the number of iterations to \(100\).

## Appendix C Feature Visualization

We further visualize the token features and the embedding of the prototypes in different datasets and model setups in Figure 6.

As shown in Figure 5(a) and Figure 5(c), we can see that even when the model is trained with only a single prototype per entity type (\(M=1\)), token features of the same entity type still tend to form different sub-clusters due to the semantic difference. Therefore, it can be confirmed that the intra-class variance is a common phenomenon in the NER task regardless of the model.

Our MProto represents each entity type with multiple prototypes. As shown in Figure 4 and Figure 5(b), for each sub-cluster of token features, there exists at least one prototype to represent this sub-cluster. In other words, our MProto can successfully model the intra-class variance of the entity token features. The visualizations show that representing each entity type with multiple prototypes rather than a single prototype is beneficial and can significantly improve the performance in the DS-NER task.

## Appendix D Transport Plan of Unlabeled Entities

To analyze the effectiveness of our denoised optimal transport algorithm, we obtain the transport plan by counting the assignment result of the unlabeled entity tokens in the train set with a checkpoint of MProto at the early stage of training.

As shown in Figure 7, most unlabeled entity

\begin{table}
\begin{tabular}{p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline  & \multicolumn{2}{c}{BC5CDR} & \multicolumn{2}{c}{CoNLL03} \\ \cline{2-5}  & Big Dict & Small Dict & KB & Dict \\ \hline \(M\) & 3 & 3 & 3 & 3 \\ \(\lambda^{c}\) & 0.05 & 0.1 & 0.05 & 0.01 \\ \(\alpha\) & 0.9 & 0.5 & 0.5 & 0.9 \\ \(\beta\) & 0.01 & 0.01 & 0.05 & 0.01 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Hyperparameters used in different datasets.

 

tokens are assigned to prototypes of their actual classes, and only a few tokens are mistakenly assigned to the 0 prototypes. With this observation, we can confirm that before the model overfits the label noise, unlabeled entity tokens tend to be similar to prototypes of their actual classes in the feature space. Therefore, they tend to be assigned to prototypes of their actual classes in our similarity-based token-prototype assignment. In our denoised optimal transport, O tokens assigned to entity prototypes are considered as label noise, and only 0 tokens assigned to O prototypes are considered as true negative samples. In this way, we can discriminate unlabeled entity tokens from clean samples. These unlabeled entity tokens are ignored in loss computation so as not to misguide the training of the model. And we can conclude that the denoised optimal transport can effectively mitigate the incomplete labeling noise in the DS-NER task.

## Appendix E Experiment on Different Number of Prototypes

We try different \(M\) (the number of prototypes per type) on BC5CDR (Big Dict) and CoNLL03 (Dict) datasets. The results are reported in Table 5. It shows that representing each entity type with \(3\) prototypes is the optimal choice on both BC5CDR (Big Dict) and CoNLL03 (Dict) datasets. Intuitively, there exists an ideal \(M\) for each entity type based on the semantic complexity of that category. And a too-large \(M\) or a too-small \(M\) will hurt the performance of MProto. Besides, setting a too-large \(M\) will reduce the number of tokens assigned to each prototype. In this case, there might be no enough token features for learning representative prototypes, which leads to underfitting.

Figure 6: The t-SNE visualization of token features and prototypes. The visualization for MProto (\(M=3\)) on BC5CDR (Big Dict) can be found in Figure 4.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \multirow{2}{*}{\(M\)} & \multicolumn{3}{c}{BC5CDR (Big Dict)} & \multicolumn{3}{c}{CoNLL03 (Dict)} \\ \cline{2-7}  & Prec. & Rec. & F1 & Prec. & Rec. & F1 \\ \hline
1 & 75.74 & **85.94** & 80.52 & 84.39 & 79.27 & 81.75 \\
2 & 77.36 & 83.43 & 80.28 & 83.42 & 79.27 & 81.29 \\
3 & **77.53** & 85.84 & **81.47** & 84.27 & **80.79** & **82.49** \\
4 & 76.78 & 84.71 & 80.55 & **84.53** & 80.10 & 82.25 \\
5 & 73.52 & 84.14 & 78.47 & 82.84 & 77.96 & 80.32 \\
6 & 76.09 & 85.49 & 80.52 & 84.53 & 79.55 & 81.97 \\ \hline \end{tabular}
\end{table}
Table 5: Experiment with the different number of prototypes.

Figure 7: The transport plan for unlabeled entities. The y-axis represents the actual entity type of the unlabeled entity tokens. The x-axis represents the class of the prototypes that are assigned to the unlabeled entity tokens.

 



## Appendix F Similarity Curve of Different Entity Types

To better analyze the effectiveness of denoised optimal transport algorithm, we additionally report the similarity curve of each entity class in Figure 8. The similarity is obtained by Equation 11.

Figure 8: The token-prototype similarity curve of each entity type over training steps.