MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition
Authors: Shuhui Wu
Yongliang Shen
Zeqi Tan
Wenqi Ren
Jietian Guo
Shiliang Pu
Weiming Lu
Venue: EMNLP 2023 Main (2023)
Keywords: Information Extraction
Named Entity Recognition
Distant Supervision

Abstract: Distantly supervised named entity recognition (DS-NER) aims to locate entity mentions and classify their types with only knowledge bases or gazetteers and unlabeled corpus. However, distant annotations are noisy and degrade the performance of NER models. In this paper, we propose a noise-robust prototype network named MProto for the DS-NER task. Different from previous prototype-based NER methods, MProto represents each entity type with multiple prototypes to characterize the intra-class variance among entity representations. To optimize the classifier, each token should be assigned an appropriate ground-truth prototype and we consider such token-prototype assignment as an optimal transport (OT) problem. Furthermore, to mitigate the noise from incomplete labeling, we propose a novel denoised optimal transport (DOT) algorithm. 
Specifically, we utilize the assignment result between *Other* class tokens and all prototypes to distinguish unlabeled entity tokens from true negatives.
Experiments on several DS-NER benchmarks demonstrate that our MProto achieves state-of-the-art performance. The source code is now available on Github.

MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition
Shuhui Wu' and Yongliang Shen't and Zeqi Tan'
Wenqi Ren and Jietian Guo and Shiliang Pu and Weiming Lu
"College of Computer Science and Technology, Zhejiang University, China
2 Hikvision Research Institute
zyg,u,,)) wwwp://wwwuuuu...eee.o.ee cn
erenwenqi,guojiete pwseeliasshhiiieee hri wewikvisionncom luwm@zju. edu. cn
Abstract
Michuel Perelstoin, - portiólio munager MuinStay International Funds
PER
OKB
Distantly supervised named entity recognition (DS-NER) aims to locate entity mentions and classify their types with only knowledge bases or gazetteers and unlabeled corpus. However, distant annotations
PER
ORG
intas-class Variance
noisy and degrade the performance of NER models. In this paper, we propose a noise-robust prototype network named MProto for the DS-NER task. Different from previous prototype-based NER methods, MProto represents each entity type with mul- tiple prototypes characterize the intra-class variance among entity representations. To op- timize the classifier, each token should be as- signed an appropriate ground-truth prototype and we consider such token- token-prototype prototype assign- ment as an optimal transport (OT) problem. Furthermore, to mitigate the noise from incom- plete labeling, we propose novel denoised optimal transport (DOT) algorithm. Specifi- cally, we utilize the assignment result between Other class tokens and all prototypes to distin- guish unlabeled entity tokens from true nega- tives. Experiments on several DS-NER bench- marks demonstrate that MProto achieves state-of-the arrt- performance. The source code is now available on Github'.
Unlabeled PERs
Others
ORGs
Figure I: The distant annotations are marked in orange, and the human annotations are marked in blue. Here we illustrate two types of issues in the DS-NER. First, un- labeled entities will introduce incomplete labeling noise in 0 class. Second, tokens the same class may fall into different sub-clusters due semantic differences.
Despite its easy accessibility, the distantly- annotated data is rather noisy, severely degrading the performance of NER models. We observe two types of issues in the distantly-suprrvised named entity recognition task. The first issue is incom- plete labeling. Due the limited coverage of dictionaries, entities not presented in dictionaries are mistakenly labeled as Other class (denoted as class). For instance, in Figure 1, "Micheal" is not covered in the dictionary and thus cannot be correctly labeled by distant supervision. These unlabeled entities will misguide NER models to overfit the label noise, particularly hurting recall. The second issue is intra-class variance. As illus- trated in Figure 1, tokens of the same class (e.g., "MainStay" and "Funds"" may fall into different sub-clusters in fcature space due to semantic dif- ferences. Traditional single-prototype classifiers do not consider the semantic difference within the same entity type. They set only one prototype for each type, which suffers from intra-class variance.
1 Introduction
Named Entity Recognition (NER) is a fundamental task in natural language processing and is essential in various downstream tasks (Li and Ji, 2014; Miwa and Bansal, 2016; Ganea and Hofmann, 2017; Shen et al., 2021b). Most state-of-the-an art NER models are based deep neural networks and require mas- sive annotated data for training. However, human annotation expensive, time-consuming, and often unavailable in some specific domains. To mitigate the reliance on human annotation, distant supervi- sion is widely applied to automatically generate an- notated data with the aid of only knowledge bases gazetteers and unlabeled corpus.
Various methods have been proposed to mitigate the noise of incomplete labeling. For example, AutoNER (Shang et al., 2018) tries to modify the standard CRF-based classifier to adapt to the noisy NER datasets. Self-learming-baas based methods (Liang
Corresponding author.
Chttps://gitthe www.o///otattoinuun/oparooooeeaa

---

al., 2020; Meng et al., 2021; Zhang et al., 2021c) learn by soft labels generated from teacher model for denoising. Positive-unlabeled learning methods treat tokens O class as unlabeled samples and optimize with an unbiasedly estimated empirical risk (Peng et al., 2019; Zhou et al., 2022). Some other studies utilize negative sampling to avoid NER models overfitting label noise in 0 class (Li al., 2021, 2022). Despite these efforts, the issue of intra-class variance has not yet been studied in previous DS-NER research.
multiple-prototype classifier and denoised op- timal transport.
2 Method
In this section, we first introduce the task formula- tion in Section 2.1. Then, we present our MProto network in Section 2.2. To compute the cross- entropy loss, we assign each token with ground- truth prototype, and the token-prototype assign- ment will be discussed in Section 2.3. Besides, to mitigate the incomplete labeling noise in 0 kens, we propose the denoised optimal transport algorithm which will be specified in Section 2.4. Figure 2 illustrates the overview of our MProto.
Unlike previous methods, we propose a novel prototype-based network named MProto for distantly-suuperviied named entity recognition. In MProto, each class is represented by multiple pro- totypes so that each prototype can represent cer- tain sub-cluster of entity type. In this way, our MProto can characterize the intra-class vari- ance. To optimize our multiple-prototyp classifier, we assign each input token with an appropriate ground-trute truth prototype. We formulate such token- prototype assignment problem as an optimal trans- port (OT) problem. Moreover, we specially design a denoised optimal transport (DOT) algorithm for tokens labeled as 0 class to alleviate the noise of incomplete labeling. Specifically, we perform the assignment between O tokens and all prototypes and regard tokens assigned to 0 prototypes as true negatives while tokens assigned to prototypes of entity classes as label noise. Based on our obser- vation, before overfitting, unlabeled entity tokens tend to be assigned to prototypes of their actual classes in our similarity- token-prototype as- signment. Therefore, unlabeled entity tokens can be discriminated from clean samples by DOT algorithm so as not misguide the training. Our main contributions are three-fold:
2.1 Task Formulation
Following the tagging- bugging-based named entity recog- nition paradigm, denote the input sequence X = [=1,-.. ,z and the corresponding distantly- annotated tag sequence as Y = [y/,..... ........y... yr]. For each token-label pair (z4,, Vi). ) 2t is the input token and yj E = {cI,-.., Cx is the class of the token. Here we let e1 be O class and others be predefined entity classes. We denote the human-annotated tag sequence Ý = [ū "........... VL.. can be considered true labels. In the distantly- supervised NER task, only distant annotations be used for training while human annotations are only available in evaluation.
Human annotations
2.2 Multi-Prototype Network
Token Encoding. For each input token sequence X, generate its corresponding features with pretrained language model:
H = fo(X)
(1)
where fo is the pretrained language model parame- terized by 0 and H [h....., hy] is the features of the token sequence.
We present MProto for the DS-NER task. MProto represents each entity type with mul- tiple prototypes for intra-class variance. And we model the token- token-prototype assignment problem as an optimal transport problem.
Prototype-based Classification. MProto a prototype-basee classifier where predictions are made based on token-prototype similarity. Visual- ization in Figure 4 shows that tokens of the same class will form multiple sub-clusters in the feature space due to the semantic difference. However, previous prototype-ban
To alleviate the noise of incomplete labeling. we propose the denoised optimal transport al- gorithm. Tokens labeled as but assigned to non-O prototypes are considered as label noise so as not to misguide the training.
classifiers which repre- sent each entity type with only single prototype cannot characterize such intra-class variance. To this end, we represent each entity type with mul- tiple prototypes. Specifically, for each class e, let
Experiments on various datasets show that our method achieves SOTA performance. Fur- ther analysis validates the effectiveness of our

---

Optimal Transport
Token Encoding
Perelstein
PER
PER
X
X
36
36
Michael
0.1
0.9
MainStay
Perelstein
International
ORG
0U9
0.1
Funds
portfolio
ORG
manager
of
3.5
Michcal
x
PER
MainStay
0.
33
05
6.1
00
00
International
portfolio
Funds
0.3
38
0.5
07
08
03
mnannger
of
ORG
Denoised Optimal Transport
The overview of MProto. For the clarity, only show tokens of three classes (PER, ORG and 0) and MProto with two prototypes per class. MProto is prototype-based classifier where predictions are made based on the similarity between token features prototypes. optimize the multiple-protee prototype classifier, we assign each token an appropriate ground-truth prototype. The top right of the figure illustrates the loken-prototype assignment for entity tokens which is considered as an OT problem. The bottom right of the figure illustrates the token protttppottyppe assignment for tokens which is solved by our DOT algorithm where 0 tokens assigned with non-O prototypes (in dashed line) are considered as label noise. The assignment is done based on cost matrix (the tables shown in the figure) where each cell the distance between a token and prototype.
Figure 2:
P. = PP,,,,.... .P.M } be the set of M prototypes representing class c. For the classification, we com- pare the similarity between token features and all prototypes, and the class of the most similar proto- type is chosen as the prediction:
However, optimizing the model only through CE loss guides tokens to be relatively close to their ground-truth prototypes in feature space, while the compactness of the token features within the same sub-cluster is considered. To this end, we fur- ther optimize the absolute distance between token features and ground-truu truth prototypes as follows:
arg max s (h;, P...
where shh, p)
|p the similarity metric and here we adopt cosine similarity. In the infer- ence process, consecutive tokens of the same type are considered a singular entity.
hpp
Σ dh Pea.m
Σ (1- s(h, Pea.m)
(4)
here we define the distance based on cosine similar- ity as d(h;, Pc.m. = 1 s(h,, Pe.m) The overall loss can be calculated follows:
Loss. To update the parameters of our MProto, we calculate the loss between token features and their corresponding ground-truth prototypes. First, we should assign each token with an appropriate ground-truth prototype. Specifically, for each token and its annotation y4 = c;, one of the prototypes of class G will be assigned as the ground truth based on the similarity between the token feature and prototypes. Such preetpprotttppe assignment is considered
t= c X
(5)
where X is the weight of the compactness regular- ization.
Prototype Updating. We update each prototype with the token features assigned to that prototype. For convenience, we denote the set of tokens as- signed to prototype Pr.m as T. At each training step we update prototypes with exponential moving average (EMA):
optimal transport problem, and the detail of solving the token-proter token-prototype assign- ment will be discussed later (in Section 2.3). Based on the assigned ground-trutt prototype Pe.. for token i, we can compute the cross-entropy loss:
P' = op- -1 + +1 a) ΣiET h,
(6)
IT
exp(s(h; (h;, p)).
gCE
Σ log Ee m exp(s(hi, (m))
where a is the EMA updating ratio. With EMA updating, the learned prototypes can be considered

---

the representative of certain sub-cluster in the feature space.
prototypes to participate in the assignment of O to- kens. Here we denote the of tokens labeled class as To and the assignment matrix between To and all prototypes as y E ||]||KMM The de- noised optimal transport is formulated as follows:
2.3 Token-Prototype Assignment for Entity Tokens
In our MProto, each class is represented by multi- ple prototypes. Therefore, how to assign token with an appropriate ground-truth prototype is an essential issue for optimization. First, we denote the set of tokens labeled as class c as Te. We aim to compute the assignment matrix 7 E R/ between To and Pe (prototypes of class c). Then the assigned prototype Pa,m for token i can be ob- tained by m arg max; Th.j. We consider such token- eeken-prototype assignment problem an optimal transport problem:
KM
30 arg min
C
70
iCTo
j=1
(8)
s.t. 7 a, 1=b
where C [Cat [Ca ..... CoK RiTolxMM is the cost matrix composed of the distances between fea- tures of the token set To and all prototypes, and [-] is the concatenation operation. The first con- straint a = E |R/T| indicates that each token is expected to be assigned to exactly one prototype. The second constraint is formulated as:
M
grc = arg min
ΣΣ
ΣΣN
ni
oig
3 Tol M
(1 [T
(1- (1-B)) B)
7"th
i=
b
1,
1]
(K - 1)M (K 1)M
s.t. 71 a, $1=a,77111 1=b
K-1
(9)
where C = dh; Po.j) ) is the cost matrix which is composed of the distances between features of the token set Tr and prototype set Pe.. a = 1 E RlTd is the assignment constraint for each token which guarantees that each token is expected to be as- signed exactly one prototype, b E RR M is the assignment constraint for each prototype which prevents the model from the trivial solution where all tokens are assigned to a single prototype. The constraint can be set based on prior knowledge. However, to keep our method simple, we simply choose even distribution (b ITel 1). The exper- iments also show that such simple constraint can already achieve satisfactory performance.
where 3 is the assignment ratio for prototypes. It indicates that: (1) we expect 3|T tokens be assigned to O prototypes, (2) the remaining tokens are assigned to non-O prototypes, (3) the token features are evenly assigned to prototypes of the same type.
Intuitively, before the model overfits the incom- plete labeling noise, unlabeled entity tokens are similar to prototypes of their actual entity type in feature space. So these unlabeled entity tokens tend to be assigned to prototypes of their actual entity type in our similarity- base eiken-prototype assign- ment. Therefore, tokens assigned to 0 prototypes can be considered as true negatives while others can be considered as label noise. We then modify the CE loss in Equation as follows:
By optimizing Equation 7, we aim to obtain an assignment where each token is assigned a similar prototype. The token-prototy lem can be solved by the miikhorm-knopp algorithm (Cuturi, 2013), which is detailed in Appendix A.
assignment prob-
exp(s(h;, Peum))
C pc
Σ log TE
Ed mm exp(s hi, pe)
2.4 Proen-Prototype Assignment for 0 Tokens
exp(s(h,, p.))
Σ wi log Ee exp(s(h;; Pel Pe)m
Incomplete labeling noise a common issue in the distantly supervised named entity recognition task. If we directly assign all O tokens to O prototypes, features of unlabeled entity tokens will be pulled closer to 0 prototypes and farther prototypes of their actual entity type, which leads to overfitting. To mitigate the noise of incomplete labeling, we specially design the denoised optimal transport al- gorithm for 0 tokens. Specifically, we allow all
TCoo
(10)
where T = U Te, is the set of tokens not labeled as O, w. = 1(p, Pa,) is the indicator with value when the assigned prototype p. for token is of class e₁ class). The first term of Equation 10 is the loss for entity tokens which is identical to traditional CE loss. The second term is the loss for O tokens where only these true negative

---

samples are considered. In this way, unlabeled entity tokens will not misguide the training of our MProto, and the noise of incomplete annotation can be mitigated.
3.2 Baselines
Fully-Supervised. We implement a fully- supervised NER model for comparison. The model is composed of a BERT encoder and linear layer as the classification head. Fully-supervised model is trained with human-annotater data, and the performance can be viewed the upper bound of the DS-NER models.
3 Experiments
3.1 Settings
NER Dataset. The distantly-annotated data is generated from two wildly used flat NER datasets: (1) CoNLL03 (Tjong Kim Sang, 2002) is an open- domain English NER dataset that is annotated with four entity types: PER, LOC, ORG and MISC. is split into 14041/3250/3453 train/dev/test sentences. (2) BC5CDR (Wei al., 2016) is biomedical do- main English NER dataset which consists of 1500 PubMed articles with 4409 annotated Chemi cals and 5818 Diseases. It is split into 4560/4579/4797 train/dev/test sentences.
AutoNER. AutoNER (Shang al., 2018) is DS-NER method that classifies two adjacent tokens as break tic. To mitigate the noise of incomplete labeling, the unknown type is not considered loss computation. For a fair comparison, we re- implement their method to use BERT-baae-cased as the encoder for the CoNLL03 dataset and use BioBERT- -aseecccae for the BC5CDR dataset.
Early Stoping. Liang et al. (2020) apply early stopping to prevent the model from overfitting the label noise. For a fair comparison, we re- implement their method to use BERT-base-cased as the encoder for the CoNLLO3 dataset and use CaBBRRT-base-assee for the BC5CDR dataset.
Distant Annotation. We generate four different distantly-annotatedd datasets based on CoNLL03 and BC5CDR for the main experiment: (1) CoNLL03 (Diet) is generated by dictionary match- ing with the dictionary released by Peng et al. (2019). (2) CoNLL03 (KB) is generated by KB matching follows (Liang et al., 2020). (3) BC5CDR (Big Dict) is generated by dictionary matching with the dictionary released by Shang et al. (2018). (4) BCSCDR (Small Dict) is generated by the first 20% of the dictionary used in BCSCDR (Big Dict). We follow the dictionary matching algorithm presented in (Zhou et al., 2022) and the KB matching algo- rithm presented in (Liang et al., 2020).
Negative Sampling. Li et al. (2021, 2022) use negative sampling to eliminate the misguidance of the incomplete labeling noise. We re-implement their method and set the negative sampling ratio be 0.3 as recommended in (Li al., 2021).
MPU. Multi-class positive-unlabeled (MPU) learning (Zhou et al., 2022) considers the negative samples unlabeled data and optimizes with the unbiasedly estimated task risk. Conf-MPU incor- porates the estimated confidence score for tokens being an entity token into the MPU risk estimator to alleviate the impact of annotation imperfection.
Evaluation Metric. We train our MProto with the distantly-annoten train set. Then the model evaluated on the aamaatte dev set. The best checkpoint on the dev set is tested on the human- annotated test set, and the performance on the test set is reported as the final result. Entity prediction is considered correct when both span and category are correctly predicted.
3.3 Overall Performance
Table shows the overall performance of our MProto compared with various baselines in four distantly-annottted datasets. We observe that our MProto achieves state-of-the-art performance on all four datasets. Specifically, our MProto achieves + .1.8%, +0.39%, +0.28% and +0.60% improve- ment in F1 score on BC5CDR (Big Dict), BC5CDR (Small Dict), CONLLO3 (KB), and CoNLLO3 (Dict) compared with previous state-of-the-art methods. We also notice that our MProto achieves consistent improvement on all datasets. In contrast, previous SOTA methods usually achieve promising results on some datasets while performing poorly in other
Implementation Details. For fair comparison, we use the BERT-base-cased (Devlin et al., 2019) for the CoNLLO3 dataset, and BioBERT-basc- cased (Lee et al., 2020) for the BC5CDR dataset. We M prototypes per class and the hy- perparameter search experiment can be found in Appendix E. More detailed hyperparameters can be found in Appendix B.

---

Model
BCSCDR (Big Dict)
BCSCDR (Small Diet)
CoNLLO3 (KB)
CoNLL03 (Dict)
BERT (Fully Sup.)
87.45 (85 115.66/8..110 10)
91.30 (90.82/91.78)
DS Matching
64.32 (86.39/51.24)
15.69 (80.02/8,700
71.40 (81.13/63.75)
63.93 (93. 12/48.67) 67)
AutoNER
73.02 (78,33/68.3) 39)
19.90 (68.34/11.66 1.64)
64.03 1
59.47 (81.89/46.9)
Early Stopping
73.66 (80.43/67. 94)
17.21 (75.60/ 9.71)
77.06 (84.03/71.16)
67.74 (86.34/55.74)
Neg Sampling
78.73 (79.30/78. 17)
24,25 (78.93/14.32)
79.30 (83.16/75.7) 16/75.78)
74.90 (83.71/67
MPU
68.22 (56.50/86.05)
73.91 (70.08/78. 18)
65.75 (58. 79/74.58)
67.65 (63.63/72.22)
MPPf-MPPU
77.22 (69.79/86.4) 42)
71,85 (8) (1/004/4444)
79.16 (68. 58/79.777 75))
8 81.89 (81.71/82.08)
MProto
81.47 (77.53/85.84)
74.30 (73.41/75.22)
79.58 (79.80779.37)
82.49 (84.27/80.79)
Table Overall performance. The results are reported in "Fl (precision recall)" format.
cases. We attribute this superior performance to two main factors: (1) The denoised optimal trans- port algorithm significantly mitigates the noise of incomplete labeling in O tokens by leveraging the similarity-baan based token token-protoon prototype assignment result, leading performance improvement across vary- ing annotation quality (with differences in dictio- nary coverage or Distant Supervision matching al- gorithms). (2) Our multiple prototype classifier can characterize the intra-class variance and can better model rich semantic entity classes, which improves the robustness across diverse data domains.
30
MeTRo
SYYLD
Mepöamp.
01-
20
Oct Proprtio
Det Fepustion
(a) CoNLL03
(b) BCSCDR
Figure 3: Results on the distantly-anootateed dataset with different proportions of the dictionaries. "NegSamp." stands for Negative Sampling. "ES" stands for Early Stopping. "DS" stands for dictionary matching.
3.4 Ablation Study
noise the DS-NER task.
We conduct the ablation study in the following three aspects. The results shown in Table 2.
3.5 Experiments on Dictionary Coverage
Multiple Prototypes. By representing each en- tity type with multiple prototypes, the Fl score of MProto improves by +0.95% on BC5CDR (Big Dict) and by +1.26% on CoNLLO3 (Dict). It confirms that the multople-prototype classifier can greatly help the MProto characterize the intra-class variance of token features.
To analyze the performance of our method with dif- ferent coverage of dictionaries, we generate distant annotations with dictionaries of different propor- tions following (Zhou et al., 2022). The distant annotation quality can be found Table and the experiment result shown in Figure 3.
The result shows that the performance of Neg- ative Sampling and Early Stopping drops signif- icantly when the coverage of the dictionary de- creases. When the dictionary size drops from 100% to 20%, the Fl score of the Negative Sampling drops by -54.48% the BC5CDR dataset and by -47.44% on the CoNLLO3 dataset. The Fl score of the Early Stopping drops by -56.13% on the BCSCDR dataset and by -51.02% on the CoNLLO3 dataset. This indicates that these methods suffer severely from incomplete labeling noise. How- ever, our MProto only suffers from slight per- formance drop compared with Negative Sampling and Early Stopping. The Fl score drops only by -7.17% on BC5CDR and by -9.29% on CoNLLO3. We can conclude that with multiple prototypes and denoised optimal transport, our MProto more robust to the incomplete labeling noise and can achieve better performance, especially on the DS-
Compactness Loss. By applying compactness loss as elaborated Equation 4, the Fl score of MProto improves by +2.23% on BC5CDR and by +0.29%. Simply optimizing the Cross-Entropy loss in Equation only encourages the token features be relatively close to the assigned ground- truth prototypes. While the compactness loss directly optimizes the distance between the token feature and the prototype to be small, making the token features of the same prototype more compact.
Denoised Optimal Transport. Compared with assigning all 0 tokens to 0 prototypes, the denoised optimal transport elaborated in Section 2.4 im- proves the Fl score of MProto by +21.89% on BC5CDR and by +34. 14% on CoNLLO3. The im- provement indicates that the denoised optimal trans- port significantly mitigates the incomplete labeling

---

BCSCDR (Big Diet)
CoNLLO3 (Dict)
Model
Loc. Fl
Cls. FI
Prec.
Rec.
F1
Loc. F1
Cls. F1
Prec.
Rec.
Fl
Default
81,80
92.21
77.53
85.84
81.47
89.04
86,31
84.27
80.79
82,49
w/o Multiple Prototypes
80.81
92.29
75.74
85,94
80,52
88.72
85.22
84.39
79.27
81.75
w/o Compactness Loss
79.64
91.05
75.05
83,92
79.24
88.79
86,06
84.09
80.40
82.20
w/o DOT
59.53
63.05
81.94
46.20
59.09
59.58
54.59
58.63
41.13
48.35
Table 2: Ablation Study.
Dataset
Prec.
Rec.
Entity
Chemical
BCSCDR (20% Dict)
87.31
12.09
21.24
Disease
BCSCDR (40% Dict)
87.66
29.06
43.65
RC5CDR (60% Dict)
88.42
40.02
55.10
Type
BCSCDR (80% Dict)
89.42
53.85
67.22
proto
BCSCDR (100% Dict)
89.45
66.95
76.58
CoNLL03 (20% Dict)
81.49
10.55
18.67
CoNLL03 (40% Dict)
84.87
20.65
33.21
CoNLL03 (60% Dict)
87.58
30.68
45.44
CoNLL03 (80% Dict)
89.02
42.42
57.46
CoNLLO3 (100% Dict)
91.62
52.15
66.47
CoNLLO3 (KB)
82.31
62.17
70.84
Table The distant annotation quality (span-level pre- cision/recall/f1) of the datasets generated with different dictionaries/KBs.
Figure 4: The t-SNE visualization of token features and prototypes on BCSCDR (Big Dict). We omit the enormous O tokens for the clarity of demonstration.
NER datasets with low-coverage dictionaries.
3.6 Visualization of Tokens and Prototypes
60
We get the token features and the embedding of the prototypes from the best checkpoint on the dev
set. To visualize high-dimensional features, we
Tetal woode
node rerttivebddd
convert token features and prototype embeddings to 2-dimensions by (-SNE.
(a) Entity token-prototype (b) trken-prototype similar- similarity
As illustrated in Figure 4, we can conclude that: (1) The intra-class variance is common phe- nomenon in the NER task. The token features of Disease class form two sub-clusters. Representing each entity type with only a single prototype can- not cover token features in all sub-clusters. There- networks cannot characterize
ity
Figure 5: The cosine similarity curve between tokens and their actual prototypes over training steps
3.7 Effectiveness of Denoised Optimal Transport
fore, single-protoyye the intra-class variance and the performance will suffer. (2) Our MProto can model the intra-class variance in the NER task. For each sub-cluster of Disease, there exists at least prototype to represent that sub-cluster. In this way, the perfor- mance of the NER method can benefit from the multiple-proootype network.
To validate the efficacy of our DOT algorithm, we present the similarity between tokens and proto- types over training steps in Figure 5. The token- prototype similarity of each entity type is obtained by:
sim
max s(x, P Pamm
(11)
More visualizations on different datasets and MProto with different prototypes can found in Appendix C.
1xel
XEX=
where Xe is the set tokens whose actual class is

---

è (the actual class of token is obtained from human annotation of CoNLL03) and denotes the cosine similarity function. As can be observed from the figure, when the model is trained using the DOT, the similarity between entity tokens and their actual prototypes cxhibits consistent increase through- out the training. thereby bolstering the entity token classification performance. In contrast, the model trained without DOT exhibits decrease in sim- ilarity, suggesting that the model overfits the in- complete labeling noise. Regarding O tokens, the similarity between O tokens and O prototypes is higher when trained without DOT. This implies the network tends to predict the 0 class, which leads to false negative. While with our DOT algorithm, this issue can be mitigated. Based on these obser- vations,
labeling noise. Various methods have been pro- posed to address the noise in distant annotations, AutoNER (Shang et al., 2018) design a new tag- ging scheme that classifies two adjacent tokens to be tied, broken, or unknown. Token pairs labeled unknown are not considered in loss computation for denoising. Negative sampling methods (Li et al., 2021, 2022) sample 0 tokens for training mitigate the incomplete labeling noise. Positive-unlabeled learning (PU-learning) (Peng et al., 2019; Zhou et al., 2022) treats tokens labeled with O class as un- labeled samples and optimizes with an unbiasedly estimated empirical risk. Self-learning based meth- ods (Liang et al.. 2020; Zhang et al., 2021c; Meng et al., 2021) train a teacher model with distant anno- tations and utilize the soft labels generated by the teacher to train new student. Other studies also adopt causal intervention (Zhang et al., 2021a) or hyper-geometric distribution (Zhang et al., 2021b) for denoising. In our MProto, we propose a novel denoised optimal transport algorithm for the DS- NER task. Experiments show that the denoised op- timal transport can significantly mitigate the noise of incomplete labeling.
conclude that the DOT algorithm plays a important role in alleviating the noise of incomplete labeling, and thereby significantly en- hancing the performance in the DS-NER task.
4 Related Work
Named entity recognition is a fundamental task in natural language processing and has been ap- plied to various downstream tasks (Li and Ji, 2014; Miwa and Bansal, 2016; Ganea and Hofmann, 2017: Wu et al., 2020; Shen et al., 2021b; Wu et al., 2022b, 2023). NER methods can be di- vided into two main categories: tagging-based and span-based. Tagging- based methods (Lample et al., 2016; Huang et al.,, 2015) predict label for each token, which perform well at detecting flat named entities while failing at detecting nested named entities. Span-based methods (Sohrab and Miwa, 2018; Yu et al., 2020; Wang et al., 2020; Shen et al., 2021a) perform classification over span representa- tions, which performs well on the nested NER task but is inferior in computational complexity. Tan et al. (2021); Shen et al. (2022); Wu et al. (2022a) design a query-based NER framework that opti- mizes entity queries using bipartite graph match- ing. Recently, some generative NER methods (Yan et al., 2021: Shen et al., b023a,b; Lu et al., 2022) have been proposed with superior performance on various NER tasks. These supervised NER meth- ods require a large amount of annotated data for training.
Prototypical Network. Our work is also related to prototypical networks (Snell et al., 2017). Pro- totyicall networks learn prototype for each class and make predictions based on the similarity be- tween samples and prototypes. It is widely used in many tasks such as relation extraction (Ding et al., 2021) and the few-shot named entity recogni- tion (FS-NER) task (Fritzler et al., 2019; Ma et al., 2022). Previous prototypical networks for the NER task are mostly single- angle-prototype networks, which do not consider the semantic difference within the same entity type. Tong et al. (2021) try to incorpo- rate multiple prototypes for O class in the FS-NER task. However, they do not consider the intra-class variance in entity classes and their clustering strat- egy will introduce huge computational complexity when training data is large. In our work, we uti- lize multiple prototypes to represent each entity type and solve the prototype assignment effi- ciently with the sinkhorn knopp algorithm. Exper- iments confirm that our MProto can successfully describe the intra-class variance.
5 Conclusion
DS-NER. To mitigate the need for human annota- tions, distant supervision widely used. The main challenge of the DS-NER task is the label noise, of which the most widely studied is the incomplete
In this paper, we present MProto, a prototype- based network for the DS-NER task. In MProto, each cat- egory is represented with multiple prototypes to

---

model the intra-class variance. We consider the token-prototype assignment problem as an opti- mal transport problem and we apply the sinkhorn- knopp algorithm to solve the OT problem. Besides, to mitigate the noise of incomplete labeling, we propose novel denoised optimal transport algo- rithm for O tokens. Experiments show that our MProto has achieved SOTA performance on vari- DS-NER benchmarks. Visualizations and de- tailed analysis have confirmed that our MPPoto can successfully characterize the intra-class variance, and the denoised optimal transport can mitigate the noise of incomplete labeling.
Ning Ding, Xiaobin Wang, Yao Fu, Guangwei Xu, Rui Wang, Pengjun Xie, Ying Shen, Fei Huang, Hai-Tao Zheng, and Rui Zhang. 2021. Prototypical represen- tation learning for relation extraction. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named en- tity recognition task. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC '19, pages 993-1000, New York, NY, USA. Association for Computing Machinery.
Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing. pages 2619-2622 Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Limitations
The intra-class variance is actually a common phe- nomenon in the NER task. Currently, only try to utilize the MProto on the distantly supervised NER task. Applying MProto the few-shot NER task and the fully supervised NER task can be further investigated.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional LSTM-CRE Models for Sequence Tagging. ArXiv preprint, abs//508..m 508.01991.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 260-270, San Diego, California. Association for Computational Linguistics.
Acknowledgements
This work is supported by the Fundamental Re- search Funds for the Central Universities (No. 226- 2023-00060), National Natural Science Founda- tion of China (No. 62376245), Key Research and Development Program of Zhejiang Province (No. 2023C01 152), National Key Research and Devel- opment Project of China (No. 201 8AAA0101900), Joint Project DH-2022ZY0013 from Donghai Lab, and MOE Engineering Research Center of Digital Library.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: A pre-trained biomedical language representation model for biomedical text mining, Bioinformatics (Oxford, England), 36(4):12344 1240.
Qi Li and Heng Ji. 2014, Incremental joint extraction of entity mentions and relations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pages 402-412, Baltimore, Maryland. Association for Computational Linguistics.
References
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems 26: 27th An- nual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held Decem- ber 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2292 23000.
Yangming Li, Lemao Liu, and Shuming Shi. 2021. Em- pirical analysis of unlabeled entity problem in named entity recognition. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Yangming Li, Lemao Liu, and Shuming Shi. 2022. Re- thinking negative sampling for handling missing tity annotations. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Papers). pages Dublin, Ireland. Association for Computational Lin- guistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171 1171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Rui- jia Wang. Tuo Zhao, and Chao Zhang. 2020. BOND: bert-assisted open-domain named entity recognition

---

with distant supervision. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 1054-1064, ACM.
Framework for Joint Entity and Relation Extraction. In Proceedings of the Web Conference 2021. WWW "21, pages 1704-1715, New York, NY, USA, Associ- ation for Computing Machinery.
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction, Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 3755-5772. Dublin, Ireland. Association for Computational Linguisties.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023a. Diffusion- NER: Boundary diffusion for named entity recogni- tion. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 3875- -3950, Toronto, Canada. Association for Computational Linguistics.
Tingting Ma, Huiqiang Jiang, Qianhui Wu, Tiejun Zhao, and Chin- Lin. 2022. Decomposed meta- learning for few-shot named entity recognition. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 1584-1596, Dublin, Ire- land. Association for Computational Linguistics.
Yongliang Shen. Zeqi Tan, Shuhui Wu, Wenqi Zhang. Rongsheng Zhang. Yadong Xi, Weiming Lu, and Yueting Zhuang. 2023b. PromptNER: Prompt lo- cating and typing for named entity recognition. In Proceedings of the s Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Pupers), pages 12492- 12507, Toronto, Canada. Association for Computational Linguisties.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang. Yu Zhang, Heng Ji, and Jiawei Han. 2021. Distantly- supervised named entity recognition with noise- robust learning and language model augmented self- training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 10367-10378, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yuet- ing Zhuang. 2022. Parallel instance query network for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics |Volume I: Long Papers), pages 947-961. Dublin, Ireland. Association for Computa- tional I. Linguisties.
Makoto Miwa and Mohit Bansal. 2016. End-to-end re- lation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pages I 1105-1116, Berlin, Germany. Association for Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. In Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Process- ing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
Minlong Peng. Xiaoyu Xing. Qi Zhang, Jinlan Fu, and Xuanjing Huang. 2019. Distantly supervised named entity recognition using uosiaiee-unlabeled learning. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 2409- 2419, Florence, Italy. Association for Computational Linguistics.
Mohammad Golam Sohrab and Makoto Miwa, 2018 Deep exhaustive model for nested named entity recognition. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 2843-2849, Brussels, Belgium. Associa- tion for Computational Linguistics.
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named en- tity tagger using domain-specific dictionary. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 2054 2064, Brussels, Belgium. Association for Computa- tional Linguistics.
Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, and Yueting Zhuang. 2021. A Sequence-to-Set Net- work for Nested Named Entity Recognition. In Pro- ceedings of the Thirtieth International Joint Con- ference on Artificial Intelligence, pages 3936-3942, Montreal, Canada. International Joint Conferences on Artificial Intelligence Organization.
Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021a. Locate and label: A two-stage identifier for nested named en- tity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 2782-2794, Online. Association for Computational Linguistics.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language inneuedddddde named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002).
Mcihan Tong. Shuai Wang, Bin Xu, Yixin Cao, Minghui Liu, Lei Hou, and Juanzi Li. 2021. Learning from miscellaneous other-class words for few-shot named entity recognition. In Proceedings s of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 11th International Joint Conference
Yongliang Shen, Xinyin Ma, Yechun Tang, and Weim- ing Lu. 202lb. A Trigger-Sense Memory Flow

---

on Natural Language Processing (Volume 1: Long Papers), pages -237, Online. Association for Computational Linguistics.
recognition via causal intervention. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Voiume I: Long Papers), pages 4803-4813, Online. Association for Computational Linguisties.
Jue Wang, Lidan Shou, Ke Chen, and Gang Chen, 2020. Pyramid: A layered model for nested named entity recognition. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 5918-599 11888,99,0000 Online. Association for Computa- tional Linguistics.
Wenkai Zhang, Hongyu Lin, Xianpei Han, Le Sun, Huidan Liu, Zhicheng Wei, and Nicholas Yuan. 2021b. Denoising Distantly Supervised Named En- tity Recognition via Hypergeometric Probabilistic Model. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 35(16): 14144444144444
Chih-Hsuan Wei, Yifan Peng. Robert Leaman, Allan Pe- ter Davis, Carolyn J. Mattingly, Jiao Li, Thomas C. Wiegers, and Zhiyong Lu. 2016. Assessing the state of the art in biomedical relation extraction: Overview of the BioCreative V chemical-diseas re- lation (CDR) task. Database: The Journal of Biolog- ical Databases and Curation, 2016:baw032.
Xinghua Zhang, Bowen Yu, Tingwen Liu, Zhenyu Zhang. Jiawei Sheng, Xue Mengge, and Hongbo Xu. 2021c. Improving entity recognition with self-collaborative denoising learning. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Process- ing, pages 10746-10757, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
-supervvid named
Shuhui Wu, Yongliang Shen, Zeqi Tan, and Weiming Lu. 2022a. Propose-and-Reffee A Two-Stage Set Predic- tion Network for Nested Named Entity Recognition. In Thirty-First International Joint Conference on Ar- tificial Intelligence, volume 5, pages 4418-4424.
Kang Zhou, Yuepei Li, and Qi Li. 2022. Distantly supervised named entity recognition via confidence- based multi-clas class positive and unlabeled learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 191-121,, Dublin, Ireland. Association for Computational Linguistics.
Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court's view generation with causality. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 763-780.
Yiquan Wu, Yifei Liu, Weiming Lu, Yating Zhang, Jun Feng. Changlong Sun, Fei Wu, and Kun Kuang. 2022b. Towards interactivity and interpretability: rationale-based legal judgment prediction framework. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 4787-4799.
Yiquan Wu, Weiming Lu, Yating Zhang, Adam Jatowt, Jun Feng, Changlong Sun, Fei Wu, and Kun Kuang, 2023. Focus-aware response generation in inquiry conversation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12585- 12599.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5808-5822, Online. Association for Computational Linguistics.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020.
Named entity recognition as dependency parsing. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6470- 6476, Online. Association for Computational Lin- guistics.
Wenkai Zhang, Hongyu Lin, Xianpei Han, and Le Sun. 202 De-biasing distantly supervised named entity

---

A Sinkhorn-Knopp Algorithm
BCSCDR
CoNLL03
We apply the sinkhorn-knopp algorithm (Cuturi, 2013) to solve the optimal transport and denoised optimal transport claborated in equation 7 and equa- tion& 8. In order to apply the sinkhorn -onnhoo----oppppp algo- rithm, we add an entropy regularizer follows:
Big Dict
Small Diet
KB
Diet
M
3
3
3
3
X
0.05
0.1
0.05
0.01
0.9
0.5
0.5
0,9
0.01
0.01
0.05
0.01
Table 4: Hyperparameters used in different datasets.
9 = arg min
ΣΣ Nh C.j + H²)(),
(12)
Adam optimizer with weight decay 0.0001 and maximum gradient norm 1.0. The maximum learn- ing is 0.0001, and the learning rate is warmed up linearly in the first 100 steps and decayed linearly afterward. The batch size is set 32. We train the model for 10 epochs in all experiments.
s.t. 7 = a, 1=b
where X" is the weight of the regularization and H(y) Ei Ti. log(Tis) is the entropy of the assignment matrix.
We specify the pscudo-code for the sinkhorn- knopp algorithm in Algorithm 1. Here the @ cor- responds to the element-wise division, a and b are vectors that represent the weights of each sample in the source and target distributions, C is the cost matrix, X is the regularization weight, and y is the assignment matrix.
For the sinkhorn knoop algorithm used in com- puting the token-prototyys assignment, we set the regularization weight X = 0.001 and the number of iterations to 100.
C Feature Visualization
The sinkhorn-knopp algorithm mainly consists of several matrix operations which can be easily accelerated by GPU devices. And we empirically find that the sinkhorn-knopp algorithm can obtain satisfying results in few iterations in our work. Therefore. applying the sinkhorn- knopp algorithm to solve the ooken-prototype assignment problem only has slight impact on the speed of the training.
We further visualize the token features and the em- bedding of the prototypes different datasets and model setups in Figure 6.
As shown in Figure 6a and Figure 6c, we can see that even when the model is trained with only a single prototype per entity type (M = 1), to- ken features of the same entity type still tend to form different sub-cluster due to the semantic dif- ference. Therefore, it can be confirmed that the intra-class variance is common phenomenon in the NER task regardless of the model.
Algorithm Sinkhorn- Kinhhorn-Koopp Algorithm
Require: a, b, C u = 1, =
Our MProto represents each entity type with mul- tiple prototypes. As shown in Figure 4 and Figure 6b, for each sub-cluster of token features, there exists at least one prototype represent this sub- cluster. In other words, our MProto can success- fully model the intra-class variance of the entity token features. The visualizations show that repre- senting each entity type with multiple prototypes rather than a single prototype is beneficial and can significantly improve the performance in the DS- NER task.
for i in 1.....,7 do
v = b 2 K n²
u' = Ø Kv
end for
return 7 diag( (u") K diag(v")
B Implementation Details and Hyperparameters
We implement our MProto with Pytorch² and Hug- gingface Transformers All experiments are ried out single RTX-3090 with 24G graphical memory. And the training of the model can be finished in approximately hours.
D Transport Plan of Unlabeled Entities
To analyze the effectiveness of our denoised op- timal transport algorithm, we obtain the transport plan by counting the assignment result of the unla- beled entity tokens in the train set with a checkpoint of MProto at the early stage of training.
We report the hyperparameters used in differ- ent datasets in Table 4. When training, we use an
"https://pytorch.oo Phttps://huugiingffae comtressfoorrrrrrrrrr..s
As shown in Figure 7, most unlabeled entity

---

Type aVio
LOC
Tyze
Enterisse cio
ren
MProto (M 1) on BCSCDR (Big (b) MProto (M 3) CoNLLO3 (Dict) MProto (M CoNLL03 (Dict) Dict)
Figure 6: The t-SNE visualization of token features and prototypes. The visualization for MProto (M = 3) on
BC5CDR (Big Diet) can be found in Figure 4.
BC5CDR (Big Dict)
CoNLLO3 (Dict)
M
Prec.
Rec.
Pree.
Rec.
00C
2436
75.74
85.94
80.52
84.39
79.27
81.75
77.36
83.43
80.28
83.42
79.27
81.29
NSC 10c
64
1750
2774
700
77.53
85.84
81.47
84.27
80.79
82.49
4
76.78
84.71
84.53
80.10
82.25
1079
$150
73.52
84.14
78.47
82.84
77.96
80.32
76.09
85.49
80.52
84.53
79.55
81.97
HBc
Chenical
Dibaase
(a) CoNLLO3 (Dict)
(b) BCSCDR (Big Dict)
Table 5: Experiment with the different number of proto- types.
Figure 7: The transport plan for unlabeled entities. The y-axis represents the actual entity type of the unlabeled entity tokens. The x-axis represents the class of the pro- totypes that are assigned to the unlabeled entity tokens.
E Experiment on Different Number of Prototypes
tokens are assigned to prototypes their actual classes, and only a few tokens are mistakenly as- signed to the 0 prototypes. With this observation, we can confirm that before the model overfits the label noise, unlabeled entity tokens tend to be simi- lar prototypes of their actual classes in the fea- ture space. Therefore, they tend to be assigned to prototypes their actual classes in our similarity- based token- prototpp assignment. our denoised optimal transport, 0 tokens assigned to entity pro- totypes are considered as label noise, and only tokens assigned to O prototypes are considered as true negative samples. In this way, we can discrim- inate unlabeled entity tokens from clean samples. These unlabeled entity tokens ignored in loss computation as to misguide the training of the model. And we can conclude that the denoised optimal transport can effectively mitigate the in- complete labeling noise in the DS-NER task.
We try different M (the number of prototypes per type) on BC5CDR (Big Dict) and CoNLL03 (Dict) datasets. The results are reported in Table It shows that representing each entity type with pro- totypes is the optimal choice on both BC5CDR (Big Dict) and CoNLLO3 (Dict) datasets. Intu- itively, there exists an ideal M for each entity type based on the semantic complexity of that category. And too-large M or a too-small M will hurt the performance of MProto. Besides, setting a too- large M will reduce the number of tokens assigned to each prototype. this case, there might be no enough token features for learning representative prototypes, which leads to underfitting.

---

04
m-
4000
(a) PER
(b) LOC
(c) ORG
(d) MISC
Figure 8: token-prototy similarity curve of each entity type over training steps.
F Similarity Curve of Different Entity Types
To better analyze the effectiveness of denoised op- timal transport algorithm, we additionally report the similarity curve of each entity class in Figure 8. The similarity is obtained by Equation 11.


Reviews:
[{"review_id": "ipzljWq7D9", "reviewer": "Reviewer_vkaK", "summary": "", "questions": "Why did not you compare your results with the NCBI-Disease dataset, very similar to BC5CDR?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n\n***R1: Inconsistencies between the data presented and the previous work.***\n\nWe appreciate the careful examination of our work and would like to address the points you raised:\n\n+ **The differences in AutoNER F1 scores:** To ensure a fair and consistent comparison across all models, we maintained a uniform encoder and chose BERT as the encoder for both our method and all the baselines. We re-implemented the AutoNER method using the BERT encoder, and the F1 score was 79.99. This is in contrast to the 84.80 originally reported by (Shang et al. 2018) using BiLSTM. This deviation can be attributed to the difference in encoders and not to the inherent capability of the model. We will provide further details on this matter in our revised version.\n\n+ **The choice of Conf-MPU-BERT over Conf-MPU-LBiLSTM:** The rationale behind our choice was to maintain uniformity in encoder selection across all evaluated models. As previously stated, all methods were evaluated using the BERT encoder to ensure a consistent basis for comparison. Thus, we chose Conf-MPU-BERT over Conf-MPU-LBiLSTM as our baseline. We apologize for any confusion, and we will clarify this point in the revised paper.\n\n***Q1: Why not compare the results with the NCBI-Disease dataset*** \n\nThank you for your insightful question regarding the choice of dataset. While both the NCBI-Disease dataset and the BC5CDR dataset are aimed at NER task within the medical domain, there are some considerations that guided our choice:\n\n+ Our initial focus was to demonstrate the capabilities of MProto in handling distant-supervised NER tasks within a specific domain. The BC5CDR dataset offers not just annotations for disease mentions, but also extends to chemical mentions. This broader spectrum of annotations allowed us to demonstrate the comprehensive capabilities of MProto in handling distantly-supervised NER tasks within the medical domain.\n+ BC5CDR has been commonly used in many recent studies [1; 2] on distant-supervised NER, offering a benchmark for straightforward comparison with other methods.\n\nHowever, we acknowledge the value of including the NCBI-Disease dataset in our evaluation and are actively working in the process of integrating this distantly-annotated dataset for evaluation. Given the limited set of directly comparable results from existing research, it will take some time in preparing the data and performing experiments on MProto and relevant baselines. We will include the results as soon as they are available.\n\nWe believe that including results from various datasets strengthens the robustness of our method, and we appreciate your suggestion to include the NCBI-Disease dataset.\n\n[1] Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In EMNLP.\n\n[2] Kang Zhou, Yuepei Li, and Qi Li. 2022. Distantly supervised named entity recognition via confidence-based multi-class positive and unlabeled learning. In ACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clear and well written. The procedure is easy to follow.", "reasons_to_reject": "There are some inconsistencies between the data presented and the previous work.\nFor instance, the authors use BC5CDR with dictionary in (Shang et al. 2018) and compare their work with AutoNER, described right in (Shang et al. 2018). By looking at the original paper, F1 on BC5CDR is 84.80, but in this paper is 79.99 (MProto's F1 is 81.47).\nSimilarly, the paper is compared with (Zhou et al. 2022), but only in its worst configuration (Conf-MPU_BERT) and not with the best one (Conf-MPU_LBiLSTM), that performs better than MProto.", "questions_to_authors": "Why did not you compare your results with the NCBI-Disease dataset, very similar to BC5CDR?", "paper_topic_and_main_contributions": "This paper describes a method to perform distantly supervised named-entity recognition (DS-NER), called MProto, that represent more than one prototype for each NER class. In this way, variance inside a single class can be intercepted and represented.\nResults presented in the paper seem to demonstrate the effectiveness of the idea.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2hiXstsfUO", "reviewer": "Reviewer_7sbu", "summary": "", "questions": "A few questions \n1. How MProto handles label dependency between two or more consecutive tokens? In other words, how MProto handles labeling of entity mention spanning over more than one token?\n\n2. How the complexity of the model increases with fine-grained DS-NER", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n\n***R1:Limited to only two datasets with only a few numbers of entity types. Noise and corresponding challenges increase multitude with more entity types. This happens primarily due to complex entity boundaries.*** \n\nWe understand the concern regarding the limitation of using only two datasets with a limited number of entity types in our current study. However, it is worth noting that the MProto has the inherent ability to handle complex entity boundaries by representing each entity type with multiple prototypes. This enables the network to capture the diverse semantics of entities effectively, even in cases where the entity boundaries are intricate and challenging.\n\nAs demonstrated in our visualization in Figure 6, the MProto's ability to represent each entity type through multiple prototypes plays a crucial role in addressing complex entity boundaries, further indicating its potential to adapt to datasets with numerous entity types.\n\nMoreover, the ablation study presented in Table 2 outlines the MProto's superior discriminative ability compared to a single-prototype counterpart. This demonstrates its adaptability and capacity to handle increased complexities and noise in entity boundaries. We believe that theoretically, MProto can maintain and enhance its performance even as the entity boundaries become more complex.\n\nWe appreciate your feedback and will include more datasets with diverse entity types for a more thorough analysis in the revised version.\n\n***R2: How DOT impacts assignments and how with and without multiple prototypes impacts assignments?***\n\nThanks for your suggestion. To clarify the impact of DOT and multiple prototypes on assignments, we provide the assignment results of different models for unlabeled entities of BC5CDR at early stage of training in the following tables. By comparing Table1 (MProto with DOT), Table2 (MProto without DOT), and Table3 (single-prototype with DOT), we can explore the effects of both factors on the assignment process. Based on these results, we can draw two main conclusions to address the concerns raised by the reviewer:\n\n| Actual type of unlabeled tokens \\\\ Type of prototypes | Other | Chemical | Disease |\n| --- | --- | --- | --- |\n| Disease    | 0 | 2334 | 279 |\n| Chemical   | 23 | 157 | 3590 |\n\n*Table 1: Token-prototype assignment for MProto (M=3). Each row in these tables represents the assignment results for unlabeled tokens belonging to a specific actual entity type. Each column reflects the number of unlabeled tokens assigned to prototypes of a particular type.*\n\n\n| Actual type of unlabeled tokens \\\\ Type of prototypes | Other | Chemical | Disease |\n| --- | --- | --- | --- |\n| Disease    | 2613 | 0 | 0 |\n| Chemical   | 3770 | 0 | 0 |\n\n*Table 2: Token-prototype assignment for MProto (w/o DOT).*\n\n| Actual type of unlabeled tokens \\\\ Type of prototypes | Other | Chemical | Disease |\n| --- | --- | --- | --- |\n| Disease    | 0 | 2102 | 511 |\n| Chemical   | 32 | 457 | 3281 |\n\n*Table 3: Token-prototype assignment for MProto (M=1).*\n\n+ **DOT Impact on Assignments:** We find that the utilization of DOT in the model (comparing Table 1 and Table 2) significantly improves the assignment results. In the model without DOT, all unlabeled tokens are assigned to prototypes of the \"Other\" type, which indicates that the model quickly overfits to label noise. In contrast, the model with DOT demonstrates a strong capacity to distinguish between unlabeled entities and 'Other' tokens, showcasing the importance of incorporating this component in the model.\n\n+ **Multiple Prototype Impact on Assignments:** Comparing the results of the single-prototype network (Table 3) to the multi-prototype network (Table 1), we find that the multi-prototype model has better discriminative ability than its single-prototype counterpart. While the single-prototype network is still able to utilize the DOT algorithm effectively, the proportion of misclassifications (i.e., assigning \"Chemical\" tokens to \"Disease\" prototypes and vice versa) increases in the single-prototype network, indicating that having multiple prototypes is beneficial for the model's performance.\n\nIn summary, the error analysis presented in the tables highlights the importance of both DOT and multiple prototypes in improving the assignment process. DOT effectively reduces the impact of label noise, enabling a better classification of unlabeled tokens, while multiple prototypes enhance the model's discriminative ability, resulting in a lower misclassification rate.\n\n***Q1: How MProto handles label dependency between two or more consecutive tokens?***\n\nThe tagging scheme of MProto can be viewed as IO tagging scheme (In-Out tagging scheme), that is, consecutive tokens of the same type are considered a singular entity.\n\n***Q2: How the complexity of the model increases with fine-grained DS-NER?***\n\nIn the context of fine-grained DS-NER, it may seem that increasing the granularity of entities would inherently increase the complexity of the model. However, our MProto provides an adaptive solution that mitigates this expected increase in complexity. Specifically, in the realm of fine-grained DS-NER, the nuanced granularity of entities tends to naturally form fewer clusters. This characteristic allows us to reduce the number of prototypes for each entity type by adjusting the hyperparameter $M$. By carefully setting this hyperparameter in accordance with the entity granularity, our MProto can effectively manage the potential increase in complexity that might arise with fine-grained DS-NER. In essence, the adaptability of the model's hyperparameter settings serves as a counterbalance to the increased granularity of entities, ensuring that the overall complexity of the model remains manageable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Addresses a very challenging issue of noise and intra-class variance in the context of distantly supervised NER task.\n2. The authors attempted to reduce noise arising from false negative Other (O) tagged entities with the help of multiple prototype network and denoised optimal transport method.\n3. With suitable experiments, authors have established that their methods yield competitive performance against SOTA in general and domain-specific datasets.", "reasons_to_reject": "Not very strong reasons to reject it. However, a few points\n1. Limited to only two datasets with only a few numbers of entity types. Noise and corresponding challenges increase multitude with more entity types. This happens primarily due to complex entity boundaries\n2. No error analysis was presented. Though authors have shown quantitative results with and without DOT, it would be interesting to see how DOT impacts assignments. Similarly for with and without multiple prototypes\n", "questions_to_authors": "A few questions \n1. How MProto handles label dependency between two or more consecutive tokens? In other words, how MProto handles labeling of entity mention spanning over more than one token?\n\n2. How the complexity of the model increases with fine-grained DS-NER", "paper_topic_and_main_contributions": "This work focuses on distantly supervised named entity recognition (DS-NER) task. Distant supervision eliminates the expensive human annotation process but at the cost of noisy annotation. This paper proposes a prototype-based network approach for the DS-NER. The proposed method, referred to as MProto, considers multiple prototypes for each entity class. The multiple prototypes for an entity type help MProto to characterize sub-clusters within each entity type. MProto assigns each input token to one of the prototypes. The main contributions of this work are the formulation of token-prototype assignments. In the proposed formulation, this work also takes care of the issue of incomplete labeling, where entities are labeled as Other due to incomplete information in the knowledge resources used for distant supervision. Towards this, the two primary contributions are -\n1. Formulation of token-prototype assignment as an optimal transport problem\n2. To take care of the noise of incomplete labeling, a proposal of denoised optimal transport\n\nExtensive sets of experiments and analyses are performed to establish the effectiveness of the MProto.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3VMSp9tz9w", "reviewer": "Reviewer_orz4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n\n***R1: Lack of high-level or intuitive summary about the (denoised) optimal transport algorithm for dealing with unlabeled entities.***\n\nThank you for your constructive feedback. In response to your suggestion, we will add a clearer and more accessible introduction to this section that captures the essence of the DOT method, before diving into the specifics. Here is a revised high-level summary:\n\nThe denoised optimal transport (DOT) algorithm serves as a key component in our approach for dealing with unlabeled entities in Distantly Supervised Named Entity Recognition (DS-NER). Intuitively, the DOT algorithm identify and isolate unlabeled entities that are likely to be label noise. This is achieved by examining the similarity between 'Other' tokens and entity prototypes, and then assigning the tokens to their closest prototype. Only 'Other' tokens assigned to 'Other' prototypes are considered true negatives. By doing so, the algorithm can effectively distinguish between genuine unlabeled entities and label noise, ultimately reducing the risk of overfitting and improving DS-NER performance.\n\n***Q1: If the approach is token or span based?***\n\nWe apologize for the confusion. Our method is indeed a tagging-based approach that predicts a tag for each token in the text. In this process, the model assigns entity types to individual tokens, and consecutive tokens of the same type are then considered as a single entity.\n\nWe appreciate your valuable suggestion and will enhance the clarity of the paper by providing more explicit details. Additionally, we plan to incorporate the source of distant labels such as dictionaries or knowledge bases earlier in the paper for the benefit of the readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper very elegantly deals with two relevant problems - learning from unlabeled data via distant labels as well as mitigating label noise. It is an interesting read for both perspectives, could inspire similar approaches and overall is well written and presented. The argumentation is clear and the main idea comes across quickly. The experiments are well fitted and seem sound, as well as the ablation and secondary analyses which justify their main ideas and choices. I also appreciate that they present the drop when reducing the coverage. \nThe presentation (i.e. chose of sections, figures etc.) is very good and clear, also the figures in the appendix concerning token-prototype similarity are really interesting.", "reasons_to_reject": "I do not find major reasons to reject.\nThe section about the (denoised) optimal transport algorithm for dealing with unlabeled entities might be a bit hard to follow or too intricate for a reader that is unfamiliar with the matter. Of course, this is not a bad thing but rather nice to learn something new. But it might help to put a more high-level or intuitive summary at the start of the section.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents MProto, a noise-robust prototype network for distantly supervised Named Entity Recognition (DS-NER).\n\nThey use dictionary or knowledge base lookups for distantly labeling raw text with NER labels. Different to other prototypical networks they allow multiple prototypes per NER class (e.g. three \"types\" of ORG prototypes) to account for intra-class variance.\nThe prototype assignment is done via optimal transport (OT) algorithms.\n\nBeing aware that the distant labels introduce a high rate of label errors or incompleteness (due to missing entries or other), they propose a novel \"denoised optimal transport\" (DOT) algorithm to distinguish the O-assignes entitiey between falsely unlabeled entity tokens and true negative examples to mitigate the noise problem and avoid overfitting to the O-class.\n\nThey test their approach on two NER datasets: CoNLL03 and BC5CDR, replacing the real human labels with the distant ones from Dictionary of Knowledge Base for training. They compare their MProto approach with the fully-supervised BERT model (trained on the obviously better real human labels) as upper bound, as well as several other DS-NER models. Though naturally weaker than the baseline, MProto beats all the other DS-NER models in their experiments.\n\nThe authors present some ablation studies, e.g. only allowing one prototype per NER class, or removing the DOT handling of O-labeled tokens, the latter showing that this is indeed a very important feature of their approach. This is also supported by a comparison of the learned class prototypes and tokens that belong to the classes (in the real fully human annotated versions of the datasets).\nThey further notice that dictionary or knowledge base coverage is an important factor for model performance through artificially reducing coverage, which of course is a downside. However, they show that their MProto still has less performance drops compared to the other DS-NER models in comparison.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "One thing that (for me) was not that clear is if the approach is token or span based. Maybe this could be made more clear?\nAlso, naming the source of distant labels (dictionary, knowledge base) came too late in the paper, which left me a bit confused for too long. Maybe this could be moved more to the beginning?"}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the paper presents a useful model for distantly supervised NER. The proposed prototype-based network, which incorporates multiple prototypes for each entity class, was praised for its ability to capture intra-class variance. The paper also introduces a denoised optimal transport algorithm to address issues of label noise.
The reviewers acknowledged the clear presentation, solid experiments, and extensive analyses performed in the paper. 

However, there were some criticisms and areas for improvement highlighted by the reviewers. One reviewer pointed out inconsistencies in the presented data and comparisons to previous work, potentially caused by the differences in encoders (via author rebuttal). Nevertheless, this should be discussed and presented in the paper properly (e.g., does the proposed method still achieve SOTA performance?) Another reviewer suggested comparing the results with more datasets and configurations to further validate the approach. There was also a call for error analysis and a high-level summary of the denoised optimal transport algorithm to aid readers unfamiliar with the topic.
Decision: Accept-Main