言語処理学会 第26回年次大会 発表論文集 (2020年3月)

Reweighting in Conditional Random Fields using an
Expert-Domain Dictionary
Shusuke Tatsumi1
Yuji Matsumoto1,3
Pontus Stenetorp2
Graduate School of Science and Technology, NAIST1
Department of Computer Science, University College London2
RIKEN Center for Advanced Intelligence Project3{tatsumi.shusuke.tk0, matsu}@is.naist.jp
p.stenetorp@cs.ucl.ac.uk

1

Introduction

For Named Entity Recognition (NER) in specialized ﬁelds, it is often the case that one lacks access to gold data. Therefore, many studies have
explored techniques such as noise reduction of ‘incomplete data’ where some of the named entities aregiven false negative labels. In these studies, they assign multiple labels (all possible label candidates) totokens with incomplete annotations. The model improves the recognition performance by learning probability distributions over the given labels. However,these models still face three main challenges. Theﬁrst problem is that conﬁrmation bias causes noisepropagation. These models perform sequence labeling learning and noise reduction in the same process
and with the same resources. Therefore, the noiseat the beginning of learning propagates to the latterstages of learning. The second problem is the tradeoﬀ between Precision and Recall. When applying
noise reduction using multiple label, Recall improves,
but Precision decreases. The third problem is false
positives in the model prediction, due to mismatches
between the resource(s) used to create the multiple
labels for training relative to the evaluation data.
For these problems, we propose the following
strategies. As a strategy for problem 1, we reducenoise from diﬀerent processes and resources. Speciﬁcally, we improve the weighting process. We use ‘entity linking, knowledge graph embedding and clustering’ as another process, and ‘hierarchical dictionary’as another resource. As a strategy for problem 2, wereduce false positive and improve Precision without
decreasing Recall. As a strategy for problem 3, werecognize word sets targeted by evaluation data fromvarious word sets on the dictionary based on ‘hierarchical structure of dictionary’ and ‘incomplete data’.
We describe the contribution of this paper below.
• We proposed a method for reweighting condi-tional random ﬁelds (CRF) using a dictionary.
This method consists of two elements, ‘weighting based on entity linking’ and ‘weighting basedon dictionary hierarchy’.
• Our method exceeded the baseline in two of thethree datasets.
• ‘Entity linking’ was eﬀective, but the eﬀect of‘dictionary hierarchy’ was limited.

2

Previous work

Figure 1: Label path for related research. In this example, the named entities ‘acute’, ‘renal’, and ‘failure’ arefalse negative. Gold is the gold data path. In Simple,
we give the false negative an ‘O’ label. In U nif orm,
we assign probability scores equally to multiple labels. In
Distribution, we assign optimal probability scores to themultiple labels, so that they are close to the Gold Pass.
We indicate the level of the probability scores by the shading of the label in the ﬁgure.

NER is a sequence labeling task that takes a word
sequence X as input and outputs a label sequence
y such as BIOES. Linear-Chain CRF[4] is a kind of
this sequence labeling model. It can be trained on
fully labeled data (single-label sequences (Figure 1GoldandSimple)) to minimize the loss function Eq.1.
∑log pw (yi | Xi )L (w) = −(1)iNext, we consider a method for processing yipossible
which is label sequence of incomplete data. In a― 1559 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Figure 2: Discrimination based on hierarchical structure

previous study[1], as in Figure 1 U nif orm, they assignscores equally to all multiple labels)( probabilityC yipossible . The loss function is Eq.2.
L (w) = −∑∑logiyipossibley∈C (pw (y | Xi )(2))

On the other hand, in previous study[3], as shown in
Figure 1 Distribution, q assigns the optimal probability scores to the multiple labels so that the pathgets to be closer to the gold path. We use this modelas the base model. Eq.3 describes the loss function.
∑∑
L (w) = −logqD (y | Xi ) pw (y | Xi )possibleiy∈C (yi)(3)

2.1

Estimation q

According to previous study[3], we ﬁnd the probability distribution q (Eq.4) of the label y for the word
Xi by k-cross validation. In the Viterbi algorithm,we estimate the score Pi, yj of Xi with yj in Eq.5. Φindicates the transition probability score from label
yi to label yi+1 . As shown in Eq.6, yj is composed ofa label set that combines a range label of the namedentity and a class type of the named entity, plus ageneral term label.
qD (y | Xi ) = ∑es(Xi ,y)y∈C(y possible ) es (X, y) =n∑

Φyi ,yi+1 +i=0n∑s(Xi ,y)Pi,yi ,i=1(4)(5)Φyi ,yi+1 , Pi,yi ∈ Ryj ∈ [{B, I, E, S} × {Class T ypes} + {O}]3

Proposed method

3.1

Reweighting(6)

be predicted as ‘O’. Speciﬁcally, as shown by Eq.7,
we add the hyperparameter α to the score of label‘O’ among all multiple labels of label yi . Dictionary identiﬁcation consists of two steps as shown in
Algorithm 1.
First, we link the entity Xi to the dictionary. We
determine that Xi , which could not be linked, is ageneral term. Then we reweight with Eq.7. Next,when Xi can be linked, we identify based on the hierarchical structure. Speciﬁcally, we reweight only
those terms that are determined to be general termswith Eq.7.
Algorithm 1 ReweighRequire:W ords = [X0 , X1 , ..., Xn ]1: procedure Reweight(W ords)
2:
for all W ords do
3:
if W ord.label is Incomp then
4:
Linked ⇐ Link(W ord.character)5:
if Linked is Unlinked then6:
Weight(W ord.labels)
7:
else
8:
Identif ied ⇐ Identify(W ord)
9:
if Identif ied is General then10:
Weight(W ord.labels)Also, we found that if we continue reweighting until the end of training, the recall will not improve
because of the inﬂuence of the dictionary biases themodel too strongly towards precision. Therefore, wereweight more early in the training and less at theend of the training. We reduce the value of α, asshown in Eq.8. η is the hyperparameter, αinit is theinitial value of α, and epoch count is the number ofepochs completed.
P (Xi , yj=‘O′ ) = P (Xi , yj=‘O′ ) + α,P (Xi , yj=‘O′ ) , α ∈ R

After calculating Pi, yj in the base model, wereweight using the dictionary. Intuitively, we adjust
the weights so that when the dictionary identiﬁesXi as a general term, the label yi is more likely to― 1560 ―α = max (0, αinit − η · epoch count)

(7)
(8)

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

3.2

Entity linking to dictionary

Table 1: Datasets conﬁguration

We link entities based on Levenshtein distance.
We associate Xi with the dictionary word with theshortest distance among the dictionary words whose
Levenshtein distance from Xi is MIN DISTANCE orless. If we cannot ﬁnd such a dictionary word, we will
not associate it. Also, we do not link the word Xi ,whose word length is less than MIN CHARACTER,
to the dictionary because it has too many linking
candidates.

3.3

Hierarchical identiﬁcation

First, we describe the preparation stage. We identify whether the word Xi is a general term or a
named entity, separately from the sequence labelingmodel. Therefore, we prepare in advance using a hierarchical dictionary and incomplete data (4.2). It
consists of three steps. In the ﬁrst step, we vectorize the hierarchical structure of the dictionary using
knowledge graph embedding. In the second step, we
perform unsupervised clustering using all named entity vectors as instances. In the third step, we classify
all clusters into general term clusters and named entity clusters using incomplete data. Speciﬁcally, weclassify each cluster as a named entity cluster if theproportion of named entities in the incomplete dataexceeds a threshold µ, and as a general term cluster
otherwise.
Next, we describe the inference stage. We link theword Xi to the dictionary and ﬁnd the cluster towhich the linked word belongs. We identify Xi as a
general term if the cluster is a general term cluster,and as a named entity if it is a named entity cluster.

4Experiment settings

4.1

Datasets and Dictionary

We evaluate with the NER datasets of the life
sciences, BC5CDR 1 , NCBI-Disease2 and CHEMDNER3 . In CHEMDNER, we evaluate only TRIVIAL, SYSTEMATIC, and FAMILY among allclasses, and evaluate others as general terms. We
show the datasets conﬁguration in Table 1.
We use CTD, the database of life sciences, for theexpert domain dictionary. CTD Chemical vocabularies contain 172,861 words, and CTD Disease vocabularies contains 12,984 words.

1 http://www.biocreative.org/tasks/biocreative-v/track-3cdr/
2 https://www.ncbi.nlm.nih.gov/research/bionlp/Data/disease/3 http://www.biocreative.org/resources/biocreativeiv/chemdner-corpus/

DomainEntity Types
ArticlesMentions

4.2

BC5CDR
Biomedical
Disease, Chemical
1,500

NCBI-Disease
BiomedicalDisease793

15,935 Chemical
12,852 Disease

6,892 Disease

CHEMDNER
Biomedical
Chemical
10,00025,610 Trivial
19,138 Systematic
11,935 Family

Creating incomplete data

We randomly remove a certain number of namedentities from the named entity sets of the gold data,
and label them as ‘O’. We express the proportion ofcorrectly annotated named entities as ‘named entity
coverage ρ’. For example, if ρ = 0.6, we keep 60 % of
all named entities, remove the remaining 40 %, andgive them an ‘O’ label.

4.3

Conﬁguration details

For sequence labeling, we set k : 2, αinit : 5.0, η :0.5. In entity linking, we set M IN DIST AN CE =5, M IN CHARACT ER = 7, and are case insensitive. For the hierarchical discrimination, we setµ : 0.001, use TransE[2] for knowledge graph embedding, and k-means for unsupervised clustering. We
use Precision, Recall, and F1 as evaluation metrics.

4.4

Comparison method

Linear-Chain BiLSTM-CRF[5] processes all words
with a single label. We give the missing entity a label‘O’. Separately, we also evaluate the performance ofmodels trained on gold data.
Multi Label BiLSTM-CRF[3] is our base model.
We assign single labels to named entities which are
kept and multiple label to other words.
We compare three of our models. Ours-Linking
does not use a hierarchical structure and reweights
only with linking. Ours-Fixed α does not decrease α
but ﬁxes it. Ours-All uses all the features proposed.
We summarize these in Table 3.
In addition, in the Multi Label BiLSTM-CRF andthe our models, we initialize the label by the styleshown in Figure 1 Simple according to previousstudy[3]. We use the result to initialize q.

5

Experimental result

5.1

Comparison of NER performance

We show the results of experiments with ρ = 0.5
in Table 2. First, in comparison with the baseline trained on incomplete data, our method outperformed the baseline in terms of F1 in two out of threedatasets. Next, we compared our method to a modeltrained on gold data. Our method was comparable― 1561 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Table 2: Comparison of NER performance
AnnotationPre.

BC5CDRRec.

F1

NCBI-Disease
Pre.
Rec.
F1

CHEMDNER
Pre.
Rec.
F1

Linear Chain BiLSTM-CRF

Gold

88.18

87.58

87.88

81.95

82.29

82.12

89.37

85.57

87.43

Linear Chain BiLSTM-CRF
Multi Label BiLSTM-CRF
Ours-Linking
Ours-Fixed αOurs-All

Incomplete
Incomplete
Incomplete
Incomplete
Incomplete

78.79
74.03
81.06
84.01
80.50

74.63
85.51
83.80
78.78
84.63

76.65
79.35
82.41
81.31
82.51

82.09
72.54
79.46
80.48
77.42

60.62
77.60
76.15
63.13
76.77

69.74
74.79
77.77
70.75
77.09

89.46
77.06
80.75
83.47
79.63

66.39
81.17
78.33
72.87
78.21

76.21
79.06
79.52
77.82
78.91

Figure 3: Precision and ρFigure 4: Recall and ρ

6

Table 3: Function comparison of our models
ModelDecreasing α

Entity Linking

Hierarchical Structure

Ours-Linking
Ours-Fixed α
Ours-All

○
×
○

○
○
○

×
○
○

to the learning model with gold data, despite learning with incomplete data.

5.2Eﬀect of decreasing α

We proposed a method for reweighting CRF using
an expert domain dictionary. Our method exceeded
the baseline in two of the three datasets. In addition, ‘Linking to the dictionary’ was eﬀective, butthe eﬀect of ‘Discrimination using the hierarchicalstructure’ was limited.

The ﬁrst author worked on this study in UCL withthe support of the Ministry of Education ‘Tobitate’.

Eﬀect of hierarchical structure

We compared Ours-Linking with Ours-All and analyzed which of linking and hierarchical structure
contributed more to the performance of NER. In
theory, Ours-All has a higher Precision than OursLinking. However, our results showed that OursLinking was higher.

5.4

Conclusion

Acknowledgments

We compared Ours-Fixed α with Ours-All andfound that α reduction was eﬀective.

5.3

Figure 5: F1 and ρρ and eﬀects of the our method

We experimented with diﬀerent ρ=[0.1, 0.2, ..., 0.9] settings and analyzed the correlation between the eﬀect of our method and ρ. We
compared Multi Label BiLSTM-CRF and Ours-All
with the data set BC5CDR. We show the resultsin Figures 3-5. Experimental results show that ourmethod is eﬀective when ρ ≥ 0.4.

References[1] K. Bellare and A. Mccallum. Learning extractors
from unlabeled text using relevant databases. AAAI,2007.
[2] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston,and O. Yakhnenko. Translating embeddings for modeling multi-relational data. NIPS, 2013.
[3] Z. Jie, P. Xie, W. Lu, R. Ding, and L. Li. Bettermodeling of incomplete annotations for named entity
recognition. NAACL, 2019.
[4] J. Laﬀerty, A. McCallum, and F. Pereira. Conditionalrandom ﬁelds: Probabilistic models for segmenting
and labeling sequence data. ICML, 2001.
[5] G. Lample, M. Ballesteros, S. Subramanian,K. Kawakami, and Chris Dyer. Neural architectures
for named entity recognition. NAACL, 2016.

― 1562 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.