Multi-Task Transfer Learning
for Fine-Grained Named Entity Recognition
Masato Hagiwara1, Ryuji Tamaki2, Ikuya Yamada2
1

Octanove Labs2

Studio Ousia

Named Entity Recognition (NER)●●Few systems deal with more than 100+ types○ cf. FIGER 112 types (Ling and Weld, 2012)
Entity typing○ (Ren et al., 2016), (Shimaoka et al., 2016), (Yogatama et al., 2015)

Can we solve NER (detection and classiﬁcation)with 7,000+ types in a generic fashion?

Challenge 1: Lack of Training Data

Lack of NER datasets
annotated with AIDA

Silver-standard datasetwith YAGO annotations

Transfer learning to AIDA

Challenge 2: Large Tag Set

Cost of CRF = O(n2) (n = # of types)

Challenge 3: Ambiguity in Types
House103544360
vsHouse107971449

WorldOrganization108294696
vsAlliance108293982

Plaza108619795
vsPlaza103965456

Hierarchical Multi-label Classiﬁcation
PhysicalEntity
ObjectWhole
Artifact
Structure
Memorial
NationalMonument

YagoGeoEntity
Location
RegionDistrict
AdministrativeDistrict
Municipality
City

The Statue of Liberty in New YorkChallenge 4: Hierarchical Types

locorgperpolitician

governormayor

professionalposition

journalist

Hierarchy-aware soft loss

locGOLDorg

xWper

Soft GOLD Labels

politician
governormayor

Type confusion weight W

Cross entropy loss

ryomaveti

golipopegorclor

GOLDryomave

goli
por
pegor

loc

ti

ci
anrnorPREDcianrnorHierarchy-Aware Soft Loss

Experiments
Datasets

Settings1)●Embeddings
bert-base-cased
2-layer BiLSTM (200 hidden units)●

Type conversion
2-layer feed-forward with ReLU

●Optimization
Adam (lr = 0.001) for pre-training
BertAdam (lr = 1e-5 with 2,500 warm-up)

2)

Pre-training
OntoNotes 5.0 (subset) for detection
Silver-standard Wikipedia for classiﬁcation
Manually-annotated subset for dev.
Fine-tuning
Manually-annotated WIkipedia
Manually-ﬁxed AIDA sample data(LDC2019E04)
Manually-annotated OntoNotes 5.0(subset)

Results
Performance on validation set

Performance on test set

Method

Prec

Rec

F1

Direct

0.45

0.42

0.43

Fine-tuned

0.65

0.57

0.61

Fine-tuned
w/o loss

0.60

0.50

0.55

RunPrec

Rec

F1

1stsubmission

0.504

0.468

0.485

Afterfeedback

0.506

0.4930.499

Error Analysis●●●●Location vs GPE○ “Southern Maryland”
OK: loc.position.region, NG: gpe.provincestate.provincestate
Ethnic/national groups○ “Syrians”
OK: no annotation, NG: gpe.country.country
Type too speciﬁc○ “Obama”
OK: per.politician, NG: per.politician.headofgovernment
Type too generic○ “SANA news agency”
OK: org.commercialorganization.newsagency, NG: org

Conclusion●Multi-task transfer learning approach for ultra ﬁne-grained NER○ Transfer learning from YAGO to AIDA
○ Multi-task learning of named entity detection and classiﬁcation○ Multi-label classiﬁcation of named entity types○ Hierarchy-aware soft loss

Improvement Ideas●

Using “type name” embeddings
○
○

●
●

Gazetteers and handcrafted features
Hierarchical model○●●●e.g., per.professionalposition.spokesperson
e.g., org.commercialorganization.newsagency

BIO+loc/org/per/... -> more ﬁne-grained types

Ensemble
Post-processing
Finally... read the annotation guideline and examine the training data!

Thanks for listening!

Masato Hagiwara1, Ryuji Tamaki2, Ikuya Yamada2
12

http://www.octanove.com/

http://www.ousia.jp/en/

Octanove LabsStudio Ousia