End-to-end named entity recognition and relation extraction using pre-trained language models
Authors: John Giorgi
Xindi Wang
Nicola Sahar
Won Young Shin
Gary Bader
Bo Wang
Venue:  (2020)
Keywords: named entity recognition
relation extraction
information extraction
information retrival
transfer learning
multi-task learning
BERT
transformers
language models

Abstract: Named entity recognition (NER) and relation extraction (RE) are two important tasks in information extraction and retrieval (IE & IR). Recent work has demonstrated that it is beneficial to learn these tasks jointly, which avoids the propagation of error inherent in pipeline-based systems and improves performance. However, state-of-the-art joint models typically rely on external natural language processing (NLP) tools, such as dependency parsers, limiting their usefulness to domains (e.g. news) where those tools perform well. The few neural, end-to-end models that have been proposed are trained almost completely from scratch. In this paper, we propose a neural, end-to-end model for jointly extracting entities and their relations which does not rely on external NLP tools and which integrates a large, pre-trained language model. Because the bulk of our model's parameters are pre-trained and we eschew recurrence for self-attention, our model is fast to train. On 5 datasets across 3 domains, our model matches or exceeds state-of-the-art performance, sometimes by a large margin.

Under review as a conference paper at ICLR 2020
END-TO-END NAMED ENTITY RECOGNITION AND RE- LATION EXTRACTION USING PRE-TRAINED LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Named entity recognition (NER) and relation extraction (RE) are two important tasks in information extraction and retrieval (IE & IR). Recent work has demon- strated that it is beneficial to learn these tasks tasksjointly jointly, which avoids the propagation of error inherent in pipeline-hased systems and improves performance. However, statc-of-the -ftth--f-he-attttt joint models typically rely on external natural language process- ing (NLP) tools, such as dependency parsers, limiting their usefulness to domains (e.g. news) where those tools perform well. The few neural, end-to-end mod- els that have been proposed paper, we propose a neural, end-to-end model for jointly extracting entities and their relations which does not rely on external NLP tools and which integrates a large, pre-trained language model. Because the bulk of our model's parameters are pre-trained and we eschew recurrence for self-atter offf-attention, our model fast to train. On 5 datasets across 3 domains, our model matches or exceeds tate-offthhe-ar performance, sometimes by a large margin.
trained almost completely from scratch. In this
INTRODUCTION
The extraction of named entities (named entity recognition, NER) and their semantic relations (rela- tion extraction, RE) are key tasks in information extraction and retrieval (IE IR). Given a sequence of text (usually sentence), the objective is to identify both the named entities and the relations between them. This information useful in a variety NI.P tasks such as question answering, knowledge base population, and semantic search (Jiang) 2012). In the biomedical domain, NER and RE facilitate large-scale biomedical data analysis, such as network biology (Zhou et al., 2014). gene prioritization (Aerts al. (006), drug repositioning (Wang & Zhang 2013) and the creation of curated databases eL al. 2015). In the clinical domain, NER and RE can aid in disease and treat- ment prediction, readmission prediction, de-identification, and patient cohort identification (Miotto
et al. 2017).
Most commonly, the tasks of NER and RE are approached as a pipeline, with NER preceding RE. There are two main drawbacks to this approach: (1) Pipeline systems are prone to error propagation between the NER and RE systems. (2) One task not able to exploit useful information from the other (e.g. the type of relation identified by the RE system may be useful to the NER system for determining the type of entities involved in the relation, and vice versa). More recently, joint models that simultaneously learn to extract entities and relations have been proposed, alleviating the aforementioned issues and achieving state-of of- ff-he--n performance (Miwa & Sasaki [004 Miwa &
Bansal 2016 Gupta et al. 2016 Li et al.] 2016 2017 Zhang et al. 2017 Adel & Schütze 2017 Bekoulis et al. 2018a guyen & Verspoor 2019 et al.) 2019).
Many of the proposed joint models for entity and relation extraction rely heavily on external natural language processing (NLP) tools such as dependency parsers. For instance, MMwa & Bansal (0166) propose recurrent neural network (RNN)-based joint model that uses a bidirectional long-short term memory network (BiLSTM) to model the entities and a tree-LSTM to model the relations be- tween entities; Li al. (2017) propose similar model for biomedical text. The tree-LSTM uses dependency tree information extracted using an external dependency parser to model relations be- tween entities. The use of these external NI.P tools limits the effectiveness of model to domains

---

Under review as a conference paper at ICLR 2020
(e.g. news) where those NLP tools perform well. As remedy to this problem, Bekoolii et al. (2018a) proposes a neural, end-to-end system that jointly learns to extract entities and relations without relying external NLP tools. In Bekoulis et al. (2018b), they augment this model with ad- versarial training. Nguyen & Verspoor (2019) propose a different, albeit similar end-to-end neural model which makes use of deep biaffine attention (Dozat & Manning 2016). Li al. (2019) ap- proach the problem with multi-turn question answering, posing templated queries to BERT-based QA model (Devlin et al. 2018)) whose answers constitute extracted entities and their relations and achieve state-of-the-- art results on three popular benchmark datasets.
While demonstrating strong performance, end-to-end systems like Bekoulis et al. (2018a b) and Nguyen & Verspoor (2019) suffer from two main drawbacks. The first is that most of the models parameters are trained from scratch. For large datasets, this can lead to long training times. For small datasets, which are common in the biomedical and clinical domains where it is particularly challenging to acquire labelled data, this can lead to poor performance and/or overfitting. The sec- ond is that these systems typically contain RNNs, which are sequential in nature and cannot be parallelized within training examples. The multi-pass QA model proposed in Li et al. (2019) allevi- ates these issues by incorporating pre-trained language model, BERT (Devlin al. 2018), which eschews recurrence for self-attention. The main limitation of their approach is that it relies on hand- crafted question templates to achieve maximum performance. This may become limiting factor where domain expertise is required to craft such questions (e.g.. for biomedical or clinical corpora). Additionally, one has to create a question template for each entity and relation type of interest.
In this study, we propose an end-to-end model for joint NER and RE which addresses all of these issues. Similar past work, our model can be viewed module (Figure D. Unlike most previous works, we include a pre-trained, transformer-bass guage model, specifically BERT (Devlin et al. (018)) which achieved state-of-the oee--o--thherrrree art performance across many NLP tasks. The weights of the BERT model are fine-tuned during training, and the entire model trained in an end-to-end fashion.
mixture of a NER module and RE lan-
Our main contributions are follows: (1) Our solution is truly end-to-end, relying on no hand- crafted features (e-g. templated questions) or external NLP tools (e.g. dependency parsers). (2) Our model is fast to train (e.g. under 10 minutes of its parameters are pre-trained and we avoid recurrence. (3) We match or exceed state-of-the-art performance for joint NER and RE on 5 datasets across domains.
a single GPU for the CoNLL04 corpus), most
THE MODEL
Figure [ illlustrates the architecture of our approach. Our model is composed of an NER module and an RE module. The NER module is identical to the one proposed by Devlin et al. (2018). For a given input sequence of N word tokens W1, W2, .... wN. the pre-trained BERT BASE model first (NER) x (NER) x22
produces a sequence of vectors, æt neural network (FFNN) for classification.
(NER) which are then fed to a feed-forward
(NER
FENNNER (x) (NER))
(I)
The output size of this layer is the number of BIOES- based NER labels in the training data, (NER) In the BIOES tag scheme, each token is assigned a label, where the B- tag indicates the beginning of entity span, I- the inside, E- the end and S- is used for any single-token entity. All other tokens are assigned the label O.
During training, cross-entropy loss is computed for the NER objective,
(NER)
log
C(TRR
NER)
n=1
(NER), is the predicted score that token n E N belongs to the ground-truth entity class and where 8n
(NER) is the predicted score for token belonging to the entity class (NER). 8n,e
2

---

Under review as a conference paper at ICLR 2020
NER Output Probabilitiss
RE Output
Probabilnies
Softmax
NER Module
RE Module
Feed Forward
Biaffine Classifier
Food Forward (Head)
Feod Forward (Tail)
BERT
Embedding
Inputs
Figure 1: Joint named entity recognition (NER) and relation extraction (RE) model architecture.
In the RE module, the predicted entity labels are obtained by taking the argmax of each score vector (NER) (NER) N (NER) The predicted entity labels then embedded produce sequence 81 82
of fixed-length, continuous vectors, e (NER) e2
(NER) which are concatenated with the (NER) eN
hidden states from the final layer in the BERT model and learned jointly with the rest of the models parameters.
(RE)
(NER)
(NER)
e;
Following Miwa & Bansal (2016) and Nguyen & Verspoor 2019), we incrementally construct the set of relation candidates, R, using all possible combinations of the last word tokens of predicted entities, i.e. words with E- or S- labels. An entity pair is assigned to a negative relation class (NEG) when the pair has no relation or when the predicted entities are not correct. Once relation candidates are constructed, classification is performed with a deep bilinear attention mechanism (Dozat & Manning 2016), as proposed by Nguyen & Verspoor (011)).
To encode directionality, the mechanism uses FFNNs to project each æ; (RE) into head and tail vector representations, corresponding to whether the 1th word serves as head or tail argument of the relation.
h
(head)
FFNN looad (x)
(RE)
(4)
(tail)
FFNN
(x: (RE)
(5)
These projections are then fed to a biaffine classifier.
(RE) Biaffine (h. (heal) j..k
ht (wall)
(6)
Biaffine(z (1,, #2) = of Un + (an x22) +
where U is an m x (RE) X m tensor, W is (RE) x (2 x m) matrix, and b is a bias vector. Here, m is the size of the output layers of FFNN and FFNN and (RE) is the set of all relation classes (including NEG). During training, a second cross-entropy loss computed for the RE objective
R Σ
CRE
log
(8)
(RE)
8
RRE) T.C
r=1
3

---

Under review as a conference paper at ICLR 2020
where (RE) is the predicted that relation candidate R belongs the ground-truth relation (RE) class and 81.0 is the predicted score for relation belonging to the relation class E (
The model is trained in end-to-end fashion to minimize the sum of the NER and RE losses.
C = LNER + LRE
2.1 ENTITY PRETRAINING
In Miwa & Bansal 2016), entity pre-training is proposed solution to the problem of low- performance entity detection in the early stages of training. It is implemented by delaying the training of the RE module by some number epochs, before training the entire model jointly.
Our implementation of entity pretraining is slightly different. Instead of delaying training of the RE module by some number of epochs, we weight the contribution of LRE the total loss during the first epoch of training
C LERR + À CRE
(10)
where À is increased linearly from 0 to 1 during the first epoch and set to for the remaining epochs. We chose this scheme because the NER module quickly achieves good performance for all datasets (i.e. within one epoch). In early experiments, we found this scheme to outperform delay of full epoch.
2.2 IMPLEMENTATION
We implemented our model in PyTorch (Paszke et al. 2017) using the BERT model from the PyTorch Transformers library Our model is available at our GitHub repository we use NVIDIAs automatic mixed precision (AMP) library Apex to speed up training and reduce memory usage without affecting task-specific performance.
[ Furthermore,
EXPERIMENTAL SETUP
3.1 DATASETS AND EVALUATION
To demonstrate the generalizability of our model, we evaluate on commonly used benchmark corpora across 3 domains. All corpora in English. Detailed corpus statisties presented in Table A.1 of of the appendix.
3.1.1 ACE04/05
The Automatic Content Extraction (ACE04) corpus was introduced by Doddington al. (2004), and is commonly used to benchmark NER and RE methods. There are 7 entity types and 7 relation types. ACE05 builds on ACE04, splitting the Physical relation into two classes (Physical and Part- Whole), removing the Discourse relation class and merging Employment-Member heimmbbershi-Suussiiinnn and www.-OnnnnnitoonAflliiiiiiii
into one class (Employment Seltloynenn--emmerssiiee SSbsiddarrr...)
For ACE04, we follow Miwa & Bansal (016)) by removing the Discourse relation and evaluating our model using 5-fold cross-validation on the bnews and nwire subsets, where 10% of the data was held out within each fold as validation set. For ACE05, we use the same test split as Miwa & Bansal (2016). We use 5-fold cross-validation on the remaining data choose hyperparameters. Once hyperparameters
chosen, train the combined data from all the folds and evaluate the test set. For both corpora, we report the micro-averaged F, score. We obtained the pre- pre-processing scripts from Miwa & Bansal 2016
httpss//gitthb...o/////////
htt/:////////////aaaaaataa//aaa https://iithub.com/ttttt

---

Under review as a conference paper at ICLR 2020
3.1.2 CoNLLO4
The CoNLLO4 corpus was introduced in Roth & Yih (2004) and consists of articles from the Wall Street Journal (WSJ) and Associated Press (AP). There 4 entity types and relation types.
We use the same test set split as Miwa & Sasaki 42114)) We use 5-fold cross-validation on the remaining data choose hyperparameters. Once hyperparameters are chosen, we train on the com- bined data from all folds and evaluate on the test set, reporting the micro-averaged Fj score.
3.1.3 ADE
The adverse drug event corpus was introduced by Gurulingappa al. (2012) serve as benchmark for systems that aim to identify adverse drug events from free-text. consists of the abstracts of medical case reports retrieved from PubMec There are two entity types, Drug and Adverse effect and one relation type, Adverse drug event.
Similar to previous work (Li al. 2016 2017 Bekoulis al. 2018b), we remove 130 relations with overlapping entities and evaluate our model using 10-fold cross- voss-validation, where 10% of the data within each fold was used as validation set, 10% as test set and the remaining data used as train set. We report the macro Fi score averaged across all folds.
3.1.4 12B2
The 2010 12b2/VA dataset was introduced by [Uzuner et al. (2011) for the 2010 i2b2/Va Workshop on Natural Language Processing Challenges for Clinical Records. The workshop contained an NER task focused on the extraction of medical entity types (Problem, Treatment, Text) and an RE task for 8 relation types.
In the official splits, the test set contains roughly twice as many examples as the train set. increase the number of training examples while maintaining rigorous evaluation, we elected to perform 5- fold cross-validation on the combined data from both partitions. We used 10% the data within each fold as validation 20% as test set and the remaining data was used train set. We report the micro F₁ score averaged across all folds.
To the best of our knowledge, 12b2/VA dataset. Therefore, we decided to compare to scores obtained by independent NER and RE systems. We note, however, that the scores of independent RE systems are not directly comparable to the scores we report in this paper. This is because RE is traditionally framed as sentence-level classification problem. During pre-processing, each example is permutated into processed examples containing two "blinded" entities and labelled for one relation class. E.g. the example: "His PCP had recently started siprofloxacinyReAtMESS @ @TREATMENTS for @ @PROBLEMS" where the model is trained to predict the target relation type, "Treatment is administered for medical problem" (TrAP).
the first evaluate joint NER and RE model the 2010
for U'IPROBLEM becomes "His PCP had recently started
This task is inherently easier than the joint setup, for two reasons: relation predictions are made on ground-truth entities, opposed to predicted entities (which are noisy) and the model only required to make one classification decision per pre-processed sentence. In the joint setup, model must identify any number of relations (or the lack thereof) between all unique pairs of predicted entities in given input sentence. To control for the first of these differences, report scores from our model in two settings, once when predicted entities are used as input to the RE module, and when ground-truth entities used.
3.2 HYPERPARAMETERS
Besides batch size, learning rate and number of training epochs, we used the same hyperparameters across all experiments (see Table A.2). Similar to Devlin et al. (2018), learning rate and batch size were selected for each dataset using a minimal grid search (see See Table A.3).
https://www.ntt nlm www.m.i..go.....
5

---

Under review as a conference paper at ICLR 2020
One hyperparameter selected by hand was the choice of the pre-trained weights used initialize the BERT BASE model. For general domain corpora, we found the cased BERTBASE weights from Devlin et al. (0188) to work well. For biomedical corpora, we used the weights from BioBERT (Lee al. 2019 which recently demonstrated state-of-the-art performance for biomedical NER, RE and QA. Similarly, for clinical corpora we use the weights provided by |Png et al. (2019), who pre-trained BERT on PubMed abstracts and clinical notes from MIMIC-II
4 RESULTS
4.1 JOINTLY LEARNING NER AND RE
Table I shows results in comparison previously published results, grouped by the domain of the evaluated corpus. We find that on every dataset besides i2b2, our model improves NER performance, for an average improvement of ~2% This improvement is particularly large the ACE04 and ACE05 corpora (3.98% and 2.41 % respectively). On i2b2, our joint model performs within 0.29% of the best independent NER solution.
For relation extraction, we outperform previous methods on 2 datasets and come within ~2% on both ACE05 and CoNLL04. In two cases, our performance improvement is substantial, with im- provements of 4.59% and 10.25% on the ACE04 and ADE corpora respectively. For i2b2, our score not directly comparable to previous systems (as discussed in section 3.1.4) but will facilitate future comparisons of joint NER and RE methods on this dataset. By comparing overall performance, we find that our approach achieves new state-of-the performance for popular benchmark datasets (ACE04, ACE05, ADE) and comes within 0.2% for CoNLLO4.
4.2 ABLATION ANALYSIS
To determine which training strategies and components are responsible for models performance,
we conduct an ablation analysis on the CoNLL04 corpus (Table We perform five different abla-
tions: (a) Without entity pre-training (see section 2.Ip i.e. the loss function is given by equation 9
(b) Without entity embeddings, i.c. equation becomes x (RE) = x (NER). (c) Replacing the two
neural networks, FFNN and FFNN with single FFNN (see equation 4 and
(d) Removing FFNN and FFNN entirely. (e) Without the bilinear operation, i.e. equation
becomes simple linear transformation.
Removing FFNN and FFNN,ai has, by far, the largest negative impact on performance. Inter- estingly, however, replacing FFNNhead and FFNN with single FFNN has only small negative impact. This suggests that while these layers are very important for model performance, using dis- tinct FFNNs for the projection of head and tail entities (as opposed to the same FFNN relatively much less important. The next most impactful ablation was entity pre-training, suggesting that low-performance entity detection during the early stages of training detrimental learning (see section 2.1). Finally, we note that the importance of entity embeddings is surprising, as previous study has found that entity embeddings did not help performance on the CoNLL04 corpus (Bekoulis) let al., (2018a), although their architecture was markedly different. We conclude that each of our ablated components is necessary to achieve maximum performance.
4.3 ANALYSIS OF THE WORD-LEVEL ATTENTION WRIGHTS
One advantage of including a transformer-base based language model is that we can easily visualize the attention weights with respect to some input. This visualization useful, for example, in detect- ing model bias and locating relevant attention heads (Vig, 2019). Previous works have used such visualizations to demonstrate that specific attention heads mark syntactic dependency relations and that lower layers tend to learn more about syntax while higher layers tend to encode more semantics (Raganato & Tiedemann 2018).
In Figure 2 we visualize the attention weights of select layers and attention heads from an instance of BERT fine-tuned within our model on the CoNL.L04 corpus. We display four patterns that are easily

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison previously published F₁ scores for joint named entity recognition (NER) and relation extraction (RE). Ours (gold): our model, when gold entity labels are used as input to the RE module. Ours: full, end-to-end model (see section 2 Bold: best scores. Subscripts denote
standard deviation across three runs. A: difference to our overall score.
Dataset
Method
Entity
Relation
Overall
A
General
ACE04
Miwa & Bansal (0016)
81.80
48.40
65.10
-5.69
Bekoulis al.
2018b)
81.64
47.45
64.54
-6.25
et al. 2019
83.60
49.40
66.50
-4.29
Ours
87.58o.2
53.99..1
70.79
ACE0S
Miwa & Bansal (2016)
83.40
55.60
69.50
-3.42
Zhang et al. (2017)
83.50
57.50
70.50
-2.42
Ll 2019
84.80
60.20
72.50
-0.42
Ours
87.210.3
58.63o.0
72.92
CoNLL04
Miwa & Sasaki
(2014)
80.70
61.00
70.85
-7.30
Bekoulis et al.
2018b
83.61
61.95
72.78
-5.37
Li al.
2019
87.80
68.90
78.35
0.20
Ours
89.461.0
66.83p.4
78.15
Biomedical
ADE
ei et al. 2016
79.50
63.40
71.45
-1.21
ct 2017
84.60
71.40
78,00
-9.66
Bekoulis et al.
(018b)
86.75 73
75.52
81.13
-6.53
Ours
89.56u.1
85.770.4
87.66
Clinical
i2b2*
Si et al. 20019
89.55
Peng al. (2019)
76.40
Ours (gold)
72.03o.1
Ours
89.266.1
63.02o.3
76.14
To the best of our knowledge, there are no published joint NER and RE models that evaluate on the i2b2 2010 corpus. We compare our model to the state-of-the-art for each individual task (see section 3.1.4).
We compare the scores achieved by their BERT BASE model.
Table 2: Ablation experiment results on the CoNLL04 corpus. Scores are reported as a micro-
averaged F₁ score on the validation set, averaged across three runs of 5-fold cross-validation (a)
Without entity pre-training (section 2. (b) Without entity embeddings (eq. 3). (c) Using a single
FENN in place of heaN and FFNN (eq. and [5) (d) Without FFN head and FFNN (e)
Without the bilinear operation (eq. 7 Bold: best scores. Subscripts denote standard deviation
across three runs. A: difference
full models score.
Model
Entity
Relation
Overall
A
Full model
86.32o.2
60.29
73.30
(a) w/o Entity pre-training
85.91
59. 59.44p.2
72.67
-0.63
(b) w/o Entity embeddings
86.07a.
59.52o2
72.80
-0.51
(c) Single FFNN
86.02a.1
60.13oo
73.07
-0.23
(d) w/o Head/Tail
83.93g2
54.67ux
69.30
-4.00
(c) w/o Bilinear
86.35
59.60po
72.98
-0.33
paying attention to the next and previous words, paying attention to the word itself, and paying attention to the end of the sentence. These same patterns have been found in pre-trained BERT models that have not been fine-tuned on a specific, supervised task (Vig 2019 Raganato & Tiedemann, 2018), and therefore, retained after our fine-tuning procedure.
interpreted:

---

Under review as a conference paper at ICLR 2020
Layer Head
4, Head 12
Layer 10, Head
Layer Head 7
Rudry
'stal
ittty
shot
Oowald
tro
days
after
Kennady
was
Ruby
said
whly
shat
Oowald
hio
daya
aft
Figure 2: Visualization of the attention weights from select layers and heads of BERT after was fine-tuned within our model the CoNLL04 corpus. Darker squares indicate larger atten- tion weights. Attention weights are shown for the input sentence: "Ruby fatally shot Oswald two days after Kennedy
assassinated.". The CLS and SEP tokens have been removed. Four major patterns are displayed: paying attention to the next word (first image from the left) and previous word (second from the left), paying attention to the word itself (third from the left) and the end of the sentence (fourth from the left).
To facilitate further analysis of our learned model, we make available Jupyter and Google Colabora- tory notebooks on our GitHub repository where users can use multiple views to explore the learned attention weights of our models. We use the BertViz library (Vig 2019) to render the interactive, HTML-based views and to access the attention weights used to plot the heat maps.
5 DISCUSSION AND CONCLUSION
In this paper, we introduced end-to-end model for entity and relation extraction. Our key contri- butions are: (1) No reliance on any hand-crafted features (e.g. templated questions) or external NLP tools (e.g. dependency parsers). (2) Integration of a pre-trained, transformer-basee language model. (3) Statc-of-the-ae thrt--rrtt performance on 5 datasets across 3 domains. Furthermore, our model is inher- ently modular. One can easily initialize the language model with pre-trained weights better suited for a domain of interest (e.g. BioBERT for biomedical corpora) or swap BERT for comparable language model (e.g. XLNet (Yang et al. 2019)). Finally, because of (2), our model is fast train, converging in approximately 1 hour or less on single GPU for all datasets used in this study.
Our model out-performed previous state-of-the-art performance on ADE by the largest margin (6.53%) While exciting, we believe this corpus was particularly easy to learn. The majority of sentences (~68%) are annotated for two entities (drug and adverse effect, and one relation (adverse drug event). Ostensibly, model should be able exploit this pattern to get near- near-perfect perfor- mance on the majority of sentences in the corpus. As a test, we ran our model again, this time using ground-truth entities in the RE module (as opposed to predicted entities) and found that the model very quickly reached almost perfect performance for RE on the test set (~99%%). As such, high performance on the ADE corpus is not likely to transfer to real-world scenarios involving the large-scale annotation of diverse biomedical articles.
In our experiments, consider only intra-sentence relations. However, the multiple entities within a document generally exhibit complex, inter-sentence relations. Our model is not currently capable of extracting such inter-sentence relations and therefore our restriction to intra-sentence relations will limit its usefulness for certain downstream tasks, such as knowledge base creation. We also ignore the problem of nested entities, which are common in biomedical corpora. In the future, we would like extend our model to handle both nested entities and inter-sentence relations. Finally, given that multilingual, pre-trained weights for BERT exist, we would also expect our model's performance to hold across multiple languages. We leave this question to future work.
add- -aa-rrra-caaaaaaaaa camara ready-versio
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Heike Adel and Hinrich Schütze. Global normalization of convolutional neural networks for joint entity and relation classification. In Proceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
Stein Aerts, Diether Lambrechts, Sunit Maity, Peter Van Loo, Bert Coessens, Frederik De Smet, Leon-Charles Tranchevent, Bart De Moor, Peter Marynen, Bassem Hassan, Peter Carmeliet, and Yves Moreau. Gene prioritization through genomic data fusion. Nature Biotechnology, 24(5): 537-544, may 2006. doi: 10. 10.1038/nbt1203,
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. Joint entity recognition and relation extraction multi-head selection problem. Expert Systems with Applications, 114: 34-45, 2018a.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. Adversarial training for multi-context joint entity and relation extraction. arXiv preprint arXiv:1808. 06876, 2018b.
Jacob Devlin, Ming-Woi Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
George Doddington, Alexis Mitchell, Mark Przybocki. Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. The automatic content extraction (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC"04), Lisbon, Portugal, May 2004. European Language Resources Association (ELRA). pdd
URL http://wwar 1rec- conf http://........ooeeeeeeeeee//e/.
Timothy Dozat and Christopher D) Manning. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.0172 2016.
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. Table filling multi-task recurrent neural net- work for joint entity and relation extraction. In Proceedings of COLING 2016, the 26th Interna- tional Conference on Computational Linguistics: Technical Papers, pp. 2537-2547, 2016.
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann- Apitius, and Luca Toldo. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedi- cal Informatics, 4585):885 892, 2012. ISSN 1532-0464. doi: http:///doi.rr/10..000 wwwciinneeaaiicle/p..... $153204641200062 Text Mining and Natural Language Processing in Pharmacogenomics.
20i.2012.04,,00 008. URL http:/ /www. s www.scienceooo www.scciiiiii.cee.oo..c..c. rect.
Jing Jiang. Information Extraction from Text, pp. 11-41. Springer US, Boston, MA, 2012. ISBN doi: 10. 00.1007/978-1-4144-3233
978-1-4614-3223-4.
URL https://doi.crg
1007/978 1- 4614 3223-22222
2
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining, arXiv preprint arXiv: 08746, 2019.
Fei Li, Yue Zhang, Meishan Zhang, and Donghong Ji. Joint models for extracting adverse drug events from biomedical text. In IJCAI, volume 2016, pp. 2838-2844, 2016.
Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji. A neural joint model for entity and relation extraction from biomedical text. BMC bioinformatics, 18( 1):198, 2017.
Gang Li, Karen E. Ross, Cecilia N. Arighi, Yifan Peng. Cathy H. Wu, and K. Vijay-Shanker miR- Tex: A text mining system for miRNA-gena relation extraction. PLOS Computational Biology, 10044110 sep 2015. doi: 10.137 1..131/joornal wcpp...0.....1....11111111 1004391.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. Entity-relation extraction as multi-turn question answering. arXiv preprint arXiv: (005:/999005599 2019.

---

Under review as a conference paper at ICLR 2020
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL http: http://xivvvvo/// wttp:o/ggaaabaaooeoeoe 0tt.11111110111111
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel TT Dudley, Deep learning for healthcare: review, opportunities and challenges. Briefings bioinformaticx, 2017.
Makoto Miwa and Mohit Bansal. End-to-end relation extraction using lstms on sequences and tree structures, arXiv preprint 1r11vv:601.007700 2016.
Makoto Miwa and Yutaka Sasaki. Modeling joint entity and relation extraction with table repre- sentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1858-1869, 2014.
Dat Quoc Nguyen and Karin Verspoor. End-to-end neural relation extraction using deep biaffine at- tention. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (eds.), Advances in Information Retrieval, pp. 729-738, Cham, 2019. Springer Interna- tional Publishing. ISBN 118-3-330-157122
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310-1318, 2013.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language pro- cessing: An evaluation of BERT and elmo on ten benchmarking datasets. CoRR, abs/ abs//9006 1906.05474, 2019. URL.http://arxtiii org/bbs 906. 05474
Alessandro Raganato and Jörg Tiedemann. An analysis of encoder representations in transformer- based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyz- ing and Interpreting Neural Networks for NLP, pp. 287-297, 2018.
Dan Roth and Wen-tau Yih. A linear programming formulation for global inference in natural language tasks. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) HLT-NAACL 2004, pp.
Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. Enhancing clinical concept extraction with con- textual embedding. CoRR, abs/1902.088911 2019. URL
902..
08691
Özlem Uzuner, Brett R South, Shuying Shen, and Scott L Dulal.. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552-555 2011.
Jesse Vig. multiscale visualization of attention in the transformer model. CoRR, abs/ 050.0594.. 2019. URL
Zh Yi Wang and Hong-Yu Zhang. Rational drug repositioning by medical genetics. Nature Biotechnology, 1080012000-10,,,, dec 2013. doi: 10. (0.18888/b....58.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint (tt::/996.0888 2019.
Meishan Zhang, Yue Zhang, and Guohong Fu. End-to-end neural relation extraction with global optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1730-1740, 2017.
XueZhong Zhou, Jrg Menche, Albert-László Barabási, and Amitabh Sharma. Human symp- tosssde network. Nature Communications, 5, jun 2014. doi: 10. 122.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 CORPUS STATISTICS
Table A.1 lists detailed statisties for each corpus used in this study.
Table A.1:
Detailed entity and relation counts for each corpus used in this study.
Dataset
Entity classes (count)
Relation classes (count)
General
ACE04
Person (12508), Organization
Physical (1202), Person-Social (362), Employynent-Membbsshhip
(4405), Geographical Entities
(4425), Location (614), Facility (688), Weapon (119), and Vehicle (209)
Subsidiary (1591), Agent- Artifact
(212), Person-Organizationn
Affiliation (141), Geopolitical
Entity-Afiliation (517)
ACE05
Person (20891), Organization (5627), Geographical Entities (7455), Location[1119), Facil- ity (1461), Weapon (911), and Vehicle (919)
Physical (1612).
Part-
Whole (1060), Person-Social
(615), Agent-Artifact (703), Mealoyyment-Memberhhhee Subsidiary (1922), Geopolitical Entity-Affliittin (730)
CoNLL04
Location (4765), Organization (2499), People (3918), Other (3011)
Kill (268), Live in (521), Lo- cated (406), OrgBased (452), Work for (401)
Biomedical
ADE
Drug (4979). Adverse effect (5669)
Adverse drug event (6682)
Clinical
i2b2
Problem (19664), Test (13831),
PIP (2203), TeCP (504), TeRP (3053), TrAP (2617), TrCP (526), TrIP (203), TrNAP (174), TrWP (133)
Treatment (14186)
A.2 HYPERPARAMETERR AND MODEL DETAILS
Table A.2
lists hyperparameters and model details that were held constant across all experiments.
Table A.3 lists those that were specific to each evaluated corpus.
Table A.2: Hyperparameter values and model details used across all experiments.
Hyperparameter
Value
Comment
Tagging scheme
BIOES
Single token entities are tagged with an S- tag, the begin- ning of an entity span with B- tag, the last token of an entity span with an E- tag, and tokens inside an entity span with an I- tag.
Dropout rate
0.1
Dropout rate applied to the output of all FFNNs and the attention heads of the BERT model.
Entity embeddings
128
Output dimension of the entity embedding layer.
FFNN FFNN
512
Output dimension of the FFNN and FFNN Lai layers Number of layers used in the FFNN of the NER module.
No. layers (NER module)
No. layers (RE module)
2
Number of layers used in the FENNs of the RE module.
Optimizer
A
Adam with fixed weight decay regulatization (Loshchilov & Hutter 2017).
Gradient normalization
r=1
Rescales the gradient whenever the norm goes over some threshold I' (Pascanu et al. (003)..
Weight decay
0.1
1.2 weight decay.
11

---

Under review as a conference paper at ICLR 2020
Table A.3: Hyperparameter values specific to individual datasets. Similar to Devlin al. (0188) minimal grid search was performed over the values 16, 32 for batch size and 2c-5, 3e-5, and 5e-5 for learning rate.
Dataset
No. Epochs
Batch size
Learning rate
Initial BERT weights
ACE04
15
16
2e-5
BERT Base (cased) (Devlin al. 2018
ACE05
15
32
3e-5
BERT-Bas (cased)
CoNLL04
10
16
3e-5
BERT Base (cased)
ADE
7
16
2e-5
BioBERT (cased) (Lee et al.
2019
12b2
12
16
2e-5
NCBI-BERT (uncased) (Peng al. 2019
12


Reviews:
[{"review_id": "SJlvvnJkcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an end-to-end joint model for named entity recognition (NER) and relation extraction (RE), using pre-trained language models. The model is very simple, with the key is to use BERT and take NER output as input to RE. The experimental results show the model, without the need for handcrafted features, get state-of-the-art results on five datasets. \n\nAlthough the paper is well written and shows good results, I would reject the paper because: \n- the idea is trivial and simple. I don't think there's significant novelty here: all the components are existing and combining them seems very trivial to me. \n- the good performance seems to be from BERT rather than the model's structure (table 2 suggests that). I thus think the contribution of the paper is pretty not significant. \n\nI think the paper does not fit this conference. It is better to be presented in a Demonstration section at a *ACL conference.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxKkZIRtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an end-to-end methods for jointly training named entity recognition (NER) and relation extraction (RE). The model leverage pre-trained BERT language models, making it very fast to train. The methods is evaluated on 5 standard NER+RE datasets with good performances.\n\nPros:\n\n- the paper is well written and very clear\n- the proposed model has two main advantages: (1) it is very fast to train due to the use of pre-trained BERT representations and (2) it does not depends on any external NLP tool (such as dependency parser)\n\nCons: \n\n- I think the main source of improvement comes from the BERT representations used as input. As proposed in the comments, this should be assessed in the paper by replacing BERT representations by non-contextual representations such as GloVE.\n- Without this ablation study, the contributions of this paper are to show that using BERT representations as input (1) leads to better performances for NER+RE  and (2) makes the model faster to train. This is not really surprising...", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeklIo0tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new joint learning algorithm that works for two tasks, NER and RE. The model is based on a pre-trained BERT model, which provides the word vectors of the input word sequence. Then it solves two tasks with two network branches: the first branch minimizes the loss for NER, and the second branch minimizes the loss for RE. The second branch uses entity labels predicted by the first branch, so joint learning may benefit both tasks. \n\nThe design of the architecture is novel, but it is also not groundbreaking. Each network branch is from known structures, but the combination is not proposed before. \n\nThe submission has evaluated the proposed algorithms on four datasets and improved SOTA performances. The ablation study justifies the design details. \n\nThe writing is generally clear. \n\nNow critics: \n\nAblation study: \n1. As pointed by one public comment, the ablation study should show how much improvement is from BERT vectors. \n\n2. I'd like to see another ablation study of whether RE helps NER. If you remove the RE component, does the NER performance suffer? \n\n\nWriting: \n3. how are predicted labels embedded? Do you learn a vector of each tag of BIOES and then take a weighted sum of these vectors with predicted probabilities as weights?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an end-to-end technique for named entity recognition, that uses pre-trained models so as to avoid long training times, and evaluates it against several baselines. The paper was reviewed by three experts working in this area. R1 recommends Reject, giving the opinion that although the paper is well-written and results are good, they feel the technique itself has little novelty and that the main reason the technique works well is using BERT. R2 recommends Weak Reject based on similar reasoning, that the approach consists of existing components (albeit combined in a novel way) and suggest some ablation experiments to isolate the source of the good performance. R3 recommends Weak Accept but feels it is "unsurprising" that BERT allows for faster training and higher accuracy. In their response, authors emphasize that the application of pretraining to named entity recognition is new, and that theirs is a methodological advance, not purely a practical one (as R1 suggests and other reviews imply). They also argue it is not possible to do a fair ablation study that removes BERT, but make an attempt. The reviewers chose to keep their scores after the response. Given the split decision, the AC also read the paper. It is clear the paper has significant merit and significant practical value, as the reviews indicate. However, given that three expert reviewers -- all of whom are NLP researchers at top institutions -- feel that the contribution of the paper is weak (in the context of the expectations of ICLR) makes it not possible for us to recommend acceptance at this time. 
Decision: Reject