End-to-end named entity recognition and relation extraction using pre-trained language models

1 INTRODUCTION . The extraction of named entities ( named entity recognition , NER ) and their semantic relations ( relation extraction , RE ) are key tasks in information extraction and retrieval ( IE & IR ) . Given a sequence of text ( usually a sentence ) , the objective is to identify both the named entities and the relations between them . This information is useful in a variety of NLP tasks such as question answering , knowledge base population , and semantic search ( Jiang , 2012 ) . In the biomedical domain , NER and RE facilitate large-scale biomedical data analysis , such as network biology ( Zhou et al. , 2014 ) , gene prioritization ( Aerts et al. , 2006 ) , drug repositioning ( Wang & Zhang , 2013 ) and the creation of curated databases ( Li et al. , 2015 ) . In the clinical domain , NER and RE can aid in disease and treatment prediction , readmission prediction , de-identification , and patient cohort identification ( Miotto et al. , 2017 ) . Most commonly , the tasks of NER and RE are approached as a pipeline , with NER preceding RE . There are two main drawbacks to this approach : ( 1 ) Pipeline systems are prone to error propagation between the NER and RE systems . ( 2 ) One task is not able to exploit useful information from the other ( e.g . the type of relation identified by the RE system may be useful to the NER system for determining the type of entities involved in the relation , and vice versa ) . More recently , joint models that simultaneously learn to extract entities and relations have been proposed , alleviating the aforementioned issues and achieving state-of-the-art performance ( Miwa & Sasaki , 2014 ; Miwa & Bansal , 2016 ; Gupta et al. , 2016 ; Li et al. , 2016 ; 2017 ; Zhang et al. , 2017 ; Adel & Schütze , 2017 ; Bekoulis et al. , 2018a ; b ; Nguyen & Verspoor , 2019 ; Li et al. , 2019 ) . Many of the proposed joint models for entity and relation extraction rely heavily on external natural language processing ( NLP ) tools such as dependency parsers . For instance , Miwa & Bansal ( 2016 ) propose a recurrent neural network ( RNN ) -based joint model that uses a bidirectional long-short term memory network ( BiLSTM ) to model the entities and a tree-LSTM to model the relations between entities ; Li et al . ( 2017 ) propose a similar model for biomedical text . The tree-LSTM uses dependency tree information extracted using an external dependency parser to model relations between entities . The use of these external NLP tools limits the effectiveness of a model to domains ( e.g . news ) where those NLP tools perform well . As a remedy to this problem , Bekoulis et al . ( 2018a ) proposes a neural , end-to-end system that jointly learns to extract entities and relations without relying on external NLP tools . In Bekoulis et al . ( 2018b ) , they augment this model with adversarial training . Nguyen & Verspoor ( 2019 ) propose a different , albeit similar end-to-end neural model which makes use of deep biaffine attention ( Dozat & Manning , 2016 ) . Li et al . ( 2019 ) approach the problem with multi-turn question answering , posing templated queries to a BERT-based QA model ( Devlin et al. , 2018 ) whose answers constitute extracted entities and their relations and achieve state-of-the-art results on three popular benchmark datasets . While demonstrating strong performance , end-to-end systems like Bekoulis et al . ( 2018a ; b ) and Nguyen & Verspoor ( 2019 ) suffer from two main drawbacks . The first is that most of the models parameters are trained from scratch . For large datasets , this can lead to long training times . For small datasets , which are common in the biomedical and clinical domains where it is particularly challenging to acquire labelled data , this can lead to poor performance and/or overfitting . The second is that these systems typically contain RNNs , which are sequential in nature and can not be parallelized within training examples . The multi-pass QA model proposed in Li et al . ( 2019 ) alleviates these issues by incorporating a pre-trained language model , BERT ( Devlin et al. , 2018 ) , which eschews recurrence for self-attention . The main limitation of their approach is that it relies on handcrafted question templates to achieve maximum performance . This may become a limiting factor where domain expertise is required to craft such questions ( e.g. , for biomedical or clinical corpora ) . Additionally , one has to create a question template for each entity and relation type of interest . In this study , we propose an end-to-end model for joint NER and RE which addresses all of these issues . Similar to past work , our model can be viewed as a mixture of a NER module and a RE module ( Figure 1 ) . Unlike most previous works , we include a pre-trained , transformer-based language model , specifically BERT ( Devlin et al. , 2018 ) , which achieved state-of-the-art performance across many NLP tasks . The weights of the BERT model are fine-tuned during training , and the entire model is trained in an end-to-end fashion . Our main contributions are as follows : ( 1 ) Our solution is truly end-to-end , relying on no handcrafted features ( e.g . templated questions ) or external NLP tools ( e.g . dependency parsers ) . ( 2 ) Our model is fast to train ( e.g . under 10 minutes on a single GPU for the CoNLL04 corpus ) , as most of its parameters are pre-trained and we avoid recurrence . ( 3 ) We match or exceed state-of-the-art performance for joint NER and RE on 5 datasets across 3 domains . 2 THE MODEL . Figure 1 illustrates the architecture of our approach . Our model is composed of an NER module and an RE module . The NER module is identical to the one proposed by Devlin et al . ( 2018 ) . For a given input sequence s of N word tokens w1 , w2 , . . . , wN , the pre-trained BERTBASE model first produces a sequence of vectors , x ( NER ) 1 , x ( NER ) 2 , . . . , x ( NER ) N which are then fed to a feed-forward neural network ( FFNN ) for classification . s ( NER ) i = FFNNNER ( x ( NER ) i ) ( 1 ) The output size of this layer is the number of BIOES-based NER labels in the training data , |C ( NER ) | . In the BIOES tag scheme , each token is assigned a label , where the B- tag indicates the beginning of an entity span , I- the inside , E- the end and S- is used for any single-token entity . All other tokens are assigned the label O . During training , a cross-entropy loss is computed for the NER objective , LNER = − N∑ n=1 log ( es ( NER ) n∑C ( NER ) c e s ( NER ) n , c ) ( 2 ) where s ( NER ) n is the predicted score that token n ∈ N belongs to the ground-truth entity class and s ( NER ) n , c is the predicted score for token n belonging to the entity class c ∈ C ( NER ) . In the RE module , the predicted entity labels are obtained by taking the argmax of each score vector s ( NER ) 1 , s ( NER ) 2 , . . . , s ( NER ) N . The predicted entity labels are then embedded to produce a sequence of fixed-length , continuous vectors , e ( NER ) 1 , e ( NER ) 2 , . . . , e ( NER ) N which are concatenated with the hidden states from the final layer in the BERT model and learned jointly with the rest of the models parameters . x ( RE ) i = x ( NER ) i ‖ e ( NER ) i ( 3 ) Following Miwa & Bansal ( 2016 ) and Nguyen & Verspoor ( 2019 ) , we incrementally construct the set of relation candidates , R , using all possible combinations of the last word tokens of predicted entities , i.e . words with E- or S- labels . An entity pair is assigned to a negative relation class ( NEG ) when the pair has no relation or when the predicted entities are not correct . Once relation candidates are constructed , classification is performed with a deep bilinear attention mechanism ( Dozat & Manning , 2016 ) , as proposed by Nguyen & Verspoor ( 2019 ) . To encode directionality , the mechanism uses FFNNs to project each x ( RE ) i into head and tail vector representations , corresponding to whether the ith word serves as head or tail argument of the relation . h ( head ) i = FFNNhead ( x ( RE ) i ) ( 4 ) h ( tail ) i = FFNNtail ( x ( RE ) i ) ( 5 ) These projections are then fed to a biaffine classifier , s ( RE ) j , k = Biaffine ( h ( head ) j , h ( tail ) k ) ( 6 ) Biaffine ( x1 , x2 ) = xT1 Ux2 + W ( x1 ‖ x2 ) + b ( 7 ) where U is an m×|C ( RE ) |×m tensor , W is a |C ( RE ) |× ( 2∗m ) matrix , and b is a bias vector . Here , m is the size of the output layers of FFNNhead and FFNNtail andC ( RE ) is the set of all relation classes ( including NEG ) . During training , a second cross-entropy loss is computed for the RE objective LRE = − R∑ r=1 log ( es ( RE ) r∑C ( RE ) c e s ( RE ) r , c ) ( 8 ) where s ( RE ) r is the predicted score that relation candidate r ∈ R belongs to the ground-truth relation class and s ( RE ) r , c is the predicted score for relation r belonging to the relation class c ∈ C ( RE ) . The model is trained in an end-to-end fashion to minimize the sum of the NER and RE losses . L = LNER + LRE ( 9 ) 2.1 ENTITY PRETRAINING . In Miwa & Bansal ( 2016 ) , entity pre-training is proposed as a solution to the problem of lowperformance entity detection in the early stages of training . It is implemented by delaying the training of the RE module by some number of epochs , before training the entire model jointly . Our implementation of entity pretraining is slightly different . Instead of delaying training of the RE module by some number of epochs , we weight the contribution of LRE to the total loss during the first epoch of training L = LNER + λ LRE ( 10 ) where λ is increased linearly from 0 to 1 during the first epoch and set to 1 for the remaining epochs . We chose this scheme because the NER module quickly achieves good performance for all datasets ( i.e . within one epoch ) . In early experiments , we found this scheme to outperform a delay of a full epoch .

Summary: The paper proposes a new joint learning algorithm that works for two tasks, NER and RE. The model is based on a pre-trained BERT model, which provides the word vectors of the input word sequence. Then it solves two tasks with two network branches: the first branch minimizes the loss for NER, and the second branch minimizes the loss for RE. The second branch uses entity labels predicted by the first branch, so joint learning may benefit both tasks.