# Research on Named Entity Recognition Based on Multi-Task Learning and Biaffine Mechanism

Authors: Wenchao Gao; Yu Li; Xiaole Guan; Shiyu Chen; Shanshan Zhao
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

Commonly used nested entity recognition methods are span-based entity recognition methods, which focus on learning the head and tail representations of entities. This method lacks obvious boundary supervision, which leads to the failure of the correct candidate entities to be predicted, resulting in the problem of high precision and low recall. To solve the above problems, this paper proposes a named entity recognition method based on multi-task learning and biaffine mechanism, introduces the idea of multi-task learning, and divides the task into two subtasks, entity span classification and boundary detection. The entity span classification task uses biaffine mechanism to score the resulting spans and select the most likely entity class. The boundary detection task mainly solves the problem of low recall caused by the lack of boundary supervision in span classification. It captures the relationship between adjacent words in the input text according to the context, indicates the boundary range of entities, and enhances the span representation through additional boundary supervision. The experimental results show that the named entity recognition method based on multi-task learning and biaffine mechanism can improve the F1 value by up to 7.05%, 12.63%, and 14.68% on the GENIA, ACE2004, and ACE2005 nested datasets compared with other methods, which verifies that this method has better performance on the nested entity recognition task.


---

## Body

## 1. Introduction

Named entity recognition tasks are mainly studied for flat entities and nested entities. In the process of many named entity recognition tasks (e.g., GENIA [ref], ACE2004 [ref], and ACE2005 [ref]), many entities may be nested, that is, there are one or more other entities inside an entity. As shown in Figure1, the sentence “Note to exclude tuberculosis” contains only the flat entity “tuberculosis”, while the entity “colon cancer” in the sentence “The patient has colon cancer” also includes the entity “colon” to form a nested form. For named entities with nested structure, due to their complex hierarchical structure, the traditional named entity model based on sequence labeling is difficult to deal with directly and effectively. Therefore, increasingly researchers began to pay attention to the problem of nested named entity recognition and proposed some models especially suitable for the task of nested named entity recognition.Figure 1
Named entity example.Sequence-based methods utilize traditional sequence labeling methods to learn nested structures. Ju et al. [1] proposed a stacked LSTM-CRF model to predict nested named entities by dynamically stacking flat NER layers to identify nested entities. Katuyar and Cardie [2] used their proposed recurrent neural network-based method to handle nested named entity recognition. Lu and Roth [3] introduced a hypergraph structure for learning nested named entities. Wang and Lu [4] further proposed a neural segmentation hypergraph to address the problem of nested entity recognition. Span-based methods are another advanced method for unified named entity recognition. The idea of this class of methods is to enumerate all possible spans and classify them. The span model of Li et al. [5] introduces a general framework by several information extraction tasks that share span representations using dynamically constructed span graphs. Sohrab and Miwa [6] enumerate all possible regions of a latent entity or span, and classify them with deep neural networks. Yu et al. [7] proposed the idea of graph-based dependency parsing to provide the model with a global view of the input through biaffine mechanism, scoring pairs of start and end tokens in a sentence. Use this tag to explore all spans so that the model can accurately predict named entities.However, these research methods have different problems. The traditional sequence labeling method identifies nested entities layer by layer. And the error of the inner entity recognition will directly lead to the wrong identification of the outer entity, which will lead to the problem of error propagation of nested entities. When the input sentence is too long or there are many entity categories in the hypergraph-based model, the hypergraph structure will become complex, resulting in difficulty in parameter optimization. The span-based method first identifies the head and tail spans of entities, constructs head-tail entity pairs, and then performs label classification. Construct head-tail pairs based on real labels during training, and predict which words are head-tail pairs during testing. This method is easy to detect nested entities in different subsequences. However, due to the emphasis on learning head and tail representations, the model lacks obvious boundary supervision for entities and lacks effective use of entity boundary information. And more entity words are not predicted, which makes the model have the problem of high precision and low recall, thus affecting the overall recognition effect. In addition, when the entity span is too long, the interactive information between the head and tail spans of the entity will gradually decay. And the problem of information interaction between the head and tail spans is also ignored to a certain extent, which affects the recognition effect.Therefore, in view of the various problems raised above, this paper proposes a named entity recognition model based on multi-task learning and double affine mechanism:(1)
To enhance boundary supervision, in addition to using the biaffine model to classify the learned head and tail spans, the model adds an additional boundary detection task to predict words as entity boundaries.(2)
The model captures the connection between adjacent words according to the context, trains the two tasks jointly under the framework of multi-task learning, and enhances the span representation through additional boundary supervision.(3)
The boundary detection module helps to generate high-quality span representations, more entity words are correctly predicted, and the recall rate of the model is improved, thereby improving the overall effect of the model.


## 2. Materials and Methods

In this work, we propose a model using Multi-Task Learning and Biaffine Mechanism (MTL-BAM). In this model, through multi-task learning, a multi-task loss is applied to simultaneously train two parts, the boundary detection module and the entity span classification module. The MTL-BAM model consists of an embedded representation module, a shared feature representation module, and a multi-task learning module. The specific model structure is shown in Figure2.Figure 2
MTL-BAM model structure.The input of the model is a sentence, and the output is the entity in the sentence and the category corresponding to the entity.Next, the research content and implementation process are described in three parts: the design of the embedding representation module, the shared feature representation module, and the multi-task learning module.


### 2.1. Embedding Representation Module

In the embedding representation module, in order to obtain the features of the input text more comprehensively, three embedding methods of BERT, CharCNN, and FastText are used. The BERT [8] method can obtain the contextual features of the sentence; the CharCNN [9] method can obtain the character-level text features; the FastText [10] method can obtain the word-level features of the sentence. Next, the three embedding methods are described in detail.


#### 2.1.1. BERT Embedding

BERT passes the words in each input sentence through the word embedding layer and converts them into vector representations. To keep the size of the vector dimension the same, padding operations are performed on training texts of different lengths before word embedding, and the length of each training text is become the same by learning from each other’s strengths. In addition to word embedding, the input of BERT also contains two embedding layers: one is sentence embedding, which is used to distinguish whether the current word belongs to sentenceA or sentence B; the other is position information embedding, which obtains the relative position information of the sentence context and expresses the sequence order in which each word appears in the sentence.The input of BERT consists of the summation of the above three embedding vectors, and the structure is shown in Figure3. Setting [CLS] and [SEP] flags for each input text, [CLS] is used to mark the beginning of a sentence and [SEP] is used to distinguish two sentences. After the above vector representation is obtained, it is input to the bidirectional encoder Transformer to realize the feature extraction of the text sequence. The BERT model is pretrained on a large number of public corpora to form a text vector representation that carries effective information. Xtlm represents the contextual embedding vector representation of the pretrained sentence at time t.Figure 3
The input representation structure of the BERT.


#### 2.1.2. CharCNN Character Embedding

The character embedding layer uses the CharCNN network encoding to map words into character-level vector representations. The specific method is: constructing the input text into a character encoding, sending it to a one-dimensional CNN model, and outputting it with a specific width after one-dimensional convolution. Perform max pooling to obtain character vector representations with specific dimensions. After the processing of the CharCNN network, the character-level vector representationXXchar is obtained, which represents the character embedding vector of the sentence at time t.


#### 2.1.3. FastText Word Embedding

The word vector model can map the sentence into a word-level vector representation. This paper uses the word vector pretrained by FastText to obtain the word representation of the sentence, whereXtword represents the word vector representation of the sentence at time t.After obtaining the character vector, word vector, and context representation, the mapped results are spliced and sent to the next network. For a sentence consisting oft tokens, output a sequence vector, the vector is shown in the following formula:(1)Xt=Xtlm;Xtchar;Xtword.Among them,t represents the current time. [; ] represents the connection, and Xtlm， Xtchar， Xtword, respectively, represent the embedding vector at time t. Xt represents the output vector at the current time t.


### 2.2. Shared Feature Representation Module

#### 2.2.1. BiLSTM [11]

The shared feature representation module first obtains BERT and other output vectors through the BiLSTM layer, to obtain more comprehensive semantic information. LSTM can effectively solve the phenomenon of vanishing gradient or exploding gradient in recurrent neural networks [12]. BiLSTM is composed of forward LSTM and backward LSTM. The LSTMs in the two directions are connected in series to obtain bidirectional word vector information. This paper adopts the BiLSTM structure to model contextual information. BiLSTM simulates the context-time interaction of sentences after obtaining the embedding vector of the embedding layer. For each sentence, the left-to-right and right-to-left order representations are computed separately.


#### 2.2.2. Head and Tail Span Representation

The head and tail span table module is based on the output of the BiLSTM layer. Use two MLPs on each hidden layer before going to the next layer, creating two different representations (s, e) as the start and end of the entity span. S carries information that identifies the head of the entity,e carries information that identifies the tail of the entity, and other redundant information is removed. The two MLP layers learn the head and tail representations of the span and the MLP layer is set to a lower dimension, so as to alleviate the overfitting phenomenon generated by the output of the BiLSTM network and obtain more features in the text.The encoded representation is input into the classifiers of MLP and softmax, which detects whether the word is the beginning or the end of an entity, and generates a span representation carrying entity head and tail information, respectively.st and et are the head and tail span representation of the entity, respectively. Their expressions are shown in formula (2) and (3), where ht represents the hidden layer output of BiLSTM, and MLPs and MLPe represent two multilayer perceptrons processing head and tail information, respectively. For st,et such token pairs, feed each such token pair to the underlying network for the associated task.(2)st=softmaxMLPsht,(3)et=softmaxMLPeht.


### 2.3. Multi-Task Learning Module

There are two subtasks in the MTL-BAM model: entity boundary detection module and entity span classification module. The two tasks are described as follows.


#### 2.3.1. Entity Span Classification Task

The model used in the entity span classification task is a biaffine mechanism. The biaffine mechanism is different from the traditional MLP mechanism, using a biaffine attention mechanism instead of bilinear, using a bilinear layer instead of two linear layers and one nonlinear layer, simpler than the traditional MLP networks. After obtaining (s, e) above, input the biaffine network to obtain a score matrix.Figure4 shows the entity span matrix constructed for the entity span classification task. In “damage to the respiratory center”, “damage” is the beginning of the entity, and “center” is the end of the entity. The constituted entity goes through the following formula to calculate all scores of the entity type contained in the current data set, and the entity type with the highest category score is clinical manifestation, then “damage to the respiratory center” is identified as the clinical manifestation entity type. This is a fixed-category classification problem, and the prior probability of the head and tail spans needs to be considered at the same time. It is known that words such as head and tail are the posterior probability of a certain category relationship, as shown in the following formula:(4)rmi=siTU1ei+U2si⊕ei+b.Figure 4
Score matrix for entity span.For entity fragmenti, rm provides the score that the current entity fragment can constitute a named entity category, in the case of restricting the entity’s start position before the end position. Among them si, ei represents the head and tail representation of the i segment, and ⊕ represents the concatenation of vectors. siT represents the transpose of the si vector. U1 indicating the posterior probability of the current word being the head and tail entity category at the same time, U2 indicating the posterior probability of the current word being the head or tail entity category, b represents the prior probability of not knowing what entity class it is. To determine the entity category spanning the head and tail of an entity, rmi represents the category score of all possible fragments that currently constitute the named entity. Then, get the category with the highest score as the category predicted for each span, as shown in the following formula:(5)y′i=argmaxrmi.After predicting the category of the entity segment, the span of all entity categories is arranged in descending order, and the postprocessing protocol is adopted: for nested entities, it is judged whether there is partial overlap between different entities, and if there is partial overlap, the entity with the highest score is retained. For theith and jth entities, si and sj, respectively, represent the starting position of the entity, ei and ej, respectively, represent the end position of the entity. If the partial overlap satisfies the case of si<sj<ei<ej, the highest-scoring entity and its category are reserved.The learning objective of entity span classification is to assign a correct class (including nonentities) to each valid interval. Therefore, it is a multiclass classification problem, and the model is optimized with Softmax cross-entropy. The loss of the model is shown in the following formulas:(6)pmic=exprmic∑c^Cexprmic^,(7)lossb=−∑t=1N∑c=1Cyiclogpmic.Among them,N represents the length of the sentence, C represents the number of entity label types, y represents the actual label type of the current word. yic is 1 if the current category is c, 0 otherwise. pmic is the output of the neural network, that is, the probability that the category is c. This output value is calculated using the Softmax mentioned above. Finally, the loss lossb of the span classification module is obtained.


#### 2.3.2. Entity Boundary Detection Task

When using the biaffine mechanism for entity span classification, the introduction of head and tail span information can easily identify nested entities. However, the span learning of head and tail makes the model lack clear boundary supervision, which reduces the number of accurate candidate entities learned and reduces the effect of entity recognition. Even for nested entities, the recognition of head and tail spans does not make the connection between internal and external entities, and the lack of accurate external boundary information will cause internal entity recognition errors and reduce the recognition effect. Therefore, a multi-task learning method is introduced, and a boundary detection module is constructed to assist entity category prediction. The boundary detection model is shown in Figure5.Figure 5
Boundary detection model.After obtaining the head and tail span representation, the shared feature representation of multi-task learning is used as input, and the ReLU activation function and Softmax classifier are input to predict boundary labels, and the training speed is faster because Softmax also incorporates the mutual exclusion information between classes. The calculation process is shown in the following formulas:(8)Ot=Us,e+b,(9)dt=softmaxOt.For each token in the sentence, hereU and b are trainable parameters. s and e represent the span representation of header information and tail information, and “,” represents the concatenation operation of vectors. dt is the calculation result of the Softmax network layer, indicating the probability that the current is “O” and “I”. We compute the loss formula (10) between the true distribution d^t and the predicted distribution dt:(10)lossd=−∑d^tlogdt.Since the model shares the same entity boundary when performing entity boundary detection and entity class judgment, the losses for the two tasks of entity boundary detection and entity span classification are jointly trained. In the training phase, the real entity boundary labels of the data are input into the model to train the entity boundary detection classifier to avoid the classifier being affected by false boundary detection during training. During the testing phase, the output of the boundary detection classifier is used to indicate which entity fragments should be considered when predicting the classification labels.


#### 2.3.3. Multi-Task Learning Loss

Therefore, the total loss of the model is the sum of the losses of entity boundary detection and entity category judgment. Total loss is defined as the following formula:(11)Multi_Loss=lossb+αlossd,α is a hyperparameter that is used as a mixing ratio parameter to control the introduction of information to control the importance of the two losses between entity boundary detection and entity category judgment.Use the boundary detection module to obtain the boundary information of the entity to obtain the internal and context information of the current entity, learn richer features through multi-task learning, optimize the head and tail span representation through the back-propagation of the loss function, and improve the classification of the entity span more accurately. The acquisition of the external information of the entity improves the recognition effect of the inner entity, and the inner entity information will also be transferred to the boundary detection model through the multi-task model to promote the boundary detection effect.By implementing the above process, the algorithm flow of this model is shown in Table1.Table 1
MTL-BAM algorithm flow.
Algorithm:MTL-BAMInput:Original datasetOutput:Named entity recognition model1:While (not traversing all original dataset sentences) do2:Xtlm is obtained using the method followed by BERT3:Xtchar is obtained using CharCNN network encoding4:The corresponding word vectorXtword is obtained through FastText5:ConnectXtlm,Xtchar,Xtword to get Xt6:While (not traversed allXt) do7:Input BiLSTM layer to get outputht8:Use two multilayer perceptrons forht to get st,et9:While (NER model parameters did not converge) do10:While (not traversed allst,et) do11:Input biaffine network training to getlossb12:Input boundary detection module training to getlossd13:Multi_Loss=lossb+αlossd


## 2.1. Embedding Representation Module

In the embedding representation module, in order to obtain the features of the input text more comprehensively, three embedding methods of BERT, CharCNN, and FastText are used. The BERT [8] method can obtain the contextual features of the sentence; the CharCNN [9] method can obtain the character-level text features; the FastText [10] method can obtain the word-level features of the sentence. Next, the three embedding methods are described in detail.


### 2.1.1. BERT Embedding

BERT passes the words in each input sentence through the word embedding layer and converts them into vector representations. To keep the size of the vector dimension the same, padding operations are performed on training texts of different lengths before word embedding, and the length of each training text is become the same by learning from each other’s strengths. In addition to word embedding, the input of BERT also contains two embedding layers: one is sentence embedding, which is used to distinguish whether the current word belongs to sentenceA or sentence B; the other is position information embedding, which obtains the relative position information of the sentence context and expresses the sequence order in which each word appears in the sentence.The input of BERT consists of the summation of the above three embedding vectors, and the structure is shown in Figure3. Setting [CLS] and [SEP] flags for each input text, [CLS] is used to mark the beginning of a sentence and [SEP] is used to distinguish two sentences. After the above vector representation is obtained, it is input to the bidirectional encoder Transformer to realize the feature extraction of the text sequence. The BERT model is pretrained on a large number of public corpora to form a text vector representation that carries effective information. Xtlm represents the contextual embedding vector representation of the pretrained sentence at time t.Figure 3
The input representation structure of the BERT.


### 2.1.2. CharCNN Character Embedding

The character embedding layer uses the CharCNN network encoding to map words into character-level vector representations. The specific method is: constructing the input text into a character encoding, sending it to a one-dimensional CNN model, and outputting it with a specific width after one-dimensional convolution. Perform max pooling to obtain character vector representations with specific dimensions. After the processing of the CharCNN network, the character-level vector representationXXchar is obtained, which represents the character embedding vector of the sentence at time t.


### 2.1.3. FastText Word Embedding

The word vector model can map the sentence into a word-level vector representation. This paper uses the word vector pretrained by FastText to obtain the word representation of the sentence, whereXtword represents the word vector representation of the sentence at time t.After obtaining the character vector, word vector, and context representation, the mapped results are spliced and sent to the next network. For a sentence consisting oft tokens, output a sequence vector, the vector is shown in the following formula:(1)Xt=Xtlm;Xtchar;Xtword.Among them,t represents the current time. [; ] represents the connection, and Xtlm， Xtchar， Xtword, respectively, represent the embedding vector at time t. Xt represents the output vector at the current time t.


## 2.1.1. BERT Embedding

BERT passes the words in each input sentence through the word embedding layer and converts them into vector representations. To keep the size of the vector dimension the same, padding operations are performed on training texts of different lengths before word embedding, and the length of each training text is become the same by learning from each other’s strengths. In addition to word embedding, the input of BERT also contains two embedding layers: one is sentence embedding, which is used to distinguish whether the current word belongs to sentenceA or sentence B; the other is position information embedding, which obtains the relative position information of the sentence context and expresses the sequence order in which each word appears in the sentence.The input of BERT consists of the summation of the above three embedding vectors, and the structure is shown in Figure3. Setting [CLS] and [SEP] flags for each input text, [CLS] is used to mark the beginning of a sentence and [SEP] is used to distinguish two sentences. After the above vector representation is obtained, it is input to the bidirectional encoder Transformer to realize the feature extraction of the text sequence. The BERT model is pretrained on a large number of public corpora to form a text vector representation that carries effective information. Xtlm represents the contextual embedding vector representation of the pretrained sentence at time t.Figure 3
The input representation structure of the BERT.


## 2.1.2. CharCNN Character Embedding

The character embedding layer uses the CharCNN network encoding to map words into character-level vector representations. The specific method is: constructing the input text into a character encoding, sending it to a one-dimensional CNN model, and outputting it with a specific width after one-dimensional convolution. Perform max pooling to obtain character vector representations with specific dimensions. After the processing of the CharCNN network, the character-level vector representationXXchar is obtained, which represents the character embedding vector of the sentence at time t.


## 2.1.3. FastText Word Embedding

The word vector model can map the sentence into a word-level vector representation. This paper uses the word vector pretrained by FastText to obtain the word representation of the sentence, whereXtword represents the word vector representation of the sentence at time t.After obtaining the character vector, word vector, and context representation, the mapped results are spliced and sent to the next network. For a sentence consisting oft tokens, output a sequence vector, the vector is shown in the following formula:(1)Xt=Xtlm;Xtchar;Xtword.Among them,t represents the current time. [; ] represents the connection, and Xtlm， Xtchar， Xtword, respectively, represent the embedding vector at time t. Xt represents the output vector at the current time t.


## 2.2. Shared Feature Representation Module

### 2.2.1. BiLSTM [11]

The shared feature representation module first obtains BERT and other output vectors through the BiLSTM layer, to obtain more comprehensive semantic information. LSTM can effectively solve the phenomenon of vanishing gradient or exploding gradient in recurrent neural networks [12]. BiLSTM is composed of forward LSTM and backward LSTM. The LSTMs in the two directions are connected in series to obtain bidirectional word vector information. This paper adopts the BiLSTM structure to model contextual information. BiLSTM simulates the context-time interaction of sentences after obtaining the embedding vector of the embedding layer. For each sentence, the left-to-right and right-to-left order representations are computed separately.


### 2.2.2. Head and Tail Span Representation

The head and tail span table module is based on the output of the BiLSTM layer. Use two MLPs on each hidden layer before going to the next layer, creating two different representations (s, e) as the start and end of the entity span. S carries information that identifies the head of the entity,e carries information that identifies the tail of the entity, and other redundant information is removed. The two MLP layers learn the head and tail representations of the span and the MLP layer is set to a lower dimension, so as to alleviate the overfitting phenomenon generated by the output of the BiLSTM network and obtain more features in the text.The encoded representation is input into the classifiers of MLP and softmax, which detects whether the word is the beginning or the end of an entity, and generates a span representation carrying entity head and tail information, respectively.st and et are the head and tail span representation of the entity, respectively. Their expressions are shown in formula (2) and (3), where ht represents the hidden layer output of BiLSTM, and MLPs and MLPe represent two multilayer perceptrons processing head and tail information, respectively. For st,et such token pairs, feed each such token pair to the underlying network for the associated task.(2)st=softmaxMLPsht,(3)et=softmaxMLPeht.


## 2.2.1. BiLSTM [11]

The shared feature representation module first obtains BERT and other output vectors through the BiLSTM layer, to obtain more comprehensive semantic information. LSTM can effectively solve the phenomenon of vanishing gradient or exploding gradient in recurrent neural networks [12]. BiLSTM is composed of forward LSTM and backward LSTM. The LSTMs in the two directions are connected in series to obtain bidirectional word vector information. This paper adopts the BiLSTM structure to model contextual information. BiLSTM simulates the context-time interaction of sentences after obtaining the embedding vector of the embedding layer. For each sentence, the left-to-right and right-to-left order representations are computed separately.


## 2.2.2. Head and Tail Span Representation

The head and tail span table module is based on the output of the BiLSTM layer. Use two MLPs on each hidden layer before going to the next layer, creating two different representations (s, e) as the start and end of the entity span. S carries information that identifies the head of the entity,e carries information that identifies the tail of the entity, and other redundant information is removed. The two MLP layers learn the head and tail representations of the span and the MLP layer is set to a lower dimension, so as to alleviate the overfitting phenomenon generated by the output of the BiLSTM network and obtain more features in the text.The encoded representation is input into the classifiers of MLP and softmax, which detects whether the word is the beginning or the end of an entity, and generates a span representation carrying entity head and tail information, respectively.st and et are the head and tail span representation of the entity, respectively. Their expressions are shown in formula (2) and (3), where ht represents the hidden layer output of BiLSTM, and MLPs and MLPe represent two multilayer perceptrons processing head and tail information, respectively. For st,et such token pairs, feed each such token pair to the underlying network for the associated task.(2)st=softmaxMLPsht,(3)et=softmaxMLPeht.


## 2.3. Multi-Task Learning Module

There are two subtasks in the MTL-BAM model: entity boundary detection module and entity span classification module. The two tasks are described as follows.


### 2.3.1. Entity Span Classification Task

The model used in the entity span classification task is a biaffine mechanism. The biaffine mechanism is different from the traditional MLP mechanism, using a biaffine attention mechanism instead of bilinear, using a bilinear layer instead of two linear layers and one nonlinear layer, simpler than the traditional MLP networks. After obtaining (s, e) above, input the biaffine network to obtain a score matrix.Figure4 shows the entity span matrix constructed for the entity span classification task. In “damage to the respiratory center”, “damage” is the beginning of the entity, and “center” is the end of the entity. The constituted entity goes through the following formula to calculate all scores of the entity type contained in the current data set, and the entity type with the highest category score is clinical manifestation, then “damage to the respiratory center” is identified as the clinical manifestation entity type. This is a fixed-category classification problem, and the prior probability of the head and tail spans needs to be considered at the same time. It is known that words such as head and tail are the posterior probability of a certain category relationship, as shown in the following formula:(4)rmi=siTU1ei+U2si⊕ei+b.Figure 4
Score matrix for entity span.For entity fragmenti, rm provides the score that the current entity fragment can constitute a named entity category, in the case of restricting the entity’s start position before the end position. Among them si, ei represents the head and tail representation of the i segment, and ⊕ represents the concatenation of vectors. siT represents the transpose of the si vector. U1 indicating the posterior probability of the current word being the head and tail entity category at the same time, U2 indicating the posterior probability of the current word being the head or tail entity category, b represents the prior probability of not knowing what entity class it is. To determine the entity category spanning the head and tail of an entity, rmi represents the category score of all possible fragments that currently constitute the named entity. Then, get the category with the highest score as the category predicted for each span, as shown in the following formula:(5)y′i=argmaxrmi.After predicting the category of the entity segment, the span of all entity categories is arranged in descending order, and the postprocessing protocol is adopted: for nested entities, it is judged whether there is partial overlap between different entities, and if there is partial overlap, the entity with the highest score is retained. For theith and jth entities, si and sj, respectively, represent the starting position of the entity, ei and ej, respectively, represent the end position of the entity. If the partial overlap satisfies the case of si<sj<ei<ej, the highest-scoring entity and its category are reserved.The learning objective of entity span classification is to assign a correct class (including nonentities) to each valid interval. Therefore, it is a multiclass classification problem, and the model is optimized with Softmax cross-entropy. The loss of the model is shown in the following formulas:(6)pmic=exprmic∑c^Cexprmic^,(7)lossb=−∑t=1N∑c=1Cyiclogpmic.Among them,N represents the length of the sentence, C represents the number of entity label types, y represents the actual label type of the current word. yic is 1 if the current category is c, 0 otherwise. pmic is the output of the neural network, that is, the probability that the category is c. This output value is calculated using the Softmax mentioned above. Finally, the loss lossb of the span classification module is obtained.


### 2.3.2. Entity Boundary Detection Task

When using the biaffine mechanism for entity span classification, the introduction of head and tail span information can easily identify nested entities. However, the span learning of head and tail makes the model lack clear boundary supervision, which reduces the number of accurate candidate entities learned and reduces the effect of entity recognition. Even for nested entities, the recognition of head and tail spans does not make the connection between internal and external entities, and the lack of accurate external boundary information will cause internal entity recognition errors and reduce the recognition effect. Therefore, a multi-task learning method is introduced, and a boundary detection module is constructed to assist entity category prediction. The boundary detection model is shown in Figure5.Figure 5
Boundary detection model.After obtaining the head and tail span representation, the shared feature representation of multi-task learning is used as input, and the ReLU activation function and Softmax classifier are input to predict boundary labels, and the training speed is faster because Softmax also incorporates the mutual exclusion information between classes. The calculation process is shown in the following formulas:(8)Ot=Us,e+b,(9)dt=softmaxOt.For each token in the sentence, hereU and b are trainable parameters. s and e represent the span representation of header information and tail information, and “,” represents the concatenation operation of vectors. dt is the calculation result of the Softmax network layer, indicating the probability that the current is “O” and “I”. We compute the loss formula (10) between the true distribution d^t and the predicted distribution dt:(10)lossd=−∑d^tlogdt.Since the model shares the same entity boundary when performing entity boundary detection and entity class judgment, the losses for the two tasks of entity boundary detection and entity span classification are jointly trained. In the training phase, the real entity boundary labels of the data are input into the model to train the entity boundary detection classifier to avoid the classifier being affected by false boundary detection during training. During the testing phase, the output of the boundary detection classifier is used to indicate which entity fragments should be considered when predicting the classification labels.


### 2.3.3. Multi-Task Learning Loss

Therefore, the total loss of the model is the sum of the losses of entity boundary detection and entity category judgment. Total loss is defined as the following formula:(11)Multi_Loss=lossb+αlossd,α is a hyperparameter that is used as a mixing ratio parameter to control the introduction of information to control the importance of the two losses between entity boundary detection and entity category judgment.Use the boundary detection module to obtain the boundary information of the entity to obtain the internal and context information of the current entity, learn richer features through multi-task learning, optimize the head and tail span representation through the back-propagation of the loss function, and improve the classification of the entity span more accurately. The acquisition of the external information of the entity improves the recognition effect of the inner entity, and the inner entity information will also be transferred to the boundary detection model through the multi-task model to promote the boundary detection effect.By implementing the above process, the algorithm flow of this model is shown in Table1.Table 1
MTL-BAM algorithm flow.
Algorithm:MTL-BAMInput:Original datasetOutput:Named entity recognition model1:While (not traversing all original dataset sentences) do2:Xtlm is obtained using the method followed by BERT3:Xtchar is obtained using CharCNN network encoding4:The corresponding word vectorXtword is obtained through FastText5:ConnectXtlm,Xtchar,Xtword to get Xt6:While (not traversed allXt) do7:Input BiLSTM layer to get outputht8:Use two multilayer perceptrons forht to get st,et9:While (NER model parameters did not converge) do10:While (not traversed allst,et) do11:Input biaffine network training to getlossb12:Input boundary detection module training to getlossd13:Multi_Loss=lossb+αlossd


## 2.3.1. Entity Span Classification Task

The model used in the entity span classification task is a biaffine mechanism. The biaffine mechanism is different from the traditional MLP mechanism, using a biaffine attention mechanism instead of bilinear, using a bilinear layer instead of two linear layers and one nonlinear layer, simpler than the traditional MLP networks. After obtaining (s, e) above, input the biaffine network to obtain a score matrix.Figure4 shows the entity span matrix constructed for the entity span classification task. In “damage to the respiratory center”, “damage” is the beginning of the entity, and “center” is the end of the entity. The constituted entity goes through the following formula to calculate all scores of the entity type contained in the current data set, and the entity type with the highest category score is clinical manifestation, then “damage to the respiratory center” is identified as the clinical manifestation entity type. This is a fixed-category classification problem, and the prior probability of the head and tail spans needs to be considered at the same time. It is known that words such as head and tail are the posterior probability of a certain category relationship, as shown in the following formula:(4)rmi=siTU1ei+U2si⊕ei+b.Figure 4
Score matrix for entity span.For entity fragmenti, rm provides the score that the current entity fragment can constitute a named entity category, in the case of restricting the entity’s start position before the end position. Among them si, ei represents the head and tail representation of the i segment, and ⊕ represents the concatenation of vectors. siT represents the transpose of the si vector. U1 indicating the posterior probability of the current word being the head and tail entity category at the same time, U2 indicating the posterior probability of the current word being the head or tail entity category, b represents the prior probability of not knowing what entity class it is. To determine the entity category spanning the head and tail of an entity, rmi represents the category score of all possible fragments that currently constitute the named entity. Then, get the category with the highest score as the category predicted for each span, as shown in the following formula:(5)y′i=argmaxrmi.After predicting the category of the entity segment, the span of all entity categories is arranged in descending order, and the postprocessing protocol is adopted: for nested entities, it is judged whether there is partial overlap between different entities, and if there is partial overlap, the entity with the highest score is retained. For theith and jth entities, si and sj, respectively, represent the starting position of the entity, ei and ej, respectively, represent the end position of the entity. If the partial overlap satisfies the case of si<sj<ei<ej, the highest-scoring entity and its category are reserved.The learning objective of entity span classification is to assign a correct class (including nonentities) to each valid interval. Therefore, it is a multiclass classification problem, and the model is optimized with Softmax cross-entropy. The loss of the model is shown in the following formulas:(6)pmic=exprmic∑c^Cexprmic^,(7)lossb=−∑t=1N∑c=1Cyiclogpmic.Among them,N represents the length of the sentence, C represents the number of entity label types, y represents the actual label type of the current word. yic is 1 if the current category is c, 0 otherwise. pmic is the output of the neural network, that is, the probability that the category is c. This output value is calculated using the Softmax mentioned above. Finally, the loss lossb of the span classification module is obtained.


## 2.3.2. Entity Boundary Detection Task

When using the biaffine mechanism for entity span classification, the introduction of head and tail span information can easily identify nested entities. However, the span learning of head and tail makes the model lack clear boundary supervision, which reduces the number of accurate candidate entities learned and reduces the effect of entity recognition. Even for nested entities, the recognition of head and tail spans does not make the connection between internal and external entities, and the lack of accurate external boundary information will cause internal entity recognition errors and reduce the recognition effect. Therefore, a multi-task learning method is introduced, and a boundary detection module is constructed to assist entity category prediction. The boundary detection model is shown in Figure5.Figure 5
Boundary detection model.After obtaining the head and tail span representation, the shared feature representation of multi-task learning is used as input, and the ReLU activation function and Softmax classifier are input to predict boundary labels, and the training speed is faster because Softmax also incorporates the mutual exclusion information between classes. The calculation process is shown in the following formulas:(8)Ot=Us,e+b,(9)dt=softmaxOt.For each token in the sentence, hereU and b are trainable parameters. s and e represent the span representation of header information and tail information, and “,” represents the concatenation operation of vectors. dt is the calculation result of the Softmax network layer, indicating the probability that the current is “O” and “I”. We compute the loss formula (10) between the true distribution d^t and the predicted distribution dt:(10)lossd=−∑d^tlogdt.Since the model shares the same entity boundary when performing entity boundary detection and entity class judgment, the losses for the two tasks of entity boundary detection and entity span classification are jointly trained. In the training phase, the real entity boundary labels of the data are input into the model to train the entity boundary detection classifier to avoid the classifier being affected by false boundary detection during training. During the testing phase, the output of the boundary detection classifier is used to indicate which entity fragments should be considered when predicting the classification labels.


## 2.3.3. Multi-Task Learning Loss

Therefore, the total loss of the model is the sum of the losses of entity boundary detection and entity category judgment. Total loss is defined as the following formula:(11)Multi_Loss=lossb+αlossd,α is a hyperparameter that is used as a mixing ratio parameter to control the introduction of information to control the importance of the two losses between entity boundary detection and entity category judgment.Use the boundary detection module to obtain the boundary information of the entity to obtain the internal and context information of the current entity, learn richer features through multi-task learning, optimize the head and tail span representation through the back-propagation of the loss function, and improve the classification of the entity span more accurately. The acquisition of the external information of the entity improves the recognition effect of the inner entity, and the inner entity information will also be transferred to the boundary detection model through the multi-task model to promote the boundary detection effect.By implementing the above process, the algorithm flow of this model is shown in Table1.Table 1
MTL-BAM algorithm flow.
Algorithm:MTL-BAMInput:Original datasetOutput:Named entity recognition model1:While (not traversing all original dataset sentences) do2:Xtlm is obtained using the method followed by BERT3:Xtchar is obtained using CharCNN network encoding4:The corresponding word vectorXtword is obtained through FastText5:ConnectXtlm,Xtchar,Xtword to get Xt6:While (not traversed allXt) do7:Input BiLSTM layer to get outputht8:Use two multilayer perceptrons forht to get st,et9:While (NER model parameters did not converge) do10:While (not traversed allst,et) do11:Input biaffine network training to getlossb12:Input boundary detection module training to getlossd13:Multi_Loss=lossb+αlossd


## 3. Results and Discussion

### 3.1. Experimental Environment Parameter Settings

This paper uses the Windows system for experiments, based on the Python platform, using pycharm as a development tool. The model is constructed using the open source deep learning framework Tensorflow, which is developed and maintained by Google’s artificial intelligence team, Google Brain. It is deployed on various servers, PC terminals, and web pages, and supports GPU high-performance numerical computing.The named entity recognition model (MTL-BAM) based on multi-task learning and biaffine mechanism is experimented on two flat entity datasets JNLPBA [13], CoNLL2003, and three nested entity datasets GENIA [14], ACE2005, and ACE2004 [15]. It is compared with BAM that only uses a biaffine mechanism for entity recognition without using a multi-task learning framework, and the experimental results are compared with other models that achieve some results in the named entity recognition tasks. The evaluation indicators used are the precision rate P, the recall rate R, and the F value.The experimental parameter settings in this paper are shown in Table2.Table 2
Experimental parameter setting.
ParameterValueBiLSTM size200BiLSTM layer3BiLSTM dropout0.4MLP size150MLP dropout0.2BERT size1024FastText embedding size300CharCNN filter widths[3–5]Char embedding size50Embeddings dropout0.5OptimiserAdamLearning rate0.001The convolution kernel window size of the convolutional layer of the CharCNN network is set to [3–5], and the character vector dimension is 50 dimensions. Use the public pretrained word vector FastText as the choice of word vector, and the word vector dimension is set to 300 dimensions. BiLSTM sets the output dimension of 3 layers to 200 dimensions, and the dimension of two fully connected layers is set to 150 dimensions, reducing the output dimension of LSTM to prevent overfitting. At the same time, to prevent overfitting, the BiLSTM layer and the MLP layer, respectively, set the dropout [16] to 0.4 and 0.2. The Adam optimizer [17] is used as the optimizer for model parameter update during training, and the learning rate is set to 0.001.


### 3.2. Analysis of the Experimental Results of Nested Named Entity Recognition

#### 3.2.1. Analysis of the Experimental Results of the GENIA Dataset

The experiments compare the performance of the MTL-BAM model and the BAM model on five different entity categories in the dataset GENIA. The experimental results on this dataset are shown in Table3. The best results in each group of experiments are shown in bold. Figure 6 is a histogram of the data distribution corresponding to Table 3.Table 3
Experimental results of entity types in GENIA dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)DNA78.8075.6777.2078.3575.9977.15RNA86.7984.4085.5886.9285.3286.11Protein82.8483.5683.2081.8585.2883.53cell_line81.0867.4273.6283.2968.3175.06cell_type77.0274.0975.5075.8775.2575.56Total81.2579.4280.3380.6280.6880.65Figure 6
Experimental results of various entity types in GENIA dataset.It can be seen from the figure and table that the recognition result of the RNA entity type is the highest, because the RNA boundary in the data set is generally represented by mRNA and RNA, so the model has learned the boundary information of RNA, and the recognition effect is the best. Therefore, it can be obtained that the entity boundary information plays an important role in the accurate identification of the entity. Except for the 0.05% drop in DNA type data, theF1 value of all other entity types has a certain effect improvement, among which the cell_type entity type with the least improvement has increased by 0.06%, and the highest cell_line entity type has increased by 1.44%, the overall F1 value increased by 0.22%, indicating that the MTL-BAM model has a certain effect compared with the BAM model. From the bold display in the table, after adding the multi-task learning model, the overall recall rate has been improved compared with before. The reason is that the boundary detection module enhances entity boundary supervision and obtains entity context representation information by obtaining entity boundary information. The span classification module provides a boundary representation, which enables the model to extract more correct entity segments. At the same time, from the perspective of accuracy, the results of the least number of RNA and cell_line have improved, indicating that the boundary detection module strengthens the connection between the inside and outside of the nested entity, so that the sparse entity learns more internal and external features, and the accuracy rate for other entity types is improved. There is a downward trend, and the reason may be that the extracted entity fragments are not accurately classified. The model should also set more effective methods of multi-task learning to strengthen the connection between internal and external entities and improve the accuracy. It shows that the multi-task model can improve the effect of the single-task biaffine mechanism.


#### 3.2.2. Analysis of Experimental Results of ACE2004 and ACE2005 Datasets

In the experiment, the performance of the MTL-BAM model on seven different entity categories of the two nested datasets ACE2004 and ACE2005 was further verified. The experimental results on the ACE2004 and ACE2005 datasets are shown in Tables4 and 5.Table 4
Experimental results of entity types in ACE2004 dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)LOC65.6963.8164.7366.6768.5767.61WEA71.4346.8856.6085.0053.1265.38GPE83.7083.9483.8285.7985.2085.49PER90.8289.1289.9688.3591.1289.71FAC72.6362.1666.9969.0062.1665.40ORG83.1378.8080.8480.3981.7081.04VEH88.2488.2488.2488.8994.1291.43Overall86.3783.7085.0084.8885.7885.33Table 5
Experimental results of entity types in ACE2005 dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)LOC65.2359.3462.1464.7161.1162.86WEA83.1885.1585.1182.6986.0084.31GPE85.2883.1584.2084.1484.5384.33PER88.7687.7488.2586.1090.0288.96FAC69.5072.0670.7673.1372.0672.59ORG85.2776.8280.8384.0477.7680.78VEH75.6469.7572.5876.3470.3073.20Overall84.2485.3684.7984.2386.1585.18It can be seen from the table that theF1 value of the MTL-BAM model on the ACE2005 dataset is 0.39% higher than that of the BAM model, and the recall rate of the other six entity labels is the same except for the FAC entity category. The overall recall rate has also improved by 0.79%. On the ACE2004 dataset, the overall F1 value of the model has increased by 0.33%. Except for the same recall rate on the FAC entity category, other recall rates have been improved, and the overall recall rate has increased by 2.08%. The recall rate of the model in this chapter is higher than that of the BAM model on both data, and the overall F1 value is also higher than that of the BAM model, which proves the effectiveness of the MTL-BAM model on multitype nested datasets.


#### 3.2.3. Analysis of Nested Data Set Comparison Experiment Results

The MTL-BAM model is compared with some existing neural network-based nested named entity recognition models. The experimental comparison results are shown in Table6. The precision and recall of the experimental results are shown, and the final comparison with other entity recognition models only uses the F1 result as a comparison.Table 6
Comparison results between MTL-BAM model and other entity recognition models on three nested datasets.
ModelGENIAACE2004ACE2005P (%)R (%)F1 (%)P (%)R (%)F1 (%)P (%)R (%)F1 (%)Katuyar and Cardie [2]79.868.273.673.671.872.770.670.470.5Ju et al. [1]78.571.374.7———74.270.372.2Zheng et al. [18]74.575.675.0——————Wang and Lu [4]77.073.375.178.072.475.176.872.374.5Yi et al. [5]——76.2——84.7——82.9Sohrab and miwa [6]93.264.077.1——————Jana et al. [19]——78.3——84.4——84.3MTL-BAM80.6280.6880.6584.8885.7885.3384.2386.1585.18In the table, Ju et al. [1] and Zheng et al. [18] are methods based on sequence annotation, Katuyar, Wang and Lu are methods based on hypergraph, Luan, and Sohrab are span-based method methods. Jana et al. [19] is an approach by using a linear model for nested label encoding. The method proposed in this chapter is higher than all above methods in recall rate and F1 value, and the accuracy rate is lower than that of Sohrab in the GENIA dataset, but because the precision rate and recall rate of this model are too different, it shows that the span-based model has a high precision and low recall rate due to incomplete span detection. Compared with Sohrab, the recall rate of this paper is improved by 16.68%, and the overall result is 3.64% higher. For the ACE2004 dataset, the F1 value is the highest result of the current model, which is 0.63% higher than the Luan model. The F1 value of the model in this chapter is the highest result on the ACE2005 dataset, and the F1 value of the model in this chapter is 0.88% higher than that of Strakova. Based on the above conclusions, the model in this paper verifies the effectiveness of recognition in nested datasets by comparing with other models. It is verified that the proposed multi-task framework enables the boundary detection module to enhance the entity boundary supervision and obtain entity context representation information by obtaining entity boundary information, and provides boundary representation for entity span classification module, which improves the effect of entity recognition. At the same time, it shows that the model in this chapter has an important contribution to improving the recall rate and balancing the F1 value.


### 3.3. Analysis of Experimental Results of Flat Named Entity Recognition

#### 3.3.1. Analysis of the Experimental Results of the JNLPBA Dataset

The experimentally compared models for the JNLPBA dataset include the models of Wang et al. [20] and Song [21]. The former proposes to share character- and word-level information between related biomedical entities across different labeled corpora. The latter uses BioBERT, a domain-specific language representation model pretrained on a large-scale biomedical corpus, with the same principles as the BERT model. Moreover, it includes some of the models mentioned above.The comparison results of the MTL-BAM and BAM models on the dataset JNLPBA based on the same experimental environment are shown in Table7.Table 7
Comparison results of MTL-BAM and BAM models on JNLPBA dataset.
ModelP (%)R (%)F1 (%)BAM72.8279.1275.84MTL-BAM72.4280.3776.19On the JNLPBA dataset, theF1 value of the model in this paper has increased by 0.35%, the recall rate has increased by 1.25%, and the accuracy has decreased to a certain extent. The experimental results verify that the model is equally effective on flat entities and nested entities.Table8 shows the experimental comparison results between the MTL-BAM model and the other entity recognition models mentioned above. Since most models only show the final evaluation index F1 value in the JNLPBA dataset, the MTL-BAM model will only compare the F1 value.Table 8
Comparison results between MTL-BAM model and other entity recognition models on JNLPBA dataset.
ModelF1 (%)Ju et al. [1]70.1Zheng et al. [18]73.6Wang et al. [20]73.52Song et al. [21]75.04MTL-BAM76.19Compared with the other entity recognition models, it is 1.15% higher than BioBERT using biomedical corpus as a pretraining model, and the verification model MTL-BAM is effective on the JNLPBA dataset on flat entities.


#### 3.3.2. Analysis of Experimental Results of CoNLL2003 Dataset

For the dataset CoNLL2003, the following models will be used for experimental comparison:The sequence annotation model proposed by Lample et al. [22] uses the BILSTM-CRF model to recognize flat entities. Strubell [23] proposed an iterative dilated convolutional neural network (ID-CNN) for entity recognition, which has better large context and structured prediction capabilities than traditional CNNs. Devlin et al. [24] proposed to use the BERT pretraining model to fine-tune a large amount of pretrained corpus to improve entity recognition. Akbik et al. [25] exploits the internal state of a trained character language model to generate a novel word embedding that enhances contextual representation to improve entity recognition.The comparison results between the data set CoNLL2003 and the BAM model based on the same experimental environment are shown in Table9.Table 9
Comparison results of MTL-BAM and BAM models on CoNLL2003 dataset.
ModelP (%)R (%)F1 (%)BAM92.9793.4193.19MTL-BAM92.7294.0793.39On the CoNLL2003 dataset, theF1 value of the model in this paper has increased by 0.2%, the recall rate has increased by 0.66%, and the accuracy has decreased to a certain extent. The experimental results verify that the model is equally effective on the flat entity CoNLL2003 and nested entities.Table10 shows the experimental comparison results between the MTL-BAM model and the other entity recognition models mentioned above. Moreover, only the F1 value was compared to the CoNLL2003 dataset.Table 10
Comparison results between MTL-BAM model and other entity recognition models on CoNLL2003 dataset.
ModelF1 (%)Lample et al. [22]90.94Strubell et al. [23]90.7Devlin et al. [24]92.8Akbik et al. [25]93.09MTL-BAM93.39It can be seen from the table that the MTL-BAM model is 0.3% higher than the other entity recognition models. The model proposed in this chapter also has a certain effect on the two flat datasets, which shows that the named entity recognition model based on multi-task learning and double affine mechanism in this chapter is versatile in various datasets. However, for the named entity recognition method of the CoNLL2003 dataset, the MTL-BAM model still has a certain gap compared with the current SOTA method [26], mainly because the method proposed in this paper is more for nested named entities.


## 3.1. Experimental Environment Parameter Settings

This paper uses the Windows system for experiments, based on the Python platform, using pycharm as a development tool. The model is constructed using the open source deep learning framework Tensorflow, which is developed and maintained by Google’s artificial intelligence team, Google Brain. It is deployed on various servers, PC terminals, and web pages, and supports GPU high-performance numerical computing.The named entity recognition model (MTL-BAM) based on multi-task learning and biaffine mechanism is experimented on two flat entity datasets JNLPBA [13], CoNLL2003, and three nested entity datasets GENIA [14], ACE2005, and ACE2004 [15]. It is compared with BAM that only uses a biaffine mechanism for entity recognition without using a multi-task learning framework, and the experimental results are compared with other models that achieve some results in the named entity recognition tasks. The evaluation indicators used are the precision rate P, the recall rate R, and the F value.The experimental parameter settings in this paper are shown in Table2.Table 2
Experimental parameter setting.
ParameterValueBiLSTM size200BiLSTM layer3BiLSTM dropout0.4MLP size150MLP dropout0.2BERT size1024FastText embedding size300CharCNN filter widths[3–5]Char embedding size50Embeddings dropout0.5OptimiserAdamLearning rate0.001The convolution kernel window size of the convolutional layer of the CharCNN network is set to [3–5], and the character vector dimension is 50 dimensions. Use the public pretrained word vector FastText as the choice of word vector, and the word vector dimension is set to 300 dimensions. BiLSTM sets the output dimension of 3 layers to 200 dimensions, and the dimension of two fully connected layers is set to 150 dimensions, reducing the output dimension of LSTM to prevent overfitting. At the same time, to prevent overfitting, the BiLSTM layer and the MLP layer, respectively, set the dropout [16] to 0.4 and 0.2. The Adam optimizer [17] is used as the optimizer for model parameter update during training, and the learning rate is set to 0.001.


## 3.2. Analysis of the Experimental Results of Nested Named Entity Recognition

### 3.2.1. Analysis of the Experimental Results of the GENIA Dataset

The experiments compare the performance of the MTL-BAM model and the BAM model on five different entity categories in the dataset GENIA. The experimental results on this dataset are shown in Table3. The best results in each group of experiments are shown in bold. Figure 6 is a histogram of the data distribution corresponding to Table 3.Table 3
Experimental results of entity types in GENIA dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)DNA78.8075.6777.2078.3575.9977.15RNA86.7984.4085.5886.9285.3286.11Protein82.8483.5683.2081.8585.2883.53cell_line81.0867.4273.6283.2968.3175.06cell_type77.0274.0975.5075.8775.2575.56Total81.2579.4280.3380.6280.6880.65Figure 6
Experimental results of various entity types in GENIA dataset.It can be seen from the figure and table that the recognition result of the RNA entity type is the highest, because the RNA boundary in the data set is generally represented by mRNA and RNA, so the model has learned the boundary information of RNA, and the recognition effect is the best. Therefore, it can be obtained that the entity boundary information plays an important role in the accurate identification of the entity. Except for the 0.05% drop in DNA type data, theF1 value of all other entity types has a certain effect improvement, among which the cell_type entity type with the least improvement has increased by 0.06%, and the highest cell_line entity type has increased by 1.44%, the overall F1 value increased by 0.22%, indicating that the MTL-BAM model has a certain effect compared with the BAM model. From the bold display in the table, after adding the multi-task learning model, the overall recall rate has been improved compared with before. The reason is that the boundary detection module enhances entity boundary supervision and obtains entity context representation information by obtaining entity boundary information. The span classification module provides a boundary representation, which enables the model to extract more correct entity segments. At the same time, from the perspective of accuracy, the results of the least number of RNA and cell_line have improved, indicating that the boundary detection module strengthens the connection between the inside and outside of the nested entity, so that the sparse entity learns more internal and external features, and the accuracy rate for other entity types is improved. There is a downward trend, and the reason may be that the extracted entity fragments are not accurately classified. The model should also set more effective methods of multi-task learning to strengthen the connection between internal and external entities and improve the accuracy. It shows that the multi-task model can improve the effect of the single-task biaffine mechanism.


### 3.2.2. Analysis of Experimental Results of ACE2004 and ACE2005 Datasets

In the experiment, the performance of the MTL-BAM model on seven different entity categories of the two nested datasets ACE2004 and ACE2005 was further verified. The experimental results on the ACE2004 and ACE2005 datasets are shown in Tables4 and 5.Table 4
Experimental results of entity types in ACE2004 dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)LOC65.6963.8164.7366.6768.5767.61WEA71.4346.8856.6085.0053.1265.38GPE83.7083.9483.8285.7985.2085.49PER90.8289.1289.9688.3591.1289.71FAC72.6362.1666.9969.0062.1665.40ORG83.1378.8080.8480.3981.7081.04VEH88.2488.2488.2488.8994.1291.43Overall86.3783.7085.0084.8885.7885.33Table 5
Experimental results of entity types in ACE2005 dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)LOC65.2359.3462.1464.7161.1162.86WEA83.1885.1585.1182.6986.0084.31GPE85.2883.1584.2084.1484.5384.33PER88.7687.7488.2586.1090.0288.96FAC69.5072.0670.7673.1372.0672.59ORG85.2776.8280.8384.0477.7680.78VEH75.6469.7572.5876.3470.3073.20Overall84.2485.3684.7984.2386.1585.18It can be seen from the table that theF1 value of the MTL-BAM model on the ACE2005 dataset is 0.39% higher than that of the BAM model, and the recall rate of the other six entity labels is the same except for the FAC entity category. The overall recall rate has also improved by 0.79%. On the ACE2004 dataset, the overall F1 value of the model has increased by 0.33%. Except for the same recall rate on the FAC entity category, other recall rates have been improved, and the overall recall rate has increased by 2.08%. The recall rate of the model in this chapter is higher than that of the BAM model on both data, and the overall F1 value is also higher than that of the BAM model, which proves the effectiveness of the MTL-BAM model on multitype nested datasets.


### 3.2.3. Analysis of Nested Data Set Comparison Experiment Results

The MTL-BAM model is compared with some existing neural network-based nested named entity recognition models. The experimental comparison results are shown in Table6. The precision and recall of the experimental results are shown, and the final comparison with other entity recognition models only uses the F1 result as a comparison.Table 6
Comparison results between MTL-BAM model and other entity recognition models on three nested datasets.
ModelGENIAACE2004ACE2005P (%)R (%)F1 (%)P (%)R (%)F1 (%)P (%)R (%)F1 (%)Katuyar and Cardie [2]79.868.273.673.671.872.770.670.470.5Ju et al. [1]78.571.374.7———74.270.372.2Zheng et al. [18]74.575.675.0——————Wang and Lu [4]77.073.375.178.072.475.176.872.374.5Yi et al. [5]——76.2——84.7——82.9Sohrab and miwa [6]93.264.077.1——————Jana et al. [19]——78.3——84.4——84.3MTL-BAM80.6280.6880.6584.8885.7885.3384.2386.1585.18In the table, Ju et al. [1] and Zheng et al. [18] are methods based on sequence annotation, Katuyar, Wang and Lu are methods based on hypergraph, Luan, and Sohrab are span-based method methods. Jana et al. [19] is an approach by using a linear model for nested label encoding. The method proposed in this chapter is higher than all above methods in recall rate and F1 value, and the accuracy rate is lower than that of Sohrab in the GENIA dataset, but because the precision rate and recall rate of this model are too different, it shows that the span-based model has a high precision and low recall rate due to incomplete span detection. Compared with Sohrab, the recall rate of this paper is improved by 16.68%, and the overall result is 3.64% higher. For the ACE2004 dataset, the F1 value is the highest result of the current model, which is 0.63% higher than the Luan model. The F1 value of the model in this chapter is the highest result on the ACE2005 dataset, and the F1 value of the model in this chapter is 0.88% higher than that of Strakova. Based on the above conclusions, the model in this paper verifies the effectiveness of recognition in nested datasets by comparing with other models. It is verified that the proposed multi-task framework enables the boundary detection module to enhance the entity boundary supervision and obtain entity context representation information by obtaining entity boundary information, and provides boundary representation for entity span classification module, which improves the effect of entity recognition. At the same time, it shows that the model in this chapter has an important contribution to improving the recall rate and balancing the F1 value.


## 3.2.1. Analysis of the Experimental Results of the GENIA Dataset

The experiments compare the performance of the MTL-BAM model and the BAM model on five different entity categories in the dataset GENIA. The experimental results on this dataset are shown in Table3. The best results in each group of experiments are shown in bold. Figure 6 is a histogram of the data distribution corresponding to Table 3.Table 3
Experimental results of entity types in GENIA dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)DNA78.8075.6777.2078.3575.9977.15RNA86.7984.4085.5886.9285.3286.11Protein82.8483.5683.2081.8585.2883.53cell_line81.0867.4273.6283.2968.3175.06cell_type77.0274.0975.5075.8775.2575.56Total81.2579.4280.3380.6280.6880.65Figure 6
Experimental results of various entity types in GENIA dataset.It can be seen from the figure and table that the recognition result of the RNA entity type is the highest, because the RNA boundary in the data set is generally represented by mRNA and RNA, so the model has learned the boundary information of RNA, and the recognition effect is the best. Therefore, it can be obtained that the entity boundary information plays an important role in the accurate identification of the entity. Except for the 0.05% drop in DNA type data, theF1 value of all other entity types has a certain effect improvement, among which the cell_type entity type with the least improvement has increased by 0.06%, and the highest cell_line entity type has increased by 1.44%, the overall F1 value increased by 0.22%, indicating that the MTL-BAM model has a certain effect compared with the BAM model. From the bold display in the table, after adding the multi-task learning model, the overall recall rate has been improved compared with before. The reason is that the boundary detection module enhances entity boundary supervision and obtains entity context representation information by obtaining entity boundary information. The span classification module provides a boundary representation, which enables the model to extract more correct entity segments. At the same time, from the perspective of accuracy, the results of the least number of RNA and cell_line have improved, indicating that the boundary detection module strengthens the connection between the inside and outside of the nested entity, so that the sparse entity learns more internal and external features, and the accuracy rate for other entity types is improved. There is a downward trend, and the reason may be that the extracted entity fragments are not accurately classified. The model should also set more effective methods of multi-task learning to strengthen the connection between internal and external entities and improve the accuracy. It shows that the multi-task model can improve the effect of the single-task biaffine mechanism.


## 3.2.2. Analysis of Experimental Results of ACE2004 and ACE2005 Datasets

In the experiment, the performance of the MTL-BAM model on seven different entity categories of the two nested datasets ACE2004 and ACE2005 was further verified. The experimental results on the ACE2004 and ACE2005 datasets are shown in Tables4 and 5.Table 4
Experimental results of entity types in ACE2004 dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)LOC65.6963.8164.7366.6768.5767.61WEA71.4346.8856.6085.0053.1265.38GPE83.7083.9483.8285.7985.2085.49PER90.8289.1289.9688.3591.1289.71FAC72.6362.1666.9969.0062.1665.40ORG83.1378.8080.8480.3981.7081.04VEH88.2488.2488.2488.8994.1291.43Overall86.3783.7085.0084.8885.7885.33Table 5
Experimental results of entity types in ACE2005 dataset.
Entity typeBAMMTL-BAMP (%)R (%)F1 (%)P (%)R (%)F1 (%)LOC65.2359.3462.1464.7161.1162.86WEA83.1885.1585.1182.6986.0084.31GPE85.2883.1584.2084.1484.5384.33PER88.7687.7488.2586.1090.0288.96FAC69.5072.0670.7673.1372.0672.59ORG85.2776.8280.8384.0477.7680.78VEH75.6469.7572.5876.3470.3073.20Overall84.2485.3684.7984.2386.1585.18It can be seen from the table that theF1 value of the MTL-BAM model on the ACE2005 dataset is 0.39% higher than that of the BAM model, and the recall rate of the other six entity labels is the same except for the FAC entity category. The overall recall rate has also improved by 0.79%. On the ACE2004 dataset, the overall F1 value of the model has increased by 0.33%. Except for the same recall rate on the FAC entity category, other recall rates have been improved, and the overall recall rate has increased by 2.08%. The recall rate of the model in this chapter is higher than that of the BAM model on both data, and the overall F1 value is also higher than that of the BAM model, which proves the effectiveness of the MTL-BAM model on multitype nested datasets.


## 3.2.3. Analysis of Nested Data Set Comparison Experiment Results

The MTL-BAM model is compared with some existing neural network-based nested named entity recognition models. The experimental comparison results are shown in Table6. The precision and recall of the experimental results are shown, and the final comparison with other entity recognition models only uses the F1 result as a comparison.Table 6
Comparison results between MTL-BAM model and other entity recognition models on three nested datasets.
ModelGENIAACE2004ACE2005P (%)R (%)F1 (%)P (%)R (%)F1 (%)P (%)R (%)F1 (%)Katuyar and Cardie [2]79.868.273.673.671.872.770.670.470.5Ju et al. [1]78.571.374.7———74.270.372.2Zheng et al. [18]74.575.675.0——————Wang and Lu [4]77.073.375.178.072.475.176.872.374.5Yi et al. [5]——76.2——84.7——82.9Sohrab and miwa [6]93.264.077.1——————Jana et al. [19]——78.3——84.4——84.3MTL-BAM80.6280.6880.6584.8885.7885.3384.2386.1585.18In the table, Ju et al. [1] and Zheng et al. [18] are methods based on sequence annotation, Katuyar, Wang and Lu are methods based on hypergraph, Luan, and Sohrab are span-based method methods. Jana et al. [19] is an approach by using a linear model for nested label encoding. The method proposed in this chapter is higher than all above methods in recall rate and F1 value, and the accuracy rate is lower than that of Sohrab in the GENIA dataset, but because the precision rate and recall rate of this model are too different, it shows that the span-based model has a high precision and low recall rate due to incomplete span detection. Compared with Sohrab, the recall rate of this paper is improved by 16.68%, and the overall result is 3.64% higher. For the ACE2004 dataset, the F1 value is the highest result of the current model, which is 0.63% higher than the Luan model. The F1 value of the model in this chapter is the highest result on the ACE2005 dataset, and the F1 value of the model in this chapter is 0.88% higher than that of Strakova. Based on the above conclusions, the model in this paper verifies the effectiveness of recognition in nested datasets by comparing with other models. It is verified that the proposed multi-task framework enables the boundary detection module to enhance the entity boundary supervision and obtain entity context representation information by obtaining entity boundary information, and provides boundary representation for entity span classification module, which improves the effect of entity recognition. At the same time, it shows that the model in this chapter has an important contribution to improving the recall rate and balancing the F1 value.


## 3.3. Analysis of Experimental Results of Flat Named Entity Recognition

### 3.3.1. Analysis of the Experimental Results of the JNLPBA Dataset

The experimentally compared models for the JNLPBA dataset include the models of Wang et al. [20] and Song [21]. The former proposes to share character- and word-level information between related biomedical entities across different labeled corpora. The latter uses BioBERT, a domain-specific language representation model pretrained on a large-scale biomedical corpus, with the same principles as the BERT model. Moreover, it includes some of the models mentioned above.The comparison results of the MTL-BAM and BAM models on the dataset JNLPBA based on the same experimental environment are shown in Table7.Table 7
Comparison results of MTL-BAM and BAM models on JNLPBA dataset.
ModelP (%)R (%)F1 (%)BAM72.8279.1275.84MTL-BAM72.4280.3776.19On the JNLPBA dataset, theF1 value of the model in this paper has increased by 0.35%, the recall rate has increased by 1.25%, and the accuracy has decreased to a certain extent. The experimental results verify that the model is equally effective on flat entities and nested entities.Table8 shows the experimental comparison results between the MTL-BAM model and the other entity recognition models mentioned above. Since most models only show the final evaluation index F1 value in the JNLPBA dataset, the MTL-BAM model will only compare the F1 value.Table 8
Comparison results between MTL-BAM model and other entity recognition models on JNLPBA dataset.
ModelF1 (%)Ju et al. [1]70.1Zheng et al. [18]73.6Wang et al. [20]73.52Song et al. [21]75.04MTL-BAM76.19Compared with the other entity recognition models, it is 1.15% higher than BioBERT using biomedical corpus as a pretraining model, and the verification model MTL-BAM is effective on the JNLPBA dataset on flat entities.


### 3.3.2. Analysis of Experimental Results of CoNLL2003 Dataset

For the dataset CoNLL2003, the following models will be used for experimental comparison:The sequence annotation model proposed by Lample et al. [22] uses the BILSTM-CRF model to recognize flat entities. Strubell [23] proposed an iterative dilated convolutional neural network (ID-CNN) for entity recognition, which has better large context and structured prediction capabilities than traditional CNNs. Devlin et al. [24] proposed to use the BERT pretraining model to fine-tune a large amount of pretrained corpus to improve entity recognition. Akbik et al. [25] exploits the internal state of a trained character language model to generate a novel word embedding that enhances contextual representation to improve entity recognition.The comparison results between the data set CoNLL2003 and the BAM model based on the same experimental environment are shown in Table9.Table 9
Comparison results of MTL-BAM and BAM models on CoNLL2003 dataset.
ModelP (%)R (%)F1 (%)BAM92.9793.4193.19MTL-BAM92.7294.0793.39On the CoNLL2003 dataset, theF1 value of the model in this paper has increased by 0.2%, the recall rate has increased by 0.66%, and the accuracy has decreased to a certain extent. The experimental results verify that the model is equally effective on the flat entity CoNLL2003 and nested entities.Table10 shows the experimental comparison results between the MTL-BAM model and the other entity recognition models mentioned above. Moreover, only the F1 value was compared to the CoNLL2003 dataset.Table 10
Comparison results between MTL-BAM model and other entity recognition models on CoNLL2003 dataset.
ModelF1 (%)Lample et al. [22]90.94Strubell et al. [23]90.7Devlin et al. [24]92.8Akbik et al. [25]93.09MTL-BAM93.39It can be seen from the table that the MTL-BAM model is 0.3% higher than the other entity recognition models. The model proposed in this chapter also has a certain effect on the two flat datasets, which shows that the named entity recognition model based on multi-task learning and double affine mechanism in this chapter is versatile in various datasets. However, for the named entity recognition method of the CoNLL2003 dataset, the MTL-BAM model still has a certain gap compared with the current SOTA method [26], mainly because the method proposed in this paper is more for nested named entities.


## 3.3.1. Analysis of the Experimental Results of the JNLPBA Dataset

The experimentally compared models for the JNLPBA dataset include the models of Wang et al. [20] and Song [21]. The former proposes to share character- and word-level information between related biomedical entities across different labeled corpora. The latter uses BioBERT, a domain-specific language representation model pretrained on a large-scale biomedical corpus, with the same principles as the BERT model. Moreover, it includes some of the models mentioned above.The comparison results of the MTL-BAM and BAM models on the dataset JNLPBA based on the same experimental environment are shown in Table7.Table 7
Comparison results of MTL-BAM and BAM models on JNLPBA dataset.
ModelP (%)R (%)F1 (%)BAM72.8279.1275.84MTL-BAM72.4280.3776.19On the JNLPBA dataset, theF1 value of the model in this paper has increased by 0.35%, the recall rate has increased by 1.25%, and the accuracy has decreased to a certain extent. The experimental results verify that the model is equally effective on flat entities and nested entities.Table8 shows the experimental comparison results between the MTL-BAM model and the other entity recognition models mentioned above. Since most models only show the final evaluation index F1 value in the JNLPBA dataset, the MTL-BAM model will only compare the F1 value.Table 8
Comparison results between MTL-BAM model and other entity recognition models on JNLPBA dataset.
ModelF1 (%)Ju et al. [1]70.1Zheng et al. [18]73.6Wang et al. [20]73.52Song et al. [21]75.04MTL-BAM76.19Compared with the other entity recognition models, it is 1.15% higher than BioBERT using biomedical corpus as a pretraining model, and the verification model MTL-BAM is effective on the JNLPBA dataset on flat entities.


## 3.3.2. Analysis of Experimental Results of CoNLL2003 Dataset

For the dataset CoNLL2003, the following models will be used for experimental comparison:The sequence annotation model proposed by Lample et al. [22] uses the BILSTM-CRF model to recognize flat entities. Strubell [23] proposed an iterative dilated convolutional neural network (ID-CNN) for entity recognition, which has better large context and structured prediction capabilities than traditional CNNs. Devlin et al. [24] proposed to use the BERT pretraining model to fine-tune a large amount of pretrained corpus to improve entity recognition. Akbik et al. [25] exploits the internal state of a trained character language model to generate a novel word embedding that enhances contextual representation to improve entity recognition.The comparison results between the data set CoNLL2003 and the BAM model based on the same experimental environment are shown in Table9.Table 9
Comparison results of MTL-BAM and BAM models on CoNLL2003 dataset.
ModelP (%)R (%)F1 (%)BAM92.9793.4193.19MTL-BAM92.7294.0793.39On the CoNLL2003 dataset, theF1 value of the model in this paper has increased by 0.2%, the recall rate has increased by 0.66%, and the accuracy has decreased to a certain extent. The experimental results verify that the model is equally effective on the flat entity CoNLL2003 and nested entities.Table10 shows the experimental comparison results between the MTL-BAM model and the other entity recognition models mentioned above. Moreover, only the F1 value was compared to the CoNLL2003 dataset.Table 10
Comparison results between MTL-BAM model and other entity recognition models on CoNLL2003 dataset.
ModelF1 (%)Lample et al. [22]90.94Strubell et al. [23]90.7Devlin et al. [24]92.8Akbik et al. [25]93.09MTL-BAM93.39It can be seen from the table that the MTL-BAM model is 0.3% higher than the other entity recognition models. The model proposed in this chapter also has a certain effect on the two flat datasets, which shows that the named entity recognition model based on multi-task learning and double affine mechanism in this chapter is versatile in various datasets. However, for the named entity recognition method of the CoNLL2003 dataset, the MTL-BAM model still has a certain gap compared with the current SOTA method [26], mainly because the method proposed in this paper is more for nested named entities.


## 4. Conclusions

From the various experimental results, it can be found that the use of the multi-task learning framework improves the final performance to a certain extent, whether on the baseline model or on other models studied in the past, and it also proves that the method is effective on the flat entity recognition task. With no performance penalty, it is a general framework that can be used for both nested and flat entity recognition tasks.In addition, the model in this paper still has many shortcomings. The current use of multi-task learning to build a boundary detection model can positively promote the entity classification module. However, for nested entities, the outer boundary can supervise the boundary of the inner entity, and the information of the inner entity has not yet guided the recognition of the outer entity, resulting in only a small improvement in the entire model. Secondly, this paper only studies flat entities and nested entities, and there are more complex entity types in practical applications, such as discontinuous entities. Therefore, in the next step, we can improve the two-way interaction ability of the two tasks. And study the use of this model or the improved model to solve the recognition of various complex entities and improve the effect of various types of entity recognition.



---

*Source: 2687615-2022-08-25.xml*