Patent ID: 11972214
Assignee: ZHEJIANG LAB
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method of Named Entity Recognition (NER)-oriented Chinese clinical text data augmentation, comprising:
data preprocessing, comprising: acquiring clinical text data without annotation information and performing word segmentation to obtain processed unannotated data; and acquiring clinical text data with annotation information for entity segmentation, annotation segmented entities to obtain processed annotation data, and performing label linearization processing;
pre-training of a span-based language model, comprising: predicting, by using unannotated data and concealing a part of information in text, the concealed part based on retained information, and introducing an entity word-level discrimination task, randomly replacing entities contained in a sentence with similar entities in an entity library, and detecting an entity word level using a discriminator network to determine whether each word in the entities is a word of original entities,
wherein said replacing entities further comprises:
defining fine-grained phrase composition structures of different types of medical entities by combining a medical dictionary and medical knowledge, wherein each fine-grained composition structure is an independent phrase with indivisible independent semantics;
segmenting entities to be replaced into independent semantic units according to predefined fine-grained information;
retrieving similar words of a same type for each independent semantic unit after segmentation, obtaining first top-k similar words of the same type arranged in an order from high degree to low degree of similarity with to-be-replaced words by calculating a semantic similarity degree, and randomly selecting one to replace an original content; and
operating on each independent semantic unit to finally complete a whole entity replacement process;
wherein the entity word-level discrimination task further comprises:
assuming that a sentence sequence X inputted into the span-based language model contains n entities S(1), S(2), . . . , S(n), and for an it entity S(t), retaining the entity with a probability P and selecting an entity with similar semantics from the entity library with a probability 1-P, namely:, S
    ^
   
   
    (
    i
    )
   
  
  =
  
   {
   
    
     
      
       
        S
        
         (
         i
         )
        
       
       ,
      
     
     
      
       p
       <
       P
      
     
    
    
     
      
       
        ℱ
        ⁡
        (
        
         
          S
          
           (
           i
           )
          
         
         ,
         ℛ
        
        )
       
       ,
      
     
     
      else
     
    
   
  
 

where  represents the entity library,  represents an entity selection strategy function, Ŝ(i) represents a selected entity with the semantics similar to the entity S(i), and p represents random variate uniformly distributed following U(0,1);
forming a new sentence g through the entity random replacement; coding X by a depth self-attention network Transformer to obtain a corresponding representation sequence H, and recording a coding result of the entity {right arrow over (S)}(i) as H(i)=(hm, . . . , hn), where m and n represent starting and ending positions of the entity Ŝ(i) in a sentence sequence, respectively, and h represents a semantic vector representation at the corresponding position; and then inputting H(i) into the discriminator network for word-level entity detection, wherein a structure of the discriminator network is two linear layers plus a ReLU activation function, and Layer Normalization and residual connection is applied to ensure stability of network training:

hi′=LayerNorm(hi+RELU(W1hi))

=LayerNorm(hi′+RELU(W2hi′))

where W1 and W2 represents weights of a first linear layer and a second linear layer, respectively, hi′ represents an output result of a first layer of the network, and  represents an output result of a second layer of the network; and
inputting  into a Sigmoid function for binary classification: when a word at a position i in X is consistent with a word at a position i in X, a prediction target being 1, and otherwise, the prediction target being 0; and
introducing a plurality of decoding mechanisms in a fine-tune stage: obtaining, in the fine-tune stage, a relationship between a text vector and the text data based on the pre-trained span-based language model, converting linearized data with entity labels into the text vector for training of a text generation model, inputting the text data that needs to be enhanced into the trained text generation model, performing text generation through forward decoding and reverse decoding in a prediction stage of the text generation model to obtain a generation result with the entity labels, and analyzing the labels to obtain enhanced data with the annotation information;
wherein decoding process of the forward decoding and the reverse decoding comprises: constructing a two-direction multi-head attention model to realize utilization of existing coding information of another process;
wherein in the forward decoding, that is, text generation from left to right, the following probability calculation is performed:

P(Y|X)=P(y1|X)P(y2|X,y1)P(y3|X,y1,y2). . . P(yn|X,y1,y2, . . . ,yn-1)

wherein in the reverse decoding, that is, text generation from right to left, the following probability calculation is performed:

P(Y|X)=P(yn|X)P(yn-1|X,yn)P(yn-2|X,yn,yn-1) . . . P(y1|X,yn,yn-1, . . . y2)

where n represents a position of a decoding result, and I represents a decoding result of a corresponding position;
wherein a segmented token is subjected to embedding, is decoded through a long-short term memory model (LSTM) structure, and is subjected to linear transformation to obtain an output result;
wherein two-direction decoding comprises text generation from two ends to a middle synchronously by maintaining decoders in two directions, wherein the forward decoding is recorded as an L2R module, the reverse decoding is recorded as an R2L module, the L2R module takes a historical prediction result in a forward direction and existing coding information of the R2L module as inputs for prediction, the R2L module takes a previous prediction result and existing coding information of the L2R module as inputs for prediction;
wherein during the two-direction decoding: assuming that the L2R module currently is to predict an nt token, the R2L module is to predict an nth to last token, after several layers of coding, an vector sequence H(l2r)>[h1(l2r), h2(l2r), . . . , hn(l2r)] of the L2R module is obtained, where hn(l2r) represents a coding result at an nth position of a sentence obtained by the L2R module, and a vector sequence of the corresponding R2L module is H(r2l)=h1(r2l), h2(r2l), . . . , hn(r2l)] where hn(l2r) represents a coding result at an nth position of a sentence obtained by the R2L module; wherein under mechanism of the two-direction decoding, hn(l2r) is taken as a query vector, H(r2l) is taken as a key and a value for an attention operation, an output of an attention mechanism is taken as a feature to predict the nth token, hn(r2l) is taken as a query vector, H(l2r) is taken as a key and a value for the attention operation, an output is taken as a feature to predict the nth token, and a final loss function is a mean of a cross entropy of the two directions;
wherein during prediction stage of the two-direction decoding, top-k results of the two directions are cached synchronously; when a next token is predicted, top-k tokens with the highest probability, top-k temporary paths in the forward direction, and top-k reverse temporary paths are both enumerated, the temporary paths are text sequences for storing temporarily generated results during the text generation, a probability value of each path is calculated as a score of the each path, after calculating scores of all combinations, top-k highest scores are retained, and in a same way top-k highest scores of paths in the reverse direction are retained; and after decoding in the two directions are both finished, a decoding in a direction with a highest score is selected as the output result;
wherein a training process of a combination of strong and weak labels comprises: adopting, by a training process of the text generation model, a mode of combining the strong and weak labels; performing confidence estimation on entity labels in weak label data generated by the text generation module, a confidence level indicating a trust degree of a current label, and the confidence estimation being as follows:, w
  =
  
   {
   
    
     
      
       1
       ,
      
     
     
      
       Y
       =
       
        Y
        h
       
      
     
    
    
     
      
       
        
         ∑
         
          log
          ⁢
          
           P
           ⁡
           (
           
            y
            i
           
           )
          
         
        
        
         
          ❘
          "\[LeftBracketingBar]"
         
         Y
         
          ❘
          "\[RightBracketingBar]"
         
        
       
       ,
      
     
     
      else
     
    
   
  
 

where Y represents a sequence (y1, y2, . . . , yl, . . . , yn), Yh represents a manually annotated strong label sequence, a confidence of the strong label sequence is set to be 1; and
wherein the loss function of the training stage of the text generation model is adjusted by performing weighted calculation based on negative logarithmic likelihood, and affecting the degree of learning of the model by the confidence level, and wherein the loss function being as follows:

Loss(Y)=w·Σ−log P(yi)

where w represents a confidence estimate.