Patent ID: 11907663
Assignee: NAVER FRANCE
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A system comprising:
a natural language processing (NLP) model trained in a training domain and configured to perform natural language processing on an input dataset;
an accuracy module configured to:
calculate a domain shift metric based on the input dataset; and
calculate a predicted decrease in accuracy of the NLP model attributable to domain shift relative to the training domain based on the domain shift metric; and

a retraining module configured to selectively trigger a retraining of the NLP model based on the predicted decrease in accuracy of the NLP model wherein one of:
(a) the domain shift metric is a proxy A domain (PAD) metric and the accuracy module is configured to calculate the domain shift metric using the equation

PAD*=1−2ε(G*d(Gf(x))), where PAD* is the domain shift metric, ε is a predetermined scalar value, and θf and θy are parameters of a first function Gf and a second function G*d learned by minimizing a loss function, and learnable parameters of Gf and G*d are learned parameters; and
(b) the domain shift metric is a reverse classification accuracy (RCA) metric and the accuracy module is configured to calculate the domain shift metric using the equation:, RCA
    *=
    
      
        
          1
          
            U
            s
            ′
          
        
        ⁢
        
          
            ∑
            
              
                x
                i
              
              ,
              
                
                  y
                  i
                
                ∈
                
                  U
                  s
                  ′
                
              
            
            
              m
              ′
            
          
          ⁢
          
                        ⁡
            
              [
              
                
                  y
                  i
                
                =
                
                  
                    C
                    ′
                  
                  ⁡
                  
                    (
                    
                      x
                      i
                    
                    )
                  
                
              
              ]
            
          
        
      
      -
      
                ⁡
        
          [
          
            
              y
              i
            
            =
            
              
                C
                _
              
              ⁡
              
                (
                
                  x
                  i
                
                )
              
            
          
          ]
        
      
    
  
  ,, where RCA* is the domain shift metric, C′ is a classifier learned on out-of-domain data and C the classifier learned on held-out in-domain data, xi and yi are data-points, Us' represents data in the training domain.