Patent ID: 11960568
Assignee: UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A multi-source domain adaptation model by aligning partial features of different domains, comprising:
a general feature extraction module;
a feature selection module for partial feature extraction and a corresponding objective function thereof;
three partial feature alignment losses, the three partial feature alignment losses comprising an intra-class partial feature alignment loss, an inter-domain partial feature alignment loss, and an inter-class partial feature alignment loss; and
two classifiers for adversarial training;, wherein:
the general feature extraction module is configured to acquire raw feature maps through conventional convolutional neural networks (CNN) and residual neural networks ResNet-101; the raw feature maps are an output of the general feature extraction module for simple and complicated images;
L1 distances between the raw feature maps of different pairs of the source domain and target domain are used as inputs of the feature selection module for partial feature extraction; the convolutional neural networks comprise three convolutional layers and two fully connected layers, and a dimension of a last layer of the networks is 2048; the residual neural networks utilize ResNet-101, with a last classification layer removed, to obtain feature maps of 2048 output dimensions of the residual neural networks;
the feature selection module for partial feature extraction is built with two fully connected layers and utilizes the L1 distances as an input to analyze highly-related feature dimensions between source domains and target domains; an output of the feature selection module is a feature selection vector for an original feature map, which is applied to the original feature map to conduct a refined feature map through dot product;
on the refined feature map, a loss function of the feature selection module for partial feature extraction, the intra-class partial feature alignment loss, the inter-domain partial feature alignment loss, and the inter-class partial feature alignment loss are calculated, respectively, and a weighted combination of the loss functions and a classification loss is used as an overall loss function;
the loss function of the feature selection module for partial feature extraction is:, L
    p
   
   =
   
    
     
      1
      
       N
       S
      
     
     ⁢
     
      
       ∑
       
        i
        =
        1
       
       
        N
        S
       
      
      
       
        
        
         
          𝔼
          ⁡
          (
          
           F
           
            S
            i
           
           k
          
          )
         
         -
         
          𝔼
          ⁡
          (
          
           F
           T
           k
          
          )
         
        
        
       
       2
      
     
    
    +
    
     
      
       (
       
        
         
          
           N
           S
          
         
        
        
         
          2
         
        
       
       )
      
      
       -
       1
      
     
     ⁢
     
      
       ∑
       
        i
        =
        1
       
       
        
         N
         S
        
        -
        1
       
      
      
       
        ∑
        
         j
         =
         
          i
          +
          1
         
        
        
         N
         S
        
       
       
        
         
         
          
           𝔼
           ⁡
           (
           
            F
            
             S
             i
            
            k
           
           )
          
          -
          
           𝔼
           ⁡
           (
           
            F
            
             S
             j
            
            k
           
           )
          
         
         
        
        2
       
      
     
    
    +
    
     
      λ
      
       r
       ⁢
       e
       ⁢
       g
      
     
     ⁢
     
      ∑
      
       
        (
        
         1
         -
         
          v
          _
         
        
        )
       
       2
      
     
    
   
  
  ,
 

where FSi=G(Si)·vi and FT=G(T)·v represent refined features of the source domains and the target domains, respectively, and k is an order of a high-order moment distance; λreg is a weight parameter for regularization; NS is a number of domains;, (
   
    
     
      
       N
       S
      
     
    
    
     
      2
     
    
   
   )
  
  =
  
   
    
     N
     S
    
    (
    
     
      N
      S
     
     -
     1
    
    )
   
   2
  
 

 is a combinatorial number, and, (
   
    
     
      
       N
       S
      
     
    
    
     
      2
     
    
   
   )
  
  
   -
   1
  
 

 is it's reciprocal; vi stands for a feature selection vector for source domain i and v is an average over the feature selection vectors of each source domain; Si represents a batch of sample from a source domain i, and T represents a batch of sample from the target domain; (*) represents the calculation of an expected value; G stands for a general feature extractor; G(Si) is the raw feature map extracted from the source domain i, and G(T) is the raw feature map extracted from the target domain;
the three partial alignment loss functions are calculated on top of class centroids on the refined feature map, which is defined as follows:, f
    
     i
     c
    
   
   =
   
    
     1
     
      n
      c
     
    
    ⁢
    
     ∑
     
      F
      
       S
       
        i
        c
       
      
     
    
   
  
  ,
 

where fic represents a centroid of class c in the source domain i;, F
  
   S
   
    i
    c
   
  
 

 stands for refined features of samples that belong to class c from the source domain i, and nc represents a number of samples in a training batch;
a class centroid of the target domain is defined as follows:, f
    
     T
     c
    
   
   =
   
    
     1
     
      n
      c
     
    
    ⁢
    
     ∑
     
      F
      
       T
       c
      
     
    
   
  
  ,
 

where fTc represents a centroid of class c in the target domain; FTc stands for the refined features of samples that belong to class c from the target domain, and nc represents the number of samples in the training batch;
to preserve information gained from previous training batches of the multi-source domain adaptation model by aligning partial features, an exponential moving average is utilized to update all of the centroids during each batch:, f
    
     i
     c
    
    b
   
   =
   
    
     
      β
      c
     
     ⁢
     
      f
      
       i
       c
      
      
       b
       -
       1
      
     
    
    +
    
     
      (
      
       1
       -
       
        β
        c
       
      
      )
     
     ⁢
     
      1
      
       n
       c
      
     
     ⁢
     
      ∑
      
       F
       
        S
        
         i
         c
        
       
       b
      
     
    
   
  
  ,
 

where βc is a weight parameter for previously calculated centroids; ficb and, F
  
   S
   
    i
    c
   
  
  b
 

 represent a maintained centroid and refined features of class c in the source domain i during batch b, respectively; the calculations of target centroids fTcb are the same as that of ficb;
the intra-class partial feature alignment loss is:, L
      c
    
    =
    
      
        
          ∑
          i
        
        ⁢
        
          
            ∑
            c
          
          ⁢
          
            
            
              
                𝔼
                ⁡
                
                  (
                  
                    
                      (
                      
                        F
                        
                          i
                          c
                        
                        b
                      
                      )
                    
                    k
                  
                  )
                
              
              -
              
                𝔼
                ⁡
                
                  (
                  
                    
                      (
                      
                        F
                        
                          S
                          
                            i
                            c
                          
                        
                        b
                      
                      )
                    
                    k
                  
                  )
                
              
            
            
          
        
      
      +
      
        
          ∑
          c
        
        ⁢
        
          
          
            
              𝔼
              ⁡
              
                (
                
                  
                    (
                    
                      F
                      
                        T
                        c
                      
                      b
                    
                    )
                  
                  k
                
                )
              
            
            -
            
              𝔼
              ⁡
              
                (
                
                  
                    (
                    
                      F
                      
                        T
                        c
                      
                      b
                    
                    )
                  
                  k
                
                )
              
            
          
          
        
      
    
  
  ;

where k stands for an order of high-order moment;
the inter-domain partial feature alignment loss is:, L
    dom
   
   =
   
    
     ∑
     i
    
    
     
      ∑
      c
     
     
      
       (
       
        
         f
         
          T
          c
         
         b
        
        -
        
         f
         
          i
          c
         
         b
        
       
       )
      
      2
     
    
   
  
  ,
 

where fTcb and ficb represent the above-mentioned centroids of class c of the target domain and the source domain i, respectively;
the inter-class partial feature alignment loss is:, L
    disc
   
   =
   
    
     ∑
     i
    
    
     
      ∑
      j
     
     
      
       (
       
        B
        -
        
         
         
          
           f
           Ti
           b
          
          -
          
           f
           Tj
           b
          
         
         
        
       
       )
      
      2
     
    
   
  
  ,
 

where fTib and fTjb represent the centroids of class i and class j of the target domain, respectively; B is a bias hyper-parameter set for restricting the degree of discreteness of different class centroids on the feature map of the target domain;
an overall loss function for the multi-source domain adaptation model by aligning partial features is:

L=Ls+λpLp+λcLc+λdomLdom+λdiscLdisc,

where Ls is a classification cross-entropy loss for source domains, which is a sum of the cross-entropy losses of two classifiers during adversarial training; λp, λc, λdom, λdisc are pre-defined weight parameters for Lp, Lc, Ldom, Ldisc, respectively.