Patent ID: 11960568
Assignee: UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 1:
2. A method for multi-source domain adaptation by aligning partial features using the multi-source domain adaptation model by aligning partial features of claim 1, the method comprising:
Step 1: preprocessing data, selecting three open datasets comprising Digit-Five, Office-31, and DomainNet for experiments;
where, Digit-Five collects subsets of five different types of hand-written digit classification datasets, which are MNISTM-M, MNIST, USPS, SVHN, and Synthetic Digits; USPS contains 9298 images and other datasets each contains 25000 images for training and 9000 images for validation;
Office-31 is a conventional multi-source domain adaptation dataset comprising 4652 images in 31 different categories that are collected from an office environment and are presented in three domains: Amazon, Webcam, and DSLR;
DomainNet contains six domains of data: clipart, infograph, painting, quickdraw, real, and sketch, and each domain contains 345 categories of images;
the data preparation process comprises image resizing, random clip, and random crop;
Step 2: extracting raw features from processed data using the general feature extraction module; where, for the Digit-Five dataset, the images are resized to 32×32, and the general feature extraction module is constructed with three convolutional layers and two fully connected layers, where a convolution kernel size is 5 and the output of the fully connected layers is a feature vector with 2048 dimensions; for the Office-31 dataset, the images are resized to 252×252, and the general feature extraction module is AlexNet pre-trained on ImageNet, and outputs a feature vector with 4096 dimensions; for the DomainNet dataset, the image size is 224×224 and the pre-trained ResNet-101 is utilized as the general feature extractor, and outputs a 2048-dimension feature vector; fSi and fT represent the feature map of the source domain i and that of the target domain, respectively;
during training, a batch size of Digit-Five is set to 128 and a batch size of 16 is utilized in the training procedure of the other two datasets, thus the dimension of feature map during training on Digit-Five is 128×2048, and 16×4096 for Office-31, and 16×2048 for DomainNet; as for the training epochs, 100 epochs of training are done on both Digit-Five and Office-31, and due to the large amount of data in DomainNet, the model is only trained for 20 epochs;
Step 3: calculating partial feature alignment losses and selecting partial features from raw features; computing an absolute difference Δi=|fSi−fT| between the feature maps of the source domains and the target domain, and utilizing the difference as the input of the feature selection module, and an output thereof is a feature selection vector vi; computing the refined feature map of the source domains by FSi=fSi·vi; where, for the target domain, an average vector v over the feature selection vectors of the source domains is regarded as the selection vector, and the refined feature map for the target domain is represented as FT;
conducting the partial feature alignment loss Lp for the partial feature selection module on top of FSi and FT, which is defined as follows:, L
    p
   
   =
   
    
     
      1
      
       N
       S
      
     
     ⁢
     
      
       ∑
       
        i
        =
        1
       
       
        N
        S
       
      
      
       
        
        
         
          𝔼
          ⁡
          (
          
           F
           
            S
            i
           
           k
          
          )
         
         -
         
          𝔼
          ⁡
          (
          
           F
           T
           k
          
          )
         
        
        
       
       2
      
     
    
    +
    
     
      
       (
       
        
         
          N
         
        
        
         
          2
         
        
       
       )
      
      
       -
       1
      
     
     ⁢
     
      
       ∑
       
        i
        =
        1
       
       
        
         N
         S
        
        -
        1
       
      
      
       
        ∑
        
         j
         =
         
          i
          +
          1
         
        
        
         N
         S
        
       
       
        
         
         
          
           𝔼
           ⁡
           (
           
            F
            
             S
             i
            
            k
           
           )
          
          -
          
           𝔼
           ⁡
           (
           
            F
            
             S
             j
            
            k
           
           )
          
         
         
        
        2
       
      
     
    
    +
    
     
      λ
      
       r
       ⁢
       e
       ⁢
       g
      
     
     ⁢
     
      ∑
      
       
        (
        
         1
         -
         
          v
          _
         
        
        )
       
       2
      
     
    
   
  
  ,
 

where k stands for the order of the moment; λreg is the weight parameter for regularization; NS is the number of the source domains;, (
   
    
     
      
       N
       S
      
     
    
    
     
      2
     
    
   
   )
  
  =
  
   
    
     N
     S
    
    (
    
     
      N
      s
     
     -
     1
    
    )
   
   2
  
 

 is the combinatorial number and, (
   
    
     
      
       N
       S
      
     
    
    
     
      2
     
    
   
   )
  
  
   -
   1
  
 

 is its reciprocal; (*) represents the calculation of an expected value;
to further conduct partial feature alignment losses, maintaining the centroids of each category and each domain on the refined feature map as follows:, f
    
     i
     c
    
    1
   
   =
   
    
     1
     
      n
      c
     
    
    ⁢
    
     
      ∑
      
       b
       -
       1
      
     
     
      F
      
       S
       
        i
        c
       
      
      1
     
    
   
  
  ⁢
  

  
   
    
     f
     
      i
      c
     
     b
    
    =
    
     
      
       β
       c
      
      ⁢
      
       f
       
        i
        c
       
       
        b
        -
        1
       
      
     
     +
     
      
       (
       
        1
        -
        
         β
         c
        
       
       )
      
      ⁢
      
       1
       
        n
        c
       
      
      ⁢
      
       ∑
       
        F
        
         S
         
          i
          c
         
        
        b
       
      
     
    
   
   ,
  
 

where, f
   
    i
    c
   
   b
  
  ⁢
     
  and
  ⁢
     
  
   F
   
    S
    
     i
     c
    
   
   b
  
 

 represent the maintained centroid and refined features of class c in the source domain i during batch b, respectively; βc is a trade-off parameter between the accumulated value and a new value of the class centroids;
on top of the centroids, calculating the intra-class, inter-domain, and inter-class partial feature alignment losses;
the intra-class partial feature alignment loss is as follows:, L
      c
    
    =
    
      
        
          ∑
          i
        
        ⁢
        
          
            ∑
            c
          
          ⁢
          
            
            
              
                𝔼
                ⁡
                
                  (
                  
                    
                      (
                      
                        F
                        
                          i
                          c
                        
                        b
                      
                      )
                    
                    k
                  
                  )
                
              
              -
              
                𝔼
                ⁡
                
                  (
                  
                    
                      (
                      
                        F
                        
                          S
                          
                            i
                            c
                          
                        
                        b
                      
                      )
                    
                    k
                  
                  )
                
              
            
            
          
        
      
      +
      
        
          ∑
          c
        
        ⁢
        
          
          
            
              𝔼
              ⁡
              
                (
                
                  
                    (
                    
                      F
                      
                        T
                        c
                      
                      b
                    
                    )
                  
                  k
                
                )
              
            
            -
            
              𝔼
              ⁡
              
                (
                
                  
                    (
                    
                      F
                      
                        T
                        c
                      
                      b
                    
                    )
                  
                  k
                
                )
              
            
          
          
        
      
    
  
  ;

where k represents the order of the high-order moment; fTcb and FTcb represent the centroid and the refined features of class c in the target domain during batch b; the loss is calculated on both source domains and the target domain, and pseudo labels generated from the classifier are regarded as target class labels;
the inter-domain partial feature alignment loss is as follows:, L
    dom
   
   =
   
    
     ∑
     i
    
    
     
      ∑
      c
     
     
      
       (
       
        
         f
         
          T
          c
         
         b
        
        -
        
         f
         
          i
          c
         
         b
        
       
       )
      
      2
     
    
   
  
  ,
 

where fTcb and ficb represent the centroids of class c of the target domain and the source domain i, respectively; only the distances between the class centroids of the target domain and the dedicated class centroids of different source domains are calculated in this period;
the inter-class partial feature alignment loss is:, L
    disc
   
   =
   
    
     ∑
     i
    
    
     
      ∑
      j
     
     
      
       (
       
        B
        -
        
         
         
          
           f
           
            T
            i
           
           b
          
          -
          
           f
           
            T
            j
           
           b
          
         
         
        
       
       )
      
      2
     
    
   
  
  ,
 

where fTib and fTjb represent the centroids of class i and class j of the target domain, respectively; the inter-class partial feature alignment loss is only calculated for the target domain; an Euclidean distance between each pair of class centroids is first calculated and then the L2 distance between the Euclidean distance and a pre-defined bias parameter B is calculated;
deriving classification probabilities from the two classifiers utilizing the refined feature maps as the input for adversarial training; calculating the cross-entropy loss Ls for each source domain and conducting the overall loss function through a weighted combination of the loss functions:

L=Ls+λpLp+λcLc+λdomLdom+λdiscLdisc,

where Ls is conducted with the sum of cross-entropy losses of the two adversarial training classifiers; λp, λc, λdom, Ldisc are pre-defined weight parameters for Lp, Lc, Ldom, Ldisc, respectively; the parameters of all of the modules of the multi-source domain adaptation model by aligning partial features are updated with the loss function L, the modules comprising the general feature extraction module, the feature selection module for partial feature extraction, and two adversarial training classifiers;
Step 4: training the general feature extraction module and the two classifiers in an adversarial manner; repeating Step 2 and Step 3, to derive the refined feature maps of the source domains and the target domain without calculating the dedicated feature alignment losses; on the refined feature map, generating, by the two classifiers, target predictions independently, calculating the absolute difference Ldis of the predictions after a softmax activation, and the cross-entropy loss Ls for the two predictions; fixing the parameters of the general feature extraction module, and updating the two classifies using Ls−Ldis as the loss function to increase the discrepancy of the target predictions; fixing the parameters of the two classifiers and re-compute Ldis as the loss function to update the feature extraction module, aiming to reduce the classification discrepancy, which contributes to the adversarial training between the feature extraction module and the two classifiers; for the general feature extraction module, repeating calculating Ldis and updating the parameters for four times, where the number of repeats is a trade-off between the general feature extractor and the two classifiers; and
Step 5: predicting on a validation set; training, by the multi-source domain adaptation model by aligning partial features, the three open datasets in Step 1 following operations from Step 2 to Step 4, and testing the validation set.