Patent ID: 11928957
Assignee: NANJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS
Field: Computer technology (Electrical engineering)
Classification: CPC G  Y | IPC G

Claim 2:
3. The audio-visual-aided haptic signal reconstruction method based on the cloud-edge collaboration according to claim 1, wherein Step (2) includes following steps:
(2-1), directly migrating an audio feature extraction network, a video feature extraction network, parameters for the audio feature extraction network and parameters for the video feature extraction network that are completely trained in the central cloud to the edge node, and taking the audio feature extraction network and the video feature extraction network as the audio attribute extraction network and the video attribute extraction network at the edge node respectively;
(2-2), taking complete audio signals, video signals and haptic signals received by the edge node as a multi-modal training data set D, D={di}i=1N, an i-th instance di=(vi, ai, hi), and (vi, ai, hi) being an i-th pair of multi-modal samples, where vi∈Rw is an i-th video signal in the multi-modal training data set, Rw is a sample space of the video signals, and w is a sample dimensionality of the video signals; ai∈Ru is an i-th audio signal in the multi-modal training data set, Ru is a sample space of the audio signals, and u is a sample dimensionality of the audio signals; hi∈Re is an i-th haptic signal in the multi-modal training data set, Re is a sample space of the haptic signals, and e is a sample dimensionality of the haptic signals; and each di has a corresponding one-hot tag yi∈RK, RK is the tag space, K is a number of categories in the multi-modal training data set;
(2-3), extracting, by means of edge node, a video attribute gv=Gv(v; θvs) and an audio attribute ga=Ga(a; θas) respectively, by using the video feature extraction network and the audio feature extraction network migrated from the central cloud, where v is a video signal, and a is an audio signal; and then, further inputting gv and ga into a multi-layer feature network to obtain a video signal feature fv=Fv(v; θv) and an audio signal feature fa=Fa(a; θa), fv and fa being associated with each other, where Fv(⋅) is the video feature extraction network at the edge node, θv represents the parameter for the video feature extraction network, Fa(⋅) is the audio feature extraction network at the edge node, and θa represents the parameter for the video feature feature extraction network;
(2-4), taking, by the edge node, an encoder of an auto-encoder model as the haptic feature extraction network, and extracting, a target haptic signal feature fh=Eh(h; θhe) for training by using the haptic feature extraction network, where h represents a haptic signal, Eh(⋅) represents the encoder at the edge node, and θhe represents a parameter for the encoder;
(2-5), fusing, by using the fusion network combining a multi-modal collaboration paradigm and the multi-modal joint paradigm, fv and fa, and obtaining the fused features,
A, the multi-modal collaboration: maximizing semantic similarities between fa, fv and fh under a constraint of a haptic modal; and
B, a multi-modal joint: deeply integrating the fa and the fv on a basis of the multi-modal collaboration paradigm, specific processes being as follows:

fm=Fm(fa,fv; θm),, where fm is a fused feature of the video signal feature and the audio signal feature that are associated with each other; Gm(⋅) is a mapping function of a multi-modal joint network, Fm(⋅) is a linear weighting of the fa and the fv; and θm is the parameter for the multi-modal joint network;
(2-6), performing a learning of the shared semantics on the video signal feature fv, the audio signal feature fa, the haptic signal feature fand the fused feature fm that are associated with each other, wherein the learning of the shared semantics includes the semantic correlation learning and the semantic discrimination learning:
the semantic correlation learning: performing, by selecting a contrast loss, a correlation constraint on fv, fa, fm and fh, reducing distances between fhand fv, fa as well as fm that are matched with fh, and enabling distances between fh and fv, fa as well as fm that are not matched with fh to be greater than a threshold δ, and defining a semantic related loss function as follows:

Lcorr=Σp≠qN,N max(0,l2(fpv,fph)+l2(fpa,fph)+δ−l2(fpv,fqh)−l2(fpa,fqh)), and

Lcorrm=Σp≠qN,N max(0,l2(fpm,fph)+δ−l2(fpm,fqh)),, where the audio signal feature fa and the haptic signal feature fh forms an audio haptic pair, the video signal feature fv and the haptic signal feature fh forms an video haptic pair, and Lcorrav is a contrast loss function of the audio haptic pair and the video haptic pair; Lcorrm is a contrast loss function of the fused feature fm and the haptic signal feature fh, fpv a p-th video signal feature, fpa is a p-th audio signal feature, fpm is a p-th fused feature, fph is a p-th haptic signal feature, and fqh is a q-th haptic signal feature; and l2(⋅)=∥⋅∥2 represents l2 norm; and
the semantic discrimination learning: selecting a full-connection layer with a softmax function as a public classifier, and adding the public classifier to the video feature extraction network, the audio feature extraction network, the haptic feature extraction network and the fusion network, ensuring a consistency and a differentiation of cross-modal semantics under a guidance of supervision information, and defining a semantic discrimination loss function as follows:, L
    
     D
     ⁢
     i
     ⁢
     s
    
   
   =
   
    
     -
     
      1
      N
     
    
    ⁢
    
     
      Σ
        
     
     i
     N
    
    ⁢
    
     
      y
      i
     
     [
     
      
       log
       ⁢
       
        p
        ⁡
        (
        
         
          f
          i
          v
         
         ;
         
          θ
          l
         
        
        )
       
      
      +
      
       log
       ⁢
       
        p
        ⁡
        (
        
         
          f
          i
          a
         
         ;
         
          θ
          l
         
        
        )
       
      
      +
      
       log
       ⁢
       
        p
        ⁡
        (
        
         
          f
          i
          h
         
         ;
         
          θ
          l
         
        
        )
       
      
      +
      
       log
       ⁢
       
        p
        ⁡
        (
        
         
          f
          i
          m
         
         ;
         
          θ
          l
         
        
        )
       
      
     
     ]
    
   
  
  ,, where LDis is the semantic discrimination loss function, p(⋅) is the public classifier, fiv is an i-th video signal feature, fia is an i-th audio signal feature, fih is an i-th haptic signal feature, fim is an i-th fused feature, and θl is a parameter for the public classifier;
(2-7), the auto-encoder model including the encoder and a decoder, learning, by comparing the haptic signal h for training with a haptic signal {tilde over (h)} obtained during a process from the encoder to the decoder, a structure of the auto-encoder model, and defining a reconstruction loss of the haptic signal as follows:, L
    
     R
     ⁢
     e
     ⁢
     c
    
   
   =
   
    
     
      1
      N
     
     ⁢
     
      
       Σ
         
      
      
       i
       =
       1
      
      N
     
     ⁢
     
      
       
       
        
         
          h
          ˜
         
         i
        
        -
        
         h
         i
        
       
       
      
      2
      2
     
    
    +
    
     α
     ⁢
     
      
       
       
        θ
        h
       
       
      
      2
      2
     
    
   
  
  ,, where LRec is a reconstruction loss function, {tilde over (h)}i is an i-th haptic signal reconstructed by the auto-encoder model, {tilde over (h)}i=Dh(Eh(hi; θhe); θhd), hi is an i-th real haptic signal; Eh(⋅) is the encoder serving as the haptic feature extract network and configured to extract haptic features; Dh(⋅) is the decoder serving as the haptic signal generation network and configured to generate the haptic features, and θh=[θhe, θhd] represents a set of parameters for the encoder, specifically, θhe is a parameter for the encoder, θhd is a parameter for the decoder, and α is a hyperparameter; and
(2-8), generating, by using the decoder Dh(⋅) of the auto-encoder model, the target haptic signal h′ from the fm to implement the reconstruction of the target haptic signal, and remapping, by the encoder Eh (⋅), the h′ to a haptic signal feature fh′, and defining a loss function of the haptic signal generated as follows:, L
    
     G
     ⁢
     e
     ⁢
     n
    
   
   =
   
    
     
      1
      N
     
     ⁢
     
      
       ∑
        
      
      
       i
       =
       1
      
      N
     
     ⁢
     
      {
      
       
        
         
         
          
           h
           i
          
          -
          
           h
           i
           ′
          
         
         
        
        2
        2
       
       +
       
        β
        [
        
         
          
           l
           2
          
          (
          
           
            f
            i
            h
           
           ,
           
            f
            i
            
             h
             ′
            
           
          
          )
         
         +
         
          
           y
           i
          
          ⁢
          log
          ⁢
          
           p
           ⁡
           (
           
            f
            i
            
             h
             ′
            
           
           )
          
         
        
        ]
       
      
      }
     
    
    +
    
     γ
     ⁢
     
      
       
       
        θ
        
         h
         ⁢
         d
        
       
       
      
      2
      2
     
    
   
  
  ,, where LGen is a generating loss function of the haptic signal, hi′=Dh(fim; θhd) is an i-th haptic signal generated by the fused feature, fim is an i-th fused feature, fih is an i-th haptic feature, fih′=Eh(hi′; θhd) is a semantic feature of hi′ extracted by the encoder, l2(fih, fih′) represents a similarity between fih and fih′, yi log p(fih′) is a classification loss of fih′, p(fih′) is a predicted tag of fih′, l2(fih, fih′) and yi log p(fih′) together form a regular term of a loss function; and β and γ are hyperparameters.