Patent ID: 11928957
Assignee: NANJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS
Field: Computer technology (Electrical engineering)
Classification: CPC G  Y | IPC G

Claim 1:
2. The audio-visual-aided haptic signal reconstruction method based on the cloud-edge edge collaboration according to claim 1, wherein Step (1) includes following steps:
(1-1), for the large-scale audio-visual database S={sj}j=1M, where M is a number of the video frames and the audio clips that are in pairs, sj=(vjs, ajs), sj is a j-th pair of the video frames and the audio clips, transferring the j-th video frame vjs and the j-th audio clip ajs to the video feature extraction network and the audio feature extraction network respectively, and extracting corresponding video features and audio features respectively; and
(1-2), connecting the video features and the audio features, and inputting the video features and the audio features into an integrated network composed of a plurality of full-connection layers and outputting integrated features, and then performing, by using the integrated features, the self-supervision learning task, wherein an objective of the self-supervision learning is to determine whether the video frames and audio clips are from the same audio-visual source; and specifically, defining a self-supervision loss function as follows:, L
    Src
   
   =
   
    
     
      -
      
       1
       M
      
     
     ⁢
     
      
       Σ
         
      
      
       j
       =
       1
      
      M
     
     ⁢
     
      p
      ⁡
      (
      
       
        g
        j
        s
       
       ;
       
        θ
        g
        s
       
      
      )
     
     ⁢
     log
     ⁢
     
      
       p
       ˆ
      
      (
      
       
        g
        j
        s
       
       ;
       
        θ
        g
        s
       
      
      )
     
    
    +
    
     
      [
      
       1
       -
       
        p
        ⁡
        (
        
         
          g
          j
          s
         
         ;
         
          θ
          g
          s
         
        
        )
       
      
      ]
     
     ⁢
     
      log
      [
      
       1
       -
       
        
         p
         ˆ
        
        (
        
         
          g
          j
          s
         
         ;
         
          θ
          g
          s
         
        
        )
       
      
      ]
     
    
   
  
  ,, where LSrc is the self-supervision loss function, gjs=[Gv(vjs; θvs), Ga(ajs; θas)] is a feature after integrating a j-th pair of video frame features and audio clip features, Gv(⋅) is a feature mapping of the video feature extraction network, θvs is a parameter for the video feature extraction network, Ga(⋅) is a feature mapping of the audio feature extraction network, θas is a parameter for the audio feature extraction network; p(⋅) represents a tag indicator, when the tag indicator is 1, it represents that the video frames and audio clips are from the same audio-visual source, when the tag indicator is 0, it represents that the video frame and audio clip are from different audio-visual sources; {circumflex over (p)}(⋅) represents a correspondence predicted value output from the integrated network; θgs represents a parameter for the integrated network composed of the plurality of full-connection layers; and the pre-trained audio feature extraction network and the pre-trained video feature extraction network are obtained by minimizing LSrc.