Patent ID: 11928957
Assignee: NANJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS
Field: Computer technology (Electrical engineering)
Classification: CPC G  Y | IPC G

Claim 3:
4. The audio-visual-aided haptic signal reconstruction method based on the cloud-edge collaboration according to claim 1, wherein Step (3) includes following steps:
(3-1), training, on a large-scale audio-visual database S={sj}j=1M stored on a central cloud, the video feature extraction network and the audio feature extraction network, specific processes being as follows:
Step 311, initializing θvs(0), θas(0) and θgs(0) that are values for θvs, θas and θgs in a 0-th iteration, respectively;
Step 312, setting a total number of iterations to be n1, giving a number of the iterations to be n=0; and setting a learning rate μ1;
Step 313, optimizing each network parameter by adopting a stochastic gradient descent method SGD:

θvs(n+1)=θvs(n)−μ1∇θvsLSrc,

θas(n+1)=θas(n)−μ1∇θasLSrc, and

θgs(n+1)=θgs(n)−μ1∇θgsLSrc,, where θvs(n+1), θas(n+1) and θgs(n+1) as well as θvs(n), θas(n) and θgs(n) are parameters for the video feature extraction network, the audio feature extraction network and an integrated network at the n+1-th iteration and the n-th iteration in the central cloud, respectively; and ∇ is a partial derivative for each loss function;
Step 314, skipping, when n<n1, to Step 313, n=n+1, and continuing a next iteration; if not, terminating the iterations; and
Step 315, obtaining, after n1 rounds of the iterations, an optimized video feature extraction network Gv(θvs) and an optimized audio feature extraction network Ga(θas);
(3-2), training the AVHR model on a multi-modal training data set received by the edge node, specific processes being as follows:
Step 321, initializing θv(0), θa(0), θm(0), θhe(0) and θl(0) that are values for θv, θa, θm, θhe, and θl in a 0-th iteration, respectively;
Step 322, starting the iterations, setting a total number of the iterations to be n2, and giving a number of the iterations to be n′=0; and setting a learning rate μ2; and
Step 323, optimizing, by adopting the stochastic gradient descent method, parameters for each feature extraction network, the fusion network, and a public classifier:

θv(n′+1)=θv(n′)−μ2∇θv(Lcorrav+Lcorrm+LDis),

θa(n′+1)=θa(n′)−μ2∇θaa(Lcorrav+Lcorrm+LDis),

θhe(n′+1)=θhe(n′)−μ2∇θhe(Lcorrav+Lcorrm+LDis+LRec),

θl(n′+1)=θl(n′)=μ2∇θlLDis, and

θm(n′+1)=θm(n′)−μ2∇θm(Lcorrm+LDis),, where θv(n′+1), θa(n′+1), θhe(n′+1), θl(n′+1), θm(n′+1) and θv(n′), θa(n′), θhe(n′), θ1(n′), and θm(n′) are respectively parameters for the video feature extraction network, the audio feature network, the encoder, and the public classifier at the n+1-th iteration and at the n-th iteration in the edge node; and the ∇ is the partial derivative for each loss function;
Step 324, optimizing, by adopting the stochastic gradient descent SGD, a parameter for the decoder:

θhd(n′+1)=θhd(n′)−μ2∇θhd(LGen+LRec),, where θhd(n′+1) and θhd(n′) are respectively parameters for the decoder at the n+1-th iteration and at the n-th iteration in the edge node; and the ∇ is the partial derivative for each loss function;
Step 325, skipping, when n′<n2, to Step 323, n′=n′+1, and continuing the next iteration; if not, terminating the iterations; and
Step 326, obtaining, after n2 rounds of the iterations, the optimal AVHR model including the optimized video feature extraction network, the optimized audio feature extraction network, an optimized haptic feature extraction network, an optimized fusion network and an optimized haptic signal generation network.