Method, electronic device, and computer program product for video reconstruction

Embodiments of the present disclosure relate to a method, an electronic device, and a computer program product for video reconstruction. The method for video reconstruction includes: receiving a video segment comprising a plurality of image frames. The method further includes: determining an audio segment corresponding to the video segment. The method further includes: acquiring a plurality of mask maps corresponding to the plurality of image frames, respectively. The method further includes: reconstructing the video segment based on the audio segment, the plurality of image frames, and the plurality of mask maps.

The present application claims priority to Chinese Patent Application No. 202210875518.4, filed Jul. 22, 2022, and entitled “Method, Electronic Device, and Computer Program Product for Video Reconstruction,” which is incorporated by reference herein in its entirety.

FIELD

Embodiments of the present disclosure relate to the field of image processing, and more particularly, to a method, an electronic device, and a computer program product for video reconstruction.

BACKGROUND

With the ongoing development of communication technology and the continuous emergence of various terminal devices, users are able to produce videos, watch videos, or communicate through videos more conveniently. In order to better enhance the user experience, it is expected to provide users with videos having at least high resolution and audio-video synchronization.

SUMMARY

Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for video reconstruction.

According to a first aspect of the present disclosure, a method for video reconstruction is provided. The method includes: receiving a video segment comprising a plurality of image frames. The method further includes: determining an audio segment corresponding to the video segment. The method further includes: acquiring a plurality of mask maps corresponding to the plurality of image frames, respectively. The method further includes: reconstructing the video segment based on the audio segment, the plurality of image frames, and the plurality of mask maps.

According to a second aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, wherein the instructions, when executed by the at least one processor, cause the device to execute actions including: receiving a video segment comprising a plurality of image frames; determining an audio segment corresponding to the video segment; acquiring a plurality of mask maps corresponding to the plurality of image frames, respectively; and reconstructing the video segment based on the audio segment, the plurality of image frames, and the plurality of mask maps.

According to a third aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions, wherein the machine-executable instructions, when executed by a machine, cause the machine to perform steps of the method in the first aspect of the present disclosure.

In the drawings, identical or corresponding numerals represent identical or corresponding parts.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the drawings show some embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms, and should not be explained as being limited to the embodiments stated herein. Instead, these embodiments are provided for understanding the present disclosure more thoroughly and completely. It should be understood that the accompanying drawings and embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the protection scope of the present disclosure.

In the description of embodiments of the present disclosure, the term “include” and similar terms thereof should be understood as open-ended inclusion, that is, “including but not limited to.” The term “based on” should be understood as “based at least in part on.” The term “an embodiment” or “the embodiment” should be understood as “at least one embodiment.” The terms “first,” “second,” and the like may refer to different or identical objects. Other explicit and implicit definitions may also be included below.

In order to provide users with videos of higher quality (e.g., with higher resolution, audio-video synchronization, etc.), there a number of available video processing methods. However, videos generated by these techniques are often blurry and of low resolution, and thus still cannot meet the expectations of users.

To address at least the above and other potential problems, embodiments of the present disclosure provide a method for video reconstruction. The method includes: receiving a video segment comprising a plurality of image frames. The method further includes: determining an audio segment corresponding to the video segment. The method further includes: acquiring a plurality of mask maps corresponding to the plurality of image frames, respectively. The method further includes: reconstructing the video segment based on the audio segment, the plurality of image frames, and the plurality of mask maps. This method makes effective use of audio information during video reconstruction, which can greatly improve the audio-video synchronization in a video while reducing the amount of computation and saving time for video processing, and can also obtain a video with higher resolution, thus greatly enhancing the viewing experience of users.

Embodiments of the present disclosure will be further described in detail with reference to the accompanying drawings below.FIG.1is a schematic diagram of an example environment100in which embodiments of the present disclosure can be implemented.

An illustration is provided below with example environment100in which user110sends video segment150to user120. In example environment100, user110may send video segment150to user120via network140. Video segment150may include a video segment stored in computing device116of user110or may include a video segment captured in real time by user110via a video capture device (such as a camera) of computing device116, and the present disclosure does not limit the source of the video and the way in which the video is acquired. In addition, depending on the actual application situation, video segment150may have different resolutions, such as 720P, 1080P, and 4K, and the present disclosure does not limit the resolution of video segment150.

In one embodiment, video segment150includes a plurality of video image frames150i(i being a positive integer greater than or equal to 1). It should be understood that the present disclosure does not limit the number of image frames in video segment150, and the video segment according to embodiments of the present disclosure may include a video of any length, and the present disclosure does not limit the length of the video segment, either. In addition, a video segment according to embodiments of the present disclosure may include a complete segment of video, or may include a portion of video from a complete segment of video, which is not limited in the present disclosure.

Computing device116of user110may encode and compress video segment150and send encoded video stream152to user120via network140. Network140includes, but is not limited to, various types of networks such as the Internet, a local area network, and a wireless network, which is not limited in the present disclosure. It can be understood that a transmitted video stream may be damaged during transmission due to the impact of transmission delays or due to the impact of conditions such as improper compression or network failures during transmission, as shown inFIG.1, where some information may be lost in video stream154received by user120(inFIG.1, a block with a filling pattern is used to cover part of video stream154to indicate that the information in the corresponding part is lost). Accordingly, video segment160obtained after decoding and decompressing the received video stream154by computing device126of user120will also lose some information, as shown in video segment160inFIG.1.

In such case, damaged video segment160may be reconstructed by employing the method for video reconstruction according to an embodiment of the present disclosure, thereby obtaining reconstructed video segment150′. For example, damaged video segment160may be reconstructed by local computing device126of user120, or by a server to which it is uploaded, thereby resulting in reconstructed video segment150′.

The present disclosure does not limit the type of a computing device that performs the method for video reconstruction according to embodiments of the present disclosure. For example, the computing device may include, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), and a media player), a multi-processor system, a consumer electronic product, a wearable electronic device, a smart home device, a minicomputer, a mainframe computer, an edge computing device, a distributed computing environment including any of the above systems or devices, etc. When performing video reconstruction, the computing device may receive video segment150including a plurality of image frames, determine an audio segment corresponding to video segment150, acquire a plurality of mask maps corresponding to the plurality of image frames, respectively, and reconstruct the video segment based on the audio segment, the plurality of image frames, and the plurality of mask maps.

Although the video reconstruction in the video transmission scenario is described above in conjunction withFIG.1, it can be understood by a person skilled in the art that the method for video reconstruction according to embodiments of the present disclosure may not be limited to the scenario described above, but may also be used as needed in any scenario where reconstruction of a video is required, and the present disclosure does not limit the application scenario. The method according to embodiments of the present disclosure makes effective use of audio information during video reconstruction, which can greatly improve the audio-video synchronization in a video while reducing the amount of computation and saving time for video processing, and can also obtain a video with higher resolution, thus greatly enhancing the viewing experience of users.

A block diagram of example environment100in which embodiments of the present disclosure can be implemented has been described above with reference toFIG.1. A flow chart of method200for video reconstruction according to an embodiment of the present disclosure is described below in conjunction withFIG.2. Method200can be performed at computing device126of user120inFIG.1or at any suitable computing device.

At block202, computing device126may receive a video segment including a plurality of image frames. As described above in conjunction withFIG.1, computing device126may receive video segment150sent from user110via network140. The processed and transmitted video segment150may be damaged during processing and/or transmission and lose information. Therefore, video segment160received by computing device126may be in need of repair through video reconstruction, and video segment160includes a plurality of image frames160i(i being a positive integer greater than or equal to 1). Further, image frames160icorrespond to video image frames150i.

At block204, computing device126may determine an audio segment corresponding to the video segment. In one embodiment, computing device126may extract a corresponding audio segment Asegfrom the received video segment by means of existing audio extraction techniques or future-developed audio extraction techniques. The extracted audio segment Asegwill be combined in a subsequent process for use in reconstructing the video segment.

At block206, computing device126may acquire a plurality of mask maps corresponding to the plurality of image frames, respectively. In one embodiment, the mask map may include a binary image, for example, a region of interest in the mask map is set to have a pixel value of 255 and the remaining regions in the image are set to have a pixel value of 0. For image frames160i, they may correspond to mask maps Mi.

The mask maps may be obtained by means of various known or future-developed techniques. In one embodiment, computing device126may perform target detection on the plurality of image frames, respectively, to obtain the region of interest (e.g., a foreground target region) through detection. Computing device126may set the pixel value of the target region detected from the plurality of image frames to a first pixel value, e.g., pixel value of 255, and set the pixel values of regions in the plurality of image frames other than the target region to a second pixel value, e.g., pixel value of 0. In this way, computing device126can obtain a mask map corresponding to at least one image frame (e.g., each image frame) in video segment160. In another embodiment, the computing device may also obtain a mask map by calculating pixel value differences between pixels in an image frame, categorizing and combining pixels between which the pixel value difference is greater than a threshold, respectively, and performing binarization processing on the categorized pixels.

It can be understood that the above implementation for acquiring a mask map is only an example, and that in other embodiments, computing device126may also upload video segment160to a server for the server to acquire a mask map corresponding to each image frame in video segment160, and then receive the mask map from the server for use in reconstructing video segment160. The present disclosure does not limit the manner in which the mask map is acquired.

At block208, computing device126reconstructs video segment160based on the audio segment Aseg, the plurality of image frames160i, and the plurality of mask maps Mi to obtain reconstructed video segment150′. By utilizing the audio segment Asegand the mask map Mi, not only can the lost information in video segment160be supplemented, but reconstructed video segment150′ can also be synchronized (e.g., frame-synchronized) with the audio segment Aseg, such that user120can obtain a video segment with higher resolution and more consistent synchronization compared with video segment150originally sent, whereby the viewing experience of user120is greatly enhanced. A specific implementation for reconstructing video segment160will be described in detail below in conjunction with the accompanying drawings.

This method makes effective use of audio information during video reconstruction, which can greatly improve the audio-video synchronization in a video while reducing the amount of computation and saving time for video processing, and can also obtain a video with higher resolution, thus greatly enhancing the viewing experience of users.

A flow chart of a method300(i.e., a specific implementation corresponding to block208inFIG.2) for reconstructing a video segment in a video reconstruction process according to an embodiment of the present disclosure will be described below in conjunction withFIG.3. Method300can be performed at computing device126of user120inFIG.1or at any suitable computing device. The process of implementing method300inFIG.3may be described in conjunction with the schematic diagram of video reconstruction architecture400inFIG.4. It can be understood that video reconstruction architecture400may be deployed at computing device126. In addition, video reconstruction architecture400may also be deployed at a server side, which is not limited in the present disclosure. Further, an illustration will be provided below through an example in which computing device126performs method300for video reconstruction according to embodiments of the present disclosure and video reconstruction architecture400is deployed at computing device126.

At block302, computing device126acquires foreground fusion information based on the audio segment, the plurality of image frames in the video segment, and the plurality of mask maps.

As shown inFIG.4, video reconstruction architecture400includes audio segment extractor410and video reconstructor420. Audio segment extractor410can be used to extract an audio segment from the video segment, and video reconstructor420reconstructs the received video segment160. Each image frame160iinFIG.4may include a foreground target and a background region. For example, the foreground target region of the image frame160iinFIG.4is a target face region, and the background region is a plurality of plants.

In one embodiment, audio segment extractor410receives video segment160and extracts an audio segment from video segment160. The manner in which the audio segment is extracted has been described above and will not be repeated here for the sake of brevity. Audio segment extractor410inputs the extracted audio segment Aseginto video reconstructor420, and video reconstructor420may acquire foreground fusion information based on the audio segment Aseg, the plurality of image frames in video segment160, and the corresponding plurality of mask maps. The specific implementation for acquiring foreground information will be described below.

At block304, computing device126may acquire background information based on the plurality of image frames in the video segment and the corresponding plurality of mask maps. In one embodiment, video reconstructor420may acquire the background information based on the plurality of image frames in the video segment and the corresponding plurality of mask maps.

At block306, computing device126performs fusion processing on the foreground fusion information and the background information to reconstruct the video segment, so as to obtain the reconstructed video segment. In one embodiment, video reconstructor420may perform fusion processing on the foreground fusion information and the background information to reconstruct the video segment.

A specific implementation of method300described above will be described in detail below in conjunction withFIG.5.FIG.5is a detailed block diagram of architecture500according to an embodiment of the present disclosure for implementing the video reconstruction method according to embodiments of the present disclosure. It can be understood that the block diagram inFIG.5is only schematic for the purpose of illustration. Depending on actual needs, other parts and components may also be included inFIG.5, which is not limited in the present disclosure. Architecture500shown inFIG.5may be implemented in computing device126or may be implemented in a server, etc., which is not limited in the present disclosure.

As shown inFIG.5, architecture500includes audio segment extractor410and video reconstructor420. Audio segment extractor410can be used to extract an audio segment from the video segment, and video reconstructor420reconstructs the received video segment160. Video reconstructor420includes audio feature extractor421, video feature extractor422, foreground information fuser423, decoder424, background information extractor425, and foreground and background information fuser426. Video reconstructor420may perform the video reconstruction method according to embodiments of the present disclosure. Accordingly, video reconstructor420may perform method300shown inFIG.3.

In one embodiment, audio feature extractor421in video reconstructor420may receive the audio segment Asegand extract an audio feature fAof the audio segment Aseg. Specifically, audio feature extractor421may extract the Mel spectrum of the audio segment Asegto obtain a two-dimensional frequency map A∈Rdimf×dof the audio segment Aseg, thereby acquiring feature information of the audio segment Aseg, e.g., feature vector fA, where dimfis the number of filters used to extend the frequency band, and d is the length of each feature vector fAextracted by audio feature extractor421. In one embodiment, dimf=64, and d=1024.

Video reconstructor420may receive the plurality of image frames160iin video segment160, where each of the image frames160imay include a foreground target and a background region. Taking image frames160iinFIG.5as an example, the foreground target region of the image frames160iis a target face region, and the background region is a plurality of plants. Video reconstructor420may also acquire a mask map430icorresponding to each of the image frames160i. The implementation of the acquisition of mask map430has been described above and will not be repeated here. Video reconstructor420may use the plurality of mask maps430ito process the corresponding plurality of image frames160iin video segment160, respectively (e.g., by means of dot multiplication of mask maps430iwith corresponding image frames160i), so as to acquire the plurality of mask-processed foreground image frames440i(as shown inFIG.5). The background pixel value in the mask-processed foreground image frames440iis 255 so that the foreground target region can be highlighted.

For the plurality of mask-processed foreground image frames440i, video feature extractor422in video reconstructor420may extract foreground video feature fVin the plurality of mask-processed foreground image frames440i. Foreground information fuser423may receive audio feature fAfrom audio feature extractor421and foreground video feature fVfrom video feature extractor422, and acquire the foreground fusion information based on the audio feature fAand the foreground video feature fV. In one embodiment, foreground information fuser423may perform normalization processing on the audio feature fAand the foreground video feature fV, respectively, and concatenate the normalized audio feature with the normalized foreground video feature to acquire foreground fusion information, and decoder424performs a decoding operation on the foreground fusion information to acquire the decoded feature FP, as shown in Equation 1 below:
FP=P(fV,fA)=P[concat(norm(fV),norm(fA))]  (Equation 1)where function P denotes the decoding processing, function concat denotes the concatenation operation, and function norm denotes the normalization processing on parameters.

As a result of the above processing, video reconstructor420may acquire foreground fusion information based on the audio segment Aseg, the plurality of image frames160iin video segment160, and the corresponding plurality of mask maps430i.

Background information extractor425in video reconstructor420may be used to acquire background information. In one embodiment, video reconstructor420may acquire complementary mask maps430′ corresponding to the plurality of mask maps430, that is, the pixel value of the target region in the image is set to 0, and the pixel values of regions in the image other than the target region are set to 255, and the corresponding complementary mask maps430′ can be acquired.

Video reconstructor420processes image frames160iin video segment160using complementary mask maps430′ corresponding to mask maps430, for example, by performing dot multiplication processing on the plurality of image frames160iin video segment160and the corresponding complementary mask maps430′ito acquire complementary mask-processed plurality of background image frames450i, where as shown inFIG.5, background image frames450imay display only images of the background region. Background information extractor425receives generated background image frames450iand extracts the image feature fTin the above complementary mask-processed plurality of background image frames450ias the background information.

Foreground and background information fuser426in video reconstructor420performs fusion processing on the foreground fusion information and the background information to reconstruct the video segment. Specifically, foreground and background information fuser426may perform convolution processing on the plurality of mask maps430iand acquire the corresponding mask features f(α), determine the complementary mask features (1−f(α)) corresponding to the mask features, and reconstruct the video segment based on the complementary mask features (1−f(α)) and the foreground fusion information as well as the mask features f(α) and the background information, so as to acquire reconstructed video segment180.

As described above, foreground fusion information includes foreground features, e.g., foreground features FPobtained through decoding by decoder424, and the background information includes background features, e.g., image features fTobtained via background information extractor425. Foreground and background information fuser426may perform convolution processing on the plurality of mask maps430ito obtain the mask features f(α), and foreground and background information fuser426may determine the complementary mask features fcon=(1−f(α)) corresponding to the mask features f and acquire the video feature FUof the reconstructed video segment180based on the complementary mask features fconand the foreground fusion information as well as the mask features f(α) and the background information fT, so as to reconstruct the video segment. The above operations may be reflected by the following Equation 2:
FU=U(FP,fT,α)=U[f(α)×fT+(1−f(α))×FP]  (Equation 2)where f(α) denotes the convolution processing on the mask maps to obtain the mask features f, and (1−f(α)) denotes the determination of the complementary mask features corresponding to the mask features f.

By the above operation, foreground and background information fuser426can obtain the features of the reconstructed video for use in reconstructing video segment180.

An example embodiment illustrating feature acquisition for use in reconstructing video160has been described above in conjunction withFIGS.3-5. This method makes effective use of audio information during video reconstruction, which can greatly improve the audio-video synchronization in a video while reducing the amount of computation and saving time for video processing, and can also obtain a video with higher resolution, thus greatly enhancing the viewing experience of users.

Video reconstructor420described inFIG.5includes audio feature extractor421, video feature extractor422, foreground information fuser423, decoder424, background information extractor425, and foreground and background information fuser426, and each of the above components may be implemented by means of a neural network model. The specific structure of the models will be described below in conjunction withFIG.6so that the implementation of the solution for video reconstruction according to embodiments of the present disclosure can be more easily understood by those skilled in the art.

FIG.6illustrates specific structural diagram600of components in the video reconstructor according to an embodiment of the present disclosure. As shown inFIG.6, audio feature extractor421, video feature extractor422, foreground information fuser423, decoder424, background information extractor425, and foreground and background information fuser426included in video reconstructor420may be implemented by means of a neural network model. The present disclosure does not limit the specific implementation of each model, and various known and future-developed neural network models may be used based on the needs of a processing task to be implemented.

For example, video feature extractor422and audio feature extractor421may perform feature extraction on image frames and audio segments in the video segment, respectively, so as to acquire corresponding image features fVand audio features fA. Foreground information fuser423may concatenate the image features fVand audio features fAafter normalization, and input the concatenated features to decoder424, and then decoder424decodes them and inputs the decoded feature information to foreground and background information fuser426. Furthermore, background information extractor425may extract the background information of the video image frames in the manner described above and input the extracted background information to foreground and background information fuser426. Foreground and background information fuser426may also receive mask maps430icorresponding to the plurality of video image frames160i, perform convolution processing on mask maps430ito acquire the corresponding mask features, and perform dot multiplication processing on the mask features and the background information to obtain a first result. Foreground and background information fuser426may also determine the complementary mask features corresponding to the mask features and perform dot multiplication processing on the complementary mask features and the foreground fusion information to obtain a second result. Afterwards, foreground and background information fuser426may also superimpose the first result with the second result, thus obtaining the feature FUof the reconstructed video. Reconstructed video180may be obtained by performing operations such as full connection on the feature FUof the reconstructed video.

The structural diagrams of the components in the video reconstruction architecture according to an embodiment of the present disclosure have been described above in conjunction withFIG.5andFIG.6. The above structure realize video reconstruction in a simple and efficient manner, reduces system power consumption, increases computing speed, and can also obtain videos with higher resolution and more consistent synchronization, thus greatly enhancing the viewing experience of users.

In one embodiment, the video reconstruction method according to embodiments of the present disclosure may further receive a reference image and migrate the style of the reconstructed video to the style of the reference image based on the style of the reference image, thereby enriching users' options and enhancing users' experience.

The reconstruction and generation of a reconstructed video with a style migration effect based on the reference image will be described below in conjunction withFIG.7andFIG.8on the basis ofFIG.5andFIG.6.

Compared with architecture500inFIG.5, video reconstruction architecture700illustrated inFIG.7adds channel converter428and image feature extractor429, wherein image feature extractor429may be used to receive reference image770and extract image features in reference image770. Channel converter428may be used to convert foreground video feature f v to the image channel where reference image770is located using the reference image features extracted by image feature extractor429. The reference image may include an image having a certain style. For example, as shown inFIG.7, reference image770inFIG.7may be an image in the oil painting style. It may be understood that the user may select various types and styles of reference images as needed, which is not limited in the present disclosure.

Image feature extractor429may use various types of neural network models known in the field and to be developed in the future to perform feature extraction on the reference image, and the present disclosure does not limit the specific structure and implementation manner of the image feature extractor. Image feature extractor429extracts the reference image features fR. Afterwards, video reconstructor420may process the foreground video features fVusing the acquired reference image features fRto convert foreground video features fVto the image channel where reference image770is located and concatenate the channel-converted foreground video features with the audio features fAto acquire channel-converted foreground fusion information. Specifically, channel converter428may convert the foreground video features fVto the image channel where reference image770is located according to the following Equation 3:

where the functions μ and σ are the mean and variance of the parameters, respectively, R denotes the reference image features, and V denotes the features of the video image frames.

With the above Equation 3, the channel-converted foreground video features T (fV) can be acquired, and the channel-converted foreground video features T (fV) can be fused together with the audio features fAvia foreground information fuser423to obtain foreground information. The specific implementation is similar to that described above in conjunction withFIG.5and will not be repeated here for the sake of brevity. In one embodiment, foreground and background information fuser426may fuse the foreground information and the background information to reconstruct the video segment. For example, foreground and background information fuser426may perform convolution processing on the plurality of mask maps and acquire the corresponding mask features, and foreground and background information fuser426may further determine complementary mask features corresponding to the mask features and reconstruct the video segment based on the complementary mask features and the channel-converted foreground fusion information T(fV), as well as the acquired corresponding mask features and the background information The specific implementation is similar to that for foreground and background information fuser426described above in conjunction withFIG.5and will not be repeated here for the sake of brevity.

Similar toFIG.6,FIG.8illustrates specific structural diagram800of components in video reconstructor420according to an embodiment of the present disclosure. As shown inFIG.8, a schematic effect diagram of channel converter428and a schematic model diagram of image feature extractor429are added inFIG.8compared withFIG.6. Each of the above components may be implemented by means of a neural network model. The present disclosure does not limit the specific implementation of each model, and various known and future-developed neural network models may be used based on the needs of a processing task to be implemented.

An effect diagram of channel converter428is illustrated inFIG.8. As can be seen, channel converter428in video reconstructor420uses the acquired reference image features fRto process the foreground video features fVto convert the foreground video features fVto the image channel where reference image770is located. The operation manner of each of the components inFIG.8can be understood with reference to the description above and will not be repeated here for the sake of brevity.

In one embodiment, the video reconstruction method according to the present disclosure may be performed by a video reconstruction model, and as described above, the video reconstruction model may include components such as video feature extractor422, audio feature extractor421, and foreground and background information fuser426. The video reconstruction model may be obtained through training. The way in which the video reconstruction model is trained will be illustrated below in conjunction withFIG.9.

It will be understood that training method900inFIG.9may be performed in computing device126or may also be performed on other computing devices such as a server, which is not limited in the present disclosure.

As shown inFIG.9, at block902, feature extraction is performed on a plurality of sample image frames in a sample video segment via the video feature extractor to acquire sample image features. The sample video segment may be obtained after processing an original sample video segment for the training of the video reconstruction model. In one embodiment, the original sample video segment may be processed using a mask map for a target region (e.g., the mouth region) to acquire the sample video segment. For example, the original sample video segment may be processed using the mask map for the mouth region to obtain a mask-processed image frame of the mouth of the target person in the sample video segment.

At block904, feature extraction is performed on a sample audio segment corresponding to the sample video segment via the audio feature extractor to acquire sample voice features.

At block906, the video reconstruction model may receive a plurality of sample mask maps corresponding to the plurality of sample image frames, respectively. The present disclosure does not limit the specific manner in which the mask map is acquired.

At block908, a training video segment is generated via the foreground and background information fuser based on the sample image features, the sample voice features, and the plurality of sample mask maps.

At block910, the video reconstruction model may be trained based on the training video segment and the sample voice features. For example, parameters in video feature extractor422, audio feature extractor421, and foreground and background information fuser426may be adjusted to obtain the video reconstruction model.

A specific implementation for training a video reconstruction model (e.g., a video reconstructor) will be described below in conjunction withFIG.10, in particular for the way of acquiring a loss function when training the video reconstruction model.

FIG.10is similar to the architecture diagram inFIG.5, and since it is the architecture employed in the training phase, a loss function determination module is added inFIG.10compared withFIG.5. Specifically, inFIG.10, during the training of video reconstructor420, training architecture1000may include audio-video projector1010, synchronization loss determiner1020, video quality determiner1030, optical flow loss determiner1040, and target part feature loss determiner1050. InFIG.10, a plurality of sample image frames1062iare received by the video reconstructor for use in training.

Each of the above loss function determiners may be a pre-designed and trained model for determining the loss function for the video reconstruction model to realize adjustment of parameters of the video reconstruction model.

During the training of the video reconstruction model, optical flow loss determiner1040may acquire first optical flow information WXbetween a plurality of training image frames1090in a training video segment. Optical flow loss determiner1040may also acquire second optical flow information WYbetween a plurality of original image frames in the original sample video segment that correspond to the plurality of training image frames1082i, and determine the optical flow movement loss Ltembased on the first optical flow information WXand the second optical flow information WY. For example, optical flow loss determiner1040may determine the optical flow movement loss Ltembased on Equation 4:

where N is the size of the selected sample batch.

Further, synchronization loss determiner1020may determine a synchronization degree loss based on a synchronization degree between a training image segment and a sample audio segment. For example, synchronization loss determiner1020may calculate the synchronization degree loss based on Equation 5 below. In one embodiment, synchronization loss determiner1020is used to determine whether the training image fragment and the sample audio fragment are fragments corresponding to each other, and synchronization loss determiner1020may include a trained classifier. The synchronization degree loss Lsynis given by:

where G(θ, A, V, R) indicates the parameters that are adjustable in the audio feature extractor, the video feature extractor, the foreground information fuser, and the foreground and background information fuser.

After obtaining the aforementioned synchronization degree loss Lsynand the optical flow movement loss Ltem, the video reconstruction model may be trained based on a weighted sum of the synchronization degree loss Lsynand optical flow movement loss Ltem.

In one embodiment, other loss functions may also be acquired to train the video reconstruction model. In one embodiment, a loss function characterizing the similarity between sample voice features and image features may be determined. For example, audio-video projector1010may calculate the similarity between the sample voice features and the image features and denote the above similarity in the form of a matrix, and each element of the matrix denotes the similarity between the voice feature and image feature in the corresponding row and column. In one embodiment, audio-video projector1010may receive sample audio features extracted by audio feature extractor421and sample video features extracted by video feature extractor1070, first project the audio features and the video features into the Euclidean space, and then calculate the similarity between the audio features and the video features using the following Equation 6. Specifically, Equation 6 is as follows:

where i, j, and k denote the indexes of the sample batch, respectively, and i is a temperature parameter. The similarity Lsimbetween the audio features and the video features may be acquired by the above Equation 6.

It can be understood that compared with the prediction phase inFIG.5, video feature extractor1070is used in the training architecture in the phase of training the video reconstructor. This video feature extractor1070may be the same as or be different from video feature extractor422, which is not limited in the present disclosure.

In one embodiment, video quality determiner1030may calculate the video quality loss Lvis. For example, Lvismay be calculated using the following Equation 7:

where G(θ, A, V, R) indicates the parameters that are adjustable in the audio feature extractor, the video feature extractor, the foreground information fuser, and the foreground and background information fuser.

In addition, target part feature loss determiner1050may determine the target part feature loss. This loss is concerned with the effect of reconstruction of, for example, facial textures, so the target part feature loss may be determined according to the reconstructed training video image features and the original video image frames. For example, the target part feature loss may be determined according to first resolution map1080in the plurality of training image frames with respect to the target part and a second resolution map in the plurality of original image frames with respect to the target part. In one embodiment, for an audio-video synchronized video, the mouth in the face may be determined as the target part, and the feature loss for the mouth may be calculated by target part feature loss determiner1050. In one embodiment, this feature loss may be determined according to Equation 8:

where SXiis the first resolution map in the plurality of training image frames with respect to the target part, and yiis the second resolution map for the ith sample image.

After the above loss function is calculated, it may be weighted. For example, as shown in Equation 9:
L=λsimLsim+λsynLsyn+λvisLvis+λtemLtem+λfaceLface(Equation 9)

By using the loss function L obtained above, the video reconstruction model can be trained. For example, the parameters in the video feature extractor, the audio feature extractor, the foreground information fuser, and the foreground and background information fuser are adjusted to obtain the trained video reconstruction model.

The architecture inFIG.10is only an example, and for a video reconstructor that can perform style conversion based on a reference image, a similar training approach and a similar loss function can be adopted so as to train a video reconstructor as shown inFIG.7. This will not be repeated here for the sake of brevity.

FIG.11illustrates a schematic block diagram of example device1100that may be used to implement embodiments of the present disclosure. Computing device126inFIG.1may be implemented using device1100. As shown in the figure, device1100includes central processing unit (CPU)1101that can perform various appropriate actions and processing according to computer program instructions stored in read-only memory (ROM)1102or loaded from storage unit1108into random access memory (RAM)1103. Various programs and data required for the operation of device1100may also be stored in RAM1103. CPU1101, ROM1102, and RAM1103are connected to each other through bus1104. Input/output (I/O) interface1105is also connected to bus1104.

Multiple components in device1100are connected to I/O interface1105, including: input unit1106, such as a keyboard and a mouse; output unit1107, such as various types of displays and speakers; storage unit1108, such as a magnetic disk and an optical disc; and communication unit1109, such as a network card, a modem, and a wireless communication transceiver. Communication unit1109allows device1100to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The various processes and processing described above, for example, method200, method300and/or method900, may be performed by CPU1101. For example, in some embodiments, method200, method300and/or method900, etc. may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as storage unit1108. In some embodiments, part or all of the computer program may be loaded and/or mounted to device1100via ROM1102and/or communication unit1109. When the computer program is loaded into RAM1103and executed by CPU1101, one or more actions of method200, method300and/or method900, etc. described above may be performed.

Embodiments of the present disclosure include a method, an apparatus, a system, and/or a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for performing various aspects of the present disclosure are loaded.

Various illustrative embodiments of the present disclosure have been described above. The above description is illustrative, rather than exhaustive, and is not limited to the disclosed various embodiments. Numerous modifications and alterations will be apparent to persons of ordinary skill in the art without departing from the scope and spirit of the illustrated embodiments. The selection of terms as used herein is intended to best explain the principles and practical applications of the various embodiments and their associated technical improvements, so as to enable persons of ordinary skill in the art to understand the embodiments disclosed herein.