Patent Description:
Videos of natural scenes observe a complicated set of phenomena; objects deform and move quickly, occlude and disocclude each other, scene lighting changes, and cameras move. Parametric models of video appearance are often too simple to accurately model, interpolate, or extrapolate video. Nonetheless, video interpolation, i.e., synthesizing video frames between existing ones, is a common process in video and film production. A related problem is video extrapolation; predicting the future by synthesizing future video frames.

A traditional solution to these problems estimates optical flow between frames, and then interpolates or extrapolates along optical flow vectors. This approach works well when optical flow is accurate, but generates significant artifacts when it is not. A new approach uses generative convolutional neural networks (CNNs) to directly hallucinate RGB pixel values of synthesized video frames. While these techniques are promising, directly synthesizing RGB values is challenging, and the results are often blurry.

<NPL> discloses a deep 3D convolutional architecture trained end to end to perform voxel-level prediction, i.e., to output a variable at every voxel of the video.

<NPL> an unsupervised learning based approach to the ubiquitous computer vision problem of image matching.

One example aspect of the present disclosure is directed to a computer- implemented method for video frame synthesis. The method includes receiving, by one or more computing devices, a video. The method includes inputting, by the one or more computing devices, a first set of sequential frame data descriptive of the video into a machine-learned video frame synthesis model. The machine-learned video frame synthesis model includes at least one convolutional neural network having a voxel flow layer. The method includes receiving, by the one or more computing devices, one or more synthesized frames from the video. The one or more synthesized frames are output by the machine- learned video frame synthesis model. The method includes providing, by the one or more computing devices, information regarding the one or more synthesized frames.

The present disclosure provides systems and methods that leverage machine-learned models (e.g., neural networks) to provide video frame synthesis. In particular, the systems and methods of the present disclosure can include or otherwise leverage a machine-learned video frame synthesis model to allow for video frames to be synthesized from videos. For example, at least one in-between synthetic frame can be interpolated from at least two existing frames of an input video and/or at least one subsequent synthetic frame can be extrapolated from the at least two existing frames of the input video.

According to the invention, the video frame synthesis model includes a convolutional neural network that has a voxel flow layer. The convolutional neural network is a convolutional encoder-decoder. The voxel flow layer describes a per-pixel three-dimensional optical flow vector across both space and time in the input video. The video frame synthesis model synthesizes one or more synthesized video frames based on the voxel flow and one or more existing frames from the input video. In one example application, the newly synthesized frames can be used to form at least a portion of a video that depicts a scene in slow motion.

More particularly, the present disclosure describes a deep voxel flow (DVF) network. The DVF network can be an end-to-end fully differentiable network that can be used, for example, to perform video frame synthesis. In certain implementations, the only training data required are triplets of consecutive video frames in which two frames are provided as inputs and the rest frame is used as a reconstruction target. Thus, the systems and methods of the present disclosure use existing videos to train CNNs in an unsupervised fashion. That is,frames are dropped from existing videos, and a loss function is employed that measures similarity between generated pixels and ground-truth dropped frames.

According to another aspect not part of the invention, the systems and methods described herein are self-supervised and can learn to reconstruct frames by borrowing voxels from neighboring frames. In this manner, the output results can be both realistic and sharp and no pre-registration is needed for the input videos. As further technical benefits, ground truth optical flow is not needed as supervision and, due to the flexible motion modeling described herein, no pre-registration is required for the input videos.

In particular, in some implementations, pixels can be generated by interpolating pixel values from nearby frames. As one example, a video frame synthesis model can include a convolutional neural network that includes a voxel flow layer. For example, the voxel flow layer can be a per-pixel, 3D optical flow vector across space and time in the input video.

In some implementations, the final pixel can be generated by trilinear interpolation across the input video volume (which can typically be just two frames). Thus, for video interpolation, the final output pixel can be a blend of pixels from the previous and next frames. Such a voxel flow layer can be, in some respects, similar to an optical flow field. However, the voxel flow layer is only an intermediate layer, and, in some implementations, its correctness is never directly evaluated. Thus, the systems and methods of the present disclosure do not require optical flow supervision.

Thus, aspects of the present disclosure address the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation). In particular, the systems and methods of the present disclosure include a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which is referred to herein as deep voxel flow. The frame synthesis methods require no human supervision, and any existing video can be used as training data by dropping, and then learning to predict, existing frames. The techniques described herein are efficient, and can be applied at any video resolution.

As one example, the systems and methods of the present disclosure can be included or otherwise employed within the context of an application, a browser plug-in, an operating system, or in other contexts. Thus, in some implementations, the models of the present disclosure can be included in or otherwise stored and implemented by a user computing device such as a laptop, tablet, or smartphone. As yet another example, the models can be included in or otherwise stored and implemented by a server computing device that communicates with the user computing device according to a client-server relationship. For example, the models can be implemented by the server computing device as a portion of a web service.

<FIG> depicts a block diagram of an example computing system <NUM> that performs video frame synthesis according to example embodiments of the present disclosure. The system <NUM> includes a user computing device <NUM>, a server computing system <NUM>, and a training computing system <NUM> that are communicatively coupled over a network <NUM>.

The user computing device <NUM> can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

The user computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the user computing device <NUM> to perform operations.

The user computing device <NUM> can store or include one or more video frame synthesis models <NUM>. For example, the video frame synthesis models <NUM> can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other multi-layer non-linear models. Neural networks can include recurrent neural networks (e.g., long short-term memory recurrent neural networks), feed-forward neural networks, or other forms of neural networks. Example video frame synthesis models <NUM> are discussed with reference to <FIG>.

In some implementations, the one or more video frame synthesis models <NUM> can be received from the server computing system <NUM> over network <NUM>, stored in the user computing device memory <NUM>, and the used or otherwise implemented by the one or more processors <NUM>. In some implementations, the user computing device <NUM> can implement multiple parallel instances of a single video frame synthesis model <NUM>.

More particularly <FIG> illustrates an example pipeline of DVF in accordance with certain aspects of the present disclosure, where a convolutional encoder-decoder predicts the 3D voxel flow, and then a volume sampling layer synthesizes the desired frame, accordingly. Specifically, the convolutional encoder-decoder is denoted as H(X; Θ), where X is the input video and Θ are the network parameters. The output of H is a 3D voxel flow field F on a pre-defined grid G: <MAT>.

Then, the original input video X is warped according to F to get the final synthesized frame Ŷ: <MAT>
where Tx,y,t is the volume sampling function operating on spatio-temporal coordinates (x, y, t). As illustrated in <FIG>, DVF learns to synthesize target frame Y ∈ RH×W from the input video X ∈ RH×W×L, where H, W, L are the height, width and frame number of the input video. The target frame Y can be the in-between frame (interpolation), or the next frame (prediction) of the input video. DVF can adopt a fully-convolutional encoder-decoder architecture, which can contain five convolution layers and three deconvolution layers. Therefore, arbitrary-sized videos can be used as inputs for DVF. The network hyperparamters (e.g., the size of feature maps and the number of channels) are specified in <FIG>.

For the encoder section of the network, each processing unit can contain both convolution and max-pooling. In certain aspects of the present disclosure, the convolution kernel sizes are <NUM> × <NUM>, <NUM> × <NUM>, <NUM> × <NUM>, <NUM> × <NUM> , respectively. The bottleneck layer is also connected by convolution with kernel size <NUM> × <NUM>. For the decoder section, each processing unit can contains bilinear upsampling and convolution. In certain aspects of the present disclosure, the convolution kernel sizes are <NUM> × <NUM>, <NUM> × <NUM>, <NUM> × <NUM>, <NUM> × <NUM>, respectively. Skip connections can be added to better maintain spatial information between the corresponding convolution and deconvolution layers. Specifically, the corresponding deconvolution layers and convolution layers can be concatenated together before being fed forward.

The convolutional encoder-decoder can output 3D voxel flow field <MAT> on grid <MAT>. In certain implementations, offset coordinates can be utilized instead of absolute coordinates to define voxel flow because offsets can be more normalized and easier to learn. This 3D voxel flow generates each target voxel (xi, yi, ti) by copying from source voxel <MAT>: <MAT>.

However, due to occlusions and the ambiguity between different flows, not all target voxels will have a source voxel projected onto them by the predicted 3D voxel flow. Trilinear interpolation can be utilized to avoid holes and other unpleasant artifacts, which can be formulated as a volume sampling mechanism: <MAT>
where X(x, y, t) is the voxel value at location (x, y, z) of the input, and Ŷ(xi, yi, ti) is the output value for voxel i∈[<NUM>,···, H × W] atlocation (xi, yi, ti). Ω is the neighboring cube of the projected source voxel <MAT> and <MAT> is the trilinear resampling weight. Because of the existence of truncation function <MAT>, the target voxel Ŷ(xi, yi, ti) is actually the weighted average of neighboring cube of the projected source voxel <MAT>, which is depicted in <FIG>.

This 3D voxel flow can be understood as the joint modeling of 2D motion field and selection mask. It can be achieved by projecting F into <MAT> and <MAT>. And Fmotlon can be further categorized into Fforward and Fbackward depending on the t of interest.

For DVF training, rather than adopting the l<NUM> loss function as , the l<NUM> loss with spatial and temporal coherence regularizations can be exploited to reduce visual artifacts. The total variation (TV) regularization can be adopted for enforcing spatial coherence and the forward-backward flow consistency is adopted for enforcing temporal coherence. Moreover, these regularizers, which are imposed on the output of the network, can be incorporated into the back-propagation scheme. This can be formalized as minimizing the following objective function: <MAT>
where D is the training set of all frame triplets, N is its cardinality and Y is the target frame to be reconstructed. P∇FP<NUM> is the aforementioned total variation term and λ<NUM> is the corresponding regularization weight. PFforward - Fbackward P<NUM> is the forward-backward flow consistency term and λ<NUM> is the corresponding regularization weight. It generally states that the motion should be steady when tracing the interpolated frame to both source frames bidirectionally. To deal with the l<NUM> norm, the Charbonnier penalty function Φ(x) = (x<NUM> + ε<NUM>)<NUM>/<NUM> can be utilized for approximation. Here, the following can be empirically set: λ<NUM> = <NUM>, λ<NUM> = <NUM> and ε = <NUM>. Learning the network can be achieved via a gradient-based optimization method solver (e.g. ADAM solver) with learning rate of <NUM>, β<NUM> = <NUM>, β<NUM> = <NUM> and batch size of <NUM>. Though a TV regularizer is described herein, a more sophisticated edge-aware regularizer, such as fast bilateral solver, could also be utilized in connection with the present framework.

In order for the DVF to be an end-to-end fully differentiable system, the gradients can be defined with respect to 3D voxel flow <MAT> so that the reconstruction error can be backpropagated through volume sampling layer. The partial derivative of reconstruction loss <MAT> is: <MAT>
where Ω is the neighboring cube of the projected source voxel <MAT> and <MAT> is the error reassignment weight <MAT>. Similarly, <MAT> and <MAT> can be obtained. This can provide a sub-differentiable sampling mechanism, allowing loss gradients to flow back to the 3D voxel flow F. Such a sampling mechanism can be implemented very efficiently on GPU, by ignoring the sum over all input locations and instead just looking at the kernel support region for each output voxel, which is depicted in <FIG>.

As described herein, the gradients of reconstruction error can be obtained by only looking at the kernel support region for each output voxel. To handle large motion in videos, more long-range correspondences can be learned by our model. Therefore, in certain aspects of the present disclosure, multi-scale Deep Voxel Flow (multi-scale DVF) can be utilized so that both large motion and small motion can be encoded.

Specifically, a series of convolutional encoder-decoder HN,HN-<NUM>,···,H<NUM> can work on video frames from coarse scale sN to fine scale s<NUM>, respectively. In certain implementations, set sN = <NUM> × <NUM>,sN-<NUM> = <NUM> × <NUM>, ···, s<NUM> = <NUM> × <NUM>. In each scale k, the sub-network Hk predicts 3D voxel flow Fk at that resolution. Large motion can have a relatively small offset vector <MAT> in coarse scale sN. Thus, the sub-networks HN,···,H<NUM> in coarser scales sN,···,s<NUM> are capable to produce desired multi-scale voxel flows FN,···, F<NUM> even for large motions.

Such multi-scale voxel flows can be fused to the finest network H<NUM> to achieve a final result. The fusion can be conducted by concatenating multi-scale voxel flow Fk to its corresponding decoder layer, which has the same spatial resolution sk. The network architecture of multi-scale DVF is illustrated in <FIG> and can be formulated as: <MAT>.

Since each sub-network Hk is fully differentiable, the multi-scale DVF can also be trained end-to-end with reconstruction loss PYk -T(Xk, Fk)P<NUM> for each scale sk.

The framework described herein can be extended to multi-step prediction in either interpolation or extrapolation. For example, if the goal is to predict the next D frames when given the current L frames, the target Y becomes a 3D volume (Y∈RH×W×D) instead of a 2D frame (Y∈RH×W). Similar to Eqn. <NUM> described herein, each output voxel Ŷ = (xi, yi, ti), i∈[<NUM>,···,H × W × D] can be obtained by performing trilinear interpolation on the input video X. The spatio-temporal structure of Y can be well modeled because 3D voxel flow <MAT>, i∈[<NUM>,···, H × W × D] is predicted via convolution such that local correlations are maintained.

Additionally or alternatively, one or more video frame synthesis models <NUM> can be included in or otherwise stored and implemented by the server computing system <NUM> that communicates with the user computing device <NUM> according to a client-server relationship. For example, the video frame synthesis models <NUM> can be implemented by the server computing system <NUM> as a portion of a web service (e.g., a video editing service). Thus, one or more models <NUM> can be stored and implemented at the user computing device <NUM> and/or one or more models <NUM> can be stored and implemented at the server computing system <NUM>.

The user computing device <NUM> can also include one or more user input component <NUM> that receives user input. For example, the user input component <NUM> can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can enter a communication.

The server computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the server computing system <NUM> to perform operations.

As described above, the server computing system <NUM> can store or otherwise includes one or more machine-learned video frame synthesis models <NUM>. For example, the video frame synthesis models <NUM> can be or can otherwise include various machine-learned models such as neural networks (e.g., deep recurrent neural networks) or other multi-layer non-linear models. Example video frame synthesis models <NUM> are discussed with reference to <FIG>.

The server computing system <NUM> can train the communication assistance models <NUM> via interaction with the training computing system <NUM> that is communicatively coupled over the network <NUM>. The training computing system <NUM> can be separate from the server computing system <NUM> or can be a portion of the server computing system <NUM>.

The training computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the training computing system <NUM> to perform operations. In some implementations, the training computing system <NUM> includes or is otherwise implemented by one or more server computing devices.

The training computing system <NUM> can include a model trainer <NUM> that trains the machine-learned models <NUM> stored at the server computing system <NUM> using various training or learning techniques, such as, for example, backwards propagation of errors. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer <NUM> can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

In particular, the model trainer <NUM> can train a video frame synthesis model <NUM> based on a set of training data <NUM>. The training data <NUM> can include, for example, the public UCF-<NUM> dataset.

In some implementations, if the user has provided consent, the training examples can be provided by the user computing device <NUM> (e.g., based on communications previously provided by the user of the user computing device <NUM>). Thus, in such implementations, the model <NUM> provided to the user computing device <NUM> can be trained by the training computing system <NUM> on user-specific communication data received from the user computing device <NUM>. In some instances, this process can be referred to as personalizing the model.

The model trainer <NUM> includes computer logic utilized to provide desired functionality. The model trainer <NUM> can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer <NUM> includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer <NUM> includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.

<FIG> illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device <NUM> can include the model trainer <NUM> and the training dataset <NUM>. In such implementations, the video frame synthesis models <NUM> can be both trained and used locally at the user computing device <NUM>. In some of such implementations, the user computing device <NUM> can implement the model trainer <NUM> to personalize the video frame synthesis models <NUM> based on user-specific data.

<FIG> depicts a block diagram of an example computing device <NUM> that performs video editing according to example embodiments of the present disclosure. The computing device <NUM> can be a user computing device or a server computing device.

For example, each application can include a machine-learned communication assistance model. An example application includes a video editing application.

<FIG> depicts a block diagram of an example computing device <NUM> that performs communication assistance according to example embodiments of the present disclosure. The computing device <NUM> can be a user computing device or a server computing device.

An example application includes a video editing application. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

The central intelligence layer includes a number of machine-learned models. For example, as illustrated in <FIG>, a respective machine-learned model (e.g., a video frame synthesis model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single communication assistance model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device <NUM>.

<FIG> depicts a block diagram of an example video frame synthesis model <NUM> according to example embodiments of the present disclosure. In some implementations, the video frame synthesis model <NUM> is trained to receive a set of input data <NUM> descriptive of a videos and, as a result of receipt of the input data <NUM>, provide output data <NUM> that describes synthesized target frames of the input video. Thus, in some implementations, the video frame synthesis model <NUM> can include a voxel flow layer <NUM> that is operable to describe a per-pixel three-dimensional optical flow vector across both space and time in the input video. The video frame synthesis model <NUM> can synthesize the one or more synthesized video frames <NUM> based on the voxel flow and one or more existing frames from the input data <NUM>.

<FIG> depicts a flow chart diagram of an example method to perform communication assistance according to example embodiments of the present disclosure. Although <FIG> depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement.

At <NUM>, a computing system receives a video, such as a video provided by a user. As examples, the communication can be loaded into a user computing device via another device (e.g., video camera, storage device, etc.) or can be captured by a camera or video camera connected to the user computing device.

At <NUM>, the computing system inputs a set of video frame data into a machine-learned video frame synthesis model. For example, a user computing device can input the video frame data into a local machine-learned video frame synthesis model. As another alternative example, a user computing device can transmit the set of video frame data over a network to a server computing device and the server computing device can input the set of communication data into a machine-learned video frame synthesis model stored at the server computing device.

At <NUM>, the computing system receives one or more synthesized frames from the video. The one or more synthesized frames can have been output by the machine-learned video frame synthesis model. For example, the user computing device can receive the synthesized frames from a local model or from the server over the network.

At <NUM>, the computing system provides information regarding the one or more synthesized frames. For example, the information can be provided for display to a user.

Claim 1:
A computer-implemented method for video frame synthesis, the method comprising:
receiving, by one or more computing devices, a video;
inputting, by the one or more computing devices, a first set of sequential frames of the video into a machine-learned video frame synthesis model (<NUM>, <NUM>), wherein the machine-learned video frame synthesis model (<NUM>, <NUM>) comprises at least one convolutional neural network comprising a convolutional encoder-decoder network denoted as H(X; Θ), where X is the input video and Θ are network parameters, wherein the convolutional encoder-decoder network is trained to predict a voxel flow field F on a pre-defined grid G, the voxel flow field F comprising a per-pixel 3D optical flow vector across space and time, wherein <MAT>
receiving, by the one or more computing devices, one or more synthesized frames Ŷ from the video, the one or more synthesized frames Ŷ generated by the machine-learned video frame synthesis model (<NUM>, <NUM>) based on the first set of sequential frames and the predicted voxel flow field by sampling the voxel flow field F; and
providing, by the one or more computing devices (<NUM>), the one or more synthesized frames, wherein the convolutional neural network is trained in an unsupervised fashion by:
removing one or more frames from the received video;
providing a plurality of the remaining frames as the first set of sequential frames;
generating the one or more synthesized frames Ŷ to correspond to the one or more removed frames;
calculating a loss function to represent a similarity between the one or more synthesized frames and the one or more removed frames; and
adjusting one or more parameters of the convolutional neural network in order to minimize the calculated loss function, wherein:
the one or more synthesized frames Ŷ are generated by the machine-learned video frame synthesis model based on the first set of sequential frames and the predicted voxel flow field F by warping one or more frames of the received video according to the predicted voxel flow field using a volume sampling function T x,y,t operating on spatio-temporal coordinates (x, y, t), wherein <MAT>