TECHNIQUES FOR PROCESSING VIDEOS USING TEMPORALLY-CONSISTENT TRANSFORMER MODEL

Techniques are disclosed for enhancing videos using a machine learning model that is a temporally-consistent transformer model. The machine learning model processes blocks of frames of a video in which the temporally first input video frame of each block of frames is a temporally second to last output video frame of a previous block of frames. After the machine learning model is trained, blocks of video frames, or features extracted from the video frames, can be warped using an optical flow technique and transformed using a wavelet transform technique. The transformed video frames are concatenated along a channel dimension and input into the machine learning model that generates corresponding processed video frames.

BACKGROUND

Technical Field

Embodiments of the present disclosure relate generally to computer science and video processing and, more specifically, to techniques for processing videos using a temporally-consistent transformer model.

Description of the Related Art

The frames of a video can include various degradations, such as noise and blurriness. For example, a video captured by a camera can contain noise due to the mechanics of camera sensors. As another example, a video and/or frames thereof can be blurry when the video and/or frames are captured by a camera that is out of focus.

Transformer models are artificial neural networks. Conventional transformer models have been applied to some computer vision tasks, such as detecting objects within images and classifying those objects. However, when transformer models are used to process the frames of a video, the processed frames can have temporally inconsistent regions that vary from frame to frame, even when no such variations should occur. The temporally inconsistent regions are, as a general matter, readily noticeable as flickering and other artifacts that can greatly reduce the quality of processed videos.

As the foregoing illustrates, what is needed in the art are more effective techniques for processing videos using transformer models.

SUMMARY

One embodiment of the present disclosure sets forth a computer-implemented method for enhancing videos. The method includes processing a first plurality of video frames using a machine learning model to generate a first plurality of processed video frames. The method further includes processing a second plurality of video frames using the machine learning model to generate a second plurality of processed video frames. A temporally first video frame included in the second plurality of video frames is a temporally second to last video frame included in the first plurality of processed video frames.

Another embodiment of the present disclosure sets forth a computer-implemented method for training a machine learning model. The method includes adding a plurality of amounts of degradation to a set of video frames to generate a plurality of sets of degraded video frames. Each set of degraded video frames includes a different amount of degradation. The method further includes performing one or more operations to train the machine learning model based on the plurality of sets of degraded video frames. The one or more operations minimize a loss function that penalizes a difference between a temporally last frame of each plurality of processed video frames generated by the machine learning model and a temporally first frame of a subsequent plurality of processed video frames generated by the machine learning model.

Other embodiments of the present disclosure include, without limitation, one or more computer-readable media including instructions for performing one or more aspects of the disclosed techniques as well as one or more computing systems for performing one or more aspects of the disclosed techniques.

At least one technical advantage of the disclosed techniques relative to the prior art is that videos processed according to the disclosed techniques generally include fewer unwanted artifacts relative to videos processed using conventional transformer models. In particular, the disclosed techniques introduce fewer temporal inconsistencies into processed videos than conventional transformer models and substantially reduce flickering between adjacent frames of processed videos. These technical advantages represent one or more technological improvements over prior art approaches.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to one of skill in the art that embodiments of the present invention can be practiced without one or more of these specific details.

System Overview

FIG.1illustrates a system100configured to implement one or more aspects of the various embodiments. As shown, the system100includes a machine learning server110, a data store120, and a computing device140in communication over a network130, which can be a wide area network (WAN) such as the Internet, a local area network (LAN), or any other suitable network.

As shown, a model trainer116executes on a processor112of the machine learning server110and is stored in a system memory114of the machine learning server110. The processor112receives user input from input devices, such as a keyboard, a mouse, a joystick, a touchscreen, or a microphone. In operation, the processor112is the master processor of the machine learning server110, controlling and coordinating operations of other system components. In particular, the processor112can issue commands that control the operation of a graphics processing unit (GPU) (not shown) that incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. The GPU can deliver pixels to a display device that can be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like.

The system memory114of the machine learning server110stores content, such as software applications and data, for use by the processor112and the GPU. The system memory114can be any type of memory capable of storing data and software applications, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash ROM), or any suitable combination of the foregoing. In some embodiments, a storage (not shown) can supplement or replace the system memory114. The storage can include any number and type of external memories that are accessible to the processor112and/or the GPU. For example, and without limitation, the storage can include a Secure Digital Card, an external Flash memory, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

It will be appreciated that the machine learning server110shown herein is illustrative and that variations and modifications are possible. For example, the number of processors112, the number of GPUs, the number of system memories114, and the number of applications included in the system memory114can be modified as desired. Further, the connection topology between the various units inFIG.1can be modified as desired. In some embodiments, any combination of the processor112, the system memory114, and a GPU can be replaced with any type of virtual computing system, distributed computing system, or cloud computing environment, such as a public, private, or a hybrid cloud.

In some embodiments, the model trainer116is configured to train one or more machine learning models, including a TempFormer model150. The TempFormer model150is a temporally-consistent transformer model for video processing tasks. In some embodiments, the TempFormer model150can be a modification of the Vision Transformer architecture (ViT). Training data and/or trained machine learning models, including the TempFormer model150, can be stored in the data store120. In some embodiments, the data store120can include any storage device or devices, such as fixed disc drive(s), flash drive(s), optical storage, network attached storage (NAS), and/or a storage area-network (SAN). Although shown as accessible over the network130, in some embodiments the machine learning server110can include the data store120.

Once trained, the TempFormer model150can be deployed to perform any technically feasible video processing tasks for which the TempFormer model150was trained. In some embodiments, the TempFormer model150can be deployed to perform video enhancement tasks, such as video denoising, deblurring, interpolation, etc. Illustratively, a video processing application146that utilizes the TempFormer model150is stored in a system memory144, and executes on a processor142, of the computing device140. In some embodiments, components of the computing device140, including the system memory144and the processor142can be similar to corresponding components of the machine learning server110.

It will be appreciated that the system100shown herein is illustrative and that variations and modifications are possible. For example, the number of machine learning servers and computing devices can be modified as desired. Further, the functionality included in any of the applications can be divided across any number of applications or other software that are stored and executed via any number of computing systems that are located in any number of physical locations.

Processing Videos Using TempFormer Model

FIG.2illustrates in greater detail the TempFormer model150ofFIG.1, according to various embodiments. As shown, the TempFormer model150includes a wavelet transform module210, a concatenation module212, a shallow feature extraction module214, a deep feature extraction module218, an image reconstruction module222, and an inverse wavelet transform module224. The TempFormer model150receives as inputs a number of consecutive video frames2021-n(referred to herein collectively as input frames202and individually as an input frame202) of a video and outputs a number of consecutive video frames2301-p(referred to herein collectively as output frames230and individually as an output frame230). The TempFormer model150can receive any technically feasible number (e.g., five) of input frames202as input and output any technically feasible number (e.g., three) of output frames230. In some embodiments, the input frames202can be input into the TempFormer model150as RGB (red, green, blue) channels.

In some embodiments, the TempFormer model150is a one-stage model that performs spatial and temporal processing simultaneously. As described, the TempFormer model150can take a number of consecutive input frames202, such as 2×m+1 frames, as inputs and output a number of consecutive output frames230, such as 2×n+1 frames. More formally, processing of video frames by the TempFormer model150can be expressed in the following form:

where Ĩ represents a frame from a temporal window of frames Blockt, which includes a set of contiguous frames and is also referred to herein as a “block” of frames, ϕ is the TempFormer model150, and Î represents a processed frame of the temporal window of frames Blockt. Although the example of m=2 and n=1 is used in some of the formulas and figures herein for illustrative purposes, n and n can be any positive integers in embodiments. To introduce communications between neighboring temporal windows of frames, m can be set to be strictly larger than n so that neighboring temporal windows share multiple common input frames. Within a temporal window of frames Blockt, input frames can exchange information in spatial-temporal transformer (STTB) blocks2201-m(referred to herein collectively as STTB blocks220and individually as a STTB block220) so that the output frames230that are output by the TempFormer model150are intrinsically temporally stable. For two neighboring temporal windows of frames, slight discrepancies can exist in the output frames230because neighboring temporal windows share a limited number of frames in common. More specifically, flickering artifacts can exist between the temporally last output frame of the temporal window of frames Blockt, namely Înt, and the temporally first output frame of the next temporal window of frames Blockt+1, namely Î−nt+1. Such flickering can be reduced or eliminated using (1) a recurrent architecture in which the temporal window of frames Blockt+1gets one processed reference frame from the previous temporal window of frames Blockt, and (2) a temporal consistency loss term, discussed in greater detail below in conjunction withFIG.6.

In operation, the wavelet transform module210decomposes each of the input frames202into wavelet sub-bands. Such a decomposition reduces the spatial resolution for computational efficiency purposes. In addition, the reduced spatial resolution enables much longer features, which can improve the performance of the TempFormer model150. In some embodiments, the wavelet transform module210halves the resolution of the input frames202to solve the problem that the size of an attention map SoftMax(QKT/√{square root over (D)}+bias) in the TempFormer model150is w2×w2, which can be a bottleneck that affects the computational efficiency of the TempFormer model150. The wavelet transform module210alleviates such a bottleneck. Although described herein primarily with respect to a wavelet transform, other types of decompositions, such as pixel shuffle, can be used in some embodiments. In some embodiments, the input frames202can also be warped using an optical flow that is calculated from the input frames202prior to performing a decomposition on the warped input frames202. Warping the input frames202using the optical flow can improve the signal-to-noise ratio of the TempFormer model150relative to conventional transformer modules, which oftentimes produce pixel misalignments in the temporal domain that appear as ghosting artifacts and blurriness. In some other embodiments, features extracted from the input frames202can be warped rather than the input frames202, themselves.

The concatenation module212concatenates the wavelet sub-bands that are output by the wavelet transform module210along the channel dimension. The channel dimension includes features from different frames. Concatenating along the channel dimension changes the input so that a transformer, shown as STTB blocks220, fuses features spatially and temporally, as discussed in greater detail below. The spatial and temporal fusing of features can reduce or eliminate temporal inconsistencies in the output frames230that are output by the TempFormer model150.

The shallow feature extraction module214includes a three-dimensional (3D) convolution layer that converts frequency channels in the concatenated sub-bands output by the concatenation module212into shallow features. That is, the shallow feature extraction module214changes the frequency of the sub-bands into features in feature space. The 3D convolution performed by the shallow feature extraction module214can also improve temporal fusion by the STTB blocks220.

The deep feature extraction module218includes a number of STTB blocks220. The STTB blocks220provide attention mechanisms that fuse features at different spatial and temporal positions of the input frames202. In particular, the STTB blocks220spatially and temporally mix the features of tokens to integrate the information of the input frames202. Each token is a patch (e.g., a 16×16 pixel patch) at a distinct position within the input frames202. As discussed in greater detail below in conjunction withFIG.4, the STTB blocks220project features of each token to a query key and value, which acts as a feature mixer. Because the wavelet sub-bands of the input frames202were concatenated along the feature channel, the features also include temporal information. As a result, the feature mixing will also produce temporal mixing.

Following the STTB blocks220is the image reconstruction module222, which includes another 3D convolution layer that transforms the features back into frequency space. Then, the inverse wavelet transform module224converts the sub-bands that are output by the 3D convolution layer into the output frames230that have the original resolution of the input frames202.

FIG.3illustrates in greater detail the STTB block2201ofFIG.2, according to various embodiments. The other STTB blocks220ofFIG.2include similar features as the STTB block2201. As shown, the STTB block2201includes joint spatial-temporal mixer (JSTM) blocks3021-6(referred to herein collectively as JSTM blocks302and individually as a JSTM block302), a patch unembedding module304, a 3D convolution layer306, a feature weights generator308, and a patch embedding module310. Although six JSTM blocks302are shown for illustrative purposes, a STTB block can include any technically feasible number of JSTM blocks in some embodiments (i.e., more than six JSTM blocks or less than six JSTM blocks).

The JSTM blocks302are attention layers that perform spatial and temporal mixing jointly. The spatial and temporal mixing fuses features from different frames spatially and temporally. In some embodiments, spatial and temporal attention is also learned simultaneously. Illustratively, the STTB block2201includes a sequence of JSTM blocks302followed by the patch unembedding304module, the 3D convolution layer306, and the patch embedding module310. The patch unembedding module304and the patch embedding module310are used to combine patches output by the JSTM block3026into an image and split an image generated using the 3D convolution layer306and weights from the feature weights generator308into patches, respectively. The 3D convolution layer306performs further feature extraction on the image that is output by the patch unembedding module304to extract deep features. The 3D convolution layer306is used, rather than a 2D convolution layer, to enhance the interaction between neighboring frames and reduce temporal inconsistency in the output frames230. Because all of the input frames202are concatenated along the channel dimension and there is a relatively large amount of temporal mixing in the TempFormer model150, each output frame230can include some patterns from neighboring output frames230. The feature weights generator308is an adaptive pooling layer that generates weights for each feature that are used to alleviate ghosting artifacts.

FIG.4illustrates in greater detail the JSTM block3021ofFIG.3, according to various embodiments. The other JSTM blocks302ofFIG.2include similar features as the JSTM block3021. As shown, the JSTM block3021includes a layer norm module402, an attention layer404, another layer norm module408, and a multilayer perceptron (MLP) layer410. As described, the JSTM block3021is an attention layer that performs spatial and temporal mixing jointly. Since the channel dimension includes features from different frames, the input images are divided into non-overlapping spatial windows with size w×w in some embodiments. An attention layer of a vision transformer can be interpreted as a spatial tokens mixer in which the weights for each token are content-dependent. In addition to spatial mixing, the JSTM block3021can also mix channels. Assume that in feature space, the channel length of each frame is c. In such cases, the temporal mixing is performed when generating the queries (Q), keys (K), and values (V) from the features of the tokens, which can be expressed as the following formula:

where X∈w2×5care the features of all frames before mixing, and {PQ, PK, PV}∈5c×5dare linear projections that project the features into {Q,K,V}∈w2×5d. Because all input frames202are concatenated along the channel dimension, each {qi,j, ki,j, vi,j}∈5dintegrates the features of all input frames202at spatial position (i,j), namely xi,j∈5c. The concatenation process can be described as:

where n∈{−2, −1, 0, 1, 2} and {qi,jIn,ki,jIn,vi,jIn}∈cis the query, key, and value of the token in frame n with spatial position (i,j).

Mixing only along the channel dimension is not enough to integrate temporal information because motions in the frames of a video introduce offsets between pairs of pixels in different frames. The following spatial mixing can place all spatial and temporal information to a reference token yi,jIn, at a spatial location (i′,j′) of frame In(•,•:V×V→)

For example, the query (qi′,j′I0), key (ki′,j′I0), and value (vi′,j′I0), of the reference token xi′,j′I0, integrate the features of all frames at position (i′,j′). In like manner, the query (qi,jI0), key (ki,jI0), and value (vi,jI0) integrate the features of all frames at position (i,j). The attention between xi′,j′I0, and xi,jI0fuses the features of all frames at both positions (i′,j′) and (i,j), which results in a spatio-temporal fusion.

The foregoing formulas written in matrix form is the computation function of the attention mechanism in a vision transformer:

where D is the length of the features of each token. For example, in some embodiments, the length D can be D=5d, where d is the length of the features for each video frame. Assuming that five frame are taken as input and concatenated along the feature dimension, the concatenated tensor has the feature length of D=5d (number of frames times the length of features for each frame). In equation (6), bias is a trainable relative position bias, which can increase the capacity of the TempFormer model150.

The MLP410layer in JSTM block3021also acts as a temporal mixer. Before feeding tokens to a next STTB block220, the 3D convolution layer306and the feature weights generator308, which is an adaptive pooling layer, are used to extract additional features. The end-to-end connection of the STTB blocks220places multiple spatial and temporal mixers together. The entire process can be expressed as:

where Weights is used to assign different weights to the features of each frame, and ⊙ represents element-wise multiplication.

FIG.5illustrates in greater detail the feature weights generator308ofFIG.3, according to various embodiments. As shown, the feature weights generator308includes an adaptive pooling layer502, a 3D convolution layer504, and a sigmoid layer506. The feature weights generator308generates weights that are used to weight each feature output by the 3D convolution layer306. Weighting of the features can alleviate ghosting artifacts in the processed output frames230relative to features that are not weighted.

FIG.6illustrates the combination of a recurrent architecture of the TempFormer model150and use of an overlap loss term during training, according to various embodiments. As shown, the recurrent architecture uses a temporally second to last frame614from a sequence of consecutive output frames612,614, and616of a video that the TempFormer model150generates after processing a sequence of consecutive input frames602,604,606,608, and610of a temporal window of frames600from the video as the temporally first frame652of a sequence of consecutive input frames652,654,656,658, and660of a subsequent temporal window of frames650from the video. The TempFormer model150then processes the input frames652,654,656,658, and660to generate a subsequent sequence of consecutive output frames662,664, and666.

In some embodiments, an overlap loss term640is used during training of the TempFormer model150to penalize a difference between a temporally last frame (e.g., output frame616) that the TempFormer model150generates for a temporal window (e.g., temporal window600) and the temporally first output frame (e.g, output frame662) that the TempFormer model150generates for a subsequent temporal window. After introducing the overlap loss term640, the number of overlapping frames between the frames612,614, and616that are output by the TempFormer model150for the temporal window of frames600and the input frames652,654,656,658, and660for the subsequent window of frames650is two, so the temporally second to last frame614corresponds to, and is used as, the temporally first input frame652. Illustratively, the TempFormer model150can be trained using a loss function that combines the overlap loss term640with losses620and670between the sequences of output frames612,614, and616and662,664, and666and corresponding sequences of ground truth reference frames630,632,634, and680,682, and684, respectively.

More formally, even when neighboring temporal windows of frames share 2(m−n) input frames, degradations in the remaining 2n+1 input frames vary in each temporal window, which is the root cause of temporal incoherency across temporal windows. In some embodiments, to solve the temporal incoherency problem, the recurrent architecture shown inFIG.6is used to enforce a connection between neighboring temporal windows. As described, the connection between neighboring temporal windows uses the temporally first input frame of a temporal window of frames Blockt+1as the temporally last output frame of a previous temporal window of frames Blockt, which can be expressed as:

The recurrent architecture spreads the information from all frames of a current temporal window of frames Blocktto a next temporal window of frames Blockt+1by propagating the temporally second to last processed frame of the current temporal window Blocktas the temporally first input frame of the next temporal window Blockt+1. The substitution of the temporally first input frame of the next temporal window with the temporally second to last processed frame from the previous temporal window provides prior knowledge to each temporal window of frames, thereby enhancing the connection between neighboring temporal blocks and achieving better temporal consistency. However, reconstruction errors can also propagate from one temporal window of frames to a next temporal window of frames. In addition, across temporal windows of frames, dynamic content and static content with periodical occlusion (e.g., when the legs of a dancer sweep over) can still be temporally inconsistent.

To solve the problem of temporal inconsistency of dynamic content across temporal windows of frames, the stride used to divide a video sequence can be modified so that neighboring temporal windows of frames share 2(m−n)+1 common input frames. In addition, the overlap loss term640is used during training of the TempFormer model150to enforce temporal consistency between the temporally last processed frame of a temporal window of frames Blocktand the temporally first processed frame of a next temporal window of frames Blockt+1. The overlap loss term640can be expressed as:

whereoverlaptis the l1 loss between the temporally last output frame of the temporal window Blocktand the temporally first output frame of the next temporal window Blockt+1. The total losstotalincludes two parts: (1) the first partblocktis the loss between the processed frames Î and the corresponding reference frames I for each temporal window of frames, shown as loss terms620and670; and (2) the second part is the overlap lossoverlaptof equation (12). In some embodiments, a hyper parameter α can be used to balance the spatial and temporal loss, as shown in the following formulas:

where T is the index of the temporal windows in the video sequence. In example ofFIG.6, the recurrent strategy after introducing the overlap loss term640overlaptcan be formulated as Blockt+1: {Î−1t+1, Î0t+1, Î1t+1}=ϕ{Ĩ0t, Ĩ−1t+1, Ĩ0t+1, Ĩ1t+1, Ĩ2t+1}.

In some embodiments, training of the TempFormer model150includes a spatial-temporal video processing phase and a temporal coherency enhancement phase. In the spatial-temporal video processing phase, one temporal window of frames is processed in each training step. During the temporal coherency enhancement phase, two neighboring temporal windows of frames (Block0and Block1) are loaded for processing in each training step. For the first temporal window, the first noisy input frame is substituted with the corresponding ground truth frame to simulate the recurrent architecture, described above. Then, the temporally first input frame (Î−2twhen five neighboring frames are included in each temporal window that is input into the TempFormer model150) of the second temporal window is replaced with the temporally second to last output frame of the first temporal window (Î00when three neighboring frames are output by the TempFormer model150), and the overlap loss of equation (12) is added to the common output frames of the first and second temporal windows (Î10and Î−11when three neighboring frames are output by the TempFormer model150).

FIG.7Aillustrates an exemplar residual700between the temporally first processed frame of a block of frames and the temporally last processed frame of a previous block of frames before temporal consistency enhancement, according to the various embodiments. As shown, the exemplar residual700indicates significant temporal inconsistency between the temporally first processed frame of the block and the temporally last processed frame of the previous block. In particular, the temporally first frame of the block and the temporally last processed frame of the previous block have inconsistent regions that vary between those frames, when such variations should not occur when both frames depict similar subject matter.

FIG.7Billustrates an exemplar residual702between the temporally first processed frame of a block of frames and the temporally last processed frame of a previous block of frames after temporal consistency enhancement using the recurrent architecture and overlap described above in conjunction withFIG.6, according to the various embodiments. As shown, the exemplar residual702indicates that there is less temporal inconsistency between the temporally first processed frame of a block and the temporally last processed frame of a previous block than indicated by the residual700, described above in conjunction withFIG.7A. Not only is the temporal consistency between neighboring temporal windows of frames improved, the coherency between neighboring frames inside each temporal window is also improved. Compared with a recurrent architecture without the overlap loss termoverlapt, use of the recurrent architecture along with the overlap loss termoverlaptgenerates processed frames in which dynamic content and static content with periodical occlusions can have substantially the same temporal stability as static content in a video sequence being processed.

FIG.8sets forth a flow diagram of method steps for training a video processing machine learning model, according to various other embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-6, persons of ordinary skill in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present disclosure.

As shown, a method800begins at step802, where the model trainer116applies different amounts of degradation to reference video frames to generate degraded video frames. In some embodiments, the model trainer116can apply different amounts of any technically feasible type of degradation. For example, the model trainer116could apply Gaussian noise to reference video frames to generate the first training video frames having different amounts of noise. In some embodiments, the model trainer116can add different amounts of any technically feasible type of blur (e.g., Gaussian blur) to reference video frames to generate blurred video frames. In such cases, the model trainer116can also add noise to the blurred video frames to generate the second training video frames having different amounts of blurriness and noise.

At step804, the model trainer116trains a video processing machine learning model using the degraded video frames generated at step802and the reference video frames. In some embodiments, the video processing machine learning model is trained to take as inputs a block of consecutive frames of a video, with a temporally first frame of the block being a temporally second to last processed frame from a previous block of consecutive frames, if any, and to output consecutive processed frames. In some embodiments, the video processing machine learning model can be the TempFormer model150, described above in conjunction withFIGS.1-5. In some embodiments, the video processing machine learning model can be trained to minimize the loss function of equation (14). In some embodiments, training of the video processing machine learning model includes the spatial-temporal video processing phase and the temporal coherency enhancement phase, described above in conjunction withFIG.6.

FIG.9sets forth a flow diagram of method steps for processing a video, according to various other embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-6, persons of ordinary skill in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present disclosure.

As shown, a method900begins at step902, where the video processing application146receives a video as input. For example, the video could be captured using a camera or rendered via a Monte Carlo rendering technique, and the video could include degradation such as noise, blurriness, and/or the like that the video processing application146can reduce or eliminate.

At step904, the video processing application146selects a predefined number of consecutive video frames from the video. For example, the video processing application146could select five frames, or any suitable predefined number of frames larger than one, that a video processing machine learning model (e.g., TempFormer model150) is trained to take as inputs. Experience has shown that using five input frames achieves a relatively desirable balance between denoising quality and computational efficiency. Using more input frames generally consumes more computational resources, resulting in longer processing times. However, any suitable number of input frames larger than one can be used in some embodiments, such as a number that is chosen based on the available computational resources and the video processing task being performed. It should be noted that changing the number of input frames will require retraining the video processing machine learning model.

At step906, the video processing application146processes (1) the selected frames and (2) a temporally second to last previously processed frame, if any, as a temporally first frame, using the video processing machine learning model to generate one or more processed frames. In some embodiments, the video processing machine learning model is the TempFormer model150, described above in conjunction withFIGS.1-5. When multiple processed frames are generated by the video processing machine learning model, the processed frames are consecutive frames.

At step908, if there are additional frames to process, then the method900returns to step904, where the video processing application146selects another predefined number of frames from the video for processing. If there are no additional frames to process, the method900ends.

In sum, techniques are disclosed for enhancing videos using a TempFormer machine learning model that is a temporally-consistent transformer model. In some embodiments, the TempFormer model includes a recurrent architecture that processes blocks of frames of a video in which the temporally first input video frame of each block of video frames is a temporally second to last output video frame of a previous block of video frames. The TempFormer model is trained using a loss function to minimize a difference between the temporally last video frame of a block of processed video frames output by the TempFormer model and the temporally first video frame of a subsequent block of processed video frames output by the TempFormer model. The TempFormer model can be trained to perform various low-level video processing tasks, such as video denoising, deblurring, interpolation, etc. After training, blocks of video frames, or features extracted from the video frames, can be warped using an optical flow technique and transformed using a wavelet or other transform technique. The transformed video frames are concatenated along a channel dimension and input into the TempFormer model that generates corresponding processed video frames.

At least one technical advantage of the disclosed techniques relative to the prior art is that videos processed according to the disclosed techniques generally include fewer unwanted artifacts relative to videos processed using conventional transformer models. In particular, the disclosed techniques introduce fewer temporal inconsistencies into processed videos than conventional transformer models and substantially reduce flickering between adjacent frames of processed videos. These technical advantages represent one or more technological improvements over prior art approaches.1. In some embodiments, a computer-implemented method for enhancing videos comprises processing a first plurality of video frames using a machine learning model to generate a first plurality of processed video frames, and processing a second plurality of video frames using the machine learning model to generate a second plurality of processed video frames, wherein a temporally first video frame included in the second plurality of video frames is a temporally second to last video frame included in the first plurality of processed video frames.2. The computer-implemented method of clause 1, further comprising performing one or more operations to train the machine learning model using a loss function that penalizes a difference between a temporally last frame of a plurality of processed training video frames generated by the machine learning model and a temporally first frame of a subsequent plurality of processed training video frames generated by the machine learning model.3. The computer-implemented method of clauses 1 or 2, wherein processing the first plurality of video frames using the machine learning model comprises performing one or more transform operations on each video frame included in the first plurality of video frames to generate a plurality of transformed video frames, concatenating the plurality of transformed video frames along a channel dimension to generate a concatenated set of transformed video frames, and inputting the concatenated set of transformed video frames into the machine learning model.4. The computer-implemented method of any of clauses 1-3, wherein the one or more transform operations include at least one of (1) one or more wavelet transform operations or (2) one or more pixel shuffle operations.5. The computer-implemented method of any of clauses 1-4, further comprising generating an optical flow based on the first plurality of video frames, and warping either the first plurality of video frames or features extracted from the first plurality of video frames based on the optical flow.6. The computer-implemented method of any of clauses 1-5, further comprising adding a plurality of amounts of degradation to a set of video frames to generate a plurality of sets of degraded video frames, wherein each set of degraded video frames includes a different amount of the degradation, and training the machine learning model based on the plurality of sets of degraded video frames.7. The computer-implemented method of any of clauses 1-6, wherein the degradation comprises at least one of noise or blur.8. The computer-implemented method of any of clauses 1-7, wherein the machine learning model comprises a transformer model.9. The computer-implemented method of any of clauses 1-8, wherein the machine learning model comprises one or more three-dimensional (3D) convolution layers.10. In some embodiments, one or more non-transitory computer-readable storage media include instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for enhancing videos, the steps comprising processing a first plurality of video frames using a machine learning model to generate a first plurality of processed video frames, and processing a second plurality of video frames using the machine learning model to generate a second plurality of processed video frames, wherein a temporally first video frame included in the second plurality of video frames is a temporally second to last video frame included in the first plurality of processed video frames.11. The one or more non-transitory computer-readable storage media of clause 10, wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform the step of performing one or more operations to train the machine learning model using a loss function that penalizes a difference between a temporally last frame of a plurality of processed training video frames generated by the machine learning model and a temporally first frame of a subsequent plurality of processed training video frames generated by the machine learning model.12. The one or more non-transitory computer-readable storage media of clauses 10 or 11, wherein processing the first plurality of video frames using the machine learning model comprises performing one or more transform operations on each video frame included in the first plurality of video frames to generate a plurality of transformed video frames, concatenating the plurality of transformed video frames along a channel dimension to generate a concatenated set of transformed video frames, and inputting the concatenated set of transformed video frames into the machine learning model.13. The one or more non-transitory computer-readable storage media of any of clauses 10-12, wherein the one or more transform operations include at least one of one or more wavelet transform operations or one or more pixel shuffle operations.14. The one or more non-transitory computer-readable storage media of any of clauses 10-13, wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform the steps of generating an optical flow based on the first plurality of video frames, and warping either the first plurality of video frames or features extracted from the first plurality of video frames based on the optical flow.15. The one or more non-transitory computer-readable storage media of any of clauses 10-14, wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform the steps of adding a plurality of amounts of degradation to a set of video frames to generate a plurality of sets of degraded video frames, wherein each set of degraded video frames includes a different amount of the degradation, and training the machine learning model based on the plurality of sets of degraded video frames.16. The one or more non-transitory computer-readable storage media of any of clauses 10-15, wherein the degradation comprises at least one of noise or blur.17. The one or more non-transitory computer-readable storage media of any of clauses 10-16, wherein the machine learning model comprises a transformer model.18. The one or more non-transitory computer-readable storage media of any of clauses 10-17, wherein the machine learning model comprises one or more three-dimensional (3D) convolution layers.19. In some embodiments, a computer-implemented method for training a machine learning model comprises adding a plurality of amounts of degradation to a set of video frames to generate a plurality of sets of degraded video frames, wherein each set of degraded video frames includes a different amount of degradation, and performing one or more operations to train the machine learning model based on the plurality of sets of degraded video frames, wherein the one or more operations minimize a loss function that penalizes a difference between a temporally last frame of each plurality of processed video frames generated by the machine learning model and a temporally first frame of a subsequent plurality of processed video frames generated by the machine learning model.20. The method of clause 19, further comprising processing a first plurality of video frames using the machine learning model to generate a first plurality of processed video frames, and processing a second plurality of video frames using the machine learning model to generate a second plurality of processed video frames, wherein a temporally first video frame included in the second plurality of video frames is a temporally second to last video frame included in the first plurality of processed video frames.