Patent ID: 12250400

The same numbers are used throughout the disclosure and figures to reference like components and features. Series 100 numbers refer to features originally found inFIG.1, series 200 numbers refer to features originally found inFIG.2, series 300 numbers refer to features originally found inFIG.3, and so on.

DETAILED DESCRIPTION

This disclosure is organized as follows. Section A describes an illustrative interpolating system for performing space-time video super-resolution (STVSR). Section B sets forth illustrative methods which explain the operation of the interpolating system of Section A. And Section C describes illustrative computing functionality that can be used to implement any aspect of the features described in Sections A and B.

A. Illustrative Interpolating System

FIG.1shows an illustrative interpolating system102for performing space-time video super-resolution (STVSR). In some implementations, the interpolating system102maps an instance of input video information104to an instance of output video information106. The input video information104includes a first number of frames, each having a first resolution. The output video information106includes a second number of frames, each having a second resolution. The second number of frames is greater than the first number of frames, and the second resolution is greater than the first resolution. For shorthand reference, the input video information104will be said to exhibit low frame rate (LFR) and low resolution (LR), while the output video information106will be said to exhibit high frame rate (HFR) and high resolution (HR). But the terms “low” and “high” are to be broadly interpreted as relative measures. That is, a frame rate or resolution is said to high because it is higher than some other specified frame rate or resolution. A frame rate or resolution is said to be low because it is lower than some other specified frame rate or resolution.

More specifically,FIG.1shows that the input video information104includes at least four input video frames, (I1L, I3L, I5L, I7L), where each numeric subscript identifies a frame number and the superscript “L” designates that these frames are low resolution frames. The output video information106includes at least seven output video frames (I1H, I2H, I3H, I4H, I5H, I6HI7H), where each numeric subscript identifies a frame number and the superscript “H” designates that these frames are high resolution frames. The output video frames I1H, I3H, I5H, and I7Hare the respective higher-resolution counterparts of the input video frames I1L, I3L, I5L, and I7L. The output video information106includes new video frames I2H, I4H, and I6Hthat do not have direct counterparts in the input video information104. The use of the interpolating system102to convert four video frames to seven video frames is merely illustrative; other implementations of the interpolating system102can convert any first number of frames to any second number of video frames, where the second number is larger than the first number.

In some implementations, the input video information104is received from a video source108that produces video information having low resolution at a low frame rate. For example, the video source108may correspond to a video camera that produces LFR and/or LR video information. Here, the interpolating system102serves the purpose of increasing the frame rate and resolution of the input video information104produced by the video source108.

In other implementations, a transmission mechanism110reduces the frame rate and/or resolution of an original stream of video information produced by the video source108. For example, the transmission mechanism110can include any type of video compression algorithm which reduces the frame rate and/or resolution of the original video information prior to transmitting the video information over a computing network, such as the Internet, or any other communication channel. Here, the interpolating system102serves the purpose of increasing the frame rate and/or resolution of the video information that is received over the transmission mechanism110. In some cases, the interpolating system102can restore the video information to its original frame rate and/or resolution, or an approximation thereof.

More specifically, consider the case of a video teleconferencing system (not shown) having at least one transmitting device and at least one receiving device. The transmitting computing device may reduce the frame rate and/or resolution of captured video information to produce LFR/LR video information, and then transmit that LFR/LR video information over a computing network. This strategy helps reduce congestion in the computing network, and better ensures that a communication session among participants will not be interrupted by bandwidth limits or other communication anomalies. The receiving computing device convert the LFR/LR video information into counterpart HFR/HR video information. Still other applications are possible; the above examples are presented by way of illustration, not limitation.

In other cases, the interpolating system102performs the more targeted task of reconstituting an approximation of at least one video frame that has been lost from an original stream of video frames. For example, assume that an original stream of video frames includes frames I1L, I2L, I3L, I4L, I5L, I6L, and so on. But assume that the frame I4Lis lost by the transmission mechanism110over the course of its transmission, e.g., because of a packet drop or any other communication failure. Here, the interpolating system102can detect the omission of the video frame I4Lbased on the timing at which it is received relative to its neighboring video frames, and/or based on metadata expressed in the received video information. The interpolating system102can apply the principles described below to reconstitute a high-resolution version of the missing video frame, together with high-resolution versions of the video frames that have been received.

With the above introduction, the processing flow of the interpolating system102will now be described, from beginning to end. Later figures and accompanying description will provide additional details regarding the introductory explanation provided with respect toFIG.1.

In some implementations, the input video information104includes four frames, each of which has a red (R) channel, a blue channel (B), and a green channel (G). Each frame has a given width (W) and a height (H). Altogether, then, the input video information104includes a block of input information112of size W×H×12. A feature-extracting component114extracts features from this block of input information112to produce a block of feature information116having a size of W×H×C. C refers to the number of features used to describe each image element of the input video information104. In some non-limiting implementations, C=96. The block of feature information116can also be viewed as a concatenation of plural instances of feature information (F1L, F3L, F7L, F7L)118associated with the four input video frames (I1L, I3L, I4L, I7L).

More specifically, in some non-limiting implementations, the feature-extracting component114can produce the block of feature information116by applying a 3×3 convolutional kernel over the block of input information112. That is, to perform this operation, the feature-extracting component114moves the convolutional kernel across the block of input information112in predetermined increments. At each position of the kernel, the feature-extracting component114forms the dot product of the values in the kernel with the values in a section of the input information112that is demarcated by the kernel at its current position. The dot product at this position represents part of the block of feature information116produced by the feature-extracting component114.

An encoder-decoder component120operates on the block of feature information116in plural respective encoding and decoding stages. To begin with, an N-stage encoding component122processes the block of feature information116in a pipeline that includes plural (N) encoding stages. Each encoding stage receives encoder input information and produces encoder output information. The encoder input information fed to the first encoding stage is the block of feature information116itself. The encoder input information fed to each subsequent encoding stage is the output encoder information provided by its immediately-preceding encoding stage.

Each encoding stage also performs an attention operation that produces an instance of encoder attention information. As will be explained below, each instance of encoder attention information generated by a particular encoding stage expresses relations among image elements that have been identified within its encoder input information. That is, each relation between a first and second image elements expresses an assessed importance that the second image element has to the interpretation of the first image element. Different implementations of the encoder-decoder component120can establish what constitutes an “image element” for analysis in different respective ways. In some implementations, an image element may correspond to feature information that derives from an individual pixel that appears in the input video information104. In other implementations, an image element may correspond to feature information that derives from a patch of pixels that appear in the input video information104. A data store124stores plural instances of encoder attention operation generated by the N-stage encoding component122.

Note that the attention operation performed by a particular encoding stage operates on input information that derives from all four of the video frames. Hence, the attention operation identifies relations that include both intra-frame relations and inter-frame relations. Intra-frame relations express relations between different image elements in any given video frame. Inter-frame relations express relations between image elements of different video frames.

A last encoding stage in the pipeline of encoding stages produces encoder output information126. The encoder output information includes plural instances of encoder output information (EN,1, EN,3, EN,5, EN,7) respectively associated with the four input video frames (I1L, I3L, I7L, I7L). The subscript N in the encoder output information indicates that the encoder information is provided by the Nth encoding stage of the N-stage encoding component122.

Next, a query-generating component128generates a query130based on the encoder output information126of the last encoding stage. The query130is a seven-slot structure that includes seven instances of encoding information from which the decoding stages will produce seven video frames of the output video information106. In some non-limiting implementations, the query-generating component128uses EN,1EN,3EN,5, and EN,7of the encoder output information126to fill in the first, third, fifth, and seventh slots of query130, respectively. The query-generating component128can fill in each remaining slot of the query130by computing an average of the encoder output information on either side of the slot. For example, for the second slot of the query130, the query-generating component128can compute the mean of EN,1and EN,3, e.g., based on ½ (EN,1+EN,3). From a higher-level perspective, the query130expresses seed information from which the decoding stages will construct seven output video frames.

Other implementations of the query-generating component128can use other algorithms to generate the query130compared to the approach described above. For example, other implementations can take into consideration more than two instances of encoder output information (associated with more than two input video frames) in constituting any slot of the query130. In addition, or alternatively, other implementations can apply different weights to the different instances of encoder output information126that are used to constitute any slot of the query130.

Next, an N-stage decoding component132of the encoder-decoder component120processes the query130in a pipeline that includes plural (N) decoding stages. Each decoding stage receives decoder input information and produces decoder output information. The decoder input information fed to the first decoding stage is the query130. The decoder input information fed to each subsequent decoding stage is the output decoder information provided by its immediately-preceding decoding stage. As will be explained in greater detail below, each decoding stage also operates on an instance of encoder attention information produced by a corresponding encoding stage. That is, each particular decoding stage has a counterpart encoding stage. Each particular decoding stage receives encoder attention information from its counterpart encoding stage.

In some implementations, the decoder output information produced by the last decoding stage (referred to below as the last-stage decoder output information) constitutes the output video information106itself. In other implementations, an optional reconstruction component134performs post-processing operations on the last-stage decoder output information, to produce reconstructed information. The interpolating system102uses the reconstructed information to produce the output video information106. The post-processing operations can include a pixel shuffle operation that distributes information imparted by the last-stage decoder output information among the three color channels of the seven output video frames. In addition, or alternatively, the post-processing operations can include any type(s) of machine-trained processing layers, such as one or more convolution layers. In any event, the reconstruction component134, if it is used at all, can perform post-processing operations that are less computationally intensive compared to other STVSR approaches. This is because the interpolating system102produces accurate results even in absence of complex post-processing operations.

In some implementations, the interpolating system102is trained to produce last-stage decoder output information that represents the video output information106itself, without further reference to the input video information104. In other implementations, the interpolating system102is trained to produce last-stage decoder output information that represents residual information, that, when combined with information extracted from the original video information104, is used to produce the output video information106. In the latter case, an optional interpolation component136is used to perform tri-linear interpolation on the four frames of the input video information104to produce a first rough approximation of the output video information106. A combining component138then combines the interpolated information produced by interpolation component136with the last-stage decoder output information produced by the N-stage decoding component132(or the reconstructed information produced by the reconstruction component134), to produce the output video information106. For example, the combining component138can add the residual information to the interpolated information to produce the output video information106.

A training system140is used to train the machine-trained model(s) used by the interpolating system102. The training system140can perform training with reference to a training set that includes LFR/LR video snippets, each video frame of which is denoted by IL. The training system140can use the interpolating system102to generate HFR/HR counterparts to the LFR/LR video snippets, each video frame of which is denoted by ÎH. The training system140can iteratively train the interpolating system102to reduce the differences between the generated HFR/HR video frames and their ground truth counterparts, each of which is denoted by IH. That is, the training system140can perform training based on the following loss function

L⁡(IˆH,IH)=IˆH-IH2+ϵ2,
where ∈ is an error term.

Advancing toFIG.2, this figure shows the encoder-decoder component120introduced inFIG.1. The encoder-decoder component120includes the N-stage encoding component122and the N-stage decoding component132. The N-stage encoding component122provides a pipeline having a plurality of encoder components (202,204,206,208), while the N-stage decoding component132provides a pipeline having a plurality of decoder components (210,212,214,216). The use of four-stage pipelines is shown by way of example, not limitation. Other implementations of the encoder-decoder component120can include less than four stages or more than four stages.

The encoder and decoder components perform processing at different resolutions. For example, the first encoder component202performs a first-stage encoding operation on the block of feature information116having an original height H and an original width W. The fourth decoder component216performs a last-stage decoding operation and operates at the same resolution as the first encoder component202. The second encoder component204and the third decoder component214operate at a resolution of H/2 and W/2. The third encoder component206and the second decoder component212operate at a resolution of H/4 and W/4. Finally, the fourth encoder component208and the first decoder component210operate at a resolution of H/8 and W/8. Other implementations can successively decrease and increase resolution at other increments compared to the increments described above. Generally, the encoder-decoder system120performs analysis at different resolutions to capture details in the input video information104at different respective scales. This strategy ultimately improves the quality of the output video information106.

Each encoder component includes a transformer encoder followed by a down-sampling component (except for the fourth encoder component208, which does not include a down-sampling component). That is, the encoder components (202,204,206,208) include respective transformer encoders (218,220,222,224), and the encoder components (202,204,206) include respective down-sampling components (226,228,230). Each decoder component includes a transformer decoder followed by an up-sampling component (except for the fourth decoder component216, which does not include an up-sampling component). That is, the decoder components (210,212,214,216) include respective transformer decoders (232,234,236,238), and the decoder components (210,212,214) include respective up-sampling components (240,242,244).

Each transformer encoder processes encoder input information fed to the transformer encoder, to produce encoder attention information. For example, the transformer encoder218of the first encoder component202converts the encoder input information (here, the block of feature information116) into encoder attention information TE,1. One implementation of the transformer encoder will be described below with reference toFIGS.3and5. Each down-sampling component down-samples the encoder attention information, to produce decoder output information, e.g., using a convolutional layer with a stride two. For example, the down-sampling component226down-samples the encoder attention information TE,1to produce first-stage encoder output information E1.

Each transformer decoder processes decoder input information fed to the transformer decoder, in combination with same-stage encoder attention information, to produce decoder attention information. For example, the transformer decoder232of the first decoder component210converts the decoder information (here, the query130), in combination with the fourth-stage encoder attention information TE,4, into decoder attention information TD,1. One implementation of the transformer decoder will be described below in connection withFIGS.4and9. Each up-sampling component up-samples the decoder attention information, to produce decoder output information, e.g., using a de-convolution operation. For example, the up-sampling component240up-samples the decoder attention information TD,1to produce first-stage decoder output information D1. Altogether, the down-sampling and up-sampling components (226,228,230,240,242, and244) achieve the step-downs and step-ups in resolution summarized above.

FIG.3shows a representative transformer encoder302for use in the transformer-based encoder-decoder component120ofFIG.2. The representative transformer encoder302includes one or more window-based encoders (304, . . . ,306). Each window-based encoder, in turn, includes a windowed encoder followed by a shift-windowed encoder. For example, the first window-based encoder304includes a windowed encoder308followed by a shift-windowed encoder310, while the Kth window-based encoder306includes a windowed encoder312followed by a shift-windowed encoder314. Further detail regarding the operation of these two types of subcomponents will be described below with reference toFIGS.5-8.

FIG.4shows a representative transformer decoder402for use in the transformer-based encoder-decoder component120ofFIG.2. The representative transformer decoder402includes one more window-based decoders (404, . . . ,406). Each window-based decoder, in turn, includes a windowed decoder followed by a shift-windowed decoder. For example, the first window-based decoder404includes a windowed decoder408followed by a shift-windowed decoder410, while the Kth window-based decoder406includes a windowed decoder412followed by a shift-windowed decoder414. Further detail regarding the operation of these two types of subcomponents will be described below with reference toFIG.9.

FIG.5shows a representative window-based encoder502for use in any one of the window-based encoders ofFIG.3. The window-based encoder502includes a windowed encoder504followed by a shift-windowed encoder506. The window-based encoder502will be explained with reference to the example presented inFIGS.6-8.

Beginning with the windowed encoder504, this component receives input information that includes either the block of feature information116provided by the feature-extracting component114or the output information generated by an immediately preceding window-based encoder (not shown inFIG.5). A first layer-normalization component508normalizes the input information by adjusting it values based on the mean and standard deviation of those values.

A window-based self-attention component (W-SAC)510receives input information supplied by the layer-normalization component508. Assume that the input information has a size of W×H×C (where, again, C refers to the number of features provided by the feature-extracting component114). The W-SAC510partitions this input information into a plurality of windows of size M×M×C. Note that each window expresses information that derives from a portion of all four input video frames of the input video information104. The W-SAC510then separately performs a self-attention operation for each window. That is, for each particular window, the W-SAC510determines the importance of each image element in the particular window with respect to each other image element in the particular window. As noted above, this operation reveals intra-frame relations among image elements in a single video frame, and inter-frame relations among image elements in different video frames.

More specifically, in one implementation, the W-SAC510can perform self-attention for a particular window using the following equation:

attn⁡(Q,K,V)=softmax(Q⁢KTd+B)⁢V.(1)

The W-SAC510produces query information Q by multiplying input vectors associated with image elements in the particular window by a query weighting matrix WQ. The W-SAC510produces key information K and value information V by multiplying the same input vectors by a key weighting matrix WKand a value weighting matrix WV, respectively. To execute Equation (2), the W-SAC510takes the dot product of Q with the transpose of K, and then divides that dot product by a scaling factor ∞d; the symbol d represents the dimensionality of the machine-learned model that implements the W-SAC510. The W-SAC510adds a position matrix B to the result of the division, and the takes the softmax (normalized exponential function) of the resultant sum. The position matrix B expresses the position of each image element in the window with respect to every other image element in the window. The W-SAC510multiples the result of the softmax operation by V. Background information regarding the general concept of attention is provided in VASWANI, et al., “Attention Is All You Need,” in 31st Conference on Neural Information Processing Systems (NIPS2017), 2017, 11 pages.

A residual connection512combines (e.g., sums) the output provided by the W-SAC510with the input information fed to the first layer-normalization component508. The windowed encoder504next processes the output information provided the residual connection512using, in order, a second layer-normalization component514, a feed-forward neural network (FFN)516, and a second residual connection518. The FNN516can use any number of layers of neurons to transform input information into output information. For example, the FNN516may represent a fully-connected multi-layered perceptron (MLP).

The shift-windowed encoder506processes the output information provided by the windowed encoder504using, in order, a third layer-normalization component520, a shift-window-based self-attention component (SW-SAC)522, a third residual connection524, a fourth layer-normalization component526, a second FNN528, and a fourth residual connection530. These components perform the same tasks as their counterparts in the windowed encoder504.

Unlike the W-SAC510, however, the SW-SAC522performs a window-shifting operation. More specifically, like the W-SAC510, the SW-SAC522partitions input information into a plurality of windows. The SW-SAC522then shifts this collection of windows a prescribed amount, e.g., by shifting the collection of windows to the left by a distance of M/2, and upward by a distance of M/2. This shifting operation changes the image elements encompassed by each of the windows. The SW-SAC522then computes attention information for the windows using Equation (1) in the manner described above. More generally, the SW-SAC522performs this shifting operation to capture dependencies among image elements not previously detected by the W-SAC510. That is, since the W-SAC510performs siloed window-based analysis, the W-SAC510does not discover relations across windows. The SW-SAC522addresses this deficiency by shifting the windows and repeating the same self-attention operations performed by the W-SAC510.

FIG.6shows an example of the operation of the W-SAC510of the windowed encoder504. Assume that the W-SAC510receives input information that represents four video frames that show a subject in an outdoor scene. A video frame602represents one such video frame. The W-SAC510partitions the input information into a plurality of M×M windows. In the simplified example ofFIG.6, the W-SAC510partitions the input information into four windows (such as representative window604), but other implementations of the W-SAC510can produce a larger number of windows than show inFIG.6. Further note thatFIG.6represents each window as a two-dimensional tile, but each window also has a depth of C features (e.g., where C may equal 96). Further note that the C features capture information across the four video frames. The W-SAC510then uses attention-generating logic606to generate attention information for each of the four windows, e.g., using Equation (1).

FIG.7shows an example of the operation of the SW-SAC522of the shift-windowed encoder506. Again assume that the SW-SAC522receives input information that represents four video frames that show a subject in an outdoor scene. A video frame702represents one such video frame. As before, the SW-SAC522partitions the input information into a plurality of M×M windows, including representative window704. The SW-SAC522then shifts this collection of windows a prescribed amount, e.g., by shifting the collection of windows to the left by a distance of M/2, and upward by a distance of M/2. The SW-SAC522then uses per-window attention-generating logic706to generate attention information for each of the newly-formed windows.

In one technique, the SW-SAC522produces nine windows as a result of the shifting operation, including a representative new window708. The SW-SAC522can pad entries in the windows with dummy values to mark parts of the windows that do not contain image elements. In another more efficient technique, the SW-SAC522can use cyclic-shifting and masking (CSM) logic710to shift the windows. As shown inFIG.8, the CSM logic710cyclically shifts the windows upward and toward the left as described above. After the shift, new information (X′, Y′, Z′) that is encompassed by the windows at their shifted positions is the mirrored duplicate of information (X, Y, Z) that was previously encompassed by the windows in their original positions. The CSM logic710then produces masks to ensure that the analysis performed by the attention-generating logic706does not compare image elements across different windows. Overall, the above-described cyclic shifting strategy avoids the need to create and process an expanded number of windows, as does the first technique. The cyclic shifting strategy also avoids the need to pad windows with dummy values, as does the first technique. Background information regarding the general topic of window shifting operations is provided in LIU, et al., “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, 11 pages.

FIG.9shows one implementation of a representative window-based decoder902. The window-based decoder902is made up of a widowed decoder904and a shift-windowed decoder906. The window-based decoder902includes the same kinds of processing components as the window-based encoder502ofFIG.5. These components generally perform the same functions as set forth above with respect toFIGS.5-8. Note, however, that the window-based decoder902includes components that perform a cross-attention operation (which was not previously described), in addition to components that perform a self-attention operation (which was previously described).

More specifically, a first layer-normalization component908receives input information from the query or from the decoder output information provided by a preceding window-based decoder (not shown). A window-based self-attention component (W-SAC)910uses Equation (1) to perform self-attention on the output information generated by the first layer-normalization component908. A residual connection912combines output information generated by the W-SAC910with the input information that is fed to the first layer-normalization component908.

A second layer-normalization component914normalizes the output information provided by the first residual connection. A third layer-normalization component916normalizes encoder attention information provided by a corresponding encoder component (not shown inFIG.9). A window-based cross-attention component (W-CAC)918then performs a cross-attention operation using Equation (1). In this case, however, the query information Q ultimately derives from the input information fed to the window-based decoder902, while the key information K and value information V derive form the encoder attention information received from the corresponding encoder component. In other words, the cross-attention operation is different than the self-attention operation because, in the case of cross-attention, all of the input information used to compute attention using Equation (1) does not originate from the same source.

A second residual connection920combines the output information generated by the W-CAC918with the input information fed to the second normalization component914. The windowed encoder904processes the output of the second residual connection920using, in order, a fourth layer-normalization component922, a first feed-forward neural network (FNN)924, and a third residual connection926.

The shift-windowed decoder906contains the same architecture as the windowed decoder904, but uses shift-window counterparts of the attention operations performed by the windowed decoder904. That is, the shift-windowed decoder906includes: a fifth layer-normalization component928, a shift-window-based cross-attention component (SW-SAC)930(which is the shift-window counterpart of the W-SAC910), a fourth residual connection932, a sixth and seventh layer-normalization components (934,936), a shift-window-based cross-attention component (SW-SAC)938(which is the shift-window counterpart of the W-CAC918), a fifth residual connection940, an eight layer-normalization component942, a second FNN944, and a sixth-residual connection946.

Different implementations of the interpolating system102were built with different numbers (K) of window-based encoders and window-based decoders in the transformer encoder302and the transformer decoder402, respectively. That is, a model-S (small) was built with K=2 encoders/decoders, a model-M (medium) was built with K=3 encoders/decoders, and a model-L (large) was built with K=4 encoders/decoders. The characteristics and performance of these three versions of the interpolating system102was then compared to three other STVSR approaches by others. A first approach (TMNet) is described in XU, et al., “Temporal Modulation Network for Controllable Space-Time Video Super-Resolution,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, 10 pages. A second approach (ZSM) is described in XIANG, et al., “Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020, 10 pages. A third approach (STARnet) is described in HARIS, et al., “Space-Time-Aware Multi-Resolution Video Enhancement,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020, 10 pages.

FIG.10shows the speed-related performance of the model-S, model-M, and model-L relative to the competing approaches, measured in terms of frames-per-second (FPS).FIG.10also compares the speed of the models with respect to a threshold of 24 FPS, which is a standard cinematic rate.FIG.11shows the sizes of the model-S, model-M, and model-L relative to competing approaches, measured in terms of the number of parameters used by the different approaches (in millions). These figures generally reveal that the interpolating system102described herein is significantly smaller and faster than the state-of-the-art competing STVSR methods while maintaining similar or better performance. For example, the model-L performs similarly to TMNet with 40% fewer parameters, model-M outperforms ZSM with 50% fewer parameters, and model-S outperforms STARNet with 96% fewer parameters. Model-S achieves a frame rate of more than 24 FPS, (the standard cinematic frame rate) operating on 720×576 frames. Model-S achieves the performance of ZSM with a 75% speedup, and outperforms STARNet with about 700% speedup. Therefore, the model-S is the only approach of those compared that enjoys a real-time inference speed.

The interpolating system102uses fewer parameters (and thus uses less memory and processing resources) compared to competing resources because it uses the same components of its architecture to simultaneously and jointly perform both temporal and spatial interpolation. “Joint” means “united” or “combined” as used herein. For example, each encoding component provides insight across both the spatial and temporal dimensions in the course of generating encoder attention information. A counterpart decoder component relies on this encoder attention information in decoding the query130. Other approaches, by contrast, dedicate particular functionality to performing temporal interpolation and other functionality to performing spatial resolution, without sharing insights between these two instances of functionality. The interpolating system102can also use a computationally light feature-extracting component114and reconstruction component134(if used at at), compared to competing approaches.

Note that the interpolating system102has been described above in the context of a transformer-based architecture. Other implementations of the interpolating systems102can use other types of machine-trained components to perform unified space-time interpolation, such as recurrent neural networks (RNN), convolutional neural networks (CNN), feed-forward neural networks (FNN), etc., or any combination thereof. The tasks that the interpolating system102can perform, regardless of the kind(s) of machine-trained components used include: (a) forming feature information that describes image elements across a plurality of video frames; (b) analyzing the feature information in a pipeline of encoding stages, to generate insight regarding the relations among image elements in both the spatial and temporal dimensions; and (c) building a set of video frames in a pipeline of decoding stages with reference to the information generated in the encoding stages, to produce output video information.

In addition, or alternatively, the N-stage encoding component122is a one-stage encoding component and the N-stage decoding component132is a one-stage decoding component. The one-stage decoding component operates on encoder attention information generated by the query-generating component128and the one-stage encoding component.

B. Illustrative Processes

FIGS.12-14show processes that explain the operation of the interpolating system102of Section A in flowchart form, according to some implementations. Since the principles underlying the operation of the interpolating system102have already been described in Section A, certain operations will be addressed in summary fashion in this section. Each flowchart is expressed as a series of operations performed in a particular order. But the order of these operations is merely representative, and can be varied in other implementations. Further, any two or more operations described below can be performed in a parallel manner. In some implementations, the blocks shown in the flowcharts that pertain to processing-related functions are implemented by the hardware logic circuitry described in Section C, which, in turn, can be implemented by one or more hardware processors and/or other logic units that include a task-specific collection of logic gates.

FIG.12shows a process1202for performing an interpolation operation. In block1204, the interpolating system102obtains input video information having a given first number of plural frames, each frame in the input video information104having a given first spatial resolution. In block1206, the feature-extracting component114generates feature information116based on the input video information104. In block1208, the interpolating system102produces output video information106based on the feature information116using a transformer-based encoding operation, followed by a query-generating operation, followed by a transformer-based decoding operation. The transformer-based decoding operation is performed based on encoder attention information produced by the transformer-based encoding operation. The encoder attention information expresses identified relations across the plural frames of the input video information104. The output video information106has a second number of frames that is higher than the first number of frames in the input video information104and has a second spatial resolution that is higher than the first spatial resolution of the input video information104.

FIGS.13and14show another process1302performed by the interpolating system102for interpolating video information. In block1304, the interpolating system obtains input video information104having a given first number of plural frames, each frame in the input video information104having a given first spatial resolution. In block1306, the feature-extracting component114generates feature information116based the input video information104. In block1308, the N-stage encoding component122encodes the feature information116in a pipeline having plural encoding stages that operate at different respective resolutions, to produce plural instances of encoder attention information and plural instances of encoder output information. Each instance of the encoder attention information expresses identified relations across the plural frames of the input video information104. In block1310, the query-generating component128produces a query130based on an instance of encoder output information produced by a last encoding stage of the plural encoding stages. In block1312, the N-stage decoding component132decodes the query130in a pipeline having plural decoding stages that operate at different respective resolutions, to produce plural instances of decoder output information. Each decoding stage that has a preceding decoding stage receives an instance of decoding input information produced by the preceding decoding stage. Further, each particular decoding stage operates on an instance of encoder attention information produced by a particular encoding stage that has a same resolution level as the particular decoding stage. In block1402ofFIG.14, the interpolating system102produces output video information106based on decoder output information produced by a last decoding stage of the plural decoding stages. The output video information106has a second number of frames that is higher than the first number of frames in the input video information104, and has a second spatial resolution that is higher than the first spatial resolution of the input video information104.

C. Representative Computing Functionality

FIG.15shows an example of computing equipment that can be used to implement any of the systems summarized above. The computing equipment includes a set of user computing devices1502coupled to a set of servers1504via a computer network1506. Each user computing device can correspond to any device that performs a computing function, including a desktop computing device, a laptop computing device, a handheld computing device of any type (e.g., a smartphone, a tablet-type computing device, etc.), a mixed reality device, a wearable computing device, an Internet-of-Things (IoT) device, a gaming system, and so on. The computer network1506can be implemented as a local area network, a wide area network (e.g., the Internet), one or more point-to-point links, or any combination thereof.

FIG.15also indicates that the interpolating system102and the training component system140can be spread across the user computing devices1502and/or the servers1504in any manner. For instance, in some cases, the interpolating system102is entirely implemented by one or more of the servers1504. Each user may interact with the servers1504via a user computing device. In other cases, the interpolating system102is entirely implemented by a user computing device in local fashion, in which case no interaction with the servers1504is necessary. In another case, the functionality associated with the interpolating system102is distributed between the servers1504and each user computing device in any manner.

FIG.16shows a computing system1602that can be used to implement any aspect of the mechanisms set forth in the above-described figures. For instance, the type of computing system1602shown inFIG.16can be used to implement any user computing device or any server shown inFIG.15. In all cases, the computing system1602represents a physical and tangible processing mechanism.

The computing system1602can include one or more hardware processors1604. The hardware processor(s)1604can include, without limitation, one or more Central Processing Units (CPUs), and/or one or more Graphics Processing Units (GPUs), and/or one or more Application Specific Integrated Circuits (ASICs), and/or one or more Neural Processing Units (NPUs), etc. More generally, any hardware processor can correspond to a general-purpose processing unit or an application-specific processor unit.

The computing system1602can also include computer-readable storage media1606, corresponding to one or more computer-readable media hardware units. The computer-readable storage media1606retains any kind of information1608, such as machine-readable instructions, settings, data, etc. Without limitation, the computer-readable storage media1606may include one or more solid-state devices, one or more magnetic hard disks, one or more optical disks, magnetic tape, and so on. Any instance of the computer-readable storage media1606can use any technology for storing and retrieving information. Further, any instance of the computer-readable storage media1606may represent a fixed or removable unit of the computing system1602. Further, any instance of the computer-readable storage media1606may provide volatile or non-volatile retention of information.

More generally, any of the storage resources described herein, or any combination of the storage resources, may be regarded as a computer-readable medium. In many cases, a computer-readable medium represents some form of physical and tangible entity. The term computer-readable medium also encompasses propagated signals, e.g., transmitted or received via a physical conduit and/or air or other wireless medium, etc. However, the specific term “computer-readable storage medium” expressly excludes propagated signals per se in transit, while including all other forms of computer-readable media.

The computing system1602can utilize any instance of the computer-readable storage media1606in different ways. For example, any instance of the computer-readable storage media1606may represent a hardware memory unit (such as Random Access Memory (RAM)) for storing information during execution of a program by the computing system1602, and/or a hardware storage unit (such as a hard disk) for retaining/archiving information on a more permanent basis. In the latter case, the computing system1602also includes one or more drive mechanisms1610(such as a hard drive mechanism) for storing and retrieving information from an instance of the computer-readable storage media1606.

The computing system1602may perform any of the functions described above when the hardware processor(s)1604carry out computer-readable instructions stored in any instance of the computer-readable storage media1606. For instance, the computing system1602may carry out computer-readable instructions to perform each block of the processes described in Section B.

Alternatively, or in addition, the computing system1602may rely on one or more other hardware logic units1616to perform operations using a task-specific collection of logic gates. For instance, the hardware logic unit(s)1612may include a fixed configuration of hardware logic gates, e.g., that are created and set at the time of manufacture, and thereafter unalterable. Alternatively, or in addition, the other hardware logic unit(s)1612may include a collection of programmable hardware logic gates that can be set to perform different application-specific tasks. The latter class of devices includes, but is not limited to Programmable Array Logic Devices (PALs), Generic Array Logic Devices (GALs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), etc.

FIG.16generally indicates that hardware logic circuitry1614includes any combination of the hardware processor(s)1604, the computer-readable storage media1606, and/or the other hardware logic unit(s)1612. That is, the computing system1602can employ any combination of the hardware processor(s)1604that execute machine-readable instructions provided in the computer-readable storage media1606, and/or one or more other hardware logic unit(s)1612that perform operations using a fixed and/or programmable collection of hardware logic gates. More generally stated, the hardware logic circuitry1614corresponds to one or more hardware logic units of any type(s) that perform operations based on logic stored in and/or otherwise embodied in the hardware logic unit(s). Further, in some contexts, each of the terms “component,” “module,” “engine,” “system,” and “tool” refers to a part of the hardware logic circuitry1614that performs a particular function or combination of functions.

In some cases (e.g., in the case in which the computing system1602represents a user computing device), the computing system1602also includes an input/output interface1616for receiving various inputs (via input devices1618), and for providing various outputs (via output devices1620). Illustrative input devices include a keyboard device, a mouse input device, a touchscreen input device, a digitizing pad, one or more static image cameras, one or more video cameras, one or more depth camera systems, one or more microphones, a voice recognition mechanism, any position-determining devices (e.g., GPS devices), any movement detection mechanisms (e.g., accelerometers, gyroscopes, etc.), and so on. One particular output mechanism may include a display device1622and an associated graphical user interface presentation (GUI)1624. The display device1622may correspond to a liquid crystal display device, a light-emitting diode display (LED) device, a cathode ray tube device, a projection mechanism, etc. Other output devices include a printer, one or more speakers, a haptic output mechanism, an archival mechanism (for storing output information), and so on. The computing system1602can also include one or more network interfaces1626for exchanging data with other devices via one or more communication conduits1628. One or more communication buses1630communicatively couple the above-described units together.

The communication conduit(s)1628can be implemented in any manner, e.g., by a local area computer network, a wide area computer network (e.g., the Internet), point-to-point connections, etc., or any combination thereof. The communication conduit(s)1628can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols.

FIG.16shows the computing system1602as being composed of a discrete collection of separate units. In some cases, the collection of units corresponds to discrete hardware units provided in a computing device chassis having any form factor.FIG.16shows illustrative form factors in its bottom portion. In other cases, the computing system1602can include a hardware logic unit that integrates the functions of two or more of the units shown inFIG.1. For instance, the computing system1602can include a system on a chip (SoC or SOC), corresponding to an integrated circuit that combines the functions of two or more of the units shown inFIG.16.

The following summary provides a non-exhaustive set of illustrative examples of the technology set forth herein.

(A1) According to a first aspect, some implementations of the technology described herein include a method (e.g., the process1302) for performing an interpolation operation. The method includes: obtaining (e.g., in block1304) input video information (e.g.,104) having a given first number of plural frames, each frame in the input video information having a given first spatial resolution; generating (e.g.,1306) feature information (e.g.,116) based the input video information; and encoding (e.g.,1308) the feature information in a pipeline having plural encoding stages that operate at different respective resolutions, to produce plural instances of encoder attention information and plural instances of encoder output information, each instance of the encoder attention information expressing identified relations across the plural frames of the input video information. The method then includes producing (e.g.,1310) a query (e.g.,130) based on an instance of encoder output information produced by a last encoding stage of the plural encoding stages. The method then includes decoding (e.g.,1312) the query in a pipeline having plural decoding stages that operate at different respective resolutions, to produce plural instances of decoder output information. Each decoding stage that has a preceding decoding stage receives an instance of decoding input information produced by the preceding decoding stage. Further, each particular decoding stage operates on an instance of encoder attention information produced by a particular encoding stage that has a same resolution level as the particular decoding stage. The method then includes producing (e.g.,1402) output video information (e.g.,106) based on decoder output information produced by a last decoding stage of the plural decoding stages. The output video information has a second number of frames that is higher than the first number of frames in the input video information, and has a second spatial resolution that is higher than the first spatial resolution of the input video information. The method is technically advantageous because it uses a model that is more compact than competing interpolating systems, and because it runs faster than competing interpolating systems.

(A2) According to some implementations of the method of A1, each given stage in the pipeline of encoding stages and the pipeline of decoding stages produces a particular instance of attention information using a transformer-based neural network.

(A3) According to some implementations of the method of A2, the transformer-based neural network performs a first kind of attention operation that involves generating window-specific attention information for individual windows within input information that is fed to the first kind of attention operation.

(A4) According to some implementations of the method of A3, the transformer-based neural network also performs a second kind of attention operation that involves generating window-specific attention information for individual windows within input information that is fed to the second kind of attention operation, the individual windows in the second kind of attention operation being shifted relative to the individual windows in the first kind of attention operation.

(A5) According to some implementations of any of the methods of A2-A4, the given stage is a given encoding stage, and wherein the given encoding stage also performs a down-sampling operation on the particular instance of attention information.

(A6) According to some implementations of any of the methods of A2-A5, the given stage is a given encoding stage, and the particular instance of attention information includes key information and value information that is generated based on feature information obtained from the plural frames of the input video information.

(A7) According to some implementations of the method of A2, the given stage is a given decoding stage, and the given decoding stage also performs an up-sampling operation on the particular instance of attention information.

(A8) According to some implementations of any of the methods of A1-A7, the operation of producing a query involves producing encoder output information for at least one added frame that is not present in the input video information.

(A9) According to some implementations of the method of A8, the operation of producing a query produces the encoder output information for the aforementioned at least one added frame based on the encoder output information produced by the last encoding stage for frames in the input video information that temporally precede and follow the added frame.

(A10) According to some implementations of any of the methods of A1-A9, the operation of producing the output video information includes: interpolating the input video information to produce interpolated video information; and combining the decoder output information produced by the last decoding stage with the interpolated video information.

(A11) According to some implementations of the method of A10, the operation of producing the video output information includes performing a reconstruction operation on the decoder output information produced by the last decoding stage prior to said combining.

(B1) According to another illustrative aspect, another method (e.g., the process1202) is described for interpolating video information. The method includes: obtaining (e.g.,1204) input video information (e.g.,104) having a given first number of plural frames, each frame in the input video information having a given first spatial resolution; generating (e.g.,1206) feature information (e.g.,116) based on the input video information; and producing (e.g.,1208) output video information (e.g.,106) based on the feature information using a transformer-based encoding operation, followed by a query-generating operation, followed by a transformer-based decoding operation. The transformer-based decoding operation is performed based on encoder attention information produced by the transformer-based encoding operation. The encoder attention information expresses identified relations across the plural frames of the input video information. The output video information has a second number of frames that is higher than the first number of frames in the input video information, and has a second spatial resolution that is higher than the first spatial resolution of the input video information. The method is technically advantageous because it uses a model that is more compact than competing interpolating systems, and because it runs faster than competing interpolating systems.

(B2) According to some implementations of the method of B1, the transformer-based encoding operation includes a pipeline having plural encoding stages that operate at different respective resolutions, and that produce plural instances of encoder attention information and plural instances of encoder output information. Each instance of the encoder attention information expresses identified relations across the plural frames of the input video information. The transformer-based decoding operation includes a pipeline having plural decoding stages that operate at different respective resolutions, and that produce plural instances of decoder output information. Each decoding stage that has a preceding decoding stage receives an instance of decoder input information produced by the preceding decoding stage. And each particular decoding stage operates on an instance of encoder attention information produced by a particular encoding stage that has a same resolution level as the particular decoding stage.

(B3) According to some implementations of the method of B2, a given encoding stage performs at least one kind of attention operation and a down-sampling operation.

(B4) According to some implementations of the method of B2, a given decoding stage performs at least one kind of attention operation and an up-sampling operation.

(B5) According to some implementations of the method of B2, the query-generating operation produces a query based on encoder output information produced by a last encoding stage of the plural encoding stages. The decoder input information fed to a first decoding stage in the plural decoding stages is the query.

In yet another aspect, some implementations of the technology described herein include a computing system (e.g., computing system1602). The computing system includes hardware logic circuitry (e.g.,1614) that is configured to perform any of the methods described herein (e.g., any of the methods of A1-A11 and B1-B5).

In yet another aspect, some implementations of the technology described herein include a computer-readable storage medium (e.g., the computer-readable storage media1606) for storing computer-readable instructions (e.g.,1608). The computer-readable instructions, when executed by one or more hardware processors (e.g.,1604), perform any of the methods described herein (e.g., any of the methods of A1-A11 and B1-B5).

More generally stated, any of the individual elements and steps described herein can be combined, without limitation, into any logically consistent permutation or subset. Further, any such combination can be manifested, without limitation, as a method, device, system, computer-readable storage medium, data structure, article of manufacture, graphical user interface presentation, etc. The technology can also be expressed as a series of means-plus-format elements in the claims, although this format should not be considered to be invoked unless the phase “means for” is explicitly used in the claims.

As to terminology used in this description, the phrase “configured to” encompasses various physical and tangible mechanisms for performing an identified operation. The mechanisms can be configured to perform an operation using the hardware logic circuitry1614of Section C. The term “logic” likewise encompasses various physical and tangible mechanisms for performing a task. For instance, each processing-related operation illustrated in the flowcharts of Section B corresponds to a logic component for performing that operation.

This description may have identified one or more features as “optional.” This type of statement is not to be interpreted as an exhaustive indication of features that may be considered optional; that is, other features can be considered as optional, although not explicitly identified in the text. Further, any description of a single entity is not intended to preclude the use of plural such entities; similarly, a description of plural entities is not intended to preclude the use of a single entity. Further, while the description may explain certain features as alternative ways of carrying out identified functions or implementing identified mechanisms, the features can also be combined together in any combination. Further, the term “plurality” refers to two or more items, and does not necessarily imply “all” items of a particular kind, unless otherwise explicitly specified. Further, the descriptors “first,” “second,” “third,” etc. are used to distinguish among different items, and do not imply an ordering among items, unless otherwise noted. The phrase “A and/or B” means A, or B, or A and B. Further, the terms “comprising,” “including,” and “having” are open-ended terms that are used to identify at least one part of a larger whole, but not necessarily all parts of the whole. Finally, the terms “exemplary” or “illustrative” refer to one implementation among potentially many implementations.

In closing, the description may have set forth various concepts in the context of illustrative challenges or problems. This manner of explanation is not intended to suggest that others have appreciated and/or articulated the challenges or problems in the manner specified herein. Further, this manner of explanation is not intended to suggest that the subject matter recited in the claims is limited to solving the identified challenges or problems; that is, the subject matter in the claims may be applied in the context of challenges or problems other than those described herein.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.