Patent Description:
Video content represents the majority of total Internet traffic and is expected to increase even more as spatial resolution frame rate, and color depth of videos increase and more users adopt streaming services. Although existing codecs have achieved impressive performance, they have been engineered to the point where adding further small improvements is unlikely to meet future demands. Consequently, exploring fundamentally different ways to perform video coding may advantageously lead to a new class of video codecs with improved performance and flexibility.

For example, one advantage of using a trained machine learning (ML) model, such as a neural network (NN), in the form of a generative adversarial network (GAN) for example, to perform video compression is that it enables the ML, model to infer visual details that it would otherwise be costly in terms of data transmission, to obtain. However, the model size remains an important issue in current state-of-the-art proposals and existing solutions require significant computation effort on the decoding side. That is to say, one significant drawback of existing GAN-based compression frameworks is that they typically require large decoder models that are sometimes trained on private datasets. Therefore, retraining these models to their original performance is not generally possible, and even when the training data is available, retraining the model would be complicated and time consuming. Moreover, the memory requirements and the inference time of exiting large decoder models make them less practical, especially in the context of video coding.

<CIT> discloses techniques for coding sets of images with neuronal networks. <NPL> discloses a concept for neuronal data compression. Both documents are drawn to fine tuning an encoder/decoder for a specific image or set of images to improve compression results, i.e., better quality with lower cost.

Further reference is made to<NPL> and<NPL>.

Enabling disclosure for the invention is found in the embodiments of <FIG> and <FIG>. The remaining embodiments are to be understood as examples which do not describe parts of the present invention.

Embodiments of this disclosure are as follows:.

A system comprising:
a machine learning (ML) model-based video encoder configured to:.

Preferably the system, wherein identifying the first decompression data comprises overfitting the first decompression data during the encoding of the first video frame subset, and wherein identifying the second decompression data comprises overfitting the second decompression data during the encoding of the second video frame subset.

Preferably the system, wherein the first video frame subset comprises video frames that are visually similar to one another, and wherein the second video frame subset comprises other video frames that are more visually similar to one another than to the video frames of the first video frame subset.

Preferably the system, further comprising an ML, model-based video decoder;
the ML model-based video encoder further configured to:.

Preferably the system, wherein the first decompression data is specific to decoding the first compressed video frame subset but not the second compressed video frame subset, and the second decompression data is specific to decoding the second compressed video frame subset but not the first compressed video frame subset.

Preferably the system, wherein the ML, model-based video decoder comprises an artificial neural network (NN).

Preferably the system, wherein the ML, model-based video decoder comprises a Micro-Residual-Network (MicroRN), and wherein the first decompression data and the second decompression data contain only weights of the MicroRN.

Preferably the system, wherein the ML, model-based video encoder comprises a High-Fidelity Compression (HiFiC) encoder, and wherein the ML, model-based video decoder includes fewer parameters than a HiFiC decoder not using the first decompression data and the second decompression data.

Preferably the system, wherein the ML, model-based video encoder comprises a HiFiC encoder, and wherein the ML model-based video decoder is configured to achieve a faster decoding time than a HiFiC decoder not using the first decompression data and the second decompression data.

Preferably the system, wherein the first decompression data is received only once for decoding of the first compressed video frame subset, and wherein the second decompression data is received only once for decoding of the second compressed video frame subset.

A method for use by a system including a machine learning (ML) model-based video encoder, the method comprising:.

Preferably the method, wherein identifying the first decompression data comprises overfitting the first decompression data during the encoding of the first video frame subset, and wherein identifying the second decompression data comprises overfitting the second decompression data during the encoding of the second video frame subset.

Preferably the method, wherein the first video frame subset comprises video frames that are visually similar to one another, and wherein the second video frame subset comprises other video frames that are more visually similar to one another than to the video frames included in the first video frame subset.

Preferably the method, wherein the system further comprises the ML model-based video decoder, the method further comprising:.

Preferably the method, wherein the first decompression data is specific to decoding the first compressed video frame subset but not the second compressed video frame subset, and the second decompression data is specific to decoding the second compressed video frame subset but not the first compressed video frame subset.

Preferably the method, wherein the ML, model-based video decoder comprises an artificial neural network (NN).

Preferably the method, wherein the ML, model-based video decoder comprises a Micro-Residual-Network (MicroRN), and wherein the first decompression data and the second decompression data contain only weights of the MicroRN.

Preferably the method, wherein the ML, model-based video encoder comprises a High-Fidelity Compression (HiFiC) encoder, and wherein the ML model-based video decoder includes fewer parameters than a HiFiC decoder not using the first decompression data and the second decompression data.

Preferably the method, wherein the ML, model-based video encoder comprises a HiFiC encoder, and wherein the ML model-based video decoder is configured to achieve a faster decoding time than a HiFiC decoder not using the first decompression data and the second decompression data.

Preferably the method, wherein the first decompression data is received only once for decoding of the first compressed video frame subset, and wherein the second decompression data is received only once for decoding of the second compressed video frame subset.

The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals. Moreover, the drawings and illustrations in the present application are generally not to scale, and are not intended to correspond to actual relative dimensions.

The present application is directed to systems and methods for providing machine learning (ML) model-based video codecs. In addition, the present application discloses a knowledge distillation (KD) approach that enables the retention of good perceptual image quality while reducing the size of the decoder. According to the present novel and inventive principles, the goal of KD is to transfer the learned knowledge of a teacher network to a smaller student network that remains competitive to the teacher network's performance. By requiring less memory and computational power than the initial teacher network, the student network could, for instance, run on less powerful devices such as mobile phones or dedicated devices. The ability to compress the generator network or decoder in the auto-encoder setting, as disclosed herein, is advantageous both in terms of memory requirements and computational efficiency. This is especially important for image and video compression, where the majority of the computation should preferably be performed on the sender (encoder) side, while the decoding should be simple. Especially in the context of video streaming, an asset will typically be encoded once for distribution, but may be decoded millions of times.

One advantage of using a trained machined learning model, such as an artificial neural network (NN) for example, to perform video compression is that it enables the machine learning model to infer visual details that it would otherwise be costly in terms of data transmission to obtain. Consequently, the resulting images are typically visually pleasing without requiring a high bitrate. Some image details synthesized when using a machine learning model-based video codec may look realistic while deviating slightly from the ground truth. Nevertheless, the present machine learning model-based video compression solution is capable of providing image quality that would be impossible using the same amount of transmitted data in conventional approaches. Moreover, in some implementations, the present machine learning model-based solution can be implemented as substantially automated systems and methods.

It is noted that, as used in the present application, the terms "automation," "automated," and "automating" refer to systems and processes that do not require the participation of a human user, such as a human editor or system administrator. Although, in some implementations, a human system administrator may review the performance of the automated systems operating according to the automated processes described herein, that human involvement is optional. Thus, the processes described in the present application may be performed under the control of hardware processing components of the disclosed systems.

It is further noted that, as defined in the present application, the expression "machine learning model" (hereinafter "ML model") refers to a mathematical model for making future predictions based on patterns learned from samples of data obtained from a set of trusted known matches and known mismatches, known as training data. Various learning algorithms can be used to map correlations between input data and output data. These correlations form the mathematical model that can be used to make future predictions on new input data. Such a predictive model may include one or more logistic regression models, Bayesian models, or NNs, for example. In addition, machine learning models may be designed to progressively improve their performance of a specific task.

A "deep neural network" (deep NN), in the context of deep learning, may refer to an NN that utilizes multiple hidden layers between input and output layers, which may allow for learning based on features not explicitly defined in raw data. As used in the present application, a feature labeled as an NN refers to a deep neural network. In various implementations, NNs may be utilized to perform image processing or natural-language processing. Although the present novel and inventive principles are described below by reference to an exemplary NN class known as generative adversarial networks (GANs), that characterization is provided merely in the interests of conceptual clarity.

<FIG> shows an exemplary system for performing machine learning (ML) model-based video compression, according to one implementation. As shown in <FIG>, system <NUM> includes computing platform <NUM> having processing hardware <NUM> and system memory <NUM> implemented as a computer-readable non-transitory storage medium. According to the present exemplary implementation, system memory <NUM> stores uncompressed video sequence <NUM> and ML model-based video encoder <NUM>.

As further shown in <FIG>, system <NUM> is implemented within a use environment including communication network <NUM> and user system <NUM> configured for use by user <NUM>. User system <NUM> includes display <NUM>, user system processing hardware <NUM>, and user system memory <NUM> implemented as a computer-readable non-transitory storage medium storing ML, model-based video decoder <NUM>. In addition, <FIG> shows network communication links <NUM> interactively connecting user system <NUM> with system <NUM> via communication network <NUM>, as well as compressed video bitstream <NUM> output by ML model-based video encoder <NUM> and corresponding to uncompressed video sequence <NUM>.

Although the present application refers to ML, model-based video encoder <NUM> as being stored in system memory <NUM> for conceptual clarity, more generally system memory <NUM> may take the form of any computer-readable non-transitory storage medium. The expression "computer-readable non-transitory storage medium," as used in the present application, refers to any medium, excluding a carrier wave or other transitory signal that provides instructions to processing hardware <NUM> of computing platform <NUM>. Thus, a computer-readable non-transitory storage medium may correspond to various types of media, such as volatile media and non-volatile media, for example. Volatile media may include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical discs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.

Moreover, although <FIG> depicts ML model-based video encoder <NUM> as being stored in its entirety in system memory <NUM> that representation is also provided merely as an aid to conceptual clarity. More generally, system <NUM> may include one or more computing platforms <NUM>, such as computer servers for example, which may be co-located, or may form an interactively linked but distributed system, such as a cloud-based system, for instance. As a result, processing hardware <NUM> and system memory <NUM> may correspond to distributed processor and memory resources within system <NUM>. Consequently, in some implementations, one or more of the features of ML model-based video encoder <NUM> may be stored remotely from one another on the distributed memory resources of system <NUM>.

Processing hardware <NUM> may include multiple hardware processing units, such as one or more central processing units, one or more graphics processing units, and one or more tensor processing units, one or more field-programmable gate arrays (FPGAs), custom hardware for machine-learning training or inferencing, and an application programming interface (API) server, for example. By way of definition, as used in the present application, the terms "central processing unit" (CPU), "graphics processing unit" (GPU), and "tensor processing unit" (TPU) have their customary meaning in the art. That is to say, a CPU includes an Arithmetic Logic Unit (ALU) for carrying out the arithmetic and logical operations of computing platform <NUM>, as well as a Control Unit (CU) for retrieving programs, such as ML model-based video encoder <NUM>, from system memory <NUM>, while a GPU may be implemented to reduce the processing overhead of the CPU by performing computationally intensive graphics or other processing tasks. A TPU is an application-specific integrated circuit (ASIC) configured specifically for artificial intelligence (AI) processes such as machine learning.

In some implementations, computing platform <NUM> may correspond to one or more web servers, accessible over communication network <NUM> in the form of a packet-switched network such as the Internet, for example. Moreover, in some implementations, communication network <NUM> may be a high-speed network suitable for high performance computing (HPC), for example a <NUM> GigE network or an Infiniband network. In some implementations, computing platform <NUM> may correspond to one or more computer servers supporting a private wide area network (WAN), local area network (LAN), or included in another type of limited distribution or private network. As yet another alternative, in some implementations, system <NUM> may be implemented virtually, such as in a data center. For example, in some implementations, system <NUM> may be implemented in software, or as virtual machines.

According to the implementation shown by <FIG>, user <NUM> may utilize user system <NUM> to interact with system <NUM> over communication network <NUM>. User system <NUM> and communication network <NUM> enable user <NUM> to obtain compressed video bitstream <NUM> corresponding to uncompressed video sequence <NUM> from system <NUM>.

Although user system <NUM> is shown as a desktop computer in <FIG>, that representation is provided merely as an example. More generally, user system <NUM> may be any suitable mobile or stationary computing device or system that implements data processing capabilities sufficient to provide a user interface, support connections to communication network <NUM>, and implement the functionality ascribed to user system <NUM> herein. For example, in some implementations, user system <NUM> may take the form of a laptop computer, tablet computer, smartphone, or game console, for example. However, in other implementations user system <NUM> may be a "dumb terminal" peripheral component of system <NUM> that enables user <NUM> to provide inputs via a keyboard or other input device, as well as to video content via display <NUM>. In those implementations, user system <NUM> and display <NUM> may be controlled by processing hardware <NUM> of system <NUM>.

With respect to display <NUM> of user system <NUM>, display <NUM> may be physically integrated with user system <NUM> or may be communicatively coupled to but physically separate from user system <NUM>. For example, where user system <NUM> is implemented as a smartphone, laptop computer, or tablet computer, display <NUM> will typically be integrated with user system <NUM>. By contrast, where user system <NUM> is implemented as a desktop computer, display <NUM> may take the form of a monitor separate from user system <NUM> in the form of a computer tower. Moreover, display <NUM> may take the form of a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a quantum dot (QD) display, or a display using any other suitable display technology that performs a physical transformation of signals to light.

By way of background, a mapping from image space to latent space may be achieved using ML model-based video encoder <NUM>, where the bottleneck values constitute the latent representation. A function g denotes the mapping from image space to latent space performed by ML model-based video encoder <NUM>, and g-<NUM> denotes the reverse mapping. An uncompressed original image x is first mapped to its latent representation y = g(x). After quantization, the resulting quantized latents y̌ are encoded losslessly to compressed video bitstream <NUM> that can be decoded into the uncompressed image x̌= g-<NUM>(y̌) that corresponds to original image x.

Image compression can formally be expressed as minimizing the expected length of the bitstream as well as the expected distortion of the reconstructed image compared to the original, formulated as optimizing the following rate-distortion objective function: <MAT> where: - log<NUM> py̌ (y̌) is the rate term and d(x, x̂) is the distortion term.

It is noted that in the notation used in Equation <NUM>, the parameters of g include g-<NUM>. Here d indicates a distortion measure and can include a combination of ℓ<NUM>, structural similarity index measure (SSIM), learned perceptual image patch similarity (LPIPS), and the like. The rate corresponds to the length of the bitstream needed to encode the quantized representation y̌, based on a learned entropy model py̌ over the unknown distribution of natural images px. By reducing the weight λ, better compression can be achieved at the cost of larger distortion on the reconstructed image.

According to one implementation of the present novel and inventive concepts, the ML model-based image compression formulation described above can be augmented with an ML model in the form of a conditional GAN. In such a case of adversarial training, D is denoted as the discriminator neural network that learns to distinguish between the ground truth x and the decoded images x̂ conditioned on the latent representation y̌: <MAT>.

The training of the discriminator is alternated with the training of image compression ML model <NUM>, in which case the rate-distortion objective augmented with the adversarial loss is optimized: <MAT> where: - log<NUM> py̌ (y̌) and d(x,x̂) remain the rate and distortion terms, respectively, while D(x̂,y̌) is the adversarial loss.

In order to take advantage of temporal redundancy in video encoding, video compression relies on information transfer through motion compensation. More precisely, a subsequent frame xt+<NUM> can be predicted from its preceding frame xt by considering motion information. As defined in the present application, the expression "motion compensation" refers to the full process that computes and encodes motion vectors, as well as any post-processing that may occur. For simplicity, it is assumed that motion compensation has been completed, and the result is an estimate of the image x̃t+<NUM> and a motion vector field m̂t+<NUM>.

<FIG> shows diagram <NUM> comparing an existing approach <NUM> to neural compression with the microdosing compression approach <NUM> introduced by the present application. The present microdosing compression approach <NUM> is based on: <NUM>) training a reduced student-decoder with data generated from a large decoder, <NUM>) overfitting the reduced student-decoder model to a specific image or set of images, and <NUM>) sending the specialized decoder weights as decompression data <NUM> alongside the image latents. To showcase the viability of the present microdosing compression approach, its incorporation into state-of-the-art models for neural image and video compression targeting the low bitrate setting is described.

First, a High-Fidelity Compression (HiFiC) or other high-performance decoder is replaced with a much smaller student-decoder. It is noted that HiFiC presently provides the state-of-the-art in low bitrate neural image compression (i.e., approximately <NUM> bits per pixel) and produces extremely competitive results at the cost of a relatively big (i.e., approximately <NUM> million parameters) decoder network. However, although a HiFiC architecture is shown to be utilized in existing approach <NUM>, that representation is merely exemplary. In other implementations, the HiFiC encoder-decoder network of existing approach <NUM> may be replaced by substantially any GAN trained network having a similar architecture based on residual blocks.

By contrast to existing approach <NUM>, microdosing compression approach <NUM> disclosed by the present application advantageously allows for a much smaller decoder (e.g., approximately <NUM> million parameters) and fifty percent (<NUM>%) faster decoding time while producing output images that are visually similar to those provided by HiFiC. Second, the application of the present microdosing KD strategy in a neural video compression framework based on latent residuals is described. In such a scenario, the reduced student-decoder is overfitted to a sequence so that a sequence specific decoder can be provided.

As shown in <FIG>, according to existing approach <NUM> to neural compression, the encoder-decoder pair is trained on a big dataset, to get an overall good performance on a variety of different content. Once the auto-encoder is fully trained, the decoder gets deployed and sent to the receiver. The big decoder then enables the decoding of any type of content. According to the present microdosing approach <NUM> by contrast, ML model-based video encoder <NUM> is configured to partition uncompressed video sequence <NUM> data into subsets <IMG>, and to learn a content-specific decoder with corresponding information <IMG> for each subset. This specialization enables the training of a ML, model-based video decoder <NUM> that advantageously requires fewer parameters, a smaller memory footprint, and using fewer computations. It is noted that ML, model-based video encoder <NUM>, uncompressed video sequence <NUM>, and ML, model-based video decoder <NUM> correspond respectively in general to ML model-based video encoder <NUM>, uncompressed video sequence <NUM>, and ML model-based video decoder <NUM>, in <FIG>. Consequently, ML model-based video encoder <NUM>, uncompressed video sequence <NUM>, and ML, model-based video decoder <NUM> may share any of the features attributed to respective ML, model-based video encoder <NUM>, uncompressed video sequence <NUM>, and ML, model-based video decoder <NUM> by the present disclosure, and vice versa.

Once ML model-based video decoder <NUM> is fully trained, and the reconstruction quality requirement of ML model-based video encoder <NUM> for the subset is fulfilled, the content-specific information (e.g., decompression data <NUM>) may be stored alongside the subset. If ML model-based video decoder <NUM> wants to decode an image x ∈ <IMG>, the subset specific decompression data <IMG>, in the form of weights, has to be sent only once per subset. A procedure for applying the present microdosing KD approach to image compression with GANs and its extension to video compression using latent space residuals is discussed below.

<FIG> shows a traditional HiFiC architecture <NUM>. Its decoder <NUM> can be divided into three sub-nets: head <NUM> including approximately two million (<NUM>) parameters, residual network (res_blocks) <NUM> including approximately <NUM> parameters, and tail <NUM> including approximately <NUM> parameters. It is noted that the coarse information of an image processed using HiFiC architecture <NUM> is saved in the latent space, and the hallucination of the texture is generated by res_blocks <NUM> of decoder <NUM>. In particular, the size of res_blocks <NUM> is due to the model having been trained on a large (private) dataset, thus such a large size is needed to capture all the textures seen during training.

However, if it is known in advance which images should be compressed (e.g., frames of a video sequence having similar features), it is possible to overfit to that data during encoding and send only the necessary weights to properly decode those images (i.e., decompression data <NUM>). That is what is implemented using the NN architecture disclosed in the present application and described by reference to <FIG>.

According to the exemplary implementation shown by <FIG>, the size of student-decoder <NUM> is significantly reduced relative to that of teacher-decoder <NUM> by training a smaller sub-network, Micro-Residual-Network (Micro-RN) <NUM>, that mimics the behavior of the res_blocks <NUM>, in <FIG>, for a specific subset of images, thereby microdosing the hallucination capability of student-decoder <NUM>. As noted above, as an alternative to a HiFiC architecture, in some implementations the present novel and inventive principles may be applied to substantially any GAN trained network having a similar architecture based on residual blocks. In such GAN trained network implementations, the residual block portion of the GAN trained network decoder could be replaced by Micro-RN <NUM>. It is further noted that student-decoder <NUM> corresponds in general to ML model-based video decoders <NUM> and <NUM>, in <FIG> and <FIG>, and those corresponding features may share any of the characteristics attributed to either corresponding feature may the present disclosure. That is to say, like student-decoder <NUM>, ML model-based video decoders <NUM> and <NUM> may include Micro-RN <NUM>.

According to the exemplary implementation shown in <FIG>, Micro-RN <NUM> is based on degradation-aware (DA) blocks, as known in the art. However, while existing methods that utilize DA blocks typically utilize a kernel prediction network to steer the weights according to a degradation vector, according to the present implementation a different set of weights <IMG> per subset <IMG>. Micro-RN <NUM> is defined by two parameters: Ch, the number of hidden channels, and B, the number of DA Blocks. In one implementation, a <NUM> x <NUM> convolution may be used. Referring to <FIG>, DConv denotes a depthwise convolution. Micro-RN <NUM> is trained with the teacher-student architecture shown in <FIG>, while the head <NUM> and tail <NUM> of student-decoder <NUM> (hereinafter "ML model-based video decoder <NUM>") are pre-trained and borrowed from teacher-decoder <NUM>.

Let x ∈ <IMG> be an image of subset <IMG> and x̃ be the image compressed by the teacher network. According to the present concepts, the following loss function is optimized: <MAT> where x̂ is the output of the student network, MSE (mean squared error) and dp are the distortion losses, and kM and kp are their corresponding weights. The perceptual loss dp = LPIPS is used. As a result, the loss forces ML model-based video decoder <NUM> to generate images that look similar to those generated by teacher-decoder <NUM> and further reduces the perceptual loss to the ground truth image. It is noted that the encoder and the entropy model, which may be modeled using a hyperprior, are frozen. Consequently, compression data <NUM> "<IMG>" only contains the weights of Micro-RN <NUM>. This advantageously leverages the powerful encoder and hyperprior of HiFiC as well as the model's knowledge of the private training data set.

To show the application of KD in neural video compression scenarios, a network such as network <NUM> in <FIG> may be used. As shown in <FIG>, network <NUM> includes two parts: Frame Prediction Network (FPN) <NUM> and Latent Residual Network (LRN) <NUM>. Given a sequence of frames (group of pictures, or GOP) to be encoded x<NUM>,. , xGOP, where x<NUM> is a keyframe (I-frame) and x<NUM>,. ,xGOP are predicted frames (P-frames), the compression of the sequence may work as follows:.

First, the I-frame (x<NUM>) may be compressed using a neural image compression network to generate the encoded latent y<NUM>. Let x̂<NUM> denote the reconstructed frame from the quantized latent y̌<NUM>. Then, for each P-frame, xt+<NUM>, <NUM> ≤ t + <NUM> ≤ GOP: (<NUM>) a temporal prediction, <MAT>, of xt+<NUM> is generated from the previous reconstructed frame, x̂t, using FPN <NUM>. FPN <NUM> works by first computing the optical flow ft+<NUM> between xt+<NUM> and x̂t. (<NUM>) Use the neural motion compression network to generate the encodings and quantized latents w̌t+<NUM> of ft+<NUM>. (<NUM>) Warp x̂t with the decompressed flow f̂t+<NUM>, and then motion compensate it to generate the temporal <MAT>.

To compute the residual between the temporal prediction and the P-Frame, LRN <NUM> is used to: (<NUM>) encode both, the prediction <MAT> and xt+<NUM>, with EI <NUM> (a pre-trained image compression encoder) and (<NUM>) compute the latent residual, xt+<NUM>, between the latents of the P-frame against the predicted frame, <MAT>, which is then quantized and entropy coded with EMI <NUM>. The final compressed bitstream of a GOP is then composed of {ŷ<NUM>, ŵ<NUM>,. , w̌GOP, r̂<NUM>,. , r̂GOP} i.e., latent of the I-frame and the compressed flow fields and latent residuals for each of the P-frames (all quantized and entropy encoded).

In the low bitrate setting, HiFiC would seem like a suitable choice for the neural image compression architecture that could be used together with the above latent space residual framework. As noted above, however, the size of the HiFiC decoder is a limiting factor. Moreover, inference time can be critical in the video where maintaining a decoding frame rate of approximately thirty frames per seconds (<NUM> fps) is often necessary. The microdosing solution disclosed in the present application advantageously advances the state-of-the-art by increasing computational efficiency while reducing inference time. During encoding, the present solution is overfitted to a specific sequence so that the <IMG> only need to be sent once for all the frames of that sequence. The present novel and inventive decoding process then proceeds by receiving and loading sequence-specific Micro-RN weights on ML model-based video decoder <NUM>, which is then fixed during the decode of the sequence. As a result of the small computational overhead imposed by the present microdosing solution, decoding time can advantageously be reduced by <NUM>% while achieving visual that is similar to bigger and slower existing decoders.

The knowledge distillation with microdosing approach described above by reference to <FIG>, <FIG>, <FIG>, and <FIG> will be further described by reference to <FIG> and <FIG>. <FIG> shows flowchart <NUM> presenting an exemplary method of performing microdosing for low bitrate video compression, according to one implementation, while <FIG> shows flowchart <NUM> describing additional actions for extending the method outlined in <FIG>. With respect to the actions described in <FIG> and <FIG>, it is noted that certain details and features have been left out of flowcharts <NUM> and <NUM> in order to not obscure the discussion of the inventive features in the present application.

Referring now to <FIG> in combination with <FIG> and <FIG>, flowchart <NUM> includes receiving uncompressed video sequence <NUM>/<NUM> including multiple video frames (action <NUM>). As shown in <FIG>, uncompressed video sequence <NUM>/<NUM> may be received in action <NUM> by ML model-based video encoder <NUM>/<NUM>. Moreover, and as noted above by reference to <FIG>, ML model-based video encoder <NUM>/<NUM> may be stored in system memory <NUM>. Thus, uncompressed video sequence <NUM>/<NUM> may be received in action <NUM> by ML model-based video encoder <NUM>/<NUM> executed by processing hardware of <NUM> of system <NUM>.

Flowchart <NUM> further includes determining, from among the plurality of video frames, a first video frame subset and a second video frame subset (action <NUM>). In some implementations, the determination of the first video frame subset and the second video frame subset, in action <NUM> may be based on similarity and dissimilarity among the video frames included in uncompressed video sequence <NUM>/<NUM>. In other words, in some implementations, the first video frame subset determined in action <NUM> may include video frames that are visually similar to one another, while the second video frame subset may include other video frames that are more visually similar to one another than to the video frames included in the first video frame subset. Continuing to refer to <FIG> and <FIG> in combination, determination of the first video frame subset and the second video frame subset in action <NUM> may be performed by ML model-based video encoder <NUM>/<NUM>, executed by processing hardware of <NUM> of system <NUM>.

Flowchart <NUM> further includes encoding the first video frame subset determined in action <NUM> to produce first compressed video frame subset <IMG> (action <NUM>). As described above, the encoding of the first video frame subset to produce first compressed video frame subset <IMG>, in action <NUM>, may be performed by ML model-based video encoder <NUM>/<NUM>, executed by processing hardware of <NUM> of system <NUM>.

Flowchart <NUM> further includes identifying first decompression data <IMG> for first compressed video frame subset <IMG> (action <NUM>). In some implementations, identifying first decompression data <IMG> comprises overfitting first decompression data <IMG> during the encoding of the first video frame subset in action <NUM>. That is to say, in some implementations, identification of first decompression data <IMG> in action <NUM>, may be performed in parallel, i.e., substantially concurrently, with the encoding of the first video frame subset to produce first compressed video frame subset <IMG> in action <NUM>. As described above, identifying first decompression data <IMG> for first compressed video frame subset <IMG>, in action <NUM>, may be performed by ML model-based video encoder <NUM>/<NUM>, executed by processing hardware of <NUM> of system <NUM>.

Flowchart <NUM> further includes encoding the second video frame subset determined in action <NUM> to produce second compressed video frame subset <IMG> (action <NUM>). As described above, the encoding of the second video frame subset to produce second compressed video frame subset <IMG>, in action <NUM>, may be performed by ML model-based video encoder <NUM>/<NUM>, executed by processing hardware of <NUM> of system <NUM>.

Flowchart <NUM> further includes identifying second decompression data <IMG> for second compressed video frame subset <IMG> (action <NUM>). In some implementations, identifying second decompression data <IMG> comprises overfitting second decompression data <IMG> during the encoding of the second video frame subset in action <NUM>. That is to say, in some implementations, identification of second decompression data <IMG> in action <NUM>, may be performed in parallel, i.e., substantially concurrently, with the encoding of the second video frame subset to produce second compressed video frame subset <IMG> in action <NUM>. As described above, identifying first decompression data <IMG> for second compressed video frame subset2, in action <NUM>, may be performed by ML model-based video encoder <NUM>/<NUM>, executed by processing hardware of <NUM> of system <NUM>.

With respect to first and second decompression data <IMG> and <IMG>, it is noted that those data are specific to the respective compressed video frame subsets they accompany. Thus, first decompression data <IMG> is specific to decoding first compressed video frame subset <IMG> but not second compressed video frame subset <IMG>, and second decompression data <IMG> is specific to decoding second compressed video frame subset <IMG> but not first compressed video frame subset <IMG>.

It is further noted that although flowchart <NUM> depicts actions <NUM> and <NUM> as following actions <NUM> and <NUM>, that representation is provided merely by way of example. In some other implementations, actions <NUM> and <NUM> may be performed in parallel prior to actions <NUM> and <NUM>, which, in some implementations, may also be performed in parallel. Thus, in some implementations, action <NUM> and <NUM> may be performed in parallel with action <NUM> and <NUM>.

In some implementations, the method outlined by flowchart <NUM> may conclude with action <NUM>. However, in other implementations, that method may be extended by one or more of the actions described by flowchart <NUM>, in <FIG>. Referring now to <FIG> in combination with <FIG> and <FIG>, flowchart <NUM> includes transmitting, to ML model-based video decoder <NUM>/<NUM>/<NUM>, first compressed video frame subset <IMG>, second compressed video frame subset <IMG>, first decompression data <IMG>, and second decompression data<IMG> (action <NUM>). As shown in <FIG>, first compressed video frame subset <IMG>, second compressed video frame subset <IMG>, first decompression data <IMG>, and second decompression data <IMG> may be transmitted to ML model-based video decoder <NUM>/<NUM>/<NUM>, in action <NUM>, by ML model-based video encoder <NUM>/<NUM>, executed by processing hardware <NUM> of system <NUM>, via communication network <NUM> and network communication links <NUM>.

Flowchart <NUM> further includes receiving first compressed video frame subset <IMG>, second compressed video frame subset <IMG>, first decompression data <IMG>, and second decompression data <IMG> (action <NUM>). As shown in <FIG>, first compressed video frame subset <IMG>, second compressed video frame subset <IMG>, first decompression data <IMG>, and second decompression data <IMG> may be received in action <NUM> by ML model-based video decoder <NUM>/<NUM>/<NUM>. In some implementations, ML, model-based video decoder may be executed by user system processing hardware <NUM>. However, and as noted above by reference to <FIG>, in some implementations, user system may <NUM> may be a dumb terminal peripheral component of system <NUM>. In those latter implementations, ML, model-based video decoder <NUM>/<NUM>/<NUM> is included as a feature of system <NUM>, and may be executed to perform action <NUM> by processing hardware of <NUM> of system <NUM>.

Flowchart <NUM> further includes decoding first compressed video frame subset<IMG> using first decompression data <IMG> (action <NUM>). As described above, the decoding of first compressed video frame subset <IMG> using first decompression data <IMG>, in action <NUM>, may be performed by ML model-based video decoder <NUM>/<NUM>/<NUM>, executed by user system processing hardware <NUM>, or by processing hardware of <NUM> of system <NUM>.

In some implementations, as noted above, ML model-based video decoder <NUM>/<NUM>/<NUM> may include an NN, such as a MicroRN for example. In implementations in which ML model-based video decoder <NUM>/<NUM>/<NUM> includes a MicroRN, first decompression data <IMG> may contain only the weights of that MicroRN for use in decoding first compressed video frame subset <IMG>. Moreover, in some implementations, first decompression data <IMG> may be received once, and only once, for decoding first compressed video frame subset <IMG> in its entirety.

Flowchart <NUM> further includes decoding second compressed video frame subset<IMG> using second decompression data <IMG> (action <NUM>). As described above, the decoding of second compressed video frame subset <IMG> using first decompression data <IMG>, in action <NUM>, may be performed by ML model-based video decoder <NUM>/<NUM>/<NUM>, executed by user system processing hardware <NUM>, or by processing hardware of <NUM> of system <NUM>.

In implementations in which ML model-based video decoder <NUM>/<NUM>/<NUM> includes a MicroRN, second decompression data <IMG> may contain only the weights of that MicroRN for use in decoding first compressed video frame subset <IMG>, Moreover, in some implementations, second decompression data <IMG> may be received once, and only once, for decoding second compressed video frame subset <IMG> in its entirety.

It is noted that although flowchart <NUM> depicts action <NUM> as following action <NUM>, that representation is provided merely by way of example. In some implementations, the decoding of first compressed video frame subset <IMG> using first decompression data <IMG>, in action <NUM>, and the decoding of second compressed video frame subset <IMG> using second decompression data <IMG>, in action <NUM>, may be performed in parallel, i.e., substantially concurrently.

Regarding the combination of ML model-based video encoder <NUM>/<NUM> and ML model-based video decoder <NUM>/<NUM>/<NUM>, it is noted that ML model-based video encoder <NUM>/<NUM> may be implemented as a HiFiC encoder, while ML model-based video decoder <NUM>/<NUM>/<NUM> is configured so as to have fewer parameters, such as ten time fewer parameters for example, than a HiFiC decoder, i.e., a large decoder that does not use first decompression data <IMG>, and second decompression data <IMG>. Moreover, ML model-based video decoder <NUM>/<NUM>/<NUM> may be configured so as to achieve a faster decoding time, such as a fifty percent (<NUM>%) faster decoding time for example, than a HiFiC decoder, i.e., a large decoder that does not use first decompression data <IMG>, and second decompression data <IMG>.

With respect to the actions described in <FIG> and <FIG>, it is noted that in various implementations, actions <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> (hereinafter "actions <NUM>-<NUM>") of flowchart <NUM>, or actions <NUM>-<NUM> and action <NUM> of flowchart <NUM>, or actions <NUM>-<NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, may be performed as automated processes from which human participation may be omitted.

Claim 1:
A method for use by a system (<NUM>) including a machine learning, ML model-based video encoder (<NUM>, <NUM>); and an ML model-based video decoder (<NUM>, <NUM>, <NUM>) comprising a degradation-aware block based Micro-Residual-Network, MicroRN, (<NUM>) defined by a number of hidden channels and a number of degradation-aware blocks of the MicroRN (<NUM>), the MicroRN (<NUM>) being used as a residual network in a generative adversarial network, GAN, trained decoder, the method comprising:
receiving, by the ML model-based video encoder (<NUM>, <NUM>), an uncompressed video sequence (<NUM>, <NUM>) including a plurality of video frames;
determining, by the ML model-based video encoder (<NUM>, <NUM>) from among the plurality of video frames, a first video frame subset and a second video frame subset;
encoding, by the ML model-based video encoder (<NUM>, <NUM>), the first video frame subset to produce a first compressed video frame subset;
identifying, by the ML model-based video encoder (<NUM>, <NUM>), a first decompression data for the first compressed video frame subset;
encoding, by the ML model-based video encoder (<NUM>, <NUM>), the second video frame subset to produce a second compressed video frame subset;
identifying, by the ML model-based video encoder (<NUM>, <NUM>), a second decompression data for the second compressed video frame subset; and
transmitting, by the ML model-based video encoder (<NUM>, <NUM>) to the ML model-based video decoder (<NUM>, <NUM>, <NUM>), the first compressed video frame subset, the second compressed video frame subset, the first decompression data, and the second decompression data;
receiving, by the ML model-based video decoder (<NUM>, <NUM>, <NUM>), the first compressed video frame subset, the second compressed video frame subset, the first decompression data, and the second decompression data;
decoding, by the ML model-based video decoder (<NUM>, <NUM>, <NUM>), the first compressed video frame subset using the first decompression data; and
decoding, by the ML model-based video decoder (<NUM>, <NUM>, <NUM>), the second compressed video frame subset using the second decompression data.