Parameter map for machine-learned video compression

A compression system trains a machine-learned compression model that includes components for an encoder and decoder. In one embodiment, the compression model is trained to receive parameter information on how a target frame should be encoded with respect to one or more encoding parameters, and encodes the target frame according to the respective values of the encoding parameters for the target frame. In particular, the encoder of the compression model includes at least an encoding system configured to encode a target frame and generate compressed code that can be transmitted by, for example, a sender system to a receiver system. The decoder of the compression model includes a decoding system trained in conjunction with the encoding system. The decoding system is configured to receive the compressed code for the target frame and reconstruct the target frame for the receiver system.

BACKGROUND

This invention generally relates to encoding and decoding videos, and more specifically to encoding and decoding videos with respect to one or more encoding parameters using a machine learned compression model.

Various online systems transmit information to and from one another over a network. The information may be in the form of images, videos that include a sequence of frames, or text. A sender typically encodes the information using an encoder into a compressed form, and the compressed information is transmitted to the receiver. The receiver can then decode the compressed information using a decoder to reconstruct the original information. A video typically includes a sequence of image frames that capture the motion of objects and background of a scene that occur due to movement of the camera or movement of the objects themselves. Compared to other types of information, video compression can be challenging due to large file size and issues such as video and audio synchronization. Video compression for lower-power devices, such as smartphones, can be even more challenging.

SUMMARY

A compression system trains a machine-learned compression model that includes components for an encoder and decoder. In one embodiment, the compression model is trained to receive parameter information on how a target frame should be encoded with respect to one or more encoding parameters, and encodes the target frame according to the respective values of the encoding parameters for the target frame. In particular, the encoder of the compression model includes at least an encoding system configured to encode a target frame and generate compressed code that can be transmitted by, for example, a sender system to a receiver system. The decoder of the compression model includes a decoding system trained in conjunction with the encoding system. The decoding system is configured to receive the compressed code for the target frame and reconstruct the target frame for the receiver system.

The encoding parameter controls one or more properties of how a target frame is encoded and decoded using the encoder and decoder of the machine-learned compression model. The encoding parameter may relate to, for example, the reconstruction quality or codelength of the target frame or how a target frame is encoded with reference to other frames in the video. In one instance, the encoding parameter is a rate parameter that indicates the desired bitrate of the compressed code for a target frame. The bitrate represents how many bits of information are processed per unit time to transmit the compressed code for the target frame. Thus, a high bitrate indicates that a higher degree of information is used to encode the target frame for a higher reconstruction quality, while a low bitrate indicates that a lower degree of information is used to encode the target frame for a lower reconstruction quality.

In another instance, the encoding parameter is a frame parameter that indicates the type of information used to encode the target frame as a I-frame, a P-frame, or a B-frame. Specifically, a target frame encoded as an I-frame may be encoded and decoded without reference to previously reconstructed frames in the video. On the other hand, a target frame encoded as a P-frame or B-frame may be encoded and decoded with reference to one or more previously reconstructed frames in the video. In particular, a target frame encoded as a P-frame is encoded with respect to changes from a previously reconstructed frame that is placed temporally before or after the target frame in the video. A target frame encoded as a B-frame, on the other hand, is encoded bi-directionally with respect to changes from one or more previously reconstructed frames that are placed temporally before and after the target frame in the video.

During the training process, the compression system iteratively trains the weights for the compression model by applying the compression model to encoding information and parameter information for the training frame to generate compressed code and a reconstructed version of the training frame. The parameter information for the training frame indicates a value for the encoding parameter for the training frame. The compression system updates the weights of the compression model to reduce a loss function. In one instance, the loss function includes a reconstruction loss indicating a difference between the training frame and the reconstructed frame, and a codelength loss indicating the codelength of the compressed code and that depends on the value of the encoding parameter. This process is repeated until a convergence criteria is reached.

By training compression models that incorporate parameter information, the resulting encoders and decoders can address a wide range of values for the encoding parameter by simply specifying the desired value as input to the encoder and the decoder during the deployment process. Responsive to receiving a desired value for a target frame, the encoder of the compression model is trained to output compressed code that, when reconstructed by the decoder, generates a high-quality reconstructed version of the target frame given the constraints or conditions set by the value of the encoding parameter. Rather than having a separate encoder-decoder for each value of the encoding parameter that may lead to inefficient and slow memory loading and unloading, information can be encoded and decoded for different values of the encoding parameter with significant weight sharing within the compression model. This allows for efficient encoding and decoding of information during deployment.

The figures use like reference numerals to identify like elements. A letter after a reference numeral, such as “110A,” indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as “110,” refers to any or all of the elements in the figures bearing that reference numeral (e.g. “client device110” in the text refers to reference numerals “client device110A” and/or “client device110B” in the figures).

DETAILED DESCRIPTION

Overview

FIG.1Ais a block diagram of a system environment100A including a compression system130, a sender system110, and a receiver system140, in accordance with an embodiment. As described in further detail, the sender system110includes an encoder, and the receiver system140includes a decoder. The encoder and decoder may be trained and provided by the compression system130. In alternative configurations, different and/or additional components may be included in the system environment100A.

The compression model130trains a compression model to generate an encoder and a decoder that can be deployed by the sender system110and the receiver system140. Specifically, the encoder can be used by a sender system110to encode information, such as images, videos, and text, into a compressed form. The compressed information can be transmitted to a receiver system140, for example, in the form of a bitstream. The decoder can be used by the receiver system140to reconstruct the original information based on the bitstream provided by the sender system110. Specifically, after training the weights of the compression model, certain components of the compression model make up the encoder and the decoder.

In one particular embodiment referred throughout the specification, the information to be encoded and decoded is a video that includes a sequence of frames. The sequence of frames are typically images of the same dimensionality, and capture motions that occur within a scene over time due to movement of the camera or the movement of the objects themselves. Specifically, motions are animated as the positions of pixels that correspond to a particular portion of the scene change throughout the sequence of frames. For example, the movement of a pedestrian walking in the left direction may be animated as the position of the pedestrian gradually shifts toward the left side of the frames. However, it is also appreciated that in other embodiments, the information can be data other than videos, such as a sequence of texts in a document.

In general, video frames, or other types of data derived from video frames may be represented as a tensor. As defined herein, a tensor denotes a multi-dimensional array of values having a width, height, and one or more channels across the depth of the tensor. Thus, a video frame in RGB color may be represented as a tensor with 3 channels, each channel including pixel intensity values for a respective color. As another example, a video frame in grayscale may be represented as a tensor with 1 channel representing pixel intensity values in grayscale. As defined herein, a “position” of an element or pixel in a tensor denotes the position within the width and height of the tensor. Thus, an element at a particular position of a tensor may have a set of values associated with the element that corresponds to the number of channels of the tensor.

In one embodiment, the compression model is trained to receive encoding information for a target frame and parameter information on how the target frame should be encoded with respect to one or more encoding parameters. In particular, the encoder of the compression model includes at least an encoding system configured to receive the encoding information and parameter information for a target frame and generate compressed code that can be transmitted by, for example, the sender system110to the receiver system140. Thus, the encoding information may include any type of information that can be used to encode the target frame into compressed code. For example, the encoding information may include image data of target frame or previously reconstructed frames.

The decoder of the compression model includes a decoding system trained in conjunction with the encoding system. The decoding system is configured to receive decoding information and the parameter information and generate a reconstructed version of the target frame for the receiver system140. The decoding information may include any type of information that can be used to decode the target frame from compressed code. For example, the decoding information may include one or more compressed codes for decoding the target frame.

The encoding parameter controls one or more properties of how a target frame is encoded and decoded using the encoder and decoder of the machine-learned compression model. The encoding parameter may relate to, for example, the reconstruction quality or codelength of the target frame or how a target frame is encoded with reference to other frames in the video. In one instance, the encoding parameter is a rate parameter that indicates the desired bitrate of the compressed code for a target frame. The bitrate represents how many bits of information are processed per unit time to transmit the compressed code for the target frame. Thus, a high bitrate indicates that a higher degree of information is used to encode the target frame for a higher reconstruction quality, while a low bitrate indicates that a lower degree of information is used to encode the target frame for a lower reconstruction quality.

In another instance, the encoding parameter is a frame parameter that indicates the type of information used to encode the target frame as a I-frame, a P-frame, or a B-frame. Specifically, a target frame encoded as an I-frame may be encoded and decoded without reference to previously reconstructed frames in the video. On the other hand, a target frame encoded as a P-frame or B-frame may be encoded and decoded with reference to one or more previously reconstructed frames in the video. In particular, a target frame encoded as a P-frame is encoded with respect to changes from a previously reconstructed frame that is placed temporally before or after the target frame in the video. A target frame encoded as a B-frame, on the other hand, is encoded bi-directionally with respect to changes from one or more previously reconstructed frames that are placed temporally before and after the target frame in the video.

In one embodiment, the parameter information for a target frame is represented as a parameter map that is one or more vectors or scalars used to input a desired value for the encoding parameter to the compression model. In one instance, the parameter map is a one-hot encoded vector, where each element in the vector represents a respective value for the encoding parameter. An element in the vector is a non-zero value (e.g., value of one) if it corresponds to the desired value for the encoding parameter for the target frame and zero otherwise. For example, a parameter map may encode five bitrate levels, and be represented by a five-element vector where each element corresponds to a bitrate level. Thus, a parameter map specifying encoding and decoding at a fourth bitrate level may be represented by the vector [0 0 0 1 0]. In one instance, the parameter map may be input to the compression model by concatenating the parameter map to input tensors to the compression model. In another instance, the parameter map is used to select a respective subset of weights of the compression model that are dedicated for a respective value of the encoding parameter represented by the parameter map.

During the training process, the compression system130iteratively trains the weights for the compression model by applying the compression model to encoding information and parameter information for the training frame to generate compressed code and a reconstructed version of the training frame. The parameter information for the training frame indicates a respective value for the encoding parameter that the training frame is configured to train for during the training process. The compression system130updates weights of the compression model to reduce a loss function. In one instance, the loss function includes a reconstruction loss indicating a difference between the training frame and the reconstructed frame, and a codelength loss indicating the codelength of the compressed code and that depends on the value of the encoding parameter. This process is repeated until a convergence criteria is reached.

By training compression models that incorporate parameter information, the resulting encoders and decoders can address a wide range of values for the encoding parameter by simply specifying the desired value as input to the encoder and the decoder during the deployment process. Responsive to receiving a desired value for a target frame, the encoder of the compression model is trained to output compressed code that when reconstructed by the decoder generates a high-quality reconstructed version of the target frame given the constraints or conditions set by the value of the encoding parameter. Rather than having a separate encoder-decoder for each value of the encoding parameter that may lead to inefficient and slow memory loading and unloading, information can be encoded and decoded for different values of the encoding parameter with significant weight sharing within the compression model. This allows for efficient encoding and decoding of information during deployment.

In one embodiment, the compression system130may train multiple compression model architectures that correspond to multiple types of encoder-decoders. Each compression model may differ in the type of information and method used for encoding and decoding. In one instance, the compression system130may train multiple compression models depending on whether the encoder-decoder is used to encode and decode a I-frame, a P-frame, or a B-frame. For example, the compression system130may train a compression model to encode and decode I-frames that are encoded without reference to previously reconstructed frames. In such an example, the encoder includes an encoding system coupled to receive encoding information including a target frame and parameter information for the target frame and generate compressed code. The decoder includes a decoding system coupled to receive decoding information including the compressed code and the parameter information and generate a reconstructed frame of the target frame.

The compression system130may train another compression model to encode and decode P-frames and B-frames that are encoded with reference to one or more previously reconstructed frames. In such an example, the encoder may include both the encoding system and the decoding system. The encoding system is coupled to receive encoding information including a target frame and previously reconstructed frames and parameter information for the target frame and generate compressed code. The decoding system is coupled to receive decoding information including the compressed code and information obtained from the previously reconstructed frames and parameter information and generate the reconstructed version of the target frame. The decoder includes the decoding system. The reconstructed target frame is then used as the previously reconstructed frame for encoding and decoding the next frame in the video. The compression system130may provide encoder-decoders to the sender system110and the receiver system140after the training process is completed.

The sender system110is a system for encoding information using an encoder. The sender system110may receive the components of the encoder from the compression system130after the training process of the compression model. The sender system110provides the encoded information to the receiver system140. For example, the sender system110may be a server of a video streaming website that streams video content to users. The encoded information may be transmitted in the form of a bitstream that is a sequence of bits. The receiver system140decodes the bitstream to reconstruct the encoded information using a decoder. For example, the receiver system140may be a client device of a user of the video streaming service that reconstructs one or more videos from the sender system110. The receiver system140may receive the components of the decoder from the compression system130.

During deployment, the sender system110encodes a set of frames in a video by applying the encoder to encoding information and parameter information for the frames in the video to generate compressed code that can be used by the decoder to reconstruct the frames of the video. Specifically, for a target frame in the video, the sender system110determines a respective value of the encoding parameter for the target frame. The sender system110applies the encoder to encoding information and parameter information for the target frame to generate compressed code. In some instances, for example, when the target frame is encoded as a P-frame or a B-frame, the sender system110further generates the reconstructed version of the target frame. The reconstructed frame is used as the previously reconstructed frame and as the encoding information for the next frame in the video.

In one instance, the sender system110assigns the respective value of the encoding parameter to each target frame based on any encoding constraints that the sender system110is subject to. For example, the sender system110may be required to transmit a video within a total file size. The sender system110may allocate the desired bitrates for the frames such that the total file size of the transmitted bits is within this constraint. The sender system110specifies the parameter information for a target frame according to the allocated bitrate level for each target frame.

The receiver system140receives the compressed code from the sender system110and decodes the set of frames for the video by applying the decoder to decoding information and parameter information for the frames in the video. Specifically, for a target frame in the video, the receiver system140may receive information in the transmitted information regarding the respective value of the encoding parameter the target frame was encoded with. The receiver system140applies the decoder to decoding information and parameter information for the target frame to generate the reconstructed version of the target frame. In some instances, for example, when the target frame is encoded as a P-frame or a B-frame, the reconstructed target frame is used as the previously reconstructed frame and as the decoding information for the next frame in the video.

FIG.1Bis a block diagram of a system environment100B including a sender system110and a receiver system140, in accordance with another embodiment. The system environment100B shown inFIG.1Bincludes the sender system110, the receiver system140, and additionally an encoded information data store150that can be used to store encoded information.

In the system environment100B ofFIG.1B, the sender system110encodes information and stores the encoded information in a data store150. The receiver system140may retrieve the encoded information from the data store150to decode and reconstruct the original information. The data store150may represent electronic storage in the form of magnetic disks, optical disks, RAM disk, analog disks, non-volatile memory, volatile memory, and the like. The electronic storage may be a standalone storage device, or may be electronic storage integrated within a device, such as a computer or a camera. For example, responsive to a user recording a video, a sender system110within the camera may encode the video and store the encoded video the memory of the camera. Responsive to a request to access the video, the receiver system140retrieves the encoded information from memory and decodes the video to present the video to the user.

Training and Deployment Process of Compression Model with Parameter Map

FIG.2illustrates a training process for a compression model200coupled to receive a parameter map for rate parameters, in accordance with an embodiment. As shown inFIG.2, the compression model200includes an encoding system210and a decoding system220among other components.

Specifically, the compression model200shown inFIG.2is configured to process a sequence of data, which can be a sequence of image frames for a video, and a parameter map encoding values for rate parameters to train the parameters of the compression model200. The compression model200may perform one or more iterations at a given timestep, and at each iteration, a target frame may be processed by the compression model200to generate a reconstructed frame for the target frame. The outputs of the compression model200are used to train the parameters of the compression model200, such that the components of the compression model200can be deployed as an encoder and/or a decoder. While the example compression model200shown inFIG.2is coupled to process one target frame at a time for ease of explanation, it should be appreciated that in other embodiments, the “target frame” can be a single frame or multiple frames, such as a group of consecutive frames, depending on the data to be encoded and decoded.

Moreover, whileFIG.2illustrates a general training process of a compression model200for ease of explanation, in practice, the compression system130may generate and train multiple compression models200that each have a different architecture depending on whether the compression model200is configured to encode and decode information as I-frames, P-frames, and/or B-frames. As described below with respect toFIGS.4A-4C, a detailed example architecture of a compression model200for processing I-frames is illustrated with respect toFIG.4A, and a detailed example architecture of a compression model200for processing P-frames or B-frames are illustrated with respect toFIGS.4B and4C. However, it is appreciated that in other embodiments, the compression model200may incorporate any other type of architectures that can be configured to receive a parameter map specifying values for encoding parameters.

The encoding system210of a compression model200shown inFIG.2is coupled to receive encoding information and the parameter information in the form of a parameter map and generate compressed code. In one embodiment, the encoding information includes image data for the target frame. The decoding system220is coupled to receive decoding information and the parameter map and generate a reconstructed version of the target frame. The decoding information includes the compressed code for the target frame. The target frame is encoded into the compressed code such that the compressed code is reconstructed by the decoding system220subject to the conditions encoded in the value of the encoding parameter in the parameter map.

In one implementation, the compressed code output by the encoding system210is generated at a codelayer that is a bottleneck layer with lower dimensionality (or lower number of nodes) than the remaining layers of the encoding system210and the decoding system220. A compressed code generated at a codelayer may be generated by applying a set of weights of the encoding system210. In particular, the compression model200may be structured such that compressed codes for at least two target frames with different values of the encoding parameter are generated at the same codelayer by applying a same subset of weights in the encoding system210. Since the compression model200is configured to receive values of the encoding parameter as a parameter map, the compressed codes for target frames with different encoding parameter values can be generated at the same codelayer instead of, for example, having separate codelayers for each bitrate level.

In another embodiment, when encoding and decoding is performed with reference to previously reconstructed frames, the encoding information may include image data from one or more previously reconstructed frames in addition to the image data for the target frame. The decoding information may include information obtained from the one or more previously reconstructed frames in addition to the compressed code. Specifically, the encoding information and the decoding information may be represented as tensors that are input to various components of the encoding system210and the decoding system220, respectively.

In one embodiment, the parameter map is encoded into the compressed code, and the decoding system220extracts the parameter map from the compressed code. However, embodiments are not limited hereto, and the parameter information can be used by the compression model200to encode and decode by other configurations. For example, in another embodiment, the parameter map may be provided only to the encoding system210and not to the decoding system220. In yet another embodiment, the parameter map may be determined in advance by the decoding system220, or it may be derived from previously reconstructed frames and therefore, the parameter map does not need to be encoded in the compressed code for the decoding system220to obtain that information. In another embodiment, the decoding system220is configured to predict the parameter map for a target frame from previously reconstructed frames and the delta between the actual parameter map and the predicted parameter map may be included in the compressed code.

In the example compression model200shown inFIG.2, the parameter map encodes different bitrate levels for the compressed code. As an example, the parameter map may be represented by a one-hot encoded vector with five elements to encode five levels, where a higher level corresponds to a higher bitrate. As described in conjunction withFIGS.1and1B, the bitrate represents how many bits of information are processed per unit time to transmit the compressed code for the target frame. Thus, a bitrate level may represent a target codelength for the compressed code, and a high bitrate level may correspond to a longer codelength, while a low bitrate level may correspond to a shorter codelength for the compressed code.

In one embodiment, one or more of the components the compression model200are configured as neural network architectures, such as artificial neural networks (ANN), convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), long short-term memory (LSTM) networks, transformer networks, and the like. Specifically, any part of the encoding system210and the decoding system220may be configured as any of these neural network models. In one embodiment, the encoding system210and the decoding system220may be configured as autoencoders described in U.S. patent application Ser. No. 15/844,424, U.S. patent application Ser. No. 15/844,449, 15/844,447, or 15/844,452, all of which are incorporated by reference herein in their entirety.

During the training process, the compression system130obtains a set of training videos. Each training video may include a set of training frames that represent a scene over time. For example, a training video may capture a car moving down a street or a pedestrian walking across a sidewalk. The training videos may differ with each other with respect to the content included in the videos. In one instance, when the parameter map represents bitrate levels, the compression system130may assign a bitrate level to each training frame. The bitrate levels may be assigned randomly to each frame or according to some predetermined criterion by the compression system130. The compression system130may also assign a frame label to each training frame indicating whether the training frame is to be processed as an I-frame, a P-frame, or a B-frame, such that a compression model200with the appropriate architecture can be applied to each training frame according to the frame label.

Returning toFIG.2, during the training process, the compression model130iteratively alternates between performing a forward pass step and a backpropagation step for training frames in a video. Specifically, during the forward pass step for a t-th iteration, the compression system130applies the encoding system210to encoding information and the parameter map ctfor a target frame xtin a training video to generate compressed code. The encoding information may include the target frame xtand/or one or more previously reconstructed frames {circumflex over (x)}t-1. In one instance, the compressed code may be further configured to include the parameter map or information obtained from the parameter map when the encoding system210is configured to propagate the parameter map to the decoding system220via the compressed code. The compression system130applies the decoding system220to decoding information and the parameter map for the target frame xtto generate the reconstructed version of the target frame {circumflex over (x)}t. The decoding information may include the compressed code and information obtained from the previously reconstructed frames {circumflex over (x)}t-1.

In the example shown inFIG.2, the parameter map ctindicates that the bitrate level for the target frame xtis the fourth bitrate level and is represented by the vector [0 0 0 1 0]. The parameter map ctis tiled and concatenated to the RGB image data tensor for the target frame xt. The parameter map ctcan also be tiled and concatenated to other input tensors that are generated within the compression model200. However, the parameter map inFIG.2is just one implementation of how parameter information can be incorporated in the compression model200, and it is appreciated that the parameter information can be represented in other types of formats or data structures. For example, in another embodiment, the parameter map may be provided as a scalar number instead of a one-hot encoded vector. As another example, as described in more detail below in conjunction with the section titled “Bitrate Level Interpolation,” the parameter information may be an arbitrary vector or a number which is then used to interpolate one-hot encoded vectors. The compression system130may store the reconstructed frame {circumflex over (x)}tfor the target frame in a frame data store270. If necessary, a frame updater280may retrieve the reconstructed frame {circumflex over (x)}tfrom the frame data store270to use as the previously reconstructed frame for a next frame in the video.

The compression system130determines a loss function based on the results of the forward pass step. During the backpropgation step, the compression system130backpropagates one or more error terms obtained from the loss function to update the weights of the compression model200. Specifically, the compression system130may update the weights of the neural network portions of the compression model200. In one instance, the error terms are the gradient of the loss function taken with respect to the weights of the compression model200, and the weights are updated to reduce the loss function. The compression system130repeats the forward pass step and the backpropagation step for multiple iterations using new training frames until a convergence criteria for the weights of the compression model200are reached, at which point the training process is completed.

In one embodiment, the loss function includes a combination of a reconstruction loss250and a codelength loss260. The reconstruction loss250indicates a difference between target frames xtand reconstructed frames {circumflex over (x)}t, and thus measures the quality of reconstruction. In one instance, the reconstruction loss250is given by:

ℒr⁡(xt,x^t;θ)=1T⁢∑T⁢μt(l)⁢D⁡(xt,x^t(l))
where l indicates the bitrate level assigned to the target frame xt, D(⋅) is a distortion metric indicating a difference between the target frame xtand the reconstructed frame {circumflex over (x)}t(l), θ indicates the collective set of parameters for the compression model200, T is the number of frames in a training batch, μtis a dynamically chosen weight during the training process that emphasizes bitrate levels that need higher training accuracy, and Lrindicates the reconstruction loss.

The codelength loss260indicates a length of the compressed code generated by the encoding system210during the forward pass step. In one instance, the compression system130assigns a regularization weight λ(l)to each bitrate level l represented in the parameter map, and the codelength loss260for a target frame xtassigned to a bitrate level l is determined as the codelength of the compressed code weighted by the regularization weight λ(l)for the bitrate level l. In one instance, the codelength loss260is given by:

ℒent⁡(xt,x^t;θ)=1T⁢∑T⁢λreg(l)⁢Rt(l)
where Rt(l)is the codelength for the compressed code for target frame xtassigned to bitrate level l, and λ(l)is the regularization weight for the bitrate level l. In particular, the regularization weight λ(l)assigned to a bitrate level l may decrease as the bitrate level l increases, and increase as the bitrate level l decreases.

Thus, by reducing a loss function that is a combination of the reconstruction loss250and the codelength loss260, the compression system130can train components of the compression model130to increase reconstruction quality for the reconstructed frame while reducing the codelength of the compressed code according to a specified bitrate level. Specifically, a target frame xtassigned to a lower bitrate level will have a higher value of the regularization weight λ(l), such that the relative weight of the codelength loss260to the reconstruction loss250will be higher than a target frame assigned to a high bitrate level. This trains the encoding system210to generate compressed codes with shorter codelength while reducing the reconstruction loss250.

FIG.3illustrates a deployment process for the compression model, in accordance with an embodiment. The compression system130provides components of the encoder to the sender system110and the components of the decoder to the receiver system140. In the example shown inFIG.3, the sender system110includes an encoder including a trained encoding system310and a trained decoding system320. The receiver system140includes a decoder including the trained decoding system320. The components shown inFIG.3may be identical or substantially similar to those shown in the compression model200ofFIG.2, but may be provided to the sender system110and the receiver system140by the compression system130after the training process of the compression model200.

During deployment, the sender system110iteratively encodes a sequence of image frames for a video using the encoder. Specifically,FIG.3illustrates the encoding process for the t-th target frame in a video. The sender system110determines a value for the encoding parameter for a target frame. The sender system110applies the encoding system310to a target frame, one or more previously reconstructed frames, and a parameter map for the target frame to generate compressed code. As described above, the parameter map may be encoded into the compressed code, such that the decoding system can extract the parameter map from the compressed code. The sender system110applies the decoding system320to the compressed code, information obtained from the one or more previously reconstructed frames, and the parameter map to generate the reconstructed frame. The reconstructed frame is stored in the frame data store370and is updated as the previously reconstructed frame for a next frame. The sender system110transmits the compressed code (bolded inFIG.3) to the receiver system140. This process may be repeated for the next frame onward.

The receiver system140iteratively decodes the sequence of images using the transmitted codes from the sender system110. The receiver system140receives the code from the sender system110. The receiver system140applies the decoding system320to the compressed code, information obtained from the one or more previously reconstructed frames, and the parameter map to generate the reconstructed frame for the target frame. The parameter map may be extracted from the compressed code. The reconstructed frame is stored in the frame data store370and is updated as the previously reconstructed frame for a next frame. The receiver system140repeats this process for the next frame as the compressed code for the next frame is transmitted from the sender system110.

Architecture of Compression Model with Autoencoder

FIG.4Aillustrates a detailed architecture for a compression model200A for processing I-frames, in accordance with an embodiment. The compression model200A shown inFIG.4Aincludes a frame encoder412and a frame decoder414. Thus, the encoding system210may include the frame encoder412, and the decoding system220may include the frame decoder414. The frame encoder412and the frame decoder414may be configured as a neural network model, as described above in conjunction withFIG.2. The frame encoder412is coupled to receive a target frame and a parameter map and generate compressed code. The framed decoder414is coupled to receive the compressed code and generate a reconstructed target frame. As previously described with reference toFIG.2, the compression model200A shown inFIG.4Ais configured to encode and decode a frame without reference to one or more previously reconstructed frames in the video.

During the forward pass step of the training process, the compression system130applies the frame encoder412to the target frame xtand the parameter map for the target frame xtof a training video to generate the compressed code for the target frame xt. The compression system130applies the frame decoder414to the compressed code to generate the reconstructed frame {circumflex over (x)}t. The compression system130determines the reconstruction loss250and the codelength loss260based on the reconstructed frame {circumflex over (x)}tand the length of the compressed code for the bitrate level assigned to the target frame xt. The compression system130backpropagates one or more error terms from the loss function to update the weights of the frame encoder412and the frame decoder414. This process is repeated until a convergence criteria is reached.

After the training process is completed, the compression model130may store the components of the encoding system210including the frame encoder412to a computer storage readable medium as the encoder and provide the encoder to the sender system110for deployment. The compression system130may store the components of the decoding system220including the frame decoder414to a computer storage readable medium as the decoder and provide the decoder to the receiver system140for deployment.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the frame encoder412to the new frame and the parameter map encoding the desired compression level to generate compressed code. The sender system110transmits the compressed code to the receiver system140. The receiver system140receives the compressed code and applies the frame decoder414to the compressed code to generate a reconstructed version of the new frame. This process is completed for the next frame in the sequence.

Architecture of Compression Model with Frame and Residue Autoencoder

FIG.4Billustrates a detailed architecture for a compression model200B for processing P-frames or B-frames, in accordance with an embodiment. One way to encode the target frame in a sequence of video frames is to characterize the target frame as a combination of a flow and a residual frame. The flow describes changes in one or more characteristics of a scene from a previously reconstructed frame to a current target frame. The flow can be applied to the previously reconstructed frame to generate a compensated frame that resembles the target frame but may not include some details of the target frame. Thus, the residual frame describes the residual difference between the target frame and the compensated frame. The encoder may encode the flow and residual frame for a target frame and transmit the compressed information to a decoder, such that the decoder can reconstruct the target frame. This process is repeated for the next frame in the sequence.

For example, the flow may be optical flow that describes the motion displacement of a block of pixels from the previously reconstructed frame to a corresponding block in the current target frame that contains the same portion of the scene (e.g., person riding a motorcycle). The displacement reflects the movement of the portion of the scene from the previously reconstructed frame to the target frame. In such an instance, the flow may be represented in the form of a motion vector that indicates the direction and magnitude of the change from the previously reconstructed frame to the target frame. The compensated frame is generated by displacing each block in the previously reconstructed frame based on the motion vectors, and the residual frame is generated by computing a difference between the compensated frame and the target frame.

The compression model200B shown inFIG.4Bis configured to encode and decode a target frame with reference to one or more previously reconstructed frames. In one instance, when the target frame is processed as a P-frame, the one or more previously reconstructed frames may include a previously reconstructed frame that is placed temporally before the target frame in the video. For example, the previously reconstructed frame may be a frame that was reconstructed immediately before the target frame that is placed immediately before the target frame in the video. In another instance, when the target frame is processed as a B-frame, the one or more previously reconstructed frames may include two or more previously reconstructed frames that include at least one frame that is placed temporally before, and another frame that is placed temporally after the target frame in the video.

In one embodiment, the compression model200B includes a flow autoencoder430B, a flow operator428, and a residue autoencoder440B. In particular, the encoding system210may include the flow autoencoder430B and the residue autoencoder440B. The decoding system220may include the flow decoder434B and the residue decoder444B. The flow autoencoder430B includes a flow encoder432B and a flow decoder434B. The flow encoder432B is coupled to receive a target frame, the zero or more previously reconstructed frames, and the parameter map for the target frame and generate the flow code. Similar to the description inFIG.2, the flow code output by the flow encoder432B may be generated at a codelayer that is a bottleneck layer with lower dimensionality (or lower number of nodes) than the remaining layers of the flow encoder432B and the flow decoder434B, and structured such that flow codes for at least two target frames with different values of the encoding parameter are generated at the same codelayer by applying a same subset of weights in the flow encoder432B.

The flow decoder434B is coupled to receive the flow code and the parameter map and generate a reconstructed version of the flow for the target frame. In other words, information in the flow code can be used to reconstruct the flow for the target frame.

The flow operator428is coupled to receive reconstructed flow for the target frame and output a compensated frame by applying the reconstructed flow to one or more previously reconstructed frames. In particular, the flow operator428applies the reconstructed flow to the one or more previously reconstructed frames based on the information that the encoder is configured to encode in the flow. For example, when the flow represents motion displacement, the flow operator428may displace pixels or blocks of pixels from the previously reconstructed frames to corresponding locations on the predicted frame based on the reconstructed flow.

The residue autoencoder440B includes a residue encoder442B and a residue decoder444B. The residue encoder442B is coupled to receive a residue frame, a compensated frame, and the parameter map for the target frame and generate a residue code. The residue frame is generated by taking the difference between the target frame and compensated frame. For example, the residue frame may be generated by taking the difference in greyscale or RGB intensity values between the target frame and the compensated frame for pixels at same locations in the compensated frame and the target frame. The residue decoder444B is coupled to receive the residue code, the compensated frame, and the parameter map for the target frame and generate a residue reconstruction for the target frame. Thus, the residue code is used to reconstruct the residue frame. The residue encoder442B may be structured such that residue codes for at least two target frames with different values of the encoding parameter are generated at the same codelayer by applying a same subset of weights in the residue encoder442B. The residue reconstruction is combined with the compensated frame to generate the reconstructed frame for the target frame. Then, the compression system130may store the reconstructed frame in the frame datastore470and the frame updater480may retrieve the reconstructed frame as a previously reconstructed frame for a next frame in the sequence when needed.

Moreover, it should be appreciated that the intermediate data structures within the compression model200B other than the target frame or the one or more previously reconstructed frames can be represented as multi-dimensional tensors. As described in conjunction withFIG.2, the parameter map may be input to the components of the compression model200B in any manner that is appropriate for incorporating parameter information. For example, the parameter map may be input to the residue encoder442B and the residue decoder444B by tiling and concatenating the parameter map to elements of the compensated frame and the residue frame or by selecting a respective subset of weights in the residue encoder442B and residue decoder444B dedicated to the specified bitrate level for the rate parameter.

During the forward pass step of the training process, the compression system130applies the components of the flow autoencoder430B to the target frame xt, zero or more previously reconstructed frames {circumflex over (x)}t-1, and the parameter map ctfor the target frame of a training video to generate the flow code for the target frame xtand the reconstructed flow {circumflex over (f)}tfor the target frame xt. The input to the flow autoencoder430B may optionally include the previous flow. The compression system130applies the flow operator428to the reconstructed flow {circumflex over (f)}tand the one or more previously reconstructed frames {circumflex over (x)}t-1to generate the compensated frame. The compression system130determines the residue frame by computing a difference between the target frame xt, and the compensated frame. The compression system130applies the components of the residue autoencoder440B to the residue frame, the compensated frame, and the parameter map ctto generate the residue reconstruction. The compression system130combines the residue reconstruction and the compensated frame to generate the reconstructed frame {circumflex over (x)}tfor the target frame xt.

The compression system130determines the loss function as a combination of the reconstruction loss250and the codelength loss260based on the reconstructed frame {circumflex over (x)}tand the compressed code generated for the target frame xt. In particular, the codelength Rt(l)of the compressed code for a target frame xtmay be a combination of the codelength of the flow code and the codelength of the residue code generated for the target frame xt. The compression system130backpropagates one or more error terms from the loss function to update the weights of the flow encoder432B, the flow decoder434B, the residue encoder442B, and the residue decoder444B. This process is repeated until a convergence criteria is reached.

After the training process is completed, the compression model130may store the components of the encoding system210and the decoding system220, or the compression model200B, to a computer storage readable medium as the encoder and provide the encoder to the sender system110for deployment. The compression system130may store the components of the decoding system220including the flow decoder434B, the residue decoder444B, and the flow operator428to a computer storage readable medium as the decoder and provide the decoder to the receiver system140for deployment.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the flow autoencoder430B to the new frame and the parameter map encoding the desired compression level to generate flow code and reconstructed flow. The sender system110generates the compensated frame and the residue frame for the new frame. The sender system110then applies the residue autoencoder440B to the residue frame, the compensated frame, and the parameter map to generate the residue code and the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store270for use in a next frame in the sequence. The sender system110transmits the flow code and the residue code to the receiver system140.

The receiver system140receives the flow code and the residue code. The receiver system140applies the flow decoder434B to the flow code and the parameter map to generate the reconstructed flow. The receiver system140generates the compensated frame. The receiver system140applies the residue decoder444B to the residue code, the compensated frame, and the parameter map to generate the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store270for use in decoding a next frame in the sequence. This process is completed for the next frame in the sequence.

Architecture of Compression Model with Flow Predictor

FIG.4Cillustrates a detailed architecture for a compression model200C for processing P-frames or B-frames, in accordance with another embodiment. The compression model200C shown inFIG.2Cfurther includes a flow predictor450for predicting the flow of a target frame. Specifically, in many instances, the flow for a set of previously reconstructed frames is a good estimate of the flow for the target frame. This is because changes in a given scene share considerable redundancy. For example, when the flow is optical flow, consecutive frames share redundancy due to the linearity of motion (e.g., constant or similar velocity between frames). For example, the motion vector for a block containing a moving cyclist in a set of previously reconstructed frames may be a good estimate for the motion vector for the target frame because the motion of the cyclist is displaced by similar amounts in one frame to another. Using the flow predictor, the decoder can predict the flow for a target frame to a reasonable accuracy using information already available to the decoder, and the flow for the target frame can be further refined by the flow delta transmitted from the encoder with a smaller number of bits.

In one embodiment, the compression model200C includes a flow predictor block450, a flow autoencoder430C, a flow operator428, and a residue autoencoder440C. Specifically, the flow predictor block450includes a flow predictor452and a flow operator428. The flow predictor452is coupled to receive a set of previous reconstructed frames, a previous flow for the set of previous reconstructed frames, and the parameter map and output the flow prediction for the target frame. The set of previous reconstructed frames may include zero or more frames that were reconstructed before the target frame (e.g., two most recently reconstructed frames), and the previous flow is the flow for these set of previous reconstructed frames. Thus, the flow prediction for a target frame is configured to be generated using information from frames that were already previously reconstructed at that time, rather than content in a current target frame.

The structure and functionality of the flow operator428is substantially similar or identical to that described with respect to the flow operator428in conjunction withFIG.4B. The flow428is coupled to receive the flow prediction and output a predicted frame by applying the flow prediction to a previously reconstructed frame (e.g., frame reconstructed immediately before). Preferably, when the flow prediction is a reasonably good estimate for the flow for the target frame, the predicted frame already closely resembles the content of the target frame.

The flow autoencoder430C includes a flow encoder432C and a flow decoder434C. The flow encoder432C is coupled to receive a target frame, the predicted frame, and the parameter map and generate a flow code. The flow decoder434C is coupled to receive the flow code, the predicted frame, and the parameter map and generate a flow delta for the target frame. The flow encoder432C may be structured such that flow codes for at least two target frames with different values of the encoding parameter are generated at the same codelayer by applying a same subset of weights in the residue encoder432C. Thus, information compressed in the flow code can be used to reconstruct the flow delta. The flow delta is used to refine the flow prediction based on the actual content of the target frame to generate the flow for the target frame. In particular, as shown inFIG.4C, the flow for a target frame is generated by combining the flow prediction with the flow delta.

The residue autoencoder440C includes a residue encoder442C and a residue decoder444C. The residue encoder442C is coupled to receive a residue frame, a compensated frame, and the parameter map and generate a residue code. The residue encoder442C may be structured such that residue codes for at least two target frames with different values of the encoding parameter are generated at the same codelayer by applying a same subset of weights in the residue encoder442C. The compensated frame is generated by the flow operator428coupled to receive the flow for the target frame and output a compensated frame by applying the flow to the previously reconstructed frame. The residue frame is generated by computing the difference between the target frame and the compensated frame, similar to that described in conjunction withFIG.4B.

The residue decoder444C is coupled to receive the residue code, the compensated frame, and the parameter map and generate a residue reconstruction for the target frame. Specifically, information compressed in the residue code can be used to reconstruct the residue frame. The residue reconstruction can be combined with the compensated frame to generate the reconstructed frame for the target frame. Then, the compression system130may store the reconstructed frame in the frame datastore270and the frame updater480may retrieve the reconstructed frame as a previously reconstructed frame for a next frame in the sequence when needed. The flow for the target frame may also be updated as the previous flow for the next frame in the sequence via a flow updater (not shown inFIG.4C).

Similar to the compression model200B inFIG.4B, it should be appreciated that intermediate data structures within the compression model200C other than the target frame or the one or more previously reconstructed frames can be represented as multi-dimensional tensors, and the parameter map may be input to the components of the compression model200B in any manner that is appropriate for incorporating parameter information. For example, the parameter map may be input to the flow predictor452by tiling and concatenating the parameter map to elements of the set of previous frames and the previous flow tensors or selecting a respective subset of weights in the flow predictor452. The parameter map may be input to the flow encoder432C and the flow decoder434C by tiling and concatenating the parameter map to elements of the target frame and the predicted frame or passed as a separate tensor. The parameter map may be input to the residue encoder442C and the residue decoder444C by tiling and concatenating the parameter map to elements of the residue frame and the compensated frame or can be passed as a separate tensor.

During the forward pass step of the training process, the compression system130applies the components of the flow predictor block450to the target frame xt, a set of previous reconstructed frames {circumflex over (x)}t-2, {circumflex over (x)}t-1, and the parameter map ctfor the target frame to generate a flow prediction. The compression system130applies the flow operator428to the flow prediction and one or more previously reconstructed frames {circumflex over (x)}t-1to generate a predicted frame. The compression system130applies the flow autoencoder430C to the target frame xt, the predicted frame, and the parameter map ctfor the target frame to generate the flow code and the flow delta for the target frame xtto generate the reconstructed flow {circumflex over (f)}t. The compression system130applies the flow operator428to the reconstructed flow {circumflex over (f)}tand the one or more previously reconstructed frames {circumflex over (x)}t-1to generate the compensated frame. The compression system130determines the residue frame by computing a difference between the target frame xt, and the compensated frame. The compression system130applies the components of the residue autoencoder440C to the residue frame, the compensated frame, and the parameter map ctto generate the residue reconstruction. The compression system130combines the residue reconstruction and the compensated frame to generate the reconstructed frame {circumflex over (x)}tfor the target frame xt.

The compression system130determines the loss function as a combination of the reconstruction loss250and the codelength loss260based on the reconstructed frame {circumflex over (x)}tand the compressed code generated for the target frame xt. In particular, similar to the compression model200B shown inFIG.4B, the codelength Rt(l)of the compressed code may be a combination of the codelength of the flow code and the codelength of the residue code generated for the target frame xt. The compression system130backpropagates one or more error terms from the loss function to update the weights of the flow predictor452, the flow encoder432C, the flow decoder434C, the residue encoder442C, and the residue decoder444C. This process is repeated until a convergence criteria is reached.

After the training process is completed, the compression model130may store the components of the encoding system210and the decoding system220, or the compression model200C, to a computer storage readable medium as the encoder and provide the encoder to the sender system110for deployment. Similarly, the compression system130may store the components of the decoding system220including the flow predictor block450, the flow decoder434C, the residue decoder444C, and the flow operator428to a computer storage readable medium as the decoder and provide the decoder to the receiver system140for deployment.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the flow predictor block450to a set of previous reconstructed frames, the previous flow, and the parameter map to generate a flow prediction and a predicted frame. The sender system110applies the flow autoencoder430C to a new frame and the parameter map encoding the desired compression level to generate flow code, the flow delta, and the reconstructed flow. The sender system110generates the compensated frame and the residue frame for the new frame. The sender system110then applies the residue autoencoder440C to the residue frame, the compensated frame, and the parameter map to generate the residue code and the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store270for use in a next frame in the sequence. The sender system110transmits the flow code and the residue code to the receiver system140.

The receiver system140receives the flow code and the residue code. The receiver system140applies the flow predictor block450to a set of previous reconstructed frames and a previous flow to generate a flow prediction and a predicted frame. The receiver system140applies the flow decoder434C to the flow code and the parameter map to generate the reconstructed flow. The receiver system140generates the compensated frame. The receiver system140applies the residue decoder444C to the residue code, the compensated frame, and the parameter map to generate the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store270for use in decoding a next frame in the sequence. This process is completed for the next frame in the sequence.

The description of the compression model200inFIG.2and example architectures of the compression models200A,200B,200C inFIGS.4A through4Care described with respect to parameter maps encoding desired compression levels. However, this is merely an example, and as described in more detail below, it is appreciated that the parameter map may encode values for any other types of encoding parameters. The weights of the compression model200may be trained by assigning the specific value of the encoding parameter to training data and inputting the training data along with the parameter map encoding the specific value of the encoding parameter to generate the outputs of the compression model200. The weights of the compression model200are updated based on one or more error terms from a loss function computed using the outputs.

Moreover, in one embodiment, information processed by various components of the compression models200A,200B,200C may be directly interpretable with respect to the types of data it encodes in the context of video encoding and decoding. For example, an element of the flow prediction tensor generated by the flow predictor452inFIG.4Cmay be interpreted to encode the predicted flow for a respective pixel and in the previously reconstructed frame, and an element in the residue reconstruction tensor generated by the residue decoder444C inFIG.4Cmay be interpreted to encode a residual pixel intensity value for a respective pixel in the target frame. However, it should be appreciated that information processed by the compression model is not limited hereto and can include information that is not readily interpretable in the context of video compression.

Specifically, in another embodiment, at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model200ofFIG.2, compression model500ofFIG.5, or compression model600ofFIG.6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task. The state information for a neural network component may be uninterpretable in the sense that it can be processed by the neural network component to store and retrieve any type of information necessary for performing the task, rather than having a specific, interpretable meaning on the information. In one instance, the state information is represented as a tensor with one or more spatial dimensions.

In one example, information received or output by a component of the compression model may be configured to be at least in part or entirely composed of state information that is not necessarily associated with an imposed meaning but is used by the component to propagate any type of information that the component deems necessary for performing the assigned task. The component may learn what types of information to place in the state information during the training process. For example, the flow prediction tensor generated by the flow predictor452inFIG.4Cmay be at least in part or entirely be composed of state information that contains data used by the flow predictor452to propagate any type of information that the flow predictor452deems necessary for predicting flow. In one instance, when information processed by the component of the compression model is represented as a tensor, at least a portion of the tensor can include interpretable information and the remaining portion of the tensor (e.g., additional channels in the tensor) may be configured as state information. For example, at least a part of the flow delta tensor output by the flow decoder434C inFIG.4Cmay include interpretable elements that encode the flow delta for each respective pixel in a target frame but can also include additional channels of state information that extract and encode information not directly interpretable by a human operator but is still useful to the residue decoder444C or other components of the compression model200C for the purpose of encoding and decoding videos.

Bitrate Level Interpolation

The compression model200described in conjunction withFIG.2configured to receive a parameter map as input allows an encoder-decoder to interpolate between bitrate levels. Specifically, the method of training and deploying the compression model200receives a specified bitrate level among a set of bitrate levels as input. For example, in the example shown inFIG.2, the components of the compression model200receive a specified bitrate level among five bitrate levels as input. In one embodiment, the compression system130generates a parameter map that when input to the compression model or the corresponding encoder-decoder, generates compressed code for a reconstructed frame that corresponds to an interpolated bitrate level between the set of bitrate levels represented by elements of the parameter map.

In one embodiment, L levels can be embedded into a Le-dimensional space represented by a one-hot encoded vector of a parameter map (e.g., Lenumber of elements). Specifically, the parameter map vlembedding a level l∈{0, 1, . . . L−1} is computed by:

During deployment, the sender system110encodes a new frame with a desired bitrate level l∈{0, 1, . . . , L−1} that is between the Lebitrate levels encoded by the elements of the parameter map by generating an interpolated embedding vlusing the equations above. The sender system110applies the encoding system210to the new frame and the interpolated parameter map to generate the compressed code. Responsive to receiving the compressed code, the receiver system140applies the decoding system220to the compressed code and the interpolated parameter map to generate the reconstructed frame. In this manner, an encoder-decoder architecture can encode an arbitrary number of bitrate levels without additional training of the compression model.

Training Process of Compression Model with Spatial Variation

FIG.5illustrates a training process for a compression model coupled to receive a parameter map for compression levels with spatial variation, in accordance with an embodiment. As shown inFIG.5, the compression model500includes an encoding system510and a decoding system520among other components. Specifically, the compression model500shown inFIG.5is substantially similar or identical to the architecture of the compression model200shown inFIG.2except that different values of the encoding parameter can be applied to different regions of an input tensor. This is particularly useful if, for example, a target frame includes certain spatial regions that might benefit from the allocation of additional bits for higher quality reconstruction. For example, these spatial regions might be regions including objects-of-interest, such as vehicles, pedestrians, cyclists, and the like. In another embodiment, the spatial regions may be at least in part content-independent. For example, spatial regions near or at the center of a frame may be encoded with higher quality (e.g., higher bitrates), and spatial regions near or at the peripheral areas of the frame may be encoded with lower quality (e.g., lower bitrates). This configuration may be, for example, useful in a virtual reality (VR) setup to match the human visual system's ability to detect at higher resolution objects in the central view. In yet another example, in setups with multiple overlapping cameras, such as cameras for an autonomous driving system, the non-overlapping areas, or areas more likely to contain objects of interest (e.g., other vehicles, pedestrians, cyclists) might be considered more important and encoded with higher quality.

In the example compression model500shown inFIG.5, the parameter map may be represented by a plurality of one-hot encoded vectors. Each encoded vector includes one or more elements each representing a respective bitrate level, where a higher level corresponds to a higher bitrate level. An encoded vector may encode the respective value of the encoding parameter for a corresponding spatial region of an input tensor. For example, each encoded vector may correspond to a respective pixel location in the input tensor, or a group of pixel locations in the input tensor that encompass the region-of-interest. In the example shown inFIG.5, each one-hot encoded vector has five elements to encode five different bitrate levels.

In one instance, the parameter map is input to the compression model500by concatenating the encoded vectors of the parameter map to elements of an input tensor and inputting the concatenated tensor to respective components of the compression model500. For example, the encoding system510may be coupled to receive encoding information and the parameter map in the form of a concatenated target frame and one or more reconstructed frames. The concatenated target frame may be a tensor having each encoded vector of the parameter map concatenated to image data for a particular region of pixels in the target frame that are assigned to the bitrate level of the encoded vector.

In another instance, the parameter map is input to the compression model500by selecting a respective subset of weights in the compression model500that are dedicated to the value of the encoding parameter indicated in an encoded vector in the parameter map. Specifically, different regions of an input tensor may be applied with different subsets of weights depending on the value of the encoding parameter for the region in the parameter map. For example, some components of the compression model500may be configured as a convolutional neural network model (CNN) that include a convolutional filter having weights repeatedly applied to a moving window of pixels of an input tensor. In one instance, the convolutional layer may include a set of convolutional filters each corresponding to a respective bitrate level, and different regions of the input tensor may be applied with different convolutional filters depending on the value of the encoding parameter for the region in the parameter map.

During the training process, the compression system130obtains a set of training videos. In particular, the compression system130may identify certain regions in the training frames and assign a bitrate level to each region in the frame. For example, the compression system130may identify pixels that are objects-of-interest in a training frame and assign encoded vectors with high bitrate levels for these pixels and assign encoded vectors with lower bitrate levels to the remaining regions of the training frame. The compression system130may also assign a frame label to each training frame indicating whether the training frame is to be processed as an I-frame, a P-frame, or a B-frame, such that a compression model500with the appropriate architecture can be applied to each training frame according to the frame label. Specifically, the compression model500may include example architectures that are identical or substantially similar to those shown in the compression models200A,200B,200C ofFIGS.4A through4C.

During the forward pass step of the training process, the compression system130applies the encoding system510to encoding information and the parameter map ctfor a target frame xtin a training video to generate compressed code. Similarly to the description ofFIG.2, the encoding system510may be structured such that compressed codes for at least two target frames with different values of the encoding parameter are generated at the same codelayer by applying a same subset of weights in the encoding system510. The compression system130then applies the decoding system520to decoding information and the parameter map for the target frame xtto generate the reconstructed version of the target frame {circumflex over (x)}t. In the example shown inFIG.5, example parameter map vectors are shown for the front row of pixels in the image, in which the left three pixels are associated with a lower bitrate level and the right three pixels (indicated with black shading) are associated with a higher bitrate level. Specifically, the parameter map ctindicates that the bitrate level for regions of the target frame xtwith black shading are associated with the fourth bitrate level and the encoded vector [0 0 0 1 0] is concatenated to the image data for the pixel locations for these regions. Optionally, the encoded vector may be concatenated to the previously reconstructed frames or other frames that are processed within the compression model. The remaining regions for are associated with a lower second bitrate level, and the encoded vector [0 1 0 0 0] is concatenated to the image data for the remaining pixel locations. The compression system130stores the reconstructed frame {circumflex over (x)}tin a frame data store570. When necessary, a frame updater580may retrieve the reconstructed frame {circumflex over (x)}tfrom the frame data store570to use as the previously reconstructed frame for a next frame in the video.

The compression system130determines a loss function based on the results of the forward pass step. Similar to the description provided in conjunction withFIG.2, the loss function may also include a reconstruction loss550and a codelength loss560that is identical or substantially similar to the reconstruction loss250and the codelength loss260of the compression model200. However, in one embodiment, the reconstruction loss550for the compression model500may be given by:

ℒr⁡(xt,x^t;θ)=1T⁢∑T⁢∑chw⁢∑l⁢wt,hw(l)⁢D⁡(xt,chw,x^t,chw(l))
where chw is an index indicating a location of an element in the reconstructed frame, l indicates the bitrate level assigned to an element xt,chwat location chw in the target frame xt, wt(l)is a weighting parameter assigned to the element at location chw that increases as the bitrate level for the location increases, and θ indicates the collective set of parameters for the compression model200, and Lrindicates the reconstruction loss and D(⋅) is the distortion loss.

During the backpropgation step, the compression system130backpropagates one or more error terms obtained from the loss function to update the weights of the compression model500, and repeats the process until a convergence criteria for the weights of the compression model500is reached, at which the training process is completed. By reducing a loss function that is a combination of the reconstruction loss550that varies according to the bitrate level for a spatial region in the reconstructed frame, the compression system130can train the components of the compression model500to increase reconstruction quality for important regions in the target frame while reducing the codelength of the compressed code.

During deployment, the sender system110iteratively encodes a sequence of image frames for a video using the encoder. The sender system110determines the bitrate levels for different regions of a target frame. The sender system110applies the encoding system510to a target frame, one or more previously reconstructed frames, and a parameter map for the target frame to generate compressed code. The sender system110applies the decoding system520to the compressed code, information obtained from the one or more previously reconstructed frames, and the parameter map to generate the reconstructed frame. The reconstructed frame is stored in the frame data store570and is updated as the previously reconstructed frame for a next frame. The sender system110transmits the compressed code to the receiver system140. This process may be repeated for the next frame onward.

The receiver system140iteratively decodes the sequence of images using the transmitted codes from the sender system110. The receiver system140receives the code from the sender system110. The receiver system140applies the decoding system520to the compressed code, information obtained from the one or more previously reconstructed frames, and the parameter map to generate the reconstructed frame for the target frame. As discussed above, in one embodiment, the decoding system520can be configured to obtain the parameter map from the compressed code or the parameter map for a target frame may be provided externally or inferred from other parameters available to the decoding system520, such as previously reconstructed frames. In another embodiment, the decoding system520can predict the parameter map for a target frame from previously reconstructed frames and the delta between the actual parameter map and the predicted parameter map may be included in the compressed code. The reconstructed frame is stored in the frame data store570and is updated as the previously reconstructed frame for a next frame. The receiver system140repeats this process for the next frame as the compressed code for the next frame is transmitted from the sender system110.

As described above, the compression model500may include any of the architectures shown inFIGS.4A through4C. For example, the compression model500may include the example architecture shown inFIG.4A. The encoding system510may include a frame encoder and the decoding system520may include a frame decoder. During the forward pass step, the compression system130applies the frame encoder to the target frame xtand the parameter map for the target frame xtof a training video to generate the compressed code. The compression system130applies the frame decoder to the compressed code to generate the reconstructed frame {circumflex over (x)}t. The compression system130determines the loss function as a combination of the reconstruction loss550and the codelength loss560and updates the weights of the compression model500.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the frame encoder to the new frame and the parameter map for the new frame to generate compressed code. The sender system110transmits the compressed code to the receiver system140. The receiver system140receives the compressed code. The receiver system140applies the frame decoder to the compressed code and the parameter map for the new frame to generate the reconstructed version of the new frame. This process is completed for the next frame in the sequence.

The compression model500may include the example architecture shown inFIG.4B. The encoding system510may include a flow encoder and a residue encoder, and the decoding system520may include a flow decoder and a residue decoder. During the forward pass step, the compression system130applies the flow autoencoder to the target frame xt, one or more previously reconstructed frames {circumflex over (x)}t-1, and the parameter map ctfor the target frame of a training video to generate the flow code for the target frame xtand the reconstructed flow {circumflex over (f)}tfor the target frame xt. The compression system130applies the components of the residue autoencoder to the residue frame, the compensated frame, and the parameter map ctto generate the residue reconstruction. Then, the compression system130combines the residue reconstruction and the compensated frame to generate the reconstructed frame {circumflex over (x)}tfor the target frame xt. The compression system130determines the loss function as a combination of the reconstruction loss550and the codelength loss560and updates the weights of the compression model500. This process is repeated until a convergence criteria is reached.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the flow autoencoder to the new frame and the parameter map to generate flow code and the reconstructed flow. The sender system110generates the compensated frame and the residue frame for the new frame. The sender system110then applies the residue autoencoder to the residue frame, the compensated frame, and the parameter map to generate the residue code and the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store for use in a next frame in the sequence of frames. The sender system110transmits the flow code and the residue code to the receiver system140.

The receiver system140receives the flow code and the residue code from the sender system110. The receiver system140applies the flow decoder to the flow code and the parameter map to generate the reconstructed flow. The receiver system140generates the compensated frame. The receiver system140applies the residue decoder to the residue code, the compensated frame, and the parameter map to generate the residue reconstruction. The reconstructed frame is generated by combining the residue reconstruction and the compensated frame. The reconstructed frame is stored in a frame data store for use in decoding a next frame in the sequence. This process is completed for the next frame in the sequence.

The compression model500may include the example architecture shown inFIG.4C. The encoding system510may include a flow predictor block, a flow encoder, and a residue encoder, and the decoding system520may include the flow predictor block, a flow decoder, and a residue decoder. During the forward pass step, the compression system130applies the flow predictor block to a set of previous reconstructed frames {circumflex over (x)}t-2, {circumflex over (x)}t-1, a previous flow, and the parameter map ctto generate a flow prediction. The compression system130applies the flow autoencoder to the target frame xt, the predicted frame, and the parameter map ctto generate the flow code and the flow delta. The flow delta is combined with the flow prediction to generate the reconstructed flow {circumflex over (f)}t. The compression system130applies the flow operator to the reconstructed flow {circumflex over (f)}tand the one or more previously reconstructed frames {circumflex over (x)}t-1to generate the compensated frame. The compression system130then applies the residue autoencoder to the compensated frame, the residue frame, and the parameter map ctfor the target frame xt, to generate the residue reconstruction for the target frame xt, as well as the reconstructed frame {circumflex over (x)}t. The compression system130determines the loss function as a combination of the reconstruction loss550and the codelength loss560and updates the weights of the compression model500. This process may be repeated until a convergence criteria is reached.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the flow predictor block to a set of previous reconstructed frames, the previous flow for the previous reconstructed frames, and the parameter map to generate a flow prediction for the new frame and a predicted frame. The sender system110applies the flow autoencoder to a new frame and the parameter map encoding the desired compression level to generate flow code, the flow delta, and the reconstructed flow. The sender system110generates the compensated frame and the residue frame. The sender system110applies the residue autoencoder to the residue frame, the compensated frame, and the parameter map to generate the residue code and the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store for use in a next frame. The sender system110transmits the flow code and the residue code to the receiver system140.

The receiver system140receives the flow code and the residue code from the sender system110. The receiver system140applies the flow predictor block to a set of previous reconstructed frames and the previous flow to generate a flow prediction for a target frame. The receiver system140applies the operator to generate a predicted frame. The receiver system140applies the flow decoder to the flow code, the predicted frame, and the parameter map to generate the reconstructed flow. The receiver system140applies the operator to generate the compensated frame. The receiver system140then applies the residue decoder to the residue code, the compensated frame, and the parameter map to generate the residue reconstruction. The receiver system140generates the reconstructed frame by combining the compensated frame and the residue reconstruction. The reconstructed frame is stored in a frame data store for use in decoding a next frame in the sequence. This process is completed for the next frame in the sequence.

It should be appreciated that throughout the training process and the deployment process of the compression model, the parameter map can be applied to different components of the compression model according to respective locations in the target frame that are associated with the encoded vectors of the parameter map. For example, in the architecture ofFIG.4B, the compensated frame and the residue frame may have the same dimensionality as the target frame. The parameter map may be input to the residue encoder and the residue decoder by concatenating the encoded vectors to respective elements of the compensated frame and the residue frame at the same locations as the region of pixels in the target frame that were assigned to these encoded vectors. As another example, the predicted frame in the architecture ofFIG.4Cmay have the dimensionality as the target frame. The parameter map may be input to the flow encoder and the flow decoder by concatenating the encoded vectors to respective elements of the predicted frame at the same locations as the region of pixels in the target frame that were assigned to these encoded vectors. As another example, the flow encoder and the flow decoder may be configured as a convolutional neural network, and a respective convolutional filter is applied to different regions of the target frame and the predicted frame depending on the selected values of the bitrate levels for these regions.

Training Process of Compression Model with Frame Parameters

FIG.6illustrates a training process for a compression model coupled to receive a parameter map for frame parameters, in accordance with an embodiment. As shown inFIG.6, the compression model600includes an encoding system610and a decoding system620among other components. Specifically, the compression model600shown inFIG.6is substantially similar or identical to the architecture of the compression model200shown inFIG.2except that the parameter map encodes values for a frame parameter that indicates the type of information used to encode and decode a target frame.

Specifically, a target frame encoded as a P-frame or B-frame may be encoded and decoded with reference to one or more previously reconstructed frames in the video. In particular, a target frame encoded as a P-frame may be encoded by characterizing a target frame with respect to flow from one or more previously reconstructed frames that are placed temporally before the target frame (e.g., placed immediately before the target frame in the video) and the corresponding residue frame. A target frame encoded as a B-frame may be encoded by characterizing a target frame with respect to flow from two or more previously reconstructed frames that are placed temporally before and after the target frame (e.g., placed immediately before and after the target frame in the video) and the corresponding residue frame.

Specifically, a B-frame may be encoded and decoded in a hierarchical manner. For example, a B-frame may be encoded and decoded as a B1-frame, a B2-frame, or a B3-frame. A B1-frame is encoded and decoded using previously reconstructed frames that were reconstructed as an I-frame and/or a P-frame, a B2-frame is encoded and decoded using previously reconstructed frames that include one frame reconstructed as an I-frame or P-frame and another frame reconstructed as a B1-frame, and a B3-frame is encoded and decoded using previously reconstructed frames that were reconstructed as a B1-frame and a B2-frame. Thus, in one embodiment, the parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map. This makes the deployment process easier since multiple models with different sets of parameters do not have to be constructed for each B1, B2, or B3frame. The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map. As an example, the second parameter map may be represented as a one-hot encoded vector including four elements, each representing whether a target frame is encoded and decoded as a P-frame, a B1-frame, a B2-frame, or a B3-frame. As another example, the second parameter map may indicate whether a target frame is encoded and decoded as a P-frame or B-frame.

Similar to the compression model200inFIG.2, in one instance, the parameter map is input to the compression model600by tiling and concatenating the parameter map to elements of an input tensor and inputting the concatenated tensor to respective components of the compression model600, or the parameter map may be input to the compression model200by selecting a respective subset of weights in the compression model600that are dedicated to the selected value of the encoding parameter indicated in the parameter map. For example, responsive to receiving a parameter map encoding a frame parameter, the encoding system610may apply a dedicated subset of weights for the frame parameter value to encoding information for the target frame to generate the compressed code for the target frame. Similarly to the description ofFIG.2, the encoding system610may be structured such that compressed codes for at least two target frames with different values of the encoding parameter are generated at the same codelayer by applying a same subset of weights in the encoding system610.

As discussed above, the parameter map may encode values for multiple encoding parameters, such as the bitrate level and whether the frame is encoded as a P-frame or B-frame. For example, the parameter map may include a first parameter map indicating desired bitrate levels and a second parameter map concatenated to the first parameter map, where the second parameter map indicates whether the frame will be encoded and decoded as a P-frame or B-frame.

During the training process, the compression system130obtains a set of training videos. In particular, the compression system130may assign a frame label to a training frame indicating whether the training frame is to be encoded and decoded as a P-frame, a B1-frame, a B2-frame, or a B3-frame. Thus, depending on the frame label for each frame in a training video and the type previously reconstructed frames for the frame, the sequence of frames may be encoded and decoded in a particular order different from the actual temporal order of the frames in the corresponding video. In one embodiment, the compression model600is configured to process a target frame with respect to flow information from one or more previously reconstructed frames and a residue frame, such as the architecture of the compression models200B,200C shown inFIGS.4B and4C.

During the forward pass step of the training process, the compression system130applies the encoding system610to encoding information and the parameter map ctfor the target frame xtin a training video to generate compressed code. The compression system130applies the decoding system620to decoding information and the parameter map for the target frame xtto generate the reconstructed frame {circumflex over (x)}t. In the example shown inFIG.6, the example parameter map ctincludes a first parameter map composed of three elements each corresponding to a bitrate level, and a second parameter map composed of two elements each corresponding to whether the frame will be encoded and decoded as a P-frame or B-frame that is concatenated to the first parameter map. Specifically, in the example shown inFIG.6, the target frame xtis encoded for a second bitrate level and encoded and decoded as a P-frame, and the corresponding parameter map [0 1 0 1 0] is concatenated to the image data for target frame xt.

Specifically, the one or more previously reconstructed frames {circumflex over (x)}t-1for the target frame xtincludes the type of frames that are specified in the parameter map input to the compression model600. For example, in the example shown inFIG.6, since the frame label assigned to the target frame xtis a P-frame, the one or more previously reconstructed frames {circumflex over (x)}t-1may include a previously reconstructed frame that was reconstructed as an I-frame. As another example, when the parameter map specifies that the target frame xtis processed as a B1-frame, the one or more previously reconstructed frames {circumflex over (x)}t-1may include two previously reconstructed frames that were reconstructed as a I-frame and a P-frame. In particular, the reconstructed I-frame may be generated, for example, using a compression model having the architecture ofFIG.4A. The compression system130stores the reconstructed frame {circumflex over (x)}tin a frame data store670. When necessary, a frame updater680may retrieve the reconstructed frame {circumflex over (x)}tfrom the frame data store670to use as the previously reconstructed frame for a next frame in the video.

The compression system130determines a loss function based on the results of the forward pass step. Similar to the description provided in conjunction withFIG.2, the loss function may also include a reconstruction loss650and a codelength loss660that is identical or substantially similar to the reconstruction loss250and the codelength loss260of the compression model200. During the backpropgation step, the compression system130backpropagates one or more error terms obtained from the loss function to update the weights of the compression model600, and repeats the process until a convergence criteria for the weights of the compression model600is reached, at which the training process is completed.

By reducing a loss function based on reconstructed frames that are generated with respect to the frame labels encoded by the parameter maps, the compression system130can train the components of the compression model600to perform P-frame and B-frame processing with a shared set of weights. Moreover, the components of the compression model600are also trained to allocate resources depending on the frame label of the target frame. For example, a target frame processed as a B1-frame may require a higher number of bits compared to frames processed as B2or B3frames in order to have good overall performance of the compression model. Since B1frames are earlier in the decoding hierarchy than B2or B3frames, more frames may use a B1frame as a reference than B2or B3frames. Thus, it may be advantageous to the compression model to allocate more bits to encoding a frame that is used as a reference to more decoded frames. Determining the correct bit allocation is also dependent on the amount of motion in the scene, and thus, there is an advantage to having the rate control handled by a compression model with machine-learned components. For instance, if there is a large amount of motion, it may be less important to allocate more bits to frames earlier in the decoding hierarchy. The compression model600may be automatically trained to allocate a higher codelength for compressing a target frame as a B1-frame, at the expense of lowering the codelength for other types of target frames.

During deployment, the rate control is less computationally intensive due to the method of training described above. In one method of encoding, the rate parameter is constant across the video. The training procedure ensures that for a fixed value of the rate parameter, strong rate-distortion (RD) performance averaged across the frames of the video is achieved without additional effort to evaluate the encoder for different values of the rate. Real-world constraints may necessitate additional effort at encoding time. For instance, the encoding phase may seek to achieve the highest quality given a maximum bitrate. In this case, the encoder may need to decrease the rate parameter for certain frames when the bitrate budget is exceeded. Another important constraint to consider is to achieve above a certain quality for all frames. In this case, if the quality drops too low, the encoder may choose to increase the rate parameter so that the quality exceeds the minimum quality threshold.

During deployment, the sender system110iteratively encodes a sequence of image frames for a video using the encoder. The sender system110determines a value for the encoding parameter for a target frame. The sender system110applies the encoding system610to a target frame, one or more previously reconstructed frames, and a parameter map for the target frame to generate compressed code. The sender system110applies the decoding system620to the compressed code, information obtained from the one or more previously reconstructed frames, and the parameter map to generate the reconstructed frame. The reconstructed frame is stored in the frame data store670and is updated as the previously reconstructed frame for a next frame. The sender system110transmits the compressed code to the receiver system140. This process may be repeated for the next frame onward.

The receiver system140iteratively decodes the sequence of images using the transmitted codes from the sender system110. The receiver system140receives the code from the sender system110. The receiver system140applies the decoding system620to the compressed code, information obtained from the one or more previously reconstructed frames, and the parameter map to generate the reconstructed frame for the target frame. In one embodiment, the decoding system620can be configured to obtain the parameter map from the compressed code or the parameter map for a target frame may be provided externally or inferred from other parameters available to the decoding system620, such as previously reconstructed frames. In another embodiment, the decoding system620can predict the parameter map for a target frame from previously reconstructed frames and the delta between the actual parameter map and the predicted parameter map may be included in the compressed code. The reconstructed frame is stored in the frame data store670and is updated as the previously reconstructed frame for a next frame. The receiver system140repeats this process for the next frame as the compressed code for the next frame is transmitted from the sender system110.

As described above, the compression model600may include any of the architectures shown inFIGS.4A through4C, in particular the compression models200B,200C shown inFIGS.4B and4C. For example, the compression model600may include the example architecture shown inFIG.4B. The encoding system610may include a flow encoder and a residue encoder, and the decoding system620may include a flow decoder and a residue decoder. During the forward pass step, the compression system130applies the flow autoencoder to the target frame xt, one or more previously reconstructed frames {circumflex over (x)}t-1, and the parameter map ctfor the target frame of a training video to generate the flow code for the target frame xtand the reconstructed flow {circumflex over (f)}tfor the target frame xt. The compression system130applies the components of the residue autoencoder to the residue frame, the compensated frame, and the parameter map ctto generate the residue reconstruction. The compression system130combines the residue reconstruction and the compensated frame to generate the reconstructed frame {circumflex over (x)}t. The compression system130determines the loss function as a combination of the reconstruction loss650and the codelength loss660and updates the weights of the compression model600. This process is repeated until a convergence criteria is reached.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the flow autoencoder to the new frame and the parameter map to generate flow code and the reconstructed flow. The sender system110generates the compensated frame and the residue frame for the new frame. The sender system110then applies the residue autoencoder to the residue frame, the compensated frame, and the parameter map to generate the residue code and the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store for use in a next frame in the sequence of frames. The sender system110transmits the flow code and the residue code to the receiver system140.

The receiver system140receives the flow code and the residue code from the sender system110. The receiver system140applies the flow decoder to the flow code and the parameter map to generate the reconstructed flow. The receiver system140generates the compensated frame. The receiver system140applies the residue decoder to the residue code, the compensated frame, and the parameter map to generate the residue reconstruction. The reconstructed frame is generated by combining the residue reconstruction and the compensated frame. The reconstructed frame is stored in a frame data store for use in decoding a next frame in the sequence. This process is completed for the next frame in the sequence.

The compression model600may include the example architecture shown inFIG.4C. The encoding system610may include a flow predictor block, a flow encoder, and a residue encoder, and the decoding system620may include the flow predictor block, a flow decoder, and a residue decoder. During the forward pass step, the compression system130applies the flow predictor block to a set of previous reconstructed frames {circumflex over (x)}t-2,{circumflex over (x)}t-1, a previous flow, and the parameter map ctto generate a flow prediction. The compression system130applies the flow autoencoder to the target frame xt, the predicted frame, and the parameter map ctto generate the flow code and the flow delta. The flow delta is combined with the flow prediction to generate the reconstructed flow {circumflex over (f)}t. The compression system130applies the flow operator to the reconstructed flow {circumflex over (f)}tand the one or more previously reconstructed frames {circumflex over (x)}t-1to generate the compensated frame. The compression system130then applies the residue autoencoder to the compensated frame, the residue frame, and the parameter map ctfor the target frame xt, to generate the residue reconstruction for the target frame xt, as well as the reconstructed frame {circumflex over (x)}t. The compression system130determines the loss function as a combination of the reconstruction loss650and the codelength loss660and updates the weights of the compression model600. This process may be repeated until a convergence criteria is reached.

During the deployment process, the sender system110encodes a new frame with a desired compression level by applying the flow predictor block to a set of previous reconstructed frames, the previous flow for the previous reconstructed frames, and the parameter map to generate a flow prediction for the new frame and a predicted frame. The sender system110applies the flow autoencoder to a new frame and the parameter map encoding the desired compression level to generate flow code, the flow delta, and the reconstructed flow. The sender system110generates the compensated frame and the residue frame. The sender system110applies the residue autoencoder to the residue frame, the compensated frame, and the parameter map to generate the residue code and the residue reconstruction, and the reconstructed frame. The reconstructed frame may be stored in a frame data store for use in a next frame. The sender system110transmits the flow code and the residue code to the receiver system140.

The receiver system140receives the flow code and the residue code from the sender system110. The receiver system140applies the flow predictor block to a set of previous reconstructed frames and the previous flow to generate a flow prediction for a target frame. The receiver system140applies the flow operator to generate a predicted frame. The receiver system140applies the flow decoder to the flow code, the predicted frame, and the parameter map to generate the reconstructed flow. The receiver system140applies the flow operator to generate the compensated frame. The receiver system140then applies the residue decoder to the residue code, the compensated frame, and the parameter map to generate the residue reconstruction. The receiver system140generates the reconstructed frame by combining the compensated frame and the residue reconstruction. The reconstructed frame is stored in a frame data store for use in decoding a next frame in the sequence. This process is completed for the next frame in the sequence.

Similar to the description provided in conjunction withFIG.2, the parameter map encoding the frame label may be input to the components of the compression model600in a manner that is appropriate for incorporating parameter information. As an example, the parameter map may be input to the residue encoder and the residue decoder by tiling and concatenating the parameter map to elements of the compensated frame and the residue frame or by selecting a respective subset of weights in the residue encoder and residue decoder dedicated to the specified bitrate level for the rate parameter.

Moreover, it is also appreciated that during the training process and the deployment process of the compression model, the compression system130can encode and decode a sequence of frames in a video using the appropriate architecture of the compression model or the corresponding encoder-decoder depending on the frame label for each frame. For example, during deployment, the sender system110may encode a frame as an I-frame using an encoder generated from the architecture of the compression model200A shown inFIG.4A. The sender system110may then encode a frame as a P-frame using an encoder generated from the architecture of the compression model600inFIG.6by specifying the frame label as a P-frame and using the reconstructed version of the I-frame as the previously reconstructed frame. The sender system110may then encode a frame as a B1-frame using an encoder generated from the compression model600by specifying the frame label as a B1-frame and using the reconstructed version of the I-frame and the P-frame as the previously reconstructed frames The receiver system140may iteratively receive the transmitted codes for each of these frames from the sender system110as well as the frame label for each target frame, and decode each frame using the appropriate architecture of the corresponding decoder.

SUMMARY