Patent Description:
In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

Systems, apparatuses, and methods for generating and implementing video encoding models for mapping encoded video frame bit-size to quantization strength are disclosed herein. In one implementation, a system includes at least an encoder, a pre-encoder, and a memory coupled to the encoder and pre-encoder. The pre-encoder runs multiple pre-encoding passes on at least a portion of an input video frame. In one implementation, the input video frame is pre-processed prior to the pre-encoding passes. Each pre-encoding pass uses a different quantization strength setting. In one implementation, the quantization strength setting refers to a particular quantization parameter (QP) used during the encoding process. For each pre-encoding pass, the pre-encoder captures the output bit-size of the encoded portion(s) of the input video frame. The pre-encoder uses the captured output bit-sizes to generate a model for mapping encoded video bitstream bit-size to quantization strength.

Before the encoder encodes the portion(s) of the input video frame, the encoder uses the model to map a specified bit-size to a corresponding quantization strength. In one implementation, the encoder provides a specified bit-size to the model and the model outputs the quantization strength value which will produce the specified bit- size. Then, the encoder encodes the portion(s) of the input video frame using the quantization strength value provided by the model so as to meet a given bit budget. In one implementation, by using the quantization strength value provided by the model, the encoder is able to make fewer quantization strength adjustments during the frame. This helps to improve the visual quality of the resulting encoded video bitstream.

Referring now to <FIG>, a block diagram of one implementation of a system <NUM> for encoding and decoding content is shown. System <NUM> includes server <NUM>, network <NUM>, client <NUM>, and display <NUM>. In other implementations, system <NUM> includes multiple clients connected to server <NUM> via network <NUM>, with the multiple clients receiving the same bitstream or different bitstreams generated by server <NUM>. System <NUM> can also include more than one server <NUM> for generating multiple bitstreams for multiple clients.

In one implementation, system <NUM> implements encoding and decoding of video content. In various implementations, different applications such as a video game application, a cloud gaming application, a virtual desktop infrastructure application, or a screen sharing application are implemented by system <NUM>. In other implementations, system <NUM> executes other types of applications. In one implementation, server <NUM> renders video or image frames, encodes the rendered frames into a bitstream, and then conveys the encoded bitstream to client <NUM> via network <NUM>. Client <NUM> decodes the encoded bitstream and generates video or image frames to drive to display <NUM> or to a display compositor.

Quantization is the mechanism used in video standards (e.g., high efficiency video coding (HEVC) standard, advanced video coding (AVC)) to control the size of an encoded video stream to meet the bandwidth requirements of a particular video application. This allows system <NUM> to send an encoded video stream from server <NUM> to client <NUM> in a consistent manner. It can be challenging to control the bit-rate of an encoded video stream while also providing an acceptable picture quality. In one implementation, the preferred bitcount of each video frame is equal to the bit-rate of the encoded video stream divided by the frame-rate of the video sequence. It is noted that the term "bitcount" is used interchangeably herein with the term "bit-size". In one implementation, server <NUM> adjusts the quantization parameter (QP) used to encode an input video sequence to control the bitcount of each frame of the encoded video stream. In this implementation, server <NUM> generates a model which maps bitcount to QP. Depending on the implementation, server <NUM> receives an indication of a desired bitcount or server <NUM> calculates a desired bitcount for each video frame. Once server <NUM> knows the desired bitcount of each video frame, server <NUM> uses the model to map the desired bitcount to a particular QP value. Then, server <NUM> sets the QP to this particular QP value when encoding a given video frame. In one implementation, server <NUM> generates a different model for each video frame (or a portion of each video frame). In other implementations, server <NUM> reuses a given model for multiple video frames.

Network <NUM> is representative of any type of network or combination of networks, including wireless connection, direct local area network (LAN), metropolitan area network (MAN), wide area network (WAN), an Intranet, the Internet, a cable network, a packet-switched network, a fiber-optic network, a router, storage area network, or other type of network. Examples of LANs include Ethernet networks, Fiber Distributed Data Interface (FDDI) networks, and token ring networks. In various implementations, network <NUM> includes remote direct memory access (RDMA) hardware and/or software, transmission control protocol/internet protocol (TCP/IP) hardware and/or software, router, repeaters, switches, grids, and/or other components.

Server <NUM> includes any combination of software and/or hardware for rendering video/image frames, generating a model mapping bitcount to QP, and/or encoding the frames into a bitstream using the QP provided by the model. In one implementation, server <NUM> includes one or more software applications executing on one or more processors of one or more servers. Server <NUM> also includes network communication capabilities, one or more input/output devices, and/or other components. The processor(s) of server <NUM> include any number and type (e.g., graphics processing units (GPUs), central processing units (CPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs)) of processors. The processor(s) are coupled to one or more memory devices storing program instructions executable by the processor(s). Similarly, client <NUM> includes any combination of software and/or hardware for decoding a bitstream and driving frames to display <NUM>. In one implementation, client <NUM> includes one or more software applications executing on one or more processors of one or more computing devices. In various implementations, client <NUM> is a computing device, game console, mobile device, streaming media player, or other type of device.

Turning now to <FIG>, a block diagram of one implementation of the software components of a server <NUM> for encoding frames of a video are shown. It is noted that in other implementations, server <NUM> includes other components and/or is arranged in other suitable manners than is shown in <FIG>. A new frame <NUM> of a video is received by server <NUM> and provided to pre-encoder <NUM> and encoder <NUM>. Each of pre-encoder <NUM> and encoder <NUM> is implemented using any suitable combination of hardware and/or software. Pre-encoder <NUM> generates quantization parameter (QP) <NUM> to be used by encoder <NUM> when encoding new frame <NUM>. In one implementation, pre-encoder <NUM> generates QP <NUM> based on model <NUM>. In one implementation, model <NUM> identifies a target QP <NUM> to use for achieving a specified bit rate or bit size of encoded frame. It is noted that the term "encoded frame" can also be referred to as a "compressed frame".

In one implementation, pre-encoder <NUM> receives pre-processed frame <NUM> and performs one or more operations on pre-processed frame <NUM>. In another implementation, pre-encoder <NUM> generates pre-processed frame <NUM> from new frame <NUM>. In one implementation, pre-processed frame <NUM> is a downsampled version of new frame <NUM>. In other implementations, pre-processed frame <NUM> represents new frame <NUM> after one or more other types of operations (e.g., filtering) are performed on new frame <NUM> other than or in addition to downsampling. In one implementation, pre-processed frame <NUM> is stored in memory <NUM>. Memory <NUM> is representative of any number and type of memory or cache device(s) for storing data and/or instructions associated with the encoding process. Depending on the implementation, pre-processed frame <NUM> corresponds to the entirety of the original new frame <NUM> or to a portion thereof.

In one implementation, pre-encoder <NUM> performs multiple pre-encoding passes on pre-processed frame <NUM> so as to generate model <NUM>. For example, in one implementation, pre-encoder <NUM> performs a first pre-encoding pass on pre-processed frame <NUM> with a first QP setting to determine the bit-size of the output frame for the first QP setting. Also, in this implementation, pre-encoder <NUM> performs a second pre-encoding pass on pre-processed frame <NUM> with a second QP setting to determine the bit-size of the resultant encoded frame for the second QP setting. It is assumed for the purposes of this discussion that the second QP setting is different from the first QP setting. Pre-encoder <NUM> can also perform additional pre-encoding passes with other QP settings. After capturing the bit-sizes of the encoded frames for the different passes at different QP settings, pre-encoder <NUM> generates model <NUM> to map QP setting to bit-size. Then, the desired bit size is provided to model <NUM> to generate a corresponding QP <NUM>.

Encoder <NUM> receives new frame <NUM> and encodes new frame <NUM> using a QP value equal to QP <NUM> generated by pre-encoder <NUM>. In one implementation, when encoder <NUM> starts encoding new frame <NUM>, encoder sets the starting QP value to be equal to the QP <NUM> generated by model <NUM>. Adjustments can be made to the starting QP value during the encoding of new frame <NUM> if encoder <NUM> determines that the amount of encoded data being generated is drifting too far from the target bit-size. The output of encoder <NUM> is encoded frame <NUM> which is conveyed to one or more clients (e.g., client <NUM> of <FIG>) and/or stored in memory. In one implementation, pre-encoder <NUM> generates a new model <NUM> and QP <NUM> for each new frame <NUM>. In another implementation, pre-encoder <NUM> reuses model <NUM> and QP <NUM> for one or more subsequent frames. In a further implementation, pre-encoder <NUM> generates model <NUM> from a portion of new frame <NUM> and then uses model <NUM> to generate QP <NUM> to be used for the encoding of other portions of new frame <NUM>. In a still further implementation, pre-encoder <NUM> generates multiple models <NUM> for new frame <NUM>, with each model <NUM> representing a different portion of new frame <NUM>. In this implementation, pre-encoder <NUM> generates multiple QP values <NUM> to be used for encoding the different portions of new frame <NUM>. Generally speaking, pre-encoder <NUM> generates QP value <NUM> for encoding new frame <NUM> based on a bit rate that has been selected for the resulting encoded bit stream. By selecting an appropriate starting QP value <NUM>, encoder <NUM> will typically make fewer QP adjustments during encoding of new frame <NUM>. This will help to improve the visual quality of the resulting encoded frame <NUM>.

Referring now to <FIG>, a block diagram of one implementation of encoding logic <NUM> is shown. An original input video frame is received by encoder <NUM> and downscaling unit <NUM>. In one implementation, the original input video frame is in the YUV color space. In other implementations, the original input video frame is encoded in other color spaces. Encoder <NUM> encodes the input video frame and generates an encoded bitstream which is conveyed to memory <NUM>. Memory <NUM> is representative of any number and type of memory devices. In one implementation, the encoded bitstream is formatted according to the high efficiency video coding (HEVC) standard. In one implementation, the encoded bitstream is formatted according to the advanced video coding (AVC) standard. In other implementations, the encoded bitstream is formatted according with other video coding standards.

Downscaling unit <NUM> generates a downscaled frame from the original input video frame and conveys the downscaled frame to memory <NUM>. A downscaled frame can be used to generate a model for mapping bitcount to QP due to the relationship shown in equation <NUM>. As indicated by equation <NUM>, for a given video frame, the ratio between the bitcount of a low resolution version of a given video frame and the bitcount of a high resolution version of the given video frame is a constant for a given QP. Accordingly, a downscaled frame is processed by pre-encoding unit <NUM> at different QPs and the relationship between the resulting bitcounts will be representative of the relationship between bitcounts for different QPs used to encode the original frame. In other implementations, downscaling unit <NUM> performs other types of filtering and/or preprocessing on the input video frame in addition to or other than downscaling. For example, in other implementations, downscaling unit <NUM> performs denoising, grayscale conversion, and/or other types of pre-processing steps.

The downscaled frame is conveyed to pre-encoding unit <NUM>. In one implementation, pre-encoding unit <NUM> performs at least two separate encodings of the downscaled frame using different QPs. In some implementations, pre-encoding unit <NUM> performs at least two separate encodings of a portion of the downscaled frame using different QPs. Then, based on the sizes of the encoded frames (or sizes of the encoded portions of the frame), pre-encoding unit <NUM> creates a model to map output bit-size to QP. The statistics for this model, labeled "encode statistics", are stored in memory <NUM>. The encode statistics are also conveyed to encoder <NUM>. Encoder <NUM> uses the encode statistics when determining which QP to select for the input video frame so as to meet a desired bit-size for the resulting encoded frame. In one implementation, the desired bit-size for each encoded frame is determined based on a desired bit-rate of the encoded bitstream generated by encoder <NUM>. For example, in one implementation, the desired bit-rate is specified in bits per second (e.g., <NUM> megabits per second (Mbps)) and the frame rate of the video sequence is specified in frames per second (fps) (e.g., <NUM> fps, <NUM> fps). In this implementation, encoder <NUM> divides the desired bit-rate by the frame rate to calculate a desired bit-size for each encoded frame.

It is noted that in other implementations, encoding logic <NUM> performs variations to the above-described techniques for selecting an optimal quantization strength for encoding video data to meet a given bit budget. For example, in another implementation, pre-encoding unit <NUM> encodes a portion of a frame with different quantization strength settings. Pre-encoding unit <NUM> then captures the bit-size of the encoded portion at the different quantization strength settings. In a further implementation, pre-encoding unit <NUM> encodes two or more frames with different quantization strength settings and then captures the bit-sizes of corresponding encoded frames.

Turning now to <FIG>, one example of a graph <NUM> of mapping bitcount to QP is shown. Plot <NUM> represents the bitcounts for different values of QP used for encoding a particular frame, a portion of a frame, or two or more frames of a video sequence. In one implementation, the relationship between bitcount and QP is modeled using the formula: bitcount = α2β*QP. In one implementation, the values for α and β are determined by experimentation. For example, in one implementation, the values for α and β are determined by pre-encoding a portion or the entirety of a frame using two or more different QP's in two or more different pre-encoding passes. The sizes of the encoded portion or entirety of the frame are captured, and then the QP's and captured bit-sizes are used to solve for the values of α and β in the above equation. Next, the values for α and β are used to generate a model which determines the relationship between bitcount and QP. When the encoder determines a desired bitcount for a given video frame, the encoder uses the model to map the desired bitcount to a particular QP value. The encoder then uses the particular QP value when encoding the given video frame so as to generate an encoded video frame with the desired bitcount.

Referring now to <FIG>, one implementation of a graph <NUM> for determining a QP value to match a target bitcount is shown. In one implementation, control logic or a pre-encoding unit implements two pre-encoding passes of a video frame with different QP values QP1 and QP2. The control logic captures the bitcounts for these two pre-encoding passes. In other implementations, the control logic performs more than two pre-encoding passes of a video frame with different QP values. The bitcount b1 corresponds to the pre-encoding pass with QP1 and is labeled with <NUM> on plot <NUM>. The bitcount b2 corresponds to the pre-encoding pass with QP2 and is labeled with <NUM> on plot <NUM>. After capturing bitcounts b1 and b2 for the pre-encoding passes with QP1 and QP2, respectively, the control logic generates plot <NUM> to map the relationship between bitcount and QP. Then, when a desired bitcount target <NUM> is determined, the control logic maps the target <NUM> to a QP value of QP3 on plot <NUM>. Then, the encoder sets the QP to the QP3 value when encoding the video frame and/or one or more subsequent video frames. Alternatively, the encoder uses the QP3 value when encoding one or more portions (e.g., blocks, coding units) of the video frame and/or one or more portions of other video frames.

Turning now to <FIG>, one implementation of a method <NUM> for generating a model to map bit size to a quantization parameter (QP) is shown. For purposes of discussion, the steps in this implementation and those of <FIG> are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method <NUM>.

A pre-encoder performs a first encoding of at least a portion of a first video frame using a first quantization parameter (QP) setting (block <NUM>). Then, the pre-encoder captures a first bit size of the encoded portion of the first video frame (block <NUM>). It is noted that the term "bit size" can also be referred to as "bitcount" herein. Next, the pre-encoder performs a second encoding of at least the portion of the first video frame using a second QP setting (block <NUM>). It is assumed for the purposes of this discussion that the second QP setting is different from the first QP setting. It is noted that the first and second encodings can also be referred to as "pre-encodings". Then, the pre-encoder captures a second bit size of the encoded portion of the first video frame (block <NUM>). Next, the pre-encoder generates a model from for mapping bit size to QP based on the relationships between the first and second QP's and the first and second bit sizes (block <NUM>). After block <NUM>, method <NUM> ends. It is noted that the "model" can also be referred to herein as a "mapping". In one implementation, the model is generated by solving for the values of α and β in the equation: bit-size = α2β*QP. For example, the values of α and β are solved using the first and second bit-sizes generated by the first and second encodings using the first and second QP's, respectively. In other implementations, the model is generated using other techniques.

In some implementations, the pre-encoder performs more than two different encodings with more than two different QP settings. The pre-encoder then uses the more than two different QP settings (and corresponding bit sizes) to generate the model of QP versus bit size. It is noted that method <NUM> can be performed on a regular or periodic basis, depending on the implementation. In one implementation, method <NUM> is performed for each portion of a given video frame. In another implementation, method <NUM> is performed for each video frame of a video stream. In a further implementation, method <NUM> is performed once every N video frames, wherein N is a positive integer greater than one. The frequency with which method <NUM> is performed can alternate between these examples based on one or more factors. In other implementations, subsequent iterations of method <NUM> are performed according to other schedules.

Referring now to <FIG>, one implementation of a method for encoding a video frame is shown. An encoder receives a video frame for encoding (block <NUM>). The encoder determines a preferred encoded video frame size for the video frame (block <NUM>). Depending on the implementation, the encoder receives a preferred encoded video frame size from another processing unit, the encoder retrieves the preferred encoded video frame size from a register or other memory location, the encoder calculates the preferred encoded video frame size based on a target bit rate of an encoded bitstream, or the encoder determines the preferred encoded video frame size in other manners.

The encoder determines which QP to select for encoding the received video frame based on a mapping between encoded video frame size and QP (block <NUM>). One example of how to generate a mapping which maps bit-size to QP is described in the above discussion regarding method <NUM> (of <FIG>). Next, the encoder generates an encoded video frame by setting a QP to a QP value provided by the model for the preferred encoded video frame size, wherein the encoded video frame represents the received video frame (block <NUM>). Depending on the implementation, the mapping is implemented in a variety of different manners. In one implementation, the mapping is setup directly by a software application. In another implementation, the mapping is implemented as a lookup table process derived by running a plurality of experiments. In a further implementation, the mapping is implemented as a closed-loop control method which checks whether the stimuli applier drives the system closer to a desired outcome (e.g., a preferred bit-rate). In other implementations, the mapping is implemented using machine learning techniques, a neural network, a regressive model, or other types of models that use quality metrics (e.g., peak signal-to-noise ratio (PSNR), structural similarity (SSIM) index, video multimethod assessment fusion (VMAF)) to optimize settings which impact perceptual quality.

After block <NUM>, the encoder conveys the encoded video frame to a decoder to be displayed (block <NUM>). After block <NUM>, method <NUM> ends. It is noted that method <NUM> can be repeated for each video frame received by the encoder. It is also noted that the mapping can be updated for each portion of the subsequent video frame, for each subsequent video frame, or after two or more video frames have been encoded.

In various implementations, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions can be represented by a high level programming language. In other implementations, the program instructions can be compiled from a high level programming language to a binary, intermediate, or other form. Alternatively, program instructions can be written that describe the behavior or design of hardware. Such program instructions can be represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog can be used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

Claim 1:
A system comprising:
a pre-encoder (<NUM>) configured to generate a mapping of frame size, measured as a bitcount, to quantization parameter, QP, based on a plurality of pre-encoding passes of a portion of a first video frame using different QP settings, wherein the mapping defines an exponential relationship between frame size and QP; and
an encoder (<NUM>) configured to:
determine a preferred encoded video frame size for the first video frame (<NUM>);
determine a given QP value from the mapping based on the preferred encoded video frame size;
generate an encoded video frame by setting a QP to the given QP value for at least a portion of the encoded video frame, wherein the encoded video frame is an encoded version of the first video frame; and
convey the encoded video frame to a decoder to be displayed.