Patent Description:
Unlike general video streaming applications where some delay is tolerated, cloud-based gaming requires that real-time interactivity be maintained. To address this, some cloud-based gaming platforms call for every frame to be encoded very close to a target frame size that matches the currently available network bandwidth. Encoding the frame at a size other than the target frame size can result in either undershooting (resulting in less bits than allowed) or overshooting (resulting in more bits than allowed), with overshooting potentially leading to packet losses and delays that degrade the gaming experience.

<CIT> discloses a multimedia system that allocates, during encoding of a multimedia stream, bits to portions of frames based on quality metrics and bit usages for different quantization parameters (QPs). An encoder of the multimedia system encodes a frame in a first pass with a first QP and in a second pass with a second QP. A comparator of the multimedia system measures and compares quality metrics, such as mean squared error, for each portion of the frame encoded with the first QP and the second QP. The comparator compares the difference between the quality metrics for each portion encoded with each QP to a threshold. If the difference in quality metrics for a portion exceeds the threshold, the comparator selects the portion for inclusion in a subset of portions to be encoded with the second QP.

<CIT> discloses systems, apparatuses, and methods for generating a model for determining a quantization strength to use when encoding video frames. A pre-encoder performs multiple encoding passes using different quantization strengths on a portion or the entirety of one or more pre-processed video frames. The pre-encoder captures the bit-size of the encoded output for each of the multiple encoding passes. Then, based on the multiple encoding passes, the pre-encoder generates a model for mapping bit-size to quantization strength for encoding video frames or portion(s) thereof. When the encoder begins the final encoding pass for one or more given video frames or any portion(s) thereof, the encoder uses the model to map a preferred bit-size to a given quantization strength. The encoder uses the given quantization strength when encoding the given video frame(s) or frame portion(s) to meet a specified bit-rate for the encoded bitstream.

<CIT> discloses a video encoding method for encoding a stream of baseband video data. The stream of baseband video data is received as a plurality of coding units. Statistics of each coding unit in the plurality of coding units are gathered. A quantization parameter (QP) for each coding unit is determined from the corresponding statistics. The coding unit is trial encoded using the QP to generate a trial encoded coding unit; and the QP is updated based on the trial encoded coding unit. Trial encoding the coding unit and updating the QP are repeated until the trial encoded coding unit meets a predetermined criterion. Then the coding unit is final encoded using the updated QP to generate a final encoded coding unit.

<CIT> discloses systems, apparatuses, and methods for calculating a quantization parameter (QP) for encoding video frames to meet a given bit budget. Control logic coupled to an encoder calculates a complexity indicator that represents a level of difficulty in encoding a previous video frame. The complexity indicator is based at least in part on a first parameter associated with the previous video frame and corresponds to one or more of a variance, an intra-prediction factor, and an inter-to-intra ratio. The complexity indicator is then used by the control logic to calculate a preferred QP to use to encode the current video frame to meet a given bit budget. By using the preferred QP generated based on the complexity indicator, the encoder is able to make fewer QP adjustments during the frame. This helps to improve the visual quality of the resulting encoded video bitstream.

The proposed solution relates to a computer-implemented method of claim <NUM>.

A computer-implemented method of the proposed solution includes performing a first pass encoding on a first frame of a set of frames for a streaming application and determining an initial quantization parameter based on the first pass encoding, i.e., using the result of first pass encoding. The method also includes deriving an alternative quantization parameter for the first frame based on at least one other frame of the set of frames. For example, an alternative quantization parameter for the first frame may be derived by using the initial quantization parameter as a start value for determining the alternative quantization taking into account an encoding result of at least one other frame of the set of frames. This may, for example, include determining one or more differences of the first frame to at least one previously encoded frame of the set frames and calculating an alternative quantization parameter being higher or lower than the initial quantization parameter depending on the one or more determined differences (e.g., differences in complexity). Additionally, the method includes performing a first encoding of the first frame using the initial quantization parameter to produce a first encoded frame and performing a second encoding of the first frame based on the alternative quantization parameter to produce a second encoded frame. Also, the method includes selecting between the first encoded frame and the second encoded frame for transmission as part of an encoded stream based on the streaming application. Further, the method includes transmitting the encoded stream. Based on the proposed solution a multi-try encoding process may be proposed including selecting an applicable encoded frame from multiple encoded frames for inclusion in a resulting stream.

In the method of the proposed solution, determining an initial quantization parameter based on the first pass encoding can include determining an estimated complexity for the first frame based on the first pass encoding. Additionally, the method includes determining a historical complexity associated with the at least one other frame and comparing the estimated complexity to the historical complexity. A historical complexity may for example be determined by a complexity of at least one previously encoded frame of the set of frames (and thus of at least one encoded frame preceding the first frame). In an example embodiment, the determined historical complexity may be calculated as an average for one or more historical complexities of previously encoded frames of the set of frames and comparing this average to the estimated complexity of the first frame currently to be encoded.

Further, the method includes determining a value for the alternative quantization parameter less than the initial quantization parameter in response to the historical complexity being greater than the estimated complexity. Also, the method includes determining a value for the alternative quantization parameter greater than the initial quantization parameter in response to the historical complexity being less than the estimated complexity.

In the method, the streaming application can include at least one of a target (frame) size (e.g., in bits), a target efficiency, a target reliability, or a target latency. For the streaming application one or more requirements, settings, or preferences may be predetermined, such as a target quality for the encoded frames, a target efficiency for the encoded frames, a target reliability for the encoded stream, or a target latency for the encoded stream. To help assure that, for example, a resulting encoded stream meets these one or more requirements, settings, or preferences of the streaming application, each frame may be encoded according to a rate control scheme so that each encoded frame is compressed to be close in size to a predetermined target bit size. In an example embodiment, this may result in selecting between the first encoded frame and the second encoded frame depending on determined frame sizes for first and second frames having a size between a target frame size and a tolerance. For example, in the method, selecting between the first encoded frame and the second encoded frame for transmission in an encoded stream based on the streaming application can thus include determining a target frame size based on the streaming application and comparing each of the first and second encoded frames to the target frame size. Selecting between the first encoded frame and the second encoded frame for transmission in an encoded stream can also include selecting the encoded frame that does not exceed an allowed number of bits based on the available bandwidth.

In the method, the alternative quantization parameter can be one of a plurality of alternative quantization parameters. Accordingly, a plurality of alternative quantization parameters may be derived based on the initial quantization parameter for encoding the first frame.

Further, the method can include deriving the plurality of alternative quantization parameters according to the set of frames. In an example embodiment, different, in particular two or more, alternative quantization parameters may be calculated based on different previously encoded frames of the set of frames.

Additionally, the encoded stream may be encoded based on a target bitrate. The method can additionally include comparing each the first and second encoded frames to the target bitrate and selecting between the first encoded frame and the second encoded frame further in response to the comparison of each the first and second encoded frames to the target bitrate.

In general, the encoded stream can be associated with a gaming session. Further, the first frame can represent at least a portion of a virtual environment associated with the gaming session. Additionally, the encoded stream may be transmitted to a client device associated with the gaming session.

Furthermore, a computer-implemented method is disclosed which includes determining an initial quantization parameter associated with a first frame of a set of frames. Additionally, such a method includes deriving an alternative quantization parameter for the first frame based on the initial quantization parameter. Accordingly, an alternative quantization parameter for the first frame may be derived by using the initial quantization parameter as a start value for determining the alternative quantization. This may, for example, include determining one or more deltas (delta values) to be added to or subtracted from the initial quantization parameter for the first frame for determining the one or more alternative quantization parameters. The additionally disclosed method also includes performing both a first encoding of the first frame based on the initial quantization parameter to produce a first encoded frame and a second encoding of the first frame based on the alternative quantization parameter to produce a second encoded frame. Both the first and second encodings may thus be performed by using the initial quantization parameter or alternative quantization parameter, respectively in an encoding operation. The additionally disclosed method additionally includes selecting between the first encoded frame and the second encoded frame for transmission as part of an encoded stream in response to a comparison of the first encoded frame and the second encoded frame to a target frame size. Also, the method can include transmitting the encoded stream.

In the method, determining the initial quantization parameter can include performing a first pass encoding on the first frame to produce a first pass encoded frame and determining the initial quantization parameter using the first pass encoded frame. Further, the method can include determining an estimated complexity for the first frame based on a statistic of the first pass encoded frame and determining the initial quantization parameter based upon the estimated complexity. Accordingly, the method can include determining an estimated complexity for the first frame by at least one parameter of a determined statistic of the first pass encoded frame being indicative for an amount or value of information in a frame that differs from one or more reference frames associated with the frame (e.g., used to encode the frame). The determined statistic of the first pass encoded frame may, for example, include a bit size of the first pass encoded frame, an energy of the first pass encoded frame, a signal-to-noise ratio of the first pass encoded frame, or any combination thereof. The initial quantization parameter shall then depend from the estimated complexity.

Further, in the method, deriving the alternative quantization parameter can also include determining a historical complexity associated with at least one other frame of the set of frames and comparing the historical complexity to the estimated complexity of the first frame. A historical complexity may for example be determined by a complexity of at least one previously encoded frame of the set of frames (and thus of at least one encoded frame preceding the first frame). In an example embodiment, the determined historical complexity may be calculated as an average for one or more historical complexities of previously encoded frames of the set of frames and comparing this average to the estimated complexity of the first frame currently to be encoded.

Also, the method can include selecting the alternative quantization parameter from between a first candidate alternative quantization parameter and a second candidate alternative quantization parameter in response to the comparison of the historical complexity to the estimated complexity of the first frame. As an example, the first candidate alternative quantization parameter can be greater than the initial quantization parameter and the second candidate alternative quantization parameter can be less than the initial quantization parameter. Further, the method can include selecting the second alternative quantization parameter in response to the historical complexity being greater than an estimated complexity of the first frame. Additionally, the method can include selecting the first alternative quantization parameter in response to the historical complexity being less than an estimated complexity of the first frame.

Methods herein can further include decoding the encoded stream to produce a decoded stream and displaying the decoded stream. Generally, the first encoding and the second encoding can both be performed within a predetermined duration. Additionally, the first encoding and the second encoding can be performed concurrently. Also, methods herein can include determining a complexity of the selected encoded frame and storing a representation of the complexity in a memory buffer. Further, methods herein can include performing a second (multi-try) encoding operation on a second frame of the set of frames using the stored representation of the complexity. Methods herein can further include determining an applicable quantization parameter of the selected encoded frame and storing the applicable quantization parameter in a memory buffer. Methods herein can also include performing a second (multi-try) encoding operation on a second frame of the set of frames using the stored applicable quantization parameter. According to embodiments, a representation of the complexity for a selected (applicable) encoded frame may thus be determined and stored in a memory buffer for use as a historical complexity in the encoding of a subsequent frame of the set of game frames. Likewise, in embodiments, a selected quantization parameter associated with the selected encoded frame (i.e., the quantization parameter used to encode the selected encoded frame) may be stored in a memory buffer for use as a historical quantization parameter in the encoding of a subsequent frame of the set of game frames.

According to example embodiments, a system can include one or more processors and a memory coupled to the one or more processors. The memory can store executable instructions configured to manipulate the one or more processors to perform the methods disclosed herein.

In additional example embodiments, a system can include a network interface couplable to a network and an encoder coupled to the network interface. The system may be configured to perform the methods disclosed herein.

Techniques and systems described herein address the demands of providing a video stream from one or more servers to a client system while maintaining the requirements, settings, and preferences of the streaming application. In particular, such techniques and systems described herein allow for encoding a set of frames at a server using a multi-try encoding operation to determine an applicable quantization parameter for each frame of the set of frames based on the streaming application. To better ensure that the resulting video stream meets the requirements, settings, and preferences of the streaming application, an initial quantization parameter for each frame is determined based on an estimated complexity determined during a first pass encoding. From the initial quantization parameter, additional, alternative quantization parameters are derived for each frame based on the encoding history of previously encoded frames from the set of frames. Encoders then perform multiple encodings of each frame according to the respective initial and alternative quantization parameters to produce multiple encoded frames. An applicable encoded frame is then selected from these encoded frames according to the streaming application. In this way, each frame is more accurately allocated the number of bits needed to result in a video stream that better meets the requirements, settings, and preferences of the streaming application.

To facilitate understanding, the techniques of the present disclosure are described in the example context of a cloud gaming system. A cloud-based or other remote server renders a stream of video frames representing the visual content of a video game instance being executed at that server or a related server and then encodes each frame using a multi-try encoding process described herein to generate a bitstream representing a stream of encoded rendered game frames for transmission to one or more client devices via one or more networks. However, it will be appreciated that the systems and techniques described herein are not limited to this example context, but instead may be implemented in any of a variety of video stream encoding/decoding systems using the guidelines provided herein.

<FIG> illustrates a cloud-based gaming system <NUM> for single-player or multiplayer (including massively multiplayer) gaming, according to some embodiments. Cloud-based gaming system <NUM> includes one or more servers <NUM>, with each server <NUM> communicatively coupled to one or more client systems <NUM> by at least one network <NUM>. Network <NUM> is configured to allow for the transmission and reception of data between any of servers <NUM> and client systems <NUM> and includes, for example, wired and wireless networks, such as Ethernet, the Internet, Wi-Fi, or any combination thereof. In embodiments, each server <NUM> is communicatively interconnected via a high-bandwidth, low-latency inter-server messaging bus. Servers <NUM> are typically distributed in one or more data centers over a geographical area so as to reduce transmission latency through physical proximity. Though in the illustrated embodiment, three servers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are presented supporting four client systems <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>, in other embodiments, any number of servers may be implemented supporting any number of client devices. It will be appreciated that in a typical real-world implementation, the quantity of servers <NUM> and quantity of client systems <NUM> typically will be considerably higher than presented in the example of <FIG>.

In the depicted embodiment, each server <NUM> includes a computing device such as, for example, a cloud server, virtual server, or any combination thereof configured to support one or more client gaming sessions executed on one or more client systems <NUM>. A "client gaming session", as used herein, includes a gaming application being played, partially simulated, or fully simulated on client system <NUM>. Each server <NUM> is configured to support this client gaming session by executing a corresponding game platform instance <NUM> that facilitates the execution of at least a portion of the gaming application being played, partially simulated, or fully simulated on the client system <NUM>. Such facilitation can include performing one or more operations related to the gaming application, such as, for example, rendering one or more frames related to the gaming application, performing calculations related to the gaming application (e.g., lighting calculations, artificial intelligence calculation, physics calculations, shading calculations, input calculations, and the like), providing access to files, or any combination thereof, to name a few. The game platform instance <NUM> provides various software and hardware resources to achieve such facilitation, such as communication/network management, resource management, media rendering encoding, and the like. In this way, game platform instance <NUM> simulates the execution of one or more operations of the gaming application for a corresponding player as though that gaming application was being played on a local gaming device, such as a personal computer ("PC"), game console, smartphone, tablet computer, automotive entertainment system, and the like.

Each client system <NUM> represents the hardware and software resources utilized to receive player input through manipulation of one or more input/output devices for at least one player, as well as to present the video and audio content representing the visual and auditory content, respectively, of the gameplay for the at least one player. Examples of a client system <NUM> include one or more desktop computers, notebook computers, tablet computers, virtual-reality systems, augmented reality systems, a compute-enabled cellular phone (i.e., a "smartphone"), a compute-enabled television (i.e., a "smart TV"), or any combination thereof, to name a few. As illustrated with reference to client system <NUM>-<NUM>, each client system <NUM> includes one or more client devices <NUM>. In the illustrated embodiment, client system <NUM>-<NUM> comprises a first client device <NUM>-<NUM>, which is communicatively coupled to, or otherwise associated with, display <NUM>, at least one input device <NUM> (e.g., a gamepad, joystick, keyboard, mouse, touchscreen), one or more network interfaces configured to couple to the network connecting the client system <NUM> to a corresponding server <NUM>, one or more processors, memory, storage, speakers, and other computing resources to render, process, and display scenes of a virtual environment. As illustrated with reference to client system <NUM>-<NUM>, each client system <NUM> can include a decoder <NUM> configured to decode one or more frames related to a virtual environment. Decoder <NUM> can include hardware and software configured to decode one or more encoding streams (e.g., game streams <NUM>) received from servers <NUM> so as to produce a decoded set of frames or decoded stream. Each decoder <NUM> is configured to decode any encoded frame encoded by any method or process disclosed herein. In embodiments, client system <NUM>-<NUM> further comprises a smartphone client device <NUM>-<NUM>, and a wearable virtual reality client device <NUM>-<NUM>, each of which may operate as an integrated mobile computing device having input facilities, output facilities, display facilities, and communication facilities analogous to those noted above with respect to client system <NUM>-<NUM>. In certain embodiments, client systems <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may include one or more facilities such as accelerometers, Global Positioning System (GPS) devices, and the like that are used to acquire motion data representing a movement of the client device, as well as a rate or acceleration of such movement.

While certain aspects described herein will be discussed with specific reference to cloud gaming scenarios, it will be appreciated that in certain embodiments the described techniques may be utilized in various non-gaming scenarios, such as if one or more of servers <NUM> and client systems <NUM> operate to render, process, and display other types of informational, educational, recreational and/or artistic content. It will therefore be further appreciated that while techniques are discussed herein with respect to the rendering of content that may utilize particular examples relevant to cloud gaming and gaming content, such discussions and techniques may be applied to such non-gaming scenarios. Examples provided herein may refer to scenarios involving the rendering, processing, and display of gaming content due to particular bandwidth and network latency issues relevant to such content and should not be construed to indicate that the techniques described are limited to those scenarios.

During operation, each server <NUM> executes a gaming platform instance <NUM> for one or more client gaming sessions. Executing game platform instance <NUM> includes rendering a set of frames that includes one or more gaming frames associated with the gaming application being executed on one or more respective client systems <NUM>. Each rendered gaming frame depicts at least a portion of a virtual environment used in the gaming application executed on the client system <NUM>. For example, each rendered gaming frame can depict at least a portion of a virtual environment displayed on a display <NUM> of a client system <NUM> during the client gaming session.

Each server <NUM> is configured to encode each rendered gaming frame via encoder <NUM> so as to generate a respective encoded set of frames (also referred to herein as "game stream" <NUM>). Each server <NUM> is configured to encode a game stream <NUM> through, for example, compression, reorganization, and manipulation of each frame rendered by gaming platform instance <NUM>. In embodiments, each encoder <NUM> of a server <NUM> implements one or more codecs so as to encode one or more rendered frames according to the one or more codecs. Such codecs can include H. <NUM>, VP9, AV1, or any combination thereof, to name a few. According to embodiments, each server <NUM> is configured to encode each frame rendered by gaming platform instance <NUM> using a multi-try encoding process which includes performing multiple encodings on a frame and selecting between those multiple encodings according to the current streaming application. As discussed in detail below with reference to <FIG>, a multi-try encoding process includes determining an initial quantization parameter and alternative quantization parameters for a current frame and performing multiple encodings of the current frame according to the determined initial quantization parameter and the alternative quantization parameters to produce multiple encoded frames. The multi-try encoding process further includes selecting an applicable encoded frame from the multiple encoded frames for inclusion in a resulting game stream <NUM>. Each resulting game stream <NUM> corresponds to a gaming application being executed on one or more client systems <NUM> and is provided to these corresponding client systems <NUM> via network <NUM>. The corresponding client systems <NUM> are each configured to decode a received game stream <NUM> via a decoder <NUM> and display the resulting decoded set of frames <NUM> on, for example, a display <NUM>. Each client system <NUM> is configured to decode a respective game stream <NUM> by compression, reorganization, and manipulation of the frames within the encoded stream according to one or more various video codecs including lossless and lossy codecs. According to embodiments, each client system <NUM> includes a decoder that implements one or more codecs so as to decode a received game stream <NUM> according to the one or more codecs. Such codecs can include H. <NUM>, VP9, AV1, or any combination thereof, to name a few. Though three game streams <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> are depicted in the illustrated embodiment, in other embodiments, servers <NUM> can generate any number of game streams <NUM> each corresponding to one or more client gaming sessions.

Referring now to <FIG>, a computing device <NUM> configured to encode and stream frames within a cloud-based gaming system is illustrated. In embodiments, computing device <NUM> implements aspects of cloud-based gaming system <NUM> as described in <FIG>. For example, computing device <NUM> may be similar or the same as server <NUM> described in <FIG>. Computing device <NUM> includes one or more software and hardware components for bi-directional communications including components for encoding a set of game frames <NUM> such as to produce a game stream <NUM>. In some embodiments, computing device <NUM> is part of an electronic device that supports encoding of a set game frames <NUM>, including, for example, a desktop computer, a notebook computer, a tablet, a server, or a game console, to name a few. In embodiments, computing device <NUM> includes processor <NUM>, modem <NUM>, and memory <NUM>. Memory <NUM> includes an electronic storage device, such as for example, a solid-state drive, a hard disk drive, random access memory ("RAM"), read-only memory ("ROM"), electronically erasable programmable ROM ("EEPROM"), optical storage device, or any combination thereof. Memory <NUM> includes instructions and data related to the operation of game platform instance <NUM>, encoders <NUM>, and rate control unit <NUM> such as, for example, codecs, reference frames, gaming engines, gaming applications, constants, and the like. Modem <NUM> is configured to be communicatively coupled to one or more client systems <NUM> via a network <NUM> and further configured to transmit a game stream <NUM> to the one or more client systems <NUM>. According to embodiments, processor <NUM>, modem <NUM>, and memory <NUM> are internally in electronic communication via one or more interfaces (e.g., a bus <NUM>).

According to embodiments, processor <NUM> includes one or more control processing units ("CPUs"), microprocessors, field-programmable gate arrays ("FPGAs"), graphics processing units ("GPUs"), application specific integrated circuits (ASICs), or any combination thereof and is configured to render and encode gaming frames for use in a client gaming session on cloud-based gaming system <NUM>. Processor <NUM> operates to execute a game platform instance <NUM>, the same or similar as game platform instance <NUM>, associated with a current client gaming session and configured to support a gaming application executed on one or more client systems <NUM>. Game platform instance <NUM> includes graphics hardware and software (not shown for clarity) to render a set of game frames <NUM> associated with an environment of the gaming application executed on the one or more client devices. Such graphics hardware and software include, for example, graphics cores, processing cores, pixel shaders, video random access memory ("VRAM"), GPUs, physics engines, lighting engines, tessellation engines, and the like. Each rendered game frame of the set of game frames <NUM> represents at least a portion of a virtual environment associated with the gaming application executed on the client device. For example, if the gaming application is a racing game, each game frame of the set of game frames <NUM> represents at least a portion of a racetrack, car, or surrounding area.

Each rendered game frame of the set of game frames <NUM> is provided to encoders <NUM> for encoding into a game stream <NUM>. Encoders <NUM> includes one or more software and/or hardware encoders (e.g., encoders <NUM>-<NUM> to <NUM>-N) configured to encode game stream <NUM> according to, for example, interframe and intraframe techniques. Though in the illustrated embodiment, three encoders <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-N are presented, in other embodiments, any number of encoders may be implemented. To reduce the bandwidth needed to transmit a game stream <NUM> between computing device <NUM> and one or more client systems <NUM>, encoders <NUM> encode game stream <NUM> by compressing one or more game frames of the set of game frames <NUM>. Compressing a game frame includes comparing the game frame to one or more reference frames stored in memory buffer <NUM> and encoding one or more of the differences between the game frame and the one or more reference frames into game stream <NUM>. Encoders <NUM> are further configured to encode the reference frames used into game stream <NUM>. In embodiments, encoders <NUM> are configured to encode game stream <NUM> based on one or more streaming applications. A "streaming application" as used herein, includes one or more requirements, settings, preferences, or any combination thereof, for a resulting game stream <NUM> associated with a current client gaming session. These one or more requirements, settings, preferences are based on, for example, a gaming application associated with the client gaming session, hardware capabilities, codecs implemented by encoders <NUM>, or any combination thereof and include, for example, a target quality for the encoded frames of a game stream <NUM>, a target efficiency for the encoded frames of a game stream <NUM>, a target reliability of a game stream <NUM>, or a target latency for a game stream <NUM>. To help assure that a resulting game stream <NUM> meets one or more requirements, settings, or preferences of the streaming application, encoders <NUM> are configured to encode each frame according to a rate control scheme so that each encoded frame is compressed to be close in size to a predetermined target bit size. The rate control scheme is implemented by rate control unit <NUM> which includes one or more processors, hard-coded logic, programmable logic, or any combination thereof, configured to control, or adjust, the bitrate and a number of respective bits (i.e., degree of compression) at which to encode a current frame of the set of game frames <NUM>. Rate control unit <NUM> controls or adjusts the bitrate and the number of respective bits by determining an applicable quantization parameter ("QP") for the current frame.

In embodiments, rate control unit <NUM> determines the applicable QP for a current frame based on a multi-try encoding operation performed by encoders <NUM>. The multi-try encoding operation includes encoders <NUM> performing multiple encodings on a current frame to produce multiple encoded frames. Each of the encoders <NUM> performs an encoding on the current frame according to a respective QP to produce multiple encoded frames (i.e., a plurality of encoded frames), with one encoder using an initial QP and one or more other encoders using alternative QPs. For example, a first encoder <NUM>-<NUM> performs an encoding on the current frame using an initial QP and a second encoder <NUM>-<NUM> performs an encoding on the current frame using an alternative QP. Rate control unit <NUM> then selects an applicable encoded frame from the resulting encoded frames based on the streaming application, with the applicable encoded frame representing the applicable QP (i.e., the QP used to encode the applicable encoded frame). It is important that rate control unit <NUM> selects an encoded frame from the resulting encoded frames that best meets one or more requirements, settings, or preferences of a streaming application application. For example, rate control unit <NUM> selects a resulting encoded frame that does not exceed an allowed number of bits based on the available bandwidth.

Rate control unit <NUM> determines the initial QP for a current frame based on a complexity of the current frame. A "complexity", as used herein, refers to an amount or value of information in a frame that differs from one or more reference frames associated with the frame (e.g., used to encode the frame). In this way, the higher the complexity of the frame, the more information within the frame exists for the encoders <NUM> to encode. In embodiments, rate control unit <NUM> determines an initial QP for a current frame according to the equation: <MAT> wherein t represents a predetermined, desired frame size, or degree of compression, for an encoded frame, a<NUM> and a<NUM> represent predetermined constants derived from past encoded frames' data, and c represents a complexity, or distortion, of the current frame. In embodiments, rate control unit <NUM> is configured to determine an estimated complexity for a current frame based upon a first pass encoding of the current frame by encoders <NUM>. In some embodiments, encoders <NUM> perform the first pass encoding at a lower resolution such as by first downsampling the current frame before the first pass encoding. The first pass encoding produces a first pass encoded frame that includes one or more statistics such as, for example, a bit size of the first pass encoded frame, an energy of the first pass encoded frame, a signal-to-noise ratio of the first pass encoded frame, or any combination thereof, to name a few. According to embodiments, rate control unit <NUM> is configured to determine an estimated complexity for the current frame based on the statistics of the first pass encoded frame. For example, rate control unit <NUM> determines an estimated complexity for the current frame based on the bit size of the first pass encoded frame. From the estimated complexity, rate control unit <NUM> determines an initial QP for the current frame. For example, rate control unit <NUM> determines an initial QP for the current frame according to the equation presented in EQ1 using the estimated complexity as complexity c.

From the initial QP, rate control unit <NUM> derives one or more alternative QPs for the current frame. The rate control unit <NUM> derives the one or more alternative QPs based on the complexities of one or more previously encoded frames from the set of game frames <NUM> (also referred to herein as "historical complexities" <NUM>), the applicable QPs of one or more previously encoded frames from the set of game frames <NUM> (also referred to herein as "historical QPs" <NUM>), one or more codecs implemented by encoders <NUM>, or any combination thereof. For example, rate control unit <NUM> derives one or more alternative QPs for a current frame by comparing the estimated complexity of the frame to one or more historical complexities <NUM>. In embodiments, deriving the alternative QPs includes rate control unit <NUM> determining whether an alternative QP is to be greater than or less than the initial QP of the current frame. For example, in response to a historical complexity being less than the estimated complexity of the current frame, rate control unit <NUM> determines that an alternative QP greater than the initial QP is to be derived. Such a determination typically is necessary when an odd number of alternative QPs are to be derived. Deriving the alternative QPs also includes determining one or more differences, or deltas, relative to the initial QP. In embodiments, these differences, or deltas, are based on one or more codecs implemented by encoders <NUM>, hardware specifications, or both. Rate control unit <NUM> applies the deltas to the initial QP to determine the values for one or more alternative QPs. In other words, rate control unit <NUM> determines values for alternative QPs relative to the initial QPs according to the determined deltas.

Multiple encoders <NUM> then perform multiple encodings on the current frame according to the initial QP and the alternative QPs, respectively, to produce a plurality of encoded frames. In embodiments, the encoders <NUM> performing the multiple encodings are each configured to complete their respective encodings within a predetermined time frame. That is to say, the encoders <NUM> complete all the multiple encodings within a predetermined time frame. According to embodiments, encoders <NUM> are configured to perform one or more of the multiple encodings concurrently. Rate control unit <NUM> then selects an applicable encoded frame from the plurality of encoded frames that best meets one or more requirements, settings, or preferences of a streaming application. In embodiments, rate control unit <NUM> selects an applicable encoded frame from the plurality of encoded frames by comparing each of the encoded frames to a target bitrate that represents the one or more requirements, settings, or preferences of a streaming application. That is to say, a target bitrate that helps encoders <NUM> to produce a game stream <NUM> that meets one or more requirements, settings, or preferences of a streaming application. According to embodiments, rate control unit <NUM> compares each of the encoded frames to a range derived from a target bitrate and an overshoot factor. An "overshoot factor," as used herein, represents a tolerance within cloud-based gaming system <NUM> for overshoots (i.e., when at least a portion of a game stream <NUM> requires more bandwidth than allowed).

In embodiments, the applicable encoded frame is included in game stream <NUM> for transmission to one or more client systems <NUM>. According to embodiments, a representation of the complexity for the applicable encoded frame is determined and stored in memory buffer <NUM> for use as a historical complexity <NUM> in the encoding of a subsequent frame of the set of game frames <NUM>. Likewise, in embodiments, the applicable QP associated with the applicable encoded frame (i.e., the QP used to encode the applicable encoded frame) is stored in memory buffer <NUM> for use as a historical QP <NUM> in the encoding of a subsequent frame of the set of game frames <NUM>.

Referring now to <FIG>, a flow diagram for a multi-try encoding operation <NUM> is illustrated. For ease of illustration, the multi-try encoding operation <NUM> is described with reference to the computing device <NUM> of <FIG> implemented as a server <NUM> in the cloud-based gaming system <NUM> of <FIG>. In operation <NUM>, the game platform instance <NUM> renders a set of frames <NUM> for a client gaming session associated with a gaming application running on one or more client systems <NUM>. Each frame of the set of frames <NUM> represents at least a portion of a virtual environment related to the gaming application. To facilitate the transmission of the set of frames <NUM> to the client system executing the gaming application, encoders <NUM> encode each frame of the set of frames <NUM> using a multi-try encoding operation <NUM>. In doing so, the resulting game stream <NUM> can better meet the one or more requirements, settings, or preferences of a streaming application. For example, the bandwidth of a resulting game stream <NUM> can be more closely matched to the actual available bandwidth.

For a current frame (also referred to herein as a "first frame") <NUM> of the set of frames <NUM>, rate control unit <NUM> performs a determination <NUM> of an initial quantization parameter for the current frame <NUM>. In embodiments, rate control unit <NUM> performs the determination <NUM> of an initial quantization parameter for the current frame <NUM> based on a first pass encoding of the current frame (i.e., first frame) <NUM>. For example, referring now to <FIG>, a flow diagram of an example determination <NUM> of an initial quantization parameter is illustrated. For a current frame (also referred to herein a "first frame") <NUM> of a set of frames <NUM>, similar or the same as current frame <NUM> and set of frames <NUM>, respectively, the encoders <NUM> perform a first pass encoding <NUM> based on rate control <NUM>. Rate control <NUM> includes rate control unit <NUM> determining initial QPs for the one or more encoders <NUM> to encode the set of frames <NUM>. For the current frame (i.e., first frame) <NUM>, rate control unit <NUM> first determines a first pass QP based on one or more codecs implemented by encoders <NUM>, target bitrate <NUM>, or both. Target bitrate <NUM> represents a bitrate for a resulting game stream <NUM> that helps the game stream <NUM> meet the one or more requirements, settings, or preferences of the streaming application. According to some embodiments, the first pass QP is a constant value determined from the codecs implemented by the encoders <NUM>. After rate control unit <NUM> has determined the first pass QP, encoders <NUM> perform the first pass encoding <NUM> on the current frame <NUM> according to the first pass QP so to produce a first pass encoded frame <NUM>. The first pass encoded frame <NUM> includes one or more statistics such as, for example, the bit size of the first pass encoded frame, an energy of the first pass encoded frame, a signal-to-noise ratio of the first pass encoded frame, or any combination thereof, to name a few. Based on the statistics of the first pass encoded frame <NUM> and the target bit rate, rate control unit <NUM> adjusts the rate control <NUM> by determining an initial QP for the current frame <NUM>. That is to say, rate control unit <NUM> determines an initial QP for the current frame <NUM> based on the statistics of the first pass encoded frame <NUM> and the target bit rate. According to embodiments, determining an initial QP includes rate control unit <NUM> determining an estimated complexity of the current frame <NUM> according to the statistics of the first pass encoded frame <NUM>. For example, rate control unit <NUM> determines an estimated complexity for the current frame <NUM> based on the bit size of the first pass encoded frame <NUM>. In embodiments, rate control unit <NUM> then determines an initial QP from the estimated complexity and the target bitrate <NUM>.

Referring again to <FIG>, after an initial QP is determined for the current frame <NUM>, rate control unit <NUM> performs a derivation <NUM> of one or more alternative QPs for the current frame <NUM>. According to embodiments, rate control unit <NUM> performs the derivation <NUM> of one or more alternative QPs based on the initial QP produced from the determination <NUM>. In embodiments, rate control unit <NUM> performs the derivation <NUM> of one or more alternative QPs by first determining the number of alternative QPs to produce. In other words, rate control unit <NUM> determines how many alternative QPs are to be derived from the initial QP. In some embodiments, the number of alternative QPs is a predetermined value. According to embodiments, rate control unit <NUM> determines whether one or more of the alternative QPs to be derived will be greater than the initial QP or less than the initial QP. For example, when the number of alternative QPs to be derived is <NUM>, rate control unit <NUM> determines whether that alternative QP will be greater or less than the initial QP. That is to say, for example, rate control unit <NUM> selects between a candidate alternative QP greater than the initial QP and a candidate alternative QP less than the initial QP. Rate control unit <NUM> determines whether one or more of the alternative QPs will be greater than the initial QP or less than the initial QP by comparing one or more historical complexities <NUM> associated with the set of frames <NUM> to the estimated complexity of the current frame <NUM>, comparing one or more historical QPs <NUM> associated with the set of frames <NUM> to the initial QP of the current frame <NUM>, or both. In embodiments, this comparison includes, for example, determining one or more trends, averages, modes, medians, or any combination thereof, in the historical complexities <NUM> and historical QPs <NUM> and comparing those trends, averages, modes, medians, or any combination thereof, to an estimated complexity or initial QP of the current frame <NUM>. For example, comparing one or more historical complexities <NUM> associated with the set of frames <NUM> to the estimated complexity of the current frame <NUM> includes determining an average for one or more historical complexities <NUM> and comparing the average to the estimated complexity of the current frame <NUM>. Based on the comparison of the one or more historical complexities <NUM> associated with the set of frames <NUM> to the estimated complexity of the current frame <NUM> or the one or more historical QPs <NUM> associated with the set of frames <NUM> to the initial QP of the current frame <NUM>, rate control unit <NUM> determines whether one or more of the alternative QPs will be greater than the initial QP or less than the initial QP (e.g., selects between a first candidate alternative QP greater than the initial QP and a second candidate alternative QP less than the initial QP). For example, in response to the comparison indicating that the current frame <NUM> has a higher complexity than previously encoded frames from the set of frames <NUM>, rate control unit <NUM> determines that one or more of the alternative QPs will be greater than the initial QP, such as, for example, to compensate for the increase in complexity. Likewise, for example, in response to the comparison indicating that the current frame <NUM> has a lower complexity than previous encoded frames from the set of frames <NUM>, rate control unit <NUM> determines that one or more of the alternative QPs will be less than the initial QP. As another example, in response to the initial QP of the current frame <NUM> being less than one or more historical QPs <NUM>, rate control unit <NUM> determines that one or more of the alternative QPs will be greater than the initial QP such as, for example, to aid in avoiding overshoots in the resulting game stream <NUM>.

In embodiments, rate control unit <NUM> further derives the alternative QPs by determining one or more differences, or deltas, relative to the initial QP. In embodiments, these differences, or deltas, are based on one or more codecs implemented by encoders <NUM>, hardware, or both. Based on these deltas, rate control unit <NUM> determines the values for one or more alternative QPs. In other words, rate control unit <NUM> determines values for alternative QPs relative to the initial QPs according to the determined deltas.

After rate control unit <NUM> determines the one or more alternative QPs for the current frame <NUM>, encoders <NUM> perform multiple encodings <NUM>-<NUM> to <NUM>-N on the current frame according to the initial QP and the alternative QPs to produce a plurality of encoded frames. According to embodiments, respective encoders of encoders <NUM> perform an encoding <NUM> according to either the initial QP or a respective alternative QP. For example, encoder <NUM>-<NUM> performs an encoding <NUM> on the current frame <NUM> according to the initial QP, and encoder <NUM>-<NUM> performs an encoding <NUM> on the current frame <NUM> according to an alternative QP. In embodiments, encoders <NUM> are configured to perform all the encodings <NUM>-<NUM> to <NUM>-N within a predetermined time frame or amount of time. Further, in some embodiments, encoders <NUM> are configured to perform one or more of the encodings <NUM> concurrently.

From the resulting encoded frames produced by the multiple encodings <NUM>, rate control unit <NUM> performs a selection <NUM> of an applicable encoded frame according to the streaming application. For example, rate control unit <NUM> selects the encoded frame that best helps the resulting game stream <NUM> meet the one or more requirements, settings, or preferences of the streaming application. In embodiments, rate control unit <NUM> selects the applicable encoded frame based on the target bitrate and a tolerance associated with the cloud-based gaming system <NUM>. As an example, referring now to <FIG>, an example selection <NUM> of an applicable encoding is presented. At block <NUM>, rate control unit <NUM> receives the plurality of encoded frames produced from the multiple encodings <NUM>. At block <NUM>, rate control unit determines whether any of the encoded frames of the plurality of encoded frames have a size between a target frame size and a tolerance. That is to say, rate control unit <NUM> determines whether there is a group of encoded frames within the plurality of encoded frames that each have a size between a target frame size and a tolerance. In some embodiments, rate control unit <NUM> determines the target frame size based on the target bitrate with the target frame size indicating how many bits an encoded frame is allowed without exceeding the available bandwidth. The tolerance represents the number of bits the cloud-based gaming system <NUM> can tolerate according to its overshoot factor. In embodiments, rate control unit <NUM> determines the tolerance according to the equation: <MAT> wherein f represents the overshoot factor for the cloud-based gaming system <NUM> and T represents the target frame size. If any of the encoded frames from the plurality of encoded frames has a size between the target frame size and the tolerance (e.g., between [T, Tolerance]), rate control unit <NUM> moves to block <NUM>. At block <NUM>, from the group of encoded frames having a size between the target frame size and the tolerance, rate control unit <NUM> selects the encoded frame closest in size to the target frame size to produce applicable encoded frame <NUM>. If none of the encoded frames from the plurality of encoded frames have a size between the target frame size and the tolerance rate, rate control unit <NUM> moves to block <NUM>.

At block <NUM>, rate control unit <NUM> determines whether the size of any of the encoded frames is between <NUM> and the target size (e.g., between [<NUM>, T]). That is to say, rate control unit <NUM> determines whether there is a group of encoded frames within the plurality of encoded frames that each have a size between <NUM> and a target frame size. If any of the encoded frames from the plurality of encoded frames have a size between <NUM> and the target frame size, rate control unit <NUM> moves to block <NUM>. At block <NUM>, from the group of encoded frames having a size between <NUM> and the target size, rate control unit <NUM> selects the encoded frame having a size closest to the target frame size to produce applicable encoded frame <NUM>. If none of the encoded frames from the plurality of encoded frames have a size between <NUM> and the target frame size, rate control unit <NUM> moves to block <NUM>. At block <NUM>, from the plurality of encoded frames (i.e., all the encoded frames produced during the multiple encodings), rate control unit <NUM> selects the encoded frame having a size closest to the target bitrate to produce applicable encoded frame <NUM>.

Referring again to <FIG>, in embodiments, the applicable encoded frame <NUM> is transmitted as part of a resulting game stream <NUM> to one or more client systems <NUM>. According to embodiments, rate control unit <NUM> stores the applicable QP of the applicable encoded frame <NUM> (i.e., the QP used to encode the applicable encoded frame <NUM>) in memory buffer <NUM> for use as a historical QP <NUM> to encode subsequent frames from the set of frames <NUM>. In embodiments, rate control unit <NUM> determines a complexity for the applicable encoded frame <NUM> based on the applicable QP, the reference frames used to encode the applicable encoded frame <NUM>, or both. Rate control unit <NUM> stores the determined complexity for the applicable encoded frame <NUM> in memory buffer <NUM> for use as a historical complexity <NUM> to encode subsequent frames from the set of frames <NUM>.

Referring now to FIG. <NUM>, a flow diagram illustrating an example multi-try encoding method <NUM> is presented. For ease of illustration, the method <NUM> is described with reference to the computing device <NUM> of <FIG> implemented as a server <NUM> of the system <NUM> of <FIG>. At block <NUM>, encoders, the same or similar as the encoders <NUM>, receive a current frame (i.e., a first frame). The current frame represents at least a portion of a virtual environment associated with a client gaming session. At block <NUM>, a rate control unit, similar or the same as rate control unit <NUM>, determines an initial QP for the current frame. In embodiments, the rate control unit determines the initial QP based on an estimated complexity of the current frame and a target bitrate. The target bitrate represents a bitrate that aids in a resulting game stream <NUM> meeting the one or more requirements, settings, or preferences of a streaming application. According to embodiments, the rate control unit determines an estimated complexity of the current frame based on a first pass encoding of the current frame performed by the encoders. From the first pass encoding, the encoders produce a first pass encoded frame including one or more statistics such as, for example, the bit size, an energy, a signal-to-noise ratio, or any combination thereof, of the first pass encoded frame, to name a few. Based on the statistics of the first pass encoded frame, the rate control unit determines an estimated complexity for the current frame which is then used to determine an initial QP for the current frame.

At block <NUM>, after determining an initial QP for the current frame, rate control unit <NUM> derives one or more alternative QPs for the current frame based on the initial QP. In embodiments, the rate control unit determines whether one or more alternative QPs will be greater or less than the initial QP for the current frame. For example, the rate control unit selects between a first candidate alternative QP greater than the initial QP and a second candidate alternative QP less than the initial QP. One of ordinary skill in the art will appreciate that such a determination is necessary when an odd number of alternative QPs are derived, such as, for example, when only one alternative QP is derived. According to embodiments, the rate control unit determines whether one or more alternative QPs will be greater or less than the initial QP based on a comparison of the estimated complexity of the current frame to one or more historical complexities <NUM>, a comparison of the initial QP for the current frame to one or more historical QPs <NUM>, or both. For example, in response to the initial QP for the current frame being less than one or more historical QPs <NUM>, rate control unit <NUM> determines that an alternative QP will be greater than the initial QP such as, for example, to help prevent overshoots in the resulting game stream <NUM>. According to embodiments, the rate control unit also determines one or more differences, or deltas, relative to the initial QP. In embodiments, these differences, or deltas, are based on one or more codecs implemented by the encoders, hardware specification, or both. Based on these deltas, the rate control unit determines the values for one or more alternative QPs relative to the initial QP.

At block <NUM>, the encoders perform multiple, distinct encodings on the current frame according to the initial QP and the alternative QPs to produce a plurality of encoded frames. In embodiments, the multiple encodings include one or more respective encoders encoding the current frame according to the initial QP and one or more respective alternative QPs. According to embodiments, the encoders perform all of the multiple encodings within a predetermined amount of time. Further, in embodiments, one or more encoders perform their respective encodings concurrently. At block <NUM>, the rate control unit selects an encoded frame from the plurality of encoded frames according to the streaming application to produce an applicable encoded frame. That is to say, the rate control unit selects an applicable encoded frame that best helps game stream <NUM> meet one or more requirements, settings, or preferences of a streaming application. In embodiments, selecting the applicable encoded frame includes the rate control unit determining whether the size of one or more encoded frames of the plurality of encoded frames are within ranges based on a target frame size determined from the target bitrate and a tolerance. In other words, the rate control unit determines whether any sub-groups of encoded frames from the plurality of encoded frames are within these ranges. For example, the rate control unit determines whether the size of any encoded frames from the plurality of encoded frames is between the target frame size and the tolerance or between <NUM> and the target frame size. In response to the rate control unit determining that the size of one or more encoded frames is within a range, the rate control unit then selects the encoded frame from the sub-group closest in size to the target frame size to produce the applicable encoded frame. In response to the rate control unit determining that the size of none of the encoded frames of the plurality of encoded frames is within a range, the rate control unit selects the encoded frame from the plurality of encoded frames closest in size to the target frame size to produce the applicable encoded frame.

At block <NUM>, the rate control unit determines a complexity for the applicable encoded frame based on the target bit rate, the reference frames used to encode the applicable encoded frame, or both. The complexity is then stored in a memory buffer (such as by storing a representation of the complexity in the memory buffer), similar to or the same as memory buffer <NUM>, for use as a historical complexity for encoding a subsequent frame from the set of frames. Likewise, the rate control unit stores the applicable QP of the applicable encoded frame (i.e., the QP used to encode the applicable encoded frame) in the memory buffer for use as a historical QP for encoding a subsequent frame from the set of frames. A computer device, the same or similar as computing device <NUM>, transmits the applicable encoded frame to one or more client systems <NUM> as part of game stream <NUM>.

In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software comprises one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium can include, for example, a magnetic or optical disk storage device, solid-state storage devices such as Flash memory, a cache, random access memory (RAM), or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer-readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.

Claim 1:
A computer- implemented method, comprising:
performing a first pass encoding on a first frame (<NUM>, <NUM>) of a set of frames (<NUM>, <NUM>) for a streaming application;
determining an initial quantization parameter (<NUM>) based on the first pass encoding;
deriving an alternative quantization parameter (<NUM>) for the first frame based on at least one other frame of the set of frames;
performing a first encoding of the first frame using the initial quantization parameter to produce a first encoded frame (<NUM>) and performing a second encoding of the first frame using the alternative quantization parameter to produce a second encoded frame;
selecting between the first encoded frame and the second encoded frame for transmission as part of an encoded stream (<NUM>) based on the streaming application; and
transmitting the encoded stream,
wherein determining an initial quantization parameter based on the first pass encoding comprises determining an estimated complexity for the first frame based on the first pass encoding; and
characterized by
deriving an alternative quantization parameter for the first frame based on at least one other frame of the set of frames comprises:
determining a historical complexity associated with the at least one other frame; and
comparing the estimated complexity to the historical complexity, wherein in response to the historical complexity being greater than the estimated complexity, a value for the alternative quantization parameter less than the initial quantization parameter is determined, and, in response to the historical complexity being less than the estimated complexity, a value for the alternative quantization parameter greater than the initial quantization parameter is determined.