Patent Description:
As video becomes increasingly more common in a wide range of applications, video streams may need to be encoded and/or decoded several times depending on the application. For example, different applications and/or devices may need to comply with bandwidth or resource constraints. In order to meet these demands requiring several combinations of settings without being prohibitively expensive, high efficiency codecs have been developed that compress video into several resolutions. With codecs such as scalable VP9 and H. <NUM>, video bitstreams may contain multiple spatial layers that allow a user to reconstruct the original video at different resolutions (i.e. the resolution of each spatial layer). By having scalable capability, video content may be delivered from device to device with limited further processing.

One aspect of the disclosure provides a method for allocating bit rate. The method includes receiving, at data processing hardware, transform coefficients corresponding to a scaled video input signal, the scaled video input signal including a plurality of spatial layers, the plurality of spatial layers comprising a base layer. The method also includes determining, by the data processing hardware, a spatial rate factor based on a sample of frames from the scaled video input signal. The spatial rate factor defines a factor for bit rate allocation at each spatial layer of an encoded bit stream formed from the scaled video input signal. The spatial rate factor is represented by a difference between a rate of bits per transform coefficient of the base layer and an average rate of bits per transform coefficient for the plurality of spatial layers. The method also includes reducing a distortion for the plurality of spatial layers of the encoded bit stream by allocating a bit rate to each spatial layer based on the spatial rate factor and the sample of frames.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the method also includes receiving, at the data processing hardware, a second sample of frames from the scaled video input signal; modifying, by the data processing hardware, the spatial rate factor based on the second sample of frames from the scaled video input signal; and allocating, by the data processing hardware, a modified bit rate to each spatial layer based on the modified spatial rate factor and the second sample of frames. In additional implementations, the method also includes receiving, at the data processing hardware, a second sample of frames from the scaled video input signal; modifying on a frame-by-frame basis, by the data processing hardware, the spatial rate factor based on an exponential moving average, the exponential moving average corresponding to at least the sample of frames and the second sample of frames; and allocating, by the data processing hardware, a modified bit rate to each spatial layer based on the modified spatial rate factor.

In some examples, receiving the scaled video input signal includes receiving a video input signal, scaling the video input signal into the plurality of spatial layers, partitioning each spatial layer into sub-blocks, transforming each sub-block into transform coefficients, and scalar quantizing the transform coefficients corresponding to each sub-block. Determining the spatial rate factor based on the sample of frames from the scaled video input signal may include determining variance estimations of each scalar quantized transform coefficient based on an average across all transform blocks of frames of the video input signal. Here, the transform coefficients of each sub-block may be identically distributed across all sub-blocks.

In some implementations, the method also includes determining, by the data processing hardware, that the spatial rate factor satisfies a spatial rate factor threshold. In these implementations, a value corresponding to the spatial rate factor threshold may satisfy the spatial rate factor threshold when the value is less than about <NUM> and greater than about <NUM>. The spatial rate factor may include a single parameter configured to allocate the bit rate to each layer of the encoded bit stream. In some examples, the spatial rate factor includes a weighted sum corresponding to a ratio of a product of variances, wherein the ratio includes a numerator based on estimated variances of scalar quantized transform coefficients from a first spatial layer and a denominator based on estimated variances of scalar quantized transform coefficients from a second spatial layer.

Another aspect of the disclosure provides a system for allocating bit rate. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed by the data processing hardware cause the data processing hardware to perform operations. The operations include receiving transform coefficients corresponding to a scaled video input signal, the scaled video input signal including a plurality of spatial layers, the plurality of spatial layers comprising a base layer. The operations also include determining a spatial rate factor based on a sample of frames from the scaled video input signal. The spatial rate factor defines a factor for bit rate allocation at each spatial layer of an encoded bit stream formed from the scaled video input signal. The spatial rate factor is represented by a difference between a rate of bits per transform coefficient of the base layer and an average rate of bits per transform coefficient for the plurality of spatial layers. The operations also include reducing a distortion for the plurality of spatial layers of the encoded bit stream by allocating a bit rate to each spatial layer based on the spatial rate factor and the sample of frames.

This aspect may include one or more of the following optional features. In some implementations, the operations also include receiving a second sample of frames from the scaled video input signal, modifying the spatial rate factor based on the second sample of frames from the scaled video input signal, and allocating a modified bit rate to each spatial layer based on the modified spatial rate factor and the second sample of frames. In additional implementations, the operations also include receiving a second sample of frames from the scaled video input signal; modifying on a frame-by-frame basis the spatial rate factor based on an exponential moving average, the exponential moving average corresponding to at least the sample of frames and the second sample of frames; and allocating a modified bit rate to each spatial layer based on the modified spatial rate factor.

In some implementations, the operations also include determining that the spatial rate factor satisfies a spatial rate factor threshold. In these implementations, a value corresponding to the spatial rate factor threshold may satisfy the spatial rate factor threshold when the value is less than about <NUM> and greater than about <NUM>. The spatial rate factor may include a single parameter configured to allocate the bit rate to each layer of the encoded bit stream. In some examples, the spatial rate factor includes a weighted sum corresponding to a ratio of a product of variances, wherein the ratio includes a numerator based on estimated variances of scalar quantized transform coefficients from a first spatial layer and a denominator based on estimated variances of scalar quantized transform coefficients from a second spatial layer.

Enabling disclosure can be found in <FIG> and corresponding passages.

<FIG> is an example of a rate allocation system <NUM>. The rate allocation system <NUM> generally includes a video source device <NUM> communicating a captured video as a video input signal <NUM> via a network <NUM> to a remote system <NUM>. At the remote system <NUM>, an encoder <NUM> and an allocator <NUM> convert the video input signal <NUM> into an encoded bit stream <NUM>. The encoded bit stream <NUM> includes more than one spatial layer L<NUM>-i where i designates the number of spatial layers L<NUM>-i. Each spatial layer L is a scalable form of the encoded bit stream <NUM>. A scalable video bit stream refers to a video bit stream where parts of the bit stream may be removed in a way that results in a sub-stream (e.g., a spatial layer L) that forms a valid bit stream for some target decoder. More particularly, a sub-stream represents the source content (e.g., captured video) of the original video input signal <NUM> with a reconstruction quality that is less than the quality of the original captured video. For example, the first spatial layer L<NUM> has a 720p high definition (HD) resolution of <NUM> x <NUM> while the base layer L<NUM> scales to a resolution of <NUM> x <NUM> as an extended form of video graphics adapter resolution (VGA). In terms of scalability, generally a video may be scalable temporally (e.g., by frame rate), spatially (e.g., by spatial resolution), and/or by quality (e.g., by fidelity often referred to as signal-to-noise-ratio SNR).

The rate allocation system <NUM> is an example environment where a user <NUM>, 10a captures video at the video source device <NUM> and communicates the captured video to other users <NUM>, 10b-c. Here, prior to the users 10b, 10c receiving the captured video via video receiving devices <NUM>, 150b-c, the encoder <NUM> and the allocator <NUM> convert the captured video into the encoded bit stream <NUM> at an allocated bit stream rate. Each video receiving device <NUM> may be configured to receive and/or to process different video resolutions. Here, a spatial layer L with a greater layer number i refers to a layer L with a greater resolution, such that i = <NUM> refers to a base layer L<NUM> with the lowest scalable resolution within the bit stream of more than one spatial layer L<NUM>-i. Referring to <FIG>, the encoded video bit stream <NUM> includes two spatial layers L<NUM>, L<NUM>. As such, one video receiving device <NUM> may receive the video content as a lower resolution spatial layer L<NUM> while another video receiving device <NUM> may receive the video content as a higher resolution spatial layer L<NUM>. For example, <FIG> depicts a first video receiving device 150a of the user 10b as a cell phone receiving the lower spatial resolution layer L<NUM> while the user 10c with a second receiving device 150b as a laptop receives a higher resolution spatial layer L<NUM>.

When different video receiving devices 150a-b receive different spatial layers L<NUM>-i, the video quality of each spatial layer L may be dependent on a bit rate BR and/or an allocation factor AF of the received spatial layer L. Here, the bit rate BR corresponds to bits per second and the allocation factor AF corresponds to bits per sample (i.e. transform coefficient). In the case of a scalable bit stream (e.g., the encoded bit stream <NUM>), a total bit rate BRtot for the scalable bit stream is often constrained such that each spatial layer L of the scalable bit stream suffers similar bit rate constraints. Due to these constraints, the bit rate BR associated with one spatial layer L may compromise or tradeoff the quality of another spatial layer L. More particularly, if quality is compromised on a spatial layer L received by a user <NUM> via a video receiving device <NUM>, the quality may generate a negative effect on a user experience. For example, it is becoming more common to transfer video content as a form of communication via real-time communication (RTC) applications. A user <NUM> of a RTC application may often choose an application for communication based on a subjective quality of the application. Therefore, as an application user, the user <NUM> generally desires to have a positive communication experience without quality issues that may stem from inadequate bit rate allocation to a spatial layer L that the application user <NUM> receives. To help ensure a positive user experience, the allocator <NUM> is configured to adaptively communicate an allocation factor AF to determine a bit rate BR for each spatial layer L among multiple spatial layers L<NUM>-i. By analytically allocating allocation factors AF among multiple spatial layers L<NUM>-i, the allocator <NUM> seeks to achieve the highest video quality over all spatial layers L<NUM>-i for a given total bit rate BRtot.

The video source device <NUM> can be any computing devices or data processing hardware capable of communicating captured video and/or video input signals <NUM> to a network <NUM> and/or remote system <NUM>. In some examples, the video source device <NUM> includes data processing hardware <NUM>, memory hardware <NUM>, and a video capturing device <NUM>. In some implementations, the video capturing device <NUM> is actually an image capturing device that may communicate a sequence of captured images as video content. For example, some digital cameras and/or webcams are configured to capture images at a particular frequency to form perceived video content. In other examples, the video source device <NUM> captures video in a continuous analogue format that may subsequently be converted to a digital format. In some configurations, the video source device <NUM> includes an encoder to initially encode or compress captured data (e.g., analogue or digital) to a format further processed by the encoder <NUM>. In other examples, the video source device <NUM> is configured to access the encoder <NUM> at the video source device <NUM>. For example, the encoder <NUM> is a web application hosted on the remote system <NUM> yet accessible via a network connection by the video source device <NUM>. In yet other examples, parts or all of the encoder <NUM> and/or allocator <NUM> are hosted on the video source device <NUM>. For example, the encoder <NUM> and the allocator <NUM> are hosted on the video source device <NUM>, but the remote system <NUM> functions as a backend system that relays the bit stream including spatial layers L<NUM>-i to video receiving device(s) <NUM> in accordance with decoding capabilities of the video receiving device(s) <NUM> and a capacity of a connection of the network <NUM> between the video receiving device(s) <NUM> and the remote system <NUM>. Additionally or alternatively, the video source device <NUM> is configured such that the user 10a may engage in communication to another user 10b-c across the network <NUM> utilizing the video capturing device <NUM>.

The video input signal <NUM> is a video signal corresponding to captured video content. Here, the video source device <NUM> captures the video content. For example, <FIG> depicts the video source device <NUM> capturing the video content via a webcam <NUM>. In some examples, the video input signal <NUM> is an analogue signal that is processed into a digital format by the encoder <NUM>. In other examples, the video input signal <NUM> has undergone some level of encoding or digital formatting prior to the encoder <NUM>, such that the encoder <NUM> performs a requantization process.

Much like the video source device <NUM>, the video receiving device <NUM> can be any computing devices or data processing hardware capable of receiving communicated captured video via a network <NUM> and/or remote system <NUM>. In some examples, the video source device <NUM> and the video receiving device <NUM> are configured with the same functionality such that the video receiving device <NUM> may become a video source device <NUM> and the video source device <NUM> may become a video receiving device <NUM>. In either case, the video receiving device <NUM> includes at least data processing hardware <NUM> and memory hardware <NUM>. Additionally, the video receiving device <NUM> includes a display <NUM> configured to display the received video content (e.g., at least one layer L of the encoded bit stream <NUM>). As shown in <FIG>, a user 10b, 10c receives the encoded bit stream <NUM> at the bit rate BR as a spatial layer L and decodes and displays the encoded bit stream <NUM> as a video on the display <NUM>. In some examples, the video receiving device <NUM> includes a decoder or is configured to access a decoder (e.g., via the network <NUM>) to allow the video receiving device <NUM> to display content of the encoded bit stream <NUM>.

In some examples, the encoder <NUM> and/or the allocator <NUM> is an application hosted by a remote system <NUM>, such as a distributed system of a cloud environment, accessed via the video source device <NUM> and/or the video receiving device <NUM>. In some implementations, the encoder <NUM> and/or the allocator <NUM> is an application downloaded to memory hardware <NUM>, <NUM> of the video source device <NUM> and/or the video receiving device <NUM>. Regardless of an access point to the encoder <NUM> and/or allocator <NUM>, the encoder <NUM> and/or the allocator <NUM> may be configured to communicate with the remote system <NUM> to access resources <NUM> (e.g., data processing hardware <NUM>, memory hardware <NUM>, or software resources <NUM>). Access to resources <NUM> of the remote system <NUM> may allow the encoder <NUM> and/or the allocator <NUM> to encode the video input signal <NUM> into the encoded bit stream <NUM> and/or allocate a bit rate BR to each spatial layer L of the more than one spatial layer L<NUM>-i of the encoded bit stream <NUM>. Optionally, a real time communication (RTC) application, as a software resource <NUM> of the remote system <NUM> used to communicate between users <NUM>, 10a-c, includes the encoder <NUM> and/or allocator <NUM> as built-in functionality.

Referring in further detail to <FIG>, three users <NUM>, 10a-c communicate via a RTC application (e.g., a WebRTC video application hosted by the cloud) hosted by the remote system <NUM>. In this example, the first user 10a is group video chatting with the second user 10b and the third user 10c. As the video capturing device <NUM> captures video of the first user 10a talking, the captured video via a video input signal <NUM> is processed by the encoder <NUM> and the allocator <NUM> and communicated via network <NUM>. Here, the encoder <NUM> and the allocator <NUM> operate in conjunction with the RTC application to generate an encoded bit stream <NUM> with more than one spatial layer L<NUM>, L<NUM> where each spatial layer L has an allocated bit rate BR0, BR1 determined by allocation factors AF0, AF1 based on the video input signal <NUM>. Due to the capabilities of each video receiving device 150a, 150b, each user 10b, 10c, receiving the video of the first user 10a chatting, receives a different scaled version of the original video corresponding to the video input signal <NUM>. For example, the second user 10b receives the base spatial layer L<NUM> while the third user 10c receives the first spatial layer L<NUM>. Each user 10b, 10c proceeds to display the received video content on a display 156a, 156b communicating with the RTC application. Although, a RTC communication application is shown, the encoder <NUM> and/or the allocator <NUM> may be used in other applications involving encoded bit streams <NUM> with more than one spatial layer L<NUM>-i.

<FIG> is an example of an encoder <NUM>. The encoder <NUM> is configured to convert the video input signal <NUM> as an input <NUM> into an encoded bit stream as an output <NUM>. Although depicted individually, the encoder <NUM> and the allocator <NUM> may be integrated into a single device (e.g., as shown by the dotted line in <FIG>) or occur separately across multiple devices (e.g., the video input device <NUM>, the video receiving device <NUM>, or the remote system <NUM>). The encoder <NUM> generally includes a scaler <NUM>, a transformer <NUM>, a quantizer <NUM>, and an entropy encoder <NUM>. Though not shown, the encoder <NUM> may include additional components to generate an encoded bit stream <NUM>, such as a prediction component (e.g., a motion estimation and intra prediction) and/or an in-loop filter. The prediction component may generate a residual to be communicated to the transformer <NUM> for transformation where the residual is based on a difference of an original input frame minus a prediction of a frame (e.g., motion compensated or intraframe predicted).

The scaler <NUM> is configured to scale the video input signal <NUM> into a plurality of spatial layers L<NUM>-i. In some implementations, the scaler <NUM> scales the video input signal <NUM> by determining portions of the video input signal <NUM> that may be removed to reduce a spatial resolution. By removing a portion or portions, the scalar <NUM> forms versions of the video input signal <NUM> to form a plurality of spatial layers (e.g., substreams). The scaler <NUM> may repeat this process until the scaler <NUM> forms a base spatial layer L<NUM>. In some examples, the scaler <NUM> scales the video input signal <NUM> to form a set number of spatial layers L<NUM>-i. In other examples, the scaler <NUM> is configured to scale the video input signal <NUM> until the scaler <NUM> determines that no decoder exists to decode a substream. When the scaler <NUM> determines that no decoder exists to decode a substream corresponding to the scaled version of the video input signal <NUM>, the scaler <NUM> identifies the previous version (e.g., spatial layer L) as the base spatial layer L<NUM>. Some examples of scalers <NUM> include codecs corresponding to a scalable video coding (SVC) extensions, such as an extension of the H. <NUM> video compression standard or an extension of the VP9 coding format.

The transformer <NUM> is configured to receive each spatial layer L corresponding to the video input signal <NUM> from the scaler <NUM>. For each spatial layer L, the transformer <NUM>, at operation <NUM>, partitions each spatial layer L into sub-blocks. With each sub-block, at operation <NUM>, the transformer <NUM> transforms each sub-block to generate transform coefficients <NUM> (e.g., by discrete cosine transform (DCT)). By generating transform coefficients <NUM>, the transformer <NUM> may correlate redundant video data and non-redundant video data to aid in the removal of redundant video data by the encoder <NUM>. In some implementations, the transform coefficients also allow the allocator <NUM> to easily determine a number of coefficients per transform block with non-zero variance in a spatial layer L.

The quantizer <NUM> is configured to perform a quantization or a re-quantization process <NUM> (i.e., scalar quantization). A quantization process generally converts input parameters (e.g., from a continuous analogue data set) into a smaller data set of output values. Although a quantization process may convert an analogue signal into a digital signal, here, the quantization process <NUM> (also sometimes referred to as a requantization process) typically further processes a digital signal. Depending on a form of the video input signal <NUM>, either process may be used interchangeably. By using a quantization or re-quantization process, data may be compressed, but at a cost of some aspect of data loss since the smaller data set is a reduction of a larger or continuous data set. Here, the quantization process <NUM> converts a digital signal. In some examples, the quantizer <NUM> contributes to the formation of the encoded bit stream <NUM> by scalar quantizing the transform coefficients <NUM> of each sub-block from the transformer <NUM> into quantization indices <NUM>. Here, scalar quantizing the transform coefficients <NUM> may allow lossy encoding to scale each transform coefficient <NUM> in order to contrast redundant video data (e.g., data that may be removed during encoding) to valuable video data (e.g., data that to should not be removed).

The entropy encoder <NUM> is configured to convert the quantization indices <NUM> (i.e. quantized transform coefficients) and side information into bits. By this conversion, the entropy encoder <NUM> forms the encoded bit stream <NUM>. In some implementations, the entropy encoder <NUM> along with the quantizer <NUM> enable the encoder <NUM> to form an encoded bit stream <NUM> where each layer L<NUM>-i has a bit rate BR0-i based on the allocation factor AF0-i determined by the allocator <NUM>.

<FIG> is an example of the allocator <NUM>. The allocator <NUM> is configured to receive non-quantized transform coefficients <NUM> related to more than one spatial layer L<NUM>-i and determine an allocation factor AF for each received spatial layer L<NUM>-i. In some implementations, the allocator <NUM> determines each allocation factor AF based on a square-error based high rate approximation for scalar quantization. Square-error high rate approximation allows a system to determine an optimal (in the context of high-rate approximation) bit rate to allocate for a number N of scalar quantizers. Typically, the optimal bit rate to allocate for N scalar quantizers is determined by rate-distortion optimized quantization. Rate-distortion optimization seeks to improve video quality during video compression by minimizing an amount of distortion (i.e. loss of video quality) subject to a bit rate constraint (e.g., a total bit rate BRtot). Here, the allocator <NUM> applies the principles that determine the optimal bit rate for N scalar quantizers to determine an optimal allocation factor to allocate a bit rate to each of the more than one spatial layer L<NUM>-i of the encoded bit stream <NUM>.

Generally speaking, the square-error high-rate approximation for scalar quantization may be represented by the following equation: <MAT> where <MAT> depends on a source distribution of an input signal (e.g., transform coefficients) to an i-th quantizer, <MAT> is a variance of that signal, and ri is the bit rate for the i-th quantizer in units of bits per input symbol. An expression for optimal rate allocation for two scalar quantizers is derived below using a square-error high-rate approximation.

The average distortion D for a two-quantizer problem, D<NUM> equals <MAT>. Similarly, the average rate, R<NUM>, for a two-quantizer problem equals <MAT>. Here, di is a square-error distortion due to an i-th quantizer and ri is a bit rate allocated to the i-th quantizer in units of bits per sample. Although, the parameter di is a function of rate, ri, such that an equation like di(ri), would be appropriate; for convenience di is simply represented as di instead. Substituting the high-rate approximation for d<NUM> and d<NUM> into the equation for D<NUM> yields: <MAT> With equation (<NUM>), <NUM>R<NUM> - r<NUM> may be substituted for r<NUM> to yield: <MAT> By further taking a derivative of D<NUM> with respect to r<NUM> equation (<NUM>) yields the following expression: <MAT> Setting the above expression, equation (<NUM>), equal to zero and solving for r<NUM> results in an expression for the optimal rate r* for a zero quantizer represented as follows: <MAT>.

Because the expression for high-rate distortion is convex, the minimum found by setting the derivative to zero is global. Similarly, an optimal rate r* for the first quantizer can be expressed as follows: <MAT>.

To find an optimal quantizer distortion, <MAT> and <MAT>, equations (<NUM>) and (<NUM>) are substituted for the optimal rate into the respective high-rate expressions for a scalar quantizer distortion as follows: <MAT> A simplified form of equation (<NUM>) yields the following equation: <MAT>.

This same two-quantizer analysis may be extended to three quantizers by combining the zero quantizer and the first quantizer into a single quantization system (i.e. a nested system) where the combined quantizer is already solved according to equations (<NUM>)-(<NUM>). Using a similar methodology to the two-quantizer rate allocation, a three-quantizer system is derived as follows.

Since the average per-quantizer distortion for a two-quantizer system is represented as <MAT>, by substituting davg into the expression for an average three-quantizer distortion, <MAT> yields the following equation: <MAT>.

Similarly, an average rate for a three-quantizer system is represented as follows: <MAT>.

Utilizing the result for optimal distortion from the two-quantizer analysis as shown in equation (<NUM>), it follows that the three-quantizer distortion may be represented by the following equation: <MAT> Accordingly, when equation (<NUM>) is simplified and <MAT> is substituted into equation (<NUM>), equation (<NUM>) transforms into the following expression: <MAT>.

With equation (<NUM>), the derivative with respect to r<NUM> may be set to zero and solved for r<NUM> to yield the following equation: <MAT> For three quantizers, a more general representation of equation (<NUM>) may be expressed as follows: <MAT>.

Based on the first and the second quantizers, an expression for an optimal rate allocation r* for N quantizers may be derived. An expression for an optimal rate for the i-th quantizer is as follows: <MAT> By substituting the expression for the optimal rate into the high-rate expression for distortion and simplifying similarly to the two-quantizer expressions, the resulting expression for optimal distortion in terms of N quantizers is shown below.

Based on the derived expressions from equations (<NUM>)-(<NUM>), the allocator <NUM> may apply these expressions for the optimal distortion to determine an optimal allocation factor AF (i.e. contribute to an optimal bit rate BR) for each layer L of the plurality of spatial layers L<NUM>-i. Similar to the derived N quantizer expressions, multiple spatial layer bit rates may be deduced from expressions associated with a two and a three layer rate allocation system. In some examples, it is assumed that although spatial layers L<NUM>-i typically have different spatial dimensions, the spatial layers L<NUM>-i originate from the same video source (e.g., the video source device <NUM>). In some implementations, the scalar quantizers that encode a first spatial layer L<NUM> and a second spatial layer L<NUM> are assumed to be identical in structure even though values of these scalar quantizers may differ. Moreover, for each spatial layer L, a number of samples S generally equates to a number of transform coefficients <NUM> (i.e. also equates to a number of quantizers).

In the case of a two-spatial layer rate allocation system, an average distortion for two-spatial layers, D<NUM>, may be represented as a weighted sum of average distortions, d<NUM> and d<NUM>, corresponding to the first and the second spatial layer L<NUM>, L<NUM> (i.e. spatial layers <NUM> and <NUM>) as follows: <MAT> where si equals the number of samples in the i-th spatial layer Li and S = s<NUM> + s<NUM>. Similarly, an average bit rate for two-spatial layers may be expressed as follows <MAT> where r<NUM> and r<NUM> are average bit rates of the first and the second spatial layer L<NUM>, L<NUM>, respectively. By substituting the expression for N-quantizer optimal distortion (i.e. equation (<NUM>)), into equation (<NUM>) for D<NUM>, above, D<NUM> may be expressed as follows: <MAT> where <MAT>is a variance of an input signal to the j-th scalar quantizer in the i-th spatial layer Li. Solving for r<NUM> in equation (<NUM>) and substituting the result into equation (<NUM>) yields: <MAT> Furthermore, by setting the derivative of D<NUM> with respect to r<NUM> to zero and solving for r<NUM>, r<NUM> may be represented by the following equation: <MAT> To simplify equation (<NUM>) for notational convenience, <MAT>. Substituting this expression for Pi into equation (<NUM>) for <MAT> and rearranging the resulting terms forms the following expression that appears similar to the N-quantizer allocation expression: <MAT> Alternatively, equation (<NUM>) may be expressed in terms of r<NUM>* to achieve the following equation: <MAT> Based on equations (<NUM>)-(<NUM>), an optimal two-spatial-layer distortion may be expressed as follows: <MAT>.

A similar approach may develop an optimal allocation factor that applies to three spatial layers L<NUM>-<NUM>. Much like the two spatial layers L<NUM>, L<NUM>, si equals the number of samples in the i-th spatial layer Li such that S = s<NUM> + s<NUM> + s<NUM>. An average rate and a distortion for the three-spatial layers L<NUM>-<NUM>, R<NUM> and D<NUM>, respectively, may be represented as the weighted sum of average rates and distortions, r<NUM>, r<NUM>, and r<NUM> and d<NUM>, d<NUM>, and d<NUM> of spatial layers <NUM>, <NUM>, and <NUM> (e.g., three spatial layers L<NUM>-<NUM>) as follows: <MAT> and <MAT>.

When similar techniques are applied from the two-quantizer results to three quantizers, R<NUM> may be expressed as a combination of an average two-layer rate, R<NUM>, using the following equation: <MAT> where <MAT>.

Similarly, for three quantizers, the distortion may be represented as follows: <MAT> where <MAT> With equation (<NUM>) for two-layer optimal distortion <MAT>, and equation (<NUM>) for optimal N-quantizer distortion, <MAT>, equation (<NUM>) may be solved for D<NUM> to yield the following expression: <MAT> where <MAT>. Equation (<NUM>) may be solved for R<NUM> to yield the following expression: <MAT> Furthermore, combining equations (<NUM>) and (<NUM>) by substituting equation (<NUM>) into equation (<NUM>) for D<NUM> forms the following equation: <MAT> An expression for r<NUM>may be formed by taking the derivative of D<NUM>with respect to r<NUM> and setting the result equal to zero. This expression may be expressed by the following equation: <MAT> When terms are rearranged, equation (<NUM>) may look similar to the N-quantizer allocation expression as follows: <MAT> Applying this equation (<NUM>) to the first layer L<NUM> and the second layer L<NUM>, the allocation factor for each layer may be expressed as follows: <MAT> <MAT>.

Both derivations of the two-spatial layer L<NUM>-i and the three spatial layer L<NUM>-<NUM> illustrate a pattern that may be extended to multiple spatial layers to optimize rate allocation (e.g., allocation factor AF for determining a bit rate BR allocated to each spatial layer L) at the allocator <NUM>. Here, extending the above results to L spatial layers L<NUM>-L yields a universal expression represented by the following equation: <MAT> where RLis the average rate corresponding to bits per sample over L spatial layers L<NUM>-i; the total number of samples S over L spatial layers where <MAT>; si is the number of samples in the i-th spatial layer; <MAT>, where hj,i depends on a source distribution of the signal being quantized by the j-th quantizer in the i-th spatial layer; and <MAT> corresponds to a variance of the j-th transform coefficient in the i-th spatial layer.

In some implementations, equation (<NUM>) has different forms due to various assumptions. Two different forms of equation (<NUM>) are represented below. <MAT> <MAT> For instance, a value for hj,i depends on a source distribution of the video input signal <NUM> being quantized by the j-th quantizer in the i-th spatial layer Li. In examples with similar source distributions, a value for hj,i does not change from quantizer to quantizer and thus cancels due to the ratio of product terms within equation (<NUM>). In other words, hj,<NUM>= hj,<NUM>= hj,<NUM>= h. Consequently, when this cancelation occurs, the term Pi = <MAT>, which effectively eliminates this parameter from consideration since Pi always appears as a ratio where h is in the numerator to cancel like terms in the denominator. In practice, hj,<NUM> may differ from hj,<NUM> and hj,<NUM> because the base spatial layer L<NUM> uses only temporal prediction whereas other spatial layers may use both temporal and spatial prediction. In some configurations, this difference does not significantly affect the allocation factor AF determined by the allocator <NUM>.

In other implementations, the encoder <NUM> introduces transform blocks that result in transform coefficients <NUM>. When this occurs, a change may occur to the grouping of transform coefficients <NUM> that introduces a variable si'. The variable si' corresponds to an average number of transform coefficients <NUM> per transform block with non-zero variance in the i-th spatial layer Li as shown in equation (39a). Contrast, this variable si' to si of equation (39b) that corresponds to the number of samples S in the i-th spatial layer Li. Additionally, in equation (39a), the term <MAT> where σ<NUM>k,i is the variance of the k-th coefficient in a transform block in the i-th spatial layer Li. Practically speaking, equation (39a) represents the optimal bit rate allocation for the i-th spatial layer Li as an expression of a weighted sum of the ratios of the products of variances (e.g., <MAT>).

Referring to <FIG>, in some implementations, the allocator <NUM> includes a sampler <NUM>, an estimator <NUM>, and a rate determiner <NUM>. The sampler <NUM> receives, as an input <NUM> of the allocator <NUM>, non-quantized transform coefficients <NUM> with a plurality of spatial layers L<NUM>-i. For example, <FIG> illustrates the transform coefficients <NUM> generated by the transformer <NUM> communicated to the allocator <NUM> by a dotted line. With the received non-quantized transform coefficients <NUM>, the sampler <NUM> identifies frames of the video input signal <NUM> as a sample SF. Based on the sample SF identified by the sampler <NUM>, the allocator <NUM> determines an allocation factor AF for each spatial layer L. In some implementations, the allocator <NUM> is configured to dynamically determine the allocation factor AF of each spatial layer L. In these implementations, the sampler <NUM> may be configured to iteratively identify sets of frame samples SF such that the allocator <NUM> may adapt the allocation factor AF to each set of samples SF identified by the sampler <NUM>. For instance, the allocator <NUM> determines an allocation factor AF to each spatial layer L based on a first sample SF1 of frames of the video input signal <NUM> and then proceeds to adjust or to modify the allocation factor AF (e.g., shown in <FIG> as changing from a first allocation factor AF1 for the first sample SF1 to a second allocation factor AF2 for the second sample S F2) applied to each spatial layer L (e.g., if necessary) based on a second sample SF2 of frames of the video input signal <NUM> identified by the sampler <NUM>. This process may iteratively continue for the duration that the allocator <NUM> receives the video input signal <NUM>. In these examples, the allocator <NUM> modifies the allocation factor AF, which in turn, changes a spatial rate factor <NUM> (e.g., from a first spatial rate factor <NUM><NUM> to a second spatial rate factor <NUM><NUM>) based on changes between the first sample SF1 and the second sample SF2. Additionally or alternatively, the allocator <NUM> may modify the allocation factor AF on a frame-by-frame basis using an exponential moving average. An exponential moving average is generally a weighted moving average that weighs the allocation factor AF determined for the current frame with the weighted average of allocation factors AF from previous frames. In other words, here, each modification to the allocation factor AF is a weighted average with a current and previous allocation factor(s) AF.

The estimator <NUM> is configured to determine variance estimations <NUM> of each transform coefficient from the encoder <NUM>. In some configurations, the estimator <NUM> assumes that the transform coefficients <NUM> in each block from the transformer <NUM> are similarly distributed. Based on this assumption, variances of the transform coefficients <NUM> may be estimated by averaging across all transform blocks in sample frames SF of the video input signal <NUM>. For example, the expression below models the k-th transform coefficient <NUM> in the i-th spatial layer Li as a random variable Ek,i. <MAT> where εb,k,i,t is the k-th transform coefficient <NUM> in the b-th transform block in the i-th spatial layer Li in the t-th frame and Bi represents the number of blocks in the i-th spatial layer Li, and SF represents the number of sample frames used to estimate the variance. In some examples, the value of <MAT> is an estimate of the variance of the k-th transform coefficient <NUM> in the i-th spatial layer Li, independent of the transform block when all such blocks are assumed to have identical statistics. In practice, however, the statistics of transform blocks may vary across the frame. This may be especially true for video conferencing content where blocks at the edge of the frame may have less activity than at the center. Accordingly, if these non-identical statistics negatively affect an accuracy of the rate allocation result, estimating variance based on blocks centrally located in the frame may mitigate the negative affect. In some configurations, the sub-blocks for which the variances of the transform coefficients are estimated represent a subset of all sub-block in the video picture (e.g., the sub-blocks located in the center-most portion of the video picture or the sub-blocks in locations in which the video picture has changed relative to previous pictures).

The rate determiner <NUM> is configured to determine a spatial rate factor <NUM> based on the sample SF of frames from the video input signal <NUM> identified by the sampler <NUM>. In some examples, the spatial rate factor <NUM> defines a factor for determining a bit rate BR at each spatial later L<NUM>-i of the encoded bit stream <NUM>. A spatial rate factor <NUM> is a ratio between a bit rate allocated to spatial layer Li-<NUM> and a bit rate allocated to spatial layer Li. In a two-spatial example with spatial layers L<NUM> and L<NUM>, a spatial rate factor equal to <NUM>, and the bit rate allocated to spatial layer L<NUM> equal to 500kbps, the bit rate allocated for spatial layer L<NUM> equals <NUM> kbps (i.e. <NUM> times <NUM> kbps). In these implementations, the value of the spatial rate factor <NUM> is set equal to a difference between an allocation factor AF of a base layer L<NUM> and an average rate RL (e.g., the expression r*<NUM>-RL of equation (<NUM>)). Here, the allocation factor AF corresponds to bits per transform coefficient for the base layer L<NUM> (also referred to as r*<NUM>) while the average rate RL corresponds to bits per transform coefficient for the more than one spatial layer L<NUM>-i. In some configurations, experimental results for two spatial layers have indicated that the spatial rate factor <NUM> corresponds to the expression <MAT>. The spatial rate factor <NUM>, as a single parameter, may allow the allocator <NUM> to easily tune or modify the bit rates BR for each layer L of the encoded bit stream <NUM>.

Although explained with respect to two spatial layers, the allocator <NUM> may apply the spatial rate factor <NUM> and/or the allocation factor AF to any number of spatial layers L<NUM>-i. For instance, the allocator <NUM> determines the allocation factor AF and/or the spatial rate factor <NUM> with respect to each set of two spatial layers. To illustrate with three layers L<NUM>-<NUM>, the allocator <NUM> determines the allocation factor AF for the base layer L<NUM> and the first layer L<NUM> and then determines the allocation factor AF for the first layer L<NUM> and the second layer L<NUM>. Each allocation factor AF may be used to determine a spatial rate factor <NUM>, one spatial rate factor <NUM> for the base layer L<NUM> and the first layer L<NUM> and a second spatial rate factor <NUM> for the first layer L<NUM> and the second layer L<NUM> With the spatial rate factor <NUM> for each set of two spatial layers, the allocator <NUM> may average (e.g., a weighted average, an arithmetic average, a geometric average, etc.) the spatial rate factors <NUM> and/or the allocation factors AF to generate an average spatial rate factor and/or average allocation factor for any number of spatial layers L<NUM>-i.

In some examples, the spatial rate factor <NUM> must satisfy a spatial rate factor threshold <NUM> (e.g., be within a range of values) in order for the allocator <NUM> to help determine a bit rate BR based on the spatial rate factor <NUM>. In some implementations, a value satisfies the spatial rate factor threshold <NUM> when the value is within a range less than about <NUM> and greater than about <NUM>. In other implementations, the spatial rate factor threshold <NUM> corresponds to a narrower range of values (e.g., <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, etc.) or a broader range of values (e.g., <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, etc.). In some configurations, when the spatial rate factor <NUM> is out of the range of values corresponding to the spatial rate factor threshold <NUM>, the allocator <NUM> adjusts the spatial rate factor <NUM> to satisfy the spatial rate factor threshold <NUM>. For instance, when the spatial rate factor threshold <NUM> ranges from <NUM>-<NUM>, a spatial rate factor <NUM> outside this range is adjusted to a nearest maximum of the range (e.g., a spatial rate factor <NUM> of <NUM> is adjusted to a spatial rate factor <NUM> of <NUM> while a spatial rate factor <NUM> of <NUM> is adjusted to a spatial rate factor <NUM> of <NUM>).

Based on the determined spatial rate factor <NUM>, the allocator <NUM> is configured to optimize the video quality by reducing a distortion for the more than one spatial layer L<NUM>-i subject to a constraint on the total bit rate BRtot. To reduce the distortion, the allocator <NUM> influences (e.g., aids the encoder <NUM> to determine) a bit rate BR to each spatial layer L based on the spatial rate factor <NUM> computed for the sample SF of frames. For example, when the encoded bit stream <NUM> includes two spatial layers L<NUM>, L<NUM>, the allocator <NUM> determines an allocation factor AF, which in turn is used to determine a spatial rate factor <NUM> to generate a first bit rate BR1 corresponding to an equation BR1 = <MAT> and a second bit rate BR0 corresponding to an equation <MAT> where BRtot corresponds to the total bit rate available to encode the overall bitstream (i.e., all spatial layers L<NUM>, L<NUM>).

<FIG> is an example of a method <NUM> for implementing the rate allocation system <NUM>. At operation <NUM>, the method <NUM> receives, at data processing hardware <NUM>, transform coefficients <NUM> (e.g., non-quantized transform coefficients) corresponding to a video input signal <NUM>. The video input signal <NUM> includes a plurality of spatial layers L<NUM>-i where the plurality of spatial layers L<NUM>-i includes a base layer L<NUM>. At operation <NUM>, the method <NUM> determines, by the data processing hardware <NUM>, a spatial rate factor <NUM> based on a sample SF of frames from the video input signal <NUM>. The spatial rate factor <NUM> defines a factor for rate allocation at each spatial layer L of the encoded bit stream <NUM> and is represented by a difference between a rate of bits per transform coefficient for the base layer L<NUM> and an average rate RL of bits per transform coefficient of the plurality of spatial layers L<NUM>-i. At operations <NUM>, the method <NUM> reduces, by the data processing hardware <NUM>, a distortion d for the plurality of spatial layers L<NUM>-i of the encoded bit stream <NUM> by allocating a bit rate BR to each spatial layer L based on the spatial rate factor <NUM> and the sample SF of frames.

<FIG> is schematic view of an example computing device <NUM> that may be used to implement the systems and methods described in this document, for example, the encoder <NUM> and/or the allocator <NUM>.

The computing device <NUM> includes a data processing hardware <NUM>, memory hardware <NUM>, a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>.

Claim 1:
A method (<NUM>) comprising:
receiving (<NUM>), at data processing hardware (<NUM>), transform coefficients (<NUM>) corresponding to a scaled video input signal (<NUM>), the scaled video input signal (<NUM>) comprising a plurality of spatial layers L, the plurality of spatial layers L comprising a base layer L<NUM>;
determining (<NUM>), by the data processing hardware (<NUM>), a spatial rate factor (<NUM>) based on a sample SF of frames from the scaled video input signal (<NUM>), the spatial rate factor (<NUM>) comprising an allocation factor for bit rate allocation at each spatial layer L of an encoded bit stream (<NUM>) formed from the scaled video input signal (<NUM>),
wherein the allocation factor corresponds to a rate of bits per transform coefficient (<NUM>) of the base layer L<NUM>, the spatial rate factor (<NUM>) is set equal to a difference between the allocation factor and an average rate RL of bits per transform coefficient (<NUM>) for the plurality of spatial layers L;
wherein the allocation factor comprises a weighted sum of ratios of a product of variances, the ratio comprising a numerator based on estimated variances of the transform coefficients (<NUM>) from the base layer L<NUM> and a denominator based on estimated variances of the transform coefficients (<NUM>) from a second spatial layer Lj of the plurality of spatial layers, the weighted sum being over the index j;
wherein variances of transform coefficients (<NUM>) are estimated by averaging across all transform blocks in sample frames SF of the video input signal (<NUM>); and
reducing (<NUM>), by the data processing hardware (<NUM>), a distortion for the plurality of spatial layers L of the encoded bit stream (<NUM>) by allocating a bit rate BR to each spatial layer L based on the spatial rate factor (<NUM>) determined based on the sample SF of frames, the allocating being subject to a constraint on a total bit rate BRtot available to encode all of the plurality of spatial layers.