Dual PID controller based bit allocation and rate control for video coding

System and methods are provided for encoding a video stream. A video encoder may encode a frame and generate output bits to a buffer. A first PID controller may receive a first error signal and generate a preliminary bit budget for the frame, the first error signal being based on a complexity measure of the frame. A second PID controller may receive the preliminary bit budget, a second error signal and a predetermined bit rate setting, the second error signal being representative of a difference between a target number of bits stored in the buffer and a feedback status of the buffer. The second PID controller may generate a final bit budget for the frame, and provide the final bit budget for the frame to the video encoder.

FIELD

The technology described in this patent document relates generally to video coding. More particularly, systems and methods are disclosed for bit allocation and/or rate control in video coding.

BACKGROUND

PID controllers are widely used in industry to regulate temperature, pressure, flow rate, chemical composition, speed and practically every other variable for which a measurement exists.FIG. 1illustrates at100an example diagram of a PID controller102and example output signals of a process controlled by a PID controller with different tuning parameters. As shown inFIG. 1(a), the purpose of using the PID controller102is to make an output y of a process110follow a reference value r. To achieve this purpose, the PID controller102calculates an error e as the difference between the measured output y and the reference value r and attempts to minimize the error e. The PID controller102includes three elements, a proportional element104, an integral element106, and a derivative element108. The three elements produce output values—the proportional, the integral and derivative values, denoted P, I, and D, respectively. These output values can be interpreted in terms of time. P depends on a present error, I depends on the accumulation of past errors, and D is a prediction of future errors, based on a current rate of change.

The traditional basic equation for the PID control scheme is:

u⁡(t)=Kp⁢e⁡(t)+Ki⁢∫e⁡(t)⁢ⅆ(t)+Kd⁢ⅆⅆt⁢e⁡(t)
where u(t) is the output of the PID controller102, Kp, Ki, Kdare tuning parameters, and e(t) is the error.

Example output signals (y) of the process110are shown at112inFIG. 1(b). Each curve (except the reference signal) represents an output signal for a particular set of tuning parameters. All output signals of the process110oscillate around a reference signal in a decaying sinusoid. Eventually, all output signals converge to the reference signal.

SUMMARY

In accordance with the teachings described herein, system and methods are provided for encoding a video stream using proportional-integral-derivative (PID) controllers. A video encoder may encode a frame in the video stream and generate output bits for the frame to a buffer. A first PID controller may receive a first error signal from the video encoder and generate a preliminary bit budget for the frame based upon the first error signal, the first error signal being based on a complexity measure of the frame. A second PID controller may receive the preliminary bit budget for the frame, a second error signal and a predetermined bit rate setting, the second error signal being representative of a difference between a target number of bits stored in the buffer and a feedback status of the buffer, the feedback status of the buffer indicating an actual number of bits stored in the buffer. The second PID controller may generate a final bit budget for the frame based on the preliminary bit budget for the frame, the second error signal, and the predetermined bit rate setting, and provide the final bit budget for the frame to the video encoder for encoding the frame and generating the output bits for the frame to the buffer.

As further examples, the complexity measure of a current frame to be encoded by the video encoder may be determined based on a difference between a mean absolute difference of the current frame and an average mean absolute difference of the frames encoded by the video encoder previously during a predetermined period, the mean absolute difference of the current frame indicating complexity of the current frame and the average mean absolute difference of the previous frames indicating the average complexity of the previous frames.

As additional examples, the video encoder may determine a quantization parameter for a current frame to be encoded by the video encoder, based on the mean absolute difference of the current frame, the final bit budget for the current frame, the quantization parameter for the frame previously encoded by the video encoder, a mean absolute difference of the previous frame, and output bits for the previous frame. The quantization parameter for the current frame may be determined according to the following equation:

Q⁡(t)=B⁡(t-1)⁢Q2⁡(t-1)⁢MAD⁡(t)T⁡(t)⁢MAD⁡(t-1),
where t is a time index of the current frame. Q(t) is the quantization parameter for the current frame, MAD(t) is the mean absolute difference of the current frame, T(t) is the final bit budget for the current frame, t−1 is a time index of the frame previously encoded by the video encoder, Q(t−1) is the quantization parameter for the previous frame, MAD(t−1) is the mean absolute difference of the previous frame, and B(t−1) is the output bits for the previous frame. The average mean absolute difference of the previous frames may be determined according to the following equation:

avg⁢⁢MAD=1F⁢∑t=Ft⁢MAD⁡(τ),
where avg MAD is the average mean absolute difference of the frames in a previous period between t−F and t, τ is a time index, between t−F and t, MAD(τ) is a mean absolute difference of the frame at time τ, and F is the total frame number of the frames encoded previously during the predetermined period.

As further examples, the video encoder may determine a quantization parameter for the frame based on a Rate-Quantization model from a Video CODEC Test Model, Near-Term, Version 8 (TMN8). The predetermined bit rate setting may include a target bit rate in bits per second (R) and a frame rate in frames per second (F). The feedback status of the buffer may be determined based on a previous feedback status of the buffer, output bits for a previous frame, and the predetermined bit rate setting. An arithmetic component may receive a first input signal representative of the target number of bits stored in the buffer and a second input signal representative of the feedback status of the buffer, and generate the second error signal based on the first and second input signals.

As another example, a method is provided for encoding a video stream using proportional-integral-derivative (PID) controllers. A first error signal may be received at a first PID controller, the first error signal being based on a complexity measure of a frame in the video stream to be encoded by a video encoder. A preliminary bit budget for the frame may be generated by the first PID controller based upon the first error signal. A second error signal may be received at a second PID controller, the second error signal being representative of a difference between a target number of bits stored in a buffer and a feedback status of the buffer, the feedback status of the buffer indicating an actual number of bits stored in the buffer. A predetermined bit rate setting may be received at the second PID controller. A final bit budget for the frame may be generated by the second PID controller based on the preliminary bit budget for the frame, the second error signal, and the predetermined bit rate setting. The final bit budget for the frame may be provided to the video encoder for encoding the frame and generating output bits for the frame to the buffer.

As another example, a system for encoding a video stream using proportional-integral-derivative (PID) controllers may include one or more data processors and a computer-readable storage medium encoded with programming instructions for commanding the one or more data processors to perform a method. In the method, a first error signal may be received at a first PID controller, the first error signal being based on a complexity measure of a frame in the video stream to be encoded by a video encoder. A preliminary bit budget for the frame may be generated by the first PID controller based upon the first error signal. A second error signal may be received at a second PID controller, the second error signal being representative of a difference between a target number of bits stored in a buffer and a feedback status of the buffer, the feedback status of the buffer indicating an actual number of bits stored in the buffer. A predetermined bit rate setting may be received at the second PID controller. A final bit budget for the frame may be generated by the second PID controller based on the preliminary bit budget for the frame, the second error signal, and the predetermined bit rate setting. The final bit budget for the frame may be provided to the video encoder for encoding the frame and generating output bits for the frame to the buffer.

As another example, a non-transitory computer-readable storage medium may include programming instructions for causing a processing system to perform a method for encoding a video stream using proportional-integral-derivative (PID) controllers. In the method, a first error signal may be received at a first PID controller, the first error signal being based on a complexity measure of a frame in the video stream to be encoded by a video encoder. A preliminary bit budget for the frame may be generated by the first PID controller based upon the first error signal. A second error signal may be received at a second PID controller, the second error signal being representative of a difference between a target number of bits stored in a buffer and a feedback status of the buffer, the feedback status of the buffer indicating an actual number of bits stored in the buffer. A predetermined bit rate setting may be received at the second PID controller. A final bit budget for the frame may be generated by the second PID controller based on the preliminary bit budget for the frame, the second error signal, and the predetermined bit rate setting. The final bit budget for the frame may be provided to the video encoder for encoding the frame and generating output bits for the frame to the buffer.

DETAILED DESCRIPTION

FIG. 2illustrates at200an example diagram of single-PID-controller-based bit allocation and rate control for video coding. The PID controller202is used in connection with a video encoder204for bit allocation and rate control. For example, the PID controller202may adjust a bit budget for a frame to be encoded in the video encoder204based on the status of a buffer206, and thus output bits from the video encoder204to the buffer206may be adjusted so as to achieve a desired bit level in the buffer206.

A target buffer level S(t) and a feedback buffer status W(t) may be received at an arithmetic component208, such as a comparator. The feedback buffer status W(t) is updated based on a previous feedback buffer status W(t−1), output bits B(t−1) previously generated from the video encoder204, and a predetermined bit rate setting that may include a target bit rate (R) in bits per second and a frame rate (F) in frames per second. For example, W(t) may be updated according to an equation:
W(t)=W(t−1)+B(t−1)−R/F.

An error signal e(t) may be generated at the arithmetic component208based on the difference between the target buffer level S(t) and the feedback buffer status W(t). For example, the error signal e(t) may be generated according to an equation:
e(t)=S(t)−W(t).

The PID controller202receives an input R/F determined from the bit rate setting and the error signal e(t), and generates a bit budget T(t) to the video encoder204based on the bit rate setting and the error signal e(t). For example, the bit budget T(t) may be generated according to an equation:

T⁡(t)=RF⁢(1+PID)=RF⁢(1+Kp⁢e⁡(t)+Ki⁢∑τ=0t⁢e⁡(τ)+Kd⁡[e⁡(t)-e⁡(t-1)])
in which PID is the discrete-time version of PID control scheme.

The video encoder204may use the bit budget T(t) to encode a current frame, and generates output bits B(t) for the current frame to the buffer206.

FIG. 3illustrates at300an example diagram of dual-PID-controller-based bit allocation and rate control for video coding. Two PID controllers302and304are used in connection with a video encoder306for bit allocation and rate control. For example, the two PID controllers may be used together to adjust a bit budget for a frame to be encoded in the video encoder306, based on not only the status of the buffer308, but also complexity of the frame to be encoded.

The PID controller302receives from the video encoder306an error signal e1(t) which represents a complexity measure of the frame to be encoded. The complexity measure of the frame to be encoded may be determined based on a difference ΔMAD between a mean absolute difference of the frame to be encoded, MAD(t), and an average mean absolute difference of frames encoded previously during a predetermined period, avg MAD. MAD(t) indicates the complexity of the frame at time t or the current frame. avg MAD indicates average complexity of the frames encoded previously during the predetermined period. For example, if the predetermined period equals to one second, and the number of the frames encoded during the predetermined period is F, F equals in magnitude, the frame rate during the predetermined period. As an example, avg MAD may be calculated according to the following equation:
avgMAD=1/FΣt−FtMAD(τ),
where avg MAD is the average mean absolute difference of the frames encoded previously during the predetermined period,τ is a time index, between t−F and t, of one of the previous frames,MAD(τ) is a mean absolute difference of the previous frame at time τ, andF is the total frame number of the frames encoded previously during the predetermined period.

Then, e1(t) can be determined according to the following equation:
e1(t)=ΔMAD(t)=MAD(t)−avgMAD=MAD(t)−1/FΣt−FtMAD(τ).

The PID controller302may generate a preliminary bit budget T1(t) based on the error signal e1(t) and a bit rate setting that may include a target bit rate (R) in bits per second and a frame rate (F) in frames per second. For example, the preliminary bit budget T1(t) may be generated according to an equation:

A target buffer level S(t) and a feedback buffer status W(t) may be received at an arithmetic component310, such as a comparator, an adder, a subtractor and an adder-subtractor. The feedback buffer status W(t) is updated based on a previous feedback buffer status W(t−1), output bits B(t−1) previously generated from the video encoder306, and the bit rate setting. For example, W(t) may be updated according to an equation:
W(t)=W(t−1)+B(t−1)−R/F.

An error signal e2(t) may be generated at the arithmetic component310based on the difference between the target buffer level S(t) and the feedback buffer status W(t). For example, the error signal e2(t) may be generated according to an equation:
e2(t)=S(t)−W(t).

The PID controller304receives an input R/F determined from the bit rate setting, the preliminary bit budget T1(t), and the error signal e2(t), and generates a final bit budget T(t) based on the bit rate setting, the preliminary bit budget T1(t), and the error signal e2(t). For example, the final bit budget T(t) may be generated according to an equation:
T(t)=T1(1+PID2)=T1(t)(1+Kpe2(t)+K1Στ=0te2(τ)+Kd[e2(t)−e2(t−1)]).

The video encoder306uses the final bit budget T(t) to encode the current frame, and generates actual output bits B(t) to the buffer308.

FIG. 4illustrates at400an example flow chart of dual-PID-controller-based bit allocation and rate control for video coding. A first error signal may be determined at402from video data, representing the difference ΔMAD between the complexity of the current frame to be encoded, MAD(t), and the average complexity of frames encoded previously during a predetermined period, avg MAD. At404, a preliminary bit budget T1(t) may be generated based on the first error signal, i.e., ΔMAD.

A target buffer level S(t) may be received at406and a feedback buffer status W(t) may be received at408. A second error signal e2(t) may be generated at410based on the difference between the target buffer level S(t) and the feedback buffer status W(t).

A bit rate setting may be received at412. The bit rate setting may include a target bit rate (R) in bits per second and a frame rate (F) in frames per second. A final bit budget T(t) may be generated at414based on the received bit rate setting, the received preliminary bit budget T1(t), and the received second error signal e2(t). The final bit budget T(t) may be used at416to encode a current frame, and generates output bits for the current frame to a buffer.

A quantization parameter is usually used for video coding. Based on the final bit budget T(t) discussed above, a quantization parameter Q(t) may be derived based on a rate-quantization (R-Q) model from a Video CODEC Test Model, Near-Term, Version 8 (TMN8).

The R-Q model from TMN8 may be expressed in an equation:

R=eln⁢⁢2⁢σ2Q2,(1)
where R is a number of bits generated for a frame, σ is a standard deviation, and Q is a quantization parameter.

The complexity of a frame to be encoded, MAD, may be used to replace σ2in equation (1) and yield an equation:

Thus, for a current frame with a complexity MAD(t), and a previous frame with a complexity MAD(t−1),

As an example, it is assumed that for the current frame, the generated bits R(t) are equal to the final bit budget T(t). For the previous frame, the generated bits R(t−1) are assumed to be equal to output bits B(t−1) for the previous frame. Therefore, a quantization parameter Q(t) may be derived from equation (3) for video coding:

The dual-PID-controller-based bit allocation and rate control disclosed herein may be applied to many scenarios with different strategies. Table 1 provides as examples strategies for three different scenarios. The dual-PID-controller-based bit allocation and rate control disclosed herein may be applied with/without one or more of the following mechanisms: buffer overflow, maximum bit rate constraint, and big motion frame skipping.

TABLE 1ScenariosStrategiesNo.FileApplying dual-PID-controller-based bit allocation1Compressingand rate control without buffer overflow, maximumbit rate constraint, or big motion frame skippingVideoApplying dual-PID-controller-based bit allocation2Recordingand rate control with big motion frame skipping(Camcorder)(1/3 ratio limitation)VideoApplying dual-PID-controller-based bit allocation3Telephonyand rate control with buffer overflow, maximumbit rate constraint, and big motion frame skipping(1/2 ration limitation)

Table 2 illustrates example performance data from applying the dual-PID-controller-based bit allocation and rate control disclosed herein to three different streams, compared with the performance data of a TMN8 rate control applied to the same three streams. The dual-PID-controller-based bit allocation and rate control disclosed herein, applied with the strategies noted in Table 1, can achieve higher output bit rate, and better average peak signal-to-noise ratio (PSNR) that indicates better video quality.

FIG. 5illustrates example performance data from applying the dual-PID-controller-based bit allocation and rate control to a foreman sequence.FIG. 5(a)shows output bits for frames in the foreman sequence, andFIG. 5(b)shows output image of the 173th frame in the foreman sequence. As a comparison,FIG. 6illustrates example performance data from applying a TMN8 rate control strategy to the foreman sequence. As shown inFIG. 5(a), the output bits varies according to the complexity of the frames in the foreman sequence using the dual-PID-controller-based bit allocation and rate control. In contrast, as shown inFIG. 6(a), the output bits does not vary much over the frames using the TMN8 rate control. Moreover, comparingFIG. 5(b)withFIG. 6(b)shows the dual-PID-controller-based bit allocation and rate control yields better image quality than the TMN8 rate control.

Referring now toFIGS. 7-9, various exemplary implementations of the dual-PID-controller-based bit allocation and rate control are shown. Referring toFIG. 7, the dual-PID-controller-based bit allocation and rate control may be embodied in a hard disk drive700. The dual-PID-controller-based bit allocation and rate control may implement either or both signal processing and/or control circuits, which are generally identified inFIG. 7at702. In some implementations, signal processing and/or control circuit702and/or other circuits (not shown) in HDD700may process data, perform coding and/or encryption, perform calculations, and/or format data that is output to and/or received from a magnetic storage medium704.

HDD700may communicate with a host device (not shown) such as a computer, mobile computing devices such as personal digital assistants, cellular phones, media or MP3 players and the like, and/or other devices via one or more wired or wireless communication links706. HDD700may be connected to memory708, such as random access memory (RAM), a low latency nonvolatile memory such as flash memory, read only memory (ROM) and/or other suitable electronic data storage.

Referring now toFIG. 8, the dual-PID-controller-based bit allocation and rate control may be embodied in a digital versatile disc (DVD) drive800. The dual-PID-controller-based bit allocation and rate control may implement either or both signal processing and/or control circuits, which are generally identified inFIG. 8at802, and/or mass data storage804of DVD drive800. Signal processing and/or control circuit802and/or other circuits (not shown) in DVD800may process data, perform coding and/or encryption, perform calculations, and/or format data that is read from and/or data written to an optical storage medium806. In some implementations, signal processing and/or control circuit802and/or other circuits (not shown) in DVD800can also perform other functions such as encoding and/or decoding and/or any other signal processing functions associated with a DVD drive.

DVD drive800may communicate with an output device (not shown) such as a computer, television or other device via one or more wired or wireless communication links808. DVD800may communicate with mass data storage804that stores data in a nonvolatile manner. Mass data storage804may include a hard disk drive (HDD) such as that shown inFIG. 7. The HDD may be a mini HDD that includes one or more platters having a diameter that is smaller than approximately 1.8″. DVD800may be connected to memory810, such as RAM, ROM, low latency nonvolatile memory such as flash memory, and/or other suitable electronic data storage.

Referring now toFIG. 9, the dual-PID-controller-based bit allocation and rate control may be embodied in a cellular phone900that may include a cellular antenna902. The dual-PID-controller-based bit allocation and rate control may implement either or both signal processing and/or control circuits, which are generally identified inFIG. 9at904, a WLAN interface and/or mass data storage of the cellular phone900. In some implementations, cellular phone900includes a microphone906, an audio output908such as a speaker and/or audio output jack, a display910and/or an input device912such as a keypad, pointing device, voice actuation and/or other input device. Signal processing and/or control circuits902and/or other circuits (not shown) in cellular phone900may process data, perform coding and/or encryption, perform calculations, format data and/or perform other cellular phone functions.

Cellular phone900may communicate with mass data storage914that stores data in a nonvolatile manner such as optical and/or magnetic storage devices for example hard disk drives HDD and/or DVDs. At least one HDD may have the configuration shown inFIG. 7and/or at least one DVD may have the configuration shown inFIG. 8. The HDD may be a mini HDD that includes one or more platters having a diameter that is smaller than approximately 1.8″. Cellular phone900may be connected to memory916such as RAM, ROM, low latency nonvolatile memory such as flash memory and/or other suitable electronic data storage. Cellular phone900also may support connections with a WLAN via a WLAN network interface918.

This written description uses examples to disclose the invention, include the best mode, and also to enable a person skilled in the art to make and use the invention. The patentable scope of the invention may include other examples that occur to those skilled in the art.

As an example, systems and methods can be configured as disclosed herein to be used for various video encoders, such as H.263/MPEG-4, and H.264. As another example, systems and methods can be configured as disclosed herein to be applied in many different scenarios, such as file compressing, transcoding, camcorder and video telephony. As another example, systems and methods can be configured as disclosed herein for achieving improvements of video quality, as well as bit rate control. As another example, systems and methods can be configured as disclosed herein for applications that introduce low computation and memory costs for encoders.