Method of encoding video with film grain

A system for providing improved video quality and compression efficiency during encoding by detecting video segments having film grain approaching the “Red Lady” problem. The system detects when film grain approaches the level of the “Red Lady” problem by measuring frame-by-frame temporal differences (ME scores). From the ME scores, two key indicators are identified: (1) The average temporal difference in frames with an intermediate motion level higher than frames of non-noisy video; and (2) The fluctuation of the temporal differences between frames in a group is very small. When these indicators identify a high film video, a signal is provided to an encoder which allocates less bits to I frames and more bits to P and B frames than for other frames of video without comparable film grain.

BACKGROUND

Technical Field

The present invention relates to a process for improving video quality when encoding video with film grain. More particularly, the present invention relates to a solution to improving video quality when film grain is present on a level similar to the “Red Lady” noise problem.

Related Art

Film grain is hard to compress in an encoder. It requires more bits to encode than many other kinds of content for any level of video quality. Film grain may be thought of as a particular kind of spatial temporal noise. As such, film grain has low temporal predictability from one frame to any other frame of video. Thus, the encoding process is limited in its ability to leverage inter-frame estimation to achieve significant compression efficiency.

In some encoders, not enough bits are allocated to inter-predicted pictures. The result can be significant video quality artifacts such as I-frame beating and intermittent repetitive loss and recovery of spatial texture. Even encoders that can allocate significant bits might not be able to eliminate substantial noise, such as when noise is as high as in the “Red Lady” video frames.

A frame of the “Red Lady” video is illustrated withFIG. 1. The “red lady” video shows that a lady walking alongside a soccer field with a grassy background. The scene itself is simple, but the entire video is filled with film grain.

Film grain is like random noise. It requires a lot of bits to encode and is not temporally predictable, which makes high film grain video, in particular the “Red Lady” video, very difficult to encode.

A common practice to encode video with film grain is to encode a good quality I frame as a reference frame as a prediction frame for subsequent predictive frames (P or B frames). However, since the I frame and P and B frames all contain film grain this does not predict well, and many bits are needed to encode unpredicted high frequency components. If too many bits are allocated to the I frame, later P and B frames may be allocated fewer bits than they need, and their quality suffers. The good quality I frame, thus, may not help with the subsequent P and B frames.

In the Red Lady Video, the random noise level is very high. Thus, in the Red Lady Video, a beating effect will be seen, due to the quality of the pictures varying too much between frame types. Allocating more bits to the I frame does not help with reducing film grain in subsequent P and B frames in a typical encoder.

As shown inFIG. 2, the difference between two consecutive frames is mostly noise. Encoding a good quality I frame for these costs too many bits and leaves fewer bits for predictive frames. Moreover, the high quality I frame, even with additional bits allocated, is not a good reference frame because the noisy temporal differences cannot be motion predicted well. With the I frame as a reference, the film grains in the predictive frames would be poorly encoded, and create a quality disparity between I and predictive frames, as illustrated inFIG. 3. Thus allocating more bits to the I frame creates a “Dirty Window” for future film grain elimination in P and B frames.

Accordingly, it is desirable to provide better solutions for eliminating film grain comparable to the “Red Lady” video, and to avoid creating “Dirty Window” I frames.

SUMMARY

Embodiments of the present invention provide a system that enables improved video quality and compression efficiency during encoding by detecting video segments having film grain approaching the “Red Lady” problem and then optimizing the bit allocation between intra- and inter-predicted pictures using bit allocation variation between I, P and B type frames.

To optimize the bit allocation when a video clip is identified as a “Red Lady” like clip, embodiments of the present invention encode smaller I frames and allocate more bits on P and B frames. Since allocating more bits to the I frame when the “Red Lady” film grain problem occurs does not enable better prediction for encoding in the P and B frames, additional bits to the I frame are not necessary. Thus, allocating extra bits allocated to the P and B frames and not using the extra bits in the I frame enables reduction of frame grain when the “Red Lady” like film grain problem occurs, and the “Dirty Window” I frame issue will no longer be a consideration.

To identify the film grain level to determine when the optimization of bit allocation away from I frames to P and B frames should occur, a temporal analysis of motion-prediction data available is provided. For the temporal analysis, measurements of plotted frame-by-frame temporal differences (ME scores) of the received videos are determined. From the ME scores, two key indicators are identified: (1) The average temporal difference in frames with an intermediate motion level (i.e., greater than ME score of 20), is higher than frames of non-noisy video with intermediate motion; and (2) The fluctuation of the temporal differences between frames in a group is very small, unlike the non-noise video with natural motions which have higher motion differences without the noise. These two indicators are set to identify when a special bit allocation ratio between the I, P and B frames should be applied so that there will tend to be less difference between frame types for film-grain content.

The system according to embodiments of the present invention uses a preprocessing filter that analyzes video frames prior to the encoder. The preprocessing filter computes the temporal difference score and stores it in a queue of data provided with the frames to the encoder. The encoder analyzes the temporal difference scores. If it is detected that the average of all temporal differences is higher than a threshold and the variance of them are smaller than a threshold, it means the video contains significant film grain or noise. Based on the level of film grain or noise detected, the encoder allocates bits to I, P and B frames dynamically.

DETAILED DESCRIPTION

For embodiments of the present invention, if a clip can be identified as a “Red Lady” like clip, good quality can be achieved by encoding smaller I frames and allocating more bits on P and B frames.

To help understand how a determination of when a clip is a “Red Lady” type clip, several different clips of data are analyzed.FIGS. 4A-4Fshow the measured and plotted frame-by-frame temporal differences (ME scores) of various video clips. InFIGS. 4A-4F, the X axis is a frame index and the Y axis shows a scaled ME score ranging from 0 to 100.

From the measurements inFIGS. 4A-4F, two key indicators of noisy video are found: (1) The average temporal difference is at an intermediate level (greater than ME score of 20), and the ME score is higher than the one of non-noisy video with intermediate motion; and (2) The fluctuation of the temporal differences is very small, not like the non-noise video with natural motions. These two indicators are, thus, used to identify “Red Lady” like video frames from any streaming video.

FIGS. 4A-4Cillustrate the ME score levels for video with noise and limited or no motion. InFIG. 4C, the Red Lady video has a constant noise level ME score of just over 20. The “Sweep” video noise levels ofFIG. 4Bare very high with an ME noise level of near 100. The “Zone Plate” noise with no motion and a set noise level has an ME score of just over 20 and can be used as a reference for ME levels. Note that in the “Blacksmith” frame ofFIG. 4D, a normal video that does not need special treatment during encoding using embodiments of the present invention that the temporal average noise difference is far below an ME level of 20 found with the “Red Lady” video ofFIG. 4C.

Unlike the “Blacksmith” ofFIG. 4D, the “Sprinkler Lady” ofFIG. 4Emeets both the two key factors of (1) an ME score over 20 and (2) the fluctuation of the temporal differences is small even taking into account motion. The final video of “Basketball” inFIG. 4Fhas a relatively high ME score, but it is less than 20, and the motion in the video accounts for ME levels that on average may push the total ME score over 20. However with the criteria of (1) an average temporal difference with a ME score greater than 20 and (2) the fluctuation of the temporal differences very small, the video ofFIG. 4Fdoes not require embodiments of the present invention to be used during encoding.

FIG. 5is a diagram of one embodiment of components for implementing embodiments of the present invention in an encoding system. InFIG. 5, the preprocessing filter500computes the temporal difference score and stores it in the queue502. The encoding in encoder506will be delayed by the frame buffer504until temporal difference scores of N frames are collected in the queue502. The encoder506analyzes the temporal difference scores of N frames. If it is detected that the average of all temporal differences is higher than a threshold and the variance of them are smaller than a threshold, it means the video contains significant film grain or noise. Based on the level of film grain or noise detected, the encoder506allocates bits to I, P and B dynamically according to embodiments of the present invention described herein. Generally, if the level of film grain or noise is high, the encoder allocates more bits to P and B frames than for other content.

FIG. 6is a flow chart illustrating steps for implementing encoding when high film grain is detected according to embodiments of the present invention. First, in step600the video clips are received, such as the “Red Lady” clip illustrated inFIG. 4C. Next, in step601, the temporal difference score, or ME score, is determined for the video frames and the result for each frame stored in a queue to provide to the encoder. Next, in step602a determination is made to decide if film grain noise is high enough to constitute “Red Lady” type film grain that requires application of embodiments of the present invention. For the step602determination, if the average of a group of temporal differences is higher than a threshold and the variance is smaller than a threshold, the film grain noise is indicated to be significant for the frames of the video clip.

Once the determination is made in602, next in step603the determination is reviewed. If film grain noise for the clip is determined to be significant, then the program proceeds to step604. If the film grain noise is determined to be insignificant, the program proceeds to step605. In step604when high film grain noise is detected encoding is performed by allocating enough bits so that the I frame at the beginning has little additional bits and the P and B frames have additional bits for encoding. In step605when film grain noise is not detected as high, a normal bit allocation is performed by the encoder.

The results of applying the algorithm shown inFIG. 6will reduce the “dirty window” effect. The algorithm also makes some high texture clips, such as “sprinkler” ofFIG. 4E, look sharper. The algorithm does not change the quality of other non-noisy clips that do not rise to the detected level of the “Red Lady” video.

For reference, Appendix A below shows an example of coding in “C” to implement the algorithm illustrated byFIG. 6.

For components shown, like the pre-processing filter500and the encoder506, each component according to embodiments of the present invention can include a processor and memory to enable operation. The memory of each device stores code that is executable by the processor to enable the processor to perform the processes described herein. Further the memory can be used to provide data storage with the data accessible by the processor to store or retrieve when performing operations.

Although the present invention has been described above with particularity, this was merely to teach one of ordinary skill in the art how to make and use the invention. Many additional modifications will fall within the scope of the invention as that scope is defined by the following claims.