Dynamic Content Adaptive Frame Rate Conversion

A technique, as well as select implementations thereof, pertaining to dynamic content adaptive frame rate conversion is described. The technique may involve analyzing information associated with two or more video frames of a stream of video frames. The technique may also involve dynamically adjusting a frame rate of the two or more video frames of the stream of video frames based on a result of the analyzing.

TECHNICAL FIELD

The present disclosure is generally related to frame rate conversion and, more particularly, to methods, devices and apparatuses of dynamic content adaptive frame rate conversion.

BACKGROUND

Unless otherwise indicated herein, approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

Frame rate, also known as frame frequency, is the rate/frequency at which an imaging device displays consecutive images (e.g., video frames). Frame rate conversion (FRC) refers to the conversion of frame rate of a stream of video frames from a source frame rate to a display frame rate which is typically greater than the source frame rate. There are generally two approaches to FRC as shown in example600ofFIG. 6. A first approach, referred to as normal FRC herein and shown in part (A) ofFIG. 6, involves frame repetition and a second approach, referred to as motion-compensated (MC) FRC herein and shown in part (B) ofFIG. 6, involves interpolation of motion-compensated frames. In mobile applications such as smartphones, tablet computers and wearable devices, however, given the limited resources (e.g., memory and battery power) there is a need to conserve memory bandwidth and power consumption used for FRC.

SUMMARY

In one example implementation, a method may involve analyzing information associated with two or more video frames of a stream of video frames. The method may also involve dynamically adjusting a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing.

In another example implementation, a device may include a FRC module configured to perform a number of operations. The FRC module may analyze information associated with two or more video frames of a stream of video frames. The FRC module may also dynamically adjust a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing.

In yet another example implementation, an apparatus may include a FRC module and a display processing module. The FRC module may be configured to perform a number of operations. The FRC module may analyze information associated with two or more video frames of a stream of video frames. The FRC module may also dynamically adjust a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing. The display processing module may be configured to process the stream of video frames.

Thus, implementations in accordance with the present disclosure analyze the content of source video frames and dynamically adjust an output frame rate of FRC. Advantageously, both memory bandwidth and power consumption may be reduced as a result of utilizing the dynamic content adaptive FRC in accordance with the present disclosure. Accordingly, applications where resources such as memory and power are limited, such as mobile applications, can significantly benefit from techniques, methods, devices and apparatuses in accordance with the present disclosure.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Overview

FIG. 1illustrates example architecture100A, example architecture100B and example architecture100C in accordance with an implementation of the present disclosure. In each of architecture100A, architecture100B and architecture100C, content adaptive FRC is performed to dynamically adjust an output frame rate of the FRC as a result of analysis of the content of source video frames. Advantageously, this may result in reduced memory bandwidth requirement as well as reduced power consumption.

Referring toFIG. 1, architecture100A may include a source video module110, a content adaptive FRC module120, a display processing module130, a normal FRC (or motion-compensated FRC) module140and a panel150. In the example shown inFIG. 1, source video module110may provide a stream of source video frames at a source frame rate of30frames per second (fps). Content adaptive FRC module120may receive the stream of source video frames at 30 fps from source video module110to output a stream of output video frames at an output frame rate which is dynamically adjusted as a result of analysis of the content of at least some of the video frames of the stream of source video frames. The output frame rate may be, for example and not limited thereto, somewhere between 0 fps and 60 fps. Display processing module130may process the stream of output video frames without changing the frame rate thereof. Normal FRC module140may perform normal FRC (e.g., frame repetition) to result in a frame rate of 60 fps, for example, at the output thereof. Panel150may be configured to receive video frames at a fixed frame rate, and may receive the stream of video frames from normal FRC module140to display the video frames.

Referring toFIG. 1, architecture1008may include source video module110, content adaptive FRC module120, display processing module130and a panel150. In the example shown inFIG. 1, source video module110may provide a stream of source video frames at a source frame rate of 30 fps. Content adaptive FRC module120may receive the stream of source video frames at 30 fps from source video module110to output a stream of output video frames at an output frame rate which is dynamically adjusted as a result of analysis of the content of at least some of the video frames of the stream of source video frames. The output frame rate may be, for example and not limited thereto, somewhere between 0 fps and 60 fps. Display processing module130may process the stream of output video frames without changing the frame rate thereof. Panel150may be configured to receive video frames at a variable frame rate, and may receive the stream of video frames from normal FRC module140to display the video frames.

Referring toFIG. 1, architecture100C may include source video module110, display processing module130, content adaptive FRC module120and a panel150. In the example shown inFIG. 1, source video module110may provide a stream of source video frames at a source frame rate of 30 fps. Display processing module130may process the stream of source video frames without changing the frame rate thereof. Content adaptive FRC module120may receive the stream of source video frames at 30 fps from display processing module130to output a stream of output video frames at an output frame rate which is dynamically adjusted as a result of analysis of the content of at least some of the video frames of the stream of source video frames. The output frame rate may be, for example and not limited thereto, somewhere between 0 fps and 60 fps. Panel150may be configured to receive video frames at a variable frame rate, and may receive the stream of video frames from normal FRC module140to display the video frames.

In some cases, each of architecture100A, architecture1008and architecture100C may be modified in that functionalities of content adaptive FRC module120may be implemented in the midst of display processing. That is, content adaptive FRC may be performed as part of display processing by display processing module130modified accordingly. For instance, a modified display processing module130may perform some display processing, followed by content adaptive FRC which is then followed by some further display processing.

Moreover, in architecture100C, there may be a post-processing module (not shown) between content adaptive FRC module120and panel150. The post-processing module may perform color processing and/or provide one or more user interface (UI) functionalities.

Example Implementations

FIG. 2Aillustrates an example device200A for FRC in accordance with an implementation of the present disclosure. Device200A may perform various functions to implement techniques, processes and methods described herein, including at least a portion of architecture100A, architecture1008, architecture100C described above as well as process500described below. Device200A may be implemented in the form of a single integrated-circuit (IC) chip or a chipset of multiple IC chips. For instance, device200A may be a processor such as a central processing unit (CPU), a graphics processing unit (GPU) or an applications-specific IC (ASIC). Device200A may include a content adaptive FRC module202.

Content adaptive FRC module202may be configured to receive a stream of video frames, analyze information associated with two or more video frames of the stream of video frames, and dynamically adjust a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing. In some implementations, as shown inFIG. 2A, content adaptive FRC module202may include at least those components shown inFIG. 2A, such as a video content analyzer210, a dynamic frame rate output controller220and an output frame generator230.

Content adaptive FRC module202may be in the form of hardware, software, middleware, firmware or any combination thereof. For example, content adaptive FRC module202may be implemented in a hardware form of a circuit or IC chip. As another example, content adaptive FRC module202may be implemented in a software form such as one or more sets of instructions executable by a CPU, a GPU or an ASIC.

In some implementations, video content analyzer210may be configured to analyze a content of every two or more consecutive video frames of the stream of source video frames. Dynamic frame rate output controller220may be configured to dynamically adjust the frame rate of at least the two or more video frames of the stream of source video frames based on a result of analyzing by the video content analyzer210. Output frame generator230may be configured to receive the stream of source video frames (e.g., from source video module110as in architecture100A and architecture1008, or from display processing module130as in architecture100C) at a source frame rate and output a stream of output video frames at an output frame rate.

In some implementations, video content analyzer210may be configured to obtain information on the content of every two or more consecutive video frames from the stream of source video frames. In some implementations, in analyzing the content of every two or more consecutive video frames of the stream of source video frames, video content analyzer210may be configured to analyze a texture, a motion speed, a motion complexity, or a combination thereof regarding the content of every two or more consecutive video frames of the stream of source video frames.

In some implementations, in dynamically adjusting the frame rate of at least the two or more video frames, dynamic frame rate output controller220may be configured to maintain the frame rate of at least the two or more video frames at the source frame rate based on the result of the analyzing. For instance, for a still scene (e.g., no motion), dynamic frame rate output controller220may maintain the source frame rate without making adjustment. As another example, for a still scene (e.g., no motion), dynamic frame rate output controller220may decrease the frame rate by skipping one or more similar video frames from the stream of source video frames in providing a stream of output video frames. As a further example, for a complicated scene (e.g., multiple motions such as a fighting scene or sports scene), dynamic frame rate output controller220may maintain the source frame rate without making adjustment, since the effect of up-conversion may be limited.

Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, dynamic frame rate output controller220may be configured to adjust the frame rate of at least the two or more video frames to be between the source frame rate and a display frame rate based on the result of the analyzing, where the display frame rate is different from (e.g., greater than or less than) the source frame rate. For instance, for a near-still or slow-speed scene, dynamic frame rate output controller220may adjust the output frame rate to be between the source frame rate and the display frame rate.

Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, dynamic frame rate output controller220may be configured to increase the frame rate of at least the two or more video frames to a display frame rate greater than the source frame rate based on the result of the analyzing. For instance, for a normal motion scene, dynamic frame rate output controller220may increase the output frame rate to match the display frame rate. Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, dynamic frame rate output controller220may be configured to decrease the frame rate of at least the two or more video frames to a display frame rate less than the source frame rate based on the result of the analyzing.

In some implementations, output frame generator230may be configured to output the stream of source video frames as the stream of output video frames.

Alternatively or additionally, output frame generator230may be configured to drop one or more frames of a still scene of the stream of source video frames in outputting the stream of output video frames. For instance, for a still scene, output frame generator230may drop one or more source video frames that are similar to one another.

Alternatively or additionally, output frame generator230may be configured to generate one or more interpolated frames for frame rate up-conversion in outputting the stream of output video frames.

FIG. 2Billustrates an example device200B for FRC in accordance with another implementation of the present disclosure. Device200B may perform various functions to implement techniques, processes and methods described herein, including at least a portion of architecture100A, architecture100B, architecture100C described above as well as process500described below. Device200B may be implemented in the form of a single IC chip or a chipset of multiple IC chips. For instance, device200B may be a processor such as a CPU, a GPU or an ASIC. Device200B may include a content adaptive FRC module204.

Content adaptive FRC module204may be configured to receive a stream of video frames, analyze information associated with two or more video frames of the stream of video frames, and dynamically adjust a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing. In some implementations, as shown inFIG. 2B, content adaptive FRC module204may include at least those components shown inFIG. 2B, such as a video content analyzer210, a dynamic frame rate output controller220and an output frame generator230.

Content adaptive FRC module204may be in the form of hardware, software, middleware, firmware or any combination thereof. For example, content adaptive FRC module204may be implemented in a hardware form of a circuit or IC chip. As another example, content adaptive FRC module204may be implemented in a software form such as one or more sets of instructions executable by a CPU, a GPU or an ASIC.

As the structure and functionality of content adaptive FRC module204may be similar to those of content adaptive FRC module202, the following description of content adaptive FRC module204is focused on differences between content adaptive FRC module204and content adaptive FRC module202. That is, a detailed description of portions of content adaptive FRC module204that are similar to content adaptive FRC module202is not provided below in the interest of brevity.

In some implementations, video content analyzer210may be configured to receive information on the content of every two or more consecutive video frames from a video decoder (not shown). In some implementations, in analyzing the content of every two or more consecutive video frames of the stream of source video frames, video content analyzer210may be configured to analyze motion vectors, block residues, or a combination thereof regarding the stream of source video frames.

FIG. 3illustrates an example device300for FRC in accordance with a further implementation of the present disclosure. Device300may perform various functions to implement techniques, processes and methods described herein, including at least a portion of architecture100A, architecture1008, architecture100C described above as well as process500described below. Device300may be implemented in the form of a single IC chip or a chipset of multiple IC chips. For instance, device200B may be a processor such as a CPU, a GPU or an ASIC. Device300may include a content adaptive FRC module302.

Content adaptive FRC module302may be configured to receive a stream of video frames, analyze information associated with two or more video frames of the stream of video frames, and dynamically adjust a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing. In some implementations, content adaptive FRC module302may include at least those components shown inFIG. 3, such as a video content analyzer310, a dynamic frame rate output controller320and an output frame generator330.

Content adaptive FRC module302may be in the form of hardware, software, middleware, firmware or any combination thereof. For example, content adaptive FRC module302may be implemented in a hardware form of a circuit or IC chip. As another example, content adaptive FRC module302may be implemented in a software form such as one or more sets of instructions executable by a CPU, a GPU or an ASIC.

As the structure and functionality of content adaptive FRC module302may be similar to those of content adaptive FRC module202and/or content adaptive FRC module204, the following description of content adaptive FRC module302is focused on differences between content adaptive FRC module302and content adaptive FRC modules202and204. For instance, dynamic frame rate output controller320and output frame generator330may be similar or identical to dynamic frame rate output controller220and output frame generator230. Thus, a detailed description of portions of content adaptive FRC module302that are similar to content adaptive FRC modules202and204is not provided below in the interest of brevity.

In some implementations, video content analyzer310may be configured to receive information (e.g., motion vectors) from a video decoder (e.g., an h.264 decoder which is not shown). In some implementations, in analyzing the content of every two or more consecutive video frames of the stream of source video frames, video content analyzer310may be configured to analyze the motion vectors and provide a result of the analysis to dynamic frame rate output controller320.

Video content analyzer310may include a motion vector difference calculator312, a buffer314and a threshold checker316. For each motion vector received from the video decoder, motion vector difference calculator312may be configured to calculate or otherwise determine a difference between a current motion vector and a previous motion vector. Buffer314may be configured to store a value of the previous motion vector and replace it with a value of the current motion vector. Threshold checker316may be configured to receive, from motion vector difference calculator312, a value representative of the difference to determine whether the difference between the current motion vector and the previous motion vector is greater than a predefined threshold. In an event that threshold checker316determines that the difference is greater than the predefined threshold, threshold checker316may trigger dynamic frame rate controller320to perform dynamically adjust the frame rate of at least the two or more video frames of the stream of source video frames. Otherwise, in an event that threshold checker316determines that the difference is not greater than the predefined threshold, threshold checker316may take no action to trigger dynamic frame rate controller320.

FIG. 4illustrates an example apparatus400A and another example apparatus400B in accordance with an implementation of the present disclosure. Each of apparatus400A and apparatus400B may perform various functions to implement techniques, processes and methods described herein, including at least a portion of architecture100A, architecture1008, architecture100C described above as well as process500described below. Each of apparatus400A and apparatus400B may be implemented in the form of a single IC chip or a chipset of multiple IC chips. In some implementations, each of apparatus400A and apparatus400B may be an electronic apparatus which may be a computing apparatus, a portable/mobile apparatus or a wearable apparatus. For instance, each of apparatus400A and apparatus400B may be a smartphone, smartwatch, a computing device such as a tablet computer, a laptop computer, a notebook computer, or a wearable apparatus. Each of apparatus400A and apparatus400B may include at least those components shown inFIG. 4, such as a content adaptive FRC module410and a display processing module420. Although not shown inFIG. 4, each of apparatus400A and apparatus400B may optionally include additional components such as, for example, a source video module, a normal FRC module, a motion-compensated FRC module, a display module or device and/or a video decoder, and in some implementations one, some or all of these components may be external to either or both of apparatus400A and apparatus400B.

In each of apparatus400A and apparatus400B, display processing module420may be configured to process a stream of video frames. However, in apparatus400A display processing module420may be coupled to receive the stream of video frames from content adaptive FRC module410, and in apparatus400B display processing module420may be coupled to provide the stream of video frames to content adaptive FRC module410.

Content adaptive FRC module410may be implemented by either content adaptive FRC module202of device200A, content adaptive FRC module204of device200B or content adaptive FRC module302of device300. That is, content adaptive FRC module410may adapt the structure/architecture of either content adaptive FRC module202of device200A or content adaptive FRC module204of device200B. In any case, content adaptive FRC module410may be configured to analyze information associated with two or more video frames of the stream of video frames. Content adaptive FRC module410may be also configured to dynamically adjust a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing.

In some implementations, in analyzing the information associated with the two or more video frames, content adaptive FRC module410may be configured to analyze a content of every two or more consecutive video frames of the stream of video frames. In some implementations, in analyzing the content of every two or more consecutive video frames of the stream of video frames, content adaptive FRC module410may be configured to analyze a texture, a motion speed, a motion complexity, or a combination thereof regarding the content of every two or more consecutive video frames of the stream of video frames. [0050]Optionally, each of apparatus400A and apparatus400B may also include a video decoder430configured to provide information regarding the stream of video frames, and content adaptive FRC module410may be configured to receive the information regarding the stream of video frames from video decoder430. Specifically, in apparatus400A, content adaptive FRC module410may be configured and coupled to receive a stream of source video frames from a source external of apparatus400A in addition to receiving information regarding the source video frames from video decoder430. In contrast, in apparatus400B, content adaptive FRC module410may be configured and coupled to receive the stream of source video frames from display processing module420in addition to receiving information regarding the source video frames from video decoder430. In such cases the information associated with the two or more video frames may include the information received from video decoder430. In some implementations, the information received from video decoder430may include information on motion vectors, block residues, or a combination thereof regarding the stream of video frames.

In some implementations, in dynamically adjusting the frame rate of at least the two or more video frames, content adaptive FRC module410may be configured to maintain the frame rate of at least the two or more video frames at a source frame rate of the stream of video frames based on the result of the analyzing. Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, content adaptive FRC module410may be configured to adjust the frame rate of at least the two or more video frames to be between a source frame rate of the stream of video frames and a display frame rate based on the result of the analyzing, with the display frame rate being different from (e.g., greater than or less than) the source frame rate. Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, content adaptive FRC module410may be configured to increase the frame rate of at least the two or more video frames to a display frame rate, which is greater than a source frame rate of the stream of video frames, based on the result of the analyzing. Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, content adaptive FRC module410may be configured to decrease the frame rate of at least the two or more video frames to a display frame rate, which is less than a source frame rate of the stream of video frames, based on the result of the analyzing.

FIG. 5illustrates an example process500in accordance with an implementation of the present disclosure. Process500may include one or more operations, actions, or functions as represented by one or more blocks such as blocks510,520and530. Although illustrated as discrete blocks, various blocks of process500may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation. The blocks may be performed in the order shown inFIG. 5or in any other order, depending on the desired implementation. Process500may be implemented by content adaptive FRC module120, device200A, device200B, device300, apparatus400A and apparatus400B. Solely for illustrative purpose and without limiting the scope of the present disclosure, process500is described below in the context of process500being performed by content adaptive FRC module120. Process500may begin at510.

At510, process500may involve content adaptive FRC module120receiving a stream of video frames from source video module110(e.g., in architecture100A and architecture100B) or from display processing module130(e.g., in architecture100C). Process500may proceed from510to520.

At520, process500may involve content adaptive FRC module120analyzing information associated with two or more video frames of the stream of video frames. Process500may proceed from520to530.

At530, process500may involve content adaptive FRC module120dynamically adjusting a frame rate of at least the two or more video frames of the stream of video frames based on a result of the analyzing.

In some implementations, in analyzing the information associated with the two or more video frames, process500may involve content adaptive FRC module120analyzing a content of every two or more consecutive video frames of the stream of video frames. In some implementations, in analyzing the content of every two or more consecutive video frames of the stream of video frames, process500may involve content adaptive FRC module120analyzing a texture, a motion speed, a motion complexity, or a combination thereof regarding the content of every two or more consecutive video frames of the stream of video frames.

In some implementations, the information associated with the two or more video frames may include information received from a video decoder regarding the stream of video frames. In some implementations, the information received from the video decoder may include information on motion vectors, block residues, or a combination thereof regarding the stream of video frames.

In some implementations, in dynamically adjusting the frame rate of at least the two or more video frames, process500may involve content adaptive FRC module120maintaining the frame rate of at least the two or more video frames at a source frame rate of the stream of video frames based on the result of the analyzing.

Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, process500may involve content adaptive FRC module120adjusting the frame rate of at least the two or more video frames to be between a source frame rate of the stream of video frames and a display frame rate based on the result of the analyzing, with the display frame rate being different from (e.g., greater than or less than) the source frame rate.

Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, process500may involve content adaptive FRC module120increasing the frame rate of at least the two or more video frames to a display frame rate, which is greater than a source frame rate of the stream of video frames, based on the result of the analyzing.

Alternatively or additionally, in dynamically adjusting the frame rate of at least the two or more video frames, process500may involve content adaptive FRC module120decreasing the frame rate of at least the two or more video frames to a display frame rate, which is less than a source frame rate of the stream of video frames, based on the result of the analyzing.

Additional Notes