Patent Description:
For example, in PTL <NUM>, it is stated that converting a <NUM> frame-rate image stream into a <NUM> frame-rate image stream by inserting an intermediate frame image between every pair of two consecutive frames in the <NUM> frame-rate image stream makes display of moving images smooth. Further, in PTL <NUM>, it is stated that inserting a black image frame into a portion corresponding to the intermediate frame image achieves a sharp image quality peculiar to movie content (film images). In the case where a black image frame is inserted as the intermediate frame image, there is a disadvantage that overall luminance is lowered.

PTL <NUM> discloses an image signal processing apparatus, which performs a luminance value conversion process of an input image signal. The luminance of a frame is changed according to whether the luminance is above or below a threshold, leading to the generation of an output image signal with adapted luminance characteristics.

PTL <NUM> discloses an image processing apparatus to perform a frame rate conversion process based on a phase adjustment, such as for a <NUM> cinema image that cannot be displayed on a display device displaying a <NUM> video. The apparatus can calculate a motion vector (V) and an interpolation phase (K = <NUM>/<NUM> = <NUM>) for two consecutive frames. Data for an intermediate frame can be generated, but the intermediate frame is out of phase (not integral). A mapping operation is performed to have the intermediate frame match the phase of the consecutive frame with a calculation of a corrected interpolation phase coefficient. Further phase adjustments further away from <NUM> by finer parameters help to eliminate judder.

PTL <NUM> discloses a system for adapting film judder correction or motion compensation. Positions of elements in additional frames to be inserted corresponding to elements in two or more consecutive frames are determined based on light intensity variables. For example, for a it is preferred to have a determined position closer to a position of an element in a first video for a higher light intensity, and comparitively farther for a lower light intensity.

An object of the present technology is to make it possible to make a favorable display of, for example, images of low frame-rate content, with a high-luminance and high-contrast television set.

The concept of the present technology lies in an image processing device including a target-object detection processing section that, on the basis of an image stream having a first frame rate, detects, for each frame, an object having luminance exceeding a luminance threshold value and/or a motion amount exceeding a motion-amount threshold value, as a target object; and an interpolated-image insertion processing section that acquires an image stream having a second frame rate larger than the first frame rate by inserting, between every pair of two consecutive frames in the image stream having the first frame rate, a predetermined number of frames of interpolated images obtained by performing motion compensation for causing the target object to sequentially move.

In the present technology, by the target-object detection processing section, on the basis of the image stream having the first frame rate, for each frame, the object having luminance exceeding the luminance threshold value and/or a motion amount exceeding the motion-amount threshold value is detected as the target object. For example, the luminance threshold value may include a maximum value of object luminance at which judder (strobing) is unnoticeable in the image stream having the first frame rate. Further, for example, the motion-amount threshold value may include a maximum value of an object motion amount at which judder (strobing) is unnoticeable in the image stream having the first frame rate. Further, for example, the luminance threshold value and the motion-amount threshold value may each include a variable that changes according to an environment.

By the interpolated-image insertion processing section, the image stream having the second frame rate larger than the first frame rate is acquired by inserting, between every pair of two consecutive frames in the image stream having the first frame rate, the predetermined number of frames of interpolated images obtained by performing the motion compensation for causing the target object to sequentially move. For example, the interpolated-image insertion processing section may cause a movement amount of the target object in the predetermined number of interpolated images to be changed according to the luminance and/or the motion amount of the target object.

Further, for example, the image stream having the first frame rate may include an image stream having a frame rate of <NUM> and related to movie content. In this case, for example, the second frame rate may include an image stream having a frame rate of <NUM>.

In the present technology, in such a way as described above, the image stream having the second frame rate larger than the first frame rate is acquired by inserting, between every pair of two consecutive frames in the image stream having the first frame rate, the predetermined number of frames of interpolated images obtained by performing the motion compensation for causing the target object, which is the object having the luminance exceeding the luminance threshold value and/or the motion amount exceeding the motion-amount threshold value, to sequentially move. Thus, even with a high-luminance and high-contrast television set, it is possible to make a favorable display of, for example, images of movie content, in a state in which judder is unnoticeable but movie-like motion characteristics remain.

In addition, in the present technology, for example, a display panel that displays the image stream having the second frame rate may further be provided. In this case, for example, the display panel may include a <NUM>-size or <NUM>-size display panel. Further, in the present technology, the image stream having the first frame rate may be acquired by reception of a broadcasting signal, reproduction from a storage, or communication. Further, in the present technology, the target-object detection processing section may detect, as the target object, an object having a size exceeding a size threshold value in addition to luminance exceeding the luminance threshold value and/or a motion amount exceeding the motion-amount threshold value.

Hereinafter, a mode for carrying out the present invention (hereinafter referred to as an "embodiment") will be described. Note that the description will be made in the following order.

<FIG> illustrates an example of frame rate conversion for converting a <NUM> (<NUM> P) frame-rate movie-content image stream into a <NUM> (<NUM> P) frame-rate image stream, according to a conventional technology. In display of the <NUM> P frame-rate image stream, motion amounts are large, and judder is noticeable. The <NUM> P frame-rate image stream is obtained by inserting, between every pair of two consecutive frames in the <NUM> P frame-rate image stream, four frames of interpolated images obtained by performing motion compensation for sequential movements on the whole of the images. In this case, the interpolation makes motion amounts between the frames small, so that judder becomes unnoticeable, but motions become less peculiar to movies.

<FIG> illustrates an example of frame rate conversion for converting a <NUM> (<NUM> P) frame-rate movie-content image stream into a <NUM> (<NUM> P) frame-rate image stream, according to the present technology. In this example, a first frame rate corresponds to <NUM>, and a second frame rate corresponds to <NUM>. The <NUM> P frame-rate image stream is obtained by inserting, between every pair of two consecutive frames in the <NUM> P frame-rate image stream, four frames of interpolated images obtained by performing motion compensation for sequential movements only on a target object that has high luminance and a large motion amount, that is, in which judder is noticeable. In this case, since the motion compensation is performed only on a portion corresponding to the object in which judder is noticeable, the judder becomes unnoticeable and motions peculiar to movies can be maintained.

<FIG> illustrates two consecutive frames including an n-th frame and an (n + <NUM>)th frame of the <NUM> P frame-rate image stream, and <FIG> illustrates <NUM> P frame-rate images corresponding to the two consecutive frames. In this case, five frames including 5n-th to (5n + <NUM>)th frames correspond to the n-th frame of the <NUM> P frame-rate image stream, the image of the 5n-th frame is the same as the image of the n-th frame, and the images of four frames including (5n + <NUM>)th to (5n + <NUM>)th frames are interpolated images obtained by the motion compensation performed only on the target object.

Further, similarly, five frames including <NUM>(n + <NUM>)th to (<NUM>(n + <NUM>) + <NUM>)th frames correspond to the (n + <NUM>)th frame of the <NUM> P frame-rate image stream, the image of the <NUM>(n + <NUM>)th frame is the same as the image of the (n + <NUM>)th frame, and the images of four frames including (<NUM>(n + <NUM>) + <NUM>)th to (<NUM>(n + <NUM>) + <NUM>)th frames are interpolated images obtained by the motion compensation performed only on the target object.

<FIG> schematically illustrates an example of generation of interpolated images. This example is an example in a case where a high-luminance object having a large motion amount and a low-luminance object having a large motion amount exist in the <NUM> P frame-rate image stream. Note that it is sufficient if the high-luminance object has high luminance in its partial portion, and the high-luminance object does not need to entirely have high luminance. This similarly applies to the following description.

The high-luminance object also has a large motion amount, and thus, it is detected as a target object. In each of the interpolated images of the four frames including the (5n + <NUM>)th to (5n + <NUM>)th frames, the high-luminance object is motion compensated and is placed at a sequentially moved position. By contrast, the low-luminance object has a large motion amount but has low luminance, and thus, it is not motion compensated. The low-luminance object is placed at a position same as that in the image of the 5n-th frame, in each of the interpolated images of the four frames including the (5n + <NUM>)th to (5n + <NUM>)th frames.

<FIG> schematically illustrates another example of generation of interpolated images. This example is an example in a case where a high-luminance object having a large motion amount and another high-luminance object having a small motion amount exist in the <NUM> P frame-rate image stream.

The high-luminance object having a large motion amount is detected as a target object, and in each of the interpolated images of the four frames including the (5n + <NUM>)th to (5n + <NUM>)th frames, this high-luminance object is motion compensated and is placed at a sequentially moved position. By contrast, the high-luminance object having a small motion amount is not motion compensated and is placed at a position same as that in the image of the 5n-th frame, in each of the interpolated images of the four frames including the (5n + <NUM>)th to (5n + <NUM>)th frames.

As described above, a target object to be motion compensated in the interpolated images is an object having high luminance and a large motion amount. <FIG> illustrates correspondence between object luminance and a luminance parameter. A luminance threshold value Lobj0 is a maximum value of object luminance in the <NUM> P frame-rate image stream at which value judder is unnoticeable. This luminance threshold value Lobj0 is an environmental parameter (variable) that changes according to brightness of a surrounding area (environment under which the images are viewed). With an increase in the object luminance beyond the luminance threshold value Lobj0, the luminance parameter lineally increases from zero. Note that it can also be considered that the luminance parameter is caused to nonlinearly change.

<FIG> illustrates correspondence between an object motion amount and a motion-amount parameter. A motion-amount threshold value Mobj0 is a maximum value of an object motion amount in the <NUM> P frame-rate image stream at which value judder is unnoticeable. This motion-amount threshold value Mobj0 is an environmental parameter (variable) that changes according to a size of a display panel and a viewing distance. With an increase in the object motion amount beyond the motion-amount threshold value Mobj0, the motion-amount parameter lineally increases from zero. Note that it can also be considered that the motion-amount parameter is caused to nonlinearly change.

In the present technology, an object with respect to which a value obtained by multiplying the luminance parameter by the motion-amount parameter is larger than zero is detected as a target object to be motion compensated. In this case, as illustrated in <FIG>, for an object with respect to which a value obtained by multiplying the luminance parameter by the motion-amount parameter is equal to zero, the motion compensation is Off, that is, the motion compensation is not performed; whereas, for an object with respect to which a value obtained by multiplying the luminance parameter by the motion-amount parameter exceeds zero, the motion compensation is On, that is, the motion compensation is performed.

Note that it can also be considered that the effect of the motion compensation is not controlled between the two stages Off and On but is caused to change according to the value obtained by multiplying the luminance parameter by the motion-amount parameter. <FIG> illustrates a case in which the effect of the motion compensation on a target object is caused to lineally change according to the value obtained by multiplying the luminance parameter by the motion-amount parameter. Note that, while detailed description of <FIG> is omitted, <FIG> are the same as <FIG>, respectively. Note that it can also be considered that the effect of the motion compensation is caused to nonlinearly change.

The maximum of the effect of the motion compensation here is the same as that in the state in which the motion compensation is On as in <FIG>. The motion compensation examples illustrated in above-described <FIG> and <FIG> each illustrate a case where the effect of the motion compensation is set to its maximum. Further, when the value obtained by multiplying the luminance parameter by the motion-amount parameter is equal to zero, the effect of the motion compensation is minimum, that is, the effect of the motion compensation is the same as that in the state in which the motion compensation is Off as in <FIG>.

<FIG> illustrates examples of changes in the effect of the motion compensation. <FIG> illustrates a case where the effect of the motion compensation is maximum. In this case, when the position of a target object in the image of the 5n-th frame is denoted by R0 and the position of the target object in the image of the <NUM>(n + <NUM>)th frame is denoted by R1, a distance between R0 and R1 is equally divided into, for example, five sections, and the positions of the target object in the interpolated images of the four frames including the (5n + <NUM>)th to (5n + <NUM>)th frames are set to boundaries of the five sections.

<FIG> illustrates a case where the effect of the motion compensation is smaller than that in the case of <FIG>. In this case, a position slightly closer to R0 than R1 is denoted by R2, a distance between R0 and R2 is equally divided into, for example, four sections, and the positions of the target object in the interpolated images of the four frames including the (5n + <NUM>)th to (5n + <NUM>)th frames are set to boundaries of the four sections and R2. Further, <FIG> illustrates a case where the effect of the motion compensation is smaller than that in the case of <FIG>. In this case, a position further slightly closer to R0 than R2 is denoted by R3, a distance between R0 and R3 is equally divided into, for example, four sections, and the positions of the target object in the interpolated images of the four frames including the (5n + <NUM>)th to (5n + <NUM>)th frames are set to boundaries of the four sections and R3.

<FIG> illustrates a configuration example of a television receiver <NUM> as an embodiment. The television receiver <NUM> includes a video input unit <NUM>, an image-quality adjustment unit <NUM>, a frame-rate conversion unit <NUM>, a panel drive circuit <NUM>, and a display panel <NUM>.

The video input unit <NUM> acquires a video signal by reception of a broadcasting signal, communication, or reproduction of a storage (disk). It is assumed here that the video signal corresponds to an image stream having a frame rate of <NUM> and related to movie content. In the case of the broadcasting signal, the <NUM> frame-rate video signal related to the movie content is transmitted thereto in a state of being converted into a <NUM> (<NUM> P) frame-rate video signal by means of, for example, the <NUM>-<NUM> pull-down method at a broadcasting station side. The video input unit <NUM> receives the <NUM> frame-rate video signal and extracts individual frames of the original <NUM> frame-rate video signal related to the movie content from the received <NUM> frame-rate video signal, thereby reconstructing the <NUM> frame-rate video signal related to the movie content.

The image-quality adjustment unit <NUM> performs image-quality adjustment processing for adjusting brightness, contrast, sharpness, and the like on the video signal acquired by the video input unit <NUM>. The frame-rate conversion unit <NUM> performs processing for converting the frame rate from <NUM> to <NUM> on a video signal SVa output from the image-quality adjustment unit <NUM>, and outputs a video signal SVb corresponding to an image stream having the frame rate of <NUM>.

<FIG> illustrates a configuration example of the frame-rate conversion unit <NUM>. The frame-rate conversion unit <NUM> includes a luminance comparison processing section <NUM>, a motion-amount comparison processing section <NUM>, a target-object detection processing section <NUM>, and an interpolated-image insertion processing section <NUM>.

The luminance comparison processing section <NUM> receives an input of the video signal SVa corresponding to the image stream having the frame rate of <NUM> and related to the movie content. The luminance comparison processing section <NUM> detects, for each frame, luminance in a predetermined unit, for example, in a unit of a macro block, compares the luminance of each unit portion with the luminance threshold value Lobj0 (see <FIG> and <FIG>), and acquires information regarding unit portions in the image that have luminance exceeding the luminance threshold value Lobj0. The information regarding the unit portions includes information regarding the luminance parameter (see <FIG> and <FIG>) in addition to position information.

The motion-amount comparison processing section <NUM> receives an input of the video signal SVa corresponding to the aforementioned image stream having the frame rate of <NUM> and related to the movie content. The motion-amount comparison processing section <NUM> detects, for each frame, a motion vector in a predetermined unit, for example, in a unit of a macro block, compares a motion amount of each unit portion with the motion-amount threshold value Mobj0 (see <FIG> and <FIG>), and acquires information regarding unit portions in the image that have a motion amount exceeding the motion-amount threshold value Mobj0. The information regarding the unit portions includes information regarding a motion parameter (see <FIG> and <FIG>) in addition to the motion vector and the motion amount (magnitude of the motion vector).

The target-object detection processing section <NUM> is supplied, for each frame, with the information acquired by the luminance comparison processing section <NUM> and associated with the unit portions in the image that have luminance exceeding the luminance threshold value Lobj0, and the information acquired by the motion-amount comparison processing section <NUM> and associated with the unit portions in the image that have a motion amount exceeding the motion-amount threshold value Mobj0. On the basis of the information supplied in such a way as described above, the target-object detection processing section <NUM> detects, for each frame, an object that exists in the image and has luminance exceeding the luminance threshold value Lobj0 and a motion amount exceeding the motion-amount threshold value Mobj0 (corresponding to the high-luminance object having a large motion amount in the examples of <FIG> and <FIG>) as a target object. Note that, while detailed description of a method for detecting an object is omitted, any of conventionally known methods may be used, for example.

The interpolated-image insertion processing section <NUM> is supplied with information regarding each target object detected by the target-object detection processing section <NUM>. This information includes the position information, the motion vector, the luminance parameter, and the motion parameter that are associated with the target object. Further, the interpolated-image insertion processing section <NUM> also receives an input of the video signal SVa corresponding to the aforementioned image stream having the frame rate of <NUM> and related to the movie content.

The interpolated-image insertion processing section <NUM> acquires the image stream having the frame rate of <NUM> by inserting, between every pair of two consecutive frames in the image stream having the frame rate of <NUM>, four frames of interpolated images, and outputs the video signal SVb corresponding to the image stream having the frame rate of <NUM>. In this case, the interpolated-image insertion processing section <NUM> performs, in the four frames of interpolated images, the motion compensation for causing only target objects to sequentially move, on the basis of the information regarding each target object (see <FIG> and <FIG>).

Further, in this case, the interpolated-image insertion processing section <NUM> performs the motion compensation as illustrated in <FIG> or the motion compensation as illustrated in <FIG>, depending on, for example, predetermined settings. In the motion compensation as illustrated in <FIG>, motion compensation that constantly makes its movement amount maximum is performed for each target object, regardless of the magnitude of the value obtained by multiplying the luminance parameter by the motion-amount parameter (see <FIG>). By contrast, in the motion compensation illustrated in <FIG>, motion compensation is performed such that the larger the value obtained by multiplying the luminance parameter by the motion-amount parameter is, the larger the movement amount becomes (see <FIG> in this order).

Note that a portion or the whole of the processing of each section of the frame-rate conversion unit <NUM> can also be performed by software processing executed by a computer.

Referring back to <FIG>, the panel drive unit <NUM> drives the display panel <NUM> on the basis of the video signal SVb acquired by the frame-rate conversion unit <NUM> and corresponding to the image stream having the frame rate of <NUM> to cause the display panel <NUM> to display the image stream having the frame rate of <NUM>. The display panel <NUM> is a liquid crystal display panel, an organic EL display panel, a CLED (Crystal LED) panel, or the like. The display panel <NUM> is, for example, a <NUM>-size display panel, an <NUM>-size display panel, or the like.

Operation of the television receiver <NUM> illustrated in <FIG> will be briefly described. In the video input unit <NUM>, a video signal corresponding to an image stream having the frame rate of <NUM> and related to movie content is acquired by reception of a broadcasting signal, communication, or reproduction of a storage (a disk), and the acquired video signal is supplied to the image-quality adjustment unit <NUM>. In the image-quality adjustment unit <NUM>, image-quality adjustment processing for adjusting brightness, contrast, sharpness, and the like is performed on the video signal corresponding to the image stream having the frame rate of <NUM>. A video signal SVa obtained as a result of the image-quality adjustment processing is supplied to the frame-rate conversion unit <NUM>.

In the frame-rate conversion unit <NUM>, processing for converting the frame rate from <NUM> to <NUM> is performed on the video signal SVa output from the image-quality adjustment unit <NUM>, to thereby acquire a video signal SVb corresponding to the image stream having the frame rate of <NUM>. In this case, four frames of interpolated images are inserted between every pair of two consecutive frames in the image stream having the frame rate of <NUM> to acquire the image stream having the frame rate of <NUM>. Here, in the four frames of interpolated images, only a target object having luminance exceeding the luminance threshold value Lobj0 and having a motion amount exceeding the motion-amount threshold value Mobj0 is motion compensated so as to sequentially move.

The video signal SVb acquired by the frame-rate conversion unit <NUM> and corresponding to the image stream having the frame rate of <NUM> is supplied to the panel drive circuit <NUM>. In the panel drive unit <NUM>, the display panel <NUM> is driven on the basis of the video signal SVb, and the image stream having the frame rate of <NUM> and related to the movie content is displayed on the display panel <NUM>.

As described above, in the television receiver <NUM> illustrated in <FIG>, the frame-rate conversion unit <NUM> inserts four frames of interpolated images obtained by performing the motion compensation for causing only the target object having luminance exceeding the luminance threshold value and a motion amount exceeding the motion-amount threshold value to sequentially move, between every pair of two consecutive frames in the image stream having the frame rate of <NUM>, thereby acquiring the image stream having the frame rate of <NUM>. Therefore, even with a high-luminance and high-contrast television set, it is possible to make a favorable display of images of movie content in a state in which judder is unnoticeable but suitable judder remains.

Note that, in the above-described embodiment, an object having luminance exceeding a luminance threshold value and a motion amount exceeding a motion-amount threshold value is determined as a target object to be motion compensated, but it can also be considered that an object having luminance exceeding the luminance threshold value or an object having a motion amount exceeding the motion-amount threshold value is determined as a target object to be motion compensated.

Further, in the above-described embodiment, a target object to be motion compensated is determined without taking a size of the object into consideration, but it can also be considered that only an object having a size exceeding a size threshold value is determined as a target object to be motion compensated.

Further, although not described above, in a case where a user sets a movie mode in image-quality mode setting, or in a case where movie content is detected in an automatic mode, the above-described frame rate conversion in the present technology can be applied.

Claim 1:
An image processing device (<NUM>) comprising:
a target-object detection processing section (<NUM>) that is configured to, on a basis of an image stream having a first frame rate, detect, for each frame, a moving object having luminance exceeding a luminance threshold value in a partial portion of the moving object and/or a motion amount exceeding a motion-amount threshold value, as a target object; and
an interpolated-image insertion processing section (<NUM>) that is configured to insert between every pair of two consecutive frames in the image stream having the first frame rate, a number of frames of interpolated images obtained by performing motion compensation for causing the target object to sequentially move, to generate an image stream having a second frame rate larger than the first frame rate.