Patent Publication Number: US-2023162322-A1

Title: Image processing method and image processing circuit

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to a signal processing method. 
     2. Description of the Prior Art 
     For the purpose of increasing a frame rate to obtain a better display effect, an interpolated frame between two frames may be generated. Motion vector(s) and related image contents may be generated by motion estimation and motion compensation (MEMC). The motion estimation may cause a halo effect when an object moves fast in the image, however. A common MEMC process for eliminating the halo effect has the following problems: a width of the halo is too large; edges of the halo are misaligned; and strength of halo reduction is too strong and thereby introduces edge flicker. 
     SUMMARY OF THE INVENTION 
     It is therefore an objective of the present invention to provide an image processing method, which can determine whether an area near an edge of the interpolated frame is located in a cover area or an uncover area, and adjust the image content of the interpolated frame accordingly, to effectively alleviate the halo effect caused by the interpolated frame, to solve the problems of the related art. 
     According to one embodiment of the present invention, an image processing method comprises the steps of: receiving an image signal, wherein the image signal comprises a first frame and a second frame; performing motion estimation on the first frame and the second frame to generate an interpolated frame; for each of a plurality of areas of the interpolated frame, determining whether there is a block in the first frame that moves to the area, and determine whether there is a block in the second frame that moves to the area, to determine whether the area belongs to a cover area or an uncover area; and in response to a determination result indicating that the area belongs to the cover area or the uncover area, adjusting image contents of the interpolated frame. 
     According to one embodiment of the present invention, an image processing circuit comprising a receiving circuit and a MEMC circuit is disclosed. The receiving circuit is configured to receive an image signal, wherein the image signal comprises a first frame and a second frame. The MEMC circuit is configured to perform motion estimation on the first frame and the second frame to generate an interpolated frame; and for each of a plurality of areas of the interpolated frame, determine whether there is a block in the first frame that moves to the area, and determine whether there is a block in the second frame that moves to the area, to determine whether the area belongs to a cover area or an uncover area; and in response to a determination result indicating that the area belongs to the cover area or the uncover area, adjust image contents of the interpolated frame. 
     These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a diagram illustrating an image processing circuit according to one embodiment of the present invention. 
         FIG.  2    is a diagram illustrating modification of motion vectors of an outer region of an edge of a foreground in an interpolated frame according to an embodiment of the present invention. 
         FIG.  3    is a flowchart illustrating an image processing method according to one embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
       FIG.  1    is a diagram illustrating an image processing circuit  100  according to one embodiment of the present invention. As shown in  FIG.  1   , the image processing circuit  100  comprises a receiving circuit  110  and a motion estimation and motion compensation (MEMC) circuit  120 . In this embodiment, the image processing circuit  100  may perform operations of increasing frame rates, for example, the image processing circuit  100  receives an image signal Din and generates a plurality of interpolated frames according to a plurality of frames of the image signal Din, where the plurality of frames and the plurality of interpolated frames are transmitted to a display panel  104  through a backend processing circuit  102  for displaying. 
     Regarding detailed operations of the image processing circuit  100 , the receiving circuit  110  receives the image signal Din, where the image signal Din comprises a first frame T 1  and a second frame T 2  shown in  FIG.  2   . Then, the MEMC circuit  120  performs motion estimation on the contents of the first frame T 1  and the second frame T 2  to determine a moving speed of each block, and performs motion compensation operation to generate the interpolated frame T_I and related information (e.g., which block of the first frame T 1  and/or the second frame T 2  is moved to the block of the interpolated frame T_ 1 ). Because a person skilled in the art should understand operations of generating the interpolated frame T_I, and the focus of the present invention is not on the part of generating the interpolated frame T_I, so the details are omitted here. In addition, since a foreground object in the first frame T 1  and the second frame T 2  may move relative to the background, a conventional motion estimation manner may make motion vector(s) of a junction of the foreground object and the background in the interpolated frame T_I inaccurate, thereby causing the halo effect mentioned in the related art. Therefore, after the motion vector(s), moving speed(s) and related image information of the interpolated frame T_I are generated in this embodiment, the MEMC circuit  120  further determines if an area near the edge of the interpolated frame T_I is in the cover area or the uncover area, and then performs motion vector or moving speed modification on the area located in the cover area and the uncover area, to prevent the first frame T 1 , the interpolated frame T_I and the second frame T 2  from the halo effect when being displayed on the display panel  104 . 
     Specifically, the MEMC circuit  120  may determine edges of the foreground in the first frame T 1  and the second frame T 2  according to at least one motion vector of the first frame T 1  and the second frame T 2 , such as the foreground edge P 1  in the first frame T 1  and the foreground edge P 2  in the second frame T 2  shown in  FIG.  2   . In one embodiment, the foreground shown in  FIG.  2    may comprise at least one block, wherein a size of each block can be determined according to the designer&#39;s consideration, for example, the size of each block can be 8*8 pixels, 16*16 pixels, etc. Then, the MEMC circuit  120  refers to the foreground edge P 1  in the first frame T 1  and the foreground edge P 2  in the second frame T 2 , or the moving speed of the block corresponding to the foreground of the first frame T 1  and the second frame T 2  to determine a range near the foreground edge of the interpolated frame T_I. For example, the MEMC circuit  120  may calculate a center point of the foreground edges P 1  and P 2  as the foreground edge of the interpolated frame T_ 1 , and then select a range according to the foreground edge of the interpolated frame T_ 1 . In one embodiment, the determined range near the foreground edge of the interpolated frame T_ 1  includes an area larger than an area from the foreground edge P 1  of the first frame T 1  to the edge P 2  of the second frame T 2 , for example, the range between BC 1  and BB 1  shown in  FIG.  2   . 
     Then, for a plurality of areas within a range near the foreground edge of the interpolated frame T_ 1 , the MEMC circuit  120  determines whether each area is located in the cover area or the uncover area, wherein the cover area or the uncover area is an area between the foreground edges of the first frame T 1  and the second frame T 2 . For example, the cover area is the area between the foreground edge P 1  and the foreground edge P 2 . Taking the left side of the foreground in  FIG.  2    as an illustration, it is assumed that the interpolated frame T_I includes areas A, B and C, where the size of each of the areas A, B and C can be equal to one block of the first frame T 1  or any other suitable size, the MEMC circuit  120  determines whether there is a block in the first frame T 1  and whether there is a block in the second frame T 2  that moves to the areas A, B, and C of the interpolated frame T_I, so as to determine whether the areas A, B and C belong to the cover area or the uncover area. Specifically, for any one of the areas A, B and C, if only one of the first frame T 1  and the second frame T 2  has a block that moves to the area of the interpolated frame T_ 1 , it is determined that the area belongs to the cover area or the uncover area. If both the first frame T 1  and the second frame T 2  have blocks moving to this area of the interpolated frame T_ 1 , it is determined that this area does not belong to the cover area, and also does not belong to the uncover area. In detail, for the area A, because the area A is mainly moved by a block BA 1  of the first frame T 1  (that is, most of the pixel values of the area A are generated according to the block BA 1 ) , and the second frame T 2  does not have any block that moves to the area A, the MEMC circuit  120  determines that the area A belongs to the cover area. For the area B, because both the first frame T 1  and the second frame T 2  have blocks that move to the area B (for example, most of the pixel values of the area B are generated according to the blocks BB 1  and BB 2 ) , the MEMC circuit  120  determines that the area B does not belong to the cover area or the uncover area. For the area C, because both the first frame T 1  and the second frame T 2  have blocks that move to the area C (for example, most of the pixel values of the area C are generated according to the blocks BC 1  and BC 2 ), the MEMC circuit  120  determines that the area C does not belong to the cover area or the uncover area. 
     Similarly, the right side of the foreground of the first frame T 1  and the second frame T 2  has a foreground edge P 1 ′ and a foreground edge P 2 ′ respectively. The MEMC circuit  120  may calculate a center point of the foreground edges P 1 ′ and P 2 ′ as the foreground edge of the interpolated frame T_ 1 , and then select a range according to the foreground edge of the interpolated frame T_ 1 . In one embodiment, the determined range near the foreground edge of the interpolated frame T_ 1  includes an area larger than an area from the foreground edge P 1 ′ of the first frame T 1  to the edge P 2 ′ of the second frame T 2 , for example, the range between BB 2 ′ and BC 2 ′ shown in  FIG.  2    . Taking the right side of the foreground in  FIG.  2    as an illustration, it is assumed that the interpolated frame T_I includes areas A′, B′ and C′, where the size of each of the areas A′, B′ and C′ can be equal to one block of the first frame T 1  or any other suitable size, the MEMC circuit  120  determines whether there is a block in the first frame T 1  and whether there is a block in the second frame T 2  that moves to the areas A′, B′, and C′ of the interpolated frame T_I, so as to determine whether the areas A′, B′ and C′ belong to the cover area or the uncover area. Specifically, for any one of the areas A′ , B′ and C′, if only one of the first frame T 1  and the second frame T 2  has a block that moves to the area of the interpolated frame T_ 1 , it is determined that the area belongs to the cover area or the uncover area. If both the first frame T 1  and the second frame T 2  have blocks moving to this area of the interpolated frame T_ 1 , it is determined that this area does not belong to the cover area, and also does not belong to the uncover area. In detail, for the area A′, because the area A′ is mainly moved by a block BA 2 ′ of the second frame T 2  (that is, most of the pixel values of the area A′ are generated according to the block BA 2 ′), and the first frame T 1  does not have any block that moves to the area A′, the MEMC circuit  120  determines that the area A′ belongs to the uncover area. For the area B′, because both the first frame T 1  and the second frame T 2  have blocks that move to the area B′ (for example, most of the pixel values of the area B′ are generated according to the blocks BB 1 ′ and BB 2 ′) , the MEMC circuit  120  determines that the area B′ does not belong to the cover area or the uncover area. For the area C′, because both the first frame T 1  and the second frame T 2  have blocks that move to the area C′ (for example, most of the pixel values of the area C′ are generated according to the blocks BC 1 ′ and BC 2 ′), the MEMC circuit  120  determines that the area C′ does not belong to the cover area or the uncover area. 
     After determining whether a plurality of areas near the foreground edge of the interpolated frame T_ 1  belong to the cover area or the uncover area, the MEMC circuit  120  performs image adjustment on the area belonging to the cover area or the uncover area. For example, since the area A is determined to belong to the cover area, the MEMC circuit  120  can change the previously determined motion vector or the moving speed of the area A of the interpolated frame T_ 1  according to the motion vector or the moving speed of the block BA 1 , to improve image quality near the foreground edge of interpolated frame T_I. For example, assuming that the block BA 1  of the first frame T 1  has a motion vector MV 1  (i.e., the amount of movement of the block BA 1  relative to a block of a reference frame), the motion vector of the area A can be directly modified to be the motion vector MV 1  (i.e., the amount of movement of the block relative to the reference frame). Similarly, since the area A′ is determined to belong to the uncover area, the MEMC circuit  120  can change the previously determined motion vector or the moving speed of the area A′ of the interpolated frame T_ 1  according to the motion vector or the moving speed of the block BA 2 ′, to improve image quality near the foreground edge of interpolated frame T_I. For example, assuming that the block BA 2 ′ of the second frame T 2  has a motion vector MV 2  (i.e., the amount of movement of the block BA 2 ′ relative to a block of a reference frame) , the motion vector of the area A′ can be directly modified to be the motion vector MV 2  (i.e., the amount of movement of the block relative to the reference frame). In addition, since the areas B, C, B′ and C′ do not belong to the cover area or the uncover area, the MEMC circuit  120  will not additionally adjust the motion vectors and moving speeds of these areas, that is, the areas B, C, B′ and C′ maintain the previously determined motion vectors and moving speeds. 
     In light of above, after the MEMC circuit  120  determines the interpolated frame T_I, the MEMC circuit  120  will further determine whether the areas near the foreground edge of the interpolated frame T_I belong to the cover area or the uncover area, so as to determine whether to perform image content adjustment on the plurality of areas, so this embodiment can effectively improve the image quality near the foreground edge of the interpolated frame T_I, and avoid the halo effect mentioned in the prior art 
       FIG.  3    is a flowchart illustrating an image processing method according to one embodiment of the present invention. Referring to  FIG.  1    to  FIG.  3    and the above embodiments, the flow is described as follows. 
     Step  300 : the flow starts. 
     Step  302 : receive an image signal, wherein the image signal comprises a first frame and a second frame. 
     Step  304 : perform the motion estimation upon the first frame and the second frame to generate an interpolated frame. 
     Step  306 : determine a foreground edge of the first frame and a foreground edge of the second frame according to at least one motion vector of the first frame and the second frame. 
     Step  308 : determine a plurality of areas near a foreground edge of the interpolated frame according to the foreground edge of the first frame and the foreground edge of the second frame. 
     Step  310 : for each area of the plurality of areas of the interpolated frame, determine whether there is a block in the first frame that moves to the area, and determine whether there is a block in the second frame that moves to the area, to determine whether the area belongs to a cover area or an uncover area. 
     Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.