Patent Publication Number: US-2022237746-A1

Title: Image processing method and image processing device using the same

Description:
This application claims the benefit of People&#39;s Republic of China application Serial No. 202110087876.4, filed on Jan. 22, 2021, the subject matter of which is incorporated herein by reference. 
     BACKGROUND OF THE INVENTION 
     Field of the Invention 
     The invention relates in general to a processing method and a processing device using the same, and more particularly to an image processing method and an image processing device using the same. 
     Description of the Related Art 
     The captured frame is generally affected by changes in the environment, the camera itself and the motion of the captured object, and thus there is the pixel information difference between the captured frame and the previous frame, wherein the pixel information difference is caused by noise or the motion of the captured object. In order to obtain a clearer frame, noise reduction is generally performed on the captured frame. However, when the noise reduction effect is strong, the noise could be reduced together with the pixel information corresponding to the action of the captured object, resulting in a blurry frame after the noise reduction. When the noise reduction effect is weak, the noise could not be effectively reduced. Therefore, the industry believes that it is necessary to study a new image processing method and an image processing device using the same to improve the aforementioned conventional problems. 
     SUMMARY OF THE INVENTION 
     The invention is directed to an image processing method and an image processing device using the same capable of resolving the problems disclosed above. 
     According to an embodiment of the present invention, an image processing method is provided. The image processing method includes the following steps: obtaining a block-size value; dividing a first frame into a plurality of first blocks according to the block-size value; dividing a second frame into a plurality of second blocks according to the block-size value; obtaining a noise reduction intensity array according to a first pixel information of each first block and a second pixel information of each second block; and obtaining an output frame according to the noise reduction intensity array, the first frame and the second frame. 
     The image processing device includes a block-size value obtainer and a noise reducer. The block-size value obtainer is configured to obtain a block-size value. The noise reducer is configured to: divide a first frame into a plurality of first blocks according to the block-size value; divide a second frame into a plurality of second blocks according to the block-size value; obtain a noise reduction intensity array according to a first pixel information of each first block and a second pixel information of each second block; and obtain an output frame according to the noise reduction intensity array, the first frame and the second frame. 
     The above and other aspects of the invention will become better understood with regard to the following detailed description of the preferred but non-limiting embodiment (s). The following description is made with reference to the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows a functional block diagram of an image processing device according to an embodiment of the present invention; 
         FIG. 2  shows a flowchart of an image processing method of the image processing device of  FIG. 1 ; 
         FIG. 3  shows a functional block diagram of the image processing function of the image processing device of  FIG. 1 ; 
         FIG. 4  shows a schematic diagram of a first frame of  FIG. 3 ; and 
         FIG. 5  shows a functional block diagram of the image processing of the image processing device according to another embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Detailed descriptions of the invention are disclosed below with embodiments accompanying drawings. However, the descriptions are for exemplary purpose only, not for limiting the scope of protection of the present invention. 
     Referring to  FIGS. 1 to 4 ,  FIG. 1  shows a functional block diagram of an image processing device  100  according to an embodiment of the present invention,  FIG. 2  shows a flowchart of an image processing method of the image processing device  100  of  FIG. 1 ,  FIG. 3  shows a functional block diagram of the image processing function of the image processing device  100  of  FIG. 1 , and  FIG. 4  shows a schematic diagram of a first frame (or picture, image) F P  of  FIG. 3 . 
     The image processing device  100  includes a block-size value obtainer  110  and a noise reducer  120 . The image processing device  100  is, for example, various electronic devices capable of image processing, such as a camera, a notebook computer, a desktop computer, a tablet computer and a communication device (for example, a mobile phone). 
     The block-size value obtainer  110  and the noise reducer  120  could be, for example, a physical circuit formed by a semiconductor manufacturing process, wherein the physical circuit is such as a semiconductor chip, a semiconductor package, etc. In addition, the block-size value obtainer  110  and the noise reducer  120  could be integrated into a single component, or the block-size value obtainer  110  and/or the noise reducer  120  could be integrated into a processor or a controller. 
     In an embodiment, the block-size value obtainer  110  is configured to: (a) obtain a block-size value BW. The noise reducer  120  is configured to: (b) divide the first frame F P  into a plurality of first blocks A P,n×m  according to the block-size value BW; (c) divide the second frame F C  into a plurality of second blocks A C,n×m  according to the block-size value BW; (d) obtain a noise reduction intensity array TNR according to a first pixel information B P,n×m  of each first block A P,n×m  and a second pixel information B C,n×m  of each second block A C,n×m;  and (e) obtain an output frame F O  according to the noise reduction intensity array TNR, the first frame F P  and the second frame F C . As a result, compared with the conventional noise reduction method for each pixel, the embodiment of the present invention divides the frame into multiple blocks and processes them individually, and accordingly it could improve the noise reduction capability and the noise reduction effect. The larger the block-size value BW is, the higher the noise reduction capability is. 
     The following is a further description of the process of the image processing method (or noise reduction method) with reference to  FIG. 2 . 
     In step S 110 , the block-size value obtainer  110  obtains the block-size value BW. For example, the block-size value obtainer  110  could use the following formula (1) to obtain the block-size value BW. In formula (1), α represents an adjustment value for noise floor, wherein the adjustment value could be any real number equal to or greater than 1, σ represents a noise intensity value, and d C  represents a noise judgment critical value. 
         BW ≥(α×σ)/ d   c    (1)
 
     The noise intensity value σ is, for example, a standard deviation, and the value of the noise intensity value σ could depend on the characteristics of the camera that captures (photographs) the first frame F P  and the second frame F C  and/or the capturing environment (for example, ambient light). The aforementioned camera could be the image processing device  100 . The noise intensity value a could be obtained by measuring the characteristics of the image processing device  100 . However, as long as the noise intensity value σ could be obtained, the embodiment of the present invention does not limit the method of measuring the characteristics of the image processing device  100 . 
     The noise judgment critical value d C  is, for example, the critical value at which the change of the frame pixel information belongs to (or tend to) to noise or the pixel dynamic change. The term “pixel dynamic change” is, for example, caused by the motion of the captured object. In an embodiment, the noise judgment critical value d C  could be a pixel information difference value. In case of the pixel information being the brightness which is set to 16, when the difference between the pixel brightness of the first frame F P  and the pixel brightness of the second frame F C  exceeds 16 levels (the gray scale value is, for example, a total of 255 levels), it means it is possible that the pixel information difference of the second frame F C  could be caused by the pixel dynamic change. When the difference between the pixel brightness of the first frame F P  and the pixel brightness of the second frame F C  does not exceed 16 levels, it means it is possible that the pixel information difference of the second frame F C  could be caused by noise. The embodiment of the present invention does not limit the noise judgment critical value d C , and it could depend on the captured object and/or the capturing environment. 
     The adjustment value α, the noise intensity value σ and the noise judgment critical value d C  could be obtained or set in advance before executing the image processing procedure. When the capturing environment and/or camera remain unchanged (for example, maintain the original state), the adjustment value, the adjustment value α, the noise intensity value σ and the noise judgment critical value d C  could remain unchanged (for example, maintain the original state). 
     In step S 120 , the noise reducer  120  divides the first frame F P  into N×M first blocks A P,n×m  according to the block-size value BW, wherein the subscript n is a positive integer ranging between 1 and N, and the subscript m is between a positive integer ranging between 1 to M. The resolution of each first block A P,n×m  is BW×BW. 
     As shown in  FIG. 4 , in case of BW being equal to 8 (that is, the size (or resolution) of each first block A P,n×m  is 8×8) and the resolution of the first frame F P  is 1920×1080, the noise reducer  120  determines the number of the first blocks A P,n×m  according to the following formulas (2) and (3), wherein R and Q represent the resolution (R×Q) of the first frame F P . Accordingly, the obtained N is equal to 240 (calculation formula: 1920/8=240), and M is equal to 135 (calculation formula: 1080/8=135). 
         N=R/BW    (2)
 
         M=Q/BW    (3)
 
     In step S 130 , the noise reducer  120  obtains a first average brightness value of a number of first pixels (for example, BW pixels) of each first block A P,n×m  and uses the first average brightness value as the first pixel information B P,n×m . In other words, each first block A P,n×m  uses the first pixel information B P,n×m  to represent the block information of the entire first block A P,n×m . 
     In step S 140 , the noise reducer  120  divides the second frame F C  into N×M second blocks A C,n×m  according to the block-size value BW. In case of BW being equal to 8 (that is, the size (or resolution) of each second block A C,n×m  is 8×8) and the resolution of the second frame F C  is 1920×1080, the noise reducer  120  determines the number of the second blocks A C,n×m  according to the following formulas (2) and (3), wherein the resolution of the second frame F C  is R×Q. 
     The first frame F P  is, for example, a previous frame, and the second frame F C  is, for example, a current frame. The previous frame could be a frame that has been processed for noise reduction or a frame that has not been processed for noise reduction. The image processing method of the embodiment of the present invention for the previous frame and the next frame belong to a time-domain noise reduction method. 
     In step S 150 , the noise reducer  120  obtains the second average brightness value of a number of second pixels of each second block A C,n×m , and uses the second average brightness value as the second pixel information B C,n×m . In other words, each second block A C,n×m  uses the second pixel information B C,n×m  to represent the block information of the entire second block A C,n×m . 
     In step S 160 , the noise reducer  120  could obtain the noise reduction intensity array TNR according to the first pixel information B P,n×m  of each first block A P,n×m  and the second pixel information B C,n×m  of each second block A C,n×m . 
     For example, as shown in  FIG. 3 , the noise reducer  120  obtains a pixel information difference value d n×m  between the first pixel information B P,n×m  of each first block A P,n×m  and the second pixel information B C,n×m  of the corresponding second block A C,n×m  and uses the pixel information difference value d n×m  as a number of noise reduction intensity values T r×q  of the noise reduction intensity array TNR, wherein the subscript r is a positive integer ranging between 1 and R, and the subscript q is a positive integer ranging between 1 and Q. 
     In an embodiment, as shown in  FIG. 3 , the noise reducer  120  could perform low-pass filtering process and/or signal enhancement process on each pixel information difference value d n×m  to obtain the pixel information difference value d′ n×m . In terms of the low-pass filtering process, the noise reducer  120  could filter out a part that belongs to the noise (low frequency) of the pixel information difference value d n×m . In terms of signal enhancement process, the noise reducer  120  could enhance a part that belong to the signal of the pixel information difference value d n×m , and the signal part is the part other than the noise part of the pixel information difference value d n×m . After processing, the noise reducer  120  generates N×M pixel information difference values d′ n×m . 
     In an embodiment, as shown in  FIG. 3 , the noise reducer  120  could perform an amplification process on the pixel information difference value d′ n×m  to obtain the noise reduction intensity array TNR having the same size (or resolution) as the first frame F P  or the second frame F C . For example, in case of the resolution being 1920×1080, the noise reducer  120  obtains R×Q noise reduction intensity values T r×q  according to at least one of N×M pixel information difference value d′ n×m , and uses R×Q noise reduction intensity values T r×q  to form the noise reduction intensity array TNR. 
     In step S 170 , the noise reducer  120  obtains the output frame F O  according to the noise reduction intensity array TNR, the first frame F P  and the second frame F C . For example, the noise reducer  120  obtains the output frame F O  according to the following formula (4). 
         B   O,r×q =(1− T   r×q )× B   P,r×q +( T   r×q   ×B   C,r×q )   (4)
 
     In formula (4), T r×q  represents the (r×q)-th noise reduction intensity value of the noise reduction intensity array TNR, B P,r×q  represents the pixel information of the (r×q)-th pixel of the first frame FP, and B C,r×q  represents the pixel information of the (r×q)-th pixel of the second frame F C . The noise reduction intensity value T r×q , pixel information B P,r×q , and pixel information B C,r×q  are known (or given) information, so the noise reducer  120  could obtain the pixel information B O,r×q  of the (r×q)-th pixel of the output frame F O  according to formula (4). 
     The noise reduction intensity value T r×q  is an arbitrary real number ranging between 0 and 1. It could be seen from formula (4) that the larger the noise reduction intensity values T r×q  is, the higher the possibility (degree) that the pixel information change belongs to the dynamic change is, and the pixel information B O,r×q  of the (r×q)-th pixel of the output frame F O  is closer to the pixel information B C,r×q  of the second frame F C  (that is, the lower the degree of noise reduction). On the contrary, the less the noise reduction intensity value T r×q  is, the lower the possibility (degree) that the pixel information change belongs to the dynamic change is, and the pixel information B O,r×q  of the (r×q)-th pixel of the output frame F O  is closer to the pixel information B P,r×q  of the first frame F P  (that is, the larger the degree of noise reduction). 
     When all (R×Q) pixel information B O,r×q  is obtained, the output frame F O  is completed. 
     Then, the noise reducer  120  outputs the output frame F O  to a display (not shown) to display the output frame F O . Alternatively, the noise reducer  120  or the controller replaces the second frame F C  with the output frame F O . When the image processing device  100  performs the noise reduction processing on another frame (for example, the next frame or the previously processed output frame F O ), the first frame F P  of  FIG. 2  could be substituted (or replaced) with the output frame F O  (processed by the noise reduction), and the second frame F C  could be substituted (or replaced) with the said another frame. In another embodiment, the first frame F P  of  FIG. 2  could be substituted (or replaced) with the second frame F C  (without noise reduction processing), and the second frame of  FIG. 2  could be substituted (or replaced) the said another frame. 
     The image processing device  100  could process a streaming video according to the aforementioned process, and the streaming video includes several frames, wherein in any two adjacent frames, the current frame could substitute into the second frame F C  of  FIG. 2 , and the previous frame could substitute into the first frame F P  of  FIG. 2 , and the previous frame could be the frame processed by the same noise reduction process or the frame without the noise reduction process. 
     Referring to  FIG. 5 ,  FIG. 5  shows a functional block diagram of the image processing of the image processing device  100  according to another embodiment of the present invention. The image processing device  100  could perform the noise reduction processing for the second frame F C  according to the several noise reduction intensity arrays TNR ,t  obtained by several different block-size values BW. The functional block in each dashed frame of  FIG. 5  corresponds to the functional block in the dashed frame R 1  of  FIG. 3 . 
     For example, as shown in  FIG. 5 , the image processing device  100  executes the same image processing procedure as described above with the block-size value BW which is equal to 1 to obtain the corresponding noise reduction intensity array TNR ,1 . The image processing device  100  executes the same image processing procedure as described above with the block-size value BW which is equal to 2 to obtain the corresponding noise reduction intensity array TNR ,2 . The image processing device  100  executes the same image processing procedure as described above with the block-size value BW which is equal to t to obtain the corresponding noise reduction intensity array TNR ,t , wherein the subscript “t” is, for example, arbitrary positive integer. The noise reducer  120  could obtain a comprehensive noise reduction intensity array TNR ,T  according to several noise reduction intensity arrays TNR ,t . For example, the noise reducer  120  adds several noise reduction intensity arrays TNR ,t  to obtain the comprehensive noise reduction intensity array TNR ,T . Then, the noise reducer  120  obtains the output frame F O , by using the above formula (4), according to the comprehensive noise reduction intensity array TNR ,T , the first frame F P  and the second frame F C . In another embodiment, the method for obtaining the comprehensive noise reduction intensity array is not limited to the addition of several noise reduction intensity arrays, and other mathematical operations (such as addition, subtraction, multiplication, division, averaging or a combination thereof) or equations (formulas) could also be used to obtain the comprehensive noise reduction intensity array TNR ,T . In addition, the embodiment of the present invention does not limit the total number of the noise reduction intensity array TNR ,t , and it could be any positive integer equal to or greater than two. In addition, the block-size value BW in  FIG. 5  is not limited by the above formula (1). In other words, each block-size value BW in  FIG. 5  could be greater than, equal to, or less than the value of (α×σ)/d C . 
     While the invention has been described by way of example and in terms of the preferred embodiment (s), it is to be understood that the invention is not limited thereto. On the contrary, it is intended to cover various modifications and similar arrangements and procedures, and the scope of the appended claims therefore should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements and procedures.