Abstract:
An image signal processing method includes: receiving an original color filter array (CFA) image and a pixel binned CFA image; computing a specific information of the pixel binned CFA image; and processing the original CFA image according to the specific information. The associated image signal processor includes an input terminal, an operating unit and a processing unit, wherein the input terminal is for receiving an original CFA image and a pixel binned CFA image, the operating unit is for computing a specific information of the pixel binned CFA image, and the processing unit is for processing the original CFA image according to the specific information and utilizing the pixel binned CFA image.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The embodiments disclosed by this present invention relate to pixel binning of image signal processing, and more particularly, to an image signal processing method which refers to both an original color filter array image and a pixel binned color filter array image, and an associated device. 
     2. Description of the Prior Art 
     In image signal processing, the image quality of a color filter array (CFA) image is affected by photon noise, dark current noise and readout noise. To eliminate/reduce noise in the CFA image, often at least one de-noising process is executed in a Bayer domain or some other color space. 
     U.S. Pat. No. 8,295,631B2 discloses a pixel value technique in the Bayer color filter array which uses weighted means to adjust luminance and chrominance values according to the local edge response to eliminate noise. In a situation where the noise floor is unknown and line buffer resources are limited, the effects of the de-noising method are suppressed and may experience the side effects of distortion or vague images. 
     U.S. Pat. No. RE44482E1 discloses an active CMOS comprising a selection circuitry for selecting saved charge to achieve pixel binning, which can reduce the readout noise and the required exposure time. As the pixels are binned, however, the resolution of the image will be affected. A novel technique is therefore needed to overcome the problems of the prior art. 
     SUMMARY OF THE INVENTION 
     An image signal processing method which refers to both an original CFA image and a pixel binned CFA image and an associated device are disclosed to solve the abovementioned problems. 
     According to a first exemplary embodiment of the present invention, an image signal processing method is disclosed which comprises: receiving an original CFA image and a pixel binned CFA image; using an operation unit to compute a specific information of the pixel binned CFA image; and processing the original CFA image according to the specific information. 
     According to a second exemplary embodiment, an image signal processor is disclosed which comprises an input terminal, an operation unit, and a processing unit, wherein the input terminal is arranged to receive the original CFA image and the pixel binned CFA image, the operation unit is arranged to compute a specific information of the pixel binned CFA image, and the processing unit is arranged to process the original CFA image according to the specific information. 
     The spirit of the present invention utilizes the advantages of the pixel binning technique to improve the flow of digital image signal processing (ISP) and reduce the noise of the output image without losing resolution. The present invention can thereby enhance the visual experience and simultaneously maintain or deduce the noise level of visual perception via making it easier to identify texture of the image. The disclosed technique can be combined with the current ISP architecture rather than discarding the original architecture, so the design cost is saved. 
     These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a diagram illustrating an image signal processor in an image processing system of the present invention. 
         FIG. 2  is a diagram illustrating a pixel binning operation. 
         FIG. 3  is a diagram illustrating a first embodiment of the image signal processor of the present invention. 
         FIG. 4  is a flowchart illustrating a Bayer domain de-noising method of the present invention. 
         FIG. 5  is a flowchart illustrating a method of color interpolation of the present invention. 
         FIG. 6  is a diagram illustrating a second embodiment of the image signal processor of the present invention. 
         FIG. 7  is a flowchart illustrating a method of enhancing the edge of a YUV image according to the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     Certain terms are used throughout the description and following claims to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should not be interpreted as a close-ended term such as “consist of”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is coupled to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections. 
     As is well-known, under the cover of the CFA, each pixel of a charge coupled device only records a single color. The analog signal will be affected by interference due to noise (e.g. the readout noise) before it is output to the ISP. Errors in the ISP may occur, generating distortion which is observable to the human eye, especially for images shot in a low-light environment where the noise level is very high. 
       FIG. 1  is a diagram illustrating an image signal processor in an image processing system  100  according to an exemplary embodiment of the present invention. In the image processing system  100 , an image sensing unit  102  outputs an original CFA image I and a pixel binned CFA image, wherein the pixel binned CFA image and the original CFA image I display the same view/image content with different resolution. In practice, the image sensing unit  102  may only provide the original CFA image I and further compute the pixel binned CFA image. As the size of the pixel binned CFA image is smaller than the original CFA image I, a color interpolation unit  104  performs an interpolation processing on the pixel binned CFA image to restore its size to that of the original CFA image I. The color interpolation unit  104  can be any kind of interpolation algorithm, e.g. bilinear interpolation. The color interpolation unit  104  outputs a CFA image I B  which is outputted with the original CFA image I to an image signal processor  106 . It should be noted that the color interpolation unit  104  can be merged into the image sensing unit  102  and the image signal processor  106 , i.e. any substantially identical device falls within the scope of the present invention. The main technical feature of the present invention is the image signal processor  106  which will be described in detail later. 
       FIG. 2  is a diagram illustrating an example of the abovementioned pixel binning operation. For the example of the Bayer pattern CFA, each 2*2 pixel can be regarded as the basic Bayer pattern unit. Each basic Bayer pattern unit comprises four color channels. As shown in  FIG. 2A , when a pixel binning operation is executed on the original CFA, the induced charges of the four identical color channels (e.g. four Grs) of the 4*4 pixel is combined as a super pixel to generate the pixel binned CFA shown in  FIG. 2B . The overall resolution is reduced to W/2×H/2 from W×H. However, as the number of times of readout and the scale of readout noise interference for each pixel decrease, the Signal-to-Noise Ratio (SNR) can be improved and thus the required exposure time can be shortened. That is, although the resolution of the pixel binned CFA is lower than the original CFA, the noise level is lower as well. The present invention utilizes the advantages of higher resolution from the original CFA and the lower noise level from the pixel binned CFA in ISP pipeline. 
       FIG. 3  is a diagram illustrating a first embodiment of the image signal processor of the present invention. The image signal processor  300  generates the output image I O  according to the original CFA image I and the CFA image I B . A preprocessing unit  302  executes processes comprising Lens Shading Correction and Auto White Balance on the original CFA image I and the CFA image I B . The post-processing unit  312  comprises Color Calibration, Gamma Correction, GbGr Unbalance Correction and Dead Pixel compensation. The architecture is not limited in the present invention. The pre-processing unit  302  and the post-processing unit  312  are not necessary elements and their related technical details are omitted here. An after pre-processed original CFA image I pre  and an after pre-processed CFA image I Bpre  are outputted by the pre-processing unit  302  according to the original CFA image I and the CFA image I B . The essence of the present invention is to refer to the after pre-processed CFA image I Bpre  to help a Bayer de-noising unit  304 , a color interpolation unit  306  and a RGB domain de-noising unit  310  perform corresponding processes, in which the obtained results are better than the results would be without referring to the after pre-processed CFA image I Bpre . The related details of the Bayer de-noising unit  304 , the color interpolation unit  306  and the RGB domain de-noising unit  310  are described later. 
       FIG. 4  is a flowchart illustrating a Bayer domain de-noising method of the present invention. If the same result can be obtained, the steps of the flow in  FIG. 4  do not need to be followed step by step, and do not need to be executed consecutively, i.e. other steps can be inserted. In addition, some steps in  FIG. 4  can be omitted according to different embodiments or design requirements. When the Bayer de-noising method  400  is applied in the Bayer de-noising unit  304 , the noise of the after pre-processed original CFA image I pre  is reduced. The pixel content of the after pre-processed original CFA image I pre  can be divided into an edge pixel part and a non-edge pixel part. The first step of the method is judging whether each pixel of the after pre-processed original CFA image I pre  is an edge pixel part or a non-edge pixel part. As the after pre-processed CFA image I Bpre  has lower noise, i.e. higher credibility, in step  404 , the Bayer de-noising method  400  uses the after pre-processed CFA image I Bpre  as a reference for calculating an edge information. In step  406 , it can be determined whether each pixel is an edge pixel or non-edge pixel according to the pixel information computed in step  404 . It should be noted that the mechanism for determining a pixel as edge pixel can be performed based on a portion of the pixel or the entire image. 
     The after pre-processed original CFA image I pre  preserves the edge more completely; therefore, for the edge pixel, step  408  inputs the after pre-processed original CFA image I pre  into a noise filter in the Bayer de-noising unit  304  (which is not shown in  FIG. 3 ) to de-noise I pre . The specific de-noising method, for example, median filtering, can be performed on adjacent pixels with the same color channel according to the estimated direction of the edge; however, this is not limited. For the non-edge pixel, step  410  inputs the after pre-processed CFA image I Bpre  into the noise filter in the Bayer de-noising unit  304  to de-noise I pre  accordingly to reduce the noise in non-edge region to which human eyes are more sensitive. In addition, step  404  in this embodiment can be changed to compute the scale of noise interference or the scale of texture complexity. The following steps can be amended accordingly. 
     For the color interpolation unit  306 , a method similar to the Bayer de-noising method  400  can be utilized. In the original CFA image I outputted by the sensor, each pixel only records the color channel information of either red, green, or blue, so color interpolation is needed to complement the information of the other two color channels that each pixel lacks.  FIG. 5  is a flowchart illustrating a method of color interpolation according to the present invention. If the same result can be obtained, the steps of the flow in  FIG. 5  do not need to be followed step by step, and do not need to be executed consecutively, i.e. other steps can be inserted. In addition, some steps in  FIG. 5  can be omitted according to different embodiments or design requirements. The color interpolation method  500  can be applied in the color interpolation unit  306  in  FIG. 3 . In step  504 , the after pre-processed CFA image I Bpre  is referred to for computing an edge information of the pixel content outputted after the Bayer de-noising unit  304 . In step  506 , the edge information is judged to determine if a pixel has a specific direction (e.g. horizontal or vertical). If yes, then the flow moves to step  508  to input the pixel content outputted after the Bayer de-noising unit  304  into the color interpolation unit  306  to perform a color interpolation process which conserves directionality of the edge. Otherwise, the pixel is considered in a smooth region, and the flow moves to step  510  to input the after pre-processed CFA image I Bpre  into the color interpolation  306  to perform color interpolation which does not conserve directionality of the edge to reduce the noises. In addition, step  504  in this embodiment can be changed to compute the scale of noise interference or the scale of texture complexity. The steps following step  504  can be amended accordingly. 
     For the RGB domain de-noising unit  310  in  FIG. 3 , a method similar to the Bayer de-noising method  400  can be applied to de-noise the output of a color interpolation unit  306  utilizing the output of a color interpolation unit  308 . As a skilled person in the art would be able to implement a RGB domain de-noising unit  310  which embodies the principles of the disclosure. 
       FIG. 6  is a diagram illustrating a second embodiment of the image signal processor of the present invention. As in the image signal processor  300 , the image signal processor  600  generates the output image I O  according to the original CFA image I and the CFA image I B . However, in the image signal processor  600 , the color space conversion units  614  and  616  are used to convert an image from Bayer domain to YUV domain, and output an original YUV image I YUV  and a binnned YUV image I BYUV . After that, a YUV edge enhancement unit  610  enhances the edges of the original YUV image I YUV  by referencing the binned YUV image I BYUV  then outputs the result to the post-processing unit  312 . 
       FIG. 7  is a flowchart illustrating the method of enhancing edges of YUV of the present invention. If the same result can be obtained, the steps of the flow in  FIG. 7  do not need to be followed step by step, and do not need to be executed consecutively, i.e. other steps can be inserted. In addition, some steps in  FIG. 7  can be omitted according to different embodiments or design requirements. The YUV edge enhancement method  700  can be applied in the YUV edge enhancement unit  610  in  FIG. 6 . In step  704 , an edge information of the Y channel of the original YUV image I YUV  is computed by referring to the binned YUV image I BYUV . In step  706 , it is determined if the corresponding pixel is an edge area according to the edge information. If yes, the flow moves to step  708  to decide if the edge area that the corresponding pixel belongs to is in a bright side (i.e. where the luminance is higher). If yes, the YUV edge enhancement unit  610  performs a process of enhancing bright side contrast of the original YUV image I YUV , e.g. increasing the luminance of the original YUV image I YUV ; otherwise (i.e. the luminance is lower and the pixel is in a dark side), the YUV edge enhancement unit  610  performs a process of enhancing dark side contrast of the original YUV image I YUV , e.g. decreasing the luminance of the original YUV image I YUV . In this way, the edge contrast and image sharpness can be improved while reducing false-positives of edge pixel detection (i.e. mistaken noises as edge pixels). The above embodiments are not limited to YUV domain, and the present invention can be applied in other color spaces with luminance, e.g. HSI, HSL, HSV and HSB. 
     Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.