Patent Description:
<FIG> of the drawings shows a high-level block diagram of a camera <NUM> in accordance with the prior art. As will be seen, the camera <NUM> comprises a lens <NUM> for focusing rays of light onto an image sensor <NUM>. The image sensor <NUM> may be a charge-coupled device (CCD) or an active-pixel (CMOS) sensor capable of converting the incoming light into pixels which are fed into a downstream image signal processing pipeline <NUM>. Typically, the signal processing pipeline performs operations such as demosaicing, noise reduction, auto exposure, autofocus, auto white balance and image sharpening on pixels, prior to output. <CIT> relates to an image processing apparatus configured to calculate an absolute difference in pixel values between each pixel (target pixel) of an image and each of neighboring pixels of the target pixel, and a sum of the absolute differences with respect to the target pixel; calculate a threshold using the sum of the absolute differences with respect to the target pixel and a weight previously determined, the threshold being used as a reference for determining whether the pixel values of the neighboring pixels are used in a smoothing process for the target pixel. Patent documents <CIT> and <CIT> disclose denoising based on sum of intensity differences between a current pixel and its neighboring pixels.

According to a first aspect of the invention, a method, performed by an image processing system, for reducing noise in a digital image is provided.

According to a second aspect of the invention is provided the system for implementing the above method.

Other aspects of the invention, will be apparent from the written description below.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention.

Reference in this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Similarly, various requirements are described which may be requirements for some embodiments but not others.

Moreover, although the following description contains many specifics for the purposes of illustration, anyone skilled in the art will appreciate that many variations and/or alterations to said details are within the scope of the present invention. Similarly, although many of the features of the present invention are described in terms of each other, or in conjunction with each other, one skilled in the art will appreciate that many of these features can be provided independently of other features. Accordingly, this description of the invention is set forth without any loss of generality to, and without imposing limitations upon, the invention.

Disclosed is a method for reducing noise in a digital image, the method comprising calculating a spatial weight for each pixel s in a current frame, wherein said spatial weight is computed based on a summation of diffused values associated with the pixel s; and selectively filtering an intensity I of the pixel s spatially using said spatial weight.

The spatially diffused values comprise intensity differences between the pixel s and its neighbors in a spatial domain.

The spatially diffused values are based on pixel intensity and are calculated according to the formula: <MAT> where I represents pixel intensity; and (t,s) specified each pixel s pixel in the current frame.

Filtering the intensity I of the pixel s spatially is according to the formula: <MAT> where λ is a constant.

Selectively filtering the intensity I of the pixel s spatially comprises selecting a constant value rho such that: <MAT> <MAT>.

Rho is selected to be proportional to a variance of noise in the image.

In one or more example methods, the method comprises selectively filtering the spatially filtered sample temporally based on a temporal weight calculated between the intensity I of the pixel s in the current frame and its value from and immediately preceding frame.

In one or more example methods, the temporal weight is calculated according to the formula: <MAT> where I(t, s) is the intensity of the pixel s in current frame I(t-<NUM>, s) is its intensity from the immediately preceding frame (t-<NUM>).

In one or more example methods, the method comprises outputting a temporally filtered pixel calculated in accordance with the following formula: <MAT> where λ is a constant.

In one or more example methods, the method comprises enhancing contrast in the digital image by applying a contrast enhancement function H(s) to each pixel s with value was altered due to the spatial filtering.

In one or more example methods, the contrast enhancement function comprises an edge detection algorithm.

Also disclosed is an image processing system comprising at least one image sensor; and an image processing pipeline configured to perform a method for reducing noise in a digital image, the method comprising calculating a spatial weight for each pixel s in a current frame, wherein said spatial weight is computed based on a summation of diffused values associated with the pixel s; and selectively filtering an intensity I of the pixel s spatially using said spatial weight.

The spatially diffused values are calculated according to the formula: <MAT> where I represents pixel intensity; and (t,s) specified each pixel s pixel in the current frame.

In one or more example image processing systems, the method further comprises selectively filtering the spatially filtered sample temporally based on a temporal weight calculated between the intensity I of the pixel s in the current frame and its value from and immediately preceding frame.

In one or more example image processing systems, the temporal weight is calculated according to the formula: <MAT> where I(t, s) is the intensity of the pixel s in current frame I(t-<NUM>, s) is its intensity from the immediately preceding frame (t-<NUM>).

In one or more example image processing systems, the method comprises outputting a temporally filtered pixel calculated in accordance with the following formula: <MAT> where λ is a constant.

In one or more example image processing systems, the method comprises a contrast enhancement configured to apply a contrast enhancement function H(s) to each pixel s with value was altered due to the spatial filtering.

In accordance with one embodiment of the invention, and referring to <FIG> of the drawings, a camera pipeline comprises a lens <NUM>, and an image sensor <NUM> which are components similar to what has already been described with respect to the prior art camera pipeline <NUM> of <FIG>. Thus, the same reference numerals have been used to indicate these components. However, unlike the prior art camera pipeline <NUM>, the output of the image sensor <NUM> is fed to a modified image processing pipeline <NUM> which includes a three-dimensional noise reduction block <NUM>, and a contrast enhancement block <NUM>.

As will be described, the three-dimensional noise reduction block <NUM> performs spatial filtering in two dimensions, and temporal filtering in one dimension, hence the noise reduction techniques are performed by the block <NUM> is referred to herein as three-dimensional noise reduction.

Referring to <FIG>, in one embodiment, the 3D noise reduction block <NUM> may comprise a spatial filtering block <NUM> configured to receive input frames <NUM>, whereupon a spatial filtering technique is applied on a frame-by-frame basis, as will now be described. The technique begins by receiving the current frame at block <NUM>, a copy of which is immediately saved at block <NUM> to a temporal buffer. The saved copy of the current frame is used in a temporal filtering block which will be described later. Next, for each pixel in the current frame, differences between the current pixel and its neighbors are calculated. The calculation of the differences between the current pixel and its neighbors is performed at block <NUM>. To illustrate how the differences are calculated, consider the current frame numeral <NUM> depicted in in <FIG> of the drawings. For illustrative purposes the current frame shows only nine pixels P(x,y). In reality, one of ordinary skill in the art would understand that the current frame may comprise more pixels based on the resolution of the image. Consider the pixel P(<NUM>,<NUM>) which is located at the center of the frame. The arrows leading from the pixel P(<NUM>,<NUM>) indicate each of its neighboring pixels. Calculating the differences between the occurrence pixel and its neighbors in this case simply means calculating the following differences:.

If the current pixel has no neighbors, such as is the case for the pixel P(<NUM>,<NUM>) , then padding of pixels may be performed. In one embodiment, the particular padding technique used comprises simply taking the value of the current pixel as the value of the padded pixels.

Referring again to <FIG> of the drawings, at block <NUM> the sum of the differences calculated in the previous step is computed. This sum of the computed differences is used in block <NUM> to selectively apply spatial filtering to the current pixel as follows:.

If the sum is greater than a threshold then the current pixel is not changed, in other words no spatial filtering is applied to the current pixel. However, if the sum is less than threshold, then the current pixel is replaced with a value corresponding to the sum of the computed differences calculated at block numeral <NUM>.

In the technique described above, the spatial filtering block numeral <NUM> calculates for each pixel "s" in frame time t, a new pixel value is in the spatial domain where the pixel's neighbors are included in the summation of diffused values. In the example described, all the current pixel's neighbors (A, B, C, D, E, F, and G -see frame <NUM>) are used in the calculation. In some embodiments, not all the pixel's neighbors is used in the calculation. For example, only the four pixels B, D, E, and F may be used.

To achieve the filtering, firstly the current pixel is filtered with the surrounding pixels in the same frame as follows: <MAT> where I represents pixel intensity; and (t,s) is the current pixel in the current frame. Thus, the above calculation sums the "differences" over all 'p's - the surrounding pixels in the same frame as discussed above. 's' is the center pixel P(<NUM>,<NUM>), and 'p's are the neighboring pixels;.

The spatially-filtered sample is output in accordance with the following equation: <MAT> where the "Spatial_filtered_sample" is the output of spatial filtering on the current pixel.

As noted above, the spatial filtering is selective in that the current pixel value is only altered in certain cases. In one embodiment, spatial filtering of the current pixel only occurs if the sum of differences is below a threshold rho as follows:
For a constant "rho", proportional to the variance of the noise: <MAT> <MAT>.

In one or more embodiments, rho may be set to be <NUM> or <NUM>. The above calculations are performed for each channel within the color space separately. For example, for the RGB color space, the above calculations are performed separately for each of the R, G, and B pixels.

In accordance with the above spatial filtering, if the sum of the differences is greater than rho then the current pixel is left unchanged (ψ (x) = <NUM>), otherwise and adapted diffusion is applied to the current pixel in terms of the formula ψ (x) = (<NUM> - (x / rho)^<NUM>)^<NUM>.

Referring again to <FIG> of the drawings, temporal filtering is applied at block <NUM>. In one embodiment, for temporal filtering, each pixel in a new current frame is (t) received at block <NUM> is compared to the corresponding co-sited pixel from the previous frame (t-<NUM>) that was saved in the temporal buffer at block numeral <NUM>. In particular, I(t, s), the intensity of each pixel of the current frame is compared to the intensity of the co-sited pixel I(t-<NUM>, s) of the previous frame time (t-<NUM>). For example, referring to <FIG>, the current frame is represented by reference numeral <NUM>, whereas the previous frame is represented by reference numeral <NUM>; and the co-sited pixels the same have the same (x,y) coordinates. Temporal filtering may then be performed in accordance with the following formula: <MAT>.

The output of the temporal filtering is a temporally filtered pixel calculated in accordance with the following formula: <MAT>.

This new pixel "s" and co-sited past frame pixel "p" are used in above spatial - temporal calculation. The result of the calculation is stored in the past frame storage (temporal buffer). It serves as the input for the following spatial contrast enhancement and may be sent out as the output of the spatial-temporal filter if contrast enhancement is not required.

In one embodiment, for each pixel s in the temporal buffer, an adaptive diffusion in the spatial domain is calculated as follows: <MAT> <MAT> if |T_weight(s)| > <NUM>; This means we filtered the current sample; and <MAT> if |T_weight(s)| equals <NUM>; This means this sample is sitting on an edge;
where H(s) is the contrast enhancement function described below and it is weighted by a constant k.

In one or more embodiments, the contrast enhancement function may utilize edge detectors, in the form of Laplacian and Sobel operators. The Laplacian is a second order derivative operator so that it is very sensitive to random noise. For this reason, it is better to modulate the output of the Laplacian operator by a more structured first order Sobel operator. The Laplacian operator also provides the sign of the contrast enhancement.

The Sobel is defined as the sum of absolute values of the horizontal and vertical gradient operator: <MAT>.

In one or more embodiments, small Sobel values may be set to zero and large Sobel values may be claimed as follows: <MAT> where c1 is a constant; and <MAT> where c2 is a constant.

In some embodiments, c1 may be <NUM> and c2 any be <NUM>.

The contrast enhancement value H(s) may then be calculated as follows: <MAT>.

The contrast enhancement techniques described above is performed by the contrast announcement block <NUM> shown in <FIG> of the drawings.

<FIG> and <FIG> show representations of a Laplacian and Sobel operator, respectively used for edge detection in accordance with an embodiment of the invention.

As will be appreciated by one skilled in the art, the aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module," or "system.

The title, background, brief description of the drawings, abstract, and drawings are hereby incorporated into the disclosure and are provided as illustrative examples of the disclosure, not as restrictive descriptions. It is submitted with the understanding that they will not be used to limit the scope or meaning of the claims. In addition, in the detailed description, it can be seen that the description provides illustrative examples and the various features are grouped together in various implementations for the purpose of streamlining the disclosure. The method of disclosure is not to be interpreted as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. Rather, as the claims reflect, inventive subject matter lies in less than all features of a single disclosed configuration or operation. The claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately claimed subject matter.

Claim 1:
A method, performed by an image processing system, for reducing noise in a digital image, comprising:
calculating a spatial weight for each pixel s in a current frame, wherein said spatial weight is computed based on a summation of diffused values associated with the pixel s; and
selectively filtering an intensity I of the pixel s spatially using said spatial weight, wherein the spatially diffused values are calculated according to the formula: <MAT> where I represents pixel intensity; (t,s) specifies each pixel in the current frame t,
and the spatial diffused values comprise intensity difference between the pixel s and its neighbouring pixels p in a spatial domain,
wherein filtering the intensity I of the pixel s spatially is according to the formula: <MAT> where λ is a constant,
wherein selectively filtering the intensity I of the pixel s spatially comprises selecting a constant value rho such that: <MAT> <MAT> and
wherein rho is selected to be proportional to a variance of noise in the image.