Image processing apparatus to reduce noise in an image

An image processing apparatus to execute a noise reduction process on an image includes setting and calculation units. The setting unit sets a parameter for a pixel of interest in the image. The calculation unit calculates a pixel value of the pixel of interest by using the parameter. Where the pixel of interest is not a pixel in a corner portion where two different edges intersect, the calculation unit averages a plurality of pixels, thereby calculating the pixel value of the pixel of interest subjected to the noise reduction process. Where the pixel of interest is a pixel included in the corner portion, the averaging is not executed and the pixel value of the pixel of interest is calculating by treating the pixel value of the pixel of interest as it is as the pixel value of the pixel of interest subjected to the noise reduction process.

BACKGROUND OF THE INVENTION

Field of the Invention

The present disclosure relates to a technique for reducing noise in an image.

Description of the Related Art

As a technique for reducing noise included in an image, a method for reducing noise without impairing the sharpness of an image such as an edge is conventionally known. The publication of Japanese Patent Application Laid-Open No. 2006-14024 discusses a method for performing a smoothing process on, among pixels in a predetermined range, only a pixel where the difference between the signal value of the pixel and the signal value of a pixel of interest is less than or equal to a threshold.

According to the method discussed in Japanese Patent Application Laid-Open No. 2006-14024, in the case of an edge having low contrast, the difference between pixel values is less than or equal to the threshold. Thus, the pixels may be averaged, and an edge part may blur. If, on the other hand, the threshold, which is to be compared with the difference is set small in order to detect pixels averaged with high accuracy relative to an edge having low contrast, the number of pixels to be used for the averaging with respect to all pixels becomes small, and a noise reduction effect is reduced.

SUMMARY OF THE INVENTION

The present disclosure is directed to appropriately executing a noise reduction process also on an edge in an image.

According to an aspect of the present invention, an image processing apparatus to execute a noise reduction process on an image includes a setting unit configured to set a parameter for a pixel of interest in the image, and a calculation unit configured to calculate a pixel value of the pixel of interest subjected to the noise reduction process by using the parameter, wherein, in a case where the pixel of interest is not a pixel in a corner portion where two different edges intersect, the calculation unit averages a plurality of pixels, thereby calculating the pixel value of the pixel of interest subjected to the noise reduction process, and wherein, in a case where the pixel of interest is a pixel included in the corner portion, the calculation unit does not execute the averaging and calculates the pixel value of the pixel of interest by treating the pixel value of the pixel of interest in the image as it is as the pixel value of the pixel of interest subjected to the noise reduction process.

DESCRIPTION OF THE EMBODIMENTS

With reference to the attached drawings, the present disclosure will be described in detail below based on its suitable embodiments. The configurations illustrated in the following embodiments are merely illustrative, and the present disclosure is not necessarily limited to the illustrated configurations.

A first embodiment is described using as an example an image processing apparatus. The image processing apparatus is configured to detect feature points in a plurality of images obtained by continuously capturing an object, detect feature points corresponding to each other between the plurality of images, and calculate the motion of the entirety of the images. Before detecting the feature points in the images, the image processing apparatus executes a noise reduction process on each of the plurality of images. As each feature point, a corner where two line segments intersect (hereinafter referred to as a “corner”) is used. Thus, if the corner blurs due to the noise reduction process, the feature point cannot be detected with high accuracy. In the first embodiment, a description is given of a noise reduction process, which preserves sharpness of the corner.

FIG. 1is a block diagram illustrating the image processing apparatus according to the first embodiment. An image processing apparatus100includes an entirety control unit101, a bus system102, an image sensor unit103, a development processing unit104, a memory105, a noise reduction processing unit106, a feature point detection unit107, a feature amount calculation unit108, a corresponding point search unit109, and an image signal processing unit110.

The entirety control unit101is a device such as a general-purpose central processing unit (CPU) having the function of controlling via the bus system102the processing order of processing units from the image sensor unit103to the image signal processing unit110, and the transmission and reception of data. The bus system102is connected to the components and the memory105, mediates a data transfer request from each processing unit, and transfers data between the components.

The image sensor unit103includes a two-dimensional sensor composed of a plurality of light-receiving elements. In the image sensor unit103, an electric charge is accumulated receiving incident light entering the sensor through an optical system (not illustrated). The image sensor unit103reads the accumulated electric charge to convert the incident light into a digital electric signal, and outputs the digital electric signal. The development processing unit104performs signal processing on the digital electric signal received from the image sensor unit103, converts the resulting signal into image data in a predetermined format such as a red, green, and blue (RGB) format or a YUV format, and outputs the image data.

The memory105is a storage device such as a dynamic random-access memory (DRAM) for temporarily storing data to be processed by each component, such as image data and feature point data. Further, the memory105is also used to store a program to be used by the entirety control unit101. Each processing unit may separately include a data storage unit (not illustrated) other than the memory105.

The noise reduction processing unit106executes a noise reduction process on the image data output from the development processing unit104. The details of the noise reduction processing unit106will be described below. The feature point detection unit107detects a feature point in the image data subjected to the noise reduction process and output from the noise reduction processing unit106. The feature point detection unit107outputs data obtained by combining the coordinates of the feature point and accompanying information. The feature amount calculation unit108calculates feature amount data indicating the feature amount of the feature point, from surrounding pixel data of the coordinates of the feature point and outputs the calculated feature amount data.

Based on feature amount data of each feature point in temporally continuously captured image data, the corresponding point search unit109searches for corresponding feature points (corresponding points) among the pieces of continuously captured image data. Based on the corresponding points found by the corresponding point search unit109, the image signal processing unit110performs positioning of the continuously captured image data, or combines the continuously captured image data.

Next, the details of the noise reduction processing unit106are described.FIG. 2is a block diagram illustrating the detailed configuration of the noise reduction processing unit106. The noise reduction processing unit106includes a window setting unit200, a signal value section determination unit201, and a smoothing processing unit202.

To determine whether to execute a smoothing process on a pixel of interest, the window setting unit200sets as windows a first area and a second area defined in two different directions. In this case, the window setting unit200sets, as the windows, three pixels in a vertical direction including upper and lower pixels with the pixel of interest at the center, and three pixels in a horizontal direction including left and right pixels with the pixel of interest at the center. Using the signal values of the pixels in the windows, the signal value section determination unit201determines a signal value section where the pixel of interest is to be smoothed.

Using a window wider than the windows set by the window setting unit200, the smoothing processing unit202executes the smoothing process. In the present case, the smoothing processing unit202sets a window including 5 pixels×5 pixels with the pixel of interest at the center, calculates the average of the pixel values of the pixels included in the window, and outputs the calculated average as the pixel value of the pixel of interest subjected to the noise reduction process. The smoothing processing unit202, however, refers to the signal value section determined by the signal value section determination unit201, and if the pixel value of the pixel of interest is outside the determined signal value section, the smoothing processing unit202does not perform the smoothing process.

The details of blocks of the noise reduction processing unit106are described.FIG. 3illustrates the pixel of interest as a target of the noise reduction process and an area near the pixel of interest. In the present embodiment, the smoothing processing unit202sets a window302, which includes 5 pixels×5 pixels, for the pixel of interest. The window setting unit200sets a first area300, which includes upper and lower pixels, and a second area301, which includes left and right pixels, as the windows for determining the signal value section. The first area300and the second area301are pixel groups defined in different directions from each other. In this case, the first area300and the second area301are orthogonal to each other.

In the following description, as illustrated inFIG. 4, (x, y) is used as coordinates representing the position of the pixel of interest, and I(x, y) is used as a sign representing the pixel value of the pixel position (x, y).

FIG. 5is a block diagram illustrating the internal configuration of the signal value section determination unit201. The signal value section determination unit201includes representative value calculation units600aand600b, a condition determination unit601, multiplexers602ato602c, a constant number calculation unit603, a subtraction unit604, and an addition unit605. The signal value section determination unit201determines the signal value section for determining whether to execute the smoothing process on the pixel of interest.

Values I(x, y), I(x, y−1), and I(x, y+1) of the pixels included in the first area300are input into the representative value calculation unit600a. The representative value calculation unit600aoutputs the greatest value among the pixel values I(x, y), I(x, y−1), and I(x, y+1) as a maximum value max1. Further, the representative value calculation unit600aoutputs the smallest value among the pixel values I(x, y), I(x, y−1), and I(x, y+1) as a minimum value min1. Further, the representative value calculation unit600acalculates the median of the three pixel values I(x, y), I(x, y−1), and I(x, y+1) and outputs the calculated median as a median value med1. That is, the representative value calculation unit600acalculates the maximum value max1, the minimum value min1, and the median value med1as representative values.

Values I(x, y), I(x−1, y), and I(x+1, y) of the pixels included in the second area301are input into the representative value calculation unit600b. Then, the representative value calculation unit600bcalculates a maximum value max2, a minimum value min2, and a median value med2as representative values similarly to the representative value calculation unit600a.

The maximum value max1and the minimum value min1calculated by the representative value calculation unit600aand the maximum value max2and the minimum value min2calculated by the representative value calculation unit600bare input into the condition determination unit601. The condition determination unit601compares the difference between the maximum value max1and the minimum value min1with the difference between the maximum value max2and the minimum value min2, thereby determining a condition. The condition determination unit601outputs control signals to the multiplexers602ato602cso that the representative value is selected that is the difference between the maximum value max1and the minimum value min1or the difference between the maximum value max2and the minimum value min2, whichever is smaller.

Based on the control signals from the condition determination unit601, the multiplexers602ato602cselect and output the maximum value max1, the minimum value min1, and the median value med1or the maximum value max2, the minimum value min2, and the median value med2. The multiplexers602a,602b, and602coutput a maximum value max, a minimum value min, and a median value med, respectively.

The median value med output from the multiplexer602cis input into the constant number calculation unit603. Then, the constant number calculation unit603outputs a constant q. In the present case, the constant number calculation unit603outputs the median value med as it is as the constant q.

The subtraction unit604outputs a value obtained by subtracting the constant q calculated by the constant number calculation unit603from the maximum value max output from the multiplexer602a, as a lower limit lower of the signal value section. The addition unit605calculates an upper limit upper of the signal value section by adding the constant q calculated by the constant number calculation unit603to the minimum value min output from the multiplexer602b. Thus, the signal value section determination unit201determines the lower limit lower and the upper limit upper based on the minimum value min and the maximum value max in the pixel groups near the pixel of interest and outputs the lower limit lower and the upper limit upper together with the median value med output from the multiplexer602c, to the smoothing processing unit202.

Next, the details of the smoothing process performed by the smoothing processing unit202are described. The smoothing processing unit202performs the smoothing process using a pixel group included in the window302, which is wider than the first area300and the second area301set by the window setting unit200. The smoothing processing unit202makes a calculation represented by formula (1) with a non-linear function f(v). The non-linear function f(v) is represented by formula (2), using the lower limit lower and the upper limit upper input from the signal value section determination unit201.

If a pixel value is in the range of signal values determined by the lower limit lower and the upper limit upper, the non-linear function f(v) outputs one as a result. In other cases, the non-linear function f(v) outputs zero as a result. That is, the pixel values of pixels not included in the set range of signal values are excluded from the 5 pixels×5 pixels including the pixel of interest. If the results of the non-linear function f(v) represented by formula (2) are zero with respect to all of the 5 pixels×5 pixels including the pixel of interest, the smoothing process is not performed, and a constant c is output. In the present embodiment, the constant c is the pixel value of the pixel of interest. If, on the other hand, none of the 5 pixels×5 pixels including the pixel of interest satisfy zero, the smoothing process is executed by calculating the average value of the pixels where the results obtained by the non-linear function f(v) are not zero.

Next, referring toFIG. 6as an example, the effects of the image processing apparatus according to the present embodiment are described. An image700includes 9 pixels×9 pixels, i.e., 81 pixels. For ease of description, the image700is an 8-bit gray image in which the pixel value of each pixel is any value from 0 to 255. If the pixel value of a white pixel is 255, and the pixel value of a gray pixel is 128, the image700is an image including a corner portion701where edges intersect at the lower right end of the image. A description is given below using as a specific example a pixel group702, which is represented by pixels P0, P1, P2, P3, and P4.

FIGS. 7A and 7Bare diagrams illustrating the pixel values of the pixel group702and the signal value section determined for each pixel.FIG. 7Ais a graph plotting the pixel values obtained by scanning the pixel group702in the order of raster scanning. The horizontal axis represents the pixel position, and the vertical axis represents the pixel value. Further,FIG. 7Bis a graph plotting the lower limit lower and the upper limit upper output from the signal value section determination unit201for each pixel of the pixel group702. InFIG. 7B, a fixed value of 16 is used as the constant q.

If the pixel of interest in the image is a flat portion, the difference between the maximum value max and the minimum value min of either the first area300or the second area301is also small. Therefore, with respect to the lower limit lower set by the maximum value max−the median value med, and the upper limit upper set by the minimum value min+the median value med, the signal value section is determined such that the difference between the maximum value max and the minimum value min is increased. If the pixel of interest in the image is an edge, the difference between the maximum value max and the minimum value min of either of the first area300or the second area301is great. Therefore, with respect to the lower limit lower set by the maximum value max−the median value med, and the upper limit upper set by the minimum value min+the median value med, the signal value section is determined such that the maximum value max and the minimum value min are brought closer to each other and the difference between the maximum value max and the minimum value min becomes smaller than in a case where the pixel of interest is a flat portion. As a result, in a flat portion of the image, the signal value section is larger, and more pixels for smoothing the pixel of interest can be detected than in a case where the pixel of interest is an edge portion. This means that a noise reduction effect is strengthened in the flat portion. On the other hand, in an edge portion of the image, the signal value section is smaller, and only a pixel having a pixel value closer to the pixel of interest is detected. Thus, in the edge portion of the image, the preservation of the edge can be given priority over the noise reduction effect. As described above, in the first embodiment, according to the difference between the maximum value and the minimum value of pixel values in a predetermined area, a signal value section for detecting pixels for use in a smoothing process is set, thereby achieving a noise reduction according to a character of an image.

Further, as illustrated inFIG. 7B, at the pixel P2, which corresponds to the vertex of the corner portion701, the difference between the maximum value max1and the minimum value min1detected in the first area300and the difference between the maximum value max2and the minimum value min2detected in the second area301both have great values. Therefore, at the pixel P2, which corresponds to the vertex of the corner portion701, the lower limit lower of the set signal value section is greater than the upper limit upper. Thus, there is no pixel satisfying the condition that the pixel value is equal to or greater than the lower limit lower and less than or equal to the upper limit upper. Consequently, the result of the non-linear function f(v) represented by formula (2) is unconditionally zero, which means that the average value is not calculated by the smoothing processing unit202.

As described above, in the present embodiment, the lower limit is set based on the maximum value of the first area300or the second area301, and the upper limit is set based on the minimum value of the first area300or the second area301. In this case, if the difference between the maximum value and the minimum value is equal to or greater than a predetermined value, the lower limit and the upper limit are reversed. In the first embodiment, if the difference between the maximum value and the minimum value is equal to or greater than two times the median value, the lower limit and the upper limit are reversed. When the upper limit and the lower limit are thus set, by using a signal value section for detecting a pixel having a pixel value close to the pixel value of a pixel of interest without setting a special determination circuit at a previous stage, it becomes possible to substantially determine whether a smoothing process is to be executed. Only if the difference between the maximum value and the minimum value is equal to or greater than the predetermined value, a noise reduction process is executed by smoothing. In the present embodiment, a corner portion is detected as a feature point. Accordingly, the corner portion is detected with high accuracy, and smoothing is not performed, whereby it is possible to detect the feature point with high accuracy at a subsequent stage.

In the above description, as the first area300and the second area301, areas other than an area extending in the vertical direction and an area extending in the horizontal direction with the pixel of interest at the center as illustrated inFIG. 3Amay be used. For example, as illustrated inFIG. 3B, a first area400may be an area including three pixels from the upper right to the lower left with the pixel of interest at the center. In this case, as a second area401, an area including three pixels positioned orthogonal to the first area400and from the upper left to the lower right with the pixel of interest at the center is selected.

Further, in the present embodiment, the median value med is the constant q for calculating the signal value section. Alternatively, the constant q may be calculated based on a table or a mathematical formula using the median value med, or the constant q may be calculated without using the median value med. The greater the value of the constant q, the wider the signal value section, and the greater the number of pixels for use in smoothing. Thus, the effect of the noise reduction process is strengthened. Further, the smaller the value of the constant q, the smaller the number of pixels for use in smoothing, and the less likely the smoothing process is to be executed. In a case where reduction of blur in an edge portion or preservation of a corner portion is given priority over the effect of the noise reduction process, the constant q may be set to a relatively small value.

Further, a description has been given using as an example a method in which, if there is no pixel having a pixel value in the signal value section, the signal value section determination unit201outputs the pixel value of the pixel of interest. For example, the median value med may be the constant c.

Further, the smoothing processing unit202may also use formula (3) instead of the calculation of the average value represented by formula (1).

In formula (3), w(i, j) is a weight coefficient and can optionally specify the value of a weight for each pixel in the window302. Alternatively, the smoothing processing unit202may mix the average value calculated by formula (1) with the pixel value of the pixel of interest using any optional ratio and outputs the resulting value.

A second embodiment will be described below. As a block diagram illustrating an image processing apparatus according to the second embodiment,FIG. 1in the first embodiment is used in common.FIG. 8is a block diagram illustrating the configuration of the noise reduction processing unit106according to the second embodiment. Components having functions similar to the first embodiment are designated by the same signs, and are not described in detail. The noise reduction processing unit106includes a signal value section setting unit1000, which sets a signal value section for a signal value section determination unit1001.

For the signal value section determination unit1001, the signal value section setting unit1000sets an upper limit and a lower limit of a signal value determined in advance or an upper limit and a lower limit of a signal value notified from outside.FIG. 9is a block diagram illustrating the internal configuration of the signal value section determination unit1001according to the present embodiment. Components having functions similar to those of the first embodiment are designated by the same signs, and are not described in detail.

The signal value section determination unit1001additionally includes a lower limit comparison unit1100, an upper limit comparison unit1101, and multiplexers1102aand1102b. The lower limit comparison unit1100compares a lower limit lext, which is set by the signal value section setting unit1000, with a lower limit ltmp, which is output from the subtraction unit604. Then, the lower limit comparison unit1100outputs a control signal to the multiplexer1102asuch that the greater of the lower limits is selected. The upper limit comparison unit1101compares an upper limit uext, which is set by the signal value section setting unit1000, with an upper limit utmp, which is output from the addition unit605. Then, the lower limit comparison unit1100outputs a control signal to the multiplexer1102bsuch that the smaller of the upper limit values is selected.

Next, usingFIG. 10as an example, the effects of the image processing apparatus according to the present embodiment are described.FIG. 10is a graph plotting the lower limit lower and the upper limit upper output from the signal value section determination unit1001for the pixel group702inFIG. 6. InFIG. 10, the constant q is a fixed value of 16, and the lower limit lext and the upper limit uext, which are set by the signal value section setting unit1000, are 192 and 255, respectively. As illustrated inFIG. 10, the lower limit lower and the upper limit upper of the signal value section are reversed at the pixels P2, P3, and P4of the corner portion701. In the configuration of the present embodiment, it is thus possible to prevent the smoothing of pixels belonging to a dark portion in the image in addition to the vertex of the corner portion701.

Further, in addition to the example illustrated inFIG. 10, for example, if the lower limit lext and the upper limit uext, which are set by the signal value section setting unit1000, are 0 and 225, respectively, it is possible to prevent the smoothing of pixels belonging to a high-luminance portion in the image inFIG. 10.

As described above, according to the present embodiment, the signal value section setting unit1000is provided so that it is possible to adjust the range of pixel values where the smoothing processing unit202performs smoothing.

Other Embodiments

The above embodiments have been described using as an example a case where the present disclosure can be achieved by a circuit (e.g., an application-specific integrated circuit (ASIC)) for achieving one or more functions. The present disclosure, however, can also be achieved by the process of supplying a program for achieving one or more functions of the above embodiments to a system or an apparatus via a network or a storage medium, and causing one or more processors of a computer of the system or the apparatus to read and execute the program.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2017-166097, filed Aug. 30, 2017, which is hereby incorporated by reference herein in its entirety.