Adhering substance detection apparatus and adhering substance detection method

An adhering substance detection apparatus includes calculating, first and second detecting, and generating units. The calculating unit calculates a variation in a feature value related to luminance in past and current captured images captured by an image capturing device, based on the luminance of pixels included in the captured images. The first detecting unit detects a first region in which the variation calculated by the calculating unit falls within a threshold range, and the feature value in the current captured image falls within a threshold range. The second detecting unit detects a second region in which an irregularity in a distribution of the luminance of the pixels included in the captured image satisfies a predetermined irregularity condition. The generating unit generates a sum region being a sum of the first and second regions, as an adhering substance region corresponding to an adhering substance adhering to the image capturing device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims benefit of priority of the prior Japanese Patent Application No. 2018-246916, filed on Dec. 28, 2018, the entire contents of which are incorporated herein by reference.

FIELD

The embodiment discussed herein is directed to an adhering substance detection apparatus and an adhering substance detection method.

BACKGROUND

Conventionally having been known is an adhering substance detection apparatus that detects an adhering substance adhering to a lens, based on a temporal change in the luminance values of segments that are divisions of the area of the captured image (see Japanese Laid-open Patent Publication No. 2014-30188, for example).

However, in the conventional technology, there has been some room for improvement in highly accurate detection of an adhering substance. For example, when the adhering substance is a mixture of a water drop and mud, the region corresponding to the water drop has higher luminance than the region corresponding to the mud. Therefore, in of attempt to detect one of such regions, the detection of the other may fail.

SUMMARY

An adhering substance detection apparatus according to an embodiment includes a calculating unit, a first detecting unit, a second detecting unit, and a generating unit. The calculating unit calculates a variation in a feature value related to luminance in past and current captured images captured by an image capturing device, based on the luminance of pixels included in the captured images. The first detecting unit detects a first region in which the variation calculated by the calculating unit falls within a predetermined threshold range, and the feature value in the current captured image fails within a predetermined threshold range. The second detecting unit detects a second region in which an irregularity in a distribution of the luminance of the pixels included in the captured image satisfies a predetermined irregularity condition. The generating unit generates a sum region that is a sum of the first region detected by the first detecting unit and a second region detected by the second detecting unit, as an adhering substance region corresponding to an adhering substance adhering to the image capturing device.

DESCRIPTION OF EMBODIMENT

An adhering substance detection apparatus and an adhering substance detection method according to an embodiment of the present invention will now be explained in detail with reference to the appended drawings. However, the embodiment described below is not intended to limit the scope of the present invention in any way.

To begin with, a general outline of an adhering substance detection method according to an embodiment will be explained with reference toFIG. 1.FIG. 1is a schematic giving a general outline of the adhering substance detection method according to the embodiment.FIG. 1illustrates a captured image I that is captured by a camera that is onboard a vehicle, with some adhering substance that is a mixture of water and mud adhering to the lens of the camera, for example. In the captured image I illustrated inFIG. 1, while a region containing a larger amount of mud (mud illustrated inFIG. 1) is represented as a black region with no gradation, because such a region penetrates almost no light, a region containing a larger amount of water (water illustrated inFIG. 1) is represented as a blurred region, because such a region penetrates a slight amount of light. In other words, the region containing a larger amount of water has higher luminance, compared with the region containing a larger amount of mud. Examples of the adhering substance are not limited to water and mud, and may include any adhering substance represented as a black region with no gradation, and a blurred region in the captured image I.

Conventionally, when the adhering substance is substance such as a mixture of mud and water, if the luminance threshold is set to a level for detecting the mud, the region blurred with water may not be detected, because the threshold is set to a low level. If the luminance threshold is set to a level for detecting both of the water and the mud, a larger number of objects other than the adhering substance may be detected erroneously, because the threshold is set to a high level. Examples of the blurred region include a region including blurred objects in the background, a region having become blurred due to the different concentrations of the mud contained in the water, and a region having become blurred due to the three-dimensional shape of a water drop, for example.

To address this issue, the adhering substance detection apparatus1according to the embodiment (seeFIG. 2) detects adhering substance such as a mixture of water and mud highly accurately, by performing an adhering substance detection method. Specifically, the adhering substance detection method according to the embodiment performs a first detection process for detecting a black region with no gradation, which corresponds to mud, for example, and a second detection process for detecting a blurred region, which corresponds to water, for example. The sum of the two types of regions detected by these two detection processes is then established as an adhering substance region. The adhering substance detection method according to the embodiment will now be explained with reference toFIG. 1.

As illustrated inFIG. 1, to begin with, the adhering substance detection method according to the embodiment calculates a variation in a feature value related to the luminance in past and current captured images I captured by the camera, based on the luminance of the pixels included in the captured images I (S1). The feature value includes a representative value of the luminance and a dispersion of the luminance of a predetermined region. Specifically, a mean value is used as the representative value, and a standard deviation is used as the dispersion. The variation is the difference between a feature value calculated from the past captured image I and a feature value calculated from the current captured image I, for example.

The adhering substance detection method according to the embodiment then detects a first region A1in which the calculated variation in the feature value falls within a predetermined threshold range, and in which the feature value in the current captured image I falls within a predetermined threshold range (S2). In other words, by performing the first detection process, the adhering substance detection method according to the embodiment detects a first region A1that is a black region with no gradation, in which the feature value has gone through little variation from the past to the present, and in which the current feature values are small. The details of the method for detecting the first region A1will be described later.

The adhering substance detection method according to the embodiment then detects a second region A2in which an irregularity in the distribution of the luminance of the pixels included in the current captured image I satisfies a predetermined irregularity condition (S3). Specifically, by performing the second detection process, the adhering substance detection method according to the embodiment detects a region in which the irregularity in the luminance distribution is moderate, that is, a blurred region, as a second region A2. More specifically, the second region A2can be also said to be a region in which the variation in the feature value falls within the predetermined threshold range and the feature value in the current captured image is outside the predetermined threshold range. Details of the method for detecting the second region A2will be described later. The distribution of the luminance of the pixels herein means a shape in which the luminance changes along a predetermined direction of a subject image. For example, defining a predetermined coordinate (x0, y0) of the image as a point of origin, and denoting the luminance of the pixels in a horizontal direction x as L(x), the shape drawn as a graph of x-L(x) is referred to as a pixel luminance distribution in the horizontal direction, with the point of origin at (x0, y0). x0, y0 may be set to any coordinates, and the direction may be set to any direction at any angle, including a vertical direction.

The adhering substance detection method according to the embodiment then generates a sum region that is the sum of the detected first region A1and the second region A2, as an adhering substance region A12corresponding to the adhering substance (S4).

In other words, the adhering substance detection method according to the embodiment separately detects the first region A1that is a black region with no gradation, which corresponds to the mud, and detects the second region A2that is a blurred region, which corresponds to the water, and detects the adhering substance region A12by taking the sum of the first region A1and the second region A2at the final step.

In this manner, even if the adhering substance is a mixture of mud and water, for example, it is possible to detect the mud and the water, by performing separate detection processes suitable for the characteristics of the two. In other words, with the adhering substance detection method according to the embodiment, adhering substance can be detected highly accurately.

The adhering substance detection method according to the embodiment sets a plurality of segments to the captured image I, and generates the adhering substance region A12correspondingly to the segments, but this point will be described later.

Furthermore, the adhering substance detection method according to the embodiment then determines whether the adhering substance has been removed based only on the variation in the feature value related to the luminance in the adhering substance region A12, unlike the determination as to whether the adhering substance region A12adheres to. This point will be described later.

A configuration of the adhering substance detection apparatus1according to the embodiment will now be explained with reference toFIG. 2.FIG. 2is a block diagram illustrating a configuration of the adhering substance detection apparatus1according to the embodiment. As illustrated inFIG. 2, the adhering substance detection apparatus1according to the embodiment is connected to a camera10, a vehicle-speed sensor11, and various devices50. In the example illustrated inFIG. 2, the adhering substance detection apparatus1is provided separately from the camera10and the various devices50, but the configuration is not limited thereto, and the adhering substance detection apparatus1may be integrated with at least one of the camera10and the various devices50.

The camera10is a camera that is onboard a vehicle, and is provided with a lens such as a fisheye lens, and an imaging device such as a charge-coupled device (COD) or a complementary metal oxide-semiconductor (CMOS). The camera10is provided at each position where images of the front and the rear sides, and the lateral sides of the vehicle can be captured, for example, and the captured images I are output to the adhering substance detection apparatus1.

The various devices50are devices that perform various vehicle control by acquiring detection results of the adhering substance detection apparatus1. The various devices50include a display device for notifying a user of the presence of an adhering substance adhering to the lens of the camera10or of an instruction for wiping the adhering substance, a removing device for removing the adhering substance by spraying fluid, gas, or the like toward the lens, and a vehicle control device for controlling automated driving and the like, for example.

As illustrated inFIG. 2, the adhering substance detection apparatus1according to the embodiment includes a control unit2and a storage unit3. The control unit2includes an acquiring unit21, a calculating unit22, a first detecting unit23, a second detecting unit24, a generating unit25, a removal determining unit26, and a flag output unit27. The storage unit3stores therein irregularity condition information31.

The adhering substance detection apparatus1includes a computer or various types of circuits including a central processing unit (CPU), a read-only memory (ROM), a random-access memory (RAM), a data flash memory, and an input/output port.

The CPU included in the computer functions as the acquiring unit21, the calculating unit22, the first detecting unit23, the second detecting unit24, the generating unit25, the removal determining unit26, and the flag output unit27included in the control unit2, by reading and executing a computer program stored in the ROM, for example.

At least one of or the whole of the acquiring unit21, the calculating unit22, the first detecting unit23, the second detecting unit24, the generating unit25, the removal determining unit26, and the flag output unit27included in the control unit2may be implemented as hardware such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

The storage unit3corresponds to, for example, a RAM or a data flash memory. The RAM or the data flash memory is capable of storing therein the irregularity condition information31, information of various computer programs, and the like. The adhering substance detection apparatus1may also acquire these computer programs or various types of information from another computer connected over a wired or wireless network, or via a portable recording medium.

The irregularity condition information31stored in the storage unit3is information including conditions that are used as a reference in a detection process performed by the second detecting unit24, which will be described later, and includes a pattern condition for the irregularity in a luminance distribution, for example. A pattern condition is a pattern of the shape of the irregularity that is a map of the luminance distribution, or a pattern of a luminance data sequences in the luminance distribution. The detection process using the irregularity condition information31will be described later.

The acquiring unit21acquires various types of information. The acquiring unit21acquires an image captured by the camera10, and generates (acquires) a current frame that is the current captured image I. Specifically, the acquiring unit21performs a gray-scaling process for converting each pixel. of the acquired image into a gray scale value between white and black, based on the luminance of the pixel.

The acquiring unit21also performs a pixel decimation process to the acquired image, and generates an image having a smaller size than the acquired image. The acquiring unit21then generates a current frame that is an integral image of the sum and the sum of squares of the pixel values of the pixels, based on the image applied with the decimation process. A pixel value is information corresponding to luminance or an edge included in the pixel.

In this manner, by performing a decimation process to the acquired image, and generating an integral image, the adhering substance detection apparatus can increase the calculation speed of the subsequent processes. Therefore, it is possible to reduce the processing time for detecting adhering substance.

The acquiring unit21may also perform a smoothing process using a smoothing filter such as a mean filter. It is also possible for the acquiring unit21to generate a current frame having the same size as the acquired image, without applying the decimation process.

The acquiring unit21also acquires a vehicle speed, based on a signal from the vehicle-speed sensor11.

The calculating unit22calculates a variation in a feature value related to the luminance in the past and current captured images I acquired by the acquiring unit21, based on the luminance of the pixels included in the captured images I.FIG. 3a schematic illustrating the details of a process performed by the calculating unit22.

As illustrated inFIG. 3, to begin with, the calculating unit22sets a region of interest ROI and segments100to the captured image I. The region of interest ROI is a rectangular region that is set in advance based on the characteristics of the camera10, and is a region excluding the vehicle body region, for example. The segments100are rectangular regions resultant of dividing the region of interest ROI horizontally and vertically. For example, each of the segments100is a region including 40×40 pixels, but the number of pixels included in one segment100may be set to any number.

The calculating unit22calculates a feature value related to the luminance, for each of the segments100. Specifically, the calculating unit22calculates a representative value of the luminance and a dispersion of the luminance as the feature value. The representative value is a value indicating representative luminance in the luminance distribution corresponding to a subject region, and, specifically, a mean value is used. Without limitation to the mean value, trimmed mean, median, or a mode, for example, may also be used. A dispersion is a value indicating the spread of the luminance distribution corresponding to a subject region, and, specifically, a standard deviation is used for that. Without limitation to a standard deviation, a variance, a maximum/minimum width, an interquartile width, or any percentile width may be used. Hereinafter, an example in which a mean value is used as the representative value, and a standard deviation is used as the dispersion will be explained. The calculating unit22also calculates these feature values related to the luminance, for the entire region of interest ROI.

The calculating unit22then calculates a variation in the feature value in the past and current captured images I. Specifically, the calculating unit22calculates a first difference that is a difference between a mean value of the luminance in a segment100in the current captured image I and a mean value of the luminance in the segment100that is at the same position in the past image, as a variation. In other words, the calculating unit22calculates the first difference between a current mean value and a past mean value of the luminance in the respective segments100, as a variation.

The calculating unit22then calculates a second difference that is a difference between a standard deviation of the luminance in a segment100included in the current captured image I, and a standard deviation of the luminance in the segment100that is at the same position in the past image. In other words, the calculating unit22calculates the second difference between the standard deviation of the past luminance and that of the current luminance in the corresponding segment100, as a variation. Hereinafter, the past captured image I will be sometimes referred to as a past frame I0, and the current captured image I will be sometimes referred to as a current frame I1.

Referring back toFIG. 2, the first ng unit23will now be explained. The first detecting unit23detects the first region A1in which the variation in the feature value calculated by the calculating unit22falls within a predetermined threshold range, and in which the feature value in the current captured image I falls within a predetermined threshold range.

Specifically, the first detecting unit23determines whether each of the segments100is a first region A1that satisfies a predetermined condition. In other words, the first detecting unit23detects the first region A1correspondingly to the segment100.

For example, the first detecting unit23determines that a segment100satisfies the predetermined condition if the conditions (1) to (3) described below are all satisfied, and detects the segment100as the first region A1. If at least one of the conditions (1) to (3) is not satisfied, the first detecting unit23determines that the segment100does not satisfy the predetermined condition, and detects the segment100as a region to which no adhering substance adheres.

(1) The first difference is equal to or less than a first predetermined difference;

(2) the second difference is equal to or less than a second predetermined difference; and

(3) the mean value: of the luminance in the segment100is equal to or less than a first predetermined value.

The condition (1) is a condition for determining whether the degree of change in the luminance is small in the same segment100. The first predetermined difference in the condition (1) sets an upper bound to the difference in the mean values of the luminance when the adhering substance adheres to, and is a difference that is set in advance based on experiments or the like. The first predetermined difference is “5”, for example.

The condition (2) is a condition for suppressing the impact of the gain adjustment of the camera10. The second predetermined difference in the condition (2) sets an upper bound to the difference in the luminance standard deviation when the adhering substance adheres to, and is a difference that is set in advance based on experiments or the like. The second predetermined difference is “1”, for example.

The condition (3) is a condition for determining whether the luminance of the segment100in the current frame I1is at a low level. The first predetermined value in the condition (3) is a value for allowing the presence of the adhering substance in the segment100to be determined, and is a value that is set based on the mean value of the luminance in the region of interest ROI included in the current frame I1, as illustrated inFIG. 4.

FIG. 4is a schematic illustrating a relation between the mean value of the luminance in the region of interest ROI and the first predetermined value. When the mean value of the luminance in he region of interest ROI is higher, the first predetermined value is set higher. An upper bound is set to the first predetermined value, and the first predetermined value is set to the upper bound when the mean value of the luminance in the region of interest ROI is equal to or higher than a predetermined mean value that is set in advance. In this manner, even if the mean luminance value changes due to the gain adjustment carried out by the camera10based on the background of the captured image I, the first region A1can be detected highly accurately.

By detecting a segment100that satisfies conditions (1) to (3) as a first region A1, the first detecting unit23can determine a segment100in which the adhering substance is present, correctly. Therefore, it is possible to improve the accuracy of the adhering substance region A12that is generated at the final step.

The first detecting unit23may also calculate a counter value indicating the continuity of the satisfactions of the conditions (1) to (3), for each of the segments100, and detect the segment100as the first region A1when the counter value becomes equal to or greater than a predetermined threshold. Specifically, the first detecting unit23increments the current counter value if it is determined that the segment100in the current frame I1satisfies the conditions (1) to (3), and decrements the current counter value if it is determined that the segment100in the current frame I1does not satisfy the conditions (1) to (3). In other words, the first. detecting unit23updates the counter value in the segment100.

The counter value has an upper-bound counter value and a lower-bound counter value that are set in advance. The value by which the counter value is incremented and decremented every time the determination is made may be the same or different.

Furthermore, the first detecting unit23may also perform the process of detecting the first region A1only when the speed of the vehicle satisfies a predetermined condition. For example, the first detecting unit23may be caused to perform the detection process if the speed of the vehicle is equal to or lower than a predetermined vehicle speed. The predetermined vehicle speed is a vehicle speed that is set in advance, and is a vehicle speed at which the camera10is able to capture a captured image I from which the adhering substance is detectable, with a small amount of blur in the captured image I. For example, the predetermined vehicle speed is 80 km/h. In this manner, she first region A1can be detected highly accurately.

Alternatively, the first detecting unit23may be configured to perform the detection process if the vehicle C is moving, specifically, if the vehicle speed is equal to or higher than a low vehicle speed that is set in advance. In this manner, it is possible to prevent the detection process to be performed repeatedly, when the vehicle C has stopped and the same captured image I has been captured repeatedly.

Furthermore, the first detecting unit23may be configured not to perform the process of detecting the first region A1if the current frame I1is a low-illuminance image, so that the generating unit25is not caused to generate the adhering substance region A12at the subsequent stage. A low-illuminance image is a captured image I that is captured while the environment around the vehicle C is dark, e.g., while the vehicle C is driving during the nighttime or inside a tunnel.

The first detecting unit23determines that the current frame I1is a low-illuminance image when the mean value of the luminance in the region of interest ROI included in the current frame I1is equal to or lower than a predetermined low illuminance value, and the standard deviation of the luminance in the region of interest ROI included in the current frame I1is equal to or less than a predetermined low illuminance deviation. The predetermined low illuminance value is a value that is set in advance, and is “85”, for example. The predetermined low illuminance deviation is also a value that is set in advance, and is “50”, for example.

In this manner, it is possible to suppress misdetection of an adhering substance adhering to the lens of the camera10, when the image is captured in a low illuminance environment. Furthermore, by not causing the adhering substance detection apparatus1to perform the detection process when the current frame I1is a low-illuminance image in which the adhering substance region A12may not be detected correctly, the processing load can be suppressed.

Referring back toFIG. 2, the second detecting unit24will now be explained. The second detecting unit24detects a second region A2in which the irregularity in the pixel luminance distribution in the captured image I satisfies a predetermined irregularity condition. Specifically, to begin with, the second detecting unit24extracts a candidate region200that is a candidate of the second region A2, from the captured image I acquired by the acquiring unit21. Specifically, to begin with, the second detecting unit24extracts luminance information of the pixels, and edge information from the captured image I. The luminance of each pixel is expressed as a parameter ranging from 0 to 255, for example.

The second detecting unit24also detect an edge in the X-axis direction (the right-and-left direction of the captured image I) and an edge in the Y-axis direction (up-and-down direction of the captured image I) from each of the pixels, by performing an edge detection process based on the luminance of the pixels. In the edge detection process, any edge detection filter such as a Sobel filter or a Prewitt filter may be used.

The second detecting unit24then detects, as the edge information, a vector including an edge angle and edge strength information of the pixel, using a trigonometric function based on the edge in the X-axis direction and the edge in the Y-axis direction. Specifically, an edge angle is expressed as an orientation of the vector, and an edge strength is expressed as a length of the vector.

The second detecting unit24then performs a matching process (template matching) for matching the detected edge information with template information representing the contours of adhering substance, and prepared in advance, and extracts edge information that is similar to the template information. The second detecting unit24then extracts the region corresponding to the extracted edge information, that is, the candidate region200that is a rectangular region. including the contour of a blurred region that is a second region A2.

Because the candidate region200is a rectangular region surrounding the region including the matching edge information, unlike the segment100described above, the candidate region200have various sizes depending on the matching result. Furthermore, a plurality of candidate regions200may overlap each other.

The second detecting unit24then extracts the luminance distribution corresponding to a predetermined pixel array that is included in the extracted candidate region200.FIG. 5is a schematic for explaining pixel arrays for which luminance distributions are extracted. As illustrated inFIG. 5, for each of the candidate regions200extracted from the captured image I, the second detecting unit24extracts the luminance distributions corresponding to three pixel arrays H1to H3in the horizontal direction, and of three pixel arrays V1to V3in the vertical direction.

The extracted pixel arrays may be the pixel arrays in at least one of the horizontal and the vertical directions. Furthermore, the number of pixel arrays to be extracted may be two or less, or four or more, without limitation to three.

The second detecting unit24then divides the extracted candidate region200into a predetermined number of unit regions, and calculates a representative value of the luminance, for each of the unit regions. The method by which the second detecting unit24calculates the representative value will be described later with reference toFIGS. 6 and 7.

The second detecting unit24converts the luminance of each pixel included in the candidate region200into a luminance unit representing a predetermined luminance range as a unit. For example, the second detecting unit24converts a parameter representing luminance within the range of 0 to 255 into a luminance unit that is a division of this parameter range at a predetermined interval.

FIGS. 6 and 7are schematics illustrating the details of the process performed by the second detecting unit24. To begin with, a method by which the second detecting unit24sets the unit regions will be explained with reference toFIG. 6.FIG. 6illustrates the distribution of the luminance in one of the horizontal pixel arrays H.

As illustrated inFIG. 6, the second detecting unit24divides the horizontal pixel array into eight unit regions R1to R8(sometimes collectively referred to as unit regions R), for example. Each of the unit regions R1to R8may have the same width (the same number of pixels) (that is, the number of pixels that is an equal division of the pixel array), or have different widths from those the others.

The number of unit regions R into which the pixel array is divided is not limited to eight, and may be set to any number. It is preferable to keep the number of the unit regions R into which the pixel array is divided constant (eight, inFIG. 4), regardless of the size of the candidate regions200extracted from the captured image I. In this manner, although the extracted candidate regions200have various sizes, unified information can be obtained by keeping the number of unit regions R constant. Therefore, it is possible to suppress the processing load in the subsequent processes such as a determination process.

The second detecting unit24then calculates a representative value of the luminance in each of the unit regions R, as illustrated inFIG. 7. As illustrated in the top graph inFIG. 7, the second detecting unit24converts the luminance value of each pixel (e.g., ranging from 0 to 255) into a luminance unit, before calculating the representative value. Specifically, inFIG. 7, the range 0 to 255 is divided into eight luminance units. The luminance units are denoted as “0” to “7”, respectively, the middle graph inFIG. 7. In this example, the width of the luminance values is divided at an interval of 32. For example, the luminance unit “0” corresponds to the luminance values 0 to 31, and the luminance unit “1” corresponds to the luminance values 32 to 63. In other words, this conversion into a luminance unit is a process of reducing the luminance resolution. In this manner, the resolution of the luminance distribution can be reduced to the number of desirable luminance units. Therefore, it is possible to reduce the processing load in the subsequent processes. In the conversion to a luminance unit, the number of luminance units into which the luminance range is divided, and the width of the luminance unit may be set any way. Furthermore, it is not necessary for the luminance units to have equal widths.

The second detecting unit24then creates a histogram of luminance units, for each of the unit regions R1to R8. The middle graph inFIG. 7illustrates the histogram for the unit region R1. In this histogram, the luminance units “0” to “7” are assigned as the classes, and the number of pixels are assigned as the frequency.

The second detecting unit24then calculates, for each of the unit regions R1to R8, a representative luminance value based on the created histogram, as illustrated in the bottom graph inFIG. 7. For example, the second detecting unit24calculates the luminance unit corresponding to the class with the highest frequency in the histogram (class “3” inFIG. 7) as the representative luminance value in the unit region R1. In this manner, because the number of data pieces in the luminance distribution can be reduced, from the number of pixels to the number unit regions R, the processing load in the subsequent processes can be reduced.

The second detecting unit24has been explained to calculate the luminance unit appearing at the highest frequency as the representative value, but without limitation thereto, the median, the mean value, and the like in the histogram may also be used as the representative value.

Furthermore, without limitation to the calculation of the representative value based on the histogram, the second detecting unit24may also calculate a mean value from the luminance values, for each of the unit regions R, and use the luminance unit corresponding to the mean value as the representative luminance value, for example.

Furthermore, the second detecting unit24has been explained to use a luminance unit as the representative value, but may also use the mean value or the like of the luminance values in the unit region R, as the representative value, as it is. In other words, the representative value may be expressed as a luminance unit or as a luminance value.

The second detecting unit24then determines whether the candidate region200is a second region A2based on the irregularity in the pixel luminance distribution in the candidate region200. The determination process performed by the second detecting unit24will now be explained with reference toFIGS. 8 and 9.

FIGS. 8 and 9are schematics illustrating the details of the process performed by the second detecting unit24. The top graph inFIG. 8represents a luminance distribution in the candidate region200, and the representative values corresponding to the respective unit regions R1to R8are indicated on the white background inside of the respective bars.

To begin with, as illustrated in the upper part ofFIG. 8, the second detecting unit24calculates amounts of change D1to D7between the adjacent unit regions R1to R8in the luminance units. A table containing the amounts of change D1to D7is indicated in the lower part ofFIG. 8(upper table).

If the pattern of change followed by the irregularity in the luminance distribution satisfies a predetermined pattern of change, the second detecting unit24determines that the candidate region200is a second region A2. Specifically, the second detecting unit24performs this determination process by comparing each of the amounts of change D1to D7with the irregularity condition information31stored in the storage unit3.

As an example of the irregularity condition information31, an example of a table containing threshold ranges for the respective amounts of change D1to D7is indicated in the lower part ofFIG. 8(lower table). If each of the amounts of change D1to D7in the candidate region200falls within the corresponding threshold range specified for the amount of change D1to D7in the irregularity condition information31, the second detecting unit24determines that the candidate region200is a second region A2.

In other words, if the amounts of change D1to D7between the adjacent unit regions R1to R8in the luminance units satisfy the pattern of change specified as the threshold ranges in the irregularity condition information31, the second detecting unit24determines that the candidate region200is the second region A2.

In other words, before the second detecting unit24performs the determination process, the feature of blurred regions, the feature being such the luminance gradually becomes higher (or lower) toward the center of the candidate region200, is stored as a threshold range in the irregularity condition information31. In this manner, the second detecting unit24can detect a blurred region caused by the adhesion of water, as a second region A2.

Furthermore, with the use of the amounts of change D1to D7, the second detecting unit24can ignore the difference in the scales of the luminance values. Therefore, it is possible to reduce the number of erroneous determinations made when the shapes of the irregularities are similar, but the luminance values are different in scales. Furthermore, because the scales of the luminance values can be ignored, it is not necessary to establish a determination condition for each of the luminance values. Therefore, the storage capacity for storing the conditions can be reduced. Furthermore, because it is not necessary to make the determination for each of the luminance values, the processing burden can be reduced.

Furthermore, by specifying the amounts of change D1to D7with some widths by setting the maximum and the minimum thereto in the irregularity condition information31, even if the adhering substance has a distorted shape, such a region can be detected as an adhering substance region. In other words, even when the adhering substance has different shapes, such regions can be detected as adhering substance regions highly accurately.

Illustrated inFIG. 8is an example in which the threshold range is set to each of the amounts of change D1to D7in the irregularity condition information31. However, when detected are small-sized second regions A2, it is possible to set the threshold range only for some of the amounts of change D1to D7.

Furthermore, explained inFIG. 8is an example in which the second detecting unit24determines whether the amounts of change fall within the respective threshold ranges specified in the irregularity condition information31, but the second detecting unit24may also perform the determination process based on the irregularity condition information31in which the irregularity in the luminance distribution is mapped based on the threshold ranges corresponding to the amounts of change D1to D7, for example. This point will be now explained with reference toFIG. 9.

The table in the upper part ofFIG. 9indicates the threshold ranges corresponding to the amounts of change in the irregularity in the luminance distribution. The diagram in the lower part ofFIG. 9illustrates the irregularity condition information31that is mapping of the threshold ranges corresponding to the amounts of change D1to D4illustrated in the upper part ofFIG. 9. Specifically, the diagram in the lower part ofFIG. 9provides a map in which the horizontal axis represents the positions of the unit regions R1to R8, and the vertical axis represents the relative luminance. Such a map is generated in advance.

For example, the amount of change Di is specified as a threshold range of +1 to +2, so two squares at predetermined positions in the relative luminance are set as the threshold for the unit region R1. For the unit region R2, one square at a position satisfying the threshold range of the amount of change D1is set as the threshold. The amount of change D2is specified with a value +1, so a square at the level immediately above the square set for the unit region R2is set as the threshold for the unit region R3. The amount of change D3has a value of −1, so the square at the level immediately below the square set for the unit region R3is set as the threshold for the unit region R4. The amount of change D4is specified with a threshold range from −2 to −1, so the two squares at the level immediately below the square set for the unit region R4are set as the threshold for the unit region R5. By following these steps, the mapping of the irregularity condition information31is completed.

In other words, the map specified in the irregularity condition information31is information representing the shape of the irregularity in the luminance units in the unit regions R1to R5, mapped based on the amounts of change D1to D4. Because no threshold range is set for the amounts of change D5to D7, any luminance to be detected is acceptable for the unit regions R6to R8.

The second detecting unit24creates a map, based on the amounts of change D1to D7in the unit regions R1to R8included in the extracted candidate region200, following the same steps as those described above, performs a matching process of matching the map with the map of the irregularity condition information31, and determines that the candidate region200is a second region A2if the maps match.

In the example illustrated inFIG. 9, if the map based on the candidate region200has an inverted V-shape as that of the map of the irregularity condition information31, the second detecting unit24determines that the candidate region200is a second region A2that is a blurred region in which the luminance becomes lower from the center toward the periphery. By contrast, if the map based on the candidate region200has a V-shape as that of the map of the irregularity condition information31, the second detecting unit24determines that the candidate region200is a second region A2that is a blurred region in which the luminance becomes higher from the center toward the periphery.

In other words, if the irregularity in the luminance distribution in the candidate region200has an inverted V-shape or a V shape, the second detecting unit24determines that the candidate region200is a second region A2. In this manner, because the determination process can be performed depending only on the shape of the irregularity, with the factor of the luminance values (luminance units) removed, missed detection due to the scales of the luminance values can be reduced. Therefore, an adhering substance can be detected highly accurately.

If the second detecting unit24keeps determining that the candidate region200is a second region A2continuously, based on the captured images I captured in the temporal order, the second detecting unit24may determine that candidate region200is a region ascertained as a second region A2.

Specifically, every time the second detecting unit24performs a determination process as to whether a candidate region200is a second region A2, the second detecting unit24assigns a score corresponding to the determination result to the candidate region200, for each of a plurality of candidate regions200, and determines the candidate region200having a total score satisfying a predetermined threshold condition as a region ascertained as a second region A2.

More specifically, if the second detecting unit24determines that a candidate region200is a second region A2, the second detecting unit24adds a predetermined value to the total score. If the second detecting unit24determines that the candidate region200is not a second region A2, the second detecting unit24subtracts a predetermined value from the total score. The same predetermined value may be used for both of the addition and the subtraction, or different predetermined values may be used for the addition and the subtraction.

The second detecting unit24then performs a conversion process of converting the detected second region A2into a region having a size corresponding to the segments100. This point will now be explained with reference toFIG. 10.

FIG. 10is a schematic illustrating the conversion process of the second region A2, performed by the second detecting unit24. The upper part ofFIG. 10illustrates the second regions A2detected from the region of interest ROI included in the captured image I. The lower part ofFIG. 10illustrates the segments100set to the region of interest ROI. The number and the layout of the segments100into which the second region A2is to be converted by the second detecting unit24are matched with those of the segments100set by the first detecting unit23(seeFIG. 3).

As illustrated inFIG. 10, the second detecting unit24superimposes the detected second region A2(hereinafter, sometimes referred to as an original second region A2) over the segments100, and generates segments100overlapping with the original second region A2as a converted second region A2(sometimes referred to as a new second region A2). In the generation of a new second region A2, it is preferable to consider not only the condition that the segments100corresponding thereto are blurred regions, but also the condition that there has been no variation in the feature value in the segments100. More specifically, it is preferable to generate a segment100that satisfies the condition (1) mentioned above, among the conditions (1) to (3) used in the calculations of the regions A1, and that overlaps with the original second region A2, as a new second region A2. The condition (1) mentioned above stipulates that the luminance at the segment100has gone through little variation. In this manner, an adhering substance, such as a mixture of a water drop and mud, can be detected effectively, even if such a region is blurred, has gone through little luminance variation, but is not represented as a black spot without any gradation.

Specifically, if a segment100is occupied by the original second region A2by a ratio equal to or higher than a predetermined threshold, the second detecting unit24generates the segment100as a new second region A2. In other words, the second detecting unit24detects the second region A2correspondingly to the segments100.

Referring back toFIG. 2, the generating unit25will now be explained. The generating unit25generates a sum region that is the sum of the first regions A1detected by the first detecting unit23, and the second regions A2detected by the second detecting unit24, as an adhering substance region A12. The details of this process performed by the generating unit25will now be explained with reference toFIG. 11.

FIG. 11is a schematic illustrating the details of the process performed by the generating unit25. As illustrated inFIG. 11, the generating unit25generates a region that is a logical sum of the first region A1corresponding to the segments100and the second region A2corresponding to the segments100, as the adhering substance region A12. In other words, the generating unit25generates an adhering substance region A12correspondingly to the segments100. Without simply taking a logical sum, the condition (1) may be taken into consideration, as mentioned earlier, at this point in time. Specifically, the generating unit25may generate a new second region A2merely by taking over the original second regions A2as they are, and exclude the segments100not satisfying the condition (1) in acquiring the sum for the new second region A2.

In this manner, by using the adhering substance region A12, the first region A1, and the second region A2all of which correspond to the segments100, the adhering substance region A12can be generated using the unified information. Therefore, complication of the process can be avoided. Furthermore, by setting a plurality of pixels as a segment100, the units in which the processes are performed are changed from the pixels to the segments100. In this manner, the number of times the process is performed can be reduced, so that the processing load can be reduced.

The generating unit25then calculates an occupied ratio that is a ratio of the region of interest ROI occupied by the adhering substance region A12. If the occupied ratio is equal to or higher than a predetermined threshold (e.g., 40%), the generating unit25generates an adhering substance flag ON, and outputs the signal to the flag output unit27.

If the occupied ratio is equal to or higher than the predetermined threshold (e.g., 40%), the generating unit25also calculates the feature value related to the luminance in the adhering substance region A12, and outputs the feature value to the removal determining unit26

Referring back toFIG. 2, the removal determining unit26will now be explained. The removal determining unit26determines whether the adhering substance has been removed, based on the variation in the feature value related to the luminance in the adhering substance region A12generated by the generating unit25. The details of this process performed by the removal determining unit26will now be explained specifically, with reference toFIGS. 12 and 13.

FIGS. 12 and 13are schematics illustrating the details of the process performed by the removal determining unit26. InFIG. 12, it is assumed that the occupied ratio occupied by the adhering substance region A12has reached a level equal to or higher than the predetermined threshold at time t1.

The removal determining unit26calculates a variation between a feature value at the time at which the adhering substance region A12is generated, and a feature value that is based on the current captured image I, and determines that the adhering substance has been removed when the variation continues to remain at a level equal to or higher than the predetermined threshold.

Specifically, every time a new captured image I is received, the removal determining unit26calculates the feature value related to the luminance of a determination region A120included in the new captured image I, and calculates a difference between the feature value in the determination region A120and that in the adhering substance region A12. In the example illustrated inFIG. 12, the removal determining unit26calculates the difference between the feature value in the adhering substance region A12at the time t1, and the feature value in the determination region A120at time t2, as the variation, and calculates the difference between the feature value in the adhering substance region A12at the time t1, and the feature value in the determination region A120at time t3, as the variation.

In other words, the removal determining unit26determines that the adhering substance has been removed only based on the variation in the feature value in the adhering substance region A12(the determination region A120), not through the detection processes performed by the first detecting unit23and the second detecting unit24.

In this manner, because the accuracy of the removal determination does not depend on the detection results of both of the first detecting unit23and the second detecting unit24, determination errors in the removal determinations can be reduced.

The removal determining unit26may determine that the adhering substance has been removed if the number of times the condition “variation≥threshold” is satisfied has become equal to or more than a predetermined number of times, or may calculate a score for each determination result “variation≥threshold”, and determine whether the adhering substance has been removed based on the score. This point will now be explained, with reference toFIG. 13.

inFIG. 13, the vertical axis represents the score indicating the continuity related to the removal determinations, and the horizontal axis represents time. It is assumed herein that the time t1is the time at which the occupied ratio occupied by the adhering substance region A12has become equal to or higher than the predetermined threshold. The arrow in the solid line indicates a determination result of “variation≥threshold”, and the arrow in the dotted line indicates a determination result of “variation<threshold”.

As illustrated inFIG. 13, the removal determining unit26assigns an initial value to the score of the adhering substance region A12at the time t1, subtracts the score when it is determined that “variation≥threshold”, and maintains the score when it is determined that “variation<threshold”.

If the score drops to a level lower than a predetermined removal threshold within a predetermined time period D between the time t1and time tn, the removal determining unit26determines that the adhering substance corresponding to the adhering substance region A12has been removed. The predetermined time period D is a time period that is set in advance, and is a time period allowing a determination to be made as to whether a removing operation has been performed. In this manner, if the condition “variation≥threshold” remains being satisfied over the predetermined time period D, it can be determined that the adhering substance has been removed. Therefore, it is possible to avoid making an erroneous determination when “variation≥threshold” is satisfied temporarily due to the noise in the captured image I, for example. In other words, the removal determination can be performed highly accurately.

When the predetermined time period D expires while the score is at a level equal to or higher than the removal threshold, the removal determining unit26sets the score to the initial value again.

When it is determined that the adhering substance corresponding to the adhering substance region A12has been removed, the removal determining unit26generate an adhering substance flag OFF, and outputs the signal to the flag output unit27.

Referring back toFIG. 2, the flag output unit27will now be explained. Upon receiving an adhering substance flag ON from the generating unit25, the flag output unit27outputs the adhering substance flag ON indicating that the adhering substance adheres to, to the various devices50. Upon receiving an adhering substance flag OFF from the removal determining unit26, the flag output unit27outputs the adhering substance flag OFF indicating that no adhering substance adheres to, to the various devices50.

in other words, the information indicating whether the adhering substance flag is ON or OFF serves as information indicating validity of whether the various devices50can use the captured image I corresponding to the current frame, or as information indicating the reliability of control performed by the various devices50using the captured image I. Therefore, instead of the information of the adhering substance flag, the second detecting unit24may also output information indicating the validity or the reliability of the captured image I to the various devices50.

The sequence of the adhesion detecting process performed by the adhering substance detection apparatus1according to the embodiment will now be explained with reference toFIG. 14.FIG. 14is a flowchart illustrating the sequence of the adhesion detecting process performed by the adhering substance detection apparatus1according to the embodiment.

As illustrated inFIG. 14, to begin with, the acquiring unit21acquires an image captured by the camera10, applies a gray-scaling process and a decimation process to the acquired image, and acquires an integral image generated based on the pixel values of the decimated image, as a captured image I (S101).

The calculating unit22calculates a variation in the feature value related to the luminance of the current and the past captured images I, based on the luminance of the pixels included in the captured images I (S102).

The first detecting unit23then detects a first region A1in which the variation calculated by the calculating unit22falls within a predetermined threshold range, and in which the feature value in the current captured image I falls within a predetermined threshold range (S103).

The second detecting unit24then detects a second region A2in which the irregularity in the pixel luminance distribution in the captured image I satisfies a predetermined irregularity condition (S104). The generating unit25then generates a sum region that is the sum of the first region A1detected by the first detecting unit23and the second region A2detected by the second detecting unit24, as an adhering substance region A12(S105).

The flag output unit27then outputs the adhering substance flag ON input from the generating unit25to the various devices50(S106), and the process is ended.

The sequence of the removal determination. process performed by the adhering substance detection apparatus1according to the embodiment will now be explained with reference toFIG. 15.FIG. 15is a flowchart illustrating the sequence of the removal determination process performed by the adhering substance detection apparatus1according to the embodiment.

As illustrated inFIG. 15, the generating unit25calculates the feature value of the adhering substance region A12(S201). The acquiring unit21then acquires the captured image I (S202).

The removal determining unit26calculates the feature value in the determination region A120, which corresponds to the adhering substance region A12(S203). The removal determining unit26determines whether the variation that is the difference between the feature value of the adhering substance region A12and the feature value of the determination region A120is equal to or higher than a predetermined threshold (S204).

If the variation is equal to or greater than a predetermined threshold (Yes at S204), the removal determining unit26subtracts a predetermined value from the initial score (S205). The removal determining unit26then determines whether the score is less than the removal threshold (S206).

If the score is less than the removal threshold (Yes at S206), the removal determining unit26determines that the adhering substance corresponding to the adhering substance region A12has been removed (S207). The flag output unit27then outputs the adhering substance flag OFF input from the removal determining unit26to the various devices50(S208), and the process is ended.

If the removal determining unit26determines that the variation less than the predetermined threshold at Step S204(No at S204), the flag output unit27outputs the adhering substance flag ON (S209), and the process is ended.

If the score is equal to or higher than the removal threshold at Step S206(No at S206), the removal determining unit26performs Step S209, and the process is ended.

As described above, the adhering substance detection apparatus1according to the embodiment includes the calculating unit22, the first detecting unit23, the second detecting unit24, and the generating unit25. The calculating unit22calculates a variation in the feature value related to the luminance in the past and current captured images I captured by the camera10, based on the luminance of the pixels included in the captured images I. The first detecting unit23detects a first region A1in which the variation calculated by the calculating unit22falls within a predetermined threshold range and in which the feature value in the current captured image I falls within a predetermined threshold range. The second detecting unit24detects a second region A2in which the irregularity in the luminance distribution of the pixels included in the captured image I satisfies a predetermined irregularity condition. The generating unit25generates a sum region that is the sum of the first region A1detected by the first detecting unit23and the second region A2detected by the second detecting unit24, as an adhering substance region A12corresponding to the adhering substance adhering to the camera10. In this manner, adhering substance can be detected highly accurately.

Furthermore, explained in the embodiment above is an example in which the captured images I captured with a camera provided on board a vehicle is used, but the captured images I may be those captured by a surveillance camera or a camera installed on a street light, for example. In other words, the captured images I may be any captured images I that are captured with a camera on which some adhering substance can adhere to the lens of the camera.

According to the present invention, an adhering substance can be detected highly accurately.