Selecting visible regions in nighttime images for performing clear path detection

A method provides for determining visible regions in a captured image during a nighttime lighting condition. An image is captured from an image capture device mounted to a vehicle. An intensity histogram of the captured image is generated. An intensity threshold is applied to the intensity histogram for identifying visible candidate regions of a path of travel. The intensity threshold is determined from a training technique that utilizes a plurality of training-based captured images of various scenes. An objective function is used to determine objective function values for each correlating intensity value of each training-based captured image. The objective function values and associated intensity values for each of the training-based captured images are processed for identifying a minimum objective function value and associated optimum intensity threshold for identifying the visible candidate regions of the captured image.

BACKGROUND OF INVENTION

An embodiment relates generally to road recognition.

Vision-imaging systems are used in vehicles for enhancing sensing applications within the vehicle such as clear path detection systems, object detection systems, and other vision/positioning systems. Such systems utilize a camera to capture the image. For an image captured during a nighttime lighting condition, a path of travel may not be readily distinguishable from non-road and other unidentified regions of captured image utilizing the vision-based capture system that may be due to insufficient illumination. If the vision-based camera system cannot distinguish between the road of travel and non-road of travel, then secondary systems that utilize the captured image information become ineffective.

SUMMARY OF INVENTION

An advantage of an embodiment is determination of a visible region of travel utilizing by selecting a modeling technique that provides the least error among alternative techniques. The method utilizes training images for identifying an objective function that is representative of identifying a visible region from all training images. The identified visible regions are utilized for detecting a clear path. Based on the objective function, an objective function value-intensity correlation graph is generated. The correlation graph is used by a plurality of modeling techniques for determining which technique will produce an intensity threshold with the least error among the alternatives. The threshold will be used on captured images when the system is utilized in a real-time environment for identifying visible regions.

An embodiment contemplates a method of determining visible regions in a captured image during a nighttime lighting condition. An image is captured from an image capture device mounted to a vehicle. An intensity histogram of the captured image is generated. An intensity threshold is applied to the intensity histogram for identifying visible candidate regions of a path of travel. The intensity threshold is determined from a training technique that utilizes a plurality of training-based captured images of various scenes. An objective function is used to determine objective function values for each correlating intensity value of each training-based captured image. The objective function values and associated intensity values for each of the training-based captured images are processed for identifying a minimum objective function value and associated optimum intensity threshold for identifying the visible candidate regions of the captured image.

DETAILED DESCRIPTION

There is shown a block diagram of an imaging capture system used in a vehicle10for performing clear path detection at night. The vehicle includes an image capture device11, a vision processing unit12, and an output application13.

The image capture device11includes a camera or video camera which images of the environment exterior of the vehicle10are obtained and stored for processing. The image capture device11is mounted on the vehicle so that the desired region along the path of travel is captured. Preferably, the image capture device11is mounted just behind the front windshield for capturing events occurring exterior and forward of the vehicle; however, the image capture device may be mounted at any other location suitable for capturing images. The image capture device11is part of an existing system in the vehicle that is typically used for recognition of road marking, lane markings, road signs, or other roadway objects used in lane departure warning systems and clear path detection systems. The captured images from the image capture device11are also used to distinguish between a daytime lighting condition and a nighttime lighting condition.

The vision processing unit12receives the images captured by the image capture device11and analyzes the images for identifying a visible region in the path of travel of the vehicle10. The visible region may be used to determine clear path, pedestrian objects, or other obstructions. Details of the processing and analyzing the captured images will be discussed in detail herein. An output application13includes any device or application that utilizes the identified visible region in the path of travel for enhancing the driver's awareness to the clear path of travel or other applications that utilize the clear path to assist the driver with nighttime enhancement operations. For example, the output application13may be a warning system that notifies the driver of an object in the path of travel.

The system utilizes the vision processing unit12for determining a visible region in the image during a nighttime condition. A training technique is executed for determining an intensity threshold that can be applied to the captured image for identifying those pixels of the image that represent a visible region. After the threshold is identified, the intensity threshold is applied to an intensity histogram representing the captured image for identifying the visible region in the image.

FIG. 2illustrates a flowchart for identifying the visible region of the path of travel for a captured image. In step20, an image is captured by the capture image device of an environment exterior of the vehicle.FIG. 3illustrates an exemplary image captured by the vision-based imaging system. The image is shown in clarity for illustrating objects in the image; however, typically an image captured during a nighttime would have visible and non-visible regions. A region of interest is selected within the image for analyzing the lighting condition. If a nighttime condition is determined by the vision imaging system, then the routine proceeds to step21for determining the visible region in the path of travel.

In step21, an intensity histogram of the captured image is generated.FIG. 4illustrates an example of an intensity histogram. The x-axis represents the intensity value, and the y-axis represents the number of the pixels for each pixel intensity value in an image. The region of interest of a respective image is analyzed. The histogram is generated based on the intensity of each of the plurality of pixels. Each pixel within the region of interest has an associated intensity value. Each of the intensity values of the image are represented within the histogram.

In step22, an intensity threshold is applied to the intensity histogram for identifying those intensity values associated with a visible region of the captured image. For example, inFIG. 4, the intensity threshold is represented by threshold line14. The intensity values greater than the threshold line14are designated as pixels that represent the visible region. The intensity values smaller than the threshold line14are not considered part of the visible region.

In step23, a candidate visible region is identified within the captured image.FIG. 5illustrates a candidate visible region of the captured image as determined by technique described herein. The darkened regions illustrate the visible candidate region of the image as identified by the intensity values of the pixels being greater than the intensity threshold value.

In step24, smoothing is applied to the candidate visible region. The smoothing operation may include open and close smoothing operations. Any known smoothing operation may be utilized for identifying the smoothed visible region from the candidate region. Opening and closing morphological smoothing may be cooperatively applied where opening smoothes the targeted region internally and closing smoothes the targeted region externally. Open operations smooth the targeted region by eliminating narrowing sections that connects larger sections, eliminates small protrusions such as corners, and generates new gaps. Closing operations smooth the targeted region by fusing narrow breaks between large areas of the targeted region and fills gaps in the targeted region.FIG. 6illustrates the morphological smoothing operation applied to the candidate visible region ofFIG. 5.

In step25, illumination normalization is performed for enhancing the contrast of the image for better image classification.

FIG. 7illustrates a training technique for determining the intensity threshold applied to the capture image.

In step30, a plurality of training-based images is captured by an image capture device during a night time condition. The training-based images represent various scenes from different nighttime lighting conditions and locations.

In step31, an objective function is generated that is used in determining an intensity threshold for identifying a visible region in an image captured from an image capture device. The objective function is generated in a training phase. The objective function is formulated based on observations of training-based captured images. The observations may be that of an observer analyzing the images and applying a ground-truth labeling to the captured images. An example of ground truth labeling may include labeling regions of a training image as a road label region, a non-road label region, and an un-labeled region.

FIG. 8illustrates an example of ground truth labeling based on observations. The region designated as a road label is represented generally by15, a non-road label is represented generally by16, and the remaining regions which are designated as un-labeled are represented by17. Based on the logic observations of the image inFIG. 8and observations of other images, an objective function is generated that can be used for determining a night-time visibility threshold estimation. An exemplary formula based on the logic observation inFIG. 8.
f(x=w1·(1−rdvis(x))+w2·(1−unLvis(x))+w3·grdmaginvis(x)  eq. (1)
where x is the intensity threshold of the visible region of interest in the sampled nighttime captured images, rdvisis the ratio of labeled road areas in a visible region of interest in the sampled captured images over a total labeled region in the sampled nighttime captured images, unLvisis the ratio of unlabeled areas classified as being invisible over a total unlabeled region area in the sampled nighttime captured images, grdmaginvisis a sum of a gradient magnitude in the invisible region, and w is the weights of each component in the objective function.

Based on the determined objective function, an optimal intensity value threshold may be identified utilizing the generated objective function.

In step32, an intensity threshold identification process is initiated for minimizing objective values generated by the objective function. Various threshold identification techniques may be utilized for generating the intensity value threshold based on the training-based images. An intensity threshold and corresponding mean squared error associated with the respective result is calculated for each technique. The various methods are represented generally by steps33-36. It should be understood that more or less techniques as described herein may be utilized. Each of these respective techniques will be described in detail later.

In step37, the mean squared error calculated from each of the techniques in steps33-36are compared for identifying the technique producing the lowest mean squared error.

In step38, the optimal intensity value threshold associated with the lowest mean square error is selected.

In step39, the intensity value threshold selected in step38is used to identify the visible region in the captured image. That is, for a respective imaging system, once calibrated and placed into production within a vehicle or other environment, the selected intensity threshold is applied to an intensity histogram generated for any night-time image captured by the vehicle image capture device for identifying those respective intensity values representing the visible region in the captured image. The visible region is thereafter utilized for detecting a clear path in the path of travel of the vehicle. An example of the histogram is shown inFIG. 4. The respective intensity values greater than the intensity threshold14represent those image pixels associated with the visible region of the captured image.

FIG. 9illustrates a flowchart of a first method (as represented by step33inFIG. 7) for identifying the intensity threshold. The method illustrated inFIG. 9identifies the intensity threshold utilizing a fixed intensity threshold technique for all training-based images.

In step40, an objective function value-intensity value correlation graph is generated for all training images utilizing the objective function. An exemplary plot is illustrated inFIG. 10. As a result, each plotted line represents a respective image. Each plotted line is comprised of objective function values calculated from the objective function values and the intensity values of each respective image. The x-axis designates the intensity values and the y-axis designates the objective function values.

In step41, the objective function values for each correlating intensity value are summed for generating a summed objective function value plotted line. That is, for a respective intensity value illustrated inFIG. 10, each of the function values of all the images for the respective intensity value is summed. The resulting plot is illustrated inFIG. 11.

In step42, the minimum objective function value is identified as the lowest of the summed objective function values shown inFIG. 11.

In step43, the intensity value threshold (e.g., 48) is identified as the intensity value associated with the identified minimum objective function value. In addition to identifying the intensity value threshold from the graphs, the intensity value threshold may be determined by the following formula:

Ithr_fix=argmin⁡(∑s=1N⁢⁢f⁡(x,s))eq.⁢(2)
where s is a respective image and f(x, s) are the objective function values for an associated image and intensity value.

In step44, a mean squared error based on the fixed intensity threshold technique is determined. For the fixed intensity threshold technique described herein, the mean squared error may be determined by the following formula:

MSEI=∑s=1N⁢⁢(I⁡(s)-Ithr_fix)2/N.eq.⁢(3)
The mean square error is compared to the mean squared errors determined from other techniques (generated in steps33-36ofFIG. 7) for determining which training technique to use.

FIG. 12illustrates a flowchart of a second method (as represented by step34inFIG. 7) for identifying intensity threshold. The second method utilizes a fixed intensity percentage technique.

In step50, the objective function value-intensity correlation graph is generated for all training-based images utilizing the objective function shown in eq. (1). It should be understood that the objective function value-intensity correlation graph will be the same for each technique since each of the methods utilize the same objective function and training images.

In step51, a cumulative intensity histogram is generated for each training-based captured image. An example of a cumulative histogram is illustrated inFIG. 13. The cumulative intensity histogram illustrates the cumulative percentage of the intensity values of the pixels within a respective image. The x-axis represents intensity values and the y-axis represents a cumulative percentage.

In step52, an objective function value-percentage based intensity correlation chart is generated for each training-based image. An example of the correlation chart is illustrated inFIG. 14. The correlation chart identifies each percentage-based intensity value in each image with a respective objective function value. The x-axis represents the percentage of the intensity values and the y-axis represents the objective function values.

In step53, the objective function values associated with each percentage-based intensity value are averaged for generating an averaged objective function value plot line. That is, for a respective percentage-based intensity value illustrated inFIG. 14, each of the objective function values of all the images for the respective percentage based intensity values is averaged. The resulting plot is illustrated inFIG. 15. The x-axis represents the percentage of intensity values and the y-axis represents averaged objective function values.

In step54, the minimum objective function value is identified from the averaged objective function values. The minimum objective function value is the lowest of the averaged objective function values (e.g. 0.75).

In step55, the respective percentage-based intensity value is determined. The percentage-based intensity value (e.g., 0.58) is the intensity value associated with the minimum objective function value identified in step54.

In step56, a captured image is obtained from the capture image device. A cumulative intensity histogram is generated for the captured image as shown inFIG. 16. The x-axis represents intensity values and the y-axis represents the percentage-based intensity value. The percentage-based intensity value identified in step55is used to identify the intensity threshold. Utilizing the plotted curve (i.e., cumulative intensity histogram), an associated intensity threshold (e.g., 48) is identified for the associated percentage-based intensity value (e.g., 0.58). As an alternative to identifying the intensity threshold from correlation graph as illustrated, the intensity value threshold may be determined by the following formula:

Rthr_fix=argmin⁡(∑s=1N⁢⁢f⁡(hs-1⁡(r),s))eq.⁢(4)
where s is a respective image, and f(hx−1(r),s) is the objective function values based on the percent-based intensity values.

In step57, the mean squared error is determined based on the data from the fixed intensity percentage technique. For the fixed intensity percentage technique described herein, the mean square error may be determined by the following formula:

MSEI=∑s=1N⁢⁢(I⁡(s)-hs-1⁡(Rthr_fix)2/N.eq.⁢(5)
The mean square error is compared to the mean squared errors determined from other techniques (generated in steps33-36ofFIG. 7) for determining which training technique to use.

FIG. 17illustrates a flowchart of a third method (as represented by step35inFIG. 7) for identifying intensity threshold. The third method utilizes an intensity threshold regression-based technique for all training-based images.

In step60, an objective function value-intensity value correlation graph is provided for all training images utilizing the objective function as shown in eq. (1). The objective function value-intensity correlation graph is the same graph utilized for each technique since each of the methods utilizes the same objective function and training images.

In step61, a minimum objective function value is identified for each training image. An exemplary graph illustrating the minimum objective function value for each training image is shown inFIG. 18. The x-axis represents the respective training image and the y-axis represents the objective function value.

In step62, the intensity value associated with each minimum objective function value is determined. An exemplary graph illustrating the intensity value for each training image is shown inFIG. 19. The x-axis represents the respective training image and the y-axis represents the intensity value.

In step63, regression parameters are determined for the regression analysis in the training phase. The intensity regression-based threshold may be determined by the following formula:

In step64, a mean squared error is determined for each regression technique applied as a function of the known objective function value, of the respective cumulative intensity histogram for each respective training image utilized, and of the determined parameters a0-aM.

In step65, the regression technique having the lowest mean square error is identified.

In step66, a cumulative histogram for the captured image is generated.

In step67, the cumulative intensity histogram cumh(m, s) and parameters determined from the training session a0-aMare used to solve for the intensity threshold.

FIG. 20illustrates a flowchart of a fourth method (as represented by step36inFIG. 7) for identifying intensity threshold. The fourth method utilizes an intensity percentage regression-based technique.

In step70, an objective function value-intensity value correlation graph is provided for all training images utilizing the objective function as shown in eq. (1). The objective function value-intensity correlation graph is the same graphs utilized for each technique since each of the methods utilize the same objective function and training images.

In step71, an objective function value-percentage-based correlation graph is generated based on the objective function value-intensity value correlation graph.

In step72, a minimum objective function value is identified for each training image. An exemplary graph illustrating the minimum objective function value for each training image is shown inFIG. 18.

In step73, the intensity percentage value associated with each minimum objective function value is determined. An exemplary graph illustrating the intensity percentage value for each training image is shown inFIG. 21. The x-axis represents the respective training image and the y-axis represents the intensity percentage value.

In step74, regression parameters are determined for the regression analysis in the training phase. The regression-based intensity percentage may be determined by the following formula:

In step75, a mean squared error is determined for the intensity-percentage value data of each regression technique applied as a function of the known objective function value, the respective cumulative intensity histogram for each respective image utilized, and the determined parameters b0-bM.

In step76, the regression technique having the lowest mean square error is identified.

In step77, a cumulative intensity histogram for the captured image is generated.

In step78, the cumulative intensity histogram cumh(m, s) and parameters determined from the training session b0-bMare used to solve for the percentage-based intensity threshold.

In step79, the intensity threshold is determined based on the correlating percentage-based intensity threshold from the cumulative intensity histogram as shown inFIG. 16.