Surrounding view camera blockage detection

A camera monitoring system for a vehicle includes a plurality of cameras disposed around an exterior of a vehicle. An image processing module communicates with the plurality of cameras, generating overhead view images from raw images taken by at least one of the plurality of cameras. A histogram module communicates with the image processing module, generating at least one histogram from the overhead view images. A likelihood module communicates with the histogram module and determines a likelihood of blockage for at least one of the plurality of cameras. A line alignment module communicates with the likelihood module and the image processing module to determine whether a trajectory of detected feature points in a selected camera aligns with a trajectory of detected feature points in an adjacent camera. A reporting module communicates with the line alignment module and reports a camera blockage status to at least one vehicle system or controller.

FIELD

The present disclosure relates to systems and methods for detecting blockages in a vehicle's surrounding view cameras.

BACKGROUND

A variety of vehicle systems use vision-based data from cameras on the vehicle's exterior. Examples of these systems are back-up cameras, systems for detecting the presence of surrounding objects such as other vehicles or pedestrians, systems for detecting the lane markers, automated driving systems, etc. The quality of the images provided by the cameras may affect the accuracy of the data provided by the system. Problems such as obstructions from the environment (mud, rain, dirt, etc) or other obstructions (paint, tape, etc.) can cause partial or total blockage of the camera's view. In these instances, the information from the camera is compromised and may be unreliable for the vehicle system.

SUMMARY

An example camera monitoring system for a vehicle includes a plurality of cameras disposed around an exterior of a vehicle. An image processing module communicates with the plurality of cameras and generates overhead view images from raw images taken by at least one of the plurality of cameras. A histogram module communicates with the image processing module and generates at least one histogram from the overhead view images. A likelihood module communicates with the histogram module and determines a likelihood of blockage for at least one of the plurality of cameras. A line alignment module communicates with the likelihood module and the image processing module to determine whether a trajectory of detected feature points in a selected camera of the plurality of cameras aligns with a trajectory of detected feature points in an adjacent camera of the plurality of cameras. A reporting module communicates with the line alignment module and reports a camera blockage status to at least one vehicle system or controller.

The camera monitoring system may further include a front camera mounted on a front of the vehicle, a rear camera mounted on a back of the vehicle, a left side camera mounted on a left side of the vehicle, and a right side camera mounted on a right side of the vehicle.

The camera monitoring system may further include a histogram module that converts a plurality of pixels of the raw images from color pixels into gray tone values representing a luminosity based on a weighted average of red, green, and blue colors in each pixel.

The camera monitoring system may further include a histogram module that converts a plurality of pixels of the raw images into 8-bit luminance values, each luminance value being within a range of 0 and 255.

The camera monitoring system may further include a histogram module that maps the luminance values to create a luminance histogram.

The camera monitoring system may further include a plurality of pixels in the luminance histogram that are from a region of interest within at least one of the overhead view images.

The camera monitoring system may further include a likelihood module that determines a blackout ratio and a whiteout ratio for the at least one histogram.

The camera monitoring system may further include a blackout ratio that is the total pixels within a range of 0 to 40 on the at least one histogram divided by the total pixel count, and a whiteout ratio that is the total pixels within a range of 250 to 255 on the at least one histogram divided by the total pixel count.

The camera monitoring system may further include a likelihood module that sums the blackout ratio and the whiteout ratio to create a Blackout+Whiteout ratio and that determines whether the Blackout+Whiteout ratio is greater than a first predetermined threshold, indicating a possible blockage of at least one of the plurality of cameras.

The camera monitoring system may further include a first predetermined threshold that is 0.3.

The camera monitoring system may further include a line alignment module that determines a moving average of the trajectory of detected feature points in the selected camera of the plurality of cameras from the trajectory of detected feature points in the adjacent camera of the plurality of cameras and that determines whether the moving average is less than a second predetermined threshold, indicating a blocked camera.

The camera monitoring system may further include a second predetermined threshold that is 0.5.

The camera monitoring system may further include a reporting module that retains the camera blockage status for a predetermined buffer period before reporting the camera blockage status to ensure the validity of results.

The camera monitoring system may further include a predetermined buffer period that is at least 50 frames or at least 100 meters driving distance.

An example method for detecting blockages in a vehicle's surrounding view cameras includes capturing, by a plurality of cameras disposed around an exterior of a vehicle, raw images of the vehicle's surroundings; generating, by an image processing module, overhead view images from the raw images taken by at least one of the plurality of cameras; generating, by a histogram module, at least one histogram from the overhead view images; determining, by a likelihood module, a likelihood of blockage for at least one of the plurality of cameras; determining, by a line alignment module, whether a trajectory of detected feature points in a selected camera of the plurality of cameras aligns with a trajectory of detected feature points in an adjacent camera of the plurality of cameras; and reporting, by a reporting module, a camera blockage status to at least one vehicle system or controller.

The method may further comprise converting a plurality of pixels of the raw images from color pixels into gray tone values representing a luminosity based on a weighted average of red, green, and blue colors in each pixel to generate the at least one histogram.

The method may further comprise mapping a plurality of pixels from a region of interest within at least one of the overhead view images to generate the at least one histogram.

The method may further comprise determining a blackout ratio and a whiteout ratio for the at least one histogram; summing the blackout ratio and the whiteout ratio to create a Blackout+Whiteout ratio; and determining whether the Blackout+Whiteout ratio is greater than a first predetermined threshold, indicating a possible blockage of at least one of the plurality of cameras, wherein the first predetermined threshold is 0.3.

The method may further comprise determining a moving average of the trajectory of detected feature points in the selected camera of the plurality of cameras from the trajectory of detected feature points in the adjacent camera of the plurality of cameras, and determining whether the moving average is less than a second predetermined threshold, indicating a blocked camera, wherein the second predetermined threshold is 0.5.

The method may further comprise retaining the camera blockage status for a predetermined buffer period before reporting the camera blockage status to ensure the validity of results, wherein the predetermined buffer period is at least 50 frames or at least 100 meters driving distance.

Corresponding reference numerals indicate corresponding parts throughout he several views of the drawings.

DETAILED DESCRIPTION

A variety of systems within a vehicle utilize data from externally mounted cameras viewing the surrounding environment of the vehicle. For example, a vehicle's Video-Processing Module's (VPM) vision-based lane sensing feature utilizes lane marker detection information. The lane marker detection information is estimated using overhead images consisting of four camera images to provide accurate heading and offset information for the vehicle's systems such as the cruise control system, for example. The quality of the captured images has a significant effect on the accuracy of the sensing information, and, thus, the overall system's performance. Therefore, a system and method for detecting both partial and full blockages of the vehicle's cameras provides a much needed improvement to the lane sensing system by providing information relating to the integrity of the camera such that the lane sensing system can make appropriate adjustments in the confidence value applied to the camera information.

Referring toFIG. 1A, a vehicle10having a camera monitoring system12according to the present teachings is illustrated. Although the vehicle10is illustrated as an automobile inFIG. 1, the present teachings apply to any other suitable vehicle, such as a sport utility vehicle (SUV), a mass transit vehicle (such as a bus), or a military vehicle, as examples. The vehicle10may further be a self-driving vehicle. The system12is configured to inform various vehicle systems, and in particular, a lane sensing system22that one or more cameras on the vehicle is obstructed, either completely or partially. The system12generally includes a camera integrity system14, a driver alert system18, the lane sensing system22, a controller26, and a number of cameras30mounted or attached to the exterior of the vehicle10. For example, the vehicle of the embodiment illustrated inFIGS. 1A and 1Bincludes four cameras30, a front camera30a,a rear camera30b,a left side camera30c,and a right side camera30d.

The vehicle10also includes vehicle sensors34and a global positioning system (GPS)38. The GPS38determines the location and inertial/orientation data of the vehicle10. Additionally, or alternatively, an image navigation system may be used in addition to or in place of the GPS26. The image navigation system, for example, may determine a location of the vehicle10based on image data collected from the cameras34, Lidar sensors, stereo sensors, radar sensors, ultrasonic sensors, or other sensors on the vehicle10. The vehicle sensors34can also include a vehicle speed sensor that generates data indicating a current speed of the vehicle10, a vehicle acceleration sensor that generates data indicating a current rate of acceleration or deceleration of the vehicle10, a steering wheel angle or position sensor indicating a current angle of the steering wheel, or any other sensor providing vehicle data to any of the driver alert system18, lane sensing system22, controller26, and camera integrity system14. The vehicle sensors34can also include environmental sensors such as sensors to determine light level, weather data, temperature, road surface status, traffic conditions, lane markers, etc. The controller26receives data from the sensors34and GPS38and uses the data for controlling the various vehicle systems.

The controller26can be any suitable controller for monitoring or controlling one or more of the sensors34, the driver alert system18, the camera integrity system14, the GPS38, the cameras30, and/or the lane sensing system22. In this application, including the definitions below, the terms “controller” and “system” may refer to, be part of, or include processor hardware (shared, dedicated, or group) that executes code and memory hardware (shared, dedicated, or group) that stores code executed by the processor hardware. The code is configured to provide the features of the controller and systems described herein.

Now referring additionally toFIG. 1B, the front camera30amay be positioned near a front bumper42of the vehicle10or in any other location on the front of the vehicle10where the front camera30aprovides an image of objects and/or lane markers in front of the vehicle10. The rear camera30bmay be positioned near a rear bumper46or rear license plate50of the vehicle10or in any other location on the back of the vehicle10where the rear camera30bprovides an image of objects and/or lane markers behind the vehicle10. The left side camera30cmay be positioned near a front left fender54or a left side mirror58of the vehicle10or in any other location on the left side of the vehicle where the left side camera30cprovides an image of objects and/or lane markers to the left of the vehicle10. The right side camera30dmay be positioned near a front right fender62or a right side mirror66of the vehicle10or in any other location on the right side of the vehicle where the right side camera30dprovides an image of objects and/or lane markers to the right of the vehicle10.

The cameras30may be positioned on the vehicle such that a field of vision70of each camera30covers at least the side of the vehicle10on which the camera30is mounted, Additionally, the fields of vision70for the cameras30may overlap with each other. For example, the field of vision70afor the front camera30amay provide an image of the environment in front of the vehicle10and may overlap with the field of vision70cfor the left side camera30cand the field of vision70dfor the right side camera30d.The field of vision70bfor the rear camera30bmay provide an image of the environment behind the vehicle10and may overlap with the field of vision70cfor the left side camera30cand the field of vision70dfor the right side camera30d.The field of vision70cfor the left side camera30cmay provide an image of the environment on the left side of the vehicle10and may overlap with the field of vision70afor the front camera30aand the field of vision70bfor the rear camera30b.The field of vision70dfor the right side camera30dmay provide an image of the environment on the right side of the vehicle10and may overlap with the field of vision70afor the front camera30aand the field of vision70bfor the rear camera30b.

Each field of vision70for each camera30may include an image having lane markers, or lane identifiers,74. The lane sensing system22utilizes the images from the cameras30to determine the position of the vehicle10within the lane markers74. The lane sensing system22also utilizes sensor34information to determine, for example, whether a turn signal is activated, indicating a desired lane change, or whether the steering wheel is at an angle. If the vehicle is within a threshold (for example only, within 8 inches) of the lane marker74, the lane sensing system22may alert the driver through the driver alert system18or may correct the vehicle's path through the controller26. For example only, the lane sensing system22may alert the driver with an audible or visual alarm, or the lane sensing system22may correct the vehicle's path by autonomously turning the steering wheel.

Examples of the fields of vision70for the cameras30are shown inFIGS. 2A-2D.FIGS. 2A-2Dillustrate views of the front camera30a(FIG. 2A), the rear camera30b(FIG. 2B), the left side camera30c(FIG. 2C), and the right side camera30d(FIG. 2D) under normal, unobstructed conditions. As illustrated inFIGS. 2A-2D, the cameras30are able to capture images including surrounding objects and lane markers74. The vehicle's shadow is also clearly seen in the right side camera image (FIG. 2D). As previously stated and described in detail later, the images from the cameras30are combined by the camera integrity system14into a single overhead view (OHV) image78, shown inFIG. 3.

Often, due to environmental conditions, such as weather, mud, water, etc., or other conditions, such as paint, tape, etc., one or more of the cameras30may become either partially or totally blocked or the image may be degraded. Example causes of a blocked camera may be an obstacle being attached to a camera lens (for example, tape, mud, dirt, insects, etc), an obstacle obstructing the camera's view (for example, a wall, tree, vehicle, etc.), extreme bad weather conditions (for example, ice, snow, salt, etc.), light obstructing the camera's view (for example, bright lights after passing a tunnel, bridge, etc), etc. In a blocked camera scenario, the camera's field of vision is fully obstructed or not visible. If the system12is experiencing at least one blocked camera, the lane sensing system22is unable to detect an existing lane marker74or is only able to detect a few detected feature points that are not enough to result in detecting the lane marker74. The blocked camera results in a lane sensing system22output of invalid, low confidence, or blocked.

Example causes of a degraded camera may be an obstacle attached to a camera lens (for example, tape, mud, dirt, insects, etc. partially blocking the camera lens), low luminance (for example, night or tunnel driving, etc.), extreme bad weather conditions (for example, water drops, etc), etc. In a degraded camera scenario, the camera's field of vision is partially obstructed or not visual. If the system12is experiencing at least one degraded camera, the system performance is very low in terms of line marker detection because there are few feature points, weak feature points, and/or noisy feature point that increase the possibility of false candidate lane markers. The degraded camera results in a lane sensing system22output of invalid or low confidence.

FIGS. 4A-4C and 5A-5Cillustrate the raw camera image (FIGS. 4A and 5A) along with the front/rear overhead view image (FIGS. 4C and 5C) and the left right overhead view image (FIGS. 4B and 5B) for examples of a degraded or blocked camera. InFIGS. 4A-4C, the right camera30dis fully blocked with tape.FIG. 4Aillustrates the raw image from the right camera30dshowing no visibility.FIG. 4Bis a left/right overhead view image where the overhead view images from the left side camera30cand the right side camera30dhave been stitched into a single overhead view image.FIG. 4Cis a front/rear overhead view image where the overhead view images from the front camera30aand the rear camera30bhave been stitched into a single overhead view image.

InFIGS. 5A-5C, the front, right, and left cameras are partially blocked with mud.FIG. 5Aillustrates the raw image from the front camera30ashowing degraded visibility or a partially blocked image.FIG. 5Bis a left/right overhead view image where the overhead view images from the left side camera30cand the right side camera30dhave been stitched into a single overhead view image.FIG. 5Cis a front/rear overhead view image where the overhead view images from the front camera30aand the rear camera30bhave been stitched into a single overhead view image.

Now referring toFIG. 6, an example block diagram of the system12is illustrated. The system12includes the cameras30, the camera integrity system14, the lane sensing system22, and the driver alert system18. The cameras30communicate with the camera integrity system14, the camera integrity system14communicates with the lane sensing system22and the driver alert system18, and the lane sensing system22communicates with the driver alert system18. The camera integrity system further includes an image processing module82, a histogram module86, a likelihood module90, a line alignment module94, a reporting module98, and a storage module102.

The cameras30communicate with the image processing module82. The cameras30send raw images taken from the front camera30a,the rear camera30b,the left side camera30c,and the right side camera30d(for example, the images inFIGS. 2A-2D) to the image processing module82where the images are converted into a single overhead view image (for example, the image inFIG. 3).

The image processing module82, upon receipt of the raw camera images, converts each raw image into an overhead view image. For example, the raw image from the front camera30ais converted into a front overhead view image, the raw image from the rear camera30bis converted into a rear overhead view image, the raw image from the left side camera30cis converted into a left overhead view image, and the raw image from the right side camera30dis converted into a right overhead view image. The image processing module82then stitches together the front overhead view image and the rear overhead view image to create a front/rear overhead view age (for example, see the images inFIGS. 4C and 5C). The image processing module82also stitches together the left overhead view image and the right overhead view image to create a left/right overhead view image (for example, see the images inFIGS. 4B and 5B). The image processing module82then stitches together the front/rear overhead view image and the left/right overhead view image to create the single overhead view image (FIG. 3) for the four camera30images.

Image stitching is a process of combining multiple images with overlapping fields of view to produce a single image. Image stitching requires substantially similar or exact overlaps between images to produce seamless results. Algorithms combine direct pixel-to-pixel comparisons of the original images to determine distinctive features in each image. The distinctive features are then matched to establish correspondence between pairs of images. The edges of the images are then blended to reduce the visibility of seams between the images.

The image processing module82communicates with the histogram module86to provide the four individual overhead view images for each camera and the single overhead view image for the four cameras. The histogram module86converts the four individual overhead view images to luminance histograms. The histogram module86converts the pixels in each of the individual overhead view images from color pixels into gray tone values computed from RGB (red, green, blue) using the following formula:
RGB Luminance Value=0.3R+0.59G+0.11B
Each pixel is converted so that it represents a luminosity based on a weighted average of the three colors, red, green, and blue, at that pixel. While the weighted average of the red, green, and blue colors are 0.3, 0.59, and 0.11 in the equation above, the red, green, and blue colors may be weighted differently as long as the sum of the weights is equal to 1.0. Thus, every RGB pixel computes to an 8-bit luminance value between 0 and 255. The final image of gray tone values appears similar to a grayscale image, but is instead brightness adjusted to indicate appropriately what the human eye sees.

Examples of a front/rear overhead image and a left/right overhead image converted to luminance values are illustrated inFIGS. 7A and 7B. Only the regions of interest (ROI)106,110,114,118, which are defined as the sub-image of the overhead view image where the lane markers74are visible, are considered for the histogram calculations. For example the front ROI106and the rear ROI110are indicated inFIG. 7A, and the left ROI114and the right ROI118are indicated inFIG. 7B.

Thus, the histogram module86, for example, converts the overhead view image from the front camera30ainto luminance values. The histogram module86then creates a luminance histogram of the front ROI106from the overhead view luminance values image. Likewise, the histogram module86converts the overhead view image from the rear camera30b,the left side camera30c,and the right side camera30dinto luminance values. The histogram module86then creates luminance histograms of the rear ROI110, the left ROI114, and the right ROI118from the overhead view luminance value images, respectively. An example front luminance histogram is illustrated inFIG. 8A, an example rear luminance histogram is illustrated inFIG. 8C, an example right luminance histogram is illustrated inFIG. 8B, and an example left luminance histogram is illustrated inFIG. 8D.

To create the luminance histogram, the histogram module86maps the pixels of the luminance value image according to brightness. Each pixel may have a brightness level or number within the range of 0 to 255, with 0 being totally black and 255 being totally white.

The histogram module86provides the front, rear, left and right histograms (FIGS. 8A-8D) to the likelihood module90. The likelihood module90determines a blackout ratio and a whiteout ratio for each of the front, rear, left and right histograms. The blackout ratio is computed as the dark pixel count divided by the total pixel count, and the whiteout ratio is computed as the bright pixel count divided by the total pixel count. The dark pixel count is the number of pixels falling within a range of dark pixels on the histogram, and the bright pixel count is the number of pixels falling within a range of bright pixels on the histogram. With additional reference toFIG. 9, the dark pixel count may be the pixels falling within reference box122, and the bright pixel count may be the pixels falling within reference box126. For example only, the dark pixels may be within the range from 0 to 40 on the luminance histogram, and the bright pixels may be within the range from 250 to 255 on the luminance histogram. The likelihood module90then sums the blackout ratio with the whiteout ratio to obtain a Blackout+Whiteout Ratio.

The likelihood module90further communicates with the storage module102to retrieve a luminance likelihood map (FIG. 10). The luminance likelihood map is a predetermined likelihood map having thresholds that indicate the probability of an image being blacked out or whiteout, and thus, the camera being blocked or degraded. If an image has a high percentage of blackout pixels, then the camera has a possibility of being blocked. In the luminance likelihood map, the area on the left of the map, from a Blackout+Whiteout Ratio of 0 to approximately 0.4, with a greater probability the closer the Blackout+Whiteout Ratio is to 0, is a “No Blockage” area, indicated at130. The area on the right of the luminance likelihood map, from a Blackout+Whiteout Ratio of approximately 0.7 to 1, with a greater probability the closer the Blackout+Whiteout Ratio is to 1, is a “Possible Blockage” area, indicated at134. Threshold line138extends horizontally across the luminance likelihood map at a threshold value of approximately 0.33, and the mapping line142extends horizontally across the luminance likelihood map at a value of 0.5 from a Blackout+Whiteout Ratio of 0 to 0.5 and then linearly decreases in likelihood from a Blackout+Whiteout Ratio of 0.5 to 1.

The likelihood module90determines the likelihood along mapping line142which corresponds to the calculated Blackout+Whiteout Ratio from the luminance histogram. If the likelihood and Blackout+Whiteout Ratio is within area130, there is no blockage. If the likelihood and Blackout+Whiteout Ratio is within area134, the likelihood module90determines that there may be possible blockage.

The likelihood module90communicates this information with the line alignment module94. The line alignment module94also communicates with the image processing module82to obtain the overhead view images from each of the cameras30. To avoid false blockage reporting during instances such as nighttime driving (due to a high ratio of black pixels that appear in the overhead view images during nighttime driving), the line alignment module94performs additional analysis to confirm the blockage status. The line alignment module94confirms an image's camera blockage by checking an alignment accuracy of detected lines in overlapped images. The line alignment module94evaluates the individual overhead view images and detects feature points for various features in the images. For example, the line alignment module94detects feature points for lane markers74in the individual overhead view images. The line alignment module94compares the detected feature points in the potential blackout or whiteout image with the detected feature points in adjacent camera images to determine if the detected feature points in the potential blackout or whiteout image have similar trajectories as the detected feature points in adjacent camera images. This comparison may be referred to as Line Alignment (LA).

The line alignment module94also estimates the trend of alignment for the detected feature points in the potential blackout or whiteout image. The line alignment module94communicates with the storage module102to retrieve a moving average map. An example moving average map is illustrated inFIG. 11. The moving average map plots the moving average (y-axis) for each frame (x-axis). The cameras each record approximately ten frames per second. A threshold line146at a moving average of 0.5 divides a strong line alignment portion150of the moving average map from a weal line alignment portion154.

The line alignment module94plots the error, or moving average, of the detected feature points in the potential blackout or whiteout image from the detected feature points in adjacent camera images on a moving average map. If the plotted line alignment value is greater than the threshold line146, then there is strong line alignment indicating no blockage of the camera. If the plotted line alignment value is less than the threshold line146, as shown in the example ofFIG. 11, then there is weak line alignment indicating that the camera is obstructed, blocked, or degraded.

The line alignment module94reports whether the camera is blocked or degraded to the reporting module98. The reporting module98makes a determination on whether to notify the lane sensing system22, the driver alert system18, or both. If none of the cameras30are blocked, the reporting module98may either report “no error” or “not blocked” for each camera. If one or more of the cameras30is blocked or degraded, the reporting module98may report “blocked” with the appropriate camera.

In some embodiments, the reporting module98reports whether the camera is obstructed, blocked, or degraded to the lane sensing system22. If one or more of the cameras30is blocked, the lane sensing system22may stop detection and report an invalid or low confidence to the controller26.

In some embodiments, the reporting module98reports a blocked camera to the driver alert system18. The driver alert system18may activate an audible or visual alarm to alert the driver of the blocked camera.

The reporting module98may retain the results from the line alignment module94for a buffering period before reporting the results to the lane sensing system and/or the driver alert system18. For example only, the buffering period may be approximately 50 frames or the number of frames equal to 100 meters of driving distance. Since the cameras store images at approximately 10 frames per second, the buffering time is approximately 5 seconds. Thus, while the default value from the reporting module98may be “no blockage,” it may take at least 50 frames, or at least 5 seconds, to switched to a “blocked” state. The buffering time ensures that the camera is truly blocked and not temporarily or mistakenly blocked for a brief moment.

Now referring toFIG. 12, a flow diagram for a method200of detecting blockages in a vehicle's surrounding view cameras is illustrated. Method200starts at204. At208the cameras30take images of the surroundings of the vehicle10. At212, overhead view images of the camera images are generated. For example, the raw image from the front camera30ais converted into a front overhead view image, the raw image from the rear camera30bis converted into a rear overhead view image, the raw image from the left side camera30cis converted into a left overhead view image, and the raw image from the right side camera30dis converted into a right overhead view mage,

At216, a luminance histogram for each overhead view image is generated. In generating the luminance histogram, the pixels in each of the individual overhead view images are converted from color pixels into gray tone values computed based on a weighted average of the three colors, red, green, and blue, at that pixel, using the formula RGB Luminance Value=0.3R+0.59G+0.11B. As previously stated, while the weighted average of the red, green, and blue colors are 0.3, 0.59, and 0.11 in the recited equation, the red, green, and blue colors may be weighted differently as long as the sum of the weights is equal to 1.0. Thus, every RGB pixel computes to an 8-bit luminance value between 0 and 255, with 0 being totally black and 255 being totally white. The luminance histogram is then created by mapping the pixels of the luminance value image according to brightness.

At220, the Blackout+Whiteout Ratio is computed. The blackout ratio is computed as the dark pixel count divided by the total pixel count, and the whiteout ratio is computed as the bright pixel count divided by the total pixel count. The dark pixel count is the number of pixels falling within a range of dark pixels on the histogram, and the bright pixel count is the number of pixels falling within a range of bright pixels on the histogram. As previously discussed, the dark pixel count may be the pixels falling within the range from 0 to 40 on the luminance histogram, and the bright pixel count may be the pixels falling within the range from 250 to 255 on the luminance histogram. The blackout ratio is then summed with the whiteout ratio to obtain a Blackout+Whiteout Ratio.

At224, the Blackout+Whiteout Ratio is mapped on the luminance likelihood map. If the Blackout+Whiteout Ratio is less than 0.3, the likelihood falls within the “No Blockage” area130of the luminance likelihood map (FIG. 10), and the camera is reported as “Not Blocked” at228. If the Blackout+Whiteout Ratio is greater than 0.3, the likelihood falls within the “Possible Blockage” area134of the luminance likelihood map (FIG. 10), and the method200moves to232. In the present example, the Blackout+Whiteout Ratio is compared to a predetermined threshold of 0.3. Other predetermined thresholds greater than or less than 0.3 may alternatively be used. For example, the predetermined threshold may be set to 0.25 or 0.35, or another suitable predetermined threshold.

At232, the overhead view images are evaluated to determine the presence of lane markers. If valid lane markers are not present in the overhead view images, the camera is reported as “Not Blocked” at228. If there are no detected lane markers, then the line alignment module94cannot perform any additional analysis. The line alignment module94needs visible markers present to check the projection between two overlapped cameras using the visible markers' feature points. If valid lane markers are present in the overhead view images, method200proceeds to236.

At236, the alignment accuracy of detected lines in overlapped images is evaluated. During the evaluation, the individual overhead view images are scanned and feature points for various features in the images are detected within each individual overhead image. For example, the feature points for lane markers74in the individual overhead view images are detected. The detected feature points in the potential blackout or whiteout image are then compared with the detected feature points in adjacent camera images to determine if the detected feature points in the potential blackout or whiteout image have similar trajectories as the detected feature points in adjacent camera images. This comparison may be referred to as Line Alignment (LA).

At240, the moving average is calculated. The error, or moving average, of the trajectory of the detected feature points in the potential blackout or whiteout image from the trajectory of the detected feature points in adjacent camera images are plotted on a moving average map (FIG. 11). The moving average for each frame, or image, is plotted on the same moving average map. If the plotted line alignment value is greater than the threshold line146, then there is strong line alignment indicating no blockage of the camera, If the plotted line alignment value is less than the threshold line146, as shown in the example ofFIG. 11, then there is weak line alignment indicating that the camera is obstructed, blocked, or degraded.

At244, the moving average is compared to the threshold value of 0.5. If the moving average is greater than or equal to 0.5, the camera is reported as “Not Blocked” at228. If the moving average is less than 0.5, there is weak line alignment indicating that the camera is obstructed, blocked, or degraded, and the camera is reported as “Blocked” at248, In the present example, the moving average is compared to a predetermined threshold of 0.5. Other predetermined thresholds greater than or less than 0.5 may alternatively be used. For example, the predetermined threshold may be set to 0.4 or 0.6, or another suitable predetermined threshold.

At252, the “Blocked” and “Not Blocked” reporting status is buffered. The identified “Blocked” or “Not Blocked” results are retained for a buffering period before reporting the results at256to ensure that the camera is truly blocked (or not blocked) and not temporarily or mistakenly blocked (or not blocked) for a brief moment. For example only, the buffering period may be approximately 50 frames or the number of frames for 100 meters of driving distance. Since the cameras store images at approximately 10 frames per second, the buffering time may be approximately 5 seconds. However, the buffering time may also be dependent on driving speed over the 100 meters of driving distance. Thus, while the default value from the reporting module98may be “no blockage,” it may take at least 50 frames, at least 100 meters, to switched to a “blocked” state.

At256, the blockage status of the camera is reported. The blockage status may be reported to various systems of the vehicle utilizing the camera30images. For example, the blockage status may be reported to the lane sensing system22. If one or more of the cameras30is blocked, the lane sensing system22may stop detection and report an invalid or low confidence to the controller26. In some embodiments, if the blockage status is “Blocked,” this status may be reported to the driver alert system18. The driver alert system18may activate an audible or visual alarm to alert the driver of the blocked camera. The method200ends at260.

The module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules that are connected via interface circuits. For example, multiple modules may allow load balancing.