Abstract:
Systems and methods are presented for operating a vehicle camera system to detect, identify, and mitigate camera lens contamination. An image is received from a camera mounted on the vehicle and one or more metrics is calculated based on the received image. The system determines whether a lens of the camera is contaminated based on the one or more calculated metrics and, if so, determines a type of contamination. A specific mitigation routine is selected form a plurality of mitigation routines based on the determined type of contamination and is applied to the received image to create an enhanced image. The enhanced image is analyzed to determine whether the contamination is acceptably mitigated after application of the selected mitigation routine and a fault condition signal is output when the contamination is not acceptably mitigated.

Description:
RELATED APPLICATIONS 
       [0001]    This application claims the benefit of U.S. Provisional Application No. 62/003,303, filed May 27, 2014 and entitled “SYSTEM AND METHOD FOR DETECTION AND MITIGATION OF LENS CONTAMINATION FOR VEHICLE MOUNTED CAMERAS,” the entire contents of which are incorporated herein by reference. 
     
    
     BACKGROUND 
       [0002]    Cameras are increasingly being used in vehicle applications to improve the driver&#39;s situational awareness and to support various safety related functions of the vehicle (e.g., lane departure warnings). Further, the cameras used in vehicle applications often require digital processing of captured imagery. Cameras mounted to the exterior of the vehicle are susceptible to accumulating dust, dirt, road salt residue, insects, and other contaminants on the camera lens. 
         [0003]    The present invention relates to the systems and methods for addressing contamination of camera lenses, particularly with regarding to contamination of camera lenses in vehicle-mounted camera systems. 
       SUMMARY 
       [0004]    Camera lenses can become contaminated (e.g., obstructed or dirtied) by various foreign objects, materials, or conditions including, for example, fingerprints, scratches, condensation, ice, or frost. Because such contaminants obstruct the field of view of a camera system, the image quality suffers. In some cases, contaminants are not necessarily imaged by the camera because of their location on the lens. For example, contaminating particles may build up inside the focal point of the lens, such that the contaminant is not in focus at the detector array. However, such contaminants still negatively affect (i.e., degrade) image quality by scattering light. Light scattering can create image regions of reduced contrast and/or resolution, as well as distortion. 
         [0005]    When the lens is contaminated, true image content may still be available in the captured image data, but can be blurred or otherwise degraded. Further, since the contaminant itself is not clearly imaged by the camera system (e.g., on the detector array), determining which regions of the image are degraded, and to what extent, can become a challenge. Additionally, in some situations, contamination can become so severe that the image is no longer useful for visual or automated use and the camera should therefore be considered non-functional. 
         [0006]    In various embodiments, the invention provides a camera lens contamination detection and mitigation system for use in a vehicle. The system detects camera lens contamination, and mitigates the reduced image quality arising from the lens contamination using software. In some embodiments, the system also estimates the type of contamination likely. Further, if the system determines that the lens contamination is too severe to be able to restore the image quality (i.e., that the camera is “blind”), the system issues one or more warnings to the driver. 
         [0007]    In one embodiment, the invention provides a method for operating a vehicle camera system. An image is received from a camera mounted on the vehicle and one or more metrics is calculated based on the received image. The system determines whether a lens of the camera is contaminated based on the one or more calculated metrics and, if so, determines a type of contamination. A specific mitigation routine is selected form a plurality of mitigation routines based on the determined type of contamination and is applied to the received image to create an enhanced image. The enhanced image is analyzed to determine whether the contamination is acceptably mitigated after application of the selected mitigation routine and a fault condition signal is output when the contamination is not acceptably mitigated. 
         [0008]    In some embodiments, the fault condition signal causes the system to display a message on a display in the interior of the vehicle informing the operator that the captured camera image is distorted. In some embodiments, the fault condition signal partially or entirety disables one or more vehicle assistance systems (e.g., an automated parking system or a lane detection system) when the fault condition signal indicates that the image is unacceptably distorted due to lens contamination. 
         [0009]    Other aspects of the invention will become apparent by consideration of the detailed description and accompanying drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0010]      FIG. 1  is a block diagram of a vehicle system including a camera. 
           [0011]      FIG. 2  is an image output from the camera system of  FIG. 1 . 
           [0012]      FIG. 3A  is an unaltered image output from the camera system of  FIG. 1  with lens contamination. 
           [0013]      FIG. 3A  is an enhanced version of the image of  FIG. 3A  after mitigation of the image distortion caused by the lens contamination. 
           [0014]      FIG. 4  is a flowchart of a method of detecting and mitigating lens contamination in the system of  FIG. 1 . 
           [0015]      FIG. 5  is a flowchart of a method of identifying specific types of contamination in the system of  FIG. 1 . 
           [0016]      FIG. 6  is a flowchart of a method of mitigating distortion due to lens condensation in the system of  FIG. 1 . 
       
    
    
     DETAILED DESCRIPTION 
       [0017]    Before any embodiments of the invention are explained in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. The invention is capable of other embodiments and of being practiced or of being carried out in various ways. 
         [0018]      FIG. 1  illustrates an example of a vehicle automation and communication system. A camera  101  is mounted on the vehicle with a field of view including, for example, an area behind the vehicle. Such camera systems may be used to assist the driver while operating the vehicle in reverse and specifically while parking. However, in other implementations the camera  101  may be mounted on other locations on the vehicle. Similarly, some implementations may include more than one camera positioned on the vehicle. 
         [0019]    The camera  101  captures images and sends the image data to a processor  103 . The processor  103  executes instructions stored on a non-transitory computer readable memory  105  (e.g., a Flash memory, a hard disk, or other type of ROM or RAM memory unit) to provide various functionality including, for example, the methods described below. The processor  103  may also store image data on the memory  105 . The processor  103  may output the image data from the camera  101  to a user interface/display  107  for display to the user. Furthermore, in some implementations, the processor  103  provides the image data or other information to other vehicle assistance/automation system  109  (for example, an automated parking assist system, a lane monitoring system, or an adaptive cruise control system). 
         [0020]      FIG. 2  illustrates one example of an image captured by the camera  101 . As discussed in further detail below, the processor  103  executes the instructions stored on the memory  105  to divide the image area into sub-regions A through  1  for analysis and processing. In the example of  FIG. 2 , several of the sub-regions display some degree of image distortion due to camera lens contamination. More specifically, sub-regions B, D, G, H, and I are relatively undistorted. However, sub-regions A, C, E, and F includes a significant amount of image distortion due to lens condensation. 
         [0021]    As described in further detail below, the processor  103  applies a detection algorithm to detect contaminants on the camera lens. Within each sub-regions, a set of image quality metrics are determined. In some implementations, the image quality metrics include, for example, image sharpness (e.g., via edge detection or MIT estimation), color variation (e.g., by generating local histograms of each color band), and spatial uniformity (e.g. by analyzing contrast ratio or optical flow). The processor  103  determines the absolute value of the determined metrics and monitors a change in each of the determined metrics over time. By monitoring the change in each of the determined metrics over time (i.e., by monitoring a series of images captured within a certain time frame), the system can flag regions where lens contamination is likely. 
         [0022]    For example, in  FIG. 2 , the lens condensation over much of the lens causes many pixel values in multiple sub-regions to vary slowly both spatially and temporally. Since vehicle motion would normally cause the image metrics (i.e., pixel values) to change at a high frequency and with significant spatial variation in non-contaminated images, the system is able to identify and flag the slow-varying sub-regions as being contaminated. Furthermore, as discussed in further detail below, when the detection algorithm flags an image (or an image sub-regions) as being degraded (i.e., contaminated), the cause of the image degradation can be identified (e.g., condensation vs. dust contamination). 
         [0023]      FIG. 3A  shows another example of an image that has been degraded by haze or condensation on the lens (i.e., the image blur/light scatter on the left side of the image  301  and on the right side of the image  303 ). As discussed in further detail below, the detection algorithm processes the image and identifies condensation as the source of the image degradation due to its slow-varying pixel values on the outermost borders of the image. 
         [0024]    Different types of lens contamination can affect the amount of true image content that is included in the degraded image. For example, in the condensation degraded image of  FIG. 3A , true image data is included in the output image, but is blurred by the condensation on the lens. In contrast, contamination due to dust or another solid foreign object on the camera lens may entirely obstruct and eliminate true image contact such that the affected region does not vary at all over time. 
         [0025]    By estimating the type of contamination, the system is able to employ an appropriate compensating algorithm to restore image quality. For example, in  FIG. 3A , the slow-varying nature of the scattered light can be leveraged to determine a mask image that varies in the same manner as the scattered light pattern. To compensate for the lens contamination due to condensation and to mitigate the image degradation, the processor  103  subtracts the determined mask image from the original raw image data from the camera  101 , As a result, much of the blur in the image of  FIG. 3A  is removed and the processor  103  generates the enhanced image of  FIG. 3B . In the enhanced image of  FIG. 3B , the contaminated regions bordering the original camera images have been restored and the road is more uniform in color. Further, more contrast between the road and the parking lines can be observed. 
         [0026]    The mask-based method of mitigating image degradation due to lens contamination is just one example of an image enhancement technique that can be applied by the camera system to mitigate a specifically identified type of lens contamination. The systems and methods described herein are not necessarily limited to solutions for condensation or other light-scattering contamination. Accordingly, other approaches and algorithms can be used to restore other types of lens contamination. 
         [0027]    Although some of the systems and methods described herein are configured to provide some form of image enhancement to mitigate lens contamination, in some cases, lens contamination can become so severe that the compensating algorithm cannot effectively restore the original camera image for practical use. In such cases, even the enhanced image could not be effectively relied upon by vehicle assistance/automation systems such as a parking assistance system. 
         [0028]    In some such cases, the processor  103  can be configured to identify situations where the contamination is so severe that the output image cannot be relied upon. In some implementations, the processor  103  monitors the image metrics (i.e., pixel values) of the restored image as a function of time (i.e., as opposed to monitoring the metrics of the original camera image). In some implementations, the system applies the same contamination detection metric to the enhanced image as was applied to the raw image data to initially detect the lens contamination. If the system determines that the image quality has not improved enough for practical usage, the system concludes that the lens contamination is too severe to effectively restore image quality and concludes that the camera is effectively “blind” in one or more sub-regions of the output image. 
         [0029]    As described in further detail below, when the system determines that the camera is effectively “blind,” the processor  103  issues a warning to the driver (e.g., visually through the user interface  107  or by other mechanisms including, for example, haptic feedback or an audible tone or message). In some implementations, the warning includes information indicating that (a) the camera is no longer operational due to an excessive and unmitigatable degree of lens contamination, (b) vehicle assistance and automation functions depending on the camera may no longer be available, and (c) the “blind” state of the camera system will persist until the driver cleans or replaces the camera lens. Furthermore, in some implementations, the system is configured to save in the non-transitory memory  105  the date and time that the system determines the camera system to be “blind,” notifies the driver, and ceases to display video output on the user interface/display  107 . 
         [0030]      FIG. 4  illustrates a method of monitoring image output data from the camera  101  to detect camera lens contamination and to provide mitigation when lens contamination is detected. The system captures a new image (step  401 ) and divides the image into sub-regions (step  403 ) as illustrated in  FIG. 2 . The system evaluates the image data of the first sub-region and determines one or more metrics for the sub-region (step  405 ). Based on the image metrics, the system determines whether contamination is detected in the first sub-region (step  407 ). If no contamination is detected and additional sub-regions of the image remain for evaluation (step  409 ), the system move on to the next sub-region (step  411 ) and determines metrics for that sub-region (step  405 ). This is repeated until each sub-region in the image has been evaluated and then a new image is captured (step  401 ) and divided for sub-region-specific evaluation. 
         [0031]    If contamination is identified in one or more of the sub-regions (step  407 ), the system determines the type of contamination (step  413 ) and applies an appropriate mitigation technique to enhance the image to remove the distortion caused by the lens contamination (step  415 ). The system then determines one or more metrics for the enhanced image sub-region (step  417 ). If the restoration brings the image to a level of acceptable quality (step  419 ), then the system replaces the sub-region in the image with the enhanced sub-region image data and move on to the next sub-region for analysis. However, if the mitigation technique is unable to sufficiently improve the image quality and the enhanced image still fails to meeting acceptable levels (step  419 ), the system disables or limits the functionality provided by vehicle systems that rely upon image data (step  421 ) and sends a notification to the user (step  423 ) indicating that the camera system has been rendered at least partially “blind” by camera lens contamination. 
         [0032]      FIG. 5  illustrates an example of a mechanism for determining whether lens contamination is present and, if so, what type of contamination. The system compares the current image to one or more previous images captures by the camera (step  501 ) and analyzes variations in individual pixel values over time (step  503 ). As noted above, lens contamination due to a foreign object (e.g., dust or road salt) may completely obstruct the image behind the contamination. As a result, pixels that are affected by such contamination might exhibit no change or variation over time. If the system determines that one or more pixels in a given sub-region exhibit approximately zero variation over time (step  505 ), the system may conclude that the camera lens is contaminated with foreign debris (step  506 ). 
         [0033]    As also discussed above, if the camera lens is contaminated with water or condensation, the affected pixels will exhibit some variation over time. However, the variation will be slow changing. Therefore, if the system determines that one or more pixels in a given sub-regions exhibit some variation over time, but the degree of variation is below a given threshold (step  509 ), then the system may conclude that the camera lens is contaminated with condensation (step  511 ). 
         [0034]    As discussed above, the system may be configured with various different image processing techniques, each optimized to enhance an image based on a specific type of detected contamination.  FIG. 6  illustrates an example of one such method for enhancing an image after condensation contamination is detected. When the system concludes that condensation is present on the camera lens and is negatively impacting the quality of the output image (step  511 ), the system monitors a series of images over a period of time to develop a mask pattern for the affected sub-region (step  601 ). The mask pattern is then subtracted from the new camera image to enhance the image quality (step  603 ). 
         [0035]    The system then analyzes the quality of the enhanced image to determine whether the mitigation effectively restored the image quality to an acceptable level. In this example, the system is configured to apply the same metric and evaluation to the enhanced image as was applied to the raw image data. Specifically, the system analyzes variation of individual pixels within the enhanced image over time (step  605 ). If the enhanced image meets acceptable image quality standards (i.e., the pixel variation is greater than a threshold indicative of a quality image) (step  607 ), then the system concludes that the mitigation is acceptable (step  609 ). The enhanced image is displayed to the user and is made available to other vehicle assistance and automation systems. However, if the enhanced image still fails to meet acceptable image quality standards (i.e., the pixel variation is greater than zero, but fails to exceed the threshold indicative of a quality image) (step  607 ), then the system concludes that the camera system is at least partially blind in the given sub-region (step  611 ). 
         [0036]    Various implementations can be configured to respond differently to partially blind image data depending on the requirements of specific vehicle automation/assistance systems employed on the vehicle and user preference. For example, in some implementations, if one or more sub-regions of the image fail to meet acceptable standards, the image may still be displayed to the user on the user interface/display  107 . However, the system may be configured to indicate which sub-regions fail to meet quality standards by high-lighting the border of the particular sub-region(s) in a different color. In other implementations, the system may be configured to omit “blind” sub-regions from the displayed image data and will display image data only in sub-regions that meet the minimum quality standards. 
         [0037]    Some vehicle assistance/automation systems require varying degrees of image quality in order to perform effectively. For example, some systems may be configured to monitor adjacent lanes and to provide a warning to the driver when nearby vehicles are detected. These systems may require a lesser degree of image quality as there is less risk associated with an incorrect determination. In contrast, some systems are configured to automatically operate the vehicle to perform parallel parking for a driver. These systems may require a greater degree of image quality as there is substantial risk associated with incorrect operation of such a fully automated vehicle system. As such, the threshold used to determine whether a specific sub-region of an image is “blind” may be varied for individual systems that will utilize the image output. In other words, a given sub-region may be identified as “blind” and unusable for the automated parking system, but the same sub-region may be identified as acceptable for the lane monitoring/warning system. As such, in some situations, one vehicle system may be disabled due to the current amount of lens contamination while another vehicle system is allowed to continue operating. 
         [0038]    Similarly, different vehicle systems and functionality may require different fields of view. For example, a vehicle system may only analyze information regarding the road surface immediately behind the vehicle. For such systems, lens contamination that only affects the upper sub-regions of the camera output image would be of no concern. Therefore, some systems may be configured to disable certain specific functionality depending on which specific sub-regions of the camera system are determined to be effectively “blind.” 
         [0039]    Thus, various embodiments of the invention provide, among other things, a camera system that analyzes image data to detect camera lens contamination, determine a specific type of lens contamination, apply image enhancement routines to mitigate the specific type of detected lens contamination, and to disable certain vehicle functionality if the degree of lens contamination cannot be sufficiently mitigated. Various features and advantages of the invention are set forth in the following claims.