Sensor dirtiness detection

An occluded area on a vehicle sensor is identified by localizing the vehicle in space. A discrepancy between historic image data and real-time image data from the sensor is determined, and a cleaning actuator is actuated based on the determined discrepancy.

BACKGROUND

Safe and comfortable operation of the vehicle can depend upon acquiring accurate and timely information regarding the vehicle's environment. Vehicle sensors can provide data concerning routes to be traveled and objects to be avoided in the vehicle's environment. Safe and efficient operation of the vehicle can depend upon acquiring accurate and timely information regarding routes and objects in a vehicle's environment while the vehicle is being operated on a roadway. A vehicle may include one or more optical or image sensors such as camera sensors. Typically, such sensors include transparent surfaces, e.g., lenses, to protect an imaging sensor viewing an area outside of the vehicle and/or to focus incoming light beams on the imaging sensor. A transparent surface such as a camera lens is typically subject to environmental conditions, e.g., dust, insect impact, smudge, rain, fog, etc., that can impair visibility of the vehicle exterior. Further, an optical property of a transparent surface such as a lens may change due to degradation or damage, e.g., scratching, pitting, etc.

DETAILED DESCRIPTION

Introduction

Disclosed herein is a method including identifying an occluded area on a vehicle sensor by localizing the vehicle in space, determining a discrepancy between historic image data and real-time image data from the sensor, and actuating a cleaning actuator based on the determined discrepancy.

The historic image data may be based on data from at least on one of a second vehicle and a second sensor of the vehicle.

The historic image data may include 3D location coordinates.

The historic image data may include a classification for each point or a plurality of points, the method further comprising determining the discrepancy based at least in part on the classification of a point included in the real-time image data.

The classification may be at least one of a flat surface, a human, a vehicle, a construction, an object, a nature, and a sky class.

Determining the discrepancy may further include performing a perspective transformation of historic image data based on vehicle location coordinates and the vehicle orientation, identifying first feature points in the real-time image data and second feature points in the historic image data, performing a homography that includes a line-preserving projective mapping for the first and second feature points, and identifying a first portion of the real-time image data matching a second portion of the historic image data.

The method may further include identifying a classification of each feature in the historic image data, selecting static features based on the classification of the features, determining false positive and true positive classifications of the static features based on the real-time image data, and determine the discrepancy based on the determined true positive and false classifications and a confusion matrix including an average expected rate of misclassification for each class of features.

The method may further include determining a local discrepancy value for a location on a transparency of the sensor and a global discrepancy value for the transparency, and actuating the cleaning actuator upon determining that a difference between an average of the local discrepancy and the global discrepancy exceeds a threshold.

The static feature may be a feature of at least one of a flat, construction, and object classes.

Further disclosed herein is a system including a processor and a memory. The memory stores instructions executable by the processor to identify an occluded area on a vehicle sensor by localizing the vehicle in space, to determine a discrepancy between historic image data and real-time image data from the sensor, and to actuate a cleaning actuator based on the determined discrepancy.

The vehicle sensor may include a camera sensor, and the instructions may further include instructions to identify the occluded area in an optical path of the camera sensor.

The optical path may include at least one of a lens and a transparent exterior cover.

The occluded area may be an area in the optical path of the vehicle sensor that is covered by at least one of fog, water, smudge, dust, and scratch.

The occluded area may be an area of the optical path where an optical attribute of the optical path deviates from a specified optical property. The optical property may include at least one of a focal point and a distortion.

Further disclosed herein is a system including a vehicle camera sensor having an optical path, and a processor programmed to identify an occluded area on the optical path of the camera sensor by localizing the vehicle in space, determine a discrepancy between historic image data and real-time image data from the sensor, and to actuate a cleaning actuator based on the determined discrepancy.

The occluded area may be an area in an optical path of the vehicle sensor that is covered by at least one of fog, water, smudge, dust, and scratch.

The occluded area may be an area of the optical path where an optical attribute of the optical path deviates from a specified optical property. The optical property may include at least one of a focal point and a distortion.

The processor may be further programmed to identify a classification of each feature in the historic image data, to select static features based on the classification of the features, to determine false positive and true positive classifications of the static features based on the real-time image data, and to determine the discrepancy based on the determined true positive and false classifications and a confusion matrix including an average expected rate of misclassification for each class of features.

Further disclosed is a computing device programmed to execute any of the above method steps.

Yet further disclosed is a computer program product, comprising a computer readable medium storing instructions executable by a computer processor, to execute any of the above method steps.

System Elements

An occluded area on a vehicle sensor may be identified by localizing an ego vehicle in space and determining a discrepancy between historic image data and real-time image data from the sensor. A sensor cleaning system may be actuated based on the determined discrepancy. The historic image data may be based on data from a second vehicle and/or a second sensor of the ego vehicle. An occluded area on a vehicle sensor, e.g., on a lens, window, or windshield, may impair an ability of vehicle computer to detect object(s), to determine based on the received sensor data, and therefore, may impair an ability of the vehicle computer to navigate and/or localize the vehicle. Thus, the present system improves vehicle operation by detecting and/or remediating an occluded area of a vehicle sensor transparency, e.g., a lens. In the context of this disclosure, “occluded” with respect to a transparent surface such as a lens means a blockage that prevents or diminishes the passage of light. In the present context, “diminishing the passage of light” means reducing and/or manipulating (e.g., translucence) light while passing through. In the present context, “reducing” means a decrease of light intensity because of passing through the occluded area, e.g., rain drop (or film). Translucence is a physical property of allowing light to pass through a material diffusely. In addition, the blockage may result in a shift in perceived color of the environment from a transparent colored film. Additionally or alternatively, a blockage may result in a blurring of image or a localized distortion.

FIG. 1shows an example vehicle100which may include a computer110, actuator(s)120, sensors130such as a (Light Detection and Ranging) lidar sensor130, camera sensor130, GPS sensor130, radar sensor130, camera sensor130, etc., and a human machine interface (HMI140). A vehicle100may be powered in variety of ways, e.g., including with an electric motor and/or internal combustion engine. A vehicle100may include a reference point150, e.g., an intersection of a vehicle100longitudinal and lateral axes (the axes can define respective longitudinal and lateral center lines of the vehicle100so that the reference point150may be referred to as a vehicle100center point). In the present context, a vehicle100location refers to location coordinates of the vehicle100reference point150.

FIG. 1further shows a first coordinate system defined by an X axis170, Y axis180, and Z axis190, e.g., a Cartesian coordinate system, that is independent from the vehicle100location and/or orientation. The first coordinate system may be referred to as a “global” coordinate system because it is defined independently of a vehicle100and is typically defined for a geographic area, such as the coordinate system of a global positioning system (GPS) that is defined for the world. Alternatively or additionally, the first coordinate system could include any other location coordinate system providing geo-coordinates (i.e., latitude, longitude pairs) or the like.

The computer110includes a processor and a memory. The memory includes one or more forms of computer-readable media, and stores instructions executable by the computer110for performing various operations, including as disclosed herein.

The computer110may operate the vehicle100in an autonomous, semi-autonomous, or non-autonomous mode. For purposes of this disclosure, an autonomous mode is defined as one in which each of vehicle100propulsion, braking, and steering are controlled by the computer110; in a semi-autonomous mode the computer110controls one or two of vehicle100propulsion, braking, and steering; in a non-autonomous mode, a human operator controls vehicle propulsion, braking, and steering.

The computer110may include programming to operate one or more of vehicle brakes, propulsion (e.g., control of acceleration in the vehicle100by controlling one or more of an internal combustion engine, electric motor, hybrid engine, etc.), steering, climate control, interior and/or exterior lights, etc., as well as to determine whether and when the computer110, as opposed to a human operator, is to control such operations.

The computer110may include or be communicatively coupled to, e.g., via a vehicle communications bus as described further below, more than one processor, e.g., controllers or the like included in the vehicle for monitoring and/or controlling various vehicle controllers, e.g., a powertrain controller, a brake controller, a steering controller, etc. The computer110is generally arranged for communications on a vehicle communication network such as a bus in the vehicle such as a controller area network (CAN) or the like.

Via the vehicle network, the computer110may transmit messages to various devices in the vehicle100and/or receive messages from the various devices, e.g., the sensor130, actuators120, etc. Alternatively or additionally, in cases where the computer110actually comprises multiple devices, the vehicle communication network may be used for communications between devices represented as the computer110in this disclosure. Further, as mentioned below, various controllers and/or sensors130may provide data to the computer110via the vehicle100communication network.

The vehicle100actuators120may be implemented via circuits, chips, or other electronic components that can actuate various vehicle subsystems in accordance with appropriate control signals as is known. The actuators120may be used to control braking, acceleration, and steering of the vehicle100. As an example, the vehicle100computer110may output control instructions to control the actuators120.

In addition, the computer110may be programmed to communicate through a wireless communication network with, e.g., a remote computer. The wireless communication network, which may include a Vehicle-to-Vehicle (V-to-V) and/or a Vehicle-to-Infrastructure (V-to-I) communication network, includes one or more structures by which the vehicles100, the remote computer, etc., may communicate with one another, including any desired combination of wireless (e.g., cellular, wireless, satellite, microwave and radio frequency) communication mechanisms and any desired network topology (or topologies when a plurality of communication mechanisms are utilized). Exemplary V-to-V or V-to-I communication networks include cellular, Bluetooth, IEEE 802.11, dedicated short range communications (DSRC), and/or wide area networks (WAN), including the Internet, providing data communication services.

The vehicle100may include one or more sensor(s)130that provide data from detecting physical phenomena (e.g., light, sound, electricity, magnetism, etc.) from spaces encompassing at least some of an interior and/or exterior of the vehicle100. With reference toFIGS. 1 and 2A-2B, a vehicle100may include one or more camera object detection sensor(s)130, e.g., lidar, radar, and/or camera sensor130. A camera sensor130may provide image data from an area within a field of view290of the camera sensor130. A camera sensor130may include a housing210, an image sensor220, and an optical path230. The camera sensor130may receive light beams from an area within the field of view290of the sensor130and may generate an electrical signal based on the received light beams, e.g., in a conventional manner. The housing210may be formed of plastic, metal, etc., that encloses components of the camera sensor130. The image sensor220may include an electronic receiver, e.g., a charge-coupled device (CCD) or Complementary metal-oxide-semiconductor (CMOS), that receives light beams (or light rays), e.g., light reflected by an object260, and generates electrical signals based on the received light beams. The optical path230may include one or more lenses240that are transparent to light beams (e.g., to light beams within a specific wavelength range such as human visible light range) and focus incoming light beams onto the image sensor220.

With reference toFIG. 3, the computer110may be programmed to generate an image300based on electrical signals from the image sensor220. An image300, in the present context, is a digital image that is storable in a computer110memory. A digital image may include a plurality of pixels and the image data may include data such as image intensity and/or color associated with each one of the pixels. The lenses240may be formed of plastic and/or glass. Further, the optical path230may include one or more other transparent components such as a light filter, polarizer, a protection cover, etc. In another example, the optical path230may include a portion of a vehicle100transparent exterior cover, e.g., a portion of the windshield positioned within a field of view a forward-looking camera sensor130mounted behind the vehicle100windshield providing image data from an area outside of the vehicle100. In the present disclosure, “transparency” includes any of transparent components of the optical path230.

With reference toFIGS. 1-3, the computer110may be programmed to receive, from the camera130, an image300that includes an area within the field of view290of the camera130.FIG. 2Bshows a two-dimensional Cartesian coordinate system defined by an X′ axis270and a Y′ axis280which specifies coordinates of points on a surface of the lens240and/or any transparent component within the optical path230.FIG. 3shows an example image300received from the camera sensor130ofFIG. 2A, and a two-dimensional Cartesian coordinates system defined by an X″ axis320and a Y″ axis330. A light beam originated (e.g., reflected and/or illuminated) from a point x, y, z on the object260may pass through a point x′, y′ of the lens240and result in a point (or pixel) with coordinates x″, y″ in the image300. The computer110may be programmed to identify the location coordinates x′, y′ based on the location coordinates x″, y″ and optical attributes of the optical path230. For example, a lens240may cause a distortion of light beams passing through the lens240, resulting in warping images captured by a camera130. A distortion attribute of the lens240may be specified with a mathematical model, e.g., defined based on a lens240specification, e.g., fish-eye effect, barrel effect, etc. The computer110may be programmed, based on the mathematical model of the lens240, to determine the coordinates of a point x′, y′ based on the coordinates x″, y″ in the image300.

If the point specified by coordinates x′, y′ on the transparency of the sensor130is partially or fully blocked, e.g., by a smudge, fog, etc., the image data for the pixel x″, y″ of the camera sensor130may be incorrect (i.e., may not reflect the light beams received from the point x, y, z). This may result in a misclassification of the object260by a computer110that is programmed to detect objects260based on the image data received from the camera sensor130.

With reference toFIG. 1, the HMI140may be configured to receive information from a user during operation of the vehicle100. Moreover, a HMI140may be configured to present information to the user. Thus, a HMI140may be located in the passenger compartment of the vehicle100. In one example, the computer110may be programmed to output a message to the HMI140indicating that the optical path230is fully or partially occluded, e.g., because of rain, smudge, fog, etc. and/or degrading of sensor130components such as the lens240, etc.

With reference toFIG. 4, the vehicle100may include an orientation sensor130that provides data indicating a current roll, pitch, yaw, and/or vertical position of the vehicle100. An orientation sensor130, e.g., an inertial measurement unit (IMU), is an electronic device that measures an orientation of a body (e.g., a vehicle100body) using a combination of accelerometers, gyroscopes, and/or magnetometers. An orientation of the vehicle100to a reference such as ground level includes a scalar three-dimensional vector with a specified origin, e.g., at the vehicle100reference point150, indicating a direction of the vehicle100relative to a reference three-dimensional coordinates system, e.g., the global coordinate system discussed above. For example, the orientation may include an algebraic sum, such as is known, of various independent vectors, each indicating a direction of the vehicle relative to a respective reference direction, e.g., a pitch, a yaw, and a roll of the vehicle100. Additionally or alternatively, the orientation may include a three-dimensional vector including longitudinal, lateral, and vertical x, y, z coordinates with reference to the X, Y, Z axes170,180,190.

FIG. 5shows an example real-time image500received from a vehicle100camera sensor130that has an occluded area510, e.g., a smudge, water, dust, scratch, insect impact, etc., on the sensor130transparency, e.g., a lens240, windshield, etc. Additionally or alternatively, an occluded area510may be an area of the optical path230, e.g., in and/or on the lens240, the windshield, etc., where an optical attribute of the optical path230deviates from a specified optical attribute. The optical attribute (or optical property) may be a focal point, a lens240distortion model parameter, etc. A deviation of an optical attribute may be a result of aging of the glass and/or plastic, physical damage, e.g., a scratch, a degradation of a physical component, e.g., glue, of the optical path230that causes a misalignment and/or out of focus conditions of the optical path230. A deviation of an optical attribute may be a result of environmental conditions such as a change in temperature, humidity, vibration, etc. An occluded area510may be a result of a full or partial blockage. In one example, a partial blockage is a result of a translucent material such as fog, rain, etc. that allows a partial passing of light while diffusing the light. In another example, a full blockage may be a result of, e.g., smudge, bug impact, etc.

FIG. 6shows a historic image600at a same location of receiving the real-time image500from the vehicle100sensor130. The historic image data (or historical map image data or historical map data) include a collection of image data associated to a geographical area, e.g., collected by a mapping vehicle including camera sensor130, location sensors130. A geographical area (or simply area) in the context of this disclosure means a two-dimensional area on the surface of the earth. An area may have any dimensions and/or shape, e.g., rectangular, oval, circular, non-geometrical shape, etc. For example, an area may include a neighborhood, a town, an airport, etc. In the present context, a “space” is a three-dimensional (3D) volume, e.g., above a geographical area that is thereby a bottom of the space. Thus, a space can include buildings, objects, etc., within a geographical area. The historic image data may be received from a remote computer, stored in a computer110memory, received from a second camera sensor130in the vehicle100, etc.

In the present context, the ‘historic image data” (or historic map data or historic map image data) includes image data captured by a second vehicle100and received from a second vehicle100computer, a second sensor130of the vehicle100prior to a time of the real-time image data collection. In one example, historic image data, e.g., the image600, may be collected days, months, etc., before a current time, e.g., by a mapping vehicle. In another example, the historic image600data may be collected by a second vehicle100in a same location minutes or seconds before the current time and received via vehicle-to-vehicle communications. In yet another example, the historic image600data may be collected by a second sensor130of the vehicle100. In yet another example, the historic image600data may include data, e.g., image data, lidar data, etc., collected from a second sensor130of the vehicle100having a second field of view290that overlaps with the field of view290of the camera sensor130. Objects in the space may cause occlusions, e.g., a vehicle standing fully or partially in front of a traffic sign may block a view of the traffic sign by the camera sensor130. The computer110may be programmed to detect such an occlusion based on data received from the second camera sensor130, map data, etc.

Table 1 above shows an example object classification scheme. The historic image600data may further include 3D (three dimensional) location coordinates of features included in the image data, e.g., location coordinates x, y, z of points on buildings, bridges, street surface, traffic signs, etc. Further, the historic image600data may include a classification (or class) of each portion, e.g., a pixel or a plurality of pixels, of historic image600data, e.g., point of the example image600, as shown inFIG. 6. A classification may be at least one of a flat surface, a human, a vehicle, a construction, an object, and a nature class. With reference to Table 1, each class, e.g., vehicle, may have multiple sub-classes, e.g., car, truck, bus, etc. In one example, the classification of points in the historic images600may be generated by an image processing algorithm that identifies the classes based on collected images600. Additionally or alternatively, a classification may be performed based on other techniques, e.g., continuous numerical regression outputting monocular depth map, etc.

With reference toFIGS. 5-6, the computer110can be programmed to identify an occluded area510on a vehicle100sensor130by localizing the vehicle100in space, determining a discrepancy between historic image600data and real-time image500data, and to actuate a cleaning actuator120based on the determined discrepancy. Additionally or alternatively, the computer110may be programmed to transmit discrepancy data to a remote computer via the wireless communication network. The remote computer may be programmed to update the historic map data (or historic image data) based on the received discrepancy data. Additionally or alternatively, the computer110may be programmed to adjust logic of object detection, perception, etc. For example, the computer110may be programmed to ignore data received from the sensor130and operate the vehicle100based on data received, e.g., from a second sensor130, a remote computer, etc.

In the present context, “localizing” a vehicle100includes determining vehicle100location coordinates and a vehicle100orientation, i.e., a vehicle100yaw, roll, and/or pitch. With reference toFIG. 4, the vehicle100location coordinates may include longitudinal, lateral, altitude coordinates, e.g., with respect to a global location coordinate system, e.g., GPS location coordinates. The computer110may be programmed, based on conventional localization techniques, to determine the location coordinates of the vehicle100based on data received from the vehicle100camera sensor130, lidar sensor130, etc., and the historic image data, e.g., lidar point cloud data, etc.

As discussed above with reference toFIGS. 2A-2B, the occluded area510in the image500may be a result of a smudge, fog, dust, scratch, etc., on the transparency of the optical path230. For example, based on point coordinates on a perimeter of the occluded area510, the computer110may be programmed to determine the location coordinates of points on a perimeter of occluded area on the transparency (e.g., with reference to the two-dimensional coordinate system with X′, Y′ axes270,280). Thus, upon determining location coordinates x″, y″ of the occluded area510in the image500with respect to X″, Y″ axes320,330, the computer110may be programmed to determine the location coordinates of an occluded area on the transparency with respect to X′, Y′ axes270,280, as discussed above with respect toFIGS. 2A-2B.

In the present context, a “discrepancy” is a quantifier for measuring a mismatch of the real-time data compared to the historic data, e.g., a mismatch in identified classes. Additionally, a “discrepancy” may include a quantifier describing differences resulting from computation of the images, e.g. monocular depth map algorithm output. For example, when one or more pixels of the historic image600on the traffic sign520is identified as a different class, e.g., a building, etc. Thus, the computer110may be programmed to determine the discrepancy based at least in part on the classification of a point included in the real-time image data500, e.g., a point within the occluded area510of the image500. The computer110may be programmed to actuate a cleaning actuator120, e.g., a wiper, a sprayer pump, etc., upon determining that the determined discrepancy exceeds a threshold, as discussed below with reference toFIG. 12.

With reference to images700,800ofFIGS. 7-8, the computer110may be programmed based on an image processing technique such as feature registration, to identify the first and second features in the real-time and historic images500,600. Each of plus (+) signs shown in the images700,800represent an example feature of images500,600. The features may include points, lines, edges, corners, and/or other geometric entities found in the images500,600.

In order to determine a discrepancy, the features identified in images500,600may be compared. Thus, the first and second features may be matched prior to identify discrepancies between the images500,600. In the present context, “matching” means recognizing a feature identified in image500in the image600or vice versa. However, the image600may be received by the mapping vehicle100camera sensor130, second vehicle100, and/or the second sensor130of the vehicle100from a different location and/or orientation. For example, as shown inFIGS. 5-6, a location and/or orientation of an image capturing device, e.g., a second sensor130on a mapping vehicle100can be different from the location and/or orientation of the vehicle100sensor130at time of receiving the real-time image500. This may be a result of a mapping vehicle100moving in a different lane of a road compared to the vehicle100, and/or camera sensor(s)130of the mapping vehicle100, second vehicle100, or second sensor130may be at a different elevation or height from a road surface and/or have a different mounting orientation compared to the orientation of the vehicle100sensor130with respect to the vehicle100reference point150.

In the present context, “perspective” is a combination of a location and/or orientation of a camera sensor130. For example, perspectives of the example images500,600of theFIGS. 5 and 6are different. Thus, in identifying and comparing, e.g., the traffic sign520in the images500,600a perspective transformation may be performed. In one example of different perspectives,FIGS. 9A-9Cillustrate a flower viewed from different perspectives, e.g., a camera taking an image from three different location and/or orientations relative to the flower. A computer110may be programmed to identify feature points of the flower in each of the images9A-9C and to perform (or compute) a homography or projective transformation to match pixels (or points) of the flower in theFIGS. 9A-9C. As discussed above, the historic image600data may lack an image600with identical perspective as the real-time image500. Thus, utilizing a homography technique, the computer110may be programmed to identify a projective transformation between the first and second features of the images500,600, as discussed below.

The computer110may be programmed to identify first feature points in the real-time image500data and second feature points in the historic image600data, to perform a homography for the first and second feature points, and to identify a first portion of the real-time image500data matching a second portion of the historic image600data.

In the present context, a “homography” or a “perspective transformation” is a line-preserving projective mapping of points observed from two different perspectives. “Line preserving” means that if multiple points are on a same line in the real-time image500, the same points are on a same line in the historic map image600. A homography of the features identified in the images500,600may return a homography matrix that transforms the location coordinates of the feature points. In other words, the homography provides a mathematical relationship between the coordinates of the points of the images500,600.

Now turning toFIGS. 7-8, the real-time and historic images500,600may partially overlap with respect to the feature detected in the images500,600. For example, some of windows540of a building530included in the image500are missing in the image600. Thus, with reference toFIG. 10, the computer110may be programmed to identify a matched portion1000of the real-time image500and the historic map image600based on the performed homography. A matched portion1000includes features for which the computer110has identified a homography matrix, i.e., can match based on an identified mathematical mapping. In other words, the matched portion1000includes the features which are identified in the real-time image500and the historic image600.FIG. 10shows only an example of a matched portion1000. A matched portion1000may have any shape based at least on the perspectives of the images500,600relative to one another.

As discussed above, the computer110can be programmed to identify an occluded area510based on the discrepancy of real-time image500and the historic image600. However, a discrepancy may result from a moving vehicle, pedestrian, growing vegetation, moving clouds in the sky, etc. In other words, the discrepancy may be a result of an occlusion, e.g., a pedestrian standing in front of a traffic sign. In the present context, “occlusion” is resulted from a feature in the space preventing a viewing of a static feature. In one example, the computer110may be programmed to identify static features in the real-time image500and to determine the discrepancy based on the static features. In the present context, a static feature is a feature that is not moving, e.g., classified as a flat, construction, and/or object class (see Table 1). Further, a class may include static and/or non-static sub-classes. In one example, a sky sub-class is a non-static sub-class because of changes to vegetation in different seasons, changes to sky because of weather conditions and time of day, etc. In other words, a static feature is a feature that is not expected to move, e.g., a traffic sign520. For example,FIG. 11illustrates an example of a matched portion1100with static features generated based on the matched portion1000ofFIG. 10. Non-static features typically change over time, e.g., vegetation changes based on change of seasons, growing, etc.

The computer110may be programmed to identify a classification of each feature in the historic image600data (based on image processing techniques), to select static features based on the classification of the features, determine “false positive” and “true positive” classifications of the static features based on the real-time image500data. In the present context, a “true positive” or “true” classification is determined when the computer110identifies a same class for a feature in both real-time and historic map images500,600. A “false positive” (or “misclassification” or “misdetection”) is determined when the computer110identifies different classes for a feature based on the images500,600. As discussed above, a location of a feature in an image500,600may be specified with coordinates of one or more pixels in the images500,600with respect to the X″ and Y″ axes320,330. Thus, to determine a true classification or misclassification, the computer110may be programmed to compare classes identified with respect to coordinates x″, y″ of the matched features in either of the images500,600.

The computer110may be programmed to determine a true classification upon determining that the area of images500,600covered by the traffic sign520is identified in the real-time image500and the historic image600to have a class “object” and a sub-class “traffic sign,” as discussed above with reference to Table 1. The computer110determines a misclassification or false positive, upon determining a class different than “object” and/or a sub-class different than the “traffic sign” sub-class.

The computer110may be programmed to determine “false positive” or “true positive” detections based on static features, i.e., ignoring the features with a class that is determined to be non-static, as shown inFIG. 11. In one example, assuming the historic image600data to be more reliable to make such determination compared to real-time image500data, the computer110may be programmed to identify the static features based on the historic map image600and to ignore the areas (or pixels) of the image500which based on image600included non-static features.

As discussed above, in one example, the historic image600data may include classification data of the features included in the images600. Thus, a “false positive” may be determined upon determining that the computer110identifies a class for a feature in the image500that is different from the classification stored in the historic map image600data for the respective feature. In another example, the computer110may be programmed to identify a class for the features in the real-time and historic map images500,600. Thus, a misclassification or false positive is determined when results of classifications of a feature in the images500and600are different.

The computer110may be programmed to determine the discrepancy based on the determined true classifications and/or misclassifications. In one example, the computer110may be programmed to determine the discrepancy based on (i) the determined true classifications and misclassifications, and (ii) a confusion matrix1200(seeFIG. 12) including an average expected rate of misclassification for each class of features.

A confusion matrix, in the present context, is a matrix including a statistic, e.g., a percentage, of true classifications and misclassifications of various types of features in normal operating conditions. Normal conditions, in the present context, mean that substantially no occluded area510is present on the transparency of the camera sensor130and misclassifications are results of other factors such as a weather condition, ambient light condition, optical attribute of the optical path230, precision and recall of image processing technique, etc. In the present context, “precision” is a fraction of relevant instances among retrieved instances, while “recall” is a fraction of relevant instances that have been retrieved over a total amount of relevant instances. In the present context, a statistic of classification in the confusion matrix1200means a rate of a detection of a specific feature, e.g., performing a classification by image processing a specified number of times, e.g., 1000 times, on various image500data and determining a number of times that the car was classified as a car or any other type of feature. This may be performed in a lab and the resulting confusion matrix may be stored in form of a table in a memory of the computer110and treated as a nominal performance of the system without obstructions, e.g., the occluded area(s)510. Individual pixels, sub-regions, and/or whole image confusion matrices1200may be used for detecting occluded areas510. In addition to confusion matrix1200or alternatively, the computer110may be programmed to determine discrepancies based on other techniques, e.g., Jaccard Index, commonly known as PASCAL VOC intersection-over-union metric. The Jaccard Index is a statistical method for comparing a similarity and a diversity of sample sets.

FIG. 12shows an example confusion matrix1200. Each row of the example confusion matrix1200shows a true class of a feature, whereas each column of the matrix1200shows a classification result based on real-time image500data. Thus, the statistic shown in the main diagonal of the matrix1200represents true classifications, i.e., a classification of real-time image500is the same as true classification based on historic map image600data. For example, an entry1210of the matrix1200shows a true classification statistic of 35.1% for detecting traffic signs, e.g., the traffic sign520of image500. Further, entries outside the main diagonal of the matrix show a misclassification of features. For example, the matrix1200shows a statistic of misclassification of traffic signs as cars (i.e., classifying a traffic sign520as a car) to be 5.9%.

Continuing to refer to the example confusion matrix1200, a misclassification of a feature may be expected, i.e., even based on image500data received from a camera sensor130lacking an occluded area510, misclassifications may be expected. Thus, the computer110may be programmed to determine the discrepancy, in the present context, based on a deviation of true classification from an expected static as included in a confusion matrix1200. For example, the computer110may be programmed to determine whether the two distributions (i.e., confusion matrix1200and distributions calculated by the computer110based on true classifications and misclassifications) are significantly different using techniques such as Z-test, Chi-Squared, etc. In one example, the computer110may be programmed to determine no occluded area present at a location of transparency of the optical path230, at which the traffic sign520is viewed, upon determining that the traffic sign520is classified in 33% of times with a traffic sign class.

In another example, the computer110may be programmed to determine an occluded area510at a location of transparency, at which the traffic sign520is viewed, upon determining that the traffic sign520is classified in a rate that is at least 10% lower than expected true detection rate (e.g., a true detection rate of 15% which is more than 10% lower than 35.1%, i.e., a reduction in true classification exceeding a threshold of 10%).

In present context, a statistic of classification of features in the real-time images500, is a rate, e.g., specified as a percentage, of a classification compared to a number of classifications, e.g., based on multiple images500captured while the vehicle100views, e.g., the traffic sin520, and/or multiple detections including detection of the traffic sign520in multiple days because the vehicle100often passes same location and views the same traffic sign520at a same location of the transparency of the camera sensor130. For example, a percentage rate may specify a number of true classifications to a total number of classifications (i.e., true classifications and misclassifications). Additionally or alternatively, the computer110may be programmed to determine the discrepancy based on various types of classes and to determine a running average discrepancy based on type of the identified classes (i.e., each sample discrepancy determined based on expected true classification statistic for the respective class).

As discussed above, the confusion matrix1200includes a statistic of true or false classification of features. Under a temporary condition, e.g., extremely low ambient condition, a rate of true detection of a feature in the image500may be temporarily lower than a threshold, although there may not be an occluded area510on the transparency of the camera sensor130. In one example, to improve a detection of an occluded area510, the computer110may be programmed to determine a local discrepancy value for a location on a transparency of the sensor130and a global discrepancy value for the transparency, and to actuate the cleaning actuator120upon determining that a difference between an average of the local discrepancy and the global discordancy exceeds a threshold.

For example, the computer110may be programmed to determine a local discrepancy to be a discrepancy of each pixel of the image (i.e., corresponding to specific point(s) of the transparency as discussed with reference toFIGS. 2-3), and to determine the global discrepancy value to be an average of local discrepancy values determines for entire surface of the transparency of the sensor130. In one example, the computer110may be programmed to actuate the cleaning actuator120upon determining that a difference of at least a local discrepancy value and the global discrepancy value exceeds 10%.

FIG. 13shows an example process1300for operating a vehicle100cleaning actuator120. The computer110may be programmed to execute blocks of the process1300.

The process1300begins in a block1310, in which the computer110receives historic map image600data a geographical area including a current geographical location of the vehicle100. Additionally, the historic map image data may further include classification of feature points in the images600.

Next, in a block1315, the computer110receives real-time sensor130data. The computer110may be programmed to receive image600data from a vehicle100camera sensor130. Further, the computer110may be programmed to receive data from a GPS sensor130, an object detection sensor130such as a lidar, radar sensor130, etc.

Next, in a block1320, the computer110localizes the vehicle100. The computer110may be programmed to determine location coordinates and/or the orientation of the vehicle100based on received sensor130data and the received historic map image600data.

Next, in a block1325, the computer110determines the historic map image600data at the current location of the vehicle100. The computer110may be programmed to determine an image600from the historic map image data based on determined location coordinates of the vehicle100and the orientation of the vehicle100.

Next, in a block1330, the computer110determines features of the real-time image500and the historic map image600of the current location of the vehicle100. The computer110may be programmed, e.g., based on a feature registration technique, to identify the features in the images500,600.

Next, in a block1335, the computer110performs homography on identified features of the real-time image500and the historic map image600of the current location and orientation of the vehicle100. The computer110may be programmed to identify features of real-time image500that can be mapped through a perspective transformation to feature identified in the historic map image600.

Next, in a block1340, the computer110identifies a matched portion1000of the real-time and historic map images500,600. The computer110may be programmed to identify the portion1100of the image500that is included in the historic map image600. In other words, the computer110may be programmed to exclude portions of the image500that include feature which lack a perspective transformation to a feature of the historic map image600based on the performed homography.

Next, in a block1345, the computer110identifies static features of the historic map image600and generates a matched portion1100with static features. In one example, the computer110may be programmed to identify the static features of the image600based on classification data stored in the historic map image600data. Additionally or alternatively, the computer110may be programmed to classify the features of the historic map image600based on an image processing technique.

Next, in a block1350, the computer110classifies features of the real-time image. The computer110may be programmed to identify a class and/or sub-class of features based on a table of class types such as Table 1. The computer110may be programmed to store the class and/or sub-class of each feature based on the location of the feature in the image600, e.g., location coordinates x″, y″ with reference to X″, Y″ axes320,330. In one example, the computer110may be programmed to perform classification of features only in portions of the image500, in which static features are located (based on the matching location of static features identified in the historic map image600).

Next, in a block1355, the computer110determines a discrepancy. The computer110may be programmed to determine false positive and true positive classifications of static features, and to determine local discrepancy values and global discrepancy values based on the identified true classifications and misclassifications.

Next, in a decision block1360, the computer110determines whether the discrepancy exceeds a threshold. In one example, the computer110determines whether a deviation between a true classification of features in a location on the transparency of the sensor130exceeds a threshold, e.g., 10%. The threshold may be determined based on a statistical analysis process, such as the Cochran-Mantel-Haenszel test or more generally conditional logistic regression, in the case of use of classification algorithms, to determine a significant difference or a difference above or below some threshold in the current and optimal sensor and logic related error rates. Additionally or alternatively, there are other methods known in the art to distinguish the performance between two classifiers. A p-value of such a method may be determined based on tradeoffs between false positive by setting the p-value too low and the risk of not cleaning a dirty sensor by setting the p-value too high. Additionally or alternatively, a repetitive statistical method may be utilized, e.g., performing a statistical method cyclically, e.g., each minute. To achieve a high confidence, e.g., 95%, that the sensor130surface250is dirty in a specified time span, e.g., 10 minutes, given performing the repetitive method every minute, the p-value per every check may be set at 0.994883%. Other confidence levels may be warranted such as in the case to avoid wasting washer fluid. Additionally or alternatively, the threshold may be determined based on a classifier accuracy or specific inter-class error, e.g., based on normal sensor130and algorithm performance variation when the sensor130surface250is not dirty. For example, lighting, weather patterns, class variation observed over time, and/or other factors may affect the performance variation. A threshold above this variation would be useful to differentiate natural variation from sensor dirtiness and may be set above the performance variation distribution (e.g. mean+2.5*sigma of performance variation). Additionally or alternatively, the threshold may be set based on simulated safety decrease in vehicle100operation. For example, simulation and/or real-world data may be used to identify test scenarios if grouping of pixels with misclassification errors at various levels (equivalent to potential thresholds) would change the vehicle100operation that results in changes to perception or motion planning that may be potentially detrimental to vehicle100operation. In one example, upon determining that based on performed test a virtual threshold of 15% is determined, then a 10% threshold may be used, assuming a 5% safety threshold. In another example, the computer110determines whether a difference between a local discrepancy and the global discrepancy exceeds a threshold, e.g., 10%. Additionally or alternatively, class-to-class misclassification error thresholds may be set higher or lower. For example, misclassifying a roadway as a person would lead to a more severe error than classifying a bike lane as a sidewalk which may not result in changes in vehicle100motion planning. If the computer110determines that the discrepancy exceeds the threshold, then the process1300proceeds to a block1365; otherwise the process1300ends, or alternatively, returns to the block1310, although not shown inFIG. 13.

In the block1365, the computer110actuates a cleaning actuator120. The computer110may be programmed to actuate a wiper actuator120and/or a spray pump actuator120to clean the transparency of the sensor130, e.g., an outer surface of sensor130transparent cover, a lens240, etc. Following the block1365, the process1300ends, or alternatively, returns to the block1310, although not shown inFIG. 13.

Computer-executable commands may be compiled or interpreted from computer programs created using a variety of programming languages and/or technologies, including, without limitation, and either alone or in combination, Java™, C, C++, Visual Basic, Java Script, Perl, Python, HTML, etc. In general, a processor (e.g., a microprocessor) receives commands, e.g., from a memory, a computer-readable medium, etc., and executes these commands, thereby performing one or more processes, including one or more of the processes described herein. Such commands and other data may be stored in files and transmitted using a variety of computer-readable media. A file in a computing device is generally a collection of data stored on a computer readable medium, such as a storage medium, a random-access memory, etc.

The term “exemplary” is used herein in the sense of signifying an example, e.g., a reference to an “exemplary widget” should be read as simply referring to an example of a widget.