Patent Description:
Non-patent Literature "Explaining and Harnessing Adversarial Examples" provides several machine learning models, including neural networks, consistently misclassify adversarial examples-inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence.

Non-patent Literature "A Survey of Adversarial Machine Learning in Cyber Warfare" provides threat models for Machine Learning systems and describe the various techniques to attack and defend them.

Non-patent Literature "Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks" provides a model that can be used to harden DNN models by detecting adversarial examples.

Image sensors are an important component in autonomous and partially autonomous driving systems. However, in some situations image data from image sensors may be incorrectly analyzed. In, for example, situations such as adverse weather conditions (for example, rain), low light, and direct light from the sun, image analysis software may incorrectly generate a prediction regarding the existence of objects in an image, the location of objects in the image, the existence of road markings in an image, the location of road markings in the image, or a combination of the foregoing. The reason the image analysis software makes an incorrect determination in some situations is that the image analysis software has not been configured to analyze image data obtained in these situations. Therefore, in some embodiments, it is desirable that image data be ignored when the image data received from a camera is obtained in a situation that the image analysis software is not configured to analyze. In some embodiments, it is desirable that at least some autonomous functionality of a vehicle be disabled when the image data received from a camera is obtained in a situation that the image analysis software is not configured to analyze. Embodiments herein describe a method and system for detecting whether received image data was obtained in a situation that image analysis software is not configured to analyze.

The present invention provides a system for determining whether image data is within a predetermined range that image analysis software is configured to analyze, as defined by independent claim <NUM>. The system includes a camera and an electronic processor. The electronic processor is configured to receive the image data from the camera and, use the image analysis software, generate a prediction regarding the image data and a confidence value associated with the prediction regarding the image data. The electronic processor is configured to calculate perturbation values by determining a sign of a gradient of a cost function (∇xL(θ, x, y)) of the image analysis software used to generate the prediction with respect to the image. The electronic processor is configured to multiply a weight (ε) by the sign, wherein x is the input image data, y is possible classifications for the image data, and θ are parameters of the image analysis software used to generate the prediction. The electronic processor is also configured to perturb the image data using perturbation values, use the image analysis software, generate a prediction regarding the perturbed image data and a confidence value associated with the prediction regarding the perturbed image data. The electronic processor is further configured to compare the confidence value associated with the prediction regarding the perturbed image data to the confidence value associated with the prediction regarding the image data. When the difference between the confidence value associated with the prediction regarding the perturbed image data and the confidence value associated with the prediction regarding the image data is bigger than or equal to a predetermined threshold value, the electronic processor is configured to determine that the image data received from the camera lies inside the predetermined range. The image analysis software comprises a machine learning software trained to analyze the image data and the machine learning software comprises a convolutional neural network.

The present invention provides a method of determining whether image data is within a predetermined range that image analysis software is configured to analyze, as defined by independent claim <NUM>. The method includes receiving, with an electronic processor, the image data from the camera and, using the image analysis software, generating a prediction regarding the image data and a confidence value associated with the prediction regarding the image data. The method also includes calculating perturbation values by determining a sign of a gradient of a cost function (∇xL(θ, x, y)) of the image analysis software used to generate the prediction with respect to the image and multiplying a weight (ε) by the sign, wherein x is the input image data, y is possible classifications for the image data, and θ are parameters of the image analysis software used to generate the prediction. The method also includes perturbing the image data using perturbation values, using the image analysis software, generating a prediction regarding the perturbed image data and a confidence value associated with the prediction regarding the perturbed image data. The method further includes comparing, with the electronic processor, the confidence value associated with the prediction regarding the perturbed image data to the confidence value associated with the prediction regarding the image data. The method also includes determining that the image data received from the camera lies inside the predetermined range when the difference between the confidence value associated with the prediction regarding the perturbed image data and the confidence value associated with the prediction regarding the image data is bigger than or equal to a predetermined threshold value. The image analysis software comprises a machine learning software trained to analyze the image data and the machine learning software comprises a convolutional neural network.

Other aspects, features, and embodiments will become apparent by consideration of the detailed description and accompanying drawings.

Before any embodiments are explained in detail, it is to be understood that this disclosure is not intended to be limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. Embodiments are capable of other configurations and of being practiced or of being carried out in various ways.

A plurality of hardware and software based devices, as well as a plurality of different structural components may be used to implement various embodiments. In addition, embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic based aspects of the invention may be implemented in software (for example, stored on non-transitory computer-readable medium) executable by one or more processors. For example, "control units" and "controllers" described in the specification can include one or more electronic processors, one or more memory modules including non-transitory computer-readable medium, one or more input/output interfaces, one or more application specific integrated circuits (ASICs), and various connections (for example, a system bus) connecting the various components.

<FIG> illustrates a system <NUM> for determining whether image data is within a predetermined range that image analysis software is configured to analyze. The system <NUM> includes a vehicle <NUM>. The vehicle <NUM>, although illustrated as a four-wheeled vehicle, may encompass various types and designs of vehicles. For example, the vehicle <NUM> may be an automobile, a motorcycle, a truck, a bus, a semi-tractor, and others. In the example illustrated, the vehicle <NUM> includes an electronic controller <NUM>, a camera <NUM>, a steering system <NUM>, an accelerator <NUM>, and brakes <NUM>. The components of the vehicle <NUM> may be of various constructions and may use various communication types and protocols.

The electronic controller <NUM> may be communicatively connected to the camera <NUM>, steering system <NUM>, accelerator <NUM>, and brakes <NUM> via various wired or wireless connections. For example, in some embodiments, the electronic controller <NUM> is directly coupled via a dedicated wire to each of the above-listed components of the vehicle <NUM>. In other embodiments, the electronic controller <NUM> is communicatively coupled to one or more of the components via a shared communication link such as a vehicle communication bus (for example, a controller area network (CAN) bus) or a wireless connection.

Each of the components of the vehicle <NUM> may communicate with the electronic controller <NUM> using various communication protocols. The embodiment illustrated in <FIG> provides but one example of the components and connections of the vehicle <NUM>. However, these components and connections may be constructed in other ways than those illustrated and described herein. For example, it should be understood that the electronic controller <NUM> may include one or more cameras than the single camera <NUM> illustrated in <FIG> and that the cameras included in the vehicle <NUM> may be installed at various locations on the interior and exterior of the vehicle <NUM>.

<FIG> is a block diagram of the electronic controller <NUM> of the vehicle <NUM>. The electronic controller <NUM> includes a plurality of electrical and electronic components that provide power, operation control, and protection to the components and modules within the electronic controller <NUM>. The electronic controller <NUM> includes, among other things, an electronic processor <NUM> (such as a programmable electronic microprocessor, microcontroller, or similar device), a memory <NUM> (for example, non-transitory, machine readable memory), and an input/output interface <NUM>. The electronic processor <NUM> is communicatively connected to the memory <NUM> and the input/output interface <NUM>. The electronic processor <NUM>, in coordination with the memory <NUM> and the input/output interface <NUM>, is configured to implement, among other things, the methods described herein.

As will be described in further detail below, the memory <NUM> includes computer executable instructions (or software) for determining, among other things, whether image data is within a predetermined range (obtained in a situation) that image analysis software is trained to analyze. In the example illustrated in <FIG>, the memory <NUM> includes image analysis software <NUM> including a convolutional neural network (CNN) <NUM> (or, more broadly machine learning software) and autonomous functionality software <NUM>. The CNN <NUM> is trained to make predictions related to detecting and classifying objects, road markings, road signs, and the like in image data received from the camera <NUM>. The autonomous functionality software <NUM> relies on predictions made by the image analysis software <NUM> (for example, the CNN <NUM>) using the image data from the camera <NUM> to provide autonomous functionality to the vehicle <NUM> by controlling the steering system <NUM>, accelerator <NUM>, and brakes <NUM>, among other things. The autonomous functionality software <NUM> may be, for example, automated cruise control (ACC), an automatic braking system, an automated parking system, or the like. It should be understood that the memory <NUM> may include more, fewer, or different software components than those illustrated in <FIG>. While described herein as including a CNN it should be noted that the image analysis software <NUM> may include a different type of machine learning software, for example, a decision tree or a Bayesian network. In some embodiments, the image analysis software <NUM> may not include machine learning software.

The electronic controller <NUM> may be implemented in several independent controllers (for example, programmable electronic controllers) each configured to perform specific functions or sub-functions. Additionally, the electronic controller <NUM> may contain sub-modules that include additional electronic processors, memory, or application specific integrated circuits (ASICs) for handling input/output functions, processing of signals, and application of the methods listed below. In other embodiments, the electronic controller <NUM> includes additional, fewer, or different components.

<FIG> illustrates the method <NUM> for determining whether image data is within a predetermined range that image analysis software is configured to analyze. The method <NUM> is performed by the electronic processor <NUM> executing the image analysis software <NUM>. It should be understood that while the example method <NUM> is described below in terms of a CNN trained to analyze image data, the method <NUM> more generally applies to image analysis software configured to analyze image data. At step <NUM>, the electronic processor <NUM> receives image data from the camera <NUM>. At step <NUM>, the electronic processor <NUM> calculates a perturbation value that is specific to the CNN <NUM> and received image data. A perturbation value is a value that, when added to the image data, alters the image data, effectively creating noise in the image data. The electronic processor <NUM> calculates the perturbation value using the gradient of a cost function of the CNN <NUM>. The cost function is defined as L(θ, x, y), where x is the input image data, y is possible classifications for the image data, and θ is values of weights included in the CNN <NUM>. The perturbation value is determined to be the result of determining the sign (positive or negative) of the gradient of the cost function (∇xL(θ, x, y)) multiplied by a weight ε. At step <NUM>, the image data is perturbed using the perturbation value (the perturbation value is added to each pixel of the image data). At step <NUM>, the electronic processor <NUM> executes the CNN <NUM> to generate a prediction for the perturbed image data (classifying the perturbed image data into one of a plurality of categories). For example, the CNN <NUM> may determine if a lane marking on the right side of a vehicle <NUM> is solid or dashed. When the electronic processor <NUM> generates a prediction, the prediction is associated with a confidence value. The confidence value represents the likelihood that the prediction is correct.

At step <NUM>, the electronic processor <NUM> also executes the CNN <NUM> to generate a prediction regarding the image data (classifies the image data into one of a plurality of categories) and determines a confidence value associated with the prediction regarding the image data. It should be noted that while step <NUM> is illustrated in <FIG> as being performed in parallel to steps <NUM>-<NUM>, in some embodiments these steps may be performed sequentially.

At step <NUM>, the electronic processor <NUM> compares the confidence value associated with the prediction made based on the perturbed image data to the confidence value associated with the prediction made based on the unperturbed image data. At step <NUM>, when the difference between the confidence values is less than a predetermined threshold, the electronic processor <NUM> determines that the image data received from the camera <NUM> lies outside of the predetermined range that the CNN <NUM> is trained to analyze. When the image data received from the camera <NUM> lies outside of the predetermined range that the CNN <NUM> is trained to analyze, the electronic processor <NUM> determines the prediction made by the CNN <NUM> is unreliable. In some embodiments, when the image data received from the camera <NUM> lies outside of the predetermined range that the CNN <NUM> is trained to analyze and the autonomous functionality software <NUM> relies on the prediction generated by the CNN <NUM>, the electronic processor <NUM> disables autonomous functionality of the vehicle <NUM> controlled by the autonomous functionality software <NUM>.

In other embodiments, rather than disabling the autonomous functionality software <NUM> when the image data received from the camera <NUM> lies outside of the predetermined range that the CNN <NUM> is trained to analyze, the electronic processor <NUM> ignores (disregards) the image data from the camera <NUM>. In the case that the electronic processor <NUM> ignores the image data from the camera <NUM>, the electronic processor <NUM> uses data from alternative sensors to generate predictions with the CNN <NUM> (or other types of software) and execute the autonomous functionality of the autonomous functionality software <NUM>. Examples of data from alternate sensors include image data from other cameras or data from sensors such as radar sensors, LIDAR sensors, ultrasonic sensors, and the like.

It should be understood that the image analysis software <NUM> may include more than one CNN and that each CNN may be trained to detect something different in received image data. For example, one CNN may be trained to detect road markings (for example, lane markers) while another CNN is trained to detect objects such as people, animals, and vehicles. In some embodiments, when image data is determined to be outside the predetermined range a CNN is trained to analyze, the autonomous functionality controlled by the autonomous functionality software <NUM> is disabled regardless of whether or not the autonomous functionality software <NUM> relies on predictions that a CNN is making based on the image data.

In this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises. a," "includes. a," or "contains. a" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, contains the element. The terms "a" and "an" are defined as one or more unless explicitly stated otherwise herein. The terms "substantially," "essentially," "approximately," "about" or any other version thereof, are defined as being close to as understood by one of ordinary skill in the art, and in one non-limiting embodiment the term is defined to be within <NUM>%, in another embodiment within <NUM>%, in another embodiment within <NUM>% and in another embodiment within <NUM>%. The term "coupled" as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

Claim 1:
A system (<NUM>) for determining whether image data is within a predetermined range that image analysis software (<NUM>) is configured to analyze, the system (<NUM>) comprising:
a camera (<NUM>); and
an electronic processor (<NUM>), the electronic processor (<NUM>) configured to:
receive the image data from the camera (<NUM>);
use the image analysis software (<NUM>), generate a prediction regarding the image data and a confidence value associated with the prediction regarding the image data;
calculate perturbation values by determining a sign of a gradient of a cost function (∇xL(θ, x, y)) of the image analysis software (<NUM>) used to generate the prediction with respect to the image;
multiply a weight (ε) by the sign, wherein x is the input image data, y is possible classifications for the image data, and θ are parameters of the image analysis software used to generate the prediction;
perturb the image data using the perturbation values;
use the image analysis software (<NUM>), generate a prediction regarding the perturbed image data and a confidence value associated with the prediction regarding the perturbed image data;
compare the confidence value associated with the prediction regarding the perturbed image data to the confidence value associated with the prediction regarding the image data; and
when a difference between the confidence value associated with the prediction regarding the perturbed image data and the confidence value associated with the prediction regarding the image data is bigger than or equal to a predetermined threshold value, determine that the image data received from the camera (<NUM>) lies inside the predetermined range,
wherein the image analysis software (<NUM>) comprises a machine learning software (<NUM>) trained to analyze the image data and the machine learning software (<NUM>) comprises a convolutional neural network.