FAIR NEURAL NETWORKS

A system is disclosed that includes a computer that includes a processor and a memory, the memory including instructions executable by the processor to input an image acquired by a sensor to a neural network to output a prediction regarding an object included in the image. The neural network can be trained, based on (a) a distributed robust optimization that minimizes an expectation applied to probability distributions of loss functions to select training images that yield a solution with a selected uncertainty level and (b) generating additional input images based on adversarial images.

BACKGROUND

Images can be acquired by sensors and processed using a computer to determine data regarding an environment around a system. For example, a computer can acquire images from one or more image sensors that can be processed to determine data regarding objects. Data extracted from images of objects can be used by a computer to operate systems including vehicles, robots, security systems, and/or object tracking systems.

DETAILED DESCRIPTION

A neural network can be trained to determine data regarding an environment around a system. For example, a system as described herein can be used to locate objects in an environment around the system and operate the system based on the location of the objects. In some examples, sensor data can be provided to a computer to determine an object identity and an object location and determine a system trajectory based on the identity and location of the object. A trajectory is a set of locations that can be indicated as coordinates in a coordinate system that along with velocities, e.g., vectors indicating speeds and headings, at respective locations. A computer in a system can determine a trajectory for operating the system that locates the system or portions of the system with respect to the object. A vehicle is described herein as an example including a system that includes a sensor to acquire data regarding an object, a computer to process the sensor data and controllers to operate the vehicle based on output from the computer. Other systems that can include sensors, computers and controllers that can respond to objects in an environment around the system include robots, security systems and object tracking systems.

Image sensors used to acquire image data regarding an environment around a system. The image data can be input to a neural network included in a computer to determine predictions regarding the environment. For example, an image sensor can acquire image data and input the image data to a neural network. The neural network can input the image data and determine an identity and location of an object included in the image. In examples where the image sensor and the neural network is included in a vehicle, a computer in the vehicle can determine a vehicle trajectory based on the identity and location of the object. In examples where the image sensor and the neural network is included in a robot, a computer in the robot can determine a trajectory for a gripper to cause the gripper to move to a position where it can grasp the object. In examples where the image sensor and neural network is included in a security system, a computer in the security system can unlock a door based on identifying a person in the image. In examples where the image sensor and neural network is included in an object tracking system, a computer can route a package in a package sorting facility. Vehicles operating using neural networks to detect objects in an environment around the vehicle will be used as non-limiting examples herein, however, techniques discussed herein to enhance fairness in training neural networks will apply generally to any neural network image processing task.

Trained neural networks may perform well in numerous tasks involving object classification and image generation. However, small perturbations that may not affect or be perceived by a human observer may significantly degrade the performance of well-trained deep neural networks. Techniques described herein enhance neural network training by using adversarial dataset generation to generate image data that can cause a neural network to fail while appearing similar to images that are successfully processed by the neural network. Adversarial dataset generation can generate images that are imperceptibly different from input images. Imperceptibly different images can differ in size, location, orientation, brightness, contrast, or texture while a human observer typically would not detect a difference between the images based on visual inspection. Imperceptible differences can be measured using two image differences measures: 1-SSIM and PieApp. 1-SSIM and PieApp will be discussed in relation toFIG.3, below. Training a neural network using imperceptibly different images as measured by 1-SSIM and/or PieAPP can enhance neural network training by adding images to the training dataset that would not be included in a training dataset compiled based on human observation.

Neural network training also limited because the quality of the training depends upon how well the training dataset compares to the data to be encountered when the trained neural network is released into the world for use. Techniques described herein use distributionally robust optimization (DRO) which is a technique for selecting a distribution of images from a training dataset that mirrors and unknown distribution of images to be encountered in the real world. A robust neural network experiences an error rate on test datasets that is similar to the error rate encountered during training. Training neural networks with DRO and optimally designed imperceptibly different test images can enhance the performance of neural networks by increasing fairness. Fairness in the context of a neural network trained to recognize objects in mages means that the neural network achieves a same error rate on a range of images encountered in the real world as an error rate generated by the training dataset. Augmenting a training dataset using adversarial dataset generation and DRO can enhance fairness by training a neural network using a training dataset that models image data to be encountered in the real world.

A method is disclosed herein, including inputting an image acquired by a sensor to a neural network to output a prediction regarding an object included in the image and wherein the neural network is trained, based on (a) a distributed robust optimization that minimizes an expectation applied to probability distributions of loss functions to select training images that yield a solution with a selected uncertainty level and (b) generating additional input images based on adversarial images. Outputting the prediction regarding the image can include outputting an object identity and an object location. The neural network can output a confidence value that indicates a probability that the prediction regarding the image is correct. The loss function can be determined based on comparing output from the neural network with ground truth data based on the input image. The neural network can include convolutional layers and fully connected layers. The input images can be respectively input to the neural network a plurality of times and the loss functions are backpropagated through layers of the neural network to select weights that minimize the loss functions. Generating the additional input images based on the adversarial images can include determining imperceptible differences based on a structural similarity index measure.

The structural similarity index measure can generate images that are not perceptibly different to a human observer but can cause the neural network to fail. Generating the additional input images based on the adversarial images can include determining imperceptible differences based on processing the input image with a PieAPP neural network. The PieAPP neural network can generate images that are not perceptibly different to a human observer but can cause the neural network to fail. The structural similarity index measure can be based on determining a ratio of joint pixel means times joint standard deviations to summed pixel means times summed pixel standard deviations. The PieApp neural network can determine image similarity by determining pairwise preference. The structural similarity index measure and/or the PieApp neural network can generate an adversarial dataset for training the neural network. Training the neural network based on an adversarial network can achieve fairness.

Further disclosed is a computer readable medium, storing program instructions for executing some or all of the above method steps. Further disclosed is a computer programmed for executing some or all of the above method steps, including a computer apparatus, programmed to input an image acquired by a sensor to a neural network to output a prediction regarding an object included in the image and wherein the neural network is trained, based on (a) a distributed robust optimization that minimizes an expectation applied to probability distributions of loss functions to select training images that yield a solution with a selected uncertainty level and (b) generating additional input images based on adversarial images. Outputting the prediction regarding the image can include outputting an object identity and an object location. The neural network can output a confidence value that indicates a probability that the prediction regarding the image is correct. The loss function can be determined based on comparing output from the neural network with ground truth data based on the input image. The neural network can include convolutional layers and fully connected layers. The input images can be respectively input to the neural network a plurality of times and the loss functions are backpropagated through layers of the neural network to select weights that minimize the loss functions. Generating the additional input images based on the adversarial images can include determining imperceptible differences based on a structural similarity index measure.

The instructions can include further instructions where the structural similarity index measure can generate images that are not perceptibly different to a human observer but can cause the neural network to fail. Generating the additional input images based on the adversarial images can include determining imperceptible differences based on processing the input image with a PieAPP neural network. The PieAPP neural network can generate images that are not perceptibly different to a human observer but can cause the neural network to fail. The structural similarity index measure can be based on determining a ratio of joint pixel means times joint standard deviations to summed pixel means times summed pixel standard deviations. The PieApp neural network can determine image similarity by determining pairwise preference. The structural similarity index measure and/or the PieApp neural network can generate an adversarial dataset for training the neural network. Training the neural network based on an adversarial network can achieve fairness.

FIG.1is a diagram of a sensing system100that can include a traffic infrastructure node105that includes a server computer120and stationary sensors122. Sensing system100includes a vehicle110, operable in autonomous (“autonomous” by itself in this disclosure means “fully autonomous”), semi-autonomous, and occupant piloted (also referred to as non-autonomous) mode. One or more vehicle110computing devices115can receive data regarding the operation of the vehicle110from sensors116. The computing device115may operate the vehicle110in an autonomous mode, a semi-autonomous mode, or a non-autonomous mode.

The computing device115includes a processor and a memory such as are known. Further, the memory includes one or more forms of computer-readable media, and stores instructions executable by the processor for performing various operations, including as disclosed herein. For example, the computing device115may include programming to operate one or more of vehicle brakes, propulsion (i.e., control of acceleration in the vehicle110by controlling one or more of an internal combustion engine, electric motor, hybrid engine, etc.), steering, climate control, interior and/or exterior lights, etc., as well as to determine whether and when the computing device115, as opposed to a human operator, is to control such operations.

The computing device115may include or be communicatively coupled to, i.e., via a vehicle communications bus as described further below, more than one computing devices, i.e., controllers or the like included in the vehicle110for monitoring and/or controlling various vehicle components, i.e., a powertrain controller112, a brake controller113, a steering controller114, etc. The computing device115is generally arranged for communications on a vehicle communication network, i.e., including a bus in the vehicle110such as a controller area network (CAN) or the like; the vehicle110network can additionally or alternatively include wired or wireless communication mechanisms such as are known, i.e., Ethernet or other communication protocols.

Via the vehicle network, the computing device115may transmit messages to various devices in the vehicle and/or receive messages from the various devices, i.e., controllers, actuators, sensors, etc., including sensors116. Alternatively, or additionally, in cases where the computing device115actually comprises multiple devices, the vehicle communication network may be used for communications between devices represented as the computing device115in this disclosure. Further, as mentioned below, various controllers or sensing elements such as sensors116may provide data to the computing device115via the vehicle communication network.

In addition, the computing device115may be configured for communicating through a vehicle-to-infrastructure (V2X) interface111with a remote server computer120, i.e., a cloud server, via a network130, which, as described below, includes hardware, firmware, and software that permits computing device115to communicate with a remote server computer120via a network130such as wireless Internet (WI-FI®) or cellular networks. V2X interface111may accordingly include processors, memory, transceivers, etc., configured to utilize various wired and/or wireless networking technologies, i.e., cellular, BLUETOOTH®, Bluetooth Low Energy (BLE), Ultra-Wideband (UWB), Peer-to-Peer communication, UWB based Radar, IEEE 802.11, and/or other wired and/or wireless packet networks or technologies. Computing device115may be configured for communicating with other vehicles110through V2X (vehicle-to-everything) interface111using vehicle-to-vehicle (V-to-V) networks, i.e., according to including cellular communications (C-V2X) wireless communications cellular, Dedicated Short Range Communications (DSRC) and/or the like, i.e., formed on an ad hoc basis among nearby vehicles110or formed through infrastructure-based networks. The computing device115also includes nonvolatile memory such as is known. Computing device115can log data by storing the data in nonvolatile memory for later retrieval and transmittal via the vehicle communication network and a vehicle to infrastructure (V2X) interface111to a server computer120or user mobile device160.

As already mentioned, generally included in instructions stored in the memory and executable by the processor of the computing device115is programming for operating one or more vehicle110components, i.e., braking, steering, propulsion, etc., without intervention of a human operator. Using data received in the computing device115, i.e., the sensor data from the sensors116, the server computer120, etc., the computing device115may make various determinations and/or control various vehicle110components and/or operations without a driver to operate the vehicle110. For example, the computing device115may include programming to regulate vehicle110operational behaviors (i.e., physical manifestations of vehicle110operation) such as speed, acceleration, deceleration, steering, etc., as well as tactical behaviors (i.e., control of operational behaviors typically in a manner intended to achieve efficient traversal of a route) such as a distance between vehicles and/or amount of time between vehicles, lane-change, minimum gap between vehicles, left-turn-across-path minimum, time-to-arrival at a particular location and intersection (without signal) minimum time-to-arrival to cross the intersection.

Controllers, as that term is used herein, include computing devices that typically are programmed to monitor and/or control a specific vehicle subsystem. Examples include a powertrain controller112, a brake controller113, and a steering controller114. A controller may be an electronic control unit (ECU) such as is known, possibly including additional programming as described herein. The controllers may communicatively be connected to and receive instructions from the computing device115to actuate the subsystem according to the instructions. For example, the brake controller113may receive instructions from the computing device115to operate the brakes of the vehicle110.

The one or more controllers112,113,114for the vehicle110may include known electronic control units (ECUs) or the like including, as non-limiting examples, one or more powertrain controllers112, one or more brake controllers113, and one or more steering controllers114. Each of the controllers112,113,114may include respective processors and memories and one or more actuators. The controllers112,113,114may be programmed and connected to a vehicle110communications bus, such as a controller area network (CAN) bus or local interconnect network (LIN) bus, to receive instructions from the computing device115and control actuators based on the instructions.

Sensors116may include a variety of devices known to provide data via the vehicle communications bus. For example, a radar fixed to a front bumper (not shown) of the vehicle110may provide a distance from the vehicle110to a next vehicle in front of the vehicle110, or a global positioning system (GPS) sensor disposed in the vehicle110may provide geographical coordinates of the vehicle110. The distance(s) provided by the radar and/or other sensors116and/or the geographical coordinates provided by the GPS sensor may be used by the computing device115to operate the vehicle110autonomously or semi-autonomously, for example.

The vehicle110is generally a land-based vehicle110capable of autonomous and/or semi-autonomous operation and having three or more wheels, i.e., a passenger car, light truck, etc. The vehicle110includes one or more sensors116, the V2X interface111, the computing device115and one or more controllers112,113,114. The sensors116may collect data related to the vehicle110and the environment in which the vehicle110is operating. By way of example, and not limitation, sensors116may include, i.e., altimeters, cameras, LIDAR, radar, ultrasonic sensors, infrared sensors, pressure sensors, accelerometers, gyroscopes, temperature sensors, pressure sensors, hall sensors, optical sensors, voltage sensors, current sensors, mechanical sensors such as switches, etc. The sensors116may be used to sense the environment in which the vehicle110is operating, i.e., sensors116can detect phenomena such as weather conditions (precipitation, external ambient temperature, etc.), the grade of a road, the location of a road (i.e., using road edges, lane markings, etc.), or locations of target objects such as neighboring vehicles110. The sensors116may further be used to collect data including dynamic vehicle110data related to operations of the vehicle110such as velocity, yaw rate, steering angle, engine speed, brake pressure, oil pressure, the power level applied to controllers112,113,114in the vehicle110, connectivity between components, and accurate and timely performance of components of the vehicle110.

Vehicles can be equipped to operate in autonomous, semi-autonomous, or manual modes. By a semi- or fully-autonomous mode, we mean a mode of operation wherein a vehicle can be piloted partly or entirely by a computing device as part of a system having sensors and controllers. For purposes of this disclosure, an autonomous mode is defined as one in which each of vehicle propulsion (i.e., via a powertrain including an internal combustion engine and/or electric motor), braking, and steering are controlled by one or more vehicle computers; in a semi-autonomous mode the vehicle computer(s) control(s) one or more of vehicle propulsion, braking, and steering. In a non-autonomous mode, none of these are controlled by a computer. In a semi-autonomous mode, some but not all of them are controlled by a computer.

A traffic infrastructure node105can include a physical structure such as a tower or other support structure (i.e., a pole, a box mountable to a bridge support, cell phone tower, road sign support, etc.) on which infrastructure sensors122, as well as server computer120, can be mounted, stored, and/or contained, and powered, etc. One traffic infrastructure node105is shown inFIG.1for ease of illustration, but the system100could and likely would include tens, hundreds, or thousands of traffic infrastructure nodes105. The traffic infrastructure node105is typically stationary, i.e., fixed to and not able to move from a specific geographic location. The infrastructure sensors122may include one or more sensors such as described above for the vehicle110sensors116, i.e., lidar, radar, cameras, ultrasonic sensors, etc. The infrastructure sensors122are fixed or stationary. That is, each sensor122is mounted to the infrastructure node so as to have a substantially unmoving and unchanging field of view.

Server computer120typically has features in common, i.e., a computer processor and memory and configuration for communication via a network130, with the vehicle110V2X interface111and computing device115, and therefore these features will not be described further to avoid redundancy. Although not shown for ease of illustration, the traffic infrastructure node105also includes a power source such as a battery, solar power cells, and/or a connection to a power grid. A traffic infrastructure node105server computer120and/or vehicle110computing device115can receive sensor116,122data to monitor one or more objects. An “object,” in the context of this disclosure, is a physical, i.e., material, structure or thing that can be detected by a vehicle sensor116and/or infrastructure sensor122.

FIG.2is a diagram of a system for training a convolutional neural network (CNN)200. CNN200includes convolutional layers204(CONV LAYERS) and fully connected layers206(FULLY CONNECTED LAYERS). Convolutional layers204can receive as input images202(IMAGE) and convolve the images with a plurality of small (5×5, 7×7, etc.) kernels to extract features from the image202. The convolutional layers204can include pooling functions between the layers to reduce the size of the image data and localize the extracted features. The values that convolutional kernels use to process the image data are determined by programmable weights. Convolutional layers204output latent variables indicating locations of detected features to fully connected layers206. Fully connected layers206include neurons that calculate linear and non-linear functions of the input latent variables to determine predictions208(PREDICT) based on the input images202. For example, a CNN200can be trained to output predictions208regarding labels indicated by objects included in input images202, e.g., vehicles or pedestrians. The calculations performed by the fully connected layers206are determined by programmable weights.

At training time, a training dataset that includes a plurality of images202and ground truth data indicating the correct predictions the neural network should extract from the input images202. The ground truth data is determined by techniques that do not include employing the CNN200, for example, human observation and measurement of the image data. A CNN200can be trained by inputting an image202from the training dataset to the CNN200a plurality of times. Each time the image202is processed to form a prediction, a loss function210(LOSS FUNCTION) is determined. A loss function is an equation that compares the prediction to the ground truth to determine how well the CNN200has processed the image202. The loss function can be backpropagated through the layers of CNN200. Backpropagation is a technique for inputting the loss function210to the layers of the CNN200starting with the output and proceeding through the layers of the CNN200to the input. The weights can be modified based on the loss function and the image202re-processed by the CNN200and a new loss function210determined. By processing the input image202a plurality of times, weights for the convolutional layers204and fully connected layers206can be determined that minimize the loss function. The CNN200can be trained by determining the weights that minimize the loss function210for all of the images202in the training dataset. A CNN200can also be trained to output a confidence value along with a prediction. A confidence value is a probability that the accompanying prediction is correct.

FIG.3is a diagram illustrating training a CNN200with adversarial dataset generation (ADV DATASET)302and distributionally robust optimization (DRO)304. Adversarial dataset generation302is a technique for generating images that can cause a trained CNN200to fail by introducing changes in images that can be imperceptible to a human observer but measurable using 1-SSIM or PieAPP. For example, small changes in image location, scale, rotation, brightness, contrast and texture can cause an image to be unsuccessfully processed that, when unaltered, yields a prediction that successfully matches ground truth data. 1-SSIM or PieAPP can be used to quantify what changes in images will result in failure of a CNN200to properly process an image. A prediction output from a CNN200in response to an altered image unsuccessfully matches the ground truth data when the prediction fails to matches the ground truth. For example, an unsuccessful match would be failure to identify a vehicle or pedestrian in an input image. Unsuccessfully matching can alternatively or additionally mean that the CNN200outputs a low confidence value to accompany the prediction.

Generating adversarial images from an existing image included in a training dataset includes modifying the existing image using suitable image processing techniques. An existing image can be modified by performing geometric transformations on the pixels of the existing image to change the location, scale, and orientation of objects included in the image, for example. An existing image can be modified by performing multiplicative and additive changes to the pixels to change brightness and contrast of the existing image. An existing image can alternatively or additionally be modified by adding random noise to the pixels of the image to change the texture. Operations for modifying (i.e., making a change to) an image such as the foregoing examples typically include specifying size, location, orientation, brightness, contrast or texture parameters that can be adjusted to determine the amount of change to introduce to the image to generate an adversarial image. Techniques described herein generate a series of adversarial images by increasing an amount of change by increasing the values of the parameters while measuring the computational effort introduced at each step. Computational effort is a measure of computational resources required to produce an image difference as determined by 1-SSIM and/or PieAPP and can be used to determine whether a change introduced to an image would be imperceptible to a human observer.

Techniques can be used to measure computational effort in images determine an adversarial dataset generation302. These techniques measure image differences to determine that the differences introduced into an image are imperceptible to a human observer. Including images in a training dataset that include differences imperceptible to a human observer is useful because a human will not typically include images that include imperceptible differences in a training database. Images that include imperceptible differences can cause a CNN200to fail and including images that include imperceptible differences in a training dataset will result in a more robust trained CNN200that can successfully process a wider variety of input images.

One technique for determining imperceptible differences is a structural similarity index measure (1-SSIM). 1-SSIM is a reward function that determines the structural similarity of two images. Structural similarity is when two images have similar but not exactly the same structures or objects in similar but not exactly the same location, size, orientation, contrast, brightness or texture. 1-SSIM for two images x and y can be determined by the equation:

Where μxand μyare the mean intensities of the pixel values in images x and y, σxand σyare the standard deviations of the pixel values in images x and y and σxyis determined by the equation:

Where xiand yiare the pixel values in images x and y having N pixels each. 1-SSIM is a computational effort function that expresses a difference between two images as computational effort, e.g., the greater the difference between images, the greater computational effort associated with the difference. Image differences that are imperceptible to humans have low computational effort 1-SSIM values. 1-SSIM values that can indicate image differences that are imperceptible to humans can be determined empirically by testing images from a training dataset to determine 1-SSIM values that indicate imperceptible differences.

An alternative or additional measure of image similarity can be determined by a PieAPP neural network. A PieAPP neural network can determine the probability that a human observer would think that a first image is of more similar to a reference image than a second image. A PieAPP neural network is described in “PieAPP: Perceptual image-error assessment through pairwise preference” by Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1808-1817, 2018. PieAPP can be used to rank images that have been perturbed by translation, rotation, scaling and addition of texture to determine which images are likely imperceptibly different to a typical human observer. An algorithm for applying either 1-SSIM or PieAPP to training images to generate a robust adversarial dataset is discussed in relation toFIG.4, below. A robust adversarial dataset is a dataset that will effectively train a CNN200to process a wide variety of input images including images that are imperceptibly different than a dataset that does not include adversarial images.

Adversarial dataset generation302modifies images in one or more of location, scale, orientation, brightness, contrast, and texture in increasing steps, stopping when the computational effort functions determined by 1-SSIM or PieAPP indicate that the modifications would stop being imperceptible to a human observer. In this fashion, determining an adversarial dataset generation302becomes an efficient technique for enhancing training of a CNN200because adversarial dataset generation302generates images that have maximum differences from the original training dataset while remaining imperceptible to a typical human observer. Adversarial dataset generation302maximizes the likelihood that the adversarial images will enhance training of a CNN200to achieve fairness while minimizing the number of images and computing resources required to train the CNN200.

Once an adversarial dataset of images that are imperceptibly different to a typical human observer are generated by the adversarial dataset generation302, DRO304provides a framework for training CNN200using the generated images. DRO304enhances the efficiency with which a CNN200can be trained by selecting the training dataset to include a distribution of images in the training dataset that matches an unknown distribution of images in the real world. DRO304is a framework for training the CNN200to select images generated by adversarial dataset generation302. By training CNN200with images selected from adversarial dataset generation302, DRO304can avoid deploying a CNN200in the field only to find new types of images that the CNN200cannot process to identify objects, whereupon it may be necessary to acquire the new types of images and add them to a re-training dataset for re-training the CNN200. An example process for implementing DRO304is described in relation toFIG.5, below.

FIG.4is a flowchart, described in relation toFIGS.1-3of a process400for generating a robust datasetrob. Process400can be implemented by a processor of a server computer120, taking as input a training datasetand outputting a robust datasetrobthat includes adversarial images xadv. Process400includes multiple blocks that can be executed in the illustrated order. Process400could alternatively or additionally include fewer blocks or can include the blocks executed in different orders.

Process400begins at block402where a server computer120inputs an initial model θ0, which in this example is a CNN200, a learning rate α, which is user selected and in this example is 0.1, a dataset={xi, yi}i=1, . . . ,N, where xiare the images, yiare the ground truth labels, N is the number of images in the dataset, and the number of steps T1, which is user selected and in this example is 100.

At block408, process400beings an inner for-loop with i=1, 2, . . . , N by updating model θ according to the equation:

where ∇θis a gradient operator for the loss functionand(θ; xi, yi) is a loss function for model θ.

At block410, server computer120uses model θ, image xi, and ground truth labels to generate adversarial images {x′i, yi} using 1-SSIM or PieAPP as discussed above in relation toFIG.3.

At block414, server computer120increments index i and if i≤N, loops back to block408, otherwise process400passes to block416.

At block416, server computer120increments index k, and if k≤T1, loops back to block406, otherwise process400passes to block418.

At block418, server computer120outputs robust datasetrob={xi, yi, Pi}i=1,2, . . . ,Mwhich includes M images, which equals the input N images plus the adversarial images add by process400. Following block418process400ends.

FIG.5is a flowchart, described in relation toFIGS.1-4of a process500for training a model θ, which can be a CNN200, using DRO304. Process500can be implemented by a processor of a server computer120, taking as input an initial model θ0, a robust training datasetrobdetermined by process400inFIG.4, above, and outputting a trained neural network θ. Process400includes multiple blocks that can be executed in the illustrated order. Process400could alternatively or additionally include fewer blocks or can include the blocks executed in different orders.

Process500begins at block502, where a server computer120inputs an initial model θ0, which in this example is a CNN200, a learning rate α, which is user selected and in this example is 0.1, and a robust datasetrob={xi, yi, Pi}i=1,2, . . . , Mwhich includes adversarial images.

At block506, server computer120begins two nested for loops with the outer for loop having k=1, 2, . . . , T2, where in this example T2=50, and the inner loop having i=1, 2, . . . , M, where M equals the number of images in the robust datasetrob. Server computer120samples images and ground truth labels {xi, yi} proportionally to the weights Piwith replacement from the datasetrob.

At block508, server computer120updates the model θ according to the equation:

Updating the model θ with the new weights trains the model θ with the adversarial images included in the robust datasetrob. Training the model θ with adversarial images included in the robust datasetrobcan achieve distributionally robust optimization.

At block510, server computer120increments index i and if i≤M, loops back to block506, otherwise process500passes to block512.

At block512, server computer120increments index k and if k≤T2, loops back to block506, otherwise process500passes to block514.

At block514, server computer120outputs the model θ, where model θ has been trained with distributional robust optimization which enhances the fairness of model θ. Training model θ generating an adversarial database as discussed in relation toFIG.4and DRO training as discussed in relation toFIG.5can achieve fairness in model θ training in a computationally efficient fashion, generating a diverse dataset with the low computational effort value. Following block514process500ends.

FIG.6is a flowchart, described in relation toFIGS.1-5of a process600for operating a vehicle using a fairly trained CNN200, trained by techniques discussed above in relation toFIGS.2-5. Process600can be implemented by a processor of a computing device115, taking as input images acquired by sensors116, processing the images with a fairly trained CNN200, and operating a vehicle110based on predictions output from the CNN200. Process600includes multiple blocks that can be executed in the illustrated order. Process600could alternatively or additionally include fewer blocks or can include the blocks executed in different orders.

Process600begins at block602, where a server computer120downloads a fairly trained CNN200, trained according to techniques described in relation toFIGS.2-5, above, to a computing device115included in a vehicle110.

At block604, computing device115inputs image data acquired by a sensor116included in a vehicle110. The image data includes data regarding objects in an environment around the vehicle110.

At block606, computing device115inputs the image to the CNN200to determine a prediction regarding an object in the environment around the vehicle110. For example, the prediction can include data regarding a label and a location of an object with respect to the vehicle110, where the object can be another vehicle.

At block608, computing device115can determine a vehicle trajectory based on the prediction regarding the object. A vehicle trajectory can be specified in a suitable manner, typically as a line connecting points plotted according to a polynomial function that describes vehicle110locations and speeds. The vehicle trajectory can be processed by computing device115to determine lateral and longitudinal accelerations to be applied to the vehicle110to cause the vehicle to travel on the path described by the vehicle trajectory. In this example the vehicle trajectory can be determined to help the vehicle110avoid contact with the other vehicle.

At block610, computing device115can transmit instructions to vehicle controllers112,113,114to cause vehicle powertrain, vehicle steering, and vehicle brakes to operate vehicle110to travel along the vehicle trajectory determined at block608and help to avoid contact with another vehicle as determined by CNN200. Following block610process600ends.

Computing devices such as those discussed herein generally each includes commands executable by one or more computing devices such as those identified above, and for carrying out blocks or steps of processes described above. For example, process blocks discussed above may be embodied as computer-executable commands.

Computer-executable commands may be compiled or interpreted from computer programs created using a variety of programming languages and/or technologies, including, without limitation, and either alone or in combination, Java™, C, C++, Python, Julia, SCALA, Visual Basic, Java Script, Perl, HTML, etc. In general, a processor (i.e., a microprocessor) receives commands, i.e., from a memory, a computer-readable medium, etc., and executes these commands, thereby performing one or more processes, including one or more of the processes described herein. Such commands and other data may be stored in files and transmitted using a variety of computer-readable media. A file in a computing device is generally a collection of data stored on a computer readable medium, such as a storage medium, a random access memory, etc.

A computer-readable medium (also referred to as a processor-readable medium) includes any non-transitory (i.e., tangible) medium that participates in providing data (i.e., instructions) that may be read by a computer (i.e., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media and volatile media. Instructions may be transmitted by one or more transmission media, including fiber optics, wires, wireless communication, including the internals that comprise a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

The term “exemplary” is used herein in the sense of signifying an example, i.e., a candidate to an “exemplary widget” should be read as simply referring to an example of a widget.