Patent ID: 12221118

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

This specification describes how a vehicle, e.g., an autonomous or semi-autonomous vehicle, can use a trained machine learning model to generate a prediction of a level of discomfort that the vehicle is imposing onto surrounding agents in the environment.

FIG.1is a diagram of an example system100. The system100includes an on-board system110and a training system120.

The on-board system110is located on-board a vehicle102. The vehicle102inFIG.1is illustrated as an automobile, but the on-board system102can be located on-board any appropriate vehicle type. The vehicle102can be a fully autonomous vehicle that determines and executes fully-autonomous driving decisions in order to navigate through an environment. The vehicle102can also be a semi-autonomous vehicle that uses predictions to aid a human driver. For example, the vehicle102can autonomously apply the brakes if a prediction indicates that a human driver is about to collide with another vehicle.

The on-board system110includes one or more sensor subsystems140. The sensor subsystems140include a combination of components that receive reflections of electromagnetic radiation, e.g., lidar systems that detect reflections of laser light, radar systems that detect reflections of radio waves, and camera systems that detect reflections of visible light.

The sensor data generated by a given sensor generally indicates a distance, a direction, and an intensity of reflected radiation. For example, a sensor can transmit one or more pulses of electromagnetic radiation in a particular direction and can measure the intensity of any reflections as well as the time that the reflection was received. A distance can be computed by determining how long it took between a pulse and its corresponding reflection. The sensor can continually sweep a particular space in angle, azimuth, or both. Sweeping in azimuth, for example, can allow a sensor to detect multiple objects along the same line of sight.

The sensor subsystems140or other components of the vehicle102can also classify groups of one or more raw sensor measurements from one or more sensors as being measures of another agent. A group of sensor measurements can be represented in any of a variety of ways, depending on the kinds of sensor measurements that are being captured. For example, each group of raw laser sensor measurements can be represented as a three-dimensional point cloud, with each point having an intensity and a position. In some implementations, the position is represented as a range and elevation pair. Each group of camera sensor measurements can be represented as an image patch, e.g., an RGB image patch.

Once the sensor subsystems140classify a one or more groups of raw sensor measurements as being measures of respective surrounding agents, the sensor subsystems140can compile the raw sensor measurements into a set of raw data142, and send the raw data142to an agent feature extractor150.

The agent feature extractor150, also on-board the vehicle102, receives the raw sensor data142from the sensor system140and generates agent feature data152. The agent feature data152includes, for each of one or more identified surrounding agents in the environment of the vehicle102, data characterizing the agent. For example, for a particular agent, the agent feature data152can include a top-down image of the environment, e.g., a top-down image centered around the agent. As another example, for a particular agent, the agent feature data152can include motion parameters of the agent (e.g., a velocity of the agent, an acceleration of the agent, and/or a jerk of the agent), a size of the agent, and/or a distance between the agent and the vehicle102. As another example, for a particular agent, the agent feature data152can include a current location of the agent and/or a predicted future location of the agent, e.g., a predicted future location of the agent generated by an agent prediction system of the vehicle102. As another example, the agent feature date152can include feature of the environment, e.g., a roadgraph of the environment. In this specification, a roadgraph is data representing the known features of the environment, e.g., a top-down image of the environment, that can include representation of the features of the roads in the environment such as the lanes of the road, cross walks, traffic lights, stop signs, etc. In some implementations, the agent feature data152can include one or more features derived from raw data captured of the environment; for example, for a particular agent, the agent feature data152can include features representing one or more of: whether the agent is currently making a turn, whether the agent is currently in an intersection, or what the current state of a traffic light is.

In some implementations, the agent feature data152is human interpretable, i.e., every element of the agent feature data152can have a real-world meaning, e.g., scalar velocity or acceleration. In some other implementations, the agent feature data152is not human interpretable, e.g., the agent feature data corresponding to a particular agent can be a learned embedding of the raw sensor data142. In this specification, an embedding is an ordered collection of numeric values that represents an input in a particular embedding space. For example, an embedding can be a vector of floating point or other numeric values that has a fixed dimensionality.

The agent feature extractor150provides the agent feature data152to a discomfort prediction system130, also on-board the vehicle102. The discomfort prediction system130uses the agent feature data152to generate, for each of the one or more identified surrounding agents, an agent discomfort prediction132characterizing a level of discomfort imposed by the vehicle102onto the agent. For example, the agent discomfort prediction132for a particular agent can be a floating point value between 0 and 1, where 0 is the lowest discomfort level and 1 is the highest discomfort level.

In some implementations, the discomfort prediction system130also combines the one or more agent discomfort predictions132to generate an aggregate discomfort score characterizing the collective discomfort imposed by the vehicle102onto surrounding agents in the environment. This process is discussed in more detail below with reference toFIG.4.

The discomfort prediction system130can provide the agent discomfort predictions132and/or the aggregation discomfort score to a path planning system160, a user interface system170, or both.

The path planning system160, also on-board the vehicle102, generates a planned vehicle path that characterizes a path that the vehicle102will take in the future. When the path planning system160receives the agent discomfort predictions132, the path planning system160can use the agent discomfort predictions132to generate a new planned vehicle path that characterizes a path that the vehicle102will take in the future. For example, the agent discomfort predictions132may identify a particular surrounding agent that the vehicle102is causing discomfort, e.g., by driving too close to the surrounding agent. In this example, the path planning system160can generate a new planned vehicle path that navigates the vehicle102farther away from the surrounding agent, relieving the discomfort imposed on the surrounding agent.

When the user interface system170receives the agent discomfort predictions132, the user interface system170can use the agent discomfort predictions132to present information to the driver of the agent102to assist the driver in operating the agent102safely. The user interface system170can present information to the driver of the agent102by any appropriate means, for example, by an audio message transmitted through a speaker system of the agent102or by alerts displayed on a visual display system in the agent (e.g., an LCD display on the dashboard of the agent102). In a particular example, the agent discomfort predictions132may identify particular surrounding agent that the vehicle102is causing discomfort. In this example, the user interface system170can present an alert message to the driver of the agent102with instructions to adjust the trajectory of the agent102to relieve the imposed discomfort or notifying the driver of the agent of the imposed discomfort.

In some implementations, the user interface system can collect user feedback about the level172of discomfort of the vehicle102. That is, the user can provide the user discomfort level172that characterizes the current discomfort of the user as a passenger of the vehicle102. For example, the vehicle102can provide an interface for the driver or a passenger of the vehicle102to identify when an uncomfortable event has happened, and to identify a severity of the discomfort. As a particular example, the user can identify the severity of the discomfort using a scalar user discomfort level172, e.g., 0.5 for “low,” 0.75 for “medium,” and 1.0 for “high.”

The user interface system170can provide the user discomfort level1762to the discomfort prediction system130, for generating a training example134for training the discomfort prediction system130. For example, the discomfort prediction system130can generate a training example134characterizing the vehicle102from the raw sensor data, where the training label corresponding to the training example134is the reported user discomfort level172. The training system120can use the training example to train the discomfort prediction system130to generate agent discomfort predictions, treating the training example (which characterizes the vehicle102) as if it characterized a surrounding agent in the environment of the vehicle102. Example training processes are discussed in more detail below with reference toFIG.6andFIG.7.

To generate the agent discomfort predictions132, the discomfort prediction system130can use trained parameter values196that it obtains from a discomfort model parameters store194in the training system120.

The training system120is typically hosted within a data center124, which can be a distributed computing system having hundreds or thousands of computers in one or more locations.

The training system120includes a training data store180that stores all the training data used to train the parameter values of the discomfort prediction system130. The training data store180receives training examples134from agents operating in the real world. For example the training data store180can receive a training example134from the agent102and one or more other agents that are in communication with the training system120. Example training systems190are discussed in more detail below with reference toFIG.3andFIG.4.

The training data store180provides training examples182to a training system190, also housed in the training system120. The training system190uses the training examples182to update model parameters that will be used by the discomfort prediction system130, and provides the updated model parameters192to the discomfort model parameters store194. Once the parameter values of the discomfort prediction system130have been fully trained, the training system120can send the trained parameter values196to the discomfort prediction system130, e.g., through a wired or wireless connection.

FIG.2is an illustration of an example environment200containing a vehicle202and multiple surrounding agents204,206, and208.

The vehicle202is merging into the left lane from the middle lane. While doing so, and/or before doing so, the vehicle202can process sensor data captured by one or more sensors on-board the vehicle202using a discomfort prediction system, e.g., the discomfort prediction system130depicted inFIG.1, to generate agent discomfort predictions characterizing a level of discomfort imposed by the vehicle202onto each of the agents204,206, and208. For example, the agent discomfort predictions might be a scalar value between 0 and 1.

The vehicle202might generate a high agent discomfort prediction corresponding to the agent204, because the vehicle202is merging in front of the agent204. That is, by merging, the vehicle202might impose discomfort onto the agent204, e.g., because of the closeness between the vehicle202and the agent204or because the merging will require the agent204to slow down. For example, the vehicle202might generate an agent discomfort prediction of 0.9 corresponding to the agent204.

The vehicle202might generate a medium agent discomfort prediction corresponding to the agent206, because merging into the left lane might also affect the agent206. For example, the merging might require the agent204to slow down, which in turn will require the agent206to slow down. For example, the vehicle202might generate an agent discomfort prediction of 0.6 corresponding to the agent206.

The vehicle202might generate a low agent discomfort prediction corresponding to the agent208, because the vehicle202will not impose any discomfort onto the agent208(which is in the right lane) by merging into the left lane. For example, the vehicle202might generate an agent discomfort prediction of 0.1 corresponding to the agent208.

FIG.3is a diagram of an example training system300for training an agent model. The agent model training system300is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The agent model can be any model configured to receive a model input generated from sensor data captured by one or more sensors on-board a vehicle in an environment, and to process the model input to generate a model output characterizing one or more agents surrounding the vehicle in the environment. That is, the model output of the agent model is a prediction regarding one or more characteristics of the surrounding agents in the environment.

In particular, the agent model is configured to receive, for each of one or more surrounding agents in the environment, agent feature data characterizing the agent generated by an agent feature extractor320. The agent feature extractor320is configured to receive the sensor data captured by the on-board sensors of the vehicle and to process the sensor data to generate the agent feature data characterizing the surrounding agent, where the agent feature data is in the format that the agent model is configured to receive as input.

As a particular example, the agent model can include one or more of a recurrent neural network, a temporal convolutional neural network, or a boosted forest model. In some implementations, the agent model can use one or more feature value smoothing techniques to reduce noise in the agent feature data. For example, the agent model can use a low-pass filter to smooth feature values across time. In some implementations, the agent model can use feature calibration, e.g., applying a transformation to the agent feature data, in order to ensure that the agent feature data has a comparable distribution to the vehicle feature data.

For example, the agent model can be an agent discomfort model that is configured to receive, for each of one or more agents surrounding the vehicle in the environment, agent feature data and to process the agent feature data to generate an agent discomfort prediction characterizing a level of discomfort imposed by the vehicle onto the agent. For example, the agent discomfort model can be the discomfort prediction system130depicted inFIG.1.

As another example, the agent model can be an agent safety model that is configured to receive, for each of one or more agents surrounding the vehicle in the environment, agent feature data and to process the agent feature data to generate an agent safety prediction characterizing a level of safety that the vehicle is imposing onto the agent. That is, the agent safety model generates a prediction of whether the vehicle is causing the surrounding agent to be in an unsafe state or position. For example, the agent safety prediction can be a scalar value between 0 and 1, where 0 corresponds to a prediction that the vehicle is not endangering the surrounding agent at all, and 1 corresponds to a prediction that the vehicle is severely endangering the surrounding agent.

As another example, the agent model can be an agent progress model that is configured to receive, for each of one or more agents surrounding the vehicle in the environment, agent feature data and to process the agent feature data to generate an agent progress prediction characterizing a degree to which the vehicle is causing the agent not to make progress along the agent's intended route. That is, the agent progress model generates a prediction of whether the vehicle is impeding the surrounding agent and causing the surrounding agent to proceed less efficiently or quickly than desired. For example, the agent progress prediction can be a scalar value between 0 and 1, where 0 corresponds to a prediction that the vehicle is not impeding the agent along the agent's route at all, and 1 corresponds to a prediction that the vehicle is severely impeding the agent along the agent's route.

While the below description refers to the case where the agent model is an agent discomfort model, it is to be understood that the below description can apply to an agent model of any appropriate type.

Because the discomfort level of the vehicle is directly observable while the discomfort level of the surrounding agents is not, the agent model training system300trains the agent model to generate predictions corresponding to surrounding agents by processing training examples to corresponding to the vehicle itself. That is, the agent model training system trains the agent model using training data characterizing the vehicle itself and training labels characterizing the discomfort level of the vehicle itself, where the training labels are generated from observation of the discomfort level of the vehicle, e.g., using user feedback. The training system300treats the training data and training labels as if they characterized surrounding agents, and trains the agent discomfort model to generate predictions regarding surrounding agents.

The agent model training system300includes a training data store310, the agent feature extractor320, and an agent model training engine330.

The training data store310includes training examples that each include i) vehicle training data312that includes sensor data characterizing the vehicle captured by one or more sensors on-board the same vehicle, and ii) a vehicle training label314that characterizes a discomfort level of the vehicle at the time the sensor data was collected.

The training data store310provides the vehicle training data312to the agent feature extractor320, which processes the vehicle training data312to generate vehicle feature data322that characterizes the vehicle and that is in the format that the agent model is configured to receive as input. That is, while during inference the agent feature extractor320generates agent feature data characterizing a surrounding agent, during training the agent feature extractor320generates vehicle feature data322that characterizing the vehicle.

For example, if the vehicle feature data includes a top-down image of the environment centered on the vehicle, then during training the agent feature extractor320can generate a top-down image centered on the vehicle. Then, during inference, the agent feature extractor320can generate agent feature data by translating and/or cropping the top-down image so that is it centered on the surrounding agent.

In some implementations, the agent feature extractor320discards a portion of the vehicle training data312in order to generate the vehicle feature data322. The vehicle typically has more data characterizing the vehicle itself than data characterizing surrounding agents; that is, the vehicle training data312includes characteristics of the vehicle that are not included in corresponding agent data that would be provided to the agent feature extractor320at inference time. Because the agent feature extractor320is configured to generate vehicle feature data322as if it were generating agent feature data, the agent feature extractor320does not include in the vehicle feature data322any data characterizing the vehicle that does not correspond to data characterizing the agent that the agent feature extractor320will have at inference time. Thus, the agent feature extractor320can discard any data in the vehicle training data312that does not correspond to agent data that the agent feature extractor320will have access to at inference time.

The agent model training engine330obtains the vehicle feature data322and a vehicle training label314and uses them to train the agent model. In particular, the agent model training engine330processes the vehicle feature data322using the agent model to generate a vehicle discomfort prediction characterizing a predicted discomfort level of the vehicle, and determines an error between the vehicle discomfort prediction and the vehicle training label314. The agent model training engine then updates the current parameters of the agent model using the determined error, e.g., using backpropagation. Thus, by processing training examples characterizing the vehicle during training, the training system300can train the agent model to generate predictions characterizing surrounding agents at inference.

FIG.4is a diagram of an example training system400for training an agent model. The agent model training system400is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

As described above with reference toFIG.3, the agent model can be any model configured to receive a model input generated from sensor data captured by one or more sensors on-board a vehicle in an environment, and to process the model input to generate a model output characterizing one or more agents surrounding the vehicle in the environment. In particular, the agent model is configured to receive, for each of one or more surrounding agents in the environment, agent feature data characterizing the agent generated by an agent feature extractor420.

For example, the agent model can be an agent discomfort models that is configured to receive, for each of one or more surrounding agents, agent feature data and to process the agent feature data to generate an agent discomfort prediction. As another example, the agent model can be an agent safety model that is configured to receive, for each of one or more surrounding agents, agent feature data and to process the agent feature data to generate an agent safety prediction. As another example, the agent model can be an agent progress model that is configured to receive, for each of one or more surrounding agents, agent feature data and to process the agent feature data to generate an agent progress prediction.

While the below description refers to the case where the agent model is an agent discomfort model, it is to be understood that the below description can apply to an agent model of any appropriate type.

The agent model training system400is configured first to train a vehicle model that is configured to process vehicle feature data of the same form as the agent feature data and to generate vehicle model output of the same form as the agent model output generated by the agent model. That is, the vehicle feature data and the vehicle model output have the same format and describe the same characteristics as the agent feature data and the agent model output, respectively. The agent model training system400can train the vehicle model directly using supervised training because the training system400has access to ground-truth vehicle labels captured by the vehicle.

In some implementations, the vehicle model is larger than the agent model, e.g., has more trainable parameters than the agent model. As a particular example, the vehicle model can be a neural network that has more neural network layers than the agent model. In some implementations, the vehicle model receives additional input in addition to vehicle feature data422; that is, the input to the vehicle model can be larger than the input to the agent model.

After training the vehicle model, the training system400uses the vehicle model to generate labels for training the agent model. That is, the training system400processes agent feature data characterizing a surrounding agent using the trained vehicle model to generate a vehicle model output characterizing the surrounding agent; this vehicle model output is used as the ground-truth label when training the agent model.

The agent model training system400includes a training data store410, the agent feature extractor420, a vehicle model training engine430, a vehicle model execution engine440, and an agent model training engine450.

The training data store410includes training examples that each include i) vehicle training data412that includes sensor data characterizing the vehicle captured by one or more sensors on-board the vehicle, and ii) a vehicle training label414that characterizes a discomfort level of the vehicle at the time the sensor data was collected.

The training data store410provides the vehicle training data412to the agent feature extractor420, which processes the vehicle training data412to generate vehicle feature data422that characterizes the vehicle and that is in the format that the vehicle model is configured to receive as input.

As described above with reference toFIG.3, in some implementations, the agent feature extractor420discards a portion of the vehicle training data412in order to generate the vehicle feature data422. The vehicle training data412can include characteristics of the vehicle that are not included in corresponding agent training data416. Because the agent feature extractor420is configured to generate vehicle feature data422as if it were generating agent feature data, the agent feature extractor420does not include in the vehicle feature data422any data characterizing the vehicle that does not correspond to data characterizing the agent available in the agent training data416.

The vehicle model training engine430obtains the vehicle feature data422and a vehicle training label414and uses them to train the vehicle model. In particular, the vehicle model training engine430processes the vehicle feature data422using the agent model to generate a vehicle discomfort prediction characterizing a predicted discomfort level of the vehicle, and determines an error between the vehicle discomfort prediction and the vehicle training label414. The vehicle model training engine430then updates the current parameters of the vehicle model using the determined error, e.g., using backpropagation.

At the end of training the vehicle model, the vehicle model training engine430provides the trained parameters432of the vehicle model to the vehicle model execution engine440, which is configured to receive either vehicle feature data or agent feature data and to process the received feature data using the vehicle model to generate a vehicle model output.

The training data store410also includes agent training data416that includes sensor data characterizing a surrounding agent captured by one or more sensors on-board the vehicle. In some implementations, the vehicle training data412and the agent training data416are the same; that is, for a given vehicle at a given time point, the vehicle training data412characterizing the vehicle and the agent training data4116characterizing a surrounding agent in the environment is the same, e.g., is a collection of all sensor data captured by on-board sensors of the vehicle at the given time point.

The training data store410provides the agent training data416to the agent feature extractor, which processes the agent training data416to generate agent feature data424that characterizes the surrounding agent and that is in the format that the agent model is configured to receive as input.

The agent feature extractor420provides the agent feature data424to the vehicle model execution engine440, which processes the agent feature data424(which characterizes the surrounding agent) as if the agent feature data424characterized the vehicle, and generates a vehicle model output characterizing a predicted discomfort level of the surrounding agent. The training system400determines this vehicle model output to be the agent training label442that will be used as the ground-truth discomfort level when training the agent model.

The agent model training engine450obtains the agent feature data424and an agent training label442and uses them to train the agent model. In particular, the agent model training engine450processes the agent feature data424using the agent model to generate an agent discomfort prediction characterizing a predicted discomfort level of the agent, and determines an error between the agent discomfort prediction and the agent training label442. The agent model training engine450then updates the current parameters of the agent model using the determined error, e.g., using backpropagation.

FIG.5is a flow diagram of an example process500for determining the discomfort imposed on surrounding agents by a vehicle in an environment. For convenience, the process500will be described as being performed by a system of one or more computers located in one or more locations. For example, a discomfort prediction system, e.g., the discomfort prediction system130depicted inFIG.1, appropriately programmed in accordance with this specification, can perform the process500.

The system obtains sensor data characterizing the environment (step502). The sensor data has been captured by one or more sensors on-board a vehicle in the environment.

The system processes, for each of one or more surrounding agents in the environment, a network input generated from the sensor data to generate an agent discomfort prediction (step504). For example, the system can process the network input using a deep neural network. The agent discomfort prediction characterizes a level of discomfort imposed by the vehicle onto the agent. In some implementations, the network input is a machine-learned input, e.g., the network input can be learned concurrently with the training of the neural network.

The system combines the agent discomfort predictions of the respective surrounding agents to generate an aggregated discomfort score (step506). For example, the system can determine a mean, median, minimum, or maximum of the agent discomfort predictions to generate the aggregated discomfort score.

As another example, the system can process the agent discomfort predictions using a learned function to generate the aggregated discomfort score. For example, the system can learn the function by simulating operation of the vehicle or by operating the vehicle in the real-world. The system can make driving decisions according to the generated aggregated discomfort scores, determine a quality of the driving decisions, and update the learned function according to the determined quality. For example, if a gradient of the quality is available, the system can update the learned function using backpropagation. As a particular example, the system can process the agent discomfort predictions using a random forest or a recurrent neural network.

The system provides the aggregated discomfort score to a path planning system of the vehicle (step508). The path planning system can process the aggregated discomfort score, and/or each of the individual agent discomfort predictions, to generate a future path of the vehicle.

FIG.6is a flow diagram of an example process600for training a neural network to predict the discomfort imposed on a surrounding agent by a vehicle. For convenience, the process600will be described as being performed by a system of one or more computers located in one or more locations. For example, an agent model training system, e.g., the agent model training system300depicted inFIG.3, appropriately programmed in accordance with this specification, can perform the process600.

The system obtains a training example that includes sensor data captured by one or more sensors on-board a particular vehicle (step602).

The system processes the training example using a feature extractor to generate feature data characterizing the particular vehicle (step604).

For example, the feature extractor can generate the feature data using a proper subset of the sensor data, e.g., a subset of the sensor data that corresponds to the particular vehicle (as opposed to surrounding agents, the environment, etc.). That is, the subset of the sensor data characterizes one or more particular characteristics of the particular vehicle. Importantly, the sensor data also includes data corresponding to each surrounding agent that includes the same particular characteristics. Thus, the feature extractor can generate vehicle feature data and agent feature data of the same form. During training, the feature extractor generated vehicle feature data corresponding to the particular vehicle; during inference, the feature extractor can generate agent feature data corresponding to each surrounding agent.

The system obtains a training label characterizing a level of discomfort of the particular vehicle (step606). For example, the training label can be generated from a user input provided by a user at the time that the sensor data was captured.

The system processes the feature data using the neural network according to current values of the network parameters of the neural network to generate a discomfort prediction (step608). The discomfort prediction predicts the level of discomfort of the particular vehicle.

The system determines an error between the generated discomfort prediction and the training label (step610).

The system determines an update to the network parameters of the neural network according to the determined error (step612)

FIG.7is a flow diagram of an examples process700for training a first neural network having multiple first network parameters to predict the discomfort imposed on a surrounding agent by a vehicle. For convenience, the process700will be described as being performed by a system of one or more computers located in one or more locations. For example, an agent model training system, e.g., the agent model training system400depicted inFIG.4, appropriately programmed in accordance with this specification, can perform the process700.

The system obtains trained second network parameters of a second neural network (step702). The second neural network is configured to process a second network input generated from sensor data captured by one or more sensors on-board a vehicle in an environment and to generate a second network output characterizing a level of discomfort of the vehicle.

The system obtains a training example that includes sensor data captured by one or more sensors on-board a particular vehicle (step704).

The system processes the training example using a feature extractor to generate feature data characterizing a particular agent surrounding the particular vehicle (step706). In some implementations, the feature extractor is the same feature extractor that generates the second network input.

The system processes the feature data using the second neural network to generate a second network output (step708). The second network output characterizes a level of discomfort of the particular agent and will be used as the training label corresponding to the feature data for training the first neural network.

The system processes the feature data using the first neural network according to current values of the first network parameters to generate a discomfort prediction (step710).

The system determines an error between the generated discomfort prediction and the second network output (step712).

The system determines an update to the first network parameters according to the determined error (step714).

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, off-the-shelf or custom-made parallel processing subsystems, e.g., a GPU or another kind of special-purpose processing subsystem. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, e.g., a mouse, trackball, or a presence sensitive display or other surface by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return.

Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

In addition to the embodiments described above, the following embodiments are also innovative:Embodiment 1 is a method comprising:obtaining sensor data characterizing an environment, wherein the sensor data has been captured by one or more sensors on-board a vehicle in the environment;processing, for each of one or more surrounding agents in the environment, a network input generated from the sensor data using a neural network to generate an agent discomfort prediction that characterizes a level of discomfort of the agent;combining the one or more agent discomfort predictions to generate an aggregated discomfort score; andproviding the aggregated discomfort score to a path planning system of the vehicle in order to generate a future path of the vehicle.Embodiment 2 is the method of embodiment 1, wherein the network input is a machine-learned network input that was learned concurrently with the training of the neural network.Embodiment 3 is the method of any one of embodiments 1 or 2, wherein combining the one or more agent discomfort predictions comprises one or more of:determining an measure of central tendency of the agent discomfort predictions,determining an minimum of the agent discomfort predictions,determining a maximum of the agent discomfort predictions, orprocessing each agent discomfort prediction using a learned function.Embodiment 4 is the method of any one of embodiments 1-3, wherein the network input for a particular surrounding agent comprises a top-down image of the environment centered on the particular surrounding agent.Embodiment 5 is the method of any one of embodiments 1-4, wherein the neural network has been trained using i) training sensor data captured by sensors on-board one or more vehicles operating in the real world and ii) user input identifying a respective comfort level of the vehicles at a plurality of time points during the operation.Embodiment 6 is the method of any one of embodiments 1-5, wherein for each surrounding agent:the network input has been generated by processing the sensor data using a feature extractor;the feature extractor generates the feature data using a proper subset of the sensor data; andthe proper subset of the sensor data comprises first data that characterizes one or more particular characteristics of the surrounding agent.Embodiment 7 is the method of any one of embodiments 1-6, wherein the neural network has been trained using feature distillation using a second neural network that is configured to process a second network input generated from sensor data captured by one or more sensors on-board the vehicle and to generate a second network output characterizing a level of discomfort of the vehicle.Embodiment 8 is a method of training a first neural network having a plurality of first network parameters and configured to process a first network input generated from sensor data captured by one or more sensors on-board a vehicle in an environment and to generate a first network output comprising an agent discomfort prediction, wherein the agent discomfort prediction characterizes a level of discomfort of an agent surrounding the vehicle in the environment, the method comprising:obtaining a plurality of trained second network parameters of a second neural network configured to process a second network input generated from sensor data captured by one or more sensors on-board a vehicle in an environment and to generate a second network output characterizing a level of discomfort of the vehicle;obtaining a training example comprising sensor data captured by one or more sensors on-board a particular vehicle in a particular environment;processing the training example using a feature extractor to generate feature data characterizing a particular agent surrounding the particular vehicle in the particular environment;processing the feature data using the second neural network to generate a second network output characterizing a level of discomfort of the particular agent;processing the feature data using the first neural network according to current values of the plurality of first network parameters to generate a discomfort prediction that characterizes the level of discomfort of the particular agent;determining an error between the generated discomfort prediction and the second network output; anddetermining an update to the current values of the plurality of first network parameters according to the determined error.Embodiment 9 is the method of embodiment 8, wherein:the feature extractor generates the feature data using a proper subset of the sensor data;the proper subset of the sensor data comprises first data that characterizes one or more particular characteristics of the vehicle; andthe sensor data comprises second data that characterizes the one or more particular characteristics of each surrounding agent in the particular environment.Embodiment 10 is a system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 1 to 9.Embodiment 11 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of embodiments 1 to 9.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain some cases, multitasking and parallel processing may be advantageous.