Learning to Drive by Observing the Best and Synthesizing the Worst

1 INTRODUCTION . In order to drive a car , a driver needs to see and understand the various objects in the environment , predict their possible future behaviors and interactions , and then plan how to control the car in order to safely move closer to their desired destination while obeying the rules of the road . This is a difficult robotics challenge that humans solve well , making imitation learning a promising approach . Our work is about getting imitation learning to the level where it has a shot at driving a real vehicle ; although the same insights may apply to other domains , these domains might have different constraints and opportunities , so we do not want to claim contributions there . We built our system based on leveraging the training data ( 30 million real-world expert driving examples , corresponding to about 60 days of continual driving ) as effectively as possible . There is a lot of excitement for end-to-end learning approaches to driving which typically focus on learning to directly predict raw control outputs such as steering or braking after consuming raw sensor input such as camera or lidar data . But to reduce sample complexity , we opt for mid-level input and output representations that take advantage of perception and control components . We use a perception system that processes raw sensor information and produces our input : a top-down representation of the environment and intended route , where objects such as vehicles are drawn as oriented 2D boxes along with a rendering of the road information and traffic light states . We present this mid-level input to a recurrent neural network ( RNN ) , which then outputs a driving trajectory that is consumed by a controller which translates it to steering and acceleration . The further advantage of these midlevel representations is that the net can be trained on real or simulated data , and can be easily tested and validated in closed-loop simulations before running on a real car . Our first finding is that even with 30 million examples , and even with mid-level input and output representations that remove the burden of perception and control , pure imitation learning is not sufficient . As an example , we found that this model would get stuck or collide with another vehicle parked on the side of a narrow street , when a nudging and passing behavior was viable . The key challenge is that we need to run the system closed-loop , where errors accumulate and induce a shift from the training distribution ( Ross et al . ( 2011 ) ) . Scientifically , this result is valuable evidence about the limitations of pure imitation in the driving domain , especially in light of recent promising results for high-capacity models ( Laskey et al . ( 2017a ) ) . But practically , we needed ways to address this challenge without exposing demonstrators to new states actively ( Ross et al . ( 2011 ) ; Laskey et al . ( 2017b ) ) or performing reinforcement learning ( Kuefler et al . ( 2017 ) ) . We find that this challenge is surmountable if we augment the imitation loss with losses that discourage bad behavior and encourage progress , and , importantly , augment our data with synthesized perturbations in the driving trajectory . These expose the model to non-expert behavior such as collisions and off-road driving , and inform the added losses , teaching the model to avoid these behaviors . Note that the opportunity to synthesize this data comes from the mid-level input-output representations , as perturbations would be difficult to generate with either raw sensor input or direct controller outputs . We evaluate our system , as well as the relative importance of both loss augmentation and data augmentation , first in simulation . We then show how our final model successfully drives a car in the real world and is able to negotiate situations involving other agents , turns , stop signs , and traffic lights . Finally , it is important to note that there are highly interactive situations such as merging which may require a significant degree of exploration within a reinforcement learning ( RL ) framework . This will demand simulating other ( human ) traffic participants , a rich area of ongoing research . Our contribution can be viewed as pushing the boundaries of what you can do with purely offline data and no RL . 2 RELATED WORK . Decades-old work on ALVINN ( Pomerleau ( 1989 ) ) showed how a shallow neural network could follow the road by directly consuming camera and laser range data . Learning to drive in an end-toend manner has seen a resurgence in recent years . Recent work by Chen et al . ( 2015 ) demonstrated a convolutional net to estimate affordances such as distance to the preceding car that could be used to program a controller to control the car on the highway . Researchers at NVIDIA ( Bojarski et al . ( 2016 ; 2017 ) ) showed how to train an end-to-end deep convolutional neural network that steers a car by consuming camera input . Xu et al . ( 2017 ) trained a neural network for predicting discrete or continuous actions also based on camera inputs . Codevilla et al . ( 2017 ) also train a network using camera inputs and conditioned on high-level commands to output steering and acceleration . Kuefler et al . ( 2017 ) use Generative Adversarial Imitation Learning ( GAIL ) with simple affordancestyle features as inputs to overcome cascading errors typically present in behavior cloned policies so that they are more robust to perturbations . Recent work from Hecker et al . ( 2018 ) learns a driving model using 360-degree camera inputs and desired route planner to predict steering and speed . The CARLA simulator ( Dosovitskiy et al . ( 2017 ) ) has enabled recent work such as Sauer et al . ( 2018 ) , which estimates several affordances from sensor inputs to drive a car in a simulated urban environment . Using mid-level representations in a spirit similar to our own , Müller et al . ( 2018 ) train a system in simulation using CARLA by training a driving policy from a scene segmentation network to output high-level control , thereby enabling transfer learning to the real world using a different segmentation network trained on real data . Pan et al . ( 2017 ) also describes achieving transfer of an agent trained in simulation to the real world using a learned intermediate scene labeling representation . Reinforcement learning may also be used in a simulator to train drivers on difficult interactive tasks such as merging which require a lot of exploration , as shown in Shalev-Shwartz et al . ( 2016 ) . A convolutional network operating on a space-time volume of bird ’ s eye-view representations is also employed by Luo et al . ( 2018 ) ; Djuric et al . ( 2018 ) ; Lee et al . ( 2017 ) for tasks like 3D detection , tracking and motion forecasting . Finally , there exists a large volume of work on vehicle motion planning outside the machine learning context and Paden et al . ( 2016 ) present a notable survey . 3 MODEL ARCHITECTURE . 3.1 INPUT OUTPUT REPRESENTATION . We begin by describing our top-down input representation that the network will process to output a drivable trajectory . At any time t , our agent ( or vehicle ) may be represented in a top-down coordinate system by pt , θt , st , where pt = ( xt , yt ) denotes the agent ’ s location or pose , θt denotes the heading or orientation , and st denotes the speed . The top-down coordinate system is picked such that our agent ’ s pose p0 at the current time t = 0 is always at a fixed location ( u0 , v0 ) within the image . The orientation of the coordinate system is randomly picked for each training example to be within an angular range of θ0 ±∆ , where θ0 denotes the heading or orientation of our agent at time t = 0 . As shown in Fig . 1 , the input to our model consists of several images of size W ×H pixels rendered into this top-down coordinate system . ( a ) Roadmap : a color ( 3-channel ) image with a rendering of various map features such as lanes , stop signs , cross-walks , curbs , etc . ( b ) Traffic lights : a temporal sequence of grayscale images where each frame of the sequence represents the known state of the traffic lights at each past timestep . Within each frame , we color each lane center by a gray level with the brightest level for red lights , intermediate gray level for yellow lights , and a darker level for green or unknown lights1 . ( c ) Speed limit : a single channel image with lane centers colored in proportion to their known speed limit . ( d ) Route : the intended route along which we wish to drive , generated by a router ( think of a Google Maps-style route ) . ( e ) Current agent box : this shows our agent ’ s full bounding box at the current timestep t = 0 . ( f ) Dynamic objects in the environment : a temporal sequence of images showing all the potential dynamic objects ( vehicles , cyclists , pedestrians ) rendered as oriented boxes . ( g ) Past agent poses : the past poses of our agent are rendered into a single grayscale image as a trail of points . We use a fixed-time sampling of δt to sample any past or future temporal information , such as the traffic light state or dynamic object states in the above inputs . The traffic lights and dynamic objects are sampled over the past Tscene seconds , while the past agent poses are sampled over a potentially longer interval of Tpose seconds . This simple input representation , particularly the box representation of other dynamic objects , makes it easy to generate input data from simulation or create it from real-sensor logs using a standard perception system that detects and tracks objects . This enables testing and validation of models in closed-loop simulations before running them on a real car . This also allows the same model to be improved using simulated data to adequately explore rare situations such as collisions for which real-world data might be difficult to obtain . Using a topdown 2D view also means efficient convolutional inputs , and allows flexibility to represent metadata and spatial relationships in a human-readable format . Papers on testing frameworks such as Tian et al . ( 2018 ) , Pei et al . ( 2017 ) show the brittleness of using raw sensor data ( such as camera images or lidar point clouds ) for learning to drive , and reinforce the approach of using an intermediate input representation . 1We employ an indexed representation for roadmap and traffic lights channels to reduce the number of input channels , and to allow extensibility of the input representation to express more roadmap features or more traffic light states without changing the model architecture . If I denotes the set of all the inputs enumerated above , then the RNN recurrently predicts future poses of our agent conditioned on these input images I as shown by the green dots in Fig . 1 ( h ) . pt+δt = RNN ( I , pt ) ( 1 ) In Eq . ( 1 ) , current pose p0 is a known part of the input , and then the RNN performs N iterations and outputs a future trajectory { pδt , p2δt , ... , pNδt } along with other properties such as future speeds . This trajectory can be fed to a controls optimizer that computes detailed driving control ( such as steering and braking commands ) within the specific constraints imposed by the dynamics of the vehicle to be driven . Different types of vehicles may possibly utilize different control outputs to achieve the same driving trajectory , which argues against training a network to directly output lowlevel steering and acceleration control . Note , however , that having intermediate representations like ours does not preclude end-to-end optimization from sensors to controls .

Summary: The paper describes a framework for training a self-driving policy by augmenting imitation loss with additional loss terms that penalize undesired behaviors and that encourage progress. The policy takes as input a parsed representation of the scene (rather than raw images) and outputs pose trajectories for a down-stream controller. The method is trained on simulated data that includes perturbations to improve generalizability. The framework is evaluated in simulation through a series of ablations to better understand the contribution of the different loss terms.