Conditional Affordance Learning
for Driving in Urban Environments
Axel Sauer1,2

Nikolay Savinov1

Andreas Geiger1,31Computer Vision and Geometry Group, ETH Zürich
Chair of Robotics Science and System Intelligence, Technical University of Munich
3
Autonomous Vision Group, MPI for Intelligent Systems and University of Tübingen
2

Abstract: Most existing approaches to autonomous driving fall into one of two
categories: modular pipelines, that build an extensive model of the environment,
and imitation learning approaches, that map images directly to control outputs.
A recently proposed third paradigm, direct perception, aims to combine the advantages of both by using a neural network to learn appropriate low-dimensional
intermediate representations. However, existing direct perception approaches are
restricted to simple highway situations, lacking the ability to navigate intersections, stop at traffic lights or respect speed limits. In this work, we propose adirect perception approach which maps video input to intermediate representations suitable for autonomous navigation in complex urban environments given
high-level directional inputs. Compared to state-of-the-art reinforcement and conditional imitation learning approaches, we achieve an improvement of up to 68 %in goal-directed navigation on the challenging CARLA simulation benchmark. In
addition, our approach is the first to handle traffic lights and speed signs by using
image-level labels only, as well as smooth car-following, resulting in a significant
reduction of traffic accidents in simulation.
Keywords: Autonomous Driving, Sensorimotor Control, Affordance Learning

1

Introduction

An autonomous vehicle is a cognitive system and hence follows the concept of a rational agent:in order to operate safely it must accurately observe its environment, make robust decisions andperform actions based on these decisions [1]. Autonomous navigation can be described as a mapping
function from sensory input to control output. To implement this function, three major approaches
have been proposed: modular pipelines, imitation learning, and direct perception.
Most autonomous vehicles are based on modular pipelines (MP) [2, 3]. An MP splits the autonomous driving problem into smaller, easier subproblems: perception, path planning, and control.
The approach often relies on various sensors in order to produce a consistent representation of thesurrounding environment. A driving decision is then made based on this representation. While MPs
are relatively interpretable due to their modularity, they rely on complex intermediate representations which are manually chosen (e.g., optical flow), often hard to estimate with sufficient accuracy
and might not necessarily be the optimal choice for solving the sensorimotor control task. Furthermore, MPs require large amounts of annotated data which can be costly to obtain, e.g., pixel-wisesemantic segmentation for training neural networks or high-definition maps for localization.
The second approach, imitation learning (IL) [4], maps directly from raw input to control output
and is learned from data in an end-to-end fashion, skipping the intermediate step of building an
explicit environment model. In contrast to MPs, data for training IL-based navigation systems canbe collected with relative ease, i.e., by driving around and recording the video footage together withthe driven trajectory. However, IL approaches are faced with the problem of learning a very complex
mapping (from raw input to control) in a single step. Thus, a model with high capacity as well as avery large amount of training data is required for handling the large variety of real world situations
which may be encountered at test time. Besides, IL lacks transparency as it is hard to comprehend
the internal decision-making process in a neural network. This raises security concerns: a driving
system that does not act transparently might not be trusted or confidently used by humans.
2nd Conference on Robot Learning (CoRL 2018), Zürich, Switzerland.

Video Input

Control
Commands

Affordances

NeuralNetwork
Directional Input

𝑅𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝑎𝑛𝑔𝑙𝑒 = 0.01 𝑟𝑎𝑑𝐶𝑒𝑛𝑡𝑒𝑟𝑙𝑖𝑛𝑒 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒= 0.15 𝑚
𝑅𝑒𝑑 𝑙𝑖𝑔ℎ𝑡 = 𝐹𝑎𝑙𝑠𝑒…

Controller

Brake = 0.0

Figure 1: Conditional Affordance Learning (CAL) for Autonomous Urban Driving. The input
video and the high-level directional input are fed into a neural network which predicts a set ofaffordances. These affordances are used by a controller to calculate the control commands.
The third approach, direct perception (DP) [5], aims to combine the advantages of MP and IL.
Instead of predicting a detailed representation of the environment, the goal in DP is to predict a lowdimensional intermediate representation of the environment which is then used in a conventional
control algorithm to maneuver the vehicle. Thus, DP neither requires the network to learn thecomplex sensorimotor control problem end-to-end, nor does it assume the availability of datasetswith pixel-level or box-level labels that are significantly more time-consuming to obtain than imagelevel labels. Chen et al. [5] demonstrated good results when applying this approach to highway
driving using an open-source simulator. However, highway driving is a rather easy task comparedto driving in rural or urban areas. When considering navigation in urban areas, several difficulties
are added: the agent must obey traffic rules (speed limits, red lights, etc.), take into account possible
obstacles on the road (e.g., pedestrians crossing the street), and handle junctions with more than one
possible direction.
At the core of the direct perception approach lies the choice of intermediate representation to bepredicted. Ideally, this representation should be of low dimensionality while comprising all the necessary information for making a driving decision. One choice of such representation are affordances,
attributes of the environment which limit the space of allowed actions. As an example, consider the
distance to the vehicle ahead which limits the ability of the ego-vehicle to speed up. Affordances canalso be conditioned on high-level driving decisions as provided, e.g., by a conventional navigation
system. Consider a vehicle at an intersection: the distance to the centerline changes with the desired
turn we would like the vehicle to take (and thus the lane we would like it to follow).
The goal of this work is to generalize the direct perception approach to the urban setting. We develop
novel intermediate representations in the form of affordances suitable for urban navigation as wellas conditional models where the decision is not based on the image input alone, but also on highlevel navigation commands as provided by a consumer grade navigation system (e.g., “turn left at
the next intersection”). Moreover, our system is designed to drive more safely and comfortably,
alleviating the common problem of jerky driving behavior in existing approaches for autonomous
driving [6]. We coin the developed approach “Conditional Affordance Learning” (CAL) as thepredicted affordances and the model output are conditioned on high-level navigation commands.
Figure 1 shows an overview of our approach. The main contributions of our work are summarized
as follows:• We develop novel intermediate representations which are low-dimensional, yet provide the necessary information to allow for driving in urban environments.
• We deploy a conditional navigation model which allows for taking high-level directional commands prior to intersections. We further implement a control algorithm which allows for a smoothride while obeying traffic rules.
• We provide an in-depth evaluation of different network architectures and parameterizations. Our
network is able to predict all intermediate representations in a single forward pass.
• We demonstrate the benefits of video-based representation learning for navigation. Prior works
either operate frame-by-frame or do not evaluate in an online setting.
The code and our trained models can be found at https://github.com/xl-sr/CAL.
2

2

Related Work

Modular pipelines are the most popular approach to autonomous driving and are employed by mostresearch projects [2, 3]. To build the environmental model, the perception stack needs to detect
all aspects of the traffic scene that are likely to be relevant for the driving decision. These detection
tasks are usually trained and solved separately [7], most recently by exploiting deep neural networks,e.g., for object detection [8, 9], image segmentation [10, 11], or motion estimation [12, 13]. This
information can be aggregated into a model of the environment [14, 15] and a planning module
generates an obstacle-free trajectory which is executed by the control module [16].
The first implementation of the imitation learning (IL) approach in the real world, ALVINN [4],dates back to the 80s. More recently, Muller et al. [17] employ a similar system to navigate an offroad truck in a variety of terrains, weather, and lighting conditions while avoiding obstacles. Using
more computational power and a modern CNN architecture, contemporary approaches such as theone by Bojarski et al. [18] demonstrate impressive performance on real-world tasks like highway
following and driving in flat courses. Codevilla et al. [19] propose a conditional imitation learning
formulation that is able to traverse intersections based on high-level navigational commands.
The idea of direct perception is to predict several affordance indicators describing the driving scene.
The concept of affordances was originally proposed by Gibson [20] in the field of psychology andhas been applied to the autonomous driving task by Chen et al. [5], demonstrating strong performance in a racing simulation. Al-Qizwini et al. [21] improve the original approach of Chen et al. [5]by analyzing different CNNs for the mapping from image to indicators. They find GoogLeNet [22]and VGG16 [23] to perform best on this task.
Due to the difficulties of training and testing in real environments, driving simulators are often
exploited in related work. Two popular driving simulators for research are the open source car
racing game TORCS [5, 21, 24] and the commercial game Grand Theft Auto V (GTA V) [25, 26].
However, TORCS is neither photorealistic nor complex enough, lacking important scene elements
such as intersections, pedestrians and oncoming traffic. GTA V can be considered photorealistic,
but is closed source and hence limited in its ability to customize and control the environment. In thiswork, we use the recently released open-source simulator CARLA [27] which provides a trade-off
between realism and flexibility, thus addressing some of the problems of previous simulators.

3

Conditional Affordance Learning

Our autonomous driving setup consists of an agent which interacts with the environment as illustrated in Figure 2. A high-level command for the maneuver to be performed (e.g., “go straight”,“turn left”, “turn right”) is provided by a planner, mimicking one of the most common driving scenarios: a human driver (the agent) following the commands of a navigation system (the planner).
CARLA uses an A* topological planner to generate these commands. Based on the high-level command and the observation from the environment, the agent calculates the value of throttle, brakeand steering wheel. Those values are sent back to the environment model which provides the next
observation and high-level command, and the process continues until the goal is reached.
In this work, we focus on visual-based navigation: we only consider observations from a single
front-facing camera. Cameras have the advantage that they are ubiquitous and cheap. In addition,
they provide information that laser scanners or radar sensors cannot provide, e.g., the color of atraffic light or the type of a speed sign. Thus they are the only sensor that allows to solve the task
without any additional sensors. Note that a monocular setup is comparable to how humans solve
the driving task. The stereo baseline of the human visual system is insufficient to perceive accurate
depth at distances of 50 meters and beyond based on this cue alone.
In the following we will discuss the intermediate representations required for autonomous navigationin urban environments and the submodules of the CAL agent for perception and control.
3.1

Affordances

A good intermediate representation should be low-dimensional, yet expressive enough to meet thenecessary requirements for autonomous navigation: (i) the agent should be able to drive from A to B
as fast as possible while obeying traffic rules. (ii) infractions against traffic rules should be avoided3Perception
Feature Extractor
Featuremap

CAL Agent

MemoryTask Blocks

1
2
3

unconditional...

Affordances
...

...
N-2
N-1
N

...

Controller
Longitudinal Control
Lateral Control...
conditional
Directional
CommandHigh-level Planner
Position

Image
Agent in Environment

Control
Commands

CARLA

Figure 2: Overview of our System. The CAL agent (top) receives the current camera image anda directional command (“straight”,“left”,“right”) from CARLA [27]. The feature extractor converts
the image into a feature map. The agent stores the last N feature maps in memory, where N is
the length of the input sequence required for the perception stack. This sequence of feature maps,
together with the directional commands from the planner, are exploited by the task blocks to predictaffordances. Different tasks utilize different temporal receptive fields and temporal dilation factors.
The control commands calculated by the controller are sent back to CARLA which updates theenvironment and provides the next observation and directional command.
at all times. In this work, we consider 6 types of infractions: driving on the wrong lane, driving on
the sidewalk, running a red light, colliding with other vehicles, hitting pedestrians and hitting static
objects. Moreover, the speed limit must be obeyed at all times. (iii) the agent should be able toprovide a pleasant driving experience. The car should stay in the center of the road and take turns
smoothly. In addition, it should be able to follow a leading car at a safe distance.
Based on these requirements we propose the affordance set shown in Figure 3 (left). The agent
should stop for red traffic lights and change its speed according to the current speed sign. For thispurpose we define an observation area A1 in the local (x, y) coordinates of the agent, as shown in
Figure 3 (right). If a traffic light or a speed sign is within A1 , the corresponding affordance indicator
switches to True or to the speed limit value. The agent keeps an internal memory of the speed
limit. Once a speed sign is passed, the current speed limit is updated and the agent adjusts its speed
accordingly. If an obstacle is in front of the car — located within an observation area A2 — theagent should perform a hazard stop — come to a complete stop as quickly as possible. In CARLA,
possible road obstacles are cars and pedestrians. For driving in the center of the road, the position
and the orientation of the car on the road must be known. The position is defined by the distance tocenterline d, the lateral distance of the center of the vehicle’s front axle to the closest point on thecenterline. The relative angle ψ describes the orientation of the car with respect to the road tangent.
To achieve steady vehicle following, the distance to the vehicle ` ahead must be known. If a car
is located in an observation area A3 , we measure ` as the shortest distance between the boundingboxes of both cars, otherwise set ` = 50.
3.2

Perception

We formulate perception as a multi-task learning (MTL) problem: using a single neural network we
predict all affordances in a single forward pass. This yields a computationally efficient solution –for a forward pass, the system (Network + Controller) takes 50 ms on average on a NVIDIA GTX
1080 Ti GPU. This is well within real-time requirements for autonomous driving systems, wherelatency should reach levels below 100 ms [28]. Furthermore, MTL has been shown to learn internal
representations which are able to improve the network’s generalization abilities [29]. Xu et al. [30]demonstrate this effect in the context of autonomous driving. In our network, all convolutional
layers are shared between tasks. The output of the last convolutional layer is used as input for thetask-specific branches which we will call “task blocks” in the following.
4

Type

Conditional

Affordances

AcronymRange of values

discreteNo

Hazard stop-∈ {T rue, F alse}

vehicle𝒍 = 𝟏𝟓 𝒎
centerline

Red Traffic Light-∈ {T rue, F alse}

Speed Sign [km/h]-∈ {N one, 30, 60, 90}𝑨𝟑hazard stop(for pedestrian)

speed
sign
30

continuous

No

Distance to vehicle [m]`∈ [0, 50]

continuous

Yes

Relative angle [rad]ψ∈ [−π, π]Distance to centerline [m]d∈ [−2, 2]km/h𝒅𝒚𝒍𝒐𝒄𝒂𝒍𝒙𝒍𝒐𝒄𝒂𝒍𝝍
𝐴2

𝐴1
traffic
light

agent

Figure 3: Affordances. Left: We categorize affordances according to their type (discrete/continuous) and whether they are conditional (dependent on directional input) or unconditional. Right: Illustration of the affordances (red) and observation areas used by our model.
Neural network architecture. To extract features from raw input images, we use the convolutional
layers of a VGG16 network [23] pretrained on ImageNet [31]. The simple structure of the network
make it an appealing choice for a feature extractor with real-time requirements. The feature extractor
is applied to every image in the input sequence to extract a sequence of feature maps. The task blocks
are shallow networks and consist of a task-specific layer, followed by a batch normalization layer[32] and a dropout layer. For the task-specific layers we experiment with dense layers, LSTMs [33],
GRUs [34], and temporal convolutional layers. Each task block has a different temporal receptive
field and temporal dilation factors (e.g., using only every other image). The output nodes of theconditional task blocks are divided into three equally sized groups. The directional command c ∈{straight, lef t, right} is used to switch between these groups of nodes. Only the active groupis used to predict the target label. During training the gradients are only backpropagated throughthis group. We observed that training specialized submodules for each directional command leads
to better performance compared to using the directional command as an additional input to the task
networks. This observation agrees with Codevilla et al. [19]. However, note that in contrast to their
Conditional Imitation Learning approach, we use specialized task networks to predict conditional
affordances rather than conditional steering commands. Conditioning in the network has several
advantages over conditioning in the controller, see the supplementary material for a discussion.
Training. As our approach decouples affordance prediction and control, we are able to train our perception stack independently from the chosen control algorithm. To train the affordance prediction
network, we recorded a dataset with the CARLA simulator [27]. CARLA provides two towns: Town
1 for training and Town 2 for testing. For collecting training data in Town 1, we navigate throughthe city using our controller (described in section 1.2) based on the ground truth affordances. During
data collection, the directional input at intersections is chosen at random. All other traffic participants and all traffic lights are controlled by the in-game AI. The full dataset contains approximately
240 thousand images and the corresponding ground truth labels for each of the six affordances. Five
percent of the dataset is used for validation. The network is trained on mini-batches of size 32.
For optimization we use Adam [35] with a base learning rate of 5e-5. For classification we use theclass-weighted categorical cross-entropy H where the weights are chosen inversely proportional tothe class label occurrence in the training set. For regression, we use the mean average error MAE.
All six affordance loss functions are added to yield the loss function for the whole network:
L=

3
X

Hj +

j=1

3
X

MAEk .

(1)

k=1

Data augmentation is crucial to achieve generalization. Following [27], we used the following augmentations: change in color, contrast, and brightness as well as Gaussian blur and salt-and-pepper
noise. Common techniques like vertical and horizontal flipping are not used, because in the driving
context vertical flipping would correspond to switching from right- to left-hand traffic and horizontal
flipping — to driving upside down. In addition, we employ camera pose randomization. It is wellknown that for imitation learning utilizing a single fixed camera pose leads to unstable behavior[18, 36] due to control errors and unexpected observations reinforcing each other. Thus, we use athree-camera setup that records the same episode at 3 different baselines (−50 cm, 0 cm, +50 cm).
In addition, each camera is randomly rotated around its vertical axis by up to ±15◦ . This setup is
only used for collecting the dataset. During testing, we use the input of a single centered camera.
5During 10 % of the training episodes, we additionally set the distance to the preceding vehicle ` tozero to provoke rear-end collisions with other cars which otherwise would not occur in our training
set. This way the network learns to correctly predict these situation as hazard stops.
Hyperparameter search. The task block hyperparameters entail the type of layer (Dense, LSTM,
GRU, Temporal Convolution), the number of nodes within these layers, the dropout ratio p, thesequence length seq, and the dilation value dil. The architecture of our network is optimized in a
two-step procedure. First, we randomly choose type and sample values for nodes, p, seq, and dilfrom predefined value ranges (see supplementary material for the respective ranges). We initialize the network with the same set of parameters for each task block and train it until the error on
the validation set converges. Second, the overall best performing task blocks are combined into a
new model. The combined model is then trained jointly until the validation error converges. The
hyperparameters of the best performing task blocks are shown in the supplementary material.
3.3

Controller

The control algorithm is split into longitudinal control (throttle, brake) and lateral control (steering)
since it is possible to decouple these two tasks [37].
Longitudinal control. The longitudinal controller is subdivided into several states, see the supplementary material for an illustration. The states are (in ascending importance): cruising, following, over limit, red light, and hazard stop. All states are mutually exclusive. For cruising thetarget speed v ∗ is the current speed limit. Given the current speed v(t) we minimize the error terme = v ∗ (t) − v(t) using a PID controller [38] for the throttle value. The brake is set to zero. The
value of v ∗ also depends on the current road type. If the agent receives a directional input to turnleft or right, we reduce v ∗ by 10 km/h to enable smooth and safe navigation through turns. The stateswitches to following if another car is less than 35 m ahead. We chose this static threshold, because
we found that a dynamic threshold combined with the uncertainty of the network predictions leads
to undesired state switching between following and cruising. We will denote the distance to the front
car as ` and the current speed limit as vmax . We use the optimal car-following model of Chen et al.
[5] and minimize the error term




c
e(t) = v ∗ (t) − v(t) = vmax 1 − exp −`(t) − d − v(t)(2)
vmaxwith a second PID controller. If the car is driving faster than the current speed limit, the stateswitches to over limit, for a detected red light it switches to red light, for a detected obstacle itswitches to hazard stop. For over limit, red light, and hazard stop, the throttle and brake value
depend on the vehicle speed, the prediction probability and the urgency to stop. The details as well
as a description of the PID tuning procedure are provided in the supplementary material.
Lateral control. For lateral control we use the Stanley Controller (SC) [2]. The SC uses two error
metrics: the distance to centerline d(t) and the relative angle ψ(t). The control law to calculate thesteering angle δSC (t) at the current vehicle speed v(t) is given by


kd(t)
δSC (t) = ψ(t) + arctan(3)v(t)
where k is a gain parameter. To calculate the final steering output δ, we add a damping term δd which
reduces swaying on straight roads. δd (t) is calculated by using the current output of the control law
δSC (t), the steering angle at the previous time step δ(t − 1), and a positive damping constant D:
δ(t) = δSC (t) + δd (t) where δd (t) = −D ∗ (δSC (t) − δ(t − 1))

4(4)

Results

We use the driving benchmark and evaluation protocol of [27] as the basis of our evaluation. The
benchmark is composed of four different driving tasks with increasing difficulty: driving straight,
driving through a single turn, navigating through the town taking several turns, and navigating
through the town with additional dynamic obstacles (cars, pedestrians). The agent is initialized
at a pre-defined point in the town and has to reach a goal under a task-specific time limit. The time
limit equals the time needed to reach the goal when driving along the optimal path at 10 km/h.
6

Table 1: Quantitative Evaluation on Goal-Directed Navigation Tasks. We show the percentage
of successfully completed episodes per task and compare CAL to a Modular Pipeline (MP) [27],Conditional Imitation Learning (CIL) [19] and Reinforcement Learning (RL) [27].

Task

MP

CIL

RL

CAL

MP

CIL

RL

CAL

MP

CIL

RL

CAL

New town andnew weather
MP CIL RL CAL

Straight
One turn
Navigation
Nav. dynamic

98
82
80
77

95
89
86
83

89
34
14
7

100
97
92
83

100
95
94
89

98
90
84
82

86
16
2
2

100
96
90
82

92
61
24
24

97
59
40
38

74
12
3
2

93
82
70
64

50
50
47
44

Training conditions

New weather

New town

80
48
44
42

68
20
6
4

94
72
68
64

The simulation engine provides two different towns (Town 1, Town 2). Only Town 1 has been seen
during training. The evaluation distinguishes between weather conditions that were seen during
training and those that were not. For each combination of task, town, and weather condition, 25 different episodes are evaluated. We compare our Conditional Affordance Learning (CAL) approach to
Conditional Imitation Learning (CIL) [19] as well as the Modular Pipeline (MP) and Reinforcement
Learning (RL) approach presented in [27]. As stated in the original paper, the RL baseline cannot
be claimed state-of-the-art as it still has room for improvement regarding its implementation. Notethat the baselines ignore all red lights and target a cruising speed of 20 km/h. In contrast, our agent
stops for red lights and recognizes speed signs. To provide a fair comparison to prior art, we limit
our algorithm to a cruising speed of 20 km/h for the quantitative evaluation against the baselines.
We use the same model for all tasks and weather conditions without parameter fine-tuning.
4.1

Goal-Directed Navigation

Table 1 shows our quantitative comparison with respect to the state-of-the-art on CARLA on thetask of goal-directed navigation. Our agent outperforms the baselines in most of the tasks and undermost conditions. As expected, performance decreases for all methods with increasing task difficulty.
CAL particularly excels in generalizing to the new town, a scenario where most baselines did not
perform well. For the new town under training weather conditions, the performance of the CAL
agent is up to 68 % better than the best baseline.
CAL outperforms MP on almost all of the tasks except for the tasks in the training town under new
weather conditions where MP performs slightly better. In almost all tasks and conditions, CAL
shows better performance compared to CIL. Prediction of the relative angle ψ especially profits
from the temporal structure of the CAL network as shown in the supplementary material. Also,
the additional parameters of the controller (e.g., damping D) improve driving performance. CAL isoutperformed by CIL only on one task: driving straight under the training weather conditions in thetest town. Our analysis revealed that this happens due to a systematic failure of the CAL perception
stack in one specific weather condition where several trees cast shadows onto the street, affecting
the prediction of the distance d to centerline such that the agent occasionally drifts off the street.
4.2

Infraction Analysis

Table 2 shows the average distances driven between two infractions during the hardest task of thebenchmark: navigation with dynamic obstacles. Under test conditions our approach is able to outperform the baselines in almost every measure. This indicates the strong generalization ability of
our approach in comparison to the baselines. CAL performs particularly well in avoiding collisions
with other vehicles and pedestrians. In the new environment, our agent performs more than 10 times
better than the best baseline (RL). During testing, pedestrians are crossing the street more frequently
than in the training scenario. The fact that our agent is still able to perform well demonstrates thatthe hazard stop network was able to predict this affordance robustly despite its sparsity.
Occasionally, the CAL agent drives on the sidewalk. This infraction occurs primarily during rightturns. Since rights turns in CARLA are generally sharper than left turns, the relative motion of
pixels between consecutive frames is larger during a right turn. This may pose a bigger challenge to
our perception network which is operating on sequential input. Driving on the sidewalk sometimes
also results in crashes with static objects, e.g., traffic lights. Note that the infraction count for static
collisions is generally higher than that for driving on the sidewalk as the agent sometimes slides
along a railing located next to the road, continuously increasing the infraction count.
7

Table 2: Infraction Analysis. We show the average distance (in km) driven between two infractions.
’>’ indicates that no infraction occurred over the whole course of testing. Note that the total driven
distance depends on the amount and length of the episodes that an agent finishes.

Task

MP

CIL

RL

CAL

MP

CIL

RL

CAL

MP

CIL

RL

CAL

MP

New town andnew weather
IL
RL
CAL

Opposite lane
Sidewalk
Collision: Static
Collision: Car
Collision: Pedestrian

10.2
18.3
10.0
16.4
18.9

33.4
12.9
5.38
3.26
6.35

0.18
0.75
0.42
0.58
17.8

6.7
6.1
2.5
12.1
30.3

16.1
24.2
16.1
20.2
20.4

57.3
>57
4.05
1.86
11.2

0.09
0.72
0.24
0.85
20.6

>60
6.0
6.0
>60
>60

0.45
0.46
0.44
0.51
1.40

1.12
0.76
0.40
0.59
1.88

0.23
0.43
0.23
0.41
2.55

2.21
0.88
0.36
2.04
26.49

0.40
0.43
0.45
0.47
1.46

0.78
0.81
0.28
0.44
1.41

Training conditions

New weather

New town

0.21
0.48
0.25
0.37
2.00

2.24
1.34
0.31
1.68
6.72

Figure 4: Attention Analysis. Left: The network focuses at the periphery for recognizing traffic
lights. Middle: The network is anticipating a hazard stop by observing the shadow of a pedestrian.
Right: The network focuses its attention on the pedestrian to correctly predict a hazard stop.
4.3

Attention Analysis

In order to gain more insight into the decision making process of our perception stack we explore
gradient-weighted class activation maps (Grad-CAMs) [39] which expose the implicit attention of
the CNN with respect to the input image. Figure 4 shows the results for the affordances hazard stopand red light. We encode the prediction probability for a particular class with different colors(green means low probability, red means high probability). The attention given to a particular image
region is encoded in the intensity channel. The stronger the attention, the more intense the color.
Regions with very low attention are shown transparent for clarity. For imitation learning this attention analysis would be much less interpretable since predicting a single control output, e.g. throttle,
entails the detection of speed limits, red lights, cars and pedestrians - all in one signal.
Figure 4 (left) shows the agent driving through town 1 with the attention map for red light overlaid.
Note that the network focuses its attention on the road outlines and structures on the roadside wheretraffic lights are expected to be located. In Figure 4 (middle) the shadow of a pedestrian is visibleat the bottom left of the image. Interestingly, the network anticipates an impending hazard stopand focuses its attention to this region despite the fact that the pedestrian itself is not visible in theimage. Finally, in Figure 4 (right) we illustrate a hazard stop situation. The network focuses all its
attention on the pedestrian’s lower body. A possible explanation is that the network is able estimate
the pedestrian’s walking direction by observing its legs and feet. Note that no pixel-wise supervision
has been provided to the network during training. Instead, this behavior has been learned from binary
image-level labels (hazard stop = T rue/F alse) alone.

5

Conclusion

We have proposed Conditional Affordance Learning which combines the advantages of both modular pipelines and end-to-end approaches for navigating complex urban environments using high-level
directional commands as input. We tested our approach extensively in simulation and demonstrated
significant performance gains with respect to the state-of-the-art on CARLA. In the future, we plan
to extend our work in several directions. Training on more diverse environments will likely improve
the agent’s performance in all domains. An extended study of network architectures can improve
the quality of the predictions and therefore improve the driving capabilities of the agent. We alsoplan to extend the sensor setup, e.g., using a stereo camera to improve distance prediction. Moresophisticated control algorithms like model predictive control may further improve the performance,in particular when training the parameters of the controller jointly with the perception stack. Finally,
inspired by recent work [40], we plan to transfer our results from simulation to the real world.
8

Acknowledgments
We would like to thank Vanessa Diedrichs for many helpful discussions about autonomous driving
and data visualization and for her general support.

References[1] S. J. Russell and P. Norvig. Artificial intelligence: a modern approach. 2016.
[2] S. Thrun, M. Montemerlo, H. Dahlkamp, D. Stavens, A. Aron, J. Diebel, P. Fong, J. Gale,
M. Halpenny, and G. Hoffmann. Stanley: The robot that won the DARPA grand challenge.
Journal of Field Robotics, 23(9):661–692, 2006.
[3] J. Ziegler, P. Bender, M. Schreiber, H. Lategahn, T. Strauss, C. Stiller, T. Dang, U. Franke,
N. Appenrodt, and C. G. Keller. Making Bertha drive — An autonomous journey on a historicroute. IEEE Intelligent Transportation Systems Magazine, 6(2):8–20, 2014.
[4] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In NIPS, 1989.
[5] C. Chen, A. Seff, A. Kornhauser, and J. Xiao. Deepdriving: Learning affordance for directperception in autonomous driving. In ICCV, 2015.
[6] R. Metz. And the award for most nauseating self-driving car goes to ... MIT Technology Review,Jan 2018.
[7] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The KITTI dataset. The
International Journal of Robotics Research, 32(11):1231–1237, 2013.
[8] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time
object detection. In CVPR, 2016.
[9] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with
region proposal networks. In NIPS, 2015.
[10] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In ICCV, 2017.
[11] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional instance-aware semantic segmentation. In CVPR, 2017.
[12] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. PWC-Net: CNNs for Optical Flow Using Pyramid,
Warping, and Cost Volume. arXiv.org, 2017.
[13] F. Güney and A. Geiger. Deep discrete flow. In ACCV, 2016.
[14] H. Zhang, A. Geiger, and R. Urtasun. Understanding high-level semantics by modeling traffic
patterns. In ICCV, 2013.
[15] A. Geiger, M. Lauer, C. Wojek, C. Stiller, and R. Urtasun. 3D Traffic Scene Understanding
from Movable Platforms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(5):1012–1025, 2014.
[16] W. Schwarting, J. Alonso-Mora, and D. Rus. Planning and decision-making for autonomousvehicles. Annual Review of Control, Robotics, and Autonomous Systems, 1(1):187–210, 2018.
[17] U. Muller, J. Ben, E. Cosatto, B. Flepp, and Y. L. Cun. Off-road obstacle avoidance throughend-to-end learning. In NIPS, 2006.
[18] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel,
M. Monfort, U. Muller, and J. Zhang. End to end learning for self-driving cars. arXiv.org,
2016.
[19] F. Codevilla, M. Müller, A. Dosovitskiy, A. Lopez, and V. Koltun. End-to-end driving via
conditional imitation learning. In ICRA, 2018.
[20] J. J. Gibson. The senses considered as perceptual systems. 1966.
9

[21] M. Al-Qizwini, I. Barjasteh, H. Al-Qassab, and H. Radha. Deep learning algorithm for autonomous driving using GoogLeNet. In Intelligent Vehicles Symposium, 2017.
[22] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and
A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[23] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv.org, 2014.
[24] J. Huang, I. Tanev, and K. Shimohara. Evolving a general electronic stability program for car
simulated in TORCS. In CIG, 2015.
[25] S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer
games. In ECCV, 2016.
[26] S. Ebrahimi, A. Rohrbach, and T. Darrell. Gradient-free policy architecture search and adaptation. In CoRL, 2017.
[27] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban
driving simulator. In CoRL, 2017.
[28] S.-C. Lin, Y. Zhang, C.-H. Hsu, M. Skach, M. E. Haque, L. Tang, and J. Mars. The architectural
implications of autonomous driving: Constraints and acceleration. In ASPLOS, 2018.
[29] R. Caruana. Multitask learning: A knowledge-based source of inductive bias. In ICML, 1993.
[30] H. Xu, Y. Gao, F. Yu, and T. Darrell. End-to-end learning of driving models from large-scale
video datasets. In CVPR, 2017.
[31] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale
Hierarchical Image Database. In CVPR, 2009.
[32] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing
internal covariate shift. arXiv.org, 2015.
[33] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–
1780, 1997.
[34] K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation.
arXiv.org, 2014.
[35] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv.org, 2014.
[36] M. Müller, V. Casser, J. Lahoud, N. Smith, and B. Ghanem. Sim4cv: A photo-realistic simulator for computer vision applications. International Journal of Computer Vision, 2018.
[37] J. Kosecka, R. Blasi, C. J. Taylor, and J. Malik. A comparative study of vision-based lateralcontrol strategies for autonomous highway driving. In ICRA, 1998.
[38] K. H. Ang, G. Chong, and Y. Li. Pid control system analysis, design, and technology. IEEE
transactions on control systems technology, 13(4):559–576, 2005.
[39] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam:
Visual explanations from deep networks via gradient-based localization. arXiv.org, 2016.
[40] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun. Driving policy transfer via modularity
and abstraction. arXiv.org, 2018.

10

Supplementary Material:
Conditional Affordance Learning
for Driving in Urban Environments
Axel Sauer1,2

Nikolay Savinov1

Andreas Geiger1,31Computer Vision and Geometry Group, ETH Zürich
Chair of Robotics Science and System Intelligence, Technical University of Munich
3
Autonomous Vision Group, MPI for Intelligent Systems and University of Tübingen
2

Abstract: This supplementary document provides further implementation details of our CAL agent in Section 1, a detailed description of our ground truth
acquisition process in Section 2 and additional experiments in Section 3.
The supplementary video shows several navigation examples and visualizes theattention of our agent for different affordance indicators over time. The video isavailable at https://www.youtube.com/watch?v=UtUbpigMgr0.

1

Implementation Details

In this section, we list the value ranges and results of our hyperparameter search. We also provide
additional details about our longitudinal control algorithm and the PID tuning procedure.
1.1

Hyperparameter Search

Table 3 shows each hyperparameter and its range of values for the random search described in themain paper. We initialize the network with randomly sampled parameters from the respective ranges.
Table 4 shows the parameters of the best performing task blocks after optimization.
Table 3: Hyperparameter Search. Value ranges for each hyperparameter.
Hyperparameter
Acronym

Layer type
type

Number of nodes
nodes

Dropout amountp

Sequence length
seq

Dilation valuedilRange of values

Dense, GRU, LSTM,
temporal convolution[10, 200][0.25, 0.75][1, 20][1, 3]

Table 4: Best task block parameters. We optimized the layer type, the number of nodes, thedropout ratio p, the sequence length seq, and the dilation value dil.
Task

type

nodes

p

seq

dil

Red light
Hazard stop
Speed sign
Vehicle distance
Relative angle
Center distance

GRU
Temp. convolution
Dense
GRU
Temp. convolution
Temp. convolution

185
160
160
160
100
100

0.27
0.68
0.55
0.38
0.44
0.44

14
6
1
11
10
10

2
1
1
1
1
1

11

States
if hazard stop == True

hazard_stop

elif red light == True

red_light

Affordances elif speed > limit - 15

over_limit

elif veh_distance < 35

Throttle
Brake

followingelse

cruising

Figure 5: States of Longitudinal Controller. The states are ordered in descending importance from
top-to-bottom as indicated by the color intensity. All state are mutually exclusive.
1.2

Controller

Longitudinal Control. The states of the longitudinal controller are illustrated in Figure 5.
The throttle and brake values for the states over limit, red light, and hazard stop are as follows:• over limit: This state is activated, if the agent is driving more than 15 km/h faster than
the speed limit v ∗ . This situation typically occurs when entering a low-speed zone from ahigh-speed zone. To decelerate quickly, we set the throttle value to zero and calculate thebrake value depending on the current speed v(t):brake = 0.3 ·v(t)v ∗ (t)(5)

As an example, driving with full speed (90 km/h) into a 30 km/h zone yields brake = 0.9,
hence applying the brakes almost fully.
• red light: If the prediction probability P for the class red light, i.e., the actual softmax
output, is higher than a threshold Prl , the controller switches to the state red light. The
throttle is set to zero and the brake is applied with:
v(t)
(6)
30
We empirically found that a threshold of Prl = 0.9 reduces false positives while still
being able to reliable stop in front of red lights. Note that we use a smaller multiplier(0.2) compared to the over limit state as red lights typically occur in 30 km/h zones. We
consider the current speed v(t) to gradually slow down the car in front of red lights.
• hazard stop: This state is activated when an obstacle in front of the agent is detected,
i.e., when P (hazard stop) > Phs where we empirically determined the threshold Phs as
Phs = 0.7. Note the threshold is lower than that for the red light state, since preventing
crashes with road hazards is more critical for successful goal-directed navigation. When ahazard stop has been detected, the throttle is set to zero and the brake is set to one.
brake = 0.2 ·PID Controller Tuning. The PID controllers used in the cruising and f ollowing state follow thestandard PID control scheme [38]. The overall control function is given as follows
Z t
de(t)
u(t) = Kp e(t) + Kie(t0 )dt0 + Kd
,
(7)
dt
0
where Kp , Ki and Kd are the proportional, integral and derivative coefficients. To be able to tune
the coefficients of the two PID controllers, we implemented a visualization tool of speed, distanceto the centerline, and other important signals. With this direct visual feedback, it is possible to use
standard PID tuning methods. In this work, we leverage the method by Ziegler-Nichols [4]. First,
12

Table 5: Observation Areas. The observation areas are rectangular boxes. This table lists the (x, y)
coordinates of the vertices v of each observation area (provided in clocal in meters.)
Area

Detection of

v1

v2

v3

v4

A1

red light
speed sign

(7.4, -0.8)(7.4, -5.8)(14.0, -0.8)(14.0, -5.8)

A2

hazard stop

(0.0, 2.0)

(0.0, -2.0)(8.2,2.0)(8.2,-2.0)

A3

distance to vehicle(0.0, 1.6)

(0.0, -1.6)(50.0, 1.6)

(50.0, -1.6)

all coefficients are set to zero. Then, the proportional gain Kp is increased until the output of theloop starts to oscillate. Given this “ultimate gain” Ku and the oscillation period Tu , we set thecoefficients to
Kp = 0.6Ku ,Ki = Tu /2,
Kd = Tu /8.

(8)
(9)
(10)

Using these values as a starting point, we empirically fine-tune the coefficients for optimal performance, with the goal of enabling fast but smooth reactions to disturbances.
1.3

Conditioning in the Network vs. Conditioning in the Controller

Conditioning in the neural network is advantageous compared to directly conditioning the controllers. Conditioning the controllers would require the prediction of all affordances including theones which are not relevant for the current navigation task at every point in time.
Consider an intersection where the goal is to go straight. Even though the only relevant signal interms of distance to the centerline is the distance to the centerline of the straight lane, the distanceto the centerlines of the other lanes (left turning, right turning) must also be predicted in order to let
the controller select the relevant signal based on the directional command. However, predicting theirrelevant affordances is very difficult as the required image features may not be present (when goingstraight, the turning lanes leave the field of view). Furthermore, the task is also highly ambiguous(once the intersection has been traversed via the straight lane, the distances to the centerline of theleft and right turning lanes are not well defined). Thus, requiring the network to predict all affordances in every situation would introduce noise during training and hence lower the performance ofthe system. Besides, it would also increase runtime as more signals must be predicted at any point
in time.

2

Ground Truth Acquisition

The API of CARLA supplies measurements about the agent (speed, acceleration, location, orientation) and about other objects in the scene (cars, pedestrians, traffic lights, and speed limit signs).
These measurements include the current status of the traffic light (green, orange, red) and the typeof the speed sign (30, 60, 90). Location and orientation are defined in a world coordinate system
cglobal = (xg , yg , zg )| . As these measurements do not directly express the affordances we want tolearn, we implemented a procedure to convert them into the desired ground truth.
2.1

Observation Area

We define a local coordinate system clocal = (xl , yl , zl )| at the center of the front axle of the carwith the x-axis corresponding to the car’s lateral axis and the z-axis corresponding to the up vector.
The agent’s orientation ψ and the agent’s position (xego , yego ) is supplied in cglobal . Using thisinformation, we convert the position of all other objects to clocal . Next, we define the observation
areas as rectangles in the x-y plane of clocal , see Figure 3 of the main paper for an illustration. If
an object falls into an observation area it is considered “detected”.
13

Table 6: Comparison of Temporal and Non-temporal Task Blocks. The last column shows therelative change in performance. The higher the IoU and the lower the MAE the better.
Best performing task blocks
Task

Metricnon-temporal

temporalrelative Change

Hazard stop
Speed sign
Red light
Relative angle
Center distance
Vehicle distance

IoU
IoU
IoU
MAE
MAE
MAE

84.96 %
91.95 %
92.41 %
0.00797
0.096420.04497

87.41 %
92.71 %
93.95 %
0.002070.08465
0.03289+ 2.88 %+ 0.83 %+ 1.67 %- 74.03 %- 12.21 %- 26.86 %The length, width, position, and orientation of the observation areas are chosen based on the respective affordances. Thus, the observation area for red light and for speed sign is located on theright side of the agent as their respective signals are located on the right sidewalk. The observation
area for hazard stop is directly in front of the car and very short, in order to only initiate a hazard
stop if an accident could not be avoided otherwise. The observation area for vehicle distance isin front of the car and has a length of 50 m. If another vehicle is located within this area, the distance of the closest vehicle to the agent is measured. If there is no car in front, the default value forvehicle distance (50 m) is used. Table 5 lists the coordinates for each observation area.
2.2

Directional Input

CARLA provides a high-level topological planner based on the A* algorithm. It takes the agents
position and the coordinates of the destination and calculates a list of commands. This “plan” advises
the agent to turn left, right or to keep straight at intersections.

3

Additional Experiments

In this section, we provide additional experiments. First, we compare temporal to non-temporaltask blocks to assess the influence of the additional temporal information provided by video data.
Second, we provide a qualitative evaluation of our agent’s driving behavior.
3.1

Comparison of Temporal and Non-temporal Task Blocks

Table 6 shows the best performing task blocks for each task. By using a temporal task block, all
the classification and regression results improve. This demonstrates that each task profits from theadditional temporal information.
The biggest relative improvement can be seen for the relative angle task block. The error of thetemporal task block is almost four times lower than the non-temporal task block. This suggeststhat this task profits more from the temporal context than other tasks. The smallest improvement isachieved for the speed sign task. To keep computation time low during training and inference, we
therefore use the non-temporal task block for this task in our final model.
In addition, we empirically observed that there is no dominating temporal layer in terms of performance. LSTMs, GRUs and temporal convolution layers perform very similar.
3.2

Driving Behaviour

The experiments in the main text examined whether the goal was reached and if there were any ruleviolations. This section focuses on qualitative driving experience, i.e., how the driving would beperceived by a passenger. The evaluation is done for the task “navigation without dynamic objects”
to evaluate the general driving behavior without distorting the results by the challenges of stopping
for cars or pedestrians.
14

Table 7: Driving Behaviour. Qualitative evaluation of the general driving performance, the lower ametric the better.
Metrics

Unit

CIL

RL

CAL

Distance to centerline
Longitudinal jerk
Lateral jerk driving straight
Lateral jerk driving turns[m][m/s3 ]
[m/s3 ]
[m/s3 ]

0.390
0.449
0.084
0.242

0.755
1.368
0.336
0.548

0.334
0.333
0.052
0.065

We use the following metrics for evaluation:• Centerline distance: Staying in the middle of the road is the main task of every lane
keeping system. We evaluate the ability of the algorithm to minimize the deviation fromthe centerline. The reported result is the median over all episodes. The median is more
descriptive for the qualitative driving experience than the mean value since failed episodes
during which an agent drifts off the road produce large outliers.
• Jerk is the rate of change of acceleration. The jerk can be felt during driving in the formof a jolt or sudden shock. It is commonly used to quantify ride comfort [3]. A smootherride results in lower fuel consumption increased passenger comfort and more safety. A way
to quantify jerk is to compare the root mean square (RMS) jerk [2]. The analysis further
distinguishes between longitudinal and lateral jerk with lateral jerk separately evaluated forstraight roads and in turns.
Table 7 reports our results. In contrast to the previous evaluations, the results are not reported
depending on the weathers conditions or the environments. The results are very similar under all
conditions for all agents. The CAL agent is able to achieve the best performance on all four metrics.
Distance to Centerline
All agents perform on a similar level and are able to keep the distance to the centerline low. The
exception is the RL approach. When driving on a straight road, the RL agent regularly starts swaying
from side to side over the entire lane, resulting in the high value.
Longitudinal Jerk
The CAL agent performs best, followed by CIL and RL. The control parameters of the CAL agent
are freely adjustable which allows to accelerate and decelerate smoothly as well as driving at aconstant pace. The RL agent is only able to set the throttle value to either 0 or 1. This results in asudden jerk every time the agent utilizes the throttle.
Lateral Jerk while Driving Straight
On straight roads, both the CAL and the CIL agent perform similarly. When driving straight, theRL agent often outputs a steering value of 0. This leads to the agent drifting off the road. Whencorrecting against the drift, the RL agent steers abruptly, resulting in large jerk values.
Lateral Jerk while Turning
The CAL agent performs exceptionally well. There are several reasons for this. First, the agent isslowing down when approaching a turn. Second, our agent turns smoothly without abrupt changes insteering. Third, the jerk peaks are generally lower than for the other approaches. Despite this good
performance, the transition from turns to straight roads leaves room for improvement. Changes inthe directional switch result in sudden jumps in the prediction of the relative angle in some cases,
resulting in slight short-timed jerk. The CIL agent is not as good as the CAL agent, but it is generally able to drive through turns smoothly. RL, in contrast, conducts strong and abrupt steering
movements, resulting in a higher jerk value.

15

References[1] Kiam Heong Ang, Gregory Chong, and Yun Li. Pid control system analysis, design, andtechnology. IEEE transactions on control systems technology, 13(4):559–576, 2005.
[2] D Hrovat and M Hubbard. Optimum vehicle suspensions minimizing rms rattlespace,sprung-mass acceleration and jerk. Journal of Dynamic Systems, Measurement, and Control,
103(3):228–236, 1981.
[3] Quanan Huang and Huiyi Wang. Fundamental study of jerk: evaluation of shift quality andride comfort. Technical report, SAE Technical Paper, 2004.
[4] John G Ziegler and Nathaniel B Nichols. Optimum settings for automatic controllers. trans.
ASME, 64(11), 1942.

16