Poster: Security of Deep Learning based Lane
Keeping Assistance System under Physical-World
Adversarial Attack
Takami Sato⇤ , Junjie Shen⇤ , Ningfei Wang, Yunhan Jack Jia† , Xue Lin‡ , and Qi Alfred Chen
Univesity of California, Irvine; † ByteDance AI Lab; ‡ Northeastern University

Abstract—Lane-Keeping Assistance System (LKAS) is convenient and widely available today, but also extremely security andsafety critical. In this work, we design and implement the firstsystematic approach to attack real-world DNN-based LKASes.
We evaluate our approach on a state-of-the-art LKAS and ourpreliminary results show that our attack can successfully cause
it to drive off lane boundaries within as short as 1.3 seconds.

I.

I NTRODUCTION

Lane-Keeping Assistance System (LKAS) is an Level2 driving automation technology that automatically steers avehicle to keep it within the current traffic lane. Due to its highconvenience for human drivers, today it is widely available
in a variety of vehicle models such as Honda Civic, Toyota
Prius, Nissan Cima, Volvo XC90, Mercedes-Benz C-class,
Audi A4, and Tesla Model S. While convenient, such function
is extremely security and safety critical: When LKAS starts
to make wrong steering decisions, an average driver reaction
time of 2.3 secondsmay not be enough to prevent the vehicle
from colliding into vehicles in adjacent lanes or in opposite
directions, or driving off road to hit road curbs or fall down
the highway cliff. Even with collision avoidance systems, itcannot prevent the vehicle from hitting the curb, falling down
the highway cliff, or being hit by other vehicles that fail toyield. Thus, it is urgent and highly necessary to understandthe security property of LKAS.
To achieve lane keeping, the most critical step in an LKAS
is lane detection, which by default uses camera due to thenature of lane lines. So far, Deep Neural Network (DNN)
based detection achieve the state-of-the-art accuracy and isadopted in the most performant LKASes today such as Tesla
Autopilot and OpenPilot [1]. Thus, the end-to-end security of
the latest LKAS technology highly depends on the securityof such DNN models. While recent works show that DNNmodels are vulnerable to carefully crafted input perturbations,
their methods cannot be directly applied to attack DNN-based
LKASes due to 3 unique challenges. First, prior methods aremostly designed for classification or object detection, and noneof their attack formulations can be directly applied for lanedetection. Second, to affect the camera input of a LKAS, theperturbations need to be realizable in the physical world andcan normally appear on traffic lane regions. Moreover, suchperturbations must not affect the original human-perceived lane
information for stealthiness. Prior works have explored suchthreats for traffic signs [2], but not for traffic lanes.
Third, to cause end-to-end impact to a LKAS, the attack
needs to affect a sufficient number of consecutive camera⇤ The first two authors contributed equally.

frames, and most importantly, the attacks on later frames aredependent on those on earlier frames. For example, if the attack
successfully deviates the detected lane to the right in a frame,
the LKAS will control the vehicle heading accordingly, whichcauses the following frames to capture road areas more to theright and thus directly affect their attack generation. To thebest of our knowledge, no prior work considers attacking asequence of image frames with such strong inter dependencies.
The only prior effort that successfully attacked an LKAS
is from Tencent [3], where they fooled the Tesla DNN-based
LKAS to follow fake lane lines created by a line of big white
dots on road regions originally without lane lines. However, itis neither attacking the scenarios where a LKAS is designed
for, i.e., roads with lane lines, nor generating the perturbationssystematically by addressing all the three challenges above.
To fill this critical research gap, in this work we design andimplement the first systematic approach to attack real-world
DNN-based LKASes. To practically introduce perturbations,
we identify road patches as the threat model, which is specificto lane detection models and can normally appear in thephysical world. For stealthiness, we restrict the perturbations
to be within lane lines, and the color space to be on thegray scale to pretend to be a benign but dirty road patch.
We then formulate the malicious road patch generation as an
optimization problem, and design a multi-frame path bending
objective function specifically for the lane detection task.
To address the challenge from the inter-dependencies amongattacks on consecutive camera frames, we design a novel car
motion model based input generation process and a gradientaggregation technique. We evaluate our approach on a stateof-the-art LKAS, OpenPilot, and our preliminary results showthat our attack can successfully deviate a LKAS to drive off
the lane boundaries within as short as 1.3 seconds, which isfar shorter than 2.3 seconds, the average driver reaction time.
II.

T HREAT M ODEL AND P ROBLEM F ORMULATION

Threat model. We assume that the attacker can possess
the same LKAS as the one in the victim vehicles and has thefull knowledge of the LKAS via reverse engineering. Before
attacking, the attacker can also collect camera frames on thetarget road by driving her own vehicle with the LKAS.
Realizable and stealthy physical-world perturbation
design. We identify malicious road patches as the attack vector,
since they are realizable in the physical world and can normallyappear around traffic lanes. For stealthiness, we restrict theseroad patches to not cover the original lane lines and their color
to be on the gray scale to pretend to be benign but dirty.

Original camera input

Apply
car motion based
input generation
Place
t=1
patch

Generated camera input
Rotated 5° to left
Shifted 1m to rightObtaingradientsof patch area

Convertgradients
to BEV

Gradient
Aggregation

Update patch−λt=2Fig. 1. Car motion model based camera input generation from the original
camera input. The red rectangle denotes the model input area.

Currentpatch
t=3

Attack goal. By attacking the LKAS, we aim to cause thevictim car to have a lateral deviation large enough to driveout of the current lane boundaries within the common driver
reaction time, which thus fundamentally breaks the design goalof LKAS and can cause severe safety consequences. Assuming
the victim vehicle locates at the lane center before the attack,the required deviation is 0.745 meters on the highway in theUSand the average driver reaction time is 2.3 seconds.
III.

Fig. 2.

Nextpatch

Overview of the optimization pipeline of the malicious road patch.

TABLE I.
ATTACK EFFECTIVENESS WHEN THE DEVIATION GOAL IS
0.745 METERS ( DRIVING OFF LANE BOUNDARIES ON THE HIGHWAY ).
Scenario

Avg. Speed

Attack TimePatch Size (W ⇥ L)

comma2k19-1
comma2k19-2
LGSVL-1

126 km/h (78 mph)
105 km/h (65 mph)
72 km/h (45 mph)

0.9 s
1.0 s
1.3 s

3.6 m ⇥ 36 m
3.6 m ⇥ 36 m
3.6 m ⇥ 36 m

ATTACK M ETHODOLOGY

Malicious road patch

With the problem formulation above, we design the following novel techniques to address the challenges in §I.

Drive out of lane boundaries after 0.9 sec

Car motion model based input generation. To consider
the inter-dependencies among attacks on consecutive camera
frames, we need to dynamically update camera inputs according to the driving trajectory changes during the patchgeneration. To address this, we use a bicycle model to simulate
the changes to car trajectory, which is then used to update
camera inputs by applying perspective transformations to theoriginal non-attacked camera inputs. Fig. 1 shows an exampleof this generation process. On the bird’s eye view (BEV), weapply a car position shift and heading angle change from theoriginal car trajectory and then project the BEV image backto the camera perspective. Although it causes some distortionand partial missing area, the model input area, which locatesat the center, is still complete and usable.

Fig. 3.

all camera inputs to BEV to align gradients to the same scale
and take an weighted average as shown in Fig. 2.
IV.

E ARLY R ESULTS

We evaluate our method on a state-of-the-art open-source
LKAS, OpenPilot [1], which is reported to have similar
performance as Tesla Autopilot and GM Super Cruise, andbetter than all other manufacturers.We evaluate our method
on 3 scenarios and the results are summarized in Table I.
The comma2k19-1 and comma2k19-2 are real-world highway
scenarios selected from the comma2k19 dataset. The LGSVL1 is a simulated highway scenario created by LGSVL, anindustry-grade photo-realistic Autonomous Driving simulator.
As shown, our attack succeeds to cause the victim vehicle todrive out of the highway lane boundaries (over 0.745 metersdeviations) within 1.3 seconds, which is much smaller than
the average driver reaction time (2.3 seconds). Fig. 3 shows
an example malicious road patch generated by our method.

Multi-frame path bending objective function. To generate the malicious road patch, we adopt an optimizationbased method, which has shown both high efficiency andeffectiveness in previous works. Since the lateral controllerof the LKAS is not differentiable, we introduce a surrogate
objective function to deviate the car as much as possible.
The lateral controller calculates a desired driving path basedon the detected lane lines, and numerically solves a steering
angle plan to enforce this path. The desired driving pathis typically represented by a polynomial function. Assuming
the car strictly follows the path, the derivatives of the pathare essentially the wheel angles it needs to apply. Thus, weformulate our objective function:
TXXV.

C ONCLUSION AND F UTURE P LANS

In this work, we design and implement the first systematic approach to attack real-world DNN-based LKASes. We
evaluate our approach on a state-of-the-art LKAS and ourpreliminary results show that our attack can successfully cause
it to drive off lane boundaries within as short as 1.3 seconds. In
the future, we plan to (1) perform more comprehensive evaluation including more diverse scenarios, different car types,
other DNN lane detection models, (2) demonstrate the attack
in real-world experiments, and (3) design effective defenses.

2

rpt (d; {Xj |j  t}, s0 ) + ||⌦t (Xt )||2 (1)f (X1 , ..., XT , s0 ) =Malicious road patch and car trajectory for comma2k19-1.

t=1 d2D, where pt (x) is the desired driving path in the t-th frame,
Xt is the t-th generated camera inputs including the malicious
road patch, s0 is the initial state and D is the set of thepoints where the controller makes steering angle decisions,is a weight of the L2 regularization, and ⌦ is a function thatextracts the patch area in a camera input.

R EFERENCES[1] “OpenPilot: Open Source Driving Agent,” https://github.com/commaai/
openpilot, 2018.
[2] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao,A. Prakash, T. Kohno, and D. Song, “Robust Physical-World Attacks
on Deep Learning Visual Classification,” in CVPR, 2018.
[3] “ExperimentalSecurity
ResearchofTeslaAutopilot,”
https://keenlab.tencent.com/en/whitepapers/Experimental Security
Research of Tesla Autopilot.pdf, 2019.

Gradient aggregation. Based on the objective function,
we obtain the gradients of each camera input. However,
the gradient descent is not directly applicable to update themalicious road patch since the patch sizes and portions aredifferent in each camera input. To address this, we transform2

Security of Deep Learning
based Lane Keeping
Assistance System under
Physical-World
Adversarial Attack
PRESENTERS
Takami SatoJunjie Shentakamis@uci.edu

junjies1@uci.edu

RESEARCH PROBLEM & MOTIBATION• Lane-Keeping Assistance System (LKAS)
Ø Automatically steers vehicle to keep it in laneØ Level-2 driving automation
Ø Widely available in a variety of vehicle models• Honda Civic, Toyota Prius, Nissan Cima,
Volvo XC90, Audi A4, and Tesla Model S• When LKAS fails to keep in lane:Ø Avg. driver reaction time 2.3 s not enough for• Drive off-road to hit road curbs orfall down highway cliff• Crash into other cars or be crashed into
• Our work:• First systematic approach to LKAS
• Target most performant design today:
DNN-based LKASATTACK METHODOLOGYOur attack can cause vehicles
running state-of-the-art LKAS toEVALUATION SETUP

drive off highway lane after 0.9 sec.

ATTACK GOAL
• Drive out of current lane boundary within
2.3 seconds (avg. driver reaction time)• The required deviation is 0.745 m on highway
DESGIN CHALLENGES• Most prior attack targets obj. detection, not LKAS
• Need to be realizable & stealthy in physical world• Attack to consecutive frames are inter-dependent
REALIZABLE & STEALTHY PHYSICAL-WORLDPERTURBATION DESIGN• New attack vector: malicious dirty road patchØ Realizable in the physical world
Ø Can normally appear around traffic lane
Ø Not cover the original lane lines
Ø Only use gray-scale color to pretend to bebenign but dirty• SCENARIOSØ Comma2k19 dataset: real-world highway driving
Ø LGSVL: industry-grade driving simulator• TARGET LKAS: OpenPilot
Ø Open source, on par w/ Tesla & GM Super Cruise

Malicious road patch pretending to be benign but dirty

Camerainput

Drive out of lane boundaries after 0.9 sec

Detectedlane lines

Generated camera input
Rotated 5°to left

Shifted 1m to right

Eval.
Scenario

Avg.
Speed

126 km/h(78 mph)105 km/h
comma2k19-2(65 mph)72 km/hLGSVL-1(45 mph)comma2k19-1

Multi-frame Path Bending Objective Function

Desiredsteering angleAttackSuccessTimePatch Size

0.9 s

3.6m × 36m

1.0 s

3.6m × 36m

1.3s

3.6m × 36m

FUTURE PLANS• More comprehensive evaluation• Real-world experiment• Design effective defenses

Gradient Aggregation
Apply
car motion based
input generation
Placepatch

Lateral MPC
controller

EARLY RESULTS• Success criteria: car deviates 0.745 m within 2.3 s

Car Motion Model based Input Generation
Original camera input

DNN
RNN input

THREAT MODEL• Attacker possesses the same LKAS as victim• Has full knowledge of the LKAS
• Can collect road images before attack• Car Motion Model based Input Generation
Ø Simulate inter-dependency by bicycle model• Multi-frame Path Bending Objective Function
Ø Design attack as optimization problem
considering inter-dependency• Gradient Aggregation
Ø Update malicious road path while keeping
inter-dependency , realizability, and stealthiness

t=1

Obtaingradientsof patch area

Convertgradients
to BEV

Gradient
Aggregation

Update patch

Takami Sato*, Junjie Shen*, Ningfei Wang,
Yunhan Jack Jia, Xue Lin, Qi Alfred Chen
*Contributed equally−λt=2Currentpatch
t=3

NextpatchTake a picture to
download theposter abstract