Failure-Scenario Maker for Rule-based Agent using
Multi-agent Adversarial Reinforcement Learningand its Application to Autonomous Driving
Akifumi Wachi(IBM Research AI)

Autonomous driving era will arrive!
© Chombosan/Shutterstock.com

2

Safety-critical Systems• Traffic accidents in realenvironments may lead tocatastrophic and tragic results.

©RESPONSE• To guarantee reliability ofautonomous driving algorithms,
we should test them in simulators
before deployment.

©IPG

How should we test autonomous driving
algorithms before deployment?
© NASA/JPL3

Definition of Failure
There are several types of failures.
Perception [1]Mechanical Failure© CAR NEXT© ASAHI

Stop Sign → Speed Limit Sign

BlowoutEngine Trouble

In this work, failure means collisions withother cars or objects.
[1] Eykholt, Kevin et al. "Robust physical-world attacks on deep learning©models."
NASA/JPL
arXiv preprint arXiv:1707.08945 (2017).

4

How Should We Test?
How about thereliability of thesoftware?

• Simplest way is to test (almost) all possible cases.
→ Computational cost is enormous.
• Alternatively, finding failure-scenarios is aneffective and efficient approach.
© NASA/JPL5

6

Training ofAstronauts

Green card
Astronaut behaves inadversarial way such that
another astronaut fails.

© NASA

Green Card in Astronaut Training
TraineeGreen Card

Pretend tobe sick© NASA

Astronaut 1© And-Plus

Astronaut 2 (Buzz Aldrin)
is now tested.

Adversarialaction© NASA© NASA/JPL
Astronaut27

Key Idea: Adversarial TestingAdversarial car (NPC)

Tested car(player)

Adversarial car (NPC)

Adversarial cars (non-player characters, NPCs)
try to make tested car (player) fail.
© NASA/JPL8

Key Ideas: Adversarial RLAdversarial cars are trained
to make tested car fail using
multi-agent reinforcement
learning (MARL).

PlayerAdversarialNPCs• Why RL?
1. Humans donʼt have tospecify details of NPCsʼ
behaviors.

Reduction of human cost

2. NPCs make player fail in
different way fromhumans.

High coverage whencombined with humandependent approaches© NASA/JPL9

Difficulties of Adversarial Testing
What happens if we simply
train NPCs to make player fail?
NPCs try everything toattack (hit) the player.

Our ultimate goal is to improve tested algorithms.
• We need natural failure-scenarios.
• Unnatural failure scenarios are useless for
improving the algorithm of the player.
© NASA/JPL

10

Natural Failure Scenarios
To obtain natural failure-scenarios, we consider twotypes of reward function.

Personal reward

Adversarial reward

Reward that characterizes
NPCsʼ own objectives

Reward that is given toNPCs when player fails

Personal reward is defined to discourage NPCs from:• Violating traffic rules unrealistically• Getting damaged by hitting other cars or objects• etc.

© NASA/JPL

11

Overall Structure
Execution

Training

EnvironmentAction

ActionPersonal
ActionrewardPlayer

NPC 1𝑜&, 𝑎&𝑜), 𝑎)⋯

Personal
rewardDid player
succeed?

NPC NNo𝑜* , 𝑎*Adversarial
reward𝑸𝟏𝑸𝑵© NASA/JPL12

Simulation Result (AirSim)
NPC is passing player in left lane, causing player tocollide with rock.

© NASA/JPL

13

Conclusion
1. Proposed framework for testing autonomous
driving algorithms using multi-agent adversarial
reinforcement learning.
2. Proposed mechanism for obtaining natural
failure-scenario that is useful for improving
tested algorithms.
3. Demonstrated effectiveness of our proposed
method in numerical simulations.

© NASA/JPL

14

Future Work
1. Apply our method to more sophisticated tested
algorithms and more realistic environments that
include pedestrians or traffic signs.
2. Create integrated adversarial situations while
incorporating perception capabilities.

© NASA/JPL

15

Thank you!

16