Planning-oriented
Autonomous Driving
Yihan Hu* Jiazhi Yang* Li Chen*+ Keyu Li*
Chonghao Sima Xizhou Zhu Siqi Chai Senyao Du Tianwei Lin Wenhai Wang

Poster: THU-AM-131

Lewei Lu Xiaosong Jia Qiang Liu Jifeng Dai Yu Qiao Hongyang Li+*equal contribuCon +project leadarXiv: h"ps://arxiv.org/abs/2212.10156

YihanJiazhi

LiShanghai AI Laboratory | 上海人工智能实验室Keyu

Hongyang

Shanghai AI Laboratory | 上海人工智能实验室Planning-oriented Autonomous Driving

Background and Mo-va-on

Shanghai AI Laboratory | 上海人工智能实验室Background - Autonomous Driving (AD) Systems
Boundingboxes

Percep&on

What are around?

Various weathers, illumina9ons,
and scenarios
Shanghai AI Laboratory | 上海人工智能实验室

Waypoints

Predic&on

How will they goin the future?

TrajectoryPlanning

Where should I go?

Background - Design Options for Autonomous Driving (AD) Systems(a) Standalone Models
Model 1

Task 1

Waypoints

PredicCon

TrajectoryPlanning…

…
Model N

Perception

Boundingboxes

Task N

Op9miza9on metric mAP●✅ ●Object Detec8on

Typical Industry solu&ons
Independent teams for module
developments

IsolatedOptimization
Objective

Op9miza9on metric minFDEMotion Prediction❌ ●Severe error accumulation
Op9miza9on target Safety and Comfort
Planning

Shanghai AI Laboratory | 上海人工智能实验室Background - Design Op>ons for Autonomous Driving (AD) Systems(b) Multi-task Framework
Head

Task 1

Task 2…
Head

Traffic Lights
Detection

Lane
PredicCon

FeaturesTask N●

Shared feature for mul&ple tasks✅ ●Easily extended to more tasks,
Compute-efficient❌ ●ObjectDetection

Lack of tasks’ coordina&on

Shanghai AI Laboratory | 上海人工智能实验室

EncoderRaw Input
credit toTesla AI Day 2021

Background - Design Options for Autonomous Driving (AD) Systems(c.1) End-to-end Framework - Vanilla Solutions

Hidden Units

Output Units

Planner

Direct policy learning from sensor inputs,
bypassing intermediate tasks✅ ●Simple design with good performance in thesimulator❌ ●Deﬁcient in interpretability

…

…

●

ALVINN, NeurIPS 1988. CMU

CNN

FC Layers

DAVE-2, arXiv 2016. Nvidia

Shanghai AI Laboratory | 上海人工智能实验室Background - Design Options for Autonomous Driving (AD) Systems(c.2) End-to-end Framework - Explicit / Interpretable DesignFeatures

Module 1●Module 2Planner

ObjectDetec8on

Better interpretability (e.g. Bird’s-eye-view, BEV)

❌ ●Lack some crucial components1
1.

The necessities of eachcomponent is mentioned inAppendix.

MotionPrediction

PnPNet, CVPR 2020. Uber

Introducing intermediate tasks to assist
planning✅ ●Tracking

BEV seg

Seman8c
Occupancy
Forecas8ng

Planning

P3, ECCV 2020. Uber
MP3, CVPR 2021.
Uber
BEV seg

Seman8c
Occupancy
Forecas8ng
Front-view features

Planning

Command

ST-P3, ECCV 2022. SH AI
Lab
Shanghai AI Laboratory | 上海人工智能实验室

trajectory

trajectory

Mo>va>on- Towards Reliable Planning
Ours: Planning-oriented Autonomous Driving
Task 1

Planner…

…

Model 1(c.1) Vanilla Solution
Task N

Model N(a) Standalone Models

Module 1

Module 2Planner(c.2) Explicit Design
HeadTask 1

Our solution: UniAD

Tracking
Task 2…
HeadMotion

Occupancy

Mapping(c.3) Planning-oriented Design (Ours)

Task N(b) Multi-task Framework

perception

prediction(c) End-to-end Autonomous Driving

What do we want:✅ ●Unify full-stack AD tasks✅ ●Coordinate all task towards safe planning

Shanghai AI Laboratory | 上海人工智能实验室

Plannerplanning

UniAD - Overview

Which tasks?

Shanghai AI Laboratory | 上海人工智能实验室

How to construct?

How to train?

Planning-oriented Autonomous Driving

Delving into Details

Shanghai AI Laboratory | 上海人工智能实验室

UniAD - Which Tasks?
Percep&on System●

Object Detection●●

Object Tracking
Online Mapping●…

Predic&on System
Track agents
behavior
Guide withmap (lane)●

Mo=on Forecas9ng●●

Occupancy Predic9on…

Planning
Interact withenvironment●…

Find free space

Incorporate all tasks ina hierarchical manner

Shanghai AI Laboratory | 上海人工智能实验室●

Five safety-critical tasks: Model the staticand dynamic information●Task hierarchy: Tasks are well-organized toop&mize informa&on ﬂow to the planner

UniAD - How to Construct?
●Pipeline

Track agentsacross 9me●Predict long-termtrajectory●Plan ﬁnal trajectory●Avoid collision

Ego-vehicle Query

TrackFormer
Track Q

MotionFormerK, V

B
Multi-view
Vision-only Input

Agent-level
FeatureK, V

OccFormer

K, V

Planner

Map Q
K, V

MapFormer

Occ Q
Motion Q

Perception

Backbone●

Shanghai AI Laboratory | 上海人工智能实验室

Segment map elements

Scene-levelFeature

BPrediction●BPlanning

Scene-level representa9on

UniAD - How to Construct?

Pipeline

Ego-vehicle Query

TrackFormer
Track Q

MotionFormerK, VB
Multi-view
Vision-only Input

Agent-level
Feature

K, V

K, V

OccFormer

Planner

Map Q
K, V

MapFormer

Occ Q
Motion Q

Backbone●●

Entire pipeline connected by queries
Tasks coordinated with queries●

InteracCons modeled by aDenCon
Shanghai AI Laboratory | 上海人工智能实验室

B

Prediction

Perception

Unified Query
Transformer-based

Scene-level
FeatureB

Planning

First 0me to unify
full-stack AD tasks!

UniAD - How to Construct?

Percep)on

Prediction

PlanningTrackFormer - MOTR (ECCV 2022)
T-1

TNew born

BEV feat

Track-i

Decoder
Det queries

Track-j●End-to-end trainable tracking
without post-associa&on●Each query represents amap element

Track-kIterative Update

MotionFormerTrack queries

MapFormer - Panoptic SegFormer (CVPR 2022)
BEV featDecoder
Thing queries( for lanes, dividers, …)
Stuff queries( for drivable areas)
Shanghai AI Laboratory | 上海人工智能实验室

Map
Segments

MotionFormerUniAD - How to Construct?

Percep)on

Prediction

Planning

Mo0onFormer (Proposed in UniAD)targetGTt+1
PredtGTt●

Diverse relation modelings via attentions:
Agent-agent, agent-map, agent-goal

Shanghai AI Laboratory | 上海人工智能实验室●

Non-linear opCmizaCon:
Adjust ground-truth trajectory
based on upstream predic&ons

UniAD - How to Construct?

Percep)on
Prediction

Prediction

Planning

OccFormer (Proposed in UniAD)●

Encode agent-wise knowledge into the scene representa&on●

Predict occupancy as aDenCon mask to restrict the interac&ons between
the agents and their corresponding BEV features.

Shanghai AI Laboratory | 上海人工智能实验室

UniAD - How to Construct?

Percep)onPredic)onPredic)on

Planning

Planner (Proposed in UniAD)

Ego-vehicle
queryk,v●Ego-vehicle query: consistently models the ego-vehicle●Collision opCmizaCon: Steer the predicted trajectories clear ofpredicted occupancy.

Shanghai AI Laboratory | 上海人工智能实验室The Recipe - How to Train?
Two-phase training. Perception stage + End-to-end stage-The stabilized perception capability helps the end-to-end stage converge faster
Stage 1:

BEV 🔥
Encoder🔥Perception

InitStage 2:

BEV
Encoder❄🔥Perception🔥Prediction

🔥Planning

Shared matching. Matching results of tracking reused in moHon and occupancy-Consistent learning of agent idenHHes-Converging faster

Shanghai AI Laboratory | 上海人工智能实验室Planning-oriented Autonomous DrivingExperiments

Shanghai AI Laboratory | 上海人工智能实验室

UniAD - Abla>on Results
Tasks beneﬁt🚀each other and contribute to safe planning

Conclusion:●●

ID. 4-6: Track & Map → Mo9on🚀
ID. 7-9: Mo9on🚀 ↔ Occupancy🚀

●ID. 10-12: Motion & Occupancy → Planning🚀

Shanghai AI Laboratory | 上海人工智能实验室

UniAD - Results
Even outperforms LiDAR-based counterparts on planning

Planning†: LiDARbased
Camera-based

Shanghai AI Laboratory | 上海人工智能实验室

UniAD - Results
SOTA performance on all invesCgated tasks
Mul&-object Tracking

Motion Forecasting

Mapping

Occupancy Predic&on

Shanghai AI Laboratory | 上海人工智能实验室

UniAD - Visualiza>ons
Planner attends to crucial areas in complex scenes

Attention on
ForwardChange Lane!

A(en)on on
Backward

Shanghai AI Laboratory | 上海人工智能实验室

UniAD - Recover from Upstream Errors
Planner could sCll aDend to ‘undetected’ regions/objects
ObjectsinDistanceShanghai AI Laboratory | 上海人工智能实验室

Undetected

Still Attendedby TrackFormer

by Planner

One-page Summary●Planning-oriented Philosophy: An end-to-end autonomous driving (AD) framework inpursuit of safe planning, equipped with a wide span of AD tasks.

●Uniﬁed Query design: Queries as interfaces to connect and coordinate all tasks.

●

State-of-the-art (SOTA) Performance with vision-only input.

●First Step towards Autonomous Driving FoundaEon Models

Shanghai AI Laboratory | 上海人工智能实验室What’s next? beyond UniAD

Embracing Founda-on Models
for Autonomous Driving

Shanghai AI Laboratory | 上海人工智能实验室

UniAD v2
Data & Training Strategy-Multiple datasets with labels for various tasks?

Shippable Algorithm-More modules integration, extensible to applications (e.g. V2X)

Closed-loop System-Closed-loop training and tesHng in simulator & real world

Check out the latest Survey Paper!
hYps://github.com/OpenDriveLab/

End-to-end-Autonomous-Driving

Shanghai AI Laboratory | 上海人工智能实验室Beyond UniAD: DriveAGI
Data-centric Pipeline

Pre-training DriveCore

Applications

Data Collection

Autonomous Driving

Data GeneraCon

Broader Impact

Universal FoundaEon Modelfor autonomous driving
How to formulate?
What’s the objecCve goal?

ParHal photo by courtesy of online resources.
Shanghai AI Laboratory | 上海人工智能实验室Poster: THU-AM-131

THANKShttps://opendrivelab.com

Shanghai AI Laboratory | 上海人工智能实验室