公益財団法人国際高等研究所 エジソンの会 2017年7月24日

自動運転における
センシングとディジタル地図との周辺技術

東京大学 情報学環
上條俊介

Outline
1. Introduction
2. Localization and mapping with Active Sensor
3. Localization with Passive Sensors
4. Pedestrian Detection and Behavior Recognition
5. Platooning on Highway
6. Policy and Definition of Automated Vehicles
7. Research Introductions
3D-GNSS and self-localization of vehicles
P2V application for pedestrian safety
Pedestrian detection and behavior understanding by Onboard
camerasAutonomous Driving
Fig. 2: Junior, the vehicle of Stanford group in
the DARPA Urban Challenge 2007 .

Fig. 3: Merging into
dense traffic during
the qualification
events at the Urban
Challenge. (a)Photo of merging
test; (b)-(c) The
merging process.

Fig. 1: Google car

Environment perception
Localization

Map

Motion planning
Fig. 4 Basic requirements for autonomous driving
Montemerlo, Michael, Jan Becker, Suhrid Bhat, Hendrik Dahlkamp, Dmitri Dolgov, Scott Ettinger, Dirk Haehnel et al. "Junior: The stanford entry in the urban
challenge." Journal of field Robotics 25, no. 9 (2008): 569-597.

Urban Scenario of Autonomous Driving• Lane-level localization is required for autonomous
driving.
• Lane-level map is needed in self-localization andmotion planning.
• Active sensor:• LIDAR• Velodyne• Passive sensors:• GNSS• Inertial sensor• CameraWhat is Autonomous and Automated Driving• Self-Localization• Obstacle Detection• Path Planning• Autonomous or Automated ?

Stereo odometry
Translation and rotation matrix

A single feature x viewed with a stereo camera from two
different poses in consecutive image pairs.

Given x position inframe 0 and frame 1,
want therelationship(translation androtation) between
frame 0 and frame 1

3D-Map Construction from multiple data

Visualization of the scanning process: the LIDAR scanner acquires
range data and infrared ground reflectivity. The resulting maps
therefore are 3-D infrared images of the ground reflectivity. Noticethat lane markings have much higher reflectivity than pavement.

Ghost based by GPS in
multiple data

After registration• Two problems• dynamic object• multiple data fusion
Hole cased bydynamic object

Remove hole using
data fusion

After registration

J. Levinson, M. Montemerlo, S.Thrun,“ Map-based precision vehicle localization in urban environments,” Robotics: Science and Systems.
vol. 4, p. 1, June 2007.

Examples of point cloud map

Stanford parking garage
Toyota Technological Institute

University of Freiburg

Point cloud data to 3D map• Data Sources
Airborne Laser Scanning• Building segmentation

Example of roof patch segmentation with height jumps. (a) ALS building points; (b)detected roof patches shown in different colors; (c) roof patches at different heights.

• Extract the outline 3D building modelZhu, L., Lehtomäki, M., Hyyppä, J., Puttonen, E., Krooks, A., & Hyyppä, H. (2015). Automated 3D Scene Reconstruction from Open
Geospatial Data Sources: Airborne Laser Scanning and a 2D Topographic Database. Remote Sensing, 7(6), 6710-6740.

Building model height evaluation
Height differences between the ALS building roof points
and generated 3D roof models in 15 test locations

Root Mean Square Error(Up) Generated Building models, with roofs in red andwalls in gray. (down) Original ALS building roof
points in green and building models in gray.

• How about the accuracyof the boundary?

Zhu, L., Lehtomäki, M., Hyyppä, J., Puttonen, E., Krooks, A., & Hyyppä, H. (2015). Automated 3D Scene Reconstruction from Open
Geospatial Data Sources: Airborne Laser Scanning and a 2D Topographic Database. Remote Sensing, 7(6), 6710-6740.

• Various multipath mitigating receivers• Narrow-correlator (Dierendonck, 1992)• Strobe-correlator (Garin, 1997)

correlator output

Localization with GNSS

M.S. Braasch. “Performance comparison of multipath
mitigating receiver architectures”. In Aerospace Conference,
IEEE Proceedings., 2001.

Narrow-correlator
Early
late
ordinal correlator

-T

0

T

• Receiver Autonomous Integrity Monitoring algorithm (RAIM)• Check the residual of the least square and identifies the suspicious
satellites.
• Choose satellites in calculation to make the least square residual small
RG Brown., "GPS RAIM: Calculation of Thresholds and Protection Radius Using Chi-square
Methods; a Geometric Approach.", 1994.

•

NLOS detection by 3D map(M. Obst, et al., 2012.)

Pedestrian detection example

training

Haar-like features based
detection

Test

Verify each candidate byhigh dimensional CoHOG
descriptors• HOG and Haar-like features are
popular and basic.
• CoHOG has good performance at
low resolution

Monteiro, G., Peixoto, P., & Nunes, U. (2006). Vision-based pedestrian detection using Haar-like features. Robotica, 24, 4650.

Convolutional Neural Network (CNN)• How a deep neural network sees
• From local texture to global structure

Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2011). Unsupervised learning of hierarchicalrepresentations with convolutional deep belief networks. Communications of the ACM, 54(10), 95-103.

Application of CNN

The YOLO (You Only Look Once ) Detection System. Processing images with
YOLO is simple and straightforward. The system (1) resizes the input image to F × F,(2) runs a single convolutional network on the image, and (3) thresholds the resulting
detections by the model’s confidence.

The Architectureof YOLO

Detection framework of YOLO
It divides the image into an even grid and simultaneously predicts boundingboxes, confidence in those boxes, and class probabilities.
These predictions are encoded as an S × S × ( B ∗ 5 + C ) tensor.
Bounding boxes + confidences x s grid on input

Position of box (x, y)Size of box (w, h)Confidence: conf
Probability of each class: p1… pc

Final detections

Class probability map

AutomatedplatoonAutomated platoon is the string operation coupled electrically with closed
gap distance in order to reduce fuel consumption.

Reduction of CO2

Connection by V2V communication

20%

Cp Contours

40%

60%

80%

100%

Last truck

Gap distance:4m

Mid truck
Head truck

Relative Reduction rate of air drag(%)

Level of Automated Vehicles : NHTSA’s Policy
No-Automation (Level 0): The driver is in complete and sole control of the primary vehicle
controls – brake, steering, throttle, and motive power – at all times.
Function-specific Automation (Level 1): Automation at this level involves one or more
specific control functions. Examples include electronic stability control or pre-charged
brakes, where the vehicle automatically assists with braking to enable the driver to regain
control of the vehicle or stop faster than possible by acting alone.
Combined Function Automation (Level 2): This level involves automation of at least two
primary control functions designed to work in unison to relieve the driver of control of
those functions. An example of combined functions enabling a Level 2 system is adaptive
cruise control in combination with lane centering.
Limited Self-Driving Automation (Level 3): Vehicles at this level of automation enable thedriver to cede full control of all safety-critical functions under certain traffic or
environmental conditions and in those conditions to rely heavily on the vehicle to monitor
for changes in those conditions requiring transition back to driver control. The driver is
expected to be available for occasional control, but with sufficiently comfortabletransition time. The Google car is an example of limited self-driving automation.
Full Self-Driving Automation (Level 4): The vehicle is designed to perform all safetycritical driving functions and monitor roadway conditions for an entire trip. Such a design
anticipates that the driver will provide destination or navigation input, but is not expected
to be available for control at any time during the trip. This includes both occupied andunoccupied vehicles
Preliminary Statement of Policy Concerning Automated Vehicles (NHTSA)Mobile Mapping Challenge in Urban Area
 Inaccurate MMS positioning in urban area with tall buildings
 Reason: GNSS error due to Non-Line-of-Sight and Multi-Path Conventional Solution: Landmark updating method
1.

Measure 3D Coordinate of Ground Control Points (GCP)

2.

Extract GCPs from original data3.

Calculate Position Correction Vectors (PCV)

4.

Correct MMS trajectory and point cloud based on PCVs

 More than

40 GCPs are required for a 500 x 300m area

MMS error before correction

Landmark updating

Multi-path

NLOS

Labor intensive
Time consuming
CostlyMMS data after correctionMap Reconstruction FrameworkPrecision Mobile Mapping (Problem statement)
Registering MMS Data to Aerial image based on Road markings

Aerial image

Extracted road markings from both MMS scan and aerial image

MMS ScanPrecision Mobile Mapping (Results and Evaluation)

Hitotsubashi intersection, Tokyo
Airborne road markings
Original MMS road markings
Proposed method

3D point clouddata
Airborne
pointcloud

3D Building Map Reconstruction

2D map

Building segmentation

Roof segmentation
Roof segment points

Boundary points extraction
Roof edges line creation
Roof polygon generation
Roof boundary

3D building reconstruction

Reconstructed 3D building model using proposed method

3D building map of city

Height ofeachroof segment
Roofheights

Roof boundary generationfor each roof segment

Generating Multilayer Vector Map

Points that made a vector form anormal distribution

Generated Normal distribution form vector map

Points that made a vector segment
Mean of generated normal distribution
Covariance of generated normal distribution

Experimental results

Evaluation of data size

Evaluation of vector ND map-matching

Route of experiments
Velodyne’s VLP-16(16 chennel)

Our experimental vehicle
Evaluation of multilayer 2D vector structure (comparison with conventional 2D methods)

都市部における測位精度向上の課題設定• 歩行者の測位精度の向上
 5m程度の測位誤差を実現すれば、道路や交差点のどの側にいるか、どの店舗の前にいるかを十分に判定できる。即ち、1m以内の測位誤差は不要。
 外国人等の土地勘がない旅行者へのナビゲーションの利便性、商業地区での歩行場所に依存した広告等のロケーションサービス。
 商業地区の都市計画のための詳細な行動データを取得できる。
 従来の測位誤差は、20m程度以上のため、道路のどちら側の歩道を歩いているかの判定が困難、交差点のどの角にいるかの判定が困難。
 WiFiアクセスポイントからの受信強度(RSSI)計測を用いた測位は、GPS測位データ
をレファレンスとしたキャリブレーションが元となっているため、 GPS測位誤差と同等の誤差を有する。
 本技術を活用することで、 WiFiアクセスポイントを参照した測位の精度も向上できる。

• 自動車のポジショニング精度の向上
 信号制御の高度化への期待。
 車線を判定することで、右折需要と直進需要を分離して計測できる。
 自動運転においても車線判定は必須技術。
 車線判別のための測位への要求精度は1.5m（車線幅の半分程度の相当）であるが、車載のCANデータ、ジャイロデータとのヒュージョンで達成できる。

Algorithm to apply Ray-tracing to 3D map
Reflected
path

Signal Observation:- Pseudorange- RSSI (Received Signal Strength
Indicator)- Deceived positioning results

Directpath

Find
the most consistent

Our
methodGroundtruth

GPSresultPosition Assumptions:
Estimated by ray tracing- Pseudorange- RSSI (Received Signal Strength
Indicator)- Deceived positioning resuts

Evaluations with solutions to NLOS and Multipath problem
iPhone4S with WiFi
u-blox NEO-6P
Proposed (with NEO-6P)
Ground Truth
Mean
Error [m]Standard
deviation[m]

u-blox NEO-6P

19.8

14.2

3D map for
NLOS

5.7

4.3

4.7

3.0

3D map for
NLOS / Multipath

Experimental results: Vehicle sensor integration
Standard GPS (single point positioning) fusion

First experimentFirst experiment
Second experiment
Third experimentStandard GPS fusion
3D map GPS fusion
Standard GPS fusion
3D map GPS fusion
Standard GPS fusion
3D map GPS fusion

Second experiment

Maximum error
(m)
16.2
3.9
26.7
5.4
15.4
6.6

3D map GPS fusion

Third experiment

Mean error
(m)
11.2
1.0
13.7
2.2
8.8
1.7

Standard Deviation
(m)
2.5
0.7
3.5
1.4
3.4
1.4

Zigbee for V2V and P2V applications.
Zigbee equipment
Road Users
With a Transmitter:Zigbee device
SignalsBicycles and pedestrians
equip a single Zigbee
transmitter.
Observation vehicles
equip four Zigbee
receivers (and also atransmitter Zigbee).

An Observation Vehicle
With 4Receivers

Results of FOT at a Real Intersection(1)Right Turning Cases

Path Prediction - Problems
Path prediction and motion classification focusing on physical states resultin short-time prediction (~1s).
1. Trajectory-based Approach
Linear Dynamical System approximation of positional movement.
Motion-specific LDSs (walk, stop, turn, etc.).
Nicolas Schneider and Dariu M. Gavrila., "Pedestrian path prediction with recursive bayesian
filters: A comparative study," Pattern Recognition. Springer Berlin Heidelberg, 2013.

2. Image-based Approach• Visual features on image plane and non-linear/high-order Markov models.
Christoph G. Keller, and Dariu M. Gavrila., “Will the Pedestrian Cross? A Study on Pedestrian Path
Prediction,” Transactions on ITS 2014.

3. Pose-based Approach• Body parts and joints in 3D space which are robust against
sensor ego-motion and change of the observing direction.
R. Quintero, et al., “Pedestrian Path Prediction using Body Language Traits” IV 2014.

For long-time prediction, it is crucial to consider contexts which determinepedestrian movement: traffic rule, situation and pedestrian intention.

Related Work on Context-based Pedestrian
Behavior Recognition
Julian Francisco Pieter Kooij, Dariu M. Gavrila, et al.,“Context-Based Pedestrian Path Prediction” ECCV2014.

Sees Vehicle
Has-Seen-Vehicle
Boolean latentcontext variables Z

At-Curb

Situation-Critical
Motion typeState
Observations E

Distance between Ped. and Veh.

Distance-To-Curb
Head-Orientation Lateral Position

Three factors influencing pedestrian’s decision to stop :P-State
Observationt
t-1
1.Pedestrian’s awareness of approaching vehicles.
Estimation
2.Distance to the approaching vehicle.
Pedestrian-Statet
3.Distance to the curb.
Prediction

Dynamical Bayesian Network based on empirical knowledge.
P-Statet+nHead and Body Orientation Detection• For more advanced active safety systems• Recognize the type of road user: cyclist or pedestrian• From an image of pedestrian and cyclist, estimate thepose/orientation
Head position and orientation
We can estimate the direction thatthe pedestrian is paying attention to
Body orientation
We can estimate the direction thatthe pedestrian is traveling toUnlabeled data in orientation• The problem is the difficulty in labeling the datanear the class boundary• Divide dataset into 2 clusters• Strongly labeled data
• Easy to label• Will be processed as `labeled` data

• Weakly labeled data
• Difficult to label• Will be processed as `unlabeled` data

Experimental result

Temporal constraint

Temporal constraint and model constraint

Related Work on Pedestrian Behavior
at Signalized Intersections
M. Iryo-Asano, et al. “Analysis and modeling of pedestrian crossing
behavior during the pedestrian flashing green interval.” Transactions on
ITS 2015.

Signal Phase in Japan
PG: Pedestrian Green
PFG: Pedestrian Flashing Green
PR: Pedestrian Red

Analysis of the probabilistic behavior of pedestrians after the onset of the PFG.
1. Pedestrians’ decision of whether to give up crossing (logistic regression)
2. Pedestrian speed distribution by areas (gamma regression)

Explanatory variables: distance to crosswalk, crosswalk length, approaching speed, …

according
to theof relative
positional
relation
betweenthe near-side edge
line segmen
servesas a reference
the coordinate
systemsince
Inofaddition,duZ:M
easurements
pedestrianthepedestrianandthecrosswalk.
positive/negative
value
before/aft
we assume a pedestrians determines his/her behavior
changetheirdecisicoordinate
systemasX.
according to the relative positional relation between
In addition,
during PFG,w
disappearance
of pa
Z the
: pedestrian
M easurements
pedestrian positions on the sameand theofcrosswalk.
change their decision at low pz

Proposed DBN

Z:

coordinate system as X .

M easurements of pedestrian positions on the samezx tcoordinate system as X . St-1
{Cross, Wait}

Zt =

zyt

decisions. Z =Signal {PG, PFG,
disappearanceofPR}
particlest (mentz
2) Motion Transdecisions.
St(6) space,to change
the moti
DecisionIn this state
we constructa2) Motion Transition Model:zprobabilities
oftomoZt = x t
Dt-1
D
the
ﬂow
of
pedestrian
behavior.
Fir
t (6)
to
changethemotiontype
anzyt
In this state space, we construct a model in accordancewiththecross/
wait
decis
the situations.
That corresponds
to si
probabilities
of motion change
thethisﬂow
pedestrian
behavior.
a pedestrianIn
stateofspace,
we constructa modelFirst,
in accordance
the cross/wait
decision-makingmtowiththe assesses
crosswalkin thismodel.
Acc
P(M
|
M
=
t
t
−
1
Motion{Run,
Walk,
Stand}
M
Mt distanceThat
corresponds
signal phaset-1
the theﬂowsituations.
of pedestrian
behavior.
First, atopedestrian
assesseshe/andshe
makes
decisionsofmthe
mov
P(M
|
M
=
,
=
t
t
−
1
t
−
1
=
(1
+St exthe crosswalkin this model.
the situations,the tosituations.
That corresponds
to signalAccording
phase and to
distance
tocross/wait decision and motion ,tydStatecorresponds = (1 + exp(− Vm (st (wto thethis model.ofAccording
to the situations,he/crosswalk
she makesin decisions
themovement.
That
V
V
t
Velocity
moves based on that. (where
The graphic
he/shemakeswait
decisionsof themovement.
Thatchange.
corresponds
Mt 6=
torectangle:
cross/
decisionandmotion
type
Then,
he/
she
discrete
Xt-1
Xtis shown
in consider
Fig. 1. F
We alsoto cross/
wait
decisionmotionchange. representation
Then, proposed
he/shePt ofDBN
ellipse:continuous
moves
based
onand
that.
Thetypegraphical
theP
PostionWe also consider
only
the dista
shaded:observed
wecandecomposethestatetransitio
moves
based
on
that.
The
graphical
representation
of
the
time
as
an
inﬂuenti
proposed DBN is shown in Fig. 1. From this representation,time
as an inﬂuential factor. How
proposed DBN is shown in Fig. 1. From this representation,
Observationdifferent constants
follows:
we candecomposethe state transition model of the DBN
as constants and coefﬁcie
different
we candecomposethe state
Zt
Zt-1 transition model of the DBN as
the other values.
condition
follows:theotherconditionalP(D t , M t , X t |D t − 1 , M t − 1 , X t −follows:
State Transition Model
Vt −m1)P(M(sV,m, mt ,t ,dxt t,t|−m1(st , td,t P
,
M
,
X
|D
,
M
,
X
,
S
)
=
P(D
|D
,
S
P(DP(D
,
M
,
X
|D
,
M
,
X
,
S
)
t
t
t
t
−
1
t
−
1
t
−
1
t
t
t
−
t
−
1
t
t
t
t− 1
t− 1
t− 1
t
= ,b0,s= b0,st,,dSt ,m, D+ )bt1,,t − 1 ,mt t ,d= P(D, P |M)P(M, S|M, D , P, S , )D , P ⇥) P(X |X
= P(D
|D |D
, S , P, S )P(M
M
t-1

t-1

t − 1tt− 1
tt −t1 tt− 1 t − 1tt − 1 t t t −t 1 t −t 1 t
Cross/Wait
Model
Transition Model⇥ P(X|X t − t1|X
,Decision
Stt −, D1 ,t ,SM) t , MMotion
(7)
tP(X
t ,tD
t)

⇥

t

t− 1

t

t

t

Dynamics
Model
The
probabilityof staying th
The probability(7)
remaining one.
remaining one.

Experimental Result

Accuracy of Pedestrian Decision Detection• Typically, it takes around 5s for pedestrians to arrive at the crosswalk.
• The proposed model requires only 2s for estimating the pedestrian decision
with high reliability in an ideal environment.

• Large measurement errors deteriorate the system performance: slow
reaction, low accuracy.
• Wait decision is harder to detect than cross decision.