知能数理研究室
Computational Intelligence Lab

Semantic Technology andLevel 5 Autonomous Driving
Computational Intelligence (COIN) Laboratory

Toyota Technological InstituteNovember 28, 2018

Yutaka Sasaki

知能数理研究室
Computational Intelligence Lab

Toyota Technological Institute (TTI)• Established by Toyota Motor Corp. in 1981 as
one of its social contributions.
• Compact university admitting about 100
undergrads and about 45 postgrads each year.
• Located in Nagoya City.

(Disclaimer: This presentation does not express any technical standpoint of Toyota companies.)
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)2

知能数理研究室
Computational Intelligence Lab

Toyota Technological Institute (cont.)• One faculty with 3 courses:– mechanical system engineering– electronics & information engineering– material science and engineering.

• 46 faculty members closely related.
• Established Toyota Technological Institute at
Chicago (TTIC) on the University of the Chicago
campus in 2003.
– KITTI (vision data set)
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)3

知能数理研究室
Computational Intelligence Lab

TTI Research Center for Smart Vehicles• Established in 2010.
• Research goals:– Develop key technologies essential for realizing
Smart Vehicles,– Targeting cutting-edge technologies suitable foruniversity laboratories.
NB: “Smart Vehicle” means any kinds of intelligent
unmanned vehicles, including autonomous robots
and UAVs.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)4

知能数理研究室
Computational Intelligence Lab

Self-driving Technology

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)5

SAE levels of automated driving

知能数理研究室
Computational Intelligence Lab(https://www.sae.org/misc/pdfs/automated_driving.pdf)
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)(https://www.sae.org/misc/pdfs/automated_driving.pdf)

6知能数理研究室
Computational Intelligence Lab

Research areas of self-driving
SoftwareHardwareSensors:
Lidar, millimeter radar, sonar,
GPS, steering angle, velocity,
accelerometer,
camera (stereo, IR, FIR)
Body:
motor, steering, brake, accelerator,
body material, shape, ...

Object/position detection:lane, car, human, bicycle, ...
Scene understanding：object movement, intension,collision detection
Control:
steering, speed, path planning
High-level control:knowledge base, dialogue,
reasoning

InfrastructureIntelligent Transport System (ITS):
vehicle-to-vehicle communication,
driving safety support(digital traffic signal, traffic regulation info, road side sensor),
3D digital road map
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)7

知能数理研究室
Computational Intelligence Lab

Why we need driving knowledge bases?
• In the fully automated (level 5) driving,– cars should do (without a human driver):• Careful consideration of passengers/destinations• Decision making in some exceptional road conditions• Conversations, like with a taxi driver• In accidents, the car should know what to do• Avoidance of criminal usages

Almost equivalent to creating AI robots!
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)8

知能数理研究室
Computational Intelligence Lab

Comparison with autonomous robots• In a general sense, autonomous cars are a kind of
autonomous robots, sometimes called as robot cars.
• Autonomous cars are constrained by:– Strict regulations such as traffic laws.
– Underactuated manipulation: Inputs are steering andbrake/accelerator (gas) for 6 degrees of freedom.
– Highest priority in safe driving.

For knowledge-based driving, these constraints are notdisadvantages but can be advantages.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)9知能数理研究室
Computational Intelligence Lab

Three Laws of Robotics
1. A robot may not injure a human being or, throughinaction, allow a human being to come to harm.
2. A robot must obey the orders given it by humanbeings except where such orders would conflictwith the First Law.
3. A robot must protect its own existence as long as
such protection does not conflict with the First or
Second Laws.
Isaac Asimov, "Runaround“, I, Robot, 1950.

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)10

知能数理研究室
Computational Intelligence Lab

Three Laws of Autonomous Vehicles
1. An autonomous vehicle may not injure a humanbeing or, through inaction, allow a human being tocome to harm.
2. An autonomous vehicle must obey laws except
where such orders would conflict with the First Law.
3. An autonomous vehicle must obey the orders given
it by human beings except where such orders would
conflict with the First or Second Laws.
Yutaka Sasaki, presentation material, Smart Vehicles ResearchCenter Symposium 2016. (in Japanese)

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)11

Our approach to level 5 self-driving
Current self-driving
Sensor info

Control（driving knowledge isclosely tied to the control）

Introduce a KB
Sensor info

Research topics:
Control
Driving KB• How to model driving knowledge bases?
• How to acquire driving knowledge?
• How to manage the knowledge bases?
• How to achieve real time performance?
• How to evaluate KB quality?

Knowledge
acquisition Driving documents/textbooks
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)12

知能数理研究室
Computational Intelligence Lab

Advantages of driving knowledge bases• Enable:– to obey traffic laws while driving.
– to follow driving manners, e.g. hazard flashers.
– (shallow) reasoning like human drivers do.
– to make conversation, like with a taxi driver.
– to drive carefully for safety, like slowing down when seeingsome children on a side walk.
– to make decisions using common sense, e.g. inoperative
traffic signals.

• Can be updated/corrected independent of driving
control software.
– Tailoring a KB for each country/region.

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)13

知能数理研究室
Computational Intelligence Lab

Semantic Technology for self-driving

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)14

知能数理研究室
Computational Intelligence Lab

Ontology-based driving (at intersections)• Ralf Regele, Using Ontology-Based Traffic Models for More Efficient Decision Making of Autonomous Vehicles,
Fourth International Conference on Autonomic and Autonomous Systems, 2008.
• B. Hummel, Description logic for scene understanding at the example of urban road intersections, Fakultät für
Maschinenbau Universitat Karlsruhe (TH), 2009.
• Michael Hülsen et al., Traffic intersection situation description ontology for advanced driver assistance, IEEE
Intelligent Vehicles Symposium (IV), 2011.
• Edson Prestes et al., Core Ontology for Robotics and Automation, Workshop on Standardized Knowledge
Representation and Ontologies for Robotics and Automation, 2014.
• Armand et al., Ontology-based context awareness for driving assistance systems, IEEE Intelligent Vehicles
Symposium (IV), 2014.
• Brunner et al., 2017 IEEE International Conference on Vehicular Electronics and Safety (ICVES), Ontologies Usedin Robotics: A Survey with an Outlook for Automated Driving, 2017
• Lihua Zhao, Ryutaro Ichise, Seiichi Mita, Yutaka Sasaki, An Ontology-Based Intelligent Speed Adaptation System
for Autonomous Cars, The 4th Joint International Semantic Technology Conference (JIST2014), 2014
• Lihua Zhao, Ryutaro Ichise, Tatsuya Yoshikawa, Takeshi Naito, Toshiaki Kakinani, Yutaka Sasaki, Ontology-based
Decision Making on Uncontrolled Intersections and Narrow Roads, 2015 IEEE Intelligent Vehicles Symposium,
Seoul, 2015.
• Lihua Zhao, Ryutaro Ichise, Yutaka Sasaki, Zheng Liu, Tatsuya Yoshikawa, Fast Decision Making using Ontologybased Knowledge Base, 2016 IEEE Intelligent Vehicles Symposium, Gothenburg, Sweden, June 2016.
• Lihua Zhao, Ryutaro Ichise Zheng Liu, Seiichi Mita, and Yutaka Sasaki, Ontology-based Driving Decision Making:
A Feasibility Study at Uncontrolled Intersections, IEICE Transactions on Information and Systems, Vol. E100.D,
No. 7, pp. 1425-1439, 2017.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)15

知能数理研究室
Computational Intelligence Lab

Ontology-based Traffic Models[Regele, 2008]• CyberCars-2 Project (FP5)– “development of a fleet of fully autonomous cybercars which
can be used for passenger transport.”– introduction of road connection networks with conflict,neighboring and opposing relations.
conflict
oppositeneighboring• The model does not contain semantic information:– speed limitation, actual geometric information of
the roads.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)16

知能数理研究室
Computational Intelligence Lab

Logical representation of road intersections[Hummel, 2009]

Formal representations with Description Logic• Terminological box (TBox)
TwoWayRoad
OneWayRoad

Road
T

UTurnLane
OneWayLane
TwoWayLane

Lane

OneWayLaneSouth
OneWayLaneNorth

UTurn-Marking• Assertion box (ABox)
𝛼 = {𝑔𝑒𝑜𝑟𝑔𝑒𝑆𝑡: ∃ℎ𝑎𝑠𝑃𝑎𝑟𝑡. 𝑂𝑛𝑒𝑊𝑎𝑦𝐿𝑎𝑛𝑒𝑆𝑜𝑢𝑡ℎ⨆𝑇𝑤𝑜𝑊𝑎𝑦𝐿𝑎𝑛𝑒 }(George St. has some lane with driving direction southwards.)

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)17知能数理研究室
Computational Intelligence Lab

Traffic Intersection Situation Description Ontology[Hülsen et al. 2011]• Intersection ontologies contain concepts of car, crossing, road connection(lane and road), and sign at crossing (traffic light and traffic sign).

TBox:

Car
Crossing

Top

CrossingPlain
CrossingWTrafficLight
CrossingWTrafficSign

RoadConnection

Lane
Road

SignAtCrossing

TrafficLightAtCrossing
TrafficSignAtCrossing

LaneEntering
LaneExiting
LaneWtoWay
TLaCr_Green
TLaCr_Yellow
TLaCr_Red
TLaCr_Off
RightOfWaySign
YieldSign
StopSign

ABox:
hasRightOfWay

hasToYield

Rule: 𝐶𝑟𝑜𝑠𝑠𝑖𝑛𝑔 ? 𝑐𝑟 ⋀𝑛𝑒𝑔 𝐶𝑟𝑜𝑠𝑠𝑖𝑛𝑔𝑊𝑇𝑟𝑎𝑓𝑓𝑖𝑐𝐿𝑖𝑔ℎ𝑡 ? 𝑐𝑟 ⋀𝑛𝑒𝑔 𝐶𝑟𝑜𝑠𝑠𝑖𝑛𝑔𝑊𝑇𝑟𝑎𝑓𝑓𝑖𝑐𝑆𝑖𝑔𝑛 ? 𝑐𝑟

⟶ 𝐶𝑟𝑜𝑠𝑠𝑖𝑛𝑔𝑃𝑙𝑎𝑖𝑛(? 𝑐𝑟)• In a simple intersection case, it takes less than 0.5s for DL
reasoning. In a complex case, it takes 1.1-3.6s.
• Not tested with simulation or real-world data.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)18

知能数理研究室
Computational Intelligence Lab

Core Ontology for Robotics and Automation (CORA)[Prestes et al. 2014]• Drafted by IEEE Standard Association (IEEE SA)• Extended the Suggested Upper Merged
Ontology (SUMO).
SUMO

CORA

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)19知能数理研究室
Computational Intelligence Lab

Ontology-Based Context Awareness[Armand et al. 2014]• An ontology includes context concepts:– Mobile Entity (Pedestrian and Vehicle), Static Entity (Road Infrastructure
and Road Intersection), context parameters (isClose, isFollowing, andisToReach)• to understand the context information when it approaches roadintersections.
• 14 SWRL rules of spatio-temporal relationships
stop(example)vehicle(?v1)  StopIntersection(?stop1)  isToReach(?v1,?stop1)
 hasToStop(?v1,?stop1)(A vehicle that is about to reach a stop intersection has to stop at theintersection.)

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)20

TTI Ontology

知能数理研究室
Computational Intelligence Lab[Zhao et al. 2014]• Driving ontology development since 2009.
Car data

Control data Map data

Decision rules （SWRL)Car Ontology

Control Ontology

Map Ontology

Realtime reasoner (Protégé/pellet)

PreScan simulator

3

Experimental vehicle
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)21

TTI Core Ontology

知能数理研究室
Computational Intelligence Lab

https://www.toyota-ti.ac.jp/Lab/Denshi/COIN/Ontology/TTICore-0.03/(Licensed under CC Attribution-NonCommercial-ShareAlike 4.0 International)

TTI Core v0.03
Ontology– Car– Control
– Map

Data
– Car
– Control– Map

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)22知能数理研究室
Computational Intelligence Lab

TTI Core: Classes
Control class

Map class

Car class

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)23知能数理研究室
Computational Intelligence Lab

TTI Core: Object/data properties
Control property

Map property

Car property

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)24

知能数理研究室
Computational Intelligence Lab

Visualization of road connections

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)25

知能数理研究室
Computational Intelligence Lab

Instance
speed limit

GPS position

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)26

知能数理研究室
Computational Intelligence Lab

Simulator/real-world Experiments- Simulation:
Reasoning time: 242ms
Query time: 2-23ms- Real-world (Off line):
Reasoning: 117ms
Query time: 3-23ms
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)28

知能数理研究室
Computational Intelligence Lab

Ontology-based decision making
at uncontrolled intersections

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)29知能数理研究室
Computational Intelligence Lab

Targeted traffic scenario[Zhao et al. 2017]• Real world problems at uncontrolledintersections, i.e. no traffic lights.
• In Japan, there are a lot of narrow roads where
even human drives feel difficulty in driving.
Car B
Car C

Car A

2018/11/28

Not one-way(C)2018 Yutaka Sasaki, TTI (JIST2018)30

知能数理研究室

IssuesComputational Intelligence Lab• Car A has the lowest priority because it cannot
interfere traffics on the wide road bylaw.
• Car B has higher priority than Car A.
• Car C has the highest priority.
• The narrow road is difficult for two vehicles to run
freely; It is safer to stop on the left side and give way
to the other vehicle to pass by slowly.
• In this case, Car A should go out of the narrow road
Car B
Car C
before Car B for safe driving.
Level 5 autonomous cars should solve
this situation with some reasoning.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)

How?

Car A

31

Map data creation• Using OpenStreetMap data, we created a map data set for our
ontology.
• Lane connections are manually considered.
• Independent of map data used for the drive control.

TTI

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)32

Created 3D similation

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)33

Case analysis

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)34

知能数理研究室
Computational Intelligence Lab

SWRL Rules (out of 14 rules)

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)35

知能数理研究室
Computational Intelligence Lab

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)36

知能数理研究室
Computational Intelligence Lab

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)37

Sub-Ontology (SubKB)

2018/11/28

38

Real time performance
Acceleration by sub-ontology• Whole Ontology (Map)

Size of Knowledge Base
Whole Knowledge Base
407kb (or larger)
Sub-Knowledge Base
19 ~ 40kb

– Max: 965ms– Min: 305ms– Avg: 470ms• Sub Ontology (Sub-Map)– Max: 236ms– Min: 37ms
– Avg: 53ms

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)

39

Criticism?
• There are always some criticism to our approach.
– Old style– Knowledge-base is slow
– Poor handling of exceptional cases– Following a human way would lead to unsafety– Deep learning is more powerful and practical

It is time for the semantic technology community totackle the real-world problems related to self-driving.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)40

知能数理研究室
Computational Intelligence Lab

Our on-going studies:
Evaluations of Driving Ontologies

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)41

知能数理研究室
Computational Intelligence Lab

Quality evaluation of driving ontologies• What are the conditions by which authorities
approve level 5 cars to run on any publicroads?
 Open question!

• Idea– If an autonomous car passes its driver’s license
test, we can say the autonomous car has ability
and knowledge to drive.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)42知能数理研究室
Computational Intelligence Lab

Overview
Paper test
Q “The traffic sign includes the instruction sign andthe regulation sign.” (Yes/No)
Logical form
𝜆𝑥. 𝜆𝑦. 𝜆𝑧. TrafficSign(𝑥) ∧ InstuctionSign(𝑦) ∧ RegulationSign(𝑧) ∧subClassOf(𝑦, 𝑥) ∧ subClassOf(𝑧, 𝑥)

SPARQL query

query

Driving test
query

Ontology
ASK WHERE {InstructionSign rdfs:subClassOf TrafficSign.
RegulationSign rdfs:subClassOf TrafficSign .
}2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)

Ontologybased driving
43知能数理研究室
Computational Intelligence Lab

Questions beyond the scope• Commonsense question– “You can put boxes on a road” (Yes/No)• Calculation question– “3 adults and 3 children can ride the car with theriding seats of 5.” (Yes/No)• Questions with figures

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)44

知能数理研究室
Computational Intelligence Lab

Our on-going studies:
Driving Ontology Expansion

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)45

知能数理研究室
Computational Intelligence Lab

Adding knowledge to TTI core ontology
Converting a driving guideline to knowledge.
Guideline
You can drive the following types of vehicles …
License type
car type
LargeLicense…
MiddleLicense…
RegularLicense
RegularVehicle, Moped, …
……

Ontology
RegularLicense

2018/11/28

licenseToDrive

RegularVehicle

licenseToDrive

Moped(C)2018 Yutaka Sasaki, TTI (JIST2018)46

Knowledge acquisition

知能数理研究室
Computational Intelligence Lab• Manual semantic annotations of“Rules of Road”（交通教則）
 on going(If there is a traffic light at a railroad crossing, (you) can pass
the crossing according to the signal.)

Deep Learning models

Semi-automatic ontology expansion
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)47

知能数理研究室

Term extraction

Computational Intelligence Lab

・ Recurrent Neural Network (RNN) models
Term tags

Bi-directional RNN

Rules of road2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)48

知能数理研究室
Computational Intelligence Lab

Experimental settings
Textual data:
Annotated “Rules of Road”
・ #sentence : 2,150
・ #words : 43,395
・ #traffic terms: 8,822
・ #traffic term hierarchy: 8 levels deep
・ #distinct terms: : 780

2018/11/28

Learning Model:
RNN・ dimensionality : 100
・ batch size : 10
・ iterations: 20,000
・ type : LSTM (Long Short-Term Memory)(C)2018 Yutaka Sasaki, TTI (JIST2018)49

知能数理研究室
Computational Intelligence Lab

Term extraction results

Conditional Random Field

LSTMs

F-measure =2018/11/28

Bi-directional LSTMs

2∗𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛∗𝑅𝑒𝑐𝑎𝑙𝑙
𝑃𝑟𝑒𝑠𝑖𝑐𝑖𝑜𝑛+𝑅𝑒𝑐𝑎𝑙𝑙(C)2018 Yutaka Sasaki, TTI (JIST2018)50

(C)2018 Yutaka Sasaki, TTI (JIST2018)

知能数理研究室
51
Computational Intelligence Lab

Further uncontrolled cases• Our previous approachtargeted intersections.
• In a parking lot, we needa different approach.
 Less restricted- Our own simulator for parking scenario.
- Ontology rules should include parking manner, not laws.

2018/11/28

(C)2018 Yutaka Sasaki, TTI (JIST2018)

知能数理研究室
52
Computational Intelligence Lab

Current progress• Parking scenario: manner-based decisions
IN/OUT2018/11/28

知能数理研究室
Computational Intelligence Lab

Conclusions• The ontology-based driving approach leads tothe level 5 self-driving.
• A lot of open questions. Studies are ongoing.
• Reasoning time is an issue:– To accelerate, the dynamic creation of a subontology is effective.

• Semantic technology will be a core technology
for the level 5 self-driving.
2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)53知能数理研究室
Computational Intelligence Lab

Acknowledgements
Collaborators
Lihua Zhao, Ryutaro Ichise, Zheng Liu, Sei-ichi Mita, Makoto Miwa,
Tatsuya Yoshikawa, Takeshi Naito, Toshiaki Kakinani,and the past and current postgraduate students at the COIN Lab.
Naonori Nagano, Katana Suenaga, Junya Takayama, Zheming Lyu

2018/11/28(C)2018 Yutaka Sasaki, TTI (JIST2018)54